On Methods
May. 3rd, 2009 07:50 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
pecuniumWe are a nation which has come to live by polls. Even those of us who discount (or even disregard them) have to live with the effects they have; because our elected representatives have come to rely on them, and our press corps to fetishise them (want to track the rise and fall of candidates for office, follow the polls. Want to predict when the guy who is willing to take the low-road will do it... look to the polls).
Polls, however, are tricky things. They can be skewed. Some are obvious... push polls aren't actually meant to measure public opinion, but shape it. The classic is the one George W. Bush used to torpedo John McCain in S. Carolina (I paraphrase: How would knowing John McCain had fathered a black love child affect your opinion of him: would you be more likely to vote for him, less likely to vote for him, or would there be no change in your chance of voting for him).)
Even when a poll isn't being driven by such intentions, how the questions are phrased can be very affective on the results. Take the recent Pew Poll on torture. We don't have all the questions (they have released some, but not all). One of them, and the answers to it, has been used to say, "religious people support torture," as well as to say the country is about evenly split on the subject.
I think neither conclusion can be fairly drawn from the results. Why? Because the question is full of some question begging assumptions, as well as building some answer framing assumptions in the mind of the respondent.
I say this because asking questions is what I did for a living. I did it as a reporter, and then I did it as an interrogator. As an editor I trained people to to a thorough job of getting facts to fill in the details, and the narrative of a story. I sent them back to get details they'd missed.
As an interrogator I was taught to do the same thing, in a far more organised, and orderly fashion. As an interrogation instructor I taught that same skill to people.
It's the last of those which did me the most good in learning to build questions. One teaches people to ask questions by answering them. Whatever the actual subject of the question was, I answered. Some of it seemed petty (esp. when I was the student).
"Can you spell your name?"
"Yes."
"Ok, do it."
"Do what"
"Spell your name."
"Y O U R N A M E" (we got very good at responsive spelling. Whatever came after, "spell" was going to be spelled out. Sometimes the students would use this to get back at us.... ever had someone ask you to spell syllogistic systems? That was the sort of thing some students did. It was irksome, but all in all not a bad thing. It showed both spirit, a sense of play, and an understanding of the systems they were learning to use, but I digress).
I did that for somewhere between 300-400 students, in the course of 14 years, as well as time teaching maintenance classes to my, and other, units. I got very good at building good questions, and at spotting the holes in bad ones.
Which is why I don't like this one in this poll (both are .pdf pages).
Do you think the use of torture against suspected terrorists in order to gain important information can
often be justified,
sometimes be justified,
rarely be justified,
or never be justified?
Whoo-Boy!.... how many ways can that be screwed up?
First, there are some inserted bits of information which shape the question. "Suspected terrorists" limits the group. It also calls into mind the prejudices of the respondent. I happen to think the most likely perpetrators of terrorism in the US are white males, between the ages of 25-40, who are right-wing in their politics, and a trifle fundamentalist in their religious views. This is based on the actual arrests of people planning terrorism in the US.
Then again, I think bomb threats to abortion clinics (and the shooting of providers), is terrorism, much of which flies below the radar of the terrorist threat in the press.
Then we have, "to gain important information". That trips the "ticking bomb" myth, the idea that the, "suspect" has this important information (which presupposes that this is only used on "real terrorists", since someone who isn't a terrorist can't give up "important information".
Which moves a whole lot of goalposts. If the person has important information they aren't suspects. Only people with, "important information" are included in the set they are being asked to think about torturing. If that's the case innocent people (such as the respondent, her family and friends) aren't going to be tortured.
If what we get is, "important information" then there is something we need to get. The implication is (again buried) that we know the person has this information, and only a person dedicated to hurting the US would refuse to give that information up.
All of which moves the odds of the question getting a positive response up.
Which makes the poll less useful than it might otherwise be.
The response choices don't help. The hypothetical (ticking bomb, important information, can be reliably extracted with torture) is one which is likely to incline someone to waffling. This is the whole point of the "ticking bomb" debating trick. It's the camel's nose.
"If lots of people are going to die, and doing this one slightly bad thing; out of pure necessity will prevent it, would you never do it, to save the innocent?" [which has in it the hidden statement that a guilty person is strapped to the table in front of you].
Of course you would... maybe you have lots of restrictions. You have to have proof the victim knows. It has to be lots of people; maybe it has to be a family member. All of those little things add up to increase the odds of the respondent copping to saying, "rarely justified," when in fact the caveats attached to that answer are so complex that the practical answer is never; but the honest respondent says to himself, "I might be willing to do it." (see Steven Barnes at Dar Kush who admits that, given the right set of incentives he'd be willing to cross that line. I disagree with a couple of his arguments (and you can see those in comments), but he's honest enough to look at it from a personal POV, as well as a structural one, and share the answers to both).
So, all things being equal, that poll is a piece of crap.
Polls, however, are tricky things. They can be skewed. Some are obvious... push polls aren't actually meant to measure public opinion, but shape it. The classic is the one George W. Bush used to torpedo John McCain in S. Carolina (I paraphrase: How would knowing John McCain had fathered a black love child affect your opinion of him: would you be more likely to vote for him, less likely to vote for him, or would there be no change in your chance of voting for him).)
Even when a poll isn't being driven by such intentions, how the questions are phrased can be very affective on the results. Take the recent Pew Poll on torture. We don't have all the questions (they have released some, but not all). One of them, and the answers to it, has been used to say, "religious people support torture," as well as to say the country is about evenly split on the subject.
I think neither conclusion can be fairly drawn from the results. Why? Because the question is full of some question begging assumptions, as well as building some answer framing assumptions in the mind of the respondent.
I say this because asking questions is what I did for a living. I did it as a reporter, and then I did it as an interrogator. As an editor I trained people to to a thorough job of getting facts to fill in the details, and the narrative of a story. I sent them back to get details they'd missed.
As an interrogator I was taught to do the same thing, in a far more organised, and orderly fashion. As an interrogation instructor I taught that same skill to people.
It's the last of those which did me the most good in learning to build questions. One teaches people to ask questions by answering them. Whatever the actual subject of the question was, I answered. Some of it seemed petty (esp. when I was the student).
"Can you spell your name?"
"Yes."
"Ok, do it."
"Do what"
"Spell your name."
"Y O U R N A M E" (we got very good at responsive spelling. Whatever came after, "spell" was going to be spelled out. Sometimes the students would use this to get back at us.... ever had someone ask you to spell syllogistic systems? That was the sort of thing some students did. It was irksome, but all in all not a bad thing. It showed both spirit, a sense of play, and an understanding of the systems they were learning to use, but I digress).
I did that for somewhere between 300-400 students, in the course of 14 years, as well as time teaching maintenance classes to my, and other, units. I got very good at building good questions, and at spotting the holes in bad ones.
Which is why I don't like this one in this poll (both are .pdf pages).
Do you think the use of torture against suspected terrorists in order to gain important information can
often be justified,
sometimes be justified,
rarely be justified,
or never be justified?
Whoo-Boy!.... how many ways can that be screwed up?
First, there are some inserted bits of information which shape the question. "Suspected terrorists" limits the group. It also calls into mind the prejudices of the respondent. I happen to think the most likely perpetrators of terrorism in the US are white males, between the ages of 25-40, who are right-wing in their politics, and a trifle fundamentalist in their religious views. This is based on the actual arrests of people planning terrorism in the US.
Then again, I think bomb threats to abortion clinics (and the shooting of providers), is terrorism, much of which flies below the radar of the terrorist threat in the press.
Then we have, "to gain important information". That trips the "ticking bomb" myth, the idea that the, "suspect" has this important information (which presupposes that this is only used on "real terrorists", since someone who isn't a terrorist can't give up "important information".
Which moves a whole lot of goalposts. If the person has important information they aren't suspects. Only people with, "important information" are included in the set they are being asked to think about torturing. If that's the case innocent people (such as the respondent, her family and friends) aren't going to be tortured.
If what we get is, "important information" then there is something we need to get. The implication is (again buried) that we know the person has this information, and only a person dedicated to hurting the US would refuse to give that information up.
All of which moves the odds of the question getting a positive response up.
Which makes the poll less useful than it might otherwise be.
The response choices don't help. The hypothetical (ticking bomb, important information, can be reliably extracted with torture) is one which is likely to incline someone to waffling. This is the whole point of the "ticking bomb" debating trick. It's the camel's nose.
"If lots of people are going to die, and doing this one slightly bad thing; out of pure necessity will prevent it, would you never do it, to save the innocent?" [which has in it the hidden statement that a guilty person is strapped to the table in front of you].
Of course you would... maybe you have lots of restrictions. You have to have proof the victim knows. It has to be lots of people; maybe it has to be a family member. All of those little things add up to increase the odds of the respondent copping to saying, "rarely justified," when in fact the caveats attached to that answer are so complex that the practical answer is never; but the honest respondent says to himself, "I might be willing to do it." (see Steven Barnes at Dar Kush who admits that, given the right set of incentives he'd be willing to cross that line. I disagree with a couple of his arguments (and you can see those in comments), but he's honest enough to look at it from a personal POV, as well as a structural one, and share the answers to both).
So, all things being equal, that poll is a piece of crap.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2009-05-04 03:52 am (UTC)no subject
Date: 2009-05-04 01:03 pm (UTC)I removed "terrorist" and "important", but left "suspect" because presumably the person on the table has been detained for a reason.
no subject
Date: 2009-05-04 03:41 pm (UTC)I suspect, sadly, the real problem is at least as many people who said yes, think torture works. That it gets information as reliably, and faster, than actual interrogation (I lost track of how many times I've heard, "A terrorist isn't just going to talk to us, we have to make him").
So long as that is out there, any question which includes it as an option has that problem.
"If a relative was accused of terrorism, would you accept the need to torture them:
Always
Sometimes
Rarely
Never"
Is a slightly better question, because it makes them ponder several things, which more sterile questions don't.