I just read in my local news paper that the US’s war on terror has failed to weaken al Qaeda.

Interesting, I thought, and read on, just to find out that someone had made a survey of some 20 000 people in 23 countries around the world, asking them if the war on terror had weakened al Qaeda.

I managed to track down the survey to a company called Globescan. The actual survey can be found here. This survey seems rather scientific. They have a methodology page, where an account for how the data was collected for each of the involved countries, can be found in a table.

Interestingly enough, in more than half of the countries included interviews have been conducted face-to-face, suggesting the sample was only semi random, if even that. Most likely the interviewers asked people passing by in a market place, supermarket, or some other place, and since a random sample of the country’s population is less than likely to pass them in the hours or days they conducted the interviews, the sample for that country was not random. (Which in turn skews the result; imagine if only moms between 20 and 30 were asked this question, as opposed to everyone in the country, and then we’re not even discussing the difference between face-to-face and telephone interviews when it comes to interviewer influence errors.)

It is also possible to find the actual questions asked on the methodology page. One that stands out is the question “Overall would you say your feelings about al Qaeda are positive, negative or mixed?”

Imagine that question being posed to someone on a street in Egypt, Indonesia, or Pakistan (face-to-face interviews) and then imagine if the interviewee would be honest or not with respect to the far from negligible chance of being overheard…

The actual question probably quoted by my local news paper (and many like it, I’ve been able to gather) was probably “Do you think what US leaders refer to as the ‘war on terror’ has made al Qaeda stronger, weaker, or has had no effect either way?”

This question was not posed to prime ministers of the countries included in the survey, neither were it posed to members of the intelligence community or the military (at least not solely, as far as the methodology page of the survey reveals), instead the you’s and me’s of these countries were asked about the progress of the war on terror.

The question in itself is a potent and interesting one. The method to find the answer quite frankly sucks! The first question in the survey, if the interviewee was positive or not towards al Qaeda is interesting given it had been posed in a situation where anonymity were assured and spectator influence limited. Other questions trying to determine the interviewee’s personal experience and relation to the war on terror would have been interesting, but the attempt to try to make average Joe determine if the war was effective or not is, if I’m really nice, plain dumb.

However, as a survey this one is scientific and within parameters (even though I would have wanted more information on methodology). Unfortunately this kind of research (and others like it, there are other gems dealing with the war in Iraq in basically the same way, to mention one), dealing with these kinds of questions, does not exist in a vacuum.

Journalists will read these surveys and make headlines from them. You and I will read these articles and draw conclusions. We will bring them with us when we vote the next time, when we plan our vacations, and go about our lives (okay not always but now and then we will make decisions remembering “that article” about the war in Iraq or the war on terror), and most of the time we won’t have the background or know the methodology, and in light of that I believe these kinds of questions should not be asked in this way.

Globescan calls their activity reputation research, I think of it as institutionalized hearsay. With the kind help of these guys we can have what Bob next door has been saying for years served to us as a Survey, based on more than X-thousand interviews in so-and-so many countries. What a glorious example of statistics once more abused as a tool for spreading FUD.

To all of you who read about this survey and drew your own conclusions from it, I can only say this: No, we still don’t know what’s really going on!