Polls, again

Before I managed to lose about a year's worth of posts during my last host change, there was a long rant on this site about why people who make their living the way I used to generally hate the media's handling of public opinion polls. And with the election just around the corner, I thought I'd put a (hopefully) more measured one up in its place.

Basically, it boils down to this: Even when news organizations get the big stuff right (a good sample, well formed questions, etc.) -- which the national ones, unlike their local brethren, usually do, because they hire reputable firms to conduct the actual polling -- most reporters don't explain the margin of error issues involved well enough for the average viewer or reader to get any real sense of what's going on with the numbers.

Here are three things to keep in mind the next time you see a news report about who's up and who's down, starting with an obvious one that a surprising number of people still don't understand, because the press doesn't say it often enough.

1) The margin of error in a two-person race applies to both candidates. So, in a typical poll of a thousand people, which produces a plus or minus 3 MoE, the spread is six points. If Wes Clark, for example, is leading George Bush 52 to 48, that 4 point lead is within the MoE.

2) When they talk about the crosstabs, the MoE rises, sometimes dramatically. For example, if they break out the results for, say, African American women, they are discussing a very small sample -- roughly 50 people in a poll of a thousand. In order to find the MoE for that tab, you have to plug the number of people surveyed into a table like this one:

Respondents	MoE (at 50%)
1000	3
900	3
800	3
700	4
600	4
500	4
400	5
300	6
200	7
100	10
50	14

In the example above (African American women), the MoE would be 14 points, applied to both candidates. That's a pretty damned big fact for a reporter to just leave out of a story, which they all too often do.

3) Large fields are different. So far, everything I've talked about has assumed a perfect polling world: two candidates, no undecideds. But it doesn't usually work that way, particularly not during the primary season. When you compare two candidates in a large primary field (which is really what most of us are interested in), the MoE is often off-the-charts -- and again the press almost never points that out. (To find a rough MoE in a situation like that, some people recommend this quick back of the envelope calculation: Add the two candidates' percentages, multiply that by the number of people surveyed -- which is often already fairly small, since the number of registered voters in either party is going to be less that half of a sample representing "all Americans" -- and plug the resulting figure into the table. You'll find that a spread of at least 15 points isn't unusual.)

There are all sorts of other problematic technical issues to be raised about the press and the way they use polling data, but it's Sunday morning and I suspect I've bored everybody enough already. So I'll just thank you for listening, and let you know that my next survey research rant, sometime this week, will be on why those things Frank Luntz insists on calling focus groups on MSNBC's Hardball are actually anything but.