Research 2000 Called into Question     Permalink

The Daily Kos Website has been commissioning pollster Research 2000 to run polls for it. Now the founder, Markos Moulitsas, has repudiated Research 2000 and told everyone to delete all its tracking polls from their data bases due to a statistical analysis that suggests they have been curbstoning (making up the data). Kos, as he is known, didn't say anything about Senate polls, but if Research 2000 is unreliable with tracking polls, it is probably unreliable about everything.

This leaves us with a big problem. While we are not going to start daily posts of the polling data until September, when the primaries are largely over and people begin paying attention to the election, at that point we will be faced with the problem that very few pollsters are left and of those, some have other issues. SurveyUSA and PPP are probably pretty solid but the largest "nonpartisan" pollster is Rasmussen, who now works for Fox News. While Fox is theoretically a news organization, it has a definite slant on the world, that say, ABC News does not have. There is no evidence that Rasmussen makes up numbers, but there are other issues here. In particular, most pollsters, including Rasmussen, have a model of the electorate and normalize their polls to it. Very briefly, suppose Rasmussen believes that the set of likely voters (which is very different from the set of registered voters and very, very different from the set of people over 18) is 40% Republican, 30% Democratic, and 30% independent. If an actual poll turns up 200 Republicans, 200 Democrats, and 200 independents, he will weight each Republican respondent by 40/33, each Democratic respondent by 30/33 and each independent by 30/33 to compensate for the bad mix in his small sample.

In principle, such weighting is not only legitimate, but a good idea. But if the model has too many of one party and too few of the other, it will affect the results of course since the people in the underrepresented groups will count more. If the pollsters released the raw data, then others could undo this effect, but most pollsters do not release the raw data. In the case of Research 2000, it wouldn't even release it to Kos, its own client, which is very fishy. Weighting is also done for gender, age, and other demographic characteristics. Again, doing this is actually a good thing, but a lot depends on how good the model is and whether that has been fudged at all.

One might think that it is easy to tell which pollsters are accurate and which are not by comparing their polls to the final election results, but such is not the case. For the final polls a few days before the election, this is true, but for general election polls in June (which are probably not worth much anyway), there is no way to check. It is not inconceivable that a polling organization with an ulterior motive could issue biased polls most of the year (to encourage supporters of its favored party and discourage the opposition) and then as the election approached, gradually remove the bias and try to be as accurate as possible, to acquire a good reputation for next time.

Research 2000 has hired a law firm to try to intimidate the media (including the blogs) into not talking about this issue. Nate Silver of 538.com has already gotten his. I'll check my mailbox later today expectantly.

Charles Franklin, a professor of political science at the University of Wisconsin at Madison, is worried that scandals like this may hurt the polling industry as a whole. His proposal is that everyone commissioning a poll should write into the contract that the pollster must deposit the raw data in the Roper Archive at the University of Connecticut, which collects them. This would allow third parties to analyze the raw data, do statistical checks on them, like using Benford's Law, and renormalize them using different models of the electorate.

We'll keep an eye out for further developments in the Research 2000 affair. For the time being, they have been marked as suspect in the internal data base.

If you like this Website, tell your friends. You can also share by clicking this button