Rasmussen and Bias

After the 2010 elections, the New York Times statistics wizard, Nate Silver, analyzed the polls produced by various polling organizations, including Rasmussen Reports, which is the house pollster for Fox News. Silver's analysis covered only polls taken during the final three weeks of the campaign and compared them to the actual election results. For polls taken much earlier, say in June, no one knows what the true sentiment of the electorate was, so there is no way to tell if the polls were accurate or not. Also, any pollster deliberately falsifying the results for partisan advantage would be advised to reduce the bias as the election neared. After all, no one can tell if a June poll is accurate but everyone can tell if a poll released the day before the election is accurate.

Silver analyzed 105 polls released by Rasmussen Reports and its subsidiary, Pulse Opinion Research, for Senate and gubernatorial races in numerous states across the country. The bottom line is that on average, Rasmussen's polls were off by 5.8% with a bias of 3.9% in favor of the Republican candidates.

There is much to criticize about Rasmussen's methods. All polls are conducted within a 4-hour window, the person who answers the phone (even a child) is sampled, phones that are not answered are not called back, and much more. All of Rasmussen's polls are done by computer; live interviewers are never used. However, other firms that do robopolling such as SurveyUSA and PPP get much more accurate results with no bias, so the problem is not the robopolling per se.

Just to look at one methodological issue, if no one answers the phone, Rasmussen picks a different random phone number instead of calling back two, three, four or more times as other pollsters do. Why does this matter? Because 20-somethings (who skew Democratic) are often out, whereas 60-somethings (who skew Republican) are often in. By not being persistent in finally getting through to a randomly chosen phone number, the sample is inherently biased towards Republicans because they are easier to reach. This may not have been intentional but it is understandable if you want to finish your survey in 4 hours. Nevertheless, cutting corners in the name of speed and cost don't improve accuracy.

Unlike companies like Strategic Vision, which most likely made up the data (but not very well) and also Research 2000, which probably did as well, no one is suggesting that Rasmussen is making up numbers without actually doing polling. There are many reports of people called by Rasmussen. The problem with Rasmussen is most likely its model of the electorate. Very briefly, if a pollster believes that in a certain state, say, 40% of the voters are Republicans and the actual survey just happens to turn up 35% Republicans, each Republican interviewed will be given a weight of 40/35 to correct for the undersampling of Republicans. All pollsters do this to correct for under- or oversampling by party, gender, age, race, income, and other factors. This is not only legitimate, but necessary with the small samples all the pollsters use. The issue here is whether Rasmussen's model of the electorate has more Republicans in it than in reality there are (not to mention whether this is accidental or deliberate).

You can read more about Silver's analysis here and here.

The conclusion is that some people do not believe in Rasmussen's polls any more. For these people, we have produced this page, which is generated exactly the same way as the main page and the Senate page, except that first all the Rasmussen polls are temporarily removed from the database. To see if this page is more accurate than the main page and Senate page, please check back on Nov. 7, 2012.

Note: Only this page has Rasmussen-free data. All the other tables and graphs on the site include all polls, including Rasmussen's.