pageok
pageok
pageok
How did the pollsters do in predicting the popular vote?

The popular vote totals are not final. On Wednesday afternoon, the margin was 6.2% and rising. The pollsters who nailed it by reporting 6-7% included Rasmussen Reports, Pew Research, FOX News, Ipsos/McClatchy, CNN/Opinion Research, and the poll aggregator 538.

The worst performers were Gallup and Zogby with an 11% spread, followed by Battleground with a 3.5% average of its two models, and Marist, CBS News, and ABC News/Wash Post a 9% spread.

Interestingly, CNN and Fox News both outperformed the oldline networks -- ABC, CBS, and NBC.

Predicted Spread Poll

2-5 Battleground (combined models)

5 Diageo/Hotline

6 Rasmussen Reports

6 Pew Research

7 FOX News

7 Ipsos/McClatchy

7 CNN/Opinion Research

8 IBD/TIPP

8 NBC News/Wall St. Jrnl

9 Marist

9 CBS News

9 ABC News/Wash Post

11 Reuters/C-SPAN/Zogby

11 Gallup

Overall, the polls did pretty well this year. If the Obama ground game weren't so incredibly good, they might have done a tad worse. But only 2 polls were too low on Obama's margin, while 7 were too high.

In aggregating them, one would have done fairly well by throwing out the most Democratic poll among every 4 polls.

One thing struck me as well, though this opinion is probably tainted by hindsight bias: since 1960, it might well be that the campaign staff that ran the better campaign won each election, though I don't remember a few of them (eg, 1984) well enough to be certain of that. Of course, a great candidate makes a campaign look good, especially in retrospect.

UPDATE: Even though Nate Silver at 538 was not in the poll list I used for the post, I reached out to give him his props for being essentially as good as others who predicted a 6-7% spread before Tuesday's election.

Several commenters point to 538's Tuesday afternoon prediction of a 6.1% spread, issued long after the polls opened and turnout info was filtering back.

Judging his prediction by the same standard as the pollsters on the list above — subtracting his pre-election McCain prediction (46.1%) from his pre-election Obama prediction (52%), his predicted pre-election spread would be 5.9%, very slightly farther from the current spread than a couple of the pollsters above.

538 Mon PM & Tues AM

538 Mon PM & Tues AM prediction

Yet even that is complicated because his last pre-election presidential prediction post (on Monday evening) put the predicted spread at 6.0%, which simply means that the 52 to 46.1% spread actually rounded up to 6.0%:

With fewer than six hours until voting begins in Dixville Notch, New Hampshire, the national polling picture has cleared up considerably. Barack Obama is on the verge of a victory, perhaps a decisive victory, in the race for the White House.

The national polls have all consolidated into a range of roughly Obama +7. That is right about where our model sees the race as well, giving Obama a 6.8 point advantage in its composite of state and national polling. Our model notes, however, that candidates with large leads in the polls have had some tendency to underperform marginally on election day, and so projects an Obama win of 6.0 points tomorrow.

Silver's performance this year has been terrific, clearly establishing himself as the most reliable of the poll-based aggregators /predictors. He has an intuitive feel for numbers and knows when to tweak his models. In part because he appears to be the best out there, I hope that next time he releases his "final" predictions BEFORE the election.

egn (mail):
On the other hand, Nate Silver got it almost exactly.

As a commenter there put it, addressing Silver and his crew, "I hope you guys like lots of money."
11.5.2008 6:09pm
Uh_Clem (mail):

Let's not forget Research 2000, which also nailed it with a final tally of 51-46. Yeah, I know it was the poll sponsored by Kos, but credit where credit is due - it's clear that it valued accuracy more than whatever propaganda purposes it's detractors accused it of.
11.5.2008 6:36pm
Syd Henderson (mail):
You have to wonder if Nate Silver is secretly hoping Missouri and North Carolina (and the last Nebraska electoral vote) all go for McCain. If they do, he'll have the electoral vote exactly, and his popular vote for each candidate is off by only .1 percent. I'm very impressed.
11.5.2008 6:46pm
CC:
538 was amazingly accurate, and Nate's denigration of Zogby as a pollster seems to be supported by the final tally.
11.5.2008 7:12pm
James Lindgren (mail):
It depends on when you take Silver's prediction. If you take his prediction BEFORE the election (ie, Monday night), then he is beaten very slightly by several other polls. On Tuesday morning when the polls opened, Silver had a prediction of 46.1% for Bush and 52% for Obama, a 5.9% difference. (I saved the screen shot.) In the text of his last prediction post available on Tuesday morning (posted Monday afternoon), he was predicting 6.0%. That means that, though the estimates rounded to 46.1 and 52%, the difference was slightly closer to 6.0 than 5.9. So if you wanted to split hairs, you could say that he was as good as the other pollsters who picked 6% or you could say that he was slightly less accurate.

It wasn't until Tuesday AFTERNOON, after the polls were open and there was turnout info available, that he changed his prediction to 6.1%.

Personally, I would look to pre-election predictions, not pre-counting predictions. His pre-election prediction was great, but no better than several others. I hope that next time he makes his "FINAL" predictions BEFORE the election, not while most of the polls are already open and info is filtering out.

Jim Lindgren
11.5.2008 7:14pm
ronbailey (mail) (www):
I'm with egn - Nate Silver is looking like a genius about now.
11.5.2008 7:47pm
A. Zarkov (mail):
In terms of spread Pollvote did as well as 538 with a spread of 6.1.
11.5.2008 7:48pm
Jiffy:
I think that Jim Lindgren's analysis is a bit off, particularly when he compares Nate Silver to "other pollsters." Of course, Silver didn't do polling on his own, he used a model to aggregate results of polls conducted by others(including the ones in Lindgren's original post). The individual poll results appear to have been clustered around the actual outcome, and all the poll aggregators (Silver at 538, Pollster.com, and the Princeton Election Consortium) did pretty darn well predicting the actual outcome. I'm not sure you can say much about the inherent accuracy of a particular poll as a result of one election result. What you can say is that good aggregation produced pretty accurate predictions.
11.5.2008 8:14pm
Cold Warrior:
The "meta-pollsters" have done us a tremendous service. Really, all 3 mentioned by Jiffy (above) were spot-on.

That's what comes with a sound understanding of probability and statistics.

But it still amazes me that supposedly intelligent people just can't wrap their minds around these concepts.

Just Monday I saw Kathryn Jean Lopez of National Review (remember, I said "supposedly intelligent people") implying some kind of shenanigans because Zogby's one-day, ~400 voter survey on Sunday had shown McCain up by one point, but Zogby's 3-day tracker (~1200 voters, the way the survey is supposed to work) ultimately showed a comfortable Obama lead on Monday.
11.6.2008 12:44am
Jestak (mail):
Besides Nate Silver and 538.com, Mark Blumenthal's Pollster.com had a pretty good election. They were a little off in predicting the overall popular vote margin, but they did awfully well forecasting the Electoral College. I noted earlier today that every state they had as "strong" or "leaning" Democratic went to Obama, while every "strong" or "leaning" Republican state went to McCain. Of their "tossup" states, assuming that Obama's current lead in NC holds up, then Obama won every "tossup" where the Pollster.com averages had him ahead except Missouri, while McCain won every one where he was ahead in their averages saving Indiana. As Pollster.com's two "mistakes" both have 11 electoral votes, you could say that they hit the probable electoral tally on the nose.
11.6.2008 1:31am
anomie:
Faulting or discounting Silver's final prediction for not coming out until after the beginning of the election doesn't quite work. Sure, in terms of human interest in the election, we'd want to have predicted results before actual election results; but in terms of the predictive accuracy of the FiveThirtyEight model, we could determine that even now after most of the actual election results are known if we just feed Nate's model with only data that were collected before the election and are independent of any election results. I don't believe that Nate's model was set up to take election returns, exit poll results, or anything else collected/measured after the beginning of voting as input, nor do I believe that he tweaked the final numbers outside of the model. As long as that is true, his final numbers really are what his model predicts given the pre-election data.
11.6.2008 8:39am
MikeDT (mail):
I'm with Jestak, pollster.com is a great site. I started reading Mark Blumenthal after the exit poll fiasco in 2004. He was one of the voices of reason on the Democratic side that argued that the exit polls were wrong, not that the Republicans stole 200,000 votes in Ohio.

Also, this analysis on whether Nate Silver said 5.9 or 6.1, or if he said it Tuesday morning or afteroon seems really like nit picking. His model is based on the pre-election polls, not the turnout or exit polls, and as part of his modeling process he makes 10,000 simulations. So these numbers really correspond to a range of possibilities.

I think the one thing we learned during this cycle is that the poll aggregators are MUCH better at predicting the election than any single poll.
11.6.2008 2:12pm
FantasiaWHT:
I liked the Onion's take on exit polling

(graph chart)
Obama - 63%, McCain, 37%

(pie chart)
Obama 46%, McCain, 56%

(another graph chart)
Obama 12%, McCain, 6%, Thinks that polling is an utterly useless way to try and gauge American thought, 82%

The exact language was funnier, sorry, I'm paraphrasing
11.6.2008 4:56pm
Poll Watcher:
I like his work and respect Nate but as he himself would warn you: let's not mistake correlation for proof. If my cat pointed to the numbers 52 and 46, that wouldn't make him a better predictor than Zogby? Also, Nate and Pollster are analysts of polls, not pollsters.

Note: I have followed 538 and pollster obsessively throughout the campaign and think they are the best, most intelligent, empirical sites.
11.6.2008 5:51pm