pageok
pageok
pageok
Some Reasons Why Polls May Be Inaccurate

Back in the early 1990s, Gary Mauser (Simon Fraser University, British Columbia) and I wrote an article, originally published in Political Communication & Persuasion, explaining why polls are sometimes inaccurate as measures of public opinion. The article is titled 'Sorry, Wrong Number': Why Media Polls on Gun Control are So Often Unreliable, and although the focus is on polls about gun control, the article observes some general problems with polling. If Senator Obama on election day significantly underperforms, or overperforms, what the polls predict, there will be many possible causes, other than the "Bradley Effect" or the "Reverse Bradley Effect." There are many factors, other than race-consciousness of the interviewees, which may cause a gap between opinion polls and actual votes.

_quodlibet_:
Hmmm... seems to be font issues with that PDF. It doesn't display right in Adobe Acroread 8 under Linux. (Opens fine with xpdf, though.)
10.14.2008 9:22pm
commontheme (mail):
I guess I can understand why someone whose website contains the following:

Full Picture of Obama Emerging. Aug. 27. What the media hasn't told you about the socialist, racialist, Barack Obama Sr. Plus bogus claim from Time that older Jewish. voters who don't back Obama must be racist.

might be feverishly scratching around for some little straw of hope.
10.14.2008 9:24pm
PC:
Prof. Kopel, given the maturity of presidential polls, do you think they are subject to the same errors that issue polls are? Presidential polls also occur at the state and national level. They are conducted by many different groups. Do you think such a large number of polls could suffer from the same errors presented in your paper?
10.14.2008 9:32pm
Ron Hardin (mail) (www):
Another reason for trouble is that public opinion does not exist. Measuring does not cure this defect!

What you have from an election is a decision, not an opinion.

So apart from measuring what does not exist, it's measuring the wrong units.

Public opinion is a ephermeral as the news for a good reason; it takes newspapers to create its illusion.
10.14.2008 9:33pm
Syd Henderson (mail):

What the media hasn't told you about the socialist, racialist, Barack Obama Sr.



We're supposed to be concerned about Obama's father now?
10.14.2008 9:34pm
Donny:
This is true, as anyone with a passing familiarity with statistics knows. However, three things make it very unlikely that the consensus of the polls on presidential outcomes is wrong.

First, historically, the consensus on Presidential elections is almost always correct. Only 2 of the last 18 elections haven't been accurately called. If there were systematic errors, we wouldn't see that kind of historical accuracy. (Consider also the accuracy of the primary polls.)

Second, there are far more pollsters doing election polling than there are on other issues. Different pollsters use different methods (some call cell phone users, some are internet-only, they use different likely voter models, etc.). So if they all agree, they eliminate a lot of the possible sources of error.

Third, even if every poll is systematically missing some cleavage of voters, internal change in the poll is still significant. If Gallup misses youth voters, then their topline numbers will be inaccurate, but change in those numbers toward Obama still mean he's more likely to win the election.
10.14.2008 9:35pm
Paul Milligan (mail):
However, one thing is certain - if Obama loses, the left will scream 'THEFT !! THEFT !!' and point to polls ( pre and post voting ) as 'proof'.

Of course, given the Electoral system, national aggregates of polls are meaningless. A candidate could easily win, let's say Cali and NY by 90 %, and it makes no difference as compared to 51 %, in the Electroal counting.
10.14.2008 9:47pm
Donny:
No, national aggregates are not meaningless. They are highly correlated with the electoral college outcome. I think the term you meant is "dispositive."
10.14.2008 9:51pm
JosephSlater (mail):
The folks at www.fivethirtyeight.com have been arguing strongly and I believe convincingly that there is not likely to be a Bradley effect in this election.

Among other things, do we really think that in 2008, a significant number of people are telling annonymous pollsters on the phone -- or in some cases, automated phone services -- that they are voting for Obama when they are really going to vote for McCain, because they worry that the annonymous pollster or machine will think they are racist?

Plus, Obama is opening up a pretty wide lead. He's up by two touchdowns (53-39) in the newest CBS poll. And the state polling news is pretty bleak for McCain two.

Denial, anger, bargaining . . . I know they all come before acceptance, although I can't remember the order.
10.14.2008 9:52pm
Oren:
Paul, you can look at an effective way of evaluating the aggregate electoral college vote over the 51 discrete contest by going to fivethirtyeight. To boil the process down the most essential elements, you go to each state and construct a best estimate of the mean and variance of the polls. You then do 10,000 simulated elections in which the result of each state is drawn from a Gaussian distribution with that mean and variance. The result is the most statistically accurate way of aggregating the polling information as it pertains to the final electoral vote tally.

Of course, the system is only as good as the inputs -- 528 has the added advantage that all the information, including the polls and their weights, are provided in full detail.

Short answer -- there's a perfectly reasonably way to make reasonable national aggregates of data.
10.14.2008 9:53pm
Oren:
Better link. (apparently, if you don't put in the protocol string, it appends to the local URL -- how bizarre).
10.14.2008 9:54pm
Oren:
Incidentally, one of the neat things you can do with 538's methodology is make an estimate of all sorts of events. For instance, "recount" is defined as one or more decisive state with a margin <= 0.5%. A few weeks ago, the probability of a recount was almost 5% (alas, it was a closer election then).

Another interesting thing is the probability of an electoral college/popular vote split. Again, the odds are way down now that the election looks to be a blowout but at one point it was also ~5%.
10.14.2008 9:58pm
Anon21:
I've skimmed the article (I have other work to do, so I don't have time to read it all at the moment). Applying your critique to current general election polls seems incorrect to me for several reasons.

1) You explicitly limit your study to media polls, which you go on to demonstrate are often cranked out on compressed schedules, with attendant cut corners in terms of proper polling methodology. Although there are some media polls being conducted this cycle, the majority of national polling data (not by source, but by volume—e.g., the Rasmussen tracker and the Gallup tracker are both producing more data and interviewing more respondents than the CBS/NYT, CNN, Fox News, and ABC/WaPo once-every-two-week one-offs put together), and the vast majority of state polling data is being produced by either professional polling firms or educational/public interest groups that specialize in opinion polling. Especially in the case of professional pollsters, such as Rasmussen Reports and SurveyUSA, their reputation as accurate and reliable is the largest determinant of their success, and thus they have strong incentives to employ rigorous scientific polling methodology.

2) Regularly-released poll series run by professional or independent pollsters generally have a good deal of data from cross-tabulations available, either to the public or to subscribers. This allows polling experts in the media and the blogosphere (unfortunately, it's become painfully accurate this cycle that the smartest voices on this issue are being posted online—whether major media news orgs just don't hire polling experts or don't put them on TV, cable news coverage of polling suffers from a number of flaws) to point out sample problems which might undermine the accuracy of the poll's central findings. This alleviates, to some extent, the difficulty you describe in your answer with obtaining an accurate sample for the question you're trying to survey. The major caveat here is that the question of the composition of this year's electorate is still very much up in the air; I would say the general consensus is that it's likely to be younger, browner, and just generally more Democratic, but the degree to which we can count on that is certainly contested between polling experts. Nonetheless, professionally-run polls do have this major advantage over media polls: the latter are generally very jealous of their cross-tabs, although disclosure has actually been up over this election cycle and the last one (which obviously significantly postdates your research paper).

3) One of the major concerns you treat in your paper, question wording, is really not an issue at all with regard to Presidential trial heats. Question order could possibly be affecting results slightly, but I believe most pollsters ask the "top-line" question at the top of the call, or second after demographic questions, which is probably the best you can do in terms of not influencing respondents one way or the other. The somewhat related issue of pushing leaners and how the prompts for that push are worded is a thorny issue in and of itself, but once you get past the first debate voter preferences tend to stabilize quickly. "Leaned" preferences probably reflect the most likely vote choice for respondents who end up voting at all at this point in the election season.

4) You mentioned that your paper goes beyond the Bradley Effect and related phenomena, but in case any other readers are unaware, I would note that there is substantial evidence* that the Bradley Effect disappeared in the mid-1990s. Both of those links incorporate data from Obama's own performance during the primaries to show that there is simply no evidence that white voters are saying one thing to pollsters and doing another in the privacy of the voting booth. This also speaks to your point, made in the paper, that the perceived characteristics of the interviewer can influence responses; and it's worth noting that one pollster generating a lot of state polling this cycle, SurveyUSA, uses only robo-calling, which essentially eliminates that effect entirely.

5) It's true that polls are generally worse measures of public opinion than the officially acknowledged margin of error, but polls can show trends, and when they converge they are much more often right than wrong. Some pollsters are also measurably better than others, and it's possible to take that into account when weighing the various polls one sees generated each day. The superstar of adjusting polls for the broader universe of "X-factors" at play in politics is, of course, 538, and I would encourage anyone who hasn't visited the site yet to take a look. Nate Silver is fairly aboveboard with his methodology, and he does have a strong background in statistical analysis.

In conclusion, I think your concerns about public polling and its accuracy are overstated as applied to this election. While it is true that polls are not exact measures of public opinion, the general "consensus" among publicly-released polls of the past two weeks that John McCain trails by about 8 points nationally and by various margins in the key battleground states is most likely indicative of the fact that John McCain is in fact trailing by about 8 points nationally and by various margins in the key battleground states. Looking ahead, it seems overwhelmingly likely, based on the information available, that Obama will win the election unless McCain turns in a very dominating performance at tomorrow night's debate, or if some major non-campaign-related news event alters the race's dynamic during these last three weeks. Historical trends show it to be highly unlikely that McCain can make up this much ground, this late, through standard campaign tactics.

*I keep getting the comment rejected with an error when I attempt to link to the paper using standard HTML tags, so the link is: http://people.iq.harvard.edu/~dhopkins/wilder13.pdf
10.14.2008 10:01pm
PC:
Paul Milligan, plenty of people are also tracking state polls. State polls are still trending towards Obama. If the election was held today and McCain won it would be a Black Swan event.
10.14.2008 10:02pm
PC:
Oren, two other interesting poll of polls are RealClearPolitcs.com and Pollster.com. RCP is right leaning and has an 8 point Obama lead. Pollster is an aggregate with no weighting. I'm curious to see how accurate 538 is. They are some serious stats geeks.
10.14.2008 10:06pm
Oren:
PC, I do visit RCP and Pollster for some variety but I prefer 538 because neither of those two do (IMNSHO) a satisfactory job aggregating the information into a final tally.

As far as 538's "accuracy", are you refering to the accuracy of their aggregation/weighting/de-biasing techniques or are you referring to their predictive power? The former is fair game but it seems unfair to judge them on the latter since they can only use the poll data that they are given.
10.14.2008 10:19pm
byomtov (mail):
WRT the Bradley effect, I seem to recall reading that results in the 2006 TN Senate race, where Corker beat Ford, very closely matched the latest pre-election polling results.
10.14.2008 10:25pm
MarkField (mail):

We're supposed to be concerned about Obama's father now?


He's advising Barack by ouija board.
10.14.2008 10:37pm
astrangerwithcandy (mail):

I've skimmed the article (I have other work to do, so I don't have time to read it all at the moment). Applying your critique to current general election polls seems incorrect to me for several reasons.


based on the length of the screed that followed, i find this assertion dubious :)
10.14.2008 10:42pm
Norman Bates (mail):
One other interesting trend line can be obtained by ranking polls on the left-right slant of their sponsors. For example, the CBSNYT poll (the one currently showing the largest Obama lead) is probably way over on the left, Gallup left-leaning, Zogby maybe slightly right-of-center, etc. When I do this there is a clear trend for the Obama lead to shrink as the polling organization becomes less left-leaning. To me this suggests that there might well be enough left bias in the polls that the race is far more close than current polls suggest.

The thing that bothers me about this is that a number of studies have found that significant numbers of voters at least partially base their voting preferences on whom they think will win: People like to be on the side of the winner. There is a real danger that a statistically insignificant polling lead may actually generate a self-fulfilling prophecy. This may give polling organizations a dangerously large role in affecting election outcomes.
10.14.2008 11:18pm
PC:
Oren, predictive power. I'm curious to see how accurate they are. I think their methodology is well thought out.
10.14.2008 11:18pm
ResIpsaLoquitur (mail):
I'm still betting that the media are talking about the Bradley effect now for the "See, I told you so" effect later. If McCain wins, they've already made their pre-emptive racism claim.
10.14.2008 11:25pm
Syd Henderson (mail):

To me this suggests that there might well be enough left bias in the polls that the race is far more close than current polls suggest.


Or enough right bias in the right-leaning polls that the race is not as close as current polls suggest. I suggest the biases in the aggregate polls cancel each other out and Obama's got a 6-8 point lead. It also looks like it's slowly expanding,
10.14.2008 11:43pm
Paul Milligan (mail):
Oren - PC : I don't doubt that Blobama may well win. I think there's a strong association in voter's minds between 'Let's get rid of who caused this credit / financial mess' and McCain, rightly or wrongly.

However, I don't trust polls at all. I've been following 538 and others, and one thing I notice is that 538 is VERY much more volatile in their swings, and always in favor of the Dems. I have heard someswhere the the owner of 538 is an extreme liberal, I don't find the cite right now. I discoutn them to some extent, as I do Zogby for the same reason.

The main guy from 538 is from Chicago. Hmmmm.... and at this link, provided by him,

http://www.newsweek.com/id/140469

he admits being an Obama supporter. Of course, if having a political opinoin disqualified pollsters, there would be no pollsters.

Another factor - I do believe that in the psychology of Republicans or Conservatives ( 2 different critters, of course ), there is more of a mindset to 'not reply to strangers on the phone asking you for personal opinions', whereas the mindset of liberals is the opposite - they are more prone ( IMO ) to want to scream their opinions at everyone around them.

All that being said - IMO polls or perhaps interesting ( I do follow them ), but provide very little information of any real use.

I will go out on a plank here and make a prediciton of my own - this year will see a continuation of the upturn in barely-pre and barely-post election legal action, with both sides contesting every little thing. And that the Dems will sue over EVERY loss they think they can possibly get overturned in court. They NEVER lost an election, they've only had some stolen from them. Just ask them :-)
10.14.2008 11:44pm
Anon21:
Paul Milligan:

However, I don't trust polls at all. I've been following 538 and others, and one thing I notice is that 538 is VERY much more volatile in their swings, and always in favor of the Dems. I have heard someswhere the the owner of 538 is an extreme liberal, I don't find the cite right now. I discoutn them to some extent, as I do Zogby for the same reason.

The main guy from 538 is from Chicago. Hmmmm.... and at this link, provided by him,

http://www.newsweek.com/id/140469

he admits being an Obama supporter. Of course, if having a political opinoin disqualified pollsters, there would be no pollsters.

Nate has been upfront about his support for Obama since the site was launched during the primaries. Nevertheless, 538 did manage to compile a fairly impressive track record during the primaries. Nate's open about his biases, he's open about his methodology, including the periodic changes has made, and I for one am very interested to see how his model performs against the election results.
10.15.2008 12:16am
PC:
Paul Milligan, since you want to have a serious discussion about methodology, "Blobama," oh wait. You don't.

I know that numbers sometimes scare "conservatives" when the numbers don't support the ideological outcome. 538 makes no qualms about its political affiliations. If you have a critique of their methodology, please offer it.

fwiw, I think the paper that Prof. Kopel wrote is a fair criticism of issue polls involving gun control, but I have my own bias as a supporter of gun rights. I'm curious if Prof. Kopel thinks his critique applies to larger polls with a more well developed methodology.
10.15.2008 12:22am
flyerhawk:
For those who think that polls are basically meaningless I suggest we take a trip down memory lane to This day in 2004 and see what poll aggregator electoral vote had to say about the election.

Hmm. On this day in 2004 that site predicted that Bush would get 284 electoral votes. He would up getting 286. They were wrong on Wisonsin and Nevada but pretty much had the other states nailed.

Click on the states themselves and the picture becomes clearer. The aggregate poll information was very accurate, although several states did swing considerably in the final few weeks prior to the election.

As for what electoral-vote thinks now? not good for McCain

McCain needs something major to change the election. Otherwise he's toast.
10.15.2008 1:41am
dsn:
If you want to see the same results filtered by a conservative poll junkey, check out http://realclearpolitics.com/

What do you know. Obama leads there too!
10.15.2008 1:42am
Joshua:
Another thing to consider is not that the presidential polls are inaccurate, but that they may turn out to be either self-fulfilling or self-defeating prophecies. That is, that would-be voters, seeing one candidate with a commanding lead in pre-election polls, decide they have better things to do on Election Day than bother to show up at their polling place to cast a vote that, by their estimation, is now inconsequential to the foregone conclusion of the outcome. (Or, alternatively, that some of these people do show up and vote, but for some third-party "protest" candidate instead of the one they would otherwise have voted for.)

It has been suggested that this dynamic could be a huge hidden advantage for McCain in the battleground states, the reasoning being that Obama's popularity is highest among young adults, a demographic that has historically had low turnout on Election Day. Maybe this will pan out, maybe it won't. That said, it's a sad observation that McCain will probably need this dynamic to break in his favor - basically relying on the vagaries of chance and "unknown unknowns" - in order to pull off the win.
10.15.2008 2:20am
jukeboxgrad (mail):
one thing is certain - if Obama loses, the left will scream 'THEFT !! THEFT !!'


one thing is certain - when McCain loses, the right will scream 'ACORN !! ACORN !!'
10.15.2008 2:49am
Paul Milligan (mail):
"one thing is certain - when McCain loses, the right will scream 'ACORN !! ACORN !!'"

To that, I say :

NUTS ! NUTS !!!!!

:-)
10.15.2008 3:01am
jukeboxgrad (mail):
A joke that's so bad it's good. Well done.
10.15.2008 3:08am
jukeboxgrad (mail):
On the subject of predictions, I think it's worth mentioning Intrade, where McCain is currently at about 21, down from his peak of 52 about a month ago.
10.15.2008 3:14am
Oren:

That is, that would-be voters, seeing one candidate with a commanding lead in pre-election polls, decide they have better things to do on Election Day than bother to show up at their polling place to cast a vote that, by their estimation, is now inconsequential to the foregone conclusion of the outcome.

Why wouldn't this apply equally to supporters of the winning candidate that figure he can win well enough without their vote?.
10.15.2008 11:27am
PDXLawyer (mail):
one thing is certain - when McCain loses, the right will scream 'ACORN !! ACORN !!'

Is it possible that Obama might legitimately win AND ACORN is corruping the process? Something like this happened in 1972.
10.15.2008 12:27pm
Jiffy:
Another interesting poll aggregator is the Princeton Election Consortium. Along with Fivethirtyeight, they seem to have the most sophisticated methodology. Although there are some methodological differences between the two (for example, Princeton eschews "predictions" and just gives an estimate of what would happen if the election occurred today while 538 seeks to predict the outcome on election day), their current assessment of the race is similar.

By the way, one of the previous comments suggested that national results are "meaningless" because "a candidate could easily win, let's say Cali and NY by 90 %, and it makes no difference as compared to 51 %, in the Electroal counting." That's true, but, according to 538's analysis, this insight favors Obama, whose support is distributed much more "efficiently" for capturing Electoral College votes. 538 is currently estimating that it is more than 8 times more likely that Obama will lose the popular vote and win the election than it is that McCain will lose the popular vote and win the election.
10.15.2008 1:46pm
jukeboxgrad (mail):
pdx:

Is it possible that Obama might legitimately win AND ACORN is corruping the process?


Naturally. But that's not what we'll be hearing. We'll be hearing that ACORN corrupted the process and that's why Obama won.

By the way, is it possible that Bush legitimately won in 2004 AND there were a bunch of Diebold shenanigans that went unpunished?
10.16.2008 2:26am