pageok
pageok
pageok
Half the Story:

Don Surber notes, quoting a correspondent:

Today's approval numbers via Rasmussen are:

45% approval for Bush
43% approval for Pelosi

Well, that's half the numbers. Filling in the other half (which, to Surber's credit, is linked from his post), we see that Bush is at 45% job approval and 54% disapproval, while Pelosi is at 43% favorable and 39% unfavorable. So Pelosi seems ahead of Bush (+4 vs. -9) rather than behind. And the talk in the Surber post about the media's "false ... impressions," media "conventional wisdom [being] flushed down the toilet," and media "lies," seems like something of an overstatement.

The broader lesson: When you see survey results, don't look just to one number. Keep in mind that the "approve" and "disapprove" (or "yes" and "no" or whatever) likely don't add up to 100%, and that the "no answer" fraction may vary from subject to subject, as it does here. (Pelosi, understandably, isn't as well-known as Bush.) Treat each survey as yielding two numbers, and when you compare survey results, compare the pairs, at least unless the "no answer" fractions are pretty much the same in each pair.

Better yet, look at more than two numbers, if the survey gives them, for instance if it separately reports the "strongly"/"very" and "somewhat" sentiments on each side. (As it happens, in both of these surveys, those with strong feelings about the subject -- whether Bush or Pelosi -- are mostly disapproving, while those with milder feelings are mostly approving.) But look at least at the two numbers from each survey, rather than just at one.

Eli Rabett (www):
The next rule is do not look at a single poll. Polling Report as well as Dr. Pollkratz have the ones for Bush. In the latest one, he is at 30% approve, 63 % dissaprove and that brings us to the next point. That is much more representative of recent polls

The last rule is look at the who the hell is this person and what are they doing to us question. On Pelosi, as late as the middle of December, roughly half of those polled answered that they did not have enough information to form an opinion.

Thank you, these have been simple rules for looking at polls.
1.6.2007 7:01am
Sk (mail):
On the other hand, the raw % approval rating (without the disapproval rating) has been a standard measure of presidential popularity for decades. I have often seen headlines stating Bush's (or Clinton's, or Reagans, etc etc) approval rating. I have never seen a headline listing both the approval AND the disapproval rating.

In other words, using the 'industry' accepted means of measuring popularity, that comparison between Pelosi and Bush seems entirely valid.

So why the sudden interest in both numbers-in gaining a more nuanced understanding of the survey? Presumably because using only the approval rating doesn't give you the headline you want ("Pelosi Approval Lower than Bush!").

Another valuable lesson about surveys. They will say just about anything you want them to say (lies, damn lies, and statistics). Thus, anytime someone questions an irrelevant poll and wants to correct your perception of the results, you can pretty much conclude that they don't really care about the actual results-they just don't like the headline.

Sk
1.6.2007 7:20am
Steve Lubet (mail):
After 6 years on the job, Americans have had ample opportunity to evaluate President Bush's performance -- and 54% disapprove.

After one day on the job, 43% of Americans approve of Speaker Pelosi and another 18% have wisely decided to withhold judgment.

In this situation -- 6 years vs. 1 day -- the unfavorables would seem to be the more relevant standard of comparison.
1.6.2007 8:19am
Lively:
Roughly 50% of the country didn't vote for Bush. Many hate him/his agenda no matter what the topic. With 54% disapproval, that accounts for 4% of people who gripe but don't go to the polls to vote.
1.6.2007 8:33am
Rand Simberg (mail) (www):
Also note that just because you disapprove doesn't necessarily imply anything about voting patterns. I've disapproved of Bush for his entire term, and thought that the country was on the "wrong track" for decades. That doesn't mean I'm going to vote for a Democrat (especially when the Democrat is Al Gore or John Kerry).
1.6.2007 8:37am
Mahan Atma (mail):
"the raw % approval rating (without the disapproval rating) has been a standard measure of presidential popularity for decades."


But it's only valid because everybody knows who the president is...
1.6.2007 8:54am
Justin (mail):
None of the other polls have come close to a 45% approval rating since before Katrina. Yay, outliers! Way to prove a point!
1.6.2007 9:56am
Hovsep Joseph (mail) (www):
Good point, but I think Eli makes another good point about not just cherry picking aspects of a poll to report on, but cherry picking the poll itself. Pollster.com plots the approval numbers for Bush from all the polls over the past year here. And they report that his most recent trend estimate is 33.5%, making the Rasmussen poll a far outlier.
1.6.2007 10:00am
Ted Frank (www):
I find it hard to believe that 81% of the country has an informed opinion on Speaker Pelosi. It's not even clear to me that that many have an informed opinion on Bush.

I'd much prefer to see government act as leaders, rather than followers, of polls. Yesterday at AEI, John McCain publicly took a position on the war in Iraq that only 11% of Americans support in the last poll; I was in the other 89% until I heard and read the arguments.

Something like 97% of Americans think that it's something other than the basic laws of supply and demand that lead to high gas prices, which encourages all sorts of counterproductive mischief from pandering politicians.

So I'm not sure what we're supposed to make about the relative approval/disapproval ratings for Bush and Pelosi. We, as voters, bound ourselves to have the executive branch led by one for four years and one house of the legislative branch led by the other for two. Pelosi (unless there's a double impeachment) doesn't suddenly get to dictate how the executive branch gets run, even if her approval rating is thirty points higher; on many matters, such as judicial nominations, she doesn't even have a say.
1.6.2007 10:17am
Zubon (www):
Job approval, not personal? Ms. Pelosi must have had one heck of a day to get that much response. Have any bills been passed yet?
1.6.2007 10:48am
Erich (mail) (www):

Volokh,
good points on survey reading but if the MSM ever took that much time to present a survey about Bush that honestly it would be the FIRST time they did so.
1.6.2007 10:52am
jab (mail):
I find it distressing thaty Prof. Volokh would even give exposure to that low-level hackery of a blog...
Regarding cherry-picking of data, we all succumb to that at times, especially on issues we are passionate about... human nature. However, this was NOT just cherry-picking, but clearly out-right manipulation and misrepresentation of the data... and I don't think it was an honest mistake... they knew what they were doing, they just have no integrity.
So again, you are too kind, too kind to even give those hacks exposure, and too kind in the way you characterize their hackery as perhaps an honest mistake that just needs a mild scolding/lesson.
1.6.2007 11:02am
Zhid (mail) (www):
I live in the Bay Area and I hate Pelosi, but that poll is really not a negative for Pelosi. Look at her numbers compared to the Speaker than she replaces and her approval rating is a HUGE increase over Hastert's. IF these polls have any value, it's not in the raw numbers but in the comparisons over time.
1.6.2007 11:03am
Eugene Volokh (www):
sk: (1) Mahan Atma is right; as I pointed out in my post, one should "compare the [approval-disapproval] pairs, at least unless the 'no answer' fractions are pretty much the same in each pair." For the President, the no answer fractions tend to be pretty similar from poll to poll. But for other people, the no answer fractions are much higher (18% for Pelosi, 1% for Bush). That's why when you're comparing Bush's and Pelosi's numbers, just looking at the approval rating is inadequate.

(2) You write, "So why the sudden interest in both numbers-in gaining a more nuanced understanding of the survey? Presumably because using only the approval rating doesn't give you the headline you want ('Pelosi Approval Lower than Bush!')." An odd presumption, as people who have read my blog (which is hardly a Democratic stalwart) would know. Why not instead presume that I do indeed have an interest in trying to set the record straight, to the point of correcting claims (Pelosi gets less approval than Bush) even when I would have liked to believe them?
1.6.2007 11:45am
ReVonna LaSchatze:
Thanks for the always needed reminders of how to read numbers in the news. Perhaps the numerical ignorance or incompetence of the MSM is bad enough, and I do believe they have good intentions.

But throw in all those BlogPundits looking not to report news facts, but to make a name for themselves by spinning to support their pre-determined postitions. See the post on the poll below. Use all the qualifications you want, but folks end up holding on to their "number" as though that tells us something about Americans. Sad how a little weakness like numerical illiteracy,, combined with smart programming skills can override intelligent discourse.
1.6.2007 11:49am
trotsky (mail):
To add to Zhid's point, Pelosi's positives and negatives have both been rising since November, when something like half of those polled had just never heard of her.

It'll be interesting to see where they level out.

But are congressional leaders ever particularly popular? It seems like Congress is lucky when it hits a 40 percent approval rating, and I have to figure that any particular congressional leader's reputation is going to track the reputation of the body as a whole -- you know, barring an indictment or somesuch.
1.6.2007 11:50am
Elliot Reed:
While it's usually not relevant to this type of poll, you forgot to mention the other cardinal rule of poll-interpretation: always examine the exact language of the question. "Do you support X?" and "Do you support the President's proposal to X?" will get you vastly different results.
1.6.2007 11:51am
Eugene Volokh (www):
By the way, to his credit, Mr. Surber has posted an update acknowledging the point made in this post.
1.6.2007 11:51am
ReVonna LaSchatze:
Why not instead presume that I do indeed have an interest in trying to set the record straight

Personally, I gave you that. Thanks and good job.
(This kind of constant vigilance re. political spin is valuable, as in real not superficial value. Look at the past 5 years alone...* Now look ahead at other countries who will be competitors and have mastered the numbers game and tell me the ongoing numerical distortion won't hurt us?)

*Medicare restructuring
*War + FEMA budgeting
*Corporate collapses
*Energy planning
1.6.2007 11:57am
Elliot123 (mail):
Poll reporting tends to emphasize what the news media want to show. Few polls are single question polls; most have many questions, and the reporters choose the question they find most interesting. They also very often omit the exact question asked in favor of their own paraphrase of the question. So, the formula is to cherry pick and rephrase the question to make a more interesting story or support a particular agenda.
1.6.2007 12:29pm
Byomtov (mail):
Well, that's half the numbers.

No. It's not anywhere near half the numbers. There are lots of polls, enough that you're likely to get one or two that are way off. This is just simple statistics. Making an issue out of one result that is far away from the others is foolish, or worse.
1.6.2007 12:46pm
Paul Turner:
This is a second, less important flaw in the Bush-Pelosi poll comparison than ignoring the level of undecideds, but it goes to the issue to cherry picking. Rasmussen conducts an approval poll for Bush daily. The numbers vary a bit from day to day, as one would expect from a small sample. The Rasmussen site says that the Pelosi poll was January 3-4. Bush was 43 percent on January 3, but that was not the date chosen for comparing her approval with Bush's. Picking Bush's peak for the week, instead of the same day as the Pelosi poll, suggests bad faith. Rasmussen has Bush at 42 percent today.

On the other hand, the criticism that other polls rank Bush lower than Rasmussen does not impress me. Rasmussen is consistently above the poll average, a "house effect" it is frank about. The Political Arithmetik site has a good discussion of this subject. This is a Rasmussen to Rasmussen comparison, though, which seems fair.
1.6.2007 12:46pm
Byomtov (mail):
Picking Bush's peak for the week, instead of the same day as the Pelosi poll, suggests bad faith.

Indeed it does.

And if we're going to rely on Rasmussen, let's not ignore this:

As Democrats took control of Congress for the first time in 12 years, 48% of Likely Voters trust Democratic Congressional leaders on key issues more than they trust President Bush. Just 38% take the opposite view and have more trust in the President.

Wonder what Surber thinks about that?
1.6.2007 1:15pm
Hovsep Joseph (mail) (www):
Paul Turner: On the other hand, the criticism that other polls rank Bush lower than Rasmussen does not impress me. Rasmussen is consistently above the poll average, a "house effect" it is frank about... This is a Rasmussen to Rasmussen comparison, though, which seems fair.

Good point. If Rasmussen consistently shows higher approval for all subjects, then there's no real bias in highlighting the Rasmussen poll as newsworthy (the original blogger's point and Instapundit's point was that the media aren't reporting this because they're biased). But other recent polls of Pelosi are strikingly different, with about half of those polled consistently reporting that they're unsure or haven't heard of her. And that really seems more accurate to me. People who read blogs like this tend to know way more about and have firmer impressions of individual politicians than the general public which may create a distorted view of what we think the public should be responding to. But an incoming Speaker of the House is someone most people just don't have much of a feeling for. Two caveats: (1) this week's media has been saturated with coverage of Nancy Pelosi, so that may explain the lack of unsure/don't know responses in the poll by Rasmussen earlier this week and (2) I'm not sure its even useful at all to compare the approval ratings of a President and a Speaker (even one who has been in office for a while) considering their very different roles and responsibilities.
1.6.2007 1:48pm
Christopher Cooke (mail):
Of course, everyone "knows" (intrinsically, as a matter of faith) that Bush is divinely ordained, and is the most popular man in the USA and in Europe, and we all blame Pelosi for all of our problems. Any polls to the contrary are just lies.
1.6.2007 3:20pm
abean:
Who cares? Presidents get elected for four years. If public opinion was to matter quite so much, you'd think the founders would have scheduled automatic, periodic recall votes.

More rationally: this is a marginal pricing effect. What is Bush's price? Well its being determined by the fringe events. Meanwhile the government has a few million people milling about doing stuff. Too much sutff/too many to rational evaulate the totality of the President's leadership and the effectivenss of the executive.

e.g. how much credit does Bush get for not appointing another Janet Reno at justice? Moreover, people might recall that when Clinton came to power he started politicizing the US Attorneys immediately, turning the Bush appointees out of their jobs without regard for what work was pending in the office. Conversely the Bush team graceful allowed the Clinton era US Attorneys to close up shop at their own pace with an eye to what work was going around in each office.

That's a substantive difference in policy but at a level of detail that too easily escapes the notice of approval ratings.
1.6.2007 3:35pm
BobNSF (mail):
Can anyone explain why supposedly intelligent people would care about a poll conducted after a person's first day on the job?

Oh, I know! It's all about polarization! I haven't followed the vote on all the legislation already passed, but it seems to me that the poll that counts is the approval in the House. So far, the Dems seem to be pulling an awful lot of Republicans on board. Scary, eh?
1.6.2007 3:45pm
DRJ (mail):
These poll results are interesting because they suggest Americans don't have much faith in their leaders, no matter how well they know them.
1.6.2007 9:14pm
CrazyTrain (mail):
Professor Volokh -- This is why people on the "left" really dislike Instapundit more than most right-wing sites. He links blindly to stuff that he claims supports his point but does not all, and he smugly continues to assert that his point of view is right even after being pointed out that the facts are dead set against him. Justin commented on how much moderates and liberals find Instapundit to be "unreadable" and you expressed puzzlement over that; this is why. Just saying.
1.6.2007 11:51pm
ReVonna LaSchatze:
Heh! Ditto CrazyTrain.

The InstaMan is an intellectual fraud. Indeed. Needs to be called on it more often.
1.7.2007 4:49am