The Volokh Conspiracy - How About A Willingness To Misrepresent The Results Of Studies?

Would you get marked down for that? I'm speaking of this CNN item, reporting on this study [UPDATE: Reader Margaret Schoen suggests that though this is hosted on the CNN site, and includes the CNN logo on top, it may just be a Netscape News reprint of a press release; in any event the substantive analysis below stands]:

States Ranked: Smartest to Dumbest

The smartest state in the union for the second consecutive year is Massachusetts.

The dumbest, for the third year in a row, is New Mexico.

These are the findings of the Education State Rankings, a survey by Morgan Quitno Press of hundreds of public school systems in all 50 states. States were graded on a variety of factors based on how they compare to the national average. These included such positive attributes as per-pupil expenditures, public high school graduation rates, average class size, student reading and math proficiency, and pupil-teacher ratios. States received negative points for high drop-out rates and physical violence.

I have an idea: Let's have a Baseball Teams Ranked: Most Talented to Least Talented, and grade a team's talents based in part on how much it spends on salaries, how large its farm team network is, how many fights the baseball players get into, and so on. What, you say? Doesn't make sense? Some teams waste lots of money, while others spend less but do it more efficiently? A baseball player's tendency to fight may be unsporting, but doesn't show lack of talent?

My point exactly. If you want to measure how well-educated a state's students are (something that you might label "smarter" without much journalistic license), do that. You might have to use imperfect proxies, such as proficiency exam scores or graduation rates, but that's the best you can do. (After all, won-loss records are imperfect proxies of baseball talent, too, but we rely on them.)

But don't separately count per-pupil expenditures or average class size. If spending more and having smaller classes improves performance, then that will show up on the performance measures. And if it doesn't show up there, then why count it as "smart"?

I've enabled comments; please keep them on-topic, substantive, and polite.

UPDATE: Just to preempt a possible objection: It's also possible to characterize the study as misdesigned, for including the spending factors. But I'm most galled by the inaccuracy of how the study's results are misleadingly characterized. "Top-ranked," while a bit tautological, would at least not make any unsupported claims; but "smartest"? That's just not a candid way of reporting on what the study measured.

	[Eugene Volokh, June 14, 2005 at 6:03pm] Trackbacks How About A Willingness To Misrepresent The Results Of Studies? Would you get marked down for that? I'm speaking of this CNN item, reporting on this study [UPDATE: Reader Margaret Schoen suggests that though this is hosted on the CNN site, and includes the CNN logo on top, it may just be a Netscape News reprint of a press release; in any event the substantive analysis below stands]: States Ranked: Smartest to Dumbest The smartest state in the union for the second consecutive year is Massachusetts. The dumbest, for the third year in a row, is New Mexico. These are the findings of the Education State Rankings, a survey by Morgan Quitno Press of hundreds of public school systems in all 50 states. States were graded on a variety of factors based on how they compare to the national average. These included such positive attributes as per-pupil expenditures, public high school graduation rates, average class size, student reading and math proficiency, and pupil-teacher ratios. States received negative points for high drop-out rates and physical violence. I have an idea: Let's have a Baseball Teams Ranked: Most Talented to Least Talented, and grade a team's talents based in part on how much it spends on salaries, how large its farm team network is, how many fights the baseball players get into, and so on. What, you say? Doesn't make sense? Some teams waste lots of money, while others spend less but do it more efficiently? A baseball player's tendency to fight may be unsporting, but doesn't show lack of talent? My point exactly. If you want to measure how well-educated a state's students are (something that you might label "smarter" without much journalistic license), do that. You might have to use imperfect proxies, such as proficiency exam scores or graduation rates, but that's the best you can do. (After all, won-loss records are imperfect proxies of baseball talent, too, but we rely on them.) But don't separately count per-pupil expenditures or average class size. If spending more and having smaller classes improves performance, then that will show up on the performance measures. And if it doesn't show up there, then why count it as "smart"? I've enabled comments; please keep them on-topic, substantive, and polite. UPDATE: Just to preempt a possible objection: It's also possible to characterize the study as misdesigned, for including the spending factors. But I'm most galled by the inaccuracy of how the study's results are misleadingly characterized. "Top-ranked," while a bit tautological, would at least not make any unsupported claims; but "smartest"? That's just not a candid way of reporting on what the study measured.