How About A Willingness To Misrepresent The Results Of Studies?

Would you get marked down for that? I'm speaking of this CNN item, reporting on this study [UPDATE: Reader Margaret Schoen suggests that though this is hosted on the CNN site, and includes the CNN logo on top, it may just be a Netscape News reprint of a press release; in any event the substantive analysis below stands]:

States Ranked: Smartest to Dumbest

The smartest state in the union for the second consecutive year is Massachusetts.

The dumbest, for the third year in a row, is New Mexico.

These are the findings of the Education State Rankings, a survey by Morgan Quitno Press of hundreds of public school systems in all 50 states. States were graded on a variety of factors based on how they compare to the national average. These included such positive attributes as per-pupil expenditures, public high school graduation rates, average class size, student reading and math proficiency, and pupil-teacher ratios. States received negative points for high drop-out rates and physical violence.

I have an idea: Let's have a Baseball Teams Ranked: Most Talented to Least Talented, and grade a team's talents based in part on how much it spends on salaries, how large its farm team network is, how many fights the baseball players get into, and so on. What, you say? Doesn't make sense? Some teams waste lots of money, while others spend less but do it more efficiently? A baseball player's tendency to fight may be unsporting, but doesn't show lack of talent?

My point exactly. If you want to measure how well-educated a state's students are (something that you might label "smarter" without much journalistic license), do that. You might have to use imperfect proxies, such as proficiency exam scores or graduation rates, but that's the best you can do. (After all, won-loss records are imperfect proxies of baseball talent, too, but we rely on them.)

But don't separately count per-pupil expenditures or average class size. If spending more and having smaller classes improves performance, then that will show up on the performance measures. And if it doesn't show up there, then why count it as "smart"?

I've enabled comments; please keep them on-topic, substantive, and polite.

UPDATE: Just to preempt a possible objection: It's also possible to characterize the study as misdesigned, for including the spending factors. But I'm most galled by the inaccuracy of how the study's results are misleadingly characterized. "Top-ranked," while a bit tautological, would at least not make any unsupported claims; but "smartest"? That's just not a candid way of reporting on what the study measured.

Steve R (mail):
It seems to me that the "smarter" states will spend less to achieve the same or better results in student perfomance. That is to say, I would design a study leaving in class size and spending rates etc, providing an inverse relationship between those data points and the rankings.
6.14.2005 7:16pm
Matt Glassman (mail) (www):

This is an endemic problem across a variety of ranking systems in a variety of fields of research on public policy. But in each case, it seems to me to be rooted in the same misconception: that "spending more money on variable X" is equivalent to "improving variable X." Education is by far the best example of this, although undoubtedly other X's exist.

The basic fact is as follows: after you have enough money to build a school and make sure each child has the proper materials (enough desks, books, pencils,teachers, etc), the marginal value of each dollar spent on education rapidly approaches zero, and other factors (parental involvement, social conditions) begin to take over as predictors of performance.

Nevertheless, people (especially those who become president) tend to think that all problems can be wallpapered over with money - and they have an attentive public audience for it.

All of this is sad, but not the least surprising. The basic liberal complaint in modern America is enconomic inequality. As such, the basic liberal solution to problems is almost necessarily an equalization of financial resources. As such, the basic liberal measure of progress is not progress per se, but money spent. QED.

BTW, i think this blog is great.
6.14.2005 7:24pm
carpundit (www):
You're right, of course, on the facts. But in criticizing CNN for not being an accurate reporter of facts, you are letting your desires become your expectations. The poll, like most <i>television</i> news, wasn't reported to edify anyone. Rather, it was certainly reported only to fill space (or fill airtime) in a way that would hold eyes, so they can sell ads. "Smartest to dumbest" is short, punchy, and easily talked about around the water cooler. Exactly <i>not</i> like a discussion of actual public policy.
6.14.2005 7:53pm
Charlie Quidnunc (mail) (www):
They measured and reported the inputs instead of the outputs. Typical undergraduate science lab error.
6.14.2005 8:07pm
Wile E:
Who are we to judge? Those guys at CNN have a much "smarter" websight than any of us.
6.14.2005 8:30pm
Jay Tea (mail) (www):
Not to toot my own horn too loudly, but I took this "study" on over the weekend. The money is only the tip of the iceberg.

I'll recap: besides the utterly-irrelevant money factors, they also count school-age children not in public schools as well as dropouts. That means that not only dropouts get counted twice, but private schools and home schooling contribute to a state's "dumbness."

I'm no statistician, no polling expert, and certainly no educational expert. But without even trying, I discredited 6 of the 21 factors in their alleged "research." That's nearly 30% of their ranking factors shredded.

6.14.2005 8:39pm
dick thompson (mail):
Same outfit does the same kind of study for best places to live. They include the price of a home and the income but have no comparison with cost of living. Result is very high ratings for the most expensive cities without taking into account that with the price of rent/mortgate you can't afford to eat out or go out at all. The quality of everything else is useless if you can't afford to use it.

As an example, I was a consultant in a large midwestern city for a couple of years. My rent for a 3 BR apartment including utilities in a good neighborhood was $600/month. I now live in a large east coast city and my rent for a studio (1 room for everything) is $1000/month. Income is the same. Where do you think I would live better. Schools work the same way. The cost needs to be balanced with what it can buy or it is just a number with no meaning.
6.14.2005 8:49pm
Xrlq (mail) (www):
This obviously one of those cases where "smartest" and "dumbest" are defined to mean "do what I think is a smart thing to do" vs. "do what I think is a dumb thing to do." Advocacy groups put out stuff like this all the time, the only question is why CNN thought it worth reporting as news.
6.14.2005 9:26pm
Dr. Weevil (mail) (www):
Graduation rates don't prove anything, either. If they're high, it may mean that the state is doing an excellent job teaching even the less interested and less talented students, or it may just mean that they're letting students graduate who haven't learned anything.
6.14.2005 10:00pm
Does anyone take CNN seriously anymore?
6.14.2005 10:01pm
If you were smarter, Eugene, you would not have posted this. Because of course, smart == liberal. Everyone smart knows that.
6.14.2005 10:11pm
washerdreyer (mail) (www):
It's not true (though its falsity is irrelevant to your main point) that baseball talent (even on a team level) is measured based on Wins and Losses by anyone who is seriously interested in the question. It's measured by league and era adjusted runs allowed and runs scored, and/or the elements thereof.

Also, Michael, your post is a completely baseless insult against liberals
6.14.2005 10:29pm
CNN should obviously be more skeptical in their reporting rather than just printing what is essentially a press release - but the reference to the "smartest" state comes directly from the authors of the "study" who trumpet on their webpage that Massachusetts is the "smartest state."

It's obvious why the study's authors would do this... one suspects that a press release stating "Massachusetts is rated at the top of the nation in the arbitrary education index score that we made up" would not have made it to CNN.
6.14.2005 10:43pm
phillymikec73e (mail) (www):
In New Jersey we have millions of dollars that just "disappeared" from accounts that no one could figure out where it went. Does that count for "money spent?"

6.14.2005 10:49pm
Washerdreyer, I don't think my post was an insult at all liberals, only those who believe they are that way because they are smarter than everyone else.

However, I think it probably was below the standards of discourse Eugene wants on this web site (because I didn't make an actual point) so I apologize for that. There are lots of other web sites I can make stupid jokes on, and the last thing Eugene wants is for people to try to one-up the other side by getting more of their own pointless partisan posts in on each thread.
6.15.2005 9:14pm