pageok
pageok
pageok
Manipulating College Rankings by Paying Students to Retake Standardized Tests:

Paul Caron of Taxprof blog has an interesting post about Baylor University's sneaky plan to raise its US News ranking by paying admitted students to retake the SAT. The students get $300 just for retaking the text, and $1000 if they beat their previous score by at least 50 points. Caron worries that law schools might imitate Baylor's plan, and try to manipulate their US News rankings by paying admitted students to retake the LSAT. Perhaps even the payments could be on a sliding scale depending on how much the student improved over their previous score.

There is a simple solution to this problem at both the undergrad and law school level: US News and other ranking systems should only take into account scores that were compiled before the the student was offered admission. That way, the danger of manipulation would be eliminated, and universities and students won't squander time and money on test retakes.

There are many problems with the US News rankings that would be difficult to solve. This potential danger is one that can probably be forestalled fairly easily.

In the meantime, I'd like to see if my alma mater, Amherst College, would be willing to pay me to retake the SAT and retroactively improve its US News ranking from when I was a freshman. If they make it worth my while, I'd be happy to do it!

UW3L:
Great idea, Prof. Somin. I took my SAT on the 1600 scale; now that it's on the 2400 scale, getting paid to retake the test, under Baylor's payment model, could be a serious boost to my income this year.
10.16.2008 12:51am
theobromophile (www):
I'm sure colleges will just find another way around this -- especially to kids who apply early decision/early action. The letter could effectively say, "Hey, we'll admit you if you re-take the SAT (or ACT) and improve your score. There's even some financial aid in it for you!"

Of course, the US News could attempt to limit such gaming by taking the average of each test result, rather than the highest. That way, if a student took the test for, say, the third time, it would only have a nominal effect upon her average score. (IIRC, US News did the opposite of this for the LSAT a few years ago, which has resulted in schools looking more favourably upon repeat test-takers, even though evidence shows that their average (not highest) score slightly overestimates their performance in law school.)

By the way, Prof. Somin, it's "test," not "text," in the second sentence, and "alma" mater Amherst.

/random proofreading
10.16.2008 1:13am
Monty:
The SAT measures your ability to take the SAT, just as the LSAT measures your ability to take the LSAT... I don't deny the existance of a slight correlation between performance on the tests, and performance as a student, but we take them far to seriously... Rankings should reflect the quality of an educational institution, not how well its INCOMING students do on a standardized test.
10.16.2008 1:20am
TFKW:
What would the equivalent scheme be for GPAs?

(I'm really not sure, but there must be one, since grades are so much less objective.)
10.16.2008 1:54am
Cornellian (mail):
$1000 is serious money, sign me up.
10.16.2008 3:14am
Obvious (mail):
Ilya,

And this is harmful how?

A change in college ratings based on real improvements in test scores is bothersome why?

College listing trumps cost, convenience, parental and student preferences, closeness to home, and numerous other factors how often?

BTW, if it's important to you, I think you could have had a LOT more comments if you'd used the subject title "Manipulating College Rankings by Paying McCain and Obama to Retake Standardized Tests" :-)
10.16.2008 3:20am
Hoosier:
There are many problems with the US News rankings that would be difficult to solve.

Sorry, Ilya, but I would suggest that there is an easy way to solve all the problems with the USNWR rankings.
10.16.2008 3:28am
Tomm:
This invites the obvious question: what did Ilya and the other conspirators get on their SATs?
10.16.2008 8:51am
Hoosier:
College listing trumps cost, convenience, parental and student preferences, closeness to home, and numerous other factors how often?

Very, very often. It's about as pathetic as something can be if it takes place outside of New Jersey.
10.16.2008 8:58am
rgore:
The LSAT admission ticket includes a certification that must be signed by each test taker and states that the person is taking the test for the sole purpose of applying to law schools. I wonder if there is anything similar for the SAT?
10.16.2008 9:41am
Norman Bates (mail):
I suspect that the Baylor administrators who came up with this idea didn't have very high SAT scores themselves (the scores correlate highly with IQ/intelligence). After spending a lot of money, Baylor's mean SAT among incoming freshmen (plus those freshman earning some extra spending money by retaking the SAT) will hardly change at all because of regression to the mean.

The only way Baylor can offset this is by setting a cutoff SAT score and disqualifying freshmen with higher SATs from participating in this program. But that runs the risk of over-weighting low-scoring students when calculating Baylor's averege SAT score.

On this one Baylor deserves exactly what it's going to get.
10.16.2008 10:04am
Curt Fischer:

Norman Bates: After spending a lot of money, Baylor's mean SAT among incoming freshmen (plus those freshman earning some extra spending money by retaking the SAT) will hardly change at all because of regression to the mean.


I suspect that the VC poster who came up with this idea...is incorrect, because students usually are allowed to use their highest SAT score, not their most recent score, when reporting to and applying to schools. "Regression to the mean" is in such a case inoperative. An opposite effect, where test takers, if they try enough times, eventually become statistical "outliers" and land an oversized score, would in fact guarantee that average scores would increase under the Baylor program.


The only way Baylor can offset this is by setting a cutoff SAT score and disqualifying freshmen with higher SATs from participating in this program. But that runs the risk of over-weighting low-scoring students when calculating Baylor's averege SAT score.


This part is doubly wrong. Even if students could not cherry pick their best score, your proposed offset would not do what you seem to think. As you yourself mentioned, "scores correlate highly with IQ/intelligence". In that case, the cutoff would need to be in the difference between the measured score (e.g., the SAT score), and the underlying "true" value (e.g., the IQ). To be overly numerical about it, a kid with an IQ of 80 who did 50% better than other IQ-80 kids on the SAT would need to be excluded from the program, even if his score was only 500. A kid with an IQ of 160 who did worse than most other IQ-160 kids, even though he scored say a 2000, should be incended to re-take. And all that's even if kids cannot pick their highest score.

Are you still harboring any suspicions about Baylor administrators and their test scores?
10.16.2008 10:56am
Dan Weber (www):
If the metric can be so easily manipulated, then maybe we should stop using the metric.

I say more power to Baylor for making a mockery of the US News Rankings.
10.16.2008 11:22am
Norman Bates (mail):
Curt Fisher:

If you are correct, and only highest SAT scores are reported, then I concede that the Baylor system would be a one-way ratchet to increase SAT averages.

In the absence of other knowledge about a freshman's intelligence your second argument doesn't apply.

I still doubt that the scheme is cost effective. It won't raise average SAT scores much. The effect of a small increase in average SAT score on the number and quality of Baylor applicants is probably not very great.

I still harbor suspicions about the level of intellect involved in cooking up this scheme.
10.16.2008 11:40am
Hoosier:
Re: Regression to the mean--

Keep in mind that universities report the highest total that a student receives in each subtest. So a student who takes the test three times, and scores M: 680, 720, 700; and V: 680, 650, 660 is reported as a student who scored total 1400 (M/V).
10.16.2008 12:20pm
Blue:
Excellent move! Anything that puts US News rankings into disrepute is positive.

The basic problem with their approach is that, at the end of the day, their BS institutional reputation rankings will always be used to ensure the "right" results at the top of the distribution. It's just a fancy way of presenting conventional wisdom. A truly meaningful ranking system would use independent variables of student achievement and measure the amount of learning and skills that the institution added to its graduates.
10.16.2008 12:34pm
Curt Fischer:

In the absence of other knowledge about a freshman's intelligence your second argument doesn't apply.


You are right of course that it is unlikely that a school will have much information better than SAT scores to indicate a student's intelligence. In that case, it is unlikely that a school would be able to institute some sort of "overperformer-relative-to-intelligence" cutoff for test retakers.

My point, though, was that your proposal for a one-size-fits-all cutoff based only the raw SAT score number would be ineffective, regardless of whether a school has some outside knowledge about their freshmen's intelligence.
10.16.2008 12:43pm
Bill Poser (mail) (www):
For those who think that religion underlies morality, I'd like to point out that Baylor is a Baptist school with a fairly strong Christian theme, including chapel requirements. I don't think that this program reflects very well on the Baylor administration's ethics.
10.16.2008 1:22pm
Bill Poser (mail) (www):
According to the New York Times, Baylor faculty are unhappy with the SAT retake program.
10.16.2008 1:24pm
jvarisco (mail) (www):
I'm not sure I see the problem here. All of the top colleges are full of people who went through $1000+ dollar private tutoring and thus have a score higher than people at less selective universities (such as Baylor) who probably only took the test once with a lot less studying. SAT ranking is a good way of gauging generally ability in student body; it's a lot more meaningful than GPA, which varies so widely across schools. And whether you take it after admittance or before, it's not as if your IQ is going to change, so it still provides the correct measure for US News. Back when I was applying to college, average SAT scores made a big difference - I wanted to be with smart kids like myself, not idiots who couldn't read or do basic math. Considering just how abysmal a score their average of 1200 is (that's an 800 on the 1600 score, and you get 1000 or so just for putting down your name and leaving it blank) there's certainly room for improvement.
10.16.2008 1:29pm
Hoosier:
Bill Poser:

For those who think that religion underlies morality, I'd like to point out that Baylor is a Baptist school with a fairly strong Christian theme, including chapel requirements.

If you are going to take the role of the Rationalist scoffing at religion, try to avoid logical fallacies.
10.16.2008 2:36pm
Passing By:
I took my SAT on the 1600 scale; now that it's on the 2400 scale, getting paid to retake the test, under Baylor's payment model, could be a serious boost to my income this year.
I took the LSAT when it had a 48 point scale. It's a shame I don't qualify - I would happily retake the LSAT every day, for as long as Baylor was willing to pay me to do so.
10.16.2008 2:50pm
MisterBigTop (mail):
jvarisco,

If you know the words, then the analogies, sentence completions, etc., aren't difficult, at all, even for the average mind. It's quite possible for a student to dramatically increase a weak SAT verbal score by studying lots of words. Does that mean his IQ has really changed to any significant degree?
10.16.2008 2:54pm
Dan Weber (www):
, and you get 1000 or so just for putting down your name and leaving it blank

No, leaving it blank gets you a bit over 200 points on each test. (You have to actively get some wrong to get all the way down to 200.) The average is 500.

A 1200 average on V/M is 600 each, nicely above average. A 1200 on V/M/E is 400 each, below average.
10.16.2008 3:11pm
wfjag:
Sounds like a great idea. It keeps Baylor students from wandering the streets of Waco till they have some money to spend.
10.16.2008 4:23pm
jvarisco (mail) (www):
MisterBigTop, my point was that students at good schools (as opposed to Baylor) are already going to be doing that. So students at better schools end up looking even better on tests than they would otherwise; this would even the playing field a bit. There are very few Harvard students who didn't at least study some vocab before the SAT.

Keep in mind it's not like they are handing out cash - the incentives are in credit at the bookstore and scholarships. Which would not be all that useful for people who are not Baylor students.
10.16.2008 5:08pm