pageok
pageok
pageok
[Andrew Morriss (guest-blogging), August 8, 2005 at 10:18pm] Trackbacks
Compression in LSATs, a good thing?

Some of the comments to my initial post suggested that compression in LSAT scores (for which I have only anecdotal evidence) is not necessarily a bad thing. For example, some folks noted that there is a signalling value that makes it easier to apply to schools where you are likely to get in. This is true, although I also tend to agree with the comment that the application cost is such a small part of the cost of law school that this is not a major factor.)

There are some reasons to be concerned about compression.

1) The LSAT measures the ability to take timed multiple choice tests very well. (My coauthor, Bill Henderson, has an excellent paper on this subject.) Having a law school that is homogeneous in this dimension may not be a great thing. Or it may not matter.

2) A school with a compressed LSAT range is going to find it increasingly hard to move its median LSAT up (conversely, it won't slip much). Since schools put a lot of weight on the median LSAT, a lot of resources are going to end up being spent on trying to get even a very small shift. (I suspect that a rational strategy for many schools would be to focus on the GPA numbers rather than the LSAT numbers; although GPA is weighted less than LSAT by US News, there may be a much greater ability to move the numbers.

3) The shift to the 25/75 percentile numbers (or, more precisely, the average of the 2) means schools are now going to be paying a lot more attention to the keeping their 25th percentile up. That is likely to make it less likely that someone with LSATs below this year's 25th percentile is going to get in next year.

4) Compression has consequences for how law schools allocate scholarship money - one strategy is to spread money around among the people just above your median, in hopes of pulling it up. This means the people paying full freight, i.e. those below the median, are subsidizing the people above the median (assuming the school gets most of its revenue from tuition).

Strophyx (mail):
With regard to the first point, I would suggest, based on anecdotal evidence only, that compression would likely have negative consequences. Until roughly my sophomore year in college I put a great deal of stock in the standardized tests that we seemed to have been constantly subjected to since roughly the 7th grade. They must have been good, since the indicated the intellectual superiority of myself and most of my friends. (Perhaps tracking according to test scores and grades had something to do with this.) Then I met another student who had been enrolled in the same honors program as I who was singularly inept not only at standardized tests, but timed tests of any kind. Given any such test, he would prove himself to be the proverbial village idiot. Instead, professors would always offer him am alternative assessment, usually much more difficult than the test the rest of us took, at which he would always excel.

Because of the strict regimentation of the engineering curriculum, the university offered a 5-year program in which an engineering major could take enough outside courses to earn a second degree in a liberal art. As I recall, this student completed this program in 3 1/2 years, earning a BSEE, and a BA (dual majors in math and philosophy), summa cum. Compression of admission scores would tend eliminate a law school's need to deal with this sort of misfit. It would also prevent students such as myself from having their opinions of their own abilities from being unduly challenged.
8.9.2005 1:45am
Michael:
"That is likely to make it less likely that someone with LSATs below this year's 25th percentile is going to get in next year."

Or that they will designate 24% of their slots for affirmative action and 76% for high LSAT scores (no, I don't think they would really cut it so close, but I would expect a sharp rise between the 20th and 25th percentiles)
8.9.2005 3:45am
KaneCitizen (mail) (www):
What are you guys' (gender-inclusive "guys'" of course) opinions of LSAT and other test prep programs (TPR, Kaplan, etc.)?

Part of me thinks that they're good for people who just have a little bit of test anxiety, but when it really comes down to it, it seems like they skew the overall results of the tests themselves -- like what is being measured is not really what is intended to be measured.

If one group goes into an LSAT (or ACT, SAT, GMAT, or whatever) session with a leg up from having been coached with a bunch of previously used tests and one group goes into it cold, won't there be at least some elevation of the undeserving? The point spread might not be all that much, but it seems like this artificially bumps certain test prep students up above others who might have a somewhat greater natural aptitude in those skills that are being tested for, but who had never been exposed to coaching or previous editions of tests. Again, maybe not by a huge spread of points, but often thousands and thousands of scholarship dollars will rest on a few points.

Or am I fretting too much about the whole thing? Comments?
8.9.2005 10:11am
nn (mail):
Compression in tests is always bad for the very best students. It makes it harder for them to signal their ability and it penalizes those who make careless errors.

When I teach big classes, I always set the difficulty of the exam so that the average is about a 60. When doing so, I find that the good students tend to stay at the top of the class even when making a couple of careless errors. The relative distribution tends to be stable. If the average is in the 80s, there is a lot more random variation. With the hard test, the distribution is more accurate, and I can adjust the curve to be as lenient or tough as I wish. But I then have a better handle on who the best students are.

As with the SAT (where recentering has nearly destroyed the ability to sort out the best of the best)it also allows schools to play more diversity games for "balance" or affirmative action. It also penalizes schools that want to be fully meritocratic and only select on characteristics relevant to academic performance, not social balance.
8.9.2005 10:20am
Phil (mail):
I had the best LSAT (percentile 99.3, I think) of anyone in my law school (which I think was ranked somewhere in the 30s according to USNEWS). I did well but not stunningly so. Of course, because of my score I had a fellowship that eliminated any concerns about tuition. The most the LSAT (1990) at that time measured was the ability to read, think, and write quickly (and not panic while doing so). These skills are somewhat helpful while in law school or practicing law, but they are far from the only skills that one needs.
Discussions of law school stuff among lawyers always makes me think of all the generals who finished at or near the bottom of their West Point classes.
If I were in admissions I cannot imangine taking very many applicants below the 25th percentile on the LSAT, but I also cannot imagine giving any significant weight to scores above the 70th or so. I would love to hear from someone in law school admissions. Are rejections relly riding on whether someone laned in the 74th vs. 76th percentile on the LSAT?!
8.9.2005 10:26am
Aultimer:
nn - standardized aptitude/skill tests are a very different thing from classroom tests. You should worry more about ensuring nearly 100% of your students learn the basic subject matter (pre-inflated "C" scores) than about padding the transcript of a few excellent students. Or do the high achievers pay more for your services?
8.9.2005 10:30am
Medis:
Kane,

As I understand it, the LSAT is not specifically designed to be a "natural" aptitude test. Rather, it is just trying to predict law school performance as measured by law school grades.

Prep classes could thus be non-distorting in at least two ways. For one, maybe they actually teach you something useful about reading, logical reasoning, or so on. Perhaps more importantly, taking such classes may represent the sort of attributes--eg, the means and willingness to spend money and time preparing for exams--that will apply equally well to law school grades.

NN,

I don't think the claim is that the LSAT itself is compressed. Rather, the LSAT scores of people attending any given school have compressed. So, the information available to the law schools is the same, but they are using it differently.

On the main point,

(3) doesn't specify a cause for concern, and (1) does not really do much in that direction either. On (1), more homogenity on any given dimension may mean more or less homogenity on other dimensions (it all depends on how the correlations work). But as yet, that doesn't sound like a cause for concern, unless something specific comes up.

(2) and (4) are variations on the general theme that a using a lot of resources on this project is likely to be inefficient, in that schools could be devoting these resources to more worthwhile projects. That is probably correct, but not very specific to the fact of LSAT compression.

So, I am still left wondering why we should be specifically concerned about LSAT compression.
8.9.2005 10:59am
mkl:
The 4th point does show an interesting detriment of merit based scholarships from the point of the non-recipients: they are paying in excess of the fair market value of their education to subsidize others who will likely erode their class rank, reducing their employment prospects and the value of that education.

I don't think there's a good counter to this, so I guess we just ought acknowlege the generosity and self-sacrifice of those paying the freight. This isn't an argument against the value of merit scholarships to the recipients (which is clear) or to society, which arguably benefits from a higher supply of better qualified lawyers, but that the costs of this fall on those who are in fact made worse off by the practice.
8.9.2005 11:21am
nn (mail):
Aultimer -- My students learn the material very well. But for purposes of the major and for recommendations (I teach a technical subject) my methods do two things: They allow me to identify those who should drop the major (by forcing out students who flunk)and they help me be more precise when identifying the really good students who are capable of going on to PhDs.

Since I regard most American universities as excessively lenient, I find that my grading restores some of the standards that should have been in place when accepting students in the first place, though from your comments I am certain you would disagree.
8.9.2005 12:41pm
SteveL (mail) (www):
I find this difficult to envision in practice. I had a 98.7 percentile score (1989), good enough for any law school in the country. Yet my grades (especially early in my college career) limited my options. The LSAT is really the only reason I was admitted. I graduated top 15%. I happen to think it's a fair predictor, as those with high GPAs and low LSAT scores tended to end up in the bottom 50%.

Point #2: If the LSAT range is compressed, schools will be less likely to take fliers on people like me. The need to move up the median is a bonus for high LSAT, low GPA applicants.

Point #1: I'm horrible at timed multiple choice tests. The LSAT was the one exception. It tests something else.

Point #4: I was far above the LSAT median (if not top score) and was never even considered for a scholarship. Scholarships seem to go to people who have the whole package or opt for say a Rutgers when a Princeton admitted them.
8.9.2005 2:00pm
Aultimer:
nn - I don't disagree, if your comments are limited to PUBLIC American universities. Students reasonably believe they're paying for education, not for faculty to engage in communist-like allocation of talent. It can be fuzzy when the taxpayers are paying a good portion of the bill, but not at private schools.
8.9.2005 2:06pm
Anthony (mail) (www):
mkl, I don't think that non-recipients of merit scholarships are made worse off.

The purpose of merit money is to raise a school's US News rank. If employers and other stakeholders (junior faculty, etc.) care about US News rank to some extent, then increasing US News rank could lead to more employers recruiting on campus (or digging deeper into the class if they already recruit there) or better faculty members accepting offers.

I don't think it's fair to say that merit scholarship recipients are going to reduce non-recipient class ranks, at least not on a major scale. This year, for example, only two of the dozen or so Levy scholars at Penn made law review, which is a lower acceptance rate than for the 1L class as a whole. Of course this is only one year's worth of data from one school, but the results didn't really surprise me; Penn's median last year was a 170 and the 75th percentile was 171, so there likely wasn't that much difference between the average merit scholarship recipient and average non-recipient (and even less difference if you discount AA admits).

Finally, I don't think these students are paying in excess of fair market value. These schools disclose the sticker price upfront, and yet many of them still get 25 or more applications per seat in the class. Law schools, especially top law schools, clearly don't have any issues filling up their 1L classes despite the higher tuitions for non-recipients. In fact, since tuition keeps going up every year without any sort of backlash, one could argue that non-recipient tuition is too low, given that a lot of people are willing to pay even more money for those spots.
8.9.2005 3:34pm
Bruce Hayden (mail) (www):
Maybe it shouldn't matter, but I sure found a prep class helpful when I originally took the LSATs in approx. 1971/2. My performance sure went up on the practice tests. Unfortunately for me, I took the business boards the week before I took the LSAT, and didn't take a prep class there - so I scored 100 pts higher on the LSATs (on the old 800 scale), and so I got my MBA first.

I don't know if SATs are better or worse here, but I just started my 14 year old daughter on SAT prep as she starts high school in a couple of weeks. That was her graduation present from middle school. And she is already doing better on the math questions. I know that this sounds like overkill, but we have worked out that she needs to get through one vocab set every two weeks to finish by her junior year. The result is that she spends less than a half an hour Sat. morning on vocab before she watches TV. Later, we take a math test together. Sun. she reads some from either the WSJ or the Economist before watching TV (again), and then we both take a verbal test together later. In two months, I have already seen improvement.
8.9.2005 7:20pm
Guest (mail):
Brian Leiter has posted on his Law School Reports blog a thoughtful e-mail from the former dean of U. of San Francisco law school, on the rankings and what they can and can't do. Also a strong comment thereto from Stephen M (Ethesis). (I am none of the above, BTW.)
8.9.2005 7:26pm
Strophyx (mail):
I think that NN's point about the benefits from designing generally lower average scores for classroom tests is good. In grad school, many of us were shocked when by one professor's tests. A typical scale was something like: 70% A, 55% B, 40% C, under 40% D. (F's generally weren't used, since too many C's would get you dropped from the program.) As he explained, everybody occassionally has one of those days where the brain simply refuses to engage. With the traditional 95% A, 85% B, 75% C, 60% D that was common when I was in high school and as an undergraduate, a single such day would destroy any hope for the term, while with his scale superior scores in general could make-up for such a brain cramp. Standardized tests, of course are a different matter, and there doesn't seem to be any objective reason for the average score to differ noticeably from the midrange.
8.10.2005 10:03pm