pageok
pageok
pageok
[Andrew Morriss (guest-blogging), August 9, 2005 at 11:30pm] Trackbacks
Rankings & more rankings:

Brian Leiter posts a lengthy and informative email from Paul McKaskle, former dean of the University of San Francisco (mentioned in a comment to my earlier post, but I wanted to highlight it for those who don't read all the comments). Dean McKaskle makes some interesting points about rankings in general and suggests a two tier division of schools - the very top and the rest. He suggests that the biggest advantage of the very top schools for students is the presence of a larger percentage of other excellent students.

Dean McKaskle has some very good points in his criticism of the U.S. News rankings. There is a lot about U.S. News rankings that troubles me: the ease of gaming, the less-than-relevant measures, and so on. But prospective students need something on which to base their decisions about where to apply, which school to accept. U.S. News is clearly meeting a demand for information.

So what to do? One answer is that we're seeing a proliferation of information resources for prospective students. I get contacted by prospective students (not all referred by the admissions office) who see something in my web bio that interests them and prompts them to ask for more information about Case. Brian Leiter is publishing both his own assessments of quality for the top schools and lots of specific information on faculty movements in Leiter's Law School Reports, information that gives prospective students some sense of whether a school is trending up or down. Bulletin boards are allowing the exchange of information, including things like scholarship offers, among prospective students. This is making life harder for schools since they have better informed applicants who have more specific questions.

I think that one of the best things law schools could do is to encourage a proliferation of rankings (and not just 1-180 lists of schools, but all forms of rankings).

Consider business schools - there are multiple, major business school rankings that use different methodologies and examine different aspects of MBA programs. Prospective MBA students have better resources to assess schools than do prospective JD students. U.S. News' "rankings" of specialty programs are even more primitive than its rankings of law schools generally. I'd like to see a Business Week or Wall Street Journal ranking of business law programs, a Wired ranking of law and technology programs, and so on. Dual degree programs are currently unranked and proliferating. Let's get more information on the web and have it in a format like Jeff Stake's Ranking Game . That will let prospective students mix and match characteristics that they think are important. If they need to be educated about why certain things are important, by all means, let's educate them. But law schools ought to be embracing more data disclosure as a means of combating the influence of U.S. News.

Anthony (mail) (www):
I don't think a series of specialty rankings is the answer to the US News problem. Even if Business Week created the absolute best business law rankings possible and Wired published a flawless study of law and technology programs, the market for such niche rankings is naturally going to be smaller than the market for general law school rankings. While one could argue that there'd be so many specialized rankings that few people would find a generalist ranking like US News useful, I don't get the impression that most prospective students already know what type of law they want to do when they graduate. Not to mention that US News could easily respond to competing specialized rankings by creating more of its own specialty rankings, or improving on their (currently very poor) specialty ranking methodology.

Personally, I think the best way to deal with the US News problem is to compete directly with US News in print -- generalist rankings using significantly better measures of quality (eg. a reputation survey that has fewer flaws, or career placement stats that are actually valid). Honestly, I'm surprised this hasn't been done already, given how much time and effort so many law professors and other parties put into law review articles and blog posts on the topic. Webpages are nice, but considering how many web rankings there are out there that are even worse than US News (such as the Brennan rankings and the Brody rankings), it's not surprising why the web rankings currently out there haven't been able to achieve anything other than a small cult following when compared to US News.
8.10.2005 10:01am
Medis:
I agree with Anthony--I'm not sure there is much point to specialty rankings, given the nature of the JD.

To create a truly viable competitor, I suspect that some sort of pairing with a well-recognized media brand might be necessary (it seems the key to USNWR's success with students--and perhaps parents). Has anyone talked to the Wall Street Journal?
8.10.2005 11:35am
Public_Defender:
The problem with specialty rankings is that they don't really help the prospective student understand what it takes to practice in a given field. So what if you learn a whole lot of current technology law? The substance will change by the time you are admitted to the bar. And you have a whole career of CLE and on-the-job training to become an expert in a field.

As you can tell by my "name," I practice criminal defense. But insurance law, civil procedure and commercial transactions were the most useful classes I took in law school for my daily practice. Why? Because the professors taught the basics in how to analyze the subject, and because they were my last chance to see how "the other half" works.

After practicing for some time now, I realize how precious the time we spend in law school is——for most lawyers, it's the only chance we'll have to get a well-rounded view of the law. Too many specialty rankings could give students a misleading idea about what they need to practice any given specialty.
8.10.2005 11:42am
Aultimer:
Wouldn't the best competition to USNWR be rankings that directly reflect the potential student's information needs? More granular job placement data (summer and post-school) than %employed would obviate the need for admission and faculty data that serves as a proxy for the question of "Where should I go to school to get the job I want?"

If placement data were mapped to admission criteria (GPA/LSAT + law school X = % chance of job Y), USNWR would throw in the towel.
8.10.2005 11:53am
Anthony (mail) (www):
Aultimer, I agree that a set of rankings based on career placement data would provide the strongest competition to US News (although, as I realized when writing my article, it is extremely difficult to rank schools based on career placement since there are so many ways of doing it, each with their own inherent biases).

However I don't think it's realistic to expect US News to throw in the towel in the face of competition -- competition clearly hasn't deterred US News from ranking business schools. US News ranks law schools not because it wants to provide a public service or contribute to the academic literature, but because it wants to make a profit. As long as US News can continue to sell magazines by ranking law schools, it will continue to rank law schools.

Furthermore, having US News throw in the towel in the face of competition may not be the most desirable result. As Prof. Morriss implied in the original post, the problem isn't really US News ranking law schools, but US News having a near monopoly on the law school rankings market (let's be realistic, all the alternative niche rankings available online don't even come close to challenging US News at this time). Because US News has a monopoly, law schools gear their policies towards maximizing their US News rank. If US News disappeared off the face of the earth and the Brennan rankings became the only game in town, you'd have the same problematic situation, except instead of gaming LSAT medians, law schools would be increasing the number of chairs in the library. A better situation would involve US News and multiple other ranking schemes competing with each other, with each having roughly equivalent market shares -- assuming that each ranking emphasized different things, law schools would have far less of an incentive to base their policies around one set of rankings. In addition, the resulting competition in the rankings marketplace would cause all the ranking providers to continuously improve their methodologies. For example, if Judge Posner's clerkship placement study from a few years back became extremely popular and well known among prospective law students, to the point where it influenced thousands of enrollment decisions, you can be certain that within a couple of years US News would be incorporating clerkship placement in its own rankings.
8.10.2005 12:22pm
Brian G (mail) (www):
My law school's rankings:

2 years ago: 76
Last year: 99
This year: 69

Really, has the school fluctuated that much in just a few years?
8.10.2005 12:34pm
Challenge:
I think it's curious your answer is more rankings.

I think information--not rankings--is great. But I have found that US News just doesn't stratch the surface on the info I desire. I really don't care if a school is ranked 14 or 30, I want to know what that school is like and what my job prospects are likely to be when I graduate. The problem is that US News focuses on determing ranks, rather than just releasing tons of information. Posner is a recent article wrote that the while the rankings are useful for determing which tier to apply to, they are not useful in determining which school within that tier to utlimately attend. The answer is not more rankings but more information. Specifically more information about employment. They should list the schools based on selectivity, so students can guage their chances, but then leave the rankings up to the individual based on the employment information, cost, program specialties etc. Drop rankings and up the information.
8.10.2005 1:10pm
Richard Bellamy (mail):
While there aren't very many sub-rankings of law schools, here in Philadelphia there is just such valuable distinctions. Beyond Penn, I believe Villanova is usually ranked second in the region, but Temple is always ranked tops nationally for its "trial advocacy" program. This is drawing potential students away from Villanova who are interested in that specialty. Widener offers more night classes, so draws students with that benefit.

Soon, moreover, Drexel University will be opening a new law school, which a declared intent to specialize, I believe, in health law and IP, building upon the specializations it already has in those areas. As an unaccredited school when it opens (with an intent to gain accreditation before the first class graduates), Drexel will likely be bottom-tier in the U.S. News rankings, at least to start, but if it can carve out that niche, and attract law firms with those specialties, it should be able to move upward.
8.10.2005 1:10pm
Public_Defender:
One potential problem with the job placement survey is that it treats law school like a trade school. Business school is a trade school. Journalism school is a trade school. By contrast, law school (like medical school) is a professional school.

How do you measure a law school's ability to teach its students to be members of a profession? That's hard to do meaningfully. You could look at bar discipline rates for the extremes, but you could also look at the quality of the work produced, the ability to work cooperatively with colleagues and opposing counsel, commitment to service to the public and the bar, reputation of graduates among judges, private practitioners, and government practitioners, etc.

Yes, I'm a snob on this point. But part of the problem in the legal profession is the worship of the dollar as opposed to respecting our role as members of the bar.
8.10.2005 1:13pm
Challenge:
"For example, if Judge Posner's clerkship placement study from a few years back became extremely popular and well known among prospective law students, to the point where it influenced thousands of enrollment decisions, you can be certain that within a couple of years US News would be incorporating clerkship placement in its own rankings."

Anybody have a link to this study?
8.10.2005 1:16pm
Anthony (mail) (www):
Challenge, not sure if this ended up being the final version, but but here's a link:

Avery/Jolls/Posner/Roth clerkship paper
8.10.2005 1:25pm
Aultimer:
Anthony -

I was being a bit flip, but I don't think any rankings based on squishy proxy data like USNWR could compete with a proprietary and more comprehensive set of individual level credentials-to-employment data. Who cares about the "reputation" of a school unless that reputation translates into opportunities?

Your study is a fine start, but not every law school prospect expects or wants to get an AmLaw 50 job, and using aggregate data has real problems in this context.
What prospect cares if Columbia does a fine job placing typical students if the prospect isn't typical?
8.10.2005 1:27pm
Anthony (mail) (www):
I agree that few people care about "reputation" unless that "reputation" translates into actual opportunities, and reputation rankings become meaningless if one can rank schools based on those actual opportunities.

I also agree on your second point, although I can't think of any way around that particular problem. I think it's a given that no set of school rankings (or even raw data) is going to meet the needs of every individual. No matter what you do, there will be some sort of error or biased introduced.
8.10.2005 1:38pm
markm (mail):
"Really, has the school fluctuated that much in just a few years?" This is a result of structuring the report around rankings rather than scores that actually measure some quality. If you've got 50 schools that all score the same within the range of normal statistical fluctuations in the scoring, the ranking system still requires that one be #51 (say) and one be #100, although it's likely that there is no difference in the underlying quality you are trying to measure. Next year, with no change at all in the actual quality, the statistical fluctuations and measurement errors will put these 50 schools into an entirely different order. Shooting from #60 to #99 or back doesn't necessarily mean there was any change in that school, or in the competition.
8.10.2005 1:49pm
AJ (mail):
What's interesting to me is that the USNWR rankings have actually gotten worse rather than better in terms of providing useful info concerning job placement. When I started law school in 1996, one of the criteria in the USNWR rankings was the average private-sector starting salary for a graduate from a particular school. (I don't recollect if the average was presented in terms of the mean or the median, but either way it was useful info). My understanding is that USNWR abandoned this criterion because (a) it supposedly failed to take into account regional differences in salary levels; and (b) there was a fear that by emphasizing salaries, it would unduly penalize schools where a high number of graduates went into public interest jobs. I think both of these concerns are misplaced. Everyone agrees that the highest salaries for law school graduates are the starting salaries paid by the AMLaw 100 firms. The nature of this market is such that starting salaries tend not to vary very much in different regional areas. For instance, notwithstanding substantial differences in cost of living, starting salaries at AMLaw 100 firms are either identical, or very close, in NY, D.C., Philly, Chicago, Boston, Houston, San Francisco, LA, and Seattle. Most places are at 125K; some in the lower end of the AmLaw range pay less outside of NY, DC, and LA, but not a lot less - Usually 110K vs. 125K. The point is, info on average starting salaries is not overly distorted by regional costs of living - The regional differences show up in a more tightly compressed salary scale for associates moving up the ladder in lower-cost areas, but info on salaries for mid-level associates was never part of USNWR's rankings to begin with.
The concern about public interest placement likewise makes little sense. Incorporating starting salary info into the rankings obviously results in higher rankings for schools that place more graduates in AmLaw 100 firms, but this is as it should be. Its all about having options - As a general rule, among all the jobs available to a newly-minted JD, the toughest to get are jobs in AMLaw 100 firms. One of the reasons these jobs are so coveted is that they are a springboard to any other type of legal job one might pursue. In other words, even students interested in pursuing careers in public service should be interested in this info, because it gives them an idea of which schools give them the most options - A student at a school that regularly places AmLaw 100 grads can always skip the AmLaw 100 and go straight into a lower-paying public service job; however, schools that are unable to place students in the AmLaw 100 remove one option for their graduates, without necessarily creating a corresponding increase in the likelihood that their graduates will be able to obtain a desirable public interest position.
One thing this salary info would have to control for would be the large number of top-school students who clerk for their first one or two years - I imagine this could be accounted for in some way; If this statistic is not accounted for, it would make the average starting salary for grads from elite schools appear deceptively low. (I recognize that USNWR based its average on private sector salaries, but this alone doesn't solve the problem - If you look only at recent grads in private sector employment, you are excluding a large number of the best grads, who have accepted temporary public sector employment for the first year or two, with the understanding in many cases that they are all-but-assured of a lucrative big-firm position when their clerkship concludes).
8.10.2005 1:51pm
Public_Defender:
It would be useful for prospective students to know what % of students go to big law firms, public interest, government work, etc. But ranking based on salary demeans the legal profession. It could also lead schools to stop accepting students who express an interest in public interest work.

If money is your measure of success in your career, please, please, please go to business school. Stay away from law school. The legal profession already has too many people like you.
8.10.2005 2:36pm
Challenge:
"But ranking based on salary demeans the legal profession."

Please. Other things being equal, you want more money right? Why including a clearly relevant quality "demeans" the profession is beyond me.

However, I dont think lumping private and public salaries is useful because it obscures, rather than highlights, the information a student is searching for. They should be separated, with additional placement info for public service (federal versus state, for example).
8.10.2005 3:11pm
AJ (mail):
I agree that publishing the percentages of graduates who wind up in different employment categories would be very helpful. I also agree that there is no necessary connection between publishing such information and incorporating such information into the rankings. Presumably students could glean the "AmLaw 100" placement info I was discussing from simply seeing the raw percentages, rather than having those percentages incorporated into the rankings.

With that said, I'm not sure I agree that it is inappropriate to incorporate salary info into rankings. The purpose of rankings as I see them is not to serve a normative function in terms of bringing about the type of legal profession we might like to see. The purpose is to aid students in making informed decisions about where to go to school, but the school choice decision is basically a means to an end - The main reason school choice matters is because where you spend your 3 years in school affects where you spend the next thirty or forty years of your career. In my experience only those who come from families that never experienced financial pressure would suggest that financial considerations should be excluded from career choices. As I said in my first post, its all a matter of options - Students going to a school that places a large percentage of graduates in high-paying private sector jobs are not prevented from choosing a public service option instead - but that's what it is, a choice. Students at schools that are unsuccessful in placing graduates in such jobs don't somehow attain a corresponding benefit in their ability to get a lower-paying private sector job or a public service job. Given this fact, I have a hard time understanding why the range of employment options that are opened up by a degree from a particular school should should be ignored when ranking which schools are "best." If anything, this should be one of the most important factors, yet other than the generic "percentage employed" #, I don't think the current USNWR rankings account for this. Thus, in my view, reintegrating the "private sector salary range" criterion into the USNWR rankings would aid those rankings in performing the primary function they are presumably meant to perform - The system would still not be perfect, but it would be an improvement over the status quo.
8.10.2005 3:13pm
Bryan DB:
Challenge, other things are not equal. And so, including a salary ranking demeans the profession because it implies that lawyers will choose financial value over societal value, etc.
8.10.2005 3:21pm
sarah22 (mail):
Reputation or prestige rankings are going to be important as long as the best firms, government jobs, academic gigs, and pro bono placements chiefly use what school you go to and the grades you get as weeding tools. And it is so efficient and dependable to use those standards that I don't see how it will change.

It's not just working at 'big law' firms and making money - the best opportunities anywhere do the same thing: the DC public defender's office, for example, probably the best in the country, interviews at the top 10 schools, as do the best, most interesting public interest jobs.

Until the system changes in some dramatic way, my advice (which I followed!) will continue to be: if you want to have the best opportunities, go to the highest ranked law school you get into. (Of course, minute differences in the rankings don't matter, but top 15 or so is significantly better than #35, and top tier better than the others, etc.)
8.10.2005 3:24pm
Public_Defender:
Law schools have demonstrated that they are willing to make change admissions decision in order to increase their USNWR rankings. If those rankings included salary, then schools would turn away applicants who professed an interest in public interest law.

How about measuring how many times a school's graduates appear in appellate courts? Since appellate cases shape the law, it would be helpful to see which school's graduates appear most often in state and federal intermediate and supreme courts.

The appellate ranking would boost schools whose students argue cases at high levels, and it would punish schools whose students become discovery drones at large law firms.
8.10.2005 3:38pm
Aultimer:
Prospects need data that tie career placement to admission criteria and intent. A school that produces a class with 20 PDs who'd rather have been AmLaw associates needs to be ranked below a school that produces 20 PDs all of whom sought a PD first job on SOME list. That solves the problem of a "penalty" for a great school cranking out great public interest lawyers.

The attitude that the only law jobs worth having are AmLaw associate, NAACPDF litigator and Fed/Sup law clerk is far more demeaning to the profession than admitting that salary matters.
8.10.2005 4:10pm
Public_Defender:
In many markets, it's harder for students in top schools to get a PD, prosecutor or AG job than a job in a big firm. This is partly because local law schools often have clerk programs with PD and prosecutors' offices, making it very, very hard for an "outsider" to break in. My office doesn't do that, but we still had hundreds of applications for our last open position.

Does anyone have an idea about how to measure the quality of lawyering done by graduates of one law school or another? How about starting with each type of work, and looking at who does the best within that job category?

For example, look at associate to partner ratings at law firms. You could also look at which students graduates work their way to management positions at legal aid, public defender and prosecutors' offices. It would be more accurate but logistically more difficult to try to measure actual work performance and professionalism.

Finally, for those of you who think that higher salaries should increase a school's rank, what would you do to prevent schools from refusing to admit potential students who say they want to do public interest work?

Schools have demonstrated that they will make admissions decisions to increase their USNWR rank. So if money mattered in the rankings, schools would focus their attention on students with high income potential.
8.10.2005 4:53pm