Judge Sotomayor: More Data, and a New Conclusion.

In an earlier post, I compared Judge Sotomayor with other judges, including Judge Wood, who were ranked in a paper by Choi and Gulati. Choi and Gulati focused on three statistics—productivity (number of opinions written per year), quality (citations), and independence (propensity to disagree with same-party judges). I found that Judge Sotomayor did not compare well with the other judges, and concluded that "The bottom line is that Judge Sotomayor is about average, or maybe a bit below average, for a federal appellate judge." However, I also noted that a problem with the comparison is that I used data from the beginning of Judge Sotomayor's tenure (1999-2001), so as to make the comparison with Choi and Gulati's 1998-2000 data as fair as possible. If a "freshman effect" exists—if less experienced judges are worse than more experienced judges, which seems plausible—the comparison is not in fact a fair one.

To address this problem, Choi, Gulati, and I—with the help of Mirya Holman of Duke Law School and several Duke Law School students—collected and analyzed data from 2004 to 2006. We can no longer compare the results to the 1998-2000 data set because of possible changes in the composition of case loads and other factors, or at least we should be very cautious. Because we did not have time to collect 2004-2006 data for all court of appeals judges so that we can supply a complete ranking, we decided to compare Sotomayor with the other court of appeals judges who were rumored to be on President Obama's shortlist, and with a kind of control group consisting of court of appeals judges rumored to be on President Bush's shortlist in 2005. The results are at the bottom of this post (with apologies for the wretched formatting).

The freshman effect is vindicated: by 2004-2006 Judge Sotomayor has, on several measures, exceeded her colleagues. On total citations—a measure of general influence that reflects number of opinions as well as citations per opinion—Sotomayor ranks third after Lynch and Wood. On citations to top 20 opinions, Sotomayor ranks first. With 512 citations to top 20 opinions, Sotomayor would have been sixth in this category in Choi and Gulati's ranking (Wood's high numbers for 2004-2006 are also consistent with her performance in that ranking). On citations per opinion—which favors judges like Sotomayor who write few opinions—Judge Sotomayor dominates all of the shortlisted judges. She ranks third on law review citations per opinion.

Not everyone agrees that citation is a good measure of quality. And there may be hidden biases in the data. For example, if the tenth circuit hears less important cases than the second circuit does, a tenth circuit judge like McConnell may be cited less—because he hears disputes that are less important, not because his opinions are not good. In addition, judges in large circuits are hurt because the set of out-of-circuit opinions that could cite them is smaller than it is for judges in small circuits. Still, the numbers seem pretty robust, and the comparison here is with well-respected judges, not with ordinary judges. Sotomayor looks good.

We also checked how many cases received a Westlaw "red flag," which usually means that the Supreme Court has vacated all or part of a holding. Judge Sotomayor seems about average or better than average, controlling for the number of cases.

As in the first analysis, Judge Sotomayor does not rank high on productivity; she is at the median for this group. However, different circuits have different productivity norms. "Productivity" really means the number of published opinions; courts frequently dispose of cases with unpublished memoranda, and the practice across circuits diverges (reflecting perhaps how busy the circuit is, or judges' collective views about how their effort should be expended). In the Choi-Gulati dataset, the most productive second circuit judge published 109 decisions over three years; the least productive published 66; Sotomayor is somewhere around average or above average for the circuit. Meanwhile, Judge Wood sits in the super-productive seventh circuit. So given circuit norms, Sotomayor has more time to lavish on her opinions than Judge Wood does, and it is not surprising that her opinions are cited more often. In terms of overall quality, it is hard to distinguish the two judges.

We did not try to calculate independence scores because of lack of time. But we looked at dissents. The numbers here are so low that they do not convey much information. Sotomayor is about average. You can interpret this to mean that she is no more or less collegial than her colleagues, or that she is no more or less independent-minded than her colleagues.

All the usual qualifications, caveats, and disclaimers apply. But the data should put to rest the rumor that Judge Sotomayor is not a competent jurist. She holds her own among a highly respected group which includes the third (Wilkinson), eighth (Wood), eleventh (Lynch), and thirteenth (Jones) ranked judges on Choi and Gulati's composite ranking (Alito was sixteenth). If citations reflect quality, Sotomayor may well be one of the top appellate judges in the country.

Many thanks to the industrious Duke Law School students who collected the data: Christopher Battles, Heather Horst, Luke Ortner, Christopher-David McCurdy, and Seth Reynolds.


Outside Court Citations

Top 20 Outside Court Citations

Law Review Citations

Published Opinions

Red Flag


Cites / Maj. Opinion

L. Rev. Cites / Maj. Opinion

Dissents Per Pub. Opinion

Sotomayor [D]










Wood [D]










Lynch [D]










Garza [R]










Garland [D]










McConnell [R]










A. Williams [D]










Clement [R]










Wardlaw [D]










Jones [R]










Wilkinson [R]










Related Posts (on one page):

  1. Sotomayor vs. Alito Revisited:
  2. Judge Sotomayor: More Data, and a New Conclusion.
Obama picked her for one reason - to satisfy the very powerful bloc of Puerto Rican female voters and donors.
5.28.2009 12:20am
Allan (mail):

So, Judge Sotomayor is an impressive candidate, who, if conservatives in 2005 are to be believed should be confirmed on that fact alone.

But, if President Obama's standard is to be the one, principled conservatives can oppose her because she is a liberal (maybe not, but certainly somewhere to to the left of center).

Here's one journeyman attorney who hopes that the character assassination will focus on liberalism, rather than false assertions about credentials.
5.28.2009 12:20am
Thanks for the fascinating, and fairminded, followup. The numbers aren't everything but they tell a large part of the story. As for her "productivity"- hey, she's heading for a court known for its puzzlingly small caseload.
5.28.2009 12:22am
I think enough people sufficiently trashed the utility of these rankings in the last thread that there's not much reason to continue piling on here. Either you think putting random numbers together has meaning or you don't.

I will say, however, that the newest addition to this exercise is particularly bizarre: the inclusion of westlaw's "red flag." The red flag is described by Posner as "usually" meaning that the "Supreme Court has vacated all or part of a holding." That, of course, is not true. A red flag can mean lots of things. For example, in statutory interpretation cases, a red flag can pop up when the statute is amended, repealed, superseded, or held either unconstitutional or preempted. A red flag will also pop up when the case (or its reasoning) is repudiated in a later en banc case from the circuit. In short, assuming, counter factually, that the Supreme Court reversing "all or part" of a holding from an opinion you signed means you are a "worse" judge, the red flag is not a particularly great indicator of that fact.

As I mentioned in the earlier thread, this is just another example of the phenomenon of adding numbers to something to make the evaluation appear objective. But it's nonsense and is indicative of nothing.
5.28.2009 12:45am
Leo Marvin (mail):
Thanks for setting the example of honest self-criticism.
5.28.2009 2:45am
Jon Roland (mail) (www):
Jorge Hirsch came up with the "h-index" that seems to work in scientific fields: Order the cited works of the subject in descending order of number of citations to it, then count down until you reach the number n of works beyond which the number of citations to them are < n. The number n is the h-index.

The problem with using the h-index for legal opinions is that a legal citation is not always to a supporting work, whereas in scientific papers it usually is. One might need to do a sub-count of supporting and nonsupporting cites. A high index of nonsupporting cites might indicate a high level of disagreement with the cited opinion.
5.28.2009 9:09am
Anderson (mail):
Not everyone agrees that citation is a good measure of quality.

A case is frequently cited because it happens to address a particular issue that hasn't arisen in other cases, not because its author or panel has some distinction.

I can believe that there are some "superstar" judges like Posner whom lawyers have actually heard of, and expect that judge might have heard of too; but I'm dubious that celebrity and quality walk hand in hand.
5.28.2009 9:54am
Matt B. (mail):
I would be curious if Posner and Easterbrook continue to destroy everyone else to the extent they did in the original study.
5.28.2009 10:15am

Choi and Gulati focused on three statistics—productivity (number of opinions written per year), quality (citations), and independence (propensity to disagree with same-party judges)

It is astonishing that you think these criteria provide any meaningful guide.

Who cares how many opinions a judge cranks out? How are the cases assigned? How many pages per opinion? How complex were the various cases? Did Sotomayor set the calendar for argument or was it another judge? Unless a matter sits undecided for over a year then then Sotomayor was deciding exactly as many cases as she was assigned.

Citations? Seriously? Why not just try to find out if she sat at the popular kids' table in grammar school?

Disagreement with same party judgment? Jesus that is jaw-droppingly stupid. What is Sotomayor? Dem or Rep? Does it matter that she was first appointed by a Rep to the trial bench? The idea that the is some supposed intellectual consistency between all republicans or all democrats is nuts. Jeff Sessons/Arlen Spector anyone. Oh sorry Specter is now a Democrat. Brian Tester and Nancy Pelosi?
5.28.2009 12:52pm
Phillip Hallam-Baker (www):
I think it is getting to the point that we just have to declare the confirmation debate bankrupt and confirm Sotomayor as soon as possible before the GOP does even more damage to itself.

When Alito was nominated we were told that it was wrong to bring up his membership in a racist organization founded to keep women and minorities out of Princeton. Now that a Latina is nominated the same folk are telling us that it is OK to call her a racist because she must be as all Latinos are racists.

At this point analysis of her judicial abilities is irrelevant, the GOP cannot examine her record without appearing to join the 'racist' attack. Rush Limbaugh, Gingrich and co have poisoned this debate and there is no rescuing it.

It is like when MoveOn published their full page 'Betray-us' ad on the eve of Petraeus reporting on the surge in Iraq. The GOP was seriously worried about that hearing, but the MoveOn ad gave them a pass.

The President was elected with an overwhelming mandate. He gets to choose whoever he likes to sit on the court. The only useful output from the argument over Sotomayor is to evaluate Obama, not Sotomayor.

It is not a conspicuously stupid or gimmicky pick as is evidenced by the fact that the whispering campaign started during the vetting process. If she had really been as unqualified as the RNC claimed the announcement would have been a complete surprise.
5.30.2009 11:09am
SciTrek (mail):
This is my first entry into a site like this. I'm a midwest science teacher for 25 yrs+, teaching from 8th grade to college. I've long had an interest in Constitutional governance based on the Preamble. Politics fouls effective governance. Three years of teaching American History up to WW I fed that interest.

The 3 statistics used in the article above are certainly of interest. It seems a true challenge to find objective methods of ranking the truest quality of a judge which, to me, would be predictive of the judge's grand wisdom over the long term.

PRODUCTIVITY- this is no good. An unbiased, thoughtful and studious judge dealing w/ complicated cases &top lawyers would have much lower productivity than a politically-charged judge with an itchy trigger-finger ego. Or a judge who has aspirations &who thinks productivity will help in advancement could manipulate/expedite his/her caseload to polish that statistic.

CITATION- (I am not very confident on the meanings of the term in this legal conversation. I trust folks here will clarify if I have it wrong.) Surely there are legal databases of landmark or mile-marker (?) cases that have withstood the test of time and cultural change... like foundational scientific theories. Over time, science theories have been tweaked and clarified due to new evidence that has been internationally (interdenominationally) challenged, tested, skeptically scrutinized, and retested.

Could politicized judges, or their advisors, make preferred lists of citations that support their perspectives, leaving off those that don't? One judge might have a list supporting a Living Law perspective of the Constitution, while another judge has a list of Constitutionally literalist citations. They could have absolutely opposite decisions on a case but both would get high rankings using statistical citations.

Meanwhile another judge w/ a brilliant new insight (by historic measurement of the future), that is independent of citation dogma, gets a low ranking.

INDEPENDENCE- This smacks of the "cowboy" or "maverick" political spin of many recent &long past campaigns. What if Judge A had a long record of disagreeing with his peers on topics where the peers were proven to have been historically "correct"? Meanwhile Judge B's record shows disagreement with his peers when history shows that the peers were wrong. Judge A is awful but gets a high independence rating, while Judge B is a brilliant futurist yet gets a low independence rating.

SUMMATION- Even if the 3 stats above can be mathematically correlated with the mass of "good judges" it would be totally misapplied to any individual. General "truths" must be limited to general populations for there are exceptional individuals who may have been forged from uncommon experiences. Seems reasonable to write that nominees to the US Supreme Court would be exceptional, yes?

How do you folks rate the greatest judges of history, Supreme Court or otherwise? Where did they come from? What were the personal characteristics and values that made them great- that made them ahead of their times? What are the ranges for each characteristic?

Maybe those qualities can be (have been) analyzed and statistically crunched.
5.30.2009 12:05pm

Post as: [Register] [Log In]

Remember info?

If you have a comment about spelling, typos, or format errors, please e-mail the poster directly rather than posting a comment.

Comment Policy: We reserve the right to edit or delete comments, and in extreme cases to ban commenters, at our discretion. Comments must be relevant and civil (and, especially, free of name-calling). We think of comment threads like dinner parties at our homes. If you make the party unpleasant for us or for others, we'd rather you went elsewhere. We're happy to see a wide range of viewpoints, but we want all of them to be expressed as politely as possible.

We realize that such a comment policy can never be evenly enforced, because we can't possibly monitor every comment equally well. Hundreds of comments are posted every day here, and we don't read them all. Those we read, we read with different degrees of attention, and in different moods. We try to be fair, but we make no promises.

And remember, it's a big Internet. If you think we were mistaken in removing your post (or, in extreme cases, in removing you) -- or if you prefer a more free-for-all approach -- there are surely plenty of ways you can still get your views out.