pageok
pageok
pageok
The John G. Roberts Umpire Watch:
Is Chief Justice John Roberts an "umpire," a "servant of the law" who merely "applies the law" rather than "makes the law" to his personal preferences? Or is he a political conservative who will always vote for the conservative cause in an ideologically-charged case? In his confirmation hearings, Roberts expressed hope that he would be an umpire. In his cases Last Term, however, Roberts tended to vote consistently for the conservative side.

  In light of the uncertainty, I propose a new feature here at the Volokh Conspiracy: The John G. Roberts Umpire Watch. When a new decision comes down, we'll make a judgment call about whether it was one of the Court's ideologically-charged cases. (This will require umpire-like judgment, perhaps, but I think we can probably identify most of the cases relatively straighforwardly based on how most of the Justices voted.) If it was, then we'll classify Roberts' vote as either politically conservative or politically liberal.

  Then, at the end of the Term, we'll tally up the numbers. If we see votes on both sides, it will support the case that Roberts is an "umpire." Someone truly following principle will go wherever that principle will take him, and that should led to mixed views from a political standpoint. On the other hand, if all the votes end up being conservative, that will support the notion that Roberts is voting to make the law to shape his personal preferences rather than merely following it.

  I should add: Yes, of course, this kind of vote-tallying is imperfect. There are some circumstances in which a neutral umpire would always vote a particular way. I'm reminded of the pro se appellant who argued that the trial judge was obviously biased: after all, the judge had denied 100% of his motions! Still, I think that's less likely given the wide mix of cases the court hears. Even if a particular legal principle has a political valence, the mix of cases on the Court's docket is so diverse that few principles explain more than one or two votes. Given that, I think the tally should give us interesting insights into how Chief Justice Roberts approaches his job.
Kevin P. (mail):
You are conflating someone who merely "applies the law" with someone whose decisions are politically conservative or politically liberal. You're not going to get very good results from this exercise. It is possible for a decision to merely apply the law with a politically conservative or politically liberal result.

Perhaps a better exercise would be to analyze the decision and see if it actually merely applies the law and tally up votes like that.
11.28.2007 5:27pm
Kevin P. (mail):
As a further thought - with your decision point being politically conservative or liberal, only Justice Kennedy is going to emerge as an umpire.
11.28.2007 5:29pm
OrinKerr:
Kevin P.

No, I'm not conflating the two: indeed, I devote the entire last paragraph of my post to that precise issue.

The primary difficulty with your recommendation is that you really need to spend a lot of time becoming an expert in an area to know which side best applies the law. Unfortunately I don't have time to do that for every case.
11.28.2007 5:30pm
tvk:
I think we can predict before tallying that there will be many more conservative than liberal votes (especially if you narrow the population to "ideologically charged" cases). The imperfection is the big caveat here. If you believe that the Republican party platform coincides amazingly with the original meaning of the constitution, then a conservative leaning court is a neutral umpire. Vice versa with a liberal court.

Because we have no good baseline of what is a "correct" or "neutral" decision, the baseline that we refer to by default is the median Justice, Justice Kennedy.

Of course, we might see whether Roberts is intellectually consistent. That is, if he believes in free speech in one case and not in another, there might be a problem. But any skilled lawyer can simply recharacterize the level of generality to form a neutral sounding principle. For example, a rule that "ambiguity is construed in free speech's favor unless the speaker is talking about 'bong hits for Jesus'" is a perfectly neutral rule and internally consistent from one point of view. Who is to say that Roberts is incorrect and not "umpiring' to adopt it?
11.28.2007 5:33pm
Anderson (mail):
For example, a rule that "ambiguity is construed in free speech's favor unless the speaker is talking about 'bong hits for Jesus'" is a perfectly neutral rule and internally consistent from one point of view.

Niiice.
11.28.2007 5:35pm
GV_:
I think this exercise is doomed to fail. That being said, instead of waiting for the opinion to come out, why not classify the case as a political hot potato after oral argument. That way, there won't be any attempt at trying to say a case is or is not politically sensitive because of the way Roberts votes.

Did Roberts vote "liberal" in any of the controversial cases last term? I can't think of any.
11.28.2007 5:44pm
OrinKerr:
GV_

What does it mean for the exercise to fail?
11.28.2007 5:50pm
Skyler (mail) (www):
Orin, you're essentially saying "Yes, I know that my methodology has no basis in reason, but I think the results will be valid nonetheless."

Wow, I hope you don't build any bridges anytime soon using that awful type of logic.

It reminds me of American Textile Manufacturers Inst. v Donovan (45 U.S. 490), where Brennan, never a source of much logic, decided that where a study commissioned by an agency grossly overestimated the costs of a modest rule change, then that study was appropriate to use as a guide for a more strict rule change. Yeah, if Brennan built a bridge, I'd recommend the ferry.

But I guess that's what suffices for logic at the highest levels of our nation, so I guess it comes honestly among lawyers.
11.28.2007 5:55pm
Dave N (mail):
I guess this exercise will be based on a simple tally. In how many 5-4 (or 6-3) decisions did the Chief join with the usual suspects? (and you can decide for yourself which usual suspects I am referring to).
11.28.2007 5:55pm
Federal Dog:
"Someone truly following principle will go wherever that principle will take him, and that should led to mixed views from a political standpoint."

Why? Someone following principle will go wherever that principle will take him, period. I do not see what logic supports additional belief that this will lead to mixed political views.
11.28.2007 5:57pm
tvk:
Orin's proposal has been done by Cass Sunstein and others and critiqued. To rehash the debate, whether Roberts is faithful to his "umpire" characterization is difficult because in any case being analyzed, there are three questions:

1. What the "correct" principle of law is.
2. What Roberts believes the correct principle of law is.
3. Whether Roberts applies what he believes to be the correct principle of law, or applies some other principle because of his personal policy preferences.

(1) is basically impossible to discern, for obvious reasons.

Now, if (2) deviates from (1), that is not probative of bad faith, but shows that Roberts would be a bad justice. But, since (1) is impossible to discern, it is likewise impossible to prove that (2) deviates (1).

What Orin is trying to discern is whether (3) deviates from (2). That is, if Roberts believes that the law is X, but decides the law is Y because it leads to a preferable outcome, then he is not being an umpire.

The problem here is that we have no way of disentangling (2) from (3). Short of Roberts writing in his diary that "I thought the correct rule was X but I decided by impose Y by fiat instead," we have no way of discerning a divide between (2) and (3). Roberts may vote very conservatively because he believes the law is very conservative.

What measures of justices' votes normally do is conflate (1)-(3). That is, we assume that justices believe the correct law to be what the author believes the correct law to be. The argument then goes that a decision that does not follow this correct law is either because the justice is incompetent (a deviation of 2 from 1), or acting in bad faith (a deviation of 3 from 2), or both. But unless we have a measure of either (1) or (2) independent of (3)--that is, unless we can objectively determine what the correct law is, or what a justice believes the correct law is, independently of how they actually vote, there is no point of reference to make a comparison.

Orin's exercise is to place the median justice as the definition of (1). That is, the political center will define what the correct law is. But that underlying premise is highly debatable.
11.28.2007 6:00pm
Bart (mail):
Orin:

Your exercise measures political outcomes, not proper application of the law. While too many judges and justices coflate political outcomes with proper application of the law, these are two very different things.

You are also erroneously coflating Justice Roberts giving a roughly equal number of liberal and conservative political outcomes with "neutrality" (i.e. having no political opinions) when in fact such a result is more indicative of simply having political opinions on both sides of our ideological divide.
11.28.2007 6:01pm
OrinKerr:
Skyler writes:
Orin, you're essentially saying "Yes, I know that my methodology has no basis in reason, but I think the results will be valid nonetheless."

Wow, I hope you don't build any bridges anytime soon using that awful type of logic.
I'm confused. I said the methodology was imperfect but still useful and reasonable. You then said that this is "essentially saying I know that my methodology has no basis in reason." Perhaps you misunderstood the common meaning of the word "essentially"?
11.28.2007 6:04pm
OrinKerr:
Bart,

Isn't your second paragraph a direct contradiction of your first?
11.28.2007 6:06pm
wekt:
Does this project assume that an unbiased judge would rule roughly 50% liberal and 50% conservative? If so, I must disagree, at least for issues regarding the Constitution.

I think that the 'correct' decision for most Constitutional issues would turn out to be politically 'conservative' by today's standards, with the exception of the 'war' on terror, the 'war' on drugs, and other issues involving a degree of federal authority that the Founders would have shuddered at.
11.28.2007 6:13pm
G.R.:
Orin, I admire your intestinal fortitude even as I doubt the wisdom of this particular endeavor.
11.28.2007 6:15pm
Skyler (mail) (www):
Orin,

Like I said, please don't build any bridges!

If you can't see how using a method that has no basis in logic or reason can't return a valid result, then you certainly belong in law, not civil engineering because only a lawyer would think that garbage in equals pearls out.
11.28.2007 6:15pm
Jiffy:
TVK: I think you are conflating "making the law to his personal preferences" with "bad faith." Roberts might act politically rather than truly following principle without consciously recognizing he's doing so. In fact, my guess is that this is mostly what goes on when, for example, conservatives ignore the federalism implications of a federal constitutional amendment to outlaw gay marriage or, in the same context, liberals get all excited about states' rights. To return to the "umpire" metaphor, while I might try to be objective, I'm not sure I'd be the most even-handed umpire for my favorite team (something that is apparent to anyone who has seen me yelling at the umps during a baseball game).
11.28.2007 6:22pm
Jestak (mail):
I'm skeptical of the whole "umpiring" analogy as regards judging. There was a good post over at Balkinization on the issue a few weeks back; I don't have time to search for it right now but I will try to track it down later.
11.28.2007 6:24pm
Latinist:
I think people are right to raise the problem that Roberts might just think that conservatives are generally righter than liberals about how to interpret laws. But there are at least some areas where a difference should be visible.

For example, I think a general conservative opinion is more or less: "people get money for ridiculous pseudo-injuries from corporations that aren't at fault; these judgments need to be further restrained." The corresponding liberal position would be something like: "corporations often try to cheat individuals with fewer resources; we should make sure the lawsuits by cheated individuals don't get thrown out just because high-paid lawyers come up with silly technicalities." (Obviously, beliefs on both sides are in fact more complicated.) Everyone would agree that there are some legitimate and some illegitimate lawsuits: the question is how many of each.

So a biased, political conservative judge might just side with the defendant every time, in order to promote his political cause (that's how elected politicians usually act), whereas a conservative "umpire" would occasionally decide "here's one of those rare cases where the plaintiff has a point."

Of course, no one's perfectly pure or utterly biased, so it'll be hard to measure, but Orin's procedure sounds like it could be useful.
11.28.2007 6:24pm
Zacharias (mail):
Orin, you and your cute lawyers can take a flying flue. If appellate judges were chosen from among the non-bendovers instead of state certified lawyers, your motions might well be 100% rejected too.

You are a member of the good-ol-boys club. Nothing more.
11.28.2007 6:25pm
BruceM (mail) (www):
I really wish someone would do this for judge Sharon Keller of the Texas Court of Criminal Appeals. She votes in favor of the state/prosecution every single time, no matter what. And this is a court of discretionary jurisdiction (for the most part). I'd love for someone to actually compile statistics for her votes. In the rare situations when the court rules in favor of the criminal defendant, judge Keller dissents every time. I'm sure she votes for the state at least 99% of the time. I'd love to see actual statistics on it, though. It might be more like 99.99% pro-state. Even when there is a (very rare) per curiam opinion in favor of the defendant, judge Keller will not sign on and dissent without comment. I think once, about 4-5 years ago, there was a unanimous decision in favor of the defendant which she may have signed on to, but I could be wrong. If so, it was probably a case where the state confessed error.
11.28.2007 6:27pm
OrinKerr:
Skyler,

For someone with such a commitment to logic and reason, you seem unusually willing to simply assume your conclusion. (And besides, I earned an A+ in my mechanics of solids class, so I think my bridges are probably staying up. ;-) )
11.28.2007 6:27pm
Skyler (mail) (www):
Okay, Orin. I must be missing your point. Can you explain it please?
11.28.2007 6:31pm
another skeptic:
Prof. Kerr,

How would you classify Wisconsin Right to Life? I saw most media classifying the result as "conservative," either because it was a win for a Right to Life group, or because it was portrayed as a win for "corporate" contributions to campaigns.

But wasn't free speech once considered liberal?

I know, many people say campaign finance is different, but those critics should take a look at the amicus list in WRTL. WRTL was supported not only by those crazy right-wingers at the ACLU and the AFL-CIO, but also by the Alliance for Justice! So the leading Left-side group in the confirmation wars was supporting the view ultimately adopted by several Justices whose confirmations they opposed!

I don't see how something chalked up as a "win" by the Alliance and the AFL-CIO can be classified as politically "conservative."

Same thing goes for many of the speech cases, such as Hill v. Colorado, which upehld anti-abortion speech restrictions, or Bong Hits, in which many conservative groups backed the student (because conservative students are often the ones in trouble with principals).

Also, State Farm and all the punitive damages cases are fun tests — Stevens backs corporate America against injured plaintiffs, and Scalia and Thomas are on the other side? So Roberts might get tagged with a politically conservative tally for joining Stevens over Scalia?

So how would you classify such cases?
11.28.2007 6:38pm
tvk:
Jiffy, your example is a mix of both types of deviation that I raised. That is, a justice may get a decision wrong because he is purely incompetent (believes the law to be Y when it is X), purely corrupt (holds the law to be Y even though he believes it is X), or partly incompetent and partly corrupt (kind of believes the law to be Y, and is subconsciously pushed over the edge to actually hold it because he prefers Y as policy, even though ultimately X is the correct law).
11.28.2007 6:42pm
Davide:
Orin:
Please explain the following:

(1) "When a new decision comes down, we'll make a judgment call about whether it was one of the Court's ideologically-charged cases. (This will require umpire-like judgment, perhaps, but I think we can probably identify most of the cases relatively straighforwardly based on how most of the Justices voted."

Please explain how this calculus will work precisely -- for example, if Scalia and Thomas vote on one side, does that make it "ideologically charged," for you? Otherwise, isn't this some sort of ad hoc after the fact conclusion that will be manipulated by personal values?

(2) "If it was, then we'll classify Roberts' vote as either politically conservative or politically liberal."

Please explain (a) how you'll classify the result; and (b) how you'll determine whether the decision should have been that way or not.

(3) "Then, at the end of the Term, we'll tally up the numbers. If we see votes on both sides, it will support the case that Roberts is an "umpire." Someone truly following principle will go wherever that principle will take him, and that should led to mixed views from a political standpoint."

Please explain how you justify assuming (a) the "proper" distribution of judgments ex ante; (b) the percentage you think an "umpire" would have on each side (liberal/conservative); and (c) what acceptable deviation, if any, you think there should be.

I'm interested in your responses, particularly to (3).

If you can't answer in a principled way, please explain what your exercise will show.
11.28.2007 6:46pm
Steve:
I would think the ACLU's position ought to be a pretty reliable signifier of the liberal side of any case, regardless of whether they might happen to be supporting a conservative organization at the time.
11.28.2007 6:48pm
Le Messurier (mail):
OK's next comment should be "Never mind".
Wanna take odds that he can't give it up?
11.28.2007 6:57pm
Cold Warrior:
What a bunch of complainers ... I think it's a fun idea.

I will admit that there's a problem in how to characterize decisions, and that "liberal vs. conservative" probably isn't the best way to break things down. So maybe we won't find out anything useful if we call decisions "conservative" or "liberal."

But let's stick to the umpire analogy. There are home plate umps famous for giving pitchers the high strike, there are home plate umps know as having a tight strike zone, etc. His claim seems to be that he's not a results-oriented jurist; he just calls 'em as he sees 'em. So if we perceive a bias toward calling the low and outside pitch a strike, well, then we've learned something interesting.
11.28.2007 6:58pm
frankcross (mail):
This methodology does not make any assumption that the median justice is correct. It does make an assumption that the legally "correct" result will not be uniformly ideological. That the "correct" result will be sometimes liberal, sometimes conservative and that, over the long haul of cases, about equally one or the other. If that assumption is correct, a systematic ideological pattern of decisions would reflect a bias.

Obviously, the assumption may be incorrect, in either direction. However, there is no neutral definition of "correct" decision. There is much disagreement, often characteristic of the evaluator's politics. I would say an assumption that the "correct" outcomes are equally distributed is certainly as reasonable as an assumption that the "correct" outcomes are what a given evaluator thinks correct.

However, step one should be made ex ante, not after viewing how the justices voted.
11.28.2007 6:59pm
GV_:
What does it mean for the exercise to fail?

That we learn nothing useful from the data.
11.28.2007 7:08pm
G.R.:
With regard to the methodology, I suspect that Orin has in mind something similar to David Shapiro's well-known analysis in "Mr. Justice Rehnquist: A preliminary view." 90 Harv. Law Rev. 293 (1976). If I recall correctly, and based on some quick web searches of abstracts -- I haven't gone to Westlaw for this -- Shapiro's criticism was that Rehnquist without exception resolved (or voted to resolve) disputes between the government and the individual for the government, disputes between the federal government and the states for the states, and disputes over federal jurisdiction against the existence of jurisdiction; and that this adversely affected the quality of Rehnquist's jurisprudence.

Orin hasn't (yet) defined his criteria as precisely as Shapiro did, but there is a general precedent for his methodology.
11.28.2007 7:11pm
Anonymouseducator (mail) (www):
Roberts himself raised the whole "umpire" issue. What did he mean by it?
11.28.2007 7:12pm
A.:
The idea that Roberts is to be a neutral umpire is posited in contrast to some bogeyman of "ideological, activist judges." If we believe that any of the other justices approach this strange species, then we can look to Roberts' record of voting with or against the "biased" justices. If he consistently votes with (or against) one of the more biased justices, there is reason to infer bias (unless the bias of the biased justice is to vote wrong every time, rather than systematically to the left or right of the "correct" answer, which seems silly). Such an analysis becomes even clearer when all the justices to each other, and when looking only at cases that weren't unanimous.
11.28.2007 7:13pm
AF:
This post makes me think of this article, in which Supreme Court expert Tom Goldstein writes:


As one of the most conservative in recent memory, the just-completed 2006 Supreme Court term has served as a rallying cry for progressives. But will the Roberts Court help bring Democratic voters to the polls for the 2008 election? Because the public's interest in the court is notoriously weak and its memory short, the upcoming term is more relevant to whether the court can be a mobilizing force. And the cases lined up for the new term, which begins next week, strongly suggest that the highest-profile decisions will actually make the court look liberal.

The upshot is that the upcoming term won't end as this one did, with headlines and TV reports describing the court as profoundly conservative, triggering praise from the right and howls of protest from the left. Instead, we will see (mistaken) talk of the court's "surprising" tack back to the left. In fact, this commentary will be wrong: The justices and their views will be exactly the same come June 2008; it is the cases that will be different. But the misleading sense of direction that's likely when the term ends next June could make the court a galvanizing campaign issue for Republicans, not Democrats.


If Goldstein is right, this term is not a good year to perform Orin's experiment. Unless, of course, the point is to get a "misleading sense of direction."
11.28.2007 7:15pm
Christopher Cooke (mail):
Orin

I think you are running into some objections by people who think the results will undermine their pre-conceived opinions about Chief Justice Roberts, or who don't want to know the results of your imperfect poll because it might cause others to know this information about him. I see no harm in forging ahead with the study, and then we can see if it provides useful or inaccurate/unhelpful information. This is similar to what Sunstein did, except he focused on the Court as a whole in trying to determine who were the most "activitist" justices.
11.28.2007 7:26pm
AF:
If we see votes on both sides, it will support the case that Roberts is an "umpire." Someone truly following principle will go wherever that principle will take him, and that should led to mixed views from a political standpoint. On the other hand, if all the votes end up being conservative, that will support the notion that Roberts is voting to make the law to shape his personal preferences rather than merely following it.

I disagree with this analysis, because it is an election year. We already know, as Orin acknowledges, that last year "Roberts tended to vote consistently for the conservative side." If that changes this year, it does not necessarily support the conclusion that Roberts has become an "umpire." It equally supports the conclusion that he has an eye on the election.
11.28.2007 7:35pm
John Jenkins (mail):
@Christopher Cooke,

Actually I think what Prof. Kerr is running up against is the fact that this exercise says more about the evaluators than the person being evaluated. If [name your favorite conservative] does the evaluation, it makes Chief Justice Roberts look like a principled jurist. If [name your favorite liberal] does it, it makes Chief Justice Roberts look like an ardent conservative, principles be damned.

The reverse would be true for Justice Ginsburg

As an analysis of the VC readers, it might be an interesting exercise. As an analysis of CJ Roberts, it probably won't be.
11.28.2007 7:39pm
eric (mail):
Why just Roberts?

Do it for everyone. I think it would be interesting. My predictions, for left to right: Stevens, Ginsburg, Breyer, Souter, Kennedy, Alito, Roberts, Thomas, Scalia.
11.28.2007 7:43pm
GV_:
Roberts tended to vote consistently for the conservative side.

Why use the qualifiers "tended to" and "consistently"? When did he not vote for the "conservative side"?
11.28.2007 7:47pm
bittern (mail):
Would it help things if we made the strike zone more than one-dimensional? Can we V.C. umpires normally decide if a non-unanimous ruling/dissent:

A. Claims a purely textualist/originalist basis?
B. Seems primarily a matter of following precedent?
C. Favors the more powerful?
D. Favors feds/executive/police?

This would be similar to evaluating whether a baseball umpire calls the top and bottom and two sides of the strike zone, for lefty and righty hitters, the same for two teams. I think this would work better, but cannot name the fancy maths that we would use.
11.28.2007 7:54pm
bittern (mail):
Eric asks Why just Roberts?

You could do everybody, but we'd be holding some up to a test they never asked for. My poorly informed understanding is that neither Stevens, Breyer, Kennedy, nor Alito ever claimed to be baseball umps.
11.28.2007 7:58pm
eric (mail):
Bittern,

Yes, but the umpire quote is ambiguous at best. A neutral judge might vote 65% conservative or liberal and still be neutral. Plus, it gives us context.

Moreover, this informal study could be used to counter Sunstein's poorly designed agency intensive version.
11.28.2007 8:03pm
eric (mail):
To clarify the above post, I know Sunstein was looking for "judical activism" (as he defined it). This informal study would give me a left to right look at the court. It would not really counter Sunstein.

I also propose eliminating 9-0 decisions from scrutiny. These should be viewed as so clear that there is no ideological tilt to them. It make no senese to call a 9-0 decision conservative because it rejected a bad liberal argument.
11.28.2007 8:07pm
hattio1:
Those arguing that this will tell more about the evaluater than about Roberts come up against a significant problem. VC readers, while tending conservative, are fairly well represented across the political spectrum. If everybody on VC finds it a "political" case, it probably is. And remember, we're not asking what the "correct" decision is in any one case, we're asking was it liberal or conservative. Once again, given the fairly broad spectrum, that shouldnt' be hard.
Remember, Professor Kerr is not saying that HE will decide which cases are political and which outcomes were conservative or liberal. He was stating that "we" will, presumably the VC readership.
11.28.2007 8:11pm
Tareeq (www):
It would help to have some definition of what "conservative" and "liberal" mean in this context. Scalia's vote in Texas v. Johnson, the classic example of what I think Orin means, seems "liberal" and when it's discussed in the context of appellate decisionmaking might be held up as that of a "neutral umpire." But change the facts slightly (Fred Phelps burning a cross at Bill Clinton's funeral?), and it's perhaps a "conservative" vote. Orin's idea is an interesting one, but at its heart seems founded on an assumption that the nine nerds in whom we vest so much power aren't as conscientious as we are.
11.28.2007 8:13pm
hattio1:
Must agree with Eric on unanimous decisions.
11.28.2007 8:14pm
bittern (mail):
eric, I quite agree that a hypothetical nonbiased judging protocol (e.g., always follow precedent) would give something other than a 50-50 liberal-conservative split. The umpire claim is that he isn't biased, that his rulings are some mixture of following precedent, text, and original intent, without picking the combination on the basis of which side wins. What I think folks would love to debate is whether Roberts is biased. That's why I proposed the multidimensional scoring. One could try to see how much of his conservative lean is explicable by factors appropriate for an umpire. In a baseball game, perhaps one pitcher throws high and his opponent throws inside; by sorting out those categories, you can get a good idea whether a high rate of called strikes is due to bias.
11.28.2007 8:28pm
2L:
There is something circular about your test. So if the justices vote how we'd "expect" them to vote, it's ideologically-charged and thus he'd be a conservative. If they don't, it's then not an "ideologically-charged" decision, and thus they can't vote "inconsistently" with how we'd expect them to vote. So basically, there's no way, under your test, that Roberts won't be a "conservative."
11.28.2007 8:44pm
Duffy Pratt (mail):
Why are people sweating whether Roberts is a neutral umpire? Almost by definition, cases that reach the Supreme Court have precedents, and arguments following from those precedents, which would make it fairly reasonable to decide the case either way. If, under these circumstance, a judge is following some sort of neutral principle, its almost certainly not one that is mandated by law. Pretending otherwise seems kind of silly.
11.28.2007 8:51pm
juris_imprudent (mail):
Are you people kidding me? 50 odd posts and nobody brings up the difference in the strike zone between the AL and the NL?
11.28.2007 10:04pm
Joshua:
wekt (some distance farther up the page): Does this project assume that an unbiased judge would rule roughly 50% liberal and 50% conservative? If so, I must disagree, at least for issues regarding the Constitution.

Indeed, Orin's project speaks to what I like to call the "Great Unspoken Assumption" of U.S. Supreme Court judicial confirmation politics at all levels, one that seems to be shared by liberals and conservatives alike: namely, that strictly applying the law and the Constitution, and eschewing judicial activism, tends to result in a jurisprudence that is, on the whole, more favorable to conservatives than to liberals. This Great Unspoken Assumption implies that there's really no such thing as a "neutral umpire", as being neutral in a legal sense naturally skews toward conservatism in an ideological sense.

Why would conservative politicians, and the conservative constituencies who elect them, so enthusiastically favor "neutral umpire" judicial nominees, as opposed to more overtly conservative activist ones, if they didn't believe legally neutral judges and justices could be counted upon to deliver generally ideologically conservative results?
11.28.2007 10:12pm
OrinKerr:
Cooke writes:
I think you are running into some objections by people who think the results will undermine their pre-conceived opinions about Chief Justice Roberts, or who don't want to know the results of your imperfect poll because it might cause others to know this information about him.
Based on the nerve I seem to have hit, I tend to think that's probably right. All I'm doing is taking a well-established methodology and doing it (a) as the Term is unfolding rather than ex post and (b) in a prominent forum not in a law review. Funny that some people seem to object so strenuously.
11.28.2007 10:18pm
OrinKerr:
Davide,

Please explain why you need explanations on those items.
11.28.2007 10:21pm
JosephSlater (mail):
Roberts himself raised the whole "umpire" issue. What did he mean by it?

I think Annoymouseducator gets the point.
11.28.2007 10:22pm
Bart (mail):

Bart: Your exercise measures political outcomes, not proper application of the law. While too many judges and justices coflate political outcomes with proper application of the law, these are two very different things.

You are also erroneously coflating Justice Roberts giving a roughly equal number of liberal and conservative political outcomes with "neutrality" (i.e. having no political opinions) when in fact such a result is more indicative of simply having political opinions on both sides of our ideological divide.

OrinKerr: Bart, Isn't your second paragraph a direct contradiction of your first?

How so?

My first paragraph argued that political outcomes are not the same as the proper application of the law.

My second paragraph observed that your measurement of "neutrality" (not applying your own opinions to a decision) is not the same as applying your own opinions to decisions to arrive at a roughly equal number of "liberal" and "conservative" political outcomes.

These paragraphs do not address the same issues. I do not see how they contradict one another.
11.28.2007 10:38pm
Jed Adam Gross (mail):
Orin,

You may be employing a well-established methodology for assessing jurists, but is a parallel method used to assess umpires?
11.28.2007 10:39pm
OrinKerr:
Bart: Sorry, i inverted your 1st paragraph when I read it -- my error.
11.28.2007 10:41pm
Skyler (mail) (www):
Orin justified,

All I'm doing is taking a well-established methodology and doing it (a) as the Term is unfolding rather than ex post and (b) in a prominent forum not in a law review. Funny that some people seem to object so strenuously.


If everyone else jumped in front of a truck, would you jump too?

I'm not a conservative, I don't know anything about Roberts, and I don't know of any other people using this methodology. But that others use it doesn't make it right.
11.28.2007 10:42pm
Gerg:
Doesn't this amount to a cynical assumption that politics is never principled?
11.28.2007 10:54pm
OrinKerr:
Skyler,

You seem to imagine that I argued, "my methodology is okay because it has been used before." Read my comment again, and you will see that I do not make that argument.
11.28.2007 11:37pm
Thoughtful (mail):
Orin,

At the end of a baseball season, should we determine whether or not the umpires were neutral or biased based on how close they come to calling 50/50 on balls and strikes? Or do I misunderstand your methodology?
11.29.2007 12:46am
Laura S.:
Orin,

You present two bits of information:

1) Roberts consistently stands opposed to Stevens, Ginsburg, Souter, and Breyer.
2) Roberts claimed to be an umpire, not a ideologue

From which you then conclude, "#1 implies #2 is false"

What an assumption. We ought to consider that BGSS are the four horsemen of the left. Thus #2 implies the BGSS are not moderates but actually ideologues. Thus, #1 is irrelevant.
11.29.2007 1:12am
Laura S.:
Lets go back to Ginsburg reading her dissent from the bench. Roberts sits there shocked at the melodramatic histrionics that the four horsemen have taken to using to deceive the public into believing that the court is at risk of becoming radically conservative.

What I'm wondering is: being a rather well connected and aware law professor why are you falling for this tripe?

Or are you being open-minded to those left leaning people within your readers who persistently nag you into abandoning reason by insulting your integrity?
11.29.2007 1:28am
Latinist:
Thoughtful:
I don't think Orin said anything about expecting 50/50 results. But without having an exact sense beforehand of how often conservatives are right, we can still get something from the information. If you found an umpire had called, say, 99% strikes all year, you'd probably suspect he wasn't being fair; even more so if you found he had called 99% strikes only when the Yankees were pitching.
11.29.2007 1:28am
KRS:
Under this metric, a self-aggrandizing squish who is more interested in his own power and in writing florid prose than in rendering decisions based on anything that resembles law might also look like an "umpire"... A blind buffoon who stands behind the plate calling the game being played in his head will probably not look biased, but that doesn't make him a respectable umpire.

On most cases, the court is unanimous or close to unanimous. The 7-2 and 9-0 cases sometimes have "liberal" results and sometimes have "conservative results. Some of them may be politically charged, but the answers are clear enough, and all 9 of the Justices probably come out of those looking like umpires.

In the politically charged cases, though, that present tough and relatively open questions, on the margins conservative judicial philosophies tend to correlate with conservative results and liberal judicial philosophies tend to correlate with liberal results. Criminal defendants and individual plaintiffs need judges who will ignore the text of the laws and do "justice" in light of the noble purpose that they presume is there. Large corporations get their legislation through Congress through bribes, apathy, or lobbying, and then they look to textualist judges to enforce that legislation to the letter. Judges who want to know what "equal protection" and "cruel" meant in 1868 and 1787 are going to be less likely to scale back the death penalty and recognize novel forms of discrimination than judges who want to know what they can get away with making up in light of the current fashion among elites.

There are exceptions, of course. The Apprendi/Blakely line of cases produced some interesting bedfellows, as did Kelo, and the Scalia/Stevens dissent in Hamdi was quite impressive (and perhaps umpire-like).

Given the court's docket, and given the sorts of cases you will probably label as "ideologically charged," JGR will probably come out looking like an "ideologue" under your criteria. I don't think it follows that he's unprincipled or not an "umpire" somehow.
11.29.2007 1:58am
egn (mail):

Lets go back to Ginsburg reading her dissent from the bench. Roberts sits there shocked at the melodramatic histrionics that the four horsemen have taken to using to deceive the public into believing that the court is at risk of becoming radically conservative.


I hate to bring in a paraphrase of the "Clinton did it" retort, but I don't suppose that Scalia reading his hateful Lawrence dissent from the bench was "melodramatic histrionics used to deceive the public into believing that Justice Kennedy is going to personally start marrying gays"?
11.29.2007 2:02am
AF:
All I'm doing is taking a well-established methodology and doing it (a) as the Term is unfolding rather than ex post and (b) in a prominent forum not in a law review. Funny that some people seem to object so strenuously.

My problem with the exercise is not with its methodology, but that it proposes to draw conclusions about CJ Roberts' jurisprudence based on a single term, which also happens to be an election year. The fact is that Roberts has been consistently conservative for two terms. If he is less conservative this year, that will not provide strong evidence that he is not a conservative. Again, I refer to Tom Goldstein's article, which based on an analysis of the cases already granted this term predicts that

we will see (mistaken) talk of the court's "surprising" tack back to the left. In fact, this commentary will be wrong: The justices and their views will be exactly the same come June 2008; it is the cases that will be different. But the misleading sense of direction that's likely when the term ends next June could make the court a galvanizing campaign issue for Republicans, not Democrats.

I will stop short of suggesting that this exercise is being proposed in order to generate such a "surprising" result. But if Goldstein -- probably the single most prominent student of the Supreme Court in the country -- is to believed, doing this experiment this term will lead to misleading results.
11.29.2007 2:05am
OrinKerr:
AF,

There's no reason this would only be done for one year. It only occurred to me this year because last year Roberts' votes were so surprisingly predictable.
11.29.2007 2:23am
Laura S.:
Egn writes:

I hate to bring in a paraphrase of the "Clinton did it" retort, but I don't suppose that Scalia reading his hateful Lawrence dissent from the bench was "melodramatic histrionics used to deceive the public into believing that Justice Kennedy is going to personally start marrying gays"?

Scalia's Lawrence dissent is indeed another example of melodramatic histrionics. My point isn't not "oooh Ginsburg was deceptive". My point is that reacting to what happened last term as if it informs our knowledge of who Roberts is--is silly.

The horseman had their theatrics just like Scalia did (for the same reason).

Orin has even discussed this:
http://volokh.com/posts/1193341791.shtml#285331

Maybe they learned from each other. Either way: The strong split between Roberts and the four horsemen isn't meaningful. BGSS aren't playing fair, if they seem 'moderate' its only some kind of red-shift effect of the course moving away from them. But they aren't moderate.

Ergo, their consistent 5-4 siding against the Roberts is a sign of their ideology not of Roberts bias.
11.29.2007 2:44am
Steve:
It's amusing how both sides are raising all sorts of anticipatory objections because they're afraid it won't come out their way.

If someone would just stipulate that "umpire" is one of those things you say in order to get confirmed, we could probably dispense with the whole operation.
11.29.2007 2:51am
Visitor Again:
The strong split between Roberts and the four horsemen isn't meaningful. BGSS aren't playing fair, if they seem 'moderate' its only some kind of red-shift effect of the course moving away from them. But they aren't moderate.

Sure they're moderate, and only someone with a very limited perspective of a few years or a very short memory would suggest otherwise.
11.29.2007 3:11am
OrinKerr:
Laura S.,

I'm curious: What's your test for determining if a conservative is just voting for whatever position is conservative, as opposed to being principled? Is your view that there is no distinction, as anyone who is principled will always, 100% of the time, vote in a conservative direction? I mean really, 100% of the time, with no exceptions, ever, under any circumstances?
11.29.2007 3:11am
tvk:
Orin, isn't your test backwards? That is, the question is not whether someone who is principled must vote 100% of the time in a conservative direction; the question is whether someone who is principled cannot vote 100% of the time in the conservative direction. If Chief Justice Roberts can vote 100% of the time in a conservative direction and yet remain principled, then the fact that he does so would tell us very little. In statistics, this is the single-member population problem.
11.29.2007 5:07am
tvk:
Example borrowed from wikipedia (http://en.wikipedia.org/wiki/Prosecutor%27s_fallacy):

You win the lottery jackpot. You are then charged with having cheated, for instance with having bribed lottery officials. At the trial, the prosecutor points out that winning the lottery without cheating is extremely unlikely, and that therefore your being innocent must be comparably unlikely.

CJR rules consistently for the conservative side. He is then charged with being unprincipled. Democrats point out that consistently ruling for the conservative side while remaining principled is extremely unlikely (but not impossible); and therefore the chance of CJR being principled must be comparably unlikely.
11.29.2007 5:47am
Ralph Phelan (mail):
All I'm doing is taking a well-established methodology...

(1) Choosing which data you're going to analyze after you've collected it and looked at it is not a "well-established methodology" anywhere outside of parapsychology research.

You have got to decide if a case is political before the ruling comes out or all you are going to measure is your own unocnscious biases.

(2) By assuming that a neutral umpire would rule for liberals as often as for conservatives you are building a tendentious conclusion into your study's methodolgy. I know that doing so is a "well-established methodology" in the social "sciences," but I still don't like it.

Neither your nor Cass &Sunstein's studies would mean anything if it were in fact true that liberals are more likely than conservatives to bend the rules to get the policy outcome they want. And how else can one explain Roe vs. Wade?
11.29.2007 8:19am
Ralph Phelan (mail):
I don't suppose that Scalia reading his hateful Lawrence dissent from the bench was "melodramatic histrionics used to deceive the public into believing that Justice Kennedy is going to personally start marrying gays"?

Maybe so.
So what?
We're discussing Roberts, not Scalia.
Got an example of Roberts doing it?
11.29.2007 8:24am
Ralph Phelan (mail):
It's amusing how both sides are raising all sorts of anticipatory objections because they're afraid it won't come out their way.
Maybe both sides are raising objections because such a badly designed study is a loose cannon that will only add noise and random damage to the debate.
11.29.2007 8:28am
Ralph Phelan (mail):
I maight add that in addition to the anticipatory objections we've seen suggestions to improve the plan to the point of usefulness, such as deciding which cases are "political" before the decisions come down, and replacing the extremely crude "liberal/conservative" analysis with:
A. Claims a purely textualist/originalist basis?
B. Seems primarily a matter of following precedent?
C. Favors the more powerful?
D. Favors feds/executive/police?
11.29.2007 8:33am
libertarian soldier (mail):
But aren't you limiting your assessment to a degree that will greatly limit its accuracy? Take grants for cert: if Roberts is consistantly in a losing group of three with Scalia and Thomas, wouldn't that say one thing, while being consistantly in the six + majority say something else? And if your response is "well, regardless of its value, its rarely possible to know who voted how on cert petitions so we can't count them", doesn't that mean you are making a proposal that will be knowingly inaccurate and hence not useful?
11.29.2007 8:33am
Skyler (mail) (www):
I find it hard to believe that any justice will vote on any basis besides principle. I may not like their principles and I might ascribe evil to their results and perhaps their motives, but I never for one minute believe that they don't think that they are acting according to their own principles.

What you're trying to do is meaningless, based on nothing, to get a result that is misleading. You might as well just come to the conclusion that you want to reach now, because it's just as valid.
11.29.2007 8:45am
blcjr (mail):
It seems to me that right in the first paragraph of Orin's post is a fatal assumption:
Is Chief Justice John Roberts an "umpire," a "servant of the law" who merely "applies the law" rather than "makes the law" to his personal preferences? Or is he a political conservative who will always vote for the conservative cause in an ideologically-charged case? In his confirmation hearings, Roberts expressed hope that he would be an umpire. In his cases Last Term, however, Roberts tended to vote consistently for the conservative side.
Quite a few people believe that "applies the law" is more or less synonymous with "conservative" and "makes the law" is more or less synonymous with "liberal." If so, then Roberts tending to vote with the conservatives simply proves that he's doing what he said he'd do.

The fatal assumption is in assuming that "applies the law" will result either result in decisions that are politically neutral, or will result -- on average -- in just as many decisions that are politically liberal as decisions that are politically conservative. If that were the case, "applying the law" versus "making the law" would not be such a political touchstone for conservatives.

Not to in any way denigrate Justice Roberts, but his response (however the question was framed) avoided the question nicely, in a way that would sound good to anyone who was politically conservative. Saying that he would "apply the law" rather than "vote conservative" was just doublespeak for saying that he would "vote conservative." Or at least that's the way it sounded to this conservative.
11.29.2007 9:02am
Ralph Phelan (mail):
And if your response is "well, regardless of its value, its rarely possible to know who voted how on cert petitions so we can't count them", doesn't that mean you are making a proposal that will be knowingly inaccurate and hence not useful?
Incomplete information is the norm in most fields of scholarship, and is generally better than nothing.

Skyler's objection is much more on point: the question is not whether justices are principled but whether their principles are good ones. Trying to determine observationally what those prinicples are is good idea. But you have to do it well. If you start out measuring only liberal/conservative, then the answer you get will of course be one of "liberal, moderate or conservative" and you will have learning nothing about "principle". At best you will measure the correlation of Roberts' position on the liberal/conservative scale with that of the court as a whole. How that relates to "principle" is a theoretical question whose answer less than obvious.

Kerr, please take the word of a guy who spends his days staring at reams of manufacturing data trying to figure out what parameters matter and why - this is an interesting idea, but it needs a lot of work.
11.29.2007 9:04am
Grange95 (mail):

Someone truly following principle will go wherever that principle will take him, and that should [lead] to mixed views from a political standpoint. On the other hand, if all the votes end up being conservative, that will support the notion that Roberts is voting to make the law to shape his personal preferences rather than merely following it.


As a lawyer and a long-time basketball referee (and occasional rec league umpire), I think you misunderstand how to determine if a referee/umpire is biased. The most important thing for an umpire is to be consistent in making judgment calls. To use the balls/strikes analogy, the key is for the umpire to make the strike zone as consistent as possible so that all the players know what is or is not a strike. That strike zone might favor one pitcher over another, or one team's hitters over the other's, but we wouldn't consider that to be a sign of a biased umpire. If the umpire changes his strike zone to favor or disfavor a particular team, or if the umpire ignores rules to assist a team he prefers, then we can say the umpire is biased. Simply counting up the balls and strikes for each team is irrelevant to determining bias, unless you can review each ball/strike call on its own merits (or in basketball, coaches/fans often incorrectly focus on total foul counts to show bias, when in fact you would need to review each call/no-call on its merits to detect bias; the referee could call fouls consistently in a manner that happened to favor one team's style of play, but yet was not biased).

Applying this analogy to the court, CJ Roberts could set a consistently conservative "strike zone" for his interpretation of laws or for his application of Constitutional principles, resulting in his vote being consistently "conservative"; this would not necessarily mean CJ Roberts is more or less biased than any other Justice. Actually, in some ways, a consistent conservative approach from CJ Roberts is more "umpire-like" than the "case by case" vacillation from some of the "centrist" judges (Justice Kennedy and ex-Justice O'Connor, perhaps?).

That all being said, I think the umpire analogy has a certain shallow appeal, but it really doesn't hold up too well to deeper application. After all, umpires and referees don't write the rules they are interpreting, and have to answer to supervisors for bad calls or incompetence.
11.29.2007 9:24am
krs:
I'm never posting again when I'm tired... I'm glad to see that my post didn't provoke a response... much of it was just wrong and I shouldn't have posted it. Suffice it to say that I think this experiment will make Justice Roberts look un-umpire-like, and I think that result will be misleading... though for lots of reasons other than the ones I posted.
11.29.2007 10:18am
Steven H (mail):
I think the usefulness of this study largely turns on whether you believe that the first two sentences of the post present a true dichotomy:

Is Chief Justice John Roberts an "umpire," a "servant of the law" who merely "applies the law" rather than "makes the law" to his personal preferences? Or is he a political conservative who will always vote for the conservative cause in an ideologically-charged case?


If it is the case that a true umpire would NOT always vote conservative in ideologically-charged cases, then we could compare the Chief Justice's practice with that of a true umpire by taking account of whether he ever voted against his political preferences.

But even then, there are several problems. First, when the Chief Justice fails to vote "conservative," he may still be voting on personal preference: people (even Justices) are complex, and even the most "conservative" Justice may disagree with the party line on some issue. Second, as pointed out above, the Chief Justice may believe that the conservative answer IS the law. He may even be right. Thus, he could be a true umpire even voting 100% conservative in ideologically-charged cases. In fact, I think that even if any observer would posit that the Chief Justice is simply applying his own policy preferences, the Chief Justice would nonetheless believe that he was consistently applying the law.

At the end of the day, it's very hard to know what we could even mean by "applying the law" v. "making the law," since so much of the Court's job is to "make" law. If one takes seriously the idea that circuits split (at least sometimes) because equally good answers are available to a given legal question, then it's to hard give any account of what the Court does that doesn't involve policy.
11.29.2007 10:44am
OrinKerr:
Skyler writes:
I find it hard to believe that any justice will vote on any basis besides principle. I may not like their principles and I might ascribe evil to their results and perhaps their motives, but I never for one minute believe that they don't think that they are acting according to their own principles.

What you're trying to do is meaningless, based on nothing, to get a result that is misleading. You might as well just come to the conclusion that you want to reach now, because it's just as valid.
I'm curious: Do you think I have come to such a silly idea because (a) I lack your deep understanding of the Supreme Court and how the Justices decide cases, (b) I am not nearly as smart as you, or (c) I have some sort of secret motive (and if I have a secret motive, what do you think that secret motive is?).
11.29.2007 10:48am
Dave N (mail):
To choose an important case this term, which view is "conservative" and which one is "liberal" in Medellin v. Texas? The crux of the case is how the treaty power (consular notification) works within the context of state's rights and federalism.

The Bush Administration intervened on Medellin's side. While I realize that this case is a classic "power of the federal government/power of the states" case, can anyone elucidate which vote Roberts might cast would be ideological and which one would be principled?
11.29.2007 10:54am
frankcross (mail):
This little analysis is imperfect, but it's a blog not peer reviewed literature.

Here's a more rigorous fact. The Penn Law Review compared the ability of a statistical program to predict votes (that program used just a couple of variables, most importantly justice ideology) and legal experts predicting votes. The statistics did somewhat better. Ideology, whether measured by press assessments or past ideologically consistent voting patterns is demonstrably the best predictor of Supreme Court votes.
11.29.2007 10:55am
Steve:
CJR rules consistently for the conservative side. He is then charged with being unprincipled. Democrats point out that consistently ruling for the conservative side while remaining principled is extremely unlikely (but not impossible); and therefore the chance of CJR being principled must be comparably unlikely.

Isn't the fallacy obvious? Someone is going to win the lottery, even though it's very unlikely that it would be any specific person. But that doesn't mean there's always going to be some Supreme Court Justice who votes for the conservative side every time.
11.29.2007 10:55am
Prufrock765 (mail):
Grange made a great point--one of the hallmarks of a good official (my sports are wrestling and football) and a good judge is consistency. Consistency is not the ONLY such characteristic but it is a necessary one. So how much consistency is too much?

Why exclude 9-0 decisions? Why not 8-1? And would your answer change if it were, mutatis mutandis, a death penalty case and former Justice T. Marshall was the "one"?

Can someone at least give an actual example of a case (even on the DC Circuit) where JGR has violated the "umpire" test so that we can at least start to evaluate the methodology?
11.29.2007 11:04am
JosephSlater (mail):
Quite a few people believe that "applies the law" is more or less synonymous with "conservative" and "makes the law" is more or less synonymous with "liberal."

Hmm. Quite a few people don't believe either of those claims.
11.29.2007 11:05am
AF:
There's no reason this would only be done for one year. It only occurred to me this year because last year Roberts' votes were so surprisingly predictable.

Fair enough. There are two things I am unclear about, however. First, is the claim that an ex ante analysis is more reliable than an ex post analysis? I do not think that is the true in this context; any selection bias comes from the Court's own certiorari process, not the study design (which can include all cases decided by the Court). Second, is the claim that our "judgment call" about whether a case is "one of the Court's ideologically-charged cases" is more accurate than objective measures like voting patterns? I would question that claim. It seems to me that voting patterns are a more accurate indicator of genuine controversy than the judgment call of Volokh Conspiracy readers.
11.29.2007 11:26am
Jake (Guest):
The baseline problem seems to me to be a fatal flaw to this proposal. However, since Kerr seems determined to treat the baseline criticism as an ad hominem attack, let me suggest another problem:

Posit a test like the one proposed by Kerr, except that instead of an ad hoc determination of "ideological" cases, we adopt a strict definition: a case is an ideological case if it is a five-four decision in which Breyer, Souter, Stevens, and Ginsburg are on one side.

Roberts is a "good umpire" if these cases came out fifty-fifty, and a "bad umpire"/"lying hypocrite" if he is consistently on the anti-BSSG side.

Anybody spot the problem with this? Does the term "selection bias" jump unbidden to your mind? Do we have any reason to think that an ad hoc "ideological case" filter will remedy the problem?
11.29.2007 11:45am
CJColucci:
But what kind of umpire is CJ Roberts supposed to be? The story is told about Hall of Fame umpire Bill Klem that when he was out drinking with his fellow umpires, one said: "I calls 'em like I sees 'em." The second responded: "I calls 'em like they are." Klem put down his drinked, paused, and said: "They ain't nothin' 'til I calls 'em."
11.29.2007 12:12pm
Ralph Phelan (mail):
Quite a few people believe that "applies the law" is more or less synonymous with "conservative" and "makes the law" is more or less synonymous with "liberal."

Hmm. Quite a few people don't believe either of those claims.

Therefore an experiment intended to prove or disprove the above hypothesis would be extremely useful.

But an experminent whose design assumes an answer is totally useless.

Kerr has implicitly assumed that the above is false (an unbiased umpire should rule both ways equally often). If he finds Roberts ruling "conservative" more than half the time he has learned nothing beyond the fact that Roberts rules conservative more than half the time. Any conclusion that Roberts has a conservative bias would come not from the data but from Kerr's implicit assumptions.

Now if Kerr took Bittern's suggestion of scoring "textualism" and "precedent" for various decisions, and measured the degree to which Roberts' preference for textualism over precedent or vice versa depends on whose ox is being gored, then we might actually learn something.

But if all you measure is "liberal" vs. "conservative" you don't have enough degrees of freedom to draw any conclusions about anything else.
11.29.2007 12:15pm
Ralph Phelan (mail):
First, is the claim that an ex ante analysis is more reliable than an ex post analysis? I do not think that is the true in this context; any selection bias comes from the Court's own certiorari process, not the study design (which can include all cases decided by the Court).

Kerr initially said he was going to select what he considered "politically charged" cases, which would have to be done ex-ante to avoid Kerr injecting his own selection biases. If he does all cases, or defines an objective procedure for deciding which cases he analyzes (e.g. number and indentity of amicus briefs) he can safely do an ex post analysis.

The certiori problem is unavoidable.
11.29.2007 12:19pm
Davide:
Davide,

Please explain why you need explanations on those items.

Orin:

With regard to (1), without your explanation, I have no way to understand whether you have any principled basis whatsoever to decide if a case is "ideologically-charged." Further, without an explanation, there is little way to know if your methodology is sound.

With regard to (2), if you can't explain how you will classify Roberts' vote, then it seems likely that your methodology is unsound.

With regard to (3), if you can't explain (a) the "proper" distribution of judgments ex ante; (b) the percentage you think an "umpire" would have on each side (liberal/conservative); and (c) what acceptable deviation, if any, you think there should be, then you have apparently conceded the futility of your exercise and its lack of a supportable methodology.

If you think I'm off, I asked you to explain what you think you your exercise will show. You didn't. I'd love to hear an explanation that justifies it despite an inability to answer the preceding queries.
11.29.2007 12:22pm
Skyler (mail) (www):
Orin, clearly you're smart and to say otherwise would be foolish. I would even say that you're smarter than me, and that's not a very bold assertion either. I ascribe no secret motive to your actions either.

But being smart doesn't make you right. Einstein believed in a god. He was very smart, but that didn't mean that his conclusion was improved by his intelligence. It was simply a belief that he held absent concrete proof. Many might agree with him that there is a god, but they're still wrong, no matter how smart.

Likewise, you may think that this methodology will yield results that have meaning. It is based on nothing but pure hope on your part, not on any basis in reality.

Here, you're assuming that errors that need to be corrected by the Supreme Court will be evenly biased either for one political spectrum or another. It doesn't take into account any potential trend of the court to correct decades of influence by the one political philosophy. You're assuming that lower courts errors will be evenly biased and that the courts will select errors to be corrected in an evenly biased manner.
11.29.2007 12:24pm
Davide:
Orin,

Most specifically, without principled answers to the preceding queries, I do not think that your assessment
will "give us interesting insights into how Chief Justice Roberts approaches his job," as you claim. Rather, I think it will provide unimportant and uninsightful answers to queries that the Harvard Law Review already answers in each annual issue, i.e. with what judges does Roberts typically vote. That is a mundane assessment of little value in determining (beyond the obvious numbers) how Roberts approaches his job.

On the negative side, it would seem like fodder for generalizations of little value (he's a "conservative" in voting cases, a "liberal" in ERISA cases, etc). Such generalized assessments will come in time anyway from general conversation. Academic review, I thought, was supposed to generate better insight.
11.29.2007 12:27pm
Ralph Phelan (mail):
If Umpire Roberts calls fewer balls against Daisuke Matsuzaka than he does against Jae Seo, is he biased against Seo, or does Seo just suck?

If your study can't answer the equivalent question, what's the point?
11.29.2007 12:38pm
Ralph Phelan (mail):
Skyler said:
Orin, clearly you're smart
But he's also clearly not a statistician, and seems unwilling to consider the possibility that he's out of his depth.
11.29.2007 12:42pm
Duffy Pratt (mail):
The constitution and the laws have enabled a two party system, under which conservatives and liberals co-exist in a shifting balance. Our system has thrived because it can encompass both a ruling party and an opposition party, and indeed sometimes has a bit of each at the same time. Given this structure, I think its fairly reasonable to assume that an "umpire" who simply followed the law would not give results that skewed to far in favor of one side or the other. If the law did that, then one side of the political spectrum would vanish or be squashed (which is what happens in one party systems).

Also, the question isn't whether Roberts is following principle. The question is whether, as he claims, he is simply following the law. As I've said before, his claim is absurd and misunderstands what the Supreme Court does.
11.29.2007 12:47pm
OrinKerr:
Davide, Skyler,

You both sincerely puzzle me. Davide, your request is rather absurd. All I have done is announce that I plan to conduct a study; you respond by insisting that you must hear all of the details about the study before it conducted. How strange.

Skyler, you seem blissfully unaware that I am way ahead of you on this; you repeatedly accuse me of taking positions that I have never taken (indeed, I have expressly disavowed, repeatedly), and yet you never seem to back down.

More broadly, both of you are missing the main point here. I am not arguing that a 50/50 distribution proves neutrality. Rather, I'm suggesting that evidence that someone votes 100% in one direction 100% of the time may provide support for the view that this person is probably not being 100% neutral.
11.29.2007 12:49pm
Skyler (mail) (www):
Well, we'll see what happens, I guess.
11.29.2007 12:56pm
Ralph Phelan (mail):
I am not arguing that a 50/50 distribution proves neutrality. Rather, I'm suggesting that evidence that someone votes 100% in one direction 100% of the time may provide support for the view that this person is probably not being 100% neutral.

I'm having trouble making anthing meaningful of this distinction, given that the number's probably going to be somewhere in between. If you find that Roberts votes "conservative" 60% of the time, what if anything will that mean?
11.29.2007 1:01pm
OrinKerr:
Ralph, if the number is in between, then that means there are votes on both sides. The original post explained the significance of that, I believe.
11.29.2007 1:06pm
Ralph Phelan (mail):
Given this structure, I think its fairly reasonable to assume that an "umpire" who simply followed the law would not give results that skewed to far in favor of one side or the other.

Given my belief that one of the parties has systematically and deliberately twisted and ignored the law over the last four decades, and that the Supreme Court should spend the next few years reversing a huge backlog of dumb decision, I do not think it's reasonable to so assume.

Therefore showing that Roberts votes in a way that favors the party I consider more honest (and that I recognize many other people don't) will supply no information to settle the difference in opinion between us.
11.29.2007 1:07pm
Ralph Phelan (mail):
Ralph, if the number is in between, then that means there are votes on both sides. The original post explained the significance of that, I believe.
If we see votes on both sides, it will support the case that Roberts is an "umpire."

So, anything short of 100% conservative and he's an "umpire"? This makes your study even more sensitive to ex-post selection bias than I thought.
11.29.2007 1:10pm
A.:
Orin, why do you think that Roberts' record compared to your judgment of the ideological nature and slant of a decision is more revealing of his neutrality (or lack thereof) than a comparison of his record to that of other justices? Is it because the latter approach would necessitate a judgment on the neutrality of the other judges to be most revealing (which judgment, however, could be made by each reader/blogger for himself)?
11.29.2007 1:19pm
PLR:
Therefore showing that Roberts votes in a way that favors the party I consider more honest (and that I recognize many other people don't) will supply no information to settle the difference in opinion between us.

It's not intended to settle your differences.
So, anything short of 100% conservative and he's an "umpire"? This makes your study even more sensitive to ex-post selection bias than I thought.

Huh?
11.29.2007 1:20pm
Laura S.:

I'm curious: What's your test for determining if a conservative is just voting for whatever position is conservative, as opposed to being principled? Is your view that there is no distinction, as anyone who is principled will always, 100% of the time, vote in a conservative direction? I mean really, 100% of the time, with no exceptions, ever, under any circumstances?

Well such a person would be principled, but I wouldn't regard their positions as being internally consistent--in particular the welding of classical economics with progressive moral intervention seems strained to me.

Perhaps to satisfy your intellectual curiosity you should focus on when Roberts ends up in the minority.
11.29.2007 1:28pm
OrinKerr:
Laura S.,

While I appreciate the fact that you responded, I fear that I do not comprehend what you just wrote.
11.29.2007 1:38pm
A.S.:
I'm not very interested in the result of the experiment (a Justice who votes more often than average for the conservative side is not necessarily being unprincipled or un-umpire-like; it likely simply means that the umpire has a principled but conservative strike zone). I'm more interested in seeing what cases Orin believes are "ideologically-charged" and which votes in those cases are supposed to be conservative or liberal. As far as I can tell, we don't have to wait for the cases to be decided for Orin to do that.

So how about it, Orin - why don't you post a list of the cases accepted so far this term and tell us (a) whether the case is "ideologically-charged" and (b) which votes are conservative or liberal?
11.29.2007 2:06pm
tvk:
Orin,

"someone votes 100% in one direction 100% of the time may provide support for the view that this person is probably not being 100% neutral"

The problem is that you have to define "neutral" somehow, and right now you are dodging that question. Right now, you are implicitly defining "neutral" as "not 100% conservative"; but you can't justify why you think the 100% conservative platform is "incorrect" or "unprincipled" in some sense.
11.29.2007 2:20pm
Ralph Phelan (mail):
So, anything short of 100% conservative and he's an "umpire"? This makes your study even more sensitive to ex-post selection bias than I thought.


Huh?

Kerr said what he thinks it does or does not mean if Roberts votes exactly 50% conservative, and what he thinks it means if Roberts votes exactly 100% conservative. I asked him what it would mean if he voted somewhere in between, using 60% as an example.

Kerr replied "Ralph, if the number is in between, then that means there are votes on both sides. The original post explained the significance of that, I believe."

The only thing I could find in his original post that I think he might be referring to is the phrase "If we see votes on both sides, it will support the case that Roberts is an "umpire.""

If this is what he meant, then just one "liberal" vote is enough to 'support the case the Roberts is an "umpire."'

I hope that's not what he meant. Because if it is, then the most important determinant of his result is sample size: If Roberts votes liberal once in a very long while, then the larger your sample the more likely you are to catch him doing it and call him an "umpire." When you're designing a study, you really, really don't want to anwser to be dependent on sample size. Significance level yes (of course), but reducing the sample size shouldn't change what answer you're most likely to get.

If Kerr didn't mean the above, but instead meant 'the more the votes are evenly distributed between both sides, the better the evidence Roberts is an "umpire,"' then he is implicitly assuming a perfect umpire would vote 50/50.

If he's not saying:
(1) The probability that Roberts is a fair umpire is (100% - fraction of conservative votes)*2 [the "fair umpires break 50/50" interpretation]
nor is he saying:
(2) The probability that Roberts is a fair umpire is 100% if he's ever seen to vote against the conservative position [the "if you watch him long enough he'll look like a fair umpire" interpretation]

then just what the heck is he saying?

What does it mean if Roberts gets a 60% conservative voting record? What if it's 95% Does Kerr even know? It's best to decide what a given result would mean before the data is collected.
11.29.2007 2:26pm
srg:
I think many, perhaps most, affirmative action policies are a bad idea but constitutional.

I think this whole thread boils down to: why does it appear to be so rare (is it really?) that Supreme Court justices are willing to uphold laws of which they disapprove, and vice-versa?
11.29.2007 2:26pm
strategichamlet (mail):
"I think this whole thread boils down to: why does it appear to be so rare (is it really?) that Supreme Court justices are willing to uphold laws of which they disapprove, and vice-versa?"

I think the point of this exercise is that Roberts with his umpire analogy implied that while other justices won't uphold laws of which they disapprove/strike down laws which they like, HE would be substantially more impartial. The question we want to answer then is, is he really different, or (it seems more likely) was this umpire stuff just a lot of hooey.
11.29.2007 2:35pm
Davide:
Orin:

You wrote, "your request is rather absurd."

One could say that about your proposed exercise. I thought commenter on the Volokh board should be civil. In that effort, I will refrain from responding to this comment.

Next, you wrote:

"All I have done is announce that I plan to conduct a study; you respond by insisting that you must hear all of the details about the study before it conducted. How strange."

That is incorrect. I asked you some fairly basic questions about your proposed exercise, which you apparently cannot answer. They were, to repeat:

(1) how you know whether a case is "ideologically-charged" at all. This is a fundamental issue to your study, not a "detail." You first wrote that doing so "will require umpire-like judgment, perhaps, but I think we can probably identify most of the cases relatively straighforwardly based on how most of the Justices voted."

Really? I don't think you can. I asked you to explain it and you can't/won't. An inability to do this moots your entire project. Refusing to explain it surely doesn't make your effort seem more legitimate.

(2) I next asked you to explain how you'd categorize a Roberts decision as either "conservative" or "liberal." Again, this is no "detail." It's the central tenet of your project. An inability to explain it, again, seems to show your effort is not useful.

(3) Finally, I asked you if you could explain how you know the baseline distribution of decisions absent partisan tilt. Your refusal/inability to answer again does not bode well for your project.

Finally, you claim that I am "missing the main point here. I am not arguing that a 50/50 distribution proves neutrality. Rather, I'm suggesting that evidence that someone votes 100% in one direction 100% of the time may provide support for the view that this person is probably not being 100% neutral."

This comment misses my point completely.

My point is that your study lacks an articulable, principled basis. If it has such a basis, you can't or wont' provide it. My point, therefore, is that your study has little to no value and that its conclusions will not, pace your claim, "give us interesting insights into how Chief Justice Roberts approaches his job." On the contrary: it seems unlikely to do so and likely only to add uninformative adjectives to his decisions ("liberal" or "conservative").

Your point, Orin, misses mine utterly. You do not defend your study at all. Instead, you claim a trivial truth: someone who votes 100% of the time in one "direction" (whatever THAT is) isn't "100% neutral."

This point is of little to no value. You have not defined(and cannot define?) "direction"; you have not defined (and cannot define?) "neutral." Your measure is thus unsound.

your inability and/or unwillingness to answer fairly basic questions about your study (and your apparent hostility to questions about it) are, to use your word, "strange," particularly when they come from someone in an academic field.
11.29.2007 2:45pm
OrinKerr:
Davide,

You appear to assume that my decision to decline to answer a series of seemingly hostile questions from an anoymous blog commenter is evidence that I have no response to them. I think I get the game: you presumably know that I will not take the time to answer all of your interrogatories, so you can then announce victory when you ask more than I will answer. (Ah, New York litigators.)

In any event, the empirical study inthis essay has a good example of the rough category of inquiry I have in mind. It's on a different topic, the question of "judicial activism" in constitutional decisions rather than politics in decisions generally. Still, the basic methodology is similar. Here's what I wrote about OT2001, in response to the claim that the Rehnquist Court was a conservative activist court:
IT'S ALSO A MISTAKE TO ASSUME THAT EVERY REHNQUIST Court decision striking down a legislative act features the conservative justices acting over the dissents of more liberal colleagues. Based on my review of the last Supreme Court term, it's more often the other way around. I recently examined last year's Supreme Court cases looking for decisions in which the more conservative and more liberal justices disagreed about the constitutionality of existing laws or administrative acts. In cases that split the justices into relatively predictable ideological camps, I asked, which group voted to invalidate the other branches more often? The more conservative justices (William Rehnquist, Antonin Scalia, Clarence Thomas, and sometimes Anthony Kennedy and Sandra Day O'Connor) or the more liberal ones (David Souter, Stephen Breyer, John Paul Stevens, and Ruth Bader Ginsburg)?

Roughly a dozen of the court's 83 cases involved fairly clear ideological splits on the scope of constitutional rights. (Many cases did not involve constitutional questions at all, and those that did usually failed to produce neat camps of left versus right.) Within the dozen or so cases, however, the more liberal justices favored striking down the other branch of government almost twice as often as the conservatives did. The trend was strongest in criminal cases.

For example, the more liberal justices struck down state laws regulating the right to counsel for defendants facing suspended sentences in Alabama v. Shelton, invalidated laws that allowed for capital punishment of mentally retarded defendants in Atkins v. Virginia, and voted against a Kansas law designed to rehabilitate sex offenders in prison in McKune v. Lile. Similarly, the liberal justices voted to block Cleveland's school choice program in Zelman v. Simmons-Harris and voted to invalidate efforts to regulate adult bookstores in Los Angeles v. Alameda Books. In all of these cases, the more conservative justices voted to uphold laws that the more liberal justices voted to strike down.

In a few cases, the opposite occurred: The conservative justices voted to strike down laws that the more liberal justices wished to uphold. Two of these decisions involved government regulation of businesses. The conservative justices invalidated restrictions on pharmaceutical advertising in Thompson v. Western States Medical Center and voted against uncompensated land-use restrictions in Tahoe-Sierra Preservation Council v. Tahoe Regional Planning Agency. And in one federalism case, Federal Maritime Commission v. South Carolina State Ports Authority, the conservative justices ruled that a federal agency cannot decide a suit against a state agency without the state's consent.

I don't want to make too much out of one Supreme Court term. The results of a single term can only be suggestive, and, as noted earlier, judicial invalidation alone provides a poor proxy for activism. Further, defining ideological camps is itself subjective and a bit arbitrary; labels like "conservative" and "liberal" invariably lack nuance. But, at the very least, the pattern suggests a certain myopia among those who rely on the number of laws the Supreme Court invalidates as a sign of its conservative activism. True, the conservative justices occasionally vote to strike down legislative acts that the more liberal justices would uphold. But the opposite happens as well. And at least based on the last term, ideological splits more often feature the conservatives voting to uphold the laws than the other way around.
11.29.2007 3:11pm
Davide:
Orin,

Your comments, I'm afraid, are the ones that puzzle me. You're the academic and you're the one claiming that your study will have value. I doubt it.

You are responding with hostility: calling requests "absurd," or resorting to ad hominems in response to "anonymous blog commenters." Such sniping, of course, is irrelevant to my queries and to your non-responses.

On the substance: whether you can't or won't respond, the issue is the same. Your study appears to lack merit. Your failure to support to it doesn't move your (the proponent's) burden to show otherwise. Or should we simply accept your ipse dixit that your study will be illuminating? That's an intriguing academic methodology.

Your article has no methodology and confirms the failings of this project.

Your whole problem (which you refuse to admit) is that you have no principled method to (a) identify a "conservative" decision (b) identify a "neutral" decision or (c) identify an "ideologically-charged" case. Your earlier article contains the same conceptual error. It almost admits it when it states that "IT TURNS OUT TO BE MUCH EASIER TO DESCRIBE ACTIVISM in theory than to identify it in practice. This is true for two reasons. First, reasonable people can disagree on many legal questions. Law is not like mathematics: It is an inexact science."

The problem here is that your study is no science at all. If you put bounds to your methodology, then it allows for criticism, measurement and analysis. An inability to do that renders it what it most likely is: a gestalt impression. Nothing more, nothing less. About as good as any other reasonably educated lawyer's gestalt impression.


Your method is "I say so." That's fairly meaningless. Like your statement (which verges on the Justice Stewart "I know it when I see it" form of analysis: "try this experiment: Pick up a Federal Reporter from the 1970s and compare an opinion by Henry Friendly of the federal appeals court in New York with a decision by Skelly Wright of the court in Washington, D.C. In all likelihood, the former will absorb the existing law and carefully apply it; the latter will be more likely to ignore precedents and create new law. That's roughly the difference I have in mind").


Perhaps you balk because you can't identify any of these terms with exactitude. No one can. But your refusal to do so doesn't let anyone else critique your analysis, replicate it or use it for any useful purpose.

For instance, let's say someone wanted to use your same definition of "ideologically-charged" cases to analyze Justice Brennan. Can't be done-- you never defined the term. Your "analysis" is a one-off.

Or let's say someone wants to see if Justice Kennedy is a "conservative" using your analysis. Can't be done. Why? You never defined your term.

Or, let say we want to see which Justice deviate from "neutral" the most using a different data set (cases from the 60s). Again, can't be done. You haven't defined the term at all.

This doesn't mean your insights are valueless. But it does mean your "study" isn't one. It's more like an opinion piece for Legal Affairs or the NY Times. No more, no less.
11.29.2007 3:33pm
OrinKerr:
Davide,

At bottom, your argument is that my proxy is imperfect. Of course -- I've said that from the beginning. The only question is whether my inquiry may be useful, despite its imperfections. On that question, you respond: "This doesn't mean your insights are valueless." I gather that is a sign of agreement.
11.29.2007 3:46pm
PLR:
Ralph at 2:26: Thanks for the explanation...
What does it mean if Roberts gets a 60% conservative voting record? What if it's 95% Does Kerr even know? It's best to decide what a given result would mean before the data is collected.

... but that doesn't make much sense to me. Data gathering precedes data interpretation, even at the risk of the data not supporting the hypothesis.
At bottom, your argument is that my proxy is imperfect. Of course -- I've said that from the beginning. The only question is whether my inquiry may be useful, despite its imperfections. On that question, you respond: "This doesn't mean your insights are valueless." I gather that is a sign of agreement.

Reminiscent of the Sunstein study of "activism" in agency review cases. I'm not sure exactly what it's measuring, but it's interesting anyway.
11.29.2007 3:58pm
Cory J (mail):
This might be a silly question, but the post indicates to me that Orin is looking for input ("[W]hen a new decision comes down, we'll make a judgment call about whether it was one of the Court's ideologically-charged cases.") We, We, We. I don't think he's referring to the co-bloggers; I look at this like a fun little exercise for the community.

As people have noted, it's obviously hard for one person to determine the conservative position, but for purposes of discussion within the VC readership, eliciting responses seems reasonable (and fun!) to me.

I'm aware the same problems will inhere regardless, but a lot of the comments seem to assume Orin will declare by himself with no explanation what is and what isn't a conservative result, case closed. I don't think the point is for Orin to be the final arbiter but rather to open it up for discussion.

It seems to me a lot of the comments are objecting that Orin will have "final say" and this will unduly influence others or be "proof" of Roberts' credentials/credibility? Just a thought.
11.29.2007 3:59pm
Cory J (mail):
Change "question" to "observation" in my prior post.
11.29.2007 4:01pm
anym_avey (mail):
I'm not sure what this had to do with the discussion, but it's pretty spectacular:

But being smart doesn't make you right. Einstein believed in a god. He was very smart, but that didn't mean that his conclusion was improved by his intelligence. It was simply a belief that he held absent concrete proof. Many might agree with him that there is a god, but they're still wrong, no matter how smart.

You had better be right, because if you're wrong, that statement right there will be Exhibit A at your trial and the court will not supply any councillor for your defense.
11.29.2007 4:02pm
Davide:
Yes, of course, I agree: how could your insights not have value? Law is an art and no science. Someone who clerked for Kennedy will have useful things to say. Knowledge in the criminal law field will provide further utility to your comments. I'll read it with interest.

None of that, however, means that (1) its use of the terms "liberal," "neutral," or "conservative" is going to be apt; and (2) that the article will provide a principled way to determine if Roberts is an umpire, a conservative, a liberal, a nervous nelly, or anything else. Nor would I credit any conclusions in that article to have any greater weight than those I would give to any other experienced attorney familiar with Supreme Court matters.

It's a shame, really: my questions weren't meant as a "gotcha." They were meant to see if you had something new/interesting to bring to the table to evaluate Roberts.

I was curious particularly because you clerked for Kennedy, the famed "swing" of the current day. Perhaps you know (as I was once told by a former Kennedy clerk) that some view Kennedy as not-very-competent jurist who swings in the middle because he's (a) not very smart and (b) is very politically sensitive [to the New York Times and others]. Do you think that Kennedy is more "neutral" (using whatever calculus you like) than, say, Scalia? And do you admire him more as a jurist accordingly? I'm honestly curious -- I have no axe to bear here.
11.29.2007 4:03pm
anym_avey (mail):
Back on topic: Maybe I missed it, but I read most of the posts, and I don't see this question answered: How do we determine neutrality versus ideology in a case that is politically charged? The 500-pound gorilla in the corner of the room is the upcoming DC gun ban case. Obviously, "conservatives" (in general) would support the strong version of an individual right, and "liberals" (in general) would tend more toward a collective right that correspondingly allows the individual right to be limited or cut off by the state.

The analyses I've read to date suggest that the individual right is the better interpretation of the Second Amendment in the context of how the "militia" would have been understood in the days before a standing army ever existed, as well as in the context of various comments from the founding fathers and their contemporaries. But the debate continues anyway. So what happens if the Supremes deliver a 5-4 or 6-3 in favor of the individual right interpretation? Is this a "conservative" victory because it delivered an outcome that "conservatives" are more likely to favor? Or is it neutral because the principle of the law's intent was upheld?

And then we're back to the basic question of determining what the real principle of the law really was, and who gets to arbitrate that?
11.29.2007 4:12pm
Ralph Phelan (mail):
Having heard nothing from Kerr I'm assuming that my "anything short of 100% conservative and he's an 'umpire'" understanding of his intent is correct.

So let me get this strait.

After the term is over and you know what all the votes are you (or you &other readers here) are going to pick a subset of the decisions, and then assign the labels "liberal" and "conservative" to one side or the other of those votes.

If Roberts' vote on any of the cases you chose to include in your data set happens to be in the direction you chose to label "liberal" then you'll say he's an unbiased umpire, but if none of them are then you'll say he isn't.

And the point of this exercise is????
11.29.2007 4:35pm
Skyler (mail) (www):
This kind of reminds me of a class I took at Stanford one time on "Design for Manufacturability." In the class they described an elaborate system, similar to ORM, or FMEA, or for law people it kind of resembles a BPL, to decide what the important features of the design should be. The analyzer was to assign numerical values to these features, and by doing some rudimentary math about priorities and importance he was to rank the features by order of impact.

Fine. Very nice tool. Except that that you could just as easily do it without the structure of the mathematics and the form because they taught that if you got to the end and didn't like the results, you should just change the inputs. I've since seen this type of tool used in a growing variety of applications.

I'm convinced that these tools are good only for use in a court of law where you can use it to suggest that you weren't just making things up, that you had something that to an ignorant person would appear to be quasi-scientific and unbiased. The reality is that it is designed to be biased and to justify a bias.

But I'm sure in products liability cases a lot of engineers get their companies off the hooks this way. Oh well, whatever works.

I think this is the same thing we're seeing here. An analysis that is admitted to being of no statistical or logical merit, yet claiming to be useful nonetheless.

I'm surprised that Orin clerked for Kennedy because this seems a lot more like Posner!
11.29.2007 4:49pm
OrinKerr:
Skyler writes:
The reality is that it is designed to be biased and to justify a bias. . . . I think this is the same thing we're seeing here.
What's my biased goal again? I don't think you were specific on this before, but I'm always interested to hear what biases I am believed to demonstrate in a particular post.
11.29.2007 5:00pm
Duffy Pratt (mail):
If you read the original post, its pretty clear that the proposal was to do the evaluation on an ongoing basis, not after the term was over.

It seems to me that in many cases, you can decide that the case is a) politically charged, and b)what the desired results of the liberals and conservatives are on the case. So that could all be taken care of before the Court votes.

Also, if you read back just a couple of posts, you might see that Prof. Kerr specifically said that one should not presuppose agreement or acquiescence from his failure to respond. If he does agree with your "anything short of 100%" rule, then let him say so. I ignored your response to my last post, but that certainly doesn't mean that I agree with it.

Perhaps the point of the exercise is to generate some fun activity on the blog, and have a nice ongoing discussion on a topic that interests people. BTW, what is the point of this blog? And, despite what people think, I really doubt that Professor Kerr's goal is to end up with a definitive label of Roberts as either Conservative or Liberal. It seems to me that people who followed this exercise might come to their own interesting conclusions along the way, and share those with others. The end result of the study does not have to be its "point." But I could be wrong about Prof. Kerr's intentions, and he can certainly speak for himself.
11.29.2007 5:07pm
Skyler (mail) (www):
Orin, I didn't say you have a biased goal. But the nature of the process is such that there can be no result that isn't a product of some sort of bias. I don't know anything about you except what I've read in these comments. I've learned you work for Kennedy at one time. That's about it. I don't know what your bias is. But in order to have a result, by your own definition of the process, there must be some sort of bias. Therefore the result is meaninglessness masquerading as a quasi-scientific result.

I guess the sticking point here is that you think that the effect of your bias (umpiring) will not significantly the results, whereas I think that there can be no meaningful result because of that bias (umpiring).
11.29.2007 5:08pm
Skyler (mail) (www):
I like your point, Duffy.

Well, let's wait and see what happens.
11.29.2007 5:10pm
Deoxy (mail):
The basic premise is flawed; tvk's second post (waaay up at the top) really laid out why VREY well.

The point that really gave it away for me, though, is when you compared judicial decisions to a coin-flip!! (in your other post on this topic) "If a coin flip always comes up heads, after a while you're gonna wonder about the coin."

This is not remotely like a coin-flip. There IS a "right" decision (or several that are at least similar in difficult cases), and other outcomes are "wrong". Where those fall on the political spectrum is immaterial.

Now, when a case is decided in a way that is clearly wrong (Kelo, for instance), then there's something to be said, but before you can make the case that a decision is political, you must first make the case that it is incorrect.
11.29.2007 5:24pm
OrinKerr:
Skyler,

I think what you're trying to say, to use an engineering analogy, is that you fear the results will have a low signal-to-noise ratio. (It seems that you use the term "bias" to mean what would generally be thought of as "noise.") My view is that the results will indeed have a lot of noise, but if we adjust our sensors appropriately, we may nonetheless get at least some information about the signal.
11.29.2007 5:27pm
Skyler (mail) (www):
Exactly.
11.29.2007 5:35pm
OrinKerr:
Glad to reach agreement!
11.29.2007 5:40pm
Terrivus:
I'm a little late to this comment party (and haven't had the time to read all of them), but... I honestly thought this was an ironic post, and I was waiting for the twist at the end. Which never came.

I have a question for Prof. Kerr. If you were to perform this analysis for last year's Term, using your methodology, would you have included the Chief's votes in the following cases?

Lopez
Duenas-Alvarez
Cunningham
Watters
Winkelman
Atlantic Research

In each of these, the Chief took the "liberal" position (dissenting in Watters). But would they make it onto your radar, since they weren't "ideological" cases? And would they have, had the vote been 5-4 -- therefore nudging the case into "ideological" status?

Similarly, what about the Chief's votes in the following cases?

Scott
Credit Suisse
Bell Atlantic
Burton
Bockting

In each of these, the Chief voted for the "conservative" position -- but so did at least one member of the "liberal" bloc. Are the cases thus "nonideological," therefore falling outside your calculus? How many more liberal Justices would have had to dissent in order for one of them to count as "ideological"?

And finally, what to make of cases that were 5-4, but not according to the usual lineup? Ideological? Nonideological?

My point is that, just as most of the media did last year, your methodology for deciding what is an "ideological" case verges on circularity and threatens to exclude pertinent votes that would otherwise affect your analysis. This is all well and good for certain journalists and pundits who already have their conclusion in mind, but I hope you don't fall prey to the same flaw.
11.29.2007 6:13pm
Cory J (mail):
Completely OT: What post has generated the most comments in Volokh Conspiracy history?
11.29.2007 6:19pm
frankcross (mail):
Skyler, if you examine the research, you will see that these studies do not have a low signal to noise ratio. If that were true, one would expect erratic results. In fact, they are entirely consistent. This means it cannot be noise. Might it be an unmeasured third variable? Sure, but try to come up with one that hasn't already been considered.
11.29.2007 6:56pm
SIG357:
If it was, then we'll classify Roberts' vote as either politically conservative or politically liberal.





I'm not sure how this is going to work. If he votes to overturn or curtail Roe and its offspring, will that be the action of an umpire or a conservative? In some cases the two will be indentical, surely.

The same applies to the Second Amendment. If he rules that it is an individual right, will that make him a right wing zealot or a dispassionate umpire applying the Constitution fairly?
11.29.2007 7:28pm
SIG357:
In his confirmation hearings, Roberts expressed hope that he would be an umpire. In his cases Last Term, however, Roberts tended to vote consistently for the conservative side.




The implication is that these two things are contradictory. But that has not been demonstrated. It's at least theoretically possible that our imagined impartial umpire would have voted for "the conservative side" in the cases in question.
11.29.2007 7:37pm
The General:
there's a simple way to do this:

1) politically liberal result = whatever the hell a liberal wants to do with his/her body or your money.

2) politically conservative result = proper application of statutory law and the Constitution as it is written, rather than on the basis of the political preferences of a particular Justice.

See? Simple.
11.29.2007 7:48pm
Angeleno:
It's remarkable how resistant many commenters are to Orin's unremarkable premise that "ideologically conservative" does not always equal "legally correct." Indeed, Orin recently wrote (in a comment thread on another post about why he doesn't always take "libertarian" positions on this blog) that he is careful to make the distinction between what he believes the best policy should be and what the law is. I can't see how anyone attempting to be objective would think it's proper to do otherwise. It's just bizarre to think that a particular ideological position would be correlated strongly with "legally correct" positions, unless one has a belief that the proper metholology to figure out what the law is is to fit it to a set of ideological preconceptions.
11.30.2007 1:08am
Angeleno:
Orin,
Do you think the "umpire" metaphor actually would make sense even if he were clearly following an unbiased set of principles?

It seems to me it's hard to make sense of the application of that metaphor to an appellate justice's work. The judges are not generally finding difficult facts (such as "was the pitch in the strike zone?" or "was the batter tagged before he touched the base?" or "is his testimony more credible than hers?" as a trial judge might have to do in a bench trial) but are often making difficult judgments about what rules even apply (such as what test to use to determine whether federal or state action violates a particular constitutional provision).

Umpires might do some of this on the margin surreptitiously (making their own judgments about what exactly the strike zone is, despite the relative clarity of the rulebook), and perhaps even engage in it openly in some cases, though I can't think of where offhand. But this metaphor seems shaky at best to me. Is there a way to apply the "umpire" idea that actually is coherent and/or analytically useful?
11.30.2007 1:23am
OrinKerr:
Angeleno writes:
Orin,
Do you think the "umpire" metaphor actually would make sense even if he were clearly following an unbiased set of principles?
I do. To pick an example, Let's say Roberts decides that he will be a textualist in every statutory case and will follow precedent as much as possible in every constitutional case. In that case, he would be acting as an "umpire," in that he would lay out a set of explicit neutral principles for determining legal meaning that he would aim for in each case. If he honestly applies those principles to every case, then he will indeed be pretty umpire-like. Of course there will be judgment calls, but then there are judgment calls for umpires, too. Now, of course, we can debate whether we like the rules to which he has decided to umpire. You could say you want him to call cases based on (say) an intentionalist approach instead of a textualist approach. But still, it would be fairly umpire-like.

Cory J asks:
Completely OT: What post has generated the most comments in Volokh Conspiracy history?
I think it was a gay marriage thread by guest-blogger Maggie Gallagher, which if I recall correctly exceeded 500 comments.
11.30.2007 2:21am
TheWhaler (mail):
I suppose the real question is:

Does he call'em as he sees'em? Or does he call'em as they are?
11.30.2007 2:31am
Ralph Phelan (mail):
the results will have a low signal-to-noise ratio.
If you could get true random noise that would be an improvement!
What you're going to have is a signal of unknown size, a systematic bias of unknown size, and no way of disentangling them because you're only measureing one variable.

Roberts will vote conservative N% of the time.
N will problably not be 50%.
Some will argue that this means he's biased.
Others will say no, it means liberals try to twist the law and he won't let them.
No agreement will be reached.

There - we've already gotten all the information that can be possibly be extracted from your results.

So actually "collecting" the "data" seems rather a pointless exercise.
11.30.2007 8:10am
Angeleno:
Thanks for the response, Orin. That makes some sense to me in theory. But did he actually articulate a real principle, or set of principles, of interpretation in his confirmation hearings, or anywhere else? (I really don't recall, but I don't think he has . . .) If so, that's ok. But if not, what's the point of trying to judge his "umpiring" on the terms you suggest?

I was quite cynical about his umpiring comment at the time -- it seemed, without more information, just something to say to get confirmed.
11.30.2007 9:01am
Angeleno:
Ralph Phelan,
The point of empirical research is not to arrive at a single authoritative interpretation of the data. It's to collect data, to analyze the trends it reveals, and to allow people to interpret it, even in competing ways (and even in ways that challenge its validity or usefulness). Why are you so offended by this exercise, even before the project's details have been fully described?
11.30.2007 9:08am
Ralph Phelan (mail):
Because the details described so far indicate a total waste of time. If you already know that there is no possible result that will change anyone's mind about anything it's really a waste.

And lots of proposals for ways to change the details to make it have a chance of actually accomplishing something have been totally ignored.
11.30.2007 9:54am
Mary Katherine Day-Petrano (mail):
Chief Justice Roberts is conservative, but didn't we all already know that? I have read a growing number of his opinions, and I don't see where he is not being an umpire. Do I disagree with some of his opinions? Yes. I think all of us disagree at some time or another with particular Judges or Justices opinions or the way a case outcome turns out, because maybe WE would not have called it that way.

What I don't understand is why some people, and this includes the media, are picking on Chief Justice Roberts so much. This morning there are new newspaper articles criticizing the Chief Justice for not doing a tell-all for the public's prurient interest about the state of his health. So what about his health? If the Chief Justice is readily able to meet the essential functions of his job, why should the public have any reason to dig into the private facts about his health? Does anyone see anything that suggests the Chief Justice is not able to perform his job? I don't; he is doing quite well as far as I can see.

He is probably the one Justice or Judge I most respect, and the reason why is because when I didn't know much about him during his confirmation process, and had heard he was so-called 'anti-disability discrimination law,' some of his supporters thought enough to respond to my inquiries about his record and try to convince me he was fair by pointing me to a Rehabilitatation Act case he decided on the D.C. Circuit. It was a good case, not one that I disliked. I think that case was a great example of the Chief Justice being what he told us he is--an umpire.

Rather than just tell me my viewpoint or ideas were stupid or not worthy like so many other Judges, lawyers, and their supporters, or ridicule or mock my disabilities as means to discredit my beliefs and opinions like so many others, Chief Justice Roberts' supporters did what Judges, Justices, and lawyers are supposed to do--CONVINCE me by persuading me. Not running roughshod over a viewpoint a person strongly holds. Setting such a classy example of what lawyers, Judges, and Justices are supposed to be like demonstrates the Chief Justice deserves to be our Chief Justice.
11.30.2007 10:06am
CJColucci:
I suppose the real question is:

Does he call'em as he sees'em? Or does he call'em as they are?



They ain't nothing 'til he calls 'em.
11.30.2007 10:59am
OrinKerr:
Ralph Phelan writes:
If you already know that there is no possible result that will change anyone's mind about anything it's really a waste.
What a fascinating comment. Just speaking for myself, I can tell you that the results will change my mind. That's why I am interested in the results: I want to see what the evidence is, and then I will evaluate the evidence, assess its strength, and use it to have a greater understanding of reality. Maybe this method is deemed quirky or weird to some readers, but I guess I'm still a scientist about that sort of thing. Of course, if some people reach conclusions despite evidence instead of because of it, taking evidence is no doubt a pointless exercise.
11.30.2007 12:59pm
Ralph Phelan (mail):
speaking for myself, I can tell you that the results will change my mind.

Sigh.

Could you please get a bit specific about how?

If Roberts scores 60% conservative, what will you think that means?

If Roberts scores 95% conservative, what will you think that means?

Are there measures other than the total percentage of conservative votes that you will be looking at? If so, what are they, and what do you think different possible values would mean?
11.30.2007 1:05pm
A.S.:
It seems to me that in many cases, you can decide that the case is a) politically charged, and b)what the desired results of the liberals and conservatives are on the case. So that could all be taken care of before the Court votes.

As I wrote above, I agree with this completely. So let's begin now!

I would propose that Orin list some of the cases he thinks are "ideological" and then tell us what he thinks are the "liberal" and "conservative" positions. I don't think he has to do this all in one post - maybe take a case per post. In fact, doing it that way will make it easier for the commenters to get a little understanding of the case and to keep everything straight.

Some cases will be easy - the Second Amendment case is obviously ideological, with "conservative" being the individual rights position and "liberal" being the collective rights position. (Query: even if one takes the individual rights position, do you also need to strike down the statute to be "conservative", or is a remand sufficient?)

So enough talk about the merits of the project. Let's go!
11.30.2007 1:37pm
OrinKerr:
Ralph,

I believe I answered your questions in the inital post, about 160 comments ago.
11.30.2007 4:18pm
Ralph Phelan (mail):
And, having reread it yet again, I still can't figure out how to apply the language to specific cases. Could you please indulge my denseness and tell me:

If Roberts votes "conservative" 60% of the time, is he an "umpire"?
If Roberts votes "conservative" 95% of the time, is he an "umpire"?
11.30.2007 4:33pm
TheWhaler (mail):
CJColucci sez:

They ain't nothing 'til he calls 'em.

A reply, which I believe raising interesting philosophical question. I happen to disagree. Just as I think rights are prepolitical, so too do I believe stikes are strikes, no matter what the umpire calls. In your view, there's no such thing as a bad call, since it's right by the very act of the call. Whereas clearly, if the ball was thrown down the middle, above the knees and below the team logo, it's a strike. Even if Lt Frank Drebin is the umpire.

It's the same distinction that Hayek made between law and legislation.
11.30.2007 5:16pm
Cory J (mail):
Orin,

Thanks for the response. I was interested in light of this post and the Ron Paul thread.
11.30.2007 5:35pm
OrinKerr:
Ralph writes:
And, having reread it yet again, I still can't figure out how to apply the language to specific cases. Could you please indulge my denseness and tell me:

If Roberts votes "conservative" 60% of the time, is he an "umpire"?
If Roberts votes "conservative" 95% of the time, is he an "umpire"?
Here's what I wrote in the post:
If we see votes on both sides, it will support the case that Roberts is an "umpire." Someone truly following principle will go wherever that principle will take him, and that should led to mixed views from a political standpoint. On the other hand, if all the votes end up being conservative, that will support the notion that Roberts is voting to make the law to shape his personal preferences rather than merely following it.
If Roberts votes 60% conservative and 40% liberal, then plainly that's votes on both sides -- indeed, it's almost an equal balance of votes!!! From the original post: "If we see votes on both sides, it will support the case that Roberts is an "umpire."" Obviously it won't "prove" it -- no one suggested anything so utterly absurd, nor do I understand how you could think so. But it will be some evidence for that.

If we see 95% conservative, then that will be very close to "all". In the case of "all," recall what I wrote: "On the other hand, if all the votes end up being conservative, that will support the notion that Roberts is voting to make the law to shape his personal preferences rather than merely following it." Now, what to do with the fact that it will be 95% instead of 100%? Obviously different people will draw different conclusions. Some will think 95% is close to 100, some won't. I think it will probably be a bit inconclusive: that's the nature of the scientific enterprise, as you cannot always get really clear data to prove or disprove a hypothesis.
11.30.2007 5:42pm
courtwatcher:
I think some of the commenters have the misperceptions that (1) in order to be valuable, a research study must conclusively prove a hypothesis, and
(2) anyone who undertakes empirical research must also believe point #1, and
(3) because Orin Kerr has proposed to do an empirical examination, he must intend and contend that his study "conclusively proves" a hypothesis. He has made clear that he disagrees with #1 and #2 (as he should) and that therefore #3 is incorrect.

No one looking at this proposed study, including its creator, believes it will necessarily "answer" the question of whether Roberts is an "umpire" -- it is just possible (or, I suspect Orin might say "likely") that the study will provide evidence that tends to support or refute that hypothesis. And there's nothing inherently wrong with an exercise of that sort -- on the contrary, this is true of all scientific and social-scientific research, to varying degrees.

Mostly, I'm puzzled that people find this so objectionable. For those who think it's a "waste of time," it may be that this exercise won't yield anything you (or others) find useful in the end for a variety of reasons, but it's not your time that's being wasted, except to the extent you choose to continue to argue about it.
11.30.2007 8:21pm
Wondering Willy (mail):
Orin's proposed study is objectionable because of the value judgment that is implied. If C.J. Roberts votes as a consistent judical conservative, then the study suggests he essentially lied when he said he was being an umpire. The gist of the whole thing is that if you're too conservative, you're not approaching judicial decisions as an umpire. That, quite frankly, is b.s., and it's why most people here are aghast at Prof. Kerr's failure or unwillingness to acknowledge that criticism.

My personal opinion is that if a judge is a true umpire, he will reach the judicially conservative result (but not necessarily the POLITICALLY conservative result) 100% of the time. I think when Chief Justice Roberts made his umpire analogy, he was meaning to communicate that he is judicially conservative.

There is no good way to tell if a judge is being an umpire or not because you can't read a judge's mind. Maybe Ginsburg, for example, really does believe the law dictates all the kooky crap you will find in her opinions. If she really does believe the law commands the results she reaches, then she is most certainly acting as an umpire, but, to extend the analogy, has poor eyesight.

The key here is that an umpire can get a call wrong either because he is a biased umpire acting in bad faith or a bad umpire acting in good faith.
11.30.2007 9:13pm
OrinKerr:
Wondering Willy,

What is your definition of a judicial conservative? What is a judicially conservative result?

In terms of your last point, I would think you would applaud such efforts as this. If judges have poor self-awareness, commentary on their work can help them be more self-aware. If so, isn't that a good thing?
11.30.2007 9:47pm
Mike& (mail):
My personal opinion is that if a judge is a true umpire, he will reach the judicially conservative result (but not necessarily the POLITICALLY conservative result) 100% of the time.

Isn't it at least one point of the inquiry to ask: "Does John Roberts' jurisprudence reflect judicial or political conservatism?" Why is that an illegitimate inquiry?

Is because it is impossible to define "political conservative" and "judicial conservative"? (I think the latter is very hard to define, but you used that term, so I assume you have a definition in mind.)

There is no good way to tell if a judge is being an umpire or not because you can't read a judge's mind.

If this view were accepted in the criminal law context, you could never convict a person of an intent crime. How can you read anyone's mind? This goes back to Wittgenstein: How can you know people have minds?

We all make the epistemic assumption that a) other people have minds and b) we can learn the contents of those minds through their actions and language. In fact, that is what's done in criminal cases every day. "How do you know A intended to kill B when we can't read A's mind?"

Are you thus willing to concede that crimes should not be based on intent? Thus, e.g., a person who "accidentally" runs over someone, killing him, should be charged with first-degree murder? After all, how can you know the guy did it on "accident" since you can't read his mind? Or should a guy who beats someone to death with a lead pipe (by hitting him over the head 15 times) be charged with negligent homicide, since, again, it's not possible to read his mind? Maybe it was an accident. How can we know?

I would imagine most would say, "People don't accidentally pound someone 15 times over the head with a lead pipe." But, if I'm reading what you wrote correctly, that counter-intuitive answer would actually be the correct one.
12.1.2007 4:24pm