pageok
pageok
pageok
Law Review Article Accuses Louisana Supreme Court Justices, Is Itself Found To Have Serious Errors:

Dan Slater (Wall Street Journal's Law Blog) reports on an interesting story about errors in the academy. The Tulane Law Review recently published an article that purported to compare Louisiana Supreme Court Justices' voting records with the campaign contributions to them from litigants and lawyers; the article asserted that

some of the justices have been significantly influenced — wittingly or unwittingly — by the campaign contributions they have received from litigants and lawyers appearing before these justices.

The New York Times reported on the story before the article was published, as did WSJ Law Blog.

1. Now it turns out that there many of the cases were miscoded — a rebuttal asserts that "in thirty-seven of the 186 opinions included in the study, the information about the case on which Palmer and Levendis based their conclusions is just plain wrong, such as how a Justice voted or even if the Justice was on the panel that decided the case." [UPDATE: After I posted this post, the rebuttal I linked to was replaced with a corrected version, and I adjusted the quote. The original version said 40 of the 186 opinions were miscategorized; the revised version says 37.]

The authors acknowledge that there were errors; one of the authors asserted that "with all the mistakes now corrected, ... the study's conclusions, broadly speaking, are the same," but the revised study and the revised dataset has not yet been publicly distributed. (I e-mailed that author mentioning I was going to blog about the controversy, and asking whether he could provide a response that I could link to; but though he originally offered to pass along the revised dataset, he later said that because of a newly arising lawsuit threat, he was told not to distribute the data until it could be independently reviewed. This may well be sensible, but at this point all that can be said with confidence is that the original data is wrong, and that this has been admitted by the researchers.)

Incidentally, while it's not clear from the article exactly how the data was gathered, it looks like part of the problem might be lack of checking by the authors or by the law review: The article asserts (p. 1298) that "Each case was thoroughly read and analyzed by a researcher. Once the cases and contribution information were gathered, we entered our observations into a standard data table." If the article "a" in "a researcher" is precise, and if the researchers were research assistants and not the authors (seemingly likely, given the thanks in footnote * and the use of the term "a researcher" rather than "one of the coauthors"), this suggests that each case was read only by one research assistant, with no further verification.

I also asked the current Tulane Law Review editor-in-chief, and he reported that to his knowledge the law review cite-checking process did not check the underlying database. This is probably consistent with standard law review practice; law reviews generally check all the citations that appear in the text of the article, but I'm unaware of any practice of checking the case data that appears in databases that aren't published within the body, footnotes, or appendix of the article. Nonetheless, it's unfortunate that this happened, since cite-checking often (though not always) does uncover factual errors such as the ones that appear to have been present here.

2. But there's more than this to the situation, I think. Even if the data were correct, the article would still be drawing what strikes me as an unsupported inference from correlation to causation. The article asserts in footnote 14 that

It is worth observing that this Article does not claim that there is a cause and effect relationship between prior donations and judicial votes in favor of donors' positions. It asserts instead that there is evidence of a statistically significant correlation between the two.

But many other passages in the article seem to argue that there was indeed causation. The opening paragraph says (as I quoted above) that "This empirical and statistical study of the Louisiana Supreme Court over a fourteen-year period demonstrates that some of the justices have been significantly influenced — wittingly or unwittingly — by the campaign contributions they have received from litigants and lawyers appearing before these justices." (Emphasis added, in this quote and the later ones.) "Some justices may sincerely believe that they have not been influenced by the money they take, but sincerity makes no difference if the reality is otherwise." "Some studies have tracked the rise in contributions made to judicial candidates, but few have attempted to determine whether these increasing contributions actually influence subsequent adjudications involving contributors. This Article demonstrates that the debate is not evenly balanced." (This quote is just a sentence before footnote 14.) There are many more examples.

The trouble, as this other critique of the article points out, is that there's a perfectly plausible alternate explanation for correlation between contributions and voting patterns — that contributors contribute money to the election of those candidates whose ideologies they agree with, rather than that the elected candidates then decide based on the identities of their contributors.

Say, for instance, that we discover that a liberal state supreme court justice often votes in favor of plaintiffs in individual tort cases, employment cases, or environmental cases; and say that we find that those plaintiffs are often represented by law firms that have contributed money to the justice. It's of course possible that the justice is influenced by the identities of his contributors. But it's also possible that the justice is voting solely based on his view of the law — and the contributors contributed to him because they share his view of the law (or in any event find his view of the law to be good for them and their clients). Simple evidence of a correlation, even a very strong correlation, cannot distinguish between these two explanations, and thus can't show that contributions "influence" the justices' votes.

The authors' attempts to control in a way that eliminates the alternative explanation strikes me as quite weak. The authors look separately at voting patterns in favor of plaintiffs and in favor of defendants, and they conclude that

In cases where the defendant was the net contributor, Justice [A] ruled for the defendant's position 66% of the time, and Justice [B] 86% of the time. On the other hand, in cases where the plaintiff was the net contributor, Justice [A]'s vote was for the plaintiff's position 66% of the time, and Justice [B]'s vote was for the plaintiff's position 63% of the time. This is a swing of 32% for Justice [B] and 49% for Justice [A] when the net donor changes from being a defendant to a plaintiff. The marked shift favoring the net contributor, irrespective of being plaintiff or defendant, strongly indicates that it is the donation, not the underlying philosophical orientation, that accounts for the voting outcome.

But no judge has any inherent philosophical orientation for plaintiffs or defendants as a broad category. For instance, some judges may support cities as defendants (for instance, voting in favor of broad municipal immunity against various claims) and cities as plaintiffs (for instance, voting in favor of broad municipal power to get injunctions against nuisances) — not because the law firms that represent small cities have donated money to the judge, but because the judge generally supports municipal government authority.

Likewise, say a law firm tends to represent employers as defendants in many employment cases, but also represents employers as plaintiffs in other employment cases, such as cases enforcing nondisclosure or anticompetition agreements. In this situation, a judge with a generally pro-employer perspective will vote in ways this law firm likes — which may mean the law firm will try to help elect the judge even if the judge never pays attention to who contributed to him. (Recall that the study considered donations from lawyers as well as donations from litigants.)

3. Of course, it's certainly possible that judges' decisions may be influenced by whether the litigants or the lawyers have contributed money — or time or an endorsement or other things — to the judges' election. I don't want to suggest that anything I say above proves this effect is absent, either on the Louisiana Supreme Court or elsewhere. But the study's claim that it has "demonstrate[d]" such a "significant[] influence[]" is not adequately supported. And before one claims that identified judges have indeed been so influenced (not just that judges generally might be influenced, a proposition that our knowledge of human nature tells us must be true at least in some instances), it seems to me that one should have significantly more evidence than the article adduces. And this would be true even if the article's underlying database were accurate, or even if the authors can replicate their correlations after the database has been corrected.

4. Finally, note that the law school's dean has written a letter of apology to the Louisiana Supreme Court Justices, and the law review has noted the error on its site. (I don't know what further attempts at correction the law review might be taking.)

Thanks to How Appealing for the pointer.

Related Posts (on one page):

  1. Why Speculate, When You Can Look It Up?
  2. Law Review Editors, Take Note:
  3. Making Data Available:
  4. Law Review Article Accuses Louisana Supreme Court Justices, Is Itself Found To Have Serious Errors:
Frog Leg (mail):
I also asked the current Tulane Law Review editor-in-chief, and he reported that to his knowledge the law review cite-checking process did not check the underlying database. This is probably consistent with standard law review practice; law reviews generally check all the citations that appear in the text of the article, but I'm unaware of any practice of checking the case data that appears in databases that aren't published within the body, footnotes, or appendix of the article.


But this ignores the threshold question: why aren't these cases cited in the article, in an appendix or otherwise? We had a similar type of study in one article I helped to edit, and the author was made to put the cases in an appendix, and we checked them there.
9.22.2008 5:24pm
ejo:
isn't there a point where you just admit you were all wet and give up. why keep digging? what is the purpose of the whole enterprise of the law review if all it does is cite, as opposed to fact, check.
9.22.2008 5:32pm
Milhouse (www):
Of course the same argument applies to politicians. When a politician's votes match up with his donors' interests, one of several things could be happening. The googoo types (such as John McCain) leap to the conclusion that the politician is not voting her true views, but rather has sold her vote for cash. But there are at least two other plausible explanations, both of which are innocent: 1) The donor looked for a candidate who already agreed with his views, and gave her money to get elected, or to keep her seat. 2) The donor gave to a hostile politician, whereupon the politician took an interest in the donor's situation, gave him time to explain his point of view, perhaps let the donor take her to see things for herself, and as a result came to realise that her previous view was ill-founded. (That's called lobbying; AKA the right to petition, which is explicitly protected by the first amendment.)
9.22.2008 5:49pm
Lior:
There has been a lot of discussion here recently on the students' "editing" of law review papers. I seem now that the situation is worse than I thought. Not only do the "editors" focus on irrelevant things like formatting of citations, they ignore essential things like checking the correctness of the article.

The objections being raised by Prof. Volokh here should have been raised by whoever reviewed the article for publication, so that the authors could address them ahead of time.
9.22.2008 6:00pm
Jack S. (mail) (www):

The objections being raised by Prof. Volokh here should have been raised by whoever reviewed the article for publication, so that the authors could address them ahead of time.


generally the editorial staff of the law review, meaning 3L's and probably the advising professor.

Perhaps since it was a Tulane professor, a few shortcuts were taken.
9.22.2008 6:10pm
JohnO (mail):
Eugene has it exactly right -- the analytical flaw in the cause-and-effect argument is so painfully obvious that I don't know how this didn't get picked up somewhere along the line. And ten sentences stating that the justices were in fact influenced can't be nagated by a footnote saying that there is no coaim of cause and effect.
9.22.2008 6:34pm
one of many:
I'm not certain that's a valid complaint Lior.

From what I can tell the problem was not in the underlying data but in the coding of that data, analogous to raw data being entered incorrectly into charts. Although it varies with the discipline, there is no consensus that raw data be included in academic articles and checking to be sure that charted data confirms the article is sufficient for most journals. The biggest difference is that here the raw data was easily available and didn't match the "processed" data.
9.22.2008 6:36pm
Tony Tutins (mail):
1. If lawyers could do math, they'd go to medical school. So a special effort would need to be made to check the data analysis in a law review submission. Conversely, why put a math-heavy paper in a law review?
2. Cite checking checks authors' assertions against commonly available sources. The unpaid laborer asks: Is there a cite for every thought? Does the source actually stand for the cited proposition? Is the cite formatted properly? These are all within the grasp of typical 2Ls. A paper like this should be submitted to a peer-reviewed social science journal, where the reviewers would be capable of critiquing the analysis.
9.22.2008 6:46pm
Eugene Volokh (www):
Tony Tutins: The chief error in the data was in noting how the judges voted in each case -- the sort of thing that law students are just fine at checking. Likewise, law review editors should have recognized the inconsistency between footnote 14, which disclaims any attempt to show causation, and the text, which asserts such causation. I think law review editors should also recognize that correlation doesn't show causation, though I realize that not all of them do.
9.22.2008 7:17pm
Brad V (www):
Tulane Maritime Law Journal is the way to go.
9.22.2008 7:26pm
Lior:
It's clear that the reviewers couldn't check the data. But failing to differentiate between correlation and causation is a separate problem.
9.22.2008 9:17pm
Brian G (mail) (www):
Big deal. Law firms will still hire people who are on law review cite checking for two years before people who excelled at clinic and actually litigated cases prior to graduating.

Law review is a waste of time in law school. I am quite sure these types of errors go on all the time, but go unnoticed because, after all, who cares about substantive due process conflicts during the infancy of the Industrial Revolution?
9.22.2008 9:17pm
Bama 1L:
Umm, if law review jumps you to the head of the downtown firms' hiring lines, doesn't that make it not a waste of time?
9.22.2008 10:37pm