I just wanted to stress that the Tulane Law Review article incident isn't just an interesting story of academic error -- it's also a story of law review embarrassment. I'm pretty sure that no law review likes to have to post on its front page,
Erratum
The Louisiana Supreme Court in Question: An Empirical Statistical Study of the Effects of Campaign Money on the Judicial Function published in Volume 82 of the Tulane Law Review at 1291 (2008), was based on empirical data coded by the authors, but the data contained numerous coding errors. Tulane Law Review learned of the coding errors after the publication. Necessarily, these errors call into question some or all of the conclusions in the study as published. The Law Review deeply regrets the errors.
I assume the law review will also have to publish a print correction. The incident also led the law school dean to have to feel obligated to publicly apologize for the errors in the article, and though the apology said the law review members did nothing wrong, the matter can't have been great for relations between the dean and the journal. And I suspect the incident in some measure tarnished the law review's brand with local employers, especially those who are friendly with the judges whom the article criticized based on inaccurate information (and an unsound confusion of causation and correlation).
Of course, law reviews must accept the risk of public hostility when they publish articles that criticize much-liked people and institutions. That's part of law review editors' responsibilities as participants in the scholarly publishing process. But the hostility is likely to be considerably higher when the criticisms prove to be based on error. And it's one thing to incur unjustified hostility in the service of truth, and quite another to incur justified condemnation because one's institution has been mistaken.
So it seems to me that there are three important lessons here:
1. When the author's article rests on data that you can check, check it. Here, the data was information about who voted which way in certain cases, and who got what contributions from whom -- something cite-checkers are amply competent to check; and checking the data for fewer than 200 cases is not a crushing burden.
If the data had been in footnotes or in an appendix, as it is in many articles, the law review would have checked it. That the data never made its way into a print article is no reason to skip checking it (as this incident illustrates). The printed article, after all, relied on the data, and errors in the data infected the information reported by the article. Had the law review done the cite-checking, they might have avoided the embarrassment to themselves, their dean, and (incidentally) the authors.
2. Look closely through the article's description of what it's saying, and watch out for self-contradiction (especially when the article is controversial enough that authors might be tempted into some self-contradictory self-protection). So when a footnote says,
It is worth observing that this Article does not claim that there is a cause and effect relationship between prior donations and judicial votes in favor of donors’ positions. It asserts instead that there is evidence of a statistically significant correlation between the two,
but the rest of the article repeatedly suggests causation -- for instance, saying that "This empirical and statistical study of the Louisiana Supreme Court ... demonstrates that some of the justices have been significantly influenced -- wittingly or unwittingly -- by the campaign contributions" (emphasis added) -- you should note the contradiction, and insist that the authors revise their claims to be internally consistent.
3. Finally, remember that correlation is not causation. If authors give evidence of correlation and from there makes claims of causation, make sure that the evidence adequately supports the claims, for instance by controlling for possible confounding factors. If the claim is that X (here, contributions) causes Y (voting patterns), consider what things may cause both X and Y (for instance, even though ice cream sales and the rate of forcible rape are closely correlated, might something else cause both, rather than ice cream sales causing rape?). Look also whether the causation leads the other way, which is to say that Y or predictions of Y can cause X: For instance, might a contributor's prediction of a judge's voting patterns lead him to contribute to the judge's election campaign, even if the contribution in no way influences the judge's vote? And if there are possible other explanations, does the author deal adequately with them.
Coming up with these alternative explanations doesn't require an understanding of statistics; even law review editors with little mathematical skill can do this. And law review editors should ask such skeptical questions just as they should look for counterarguments to authors' key doctrinal or normative assertions, and make sure that the authors deal with at least the main such counterarguments. If the authors do a poor enough job of dealing with these counterarguments, you should reject the article; or if you think the article is basically sound but needs to respond to those counterarguments, you should insist that the authors deal with them.
Authors should rightly have a great deal of discretion in how they craft their arguments. But when they don't adequately respond to the obvious counterarguments to their main assertion -- for instance, when they claim causation based on correlation, but don't control for obvious confounding factors -- part of your job is to call them on this.
And if you don't, when others call the authors on the errors, the result can be embarrassment for you as well as for the authors.
Related Posts (on one page):
In this case, the error was found out much more quickly and publicly admitted. So, some things worked right. I think the law review approach (trust nothing and check all sources) is far, far superior to academic peer review.
That being said, an error was made due to some simple undergraduate level mistakes. That's not quality publishing, when making the same error could flunk a Senior Seminar paper at 4 year liberal arts college.
cro: Unfortunately, I don't think the discovery of these errors stems from the law review approach. Rather, it stems from (1) the article's being a criticism of particular influential people, who promptly asked others to check it, and (2) the article's database being easy to check, since it refers to easily available sources (published court opinions) rather than to original sources buried in archives across the country.
Basic, I know, but still...
The fault here lies entirely with the authors.
Has anyone outside of a political or state connection with Louisiana confirmed the existence and/or scope of the coding errors?
Of course, the causation/correlation problem is visible on the face of the article, and thus easily confirmable by all of us.
Not necessarily. Typically I do a job because I am getting paid, but I get paid after doing the job. It's true that the promise of getting paid comes first, but that promise may be very secret whereas the getting paid part might be highly visible.
My reasoning? Simple logic and statistics: Professors are democrat 10-1. Law students, especially those on the Law Review are probably democrat by 30-1.(And personally, I would also bet money that on most Law Reviews, republicans/conservatives aren't represented at all.) And of course, the lack of attention to detail, like fact checking, faulty statistical analysis (can anyone say global warming), are just part and parcel of what is accepted as scholarship and scientific concensus, as long as it advances the 'correct' agenda.
The editors reviewed the article and it was just too good to check. If I ever need a lawyer, and I have in the past, I'll just make sure that it isn't one from Tulane.
And of course, law professors wouldn't write an article like this about democrat judges. It. Just. Does. Not. Happen.
Of course, I could be wrong.
Why would you possibly think this? Do you believe that democrats are just inherently so much smarter than republicans that the republicans are incapable of making the law review?
And as LTEC points out, some causal event must happen first, but it needn't be the payment itself. The implicit promise of compensation could be the causal event.
I like the idea of law review editors checking data, when that is feasible. It's not done by peer reviewed journals, but this could be a real contribution of law journals to empirical research.
As for correlation/causation, it's very true, but I would just note that people in journals attribute causation all the time. With less evidence than a statistical correlation. I'm perplexed by how people criticize those who use correlations as evidence of causation but give a pass to those who use only anecdotal evidence.
EV, I hope that this is not so. I expect it will not impact the hiring decision of private law firms in the local market. It might make a state supreme court clerkship difficult for Tulane grads to get for a couple of years; you'd like to think judges would not take it out on young law students but who knows.
I am struck by the court's using its own website as a platform to lash back out at private critics, and to gloat about the Tulane dean's apology. You get the impression that an article critical of the court would NEVER have been published at a state school, even if it were very well grounded.
The further irony is that while they denounce the very idea that campaign contributions influence their behavior, they are raising money as we speak (I got an invitation to a fundraiser just last week); and two judges are campaigning for an open seat on the supreme court, one of whom is running an ad denouncing "greedy trial lawyers." And about 10 years ago, Tulane's environmental law clinic notched up an impressive string of victories in environmental litigation- which were promptly followed first by industry protests, and second by the state supreme court adopting the most restrictive student-practice rules in the nation. You know, for the protection of the public.
So yeah, maybe if contributions don't determine how they vote after getting elected it's only because they sold themselves to contributors promising voting outcomes before they get elected. What's the difference?
But if the article contained bad scholarship, that much needed to be admitted. I hope someone does a more thorough followup- I think this topic is one deserving of study. But I get the feeling nobody in Louisiana is going to be eager to look at this issue again.
For some pieces, it's not really feasible to try to recheck the author's coding for accuracy. For example, if an author has gone through hundreds--and perhaps over a thousand--circuit court opinions using some form of agency deference and categorized each along, say, five different variables, my sense is that editors are probably going to take the author's word for it (and perhaps post the raw data on line or in an appendix as a substitute for thorough cite checking). Needless to say, this opens to the door for serious errors to creep in--e.g., what happens when the author miscategorizes many of the cases in the study?
One obvious benefit of student editing, as compared to peer review, is that students are generally willing and able to do the hundreds of hours of grunt work involved in cite checking. But the recent trend towards empirical pieces puts a strain on those resources, which are great but not unlimited.
EV, do you have a sense of how law reviews should cite check pieces like this one? It sounds like this particular article might have not have involved so many cases that regular cite checking was impracticable (and, as noted, may have had other red flags like internal inconsistency). But what if the study had been nationwide and had relied on 1,000 cases? Slog through it (or don't accept the piece for publication if you're unwilling to do so)? Trust the author, but publish the data in an appendix? Some hybrid (e.g., spot check some portion of the cases to see if they author is generally accurate)? It seems like an issue of increasing importance going forward.
That's a ridiculous statement and I'm not sure what you think the nature of law review is that conservatives would be discouraged from participating in it. I attended law school in Texas, and there were a fair number of conservatives on law review. In fact, it was probably an even split.
A social science is one in which "data" is the plural of "anecdote".
Instead of having strongly positive connotations, the brand may, at least for a time, have negative connotations, at least to many Louisiana lawyers and judges. That may be unfair to individual law review members, but my sense is that this is the way branding works in people's minds.
Forcible rape causes ice cream sales.
Baskin Robins in a major benefactor of this principle, and I'm very surprised victim advocates haven't started a boycott already.
Also, as a cite checker, I beg people to distinguish between editors (who are responsible for the flow and logic of the piece) and cite checkers, who are responsible for, well, the cites -- and perhaps some very light textual editing (for grammar, spelling, and so on).
First, there is no indication, or discussion even, of whether the actual votes of the justices in a particular case were objectively wrong, outside of the majority view, or against the justices' judicial view, such that the campaign contribution could have had any measurable influence upon a justice's particular vote. I mean, proof of influence from a campaign contribution would have to involve a showing that a justice voted a particular way due mainly to the fact of the contribution, and not due to his or her appreciation of the legal principles and facts involved in the case. Without such a showing or evidence of quid-pro-quo, then there can be no reasonable or non-speculative, or non-defamatory, discussion of causation, so the correlation produced is simply of no moment. Indeed, there is no discussion of the legal and factual issues presented in any of the cases used by the authors in their so-called study.
Second, the authors of the article looked only at contributions from lawyers, rather than include those from PACs or business and insurance groups, or individuals. However, it is well known that business and insurance groups have a keen interest in Louisiana judicial elections, and donate large amounts of money to judicial campaigns. Thus, focusing solely on lawyer contributors, and not including other types of contributors, is at the outset an inquiry of very limited and distorted value, if any.
Third, the authors of the article clearly presume that a justice who has a known judicial philosophy of, say, being pro-defendant and then votes pro-defendant in nearly every case is de facto a less-biased and less-easily-influenced judge than one who votes roughly equally for the plaintiff and for the defendant. The underlying premise of the article is that pro-defendant, a/k/a conservative judges are not influenced by contributions whereas voting for the plaintiff means the judge is influenced by contributions and renders the judge a pro-plaintiff, a/k/a liberal judge. The authors' parameters were clearly pre-selected to support such a correlation, but such a premise is preposterous in my view, ignores reality, and is tantamount to claiming that defendants are always innocent of any wrong-doing and plaintiffs are always bringing frivolous law suits them.
The article has already done its damage: Kitty Kimball is getting very bad press and has been attacked during the current judicial elections. Her opponent still uses the article on his website: http://judgejeff.com/