pageok
pageok
pageok
Debating Affirmative Action in Law Schools:
Writing in Slate, Emily Bazelon contends that the critiques of Richard Sander's Stanford Law Review article on affirmative action "pounce" on the argument, "destroy" it, and "throw" so many "punches" that the debate became a "bloodletting" that left Sander speechless. I haven't followed the back-and-forth very closely — some of Sander's assumptions seemed a bit off to me when I first skimmed his piece, but this is hardly my field — but I'm wondering if any VC readers have read the critiques and agree with Bazelon's assessment. I have enabled comments; links to the various responses are availible from Slate.
Anthony (mail) (www):
I haven't read all the rebuttal studies mentioned in the Slate article, but several things struck me as odd:


"What does Sander have to say for himself, once the bloodletting is done? Not much that helps him. In a rebuttal that Stanford also will publish, he wags a finger at Ayres and Brooks, chiding that "this is not their best work or their finest hour." He also compares his work to Galileo's. That's not really the comparison that springs to mind."


Did the author even read Sander's 31 page response? I haven't gotten a chance to read the whole thing yet, but by the third page Sander already identifies several critical flaws (such as mathematical errors) with one of the studies cited in the Slate article:

By failing to mention any of Sander's substantial critiques while mentioning other random quotes, the author is obviously misrepresenting Sander's response. Perhaps that is why she didn't link to the Sander response paper while linking to the papers criticizing him?


"Wilkins also takes a longer view and argues that Sander's concern about the fate of black law students turns out to be misplaced: Five years to 15 years after graduation, they earn significantly more on average than other black college graduates."



How exactly is this relevant? Correct me if I'm wrong, but I don't recall Sander (or anyone for that matter) claiming that attending law school caused black wages to become lower than those of the average black college graduate.


"And Wilkins points out that Sander never questions the utility and value of the bar exam itself—which probably puts him in the minority of those who have taken it, assuming he has. One way to increase the number of black lawyers might be to write a test that relies less on trick multiple-choice questions, or to convince the state bar associations that administer the exam to quit failing more and more would-be lawyers each year, as several have taken to doing. (Fewer new lawyers means less competition.)"


So Sander is wrong because he doesn't want to dumb down the bar exam? Once again, how exactly is this a criticism?


"But Ayres and Brooks do the next best thing—they compare black students who went to their first-choice school, which is presumably more elite, to those who attended a lower-choice school. The entering credentials of the first-choice and lower-choice groups turn out to be quite similar. And so do their bar-passage rates. The wrinkle in the story is that the black students who do the best on the bar exam, relatively speaking, are the ones who attend the top schools and historically black schools like Howard law school in Washington, D.C."


The article didn't link to the Ayres/Brooks paper so I haven't gotten a chance to read it, but wouldn't white students who attend their first choice school do better on the bar exam also? Studies at the undergraduate level have proven that there are many benefits associates with attending one's first choice school, so it would be highly unusual for white law students attending their top choice to not also have higher bar passage rates. Not to mention that if Ayres/Brooks found that whites do not do better at their first choice schools, you'd think this author would mention it.

I haven't looked at the other response papers so I won't comment on the substance, but it's clear that this author already has an agenda
4.30.2005 7:28pm
Fabian:
How, pray tell, can Anthony know that Sander's critiques are "substantial" without reading them?

In fact, Sander's response that Anthony links to is solely limted to a critiques of a single piece, which piece was cited in a single sentence in the Slate article. Inasmuch as the critique Anthony links to does not contain the quote attributed to Sanders in Slate and does not address Ayers and Brooks at all is is apparent that it is not the critique that is quoted in Slate.

Apparently, then, that neither the Ayers/Brooks piece nor Sanders response is yet available for linking. And it seems that Anthony's snarky comments about the lack of links are unfounded -- as, is his conclusion based on the lack of linking that "it is clear that the author already has an agenda". Anthony, you owe the author an apology.
4.30.2005 8:26pm
E.S.:
Unfortunately, affirmative action is one of several topics where virtually any and all authors will have some agenda. So, Anthony, it does no good to point out that the author has an agenda. Even the Volokh Conspiracy has an agenda when posting about the Sander series. They could have posted a clearing house on articles pro and con about AA, but instead cherry-pick this article.

Not that I blame them, or most anyone who has such an agenda. Disagreements about AA, at their core, are virutally all strongly ideological ones (a point which makes Sander's article "unique" in the sense you expect his ideology to side with AA). So it's difficult, if not impossible, to dispassionately assess the responses to Sander. If you don't like AA, you're likely going to view the responses to Sander as deficient. If you support AA programs, then the rebuttals will be solid and convincing.

The only real way to identify whether AA has been truly helpful or hurtful (or had no effect) for blacks would be to have parallel timelines, one with AA, one without. Do you really think that a strongly libertarian individual would concede that AA is good, even in the face of data that "conclusively" proved AA had helped blacks? Similarly, do you think a staunch conceptional supporter of AA would simply accept data that "thoroughly disproved" that AA was beneficial? Likely not. Their ideological committments will typically lead them to reject data that fails to conform with their conceptual norms. I do not believe there can be any dispassionate assessment of the effects of AA, by any side of the ideological spectrum.
4.30.2005 11:23pm
Robert Lyman (mail) (www):
I do not believe there can be any dispassionate assessment of the effects of AA

That's a rather bleak view, and I can't see any reason it couldn't be extended to pretty much any disagreement, could it not? Yet it is surely possible to dispassionately consider facts and evidence before coming to a conclusion. True, people will tend to believe data which supports their world view, but that doesn't justify the breathtaking cynicism of that sentence.

And as a conservative/libertarian, let me say that I would not concede that AA is "good" even if I were totally convinced that it were wonderful for black lawyers, because I think it is bad for the country as a whole. A staunch conceptual supporter of AA might support it even if it seemed to be bad for black students for some other reason; perhaps as a symbolic form of reparations or as a way of keeping race on the political adgenda

It is important to separate the reaction to the study from the policy proposals that any one person supports.
4.30.2005 11:32pm
Xavier (mail):
The discussion surrounding the Sander article is excessively politicized, but Sander himself is very balanced. Before doing the study, he was a supporter of affirmative action and even now he only supports scaling it back. If Sander's study is flawed, it's not because he has an anti-affirmative action agenda.
5.1.2005 12:01am
John Steele (mail):
I've long supported affirmative action, and still do, but I think that Sander has been treated shabbily by many of his critics—and the Bazelon piece is just awful.

If you take the affirmative action issue seriously (who doesn't?), then you should visit Sander's home page and read his main paper. In it, Sander takes a system-wide look at affirmative action and looks at a variety of effects flowing from the practice. His article is not, as some critics would have it, a single, narrow, speculative conclusion that can be simply "debunked" or simply "disproven." It's a web of inter-related observations, some based on hard data and some based on explicit assumptions, about the entire system of affirmative action. Based on the responses so far, don't be surprised if large parts of Sander's analysis are taken for granted by his critics, who will focus on narrow portions of his work.

Sander Site

That site also has a response by Professor David Chambers and three others, and a draft reply by Sander which, as Anthony notes above, leveled many criticisms, not the least of which were basic errors in arithmetic that had dramatic effects in Chambers's conclusions. Those two articles contain a sharp exchange, but you should judge the matter for yourself once the final versions are released. I wouldn't be surprised if the final version of the response by Chambers, et al., will be significantly rewritten in light of Sander's reply. (By the way, Sander's reply to Chambers, et al., is misidentified above by Anthony as being Sander's reply to Ayres/Brooks.)

That site does not have the three response critiques by Professors Ayres &Brooks, Professor David Wilkins, and Professor Michele Dauber; nor does it have Sander's reply to those critiques. Not all of those still-unreleased documents may even be in final draft as of today. Bazelon apparently has gained access to the unreleased papers (or draft papers) from someone close to the debate and that's what Bazelon's talking about. I don't know who her inside source is, but it's clearly not Sander. Regardless of Bazelon's sources, how can we evaluate Bazelon's triumphalist rhetoric about Sander's critics until we gain access to the source documents themselves? My suggestion? Let's wait and read those critiques and Sander's replies before we judge the issue. I cannot imagine that Bazelon is correct that Sander has nothing to say to his critics—but we'll know soon enough.

Bazelon also makes reference to a new article by Daniel Ho, soon to be published in the Yale Law Journal. Again, rather than conclude, as Bazelon does, that Sanders has no reply to Ho, let's see what Sander has to say.

So, until the articles themselves are available, Bazelon's article contributes precisely zero—unless, perhaps, the goal is to poison the well. One last time: let's wait to see what the articles say. To date, Sander has been very open about his data, his methodology, his assumptions, and his conclusions. He's been thorough and painstaking in his analyses and responses. I think that Sander—and his critics—ought to be judged on what they wrote, not on a slanted, conclusory summary by Bazelon.

This is such an important issue, and if we all are as patient and as thorough as Sander has been, we will all benefit from this debate, regardless of whether we draw the same conclusions he does. Like a good academic, Sander is not simply focusing on a desired result in a political debate and then using rhetoric suited to that end, but rather is asking systematic questions about what the data show is really happening in the world. For that, Sander deserves just a little respect.
5.1.2005 2:44am
John Doe (mail):
I refuse to share my expert opinion because VC's flippant enabling of comments is haughty and offensive.
5.1.2005 4:10am
anonymous (mail):
After reading Sander's paper and Bazelon's response I have come to a couple conclusions. The first is that Sander's papers seems correct. When you take the argument on its face value one would reason that the numbers would support it. Of course, as he admits, the black applicants that get pulled up into the ranks of Yale and Harvard are going to be allright. You have the best teachers in the country and the best students in the country. I think if you throw anyone into that situation they will do well. I think the real focus of his paper is on all the students that wouldn't have otherwise made it into law school or get sucked up in the cascade effect.

I think it is natural to assume that if you take someone who is less qualified and put them into a highly competitive environment they will tend to be the ones to sink and not swim. I think that he really changes the focus of the question. Yes when we deal with the top ten law schools those that get into them are greatly benefited. He takes the focus off of them and looks at all applicants. He shows that getting pushed from a school ranked around 80 to a school ranked around 50 doesn't help.

Lastly he shows that grades are extremely important at everywhere except the very top schools. I am at a 2nd tier school and I have very good grades. My GPA at the moment is at an A- level. At these schools the disparity between job opportunities and grades is magnified. The top 10% of our school will have a shot at a big firm job. The other 90% won't. So the kid who makes worse grades because he got bumped up into our school is really screwed. Their job options are more limited then they would have been had they gone to a 'lesser' school and outperformed.

Sander's really asks the question of fairness. Is it fair to help out those at the top at the expense of those under them?

Lastly I have no doctrate in math and I haven't personally seen the all the numbers that all of these opposing studies have looked at. I think the most important thing about Sander's piece is hopefully how it will change the debate. It will take the stigma out of being anti-affirmative action. Peopl often equate an anti-affirmative action view with a racist view. Hopefully this will prompt more empirical studies that actually look at how these systems are effecting the legal profession. Also this should hopefully create more openness in the process.
5.1.2005 4:12am
William Henderson (www):
John Steele is right. Sander's reply to his critics (four response articles forthcoming in Stanford Law Review) is not generally available. However, in her article in Slate, Emily Bazelon claims to have read all four response articles and Sander's reply. She goes on to write, "What does Sander have to say for himself, once the bloodletting is done? Not much that helps him."

I have also read Sander's reply in draft form (Sander sent me a copy about a month ago for comments). Suffice to say that Sander has lots to say that is very responsive and specific (and, yes, even persuasive) regarding these critiques. It is not appropriate for me to elaborate, but I am confident that any reader who goes to the trouble of actually reading Sander's 117-page Stanford Law Review article, the four response critiques, and Sander's reply, will realize that Bazelon has not accurately characterized this important and valuable exchange.

Further, when these commentaries become publicly available, I would encourage readers to do just that: Read them in full. Because of the politicized nature of this topic, there is a high probability that secondhand accounts, from either the left or right, are simply getting it wrong.

For my own take on Sander's views, you can read the January 10th-14th exchange I had with Sander as part of the Legal Affairs Debate Club.
5.1.2005 4:52am
BSKB (mail) (www):
I can think of all types of problems with AA, the first and foremost amongst these is articles like this. Sanders' work may be controversial, but carries some merit. But, that merit is shot down by people who dislike the idea of critiquing AA. I am at a tier 4 school right now and I can assure you that the rachet effect he describes is real and observable. I can go to any number of lower tier law schools and see blacks under represented due to forced representation in higher tiers. It's sad because the diversity they would provide is valuable wherever it ends up, but the school where at least some individuals would best fit does not actually get them. I don't worry about tier 1 black students, nor do I worry about white students who would otherwise fit into the upper echalon, I worry about the bottom half because that is where the effects are truly felt. This is often ignored because these schools are treated as marginal and therefore it's ok to be a little bit less fair to. Too bad it's at the expense of others.
5.1.2005 6:15am
Justin (mail) (www):
Anonymous, you *way* overestimate the importance of *law school* in a *legal education*. Quick, where did Johnny Cochran go to law school? What journal did Ted Shaw write for?

While the system of course favors people with good grades from top law schools in giving them opportunities, I dispute that other than a general correlation between LSAT score and intelligence, the LSAT, the primary driver of law school admissions, has anything to do with how good of a lawyer or advocate one can be.

I'm someone whose skeptical (because its too small, not because its too big) of affirmative action as a way of repairing the harm done to african american communities (though I think its a legal, if "silly" approach under the 14th amendment). Yet how friggin' paternalistic is it to say to people who belong to a culture society has screwed over the last several hundred years and say, "I know you want to go to law school, and I know society has put you on a lesser footing in doing so, but we're not even going to let you try. After providing you with inadequate education and training, you just can't handle it. Trust us. We wouldn't do anything that's not good for you. No law school for you."

Not to mention that if you continue to "let" Harvard and Yale use affirmative action, then lower schools will end up with no minorities whatsoever. And if you keep Harvard and Yale from using affirmative action, you've denied african americans who have had nothing but success there the chance to go there, depriving these communities of their leaders.
5.1.2005 1:24pm
Jesse Rothstein (www):
As someone who works in the area--I am an empirical economist, and am currently working on my own analysis of the data that Sander uses--I have read the relevant pieces carefully. (By the way, many of them are available on the web: Just google "Daniel Ho Yale," for example.)

My take on the debate is that Sander is dead wrong--or, rather, that the analysis he performs is so poorly related to the questions he wants to answer that we learn nothing about the latter from the former--but that the extant critiques, and particularly the Bazelon piece, largely miss the point.

Sander's argument has two parts. First, affirmative action leads those black students who would have attended law school in any case to go to more difficult law schools, where they do worse and are less likely to end up becoming lawyers. Second, a few black students are permitted to attend law school via affirmative action who would not have been admitted otherwise, but that these students are quite unlikely to pass the bar. His assessment is that the first effect depresses the number of black lawyers substantially, and the impact of the second is smaller than might be expected.

Most of the critiques focus on the adding-up issue, whether the net effect on the number of black lawyers is positive or negative. They largely--but not entirely--grant Sander's claim that AA is bad for blacks who would have attended law schools without it. This, in my view, is a shame, as this is by far the weakest part of Sander's argument. He relies on an econometric analysis here that doesn't pass the laugh test: No serious econometrician would believe for a second that his analysis establishes the facts he claims for it. (A hint that this component of Sander's paper might not be particularly strong comes from his decision to publish the paper in a law review: Without intending any offense to the editors of the Stanford Law Review, I don't believe that they are particularly well suited to evaluate an econometric study.)

In principle, Sander might have obtained the right answer even with bad methods. But that's not my read of the data. I've been working on implementing more plausible tests of Sander's "mismatch" hypothesis. Based on the analyses I've done so far, there is essentially no support in the data for the claim that affirmative action is bad for black law students above, say, the 10th percentile of the LSAT distribution. There may be a negative impact at the very bottom of the distribution, but I'm not prepared to take a stance on that until I finish my analysis.

Unfortunately, I don't have a paper finished, so I can't provide documentation for these claims just yet. But Daniel Ho's paper is pretty good in this regard.

It is a shame that this debate is so politicized: It leads partisans on both sides to grab onto arguments that they should be ashamed to stand behind. The Bozelon piece was, in my view, particularly poor in this regard: She made awfully strong claims that she didn't provide much support for. But the Sander piece is no better. Anyone who takes it to have provided any real information about the impacts of affirmative action on black bar passage rates, or on the number of black lawyers, has been bamboozled.
5.1.2005 2:29pm
MDP (mail) (www):
Bazelon: One way to increase the number of black lawyers might be to write a test that relies less on trick multiple-choice questions ...

Her breathtaking condescension towards blacks calls to mind the "secret sin" theory of political types.
5.1.2005 3:45pm
Challenge:
"The Bozelon piece was, in my view, particularly poor in this regard: She made awfully strong claims that she didn't provide much support for."

Sounds a lot like your post.
5.1.2005 4:04pm
Jesse Rothstein (www):
I said: "The Bozelon piece was, in my view, particularly poor in this regard: She made awfully strong claims that she didn't provide much support for."

Challenge said: "Sounds a lot like your post."

Fair enough. I tried to make that clear, with my apology for not providing a paper just yet. But I thought other readers might be interested that at least some econometrically-minded readers--you can click through to my CV to decide whether I know enough to speak on this issue--see Sander's approach as suspect. I'd encourage you to give his paper a careful read before relying too much on his conclusions, and to pay particular attention to the underlying statistical assumptions.

Still, I can describe to you my basis for saying that there's no evidence in the data for a negative effect of AA except at the bottom of the distribution. Sander's basic approach is to estimate the gap in outcomes among black and white students with the same LSAT scores and undergraduate GPAs. (He doesn't do exactly this, as he imposes several functional form and linearity assumptions that seem to me unjustified, and he sets up the regressions in a very odd way, but this is the essence of his approach.) He then labels the gap the effect of affirmative action. This labelling is quite suspect, as there is extensive evidence that black students, on average, underperform white students with similar test scores even when they attend the same schools. So one would expect there to be a substantial gap even in the absence of affirmative action. But it turns out that when one is careful to compare students with exactly the same qualifications (rather than in relatively wide bins), the gap in bar passage rates is only a couple percentage points at any point above the 20th percentile of the qualifications distribution. Again, until I finish my paper, you'll have to take my word for it, or pull down the Bar Passage Survey data set from Sander's web page and analyze it yourself.
5.1.2005 5:12pm
Anonymous Law Student:
That's true, Prof. Rothstein, however, Prof. Sander's paper is particularly focused on what effect affirmative action has on the bottom of the credentials distribution- namely, that it causes minorities to be drastically overrepresented in that area.

Equally important, even setting aside the affirmative action debate, is Prof. Sander's argument that students at the bottom of the class substantively learn less than they would if they were in a cohort to which they were more equal. This dovetails with Prof. Sander's previous study of academic support programs at law schools, which determined that the most effective academic support programs were those that took underperforming students and placed them in special (well, remedial) classes for the remainder of their first year. Prof. Sander determined that this had a positive, and fairly large, effect on those students' continued academic success.

Setting aside the affirmative action debate, perhaps Prof. Sander's article will best be used to end some of the egalitarian course/section assignments so common in law schools. "Tracking" by ability works quite well in education up to the graduate level (witness the success of gifted/talented programs for grade and middle school, honors/AP/IB classes in high school, and "honors colleges" and the like at universities), but seems to be wholly absent in law schools, even those with a wide range of student ability.
5.1.2005 6:20pm
Jesse Rothstein (www):
Perhaps, "Anonymous Law Student." But I don't take Sander to be talking particularly about the bottom of the credentials distribution at all. What he seems to be saying is that there is a large negative effect of being at the bottom of a particular school, and that affirmative action ensures that most blacks will wind up in the bottom of their classes. That doesn't have much at all to do with the bottom of the overall distribution.

In any case, I didn't mean to imply that I believe that there is a real, substantial black-white gap at the bottom of the overall distribution--there are some statistical issues I'd like to explore before I am willing to conclude that. And Sander certainly hasn't convinced me that there is a negative impact of being at the bottom of one's law school class on bar passage rates.
5.1.2005 6:42pm
Challenge:
"This labelling is quite suspect, as there is extensive evidence that black students, on average, underperform white students with similar test scores even when they attend the same schools."

And this is an argument for the efficacy of AA how? It seems to me if there is a gap even among similarly qualified whites and blacks, then that the gap would be further exacerbated by the mismatch problem. It may be true that Sander has over-estimated the gap created by AA, but your explanation isn't exactly a ringing endorsement of continuing AA. It seems the best thing one could do would be to find ways on reducing that gap in performance, not in promoting "solutions" which widen it.
5.1.2005 7:21pm
Want to pin it down (mail):
Jesse:

For those of us without quantitative backgrounds, can you break it down for me? Could you answer these questions?

1. As far as Jesse best understands Sander, Sander claims that ??? percentage of African-American class rank, drop-out before degree, and bar passage are explained by the gap in entering credentials. [I thought Sander said it was only half, but the Bazelon article suggests that Sander says it 100%]

2. As far as Jesse Rothstein best understands it, ??? percentage of African-American class rank drop-out before degree, and bar passage are explained by the gap in entering credentials.

3. As far as Jesse Rothstein best understands Sander's critics, they claim that ?? percentage of African-American class rank drop-out before degree, and bar passage are explained by the gap in entering credentials? [From reading Bazelon, I can't tell if the correct answer here is 0, or if it's "they can't say," or if it's "they don't offer any opinions on that.]

4. If an African-American law school applicant approached Jesse Rothstein and said that he/she had been admitted to a very top school (e.g., ranked 3-5) where the applicant's LSAT and GPA were significantly below the median for white admitteees, and to a school ranked 23rd where the student's LSAT and GPA were at the median for white admittees, but was wondering about this so-called "mismatch effect" he/she had been reading about, what would you say to that student? What, if anything, could a well-informed statistician say?
5.1.2005 7:51pm
A law student:
I'm curious as to where all of this data comes from.

Are law school grades public record?

If not, I hope they're not relying on some sort of voluntary survey of law students.
5.1.2005 9:21pm
Cheburashka:
Let's put the affirmative action question aside and focus on the extraordinary result that Ayers &Brooks have reached (and that's plainly the right thing to do; there just aren't enough black students at first or second tier schools to produce a statistically significant result):

Regardless of race, law school tier is entirely irrelevant in one's chances of passing the bar.

Could it be, perhaps, that what law school one attends is also entirely irrelevant to one's developed skill as an attorney?

Columbia, I want my $80,000 back!
5.2.2005 12:33am
Justin (mail) (www):
I don't know when you graduated from Columbia, Cheb, but I didn't learn one darn thing about the bar exam while here.
5.2.2005 1:51am
Jim Lindgren (mail):
I am someone who both favors affirmative action in law school admissions and would like a rational, fact-based discussion of how it might be tweaked to work better--whether increased slightly to increase any positive critical mass effect or decreased slightly to reduce any negative mismatch effect. With Grutter now happily settling the legality of affirmative action, I was hoping for such a discussion, but the initial public responses to Sander's work have certainly shaken my belief that such a rational discourse is possible.

I am posting to second the comments above from William Henderson. As Henderson indicates, the Elizabeth Bazelon article grossly misrepresents the nature of Richard Sander's responses to critics. I have rerun many of Sander's analyses, and (like Henderson) I have read drafts of some of the critiques and drafts of some of Sander's responses.

Last fall, when I read Sander's article and reran his data, I had at least five problems with his analysis or his article, which I will mostly not detail here. One is that Sander's finding in his study of 21 law schools that, compared to whites, African-Americans do as well in grades as their credentials predict depends on effectively coding those who do not disclose their race as if they were white. If one does not make that assumption and codes the unknowns as unknown, then African-Americans do slightly worse than their credentials would suggest, which means that not all of the differences that Sander documents are due to mismatch. In the earlier draft of the Ayres &Brooks paper (at least), they pick up on my finding and credit me for it (they were tipped off because Sander added a note indicating this coding effect to his final Stanford paper).

Yet most of the critiques of Sander's work that I have read so far have been surprisingly ineffective, except on one point that is in one section of Sander's long article: What would happen to the number of African-American lawyers if affirmative action were ended? As I've been quoted in the press, I would give very little weight to such counter-factual estimates because so much would change in a world without affirmative action. I am thus skeptical of Sander's estimates, as well as of the estimates of the Lempert group and of Ayres and Brooks (though my guess might be closer to Ayres &Brooks' estimate than any other, but for me it would be just a guess). As I have earlier suggested to reporters who called, I think that without AA many minorities would choose another field rather than law, in my opinion one of the few sound points that the Bazelon piece does make effectively.

I was particularly struck by Bazelon's endorsement of Daniel's Ho's Comment in the Yale Law Journal. I read the original version of Ho's comment, the presumed final version now on the web, and the first draft of Sander's response. Bazelon writes:


To begin with, there is the problem of "post-treatment bias," which means that it's a bad idea to control for a factor that is itself a consequence of the cause you're studying. That no-no is explained by Daniel Ho, a Yale law student with a Harvard Ph.D., in a forthcoming issue of the Yale Law Journal. (Here's Ho's piece; here's my brief summary.) When Ho ran his own tests, he found that attending a more elite school has "no detectable effect" on the rate at which similarly qualified black students pass the bar.

Interestingly, this is one of the five potential problems that I noticed when I read Sander's article last fall. His theoretical causal model is that low credentials at higher credentialed schools cause low law school grades, which in turn cause higher bar failures. The standard way to model this statistically is with structural equation modeling (SEM), such as LISREL or AMOS, which can fit just such multiple causal paths—an analysis that I planned to do as part of my critique of Sander.

So, after Ho criticizes Sander for positing a model that treats intermediate causes like other predictors, does Ho fit a statistical model that treats intermediate causes such as law school grades properly? NO! Ho just omits from his analysis the strongest predictor of bar passage, law school grades. So Ho makes a legitimate criticism, but doesn't follow it up by testing the actual causal model that Sander posits to see whether Ho's criticism actually matters. Sander, in his draft response to Ho in the Yale Law Journal, does do the SEM analysis that Ho's criticism implicitly suggests should be done and finds that Sander's conclusions are largely unchanged. I have not rechecked Sander's SEM analysis on this point, but for Bazelon to suggest that Sander is without a substantial response is both false and misleading.

So what analysis does Ho do instead, while omitting the strongest predictor (law school grades) of what he wants to predict (bar passage)? Ho does a matching analysis, using law school "tier" as a proxy for student credentials and as a test of Sander's mismatch hypothesis. For example, Ho matches the gender, LSAT, and UGPA of African-American students who go to tier 3 schools with African-Americans who go to tier 2 schools, and determines that they do similarly on passing the bar.

Yet the law school "tiers" that Ho uses are not actually tiers; they are nonhierarchical clusters. They are not like the US News tiers, which are hierarchically ranked on school quality. If you read the Codebook for the Bar Passage Study, you see that the clusters are not rank-ordered, but rather clustered by racial makeup, cost (which correlates so highly with public/private that the LSAC says that it is the same variable), size, faculty/student ratio, selectivity (which I think mean admission acceptance rate), mean LSAT, and mean UGPA.

The schools are then divided into nonhierarchical clusters, based on similarities that are mostly not using UGPA and LSAT. Each cluster except the first appears to have a range of academic tiers in it. Indeed, two clusters each have 50 schools in them, almost all schools in one tier are private and almost all schools in the other are public, so nearly 2/3rds of schools fall into just two clusters, with much heterogeneity in real academic tier within each cluster. Because the LSAC doesn't provide the data school by school, we can't even tell which clusters of schools are significantly different from each other on LSAT or UGPA. We can tell that if one reranks these nonhierarchical clusters by UGPA, cluster 2 has an UGPA of 3.319 and cluster 3 has an UGPA of 3.300, a difference so slight that even without school-by-school data and with 12,000+ students in these two clusters, the difference on UGPA between adjacent clusters is not significant.

By comparison to the small credential differences between clusters, the mean black/white difference in LSAC bar passage data is large--about .4 on UGPA and 9 points on the old 48 maximum point LSAT scale (these differences are also fairly close to the black/white differences within each tier except for tier 6, the historically black schools).

Yet the LSAT and UGPA differences between adjacent "tiers" are much smaller than these mean black/white differences, about 2-3 LSAT points on average between adjacent tiers, and about .13 on UGPA. The only comparisons between tiers that are as large as the usual black/white differences would compare:


(1) the traditionally black schools (tier 6, the smallest tier) with tier 1, 2, and 3 schools (on LSAT and UGPA);

(2) tier 1 schools with tier 5 schools (on UGPA and LSAT).

Yet the traditionally black schools are so different from the other schools that it would be hard to generalize from their experience to that of most law schools, and Sander's data and that of Bok and Bowen suggest that things might be a bit more positive for graduates of tier 1 schools.

None of the other differences between tiers are large enough to be as large as the mean black/white differences. Thus, while Ho is often actually mostly measuring cost (or in effect public/private distinctions), he is pretending that they are student credentials distinctions. Ho even recognizes that the tiers are not based solely on eliteness because he renames the bottom tier, tier 6, as "Historically Black Colleges or Universities."

The credentials differences between all adjacent tiers are too small to test Sander's mismatch hypothesis using Ho's method. If one perhaps limited the analysis to matching students who went to different tiers as far apart as the typical black/white entry credentials (about 4 tiers difference), then perhaps one might BEGIN to have something to say about Sander's hypothesis using matching methods. But if you read footnote 24 of Ho's comment, you will see that HE EXCLUDED ALL SUBSTANTIAL MISMATCHES of more than one tier difference! In other words, if the differences in tiers were large enough to begin to test Sander's mismatch hypothesis, Ho systematically excluded those matching students from the analysis! (The reason that Ho gives for omitting the most salient cases is because of unobserved variables, which is a more serious problem for Ho's approach than for Sander's because Ho relies so heavily on tiers. Yet it is precisely with differences of more than one tier that credentials differences are likely to be large enough to offset the effect of unobserved variables.) There really aren't enough cases with enough differences between tiers across a wide enough range of schools to test the Sander hypothesis using the LSAC bar passage data. Omitting the most salient cases (as Ho does) leaves only differences in credentials that are too small to test what Ho wants to test. Thus Ho cannot refute Sander's hypothesis because he does not actually test it.

Ho raises an interesting method to test differences in bar passage: matching the credentials of students who went to much more elite schools than others with the same credentials. Unfortunately, he doesn't have the data to do the analysis that he wants to do because his tiers are not tiers, but rather nonhierarchical clusters. This kind of embarrassing error to read the codebook carefully (or to understand the implications of what he read) has been all too common among Sander's critics. Ho's analysis depends on law schools being nicely divided into 6 hierarchical tiers of substantially different entry credentials, but the LSAC is quite explicit that it uses mostly criteria other than LSAT and UGPA to determine its clusters, and that the clusters are not hierarchical. Even if Ho failed to read the Codebook, he should have spent enough time with his data to realize that the credentials differences between adjacent clusters were too small to test the effect of credentials differences between African-Americans and whites.

The reason that Ho's paper is not a test of Sander's mismatch hypothesis is the same reason that several other critiques of Sander's work are so similarly unsuccessful. The LSAC has released its bar passage data only by clusters that are determined mostly by criteria other than student entry credentials or eliteness of school, not school by school. No truly effective refutation of Sander's bar passage claims is likely to occur until the school-by-school data are released. The data collected by the LSAC was promised to schools participating in the bar passage study, but so far the database has been released in a form that makes it largely unusable for sophisticated analyses such as Ho wants to do; the data are so bowdlerized that one can't even replicate the relatively straightforward analyses that the LSAC itself did. Because the clusters in the LSAC bar passage study do not split the schools enough to test the mismatch hypothesis, we are in an odd situation. My sense is that the LSAC wants to help those of us who want to test (and if warranted refute) Sander, but they haven't released the school-by-school data on bar passage that would make such a refutation possible. Without the data they need from the LSAC, Sander's existing critics are in a weak position, and many of their arguments are as ineffective as Ho's. I think that Sander may be vulnerable on several fronts, but his public critics so far are generally using either inappropriate data or inappropriate techniques to challenge him.

There are some genuine insights in Ho's Yale comment, such as his noticing something I had noticed earlier: Sander's failure to fit a causal model that actually fits his hypothesized causal paths. But Ho never follows through to use a technique designed for testing his criticism, such as SEM. And in his seemingly effective Yale response to Ho, Sander now does an SEM analysis that Ho should have done to prove that Ho's criticism actually matters (Sander's new analysis suggests that it doesn't).

Then Ho has the interesting idea to try matching, but he uses "tiers" based mainly on non-credentials issues such as cost (which translates to public v. private) as surrogates for student entry credentials and then systematically excludes the most relevant cases of substantial mismatch (tier differences of more than one tier).

Although unmeasured variables are always problematic in any archival study (including Sander's), their problems are particularly acute for matching studies such as the one Ho proposes to do, because if two students are matched by everything except the eliteness of school, it is highly likely that the student attending the more elite school is higher on unmeasured credentials (recommendations, work experience, etc.) than an otherwise matched student at a less elite school.

Thus, while matching studies are worth doing here, they would be systematically biased against supporting Sander's mismatch hypothesis, should it exist. But even if I might favor other methods of testing Sander's mismatch hypothesis, matching analysis is certainly worth doing. After all, it is best to use multiple methods to examine any important hypothesis--of which matching should probably be one. Now we just need to get the LSAC to release the school-by-school data so that Ho can actually test Sander's hypothesis.

Although with articles like Bazelon's we are not off to an auspicious beginning, I suspect that eventually we will be able to sort out which parts of Sander's thesis are probably correct and which parts are not.
5.2.2005 1:51am
Jim Lindgren (mail):
Jesse:

Sorry, but if I understand your comments correctly, I think you are proposing to make exactly the same error in your article as Ho makes. The LSAC bar passage data is not broken down by hierarchical academic tiers. It is broken down by nonhierarchical clusters based mostly on variables other than student LSAT and UGPA. If you read the codebook, you will see that you can't do what you are proposing to do. See my long critique of Ho's Comment above. I hope that this will save you from committing the same embarrassing error as Ho commits.

If you are able to get the LSAC to release the data so that you can validly do what you propose doing, let us know.


Cheburashka:

You raise sample sizes. The bar passage study has about 150 African-American students in tier 1 and 280 in tier 2, which would be enough for significance if the effect were fairly strong.
5.2.2005 2:05am
Jacob Zipfel (mail):
Dr. Lindgren,

Thank you very much for your explanation. I read Sander's full article, plus 2 of the critiques and Sander's main counter. But, its great when you come along and explain the background methodology in a manner a layman can understand. Your critique of Ho's data &methodology seems damning. While I am not too familiar with statistics, your point about his omission of the strongest predictor (law school grades) is a mistake some undergrad taking their first course in research methods might make. Additionally, his use of tiers or clusters from the LSAC data is borderline fraudulent. Its one of the first things you learn when undertaking social science research. Make sure the data you use actually represents the phenomena you are attempting to test (or at least close!) My methods teacher would have run me out of the classroom if I had tried what Ho is doing.

Question: Do you have a link to a site/document listing out your other four problems with Sander's work?

Another Question: How far could one get with FOIA requests to public universities to start tracking students (not in an identifiable manner of course) over a period of time? - to start establishing some good datasets. (Obviously this would omit private universities, but it could be a starting point.)

On a lighter side, I couldn't help from plugging my GPA and LSAT scores into the general admissions equation in the Sander's article. It may be uncouth for me to say this, but change my race and I would have been a shoo-in for law school in Michigan (according to Sander's tables), maybe even Harvard or Yale. I do acknowledge (at least in a normative manner) a need for representation of all (racial) groups. But practically speaking, it just sucks to know that for some people with the same credentials as myself, they get into top ten (or likely top five) schools for the same scores that get me into the top 15-25. Disclaimer: Yes, I realize this applies to legacies as well.
5.2.2005 2:33am
Anon Agan (mail) (www):
When discussing the "The LSAC" it is important to realize that a respected academic scholar at UCLA had to rewrite her paper since the data analysis methods they had suggested did not account for regression to the mean.

Which about catches it for law review editors and statistics knowledge.
5.2.2005 8:54am
Jesse Rothstein (www):
Jim:

I agree with you about the problem with the tiers/clusters. Although my read of the data is that Tiers 1 and 2--which sometimes are numbered differently; in any case, the ones labeled "elite" and "public ivy" in some of the LSAC analyses--are more distinguishable from the others in terms of entering credentials than you portray.

In any case, I don't propose to rely on the clusters at all for the analysis I describe above. Rather, I intend to ignore the clusters, simply comparing blacks and whites with the same entering credentials, who on average attend schools of different selectivity. This will, if anything, be biased toward finding evidence of mismatch, as there is extensive evidence in the "validity" literature that whites outperform blacks with the same credentials even at the same schools.

I do fall squarely into your criticism of Ho for not taking account of law school GPA in analyses of bar passage rates, but here I think your criticism is overstated. My (and Ho's) approach is to examine the "reduced form", where you propose to examine the structural model. I'm not religious about this, but here it seems to me that the reduced form is more informative. My reason for thinking this is that I suspect the law school GPA captures not just the effect of law school difficulty but also unobserved differences in entering credentials. The right structural model could deal with this, but it seems to me to be safer to leave that variable out: If attending selective law schools reduces blacks' success rates, either through reducing their GPAs or through another channel, the reduced form model should capture that.

As for the LSAC school-by-school data: My understanding is that they actually destroyed the school identifiers, and don't know either which student in the data set went to which school or even which schools are part of each cluster. So the data availability problem appears to be not the result of current intransigence, but of past excessively-legalistic adherence to privacy protections.

"Want to pin it down": Your questions are good ones, but they'd take me a while of looking through the Sander paper to figure out the exact answers. Unfortunately, I have to go teach...
5.2.2005 9:57am
Jim Lindgren (mail):
Jacob:

Thanks for your very kind comments.

I haven't yet released my other criticisms, though Sander knows of some of them. It's amazing what one can find if you really get deeply into the data.

I don't know about FOIA requests from state universities, though I think people have occasionally gotten some info from this method.

Two points on running your own numbers in Sander's model:

(1) Very few whites are actually denied places in top schools because diversity is also a criterion for admission--only those right on the (formerly) traditional credentials borderline.

(2) At most schools, legacy preferences are much, much smaller than affirmative action preferences. If AA was only as strong a plus as being a legacy, I doubt that AA would have ever been very controversial.
5.2.2005 11:51am
Jim Lindgren (mail):
Jesse:

Thanks for your comments. You write in response:


In any case, I don't propose to rely on the clusters at all for the analysis I describe above. Rather, I intend to ignore the clusters, simply comparing blacks and whites with the same entering credentials, who on average attend schools of different selectivity. This will, if anything, be biased toward finding evidence of mismatch, as there is extensive evidence in the "validity" literature that whites outperform blacks with the same credentials even at the same schools.


Perhaps I misread your earlier post. You wrote:


My take on the debate is that Sander is dead wrong . . . .

Sander's argument has two parts. First, affirmative action leads those black students who would have attended law school in any case to go to more difficult law schools, where they do worse and are less likely to end up becoming lawyers. Second, a few black students are permitted to attend law school via affirmative action who would not have been admitted otherwise, but that these students are quite unlikely to pass the bar. His assessment is that the first effect depresses the number of black lawyers substantially, and the impact of the second is smaller than might be expected.

Most of the critiques focus on the adding-up issue, whether the net effect on the number of black lawyers is positive or negative. They largely--but not entirely--grant Sander's claim that AA is bad for blacks who would have attended law schools without it. This, in my view, is a shame, as this is by far the weakest part of Sander's argument. ...

I've been working on implementing more plausible tests of Sander's "mismatch" hypothesis. Based on the analyses I've done so far, there is essentially no support in the data for the claim that affirmative action is bad for black law students above, say, the 10th percentile of the LSAT distribution. There may be a negative impact at the very bottom of the distribution, but I'm not prepared to take a stance on that until I finish my analysis.


OK, so you originally said that you were testing Sander's claims about passing the bar and becoming lawyers, so I naturally assumed that you were using the LSAC bar passage data, because that is the only data that I'm aware of that includes dropouts from law school, a significant part of the analysis on who actually becomes a lawyer.

But in your latest post you say that you are not using clusters, but rather are looking at students who "on average attend schools of different selectivity." But I didn't think that the LSAC bar passage data had any such variable.

My question is: Which database are you using and where do you come by your school "selectivity" data?

Thanks,

Jim
5.2.2005 12:32pm
Justin (mail) (www):
Jacob Zipfel,

you do realize that if you were born black, how unlikely it would be for you to have the resources and opportunities to have your gpa and lsat, right? I agree that this can apply to legacies coked up on long island ;).
5.2.2005 1:16pm
Jesse Rothstein (www):
Jim:

The part you quote from me is taken from an earlier post, in which I attempted to describe Sander's approach. As I mentioned, I don't think his approach is sound, and I don't plan to use it myself. (On the other hand, I can see how my "I've been working on implementing more plausible tests..." might have been misleading.)

I do plan to use the LSAC BPS data, and to test the hypothesis that Sander is implicitly trying to test: That blacks perform worse on the bar than do whites with similar entering credentials. Sander notes that blacks do seem to attend more selective schools--as best as we can identify selectivity in the BPS data--than do whites with similar entering credentials, and he labels the "black effect" on school selectivity and on bar passage the impact of affirmative action.

I'm not convinced that any black-white gap in bar passage rates among students with similar entering credentials can be attributed solely (or even primarily) to affirmative action, given the extensive evidence that blacks underperform within the same schools. I'm willing to believe Sander, however, that there is a black effect on school selectivity, and to attribute that to affirmative action. In that case, the black effect in bar passage reflects underlying black underperformance plus the impact of mismatch, so is upward biased relative to the mismatch effect on its own.

My only claim was that, when you use more flexible models than Sander's, and when you estimate things without the law school GPA--which, I think we agree, just mucks things up in Sander's analysis, though we disagree about how to correct for it--you don't see much of a black-white gap except at the bottom of the distribution. That is, we can write the black-white gap as a function of the LSAT and UGPA:

gap(LSAT, UGPA) =
Pr(pass bar | white, LSAT, UGPA) - Pr(pass bar | black, LSAT, UGPA).

For values of LSAT &UGPA above about the 20th percentile of the distribution, gap(LSAT, UGPA) is small, around 2 percentage points. Given that this is an upward-biased estimate of the mismatch effect, I read this as evidence that that effect is quite small.

(My statement about how blacks and whites "on average attend schools of different selectivity" was simply a statement that I am operating from the assumption that a similarly defined gap in school selectivity, if we had a good measure of that, would be large. I realize that the BPS data allow only imperfect tests of this--although I think the clusters are more informative than you seem to think they are--but I suspect that it is true even if hard to test well.)

Now, for very low values of LSAT and UGPA, the bar passage gap is quite large. But I suspect that this reflects sample selection, as it may be that many white students with these low credentials are unable to get into law school at all. I'm working on obtaining supplementary data that would allow me to evaluate that.

Does this clear up my earlier posts at all?

(I should mention, by the way, that I am working on this project with one of your colleagues, Albert Yoon.)
5.2.2005 1:41pm
Cheburashka:
Jim, you write: You raise sample sizes. The bar passage study has about 150 African-American students in tier 1 and 280 in tier 2, which would be enough for significance if the effect were fairly strong.

It seems to me that with numbers like that the effect would have to be extraordinary; how many years are those numbers over - are they even enough to overwhelm year-over-year changes?

I suppose that even when a study produces a statistically significant result, if the sample size is small and the time frame short, its unlikely to convince many people that the effect is real rather than an artifact of regression analyses.

Three more things:

Sander is plain wrong when he says that law firms hire more based on grades than based on law school prestige. The large prestigious firms only recruit out of a small number of schools in the first and regional-second tiers. There may be _some_ truth to Sanders' statement in the case of smaller firms hiring out of the lower tiers, but this has more to do with those schools being very difficult to rank comparatively.

Second, regarding the racial mixture of lawyers in general, I think its rather obvious that the number of black lawyers will drop in the absence of affirmative action, because the law schools and the profession of law compete with other professional schools and professions. If law schools abandon affirmative action, but business, accounting, and medical schools keep it, it seems to me that the cascade effect would become a simple drain as black students choose to go to Harvard med rather than Fordham law, and so forth down the tiers.

Third, if any of you statistical types are looking for an additional variable to track, I suggest looking at ethics complaint and disbarrment rates.
5.2.2005 1:51pm
Anonymous Law Student:
Cheburashka,

I think that you're allowing yourself to be blinded by what Skadden or Latham does when picking associates. Even though those are huge law firms, they represent a very tiny fraction of legal practice. Thus, what they do is not really the most relevant issue for determining what, as a whole, happens with lawyers. If you look at Prof. Sander's regression results, note the huge dropoff in salary after his tiers one and two (out of six or eight, depending on which model he's using at the time).

Justin- do you realize how racist your assumptions are? The African-American middle class is far, far larger than the downtrodden, inner city residents.
5.2.2005 2:18pm
Cheburashka:
ALS: A tiny fraction, true, but the tiny fraction that we care about.

One of the purported advantages of affirmative action is that it has downstream integrative effects - enforced racial balancing in the academy leads to natural racial balancing in the workplace.

Indeed, if affirmative action were eliminated and the 1st tier schools drained of their black students, those elite law firms from which associates launch careers in business and politics would be virtually devoid of incoming black associates.

Of course, that's the case anyway. (It doesn't take a regression analysis to figure out that affirmative action in professional schools is a pointless charade performed for the benefit of institutional appearance.)

So if you'll allow me to bring this back to the point, or lack thereof:

1. What possible benefit can there be to doing aggregate studies on the effect of a policy on what amount to only a few hundred people a year?

2. Does it matter at all to society what the racial balance is of the 5th tier law grads hanging around housing court trying to drum up business?
5.2.2005 2:44pm
AF (mail):
Professor Lindgren,

Thank you for your comments on Ho's article, which I am familiar with. I am not qualified to comment on your methodological critiques of matching (not being a statistician or having seen the codebook) but I want to see if I can clarify exactly what you're claiming about Ho's critique of Sander's mismatch hypothesis.

Ho's article makes two argument: first, Sander cannot validly conclude from his own model that AA is harmful to black law students, and second, a matching analysis detects no harmful effects of affirmative action on black law students.

I take it that you agree with Ho's first argument, his critique of Sander, and, in fact, that you have already made a similar critique of Sander's study. This is a fundamental critique, namely, that Sander's own results cannot be interpeted to mean that affirmative action causes black students to fail the bar. If valid, this critique means there is no evidence for Sander's mismatch hypothesis. It still remains to be proved; it does not need to be disproved. The working hypothesis is the still null hypothesis that affirmative action has no effect on bar passage reads.

I read your methodological critique of Ho's second argument in this context. Matching is *not* intended to disprove the mismatch hypothesis; it is intended to disprove the null hypothesis, and does not do so. Your critique goes to the weight we can give to the fact that Ho's matching does not disprove the null hypothesis. You say that we can't give it much weight and that the effect of affirmative action on bar passage remains highly uncertain.

Is that a fair characterization of your critique?
5.2.2005 2:44pm
Anonymous Law Student:
Cheburashka-

I think your basic assumption about what "we" care about is incorrect, as far as it relates to Prof. Sander's article- he is very clear that he is concerned about the aggregate effect on the supply of lawyers generally, not simply at the huge law firms.

Also, not even all supporters of affirmative action buy into the "let's place people at the very top!" argument you make- remember, there's a large proportion of people who want affirmative action so that these people will return to serve "their communities" as lawyers, which, if accepted, makes it very important what the racial balance of the "5th tier law grads hanging around housing court trying to drum up business" is.

Secondly, having a sample size of a few hundred people is not at all a problem, if the effect is large enough. From some data I've examined myself (from the 1995 National Survey of Law Student Performance), even a very small number of minorities produces results statistically significant at the 5% level for a slew of academic indicators, including entering undergraduate GPA, LSAT scores, and first semester grades.

More generally, you might be surprised at how little data is actually needed to produce statistically significant results when effects are large enough. While it isn't regression, as Prof. Sander used, try a google search for "t distribution" or "Student's curve" and read about how tiny the samples can be with those.
5.2.2005 3:33pm
Cheburashka:
ALS: I think you've missed both points.

First, I could care less what Sander cares about. Sander cares about what he can measure. Do I care about a policy whose effect is distributional among only a few hundred people a year in this country of hundreds of millions? Not really.

Second, just because a study produces a statistically significant result doesn't mean its proven anything. The law school application pool changes dramatically from year to year. If you're only looking at a small number of years, what can you be said to have proven even if your result is statistically significant?

There's a famous death penalty study that was done many years ago that concluded that each execution saved a certain number of lives.

I bring that up because through it I was taught a great lesson in statistics. I took criminal law at Columbia from the professor who had advised that paper when it was a thesis project. He pointed out in lecture that if you took out a mere two years from the late 60s, the result (and it was surely a statistically significant one) disappeared entirely. If the entire result was being generated by two years out of a hundred, what was the study proving? What did it mean - in human terms - to say that the result was statistically significant?

Not much, I think.
5.2.2005 4:56pm
Challenge:
1) Very few whites are actually denied places in top schools because diversity is also a criterion for admission--only those right on the (formerly) traditional credentials borderline."

This is an interesting point. If very few whites are denied spaces, then it follows that very few blacks benefit from affirmative action. Given the supposed insignificance of these numbers, maybe you would like to rethink your support of affirmative action.

In the Grutter v. Bollinger case the plaintiff had a mere 9% chance of admission. Let me see if you can take a guess what her chances would have been if she had been black or Hispanic. 30%? 40%? No, every applicant in her LSAT/GPA grid who was black or Hispanic was accepted. Yes, diversity is just "one of many factors" considered.

Let's say the above disparity was reversed--9% chance of admission if black, 100% chance if white. Now, because "very few" blacks would be denied admission--indeed only those applicants at the credentials borderline--does this change the underlying morality of the situation? Of course not. What I would think would matter, especially on a libertarian-leaning blog like Volokh, would be that individual rights are abridged. All of these econometric justifications in the name of sloppy social engineering make me want to puke. I have been patient, and read through the critiques, but the gist of things is that the critics of Sander's analysis seem to be criticizing a conclusion that, to my understanding, he never made--that affirmative action negatively impacts blacks in the top schools.
5.2.2005 6:10pm
Challenge:
"Second, regarding the racial mixture of lawyers in general, I think its rather obvious that the number of black lawyers will drop in the absence of affirmative action, because the law schools and the profession of law compete with other professional schools and professions. If law schools abandon affirmative action, but business, accounting, and medical schools keep it, it seems to me that the cascade effect would become a simple drain as black students choose to go to Harvard med rather than Fordham law, and so forth down the tiers."

Interesting. I don't know why it would be considered a bad thing if there were more black doctors but fewer black lawyers. Maybe that'd be a good thing. Moreover, I think your analysis leads us to the obvious conclusion--abandon racial preferences in university admissions and private hiring, as the Constitution and the Civil Rights Act of 1964 demand.
5.2.2005 6:17pm
Anonymous Law Student:
You're framing the question improperly yourself, Cheburashka. The pool is not every man, woman, and child in the U.S., but those applying to law school in that year, which is far smaller. Also, I question your blithe assertion that you don't really care about a policy that could substantially harm "only a few hundred people," as Prof. Sander claims. Erroneous executions harm, at most, a handful of people per year, out of the "hundreds of millions" of people in the country, yet we care about them. Obviously the harm there is greater, but it isn't scale that should be important, particularly when the small amount in question is, in fact, a very large percentage of African-American law students in any given year.

And, I think that you're misunderstanding the basic point about what statistics can and cannot show, particularly given your unsupported assertion that the law school applicant pool varies dramatically in any given year. There is variance, but that isn't the same as "dramatic," or more prosaically, significant variation. The onus is on you to demonstrate that the predictive power of Prof. Sander's model vanishes when examined over a larger period of time; regardless, it should concern people over the effects that were identified over that (I believe) five year period. Now, certainly you can charge that this is not a long enough time frame, but it needs to be supported by more than an ipse dixit.

Challenge- your point is, in my view, the correct one, and reminds me of an exchange during the Grutter oral argument, where the attorney for the University of Michigan said that very few whites were harmed by affirmative action. Justice Scalia asked her if she could name any instance where the Court had overlooked what it had identified as a constitutional violation because it only harmed a handful of people. She stood there silently for a few seconds before the Chief Justice told her that she could consider it a statement rather than a question.
5.2.2005 6:29pm
Jim Lindgren (mail):
Jesse:

Thanks for responding.

I am a big fan of the work of your collaborator and my colleague, Albert Yoon (and I have read nearly all of it). I haven't seen any of yours yet.

1. The grade gap.

You write:


I'm not convinced that any black-white gap in bar passage rates among students with similar entering credentials can be attributed solely (or even primarily) to affirmative action, given the extensive evidence that blacks underperform within the same schools.

And similarly, you write:


there is extensive evidence that black students, on average, underperform white students with similar test scores even when they attend the same schools. So one would expect there to be a substantial gap even in the absence of affirmative action.

What evidence of a "substantial gap" are you referring to? I have done several unpublished studies of single law schools, and I have rerun the numbers from Sander's study of 21 law schools. I find either no race effect or only a very small one, depending on the school and the model specified. I don't know the whole literature, but I was under the impression that only people who mistakenly lumped law schools together found large racial or ethnic differences. You refer to a "substantial gap" and "the extensive evidence that blacks underperform within the same schools."

Further, as I have been arguing for years (and as you hint yourself in passing), some of the supposed racial performance gap on grades disappears when you don't impose a linearity assumption on LSAT (since LSAT is seldom linear in its effects). From the data I've seen, there is probably a performance gap (contrary to Sander's main conclusion), but from what I've seen it is small, nothing close to the bar passage gap that Ayres & Brooks, the LSAC, and Sander separately document.

2. Splitting the bottom 20% from the top 80%.

You also write:

My only claim was that, when you use more flexible models than Sander's, and when you estimate things without the law school GPA--which, I think we agree, just mucks things up in Sander's analysis, though we disagree about how to correct for it--you don't see much of a black-white gap except at the bottom of the distribution. . . .

That is, we can write the black-white gap as a function of the LSAT and UGPA:

gap(LSAT, UGPA) =
Pr(pass bar | white, LSAT, UGPA) - Pr(pass bar | black, LSAT, UGPA).

For values of LSAT &UGPA above about the 20th percentile of the distribution, gap (LSAT, UGPA) is small, around 2 percentage points. Given that this is an upward-biased estimate of the mismatch effect, I read this as evidence that that effect is quite small.

I just ran frequencies on the LSAC bar passage data. In the file as a whole, 21.6% of students have an LSAT of 32 or below, so that would be about 20% of the overall distribution. As I read it, 75% of African-Americans fall within that bottom 21.6%, LSATs 32 and below. So are you saying that if you ignore 3/4ths of African-Americans (the ones with the lowest LSATs and thus the ones most subject to mismatch effects), then the mismatch effect is small? But if Sander is arguing about the effect of affirmative action on African-Americans, then why would you generalize how affirmative action works only after setting aside 3/4ths of African-Americans.

Further, are you sure that you have coded your outcome variable properly? I just split the file by LSAT 32 or below and 32.50 and above. I coded graduating and passing the bar on the first time v. otherwise (including those who ceased providing information because they dropped out, flunked the bar, or other reasons). Using logistic regression, with predictor variables of UGPA, LSAT, and black/nonblack, I get 35% lower odds of graduating and first-time passing for African-Americans with over 32 LSATs and 32% lower odds of graduating and passing for blacks with LSATs of 32 or lower.

If one excludes grades (as you at one point say should be done), the relative decrease in the odds of first-time passage is 40% for the bottom 20% of LSATs and 43% for the top 80% of LSATs. In other words, I get the same pattern for those in the bottom 20% as those in the top 80% of LSATs. But you say the white/black bar passage gap for the top 80% is only 2%, when I find the black/nonblack odds of passage gap for that group is 35%-43% (depending on whether you exclude UGPA or not).

Are you certain that you haven't made some basic error in your analyses, such as excluding the dropouts?

3. Your Models v. Sander's.

Last, Jesse, you started your comments by writing:


My take on the debate is that Sander is dead wrong--or, rather, that the analysis he performs is so poorly related to the questions he wants to answer that we learn nothing about the latter from the former . . . .

Yet the analysis that you now disclose that you are doing is very much like what Sander does in his Table 6.2 -- trying to explain the black/white bar passage gap using UGPA and LSAT. Your introducing a split between the bottom 20% and the top 80%, a distinction for which there is no particular theoretical justification, would seem to be not enough of a difference to justify your strong language saying that Sander is "dead wrong" or is using the wrong models to examine his questions. Your adding a questionable dichotomous variable to a predictive model is hardly such a different sort of analysis that it would avoid any major defects in Sander's analysis that could have prompted your strong statement.
5.2.2005 6:32pm
Ambrose (mail):
As a southern white boy who graduated from Michigan I think there is more to a legal education than good teachers at a good school. I would suggest that the differences are cultural, not racial. After all American blacks are not even seen as black in the rest of the world, there has already been so much mixing that there are few, if any, racial factors. Perhaps law schools don't do a good job educating their "black" students. If there is a difference in the bar results, it is the school that has failed the students.
5.2.2005 7:52pm
Challenge:
Is it just me or did Jesse just receive a "bloodletting" from Professor Lindgren. ;)
5.2.2005 7:55pm
Jesse Rothstein (www):
Jim:

I'm not sure what I did to offend you, but if I insulted you without so intending, I apologize. My sense is that you and I agree about more than we disagree about, so I'm not sure what went wrong.

I can answer two of your points at once: If you are interested in seeing my work, you can go to my web page, linked above. One of the papers there is called "College Performance Predictions and the SAT" (published in the Journal of Econometrics) in which I present evidence that black students do less well in college than do white students with the same SATs and high school GPAs (though that is not the main point of the paper). I am by no means the only person to find this; there are a variety of references in that paper to other research that obtains the same result. (Also, although in that paper I impose linearity, that is only because I get the exact same results with nonparametric methods, and no indication of nonlinearity.) I am less familiar with the literature on law schools, but my sense was that the same result has been found there as well. Perhaps I am wrong about that, however; as I said, I am less familiar with that literature.

I agree with you, also, that the majority of blacks are at the bottom of the LSAT distribution. So I wouldn't argue, as you describe, that one should "generalize how affirmative action works only after setting aside 3/4ths of African-Americans." I would simply argue that the fact that the results are different in the two groups--if you grant me that they are for the moment--suggests a more nuanced view of the issue than one would have taken away from the Sander article. I also suspect that the comparison between blacks and whites with the same scores is particularly biased by endogenous sample selection at the very bottom of the pool: My guess is that many whites with very low scores are unable to get into any law school, while this happens to fewer blacks. But this is pure speculation; as I've been saying all along, we haven't yet finished our analysis. Perhaps I should never have posted given that, but I've tried to be clear about the preliminary nature of the results.

I also can't understand your taking issue with my characterization of Sander's methods as wrong, as you have throughout this thread seemed to argue the same thing. I see a few basic problems with his analysis:

1) He never presents a reduced form analysis, but only a sort of poor-man's structural analysis: He labels the black effect on law school GPA and cluster the affirmative action effect, then multiplies through these effects by the coefficients of law school GPA and cluster in a regression for bar passage to get the affirmative action effect on bar passage. This strikes me as an extremely odd way to approach things. If one is willing to assume that blacks and whites are comparable conditional on LSAT and UGPA, then the reduced form analysis is the right way to go. If not, then it is hard to see how Sander's structure helps things; my guess is that it hurts substantially. (I see law school GPA as extremely likely to be endogenous, capturing unobserved qualifications, which would tend to bias Sander toward finding large AA effects.)

2) He imposes a linear model (or, rather, a logistic model that is linear in the variables). As you argue below, this is likely to be misleading.

3) He performs a lot of the analysis in bins. One of the important statistical facts that anyone working on black-white academic performance should know is that with normal distributions with same variance but different means, within any bin, the black mean is lower than the white mean. This, again, will tend to bias him toward finding larger AA effects.

4) The aforementioned failure to distinguish between the top and bottom of the distribution, so that he winds up drawing conclusions about the top schools that the data don't support.


My sense is that you agree with each of these (excepting, perhaps, the last). I see them--particularly the first--as extremely serious problems, enough to justify my claim that we learn nothing about the effect of affirmative action on dropout rates from Sander's analysis. (I don't disagree with his conclusions about cascade effects, although I don't see those as particularly original points.) The analysis that I have described is different in each point from Sander's. While I am the first to acknowledge that it is imperfect, the likely biases at least work against the conclusion that I am arguing for.

Finally, you ask whether I am certain that I have not made a basic error. I am not wholly certain--as I have mentioned throughout this thread, all of the results I describe are preliminary. But I am certain that I have coded bar passage correctly, or at least as correctly as can be done with these data: Anyone who was not observed to pass the bar is coded as not passing. I suspect that the difference between the 35-40% results that you present and the 2 percentage points results that I claimed derives from your confusion of odds ratios (your numbers) with probabilities of passing (my numbers). I haven't estimated logistic models, so I can't be absolutely sure, but the numbers seem in the right ballpark. For some purposes, one might want to present odds ratios, but that's not what I presented.
5.3.2005 10:42am
Challenge:
"I agree with you, also, that the majority of blacks are at the bottom of the LSAT distribution. So I wouldn't argue, as you describe, that one should "generalize how affirmative action works only after setting aside 3/4ths of African-Americans."

Actually that is is precisely what you have been arguing by propping up the efficacy of AA by ONLY look at the top 1/4.
5.3.2005 11:12am
Challenge:
"I'm not sure what I did to offend you, but if I insulted you without so intending, I apologize."

I think Lindgren was quite civil and stuck to the debate at hand. It also seems like, at least to this non-econometrician, that you have refused to address the many deficits in your analysis which Lindgren has brought to attention.
5.3.2005 11:19am
Jesse Rothstein (www):
Challenge, you write that "Actually that is is precisely what you have been arguing by propping up the efficacy of AA by ONLY look at the top 1/4."

I'm not sure what you are referring to. Here's what I wrote in my first post here:

Based on the analyses I've done so far, there is essentially no support in the data for the claim that affirmative action is bad for black law students above, say, the 10th percentile of the LSAT distribution. There may be a negative impact at the very bottom of the distribution, but I'm not prepared to take a stance on that until I finish my analysis.


I've not made one claim here about the "efficacy of AA." I'm focused on one particular question: Does affirmative action cause black students to do less well (i.e. to pass the bar exam less often) than they would if there were no affirmative action? I can imagine people supporting AA even if the answer turns out to be yes, and there are certainly people who would oppose AA even if the answer is no. I gather that you fall in the latter category. In that case, it is hard to see why the answer to this question is of great import to you.
5.3.2005 11:50am
AF (mail):
I have been enjoying the informative interchange between Professors Lindgren and Rothstein. At the risk of being repetitive, I have a question for both you regarding how a non-statistician can bottom-line the current state of your debate. The question is:

Do you both agree that Richard Sander's article provides little or no evidence for his claim* that affirmative action causes black law students to do worse on the bar than they would if there were no affirmative action?

Note that I am not asking whether you think there is any such evidence, only whether Richard Sander's article provided it. And I am not asking for your opinion on the significance of my question. I am just asking for a yes or no answer.

I don't mean to be rude--it's just that all the nuanced technical debate can sometimes obscure the clear points of consensus and their implications for the non-specialist's evaluation of Sander's article.

It would seem that the answer to my question is "yes."

Professor Lindgren has written:
"What would happen to the number of African-American lawyers if affirmative action were ended? As I've been quoted in the press, I would give very little weight to such counter-factual estimates because so much would change in a world without affirmative action."

Professor Rothstein has written:
"My take on the debate is that Sander is dead wrong--or, rather, that the analysis he performs is so poorly related to the questions he wants to answer that we learn nothing about the latter from the former."

If, in fact, the answer to my question is "yes," then the complex methodological debate you are having over how to measure the effects of affirmative action on bar passage is *not* a debate over whether Sander's claim* that affirmative action causes black law students to fail the bar is valid.

You both seem to agree that Sander's claim* is not valid.

While I think what I have written is clearly implied from what you have already said, I would very much like to hear it directly from you, particularly Professor Lindgren. Again, not to be rude, only to clarify the debate.

*Richard H. Sander, A Systemic Analysis of Affirmative Action in American Law Schools, 57 STAN. L. REV. 367, 447 (2004) ("[R]acial preferences in law school admissions significantly worsen blacks' individual chances of passing the bar . . . .").
5.3.2005 2:40pm
Jesse Rothstein (www):
AF: Yes, I agree.
5.3.2005 2:55pm
Challenge:
"I'm focused on one particular question: Does affirmative action cause black students to do less well (i.e. to pass the bar exam less often) than they would if there were no affirmative action?"

No, you're not. You are interested only in a fraction of the effect Sander describes--on blacks in the highest schools with respectable credentials. You've been pretty consistent. You wish to ignore the effect affirmative action has on the vast majority of blacks, preferring to emphasize the effect for the elite.

It's curious that you would argue against Sander's hypothesis by making a point which underscores it. I do not think Sander is opposed to the idea that affirmative action hurts the upper echelon of blacks and Hispanics less than it does for those in the 2nd, 3rd, and 4th tiers. But, assuming the validity of your criticism, lumping all of those groups together would reduce the estimated effect for the bottom 80%, even though it would exagerate the effect for the top 20%. Translation: If you're right and Sander's work is less relevant to the top 20%, then the suspected negative effect on achievement for the bottom 80% is much more pronounced than Sander has estimated. That may be a flaw in Sander's work, but it is not the fatal flaw you and his other critics have attempted to find. In fact, it may be a "flaw" which results in a great under-estimation of the negative effect on the vast majority of blacks and Hispanics, even if it over-estimates the effect for the elite minority in the best schools.
5.3.2005 4:32pm
AF (mail):
Challenge--

You are missing the important point that there is scant evidence for Sander's hypothesis in Sander's own article. That is the criticism made by Daniel Ho in his article and accepted by both Rothstein and Lindgren (who, by his own account, made the same point before Ho did.)

Thus, the flaws in Sander's work are clearly fatal. The question being debated is whether there is any evidence for the unsupported claim Sander made.

Note that the debate between Rothstein and Lindgren are having is over the gap in bar passage between black and white students who enter law school with similar characteristics (mainly undergraduate GPA and LSAT). Even if this black/white gap existed across the entire universe of law students, it could not necessarily be attributed to affirmative action.

Indeed, Professor Rothstein, I share Professor Lindgren's confusion as to why you are using the black/white gap to test the mismatch hypothesis at all. If, as you noted, the black/white gap exists in other contexts where the mismatch hypothesis is not at issue, why would we ever expect the existence of a black/white gap to confirm the mismatch hypothesis?
5.3.2005 5:00pm
Challenge:
Lindgren and Rothstein disagree over the gap among simiarly qualified whites and blacks, so what? If there is no gap, then that supports Sander's analysis on that point. If there is one, then that doesn't mean there isn't a gap due to AA, or that Sander's analysis doesn't capture some of it. Rothstein's point was that he didn't control for this gap. Lindgren says there is no gap.

Rothstein has relied on some pretty dubious claims in order to join in on the chorus. Most of Lindgren's criticisms, as far as I can tell, are being held in reserve until he can verify and confirm them.

I have seen no "devastating" critiques of Sander's methodology displayed on this thread. Maybe Sander's methodology is flawed, but it is certainly no more flawed than Rothstein's sorry attempts at criticism.
5.3.2005 5:23pm
AF (mail):
The devastating critique of Sander is that the only evidence he offers for the mismatch hypothesis is the black/white gap in grades and test scores. He provides no evidence that the black/white gap is attributable to affirmative action, yet he attributes it to affirmative action.
5.3.2005 5:37pm
AF (mail):
In the previous post, by the "mismatch hypothesis," I mean the hypothesis that affirmative action results in fewer black lawyers.
5.3.2005 5:39pm
Challenge:
If there is no gap among similarly qualified whites and blacks (which Sander explicitly states he believes in his commments on this blog, and which Lindgren similarly concurs on this very thread), then what possible explanation would be there besides AA?
5.3.2005 5:40pm
Challenge:
And on this point, I must add that Rothstein has now confessed his ignorance of the literature on this question.
5.3.2005 5:42pm
AF (mail):
Challenge--

Let's make sure we're on the same page here. Here are the undisputed facts:

1) White students get better law school grades and do better on the bar than black students with similar LSAT scores and undergraduate GPAs.* This is the black/white gap.

2) White and black students with similar LSAT scores, undergraduate GPAs, and law school GPAs perform similarly on the bar.

From these facts, Sander infers that affirmative action causes black law students to fail the bar.* However, there is absolutely no reason to believe that affirmative action, rather than some other difference between whites and blacks, explains the black/white gap.

I don't know what the reasons are for the black/white gap, but that doesn't mean affirmative action is the cause. Race is a significant causal factor in all sorts of contexts where affirmative action is not at issue. For example, white students get better freshman grades than black students at the same colleges with similar SATs and high school GPAs. That black/white gap cannot be explained by affirmative action.

*Though as Professor Rothstein notes, this is only true at the bottom of the LSAT distribution, where, as Professor Lindgren points out, most black students are.
5.3.2005 5:59pm
Challenge:
"Let's make sure we're on the same page here. Here are the undisputed facts:

1) White students get better law school grades and do better on the bar than black students with similar LSAT scores and undergraduate GPAs.* This is the black/white gap."

That is NOT an indisputed fact, it has been disputed on this very thread by Lindgren, who is a critic of Sander's work!

Both Sander and Lindgren do not believe in a pre-AA achievement gap among similarly qualified whites and blacks (or in Lindgren's case, he doesn't seem to believe in a "substantial" gap).

Sander:

"Consequently, blacks as a group have academic trouble in law school in very consistent and predictable patterns. At American law schools that use large racial preferences, half of all black students end up in the bottom tenth of their first-year class. Put a little differently, the median black student performs in the first-year at about the 7th percentile of the median white student. The gap is statistically no different in legal writing classes than in classes with timed exams. And, when we adjust for dropouts, the black-white gap gets slightly wider over the second and third year of law school.

It's important to note that this performance gap has nothing to do with race per se; whites who attend law schools where their credentials are far below most of their peers have pretty much the same types of troubles. The performance gap is a function of preferences. (Now, it's true that the preferences come about in the first place because of the black/white credentials gap - but that is another story, which I'm happy to address later if readers are interested.) There is no credible evidence I've seen that, if schools used race-blind admissions, blacks would underperform whites at all. (I will discuss this issue further, and respond to some reader commentary, this Friday.)"

Lindgren, on this very thread, stated:

"What evidence of a "substantial gap" are you referring to? I have done several unpublished studies of single law schools, and I have rerun the numbers from Sander's study of 21 law schools. I find either no race effect or only a very small one, depending on the school and the model specified. I don't know the whole literature, but I was under the impression that only people who mistakenly lumped law schools together found large racial or ethnic differences. You refer to a "substantial gap" and "the extensive evidence that blacks underperform within the same schools."

Further, as I have been arguing for years (and as you hint yourself in passing), some of the supposed racial performance gap on grades disappears when you don't impose a linearity assumption on LSAT (since LSAT is seldom linear in its effects). From the data I've seen, there is probably a performance gap (contrary to Sander's main conclusion), but from what I've seen it is small, nothing close to the bar passage gap that Ayres &Brooks, the LSAC, and Sander separately document."


Rothstein responded to Lindgren:

"I am by no means the only person to find this; there are a variety of references in that paper to other research that obtains the same result. (Also, although in that paper I impose linearity, that is only because I get the exact same results with nonparametric methods, and no indication of nonlinearity.) I am less familiar with the literature on law schools, but my sense was that the same result has been found there as well. Perhaps I am wrong about that, however; as I said, I am less familiar with that literature."
5.3.2005 6:32pm
AF (mail):
Challenge--

The white/black gap I'm referring to is for all students, not students within individual schools. Lindgren is talking about individual schools.

I misspoke, however, when I said the gap existed for law school grades. It seems to exist for bar results but not law school grades. In the passage you quote, Sander is referring to grades, not bar passage. He does document a white/black gap for bar passage; indeed, that is the basis for his claim that affirmative action causes blacks to do worse on the bar.
5.3.2005 7:17pm
Jim Lindgren (mail):
Jesse:

You have done nothing to offend me, which is why our exchanges have been (and I suspect will continue to be) so civil.

I appreciate your distinguishing your analysis meaningfully from Sander's in several respects (as you now have), which was not entirely clear from your earlier postings. I will await your paper to assess these moves.

I have written a long post in which I discuss what I believe to be the sources of our discrepancies, but your certainty that the coding of your bar passage variable is correct gives me pause. Thus I am posting this briefer note first.

You say that:


"I am certain that I have coded bar passage correctly, or at least as correctly as can be done with these data: Anyone who was not observed to pass the bar is coded as not passing."


From the LSAC bar passage data I'm using, I can't figure out how that can be true, but I can get very close to your estimate if I mistakenly exclude those who didn't graduate from law school from the analysis, as I previously suggested you might have done. But you have now presumably looked into that possibility since I pointed to that to check, and you are still certain your coding is correct.

It should be a snap to determine which of us is right if you will email me a copy of the bar passage database that you are using and an indication of which dependent and independent variables you are referring to in your analyses to get to your estimate of about a 2% black/white gap for those in the top 80% of LSAT or LSAT/UGPA.

Or if you used the database downloaded from Rick Sander's website, as I expect you did, then just post here which of his variables you used, and how you recoded them, including how you divided the cases into the top 80% and the bottom 20%, and which statistical models you ran.

In addition, would you please post here the frequencies for each value of the dependent bar passage variable that you used in your analyses (ie, # passing, # not passing), indicating whether this total is for all ethnicities, or just for whites and blacks combined. That should make things clearer for anyone who can download the Sander database to check for themselves.

I hope that you will appreciate my cautious way of proceeding in this matter. I have no doubt that we can get to the bottom of our differences quite simply. That is one of the nice things about social science done in a cooperative spirit.

Respectfully,

Jim

James Lindgren
Benjamin Mazur Research Professor
Northwestern University
5.3.2005 9:12pm
Challenge:
"The white/black gap I'm referring to is for all students, not students within individual schools. Lindgren is talking about individual schools."

The discussion was brought up that within the same schools, as for obvious reasons comparing different schools to one another can be problematic. If blacks are doing the same as their white peers when they're similarly qualified, and that this holds for each college, then it can indeed be said that there is no racial gap among similarly qualified whites and blacks.

"I misspoke, however, when I said the gap existed for law school grades. It seems to exist for bar results but not law school grades. In the passage you quote, Sander is referring to grades, not bar passage. He does document a white/black gap for bar passage; indeed, that is the basis for his claim that affirmative action causes blacks to do worse on the bar."

I think you misspoke again. The gap, among blacks and whites broadly, occurs in dropout rates, grades, and bar passage rates. Sander attributes this gap to affirmative action "mismatch" and does not correct for any original "race gap." He does not believe one exists, and Lindgren seems to support that contention, or concedes that it may exist but that it is much smaller than thought in some literature. Again, that is not an "indisputable" fact. Maybe it is correct, and maybe it is not. I think it is telling that when analysis is properly confined to a given school that the effect is non-existent or is very small. Perhaps it is acceptable to aggregate data like that, and compare apples to organges, but it strikes me as poor methodology. Maybe it isn't. I'd like to know what Lindgren and even Rothstein think about that question.
5.3.2005 10:01pm
Challenge:
"some of the supposed racial performance gap on grades disappears when you don't impose a linearity assumption on LSAT (since LSAT is seldom linear in its effects)."

This struck me. Of course the LSAT is not linear! Does anybody really think there is much of a difference between a 170 and a 180? Yet I doubt people would say the same for 145s and 155s. Raw score would probably be more linear, but I would again think you would see some serious decreasing marginal returns there as well.
5.3.2005 10:10pm
Jim Lindgren (mail):
Jesse:

Just to clarify my last post on your Bar Passage data on the probability of becoming a lawyer.

When I said that I was able to get close to your estimate (of a 2% black/white gap for the top 80% of LSATs in the probability of becoming lawyers) if I micoded bar passage by excluding those who dropped out of law school, I was referring to probit (probability) models, which I always assumed that you used, not logistic regression (odds) models.
5.3.2005 11:13pm
Cheburashka:
From ALS:

You're framing the question improperly yourself, Cheburashka. The pool is not every man, woman, and child in the U.S., but those applying to law school in that year, which is far smaller. Also, I question your blithe assertion that you don't really care about a policy that could substantially harm "only a few hundred people," as Prof. Sander claims. Erroneous executions harm, at most, a handful of people per year, out of the "hundreds of millions" of people in the country, yet we care about them. Obviously the harm there is greater, but it isn't scale that should be important, particularly when the small amount in question is, in fact, a very large percentage of African-American law students in any given year.


Who's "we"? I'm sorry but I don't see your point. The impact of affirmative action on minority bar passage rates, based simply on the miniscule numbers allegedly involved, has to be about the least important issue facing legal pedagogy today.

And, I think that you're misunderstanding the basic point about what statistics can and cannot show, particularly given your unsupported assertion that the law school applicant pool varies dramatically in any given year. There is variance, but that isn't the same as "dramatic," or more prosaically, significant variation. The onus is on you to demonstrate that the predictive power of Prof. Sander's model vanishes when examined over a larger period of time; regardless, it should concern people over the effects that were identified over that (I believe) five year period. Now, certainly you can charge that this is not a long enough time frame, but it needs to be supported by more than an ipse dixit.


Ah, but you see I do have something better than a mere ipse dixit: practitioner's experience.

My experience tells me that even when the results are statistically significant, I can safely look skeptically on, and indeed ignore entirely, econometric analyses that purport to reach sweeping results based on small numbers and short periods of time.

My experience also tells me that the year-over-year changes in law school demographics are extraordinary. Applications for the class of 2005, and consequently the credentials of students, shot up over the preceeding years as a result of the internet and stock market crashes. There was a surge in white female law applicants in the mid and late 90s (I suspect Hillary-inspired, but that's just a guess). What do you think the effect of either of those trends would have been on the ratio of black to white credentials among law students?

In the early 90s my alma mater, Columbia, redesigned its 1st year curriculum for the express purpose of being more appealing and friendly to female and minority law students - what effect would that have had on Sander's underqualification-low grades-discouragement-bar-failure chain of causation?

What about cultural changes among black college students? Changing levels of school cost, financial aid, and expected income? Changes in race-sensitive policies at workplaces and other professional schools and their impact on the applicant pool? And so on.

To bring this back to the statistics, a analysis can only talk about that which it is analyzing - in this case, five years. Sander cannot have controlled for the year-over-year demographic shifts I mentioned. He cannot have controlled for the curricular changes that were taking place. He can speak only about that small number of years.

And are we sure that its true of each those years, or could it be true in only some of them and in the aggregate? Could it, like the death penalty study, be an artifact that disappears when we delete a year or two?

Is it geographically uniform, or does it vary by region?

The point of this is not that econometric analysese are useless. It is that in our big, bizarre, and always changing universe, to say that a statistically significant effect was found in a miniscule slice really doesn't tell you anything.
5.4.2005 1:43am
Challenge:
I share's Cheb's skepticism of econometric analysis, but for a different, though related, reason. I believe the greater barrier is in the complexity econometrics tries to reduce. Not that some weight can't be given to models like Sander's and his critics. But I think both the general public and the "experts" overvalue their usefulness.
5.4.2005 2:35am
Jim Lindgren (mail):
I just reread the longer version of the Daniel Ho's Yale piece and I noticed something amazing. He is purporting to test Sander's hypothesis that AA and mismatch leads to such poor performance in law school that fewer African-Americans become lawyers because fewer graduate and pass the bar. But it is clear from the bar passage means in Ho's longer paper that Ho has excluded dropouts and those too discouraged to take the bar from his bar passage outcome variable!

Wow--that's a pretty big mistake to make in a published paper! And there is nothing in the Yale version of the piece that hints that he hasn't made exactly the same mistake there.

It's stunning that so many people could have read these papers by Ho without realizing all the major problems with Ho's analysis--

(1) Ho excludes any substantial mismatches of more than one tier from his mismatch tests in the YLJ when these would be the cases most important to include.

(2) Ho excludes dropouts and nontakers from his mismatch tests, even though these are important parts of Sander's mismatch hypothesis.

(3) Ho confuses nonhierarchical clusters for hierarchical academic tiers, which are absolutely crucial to his matching method.

It just goes to show that you can't believe everything you read in the Yale Law Journal or in Slate.

And No, I don't believe that peer review necessarily would have caught these devastating mistakes in Ho's work, since his work was read and evaluated by many eminent professors who review papers for journals.
5.4.2005 3:13am
Steve Sailer (mail) (www):
There appear to be two ways blacks might be hurt by affirmative action:

The first is "mismatching" where blacks who would have thrived at a slower-paced law school are thrown in over their heads at an elite law school and end up learning less. That's an interesting theory but I can certainly see how it would be hard to prove or disprove considering how little data we have in general on how much value education adds.

The second, however, seems more straightforward: college graduates who don't belong in law school at all are more likely to be admitted if they are black than if they are white. A friend of mine who was witty, charming fellow, but no scholar, spent three years at law school and then several more years working menial jobs while he took and repeatedly flunked the bar exam. He ended up wasting a half dozen years of his life during which he could have taken his college degree and made good money as a salesman, a job he was better suited for.

This problem of sending academic cannon fodder off to be slaughtered by the bar exam would seem like it inevitably must be worse for blacks than for whites due to affirmative action-driven law schools having lower standards for blacks. Has anyone quantified the size of this effect?
5.4.2005 5:08am
Challenge:
Jim, why don't you do Volokh's readers a favor and post something about your findings and criticisms on the blog. Just a thought....
5.4.2005 5:18am
Jim Lindgren (mail):
Challenge:

I will post on this topic some day, but I might wait a few weeks until I see the final drafts of the Stanford articles.

Thanks,

Jim
5.4.2005 12:42pm