pageok
pageok
pageok
Responding to Critics (3): Selection-Bias Blues

I've posted in this space data that shows blacks who pass up the best law school that admits them, and go to their "second-choice" school, are closer in credentials to their classmates and have much better outcomes during and after law school. The postings have generated much discussion. Professor Dirk Jenter, while defending me from the social science nihilism of "Mahan Atma", offers a pointed critique of the "second-choice" analysis: isn't the analysis contaminated by self-selection? The students going to their second-choice schools are, of course, doing so consciously; maybe that means they're a group that believes they will optimize performance at a less elite school, which makes their subsequent, superior performance at those schools and on the bar exam less surprising.

Selection-bias problems are an ever-present danger in this type of observational data, creating pitfalls which more than one of my critics have fallen into. It is probably not possible to eliminate entirely all danger of selection-bias in this comparison of first- and second-choice students, but I am pretty confident that there's little or no such bias here, for several different reasons. (See my "Reply to Critics" for a fuller discussion)

First, these students chose responses indicating that financial or geographic factors led them to turn down their first choice school and go somewhere else. And their other answers to the detailed surveys they completed were consistent with those answers -- although they cared about school "eliteness" almost as much as other students, they cared about "cost" and "financial aid" a lot, too. So, the motivations of these students didn't seem related to some kind of strategy of seeking out a less competitive environment.

Second, we have a wealth of data about the strategies of these students as they started law school; in every way I've been able to measure, they seem to be approaching law school with strategies and expectations that are indistinguishable from all the other black students. For example, both the second-choice and other students are equally likely to respond that they are "very concerned" about getting good grades in law school (89% vs. 88%), and both groups are equally likely to think they are going to end up in the top tenth of their law school classes (37% vs. 38%). Blacks in general express more concern in the survey data about passing the bar -- but, ironically enough, both black going to second-choice schools and all the other blacks tend to think that going to a more elite school will improve their chances on the bar. All of this data points against selection bias.

Third, it is important to keep in mind that this entire exploration of the "second-choice" phenomenon is a way of confirming the hypotheses I developed and tested with entirely different data in my original article. I didn't observe this high performance among blacks going to second-choice schools, and then construct a theory around it; this data was brought to light by others after Systemic Analysis had gone to press. In Systemic Analysis, I'm comparing blacks (as a group that generally is boosted into more elite schools by racial preferences) against whites (who sometimes receive preferences, but generally don't), while controlling for entering credentials. Certainly there's no self-selection process there (or only a little, accounting for students with mixed-race backgrounds). What's nice about the first-choice/second-choice analysis is that it avoids arguable pitfalls of the white/black analysis, and vice versa. But both methods produce essentially identical results.

"Michael" raises another interesting issue. In estimating the average "credentials gap" facing blacks at their second-choice schools (and comparing that with the credentials gap facing other blacks), I use the six loosely-defined "tiers" in the LSAC-BPS database. The creators of this database grouped schools into "clusters" by using some indicators of prestige (e.g., student scores) and some indicators unrelated to prestige (e.g., public sector vs. private sector). The six tiers certainly correlate substantially with school prestige, but they also undoubtedly overlap. So the most elite Tier 2 schools are almost certainly higher-ranked than the least elite Tier 1 schools, even though Tier 1 as a whole is clearly more elite than Tier 2 as a whole.

Consequently, one needs to be careful about using the tiers in sensible ways. In the second-choice analysis, I know each student's grades (standardized by school) and individual outcomes (e.g., graduation), but in terms of school identity I only know what tier they are in. I compared each student's index score to the median index score of students in the same tier to estimate the typical credentials gap between students and their classmates. This is, of course, a rough measure -- but the key concern is whether there's some reason to think it's biased in a way that helps my analysis. The answer is: I don't think so. Blacks other than the second-choice students should be distributed more or less randomly across the six tiers. The blacks going to second-choice schools should also be pretty randomly distributed -- with one exception. In Tier 1, they are more likely to be in schools near the bottom of that elite tier rather than at the top (since they have generally rejected a more elite school, they are unlikely to be going to Harvard or Yale). But that distortion would mean I was overstating their actual mismatch with their fellow students, which cuts against my analysis, not in favor of it. And, in any case, for the whole group of students this is likely to be a small distortion indeed.

Consider, by way of contrast, an analysis done by Dan Ho (a recent Yale graduate and Harvard Ph.D. whose critique, and my reply, are in the June issue of the Yale Law Journal). Ho compared blacks who had the same index scores but attended schools in adjacent tiers. Ho's argument went like this: if two blacks with the same entering credentials went to adjacent tiers, and passed the bar at the same rate, then there's no penalty to blacks from going to higher tier schools, and Sander is wrong. But of course, if the BPS tiers overlap, then for any analysis which selects blacks from, say, Tier 1 and Tier 2 schools who have matching index scores, it's quite likely that these students are actually going to schools of equivalent eliteness. Here the bias is likely to be quite significant, and it all is in the direction of the results Ho wants. Ho's analysis is invalid for other demonstrable reasons, but I offer it here as an example of an improper use of tiers.

A final point: "Michael" also asks whether we know the blacks going to their second-choice schools actually got into their first-choice school. The answer is yes, we do -- the questionnaire asks that question, and we used it as a filter.

Tomorrow I'll address some critiques of Systemic Analysis itself.

Jadagul (mail):
Something I've been wondering about: if students are turning down their first-choice schools for financial reasons, say, does that mean that second-choice students are more likely to have attended law school on scholarships (or simply gone to cheaper law schools)? If that's true, would they have been able to spend less time working their ways through law school, and thus more time studying? Would this affect their performances?

Similarly, it's possible that when students chose schools based on geographic factors, these geographic factors made school easier on them, and thus helped them get better grades? I'm basically asking a variation on Professor Jenter's question: could whatver factor caused the student to choose his second-choice school over his first-choice school made it easier for him to learn/achieve at his second-choice school, and thus independently increased his chances of passing the bar, becoming a lawyer, etc.?
6.14.2005 11:47pm
AF (mail):
Professor Sander,

All of the evidence you have presented for your mismatch hypothesis--that law school grades strongly affect bar passage, that blacks students do worse on the bar than white students with the same pre-law school characteristics, and now, that second-choice students do better than first-choice students--is indirect.

It seems to me, however, that there is direct test of the mismatch hypothesis available--and it disconfirms the hypothesis.

According to the mismatch hypothesis, students who go to prestigious law schools for which they are underqualified will do worse (ie, graduate and pass the bar at lower rates) than they would have done had they gone to less prestigious law schools that are more their speed. The direct test of this hypothesis is to measure the effect of law school prestige on graduation rates and bar passage. The mismatch hypothesis predicts that, holding all pre-law-school characteristics equal, law school prestige should relate negatively to graduation rates and bar passage.

But it doesn't. All else equal, law school tier has an insignificant or positive effect on graduation rates and bar passage, thus disconfirming the mismatch hypothesis.

That is Dan Ho's basic criticism of Systemic Analysis: in marshalling your various indirect arguments for the mismatch hypothesis you miss the direct test, which is the effect of law school prestige on bar passage (and graduation rates).

This basic criticism of Ho's does not depend on the matching analysis which you claim to be flawed; your own regression analysis finds no negative effect of law school tier on bar passage.

In comments to a previous post here on the volokh conspiracy, (see "Debating Affirmative Action in Law Schools"), Jim Lindgren defended you from Ho by arguing that the law school tiers in the data set are non-hierarchical and therefore tell us essentially nothing about law school prestige. In contrast, you maintain that "[t]he six tiers certainly correlate substantially with school prestige."

I'm interested to hear your explanation why the lack of negative effects of law school tier on bar passage does not disconfirm the mismatch hypothesis.
6.15.2005 1:30am
Jadagul (mail):
Mr. AF: I assume that Professor Sander's response would be similar to the one that he offers in his most recent post. Your tier in general corellates fairly well with prestige; but if we also know your scores, we can guess where in the tier you fall. Thus, as per the above post, "for any analysis which selects blacks from, say, Tier 1 and Tier 2 schools who have matching index scores, it's quite likely that these students are actually going to schools of equivalent eliteness," because the scores get you into a school in Tier 2 that is relatively better than the school they'd get you into in Tier 1.

In other words, the average Tier 1 school is better than the average Tier 2 school; but the average student in Tier 1 with a such-and-such on the LSAT, and such-and-such college grades, etc., goes to a school of the same quality as the school of the average student in Tier 2 with the same LSAT, same grades, etc.

My apologies to Professor Sanders if I've misrepresented his position.
6.15.2005 1:37am
Jacob Zipfel (mail):

According to the mismatch hypothesis, students who go to prestigious law schools for which they are underqualified will do worse (ie, graduate and pass the bar at lower rates) than they would have done had they gone to less prestigious law schools that are more their speed. The direct test of this hypothesis is to measure the effect of law school prestige on graduation rates and bar passage. The mismatch hypothesis predicts that, holding all pre-law-school characteristics equal, law school prestige should relate negatively to graduation rates and bar passage.



I believe this misses the mark. You can't hold all pre-law school characteristics equal, because broadly speaking that is the issue at point. Better law schools attract better candidates who are better equipped to succeed in the more competetive environment. The issue is that blacks are admitted to law schools for which they are not adequately prepared for (for a variety of reasons outside the scope of this debate) and as such do not do as well on average. By choosing to go to their 2nd choice schools, they are put into environments into which they have a better chance of succeeding in and in passing the bar. Professor Sander is trying to show just that.
6.15.2005 3:07am
Aultimur:
Rather than identifying "mismatch," hasn't Sander just demonstrated of how poorly most (elite) law schools serve half of their customers? Sure, the prospects choose to apply, but who's in the best position to know which students are really the ones the school intends to teach/serve well? Finding a student-school match isn't like buying shoes - these schools should be ashamed that they serve the "bottom" half so poorly. None of the elite school websites I just browsed say anything about poorly serving the bottom half of the class, or minorities - in fact, they seem to imply everyone gets a good education.
6.15.2005 10:27am
Mahan Atma (mail):
Prof. Sander,

I find it telling that you dismiss my criticism of your statistical analysis as "social science nihilism" without actually addressing the substance of my remarks.

This isn't "nihilism" at all; it's simple statistical theory. You have a non-random sample, and you have not in any way randomly assigned a control group. As a result, there is no probability distribution underlying any of your statistics. Ergo it is absurd to perform statistical inference -- a method that requires the assumption of a probability distribution.

How exactly is this "nihilism"? It isn't; any social scientist with a random sample from a population, or who randomly assigns a control group, can meet the basic assumptions of the statistical models. You cannot do so with the LSAC-BPS dataset. So you have no basis for doing statistical inference with this data.

This is a very basic and fundamental criticism of your work. Your failure to respond to it simply underscores your lack of understanding of the statistical theory underlying the methods you use.
6.15.2005 10:58am
SelectionBiasQuestioner:
Here's a selection bias question. What do we know about the giving of scholarships that seems to drive much of the mismatch hypothesis? It is a possibility (one that at least needs to be investigated) that it is harder to get a scholarship at a lower ranked school than it is to get in to a higher ranked one. The result would be (if this is true -- I don't know) that the group getting scholarships (some of whom would take them) at lower ranked schools would be a better group of students.

A second question, though this isn't really selection bias. Is it possible that many schools give scholarship winners special treatment, either formally or informally?
6.15.2005 11:01am
Dirk Jenter (mail):
Professor Sanders:

Thank you very much for your detailed reply, this discussion is very interesting. You are certainly right, it is virtually impossible to rule out all channels of self-selection bias in this kind of study. If indeed financial or geographic factors led the students to turn down their first choice school and go somewhere else, then your evidence seems very supportive of the mismatch hypothesis.
6.15.2005 11:44am
Mahan Atma (mail):
Dirk,

Re your comment yesterday that the article I posted confirms the statistical argument you made: I don't see anything in the article that does.

Can you please quote the text you're talking about?
6.15.2005 12:00pm
SANDER'S HYPOTHESIS INCOMPLETE (mail):
There are a few problems with Sander's recent post:

1. "Credentials" are not objective as he presumes. If X has a 3.6 and Y has a 3.6, they have the same GPA. But if X went to Harvard undergrad and Y went to Wake Forest, X's 3.6 perhaps counts for more. But it is not necessarily true that Harvard undergrads are smarter or more qualified than Wake Forest undergrads. If Y went to a high school with no AP courses, and X went to a high school with many AP offerings, then X could have had a 4.47 GPA, while the most Y could have had is a 4.0. If you add in that X is affluent and so could afford to take a Kaplan course for the SATs, on which he got a 1370 (average for his geographic area), and that Y was poor and got a 1250 (400 > the average for his geographic area), you have a credentials gap at the high school level that affects the prestige of one's college. But the credentials gap at the high school level reflects resources of X's parents and community, as property values fund local high school budgets.

2. A credentials gap problem also exists at the college level. Let's say that A and Z, both have a 160 IQ and attend Wake Forest, and Y has a 3.6, but Z has a 3.2, totally due to diligence/laziness. In other words, Z could have had a 3.6 with ease. Then let's say that both Z and Y get a 180 on the LSAT. If Z cannot get into U Chicago law because of his GPA, but Y can, you might not see a problem: after all Z was lazy. But what if Wake Forest was absurdly easy, Z was egregiously lazy, and A had a 100 IQ and didn't do very much work during undergrad in comparison to students at other colleges. Then the fact that Z would have failed at any other college, yet graduated at Wake Forest with a 3.2 implies that A's 3.6 isn't much of an achievement at all. But when A applies to U Chicago, his 3.6 is multiplied against his 180 LSAT to the same effect as any other candidate with a 3.6 and a 180 LSAT from any other college in the nation: in other words, colleges where students actually learn stuff and do work.

3. Let's now turn to Harvard undergrad, where X earned his 3.6. Let's say that W was also there, and W earned a 3.6. But W took harder courses. And W didn't just work hard, W actually understood the material. More than that W had a 1600 SAT, a 4.0 GPA, took no AP courses in high school, and was desperately poor. Unlike X, W is not at Harvard because of wealth or luck at all. He gets a 180 LSAT.

4. Let's say that X and W and A attend U Chicago Law, and Z and Y attend George Mason. In terms of their law school GPAs and career outcomes, they rank like this: W, Z, Y, X, A.

5. I think the observations to be made are these:

a. A had high credentials, but should not have attended an elite school.

b. W deserved to attend an elite college and an elite law school.

c. X effectively purchased admittance to an elite college, which unfairly boosted his credentials at the expense of Y, who was denied the credentials boost that would have placed him at the appropriate law school.

d. Z was lazy, but was much more intelligent than his credentials indicated.

6. Assuming that Y, A, and Z are black and W and X are white, Sander is, I think, discussing this pehnomenon: (1) Y ended up at a lesser law school than he would have liked, and does better than we expect, because we confuse him for A; (2) Z ends up at a law school on par with his credentials and does better than we expect, because we do not realize he is a lazy version of W; (3) we cannot explain why X did so poorly in comparison to Z.

7. But let's note that here blacks are not NECESSARILY better off they go to schools on par with their credentials:
(A) Z should really be in a classroom with W.
(B) X never should have gotten into an eliet college and Y should have.
(C) A, on the other hand, the stereotypical beneficiary of affirmative action, shouldn't receive affirmative action.

Sander's hypothesis is incomplete, in that it only points out (C), while ignoring (A) and (B).
6.15.2005 8:52pm
SANDER'S HYPOTHESIS INCOMPLETE (mail):
Please substitute A for Y where I wrote:


2. A credentials gap problem also exists at the college level. Let's say that A and Z, both have a 160 IQ and attend Wake Forest, and [A] has a 3.6, but Z has a 3.2, totally due to diligence/laziness. In other words, Z could have had a 3.6 with ease. Then let's say that both Z and [A] get a 180 on the LSAT. If Z cannot get into U Chicago law because of his GPA, but [A] can, you might not see a problem: after all Z was lazy. But what if Wake Forest was absurdly easy, Z was egregiously lazy, and A had a 100 IQ and didn't do very much work during undergrad in comparison to students at other colleges. Then the fact that Z would have failed at any other college, yet graduated at Wake Forest with a 3.2 implies that A's 3.6 isn't much of an achievement at all. But when A applies to U Chicago, his 3.6 is multiplied against his 180 LSAT to the same effect as any other candidate with a 3.6 and a 180 LSAT from any other college in the nation: in other words, colleges where students actually learn stuff and do work.
6.15.2005 9:04pm
Eric Watson (mail):

Here's a selection bias question. What do we know about the giving of scholarships that seems to drive much of the mismatch hypothesis? It is a possibility (one that at least needs to be investigated) that it is harder to get a scholarship at a lower ranked school than it is to get in to a higher ranked one. The result would be (if this is true -- I don't know) that the group getting scholarships (some of whom would take them) at lower ranked schools would be a better group of students.

A second question, though this isn't really selection bias. Is it possible that many schools give scholarship winners special treatment, either formally or informally?


The answer is yes. Howard University, a historically black university, used to offer free rides including free books (and free laptops in some cases) to undergraduate applicants with a 1500-1600 SATs (out of 1600). The students who attended often had standing offers from Ivy League schools, but followed the money instead. Because Howard was explicitly bringing in National Merit scholars and whatnot to increase its number of Rhodes and Fulbright scholars and boost its average SAT score (and thus ranking) -- which is around 1100, the competition for these greedy high-achievers consisted of students with 900s on their SATs, which is by all accounts below average. In line with Sanders' hypothesis, their averages at Howard were higher than their averages at Cornell or Harvard, say, would have been because of the increased competition at an Ivy league. Then again, that doesn't mean they weren't qualified to be admitted to the Ivy Leagues -- and their higher college GPAs are not a benefit to them because the trade-off in GPA does not compensate for the lack of prestige of the institution they attend and they probably would have done just fine at an Ivy League school. And by just fine, I mean in the top half. Top half means better than a sizable quantity of the supposedly more qualified white students who were admitted.
6.15.2005 9:25pm