pageok
pageok
pageok
Cross v. Lott on Alito:

A few days ago I posted on "Alito By the Numbers" and linked to a news column by John Lott. For those who may not have followed the subsequent Comments but may be interested, Frank and John have a vigorous dialogue in the Comments to that post. Check it out here if you are interested.

I have added links to the underlying Choi and Gulati papers as an Update to the original post.

frankcross (mail):
Todd, I think you're being a troublemaker. I've said my piece. And I would like to divert the discussion from whether the original post was or was not misleading to the specifics of the quantitative analysis. That's what I'm interested in. There are lots of interesting issues here, such as the value and validity of these measures and how they are best used to evaluate the Alito nomination.
11.6.2005 3:25pm
Defending the Indefensible:
The problem is reliance on any econometric analysis by John Lott is per se suspect, and not really worth engaging, as it would in effect require independent access and analysis of all his data merely to confirm whether the conclusions are justified, and furthermore to determine whether his methodology was cherry-picked to obtain a desired result.

In other words, in the absense of basic trust, there cannot really be fruitful discussion. This is regardless of whether one might agree with Lott's policy preference.
11.6.2005 4:39pm
JonC:

...it would in effect require independent access and analysis of all his data merely to confirm whether the conclusions are justified, and furthermore to determine whether his methodology was cherry-picked to obtain a desired result.


Can't this be said of any social scientist or econometrician? Or do you think there are some out there we can just trust implicitly without scrutinizing their analysis or bothering to try to replicate their results?

Furthermore, the data in question aren't Lott's. They're Choi's and Gulati's.
11.6.2005 7:09pm
Antonin:
Can't this be said of any social scientist or econometrician?
Some years ago, John Lott published a book called More Guns, Less Crime, in which he referred to a study he performed that purported to show that in only 2% of defensive gun users did the user fire the gun at all. He has repeatedly referred to this statistic in print. Unsurprisingly, pro-gun-control scholars asked to see his data, but it turned out the study had never been conducted at all and the data didn't exist. Lott has never admitted this, but the circumstantial evidence is overwhelming: his unwillingness to provide the data; his claims that everything had been lost in a computer crash and no paper records or backups of any kind were kept, that he couldn't remember the name of a single student who worked on the product, and that there were no records of such students being hired or paid because he paid them all cash out-of-pocket; and his use of an anonymous online persona, "Mary Rosh", to defend his work.

If there's any unforgiveable scientific sin, it's fabricating data.

http://en.wikipedia.org/wiki/More_Guns,_Less_Crime

(scroll down to "Criticism")
11.6.2005 9:17pm
Zywicki (mail):
Frank:
Sorry--I didn't intend to be a troublemaker. Since you guys had a back and forth, I just didn't want to leave the matter completely hangin. I agree with you that what is important is the underlying quantitative analysis, not whether the original piece was misleading. I apologize if I implied otherwise.
11.7.2005 6:59am
John Lott (mail) (www):
1) I do not understand at all why Todd is viewed as a troublemaker here. He found the discussion interesting and suggested that people look at it. What was agreed to at the end was quite different than the original charges that were leveled.
2) As to the other charges leveled in this posting, alll the data used in my book, More Guns, Less Crime, has been made available. Every single regression can be rerun and replicated. I have redone the survey data that was mentioned in one sentence in the book and made that available to anyone who wants to see it (www.johnlott.org). If you would like to see others who have found similar results on right-to-carry laws, see this: http://johnrlott.tripod.com/postsbyday/RTCResearch.html. If you would like to see a discussion of the survey data and other issues, see this: http://johnrlott.tripod.com/postsbyday/topic-mysurveys.html.
11.7.2005 8:08am
Tim Lambert (mail) (www):
The big give away with Lott's MGLC work is that he changed his model as more data came in. The headline result in his original paper involved looking at changes in crime levels, but as more data became available he stopped reporting the results of his original model. Why? Because it no longer showed benefits from carry laws. Not a problem for Lott, since he he just searched for a model that did show benefits and reported that.

As for the survey, the data is so simple that anyone who cares can check the accuracy of his claims in just five minutes. Lott has repeatedly claimed that the survey he did in 2002 found that 95% of defenders just brandished the gun. But if you download his data from johnlott.org (it's in a spreadsheet, so all you need is something like Excel to look at it), you will find that in 12 out of 13 DGUs the defender just brandished the gun. Try dividing 12 by 13 and see if you get 95%. More here.
11.7.2005 11:30am
John Lott (mail) (www):
Mr. Lambert is inaccurate as usual. After critiques that people had raised I did add a few additions over time, but they were as a response to points that others had raised and they definitely did not alter the results reported in the tables. The data is available on my website at www.johnlott.org for anyone to see that this claim is false. For example, rather than just simple geographic (state or county) and year fixed effects, I tried breaking down the year fixed effects by geographical region so as to allow for any trends that might vary by region. This was done because of claims that cocaine might have increased crime rates in some regions relative to others (the work that Bronars and I had done comparing neighboring county crime rates should have more than dealt with this problem, but I decided to give people what they wanted here.) After work by Carl Moody (http://johnrlott.tripod.com/postsbyday/RTCResearch.html the direct link is too long to be posted here), I added prison population in addition to the arrest rate and other crime data that I already had. (Of course, the city level data that I added in the 2000 addition of my book had extremely detailed police data on everything from unionization of police to the type of police strategies used.) But as anyone who has read my book knows, I reported thousands of specifications for the county level data showing all the possible combinations of sets of the control variables and the results were remarkably stable (this is following a similar approach that Bartley and Cohen had done previously with this data and I found similar results).

12/13=92.3%. The difference between that and 95% is weighting for state level population differences built up from the county level demographic data that I had. Again, the data has been made available at www.johnlott.org and a discussion is available at http://johnrlott.tripod.com/postsbyday/topic-mysurveys.html. Sample size, levels of significance and other points were discussed in detail.
11.7.2005 12:34pm
John Lott (mail) (www):
Here are some direct links to what I discussed.

1) Right-to-carry research
2) Discussion of Survey Data
3) Carly Moody's Research
11.7.2005 1:16pm
Tim Lambert (mail) (www):
Anyone can check for themselves that the basic results reported in table 4.1 of MGLC were not updated in the section entitled "Updating the Basic Results". And it's pretty obvious why -- you reran those regression with the extra years and didn't like the results you got so you chose not to report them.

In Bias against Guns you said that you didn't use state level population differences to weight the data. Now you are saying that you did. How about you present the details of your weighting calculations? I did the weighting following the procedure described in your book and certainly did not get 95% as a result.
11.7.2005 10:02pm
John Lott (mail) (www):
The first edition of MGLC spent a great deal of time explaining why looking at the simple before and after averages does not give an accurate description of the impact of legal changes. Since you missed that discussion, let me give you a couple simple examples. Suppose that crime rates were falling before the law and continued to fall at the same rate after the law was enacted. Doing a simple before and after average would indicate the the crime rate was lower after the law than it was before, but I would trust that most people would agree that the law did not have any impact. One other example for you. Suppose that crime rates were rising before the law and falling afterwards so that it looks like an inverted "V" with the point right at where the law went into effect. What would the before and after averages show? They would show no impact from the law, but obviously this would be the case where the law had the biggest impact. The point of first reporting the before and after average estimates is not because they are a particularly useful estimate, but because that is the way that many people who don't understand econometrics very well estimate these things. Many people just use a dummy variable for the laws that they test and don't understand what this implies with how the variables that they are measuring are changing over time. The point of the book was to take a very simplistic estimate and then provide more useful ones. I hope that this is clearer now. If it still isn't clear, there are graphs in the book that try to illustrate this point (as one example see pages 135 to 138 in the first edition).
11.8.2005 12:24am
Tim Lambert (mail) (www):
It's always possible to construct a rationale to justify whatever model you choose. The fact remains that the main results reported in the original paper were for the step model. If this model is wrong then it was wrong then. But you only decided that it was wrong and decided not to even report the results when it stopped showing more guns less crime.


In Bias against Guns you said that you didn't use state level population differences to weight the data. Now you are saying that you did. How about you present the details of your weighting calculations? I did the weighting following the procedure described in your book and certainly did not get 95% as a result.
11.8.2005 10:58am
John Lott (mail) (www):
What is the point of arguing with you? Do you have any possible response to the logical arguments? In the vain hope that you would not completely ignore them, I wrote them out again for you in case you didn't remember them. Yet, you completely ignore the logical arguments and misstate was was written. Anyone who reads the book knows the effort that I went to great lengths in the FIRST EDITION to explain why looking at before and after averages is misleading.
11.8.2005 1:52pm
Tim Lambert (mail) (www):
The relationship between carry laws and crime rates is discussed in chapter 4 of MGLC, That chapter starts with TWENTY PAGES of discussion of the results using before and after averages. Finally, starting on page 70 there is a page or two on trends. You does not say that the preceeding 20 pages wre misleading -- he states that he wants to look at trends as well because "If changes in the law affect behavior with a lag, changes in the trend are probably more relevant". After that we have another 20 odd pages, mostly looking at before and after averages. Are really now arguing that 90% of the key chapter in MGLC is misleading?

In Bias against Guns you said that you didn't use state level population differences to weight the data. Now you are saying that you did. How about you present the details of your weighting calculations? I did the weighting following the procedure described in your book and certainly did not get 95% as a result.
11.8.2005 10:14pm
John Lott (mail) (www):
Are you really this dense? Let me go through this again for you. Most people use a simple dummy variable for the impact of a law. I did this initially just to show the basic point. However, I explain what the problems are with this approach, something that you have continued to fail to address in any of this discussion, and why that using this simple dummy variable can be extremely misleading despite its frequent use. The point is to start with what is the simplest and gradually make it more accurate. Despite your inaccurate claims, there was no change in the model specification between the first and second editions. The second edition of the MGLC starts with the specifications that avoid the points raised above regarding the dummy variable approach. What might be productive is some discussion on your part about the logical issues that I raised with the use of a dummy variable.
11.9.2005 1:50pm
John Lott (mail) (www):
Why no response Lambert?

Let me also note that Frank Cross never sent me a copy of his critiques of Choi and Gulati. He stated that I was "quite misleading" in using their work, though all I was doing was citing what they had found. It turned out that he felt that I should have understood this critiques of their work, though I had never seen them. Since he didn't spell out in the previous debate what his critiques were, I asked him to at least email them to me and he has done neither.
11.13.2005 4:08pm
Tim Lambert (mail) (www):
Why haven't you responded with details of your weighting calculations? Did you actually do them?

It is obvious to anyone who looks at chapter 4 that you didn't just start with the dummy variable approach -- 90% of the chapter uses that approach. If the dummy variable approach is extremely misleading, then your book is extremely misleading. The update in the 2nd edition drops the dummy variable specification. That is changing the specification.
11.15.2005 9:19am
John Lott (mail) (www):
OK, Lambert, since you fail to address my points I assume that you concede that you were just making things up when you said that I "changed his model as more data came in." You concede that you were wrong in saying that I was making ex post rationalizations for the problems with looking at simple before and after averages. You concede that you have no logical arguments against my points for why simple before and after averages are likely to be quite misleading.

Second point, I have provided links to everyone one above on the weighting issues, but since you haven't followed those let me make things clear in case they weren't previously. In the first survey I aggregated the county data up to the state level for the different demographic groups. In the second survey, because the sample wasn't as large, I aggregated the county level up to national level for those demographic groups.
11.20.2005 3:36pm
Tim Lambert (mail) (www):
I concede none of those things. You did change your model, dropping the before and after averages. You did make ex post rationalizations to justify this since your arguments did not appear in MGLC 1st edition. And I don't have to disprove your arguments against before and after averages -- if they really are that misleading you shouldn't have based chapter 4 of MGLC about them.

I know how you said that you weighted things in the 2002 survey. Trouble is, if followed that procedure and you don't get 95%. If you follow that link you'll see I have a spreadsheet containing all the calculations. How about you present your weighting calculations so we can see where you went wrong?
11.21.2005 1:34am
John Lott (mail) (www):
You refuse to respond to any of the points that I made above and you refuse to concede anything. That is a productive combination. 1) The arguments about problems with before and after averages were made in the first edition. 2) You don't even deal with whether the arguments are correct or not.

Since you refuse to deal with the arguments: "Let me go through this again for you. Most people use a simple dummy variable for the impact of a law. I did this initially just to show the basic point. However, I explain what the problems are with this approach, something that you have continued to fail to address in any of this discussion, and why that using this simple dummy variable can be extremely misleading despite its frequent use. The point is to start with what is the simplest and gradually make it more accurate. Despite your inaccurate claims, there was no change in the model specification between the first and second editions. The second edition of the MGLC starts with the specifications that avoid the points raised above regarding the dummy variable approach. What might be productive is some discussion on your part about the logical issues that I raised with the use of a dummy variable."
11.21.2005 11:15pm
Tim Lambert (mail) (www):
I have responded to your points -- you have just ignored my responses. Once again: you misrepresent the content of chapter 4 of MGLC. You do not just start with the dummy variable model and move on to a different model. In fact, 90% of the chapter is about the dummy variable model. Nor do you, in the 1st edition of MGLC say that the dummy variable model is "extremely misleading". Also telling is that chapter 4 already included an update on the model with more data, and in that case you updated the dummy variable model. Of course, in that update it was still giving results that you liked...

And why do continue to evade questions on the weighting of your survey? If you really did do some calculations to come up with the 95% number why can't you show them to us? What are you hiding?
11.23.2005 10:39am