A Response to Orin on Tradesports.--

1. Response to Orin's Response.

Orin wrote that about two hours before his 8:38pm post Roberts futures were trading at 1% and (at 8:38pm) were trading at 99.5%. He concluded: "Looks like Tradesports was right after all — at least after the announcement was made."

In response, I pointed out (1) that at 6:38pm (two hours before his post), Roberts futures were trading at 10%, not 1% (but I noted that they had been trading at 1% at 5:55pm); and (2) that "At 6:48pm (almost two hours before Orin's post and nearly an hour before the final press leak), Roberts became one of the top two favorites for the first time, trading at 30. At 7pm, for the first time, Roberts became the favorite at 41.9%. After remaining one of three favorites for the following 40 minutes, Roberts became the favorite for good at 7:40pm."

Now in response, Orin acknowledges only my first point:

But as best I can tell, the only apparent ground for his characterization is that I said Roberts was trading at 1% "about 2 hours" before my post, when the exact time Roberts was trading at 1% was 2 hours and 45 minutes before my post. But even assuming we construe "about" to exclude a 45-minute error window, what difference does that make?"

The main reason I wrote my update was to point out that Orin was "incorrect or misleading" in that the trading on Tradesports indicated that Roberts had a very substantial likelihood of being the nominee, not only after the press announced that Roberts would be nominated, but before the press announced it.

Orin points out that KJ Lopez had a post at the Corner at 5:24pm predicting Roberts as the nominee. But, of course, hers was one of many posts that predicted one or another nominee (indeed, Lopez herself was pushing Clement earlier in the day, as were traders on Tradesports). Her reasoning on NRO's The Corner and at NRO's Bench Memos was far from compelling. Here is Lopez's 5:24pm post that Orin notes:

Prediction: Bush will nominate a white male tonight. Just because they said he couldn't.

John Roberts. Bill Pryor! John Bolton!! Karl Rove.

I'm serious about one of those guys, by the way (and, no, I'm not being a right-wing lunatic this time, so this one's an easy guess).

Lopez follows this up at NRO's Bench Memos by pointing out that Luttig's family is in Washington. Her colleague at Bench Memos, Jonathan Adler floats Maurren Mahoney and makes a brief case for Edith Jones. Robert Alt on Bench Memos comments on Luttig's presence with his family in Washington.

At 6:14pm, Lopez follows up her prediction of Roberts or another white male by saying:

Did Major Garrett Just Say that he expects it won't be anyone who can be perceived as "an angry white man"? I assume there's no other kind of white male? With every comment like that I become more and more convinced Bush will do the impossible.

Here's my question: Supposedly when O'Connor was chosen everywoman got goosebumps, wanted to hug her...will men have the same reax if John Roberts is the nominee?

So the reasons Lopez actually gives for Roberts are only (1) that Bush will nominate a white male "Just because they said he couldn't" and (2) "With every comment like that [about angry white males] I become more and more convinced Bush will do the impossible."

At 6:17, Lopez says that she is told it's not Luttig. At 6:27, Lopez says that Roberts is in town, but she wonders where Mary Ann Glendon is, saying, "I can dream."

My point in reviewing some of this (and there is more) is that, while in hindsight Lopez was correct, her publicly stated reasons were far from compelling, and even her colleagues at Bench Memos and The Corner were not giving up on other speculations. The advantage of a market is that it can sift through information and assign a value to it.

Markets are good at evaluating publicly traded stocks, not because no analyst is correct in hindsight, but because it is difficult to know a priori which analyst is correct and which isn't.

So Orin asks whether markets such as Tradesports "just mirror the collective common wisdom of newspapers and blogs." Yet aggregating the collective wisdom and putting a probability on it is a very valuable function in itself. Markets should do this better than most experts reading through the blogs and newspapers and trying to figure out whom to trust.

Orin ends by saying:

I suppose this means that you could try to use Tradesports as a way of monitoring what a few newspapers and blogs are saying, but on the whole this seems like a quite modest function. It seems easier to just scan the headlines at How Appealing.

I just scanned the headlines at How Appealing. Before the press's announcement of Roberts as the nominee, there were headlines suggesting that Clement would be the nominee and then one saying she wouldn't. So How Appealing would not have been a way to aggregate opinion on who the nominee would be, except for the time earlier in the day when Clement was the conventional wisdom choice at How Appealing, The Corner, and Tradesports.

2. The Larger Question.

Part of Orin's and my disagreement tonight is that we are talking past each other.

Orin questioned the predictive ability of trading markets. I responded:

markets are frequently wrong; . . . The question is whether experts can usually do a better job than markets. It would seem that the answer is generally No.

Orin responded:

I think the question is whether markets do a good job predicting the discretionary decision of one person, as compared to predicting the collective outcome of the individual decisions of many. I can see markets doing a good job predicting collective decisionmaking, but I don't see the advantage they have in predicting what one person is thinking. Thus, it seems to me that Tradesport users incorrectly predicted Rehnquist would decide to retire because that's what newspapers were incorrectly predicting at the time; ditto for the idea that Clement would be nominated to replace Rehnquist.

My claim is a comparative one. Markets should do a better job than experts who lack actual inside information of the choice for a number of reasons that are raised in the comments to one of Orin's earlier posts. I would think this would be true both for predicting individual decisions and for predicting voting or future big market decisions, though I don't know what the state of the empirical evidence is on this narrower point.

Orin says that "the question is whether markets do a good job predicting the discretionary decision of one person, as compared to predicting the collective outcome of the individual decisions of many." This way of stating the issue potentially conflates two questions:

(1) the relative difficulty of predicting individual v. collective decisions; and
(2) the relative effectiveness of experts v. electronic markets.

For obvious reasons, predicting one person's decision is probably harder than predicting the aggregate decisions of many people, though sometimes that is not true (I suspect that for some Supreme Court decisions, I could better predict how Scalia would vote than how the Court as an aggregate would vote. Similarly, I expect that I could also better predict how Senator Hatch would vote on Bush's next Court of Appeals nominee than how the Senate Judiciary Committee would vote.)

But if all that Orin were claiming is that individual decisions are usually harder than agggregate decisions to predict (by markets or by experts), then I would suspect that this is generally true.

Yet Orin is questioning the effectiveness of trading markets: "the question is whether markets do a good job predicting the discretionary decision of one person, as compared to predicting the collective outcome of the individual decisions of many."

I am claiming that markets (however "good" or bad they are in absolute terms) should be better than experts on balance, or at least better than experts who lack actual first-hand knowledge of the forthcoming decision. So Orin and I may be talking past each other. I am asserting that I would expect a comparative advantage for markets over experts; Orin is questioning whether markets would "do a good job" predicting individual decisions compared to group ones.

Stephen Lindholm (www):
It seems to me that the reason Roberts name came out two hours before the press article went on the wire is that the name was leaked. I really don't see a mystery -- it has nothing to do with market efficiency, or blogger reasons, or what have you. Tradespot was just an incentive for people with inside information to use it. Bush certainly kept the name close to his chest until a few hours beforehand.
7.20.2005 5:24am
Carl (mail):
What an enormous meandering post, not nearly as tightly-edited and on-point as the usual VC stuff...

I think the debate is weirdly framed. Of course the expert makes better predictions than a market. That's the definition of an expert.

But this is like saying of course the fastest horse wins the Kentucky Derby. Sure, there's little doubt of that. The problem for the bettor is not whether or not the fastest horse exists, or whether or not he can win, but rather how to decide which horse is the fastest.

In this case, the problem is not so much whether an expert exists whose prognostication is accurate but how non-experts can decide which nominal "expert" is the real expert.

You could just ask them, of course. But alas, human nature being the way it is, in that situation nominal experts are strongly motivated towards "sales" speech rather than their best opinion, inasmuch as their reward in this situation comes more from earning your immediate trust than in being right.

Perhaps a market is, in addition to an efficient way to disseminate expert opinion, also an efficient bullshit filter which strips out salesmanship from the pooled expert opinion that must underly any statistically significant trend of correct prediction.
7.20.2005 6:11am
Jeremy (mail):
Tradesports generally reflects only the aggregate conventional wisdom. The smartest and best informed people in the world woke up thinking "Edith Clement," then thought "Edith Jones," then thought "Mike Luttig," then heard it was John Roberts. The market basically reflected news stories and expert speculation throughout the day, and is nothing more than an aggregation of opinion of experts trusted and believed by the traders. It's rather like RealClearPolitics' "metapolls" where they average poll results. Nothing fancy about it.

People with insider info could have started the markets moving for Roberts before it became common knowledge, but no one seems to have done that in a discernible way. Frankly, it's difficult to see how a thinly traded market that is difficult for Americans to use can be of much use predicting American political events. Sure, it did ok in the 2004 election, but 46 states were never in doubt.
7.20.2005 9:20am
anonymous coward:
The relevant difference between, say, the presidential election and SCOTUS nomination isn't collective vs. individual decisionmaking but tons of existing data (e.g. polls) vs. lack of data.

Apparently we're supposed to believe that some trades at 30 (and some substantially lower) on a thin, wide market has some powerful meaning. And how many trades was that, anyway? Enough to meaningfully draw a distinction between the solitary judgment of an expert and the collective wisdom of the market?
7.20.2005 9:29am
this is an incredibly boring debate.
7.20.2005 10:23am
Eh Nonymous (mail) (www):
Ted: how bizarre that you wanted to speak up and say so, then. Failure to comment does a much better job of showing your disinterest; and we enjoy it better too!

Carl: interesting perspective.

Jim: Thank you for a fascinating, insightful, and well-reasoned post. You said all that ought to be said, and covered it with helpful, accurate quotes and with precision. Exactitude is often missing on blog discussions in general; people are vague, realize they didn't use formal statements of logic, accuse each other of things without saying when those things occurred, and generally behave badly. Me, for example, right now.

This post was excellent, and higher quality than I have come to expect from VC, in direct contrast to commenter Carl.

Thanks for showing that some of the people on VC read each other, not just to carp, and that some of you are paying attention, trying to understand areas of disagreement, and publicly disclosing your analyses.

Here's to even more fruitful discussion and disagreement on VC, with us readers reaping the benefits.


Eh N.
7.20.2005 10:38am
Amber (mail):
I agree with Ted. Please, no more Tradesports posts!
7.20.2005 11:08am
Please use the hide/show function for posts like this.
7.20.2005 11:30am
Dem: and other blogs also started talking Roberts up around the same time as NRO and before the jump in the markets. And, while Roberts price began going up as he was mentioned more on blogs, it did not get above the mid-30's (ie, to the point where the market seemed to agree Roberts would be the nominee) until after Drudge, Redstate and others had posted that Roberts was definitely going to be the nominee. Confirmthem, for example, (Redstate affiliate) had the news at 7:40, and Jim's chart shows the jump happened at sometime after 7:42 (unclear exactly what time.) So, all tradesports shows is a reaction to the rumors/news coming from about 5 blogs. I happened to check one of the blogs with the early info. right around 7:40 and had the news before my friend, who I was talking to on the phone, who had just loaded tradesports.

So, I just don't see how looking at tradesports has any more value than reloading redstate and scotusblog. Yes, tradesports is only one site, which means less reloading. But, it also only reacts to the other site so that means you'll get the info. a few minutes later.

If a market involves a lot of different pieces of information then it would have some value in aggregating perceptions about which sources of information are accurate. (Though, it still wouldn't show whether a proposition is, in fact, true.) But, when a market is moved by only about 5 web sites (as in the case of a SCOTUS nominee), I don't think it has much value.
7.20.2005 12:45pm
NR (mail):
I agree with WB. Please use the hide/show function for posts this long, and please enable comments so that discussions can develop sequentially below the fold, rather than dominating the main page.
7.20.2005 12:47pm
ed johnson (mail) (www):
I think Jim is basically right, although I don't like the terms "best estimate," because it's not clear what that means. The value of trading markets is based on a simple arbitrage argument:

THEOREM: It is impossible for a reasonably informed outsider to consistently out-predict a functioning (i.e. fairly liquid) prediction market (such as tradesports) simply by reading weblogs and such.

PROOF: If it were possible, then reasonably informed outsiders would do it and make lots of money. People would enter the market until it was no longer possible to make any significant profit.

How difficult or usefull this will be varies from case to case. If Orin thinks it's easy to out-predict tradesports, then good for him, he should be able to make some money! If he's just saying that you can do as well as tradesports by reading lots of papers, then that's ok too...I'd say that tradesports is still valuable because it agregates all that information for you and assigns a numerical value to it. It's a very quick and easy way of summarizing public wisdom on a subject. The fact that Roberts wasn't on the radar until late yesterday just shows that outsiders couldn't have easily predicted the nominee before yesterday. Nothing wrong with that.
7.20.2005 1:54pm
anonymous coward:
That arbitrage argument doesn't work if the markets are too thin and too wide to make much money by being better-informed. (Or, on the flip side, if the "reasonably informed outsiders" don't have enough money to move the market away from the less-informed.) You can't create an efficient market by appealing to theory.
7.20.2005 2:28pm
ed johnson (mail) (www):
That's all exactly right, anonymous coward. My "theorem" obviously isn't a real theorem in terms of rigor. But it shows the basic idea of what prediction markets can hope to achieve. Personally, I think the tradesports markets come reasonably close to achieving it. Critics such as Orin haven't been arguing that the prediction markets are too thin, they have been arguing that they are flawed in some more fundamental sense.
7.20.2005 4:28pm
Jennifer Carter (mail) (www):
With apologies to all who find this discussion boring, this fall the Harvard Journal of Law &Public Policy will publish an (in our view) interesting article by AEI-Brooking's Bob Hahn on the future of using information markets in public policymaking. An abstract and the manuscript are available on SSRN.
7.20.2005 4:41pm
ed johnson (mail) (www):
I think I understand one reason why this debate is confusing.

We sometimes seem to be debating whether prediction markets are "right" or "best" in some sense. This is miguided; with hindsight, we can see that Roberts was the correct prediction and any other prediction was wrong. Any prediction that put less than 100% on Roberts was not the "best prediction."

The right question to ask is whether it is easy to out-predict the markets. If it is hard, then they must have some predictive value.
7.20.2005 5:13pm
Missing the point here, somewhere.

Jim rambles about efficient markets. Orin points out that this is not an efficient market. Jim rambles some more.

Everyone misses the point that the market merely exposes inside information via rewards. I don't believe the 'crowds' explanation makes any more sense than lots of other cliches. This is how I pick stocks: watch executive compensation. They're selling? short it. They're taking a lot of stock in place of cash? ... Naturally, you have to correct for particular situations, but watch the ones that know what is going to happen.

The great thing, the single greatest thing (in terms of efficiency - I'm omitting legal and liberty arguments), I'd assert, about markets is that they reward the act of sharing information.

The Invisible Hand didn't order you to leak, but... that's some tastey candy, huh?
7.20.2005 11:03pm