pageok
pageok
pageok
Evaluating Billy Beane and Moneyball:

This analysis of Oakland A's general manager Billy Beane's famous "Moneyball" picks in the 2002 major league draft concludes that the Oakland GM's use of statistical methods to evaluate players has had mixed results. In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance. Moreover, focusing on just one year's draft picks ignores the fact that Beane has actually been GM of the A's for 7 years. It also ignores the reality that he uses statistical analysis to evaluate free agent signings as well as draft picks.

A better measure of the Beane/Moneyball record is how well the A's have done overall during his tenure relative to the available financial resources. This approach considers the full impact of all of Beane's player acquisition decisions during his time in Oakland. Here, I compare the A's record during Beane's 7 full seasons as A's GM (1999-2005), to that of the New York Yankees and Boston Red Sox, usually regarded as the 2 most successful American League teams of the era.

From 1999 to 2005, Yankees won an average of 97 (of 162) games per season, for a total winning percentage of .602. The A's won 94 games per year for a winning percentage of .581. The Red Sox, in turn, won 92 games per year, for a .567 winning percentage. The A's made 4 playoff appearances during that time and narrowly missed the playoffs in two other seasons, exactly the same as the Red Sox; the Yankees reached the playoffs in all 7 years. Thus, the A's record under Beane is slighly worse than that of the Yankees and slightly better than that of the Red Sox. Quite impressive for the A's considering the strength of the comparison group.

But here's the kicker: During the period in question the Yankees payroll averaged $138 million per year, the Red Sox $103 million, and the A's a mere $42 million (figures calculated from data available here). The A's achieved roughly the same results as the other two teams, while spending only one third as much as the Yankees and about 40% of what the Red Sox shelled out. To put it another way, the A's during Beane's tenure spent $450,000 per win, while the Yankees spent $1.42 million and the Red Sox spent $1.12 million. Given that the Yankees and Red Sox are both regarded as well-run franchises among the best in baseball, Beane's vastly superior use of resources compared to them is extremely impressive.

To be sure, the above analysis is oversimplified. Ideally, we should compare Beane's A's to all other AL teams and not just to the Yankees and Red Sox. I do not have time to do all the calculations right now, but I highly doubt that any of the other AL franchises even comes close to the A's performance in wins relative to spending (a quick eyeball analysis suggests that only the White Sox, Angels, Twins, and possibly Indians could even approach the Yankees and Red Sox). National League teams are not a good comparison group because they rarely face AL teams in regular season games and so are not playing against the same competition.

Another potential methodological flaw is the implicit assumption that the relationship between increased spending and wins is linear. In reality, doubling spending will not double your wins even if the money is spent optimally. Nonetheless, there is no doubt that Beane got far greater bang from his bucks than the Yankees and Red Sox did from theirs, even if we take nonlinearity into account.

It is also true that not all the A's success can be attributed to Beane. However, Beane did pick nearly all the players, and the few left over from the prior regime were picked by his predecessor Sandy Alderson, who had begun to use statistical analysis himself and chose Beane as his successor in part because of the latter's interest in the new methods.

Finally, naysayers will point to the A's relatively weaker playoff performances during this time period. Both the Red Sox and Yankees did much better in the playoffs than the A's, who lost all four of their first round series. However, as Billy Beane himself pointed out in Moneyball, playoff series outcomes are heavily influenced by chance because of the small number of games involved and the high degree of randomness in baseball. The A's lost all four series by narrow 3-2 margins and several times were victimized by very bad luck. Had they been slightly luckier, their playoff record might have been much better.

Ted Frank (www):
One can evaluate Beane's draft record separate from Beane's overall record. Beane may be very good at free agent signings (and, more importantly, non-signings) and trades, but poor at drafting, and, if so, the poor drafting would merit criticism.

In my view, evaluating Moneyball based on the performance of just seven draft picks is a weak methodology, given the small numbers involved and the great influence of chance factors, such as fluke injuries, on individual player performance.

Moreover, there's no control: how well did other teams' first-round draft picks do (most teams' first-round draft picks have mixed records) and how much did it cost them.

Regardless of the success of the draft, Beane's draft seemed to flunk some basic precepts of game theory and economics. A draft is a Dutch auction, and spending a first-round pick on someone no other team is going to pick in the first five (or even ten) rounds is overpaying, even if only in terms of opportunity cost, even if the player turns out to be a hidden-gem success.
8.20.2006 9:28am
Jacob (mail):
I doubt Ilya was trying to include an exhaustive list ('twould be impossible) of all the reasons the A's cannot be compared to the Red Sox and Yankees of the era, but amongst the most significant ones is the imbalanced schedule. Boston and New York face each other twenty games a season. Oakland has nowhere near as many games against so tough an opponent.

Beane's teams have performed fantastically for their circumstances, however. Yet I wince when I see their unfortunate playoff outcomes described as victimization by bad luck. In school we just used the phrase "sample size" (162 >>> 5).
8.20.2006 10:01am
El Blogero (mail):
I have actually become exhausted by all the attempts to deify Beane or mystify his methodology. The above comparison is off for many reasons, including:

-- the A's have not been as successful as either the Yankees or RedSux during this period (just just their respective AL and World Series pennants). That is the definition of success in any sport, not average games won (just ask the 2001 Seattle Mariners how much historical weight goes with the title of "most wins by a team that did not win its league's pennant.")

-- the A's spend much of the seasons playing the Mariners, Angels and Texas, which no one can compare with 19 games between the Yanks and Sux (in the season series against each other, the Yanks and Sux usually end up close to even or over/under by about 2 games).

-- By the "average wins per dolar spent" standard, I am sure that there are other teams, including losing teams, that could celebrate their frugality. For example, how much did Atlanta spend per win during this period, and are they the mark for success in the NL (despite the fact that another team has been the NL representative in the World Series in each of those years)? How about Minnesota, inicluding its success in reaching the post-season during these years?

-- Lastly, the above comparions is based at least in part on the assumption that is qualitively better to spend little for 7 years and end up without zero championships than to spend a lot in a couple of those 7 years and win a championship (see 2001 Diamondbacks). Is one better than the other for the fans? How about the team?

The bottom line is that Beaneball should be evaluated based on the usual measures for success, not a lower "we were only planning for the 162 game season" measure (and, to be fair, the Yankees spending vs. World Series should also be judged on the same measure-if they win, well spent, if not, failure).
8.20.2006 11:03am
Ted Frank (www):
The Red Sox and Yankees have also got to beat up on the Blue Jays and Devil Rays in the same time. The Yankees and Sox do not play each other 19 times a year.

Oakland has a much lower dollars per marginal win than Atlanta.

Once upon a time baseball valued the 162-game season more than the exhibition playoff afterwards. The wildcard changed that. A five-game series between two very good teams is essentially a coin-toss that no general manager can hope to influence. While Oakland hasn't had much luck in the coin tosses, they've been almost as good as any team in baseball in the last decade, and much better than their peers with comparable budgets.
8.20.2006 11:12am
Joel B. (mail):
The idea expressed by Jacob that the A's are somehow at a huge advantage being in the AL West compared to the AL East seems silly. Certainly over the period of the past 7 years all 4 teams in the AL West have been fairly evenly matched. Is there anyone who doubts that for the pas few years the Angels have been an excellent team? The Rangers themselves are a strong team, and the Mariners have been off and on but far mor consistent than the Devil Rays or Blue Jays.

It seems like the only division which is fair to criticize, is the AL Central, which at times seems like it has 2 .350 level teams and 2-3 .600 level teams, which is probably a result to some degree of the divisional play. The AL West especially has the most bunching of all the divisions (no team is effectively out of it yet) and all the teams are fairly strong.
8.20.2006 11:52am
Hei Lun Chan (mail) (www):
Many critics also argue that Moneyball is mostly a theory about finding undervalued hitters, while the success of the A's in those years is mainly due to the dominance of their three starting pitchers, Hudson, Zito, and Mulder.
8.20.2006 12:37pm
Jacob (mail):
Well, I never said the A's had a huge advantage. My point was that comparison between the divisions (at least based on strict win-loss records) is impossible.

Ilya was working off the assumption that the Yankees and Red Sox were likely the two best teams, and his conclusion was that the A's were comparable. The win-loss record (if it shows anything) would support that. If we then accept that these are the three best teams in the AL, then we must face the fact that this season the Yankees will play 28 games against the best teams, the Red Sox will play 29 games against the best teams, and Oakland will merely play 19 games against the best teams. The advantage garnered by that (I make no claims to how great it is) might indeed be balanced by the paucity of games against the Devil Rays, but it's unlikely we're going to be able to figure it out.

What we can do is look at these teams' records outside their own division, but then we're going to be relying on a very small number of games against the AL Central in any given season. To get a reasonable sample we'll have to aggregate over several seasons, which forces us to assume that a win against the Twins in season X is equal to a win against the Twins in season Y.

So no, I'm not certain the Yankees and Red Sox have been better over the last seven years than the A's. I have to rely on my own subjective impressions, and reluctantly admit that I can't back them up with all the statistical evidence I'd hope. And I'm pretty sure I made Rob Neyer cry, too.


Also, I'm sure there's some big moral debate to be had regarding how much one should value the regular season vs. the playoffs, like Ted Frank and El Blogero may end up having. But surely we can all agree that the measure of Beane's success as a manager can't be his ability to foresee which guys happen to play better in October than in April? Building a team that's most likely to win any given game during the season is going to give one a pretty good chance at getting a ring at the end of it. With the playoffs so short, that's pretty much all a manager can do...improve the odds his team will happen to win four games out of seven.
8.20.2006 12:55pm
John M (mail):
You give the article too much credit. It seems the best way to determine if the A's had a good 2002 draft, at least as of 2006, is to actually compare the A's results to the results of other teams. There were 41 players chosen in the first round and sandwich round in 2002. Of those 41, 12 have become somewhat established as MLB players:

Prince Fielder, Mil
Jeff Francis, Col
Jeremy Hermida, Fla
Joe Saunders, LAA
Khalil Greene, SD
Scott Kazmir, NYM--now plays for TB
Nick Swicher, Oak
Cole Hamels, Phi
Jeff Franceur, ATL
Joe Blanton, OAK
Matt Cain, SF
Mark Teahen, OAK--now for KC

Others have been up in the majors, but the 12 above meet the somewhat subjective definition of guys who will be MLB players for the foreseeable future. Of the 12, three were drafted by the Oakland A's with their seven picks. That means the other MLB teams have thus far gone 9/34. Or about 26 percent. Oakland went 3/7, or 42 percent. Even judging Beane by one round of one draft, he did better with his seven picks than did the other 29 teams with their 34 picks, on average.

I had to laugh out loud that Jeremy Brown is considered a strike against Beane. It's been a while since I read Moneyball, but my recollection is that, as the article notes, the scouts were simply incredulous that the short, slow, fat Brown had been drafted in the first round. No other MLB team had him anywhere near the first round on its draft board. He was, if memory serves, a guy expected to be drafted no earlier than the 10th round or so. Now, one can debate whether Beane should have maximized the value of his pick by trading down, but it's worth noting that drafting baseball players is a really, really tough business. There are first round busts, several of them, every year. Jeremy Brown was considered a marginal prospect by nearly every team but the A's, and four years later, he has played in the major leagues. That's nothing to sniff at.
8.20.2006 2:04pm
Jake (Guest):
Just to pull some other small market teams into the comparison:
The Twins have also been around $450,000 per win from that time period. The Marlins have been around $470,000. The Brewers, nobody's have been at $580,000, while the Royals have been around $525,000.

I think it's problematic to compare people along one axis ($ per win) when they're actually competing along another axis (wins/pennants).

Brian Cashman is not being asked to do his best on a budget--if he were, he may well do just as well as Billy Beane. On the other hand, if Beane were handed the keys to the Yankees there's no guarantee that his techniques would scale. Look at Dan Duquette of the Expos and Duquette of the BoSox.
8.20.2006 2:53pm
Jim Chen (mail) (www):
A group of us — Tom Bell, Al Brophy, Paul Caron, and Ronen Perry — are taking a serious look at the Moneyball approach to law school evaluation and administration. Our blog is called MoneyBall and can be reached either at http://money-law.blogspot.com or through the Jurisdynamics Network.

In short, if you enjoyed Moneyball, you'll enjoy MoneyLaw.
8.20.2006 3:01pm
Adam J. Morris (mail):
Part of the problem with the book is that Michael Lewis was a victim of "new convert syndrome." He ended up over-selling the argument, and unnecessarily denigrating the work others have done.

For example, Lewis referred to the A's 2001 draft as "an expensive disaster." But the A's two 2001 first rounders -- Bobby Crosby and Jeremy Bonderman -- are both very good major leaguers, better than any of the 7 first rounders from the "Moneyball" draft. And the two players with the most success from the "Moneyball" draft -- Nick Swisher and Joe Blanton -- are guys that the scouts liked, as well.

Lewis wrote an entertaining book, but, like some of his other works, he tended to too often paint things in stark black and white, rather than recognize the shades of grey.
8.20.2006 3:10pm
Chris S.:
John M. -- Actually, trading down is not an option in MLB. It's what prevents KC from selling their top pick every year, and why agents instead just make it clear that their player will not sign with X team or for X dollars. There are players that the lower revenue teams simply cannot draft because they are unsignable; that makes judging based on drafts all the more difficult -- teams are not truly choosing from the same pools as each other.

I think Moneyball is really just about identifying and exploiting undervalued commodities. Some teams are better at it than others, and evaluating results requires a bit of context.
8.20.2006 4:27pm
John McG (mail) (www):
One of my furstrations with the sabermetric crowd is that they start with the presumption that we need to dispense with sentimentality in analzing baseball. What matters are results.

But then, when it comes time to evaluate the A's playoff performace, we're told that we should not be so harsh because they have been victims of bad luck.
8.20.2006 4:30pm
Jim Chen (mail) (www):
Ilya,

Your post compelled an entire post in response. I hope to see you and your readers over at MoneyLaw.

Jim
8.20.2006 5:07pm
Jim Chen (mail) (www):
Oops. The link I posted a few minutes will probably no longer work, given that I had to fix an embarrassing mistake in my post's title.

Here's the properly functioning link.
8.20.2006 5:12pm
Not a lawyer, but ...:
A draft is a Dutch auction, and spending a first-round pick on someone no other team is going to pick in the first five (or even ten) rounds is overpaying, even if only in terms of opportunity cost, even if the player turns out to be a hidden-gem success.

No, it's not a Dutch auction. The budget that Beane was given wasn't anywhere near enough to pay million-dollar signing bonuses for even one prospect, let alone several; he HAD to find players that other teams thought were not valuable. And, as mentioned in the book, he used a high draft choice as his way of telling a seemingly unlikely prospect that the A's really believed in him, as well as bestowing prestige on the player.
8.20.2006 6:13pm
James Lindgren (mail):
The Red Sox are not a good comparison because they have been using somewhat similar criteria to Oakland in recent years. In other words, the Red Sox are partly Moneyball followers as well.
8.20.2006 6:43pm
James Lindgren (mail):
John M,

I don't know about the facts you cover, but your general approach is definitely the right one to take.

It would be best to use some objective criteria for "making it": e.g., for hitters: games played OR at bats OR total number of bases hit; for pitchers: innings pitched.
8.20.2006 6:48pm
Kurt2 (mail):
The advantage garnered by that (I make no claims to how great it is) might indeed be balanced by the paucity of games against the Devil Rays, but it's unlikely we're going to be able to figure it out.

Well, it might be hard to figure out if it weren't such a blowout. Boston is really famous and everything, but since 1999 the AL East's overall winning percentage is .502; the AL West has gone .531. That's including the Yankees and the A's; if you excludde them the records are .478 and .498. Tampa Bay's *best* record in that time is 70-91; all AL West teams combined have been worse three times in seven years. The AL West has had a better overall record every year since 1999, including this year. Yeah, I think all those Tampa games might make up for it.
8.20.2006 7:47pm
Ilya Somin:
The Red Sox are not a good comparison because they have been using somewhat similar criteria to Oakland in recent years. In other words, the Red Sox are partly Moneyball followers as well.

The A's outperformed the Red Sox by much greater margins before the Sox adopted Moneyball principles starting in 2003. I did not break down the stats by pre- and post-Moneyball Red Sox performance, but in 1999-2002, the A's greatly outperformed the Sox in every year except 1999 (Billy Beane's first as GM). The Red Sox won 94, 85, 82, and 93 games during those 4 years. The A's 87, 91, 102, and 103. On average the A's outdid the Sox by over 7 games per year during Boston's pre-Moneyball days. Since 2003, the Red Sox have slightly outperformed the A's in terms of wins. In both periods, of course, the Sox spent vastly more money than Oakland.
8.20.2006 11:57pm
Zywicki (mail):
One anomaly that always jumped out at me was Miguel Tejada--Oakland's best player at the time of Moneyball and about as much of a non-Beane hitter as one could imagine. So I've always wondered whether Moneyball may really have more bite for complementary or less-obvious players, rather than true superstars whose value is apparent.

I'm less concerned by the criticism that the A's built so much on their Big 3 at this time, in that this may be more consistent with Moneyball thinking. Given the A's budget constraint, Beane basically chose to allocate resources to starting pitching over relievers, based on the idea that almost any decent pitcher could be turned into a capable reliever. So by refusing to overpay for relief, he was able to afford to keep his starters (and I think he made some trades in there too), and by appropriately valuing starting pitching to avoid trading starters for other lower-valued investments. For position players the tradeoff at the margin tells us first, to trade off hitting for fielding and second, OBP for BA.

So I think the real criteria for measuring the thesis is in the counterfactual of where the team would otherwise have finished--holding budget constant.

The most mysterious part of the story to me is how the A's have been so dominant in August every year (best month record of any team in any month over the past five years), but this hasn't carried over to September and October. I could see a story that their post-season woes were casued by a lack of depth--i.e., that budget pinches not in your starting lineup, but with respect to bench strength and middle relief. This is where it seems to me that the Yankees had been able overwhelm their opponents with money is by being deeper, and so being able to deal with injuries and single game/batter matchups better in a short series. But if it were merely depth, then it is hard to see why the A's consistently play so well in August.
8.21.2006 9:35am
KeithK (mail):
As a baseball fan, I'd rather have a team that is consistently successful but doesn't win the championship (like the A's) rather than one that wins it all one year but is weak for a long stretch thereafter (Diamondbacks or Marlins). Winning it all is a blast, but to me the joy of baseball is more about the day in and day out pennant race. (Of course, I'm a Yankee fan, so feel free to discount for that.)
8.21.2006 1:48pm
Johnny LaRue (mail):
Brian Cashman is not being asked to do his best on a budget--if he were, he may well do just as well as Billy Beane. On the other hand, if Beane were handed the keys to the Yankees there's no guarantee that his techniques would scale. Look at Dan Duquette of the Expos and Duquette of the BoSox.
*******
This is a good point. On those rare recent occasions when Beane is given permission and a "blank check" to go get a high priced free agent in the off-season, he seems to whiff pretty badly (Kendall, Loaiza), but still does very well at the sluggers flea market (F. Thomas, Jaha, even Durazo for a while).

On the other hand, Cashman/Steinbrenner (never sure who's calling the shots) blew WAAAAAY too much cash/prospects on Pavano and R. Johnson but still can find a few useful cheap retreads that no one else wanted (Chacon, Small, e.g. in 2005).
8.22.2006 5:20pm