Complex systems scientist Cosma Shalizi reviews economic journalist Justin Fox‘s book The Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Street for American Scientist magazine in the article Twilight of the Efficient Markets:

The Myth of the Rational Market, by Justin Fox, is an account—popular but thorough—of the roots, rise, triumph and ongoing fall of the theory of efficient markets in finance. This school of thought is an exemplary specimen of a type of social science that flourished after World War II: It has mathematical models at its center, has supposedly been empirically validated by statistical analyses, is indifferent to history and to institutions, and takes as an axiom that people are intelligent, farsighted and greedy. Unlike many economic theories, the efficient-market school has been influential beyond academia. It helped reshape ideas about how companies should be run, how executives should be paid, and indeed how the economy should be regulated (or not) to promote the general welfare. (In comic-book form: A mild-mannered social science by day, at night efficient-market theory puts on a cloak of ideology and struggles for the Capitalist Way.) The theory contributed, arguably, to setting up the crisis that has gripped the world economy since 2007. Its story is of much more than just scholarly interest.

The founding principles of efficient-market theory are easily described. The assumption on which all else rests is that, unless one has private knowledge, there is no way to profit from financial markets without risk. …

… Therefore, says efficient-market theory, securities prices are unpredictable. Current prices are supposed to be optimal forecasts, on the basis of currently available data, of the present value of future returns, because changes in optimal forecasts are, themselves, unpredictable. (If you know that tomorrow your forecast of next year’s gasoline price will be higher than today’s forecast by $1, you should raise your current forecast.) As Paul Samuelson put it, “properly anticipated prices fluctuate randomly.” The efficient-market hypothesis, as a technical term, is the claim that market prices cannot be predicted, either from past prices alone or from past prices combined with other publicly available information. One of the early triumphs of the school was the demonstration that stock prices look very much indeed like random walks.

… A vast superstructure was erected on these foundations, beginning in the 1950s and really taking off in the 1960s and 1970s. Particularly impressive wings of that edifice were devoted to the design of portfolios to balance risk against return and to the valuation of derivative securities (“contingent claims” or bets on the value of other securities), especially options to buy or sell stocks at given prices by given dates. As Fox notes, scholars of finance achieved acclaim, and were awarded substantial consulting fees, for solving pricing problems that by hypothesis were already being solved by the markets themselves! (Donald Mackenzie’s An Engine, Not a Camera explores this paradox in depth.)By the 1980s and 1990s, these ideas had led to changes in the way the investment industry worked, new concepts of corporate governance and new kinds of financial firms, which aimed to systematically identify arbitrage opportunities—deviations from what the theory said prices should be—and to earn a profit even as they eliminated those opportunities. More diffusely, the academic prestige of efficient-market theory provided, at the least, rhetorical support for deregulating markets, especially financial markets, and delegating more and more authority to them. This was aided by a conflation—subscribed to by many scholars—between those markets having informationally efficient prices (that is, unpredictable ones) and those markets allocating capital efficiently (directing savings to where the money can be used most profitably). The latter is the more usual economic notion of efficiency, but informationally efficient prices are neither necessary nor sufficient for efficient allocation.

The whole edifice, however, has turned out to be built, if not on sand, then at best on loose fill. More rigorous testing on larger data sets has shown that the capital asset pricing model does not fit the data; beta in particular does not predict returns at all. The response has been to identify variables that do predict returns and presume that they must be risk factors, although the extra risk has never been demonstrated. Prices are hard to predict, although not impossible, especially with high-frequency data (arriving minute-by-minute or faster). One reason markets are hard to predict is that they change much more than forecasts of future earnings should, and often they change on no detectable information at all. (Defenders claim that this just shows scholars aren’t smart enough to grasp information known to everyone in the market.) Economists taking a behavioral approach have shown that actual investors don’t act like the cool, farsighted calculators that efficient-market theory demands; worse, it turns out that having a handful of smart arbitrageurs around is actually not enough to swamp the “noise traders”—it really is the case that, as the saying goes, “markets can stay irrational longer than you can stay solvent.”

This leaves us at an impasse. Efficient-market theory ought, with any methodological justice, to be relegated to the Museum of Nice Tries. But there is no unified replacement theory, and developing one will be arduous, involving empirical and theoretical work on all scales, from the experimental psychology of individual investors, through the institutional constraints under which money managers work, to solving for the aggregated effects of market participants’ interactions. In the meantime, efficient-market theory provides a ready basis for precise calculations, and one that is moreover now built into the academic field of finance and into the practice and even infrastructure of the markets.