Tag Archives: Cosma Shalizi

Steven Johnson on the source of good ideas

Two short videos by science writer Steven Johnson on his book Where good ideas come from: the natural history of innovation.

An animated promotional video for his book:

And him giving a TED talk.

Steve Johnson has posted some of the responses to his ideas on his blog.

I haven’t read the book, but complex systems scientist Cosma Shalizi has a rich review that addresses many of the books strengths and weaknesses.  He introduces the book as:

This is 100-proof American evolutionist, naturalistic liberalism, which is to say, Pragmatism. It is a celebration of the virtues of openness, experimentation (including failed experiments), giving “slow hunches” chances to develop, to serendipitously blending ideas from diverse intellectual backgrounds and disciplines, and the continuity of human culture and thought with processes in the natural world. It’s a view of the social life of the mind, illustrated by engagingly-told anecdotes from the history of science and technology; apt references to a wide range of scholarly studies; long, admiring quotations from Darwin; the natural history of coral reefs and the evolution of sexual reproduction. (The broader history of culture, especially the fine arts, is occasionally alluded to, and there are abundantly merited plugs for his old teacher Franco Moretti’s studies on the evolution of genres and “distant reading”; but mostly it’s a science-and-technology book.) Johnson has painted a crowd scene: good ideas hardly ever come from isolated individuals thinking very hard and having flashes of inspiration; they come from people who are immersed in communities of inquiry, and especially from those who bridge multiple communities. The picture is an attractive one, which I actually think (or perhaps “fervently pray”) has a lot of truth to it.

Paradoxes of efficient market theory

Complex systems scientist Cosma Shalizi reviews economic journalist Justin Fox‘s book The Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Street for American Scientist magazine in the article Twilight of the Efficient Markets:

The Myth of the Rational Market, by Justin Fox, is an account—popular but thorough—of the roots, rise, triumph and ongoing fall of the theory of efficient markets in finance. This school of thought is an exemplary specimen of a type of social science that flourished after World War II: It has mathematical models at its center, has supposedly been empirically validated by statistical analyses, is indifferent to history and to institutions, and takes as an axiom that people are intelligent, farsighted and greedy. Unlike many economic theories, the efficient-market school has been influential beyond academia. It helped reshape ideas about how companies should be run, how executives should be paid, and indeed how the economy should be regulated (or not) to promote the general welfare. (In comic-book form: A mild-mannered social science by day, at night efficient-market theory puts on a cloak of ideology and struggles for the Capitalist Way.) The theory contributed, arguably, to setting up the crisis that has gripped the world economy since 2007. Its story is of much more than just scholarly interest.

The founding principles of efficient-market theory are easily described. The assumption on which all else rests is that, unless one has private knowledge, there is no way to profit from financial markets without risk. …

… Therefore, says efficient-market theory, securities prices are unpredictable. Current prices are supposed to be optimal forecasts, on the basis of currently available data, of the present value of future returns, because changes in optimal forecasts are, themselves, unpredictable. (If you know that tomorrow your forecast of next year’s gasoline price will be higher than today’s forecast by $1, you should raise your current forecast.) As Paul Samuelson put it, “properly anticipated prices fluctuate randomly.” The efficient-market hypothesis, as a technical term, is the claim that market prices cannot be predicted, either from past prices alone or from past prices combined with other publicly available information. One of the early triumphs of the school was the demonstration that stock prices look very much indeed like random walks.

… A vast superstructure was erected on these foundations, beginning in the 1950s and really taking off in the 1960s and 1970s. Particularly impressive wings of that edifice were devoted to the design of portfolios to balance risk against return and to the valuation of derivative securities (“contingent claims” or bets on the value of other securities), especially options to buy or sell stocks at given prices by given dates. As Fox notes, scholars of finance achieved acclaim, and were awarded substantial consulting fees, for solving pricing problems that by hypothesis were already being solved by the markets themselves! (Donald Mackenzie’s An Engine, Not a Camera explores this paradox in depth.)By the 1980s and 1990s, these ideas had led to changes in the way the investment industry worked, new concepts of corporate governance and new kinds of financial firms, which aimed to systematically identify arbitrage opportunities—deviations from what the theory said prices should be—and to earn a profit even as they eliminated those opportunities. More diffusely, the academic prestige of efficient-market theory provided, at the least, rhetorical support for deregulating markets, especially financial markets, and delegating more and more authority to them. This was aided by a conflation—subscribed to by many scholars—between those markets having informationally efficient prices (that is, unpredictable ones) and those markets allocating capital efficiently (directing savings to where the money can be used most profitably). The latter is the more usual economic notion of efficiency, but informationally efficient prices are neither necessary nor sufficient for efficient allocation.

The whole edifice, however, has turned out to be built, if not on sand, then at best on loose fill. More rigorous testing on larger data sets has shown that the capital asset pricing model does not fit the data; beta in particular does not predict returns at all. The response has been to identify variables that do predict returns and presume that they must be risk factors, although the extra risk has never been demonstrated. Prices are hard to predict, although not impossible, especially with high-frequency data (arriving minute-by-minute or faster). One reason markets are hard to predict is that they change much more than forecasts of future earnings should, and often they change on no detectable information at all. (Defenders claim that this just shows scholars aren’t smart enough to grasp information known to everyone in the market.) Economists taking a behavioral approach have shown that actual investors don’t act like the cool, farsighted calculators that efficient-market theory demands; worse, it turns out that having a handful of smart arbitrageurs around is actually not enough to swamp the “noise traders”—it really is the case that, as the saying goes, “markets can stay irrational longer than you can stay solvent.”

This leaves us at an impasse. Efficient-market theory ought, with any methodological justice, to be relegated to the Museum of Nice Tries. But there is no unified replacement theory, and developing one will be arduous, involving empirical and theoretical work on all scales, from the experimental psychology of individual investors, through the institutional constraints under which money managers work, to solving for the aggregated effects of market participants’ interactions. In the meantime, efficient-market theory provides a ready basis for precise calculations, and one that is moreover now built into the academic field of finance and into the practice and even infrastructure of the markets.