The Uses of Knowledge: Prediction

A number of social scientific studies and discussions have come to the conclusion that "experts" in the softer fields such as social science or history tend to be not as good as reasonably well informed generalists when it comes to predicting future events. Yet these experts are the ones called in as consultants when governments or businesses face decisions in complicated situations. This conclusion obviously relates to the accompanying discussion of Science, Knowledge, and Ignorance.

Many reasons are cited for the failures of experts. One is the very fact that they see themselves as more informed and therefore more capable on topics related to their specialities than the average well-informed reader of a paper such as the New York Times. They resist, therefore, making predictions paralleling those that anyone could read in a newspaper. To preserve their status they infer that their predictions have to be special, adding to the discussion possibilities others have not thought about. Secondly, the expert is more invested in the subject at issue than others. This leads him or her to have developed well-defined positions before questions are asked. In their answers, experts will generally strive to defend these preformed positions. An expert will also have a tendency to over value the aspects of a situation with which he or she is particularly identified. For example, if asked a question about the future of India, an expert on Hinduism will have a pre-determined tendency to see any future for India to be heavily influenced by one or another aspect of Hinduism. Finally, because the expert feels that he must provide answers, he will feel pressured to take strong positions on issues when in fact there is little basis for confidence in any prediction.

Experts find scenarios will more variables more likely than those with less, even though a consideration of simple probabilities reveals that this cannot be right. A group of professional planners and forecasters were asked, for example, whether diplomatic relations between the United States and the USSR would rupture in 1983. A similar group was asked whether the Russians would attack Poland and a rupture would occur in 1983. The probability of the second scenario was judged to be higher than the first. The experts were simply seduced by the detail. They therefore failed to heed the most useful and common sense principle for forecasters: predictions are more likely to be realized if they require no more than one or two events to occur between the present and the predicted future.

[In large part, the foregoing discussion is summarized from Louis Menand's "Everybody's an Expert", The New Yorker, December 5, 2005. Menand's piece is primarily a review, in turn, of Tetlock's, Expert Political Judgment: How Good Is It? How Can We Know?. Tetlock's psychological study is based on a long-term study of a selected group of professional economic and political forecasters and commentators over a twenty-year period. (This involved a consideration of 82,000 forecasts.) He found that in detailed comparisons, the forecasters often did worse than chance. Menand adds that Tetlock's work supports hundreds of other similar studies. The reader may be interested in a related line of inquiry addressed by James Surowiecki in The Wisdom of Crowds. According to Surowiecki, properly organized, a large group of nonexperts will do better than experts, even small groups of experts, in many problem solving or forecasting situations.]

This devastating criticism of expert prediction cuts close to home, in that the author was at least on the margins of the forecasting world in his Hudson years (The Year 2000). Even though in retrospect, the Hudson predictions have not held up well, he has continued to believe that he might add something to discussions of future possibilities because of his more informed analytic and ultimately predictive powers. The evidence discussed here certainly casts doubt on this assumption. Moreover, the criticism of expert prediction has seemed to be confirmed by recent experience. Last year I made regular entries in a blog concerned primarily with the Iraq and Afghanistan conflicts. I thought I should be able to have something useful to say on the efforts of the United States and the other actors, since I have a degree in Middle Eastern Studies and spent years working on National Security Policy. However, in reviewing my informed comments and predictions, I found that I was all over the map, as unable to separate the wheat from the chaff as many others, including other experts and members of the Administration.

The discussion of the weakness of expert prediction makes less likely many of the contributions I have assumed I would be able to make in the future. It also leads to the more general question of what the sometime social scientist, or cultural expert, or historian has to offer. How and why should he or she contribute to the discussion of issues that face the country and the world?

In order to begin an analysis of the legitimate roles of the expert and expertise, let us take the example of an expert on Iran, which I once was. I could have legitimately spent a lifetime developing my knowledge of all things Iranian, contributing from time to time to the general fund of knowledge that Iranians and non-Iranians have of their country's culture, history, literature, natural resources, geography etc. This would have provided an improved basis on which decision makers and purveyors of news would develop their decisions and communications. It appears it should help, for example, for such people to know that Iran was for most of its post-Islamic history Sunni rather than Shiite. Only after 1500 A.D. were most Iranians forcibly converted to Shiism by a Turkic conqueror. Iran's great poetic tradition was produced by Sunnis rather than Shiites, to the extent it was informed at all by religious creeds. In fact, many of its cultural heroes were Sufis, mystics who sought a direct relation with God and discountenanced the orthodox religious schools. Until very recently, in fact, the religious experience of many Iranians has been Sufic, a form of Islam heatedly rejected by the Shiite religious establishment in Iran today. I know these things, but to what extent is such knowledge really helpful? Is it merely curious knowledge useful for cocktail parties?

Remembering the accompanying discussion of science and knowledge, we seem to be talking about the value of background knowledge in policy making. Improved knowledge of Iran, such as that of the factoids sketched above, will let the world have a more complex understanding of Iran and Iranians today. It will mean, for example, that Western publics and their leaders will not be surprised if it turns out that young Iranians are less devoted to their present religious rulers than they appear to be on the surface. However, these factoids do not actually mean that an expert who knows them is able to predict with a high degree of probability that events will turn in one or another possible direction. He may be quite poorly positioned to do so. He is likely to be more aware of these factoids than even the average Iranian, and to spend more of his time with Iranians who are aware of them. In particular, he is likely to have middle class Iranian friends, in Iran or overseas, who are attracted to what are minority Iranian traditions, and thereby apt to overestimate their possible role in contemporary Iran. His emotional attachment to Iranians who think like he does, to their interests and judgments, can seriously impair his ability to add important insights to what faces a decision maker who must deal with the Iran that exists today.