The Problem with Prediction Registries

Michael Mandel

David Brin (whom I like very much) has advocated 'prediction registries'-that is, keeping track of the predictions of experts and pundits, so we can see which ones were right and which ones are wrong.

A tempting idea. But Brin points out that such registries are not around the corner:

Before such an archive can be built, somebody must fund several man-years of R&D, all toward a system that can be seen as credible even when partisans see their own side proved wrong! That's not easy to do. It will take breakthroughs in the representation of opposing ideas. But the payoff could be stunning.

There's an even more basic problem with prediction registries--deciding on the time frame of the prediction. Consider, for example, the repeated prediction by most economists that the trade deficit is unsustainable, and will lead to a massive slide in the dollar. The December 4, 2004, cover story of The Economist was called "The Disappearing Dollar." The basic idea of the story was simple:

Over the next few years it seems an excellent bet that there will be a large drop in the dollar.

In many ways, this is the classic wrong call. Almost as soon as the The Economist was on the stands, the dollar started to appreciate against the euro and the yen (the dollar hit its low against the euro on 12/30/04, and has risen by 12% since then. The dollar hit its low against the yen on 1/14/05, and has risen by about 15% since then).

However!! The actual prediction was "over the next few years." So if the dollar dropped sharply in, say, 2008, the writers of the article would be entitled to say that they were right.

And if you think it's just journalists who suffer from this problem, note that the same article cited a 2004 paper by Noriel Roubini and Brad Setser (hi Brad!), which said:

The rapid deterioration of US net external debt position implied by large trade and current account deficits cannot continue indefinitely.


Even if the United States continues to be able to borrow on terms that other, comparable, debtors could not imagine, our analysis suggests that the U.S. is on an unsustainable and dangerous path.


the tensions created by this system are large, large enough to crack the system in the next three to four years.

Note that Roubini and Setser give themselves a window of the "next three or four years" for all those horrible things to happen.

Certainly nothing horrible has happened over the past 15 months or so, since they wrote the paper. The trade deficit has continued to grow, the dollar has appreciated, the U.S. economy has chugged along as happy as a pig in the mud. So in the short run the trade deficit was clearly sustainable. Not only that, it's unlikely that Roubini and Setser would have predicted an appreciation of the dollar.

However!! If there was to be a sharp downward depreciation in the dollar anytime between now and 2008, it would be consistent with the Roubini-Setser prediction. Moreover (and here's where the fun part comes in), even if there was a gradual depreciation of the dollar so that the trade deficit came down gently, I suspect that Roubini and Setser would deem that consistent with their prediction of "unsustainability" as well. Perhaps Brad could address that question.

In any case, I don't really mean to pick on them, though I do mean to pick on the Economist. It's just very hard to identify truly falsifiable predictions "in the wild" as it were.

P.S. As far as I know, nobody has ever systematically studied the accuracy of cover stories in either The Economist or BusinessWeek.

    Before it's here, it's on the Bloomberg Terminal. LEARN MORE