Refereeing Reinhart-Rogoff Debate
The discovery of an error in an influential research paper by Harvard University economists Carmen Reinhart and Kenneth Rogoff has sparked an academic firestorm. It’s time to sort through the wreckage.
To recap, Reinhart and Rogoff compiled an impressive collection of historical data on government-debt levels around the world. It initially covered a sample of 20 advanced economies over the postwar period, then expanded to include much of the past two centuries, as well as a separate sample of emerging markets. The data trove provided the first serious testing ground for exploring the links between levels of public debt and other things that matter for our well-being, such as financial crises and the rate of economic growth.
Their approach was simple. They measured typical economic-growth rates across countries during episodes when public-debt levels were low, medium, high or very high in relation to gross domestic product. Their conclusion: “Median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise; average (mean) growth rates are several percent lower.”
Enter their critics. Thomas Herndon, a tenacious 28-year-old graduate student at the University of Massachusetts Amherst, aided by professors Michael Ash and Robert Pollin, has highlighted three issues in Reinhart and Rogoff’s analysis of the postwar experience of advanced nations.
The biggest howler is the least consequential. By highlighting the wrong cells in an Excel spreadsheet, Reinhart and Rogoff actually took an average over 15 countries, rather than the full sample of 20. Embarrassing? Yes. Important? No. Of the five missing countries, only one -- Belgium -- had ever experienced very high debt. Adding it barely changed the findings because Belgium’s economic growth during its high-debt episode was roughly similar to that in other highly indebted nations.
The second issue is more substantive. Some of the early postwar data for Australia, Canada and New Zealand were not included in Reinhart and Rogoff’s original analysis because, as they have since explained, those data weren’t available when they wrote their 2010 draft. These countries experienced reasonably rapid growth during episodes of very high debt, so including them weakens the result.
The final issue is the toughest: How should we aggregate the data into an informative bottom line? To Reinhart and Rogoff’s critics, the natural approach is to take the average for each debt level across all years in all countries. This would, for example, give a country with 10 years of very high debt 10 times the weight of a country with only one year. Instead, Reinhart and Rogoff took an average growth rate for each country experiencing very high debt, then calculated the average across countries. In their approach, all countries with any experience of very high debt get the same weight.
Which approach makes more sense? That depends on the question you want to answer. Reinhart and Rogoff are trying to find the average country’s growth rate during episodes of very high debt. Their critics are seeking the average growth rate of GDP when debt is very high. These are subtly different.
From a statistical perspective, your preference might depend on your judgment about what drives differences in economic growth at a given level of debt. If you think broad country characteristics such as geography or quality of governance are the most important, you might choose Reinhart and Rogoff’s approach of averaging out the national idiosyncrasies to determine the experience of the “typical country.” If you believe that country and time-specific factors such as domestic-policy decisions matter most, then you might want to weight all years equally to average out these one-time influences.
But let’s not get lost in the trees. In the end, all the corrections advocated by the critics shift the average GDP growth for very-high-debt nations to 2.2 percent, from a negative 0.1 percent in Reinhart and Rogoff’s original work. The finding remains that economic growth is lower in very-high-debt countries (see chart). It has been disappointing to watch those on the left seize on the embarrassing Excel errors but ignore this bigger picture.
The negative correlation between public debt and economic growth is also present in the other samples studied by Reinhart and Rogoff, and it is a finding whose broad contours are consistent with other empirical analyses. Although the critics are right that it’s difficult to pin down the exact strength of the debt-growth correlation, that’s no reason to discard the balance of evidence suggesting that it is negative.
Equally, it’s time to abandon the more specific claim that there is a threshold of 90 percent of GDP beyond which the negative effects of public debt on economic growth become particularly evident. This was always a stretch, and is now quite clearly inconsistent with the balance of the evidence. Unfortunately, it’s the sort of sound bite that the media and our politicians find irresistible.
Lost in all this sound and fury is the real question that we should be debating: Is it appropriate to infer that high debt is driving slower growth, and hence governments need to take greater care before taking on debt? Or is lower GDP growth, or perhaps some other factor, the reason that debt burdens rise? If the observed correlations reflect the latter reason (and there are hints that it may), then the whole exercise has little relevance to public policy.
And even if higher debt causes lower growth, does this mean the government should borrow more or less? For a family, more borrowing means a bigger debt burden. But for a government, that may not be true. Research by the economists Brad DeLong and Lawrence Summers suggests that in a depressed economy, borrowing to pay for fiscal stimulus can actually reduce the debt-to-GDP ratio by preventing a deeper decline from hurting the long-term economic prospects of the unemployed.
It’s a symbol of the weakness of our public discourse over statistical analysis that the blooper reel has so dominated, crowding out the more important discussion of how these facts should inform public policy.
What about the broader issues that all this raises for how we should treat empirical economic studies? Stay tuned. We’ll turn to that issue in our next column.
(Betsey Stevenson is an associate professor of public policy at the University of Michigan. Justin Wolfers is a professor of public policy and economics at the University of Michigan, and a nonresident senior fellow of the Brookings Institution. Both are Bloomberg View columnists. The opinions expressed are their own.)
To contact the writers of this article: Justin Wolfers at firstname.lastname@example.org Betsey Stevenson at email@example.com
To contact the editor responsible for this article: Mark Whitehouse at firstname.lastname@example.org