Blog Post

Blogs review: The Reinhart and Rogoff debacle

What’s at stake: The authors of the widely acclaimed book on the history of financial crises, This Time is Different, have faced the mother of all academic backlashes after a group of economists identified important flaws (including basic coding errors) in their analysis of the relationship between public debt and economic growth.

By: Date: October 14, 2013 European Macroeconomics & Governance Tags & Topics

What’s at stake: The authors of the widely acclaimed book on the history of financial crises, This Time is Different, have faced the mother of all academic backlashes after a group of economists identified important flaws (including basic coding errors) in their analysis of the relationship between public debt and economic growth. They, in particular, debunked the now popular notion that there is a debt threshold (90% of GDP) after which economic growth decreases in a nonlinear way. The availability of the dataset (in Stata) has also allowed an econometrician raise serious doubts on the idea that the causality runs from debt to GDP. While the backlash has, until now, centered mostly around this particular work on debt and GDP growth, similar data issues have been identified in the book This Time is Different.

The 90% threshold, the austerity narrative, and the twitter backlash

Paul Krugman writes that the intellectual edifice of austerity economics rests largely on two academic papers that have now been debunked. One was Alesina/Ardagna (see our previous review here) on expansionary fiscal contractions. The other paper, which has had immense influence, was Reinhart/Rogoff on the negative effects of debt on growth. Very quickly, everyone “knew” that terrible things happen when debt passes 90 percent of GDP.

Mike Konczal writes that from the beginning there have been complaints that RR weren’t releasing the data for their results (e.g. Dean Baker). Konczal knew of several people trying to replicate the results who were bumping into walls left and right – it couldn’t be done. After trying to replicate the RR results and failing, Thomas Herndon, Michael Ash, Robert Pollin (HAP) reached out to RR and they were willing to share their data spreadsheet. This allowed Herndon et al. to see how RR’s data was constructed.

Peter Coy (HT Tim Duy) notes that the Twitterverse exploded with chatter about the new research paper claiming to find holes in a landmark academic paper that’s been cited to justify extreme austerity measures. By mid-afternoon the authors had emailed reporters a quick reply defending their work.

Data issues in “Growth in a time of debt”

Mike Konczal writes that HAP find that three main issues stand out. First, RR selectively exclude years of high debt and average growth. Second, they use a debatable method to weight the countries. Third, there also appears to be a coding error that excludes high-debt and average-growth countries. All three bias their results, and without them you don’t get their controversial result about low growth for countries with a debt to GDP ratio higher than 90%.

Ritwik Priya writes that removing the asymmetric weights and including all episodes is really where the meat and juice of the debate lies. The excel error only changes the results from -0.1 to 0.1 (see Ritchie King for a picture of the now infamous excel spreadsheet and some tips on how to use excel properly!), while this changes the result further from 0.1% to 2.2%. 2% vs 3% simply lacks the jaw-drop value of less than 0% vs. 3%, and is actually not even statistically significant.

HAP writes that RR adopts a non-standard weighting methodology for measuring average real GDP growth within their public debt/GDP categories. After assigning each country-year to one of four public debt/GDP groups, RR calculates the average real GDP growth for each country within the group, that is, a single average value for the country for all the years it appeared in the category. The country averages within each group were then averaged, equally weighted by country, to calculate the average real GDP growth rate within each public debt/GDP grouping. The problem is that equal weighting by country gives a one-year episode as much weight as nearly two decades in the above 90 percent public debt/GDP range.

Reinhart and Rogoff argue that their weighting procedure is hardly unconventional. We do not want to excessively weight Greece, for example, which has debt over 90% for 19 years in the 1946-2009 sample. The post-war Advanced Economy experience would quickly reduce to the experiences of Greece and Japan. Our approach has been followed in many other settings where one does not want to overly weight a small number of countries that may have their own peculiarities.

Josh Barro writes that because there were only seven countries in the data set that RR used to calculate average GDP growth under high debt conditions, and because they weighted each country’s average growth equally, getting New Zealand wrong by more than 10 percentage points was a very big deal, shaving 1.5 percentage points off their estimate of average growth. Ritwik Priya writes that it’s true that any data analysis exercise will have to make these choices about weighting. But for precisely this reason data analysis exercises offer, or should offer, at the very least, footnotes or explanatory pieces detailing these choices and a short note on why they were preferred to other choices.

Reinhart and Rogoff writes that the charge of selective omissions is the one they object to in the strongest terms. The “gaps” are explained by the fact there were still gaps in our public data debt set at the time of this paper, a data set no one else had ever been able to construct before and which we now have filled in much more completely.

The conceptual issues in “Growth in a time of debt”

Coppola writes that the Rortybomb blog delivered the killer punch to RR. Econometric analysis by Arindrajit Dube demonstrated that even with good data, the economic analysis was flawed and the conclusions unjustifiable. High public debt cannot reliably be shown to cause low growth. But low growth can reasonably reliably be shown to cause high public debt. Ritwik Priya writes that it is worth pointing out that the main reason RR come up with the 90% figure is because their intervals are of 30% i.e. they split the data into buckets of 0-30%, 30-60% and so on. This is purely a modeling choice artifact, and the actual tipping point, assuming any exists, may be 80% or 110% or 93.7%.

Arindrajit Dube notes that there is a visible negative relationship between growth and debt-to-GDP, but as HAP point out, the strength of the relationship is actually much stronger at low ratios of debt-to-GDP.  This makes us worry about the causal mechanism. After all, while nonlinearity may be expected at high ratios due to a tipping point, the stronger negative relationship at low ratios is difficult to rationalize using a tipping point dynamic.

Arindrajit Dube writes that while it is difficult to ascertain causality from plots like this, we can leverage the time pattern of changes to gain some insight. Here is a simple question: does a high debt-to-GDP ratio better predict future growth rates, or past ones? As is evident from the diagram below, current period debt-to-GDP is a pretty poor predictor of future GDP growth at debt-to-GDP ratios of 30 or greater—the range where one might expect to find a tipping point dynamic.  But it does a great job predicting past growth. This pattern is a telltale sign of reverse causality.

The do-file for reproducing Arindrajit Dube’s analysis is here.

Data issues in “This Time is Different”

A number of authors like Cardiff Garcia and Joseph Cotterill, Adam Posen, Ryan Avent (on Twitter) think that the criticism of this finding should be kept separate from judgments about their earlier book, “This Time is Different”, which remains a highly valuable contribution to the study of finance crises. Paul Krugman writes the book had a sound empirical strategy: it focused only on extreme events, then described what happened around those events. Because of the severity of the shock, it was reasonable to infer that whatever happened around crises was in fact crisis-related, so problems of causation were sidestepped.

But several authors have noted that there were also signs of sloppiness in the construction of the dataset for “This Time if Different”.

Andrew Jalil reveals the major inconsistencies in the banking panics series of “This Time is Different” based on his reading of contemporary news reports surrounding each of the banking panics episodes identified by RR. Jalil notes that the book actually provides two, sometimes contradicting, versions of the banking crisis series (Table A.3.1 and Table A.4.1). One version, for example, identifies December 1861 and April 1864 as banking crises, whereas the other does not contain. [April 1864 should not be classified as a banking panic since what happened was just a serious disturbance on stock markets that was unrelated to the states of banks and did not turn into a banking panic]. The RR series also happens to classify certain foreign crises as domestic ones.  The banking panic that took place in England in 1825 is, for example, wrongly classified as one that affected the US.

David Lopez-Salido and Edward Nelson write that the account of postwar U.S. financial crises by RR is also questionable on several counts. They treat the 1970s as free of financial crises in the United States, even though the mid-1970s witnessed banking stresses that saw banks’ equity capital ratio plunge to a postwar low. RR do not treat 1982 and 1983 as years of financial crisis, despite the pressure in those years on U.S. commercial banks brought by the LDC debt position. And the savings and loan crisis is referred to by R&R as the S&L crisis of 1984, even though S&L failures actually were lower in 1984 than in any other year in 1981-1989.


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article More on this topic More by this author

Blog Post

Silvia Merler

The Italian referendum

What’s at stake: on 4 December, Italy will hold a referendum on a proposed constitutional reform approved by Parliament in April. The reform, which was designed in tandem with a new electoral law, aims to overcome Italy’s “perfect bicameralism” by changing the structure and role of the Italian Senate. It also changes the distribution of competences between the state and regions. After the shocks of Brexit and the US election, polls are now drifting towards a defeat of the government’s position in Italy.

By: Silvia Merler Topic: European Macroeconomics & Governance Date: November 28, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Trumpocalypse now: first reactions

What’s at stake: this question should probably be re-formulated as “what’s NOT at stake?” On Tuesday 8 November, the US elected Donald Trump as its next President. Several aspects of Trump’s political and economic agenda appear extreme (we have previously focused on his stance on trade). After the initial shock, we review economists’ opinions on what has happened and what may happen. We will be coming back to this topic regularly.

By: Silvia Merler Topic: Global Economics & Governance Date: November 21, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Brexit and the law

What’s at stake: last week, the UK High Court ruled that the triggering of Article 50 - and therefore the Brexit process - should involve the UK Parliament. The Government will appeal the decision but this has created a new wave of uncertainty about the timing of Brexit, and on what this involvement can mean in practice. We review the different opinions.

By: Silvia Merler Topic: European Macroeconomics & Governance Date: November 14, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Monetary policy at the time of elections

What’s at stake: At this week’s meeting, the Federal Reserve left interest rates unchanged. While this was largely expected, the economic blogosphere has been discussing whether and to what extent this is linked to the election, and what can be expected for the future.

By: Silvia Merler Topic: Global Economics & Governance Date: November 7, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Should we rethink fiscal policy?

What’s at stake: there has been quite some discussion recently on whether we should rethink the framework of fiscal policy in order to make it more appropriate and effective in a world where demand seems to be chronically anemic, inflation is low and the interest rates are likely to stay close to zero (if not negative) for a long time. According to some of the authors, in the Eurozone these concerns are particularly pressing.

By: Silvia Merler Topic: Global Economics & Governance Date: October 24, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Brexit, the pound and the UK current account

What’s at stake: UK PM Theresa May announced the intention to trigger article 50 by March 2017, the Pound Sterling crashed, and a dispute among Tesco and Unilever has resulted in Marmite shortage. Brexit means Brexit, and it continues to be highly discussed. It would be impossible to summarise all the economic blogosphere on Brexit. Our aim is to periodically update our readers on selected important aspects of what promises to be a long-lived topic of discussion. This time we are looking at economists’ view on the Pound crash and the UK current account.

By: Silvia Merler Date: October 17, 2016
Read article More by this author

Blog Post

Silvia Merler

The Deutsche Bank Frenzy and what it says about European banks

What’s at stake: The IMF recently published its Fall Global Financial Stability Report, which points to a decrease in short-term risk but building of medium-term ones. At the same time, European market has been nervous last week on the news that Deutsche Bank (Germany’s biggest bank) has been demanded USD14bn by the US Department of Justice to settle allegations that the bank mis-sold mortgage-backed securities before the financial crisis. While reports point to a possible USD5.4bn settlement, this turmoil raises a question of whether the European financial system is still weak, eight years since the crisis. We try to summarize the reactions in the blogosphere.

By: Silvia Merler Topic: Finance & Financial Regulation, Global Economics & Governance Date: October 10, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Trumping Trade

What’s at stake: Trade is a central topic in the US presidential campaign, with both candidates expressing some degree of criticism about past trade policy. But while Hillary Clinton’s position could be described as a cautious scepticism, Donald Trump’s trade plans are more openly protectionist. His proposals include high tariffs on imports, renegotiating trade agreements and possibly US withdrawal from the WTO. After the first presidential debate, we review economists’ reactions and their assessment of Trumps trade policies.

By: Silvia Merler Topic: Global Economics & Governance Date: October 3, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Big in Japan

What’s at stake: This week saw two important Central Banks’ meetings, whose outcomes could hardly be more different. While the U.S. Federal Reserve left interest rates unchanged, the Bank of Japan introduced a big shift in its easing framework. BOJ committed itself to overshoot its inflation target of 2 percent, and introduced a targeting of the yield on ten-year Japanese government debt, initially at about zero percent. We review the economic blogosphere reaction to this latest monetary policy action.

By: Silvia Merler Topic: Global Economics & Governance Date: September 26, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

The US infrastructure investment debate

What’s at stake: Infrastructure investment has been and will continue to be a prominent campaign theme in the run up to the US elections. Both Hillary Clinton and Donald Trump have promised significant public investment in infrastructure. For some time, the discussion has revolved around the opportunities and costs of increased government infrastructure spending.

By: Silvia Merler Topic: Global Economics & Governance Date: September 19, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

The Apple of Discord

What’s at stake: On August 30th, following the results of an in-depth state aid investigation started in 2014, the European Commission concluded that Ireland granted undue tax benefits of up to €13 billion to Apple. The decision is based on state aid grounds: the Commission argues that two tax rulings issued by Ireland effectively granted Apple preferential treatment, which amounted to state aid. The Commission ordered Ireland to recover up to €13 billion (plus interest) from Apple, but the decision is controversial and opinion differ as to the effects it will have. We summarize reactions.

By: Silvia Merler Topic: Innovation & Competition Policy Date: September 12, 2016
Read article More on this topic More by this author

Blog Post

Silvia Merler

Post-Jackson Hole low morale

What’s at stake: this year’s edition of the Jackson Hole symposium was awaited as an occasion to discuss how to redesign monetary policy for the future. We documented the state of academic and policymaking discussion on the topic in a previous review. But it seems the meeting has left many with the impression the Fed is not yet ready to start “rethinking normality”.

By: Silvia Merler Topic: Global Economics & Governance Date: September 5, 2016
Load more posts