Blog Post

Blogs review: The popularity of Randomized Control Trials

What’s at stake: The popularity of Randomized Control Trials (RCTs) in academia has led to an impressive increase in the amounts governments and international institutions spend on providing evidence from RCTs. While valued for their research design, they remain criticized for having little predictive value beyond the context of the original experiment and the difficulties they face in evaluating complex interventions.

By: , and Date: January 14, 2014 Topic: Global Economics & Governance

The coming of age of RCTs

Joshua Angrist and Jörn-Steffen Pischke write that empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact. The primary engine driving improvement has been a focus on the quality of empirical research designs. The advantages of a good research design are perhaps most easily apparent in research using random assignment.

The Economist writes that RCTs are like drug trials for economics. If you want to discover if, say, using identity cards would improve the delivery of subsidized rice to the poor and reduce theft, then you take a collection of comparable villages or households, and randomly assign ID cards to some and not to others. Then wait to see what occurs. Anders Olofsgård writes that the recent focus on impact evaluation within development economics has led to increased pressure on aid agencies to provide evidence from RCTs. Free exchange reports that the World Bank runs RCTs. So do regional bodies like the Inter-American Development Bank. Even governments deploy them.

Good estimates from the wrong place versus bad estimates from the right place

Joshua Angrist and Jörn-Steffen Pischke write that the rise of the experimentalist paradigm has provoked a reaction about the question of external validity – the concern that evidence from a given experimental or quasi-experimental research design has little predictive value beyond the context of the original experiment. The criticism here – made by a number of authors including Heckman (1997); Rosenzweig and Wolpin (2000); Heckman and Urzua (2009); and Deaton (2009) – is that in the quest for internal validity, design-based studies have become narrow or idiosyncratic.

Lant Pritchett and Justin Sandefur write that external validity is such a big issue that OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Berk Özler explains that the variation across contexts – that Pritchett and Sandefur refer to – is more or less defined as variation in well-identified effect sizes (ES), while variation within is the difference between a not well-identified estimate and the well-identified one within the same context. In other words, if you ran an OLS of vouchers on student performance using the baseline data in your experiment that provided vouchers and compared the biased ES with the experimental one, that would be your within variation. If you compared the experimental ES in Colombia with one from Tanzania, that would be your across variation.

Abhijit V. Banerjee and Esther Duflo write that the trade-off between internal and external validity is present as well in observational studies. Joshua Angrist and Jörn-Steffen Pischke write that empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design.

The role of interactions among components

Martin Ravallion writes that assessing a set of development policies and projects by evaluating its components one-by-one and adding up the results requires some assumptions that are pretty hard to accept. For one thing, it assumes that there are negligible interaction effects amongst the components. For another thing, it assumes that we can either evaluate everything in the portfolio or we can draw a representative sample, such that there is no selection bias in which components we choose to assess.

A bulletin of the World Health Organization explains that challenges in global health lie not in the identification of efficacious interventions, but rather in their effective scale-up. This requires a nuanced understanding of how implementation varies in different contexts. Context can have greater influence on uptake of an intervention than any pre-specified implementation strategy. Ann Oakley and al. write that most RCTs focus on outcomes, not on the processes involved in implementing an intervention and argue that that including a process evaluation would improve the science of many RCTs. Because of their multifaceted nature and dependence on social context, complex interventions pose methodological challenges, and require adaptations to the standard design of such trials.

Have RCTs diverted an important amount of resources?

Martin Ravallion writes that our evaluative efforts are progressively switching toward things for which certain methods are feasible, whether or not those things are important sources of the knowledge gaps relevant to assessing development impact at the portfolio level. There is a positive “output effect” of the current enthusiasm for impact evaluation, but there is also likely to be a negative “substitution effect.” As Jishnu Das, Shantayana Devarajan and Jeffrey Hammer have pointed out it is not clear that RCTs are well suited to programs involving the types of commodities for which market failures are a concern, such as those with large spillover effects in production or consumption.

Joshua Angrist and Jörn-Steffen Pischke write that related to the external validity critique is the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important. Given, for example, that the recent economic crisis has spawned intriguing design-based studies of the crisis macroeconomics (see Gabriel Chodorow-Reich’s 2013 job market paper for example), this claim seems less relevant today than when Noam Scheider complained about an evolution epitomized by the success of Freakonomics.

 

 

 


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article More by this author

Blog Post

It’s hard to live in the city: Berlin’s rent freeze and the economics of rent control

A proposal in Berlin to ban increases in rent for the next five years sparked intense debate in Germany. Similar policies to the Mietendeckel are currently being discussed in London and NYC. All three proposals reflect and raise similar concerns – the increase in per-capita incomes is not keeping pace with increases in rents, but will a cap do more harm than good? We review recent views on the matter.

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: July 8, 2019
Read article More on this topic More by this author

Blog Post

The breakdown of the covered interest rate parity condition

A textbook condition of international finance breaks down. Economic research identifies the interplay between divergent monetary policies and new financial regulation as the source of the puzzle, and generates concerns about unintended consequences for financing conditions and financial stability.

By: Konstantinos Efstathiou Topic: Finance & Financial Regulation Date: July 1, 2019
Read article More on this topic More by this author

Blog Post

The June Eurogroup meeting: Reflections on BICC

The Eurogroup met on June 13th to discuss the deepening of the economic and monetary union (EMU) and prepare the discussions for the Euro Summit. From the meeting came two main deliverables: an agreement over a budgetary instrument for competitiveness and convergence and the reform of the European Stability Mechanism (ESM) treaty texts. We review economists’ first impressions.

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: June 24, 2019
Read article More on this topic More by this author

Blog Post

The campaign against ‘nonsense’ output gaps

A campaign against “nonsense” consensus output gaps has been launched on social media. It has triggered responses focusing on the implications of output gaps for fiscal policy under EU rules, especially for Italy. But the debate about the reliability of output-gap estimates is more wide-ranging.

By: Konstantinos Efstathiou Topic: European Macroeconomics & Governance Date: June 17, 2019
Read article More on this topic More by this author

Blog Post

The inverted yield curve

Longer-term yields falling below shorter-term yields have historically preceded recessions. Last week, the US 10-year yield was 21 basis points below the 3-month yield, a feat last seen during the summer of 2007. Is the current yield curve a trustworthy barometer for future growth?

By: Inês Goncalves Raposo Topic: Global Economics & Governance Date: June 11, 2019
Read article More on this topic More by this author

Blog Post

The 'seven' ceiling: China's yuan in trade talks

Investors and the public have been looking at the renminbi with caution after the Trump administration threatened to increase duties on countries that intervene in the markets to devalue/undervalue their currency relative to the dollar. The fear is that China could weaponise its currency following the further increase in tariffs imposed by the United States in early May. What is the likelihood of this happening and what would be the consequences for the existing tensions with the United States, as well as for the global economy?

By: Inês Goncalves Raposo Topic: Global Economics & Governance Date: June 3, 2019
Read article More on this topic More by this author

Blog Post

The next ECB president

On May 28th, EU heads of state and government will start the nomination process for the next ECB president. Leaving names of possible candidates aside, this review tries to isolate the arguments about what qualifications the new president should have and what challenges he or she is likely to face.

By: Konstantinos Efstathiou Topic: European Macroeconomics & Governance Date: May 27, 2019
Read article More on this topic More by this author

Blog Post

The latest European growth-rate estimates

The quarterly growth rate of the euro area in Q1 2019 was 0.4% (1.5% annualized), considerably higher than the low growth rates of the previous two quarters. This blog reviews the reaction to the release of these numbers and the discussion they have triggered about the euro area’s economic challenges.

By: Konstantinos Efstathiou Topic: European Macroeconomics & Governance Date: May 20, 2019
Read article More by this author

Blog Post

Is an electric car a cleaner car?

An article published by the Ifo Institute in Germany compares the carbon footprint of a battery-electric car to that of a diesel car, and argues a higher share of electric cars will not contribute to reducing German carbon dioxide emissions. Respondents rejected the authors’ calculations as unrealistic and biased, and pointed to a series of studies that conclude the opposite. We summarise the article and responses to it.

By: Michael Baltensperger Topic: Energy & Climate, Innovation & Competition Policy Date: May 13, 2019
Read article More on this topic More by this author

Blog Post

All eyes on the Fed

Last week the US Federal Reserve left the federal funds rate unchanged and lowered the interest rate on excess reserves. We review economists’ recent views on the monetary policy conduct and priorities of the United States’ central bank system.

By: Inês Goncalves Raposo Topic: Global Economics & Governance Date: May 6, 2019
Read article More on this topic More by this author

Blog Post

Is this blog post legal (under new EU copyright law)?

How new EU rules on using snippets from news publishers and on copyright infringement liability might affect circulation of information, revenue distribution, market power and EU business competitiveness.

By: Catarina Midoes Topic: European Macroeconomics & Governance Date: April 8, 2019
Read article More on this topic More by this author

Blog Post

Secular stagnation and the future of economic stabilisation

Larry Summers’ and Łukasz Rachel’s most recent study documents a secular fall in neutral real rates in advanced economies. According to the authors, this fall would be even more marked in the absence of offsetting fiscal policies. Policymaking in a world of permanently low interest rates may be hard to navigate, especially in troubled waters. We review economists’ views on the matter

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: April 1, 2019
Load more posts