Blog Post

Blogs review: The popularity of Randomized Control Trials

What’s at stake: The popularity of Randomized Control Trials (RCTs) in academia has led to an impressive increase in the amounts governments and international institutions spend on providing evidence from RCTs. While valued for their research design, they remain criticized for having little predictive value beyond the context of the original experiment and the difficulties they face in evaluating complex interventions.

By: , and Date: January 14, 2014 Topic: Global Economics & Governance

The coming of age of RCTs

Joshua Angrist and Jörn-Steffen Pischke write that empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact. The primary engine driving improvement has been a focus on the quality of empirical research designs. The advantages of a good research design are perhaps most easily apparent in research using random assignment.

The Economist writes that RCTs are like drug trials for economics. If you want to discover if, say, using identity cards would improve the delivery of subsidized rice to the poor and reduce theft, then you take a collection of comparable villages or households, and randomly assign ID cards to some and not to others. Then wait to see what occurs. Anders Olofsgård writes that the recent focus on impact evaluation within development economics has led to increased pressure on aid agencies to provide evidence from RCTs. Free exchange reports that the World Bank runs RCTs. So do regional bodies like the Inter-American Development Bank. Even governments deploy them.

Good estimates from the wrong place versus bad estimates from the right place

Joshua Angrist and Jörn-Steffen Pischke write that the rise of the experimentalist paradigm has provoked a reaction about the question of external validity – the concern that evidence from a given experimental or quasi-experimental research design has little predictive value beyond the context of the original experiment. The criticism here – made by a number of authors including Heckman (1997); Rosenzweig and Wolpin (2000); Heckman and Urzua (2009); and Deaton (2009) – is that in the quest for internal validity, design-based studies have become narrow or idiosyncratic.

Lant Pritchett and Justin Sandefur write that external validity is such a big issue that OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Berk Özler explains that the variation across contexts – that Pritchett and Sandefur refer to – is more or less defined as variation in well-identified effect sizes (ES), while variation within is the difference between a not well-identified estimate and the well-identified one within the same context. In other words, if you ran an OLS of vouchers on student performance using the baseline data in your experiment that provided vouchers and compared the biased ES with the experimental one, that would be your within variation. If you compared the experimental ES in Colombia with one from Tanzania, that would be your across variation.

Abhijit V. Banerjee and Esther Duflo write that the trade-off between internal and external validity is present as well in observational studies. Joshua Angrist and Jörn-Steffen Pischke write that empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design.

The role of interactions among components

Martin Ravallion writes that assessing a set of development policies and projects by evaluating its components one-by-one and adding up the results requires some assumptions that are pretty hard to accept. For one thing, it assumes that there are negligible interaction effects amongst the components. For another thing, it assumes that we can either evaluate everything in the portfolio or we can draw a representative sample, such that there is no selection bias in which components we choose to assess.

A bulletin of the World Health Organization explains that challenges in global health lie not in the identification of efficacious interventions, but rather in their effective scale-up. This requires a nuanced understanding of how implementation varies in different contexts. Context can have greater influence on uptake of an intervention than any pre-specified implementation strategy. Ann Oakley and al. write that most RCTs focus on outcomes, not on the processes involved in implementing an intervention and argue that that including a process evaluation would improve the science of many RCTs. Because of their multifaceted nature and dependence on social context, complex interventions pose methodological challenges, and require adaptations to the standard design of such trials.

Have RCTs diverted an important amount of resources?

Martin Ravallion writes that our evaluative efforts are progressively switching toward things for which certain methods are feasible, whether or not those things are important sources of the knowledge gaps relevant to assessing development impact at the portfolio level. There is a positive “output effect” of the current enthusiasm for impact evaluation, but there is also likely to be a negative “substitution effect.” As Jishnu Das, Shantayana Devarajan and Jeffrey Hammer have pointed out it is not clear that RCTs are well suited to programs involving the types of commodities for which market failures are a concern, such as those with large spillover effects in production or consumption.

Joshua Angrist and Jörn-Steffen Pischke write that related to the external validity critique is the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important. Given, for example, that the recent economic crisis has spawned intriguing design-based studies of the crisis macroeconomics (see Gabriel Chodorow-Reich’s 2013 job market paper for example), this claim seems less relevant today than when Noam Scheider complained about an evolution epitomized by the success of Freakonomics.

 

 

 


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article More on this topic More by this author

Blog Post

A few good (wo)men – on the representation of women in economics

Last week, the American Economics Association Annual Meetings held a session on Gender Issues in Economics and later announced that a new code of professional conduct is in the pipeline. In this blogs review we revise the recent contributions on female representation and perception in economics.

By: Inês Goncalves Raposo Topic: Global Economics & Governance Date: January 15, 2018
Read article More on this topic More by this author

Blog Post

The Republican Tax Plan (2): The debate rumbles on

Reactions to the Republican tax plans continue, concentrating on different aspects of the proposed legislation. We review the latest contributions.

By: Silvia Merler Topic: Global Economics & Governance Date: December 18, 2017
Read article More by this author

Blog Post

The DSGE Model Quarrel (Again)

Dynamic Stochastic General Equilibrium models have come under fire since the financial crisis. A recent paper by Christiano, Eichenbaum and Trabandt – who provide a defense for DSGE – has generated yet another wave of reactions in the economic blogosphere. We review the most recent contributions on this topic.

By: Silvia Merler Topic: European Macroeconomics & Governance, Global Economics & Governance Date: December 11, 2017
Read article More by this author

Blog Post

The Bitcoin Bubble

The price of bitcoin has just passed $11,000. A year ago it was worth less than $800. Economists and commentators are thus increasingly concerned that this may be a bubble waiting to burst. We review recent opinions on the topic.

By: Silvia Merler Topic: Finance & Financial Regulation, Global Economics & Governance Date: December 4, 2017
Read article More on this topic More by this author

Blog Post

The Republican Tax Plan

As the Trump administration’s tax plan continues its way through the legislature, we review economists’ and commentators’ recent opinions on the matter.

By: Silvia Merler Topic: Global Economics & Governance Date: November 27, 2017
Read article More on this topic More by this author

Blog Post

Has the Phillips curve disappeared?

The Phillips curve prescribes a negative trade-off between inflation and unemployment. Economists have been recently debating on whether the curve has disappeared in the US and Europe. We report some of the most recent views.

By: Silvia Merler Topic: Global Economics & Governance Date: November 21, 2017
Read article More on this topic More by this author

Blog Post

Powell's Federal Reserve

With the appointment of Jerome Powell as the next Fed’s chairman, President Trump break a tradition of bipartisan re-nomination and chooses someone who is not an economy by formation. We review economist’s opinions on this choice and the challenges ahead.

By: Silvia Merler Topic: Global Economics & Governance Date: November 13, 2017
Read article More on this topic More by this author

Blog Post

The Bank of England’s dovish hike

For the first time since 2007, the Bank of England raised interest rates, with a hike of 25 basis points. At the same time, it provided forward guidance that outlines a very gradual path for future increases. We review the economic blogosphere’s reaction to this decision.

By: Silvia Merler Topic: European Macroeconomics & Governance Date: November 6, 2017
Read article More on this topic More by this author

Blog Post

The capital tax cut debate

How much do workers gain from a capital gains tax cut? CEA chairman Hasset claims the tax cut will cause average household labour income to increase by between $4000 and $9000. Several commentators note this implies that more than 100% of the incidence of the tax is on labour. This question has triggered a heated discussion in the economic blogosphere, which we review here.

By: Silvia Merler Topic: Global Economics & Governance Date: October 30, 2017
Read article More by this author

Blog Post

Bailout, bail-in and incentives

Ever since the outbreak of the global financial crisis, more and more rules have been developed to reduce the public cost of banking crises and increase the private sector’s share of the cost. We review some of the recent academic literature on bailout, bail-in and incentives.

By: Silvia Merler Topic: Finance & Financial Regulation, Global Economics & Governance Date: October 23, 2017
Read article More on this topic More by this author

Blog Post

An irrational choice: behavioural economist wins Nobel Prize

Richard Thaler was awarded this year's Nobel Prize in Economics for his contributions to the field of behavioural economics. His work documents a set of cognitive biases affecting economic decision-making and casts doubt on commonly-held assumptions about the rational ‘homo economicus’ that inhabits economic models and theories. What are the implications for the economics discipline and public policy?

By: Konstantinos Efstathiou Topic: Global Economics & Governance Date: October 16, 2017
Read article More on this topic More by this author

Blog Post

On the cost of gun ownership

On 1 October 2017, 59 people were killed and another 489 injured in what is currently the deadliest mass shooting in US modern history. The author reviews recent contributions on the economic cost of gun violence, as well as the impact of regulation.

By: Silvia Merler Topic: Global Economics & Governance Date: October 11, 2017
Load more posts