Blog Post

Blogs review: The popularity of Randomized Control Trials

What’s at stake: The popularity of Randomized Control Trials (RCTs) in academia has led to an impressive increase in the amounts governments and international institutions spend on providing evidence from RCTs. While valued for their research design, they remain criticized for having little predictive value beyond the context of the original experiment and the difficulties they face in evaluating complex interventions.

By: , and Date: January 14, 2014 Topic: Global Economics & Governance

The coming of age of RCTs

Joshua Angrist and Jörn-Steffen Pischke write that empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact. The primary engine driving improvement has been a focus on the quality of empirical research designs. The advantages of a good research design are perhaps most easily apparent in research using random assignment.

The Economist writes that RCTs are like drug trials for economics. If you want to discover if, say, using identity cards would improve the delivery of subsidized rice to the poor and reduce theft, then you take a collection of comparable villages or households, and randomly assign ID cards to some and not to others. Then wait to see what occurs. Anders Olofsgård writes that the recent focus on impact evaluation within development economics has led to increased pressure on aid agencies to provide evidence from RCTs. Free exchange reports that the World Bank runs RCTs. So do regional bodies like the Inter-American Development Bank. Even governments deploy them.

Good estimates from the wrong place versus bad estimates from the right place

Joshua Angrist and Jörn-Steffen Pischke write that the rise of the experimentalist paradigm has provoked a reaction about the question of external validity – the concern that evidence from a given experimental or quasi-experimental research design has little predictive value beyond the context of the original experiment. The criticism here – made by a number of authors including Heckman (1997); Rosenzweig and Wolpin (2000); Heckman and Urzua (2009); and Deaton (2009) – is that in the quest for internal validity, design-based studies have become narrow or idiosyncratic.

Lant Pritchett and Justin Sandefur write that external validity is such a big issue that OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Berk Özler explains that the variation across contexts – that Pritchett and Sandefur refer to – is more or less defined as variation in well-identified effect sizes (ES), while variation within is the difference between a not well-identified estimate and the well-identified one within the same context. In other words, if you ran an OLS of vouchers on student performance using the baseline data in your experiment that provided vouchers and compared the biased ES with the experimental one, that would be your within variation. If you compared the experimental ES in Colombia with one from Tanzania, that would be your across variation.

Abhijit V. Banerjee and Esther Duflo write that the trade-off between internal and external validity is present as well in observational studies. Joshua Angrist and Jörn-Steffen Pischke write that empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design.

The role of interactions among components

Martin Ravallion writes that assessing a set of development policies and projects by evaluating its components one-by-one and adding up the results requires some assumptions that are pretty hard to accept. For one thing, it assumes that there are negligible interaction effects amongst the components. For another thing, it assumes that we can either evaluate everything in the portfolio or we can draw a representative sample, such that there is no selection bias in which components we choose to assess.

A bulletin of the World Health Organization explains that challenges in global health lie not in the identification of efficacious interventions, but rather in their effective scale-up. This requires a nuanced understanding of how implementation varies in different contexts. Context can have greater influence on uptake of an intervention than any pre-specified implementation strategy. Ann Oakley and al. write that most RCTs focus on outcomes, not on the processes involved in implementing an intervention and argue that that including a process evaluation would improve the science of many RCTs. Because of their multifaceted nature and dependence on social context, complex interventions pose methodological challenges, and require adaptations to the standard design of such trials.

Have RCTs diverted an important amount of resources?

Martin Ravallion writes that our evaluative efforts are progressively switching toward things for which certain methods are feasible, whether or not those things are important sources of the knowledge gaps relevant to assessing development impact at the portfolio level. There is a positive “output effect” of the current enthusiasm for impact evaluation, but there is also likely to be a negative “substitution effect.” As Jishnu Das, Shantayana Devarajan and Jeffrey Hammer have pointed out it is not clear that RCTs are well suited to programs involving the types of commodities for which market failures are a concern, such as those with large spillover effects in production or consumption.

Joshua Angrist and Jörn-Steffen Pischke write that related to the external validity critique is the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important. Given, for example, that the recent economic crisis has spawned intriguing design-based studies of the crisis macroeconomics (see Gabriel Chodorow-Reich’s 2013 job market paper for example), this claim seems less relevant today than when Noam Scheider complained about an evolution epitomized by the success of Freakonomics.

 

 

 


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article More on this topic More by this author

Blog Post

On Modern Monetary Theory

An old debate is back with a kick. The discussion around modern monetary theory first gained traction in the economic blogosphere around 2012. Recent interventions in the US and UK political arenas rekindled the interest in the heterodox theory that is now seeping into mainstream debates.

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: February 11, 2019
Read article More on this topic More by this author

Blog Post

The American tax debate

The debate over two different proposals for tax reforms: Senator Elizabeth Warren’s plan for a tax on wealth, and Congresswoman Alexandria Ocasio-Cortez’s plan for a higher top marginal tax rate on income

By: Enrico Bergamini Topic: Global Economics & Governance Date: February 4, 2019
Read article More on this topic More by this author

Blog Post

Brexit: Now for something completely different?

The life of Brexit. After a week of ECJ rulings, delayed votes, Theresa May’s errands across Europe and the vote of no confidence, we review the latest economists’ opinions to try to make sense of what has changed and what hasn’t.

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: December 17, 2018
Read article More by this author

Blog Post

Les gilets jaunes

For weeks, protesters wearing yellow motorist vests have taken to the streets of Paris to protest against the rising price of fuel. They have since taken on a wider role, and are seen as symbols of the growing popular discontent with President Macron. Silvia Merler reviews scholars’ opinions about this movement.

By: Silvia Merler Topic: Energy & Climate, European Macroeconomics & Governance Date: December 10, 2018
Read article More by this author

Blog Post

Green central banking

A few weeks ago, Silvia Merler discussed the rise of “ethical investing”. A related question emerging from the discussion is whether central banks should also “go green”. Silvia reviews the latest developments and opinions on this topic.

By: Silvia Merler Topic: Energy & Climate, Finance & Financial Regulation Date: December 3, 2018
Read article More on this topic More by this author

Blog Post

Machine learning and economics

Machine learning (ML), together with artificial intelligence (AI), is a hot topic. Economists have been looking into machine learning applications not only to obtain better prediction, but also for policy targeting. We review some of the contributions.

By: Silvia Merler Topic: Innovation & Competition Policy Date: November 29, 2018
Read article More on this topic More by this author

Blog Post

The Brexit withdrawal agreement

On November 14th the UK government cabinet approved the draft text of the withdrawal agreement, the deal reached between EU and UK negotiators. The decision was followed the next day by the resignations of several members of Parliament. We review the first reactions in the blogosphere.

By: Silvia Merler Topic: European Macroeconomics & Governance Date: November 19, 2018
Read article More on this topic More by this author

Blog Post

US mid-term elections and the global economy

Democrats won control of the House and Republicans held onto the Senate in the most consequential US mid-term elections in decades. Bowen Call reviews economists’ and scholars’ analyses of the impact this might have on the world economy.

By: Bowen Call Topic: Global Economics & Governance Date: November 12, 2018
Read article More on this topic More by this author

Blog Post

Jair Bolsonaro’s Brazil

Far-right candidate Jair Bolsonaro won the Brazilian presidential elections after a highly polarising campaign. We review economists’ and scholars’ views of what this means for Brazil going forward.

By: Silvia Merler Topic: Global Economics & Governance Date: November 5, 2018
Read article More on this topic More by this author

Blog Post

The rise of 'ethical' investing

We are used to think about the value of investment as measured by financial return. But investing with an eye to environmental or social issues and, more generally, ethical considerations, has become more prominent. We review contributions to this debate.

By: Silvia Merler Topic: Finance & Financial Regulation Date: October 29, 2018
Read article More on this topic More by this author

Blog Post

The United States-Mexico-Canada free trade agreement (USMCA)

While final ratification of the USMCA (also known as Nafta 2.0) is pending, we review economists’ assessment of the agreement.

By: Silvia Merler Topic: Global Economics & Governance Date: October 22, 2018
Read article More on this topic More by this author

Blog Post

The 2018 Nobel Prize: Growth and the environment

The 2018 Nobel Prize in Economic Sciences has been awarded jointly to William Nordhaus and Paul Romer for integrating respectively climate change and technological innovation into long-run macroeconomic analysis. We review how economists reacted to the announcement.

By: Silvia Merler Topic: Energy & Climate Date: October 15, 2018
Load more posts