Blog Post

Blogs review: The popularity of Randomized Control Trials

What’s at stake: The popularity of Randomized Control Trials (RCTs) in academia has led to an impressive increase in the amounts governments and international institutions spend on providing evidence from RCTs. While valued for their research design, they remain criticized for having little predictive value beyond the context of the original experiment and the difficulties they face in evaluating complex interventions.

By: , and Date: January 14, 2014 Topic: Global Economics & Governance

The coming of age of RCTs

Joshua Angrist and Jörn-Steffen Pischke write that empirical microeconomics has experienced a credibility revolution, with a consequent increase in policy relevance and scientific impact. The primary engine driving improvement has been a focus on the quality of empirical research designs. The advantages of a good research design are perhaps most easily apparent in research using random assignment.

The Economist writes that RCTs are like drug trials for economics. If you want to discover if, say, using identity cards would improve the delivery of subsidized rice to the poor and reduce theft, then you take a collection of comparable villages or households, and randomly assign ID cards to some and not to others. Then wait to see what occurs. Anders Olofsgård writes that the recent focus on impact evaluation within development economics has led to increased pressure on aid agencies to provide evidence from RCTs. Free exchange reports that the World Bank runs RCTs. So do regional bodies like the Inter-American Development Bank. Even governments deploy them.

Good estimates from the wrong place versus bad estimates from the right place

Joshua Angrist and Jörn-Steffen Pischke write that the rise of the experimentalist paradigm has provoked a reaction about the question of external validity – the concern that evidence from a given experimental or quasi-experimental research design has little predictive value beyond the context of the original experiment. The criticism here – made by a number of authors including Heckman (1997); Rosenzweig and Wolpin (2000); Heckman and Urzua (2009); and Deaton (2009) – is that in the quest for internal validity, design-based studies have become narrow or idiosyncratic.

Lant Pritchett and Justin Sandefur write that external validity is such a big issue that OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Berk Özler explains that the variation across contexts – that Pritchett and Sandefur refer to – is more or less defined as variation in well-identified effect sizes (ES), while variation within is the difference between a not well-identified estimate and the well-identified one within the same context. In other words, if you ran an OLS of vouchers on student performance using the baseline data in your experiment that provided vouchers and compared the biased ES with the experimental one, that would be your within variation. If you compared the experimental ES in Colombia with one from Tanzania, that would be your across variation.

Abhijit V. Banerjee and Esther Duflo write that the trade-off between internal and external validity is present as well in observational studies. Joshua Angrist and Jörn-Steffen Pischke write that empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design.

The role of interactions among components

Martin Ravallion writes that assessing a set of development policies and projects by evaluating its components one-by-one and adding up the results requires some assumptions that are pretty hard to accept. For one thing, it assumes that there are negligible interaction effects amongst the components. For another thing, it assumes that we can either evaluate everything in the portfolio or we can draw a representative sample, such that there is no selection bias in which components we choose to assess.

A bulletin of the World Health Organization explains that challenges in global health lie not in the identification of efficacious interventions, but rather in their effective scale-up. This requires a nuanced understanding of how implementation varies in different contexts. Context can have greater influence on uptake of an intervention than any pre-specified implementation strategy. Ann Oakley and al. write that most RCTs focus on outcomes, not on the processes involved in implementing an intervention and argue that that including a process evaluation would improve the science of many RCTs. Because of their multifaceted nature and dependence on social context, complex interventions pose methodological challenges, and require adaptations to the standard design of such trials.

Have RCTs diverted an important amount of resources?

Martin Ravallion writes that our evaluative efforts are progressively switching toward things for which certain methods are feasible, whether or not those things are important sources of the knowledge gaps relevant to assessing development impact at the portfolio level. There is a positive “output effect” of the current enthusiasm for impact evaluation, but there is also likely to be a negative “substitution effect.” As Jishnu Das, Shantayana Devarajan and Jeffrey Hammer have pointed out it is not clear that RCTs are well suited to programs involving the types of commodities for which market failures are a concern, such as those with large spillover effects in production or consumption.

Joshua Angrist and Jörn-Steffen Pischke write that related to the external validity critique is the claim that the experimentalist paradigm leads researchers to look for good experiments, regardless of whether the questions they address are important. Given, for example, that the recent economic crisis has spawned intriguing design-based studies of the crisis macroeconomics (see Gabriel Chodorow-Reich’s 2013 job market paper for example), this claim seems less relevant today than when Noam Scheider complained about an evolution epitomized by the success of Freakonomics.

 

 

 


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article More on this topic More by this author

Blog Post

The debate on euro-area reform

A paper jointly written by 14 French and German economists set off a debate about the reform of euro-area macroeconomic governance. We review economists’ opinions about it.

By: Silvia Merler Topic: Finance & Financial Regulation Date: April 16, 2018
Read article More on this topic More by this author

Blog Post

Latvia’s money laundering scandal

Latvia’s third largest bank ABLV sought emergency liquidity from the ECB and eventually voted to start a process of voluntary liquidation, after being accused by US authorities of large-scale money laundering and having failed to produce a survival plan. What does it mean for the ECB?

By: Silvia Merler Topic: Finance & Financial Regulation Date: April 9, 2018
Read article More on this topic More by this author

Blog Post

Milton Friedman's " The role of monetary policy" - 50 years later

In March 1968, Milton Friedman’s “The Role of Monetary Policy” - after his famous presidential address to the American Economic Association - was published in the American Economic Review. 50 years later, economists reflect on this famous work.

By: Silvia Merler Topic: Global Economics & Governance Date: April 3, 2018
Read article More on this topic More by this author

Blog Post

The Brexit Transition Deal

Michel Barnier, the European Union’s Brexit negotiator, and David Davis, Britain’s Brexit secretary, announced a transition deal on March 19. We review recently published opinions about the deal and its implications.

By: Silvia Merler Topic: European Macroeconomics & Governance Date: March 26, 2018
Read article More on this topic More by this author

Blog Post

Central banks in the age of populism

Two years of elections have shown that we live in an age of increasing political and economic populism. What are the consequences of that for central banks? We explore opinions about it, from both 2017 and more recently.

By: Silvia Merler Topic: Finance & Financial Regulation Date: March 19, 2018
Read article More on this topic More by this author

Blog Post

Are we steel friends?

The U.S. administration is considering to impose tariffs on steel (25%) and aluminium (10%), based on a national security argument. We review economists’ views about this major shift in U.S.’ trade policy.

By: Silvia Merler Topic: Global Economics & Governance Date: March 12, 2018
Read article More on this topic More by this author

Blog Post

Getting accustomed to Brexit - UK and the customs union scenario

The Labour Party’s support of customs union membership has the potential to change the course of Brexit, with 13 months left to close negotiations. This week we review the commentary around the possibility of a post-Brexit EU-UK Customs Union.

By: Inês Goncalves Raposo Topic: European Macroeconomics & Governance Date: March 5, 2018
Read article More on this topic More by this author

Blog Post

The Italian elections

Italy goes to the polls on March 4, with a new electoral law that is largely viewed as unable to deliver a stable government. We review recent opinions and expectations,  as well as economists’ assessment of the cost/coverage of parties’ economic promises. 

By: Silvia Merler Topic: European Macroeconomics & Governance Date: February 26, 2018
Read article More by this author

Blog Post

Venezuela’s hyperinflation

The International Monetary Fund forecasts Venezuelan inflation spiralling to 13,000 percent this year. As President Maduro is expected to introduce the “petro” cryptocurrency next week, we review economists’ recent (and less recent) opinions on the current crisis.

By: Silvia Merler Topic: Global Economics & Governance Date: February 19, 2018
Read article More on this topic More by this author

Blog Post

The stock market slide

The stock market dropped last week, leading to questions and debates as to the underlying reasons. We review economists’ views on the issue.

By: Silvia Merler Topic: Finance & Financial Regulation Date: February 12, 2018
Read article More on this topic More by this author

Blog Post

Economies of States, Economies of Cities

Both in Europe and the US, economists are starting to notice how the economies of cities have been sometimes diverging from the economies of states. While some areas thrive, others may be permanently left behind. Maybe it is time to adopt a more clearly sub-national perspective. We review recent contributions on this issue.

By: Silvia Merler Topic: Global Economics & Governance Date: February 5, 2018
Read article More on this topic More by this author

Blog Post

Rebuilding macroeconomics: Initial reflections on a major theory project

The ‘Rebuilding Macroeconomic Theory Project’ came to an end in the most recent volume of the Oxford Review of Economic Policy; how were the various papers’ conclusions received?

By: Konstantinos Efstathiou Topic: Global Economics & Governance Date: January 29, 2018
Load more posts