Acemoglu criticizes Randomized Control Trials (RCTs)

RCTs (or Randomized Control Trials) are probably one of the most important methods in economic development nowadays. They are used to evaluate the impact of social programs (but they can also be used to evaluate NGOs' programs or even new projects by private companies). For example, how do we know that giving a subsidy to poor families to motivate them to send their kids to school actually has a positive impact on education. Or how do we know that a government program that gives food to the poor actually is accomplishing the final goals (improved nutrition, for example). The RCTs method has been used for example to evaluate the impact of social programs like Progresa in Mexico. 

In this paper (2010) Daron Acemoglu probably makes the most serious criticism of RCTs (or counterfactual analysis, as he calls it). 

Abstract: I discuss the role of economic theory in empirical work in development economics with special emphasis on general equilibrium and political economy considerations. I argue that economic theory plays (should play) a central role in formulating models, estimates of which can be used for counterfactual and policy analysis. I discuss why counterfactual analysis based on microdata that ignores general equilibrium and political economy issues may lead to misleading conclusions. I illustrate the main arguments using examples from recent work in development economics and political economy.

An example: 

As an example of composition effects, consider the problem of estimating the importance of credit market imperfections. Banerjee and Duáo (2005) survey a large body of evidence that small and medium-sized businesses in less-developed economies are credit constrained and an extension of credit to these businesses will make them increase production. Now consider the effect of a large-scale policy of credit expansion to small- and medium-sized businesses. This policy could lead to a different type of composition effect than the one operating in partial equilibrium. For example, it may be the case that in partial equilibrium estimation focusing on firm-level variation we found that firms with better access to credit expanded, but this was at the expense of other firms that did not have access to credit (that is, partly by ìstealing businessî from others). And yet, the same response cannot take place in general equilibrium. As a consequence, when additional credit becomes available to a large fraction of firms, total output may not increase by as much or at all. One could thus imagine a situation in which partial equilibrium estimates of relaxing credit constraints are large, while the general equilibrium e§ects would be small.

This is a very constructive criticism and a call to include into RCTs well specified theory, general equilibrium considerations, and political economy factors. 


A well specified theory prior to test an econometric model is very important according to Acemoglu because if a theoretical model is not well specified prior to analyzing the data, the results of the regression analysis can be misleading. For example, we can conclude that giving a subsidy is a good thing, but actually we did not take into account that we will not have the physical infrastructure to accommodate more students. This is an application of the Lucas Critique in the context of partial equilibriums as Acemoglu calls the changes introduced by policies motivated by RCTs. 


I wonder however if Randomized Control Trials as they work right now can be a mechanism for theory building or theory enhancing, because they can give feedback on variables or interactions that current theoretical models do not include. This might be indeed a very valuable function of RCTs, helping improve current theories of economic development and growth. This paper gives a pretty good description of what theory building is from the perspective of political sciences. 


I learned about Acemoglu's paper thanks to Peter Boettke in Facebook.