Village-based schools

We conduct a randomized evaluation of the effect of village-based schools on children's academic performance using a sample of 31 villages and 1,490 children in rural northwestern Afghanistan. The program significantly increases enrollment and test scores among all children, eliminates the 21 percentage point gender disparity in enrollment, and dramatically reduces the disparity in test scores. The intervention increases formal school enrollment by 42 percentage points among all children and increases test scores by 0.51 standard deviations (1.2 standard deviations for children that enroll in school). While all students benefit, the effects accrue disproportionately to girls. Evidence suggests that the village-based schools provide a comparable education to traditional schools.

That is from the new paper "The Effect of Village-Based Schools: Evidence from a Randomized Controlled Trial in Afghanistan" by Burde and Linden (April 2012).

Why is this important? Lets say you are Minister of Education in a developing country, or governor of a province. Of course you face a budget contraint. Where should you set up the next 10 new schools?

Product innovation still needed in Microfinance

. . . [P]oor people face various limits, and their ability to capitalize on opportunities varies greatly. One of the next steps is to find simple ways to identify those differences and cater to them with the right products delivered with the right design.

Details matter. Individual characteristics of clients matter. And design matters. 

That is from a new literature survey: "Latest Findings from Randomized Evaluations of Microfinance" by Bauchet et al. It includes 10 studies around the world. See "Mostly Economics" post. HT: Mostly Economics. 

Acemoglu criticizes Randomized Control Trials (RCTs)

RCTs (or Randomized Control Trials) are probably one of the most important methods in economic development nowadays. They are used to evaluate the impact of social programs (but they can also be used to evaluate NGOs' programs or even new projects by private companies). For example, how do we know that giving a subsidy to poor families to motivate them to send their kids to school actually has a positive impact on education. Or how do we know that a government program that gives food to the poor actually is accomplishing the final goals (improved nutrition, for example). The RCTs method has been used for example to evaluate the impact of social programs like Progresa in Mexico. 

In this paper (2010) Daron Acemoglu probably makes the most serious criticism of RCTs (or counterfactual analysis, as he calls it). 

Abstract: I discuss the role of economic theory in empirical work in development economics with special emphasis on general equilibrium and political economy considerations. I argue that economic theory plays (should play) a central role in formulating models, estimates of which can be used for counterfactual and policy analysis. I discuss why counterfactual analysis based on microdata that ignores general equilibrium and political economy issues may lead to misleading conclusions. I illustrate the main arguments using examples from recent work in development economics and political economy.

An example: 

As an example of composition effects, consider the problem of estimating the importance of credit market imperfections. Banerjee and Duáo (2005) survey a large body of evidence that small and medium-sized businesses in less-developed economies are credit constrained and an extension of credit to these businesses will make them increase production. Now consider the effect of a large-scale policy of credit expansion to small- and medium-sized businesses. This policy could lead to a different type of composition effect than the one operating in partial equilibrium. For example, it may be the case that in partial equilibrium estimation focusing on firm-level variation we found that firms with better access to credit expanded, but this was at the expense of other firms that did not have access to credit (that is, partly by ìstealing businessî from others). And yet, the same response cannot take place in general equilibrium. As a consequence, when additional credit becomes available to a large fraction of firms, total output may not increase by as much or at all. One could thus imagine a situation in which partial equilibrium estimates of relaxing credit constraints are large, while the general equilibrium e§ects would be small.

This is a very constructive criticism and a call to include into RCTs well specified theory, general equilibrium considerations, and political economy factors. 


A well specified theory prior to test an econometric model is very important according to Acemoglu because if a theoretical model is not well specified prior to analyzing the data, the results of the regression analysis can be misleading. For example, we can conclude that giving a subsidy is a good thing, but actually we did not take into account that we will not have the physical infrastructure to accommodate more students. This is an application of the Lucas Critique in the context of partial equilibriums as Acemoglu calls the changes introduced by policies motivated by RCTs. 


I wonder however if Randomized Control Trials as they work right now can be a mechanism for theory building or theory enhancing, because they can give feedback on variables or interactions that current theoretical models do not include. This might be indeed a very valuable function of RCTs, helping improve current theories of economic development and growth. This paper gives a pretty good description of what theory building is from the perspective of political sciences. 


I learned about Acemoglu's paper thanks to Peter Boettke in Facebook. 

Should aid reward performance?

Chris Blattman links to a list of new and interesting papers on field experiments on governance and politics. 

Among the papers it is this one: "Should Aid Reward Performance? Evidence from a field experiment on health and education in Indonesia." 

Abstract: This paper reports an experiment in over 3,000 Indonesian villages designed to test the role of performance incentives in improving the efficacy of aid programs. Villages in a randomly-chosen one-third of subdistricts received a block grant to improve 12 maternal and child health and education indicators, with the size of the subsequent year’s block grant depending on performance relative to other villages in the subdistrict. Villages in remaining subdistricts were randomly assigned to either an otherwise identical block grant program with no financial link to performance, or to a pure control group. We find that the incentivized villages performed better on health than the non-incentivized villages, particularly in less developed provinces, but found no impact of incentives on education . . . 

The authors conclude:

In sum, the evidence presented here suggests that properly designed, performance based incentives can be a useful addition to aid programs. We found that adding performance incentives increased health outcomes, particularly in poorer areas with worse performance at baseline. Though the gains from incentives were modest, we found little downside from the incentives.

I have not read the paper entirely but is intriguing that performance incentives worked for health but did not for education outcomes. 

Context and incentives

[E]ven for programs designed on seemingly universal principles like incentives, success and failure are situational. All the more reason to test them . . . Incentives only work if the monitoring tool for administering them is corruption-proof. Source: "More than Good Intentions." 

Impact Evaluation (RCT)

Here is one suggestion for strengthening incentives for using knowledge in the practice of development: as a matter of principle, no project or policy proposal should be funded that does not have a recent Ex-ante Survey of Knowledge on Impact (ESKI)—an objective and thorough assessment of prevailing knowledge with bearing on the case for and against the intervention in its setting, based on past research, impact evaluations and experience.  This paper can be thought of as the principle input to the ex ante appraisal—explaining the rationale for the intervention, based on what we know, and identifying plausible ranges for the key parameters relevant to a successful project. It should draw on theory and evidence to understand the development problem or obstacle that the intervention targets. What is the market or governmental failure it addresses and how do we know that doing so will make things better overall? What are the implications for equity? The best World Bank lending operations already include something like an ESKI. The challenge is to raise standards and bring the idea to all operations, and not just the Bank’s.

Source HT: Chris Blattman