Integrating Ethnography into Impact Evaluation

. . . [T]his paper argues that ethnographic evaluation can make an essential contribution to impact evaluation by providing explanations as to why interventions have not been successful, as well as an evidence base for improving programme design and implementation generated by interaction with community members. 

That is from the paper "Integrating Ethnographic Principles In Ngo Monitoring And Impact Evaluation" by Bell and Aggleton (Journal of International Development, 2012). The paper is not available online for free. I thank the authors for sending me a copy. 

The authors conclude:

[Ethnographic evaluation] makes an essential contribution by providing robust explanations as to why interventions have not been very successful, as well as an evidence base for improving programme design and implementation. Key areas in which programme improvement cannot occur as a result of this deeper insight include a better understanding of (i) underlying principles and assumptions of programme approaches, (ii) the socio-cultural logics influencing young people’s decision-making, (iii) unexpected sexual health outcomes and (iv) local barriers to programme delivery and efficacy.

Of course, the key is to see this approach as a complement of experimental and cuasi-experimental ones (and the authors say that). 

Impact evaluation in practice - core elements

This is a list of core elements according to the book Impact Evaluation in Practice (The World Bank, 2011), which seems like a pretty good guide for people conducting impact evaluation in the field.  

✓ A concrete policy question—grounded in a theory of change—that can be answered with an impact evaluation

✓ A valid identification strategy, consistent with the operational rules of the program, that shows the causal relation between the program and outcomes of interest

✓ Awell-powered sample that allows policy-relevant impacts to be detected and a representative sample that allows results to be generalized to a larger population of interest

✓ A high-quality source of data that provides the appropriate variables required by the analysis, of both treatment and comparison groups, using both baseline and follow-up data

✓ A well-formed evaluation team that works closely with policy makers and program staff

✓ An impact report and associated policy briefs, disseminated to key audiences in a timely manner and feeding both program design and policy dialogues

Impact Evaluation (RCT)

Here is one suggestion for strengthening incentives for using knowledge in the practice of development: as a matter of principle, no project or policy proposal should be funded that does not have a recent Ex-ante Survey of Knowledge on Impact (ESKI)—an objective and thorough assessment of prevailing knowledge with bearing on the case for and against the intervention in its setting, based on past research, impact evaluations and experience.  This paper can be thought of as the principle input to the ex ante appraisal—explaining the rationale for the intervention, based on what we know, and identifying plausible ranges for the key parameters relevant to a successful project. It should draw on theory and evidence to understand the development problem or obstacle that the intervention targets. What is the market or governmental failure it addresses and how do we know that doing so will make things better overall? What are the implications for equity? The best World Bank lending operations already include something like an ESKI. The challenge is to raise standards and bring the idea to all operations, and not just the Bank’s.

Source HT: Chris Blattman