Evidence-Based Policy

I reached the American Journal of Evaluation via Pablo Rodriguez. In the December 2011 there is an interesting and relevant article titled "Known Knowns, Known Unknowns, Unknown Unknowns: The Predicament of Evidence-Based Policy" by Pawson, Wong, and Owen, an early draft is here. The paper discusses the complexities of evidence-based policy - it is not as straight forward as I had thought. 

The abstract of the paper:

The authors present a case study examining the potential for policies to be “evidence-based.” To what extent is it possible to say that a decision to implement a complex social intervention is warranted on the basis of available empirical data? The case chosen is whether there is sufficient evidence to justify banning smoking in cars carrying children. The numerous assumptions underpinning such legislation are elicited, the weight and validity of evidence for each is appraised, and a mixed picture emerges. Certain propositions seem well supported; others are not yet proven and possibly unknowable. The authors argue that this is the standard predicament of evidence-based policy. Evidence does not come in finite chunks offering certainty and security to policy decisions. Rather, evidence-based policy is an accumulative process in which the data pursue but never quite capture unfolding policy problems. The whole point is the steady conversion of “unknowns” to “knowns.”

 According to the authors programs are complex because:

• Programmes are active, not passive. Interventions do not work in and of themselves; they only have affect through the reasoning and reactions of their recipients.
• Programmes have long implementation chains and multiple stakeholders. Recipients are many and varied; reactions to programmes thus differ; outcomes are thus generally mixed.
• Programmes are embedded in complex social systems. Recipients are rooted in different localities, institutions, cultures, histories, all of which shape the fortunes of a programme.
• Programmes are implemented amidst the turbulence of other interventions. The policy agenda is delivered through a multitude of interventions, each one interfering with the reception of another.
• Programmes beg, steal, borrow and adapt. Practitioners work constantly to improve the delivery of interventions rather than preserving uniformity to meet evaluation and trial requirements.
• Programmes are the offspring of previous interventions. Social problems are longstanding; interventions evolve to try to combat them; the success of a current scheme depends on its history.
• Programmes change the conditions that make them work in the first place. An intervention’s success is always time limited since alleviating a problem always involves changing its concomitant causes.

Their conclusions:

There is no such thing as a typical policy intervention or a quintessential programme. Legislative instruments of the type considered here are worlds away from interventions based on, say, financial incentives or peer learning. But what they all have in common is a sprawl of ambitions, stakeholders, localities and histories. In this respect, evidence-based policy has to deal with a standard predicament. Research synthesis can only provide partial information on the medley of issues that face the decision maker. That information, as here, is likely to draw upon inquiry conducted in diverse research traditions – toxicology, social psychology, political science, socio-legal studies, etc. Accordingly, that information is also likely to be partial in research quality and political leanings. So what does this tell us about the warrant of evidence?

. . . 

Evidence does not come in finite chunks offering certainty and security to policy decisions. Programmes and interventions spring into life as ideas about how to change the world for the better. Evaluation research allows us to refine those explanations and systematic review allows a refinement of those refinements. The review process should be understood as a means of building, adjudicating and extrapolating programme theories. Evidence based policy will only mature when it is understood that it is a continuous, accumulative process in which the data pursues but never quite draws level with unfolding policy problems. The whole point is the steady conversion of ‘unknowns’ to ‘knows’. 

It may well be that policy makers are too busy, too stubborn or too stupid to digest the significance of the qualifications and contingencies that govern whether a policy will work (this subject is much studied but not pursued in this paper). Nevertheless we arrive at the inescapable, long-and-short of it – evidence-based policy deals in conditional truths and provisional explanations.