Positive Outcomes: Discovering What Works in Development
Increasing Project Impact
I recently returned from attending the conference “Perspectives on Impact Evaluation: Approaches to Development Effectiveness.” As a researcher of development impact, I jumped at the chance to attend. I was greeted by workshops on how to design and implement impact evaluations for many types of projects including agricultural production, health, microenterprise and education. 600 professionals from around the globe attended this, the first conference focused on using impact evaluations to discover what works in development.
Outputs vs. Outcomes
Traditionally, project monitoring and evaluation (M&E) has served an accounting role focused on outputs. A traditional M&E plan would verify that 1,000 books were distributed to students in an educational project. However, by focusing on outputs, donors weren’t taking the opportunity to learn if their investments were having a positive outcome: Did the project improve the well-being of the students? New types of questions have begun coming to light:
-
Did the students remain in school longer as a result of the project?
-
Did the program better prepare them for the job market?
-
Were they healthier and more prosperous in the long-term?
-
Were the outcomes sustainable?
Why is this important for NGOs?
Impact evaluation provides feedback to help improve the design of programs. Even though NGOs have been reluctant to report on poorly performing interventions, donors have been showing interest in including both negative and positive outcomes in a more open dialogue with their partners. If a donor learns that two out of five project activities didn’t have impact – why should they reinvest in those activities? Why not exchange them for activities that have proven effectiveness? This open dialogue represents a positive step toward allocating time and resources toward enhancing the impact of future projects.
How do we initiate an impact evaluation (IE)?
-
Be sure to include the impact evaluation plan in the formative stages of project design.
-
Carefully develop the question (outcome) that the evaluation is being designed to answer.
-
Choose clearly identifiable and measurable indicators for the expected outcomes.
-
Conduct a baseline survey prior to beginning project activities.
-
Select two identical groups of community members:
-
a treatment group who will receive the benefits of the project.
-
a control group who won’t receive the benefits of the project.
-
The goal will be to compare the treatment group to the control group at different points in the project cycle. The good news is that impact evaluations planned into the project from the beginning need not cost significantly more than traditional M&E plans.
Methodologies
Quantitative studies evaluate numerical data. Qualitative studies can assess outcomes through interviews with constituents. Some studies are conducted remotely by scientists in distant cities, whereas participatory methodologies encourage on-site community-led evaluations. As you can imagine, members of different camps feel strongly about the efficacy of their methodology. Fortunately, an underlying theme at the conference was the use of mixed methods. For example, combining and comparing the results of a randomized control trial with the results of a community-led evaluation for the same project can offer a more holistic vantage of outcome effectiveness – and at the same time engage the community in an inclusive process.
Practical Examples of Using Information from Impact Evaluations
What practical, evidence-based results can we hope to derive from an impact evaluation? Let’s say that you are promoting a health initiative. In an attempt to reduce the rate of diarrhea in children, you are considering four commonly used interventions: water supply, water treatment, sanitation, and hygiene.
A survey of randomized control trials presented at the conference revealed that diarrhea can be reduced by 40% through water treatment and improved sanitation. Hand washing reduced diarrhea by 30% to 50%. However, the survey indicated that community water systems didn’t deliver substantial health benefits: water could become contaminated between the source and the home. Point of use water purification, in this context, showed evidence of offering the greater health benefit of the two water interventions.
This type of clear information can help programming staff decide on the most effective allocation of resources for a given outcome. Evidence-based results like these can guide this health project’s design to maximize impact by including a sanitation program, a point-of-use water purification system, and a hand-washing campaign.
The Growth of Impact Evaluations
When I first began watching development projects unfold, I became concerned about their effectiveness and sustainability. Seven years ago, I began searching for ways to evaluate projects that would reveal what was working, when it was working, and why. A colleague introduced me to randomized control trials. These studies opened a new window for me into development impact and gave me the data necessary to find out what was working in development. Discussions with field anthropologists gave me respect for qualitative interpretation of projects. I also began to observe that projects were rarely sustainable if the community didn’t have a voice in project design; this led to exploring participatory needs assessments. It wasn’t easy gathering together this information as the community of experienced IE practitioners was still small.
What Next?
The conference was excellent because it brought together the community of IE practitioners to compare notes and share ideas. But the next step in ramping up the number of impact evaluations is for donors and NGOs to begin including these methodologies in their projects. However, one conclusion of the conference was that more people need to be trained in impact evaluation.