General

The Power of Outcome Harvesting: A Paradigm Shift in Monitoring & Evaluation?

What will you learn from this article?

  1. Outcome harvesting is an evaluation approach that documents the outcomes of a program strategy by embracing complexity.
  2. The core principles of outcome harvesting include a contribution analysis, continuous iteration, engagement with key actors, and quantitative data collection.
  3. These approaches are well suited to reveal unexpected, more nuanced outcomes, particularly in complex environments.

In the realm of project management and evaluation, the quest for effective methodologies to capture and assess outcomes is ever-present. Among various approaches, outcome harvesting has emerged as a dynamic and innovative method that places emphasis on learning and adaptation. Our recent work with Agenda for Change allowed us to explore how outcome harvesting can serve as the core foundation for a results framework.

What is outcome harvesting?

Outcome harvesting is an evaluation approach that focuses on identifying and documenting the outcomes of a program strategy or strategic intervention without necessarily knowing what these are in advance. It embraces complexity, encourages actor engagement, and aims to capture unintended and unexpected changes that have occurred. Outcome harvesting provides a flexible, realistic approach to measuring progress.

Figure 1: Outcome harvesting as an evaluation approach

The core principles of outcome harvesting

  1. Contribution Analysis: Outcome harvesting prioritizes understanding the contribution of the project to observed outcomes. It acknowledges that multiple factors influence outcomes and seeks to uncover the initiative’s role within the broader context.
  2. Iterative and Adaptive: Outcome harvesting acknowledges that outcomes can change and evolve over time, which allows for ongoing learning and adaptation.
  3. Engagement: This approach actively involves stakeholders throughout the process, recognizing their unique perspectives and knowledge. Stakeholders play a vital role in identifying outcomes and planning responses, resulting in a more inclusive and comprehensive evaluation.
  4. Qualitative Data Complements Quantitative Data : Unlike traditional evaluation methods that heavily rely on quantitative data, outcome harvesting emphasizes the collection of qualitative data. This includes narratives and stories that provide rich insights into the outcomes and the project’s contribution. Meanwhile, as shown in Figure 1, qualitative stories are complemented with quantitative data (i.e. parameters) in order to affect and evaluate progress on the basis of an organization’s Theory of Change. This process, in turn, enriches the narrative with quantitative information.

Outcome harvesting in practice

An example use case of outcome harvesting could potentially be to capture stories about systems strengthening. In this case, systems strengthening could be measured through stories being classified against building blocks through tags like #financing, #servicedelivery. The stories could be further contextualized with quantitative indicators, such as JMP service level information. In this way, outcome harvesting could be used to holistically examine how an organization’s work is impacting both systems and services.

Benefits of outcome harvesting

Outcome harvesting is particularly well-suited and useful in operations in complex environments with many actors. The approach enables a more nuanced understanding of outcomes which may not be apparent in traditional approaches. Furthermore, by adopting an iterative and adaptive approach, outcome harvesting promotes continuous learning and improvement. One of the key benefits of outcome harvesting is that it is excels at revealing unexpected outcomes, which may include both positive and negative effects. By actively involving stakeholders throughout the process, outcome harvesting fosters ownership and enhances the credibility and usefulness of evaluation results. It builds trust and encourages collaboration among diverse actors.

Figure 2: How are stories are linked to organizational vision and priority areas?

Outcome harvesting offers a new perspective on results frameworks, shifting the focus from rigid indicators to a more adaptive and learning-oriented approach. By embracing complexity, engaging stakeholders, and capturing unexpected outcomes, this methodology unlocks insights that traditional evaluation methods may overlook.

We’re curious to know, have you used outcome harvesting in any of your projects? If so, how? What were the key learnings, challenges, surprises?

Tracking global progress on water, sanitation, and hygiene in ‘One for All’ Focus Countries – a collaboration between WASHNote, IRCWash and Water for People

One for All Global Landscape

IRC and Water For People have recently joined forces as ‘One for All‘. IRC and Water for People have jointly developed a strategy for their work in water, sanitation, and hygiene systems change at local, national, and global levels. The Results Framework contains metrics for each level of work; globally, they seek to facilitate transformational change in how services are provided around the world.

Together with IRC and Water for People, we recently completed a report on the status of 24 global landscape indicators – a set of indicators looking at a range of themes, including the population with access to at least basic services, the spread of access between wealth quintiles, financial commitments and disbursements, political will, and institutional capacity, amongst others. The report aims to highlight the progress and current status of One for All Focus Countries, which include: Honduras, Bolivia, Peru, Guatemala, Tanzania, Rwanda, Malawi, Uganda, Niger, Mali, Burkina Faso, Ethiopia, Ghana, India and Bangladesh. Nonetheless, these are compared to regional and global averages as well. All analyses will be repeated in future years in order to document progress over time.

Looking at secondary data sources: JMP, OECD and others

We used secondary data sources like GLAAS, the UNICEF/WHO Joint Monitoring Program (JMP), OECD, and Sanitation and Water for All (SWA). Further building on the work of the jmpwashdata data science library, which is now available for use, we were able to creatively visualize the number and proportion of people which have access to WASH services. This was done not only for the most recent data (2020), but also projected into the future based on the Annual Rate of Change (ARC) to be able to present a possible scenario for 2030. See the figure below:

We also further disaggregated these findings depending on wealth quintiles, boding very interesting (albeit shocking) results. For example, we projected that in Ethiopia approximately 33 million people in the lowest two wealth quintiles will still lack access to basic drinking water. This equates to over half (~56%) of the population in these wealth quintiles, but less than a quarter of the total population (~23%), indicating severe disparities in equity which should be adequately addressed. However, it is important to note that the data quality for JMP’s inequality spreadsheets is not as strong as the global dataset. The figure below shows the spread of data quality for the countries we assessed, based on the JMP’s country inequalities files.

Similarly, OECD data on Aid Commitments and Disbursements for WASH was downloaded and is now available as an R package and CSV file. We found that both commitments and disbursements seem to fluctuate over time, but that there is no clear trend.

Based on these analyses, some next steps include developing clear targets for each indicator, and also supporting focus countries in reaching these goals.

References

Dickinson, N. 2021. jmpwashdata: WHO/UNICEF Joint Monitoring Programme Water and Sanitation Data. R package version 0.1.4.

GLAAS. 2021. UN-Water Global Analysis and Assessment of Sanitation and Drinking-Water Data Portal. https://glaas.who.int/glaas/data

OECD-CRS. 2023. OECD Creditor Reporting System.
https://stats.oecd.org/index.aspx?DataSetCode=CRS1

WHO/UNICEF. 2023. WASH Data. https://washdata.org/data/downloads#WLD

WASHNote