A big challenge people are always asking me about is how to measure impact when many organizations and people are involved and when success depends on flexibly changing course in response to learning what works. They are characteristics of complex environments. Attribution is highly problematic with so many actors and measurement against baselines is often simply not useful when the learning actually leads to redefining goals and true innovation. The International Development Research Centre in Canada has developed Outcome Mapping; Michael Quinn Patton frames the appropriate response as “developmental evaluation” (more information). Danny Burns draws from his considerable experience as an Institute for Development Studies action researcher in such situations in a valuable paper: Burns- Assessing Impact in Dynamic and Complex Environments Systemic Action Research and Participatory Systemic Inquiry.
He starts by emphasizing a systems approach, and that in complex environments “changing system dynamics is critical to sustainable impact.” He notably writes about “assessment” rather than “measurement”: the latter concept is almost equated with quantitative measures that can have very different meaning to different stakeholders and on their own, can be largely meaningless. “Assessment” is much more about understanding context and learning, which are critical ingredient in complex environments.
Burns modifies the famous learning cycle and nicely addresses a question I’ve always had about the usefulness of approaching complex issues with theories of change when learning from action can make the theory inappropriate: he integrates generation of new theories of change into the learning cycle. Duh. Seems obvious now!
Another challenge working in complex environments is the non-linear nature of change that makes traditional planning process designed around moving from point A to point B inappropriate. He looks at four aspects of complexity and systems that have particular impact on assessment:
Change is emergent: Figure 2 illustrates the original belief that A leads to B, which leads to C to get to goal 1. When learning early on suggests that goal 2 is a more desirable goal and it’s best approached by going from A to K, traditional planning would break down. Emergence requires assessing against the decisions during the program, not against the original logic.
Unintended consequences: Changes in one part of a system often result in unforeseen ways within the system (such as a neighborhood) beyond its boundaries (such as in an abutting neighborhood). This means assessment has to ask questions about impact on the wider system.
System dynamics: Change is constrained by system dynamics and sustainable change requires change in system dynamics. This means assessment must focus on how the system dynamic has changed, not on what changes have taken place within the system dynamic.
Tipping point: Latent change often leads to tipping points characterized by sudden major transformation. Therefore assessment is required of the underlying change as well as the surface level of change. This might involve, for example, noting how discourse has changed. Dave Snowden’s Sensemaker would be great for this.
Iterative processes are particularly critical to support complex environment assessment with respect to:
Of course all this emphasizes undertaking interventions as learning processes. Attentiveness to what is changing and new potentials must be supported through discussion, documentation and routines.