Posted by Steve Waddell in M&E on May 5, 2011
Assessing the effectiveness of networks is a notoriously under-developed field. I recently took my hand at it with the co-leadership of Horacio Trujillo and iScale. We applied our thinking to the International Land Coalition, a multi-stakeholder network aiming to promote secure and equitable access to, and control over, land through advocacy, dialogue, knowledge sharing and capacity building. The major ILC participants are NGOs and intergovernmental organizations (IGOs) like the World Bank and the Food and Agriculture Organization (FAO).
We emphasized that our work was an “assessment” of the last strategic plan period (2007-2011), rather than an “evaluation”. The latter term is associated with baselines and quantitative analysis. (Usually evaluators draw from organization-based methodologies that are highly problematic and even undermine the work of multi-stakeholder change networks.)
We didn’t have baseline data, and ILC wanted us to focus on questions about how it is doing as a whole network. The product is being shared with funders as a report, and integrated into ILC’s new strategic plan.
We did some web crawl mapping, interviewed a lot of people, looked at lots of documents, did a focus group in Nairobi and with the ILC Council in Rome, and developed four case studies – one case each was at the local, national, continental and global levels.
Gathering data is, of course, lots of work. But the really hard part of doing an assessment is to make sure you are asking the right questions, using the right lens to look at the data, and summarizing everything in a comprehensible and comprehensive actionable way. We used two frameworks to organize the data, raise questions, and create a synthesis.
We used Global Action Networks (GANs) as a framework because we thought the ILC was a GAN, as did some people at ILC. So the big GAN question was: as a GAN, how is ILC doing? We used two tools to answer this, presented in Tables 1 and 2. For each we asked: What has ILC achieved? What does it need to do better?
For example, with “Shared Visioning” of Table 2, achievements of ILC included:
On the other hand, in terms of doing better the data indicated that:
The second framework was “theory of change” (ToC). Horacio looked at ILC’s strategic objectives (SOs) and mission, and reframed them as a ToC represented in Diagram 1. This Diagram reflects that ILC’s mission around access to land is, in fact, a strategy to reduce hunger. The preceding SOs explain how ILC says it is going to realize these outcomes.
We then took the data from the GAN framework and applied it to the ToC to “rate” ILC’s performance using a “stoplight” approach where:
From all of this, we then proposed a refinement of the ILC’s ToC, represented in Figure 2. In this refined version there are the following important changes:
- replace the terms “coherent and coordinated” with “catalyzing of coherent” to better reflect our view that (1) an active verb is preferable, (2) a broader range of actions than coordination are necessary (for example, developing), (3) the core function of ILC is to create “coherence” amongst stakeholders.
- identify the path of influence that global commitments can have on catalyzing national commitments, by breaking these apart.
All of this is contained in two reports. The first report contained the full GAN analysis and a couple of dozen recommendations. The second report, for broader consumption, omitted the GAN analysis and focused on the two Figures and four Summary Recommendations that ILC should:
For Horacio and me this project was, therefore, about producing a useful assessment for ILC and also about producing a methodology for conducting such assessments. We believe that it would definitely be useful for other GANs, and adaptable to other networks as well to provide useful strategic direction.
Thanks to others on the iScale scaling for impact team who contributed to this work: Randy Kemp, Sanjeev Khagram and Saira Abbasey!
I asked International Land Coalition Director Madiodio Niasse if he’d like to comment on our assessment, and he shared the following:
The review of ILC’s 2007-2011 by iScale has generated results far beyond our expectations. We started by anticipating an ordinary evaluation in which projected results and actual achievements would be compared and analysed. We ended up engaging in a radical collective reflection on who we are, what brings together, what is our comparative advantage, and how to position ILC in the future. A constructive tension between continuity and innovation has emerged from the internal consultations. And I am sure that ILC is today more ready for change than ever before, although it will take us time to clearly articulate this in a language easily understandable by our members and partners. For these reasons I think the process facilitated by iScale is as important as its end product, the review report. Thank you.