Assessing the effectiveness of networks is a notoriously under-developed field.  I recently took my hand at it with the co-leadership of Horacio Trujillo and iScale.  We applied our thinking to the International Land Coalition, a multi-stakeholder network aiming to promote secure and equitable access to, and control over, land through advocacy, dialogue, knowledge sharing and capacity building.  The major ILC participants are NGOs and intergovernmental organizations (IGOs) like the World Bank and the Food and Agriculture Organization (FAO).

We emphasized that our work was an “assessment” of the last strategic plan period (2007-2011), rather than an “evaluation”.  The latter term is associated with baselines and quantitative analysis.  (Usually evaluators draw from organization-based methodologies that are highly problematic and even undermine the work of multi-stakeholder change networks.) 

We didn’t have baseline data, and ILC wanted us to focus on questions about how it is doing as a whole network.  The product is being shared with funders as a report, and integrated into ILC’s new strategic plan. 

We did some web crawl mapping, interviewed a lot of people, looked at lots of documents, did a focus group in Nairobi and with the ILC Council in Rome, and developed four case studies – one case each was at the local, national, continental and global levels.

Gathering data is, of course, lots of work.  But the really hard part of doing an assessment is to make sure you are asking the right questions, using the right lens to look at the data, and summarizing everything in a comprehensible and comprehensive actionable way.  We used two frameworks to organize the data, raise questions, and create a synthesis. 

The GAN Framework

We used Global Action Networks (GANs) as a framework because we thought the ILC was a GAN, as did some people at ILC.  So the big GAN question was:  as a GAN, how is ILC doing?  We used two tools to answer this, presented in Tables 1 and 2.  For each we asked:  What has ILC achieved?  What does it need to do better?

For example, with “Shared Visioning” of Table 2, achievements of ILC included:

• Broad agreement about the major roles of ILC: visioning, system organizing, learning, advocacy (reassuring some who feared other participants viewed “financing” as an ILC role)
• ILC’s vision is now reflected in two major international documents recently adopted by other organizations. 

On the other hand, in terms of doing better the data indicated that:

• ILC should shift from a focus on “visioning” to “implementing”. 

The Theory of Change Framework

Theory

of

Change

Framework

The second framework was “theory of change” (ToC).  Horacio looked at ILC’s strategic objectives (SOs) and mission, and reframed them as a ToC represented in Diagram 1. This Diagram reflects that ILC’s mission around access to land is, in fact, a strategy to reduce hunger.  The preceding SOs explain how ILC says it is going to realize these outcomes. 

We then took the data from the GAN framework and applied it to the ToC to “rate” ILC’s performance using a “stoplight” approach where:

• Green represents those items that we assess as having been realized to a significant extent,
• Yellow represents those we assess as realized to a lesser, but still meaningful degree,
• Orange represents those we assess as realized to some, but less meaningful extent, and
• Red represents those objectives we assess as not having been realized or having been realized to an unsatisfactory degree.

From all of this, we then proposed a refinement of the ILC’s ToC, represented in Figure 2. In this refined version there are the following important changes:

1. SO1:  The dotted lines aim to emphasize that this is a utilitarian, organizational development objective of different status.
2. SO2:  This civil society-focused objective is greyed-out and integrated into the next SO, reflecting our suggestion to increase focus on developing the multi-stakeholder aspect of ILC and supporting more specifically CSOs’ (and others’) capacity to engage in multi-stakeholder processes.
3. SO3: We replace “members” with “participants”, reflecting the need to broaden the ILC’s stakeholder focus, but without getting into formal governance issues.
4. SO4:  We explicitly add “other stakeholders” to reflect our recommendation that the ILC broaden its stakeholder focus.
5. SO5:  We:

• replace the terms “coherent and coordinated” with “catalyzing of coherent” to better reflect our view that (1) an active verb is preferable, (2) a broader range of actions than coordination are necessary (for example, developing), (3) the core function of ILC is to create “coherence” amongst stakeholders.
• identify the path of influence that global commitments can have on catalyzing national commitments, by breaking these apart.

All of this is contained in two reports.  The first report contained the full GAN analysis and a couple of dozen recommendations.  The second report, for broader consumption, omitted the GAN analysis and focused on the two Figures and four Summary Recommendations that ILC should:

1. Focus more rigorously on strengthening the multi-stakeholder objective of its strategy.
2. Recognize in its priorities an opportunity to shift from creating frameworks to the implementation of such networks.
3. Be more disciplined and strategic about prioritizing, setting targets and allocating resources. 
4. Consider reframing its strategic objectives as a more concise theory of change.

For Horacio and me this project was, therefore, about producing a useful assessment for ILC and also about producing a methodology for conducting such assessments.  We believe that it would definitely be useful for other GANs, and adaptable to other networks as well to provide useful strategic direction.

Thanks to others on the iScale scaling for impact team who contributed to this work:  Randy Kemp, Sanjeev Khagram and Saira Abbasey!

ILC Director’s Comment

I asked International Land Coalition Director Madiodio Niasse if he’d like to comment on our assessment, and he shared the following:

The review of ILC’s 2007-2011 by iScale has generated results far beyond our expectations. We started by anticipating an ordinary evaluation in which projected results and actual achievements would be compared and analysed. We ended up engaging in a radical collective reflection on who we are, what brings together, what is our comparative advantage, and how to position ILC in the future. A constructive tension between continuity and innovation has emerged from the internal consultations. And I am sure that ILC is today more ready for change than ever before, although it will take us time to clearly articulate this in a language easily understandable by our members and partners. For these reasons I think the process facilitated by iScale is as important as its end product, the review report. Thank you.