4. Pathway to Impact: Supporting and Evaluating Enabling Environments for Research for Development

CGIAR is a global agricultural research partnership for a food secure future. Its science is carried out by 15 research centers with 10,000 scientists working in 96 countries and a host of partners in national and regional research institutes, civil society organizations, academia, development organizations, and the private sector (CGIAR 2015a). Its work contributes to the global effort to tackle poverty, hunger and major nutritional imbalances, and environmental degradation. The 15 CGIAR Centers have different foci and operate semi-autonomously in pursuing their specific research agendas, ranging from promoting the productivity of specific crops, livestock, and fish commodities to production systems in specific agro-ecologies and research on policies natural resource management (Raitzer and Kelly 2008).

CGIAR was formed in 1971 to foster technical solutions to agricultural productivity constraints affecting developing countries (Renkow and Byerlee 2010). Research tended to focus on creating outputs, was often technology focused and supply driven; success was measured by peer-reviewed publications, citations and science products. Criticism has been mounting over the last decades, as the limitations of the output delivery model became evident: outputs do not automatically translate into impact. It was often assumed that communication and development specialists would repackage research findings after the researcher produced them and that farmers would realize the value of new technologies and happily adopt them to increase agricultural productivity (Fig. 4.1). CGIAR itself has long recognized these weaknesses and embarked on a far-reaching reform process in 2010.

The challenges of demonstrating wide-reaching impact through R4D are compounded by a rapidly growing human population, climate change and other complexities of our time. The human population has almost doubled from 3.8 billion in 1971 to 7.3 billion in 2014 (UN 2015b). With an expected extra two to three billion people to feed over the next 40 years, this will require targeted research efforts to achieve not just growing 70 % more food but making 70 % more food available on the plate to keep up with rapidly rising demand (WWAP 2012). Climate change is already affecting agriculture in many developing countries, and the effects will become increasingly challenging in the future. Higher temperatures, shifting disease and pest pressures, and more frequent and severe droughts and flooding will affect agricultural production and place increasing pressure on water and other natural resources (IPCC 2013). Climate change impacts are increasing the vulnerabilities of populations that are already struggling with food insecurity and poverty, even in the relatively conservative scenario of a global 2-degree temperature rise (Thornton et al. 2014a).

The increasing complexity of the challenges, particularly with regard to their impacts on poor and vulnerable populations, requires a rethinking of our approach to R4D. CGIAR has taken on this challenge by broadening its portfolio of major new initiatives for strategic research. A first round of some half-a-dozen ‘Challenge Programs’ were mandated to develop new R4D models over a period of up to 10 years, starting in 2002 (CGIAR 2015b). Box 4.2 describes one example of these programs, focusing on water and food.

In a second round, from 2010 onwards, 16 CGIAR Research Programs (CRPs) were set up in a 5-year first phase (CGIAR 2015c). The major reorientation of the R4D portfolio was in the move from an output focus to an outcome focus. Success was now to be measured in terms of the CRP’s contribution to behavioral changes, manifested in changes in knowledge, attitudes and skills and practices of a wide set of non-research next users, including development practitioners, farmers and policy makers.

Through approaches such as results-based management, theories of change and impact pathways, the term outcome came into focus. Organizations such as the International Development Research Center (IDRC) were early developers of M&E tools to capture and measure outcomes through their ‘Outcome Mapping’ methodology (Earl et al. 2001). Within CGIAR, ‘Participatory Impact Pathways Analysis’ (PIPA) (Douthwaite et al. 2003, 2007) was developed under the CPWF to unpack processes and mechanisms in the realm of outcomes.

Towards the end of the first phase, 4 of the 16 CGIAR research programs were tasked to develop a comprehensive, suitable and lean results-based management approach for research for development, initially for a period of 12 months. The following section describes how the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) developed and implemented its RBM trial and highlights the main lessons learnt.

This section describes the approach to results-based management taken by one research for development program of CGIAR, CCAFS. The description is combined with theoretical and practical references to development agencies that started experimenting with results-based management some 10 years earlier. Figure 4.2 illustrates CCAFS’ approach to implementing results-based management with a theory of change (TOC) approach along defined impact pathways, focusing on outcome delivery. The TOC defines several activities, such as developing the impact pathways for thematic research and regional work, trialing RBM with a subset of projects, training key partners in the impact pathways building, and analytical systems support. These led to tangible outputs, e.g. a finalized ex-ante set of impact pathways with coherently defined outcome targets, workshop reports and learning notes, facilitation guidelines (CCAFS 2015a), a RBM MEL strategy (CCAFS 2015b), and an online platform. This involved the engagement with and involvement of identified key next-users such as CGIAR Consortium Office, program partners, and fellow researchers, with the idea that these outputs would both be useable and an incentive to overcome existing barriers in the system. It was also envisaged that the outputs would facilitate changes in their practice: for example, working towards implementing more efficient and effective R4D, and proactively changing organizational norms. Moving from outcomes to impact in Fig. 4.2 requires several steps that are not elaborated because this is beyond the scope of this chapter.

CCAFS was clearly committed to an outcome-focused R4D program from its inception. It became increasingly clear that a logframe approach (LFA) was not the most ideal way of doing R4D differently. In particular, when moving from a Challenge Program to a CGIAR Research Program with increasing complexities of partnerships, engagement and CGIAR integration, the limits of a logframe became apparent (as described in section 4.5.1). The program’s vision of contributing towards development outcomes increasingly required a different approach: one that acknowledged the importance of stakeholder engagement and capacity development. As a result, monitoring the annual contribution of CCAFS and its partners towards development outcomes becomes increasingly complex.

While a wide range of MEL approaches and methodologies with an outcome focus exist (e.g. PIPA, Outcome Mapping, Outcome Harvesting), none provides a blue-print solution that can just be rolled out. The approaches were designed to address the particular needs of a specific program or organization. Thus, to adapt these approaches to a new program, it is key to select the right mix of elements creating a conceptual framework in support of the program’s specific TOC and MEL requirements. Springer-Heinze et al. (2003) advocate a holistic approach to impact evaluation and program monitoring with quantitative and qualitative elements, based on an impact pathway that can accommodate different stakeholder views, allows for reflection, and emphasizes capacity of research organizations. Cummings (1997) compares RBM, LFA and Project Lifecycle Management and would welcome more discussions and learning among the different approaches. According to Bazeley (2004), ‘The “mixing” may be nothing more than a side-by-side or sequential use of different methods, or it may be that different methods are being fully integrated in a single analysis’. Applying a mix of methodologies in a programmatic MEL framework raises certain terminological, definitional, paradigmatic and methodological issues, including over-interpretation of numbers, single dimensionality, and disregarding ‘outliers’ from the analysis (Bazeley 2004). However, mixed methods also provide opportunities to address the respective shortcomings of any single method as applied in practice.

CCAFS in its early years worked with various logframe elements in planning and reporting. Limitations of the more traditional LFA resulted in experimentation with elements of TOC that were integrated within the logframe, in order to more effectively capture the complexity of activities, partners and anticipated outcomes of the program. The limitations of this single method approach resulted in CCAFS deciding to operationalize a modular MEL approach, described in the next section. The findings and analysis section explains CCAFS’s approach over time. With the limitations in mind CCAFS is aiming for a more holistic approach in line with Springer-Heinze et al. (2003).

In addition to the use of logframe elements within the CCAFS planning and reporting system, at program design stage CCAFS also explicitly included a research theme entitled ‘Knowledge to Action’ in its portfolio (Jost et al. 2014a). The team was experimenting with strategies of getting from research outputs to development outcomes. This theme was tasked with research, not with creating an operational mechanism for CCAFS per se. It was only in year 3, when CCAFS started working in two new target regions, that opportunities presented themselves to trial a TOC approach within this new component of the R4D portfolio (Jost and Sebastian 2014; Jost et al. 2014b). Very early on it became clear that a new way of thinking needs to take effect for the whole program in order to plan for and capture outcomes more effectively and include engagement and capacity enhancement as key strategic elements (Thornton et al. 2014b). As a consequence, the ‘Knowledge into Action’ theme was then mainstreamed into the whole CCAFS program with its four research and five regional programs.

The opportunity to trial an alternative approach of RBM was taken up enthusiastically (Thornton et al. 2014b; Jost et al. 2014c). Theory of change, impact pathways and results-based management offer practical mechanisms to potentially enhance program design and its monitoring, learning and evaluation and help CCAFS to create an operational program management framework that is better suited to deal with the complexities at hand.

Closely linked to this, CCAFS and partners also started experimenting with learning-based approaches within R4D recognizing the need to include mechanisms that challenge business as usual and support institutional learning and innovation to ensure that research contributes to development outcomes, see Box 4.4.

The RBM trial project teams were thrown in at the deep end. Used to a more traditional LFA, they were tasked with shifting to a TOC and learning-based approach for planning their projects within the trial. It was quickly apparent that capacity to plan projects using this new approach had to be built within CCAFS (and wider CGIAR).

Using TOC approaches within R4D requires the strengthening of capacities of scientists to do research differently and work with non-research partners for impact, but also of institutions to facilitate such a shift. Several authors highlight the importance of building capacity for institutional learning (Hall et al. 2003; Horton and Mackay 2003; Eade 1997; Springer-Heinze et al. 2003). Eade (1997) emphasizes a capacity-building approach, training of staff in a variety of relevant skills, and the dynamic and long-term nature of the process when looking at types of social organization of NGOs engaged in development theory and practice. Johnson et al. (2003) show that participation of non-research stakeholders early on in the research process is important, as it can inform institutional learning in research organizations to change priorities and practices. It can also enhance the relevance of agricultural technologies and the capacity of these stakeholders to design their own action research processes (Johnson et al. 2003). Horton and Mackay (2003) outline the links between M&E, learning and institutional change and highlight the importance of institutional learning as a means to develop the capacities of the organization and of individual researchers, as well as empowering non-research partners as key stakeholders in the process.

CCAFS worked with expert facilitators and trainers from PIPA to implement a 1-week training course on using TOC for project and program planning (Alvarez et al. 2014). Participants were chosen strategically so that capacity would be available in the CGIAR Centers at the point in time when CGIAR proposals would need to be developed following the TOC principles. In addition to project representatives, CCAFS science officers representing all themes and regions participated, in order to build in-house capacity of TOC champions. The training, in combination with TOC facilitation guides (version 1: Jost et al. 2014d; version 2: Schuetz et al. 2014d) and learning notes (CCAFS 2015a), helped highlight the opportunities (and constraints) of rolling out RBM to a whole R4D program. An online community of practice (and wikispace) was established and allowed for continued documentation and exchange of experiences.

CCAFS’ approach to RBM is centered on adaptive management, regular communications between program and projects, and facilitated learning within projects. Besides periodic virtual meetings, trial participants were surveyed for a more in-depth and standardized reflection, and for capturing lessons and achievements from their experience (Schuetz et al. 2014b, c). These lessons also formed the basis for the progress report to CO (Thornton et al. 2014c). Ten months into the RBM trial, the progress report summarized project participant experiences, as well as the programmatic perspective.

From the programmatic level, reflections and lessons by the CCAFS Program Management Committee have been documented in the CO progress report, as well as in the series of learning notes (CCAFS 2015a). It was a great learning experience to have an RBM trial with the six projects and to be allowed to test and tryout what is required to make the shift from a LFA to an approach that is much more people-focused, learning-focused and outcome-focused. The approach to developing the IPs was simplified over time, mostly in relation to a reduction in the type and number of indicators and level of complexity so that the wider group of people who were expected to work with them would continue to buy in to the approach (Schuetz et al. 2014d).

The survey results show that there are many people within CGIAR Centers and CCAFS partners who are willing to take on the challenge to develop new ways of collaborating and working beyond delivering outputs towards outcomes (Schuetz et al. 2014b). From the survey, the RBM trial team found that the projects had made considerable progress, but also that making fundamental shifts in the way of working take time and (initially at least) additional resources. It requires iterative and continuous processes. Staffing, or the profile of project team members, and project team composition are emerging as key factors for success. Project staff has acknowledged that they may require additional skills beyond disciplinary expertise, such as skills in coordination, facilitation, engagement, communications, and participatory and learning-oriented M&E. The RBM trial team is using the findings from the survey to explore how additional support can be provided in such areas as engaging with stakeholders and using RBM.

Opportunities for changing the programmatic approach to project planning, implementation and MEL emerged when CCAFS was approaching the end of its first phase in 2014. The mandate to implement an RBM trial came at a perfect time – it was initiated in advance so that it could inform the planning of the CCAFS extension phase (2015–2016), as well as Phase 2 proposal development (2017–2022). With a time lag of several months between the RBM trial and CCAFS as a whole, the program planning process and TOCs were developed and defined for all four research and five regional programs as a first step to putting together the new program portfolio (Schuetz et al. 2014e). Figure 4.4 provides an illustration of one research theme’s impact pathway component with its regional elements, indicators and outcome targets.

Experience in CPWF also shows that an intense process is required to finalize the program portfolio and allow for the appropriate triangulation and harmonization between thematic perspective, regional context and individual project proposals to ensure programmatic coherence, cohesion and its relevance and potential for impact (Hall et al. 2014; Biswas et al. 2008). This requires intense bilateral virtual preparation between research and regional teams, facilitated face-to-face time (e.g. in the form of workshops or writeshops), and follow-up work. Intensive workshops bring together project leaders, key national and regional partners and core program staff within a respective regional or thematic focus. The workshops/writeshops can bring together selected projects in a region as a team that will continue to work together over a period of time. It is key that the agenda is designed to hone the individual project IPs towards a coherent and cohesive regional and global R4D program that complements other ongoing initiatives and contributes to the given development goals.

While it took a considerable amount of effort, the iterative development of the CCAFS TOCs and impact pathways was done in a resource-efficient way. It started off mostly virtually and intensely facilitated, building on CCAFS Phase 1 engagement and regional priorities, and was completed in five regional face-to face meetings with key next-users and stakeholders within the CGIAR research community (Schuetz et al. 2014e, f). Building on the learning-based approach to developing a suitable TOC approach for CCAFS, a series of learning notes was written to document the RBM trial experiences and the rolling out of the approach to the whole program (CCAFS 2015a). The TOC development and facilitation process, and guidance documentation were revised to make them leaner, more contextualized and easier to implement (Schuetz et al. 2014d). The TOC building process is one key component in the CCAFS MEL system that was developed to support the new approach in a comprehensive manner (CCAFS 2015b).

Building on the above, a CCAFS Monitoring, Learning and Evaluation Strategy was approved by the program’s management committee and its advisory board (Schuetz et al. 2014g). The overall goal of the CCAFS MEL strategy is to develop an “evaluative culture” within CCAFS that encourages self-reflection and self-examination, seeks evidence, takes time to learn, encourages experimentation and change so that MEL becomes an integrated mechanism. The strategy includes a conceptual framework, guided by overall program principles for partnership, engagement and communications and a modular system (see Fig. 4.5). The added value of the framework has been adapted from UNDP’s (2007) expected competencies for their managers through an RBM approach:

A modular system can best meet the demands of the program as a whole and its projects, as well as the wider CGIAR system (see below; Thornton et al. 2014d). Some elements are prescribed by CGIAR governance bodies, including the carrying out of baselines, independent impact assessments, and periodic external evaluations. Programmatic flexibility exists within the day-to-day operational MEL, as a system is required that allows enough flexibility and adaptability to be applied to the different types of projects and programs.

CCAFS has identified the following modules to guide its MEL system (Schuetz et al. 2014g):

Chapter 14 (Adaptation Processes in Agriculture and Food Security: Insights from Evaluating Behavioral Changes in West Africa) of this book describes an example of how this has been operationalized in a regional program of CCAFS.

In this section we list some practical implications for a research-for-development organization that is considering moving to an approach based on RBM and TOC (Schuetz et al. 2015).

Working along TOCs and impact pathways has major implications for M&E. It implies a move to contribution rather than attribution, to acknowledge the role and inputs of partners and other actors both in achieving outcomes and in providing evidence for those outcomes. Building in triple-loop learning can make a major contribution to reflection and to supporting adaptive management, so that project teams can better deal with uncertainty. At the same time, not everything can be measured; this highlights the need for narratives that can complement and support more quantitative information.

As part of creating a program enabling environment, embracing the three thirds principle facilitates investment into solid science, critical partnerships, ownership and buy-in by partners, and capacity enhancement at all levels both internally and externally. CCAFS has been pushing the boundaries of R4D and has been serious about taking on the expanded CGIAR mandate to deliver outcomes, see Fig. 4.6.

The three thirds principle implies different budgeting and funding structures, so that appropriate levels of resources are allocated to capacity building, communications and engagement with the wide range of different partners likely to be needed. These elements need to be budgeted for explicitly within a project life-cycle, rather than as an after-thought. At the same time, there is still much work to be done on how to monitor outcomes effectively, evaluate the real share of contribution towards the observed change, and assess value for money. Similarly, delivery of outcomes, especially at scale, may take time for research-for-development programs. Longer funding cycles could be expected to facilitate this considerably.

The CCAFS experience has highlighted several operational principles for programmatic RBM. First, there is a need to focus on people and users, on utilizing M&E as a tool to help achieve outcomes, and on accountability – it is the people within organizations that make behavioral and practice changes happen. Second, there should be an emphasis on learning through M&E activities. Robust knowledge needs to be generated that can feed into developmental policy and investment decision making, and this in turn requires a cumulative and catholic approach to choice of impact assessment methods at different levels (Maredia 2009). Third, adaptive management needs to be encouraged, as a key element of RBM. As a tool that is based on learning processes, it can improve long-run management outcomes. The challenge in using it is to find the balance between gaining knowledge to improve management in the future and achieving the best short-term outcome based on current knowledge. Fourth, the development and implementation of an online platform is a great investment for capacity development. Planning, reporting and evaluation procedures need to be as simple as possible while still providing (most of) the information needed for effective and timely management.

Solutions that are good enough rather than optimal. In many domains of knowledge and practice there is no best practice or option, particularly when the problem is complex and resources are constrained. CCAFS made considerable changes once it had started to implement an approach based on TOC and impact pathways, and in time moved towards a leaner and simpler model. Time will tell if some of the details inevitably lost in this process will need to be added back in, but the notion of “good enough” systems needs to be a key guiding principle.

Addressing tensions across scale. CCAFS is still in the process of embedding TOCs for the different organizational units of the program, in order to provide a flexible framework that allows for aggregation of output, outcomes and targets across the different units. For example, targets need to be framed locally with users and beneficiaries, and voiced in such a way as to allow the flexibility to deal with uncertainty and emerging priorities and opportunities. New investments of time and effort may be needed to identify and work with non-traditional partners to promote behavioral change in shared IPs.

Providing value for money. Many funding agencies now require that grantees demonstrate value for money. The Deutsche Gesellschaft für Zusammenarbeit states that its ‘work is systematically geared towards results, the yardstick by which we measure the success of our work. We want to help achieve tangible positive changes on the ground’ (GIZ 2015). Some have critiqued the whole notion of payment by results as applied to development and research-for-development on the basis that it provides perverse incentives that actually diminishes cost-effectiveness (see Chambers 2014). As noted above, there is much work still to do on appropriate measurement mechanisms, but this does not diminish the need to demonstrate accountability.

Balancing science and outcomes. Research is often curiosity-driven, and traditional indicators of success center on peer-reviewed publications in high-profile academic journals. In today’s highly competitive research environment another crucial success factor relates to fundraising: the ability to write and win competitive research proposals. Neither of these motivations for research is guaranteed to deliver development outcomes. For CGIAR and its research programs, it is still early days, but preliminary results suggest that “successful RBM” relates to effective and efficient research leading to outcomes, with a minimum of perverse incentives. The building of an IP with a narrative TOC forces researchers to give some thought to what lies between solid science, great technologies, and their positive developmental impact. A mix of an outcome-focused TOC with people and partners at the core, and a RBM approach that allows us to monitor, reflect, evaluate, and learn, are key elements for a programmatic MEL strategy – coupled with great science.

Requests by funding agencies for a move towards outcome-oriented research programs are having considerable impact on the way in which research is conceived, planned, implemented and evaluated. A key requirement for such work is flexibility – the flexibility to adjust so that the outcome orientation works as a support mechanism and enabler rather than a one-size-fits-all straitjacket without any space for innovation, serendipity and creativity. The shift to a R4D approach based on TOC is fostering massive change, much of it for the better, in our view. However, it also comes with considerable challenges. Defining the necessary changes, and developing new processes and mechanisms, need time and resources, which are often grossly underestimated and inadequately planned for. Some of these challenges arise because of the nature of research: the results are not known from the start, unlike in engineering where the outcomes are generally much less uncertain. Another challenge is that CGIAR is a R4D organization, not a development organization, and it is still in the process of sorting out how to balance the need to do great science with the need for impact. We need to avoid the results-based focus being to the disadvantage of the science, and development being seen to be in competition with the science. Rather, they need to be seen as complementary, enabling, and liberating.