16. Evaluating Climate Change Adaptation in Practice: A Child-Centred, Community-Based Project in the Philippines

‘Program theory’, also referred to ‘theory-based’ or ‘theory-of-change (TOC) based evaluation’ refers to developing a causal model from project activities (inputs) to a series of outcomes, then using this model as the basis for evaluation.¹⁴ It is widely used for evaluations across sectors including to evaluate aid and development interventions.¹⁵ Theory of change-based evaluation generally uses the theory established at program design, not just to trace if different steps actually occurred, but also to test the assumptions between the causal links in the model.¹⁶ Findings from these types of evaluations can also be used to improve the ‘quality’ of theories, including by investigating alternative causal explanations to that incorporated into the initial theory of change.¹⁷

When framing an evaluation around a theory of change for a child-centred, community-based CCA project, the context for evaluation is a complex system that does not allow for a “neat” or “predictable” TOC to be articulated at the outset. CCA projects are usually dynamic and emergent interventions – whilst parameters and activities are set at the design stage, the exact details of implementation need to emerge and be developed over the course of implementation. For example, community participation in adaptation planning will always result in actions and pathways that cannot be exactly predicted at the outset of a project.¹⁸ One of the potential applications of theory-based evaluation is to identify measures that can be used for monitoring over time.¹⁹ But in complex systems relevant to CCA interventions, “SMART” measures may not be able to be developed in advance, making pre-and post-comparisons difficult.

Whilst some versions of program theory evaluation rely on close adherence at a detailed level to the initial theory to guide the evaluation, we do not take this strict definition. The complex character of CCA interventions far from makes theory-based evaluation redundant. Rather, flexible application is needed, and a balance struck between evaluation questions that are closely guided by the (initial) theory of change, and an approach that is open to outcomes, and the means to achieving them, emerging during implementation itself.²⁰

‘Developmental evaluation’ was coined by Paton to describe the types of evaluations applicable in complex situations where outcomes are emergent, where activities are not set in stone, and where it is not exactly known how, why or where activities will lead.²¹ Developmental evaluation aims to “support real-time learning in complex and emergent situations” where the focus is on “adaptive learning rather than accountability.”²²

Development practitioners and researchers have widely recognised that for projects in complex environments to be successful, self-evaluation and ongoing learning is key. Developmental evaluation can be understood by considering what its purpose is not – it is not summative, in that it doesn’t aim to evaluate at the end of a program and make a judgement about whether and how the program will continue; nor is it formative, in that it is nor primarily about setting baseline data for a future summative evaluation.²³

These high complexity situations have characteristics such as dynamic, emergent, non-linear and uncertain²⁴ – a list which also fundamentally characterises climate change, its impacts on communities, and what is needed to support adaptation. Developmental evaluation has applicability where there is uncertainty, and where the program might need to change and adapt according to emerging and changing contexts. This is particularly applicable in the case of climate change adaptation, and in the case of the CC-CBA project, significant path changes were required in the aftermath of Typhoon Haiyan in December 2013.

A key characteristic of developmental evaluation is that it supports continuous learning and innovation through embedding evaluators as part of the team engaged in project delivery, in a long-term partnering relationship.²⁵ The CC-CBA design integrated the research component within the project and indeed it was the role of researchers to facilitate evidence-based, systematic reflection on project progress. Strong individual and organisational partnerships were successfully built.²⁶ The practical realities of program budgets meant that evaluative researchers could not be embedded full-time within the project implemented, but were directly involved in research design and inception and through the course of the evaluative process interacted with project implementers periodically throughout the 3-year project.

Participatory FGDs were the core of the method applied to evaluate, test assumptions, and hear the perspectives from children and youth about their experiences of the CC-CBA project. The focus groups were designed for small groups of 8–12 children, but in practice larger groups who attended were managed flexibly with a similar process.

The team conducting the focus group discussions comprised a facilitator, documenter and a few observers, all from the project or research team. The children were familiar with the facilitators, who were specifically selected as members of the implementation team who had worked closely with the children through various activities. Although familiarity between evaluators and participants is sometimes thought to adversely affect the “impartiality” of the process,³⁰ in this situation encouraging participation and ensuring children were comfortable with the adults present was considered paramount to inclusive participation, consistent with the rights-based approach to the project, and particularly important given the sensitive and potentially troubling issues discussed related to the lived experiences of children through typhoons, landslides, floods and other climate change impacts. “Bias” resulting from the familiarity was effectively managed through careful FGD design and implementation.³¹

The facilitators were well placed to encourage children to participate. However, some children were at times hesitant to speak, at least initially, when there were several adults present as observers in the background (including one to three not from the Philippines). Over repeated visits throughout the research process children became familiar and comfortable with the Manila- and Australia-based members of the team – by the end of the project activities, familiar enough to notice and ask about where we were when some or a few of us were not present. In other cases some younger children were reluctant to offer views if the groups were dominated by older children, although overall working with teachers beforehand generally ensured children within a group were fairly consistent in age. Separate FGDs were conducted with out of school youth groups. However, there were challenges in organising to hear from children with disabilities and from ethnic minorities.

Adults from the community – parents, teachers, and local government members – were generally not present at the focus groups with children, as we sought to avoid power imbalances that would discourage children from sharing their perspectives. However, in some focus groups, local government or some teachers attended, discretely in the background. In these cases, team members familiar with these adults (who were also project participants), gauged that they would not inhibit children’s participation, and considered there would be value in them hearing children’s perspectives directly.

A nested approach was taken to translate the TOC indicator set into a series of questions linking knowledge, advocacy and practice and policy. The questions in the focus group were developed to investigate qualitative processes – for example, whether, how and why were knowledge and advocacy activities have influenced practices and policy? The FGDs focussed on those project activities which directly involved children, particularly on knowledge and advocacy activities. The FGDs were also used as a tool to explore whether children were aware of or involved in other participatory, planning- and policy-oriented activities such as the PCVAs conducted by local government. Beyond FGDs, policy impact was explored further through supplementary interviews with local leaders (see section ).

In FGDs, children were specifically invited to share problems and barriers around communicating to their families, schools, community and government members about climate change adaptation, and project implementers found their responses crucial to fine-tune advocacy program activities with both children and duty-bearers. Children also shared with team members new stories of how they had influenced their family members (including for example, family members who were also Barangay leaders) to recognise the importance of climate change adaptation.

Attribution was a key consideration in designing questions – it was considered in such a situation that establishing precise counterfactuals was not a realistic exercise, but the questions explicitly probed fact (e.g. what children learned from a specific project activity) as well as alternative explanations (e.g. sources of information about climate change beyond the project).

The final topic of the focus groups was key to applying the rights-based approach to the evaluation. We explored with children their vision for what climate change adaptation would look like, including by asking what else they would like to do to prepare for the impacts of climate change, and what else they would like to see others do – family members, school, community and local governments. By giving children a voice on this open question, useful information was provided to the implementation team about ideas for future activities. By posing this discussion topic, it also prompted children themselves to think creatively and independently about how to adapt.

Supplementary interviews were conducted after the focus groups to gather additional perspectives on pathways of impact and changes that had occurred through the project. Attention to the responsibilities of duty-bearers is fundamental to rights-based programming and adults’ attitudes and actions were explored during the supplementary interviews. Parents, teachers and local governments were asked questions that were parallel to those posed in focus groups, around knowledge, advocacy and practice and policy. For example they were asked for their perspectives as ‘audience’ members of children’s advocacy, both formally through project activities (e.g. radio programs) and through other informal communication channels, such as at home. Local government members were asked specific questions to inform the “policy” sub-set of indicators, including about how children and their communities were involved in Barangay-level planning for disaster risk reduction and CCA, The policy impact of the project relied not solely on advocacy by children and their communities, but also critically by directly supporting local governments to: provide opportunities for children and community’s to share perspectives in forums (such as PCVAs); and then use these perspectives to inform their planning and budgeting for CCA activities and development of CCA-related policies and regulations. These issues were explored during supplementary interviews.

Project implementers, reflecting on these supplementary interviews in comparison with the focus groups, noted in some cases how children and youth had developed a much more sophisticated understanding of climate change science, impacts and adaptation solutions than some of the corresponding adult participants in the project. These supplementary interviews thus provided project implementers with useful information about priorities for continuing their work with duty-bearers, including particularly on advocacy activities.

The analysis of FGD and interview results was mostly conducted through structured ‘debrief’ sessions involving facilitators, documenters, observers and interviews closely after each community was visited. This approach to analysis was driven by the practical realities of project implementation – the busy schedules and limited time for team members to conduct further desk-based analysis – as well as recognising the value of involving the team in joint reflection exercise.

The main purpose of the debrief session was to foster learning through structured reflection on the FGDs and interviews. Through the debrief sessions, the team also captured additional observations from the FGDs that were not possible to capture in detail at the time of the FGD; to reflected on what went well and less so about the FGD and facilitation itself to inform future FGDs and briefings required; and to identify learnings from the FGD and interviews, and how these might help inform future program activities. Debriefs also involved capturing representative example quotes from children in a structured away against indicators areas that showed how well children’s knowledge improved, their communication and advocacy, and the impact on practice and policy – with a reminder to link to participation in the program.

Although the emphasis was on qualitative investigation, the project team also considered it could be useful to formulate scalar measures of the indicators, potentially to enable comparison and, beyond the original thinking for applying the indicators, to assist reporting for accountability purposes. This required the team working together in the debrief sessions to articulate ‘levels’ of knowledge improvement, capacity to advocate, and practice and policy impact – and then assessing how many girls and boys in each focus group were at each level. Defining levels for scalar translation was a challenging process, particularly within the timeframe of a debrief session. The notion of scalar measures also prompted discussions about the appropriate baseline, adjustments for age level, and adjustments for variations in ‘external’ factors such as the overall level of education, access to media or other information, and whether or not children had themselves experienced climate-related disasters. Nevertheless, these discussions about how to quantify changes were in themselves valuable for the team to reflect on not only what level, but qualitatively what kind of changes were expected and could be expected as a result of the project.

The learning and reflection aims were achieved through steps A to C, but an optional extra step was developed and trialled, and could be implemented if further resources and time are available. The aim of further analysis – detailed consideration of notes and transcripts from FGDs, interviews and the debrief sessions – is to produce written narrative that can be used to record, share, report and compare learnings, and be used as examples to inform CCA practice on the ground. Ideally, the draft narrative could be shared with those children and youth who participated in the FGDs to gather their further reflections and feedback. In practice however, time availability was a major constraint limiting this aspect of the method.