Rethinking Media Development through Evaluation

While the theory and practice of media assistance has evolved in recent decades, evaluation remains an unresolved issue. Pressure is mounting to find ways to prove effectiveness and impacts. This chapter introduces four principles for more effective evaluation of media assistance. They are: (1) plan early, adapt periodically, evaluate regularly; (2) involve stakeholders in planning and evaluation; (3) evaluation designs should be theoretically informed and framed; and (4) use evidence for proving and improving. These principles, which have informed a recent practical resource for practitioners (IDEAS Guide), are explored throughout the book.

This chapter considers the materiality of media assistance evaluation documents, paying attention to the histories, procedures, and anticipated future moments of reception, circulation, and uses of documents. Tracing the document-making process in this way illuminates how documents are shaped by and instrumentalized within the development bureaucracy, leading to the ubiquitous 30-page report. External consultants are used for the rhetorical value of objectivity; but these claims of objectivity need critiquing. Bureaucratic systems construct a document that complies with quality assurance, but they contribute little to achieving rigorous, insightful, useful, and relevant evidence and evaluation. Furthermore, the proceduralization of evaluation in media assistance has the effect of constraining the methods and approaches that can be used.

Since the 1990s, the dominant framework for media assistance has been based on a Western philosophy that values free media for the functioning of a liberal democracy. Through a careful examination of a media assistance program in Cambodia, this chapter shows how multiple media, communication and development approaches can operate simultaneously within a program. In particular, it shows the value of integrating communication and social changes theories and approaches to participatory communication in order to rethink the contributions of media assistance to governance and democracy in terms of power and relationships between citizens and governors. This shift has important implications for the program theories used in projects, and suggests new ways of understanding the outcomes and impacts from media assistance.

Unlike communication for development and social change approaches, “participation” is not a native concept in media assistance and is therefore not a dominant approach in the evaluation of media assistance programs. This chapter considers the benefits, barriers, and caveats when advocating for more participatory approaches to evaluation in media assistance, taking into account the ways in which participatory approaches are already used. Participatory evaluation should not be about labor shifting; it is more valuable when it is about the inclusion of key stakeholders in the framing of the evaluation’s focus, and in deciding on actions to take in response to emerging evidence.

Articulating evaluation questions is a deceptively challenging aspect of designing evaluation plans in media development. This chapter examines the creation of the “menu of common questions” tool from the IDEAS Guide, with a focus on dialogue and two-way communication aspects of governance and accountability programs. The chapter then discusses how advances in complexity thinking and evaluation can be applied to questions of impact and causal analysis in media assistance. While there is a growing body of evaluations using randomized control trials and quasi-experimental designs in media development, alternative options for systematic causal and contribution analysis are under-used options. While there is great potential for improving impact evaluation practices using these techniques, it is important not to lose sight of the importance of learning through evaluation, which is ultimately the way in which programs benefit from evaluation and become more effective.

The concluding chapter revisits the principles underlying the “post-media-missionary” approach to media development and its evaluation, and considers the implications for policy and practice. While challenges in terms of demonstrating the value and contribution of media assistance to media and political systems remain, another pressing challenge for the future is the need to learn from failure. The new “sustainable development goals” offer some opportunities for media assistance to grow, especially with the new goals relating to access to information, and to the accountability of institutions, where communication can be positioned as key to citizen voice and government responsiveness. The focus on governance and voice in the SDGs, coupled with a new agenda for innovation in development, give reasons for being optimistic about a future in which there is less fear and more openness to learning and experimentation.