A Network Based Approach to Evaluate Participatory Policy Processes: An Application to CAADP in Malawi

Donor organizations recently engaged in promoting participatory policy processes as a tool for designing efficient policy programs. Participatory policymaking is a process through which stakeholders influence and share control over priority-setting and policymaking (World Bank 2011). The implementation of participatory processes is promoted to increase the efficiency and effectiveness of policy choices, particularly in developing countries. In general, more effective and efficient policy choices are induced via specific mechanisms: 1. Increasing governmental accountability (Keefer and Khemani 2005) (i.e., incentives for governmental agents to serve societal interests and needs). 2. Reducing government capture (Bardhan and Mookherjee 2002) (i.e., government incentives to serve vested interests at the expense of the general public). 3. Increasing evidence-based policy processes (i.e., the degree to which political actors use available political knowledge for making policy choices. Political knowledge is knowledge about the technical relation between policies and induced policy outcomes). 4. Increasing policy ownership (i.e., citizens [civil society] identify with and feel committed to governmental policy); policy ownership implies a higher citizen compliance with the established policy framework and can significantly reduce political implementation costs (Adserà et al. 2003; Jones 2013; Chambote and Shankland 2011). Accordingly, it is widely accepted that the ownership of policy programs that occurs as a result of participation in policy program formulation leads to more effective implementation and adoption of the policy programs (World Bank 2011).

Understanding the nature of participatory policy processes is not only high on the research agenda at the academic level but also increasingly recognized as a key condition for efficiently providing support for the formulation of effective policy programs in political practice. The Comprehensive Africa Agriculture Development Programme (CAADP), which was initiated by the African Union, is a good example of these new developments. The inclusion of local stakeholder organizations in the planning, formulation and evaluation of sector-specific growth policies is a key principle of the program (NEPAD 2010). However, a CAADP task group focused on the evaluation of non-state actor participation reported that stakeholders had only a limited ability to use the newly created opportunities for participation. Based on information collected via a qualitative stakeholder survey and desk research, the task group emphasized that CAADP has not yet consistently achieved a high quality inclusion of non-state actors at the national, regional or local levels (Randall 2011, p. 2).

Nevertheless, although participatory policy processes are intuitively convincing as a tool for the evaluation of existing policy processes and for the design of more efficient policy processes in the future, the development of a measurement tool that allows for the comprehensive characterization of participation structures inherent to real policy processes that are ongoing in political practice is necessary. Ideally, this measurement tool is grounded in political theory to guarantee that empirically identified participation structures can be consistently related to an intervention logic (i.e., the derivation of a causal link between specific properties of the policy process and desirable policy outcomes). The latter criterion implies that the impact of the identified structures of a policy process on governmental accountability and capture and on the effective use of political knowledge and political ownership can be directly derived.

However, to the best of our knowledge, a comprehensive, micro-political-founded and quantitative evaluation framework for participatory policy processes is not currently available. In this context, the Advocacy Coalition Framework (ACF) proposed by Sabatier and Jenkins-Smith (1993) has attracted attention as an approach for analyzing policy processes, particularly among political scientists at the academic level. More recently, the ACF has been increasingly recognized by development economists (Birner and Resnick 2010). This framework includes a number of interesting aspects (e.g., this framework explicitly identifies beliefs as drivers of coalition formation and final political decisionmaking). Moreover, this framework provides a systematic approach for analyzing stakeholder interactions. However, the ACF is a qualitative approach and provides neither a theoretical model of political decisionmaking nor a theoretical model of belief formation among the actors involved in policymaking. In particular, a quantitative description of real policy processes is a necessary condition for a comprehensive evaluation of the impact of these processes on the effectiveness and efficiency of policy outputs. Hence, the ACF in its present form is not yet an appropriate tool for policy learning (i.e., a tool that is ready to identify causal links between specific patterns of stakeholder interactions and induced governmental performance).

In this context, the aim of this chapter is to initiate the development of a theoretically founded framework for analyzing participatory policy processes that can be applied empirically. In particular, following the literature on policy network analyses (Lauman and Knoke 1987; Pappi et al. 1995; Knoke et al. 1996; Pappi and Henning 1998, 1999; Henning 2000, 2009), we apply social network analysis (SNA) to measure complex interactions among stakeholder and governmental organizations. Moreover, we derive a theoretical framework for the incorporation of policy network theory into political economy models of lobbying and legislative decisionmaking. As we demonstrated in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”, we incorporate a network model of political belief formation into a political bargaining model (see Henning in this volume). The latter model combines a generalized Grossman-Helpman lobbying model and a modified legislative bargaining model of the Baron and Ferejohn type. The central component of such an integrated model corresponds to a generalized mean voter decision rule, where in addition to legislators, interest groups also have political control over policies (Pappi and Henning 1998; Henning 2000, 2009). Political control of nongovernmental organizations results via two different mechanisms: lobbying and communication learning. The first mechanism is determined by the political access structures via which nongovernmental organizations access powerful governmental organizations, and the second mechanism is determined by political communication among organizations. Empirically, the equilibrium outcome of the complete model can be derived from observed political support and communication networks.

Based on our theoretical framework, we undertake a descriptive analysis of the central components that determine policy choices. This analysis includes a network analysis of the underlying communication and access structure, which is encapsulated in the communication network and the political support network. In this regard, social network analyses provide a wide range of local and global network tools and methods for describing the characteristics of an individual network (Wassermann and Faust 1994). Previous political sociology reports demonstrated that network analysis can be used to systematically describe interaction structures among nongovernmental and governmental organizations that are engaged in a specific policy domain (Lauman and Knoke 1987; Knoke et al. 1996; Pappi et al. 1995; Pappi and Henning 1998, 1999; Henning 2000; Henning and Wald 2000). However, with a small number of exceptions (e.g., Pappi and Henning 1998; Henning 2000, 2009), these studies used SNA to describe policy domain network structures without relating the identified structures to political performance. In contrast, our framework offers innovative network tools (i.e., the network multipliers derived in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”) that go beyond a pure descriptive analysis. The framework will allow us to make direct conclusions concerning the impact of the identified network structures on different aspects that determine governmental performance. Specifically, we describe the four previously mentioned aspects of government performance and use our network-based indicators to measure these factors empirically.

The case country selected for the empirical application of our framework is Malawi. Malawi approved a policy reform, the sector investment program Agricultural Sector Wide Approach (ASWAp), based on the principles of CAADP in 2010. Data was collected via a policy network study in 2010.

In the next section, we briefly summarize the theoretical framework for modeling participatory policy processes. We describe in detail the design of the network study, including the collected data and the central network theoretical tools/measures used in our analyses. We then present the principal empirical results, and we conclude by providing an outlook on future research.

In general, we apply the theoretical framework developed in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”. We focus specifically on the combination of a modified legislative bargaining model and the political belief formation model in a policy elite network. This approach considers the political decisionmaking process as an aggregation mechanism of the policy preferences of the involved political actors. In essence, this aggregation mechanism corresponds to a generalized mean voter decision rule:where γ ^∗ denotes the final policy decision, \( {\varphi}_j^T \) denotes the total political power and \( {\widehat{\gamma}}_j^0 \) denotes the initial preferred policy position of actor j. The total political power results from political influence on powerful political actors:where \( {\overline{m}}_{gj} \) denotes the political influence of actor j on agent i. As demonstrated in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”, \( {\overline{m}}_{gj} \) is the network multiplier derived from the communication network among governmental and nongovernmental actors. Accordingly, \( {\overline{m}}_{gg} \) denotes the weight that a legislator g puts on her own initial position, while \( {\varphi}_i^L \) is the political power of agent i that is derived from the lobbying game, as described in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”. Thus, it holds:

As described in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”, \( {m}_{gi}^S \) is the support network multiplier that corresponds to the outflow of legislative power from legislator g to actor i, resulting in the equilibrium of the lobbying game. Hence, the better access an interest group i has to powerful legislators, the more successful are the lobbying activities of this organization. Further, φ _g denotes the legislative power of legislator g that is derived from the modified legislative bargaining game, as described in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”. Equations (1, 2, and 3) constitute the theoretical backbone of our policy process framework, which we have also illustrated in Fig. 1 below.

As is illustrated in Fig. 1, based on our theory, the policy process corresponds to an aggregation mechanism of the policy positions of the involved governmental and nongovernmental organizations, where the individual weight of an organization is jointly determined by political communication network structures that determine political influence \( {\overline{m}}_{ij} \), informal access structures \( {\overline{m}}_{gi}^S \) that determine lobbying power and constitutional rules that determine legislative decisionmaking powerφ _g ¹.

Based on our theoretical framework, the underlying communication and access structures, which are encapsulated in the communication network MC and the political support network MS, are central components that determine final policy choices. To describe these networks, we apply social network analysis tools, including methods for identifying overall network structures (e.g., block model analysis) (Wassermann and Faust 1994). However, our framework offers innovative network tools that go beyond a pure descriptive analysis to allow us to draw direct conclusions concerning the impacts of the identified network structures on different aspects that determine governmental performance (i.e., the network multipliers derived in Chapter “Modeling and Evaluation of Political Processes: A New Quantitative Approach”. Specifically, we develop the network-based indicators described below to empirically measure different aspects of governmental performance.

In this section, we discuss how the identified participation structures impact political performance. In particular, we assess governmental performance by applying the derived indicators to measure political accountability, capture, and ownership and effective use of political knowledge. To evaluate different dimensions of political performance, it is important to define adequate benchmark levels. Because by construction, all indicators lie in the [0–1] interval, 1 is a default benchmark value for all indices. However, for some indices, different benchmark values appear to be more appropriate. For example, when measuring accountability by the total political power of national stakeholder organizations (GA-total), it appears unrealistic to assume that perfect accountability corresponds to a situation in which stakeholders control 100% of the total power. Depending on the degree to which democratic elections imply incentives for politicians to represent society’s interests, the optimal level of accountability induced through stakeholder participation varies. Given a relatively low level of government accountability induced by elections, as is the case in Malawi (see Chapter “Voter Behavior and Government Performance in Malawi: An Application of a Probabilistic Voting Model”), we assume a benchmark value of 0.5 for all three accountability measures. Analogously, when measuring political knowledge used in the policy process by the power share of national research organizations, it also appears adequate to take a benchmark value below 1. The latter inference results from the fact a benchmark value of 1 implies that the political knowledge of research organizations is infinitely higher than that of other political organizations. Hence, because Bunda College is the only national research organization in Malawi, we took 0.05 as an appropriate benchmark value for our knowledge indicator “PK-research”. Finally, when measuring the involvement of national stakeholder organizations in political communication using the network density as an indicator, a benchmark value of 0.5 appears to be more appropriate. Please note that a density of 1 implies that every stakeholder organization communicates with every governmental organization, which would be rather unrealistic and inefficient. This inefficiency results from the fact that brokerage via national peak organizations and subordinated state agencies allows for more efficient communication between government and stakeholder organizations.

Renormalizing the calculated performance indices to the [0–1] interval using appropriate benchmark values, we present the calculated performance indices in an evaluation wheel, as demonstrated in Fig. 7 below. We consider the evaluation wheel a helpful tool for illustrating the impact of the identified participation network structures on the different dimensions of political performance. A closer evaluation of Fig. 7 reveals that overall, the policy process in Malawi implies only a moderate achievement of political performance, where most indicators values reach 50% or less of their benchmark values (see Fig. 7). At first glance, government capture appears to play only a minor rule in the CAADP policy process in Malawi, with a performance level for total capture (GC-total) reaching over 70% of the benchmark value, which corresponds to a complete avoidance of capture. However, a close evaluation partially reverses this conclusion. In particular, to construct the capture index, we used the different household types that have been separated in the Malawi CGE (i.e., small-, medium- and large-scale agricultural households and urban consumer households). To calculate the representation of different household types in the political process, we calculated the relative shares of household types in the value-added share of each economic subsector. The higher the value-added share of a subsector that is allocated to a household type, the more the socioeconomic interest groups representing this subsector represent this household type in the political process. Therefore, we matched all agricultural, agribusiness and non-agricultural interest groups in our policy network with economic subsectors. Based on these matches, we could calculate the representation rate of different household types for each organization. Finally, weighting these organizational representation rates by the total power of organizations generates the representation share of a household type in the political process. These shares are compared to the corresponding population shares of household types to derive our capture index. For example, agricultural export crops (i.e., tobacco, coffee and tea) are primarily produced by large-scale farm households (i.e., 30% of the total value-added share of these sectors ends up in the pocket of large-scale farm households, but only 3% of all Malawian households are in this household category). Thus, interest groups like TAM, CAMAL and TAMA that focus on representing these subsectors over-represent large-scale farm households, and general farm organizations, such as FUM, that represent all agricultural subsectors proportionally represent all farm households. A special case corresponds to the National Smallholder Farmers’ Association of Malawi (NASFAM), which is focused on representing small- and medium-scale farmers. Therefore, we exogenously assume that NASFAM only represents these two household types. Interestingly, overall, medium farm households (e.g., households that operate farms with a size between 0.75 and 3 hectares of land) are extremely underrepresented, as indicated by a difference of approximately 26 percentage points between the population share and the political representation share for this household type. In contrast, urban households are extremely over-represented in the political process, with the same difference of approximately 26 percentage points between the representation share of urban households and the urban population share. Thus, political participation is clearly biased against rural households and in favor of urban households. This bias can be observed in the relatively low value of the capture sub-indicator GC-urban. In absolute terms, the share of the total political power that represents the interest of urban households is nearly 3 times higher than the urban population share. However, because urban households comprise only 9% of all Malawian households, the total capture remains low.

Finally, we would like to make one further comment on the interpretation of the impact of the identified participation structures on the achievement of a political consensus. Compared to the benchmark level of 1, the current participation structures imply only a moderate potential for generating a political consensus, as indicated by a level of 41% for the central ownership indicator Ownership. However, as demonstrated in Fig. 8, political communication implies a remarkable reduction of political conflict in the complete network. This reduction is demonstrated in Fig. 8, which compares the range of the convex hull that covers all preferred policy positions before and after communication (see the outer and medium set, respectively, in Fig. 8). Moreover, Fig. 8 demonstrates that the influence fields that operate on the different organizations imply extremely homogenous policy positions (see the set covering the inner dots in Fig. 8). Thus, if governmental organizations were more open to the opinions and viewpoints that other organizations communicate in the political discourse, the communication network structure in Malawi would imply a political consensus. Therefore, an evaluation of the identified communication structures against a benchmark of 1 might lead to result that is too pessimistic with regard to the potential to achieve political ownership via political consensus.

To identify potential strategies for improving participatory and evidence-based policy processes, we simulated the variation of political performance indicators assuming changed formal and informal rules determining participation structures. We simulated political performance for all 45 legislative and lobbying scenarios; however, in the following section, we focus on the most relevant results. First, for both types of participation, we change the level of participation intensity without changing the participation structure. With respect to lobbying, the level of politician interest in political support is changed, keeping both the relative interests of politicians and the access to politicians constant. Accordingly, with respect to political communication, we change the level of own control of politicians, keeping the communication network structure and the relative own control among politicians constant. Second, we change the participation structure (i.e., we shift the participation bias from the identified donor-dominated participation structure to a CSO- and farm-dominated participation structure. Finally, we simulated how political performance changes assuming a constitutional reform from the present PDR to party leadership. We present the results for these four scenarios in Fig. 9. In detail, it is assumed in the “no lobby scenario” that politicians have no interest in political support. In contrast, improved access of national stakeholder organizations is simulated in the “CSO lead” scenario. Technically, the latter scenario is generated by recalculating the support network multipliers under the assumption that the relative interest of politicians in political support provided by national stakeholder organizations¹¹ is increased. In the “open discourse scenario,” we multiplied the own control of politicians by 0.7 and recalculated the communication network multipliers. Finally, in the “party leadership” scenario, we simulated a constitutional reform of the legislative process from the PDR to party leadership (PL). Please note that with respect to content, such a reform might correspond to a formal constitutional reform from a presidential to a parliamentary system. However, such a reform might also result from changed legislative norms without a formal change of the constitution.¹²

For all simulation scenarios, we take the best-fit scenario (see Fig. 7) as a base-run scenario and present the percent change in political performance indicators in comparison to their corresponding values in the base-run in Fig. 9.

As demonstrated in Fig. 8, lobbying has a significant positive impact on nearly all political performance indicators. Only the dominance of international donor organizations is less pronounced assuming no lobbying occurs (i.e., without lobbying, international stakeholder organizations would not be able to gain much political influence, and Malawi’s civil society would take more ownership in CAADP reforms due to a feeling that these reforms are less imposed by the international donor community). However, as demonstrated in Fig. 9, without lobbying, the final political decision regarding CAADP would also be more distant to the desires of Malawi civil society, implying a lower political consensus and less political ownership, as indicated by a decrease of approximately 30% in the indicator Ownership for the “no lobbying scenario” (see Fig. 9). Moreover, increasing the importance of lobbying without changing the relative lobbying power of organizations implies that there will be no impact on capture. Accordingly, for the no lobbying scenario, all capture indices remain unchanged in comparison to their base-run values. The main positive impacts of increased participation via lobbying can be observed for the effective use of political knowledge and for accountability. The use of political knowledge decreased by nearly 60% when comparing the no lobby scenario to the base-run scenario (see Fig. 9).

Analogously, government accountability increases significantly with lobbying, with the accountability level decreasing by approximately 50% for the no lobbying scenario compared to the base-run scenario (see Fig. 9). Interestingly, the positive impact of lobbying does not change significantly if different lobby structures are assumed. Hence, as demonstrated in Fig. 9, for the CSO scenario, the majority of performance indicators remain unchanged (e.g., they lie on the 0% line in Fig. 9). The only exemptions are accountability (GA-total), which improves by 20%, and the dominance of donor organization (PO-donor), which is significantly reduced when changing form donor-dominated to CSO- or farm-dominated lobbying structures.¹³

As described above, the high own control of politicians is a characteristic feature of the CAADP policy process in Malawi, which implies that politicians rely primarily on their own expertise when making policy choices. As demonstrated in Fig. 9, increasing the importance of public discourse for the political belief updating of politicians would imply that wisdom of the crowd effects can be better exploited in Malawi. Thus, in particular, the effective use of political knowledge would increase significantly. Please note that in this context, increasing the physical participation of stakeholders fails to automatically guarantee higher political influence of stakeholders because the political influence of stakeholders only results if politicians in fact update their beliefs based on the political views and opinions communicated by national stakeholders. Thus, as long as stakeholder participation is only formally imposed, as is partially true for the CAADP process in Malawi, the impact on performance is rather limited. A change in the political culture such that politicians increasingly consider the potential political expertise of national stakeholders is needed. However, in this context, a warning also appears necessary. An increase in the effective participation of national stakeholders in political communication only triggers wisdom of the crowd effects and improves political performance if stakeholders have a relatively high political knowledge in comparison to their total political power. Our analysis indicates that this scenario is relevant for Malawi, but this scenario does not necessarily hold true for other countries.

Finally, shifting legislative power from the government to the parliament implies a trade-off between increased political ownership and sharply decreased effective use of political knowledge. In contrast, political incentives (i.e., accountability and capture) are only slightly changed by this power shift (see the party leadership scenario in Fig. 9). Basically, this result follows from the fact that in contrast to governmental organizations, parliamentary parties have significantly less political knowledge (see Fig. 10 in the appendix). Hence, although these parties are generally more open to political opinions communicated by other organizations in the political discourse (i.e., parties have a lower own control), the overall effect of this scenario is negative in Malawi. In contrast, party leadership has a strong positive impact on political ownership. First, deciding budget allocations for CAADP policy programs under a party leadership implies that these allocations would be much closer to the allocations preferred by civil society organizations (i.e., political ownership in CAADP reforms would significantly increase), as indicated by a nearly 60% increase in the indicator Ownership (see Fig. 9). Moreover, the political power exerted by international donor organizations would be significantly lower for the party leadership scenario than for the base-run scenario. Thus, under party leadership, the civil society of Malawi would be less likely to perceive that CAADP reforms are imposed by international donors.

Although participatory and evidence-based policy processes are increasingly promoted at the academic level and in political practice, the current understanding of the impact of these processes on political performance is still in its infancy. In this context, this paper proposes a network-based framework for analyzing and evaluating participatory and evidence-based policy processes. Specifically, we consider the following points to be the main contributions of our approach: