Mapping the use of knowledge in policymaking: barriers and facilitators from a subjectivist perspective (1990–2020)

The sources to be reviewed were selected according to several criteria in line with the principles of the PRISMA-S approach, an updated and extended version of the PRISMA approach that is particularly suitable for use by authors of all disciplines (Rethlefsen et al., 2021), as detailed in Fig. 1. The approach that guides this process is based on the assumption that selecting literature, similar to selecting any other kind of data for scientific research, is a sampling process. As such, it aims first to identify the population and then to clarify the criteria for the inclusion or exclusion of sources (Yorks, 2008).

More specifically, the data selection procedure started with a keyword search that was performed on the Scopus and Web of Science online databases by using the search words that are shown in Table 1. They were applied to a large number of subject areas,³ with a timeframe set between 1990 and 2020. The subject areas and categories of the medical field were excluded not only because the main goal of the paper was specifically to map the situation in the other fields but also as a result of the specific role held by metareviews in the medical profession.⁴ The search criteria and results were reviewed and validated by both authors as well as by an external reviewer.

The two datasets obtained through this keyword search were then merged, and any resulting duplicates were removed by combining manual deduplication with the use of Zotero software. At this point, all the sources without abstracts were also removed, and the remaining 4,743 records were screened according to their relevance to the subject of study. This first screening led to the removal of 3,358 records because they did not address the relationship between evidence and policy.

As a result, 1,385 relevant studies (all papers from journals) were identified as fully eligible for the goal of this review.

The corpus of literature collected shows great variation across our chosen descriptive and analytical dimensions. First, a slight majority (53%) of the research designs are empirical analyses, followed by theoretical proposals and, finally, literature reviews, as shown in Fig. 2. This means that most of the scholars concerned with the use of knowledge in public policymaking have explored this issue empirically by examining local realities, comparatively or not, and drawing lessons from their analyses. However, it should be underlined that approximately 35% of the papers focus on theoretical proposals. This shows a potential over-theorisation in the field.⁵

Figure 3 presents the distribution of the empirical focus of the studies. Most of them are included in the evidence-based category. Fewer studies focus on the actors and processes of policy advising, followed by studies interested in the dissemination of knowledge from those who produce it and, finally, articles that focus on data-driven processes.

Table 3 presents the distribution of studies among policy sectors according to their empirical focus. The first group of studies focuses on the environmental sector, while studies on healthcare policy rank third. Furthermore, it must be noted that the last 18 policy sectors (from Development) cover only 19% of the total sample. This confirms that there is a deep, asymmetric concentration of studies on the role of knowledge in a few policy sectors.

Table 4 shows instead that just under half of the papers examined in this metaview (46%) focus on the analysis of policy processes from a theoretical or empirical point of view. As shown in Appendix I (Tables D1, D2, and D3), moreover, this distribution has not particularly changed over time, with the only exception being the prescriptive papers, whose percentage has more than halved relative to the first analysed time period.

Table 4 shows how indeed more than half of the selected literature is devoted to proposing prescriptions (telling policymakers the extent to which they should use knowledge) or, on the technical side of the use of knowledge, to analysing or suggesting tools and models through which knowledge is or should be processed to make it usable. Overall, this part of the literature is interesting from a technical point of view because it offers a broad perspective of (1) how to perform and use systematic reviews to make actual knowledge usable for policymakers; (2) how to use RCTs or design ex-ante assessments or modelling frameworks against policy uncertainty; and (3) evaluative systems through which policymakers can forecast or monitor the effectiveness of their decisions. These are all interesting dimensions in relation to knowledge utilisation that could be useful for policy scholars, especially for those interested in a prescriptive policy design perspective. However, they do not deal directly with the practical problem of the use of knowledge, which concerns whether and how it can have a key role in policymaking and which conditions can favour or disfavour this role. Thus, there is much to learn from the literature, but its prevalence in quantitative terms also shows that many scholars in different disciplinary fields are focused more on how to produce potentially usable knowledge than on understanding the conditions under which the existing knowledge is used in policymaking. The difference between these two goals is that the former mainly concerns the qualities inherent in the produced knowledge, while the latter mainly concerns the components of policymaking and how they interact with the knowledge input.

Table 5 presents the distribution of barriers and facilitators among the five analytical goal categories. Most of the facilitators are identified as an outcome of analyses that focus on the technical features of knowledge generation and use, followed by those provided in relation to the analyses of policy processes. Moreover, it is interesting that both policy process analyses and analytical framework proposals, which will be the focus of our in-depth examination of single barriers and facilitators in Sect. 4.2, display a trend inversion compared to the general predominance of facilitators: when analysing policy processes or proposing frameworks to that aim, barriers prevail. These data already suggest that most problematic dimensions are inherent in policy processes.

The data presented above are rich and could be discussed from different points of view. Here, we focus on those that are more relevant from a public policy perspective.

First, the field of knowledge utilisation is elusive and slippery, including in metareviews. It is relevant to emphasise that the potential results depend on the research questions and thus on the keyword search. We used keywords that would allow a wide-mesh sieve. Thus, we could include relevant items that could have been excluded with more specific targeting. In this way, it was possible to build a detailed map of how knowledge utilisation is treated in various streams of literature and thus to determine how much of this literature must be considered of partial, if not marginal, interest for policy scholars.

The different focus adopted by this metareview—based on the distinction between objective perspective (knowledge for policymaking) and subjective perspective (knowledge in policymaking) and the consequent choice to privilege the latter for deeper analysis—shows different results in terms of barriers and facilitators compared to those of previous studies as presented in Sect. 2.2, mostly belonging to the objective perspective and thus emphasising those factors that constrain or facilitate the use of knowledge (considered external to the policy process): clarity of evidence and proper channels for knowledge translation.

By adopting a subjective perspective, we have assumed that barriers and facilitators are not objective elements in policymaking but that the constitutive components of policymaking could constrain and facilitate the use of knowledge according to the context (and thus also to the researchers’ theoretical approach).

Being aware of these assumptions, it should be noted that, according to the picture emerging from the papers belonging to the subjective perspective, values/ideology/beliefs, problems regarding the relationships between actors and (low) policy capacities emerge as pivotal factors constraining the proper use of knowledge (see Table 6). With regard to the first mentioned barrier (values/ideology/beliefs), it was noted, for instance, that “[i]ndividuals and groups with contrasting belief systems interpret and use evidence differently—not just because they are actively pursuing particular policy results, but because of how existing beliefs, motivations, and values structure the cognitive processes through which evidence is understood and applied” (Parkhurst, 2016: 384). In terms of actors’ relations, some of the reviewed literature has detected difficulties in science communication to nonscientists. For instance, it has been pointed out that “[t]he rising number of involved actors in communication on the science-policy interface raises new challenges when speaking about quality: scholarly information has to be understood by politicians and the general public (who) do not necessarily engage in scholarly activities themselves” (Sokolovska et al., 2019: 5). The difficulties in actors’ relations, as emerged from our metareview, are twofold and mainly caused by the differences between the academic and policymaking communities, as “academics have difficulty communicating research in an applicable manner, and policymakers, in turn, tend to focus on operational motivations” (Wilkinson et al., 2017: 88). Finally, in terms of policy capacity, our reviewed literature shows that its low level is a key constraint to improving evidence-based policymaking, with an example of this being found in the short-term focus of the actors involved in policy analysis (Howlett, 2009).

Thus, according to our reviewed literature, ideational factors and a lack of trained administrators of appropriate organisations seem to matter more than the type of evidence available. The skills of policymakers are important, but they should be included in the more general capacities (individual, organisational, systemic) that characterise a specific policy setting, the government and its bureaucracy. Furthermore, when relations between actors are analysed, there is an emphasis not only on the interaction between scientists and policymakers but also on the relevance of other stakeholders, such as the community, specific groups of citizens and vested interests. For instance, in the case of water governance in the city of Ahmedabad, one of our reviewed items identified insufficient stakeholder involvement as a factor contributing to explaining the weak linkage between research and policy and, conversely, stakeholder involvement broadening as a factor that might facilitate “a long-term joint knowledge production process” (Aartsen et al., 2018: 2455).

Finally, we find that financial issues do not stand out in the selected sample. This indicates that the problem of the impact of knowledge utilisation is not related to the existence of sufficient evidence (whose lack would justify the consideration of funding as a barrier) but to other factors.

Regarding facilitators, this metareview clearly provides a very different picture than the literature based on other reviews, which are very often based on an objective perspective (cf. Section 2). On the one hand, this work reinforces in a detailed way those perspectives that call for better governance of evidence (Parkhurst, 2017). In fact, what emerges is not only that in the concrete dynamics of policymaking, the triggers to improve the role of knowledge involve factors related to policy capacities and actors’ relations but also that some specific factors could matter more than others, such as analytical capacities, the direct involvement of citizens and local stakeholders and coproduction. The importance of analytical capacities in policymaking is supported by scholars who state that the policymaking process needs and produces knowledge and that during this process, some (analytical) questions must be answered, such as “Which aspects of the problem should be considered more important? What information is relevant and should be used? Who are the principal stakeholders? Who else is affected by the situation and the possible policies? Which resources are allocated, where how and when? What matters in terms of potential consequences?” (De Marchi et al., 2016: 23). To answer these questions, policymakers need analytical capacities. These capacities are also key elements of co-production practices, which are, in turn, functional to the use of knowledge in policymaking. Our reviewed literature also outlines other elements needed for these practices to work well. Along with the development of capacity and social capital, these elements include the active participation of knowledge users (e.g. policy implementation actors such as clinicians) in the creation of knowledge, the equality of researchers and policy actors in the conduct and application of research, the reciprocity of benefits between researchers and policy actors, and the setup of infrastructures that incentivise coproduction relations (Heaton et al., 2015: 1489). Finally, some of the literature we reviewed argues that public (citizens’) participation should be reconsidered, not only to solicit valuable information from key stakeholders but also to leverage participation itself as empirical evidence. This reviewed literature identifies practical rules for conducting meetings with citizens, such as recording them to avoid losing important information, as well as the benefits of reconsidering public engagement as research, which might stem from improving science communication to broad audiences and demonstrating to citizens that policy decisions are made in consideration of their input (Hall et al., 2016). Again, however, such efforts to employ public engagement as evidence require adequate analytical capacities.

A significant element emerging from the analysis of barriers and facilitators is the asymmetric role of ideas/values and ideology: this necessary, and inescapable, component of policymaking acts more as a barrier than as a facilitator. It must be noted that values and beliefs can be only partially manipulated, as the Advocacy Coalition Framework convincingly underlines (Sabatier, 1993; Weible & Jenkins-Smith, 2016). Thus, this asymmetry could be justified exactly because values and ideas are less changeable or manipulable in policy dynamics than other dimensions such as actors’ relationships and policy capacities. However, it could be clearly observed that the main elements of policymaking do not act in isolation and thus that this asymmetry could depend on the theoretical/analytical perspective of the authors but also on the specific policy fields analysed or on the high level of politicisation of the treated case.

Thus, this asymmetric role of ideas/values and ideology cannot be taken for granted, but the reviewed literature gives this general signal that is reinforced by other evidence coming from the analysis. In fact, for example, from a focused analysis of our final sample of 224 papers, it seems that ideational maps that hamper the use of knowledge in policymaking—for instance, international assessment results in the field of education—could be overcome by means of enhanced policy capacity, such as in the form of an improvement in strategic leadership (De Lisle et al. 2014). At the same time, and more representatively in terms of the literature reviewed, divergences in values, priorities and beliefs can be surmounted by means of enhanced actor relations. For instance, the pragmatic “forma mentis” of many policymakers in the healthcare sector, who tend to look mainly at the knowledge that they find immediately usable—which can represent an obstacle to the absorption of knowledge that they do not perceive as such—can be overcome by means of improved relationships between actors, where researchers proactively learn more about policymakers’ priorities and seek to develop mutually beneficial relations with them (Wye et al., 2015). The same happens when the priorities, ideas and/or values and knowledge systems of policymakers and researchers are not the same, as is sometimes the case in the field of environmental protection. This ideational difference is indeed surmountable, according to the reviewed literature, through preemptive behaviour on the part of researchers who take action to know in advance what these priorities are (Karam-Gemael et al., 2018); by means of procedures and practices that unlike the search for consensus that is the prevalent orientation in some fora (e.g. the Intergovernmental Science–Policy Platform on Biodiversity and Ecosystem Services—IPBES) are able to accommodate pluralism, contestation and distinct perspectives and knowledge systems (Díaz-Reviriego et al., 2019); by means of collaborative, resource-intensive co-learning arrangements between policymakers and scientists that are properly organised in terms of governance (Morgan, 2017); or even by attempting at co-production, which some of the reviewed literature presents as an important practical arrangement of the relationships between actors—although not as a panacea (Sylvester & Brooks, 2020).

This evidence invites us to take seriously the hypothesis that knowledge production should be appropriate and localised (Parkhurst, 2017) and that policy capacities could also be a pivotal resource for any kind of policy process (Wu et al., 2018). Overall, the present metareview on this point indicates that to increase the likelihood of a more effective role of knowledge in policymaking, inclusive policymaking processes should be introduced, bureaucrats should be better trained and the organisation’s capacity to collect and analyse data should be increased. Again, this evidence may depend on the characteristics of the case studies analysed by the reviewed literature and thus should not be taken for granted. However, these findings indicate a path for further analysis and in-depth empirical research.

The third relevant point that emerges from our metareview is that three constitutive elements of policymaking (values/ideology/beliefs, actors’ relations and policy capacities) prevail as barriers and/or facilitators, while the other three categories (including the factors considered more relevant by the other sectoral reviews as summarised in Sect. 2.2) are residual or relevant in only a few policy fields. The limited number of papers found and scrutinised on these aspects indicates the small number of interested scholars working on them. The question is thus raised regarding why interests, institutional and processual arrangements and types of evidence emerge as less relevant in favouring or constraining the use of knowledge in policymaking. While the low relevance of the type of evidence in policy dynamics can be related to the fact that, according to a subjective perspective, it has a high chance of being an intrinsically weak component of policymaking due to the intensity and strength of the others, it is less easy to justify the low relevance of the other two components. Obviously, the usual disclaimer can be applied, i.e. the specific subjective perspective of the reviewed works: for example, it could be the case that the authors have not clearly distinguished the ideas and interests in their research or have focused more on the institutionalised actors’ interactions and less on the institutional arrangements in which these interactions happen. This could also depend on similar mistakes in coding the data. However, the evidence emerging in the review significantly indicates this difference among the six components of policymaking, and this is something that should be addressed by further research. If corroborated by further empirical investigation, this difference could represent a significant “discovery” with respect to the determinants of the use of knowledge in policymaking.

The fourth relevant point is why there are so many sectors in which there is insufficient analytical attention to the role of knowledge in policymaking. We counted 18 policy sectors that are clearly underrepresented in the sample, while constant attention was given over time to sectors such as environmental and health policy. Sectors such as labour, social work, public administration reform, transport, energy, higher education, urban development and migration emerge as uninteresting to scholars who are interested in the policy use of knowledge. This may be due to the keywords we used or the fact that the role of knowledge is less relevant in policy fields where it is more difficult to objectify it, which seems to be the case for health, the environment and climate, where the bulk of knowledge is apparently more homogeneous and internally more shared by scientists. Thus, this underrepresentation may be a consequence of the more disputable nature of the issues at stake and of the different theoretical approaches and empirical evidence coming from scholars. However, it must be emphasised that many scholars of public policy are more interested in the dynamics of policymaking, whereas the different types of knowledge utilisation are usually treated as one of the possible factors influencing policymaking. Among the various literature streams in public policy that pay attention to the role of knowledge in policymaking, the following can be noted:

The richness of these approaches in public policy, therefore, could justify the fact that so many policy fields lack consistent literature on knowledge utilisation: these approaches consider knowledge as an element of their designed policy setting, even if the role of knowledge is not directly enlightened as pivotal.

At the same time, this richness calls for a higher integration with studies specifically devoted to analysing knowledge utilisation in policymaking. In fact, on the one hand, these studies, such as the public policy streams of research listed above, share the common subjective perspective that assumes the complexity of policymaking as an inescapable feature and consequently that knowledge is not a phenomenon that can be considered external to the policy dynamics themselves. On the other hand, this metareview clearly shows that the three fundamental components of policymaking (values/ideology/beliefs, policy capacities, actors’ relations) that emerge as more relevant in constraining and/or facilitating knowledge in policymaking are exactly those on which there is a flourishing public policy literature. Greater integration would help better clarify the drivers and activated mechanisms through which this set of factors can constrain or facilitate a better use of knowledge and thus find and potentially design possible solutions to make knowledge more relevant in policymaking. In addition, overall, due to the emerging relevance of policy capacities and actors’ relations, this greater integration would help demonstrate ways to better design actors’ relations and improve policy capacities to eventually make the role of knowledge in policymaking more effective.