Skip to main content
Download PDF
Download ePub
  • Systematic review
  • Open access
  • Published:

Health systems guidance appraisal—a critical interpretive synthesis

Implementation Science volume 11, Article number: 9 (2015) Cite this article

Abstract

Background

Health systems guidance (HSG) are systematically developed statements that assist with decisions about options for addressing health systems challenges, including related changes in health systems arrangements. However, the development, appraisal, and reporting of HSG poses unique conceptual and methodological challenges related to the varied types of evidence that are relevant, the complexity of health systems, and the pre-eminence of contextual factors. To address this gap, we are conducting a program of research that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting. The focus of this paper was to conduct a knowledge synthesis of the published and grey literatures to determine quality criteria (concepts) relevant for this process.

Methods

We applied a critical interpretive synthesis (CIS) approach to knowledge synthesis that enabled an iterative, flexible, and dynamic analysis of diverse bodies of literature in order to generate a candidate list of concepts that will constitute the foundational components of the HSG tool. Using our review questions as compasses, we were able to guide the search strategy to look for papers based on their potential relevance to HSG appraisal, development, and reporting. The search strategy included various electronic databases and sources, subject-specific journals, conference abstracts, research reports, book chapters, unpublished data, dissertations, and policy documents. Screening the papers and data extraction was completed independently and in duplicate, and a narrative approach to data synthesis was executed.

Results

We identified 43 papers that met eligibility criteria. No existing review was found on this topic, and no HSG appraisal tool was identified. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to its development/reporting. We identified 30 concepts that may be relevant to the appraisal of HSG and were able to cluster them into three meaningful domains: process principles, content, and context principles.

Conclusions

Our study showed the role that the quality criteria play in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between the various concepts in the three domains.

Peer Review reports

Background

Roemer [1] defined a health system as the combination of resources, organizations, and financing and management bodies that culminate in the delivery of health services to a population. A redefinition of the term health system was proposed by the World Health Organization [2] to consist of all organizations, people, actions, and activities whose primary purpose is to promote, restore, and maintain health. This was further modified—to be more comprehensive and explicit—by a group convened by the World Health Organization’s (WHO) Alliance for Health Policy and Systems Research, to refer to delivery and financial and governance arrangements for health care and population health services and the broader context in which they are negotiated, implemented, and reformed [3].

Health systems guidance (HSG) are systematically developed statements produced at global, national, and regional levels to assist with decisions about appropriate options for addressing health systems challenges (including related changes in health systems arrangements), the implementation of these options, and the monitoring and evaluation of the implementation efforts [4]. The need to develop and use evidence-informed approaches to address national and global health system issues through the use of HSG has been articulated by many [5]. For example, the Millennium Development Goals (MDGs) reflect eight international goals (e.g., reduce child mortality, improve maternal health, and combat HIV/AIDS, malaria, and other diseases) and were developed by the United Nations in 2000. However, their attainment has been hindered by weak health systems and lack of system-specific recommendations [6, 7]. Formulating recommendations to address root causes has the potential to clear the path toward better health outcomes in line with health goals [8].

The terms guidelines and guidance both refer to any document containing a recommendation on a course of action. However, in contrast to the word “guidelines,” the term “guidance” is used in the health system context in order to make explicit the difference between the process of supporting evidence-informed judgments for health system issues from that of clinical judgments as would be expected with clinical practice guidelines (CPGs). As a policy-oriented product, HSG represents the whole body of knowledge that informs policy decisions on how health system issues should be classified or prioritized, appropriate health system governance, optimal financial arrangements, system organization, and the design and delivery of effective health programs and services [9–12].

The main bodies tasked with developing HSG are international and intergovernmental organizations (e.g., WHO, Pan American Health Organization), local ministries of health, and special national committees or agencies providing support to ministries of health. Their production is usually linked with these high-level health sector entities and less frequently in decentralized structures at the sub-national level. HSG may be aligned with specific goal-oriented taxonomies (e.g., WHO building blocks), unique national/regional governance, or tailored to financial/delivery arrangements with the common goal to optimize health care systems. However, while these organizations have demonstrated an interest in the development, appraisal, and reporting of this type of guidance as described in the WHO’s Handbook for Supporting the Development of Health Systems Guidance [13], for example, many of their members and collaborators (who are also members of the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) team) contend that these intensions have not been matched by adequate action or results, in part, due to the lack of experience in developing HSG.

Indeed, in comparison to CPGs, the development of HSG pose different conceptual and methodological challenges related to the varied types of evidence that are relevant, valued, and considered; the complexity of health systems’ internal and external relations that must be addressed; and the pre-eminence of contextual factors that directly influence the design and adoptability of recommendations [4]. As with CPGs, HSG statements or recommendations should be justified by assessments of the quality of evidence supporting them, the potential for unintended consequences, and by discussions of implementation and contextual issues. For example, the full implementation of HSG recommendations can be further hampered by “bottlenecks” like health system fragmentation and capacity limitations; these limitations should also be addressed in the guidance documents [14, 15].

The potential for positive impacts by HSG as a decision tool to improve health systems is great. However, as with CPGs [16], this potential is only as good as the quality of the HSG. Indeed, the ability to impact and optimize health system performance and efficiency through the development and adoption of HSG is hampered by the dearth of tools to guide their development, appraisal, and reporting. As a consequence, this leaves knowledge users at a loss when choosing the highest quality and/or most appropriate guidance or in creating new guidance in circumstances where there is none or navigating circumstances where the existing guidance is not credible or of poor quality. In contrast to the development, appraisal, and reporting methods for CPGs, the development of HSG is still at a rudimentary stage [4]. There has been some work related to the use of “evidence briefs” to assist policymakers and stakeholders with working through a health system problem, options for addressing it, and key implementation considerations, informed by the best available data and research evidence [17]. However, there is a need for systematically and transparently developed guidance that can feed into such context-specific documents. With the rising trends that encourage bridging the gap between research and policy and practice, this is a significant research gap in both the science and practice of knowledge translation. HSG can provide this bridge between research synthesis and policy needs for evidence.

The creation of high-quality HSG requires tools to support the development and reporting of high-quality guidance and tools able to differentiate between high- and low-quality reports. There is also value in establishing acceptable quality thresholds and creating common methodologies and nomenclature among the HSG community (developers, users, and researchers). At present, however, there is no universally acceptable gold standard approach for appraising HSG, although there are some tools (for example the Handbook for Supporting the Development of Health System Guidance) to support their development [13] and reporting. To address these gaps, we are conducting a program of research with the international health systems guidance community that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting.

The first step to this program of research, and the focus of this paper, was to conduct a knowledge synthesis of the published and grey literatures to determine concepts (items, criteria, or domains) related to HSG development, reporting, and quality. The results of the synthesis are to provide the foundational components of our HSG tool and to serve as a conceptual status report for the research community.

Methods

With the goal of generating a list of concepts that can be used to develop, appraise, and report HSG, we conducted an initial search to identify existing concepts, tools, templates, or checklists that have been used or could be used to describe, differentiate, or appraise the quality of HSG. The aim of this initial search was to determine if reviews on this topic exist, estimate an initial number of relevant papers available, highlight additional and useful search terms, and clarify inclusion and exclusion criteria and the most appropriate knowledge synthesis approach [18, 19].

After contemplating the variety of knowledge synthesis approaches available, a critical interpretive synthesis (CIS) approach was considered the most appropriate for three reasons. First, CIS is a systematic approach that facilitates the analysis of complex and diverse bodies of literature including qualitative, quantitative, and theoretical papers [20–26]. The available literature on HSG is highly heterogeneous and methodologically diverse, comprising of a mix of empirical qualitative and quantitative papers and non-empirical papers. A wider range of evidence and study designs are typical occurrences in health systems/health policy research and HSG is a nascent domain where good thinking is likely to be captured in expert opinion/views, editorial comment, policy documents, political statements, experiences of stakeholders, theoretical/discussion papers, and other colloquial forms of evidence. In contrast to CIS, conventional systematic reviews, for example, have been criticized for excluding forms of evidence traditionally considered as non-experimental [20, 27–30].

Second, the objective of a CIS is to develop new concepts and theories through a typically interpretive mode of inquiry. This is in contrast to more conventional systematic review approaches where the mode of inquiry is more aggregative and aimed at testing theories by collating, compiling, pooling, and summarizing common outcomes across a range of studies [19–21, 25–27, 31, 32]. Since our review aims at generating a candidate list of items, criteria, or domains for HSG development, appraisal, and reporting, this theory-generating approach that CIS promotes was deemed a good fit.

Third, CIS offers a more flexible, iterative, dynamic, and reflective approach requiring investigators to assess the extent to which new information or data are provided with each additional paper considered. It applies a relatively loosely defined set of processes for critically analyzing and synthesizing literature [19, 20, 23, 27, 28, 30]. This differs from the more conventional systematic review methods which have clearly stated study protocols, an exhaustive search of all available literature, standardized data extraction templates, and explicit quality appraisal checklists, as well as pre-determined focused questions, strict inclusion and exclusion criteria, and specified data boundaries [20, 23, 27–30].

With the unique conceptual and methodological challenges related to the development of HSG [4], it was essential to have a review method that was iterative, flexible, and dynamic. Our review was not simply aimed at summarizing this complex literature on HSG, so we sought to produce a logical and insightful interpretation of a purposefully sampled body of evidence. As such, the aim was to yield a comprehensive interpretation without requiring exhaustive identification of relevant items of literature as in a conventional systematic review [20, 29, 30].

Review question(s)

As per CIS methodological standards, our review questions served as compasses rather than anchors [20, 29, 33] allowing for the concepts of HSG to be derived from synthesis of the literature and constantly modifying them in an iterative manner throughout the review.

Our guiding review questions were as follows:

  1. 1.

    How have authors variously defined HSG quality in the literature?

  2. 2.

    How have authors interpreted and used criteria (or tools or instruments, checklists, systems, etc.): to describe or define HSG quality or reporting requirements, to appraise HSG quality, or to differentiate between HSG on the basis of quality?

  3. 3.

    What methods have authors and stakeholders used to develop these criteria?

  4. 4.

    What methods have authors and stakeholders used to address health system issues/challenges?

Literature search

In searching the literature, our goal was to select papers based on their potential relevance to HSG appraisal and quality, while including other papers that, though not directly relevant to health systems, were deemed important for the purpose of our review. Instead of including an exhaustive number of papers, our plan was to provide a comprehensive sampling frame of potentially relevant papers using emergent eligibility criteria [25, 32]. As per the standards of a CIS, the boundaries of our inclusion and exclusion criteria were modifiable, dynamic, and continuously shifting [20, 32, 34]. The goal was to populate the concept of HSG with new concepts rather than finding papers that reiterated ideas already captured in previously reviewed papers.

Eligibility criteria

  1. 1.

    Study content: (a) Papers that evaluated HSG or papers that report on criteria/tools that have been documented as important indicators of HSG quality and (b) papers that reported on methods for addressing health system issues/challenges.

  2. 2.

    Time frame: Papers are eligible if published in or after the year 2000 (when the first World Health Report on health systems [2] was published).

  3. 3.

    Context: Unrestricted. We sought papers that considered HSG evaluation and health system issues/challenges in various contexts (low-, middle-, and high-income countries).

  4. 4.

    Study design: Any study design.

  5. 5.

    Language: English, French, and Spanish

A combination of key and free text terms were used to search through various databases. Terms used were health systems, health policy, guidelines, guidance, health services arrangement, health services organization, health system issues, health system challenges, tools, instruments, criteria, items, domains, evaluation, appraisal, quality, and standards.

The search strategy included various electronic databases and sources: Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Library, Excerpta Medica dataBASE (EMBASE), Google Scholar, Health Systems Evidence, Latin American and Caribbean Health Sciences Literature (LILACS), PubMed, Virtual Health Library, and Web of Science. We also searched subject-specific as well as regional electronic sources: Australia’s National Health and Medical Research Council, Evidence Best Practices for Public Health, University of Massachusetts National Guidelines Clearinghouses, National Institute of Health and Care Excellence (NICE), WHO’s Evidence Informed Policy Network (EVIPNet), WHO EURO’s Health Evidence Network, and Guidelines International Network (GIN) directories.

We also searched for other grey literature, including conference abstracts (Global Symposium on Health Systems Research, Canadian Association for Health Services and Policy Research, Health Systems and Process Improvement Conference, Canada’s Health Leadership Conference, Health Systems Reform in Asia Conference, International Society on Priorities in Health Care). Further, research reports, book chapters, unpublished data, dissertations, and policy documents that were nominated by members of the team or found in unique holdings of health sciences libraries in Canada were also included. Additional papers were identified by manually searching bibliographies, while more were obtained by hand searching some key journals (i.e., Health Policy and Planning, Health Services Research and Policy, Health Research Policy and Systems, Global Healthcare Systems, Health Systems and Reform, and Health Policy). This was a complementary search strategy to account for papers not included in electronic databases or with search terms that do not allow them to be easily identified [34, 35].

Finally, we also contacted (through emails, phone, Skype or in person) experts, colleagues, and members of our research team with a known interest on this topic to identify additional papers (published, unpublished, or ongoing). To increase the scope of these key informants, we also asked initial contacts to refer us to others who can provide more information. Relevant papers were then imported to Endnote bibliographic software.

A two-step sampling process was used. First, a purposeful sampling approach [36, 37] was used to investigate the literature to determine the range of unique candidate concepts—or theoretical domains—associated with HSG. We stopped sampling at the saturation point where looking at new literature no longer contributed additional concepts [25, 37]. Second, a theoretical sampling strategy was used to interrogate the literature relevant to each of the identified concepts. Theoretical sampling does not occur at a single point in the research process but is a recurrent feature, which aligns it well with the dynamic CIS methodology. Similar to the first approach, we stopped sampling at the saturation point where looking at new literature no longer contributed additional descriptions of the identified concepts. The intent here was to thoroughly capture the depth of the concept across the literature in order to generate and develop the theoretical underpinnings of the ideas rather than collecting numerous citations of identical concepts and/or descriptions. Screening titles, abstracts, and full text was completed independently and in duplicates (DAA and SA). Disagreements were resolved by consensus.

Quality appraisal

We did not consider it appropriate to undertake quality appraisal for three principal reasons: firstly, the literature on HSG is diverse and complex; secondly, the available literature could not be hierarchically ordered in terms of study design, importance, or relevance; and thirdly, very little consensus exists on how or whether to perform quality appraisal in a CIS methodology [27, 32, 38, 39]. Instead, we chose to evaluate papers for inclusion based on a judgment of their relevance and likely contribution to concept development and theory [23, 26, 32, 40]. Our assumption was that some methodologically weak papers are theoretically and conceptually pertinent [25, 29, 34]. The goal of our review was to generate key concepts relevant to the appraisal of HSG, so we applied a flexible relevance boundary in order to be as inclusive and comprehensive as possible and include papers that could contribute to this theory generation.

Data extraction and analysis

As a result of the nature of the data, we did not use a standardized data extraction template to retrieve the relevant elements of this review from all the retained papers. Instead, and as per CIS methodology, we gave latitude for a more narrative data retrieval approach. It was, however, possible to extract some key elements from all the papers including author(s), the year of publication, geographical location of the study or affiliation of the author, purpose of the study/paper, its relevance and a summary of its main conceptual contributions, and types of items, criteria, or domains considered. Data were extracted independently and in duplicate (DAA and SA) with disagreements resolved via consensus. The extracted data were then compiled in summary form.

Again due to the anticipated diversity of methodological designs and implementation of the different items, criteria, and domains that were extracted from the papers, a narrative approach to data synthesis was executed. The data extracted were sorted and categorized into groups with themes. We sought to collect similar concepts within the same group. This systematic method of recording themes, and making connections between themes and the data collected within a comprehensive category system [41, 42], is an advocated concept-development coding approach [43]. The information was imported into the analysis software NVivo version 9 in order to analyze groups of themes that depicted similar patterns in information. We did not organize the concepts in any hierarchical order but reported the concepts that were identified in each of the papers in this review.

Modeling from the health policy analysis triangle framework that was developed by Walt and Gilson [44], we were able to connect the concepts (constructs) together into meaningful domains. The framework considers all the essential elements that interact to shape policymaking by demonstrating that health policy should focus on the processes contingent on developing and implementing change, on the content of health policy reform, and on the context within which the policy is promulgated, as well as on the actors involved in the policy reform [44, 45]. The health policy analysis triangle framework is a highly simplified model of an extremely complex set of interrelationships between the different elements of the model (process, content, context, and actors) with each element influencing or being influenced by the other [45, 46]. Development, appraisal, and reporting of HSG play an important role in health policy making by providing options and recommendations to address a health systems issue. Therefore, similar to health policymaking, HSG development, appraisal, and reporting can be seen to occur in a series of discrete yet interconnected components of process, content, and context.

Results and discussion

No existing knowledge synthesis was found on the topic, and no existing HSG appraisal tool (draft or final version) was identified. We identified a total of 43 papers that met our eligibility criteria and reported on concepts (items, criteria, domains) considered directly or conceptually relevant to HSG and/or their quality (see Fig. 1 for a flowchart of the study selection).

Fig. 1
figure 1

Flowchart of study selection

Full size image

Forty-seven percent of the retained studies were technical reports, 32 % were concept papers, 13 % were quantitative studies, and 8 % used a mixed methods approach. The authors held affiliations at the following organizations: 33 % at universities, 25 % at the WHO, 23 % at research institutes, 12 % at government agencies (United Kingdom’s National Health Services [NHS], United States’ Centre for Disease Control [CDC], United States Agency for International Development [USAID], United Kingdom’s National Institute for Health and Care Excellence [NICE]), and 7 % at national ministries of health. Fifty-eight percent (58 %) of the papers were lead authored by an individual based in Europe, 30 % in North America, 8 % in Australia, 2 % in Asia, and 2 % in Africa. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to development and reporting of HSG. Thirty concepts were identified that are considered to be a good fit for, and may be relevant to, the development, appraisal, and reporting of HSG. The Appendix provides a description of the papers, their objectives, relevance to the HSG process, and the concepts extracted.

Process principles, content, and context principles

Ostrom [47] stated that identifying key concepts and relationships among them culminates in the generation of a theory, which connects these key concepts. We were able to organize the concepts and identify relationships within and across them. Through an iterative process, we clustered the concepts together into three meaningful domains: process principles, content, and context principles.

Process principles

Process principles represent the methodological elements and the defining principles that demonstrate the development integrity of the HSG. They refer to the ways in which the HSG is initiated, developed, and formulated and can be looked upon as the “who” and “how” of the guidance [44, 46]. Here, we find the procedures and principles that were employed by the HSG developers in coming up with the guidance recommendations. Process principles are critical in the development of HSG because the methodological strategy is crucial for guidance appraisal and for differentiating across HSG of varying quality. They are also important because they can articulate the tactics that could be utilized in the development and reporting of HSG to optimize quality. Process principles also depict the subjective and/or objective belief systems in place that represent the preferences (what ought to be) of individuals, groups, or populations on the course(s) of action for addressing a health systems challenge. Process principles also refer to how the guidance is made public in a way that is consistent and comparative as this can facilitate their comprehension, which will further enhance their uptake and aid in easing their application. Therefore, when developing, appraising, or reporting HSG, a conceptual understanding of the process is fundamental. Table 1 shows concepts for HSG process principles.

Table 1 Process principles
Full size table

We were able to point out patterns and draw linkages between the process principles concepts. We offer five examples here. First, guidance should be relevant to and developed for health system areas with a clearly demonstrated and documented need, and the feedback of appropriate stakeholders will further highlight priority areas, which may potentially lead to timely interventions. Second, systematic and transparent approaches have to be applied to search for and identify relevant evidence, which should also be available in a timely manner. Third, ensuring that stakeholders from all applicable health system and sub-system levels are involved in the HSG process, that relevant evidence is sought and that appropriate outcomes are chosen can enhance the comprehensiveness of the guidance recommendations. Fourth, having clearly defined and consistent outcomes and declaring and managing interests as well as engaging in a participatory approach that incorporates the various perspectives of multiple stakeholders is also important for transparency of HSG. Finally, the ethical lens applied will be impacted by the quality of the evidence available, the composition of the involved stakeholders, the outcomes/indicators selected, the health system levels/sector involved, and whether it is a priority area that is relevant to the setting.

Content

Content (the “what” of the HSG) represents the topics, subjects, and substance that demonstrate the content integrity of the guidance recommendations [44, 46]. Here, we find statements about the impetus of the endeavor that provides direction on the HSG objectives and goals. Keeping in mind that in the absence of an appropriate economic lens, even the best-designed HSG may not achieve adequate success; we also find here the myriad of economic factors that come with the HSG process and need to be taken into account. In addition, within the content of the HSG, it is also important to consider and reflect on the fact that not all guidance recommendations go into practice as planned and events may occur that may derail intended actions. We also find here assessment considerations, usually ongoing throughout the HSG process, and which refer to those elements that assist in determining whether the guidance process was properly followed and/or records the impact/outcomes of the HSG. The content of the HSG also articulates the operationalization of the proposed recommendations and carries information on how to best reach the target users of the guidance. Table 2 shows concepts for HSG content.

Table 2 Content
Full size table

As with the process principles, we were also able to point out patterns and draw linkages between the content concepts. For example, the way the problem is defined can provide direction on the costs and resources required for implementation of the solution, reveal some potential unintended consequences, and guide the operational plan. The operational plan can also provide hints on some potential unintended consequences. A clear problem definition will also inform the process and outcomes/impact evaluation of the HSG. Designing the operational options can inform the costs and resources required. The costs, resources, and operational plan will in turn influence whether the HSG will be effective and/or cost-effective. The updating plan is also contingent on the health system issue being addressed, the effectiveness of the guidance recommendation, and the resources required.

Context principles

Context principles represent those systemic factors like the local technical, situational, structural, institutional political, and socio-cultural components of the health policy environment that can impact HSG recommendations [44, 46]. Like health policymaking, HSG does not occur in a vacuum so it is important to pay attention to the variety of contextual factors that may have significant impact on how the guidance is developed, adapted, and implemented by the end users [48, 49]. The process principles and the content of the HSG all have to be contextualized; therefore, context principles impact these other clusters of the concepts. Context principles refer to the usability in context, and here, we find concepts related to those system components that make up the setting within which the HSG is to be used. A clear overview of the context or setting that will be impacted by the guidance recommendations is essential. This will provide an understanding of why some guidance recommendations may work in some settings and not others. We find here factors that will enhance and facilitate the adherence to HSG recommendations as per protocol. We also find here factors that represent the values and moral fabric of the society and how this can either facilitate or impede the HSG recommendations. Table 3 shows concepts for HSG context principles.

Table 3 Context principles
Full size table

We were able to point out patterns and draw linkages between the context principle concepts as well. We offer some examples here. For HSG to be feasible and sustainable, it should be affordable and resonate with local values (political and socio-cultural). Information on affordability, feasibility, and sustainability may also determine whether the guidance recommendations can be transferable to other settings. Providing socio-culturally appropriate recommendations may also indicate which other health systems can adapt and use the HSG. Also, the HSG should be flexible enough to accommodate shifts in values (e.g., political, socio-cultural). Additionally, HSG that reflects the socio-cultural preferences of the target population, that is affordable, feasible, sustainable, and transferable to comparable settings, can amass support from politicians. The external factors that originate from other institutional systems may impact the feasibility and sustainability of the guidance and may be influenced by the political and socio-cultural climate.

Synthesis across domains

As is expected of a Critical Interpretive Synthesis (CIS), we were able to point out relationships between the concepts across the three clusters as shown in Fig. 2. We highlight some relationships:

Fig. 2
figure 2

Framework of health systems guidance concepts

Full size image

  • ➢ Uptake of guidance can be enhanced if it is addressing a priority area for which evidence-based reports of effectiveness and cost-effectiveness exist.

  • ➢ Consultations with appropriate stakeholders are also crucial for feasibility of implementation and sustainability of the HSG because sometimes, the individuals tasked with implementing guidance recommendations may not be committed to them and this may influence adherence. Their input into the HSG process and support for the recommendations may alleviate this concern.

  • ➢ HSG outcome(s) chosen will influence the way the problem is defined and provides information that will be useful for evaluating the HSG.

  • ➢ Including an ethical viewpoint will also impact how the problem is defined and will influence cost-effectiveness thresholds.

  • ➢ Systematic and transparent processes may provide an impetus for donor involvement (affordability).

  • ➢ Information on affordability can determine whether the health system issues can be addressed in a timely manner and inform judgments on how to evaluate the process.

  • ➢ The HSG should be flexible enough to accommodate constantly evolving evidence and changing health system priorities.

  • ➢ The guidance should provide socio-culturally appropriate solutions that are relevant to the applicable levels/sectors of the health system.

  • ➢ HSG that is in line with ethical principles, addresses priority issues, and is timely will be appealing to politicians.

  • ➢ The external alignment of guidance may affect the effectiveness and impact the operational considerations.

  • ➢ The outcomes selected, the system level/sectors involved, or the ethical values in place may expose the external factors that may be pertinent.

  • ➢ Some determinants of generalizability of the guidance are the stakeholders involved, the evidence used, transparency of the process, ethical lens considered, and outcome/indicators selected

Conclusions

This project is the first phase of a multistage approach to create an internationally useful HSG tool, AGREE for Health Systems (AGREE-HS) that will inform the development, reporting, and appraisal of HSG. Modeling after the paradigm to create a tool for clinical practice guidelines [50], our first step was to conduct a review of the published and grey literatures to identify concepts related to HSG quality. In this vein, it was our expectation that the receptiveness, adoption, and diffusion of HSG recommendations depend on the perception of their quality, and with this study, we aimed to identify those core components of good quality HSG.

We found a total of 30 potential HSG appraisal concepts that have the capacity to discriminate between high- and low-quality guidance and direct their development and reporting. We found no existing tools to support HSG appraisal and found few studies describing concepts that were directly tested to appraise HSG. Indeed, the papers we examined reflected a variety of study designs and goals; none reflected methods used to develop appraisal methods. However, the data from our studies show a convergence of ideas in the HSG research community about what constitutes good and useable HSG. Together, these data can provide the foundation of a tool that guides HSG developers in the types of information important to report in a HSG document and tactics for optimal execution. These concepts also address issues of appropriateness and completeness as well as information, which are important components in the uptake of guidance recommendations.

One strength of this study is that we used a sound knowledge synthesis strategy, the critical interpretive synthesis (CIS) approach. We acknowledged the paucity of data related to HSG appraisal, as well as the diversity in the literature sources and types of available data. Also, given that HSG is not a highly bounded topic, we did not rely on a narrow initial research question; instead, we were refining it as the review progressed. The various methodological stages did not proceed as discrete entities as there was a constant to-ing and fro-ing throughout the review and synthesis process; at times, we were concurrently searching, sampling, critiquing, and analyzing. The CIS approach is both systematic and iterative with an interpretive approach to analysis and synthesis of data that allowed us to capture and critically analyze an in-depth depiction of how to differentiate between HSG on the basis of quality, among other considerations. Through CIS methodology, we were able to further identify and include additional papers that were not directly related to HSG but made valuable theoretical contributions to the process of appraising guidance [20, 27].

A weakness of our study is in the nature of the CIS strategy, which exposes us to the risk that certain concepts may have been missed (calling saturation too soon, for example). If we have missed important elements, we anticipate they will be identified in the second stage of our research. Also, the health policy analysis triangle framework is made up of four elements: process, content, context, and actors. But for our analysis, the original framework was modified as we included “actors” in the process principles. Actors (the who of policy) refer to local, national, or international individuals or groups (governmental and non-governmental) involved in the policy process [44, 45]. We found that actors could be incorporated under the process principles concept “participatory.” Additionally and at first glance, it appears that there is the risk that some concepts may lead to potential contradictions. For example, in some jurisdictions, achieving an ethical HSG may not be aligned with achieving a feasible HSG (e.g., in contexts where there is a bias toward citizens because of race or sexual orientation). Currently, the data provides no guidance on how to reconcile a situation like this.

In our analysis, we showed the role that the process principles, content, and context principles plays in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between their various concepts. We pointed out patterns and drew linkages between the concepts in order to show that the concepts and domains do not typically occur in a linear, independent, or discrete manner. The interaction between these three clusters is an imperative consideration because they all influence the guidance process and can facilitate or impede the success of HSG recommendations.

Some of the concepts (criteria or items) and domains identified in this review may not be applicable to every jurisdiction or country so they are not intended to serve as a blueprint for all health systems to strictly apply. Guidance documents are administrative instruments that provide recommendations and implementation options typically in a step-by-step format but do not have force of law and, as such, allow for accommodations in approach [51, 52]. They are intended to provide recommendations on how to comply with governing statutes and regulations and assist staff and managers on how institutional mandates and objectives should be implemented in a manner that is fair, consistent, and effective [52]. Therefore, development of high-quality HSG will impact the type of recommendations being formulated, the degree to which they get implemented, the methods of dissemination, and the extent to which they impact on the usual operations of the health system [51].

Different players in the HSG process play different roles at different times and under different circumstances. Developers of guidance may only be looking at process principles and content, while end users in the field may focus more on usability of the guidance in their individual contexts. Thus, for example, to optimize this dual perspective and facilitate the division of labor, there is interest at the WHO to produce workbooks where the global guidance they develop (process principles and content) will be complemented by having a companion workbook for those at the receiving end (context principles). In this case, the role of the WHO will be the development of global guidance, while the country’s role (or health system’s role) will be to take this global guidance and then apply it to their local context. This makes our framework of health systems guidance concepts quite practical.

Our study has elicited 30 unique concepts. It is unclear the extent to which HSG stakeholders will view all concepts as equally important to our proposed tool, or, moreover, given the potential roles of the tool (development, appraisal, reporting), if there are varying levels of priority as a function of purpose. The next phase (stage 2) of our research program is aimed to create the beta version of the AGREE-HS by having international stakeholders prioritize these concepts and using those results to populate the tool. Our final phase (stage 3) will involve the usability testing of the beta version of the tool.

Abbreviations

AGREE-HS:

Appraisal of Guidelines for Research and Evaluation for Health Systems

CDC:

Centre for Disease Control

CINAHL:

Cumulative Index to Nursing and Allied Health Literature

CIS:

Critical Interpretive Synthesis

CPGs:

Clinical Practice Guidelines

DAA:

Denis Ako-Arrey

EMBASE:

Excerpta Medica dataBASE

EVIPNet:

Evidence Informed Policy Network

GIN:

Guidelines International Network

HSG:

health systems guidance

JL:

John Lavis

LILACS:

Latin American and Caribbean Health Sciences Literature

MB:

Melissa Brouwers

MDGs:

Millennium Development Goals

MG:

Mita Giacomini

NHS:

National Health Services

NICE:

National Institute of Health and Care Excellence

SA:

Saira Akram

USAID:

United States Agency for International Development

WHO:

World Health Organization

References

  1. Roemer M. I. National health systems of the world, vol. 1: the countries. New York: Oxford University Press; 1991. p. 97.

    Google Scholar 

  2. World Health Organization. The World Health Report 2000. Health systems: improving performance. Geneva, 2000. [accessed 9 July 2013] Available from: http://www.who.int/whr/2000/en/

  3. Hoffman S, Rottingen J-A, Bennett S, Lavis J, Edge J, Frenk J. Background paper on conceptual issues related to health systems research to inform a WHO global strategy on health systems research—a working paper in progress. Geneva: World Health Organisation; 2012. p. 2013.

    Google Scholar 

  4. Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen JA, Dröschel D, et al. Guidance for evidence-informed decisions about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9:e1001185.

    Article  PubMed Central  PubMed  Google Scholar 

  5. Lavis JN, Røttingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9:e1001186.

    Article  PubMed Central  Google Scholar 

  6. Remme JHF, Adam T, Becerra-Posada F, D’Arcangues C, Devlin M, Gardner C, et al. Defining research to improve health systems. PLoS Med. 2010;7(11):e1001000.

    Article  PubMed Central  PubMed  Google Scholar 

  7. Fryatt R, Mills A, Nordstrom A. Financing of health systems to achieve the health Millennium Development Goals in low-income countries. Lancet. 2010;375:419–26.

    Article  PubMed  Google Scholar 

  8. Murray C, Frenk J. A WHO framework for health system performance assessment. Bull World Health Organ. 2000;78:717–31.

    PubMed Central  CAS  PubMed  Google Scholar 

  9. World Health Organization. Fifty-eighth World Health Assembly, A58/22, 1–7. Geneva: Author; 2005. WHA58.34: Ministerial Summit on Health Research.

    Google Scholar 

  10. Bosch-Capblanch X. Handbook for developing health systems guidance. Swiss: TPH; 2010. Analytical framework for health systems guidance.

    Google Scholar 

  11. World Health Organization. Health system strengthening: improving support to policy dialogue around national health policies, strategies and plans. Report by the Secretariat to the 64th World Health Assembly. A64/12. Geneva: World Health Organization; 2011. Retrieved on October 4th 2013 from http://apps.who.int/gb/ebwha/pdf_files/WHA64/A64_12-en.pdf.

    Google Scholar 

  12. Wilson MG, Moat KA, Lavis JN. The global stock of research evidence relevant to health systems policymaking. Health Res Policy Syst. 2013;11(32):10–1186.

    Google Scholar 

  13. Bosch-Capblanch X. Handbook for supporting the development of health system guidance. Basel: Swiss Centre for International Health; 2011.

    Google Scholar 

  14. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder A, et al. Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet. 2004;364:900–6.

    Article  PubMed  Google Scholar 

  15. Chopra M, Lawn J, Sanders D, Barron P. Achieving the health millennium development goals for South Africa: challenges and priorities. Lancet. 2009;374:1023–31.

    Article  PubMed  Google Scholar 

  16. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al .Appraisal of Guidelines for Research and Evaluation. AGREE II Instrument. The AGREE Next Steps Consortium. 2009. Available from:http://www.agreetrust.org/resource-centre/agree-ii/. [Accessed: 2013 Aug 23].

  17. SUPPORT. Supporting policy relevant reviews and trials. 2012. Available: http://www.support-collaboration.org/index.htm. Accessed 30 January 2013.

    Google Scholar 

  18. Brettle A. Information skills training: a systematic review of the literature. Health Info Libr J. 2003;20 Suppl 1:3–9.

    Article  PubMed  Google Scholar 

  19. Heaton J, Corden A, Parker G. ‘Continuity of care’: a critical interpretive synthesis of how the concept was elaborated by a national research programme (2012). Int J Integrated Care. 2012;12.

  20. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.

    Article  PubMed  Google Scholar 

  21. Dixon-Woods M, Kirk MD, Agarwal MS, Annandale E, Arthur T, Harvey J, et al. Vulnerable groups and access to health care: a critical interpretive review. National Coordinating Centre NHS Service Delivery Organ RD (NCCSDO). 2005.

    Google Scholar 

  22. Taylor TL, Killaspy H, Wright C, Turton P, White S, Kallert TW, et al. A systematic review of the international published literature relating to quality of institutional care for people with longer term mental health problems. BMC Psychiatry. 2009;9(1):55.

    Article  PubMed Central  PubMed  Google Scholar 

  23. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.

    Article  PubMed Central  PubMed  Google Scholar 

  24. Gysels MH, Evans C, Higginson IJ. Patient, caregiver, health professional and researcher views and experiences of participating in research at the end of life: a critical interpretive synthesis of the literature. BMC Med Res Methodol. 2012;12(1):123.

    Article  PubMed Central  PubMed  Google Scholar 

  25. Entwistle V, Firnigl D, Ryan M, Francis J, Kinghorn P. Which experiences of health care delivery matter to service users and why? A critical interpretive synthesis and conceptual map. J Health Serv Res Policy. 2012;17(2):70–8.

    Article  PubMed Central  PubMed  Google Scholar 

  26. Kazimierczak KA, Skea ZC, Dixon-Woods M, Entwistle VA, Feldman-Stewart D, N’dow JM, et al. Provision of cancer information as a “support for navigating the knowledge landscape”: findings from a critical interpretive literature synthesis. Eur J Oncol Nurs. 2013;17(3):360–9.

    Article  PubMed  Google Scholar 

  27. Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35. doi:10.1186/1471-2288-6-35.

    Article  PubMed Central  PubMed  Google Scholar 

  28. Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27–44.

    Article  Google Scholar 

  29. Talseth AG, Gilje FL. Nurses’ responses to suicide and suicidal patients: a critical interpretive synthesis. J Clin Nurs. 2011;20(11–12):1651–67.

    Article  PubMed  Google Scholar 

  30. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from critical interpretive synthesis. Telemedicine and e-Health. 2012;18(2):137–44.

    Article  PubMed  Google Scholar 

  31. Noblit GW, Hare RD. Meta-ethnography: synthesising qualitative studies. London: Sage; 1988.

    Google Scholar 

  32. Markoulakis R, Kirsh B. Difficulties for university students with mental health problems: a critical interpretive synthesis. Rev High Educ. 2013;37(1):77–100.

    Article  Google Scholar 

  33. Eakin JM, Mykhalovskiy E. Reframing the evaluation of qualitative health research: reflections on a review of appraisal guidelines in the health sciences. J Eval Clin Pract. 2003;9:187–94.

    Article  PubMed  Google Scholar 

  34. Flemming K. Synthesis of quantitative and qualitative research: an example using critical interpretive synthesis. J Adv Nurs. 2010;66(1):201–17.

    Article  PubMed  Google Scholar 

  35. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994;309(6964):1286–91.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  36. Seale C. The quality of qualitative research. London: Sage Publications; 1999.

    Google Scholar 

  37. Annandale E, Harvey J, Cavers D, Dixon-Woods M. Gender and access to healthcare in the UK: a critical interpretive synthesis of the literature. Evid Policy. 2007;3(4):463–86.

    Article  Google Scholar 

  38. Sandelowski M, Docherty S, Emden C. Focus on qualitative methods. Qualitative meta-synthesis: issues and techniques. Res Nurs Health. 1997;20:365–71.

    Article  CAS  PubMed  Google Scholar 

  39. Dixon-Woods M, Shaw R, Agarwal S, Smith J. The problem of appraising qualitative research. Qual Saf Health Care. 2004;13:223–5.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  40. Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. In: Furlong J, Oancea A, editors. Special Edition of Research Papers in Education, 22 (2): 213–228. 2007.

    Google Scholar 

  41. Burnard P. A method of analysing transcripts in qualitative research. Nurse Educ Today. 1991;11:461–6.

    Article  CAS  PubMed  Google Scholar 

  42. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24:105–12.

    Article  CAS  PubMed  Google Scholar 

  43. Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. 2nd ed. Newbury Park: Sage; 1998.

    Google Scholar 

  44. Walt G, Gibson L. Reforming the health sector in developing countries: the central role of policy analysis. Health Policy Plan. 1994;9:353–70.

    Article  CAS  PubMed  Google Scholar 

  45. Walt G, Shiffman J, Schneider H, Murray SF, Brugha R, Gilson L. ‘Doing’ health policy analysis: methodological and conceptual reflections and challenges. Health Policy Plan. 2008;23(5):308–17.

    Article  PubMed Central  PubMed  Google Scholar 

  46. Buse K, Mays N, Walt G. Making health policy. Berkshire: Open University Press; 2005.

    Google Scholar 

  47. Ostrom E. Institutional rational choice: an assessment of the institutional analysis and development framework. In: Sabatier PA, editor. Theories of the policy process. 2nd ed. Boulder: Westview Press; 2007. p. 21–64.

    Google Scholar 

  48. Collins C, Green A, Hunter D. Health sector reform and the interpretation of policy context. Health Policy Plan. 1999;47:69–83.

    Article  CAS  Google Scholar 

  49. WHO, World Health Organization. Sound choices: enhancing capacity for evidence-informed health policy. Geneva. 2007

  50. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder Get al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010. DOI:10.1503/cmaj.090449

  51. World Health Organization. “Guidelines for WHO guidelines.” EIP/GPE/EqC/2003.1. Geneva: Author; 2003. Retrieved Jan 28, 2013. <http://whqlibdoc.who.int/hq/2003/EIP_GPE_ EqC_2003_1.pdf.

    Google Scholar 

  52. Health Canada. Recognition and use of standards under the medical devices regulations. 2009. Retrieved on October 1st, 2013 from http://www.hc-sc.gc.ca/dhp-mps/md-im/applic-demande/guide-ld/md_gd_standards_im_ld_normes-eng.php.

    Google Scholar 

  53. Davies E, Littlejohns P. Views of directors of public health about NICE appraisal guidance: results of a postal survey. J Public Health. 2002;24(4):319–25.

    Article  Google Scholar 

  54. Shaw CD, Kalo I. A background for national quality policies in health systems. 2002. Retrieved on August 19th 2013 from http://apps.who.int/iris/handle/10665/107458.

    Google Scholar 

  55. Arah OA, Klazinga NS, Delnoij DMJ, Ten Asbroek AHA, Custers T. Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement. Int J Qual Health Care. 2003;15(5):377–98.

    Article  CAS  PubMed  Google Scholar 

  56. Islam M. Health systems assessment approach: a how-to manual. 2007. Retrieved on August 17th 2013 from http://www.urc-chs.com/uploads/resourcefiles/Health_Systems_Assessment.pdf.

    Google Scholar 

  57. National Health and Medical Research Council - NHMRC. Standards and procedures for externally developed guidelines. 2007. Retrieved on August 16th 2013 from https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/nh56_nhmrc_standards_procedures_externally_developed_guidelines_131223.pdf.

    Google Scholar 

  58. World Health Organization - WHO. The health systems assessment approach: a how to manual: guidance on assessing health system building blocks. 2012. Retrieved on October 4th 2013 from http://www.healthsystemassessment.com/wp-content/uploads/2012/06/Section-3-Module-3-1-_HSAA-Manual-August-2012-11.pdf.

    Google Scholar 

  59. Funk M, Drew N, Faydi E. Checklist for evaluating a mental health policy. Retrieved August 19th 2013 from http://www.who.int/mental_health/policy/WHOPolicyChecklist_forwebsite.pdf.

  60. National Health Services - NHS. How to use NICE guidance to commission high quality services. 2009. Retrieved October 4th 2013 from http://www.gserve.nice.org.uk/media/027/17/HowToCommissioningGuideFinal.pdf.

    Google Scholar 

  61. Øvretveit J, Klazinga N. Guidance on developing quality and safety strategies with a health system approach. 2008. Retrieved on August 17th 2013 from http://www.euro.who.int/__data/assets/pdf_file/0011/96473/E91317.pdf.

    Google Scholar 

  62. Van der Sluijs JP, Petersen AC, Janssen PH, Risbey JS, Ravetz JR. Exploring the quality of evidence for complex and contested policy decisions. Environ Res Lett. 2008;3(2):024008. anual.

    Article  Google Scholar 

  63. Lewin S, Bosch-Capblanch X, Oliver S, kl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187.

    Article  PubMed Central  Google Scholar 

  64. Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual framework for the OECD health care quality indicators project. Int J Qual Health Care. 2006;18 suppl 1:5–13.

    Article  PubMed  Google Scholar 

  65. Oxman AD, Fretheim A, Schünemann HJ. Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Syst. 2006;4:26.

    Article  PubMed Central  PubMed  Google Scholar 

  66. National Health Services - NHS. Methods for the development of NICE public health guidance (second edition). 2009. Retrieved October 4th 2013 from http://www.gserve.nice.org.uk/media/CE1/F7/CPHE_Methods_manual_LR.pdf.

    Google Scholar 

  67. Moga C, Guo B, Schopflocher D, Harstall C. Development of a quality appraisal tool for case series studies using a modified delphi technique. Edmonton: Institute of Health Economics (IHE); 2012.

    Google Scholar 

  68. International Centre for Allied Health Evidence - iCAHE. The iCAHE guideline quality checklist. Retrieved on August 19th 2013 from http://www.biomedcentral.com/content/supplementary/1471-2288-14-63-s1.pdf.

  69. World Health Organization - WHO. Practical guidance for scaling up health service innovations. 2009. Retrieved on October 5th 2013 from http://www.expandnet.net/PDFs/WHO_ExpandNet_Practical_Guide_published.pdf.

    Google Scholar 

  70. Bosch-Capblanch X, Allen C. Health systems strengthening and conflict: transformation in fragile states. 2012. Retrieved on August 17th 2013 from http://www.evidenceaid.org/wp-content/uploads/2013/05/MMI_for_Evidence_Aid_XBC_revised_by_MC.pdf.

    Google Scholar 

  71. German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Morb Mortal Wkly Rep Recomm Rep. 2001;50:1–35.

    CAS  Google Scholar 

  72. Murray CJ, Frenk J. A WHO framework for health system performance assessment. Evidence and information for policy, World Health Organization. 1999.

    Google Scholar 

  73. Atun R. Health systems, systems thinking and innovation. Health Policy Plan. 2012;27 suppl 4:iv4–8.

    Article  PubMed  Google Scholar 

  74. Orem JN, Wavamunno JB, Bakeera SK, Criel B. Do guidelines influence the implementation of health programs?—Uganda’s experience. Implement Sci. 2012;7(1):1–16.

    Article  Google Scholar 

  75. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst. 2009;7 Suppl 1:S1.

    Article  PubMed Central  PubMed  Google Scholar 

  76. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

    Article  PubMed Central  Google Scholar 

  77. Phillips AC, Lewis LK, McEvoy MP, Galipeau J, Glasziou P, Hammick M, et al. Protocol for development of the guideline for reporting evidence based practice educational interventions and teaching (GREET) statement. BMC Med Educ. 2013;13(1):9.

    Article  PubMed Central  PubMed  Google Scholar 

  78. Lomas J, Culyer T, Mccutcheon C, Law S, Tetroe J. Conceptualizing and combining evidence for health system guidance. 2005. Retrieved on October 4th 2013 from http://www.cfhi-fcass.ca/migrated/pdf/insightAction/evidence_e.pdf.

    Google Scholar 

  79. Bryce J, Requejo JH, Moulton LH, Ram M, Black RE. A common evaluation framework for the African Health Initiative. BMC Health Serv Res. 2013;13 Suppl 2:S10.

    Article  PubMed Central  PubMed  Google Scholar 

  80. American Psychological Association -APA-. Criteria for practice guideline development and evaluation. Am Psychol. 2002;57:1048–51.

    Article  Google Scholar 

  81. Wilson P. How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet. BMJ. 2002;324(7337):598.

    Article  PubMed Central  PubMed  Google Scholar 

  82. German BACKUP Initiative - GIZ. Guidance on integrating gender-specific issues into health systems strengthening activities. 2011. Retrieved on October 4th 2013 from http://www.genderingermandevelopment.net/files/images/pdf/Gender_Mainstreaming/giz2011-en-gender-dimensions-in-health-systems.pdf.

    Google Scholar 

  83. Hoffman S, Lavis J, Bennett S. The use of research evidence in two international organizations’ recommendations about health systems. Healthcare Policy. 2009;5(1):66–86.

    PubMed Central  PubMed  Google Scholar 

  84. Perpiñán Abellán JM, Sánchez Martínez FI, Martínez Pérez JE. Quality assessment of economic evaluations in health care: a checklist and user guide. Rev Esp Salud Publica. 2009;83(1):71–84.

    Article  Google Scholar 

  85. Kingdon JW, Thurber JA. Agendas, alternatives, and public policies (vol. 45). Boston: Little, Brown. Chicago; 1984.

    Google Scholar 

  86. Murray CJ, Evans DB. Health system performance assessment: debates, methods, and empiricism. Geneva: World Health Organization; 2003. Retrieved on August 17th 2013 from http://whqlibdoc.who.int/publications/2003/9241562455.pdf.

    Google Scholar 

  87. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA, et al. Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet. 2004;364(9437):900–6.

    Article  PubMed  Google Scholar 

  88. Schünemann HJ, Fretheim A, Oxman AD, WHO Advisory Committee on Health Research. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006;4(13):1–6.

    Google Scholar 

  89. Savigny D, Adam T (Eds.). Systems thinking for health systems strengthening. World Health Organization. 2009. Retrieved on August 17th 2013 fromhttp://apps.who.int/iris/bitstream/10665/44204/1/9789241563895_eng.pdf.

  90. Swanson RC, Bongiovanni A, Bradley E, Murugan V, Sundewall J, Betigeri A, et al. Toward a consensus on guiding principles for health systems strengthening. PLoS Med. 2010;7(12):e1000385.

    Article  PubMed Central  PubMed  Google Scholar 

  91. Etienne C, Asamoa-Baah A, Evans DB. Health systems financing: the path to universal coverage. World Health Organization. (2010): Retrieved on August 17th 2013 from http://apps.who.int/iris/bitstream/10665/44371/1/9789241564021_eng.pdf.

  92. Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F, Bennett S. Building the field of health policy and systems research: framing the questions. PLoS Med. 2011;8(8):e1001073.

    Article  PubMed Central  PubMed  Google Scholar 

  93. Peters DH, Bennett S. Better guidance is welcome, but without blinders. PLoS Med. 2012;9(3):e1001188.

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the contributions of the members of the AGREE-HS team: Andy Haines—London School of Hygiene & Tropical Medicine, UK; Carmen Mihaela Dolea—World Health Organization, Switzerland; Fadi El-Jardali—American University of Beirut, Lebanon; Francoise Cluzeau—National Institute for Health and Care Excellence, UK; Govin Permanand Govin—World Health Organization, Denmark; Iván Darío Flórez Gómez—Universidad de Antioquia, Colombia; Jillian Ross—Cancer Care Ontario, Canada; Luis Gabriel Cuervo—Pan American Health Organization, USA; Mike Wilson—McMaster University, Canada; Pablo Perel—London School of Hygiene and Tropical Medicine, UK; Paidgraig Warde—Cancer Care Ontario, Canada; Pierre Ongolo—Centre for Development of Best Practices in Health, Cameroon; Sheila McNair—Program for Evidence Based Care, Canada; Ulysses Panisset—World Health Organization, Switzerland; Xavier Bosch-Capblanch—Swiss Tropical and Public Health Institute, Switzerland; and Yaolong Chen—Lanzhou University, China. This project was supported by the Canadian Institutes of Health Research (Canada). DAA received a student award from Knowledge Translation (KT) Canada. We would also like to acknowledge SA for her role as a second reviewer during the screening and data extraction.

Author information

Authors and Affiliations

  1. McMaster University, Juravinski Hospital Site, G Wing, 2nd Floor, Room 207, 711 Concession Street, Hamilton, ON, L8V 1C3, Canada

    Denis E. Ako-Arrey & Melissa C. Brouwers

  2. McMaster University, MML-417, 1280 Main St. West, Hamilton, ON, L8S 4L6, Canada

    John N. Lavis

  3. McMaster University, CRL-218, 1280 Main Street West, Hamilton, ON, L8S 4K1, Canada

    Mita K. Giacomini

Authors
  1. Denis E. Ako-Arrey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melissa C. Brouwers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John N. Lavis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mita K. Giacomini
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

on behalf of the AGREE-HS Team

Corresponding author

Correspondence to Melissa C. Brouwers.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

DAA and MB are the lead authors and were responsible for conceptualizing the theoretical and empirical formulations of each research project, literature review, study protocol, and design and collecting, analyzing, and interpreting data as well as manuscript preparation. JL and MG are co-authors and offered substantive intellectual input and expertise during each phase of the research formulation and manuscript preparation and provided feedback on earlier drafts. Members of the core and expanded scientific team (AGREE-HS team) provided operational guidance as well as feedback and suggestions on the draft versions of the paper. All authors read and approved the final manuscript.

Authors’ information

DAA is an International Public Health Servant with a PhD in Public Health Policy currently working as a post-doctoral trainee at the World Health Organization supporting the country office in Suriname in various health systems strengthening initiatives. This project was undertaken as part his doctoral studies in the PhD Policy program at McMaster University, Hamilton, Canada.

Appendix

Appendix

Table 4 Summary of selected papers
Full size table

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ako-Arrey, D.E., Brouwers, M.C., Lavis, J.N. et al. Health systems guidance appraisal—a critical interpretive synthesis. Implementation Sci 11, 9 (2015). https://doi.org/10.1186/s13012-016-0373-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13012-016-0373-y

Keywords

Implementation Science

ISSN: 1748-5908