Journal of Theoretical and Applied Electronic Commerce Research
ISSN 0718-1876 Electronic Version VOL 6 / ISSUE 1 / APRIL 2011 / 61-94 © 2011 Universidad de Talca - Chile
This paper is available online at www.jtaer.com
DOI: 10.4067/S0718-18762011000100006

RESEARCH

 

Information Systems Interoperability in Public Administration: Identifying the Major Acting Forces through a Delphi Study

 

Delfina Soares¹ and Luis Amaral²

¹ University of Minho, Department of Information Systems, dss@dsi.uminho.pt ² University of Minho, Department of Information Systems, amaral@dsi.uminho.pt

---

Abstract

Information Systems (IS) interoperability in Public Administration (PA) is a main goal and a major challenge for PA professionals. Achieving interoperability among IS that are technologically disparate and that exist in different organizational contexts is a complex task, being affected by multiple aspects, not yet satisfactorily known and characterized. The aim of this paper is to unveil the forces that influence IS interoperability initiatives in PA. The inquiry was inspired by Lewin's Field Theory. The data generation process was based on a Delphi study involving 55 experts from PA, IS/IT industry, and academy. A set of 31 forces were identified and ranked based on the level of importance they assume in IS interoperability initiatives. Thirty eight propositions describing restraining and driving influences were also formulated, as well as 24 propositions about forces' configuration in the current context of Portuguese PA that represent the specific constellation of forces acting in IS interoperability initiatives in that country. The results of this study provide an understanding of the complex of forces acting in IS interoperability, contributing to improve the study, management, and implementation of these initiatives and, consequently, to the establishment of a PA with more adequate, sustained, and sustainable levels of interoperability.

Keywords: Interoperability, E-government, Information systems, Information technologies, Public administration, Delphi study

---

1      Introduction

Since the late 90's, the development of e-government and, more specifically, the creation of an electronic Public Administration (PA) have been considered as main concerns and priorities for action by most governments all over the world, being seen as essential vectors to promote and foster a new paradigm of society currently known as the "Information Society". Since then, the quest and the need to deeply transform the way PA operates won the interest of politicians, who started to direct considerable attention to the formulation and implementation of strategies that could lead to the achievement of the intended transformation.

Information Technologies (IT) assume, in this context, an important and leading role, being recognized as a primary instrument to e-government development and to the modernization and transformation of the public sector. The aim and concern of politicians and PA professionals is to harness the potential offered by information technologies to transform the Public Administration operating paradigm, making it more efficient, more effective, more transparent, more citizen centered, and more capable of delivering cross-agency public services in an integrated way, through multiple access channels, 24 hours a day, seven days a week [5] - [6], [65]. Achieving this "ideal vision" is not a trivial task. Indeed, working according to this new paradigm imposes a fundamental and not so easily achievable condition to PA agencies that is to have the capacity to work in a collaborative way, what inevitably requires and demands the existence of interoperability capacity among their Information Systems (IS). In this context, IS interoperability emerges as a pertinent topic in the PA area, becoming a key requirement for the creation and development of the "ideal vision" typically associated with the e-government phenomenon [21].

The relevance of interoperability in PA is quite evident considering the numerous government programs, action plans, and policy documents published in recent years at national and European levels, the multiple interoperability strategies, frameworks, and platforms being developed in multiple countries, the huge number of research projects on interoperability issues funded namely by the European Commission under their last framework programs, the creation of observatories on e-government interoperability issues, and the raise in the number of conferences and scientific or practitioner commissions and associations focused on the interoperability thematic.

All these facts suggest the recognition by politicians, PA professionals, European Institutions, and other stakeholders in this process (particularly the IS/IT industry community) of the importance of creating more appropriate interoperability levels within and between PA agencies at local and central government levels, as well as at a supranational level such as the European Union.

Despite all the recognition that exists around the need and importance of IS interoperability in PA, and despite all the investments and efforts already made in order to improve the collaboration and data exchange capacity of different government agencies, the level of interoperability between agencies is still not adequate and the problems and difficulties caused by this lack of interoperability are still quite evident. Actually, from a citizen standpoint, government agencies keep operating in a very fragmented way, continuing to compel citizens to have to interact with multiple agencies in order to get their requests satisfied. Delays in service delivery, lack of transparency about the procedures involved in service provision, the need to fill and refill forms containing the same set or subset of data, and the incapacity to obtain detailed information about the status of execution of a particular service through different access channels, are evidences of the lack of interoperability which generally persists in PA.

The two paradoxical observations just reported—on the one hand, the fact that interoperability has been recognized as a need in PA and that some efforts and investments have already been made by agencies to improve their interoperability, and, on the other hand, that despite the efforts undertaken the levels of interoperability have not come up to the level it would be desired and adequate—promptly give rise to the question of what makes interoperability initiatives so complex, so challenging, and so difficult to manage and implement.

Being it because of the huge number of agencies that could be involved; the technological, semantic, organizational, or cultural differences existing among them; or even because of the change that the creation of interoperability conveys to the agencies and to their information systems, the view shared by both practitioners and researchers is that interoperability initiatives are indeed highly complex and challenging efforts.

According to [63], the exact extent and complexity of the challenges surrounding the implementation of IS interoperability initiatives in PA, are not yet, neither in theory nor in practice, properly understood. This opinion is also stressed in the white paper published by CompTIA [11], where it is mentioned that there is still no adequate explanation about the factors or forces that may facilitate or limit interoperability initiatives.

In this context, what seems to be one of the major problems underlying IS interoperability initiatives in PA is the lack of an explicit and systematic understanding about the complex of forces acting on the implementation of these initiatives. This lack of understanding makes those responsible for interoperability initiatives to face successive failures, preventing them from better managing the phenomenon of creating more adequate, more sustained, and more sustainable levels of interoperability. Besides contributing to the perpetuation of the dissatisfaction felt by citizens and other agents interacting with PA agencies, these failures also cause financial losses as well as a depreciation of the image of both the PA IS/IT professionals, the Administration itself, and even the country as a whole, what may seriously jeopardize the efforts to achieve higher levels of maturity in e-government development.

The aim of this paper is to unveil the myriad of forces that may influence the implementation of IS interoperability initiatives in PA. More specifically, and grounded on the ideas and principles advocated and proposed by Kurt Lewin in the context of his Field Theory [40] - [44], this paper seeks to provide answers to the following research questions: (1) What are the forces acting on the implementation of IS interoperability initiatives in PA? (2) What is the relative level of importance of each of those forces? and (3) Which type of influence (restraining or driving) does each force have on the initiatives?

The answers to these questions were obtained through the execution of a three round Delphi study, involving a panel of 55 experts, coming from Public Administration, IS/IT industry, and academy.

Considering that all the panelists were experienced in the implementation of initiatives that aimed to promote and create IS interoperability in the Portuguese PA context, it made sense to instantiate the answers given to the three previous research questions, in order to depict the overall constellation of forces currently acting in the Portuguese PA context. Hence, an additional research question—What is the configuration of each force in the current context of Portuguese PA?—was formulated.

This paper is organized as follows. After this introduction, section two sets down the basis for the rest of the paper, by defining the concept of interoperability and by reviewing previous works relevant to this research, i.e., to the study of the forces acting on IS interoperability initiatives in the government context. Section three describes the methodological approach of the research, starting with a description of the Delphi method's features and proceeding with a description of how the Delphi study was designed and executed. Section four presents the results of the Delphi study and, finally, section five states the main contributions of the research, as well as its main limitations and possible future research work.

2     Literature Review

The goal of this section is to set down the basis for the rest of the paper, by defining the concept of interoperability and by giving an overview of previous works on information sharing, information integration, G2G (Government-to-Government), and interoperability that deemed relevant for this research.

2.1      The Interoperability Concept

Interoperability is the central concept of this paper. As such, it is convenient to start by clarifying its meaning in this work. This clarification has proved even more relevant since, as pointed out by [10] - [11], and [48], despite being frequently used in current written and spoken discourses, the meaning of interoperability persists somewhat ambiguous and diffuse.

Although there is no universally accepted definition for interoperability, it has been recently noticed a growing acceptance and convergence of the academic and practice e-government communities around the definition proposed in the document European Interoperability Framework for pan-European e-Government Services (EIF) -version 1.0, published in 2004 by the European Commission, under the IDABC program. In the document, interoperability was defined as "the ability of information and communication technology (ICT) systems and of the business processes they support to exchange data and to enable the sharing of information and knowledge" [22] p. 5.

This definition has been revised in the draft document, published in 2008, that serves as a basis for the publication of the second version of the European Interoperability Framework (EIF V2). In this document interoperability is defined as "...the ability of disparate and diverse organizations to interact towards mutually beneficial and agreed common goals, involving the sharing of information and knowledge between the organizations via the business processes they support, by means of the exchange of data between their respective information and communication technology (ICT) systems" [23] p. 5. This new definition adds detail to the previous one, namely by stressing that (1) the interacting organizations are disparate and diverse, (2) that they interact with a common goal, and (3) that they will all benefit from it.

Issues (1) and (2) explicitly stated in previous definition are some of the key ideas that underlie the interoperability concept. The analysis, by the authors, of more than 30 interoperability definitions obtained from different sources, including general dictionaries, technical dictionaries, reports, scientific papers, and books, and from multiple research areas, namely from the military area to the hardware, software, databases, computer networks, and information systems areas, revealed the existence of a set of key ideas inherent to the concept of interoperability. The seven key ideas that characterize any interoperability scenario are: (1) the involvement of two or more entities, (2) the existence of heterogeneities among the entities, (3) the conjunct operation of the entities, (4) the existence of a shared understanding among entities, (5) the idea of being a effortless operation, (6) the existence of a common purpose among the entities, and (7) the preservation of the entities' autonomy. According to these ideas, the use of the term interoperability is associated with situations where two or more entities, that were separately developed, that are heterogeneous, and which operate in an independent way, become able to operate together to achieve a common overall objective, while maintaining as much as possible their autonomy and heterogeneity, and without requiring that any entity needs to know the detailed specificities of the other entities with which it interoperates.

Combining this set of key ideas with the definition presented in the draft version of EIF V2.0, in this paper interoperability is defined as the ability of disparate, diverse, and autonomous organizations to interact towards an agreed common goal, through the sharing of information and knowledge between the business processes they support, by means of the exchange of data between their respective information and communication technology systems, while preserving their level of autonomy and without having to make a significant effort.

An additional issue becomes evident from the three definitions presented: that interoperability is a broad construct, going beyond the technical dimension traditionally associated to the term. The definitions clearly highlight the semantic, and, particularly, the procedural and organizational dimensions involved in the interoperability construct. Indeed, the literature review carried out on the e-government interoperability area showed that, instead of defending a monolithic perspective of interoperability, as frequently happens in other areas [49], authors in this area have been increasingly recognizing and stressing the need to consider these three dimensions in the study and implementation of interoperability initiatives [21].

The technical interoperability dimension covers the technical issues of linking computer systems and services. It includes aspects such as open interfaces, interconnection services, data integration and middleware, data presentation and exchange, accessibility, and security services [22].

The semantic interoperability dimension is concerned with ensuring that the precise meaning of exchanged information among computer systems and services is understandable, even though they were not initially developed with the purpose to interoperate. Semantic interoperability enables systems to combine received information with other information resources and to process it in a meaningful manner [22].

The organizational interoperability dimension is concerned with defining business goals, modeling business processes, and bringing about the collaboration of administrations that wish to exchange information and may have different internal structures and processes. Moreover, organizational interoperability aims at addressing the requirements of the user community by making services available, easily identifiable, accessible, and user oriented [22].

Beyond technical, semantic, and organizational issues, some documents, such as [23] and [67], also highlight the prominent role of legal and political issues on e-government interoperability, arguing for their inclusion as new dimensions of interoperability.

A final observation that emerged from the literature review is the importance and pertinence of the governance issue in e-government interoperability. The issue of interoperability governance has been introduced and emphasized in documents such as [22] - [24] and [66], that clearly point out the need and relevance of proceeding to the overall governance of the interoperability. The adoption of a more simplistic approach to interoperability, in which the attention is exclusively focused on aspects of collaboration and on ensuring the ability of "being together" in the provision of public service, should not be considered sufficient. As stressed by [32], the most important aspect in PA agencies interoperability is not that they are able of "being together", but that they are able to find out "the most effective way of being together". Consequently, the attention in the quest for interoperability must be set, not only on collaboration issues, but also on matters of coordination, direction, and control of the whole process.

The way interoperability has been defined in this section, as well as the recognition of its multiple dimensions, will influence and shape the search for the forces acting on the implementation of IS interoperability initiatives in the PA context.

2.2      Studies in Information Sharing, Information Integration, G2G, and Interoperability

The literature review carried on in this study, with the aim to find out works that are focused in the identification and understanding of the barriers, problems, success factors, and risk factors involved in IS interoperability initiatives in PA and that could be valuable for this study, revealed that research in this area is still quite limited. Indeed, most of the potentially interesting studies found over the literature review were not focused on the interoperability theme in itself but on some related themes such as interconnection, information sharing, information integration in government, and G2G.

Table 1 systematizes eight studies particularly relevant to the aim of this paper. The studies are listed in chronologicallorder.

The first of these studies was published in 1996 by Sharon Dawes, a researcher of the Center for Technology in Government (CTG), University at Albany. This study became a seminal paper, being cited by most works done in this area. The aim of the study was to understand the opinions and attitudes of state government managers towards the benefits and costs of interagency information sharing. In order to do that, Dawes organized her work in two phases. In the first phase, Dawes performed a literature review to identify the benefits and barriers associated with interagency information sharing. The benefits and barriers identified are shown in Table 2.

In the second phase, Dawes conducted a survey, in the state of New York, to evaluate the attitudes and opinions of state government managers towards the benefits and barriers identified in the literature, and to examine the policies and tools used to govern information sharing activities. As a result, Dawes proposed a theoretical model of interagency information sharing. This model argues that, as a consequence of their participation in previous sharing initiatives, agencies have perceptions about the potential benefits and risks involved that will affect their performance in future similar initiatives. As new sharing experiences occur, new benefits and risks will be perceived by participants. These new perceptions form the basis for future expectations and produce lessons for a general policy and management framework that could be utilized in the future.

Some years later, Landsbergen and Wolken [39] extended Dawes' model. While having used the interoperability term in their work, the fundamental contribution of these authors was at the interagency information sharing level, by reviewing and updating Dawes' model of interagency information sharing. Based on literature review, document analysis, case studies, and interviews, Landsbergen and Wolken identified three main benefits of interoperability (effectiveness, efficiency, and responsiveness), as well as 11 barriers that, as shown in Table 3, were classified in four different categories: political, organizational, economic, and technical.

Those scholars used the information gathered in the study to expand the theoretical model for interagency information proposed by Dawes. The principal contribution of their model is its emphasis on the need for the existence of an infrastructure to support agencies in information sharing, as well as on the need for legal, managerial, and policy approaches to maintain interagency information sharing.

While keeping the focus in the interagency information sharing, the third study centered its attention in the interconnection between state and local agencies. The aim of Akbulut was to investigate the factors that influence local agency participation in electronic information sharing with state agencies. The author organized her work in two parts. The first part started with the development of a research framework. The framework included 14 factors, classified in three different groups (agency characteristics, environmental characteristics, and electronic information sharing characteristics) obtained by literature review. The framework was then tested through a survey of local agencies. The second part of the study involved the collection and analysis of qualitative data related to a major state-local electronic information sharing initiative to seek additional support for the findings of the quantitative data analysis and to identify additional factors that are not discovered in the quantitative part.

Combining the data gathered in both parts of the study, Akbulut found that: (1) five factors, of the 14 included in the framework, influence the participation of local agencies in electronic information sharing with state agencies; (2) one factor has no influence on the participation of local agencies in electronic information sharing with state agencies; and (3) eight factors may influence local government participation in electronic information sharing with state agencies.

Table 4 systematizes Akbulut's findings.

Finally, Akbulut proposes a set of 33 recommendations, clustered in those three types of characteristics, which could increase local agency participation in electronic information sharing initiatives.

The fourth study was conducted by Joia. The aim of Joia was to propose a heuristic framework to successfully implement G2G endeavors in Brazil. Based on action research and on two case studies, Joia identified three key success factors (security, organizational culture, and training) to the implementation of G2G processes in PA agencies, as well as nine barriers that could arise during G2G projects and affect its success. Joia clustered the barriers in three main categories, as shown in Table 5.

The fifth and sixth studies were authored by a set of researchers working with Dawes at the CTG. Both studies are part of a series of works intended to increase the understanding of interorganizational information integration in public agencies and they represent an effort to model the social and technical processes of interorganizational information integration.

The study by Pardo et al., published in 2004, was focused on the identification of the factors and processes involved in integrating information across levels and agencies in government as well as on the understanding of how IT and social factors interact to influence interorganizational information integration. In this study, authors conceptualize the integration across distributed information sources and organizational boundaries as a complex social process in which technological and organizational artifacts are developed and assembled for the purpose of information use. A list of 32 main components of information integration was presented. The components were clustered in four categories as depicted in Table 6.

Keeping the focus on interorganizational information integration, the study published in 2005 by Gil-García et al. examined the factors that influence the success of selected criminal justice integration initiatives. Based on literature review and six case studies, the researchers identified a list of barriers to information integration, namely Turf and resistance to change, IT and data incompatibility, Organizational diversity and multiple goals, and Environmental and Institutional Complexity. The authors also suggested seven strategies for the implementation of interorganizational information integration initiatives, specifically: Retain autonomy of the agencies, Establish and exercise a governance structure, Secure strategic partnerships, Build on long-range and comprehensive planning, Build understanding of the business process, Secure adequate financial resources, and Obtain and nurture executive leadership and legislative support.

The seventh study was conducted by Bekkers in the Netherlands. Bekkers' goals have been to identify critical factors which account for the success of IT-driven back office integration projects and to find out how should back office integration be organized: as a management or as a governance challenge. To achieve these goals, Bekkers analyzed four successful Dutch back office integration projects according to a semi-standardized format, involving (1) an analysis of the involved back offices and their core values, (2) an analysis of the perceived resource dependency between the involved back offices, (3) an analysis of the factors which contributed to the definition of a shared information domain, and (4) an analysis of the circumstances under which a project management and/or process management approach contributes to the effective integration of back offices. Based on the case studies, Bekkers concluded that back office integration should be seen as a governance problem and set out six main lessons learned from back office integration projects, namely: Managing multi-rationality in different arenas; The ongoing recognition of interdependency; It is the content that unifies: trust and external pressure as lubricant; Managing the political agenda; The dynamic allocation of costs and benefits; and The balance between project and process management.

The eighth work resulted from a study on e-government interoperability conducted under the MODINIS program funded by the European Commission. The study, entitled Study on Interoperability at Local and Regional Level, ran for 26 months, from December 2004 to February 2007. The study had four main objectives: (1) To characterize the status of local and regional interoperability in European member states; (2) To identify the key success factors of local and regional interoperability; (3) To identify the key barriers of local and regional interoperability; and (4) To issue recommendations on e-government interoperability. Based on an extensive literature review, on the analysis of case studies that represent good practice projects in European member states, and on the analysis of stakeholders' needs and feedback, the authors identified a list of 39 key factors for e-government interoperability at local and regional level. These factors were organized along four different categories as shown in Table 7.

The authors also issued 44 recommendations regarding e-government interoperability, organized in a matrix structure that groups the recommendations along two dimensions, namely the suitable level of action (European Union, National, and Local/Regional level) and the areas where a recommendation should be applied (Legislation, Funding/Financial, Policy/Management, and Technical).

While providing an interesting set of factors, barriers, or issues that could eventually be useful to whom is involved in the promotion and implementation of IS interoperability in PA, these studies have two main drawbacks.

First, only two studies ([39], [66]) focus on the interoperability theme, and from these, one study ([39]), while using the term interoperability, defines it in an information sharing perspective as being "essentially a problem of sharing information". Despite the proximity of the themes approached in the other six studies, it is not possible to assume that the factors, barriers, or issues identified in these studies are necessarily the ones that are involved in interoperability initiatives. It is important to understand which of those factors, barriers, or issues are effectively relevant to interoperability initiatives, as well as what new additional factors, barriers, or issues, not identified in the studies, must be considered in interoperability initiatives. The study [66] is the only one specifically focused on interoperability, though its emphasis is placed at the European local and regional levels of government.

Second, neither of the studies addresses the level of importance and the type of influence exerted by each of the factors, barriers, or issues, nor gives specific practical orientations or recommendations based on it. This fact can pose significant difficulties, specially to practitioners, since they will have to face and manage this vast set of elements during the implementation of IS interoperability initiatives, without knowing conveniently how relevant is each one of those elements, if they have equal or different levels of importance, if all deserve the same level of attention, or how can each of them influence the promotion and implementation of IS interoperability initiatives in PA. The existence of this kind of information could be indubitably useful, helping practitioners to deal with such a set of factors, barriers or issues.

It is the goal of this research to contribute to mitigate these drawbacks.

3      Research Design

This section describes the strategy adopted to answer the four research questions stated in the first section of this paper. The decision was to base all the data gathering process on the conduction of a Delphi study.

After a brief review of the Delphi method origins and main features, the section proceeds outlining the decisions made concerning the design of the Delphi study. The section ends with the description of the study execution.

3.1      The Delphi Method

The Delphi method originated in the 50's as result of a set of studies performed by researchers at the RAND Corporation (Santa Monica, California), who investigated the scientific use of expert opinion in inexact sciences [38]. Most of those studies ran in the context of projects in the area of military defense. Considering the military nature of the projects, the method was kept private for about twelve years [13]. Only in 1963, after U.S. Air Force's decision of declassifying this method from the category "reserved for military use", have Dalkey and Helmer published the first paper introducing and explaining the use of the Delphi method [13]. Since then, Delphi has been widely adopted in areas such as marketing, education, health, transportation, and engineering [34], [58]. In the IS field it is also possible to find numerous examples of its application [3], [7] - [9], [17] - [18], [30] - [31], [35] - [37], [51] - [55], [57], [59], [61] - [62].

Basically, the Delphi method is a group communication technique especially used to achieve consensus of opinion among a group of experts [13]. The method has been developed in order to address some of the problems associated to other forms of interaction used on more conventional techniques of group dynamics [29], [45], such as Focus Groups, Nominal Groups, Brainstorming, Fishbowl, Synectics, and Roudtables.

The Delphi method is an iterative and structured process of listing, refining, and aggregating the opinions and perceptions of a group of people, called the expert panel, that could make valuable contributions to the resolution or understanding of a complex topic or problem in order to create a consensual shared vision on the matter under discussion [14]. Typically, this iterative process consists of sending a series of successive questionnaires to a panel of experts, until a certain degree of consensus is reached among them. On each new iteration, besides the questionnaire, the experts should also be sent a summary of the results achieved in the previous round [45].

The Delphi method is characterized by six main features: sampling, anonymity, iteration, controlled feedback, consensus, and statistical aggregation of group response.

Contrary to what happens in most techniques, the sampling of participants in a Delphi study is not based on a random process that seeks to generate representative samples of a given population [34], [55]. Instead, the sampling process is focused on the identification and selection of individuals that have a deep knowledge in the area under discussion or study [34], [55]. This sampling process is usually known as "expert sampling" [46] or "purposive sampling" [16].

The anonymity among participants is another feature of the method. Owing to the use of questionnaires, the process of collecting the experts' views and opinions can take place without the need for simultaneous physical presence of experts [58]. Since the questionnaires are answered singly, it is possible to keep the individual responses anonymous. Maintaining the anonymity of answers allows panel members to express their views and perceptions in a more honest and genuine way, without being subject to social pressures that may result from the presence of experts with more dominant personalities or status [29], [38], [58], [69].

Another feature of the Delphi method is its iterative nature. As mentioned before, Delphi is based on a successive and systematic process of questionnaire sending, each of which called "round". By having to successively answer to questionnaires, experts have the opportunity to think over and over again on the topic and to discover and add new and more mature ideas on it. Although in its most conventional form the Delphi method involves the execution of four rounds, this number can vary, being quite usual to find studies with two or three rounds [59]. Also in its conventional form, the first round has an unstructured nature starting with one or more open questions that gives experts the opportunity to freely express their opinions and perceptions about the subject under analysis [58]. After the first round, experts' responses are gathered and consolidated into a set of items, based on which more structured questionnaires are designed and used in the next rounds of the study [58].

As has been stressed above, at each new round or iteration, besides the questionnaire, a set of feedback information, which summarizes the views highlighted by the group in the previous round, is also sent to experts [58]. Feedback submitted may take different forms, varying from a simple statistical summary of the group opinion, to the inclusion of qualitative justifications given by the participants who have shown more dissonant opinions and perspectives [14], [38]. The combination of the iteration and feedback characteristics gives participants the opportunity to revise and amend or reaffirm their views and perceptions as the study proceeds [45].

According to the principles underlying the Delphi method, the decision to stop the iterative process of questioning should be determined by the level of consensus achieved among the experts. The recommendation of the method is that at the end of each round the level of consensus among experts be evaluated and a decision made: to proceed to a new iteration, if the consensus level is not yet adequate to end the study, if such level is considered appropriate [45].

Once the study rounds are completed, the final opinion of the group is obtained by aggregating the views expressed by each of the individual experts in the last round [14].

Due to the features described, namely the use of experts, its iterative nature, and the existence of feedback mechanisms, the Delphi method is generally referred to as being particularly suitable for performing studies aimed at exploring and describing concepts, topics, or phenomena insufficiently mature, for which there is yet no established evidence base [3], [45]. The advantage of using a Delphi method in such contexts results from the fact that in a Delphi study the process of knowledge gathering and generation is based on the views and perceptions of experts which have practical experience on the study area [38], [45]. Additionally, since the Delphi method has an iterative nature, it is possible to include new ideas and discoveries as the study progresses [61], which allows a more thorough exploitation of the phenomenon, something impossible in an non-iterative form of questionnaire. Finally, the fact that the Delphi method is a group technique that has feedback mechanisms, allowing that the participating experts reconsider their initial choices in the light of opinions and perspectives expressed by the remaining group [61], greatly contributes to greater consistency of the results.

Thus, after a careful assessment of the characteristics of the Delphi method and its comparison with other alternative techniques, such as traditional questionnaire or focus groups, and taking into consideration the type and the exploratory nature of the research questions of this study (focused on discovering the relevant forces, on identifying their level of importance, their type of influence, and their current configuration in the Portuguese PA) the decision to perform a Delphi study seemed to be rather feasible and adequate in this research.

3.2      Delphi Study Design

Some of the main criticisms found in the literature about the Delphi method are not related to the method itself but with how the method is used (or misused) by researchers. As noted by many authors, most of the Delphi studies reported in the literature neither explain clearly how the study was planned and executed nor give enough information about the decisions and options come to during its design [34], [38], [55]. Considering these criticisms, the following paragraphs set out the decisions made for each of the eight aspects pointed out in the literature as important elements in the design of a Delphi study, namely: panel constitution, experts invitation, starting of the first round, predefined list generation, structure of the questionnaire, stopping criterion of rounds, mode of execution, and feedback information.

3.2.1      Panel Constitution

The constitution of the panel is quoted as one of the most critical issues in a Delphi study. Despite its importance, there are not in the literature clear prescriptions of how this process should be conducted.

One of the few studies that provide a more structured and detailed procedure for the constitution of the panel is authored by Okoli and Pawlowski [55]. Based on a set of recommendations proposed by Delcebq to form nominal groups, Okoli and Pawlowski defined an iterative procedure to identify the experts to be included in a Delphi panel. This procedure served as inspiration to the constitution of the expert panel used in this study.

The adaptation of Okoli and Pawlowski's procedure to the context of this study resulted in the execution of the following five steps: Step 1 - Define inclusion criteria; Step 2 - Define key searching niches; Step 3 - Populate niches with names; Step 4 - Invite experts and request indication of new experts; and Step 5 - Invite new experts.

The aim of the first step was to define the criterion that should be verified by all the individuals that could be included in the panel. Based on the "expert" definition provided in the Dictionary of the Academy of Sciences—"a person who knows from experience, and that acquired a vast knowledge in a particular subject; a specialist in a given activity" [1]—and being the term "specialist" defined, in the same Dictionary, as "a person who understands and knows a particular subject or matter" [1], the inclusion criterion adopted in the study was set as any individual who has been involved in initiatives that aim at promoting and implementing interoperability between information systems in Public Administration.

The second step was the identification of the main searching niches. A niche represents a generic area or context where it is likely to find individuals that may fit the inclusion criteria defined in Step 1. Three main niches were identified: PA agencies, IS/IT industry, and academy. Since the focus of this work is on the study of IS interoperability initiatives in PA, public administration agencies were the primary and core searching niche for the identification of experts to include in the panel. Besides those, professionals from IS/IT industry that are product suppliers or services providers of PA agencies were also considered eligible to participate in the panel, as well as academics (scholars or researchers) that could have been involved in efforts or initiatives related to the promotion of IS interoperability in PA, particularly through their participation in consultancy and advisory services.

The third step intended to identify the names of the individuals in each niche that could be invited to integrate the panel. The search that was undertaken focused on the Portuguese context. The strategies followed to identify the expert names included the searching for names (1) on finished or ongoing cross-agency projects, (2) on commission, units, institutes, and associations whose objectives, functions, and activities are related to the promotion of interoperability in PA, (3) on the participants list of the Annual Meeting of Heads of Information Systems and Informatics of Public Administration, which is a forum for debate on the issues and challenges facing the PA, (4) on the web sites of PA agencies, (5) on IS/IT publications available in the PA community of practice, and (6) on personal contact lists. A total number of 101 individuals were identified.

The 101 individuals identified were invited in Step 4. In addition to the invitation, the individuals were also asked to suggest the names of other experts that they thought could be interesting to include in the panel.

Finally, in Step 5, the thirty-nine new names suggested in Step 4 were invited to participate in the study. In total, 140 experts were invited (101 identified in Step 4 and 39 in Step 5), of which 89 (63%) belonged to Public Administration agencies, 36 (26%) to IS/IT industry and 15 (11%) to academy.

3.2.2      Experts Invitation

The second decision made during the design phase was how to invite the members of the panel to participate in the study.

The two most usual alternatives mentioned in the literature are either to send an invitation asking experts to participate in the study or to send the invitation when the study starts, i.e. to merge the invitation and the start of the first round. Pondering carefully both alternatives, the decision made was to send the invitation before the beginning of the first round. This decision stemmed from the consideration that the experts' likelihood to accept the invitations to participate in the study would be bigger, since psychologically it looks easier for a person to accept an invitation to participate in an activity that will only take place later. Moreover, it was also considered that if an expert had previously accepted to participate in the study, he was going to feel compelled to answer when later he receives the announcement of the beginning of the study, thus leading to higher response rates.

From the 140 experts invited, 55 (39%) accepted to participate in the Delphi study, of which 35 (63%) belonged to Public Administration agencies, 12 (22%) to IS/IT industry and 8 (15%) to academy.

3.2.3      Starting of the First Round

The third decision made was how to start the first round of the study. The two possible alternatives were the blank sheet approach (approach used in the conventional Delphi method) [13], [18], or the predefined list approach (variant that has been frequently adopted) [7] - [9], [19], [27], [30], [51] - [52], [54], [59]. Despite the advantages usually attributed to the blank sheet approach, namely the inexistence of any initial bias and the non-imposing of restrictions on the options and creativity of the participants, the decision was to adopt the predefined list approach. In fact, although the adoption of a blank sheet approach could seem to be the "ideal desired" approach for this study, it revealed itself, however, as a more risky alternative, not only because it usually requires the execution of a greater number of rounds [59] but also because it calls for an increased effort by participants, causing higher levels of wear that could lead to higher dropout rates. On the contrary, the existence of a predefined list in the first round could generate in participants a sense that the effort they would have to make would be less than if they were asked to identify and indicate the relevant forces by their own initiative. Thus, while not being the "ideal desired", the adoption of a predefined list was considered the "ideal feasible" option to adopt in this study.

3.2.4      Predefined List Generation

The decision to start the Delphi study with a predefined list of forces immediately raised the question of how to generate that list.

There were three alternatives envisaged at this level, namely to generate the list from a literature review, from a group discussion, or from a mixed strategy that starts with a literature review process which is followed by a group discussion evaluation. Properly weighing the pros and cons of each of the three alternatives the decision was to obtain the list of forces by a literature review process.

In order to mitigate some of the criticisms of this alternative, namely the bias that could be introduced and the incompleteness of the generated list, two additional decisions were made. The first was to maintain the list open along the rounds. This allowed experts to add new forces that they deemed relevant but that were not yet included in the list, thus mitigating the second criticism mentioned. The second decision was to order alphabetically the forces that made up the list submitted in Round 1, thus mitigating the criticism that the use of a predefined list could introduce bias in the responses of the panel members.

Summarizing, the decision about how to start the first round was to use a predefined list of alphabetically sorted forces obtained by a literature review process and kept opened along all the rounds of the Delphi study.

An extensive literature review process, encompassing multiple areas, was conducted. Besides the eight studies mentioned in section 2.2, journal and conference articles in the area of inter-organizational collaboration, organizational management, change management, public management, political science, and information systems management were also analyzed. Beyond journal and conference papers, also technical reports, deliverables of research projects, benchmarking reports, green papers, white papers, as well as documents published by governmental entities and public agencies, from different countries, about e-government, and administrative reorganization and modernization were also read.

The literature review process produced a list of 28 forces whose names are presented in Table 8. A brief description of each of those forces is provided in Appendix A.

3.2.5      Structure of the Questionnaire

The fifth decision made was how to structure the questionnaire so that it would allow gathering the data needed to answer the four research questions, namely what are the relevant forces, what are their importance levels, which are their current configurations, and which type of influence do they exert in IS interoperability initiatives.

The questionnaire was structured in three main phases, the first devoted to assessing the level of importance of each force, the second devoted to assessing the current configuration and the type of influence of each force, and the third dedicated to the identification of new relevant forces.

The level of importance was evaluated through the use of Q-sort. Q-sort is a ranking technique that follows a structured procedure that makes respondents to analyze and evaluate the importance of the various forces conjointly, being the classification assigned to a force the result of a series of comparative judgments of the multiple forces [25], [47], [50]. This procedure also makes the respondent to attribute different importance levels to each of the forces, thus preventing experts to focus their responses in the same area of the classification scale, reducing the possibility of collisions and ties among the forces ranks [9].

The evaluation of the configuration of each force and of the type of influence that each force may exert over IS interoperability initiatives was held in an interrelated form. Both elements were evaluated based on the use of dichotomous nominal scales (true/false for the configuration and driving/restraining for the type of influence). For each of the forces a proposition was made stating one possible configuration of the force, being the experts asked, first, to say if they consider the proposition true or false in the current context of Portuguese PA and, second, to say if they consider that the indicated configuration for the force was driving or restraining the implementation of IS interoperability initiatives.

Lastly, the instrument used for identifying new forces was quite simple, corresponding to a free text area where experts were invited to identify and describe new forces that they deemed to be relevant but that were not included in the existing list.

After its design, the questionnaire was pretested by six academics in order to assess its readability as well as the existence of inconsistencies.

3.2.6      Stopping Criterion of Rounds

The sixth decision made regarded the criterion to stop the iterative process of the rounds execution. According to the principles underlying the Delphi method, the stopping criterion of rounds is determined by the existence of consensus among the experts. The recommendation of the method is that at the end of each round the level of consensus be evaluated, and, based on it, a decision be made: to proceed to a new iteration if the level of consensus is not significant or to end the study if the level is considered appropriate [45]. Given the iterative nature of a Delphi study, it was decided that the assessment of the level of consensus would include two components: the level of agreement of the experts' opinion in the round and the level of stability of the global panel opinion between rounds. The level of agreement of the experts' opinion assesses the homogeneity or consistency of the opinion expressed by experts in each round [28]. The level of stability of the global panel opinion assesses whether the view expressed by the overall panel stabilizes over the rounds [55], [60] - [61].

The assessment of each of these two components of consensus was based on statistical measures. Since there were three different topics asked in the questionnaire— the level of importance of each force, its configuration, and the type of influence it exerts—different measures were used. Table 9 summarizes the statistical tests performed. All of them are nonparametric tests, since the data generated was either of an ordinal type or of a nominal type.

The Kendall's W coefficient of concordance was chosen to evaluate the level of agreement among the experts' individual rankings of importance in a given round. This coefficient is widely recognized not only by the quality of results it produces but also by the simplicity of its implementation [55]. The value of W varies between 0 (no agreement) and 1 (perfect agreement) [64]. A high value of W means that the n experts are judging de level of importance of the m forces in a similar way. Some of the studies that used this statistical measure were [7], [55], and [59].

The Spearman's rho correlation coefficient was the statistical measure used to evaluate the stability of the global panel rank of importance between rounds. This measure is used to determine the extent of association or correlation between two rankings [64]. At the end of each round, the n individual rankings were combined to generate the global panel ranking for that round, which was then used, together with the global panel ranking of the previous round to calculate the Spearman's rho coefficient. Two studies that used this statistical measure were [59] and [68].

The assessment of the level of experts' agreement regarding the configuration and the type of influence of each force was based on the use of Binomial tests. The Binomial test compares proportions and is generally used to test the occurrence of one of the two realizations of a dichotomous variable [46].

Lastly, the assessment of stability of the global panel opinion regarding the configuration and the type of influence of the forces was based on the use of McNemar tests. The McNemar test is used to test scores or proportions of dichotomous nominal variables in two-paired samples. This test is particularly suitable for studies of the type "before vs. after", allowing to test the significance of the change of opinion [46].

Besides the level of consensus, measured by the level of agreement of the experts and the level of stability of the overall opinion of the panel between rounds, an additional condition—the maximum number of rounds—was also included in the stopping criterion defined for this study. Although this condition could seem somehow controversial, as it is contradictory to the basic Delphi principle of keeping on the rounds until consensus is reached, several studies were found in the literature where no consensus was reached among experts, having the studies stopped not because of the consensus level achieved but because of the high number of rounds performed [12], [34].

Shortly, in each round of the study, the decision of stopping or not the Delphi iterative process was made based on the calculated values of Kendall's W coefficient of concordance, Spearman's rho coefficient of correlation, Binomial tests, and McNemar tests, provided that the total number of rounds was not greater than three.

3.2.7      Mode of Execution

The seventh decision made during the design phase concerned the mode of execution of the Delphi study. After a careful reflection, the decision was to execute the process via the Web. The use of a Web system seemed to have multiple advantages when compared with the use of the traditional form of mail. The advantages may occur at four levels: process efficiency level, budget level, data quality level, and response facility level.

The process efficiency gains arise, primarily, from the fact that the use of a Web system makes the entire communication process (from sending to receiving) considerably faster than when it is conducted via traditional mail [20] and, secondly, from the fact that the required data processing time is lower since the data is already in a digital format, being not necessary to make its manual insertion in the statistical analysis application [59].

At the budget level, the benefits are evident, due to the considerable costs that are associated to conventional mail sending, including costs of paper, printing, and shipping. While the use of the Web could have costs related to the acquisition and support of the Web system [59], the amount involved in this research project was insignificant, since the applicational and technological Web infrastructure was already available for use.

The advantages at the data quality level stem mainly from the fact that using a Web system makes the experts to follow the researcher intended response procedure. This was particularly relevant in this study since one of the questions used the Q-Sort technique, which involves the realization of a specific sequence of steps. The data quality could also be better because as the data will be in digital format there would be no need to manually insert it, thus minimizing the errors that frequently are associated with the manually insertion of data.

Finally, the use of a Web system is also advantageous to the experts, since the task of responding to a question that uses Q-Sort technique by traditional means poses a series of requirements, such as the need to have a totally clean table where various paper cards could be placed, which would make the process much more complex and tedious to the experts.

It is worth mentioning that the use a Web system to support the Delphi study required some special attention, namely in what regards the monitorization of the system in order to ensure that it is was always available. Another concern was to guarantee that no data was lost. To prevent such occurrence, daily copies of the database used by the Web system were made.

3.2.8      Feedback Information

The eighth decision made concerned to what feedback information should be sent to the experts in each new round of the study. The provision of feedback information is a typical and core feature of the Delphi method. It allows the sharing of visions and perceptions among the experts, thus contributing to generate a group consensual opinion about the subject. The decision made in this study was to include the following elements as feedback information:

      •   Regarding the level of importance - the rank position obtained by the each force, as well as the sum of points it scored, its average, its standard deviation, and its variance

      •   Regarding the configuration and type of influence - the percentages obtained by each force in each of the classification categories available for each of the two evaluated elements (percentage of true and false answers in what concerns the force configuration and percentages of driving and restraining in what concerns the type of influence exerted by the force)

Additionally, it was also provided to each expert access to the individual answer he provided in the previous round of the study. Some qualitative indications, such as not satisfactory, satisfactory, and very satisfactory, about the group's level of consensus reached in the previous round were also included.

3.3      Delphi Study Execution

This section briefly describes how the Delphi study was executed.

Table 10 summarizes the main figures of the study execution.

The overall duration of the Delphi study was 53 days, being the duration of each of the three rounds 13, 14, and 9 days, respectively. The time periods between rounds were of 15 days between Round 1 and Round 2 and of two days between Round 2 and Round 3.

In all the rounds the questionnaire was sent to the 55 experts that had previously accepted the invitation to participate in the study, even if they had missed to answer some round. The response rates obtained were 82% in Round 1, 65% in Round 2, and 73% in Round 3.

The list of forces submitted to evaluation contained 28 forces in Round 1, 31 forces in Round 2 (the 28 forces of the predefined list used in Round 1 plus three new forces suggested by experts in Round 1), and 31 forces in Round 3 (the same forces that were submitted in Round 2).

Table 11 contains the names of the three new forces suggested by experts in Round 1. A brief description of each of those forces is provided in Appendix B.

Taking into account the value of the statistical tests performed for each of the three topics questioned (level of importance, configuration, and type of influence) the level of consensus in Round 1 and Round 2 were evaluated as not enough. In Round 3 the statistical values calculated revealed the existence of sufficient levels of consensus, thus leading to the study termination. The detailed values of the Binomial and McNemar tests are available in Appendices

4      Delphi Study Results

The results presented in this section reflect the views, opinions, and experience of the 40 experts that participated in the third and last round of the Delphi study, of which 22 (55%) are PA professionals, 11 (27.5%) IS/IT industry professionals, and 7 (17.5%) academics. The 22 PA professionals belonged to seven different Portuguese ministries: Ministry of Justice (5), Ministry of Labour and Social Solidarity (5), Ministry of Finance and Public Administration (3), Ministry for Internal Administration (2), Ministry of Culture (1), Ministry of Science, Technology and Higher Education (1), and the Presidency of the Council of Ministers (4).

Along the study, a total of 31 forces relevant to the implementation of IS interoperability initiatives in PA were identified.

4.1      Level of Importance of the Forces

The first request made to the experts in the Delphi questionnaire was to rank the set of listed forces according to their level of importance to the promotion and implementation of IS interoperability initiatives in PA. The result of this request is systematized in Table 12. Beyond the rank position of each force, the table also contains additional descriptive statistical measures, such as the average, standard deviation, minimum, maximum, range, and interquartile range that help in the understanding and interpretation of the results.

The first evidence that stands out from the table is the high importance caught by the force Political Will. The importance of this force is notorious, not only by the first rank position it obtained, but also from the frequency of responses that it got. As can be seen from Figure 1, 28 of the 40 experts ranked that force precisely the first rank position, meaning that 28 of the 40 experts considered Political Will as the most important force among the 31 forces under evaluation. Only three experts ranked Political Will out of the top 10 positions.

Figure 1: Frequency distribution of Political Will

Looking to the values in columns SD, Range, and IQR in Table 12, it becomes clear that Political Will was the force that gathered the highest level of convergence among experts' opinions. Indeed, though the range of this force is still of 15 positions, it is the lower when compared with the ranges of the other 30 forces. In the same sense, the average value of this force also reveals the great importance attributed to this force. Not only is this value the lowest of all the forces, as the difference between the average value of this force (Political Will) and the next force in the ranking (Involvement and Commitment of Agencies) is of 3.33 points, which corresponds to the largest "jump" found in the table.

Together, these facts show the existence of a strong conviction among the experts on the importance that the force Political Will assumes in the promotion and implementation of IS interoperability initiatives in PA. Actually, a favorable and genuine Political Will, characterized by the existence of a strong sponsorship and commitment on the part of politicians, can be highly beneficial to achieve interoperability in PA, since politicians may (1) instigate and promote interoperability projects; (2) channel funds for such type of initiatives; (3) promote the public debate on fundamental but controversial questions, such as privacy and personal data protection; (4) facilitate the creation or amendment of the required legislation, and (6) unblock and make disappear many bureaucracies usually surrounding this type of initiatives.

The support of politicians can also be fundamental to "make agencies to participate" in cross-agency initiatives. Because interoperability initiatives involve the participation of various agencies, such initiatives require the harmonization of multiple and different strategies, processes, people, and technologies. This process of harmonization is not a trivial task, being most of the times difficult to achieve without the intervention of political agents.

The second evidence from the table is the existence of different groups of importance. As can be seen from the A of Average column, which contains the difference between the averages of forces in successive ranking positions, there are some considerable "jumps" between some forces, particularly between rank 1 and rank 2, between rank 5 and rank 6 and between rank 30 and rank 31. These "jumps" reveal the existence of four groups of importance. The first group, whose elements may be considered as having extreme importance, includes the Political Will force. The second group, which can be considered as comprising the very important forces, includes forces in rank positions 2, 3, 4, and 5. The third group, containing the important forces, includes the remaining forces of the list, with the exception of the force Constitutional Principles, which, alone, constitutes the fourth group, regarded as the least important. Thus, despite the importance that all forces can take in the course of interoperability initiatives, the five forces belonging to the first and second groups, namely Political Will, Involvement and Commitment of Agencies, National Structure for Interoperability Governance, Standards for Interoperability, and Intra- and Interagency Leadership, should receive special attention.

In fact, it is clearly understandable that the existence of involvement and commitment on the part of all agencies constitutes a fundamental condition to the success of interoperability. Indeed, since, by definition, interoperability always involves the conjunct operation of two or more entities, it will only be achievable if all the parts are actively involved and committed, being the presence of each of the agencies absolutely indispensable.

The importance associated to the existence of a National Structure for Interoperability Governance is also understandable. Considering, as mentioned in section 2.1, that, to achieve e-government interoperability, more important than to find out the way of "being together", is to find out "the most effective way of being together", it seems fundamental to address the interoperability endeavor in a comprehensive, rational, structured, and sustained way. This means that the interoperability must be thought in a global form: it must be ensured the consistency among the policy and the technical strategies of the multiple agencies; it must be promoted the convergence of the agencies' actions in crucial areas such as security and privacy; there must be defined and proposed appropriate funding mechanisms; there must be defined global and rigorous mechanisms for monitoring interoperability progress, and there must be identified and provided shared services common to several agencies in order to avoid the duplication of effort and costs. In other words, it seems therefore important to define and implement a governance model for interoperability focused on the overall direction and control of the multiple collective efforts for interoperability. The setting of central and continuous processes and mechanisms of governance, that allow the establishment of unified policies, consistent architectures, common specifications and standards, and compatible procedures and infrastructures, are critical factors to build a truly interoperable PA.

The high level of importance attributed to Standards for interoperability also makes sense. Given the existent disparities at technological, semantic, process, and security levels, it becomes fundamental to have norms that allow the correct exchange of data and sharing of meaning among the parts. The inexistence of standards widely agreed and adopted by the agencies will require new negotiations and agreements every time a new initiative is initiated, leading to additional and unnecessary delays, conflicts, and costs.

Equally understandable is the importance of Leadership either at an Intra- or Interorganizational level. Indeed, while being considered as a very important issue in all kinds of projects, leadership assumes an even more fundamental role in initiatives involving multiple actors, that have different missions, interests, and priorities and that operate in disparate technical, semantic, and organizational contexts, as happens in interoperability initiatives in PA. It is crucial that there is someone or some entity responsible for the global leading of the initiative; someone that is able to promote and oversee the overall initiative, to motivate all the involved agencies, to manage effectively the different interests and perceptions among the multiple agencies, and to conveniently conduct the negotiations among the various agencies. It seems also essential that this global leadership be accompanied by the existence of leadership actions at the internal level of each agency. The presence in each agency of a person that is truly committed to the initiative, that is able to introduce and "sell" internally the initiative, that understands the costs and benefits that may be involved, and, specially, that is able to manage the organizational impact that the agency may suffer as a result of its involvement in an interoperability initiative, is primordial.

The substantial importance reached by these five forces is also reinforced by the fact that, as can be seen from the first and second columns of Table 12, they have settled on the top positions right from the first round, remaining in the front five ranks along all the three rounds of the study, what denotes a strong belief of the experts on their importance to the initiatives.

Another evidence that follows from the analysis of column Range in Table 12 is that there is, for the majority of the forces, a great diversity of opinion among experts, with forces such as People's Attitude Toward Change, National Structure for Interoperability Governance, and Electronic Signature, being classified in all of the thirty one possible rank positions. It is important to note, however, that a broad range may be associated to two different realities. One reality is when to a wide range value corresponds a broad IQR, meaning that there is indeed a wide spread of opinion among experts. This happens, for example, with the forces Electronic Signature, Domain Ontology for Public Administration, and People's Attitude Toward Change. Another reality is when to a wide range value corresponds a narrow IQR, meaning a situation where the majority of experts has similar opinions, but there is one or few experts who have a considerably different view, opinion, or experience on the force. These cases are usually called outliers [46]. Forces in rank 1, 2, 3, 4, 5, 6, 25, 26, 27, 28, and 29 are forces for which there were found outliers. The maximum number of outliers found was of 8 for force in rank 1, and the minimum number was of 1 for forces in rank 26 and 29.

4.2      Configuration and Type of Influence of the Forces

The second request made to experts in the Delphi questionnaire was to state the configuration of each force in the current context of the Portuguese PA as well as the type of influence exerted by each force in the IS interoperability initiatives. As already mentioned, the way found to get the answers to these questions was based on the formulation of a statement describing a possible configuration of each force and on asking experts firstly whether that statement was true or false in the current context of Portuguese PA and secondly how the current configuration indicated could influence (driving or restraining) the implementation of IS interoperability initiatives in PA. Table 13 summarizes the responses given by experts. Considering that an expert evaluated the type of influence based on the answer he gave about the configuration of the force, the percentages concerning the type of influence exerted by each force are presented, in the table, separately for the two possible situations: the situation that corresponds to a true configuration and the situation that corresponds to a false configuration.

<

4.2.1      Configuration of Forces in Current Context of Portuguese PA

Using the percentages presented in Table 13, it was possible to advance some specific statements about the configuration of each force in the current context of Portuguese PA. The statements proposed were only those to which statistical support was found, i.e. the statements that refer to forces to which the Binomial test produced evidence values (p-values) lesser or equal than 0.05, which means that there is a statistically significant difference between the number of experts who considered the statement presented as true or as false. As can be seen from the Binomial test values available in Appendix C, this criterion (p-value ⇐ 0.05) is verified for 27 forces, thus allowing the formulation of the 27 propositions presented in Table 14.

4.2.2      Type of influence Exerted by the Forces

The percentages shown in Table 13 also allowed the formulation of statements about the type of influence exerted by each force in the promotion and implementation of IS interoperability initiatives in PA. Again, the statements formulated were only those to which statistical support has been achieved, i.e. the statements that refer to situations to which the Binomial test produced evidence values (p-values) lesser or equal than 0.05, which means that there is a statistically significant difference between the number of experts who considered that the force in the situation described would have a driving or a restraining role.

Since the evaluation of the type of influence was dependent on the evaluation of the configuration, the Binomial tests for the type of influence had to be done twice for each force, performing a total of 62 tests (2 tests for each of the 31 forces).

As can be seen from the Binomial test values available in Appendix E, the criterion p-value ⇐ 0.05 is verified in 38 situations, allowing the formulation of 38 generic propositions, 24 describing situations in which a force has a restraining or limiting influence on IS interoperability initiatives (Table 15) and 14 describing situations in which a force has a driving or facilitating influence on IS interoperability initiatives (Table 16).

4.2.3      Constellation of Forces Acting in Current Context of Portuguese PA

Taking the general results set forth previously about the level of importance of each force (Table 12) and the type of influence that a force may exert (Table 15 and Table 16) and combining them with the specific configuration that each force assumes in the current context of Portuguese PA (Table 14), it was possible to depict the global constellation of forces acting on the phenomenon of promotion and implementation of interoperability between IS in the current context of the Portuguese PA.

The simple and intuitive way found out to gather this information was through the use of a Force Field Diagram (FFD), whose concept was introduced by Kurt Lewin. As the name indicates, a FFD is a graphical representation of the set of forces that act, in a given moment, on an entity (individual, group, or organization), phenomenon, or situation. Schematically, a FFD has the configuration of a "T": on the left side of the vertical axis of the "T" are represented the forces driving or facilitating the occurrence of the phenomenon and in the right side of the axis are represented the forces restraining or limiting the phenomenon.

According to the terminology and rules of FFD diagrams construction, each of the 31 forces identified and evaluated by the experts in the Delphi study were represented by an arrow. The arrow direction is determined by the type of influence that the force exerts on IS interoperability initiatives and its length is determined by the position achieved by the item in the ranking of importance (longest arrows correspond to more important forces and shorter arrows to less important forces). The forces were presented along the vertical axis in order of importance. Thus, an arrow that is in a higher position in the vertical axis represents a force that has obtained a top position in the ranking of importance, and that was deemed more important by the panelists.

Each arrow has two text elements associated: one above the arrow which shows the name of the force and the other below the arrow which states the configuration of the force in the actual context of the Portuguese PA. To visually reinforce the type of each force, red color was used to represent restraining forces and green color to represent driving forces. The resulting FFD is depicted in Figure 2.

Figure 2: Constellation of the forces acting on the promotion and implementation of IS Interoperability initiatives in the current context of the Portuguese public administration

As may be seen from the diagram, the forces People's Attitude Toward Change, Net Benefits Perceived by Agencies, Interagency Trust, Privacy and Protection of Personal Data, Technological Incompatibility, Changes in the Statu Quo of Agencies, and Constitutional Principles are not depicted, since there was not statistical significance to allow their representation.

It is quite clear from the diagram the existence of a large number of forces restraining or limiting the progress of IS interoperability initiatives in PA. Indeed, from the 24 forces represented in diagram, only 7 forces assume, in the current context of Portuguese PA, a driving or facilitating role, being 17 of them restraining factors of such initiatives.

Looking to the top five, that contains de forces classified as extreme and very important, it can be seen that three of them (National Structure for Interoperability Governance, Standards for Interoperability, and Intra- and Interagency Leadership) are restraining the promotion and implementation of IS interoperability initiatives in the current context of Portuguese PA. Thus, according to the experts' opinion expressed in this study, these three particular forces should be carefully managed and some forms of intervention should be identified and put in place in order to mitigate the restraining or limiting effects caused by the current configuration that theses forces assume in Portuguese PA in order to be able to achieve the adequate and desirable levels of interoperability.

5      Conclusion

The work described in this paper constitutes a step forward in the search for a deeper understanding on the complex phenomenon of achieving more adequate, more sustained, and more sustainable levels of interoperability in Public Administration.

The aim of this study was to unveil the myriad of forces that may influence and determine the implementation of IS interoperability initiatives in PA. More specifically, this paper sought to identify the various aspects of the reality— forces—acting on IS interoperability initiatives in PA, as well as the level of importance assumed by each of those forces and the type of influence that each force may exert on the interoperability initiatives.

In order to achieve this aim, a Delphi study was performed. Based on the Delphi study, a set of "instruments" was produced, namely:

      1.   A list of 31 forces relevant to IS interoperability initiatives in PA (Table 12). These forces have different natures—political, legal, organizational, semantic, and technical—evidencing and supporting the arguments about the multidimensionality of the interoperability construct.

      2.   The ranking of importance of the 31 forces, showing the relative level of importance of each of the forces (Table 12). This ranking brought to evidence the existence of four clusters of forces: the extreme important, the very important, the important, and the least important forces.

      3.   A set of 24 propositions describing restraining or limiting influences of the forces on the promotion and implementation of interoperability initiatives in PA (Table 15).

      4.   A set of 14 propositions describing driving or facilitating influences of the forces on the promotion and implementation of interoperability initiatives in PA (Table 16).

Besides these four "instruments", the Delphi study also allowed the formulation of a set of 27 propositions that characterize the configuration assumed by each force in the current context of Portuguese PA (Table 14). Combining and instantiating the previous four "instruments", in the light of these 27 configuration propositions, a FFD was drawn representing the overall constellation of forces acting on the phenomenon of IS interoperability in the specific current context of Portuguese PA (Figure 2).

This study represents a valuable contribution to the implementation and management of IS interoperability initiatives in PA. Indeed, besides alerting the practitioners to the set of aspects that may affect this type of initiatives, the study also offers practitioners new insights, concerning the level of importance assumed by each of those aspects as well as about the type of influence they may exert in interoperability initiatives, that haven't, until now, been covered in the literature.

In a concrete way, PA professionals may easily use the four "instruments" produced in this study to make a diagnostic of their PA readiness status to interoperability achievement that may help them to manage its promotion and implementation. The FFD generated in this study to the specific Portuguese PA is an example of that. Indeed, looking at that FFD (Figure 2), Portuguese PA professionals have an overall perspective of what is relevant, how relevant it is, and how it is affecting the promotion and implementation of IS interoperability in PA in Portugal. In the same sense, this set of "instruments" may be used by any other particular PA to generate its specific FFD. The only thing that this other PA needs to do is to identify its specific set of configuration propositions. Having that set of propositions (a set of propositions similar to the one presented in Table 14), it is possible to generate the specific FFD for that particular PA, and thus to have an overall perspective of the specific constellation of forces acting on it, based on which the practitioners may orientate their actions in order to achieve a constellation of forces that is more favorable to the promotion and implementation of interoperability in PA.

Additionally, PA professionals can also use these "instruments" to keep track of how they are progressing towards the creation of a more favorable constellation of forces to interoperability. Indeed, if they repeat the diagnostic periodically and compare the multiple generated FFDs along the time, they will be able to understand how well are they directing their efforts in order to promote and implement IS interoperability in PA.

Besides being beneficial to the community of practice, the results of this study are relevant to the research community, contributing to the enrichment of the body of knowledge in the area of e-government interoperability.

While the inclusion of the quest of the level of importance and of the type of influence of each force constitutes a step forward toward the understanding of the complex of forces acting in interoperability initiatives, this study has however two limitations. Both of these limitations refer to ideas and principles argued by Kurt Lewin, in his Field Theory, that were not contemplated in this study, namely the principle of the interdependence of the forces and the dynamic nature of the forces.

According to Lewin, the forces that act on a phenomenon do not exist in isolation; they are interrelated and influence each other— the principle of interdependence. To know how they interact and influence each other is a fundamental element in order to better understand and represent the complex of forces acting on the phenomenon. Lewin also argued that the set of forces that act on a phenomenon, as well as its importance and type of influence, are not static elements, rather having a dynamic nature that evolves along the time. Understanding how these forces, either individually or collectively, evolve, would improve the existing knowledge about the complex of forces acting on the phenomenon of implementation of interoperability between IS in PA.

Considering the richness that the inclusion of these two additional elements could bring to a deeper understanding of the complex of forces, two future research projects could and should be done. One project should be focused on the identification of the existent interdependencies among the forces. This study should use the list of 31 forces produced in this work and try to find out the net of dependencies that exist among those forces. The other project, that could be based, for instance, on the realization of longitudinal case studies, should be focused on understanding how the constellation of forces changes and evolves along the time.

Lastly, it could also be interesting to replicate this study in other countries. The existence of similar studies in different countries would enable an inter-contextual analysis of the phenomenon of IS interoperability in distinct public contexts. This would allow the investigation of the existent differences between countries and interpret those differences in the light of the countries' characteristics, making it possible to determine how the complex of forces acting on each country can be related, for example, with the level of PA technological development of that country, with the way political and administrative spheres relate to each other, with the existence or non-existence of some structure of IT governance in the country, with the existence of partnerships with private/public institutions, with the attitude of PA employees of each country, and with cultural issues. It is considered particularly interesting to carry out these studies in countries that present different levels of maturity of e-government development, particularly with respect to the level of interoperability between PA information systems already achieved. In the European context, countries such as Austria, Slovenia, Sweden, and Finland (characterized for providing public services with high levels of sophistication) as well as countries such as Switzerland, Slovakia, Poland, and Bulgaria (whose services exhibit lower levels of sophistication), are very interesting candidates to perform the study. Additionally, countries such as USA, Canada, Australia, Singapore and South Korea could also be investigated.

 

References

[I] ACL, Dicionário da Lingua Portuguesa Contemporánea da Academia das Ciências de Lisboa. Lisboa: Verbo, 2001.

[2] A. Y. Akbulut, An Investigation of the Factors that Influence Electronic Information Sharing Between State and Local Agencies, Ph.D. dissertation, Louisiana State University, 2003.

[3] H. Akkermans, P. Bogerd, E. Yücesan, and L. Wassenhove, The Impact of ERP on Supply Chain Management: Exploratory Findings From a European Delphi Study, European Journal of Operational Research, vol. 146, no. 2, pp. 284-301, 2003.

[4] V. Bekkers, The Governance of Back Office Integration: Some Dutch Experiences, in Proceedings of the 5th European Conference on E-Government, Antwerp, Belgium, 2005, pp. 57-65.

[5] N. Benamou, A. Busson, and A. Keravel, Impact of e-Government Interoperability in Local Governments, in Proceedings of the 3rd International Conference on Electronic Government, Zaragoza, Spain, 2004, pp. 82-87.

[6] BFC. (2005). Observatory on eGovernment Interoperability: Activities in 2004. TERREGOV (Project IST 507749). [Online]. Available: http://www.terregov.eupm.net/my spip/index.php?param=12.

[7] J. Brancheau and J. Wetherbe, Key Issues in Information Systems Management, MIS Quarterly, vol. 11, no. 1, pp. 23-45, 1987.

[8] J. Brancheau, B. Janz, and J. Wetherbe, Key Issues in Information Systems: 1994-95 SIM Delphi Results, MIS Quarterly, vol. 20, no. 2, pp. 225-242, 1996.

[9] M. R. Campos, Questoes Chave da Gestão de Sistemas de Informação: Avaliação da Situação Nacional, M.S. thesis, Department of Informatics, University of Minho, Portugal, 1998.

[10] D. Chen. (2005). Practices, principles and patterns for interoperability. INTEROP NoE, Deliverable 6.1, version 1.0. [Online]. Available: http://interop-noe.org/deliv/.

[11] CompTIA. (2004). European Interoperability Framework - ICT Industry Recommendations. CompTIA White Paper. [Online]. Available: http://www.e-innovation.org/stratinc/files/library/ict/15.ICT_standards.pdf

[12] D. J. Couger, Key Human Resource Issues in IS in the 1990s: Views of IS Executives versus Human Resource Executives, Information & Management, vol. 14, no. 4, pp. 161-174, 1988.

[13] N. Dalkey and O. Helmer, An Experimental Application of the Delphi Method to the Use of Experts, Management Science, vol. 9, no. 3, pp. 458-467, 1963

[14] N. Dalkey, The Delphi Method: An Experimental Study of Group Opinion. Santa Monica: RAND Corporation, 1969.

[15] S. Dawes, Interagency Information Sharing: Expected Benefits, Manageable Risks, Journal of Policy Analysis and Management, vol. 15, no. 3, pp. 377-394, 1996.

[16] S. Dawson and L. Manderson. (1993). A Manual for the Use of Focus Groups. Boston, MA: International Nutrition Foundation for Developing Countries. [Online]. Available: http://www.unu.edu/unupress/food2/UIN03E/uin03e00.htm#Contents.

[17] S. Dekleva and J. Zupancic, Key issues in information systems management: a Delphi study in Slovenia, Information & Management, vol. 31, no. 1, pp. 1-11, 1996.

[18] G. Dickson, R. Leitheiser, M. Nechis, and J. Wetherbe, Key Information Systems Issues for the 1980's, MIS Quarterly, vol. 8, no. 3, pp. 135-148, 1984.

[19] E. Doke and N. Swanson, Decision Variables for Selecting Prototyping in Information Systems Development: A Delphy Study of MIS Managers, Information & Management, vol. 29, no. 4, pp. 173-182, 1995.

[20] C. Dommeyer and E. Moriart, Comparing Two Forms of an E-mail Survey: Embedded vs. Attached, International Journal of Market Research, vol. 42, no. 1, pp. 39-50, 2000.

[21] EC. (2003). Linking up Europe: the Importance of Interoperability for eGovernment Services. IDA Program, European Commission. [Online]. Available: http://www.epractice.eu/files/media/media 591.pdf.

[22] EC. (2004). European Interoperability Framework for pan-European eGovernment Services. IDABC Program, European Commission. [Online]. Available: http://ec.europa.eu/idabc/servlets/Docd552.pdf?id=19529.

[23] EC. (2008). Draft document as basis for EIF 2.0, European Commission. [Online]. Available: http://ec.europa.eu/idabc/servlets/Docb0db.pdf?id=31597.

[24] EPAN. (2004). Key Principles of an Interoperability Architecture, EPAN - eGovernment Working Group. [Online]. Available: http://www.dgap.gov.pt/media/0601010000/irlanda/irlia1.pdf.

[25] J. V. Exel and G. D. Graaf. (2005). Q Methodology: A sneak preview. [Online]. Available: http://www.qmethodology.net/PDF/Q-methodology%20-%20A%20sneak%20preview.pdf.

[26] J. Gil-García, C. Schneider, T. Pardo, and A. Cresswell. (2005). Interorganizational Information Integration in the Criminal Justice Enterprise: Preliminary Lessons from State and County Initiatives, in Proceedings of the 38th Hawaii International Conference on System Sciences, Island of Hawaii. [Online]. Available: http://csdl2.computer.org/comp/proceedings/hicss/2005/2268/05/22680118c.pdf.

[27] P. Gottschalk, B. Christensen, and R. Watson, Key Issues in Information Systems Management Surveys: Methodological Issues and Choices in a Norwegian Context, Scandinavian Journal of Information Systems, vol. 9, no. 2, pp. 57-66, 1997.

[28] B. Graham, G. Regehr, and J. Wright, Delphi as a Method to Establish Consensus for Diagnostic Criteria, Journal of Clinical Epidemiology, vol. 56, no. 12, pp. 1150-1156, 2003.

[29] U. Gupta and R. Clarke, Theory and Applications of the Delphi Technique: A Bibliography (1975-1994), Technological Forecasting and Social Change, vol. 53, no. 2, pp. 185-211, 1996.

[30] S. Hayne and C. Pollard, A Comparative Analysis of Critical Issues Facing Canadian Information Systems Personnel: a National and Global Perspective, Information & Management, vol. 38, no. 2, pp. 73-86, 2000.

[31] C. Holsapple and K. Joshi, Knowledge Manipulation Activities: Results of a Delphi Study, Information & Management, vol. 39, no. 6, pp. 477-490, 2002.

[32] M. Hura, G. McLeod, E. Larson, J. Schneider, D. Gonzales, D. Norton, J. Jacobs, K. O'Connell, W. Little, R. Mesic, and L. Jamison. (2000). Interoperability: A Continuing Challenge in Coalition Air Operations. [Online]. Available: http://www.rand.org/publications/MR/MR1235/.

[33] L. A. Joia, Developing Government-to-Government Enterprises in Brazil: A Heuristic Model Drawn From Multiple Case Studies, International Journal of Information Management, vol. 24, no. 2, pp. 147-166, 2004.

[34] S. Keeney, F. Hasson, and H. McKenna, A critical Review of the Delphi Technique as a Research Methodology for Nursing, International Journal of Nursing Studies, vol. 38, no. 2, pp. 195-200, 2001.

[35] A. Keller, Future development of electronic journals: a Delphy survey, The Electronic Library, vol. 19, no. 6, pp. 383-396, 2001.

[36] M. Khosrow-Pour and N. Herman, Critical issues of Web-enabled technologies in modern organizations, The Electronic Library, vol. 19, no. 4, pp. 208-220, 2001.

[37] W. King, P. Marks, and S. McCoy, The Most Important Issues in Knowledge Management, Communications of the ACM, vol. 45, no. 9, pp. 93-97, 2002.

[38] J. Landeta, Current Validity of the Delphi Method in Social Sciences, Technological Forecasting and Social Change, vol. 73, no. 5, pp. 467-482, 2006.

[39] D. Landsbergen and G. Wolken, Realizing the Promise: Government Information Systems and the Fourth Generation of Information Technology, Public Administration Review, vol. 61, no. 2, pp. 206-220, 2001.

[40] K. Lewin, Field Theory and Experiment in Social Psychology, in Field Theory in Social Sciences: Selected Theoretical Papers. New York: Harper & Brothers, 1939, pp. 262-278.

[41] K. Lewin, Formalization and Progress in Psychology, in Field Theory in Social Sciences: Selected Theoretical Papers. New York: Harper & Brothers, 1940, pp. 169-190.

[42] K. Lewin, Psychological Ecology, in Field Theory in Social Sciences: Selected Theoretical Papers. New York: Harper & Brothers, 1943, pp. 289-300.

[43] K. Lewin, Behavior and Development as a Function of the Total Situation, in Field Theory in Social Sciences: Selected Theoretical Papers. New York: Harper & Brothers, 1946, pp. 337-381.

[44] K. Lewin, Frontiers in Group Dynamics, in Field Theory in Social Sciences: Selected Theoretical Papers. New York: Harper & Brothers, 1947, pp. 301-336.

[45] H. Linstone and M. Turoff, Introduction, in The Delphi Method: Techniques and Applications, Reading: Addison-Wesley, 1975, pp. 3-12.

[46] J. Maroco, Análise Estatistica - Com Utilização do SPSS. 2nd Ed., Lisboa: Edições Silabo, 2003.

[47] B. McKeown and D. Thomas, Q Methodology. Newbury Park: Sage Publications, 1988.

[48] P. Miller, Interoperability: What is it and Why should I want it?. Ariadne Magazine, is. 24, June 2000.

[49] W. Moen, Mapping the Interoperability Landscape for Networked Information Retrieval, in Proceedings of the First ACM/IEEE-CS Joint Conference on Digital Libraries, Roanoke, 2001, pp. 50-52.

[50] E. Morgado, N. Reinhard, and R. Watson. (1999). Adding Value to Key Issues Research Through Q-Sorts and Interpretive Structured Modeling, Communications of the Association for Information Systems. [Online]. vol. 1, no. 3, Available: http://portal.acm.org/citation.cfm?id=374112.

[51] P. Mulligan, Specification of a Capability-Based IT Classification Framework, Information & Management, vol. 39, no. 8, pp. 647-658, 2002.

[52] S. Nambisan, R. Agarwal, and M. Tanniru, Organizational Mechanisms for Enhancing User Innovation in Information Technology, MIS Quarterly, vol. 23, no. 3, pp. 365-395, 1999.

[53] D. Nevo, I. Benbasat, and Y. Wand, Exploring Meta-Knowledge for Knowledge Management Systems: A Delphy Study, in Proceedings of the International Conference on Information Systems (ICIS 2003), Seattle, Washington, 2003, pp. 439-449.

[54] F. Niederman, J. Brancheau, and J. Wetherbe, Information Systems Management Issues for the 1990's, MIS Quarterly, vol. 15, no. 4, pp. 475-500, 1991.

[55] C. Okoli and S. Pawlowski, The Delphi Method as a Research Tool: An Example, Design Considerations and Applications, Information & Management, vol. 42, no. 1, pp. 15-29, 2004.

[56] T. Pardo, A. Cresswell, S. Dawes, and B. Burke. (2004). Modeling the Social & Technical Processes of Interorganizational Information Integration, in Proceedings of the 37th Hawaii International Conference on System Sciences, Island of Hawaii. [Online]. Available: http://csdl2.computer.org/comp/proceedings/hicss/2004/2056/05/205650120a.pdf.

[57] G. Pervan, Information Systems Management: An Australian View of the Key Issues, Australasian Journal of Information Systems, vol. 1, no. 2, pp. 32-44, 1994.

[58] G. Rowe and G. Wright, The Delphi Technique as a Forecasting Tool: Issues and Analysis, International Journal of Forecasting, vol. 15, no. 4, pp. 353-376, 1999.

[59] L. Santos, Factores Determinantes do Sucesso de Adopção e Difusão de Servicos de Informação Online em Sistemas de Gestão de Ciência e Tecnologia, Ph.D. dissertation, Department of Information Systems, University of Minho, Guimaraes, Portugal, 2004.

[60] M. Scheibe, M. Skutsch, and J. Schofer, Experiments in Delphi Methodology, in The Delphi Method: Techniques and Applications, Reading: Addison-Wesley, 1975, pp. 257-281.

[61] R. Schmidt, Managing Delphi Surveys Using Nonparametric Statistical Techniques, Decision Sciences, vol. 28, no. 3, pp. 763-774, 1997.

[62] R. Schmidt, K. Lyytinen, M. Keil, and P. Cule, Identifying Software Project Risks: An International Delphi Study, Journal of Management Information Systems, vol. 17, no. 4, pp. 5-36, 2001.

[63] H. Scholl and R. Klischewski, E-Government Integration and Interoperability: Framing the Research Agenda, International Journal of Public Administration, vol. 30, no. 8, pp. 889-920, 2007.

[64] S. Siegel and J. Castellan, Nonparametric Statistics for the Behavioral Sciences. 2nd Ed., New York: McGraw-Hill, 1988.

[65] E. Tambouris and K. Tarabanis, e-Government and Interoperability, in Proceedings of the 5th European Conference on E-Government, Antwerp, Belgium, 2005, pp. 399-407.

[66] E. Tambouris, K. Tarabanis, V. Peristeras, and N. Liotas. (2007, April). Study on Interoperability at Local and Regional Level - Final Version, MODINIS LOT 2 Program. [Online]. Available: http://www.epractice.eu/files/media/media1309.pdf.

[67] N. Warner. (2004). C2 Interoperability - An Australian National Whole of Government Approach, in Proceedings of the 9th International Command and Control Research and Technology Symposium. Canberra, Australia. [Online]. Available: http://www.dodccrp.org/events/9th_ICCRTS/CD/papers/076.pdf.

[68] R. T. Watson and J. C. Brancheau, Key Issues in Information Systems Management: An International Perspective, Information & Management, vol. 20, no. 3, pp. 213-223, 1991.

[69] P. Williams and C. Webb, The Delphi Technique: A Methodological Discussion, Journal of Advanced Nursing, vol. 19, no. 1, pp. 180-186, 1994.

 

Appendix A

Appendix B

Appendix C

Appendix D

Appendix E

Appendix F

Received 28 July 2010; received in revised form 15 November 2010; accepted 22 December 2010