Over 90 percent of companies are estimated to use IT Service Management (ITSM) frameworks, yet there is little research on their benefits to the Information Technology (IT) department and the business units. An international survey of 491 firms was conducted to assess the benefits of the IT Infrastructure Library (ITIL), the de-facto ITSM framework, specifically on how these benefits evolve as companies increase their adoption of the ITIL model. Also studied are the perception of challenges of the implementation and the number of ITIL processes implemented in relation to the progress of the adoption of ITIL. Results indicate that as the maturity of implementation increases, the perception of challenges decreases. Findings also show that as the maturity of implementation increases, the number of realized benefits increases, as well as the number of implemented ITIL processes. Implications for practitioners and researchers are also discussed.
Notes
Accepted after two revisions by Prof. Dr. Bichler.
This article is also available in German in print and via http://www.wirtschaftsinformatik.de: Marrone M, Kolbe LM (2010) Einfluss von IT-Service-Management-Frameworks auf die IT-Organisation. Eine empirische Studie zu Vorteilen, Herausforderungen und Prozessen. WIRTSCHAFTSINFORMATIK. doi: 10.1007/s11576-010-0257-8.
The objective of this paper is to gain an understanding on the benefits provided by the implementation of ITIL. This study compares the challenges and benefits of companies at various levels of implementation. It also looks at the number of implemented processes at the various levels. Results indicate that as the maturity of implementation increases, the perception of challenges decreases. Findings also show that as the maturity of implementation increases, the number of realized benefits increases. Insight into the perception of effectiveness of ITIL is presented here, as well as implications for practitioners and researchers.
1 Introduction
In 1980, when listing the critical success factors of Information Systems (IS), Rockart (1982) argues that “the first, and most obvious, IS critical success factor is service”. The Information Technology (IT) departments in many organizations were previously focused on the production of software applications, and in the late 1980s it started to change to a service mode of operation. For IT Service Management (ITSM), the main focus is not on the development of IT applications, but rather on the management of IT services.
Advertisement
Several studies have focused on the adoption of ITSM frameworks as well as on specific service oriented IT frameworks. Winniford et al. (2009) claim that around 45% of US companies are using an ITSM framework while 15% are planning its usage. The IT Governance Institute (2008) estimates that the IT operational framework with the highest adoption rate is IT Infrastructure Library (ITIL) with 24%, followed by Control Objectives for Information and related Technology (CobiT) with an adoption rate of 14%.
Additional to the rising adoption rates of ITSM frameworks, a factor to look at is the costs entailed by IT Services. IT Services account for an estimated 70% to 80% of the expenditure of an IT organization (Orlov 2005). Practitioners have an interest in understanding the possible benefits realized by companies which adopt an ITSM framework.
This empirical study focuses on ITIL as the most popular ITSM framework. In this research, importance is given to the understanding of how these benefits evolve as companies increase the adherence to the guidelines to the ITIL model. Also of interest is the perception of challenges of implementing ITIL, and as expressed previously, how the perceptions of challenges develop as companies increase their adherence to the model. The last point is to understand how the implementation of ITIL processes affects the maturity of the implementation of ITIL.
So far there have been no academic studies on this matter, and the research methodology of a large scale international survey has not been employed. Therefore, this research, using empirical data gathered from a survey with leading companies from various industries, sets out to understand the following:
Which effect does the total number of implemented processes have on the maturity of the ITIL implementation?
How are challenges perceived at different levels of maturity of the ITIL implementation?
How does the total number of realized benefits develop as the maturity of the ITIL implementation increases?
Advertisement
This article begins with a literature review on IT Service Management, on processes of ITIL, as well as benefits and challenges of implementation. This is followed by a description of the methodology used for this research. The data gathered in our survey is analyzed using the Kruskal-Wallis and Mann-Whitney tests to complete comparisons within the different implementation levels. Results of the survey are then analyzed and outcomes are discussed. Before the limitations and the future research sections, conclusions are drawn.
2 Literature Review
ITSM is a part of the Service Sciences that concentrates on IT Operations (Galup et al. 2009). It can be defined as “a set of processes that cooperate to ensure the quality of live IT services, according to the levels of service agreed to by the customer” (Young 2004). Conger et al. (2008) add that ITSM “focuses on defining, managing, and delivering IT services to support business goals and customer needs, usually in IT Operations”.
Service oriented IT Management can be seen as a philosophy for an orientation towards market, service, lifecycle and processes (Zarnekow et al. 2005). First, there is a market orientation which implies that there is a customer-supplier relation instead of a relationship as project partners. Second, there is a service orientation which means service providers have service portfolios instead of project portfolios. These service portfolios include all of the IT services offered by the provider. Third, the focus is on the service lifecycle hence, ITSM provides a methodical approach to the management of IT services – from design, implementation, operation to continual improvement. It does not only focus on the technical aspects of IT but also allows the alignment of services and functions provided by IT within the organization. The main focus of the management of IT services is on the costs of the whole lifecycle, not merely on the costs of development. Fourth is the process orientation, so that the IT organization is oriented on processes and not on functional structures.
There are various concepts of ITSM frameworks. The most common approach is the ITIL which is a de facto standard for IT Service Providers (Hochstein et al. 2005; IT Governance Institute 2008). A variety of ITSM frameworks have been developed using ITIL as a reference, such as Hewlett-Packard (HP ITSM Reference model), IBM (IT Process Model) and Microsoft’s MOF (van Bon et al. 2007).
The current version of ITIL, Version 3, was published in May 2007. It consists of 26 sections which are included in the following five lifecycle phases: Service Strategy, Service Design, Service Transition, Service Operation and Continual Service Improvement. The earlier version, Version 2, has a total of ten processes in two main domains: Service Support and Service Delivery. There are other operational guidance domains, but for this paper, we focus on these two key domains.
Academic research on ITSM is still in its early stages despite its numerous appearances in the popular press and practitioners’ magazines. Existing academic literature merely presents the description of the areas documented on ITIL (Cervone 2008; Hendriks and Carr 2002) or analyses adopters of ITIL through case studies (Hochstein et al. 2005; Cater-Steel et al. 2006b; Marrone et al. 2010). A few researchers have covered the topic of ITIL benefits, challenges of implementation and the effectiveness of ITIL. Relevant academic research is shown in Table 1.
Lists four benefits: improvement in quality of IT services, efficiency and optimization of processes and transparency, and comparability through process documentation and process monitoring. Lists six success factors when implementing ITSM frameworks: Demonstrating “quick wins”, Strive for continuous improvement, Market campaigns in order to create acceptance and understanding, Management support, training, and formation of virtual teams so that “new” processes would be developed simultaneously with the operational activities.
Researches the effect of the implementation of ITIL on customer satisfaction and service quality. The researchers conclude that, at the research site, a large service unit of ICT in South Africa, there was a direct correlation between customer satisfaction, service quality and the use of ITIL.
Describes the challenges of adopting ITIL as the following four factors: lack of management support, cultural change in terms of resistance, delays in choosing an appropriate tool, and management problems for resources such as time, people and money.
Monitors an IT Service provider in Croatia and applied various Key Performance Indicator (KPI) metrics before and after the implementation of a number of ITIL processes. The study concluded that the IT service provider underwent improvements which were attributable to the implementation of ITIL.
Provides an overview of ITIL and suggests the following three benefits: cost reduction, improving customer satisfaction and improving the productivity of the IT department.
Identifies six factors considered benefits of an ITIL adoption: improvement in customer satisfaction, improvement in internal processes, standardization of processes, improvement in service quality, increase in efficiency, and improvement in return on investment.
Sheds light on the challenges of implementation. Focuses on CSF and concludes that Senior Management, an appropriate Change Management strategy, a close relation with multiple vendors and effective project governance are key factors for implementation.
Identifies the following CSF: Executive management support, Interdepartmental communication and collaboration, use of consultants, training and careful software selection, creating an ITIL-friendly culture, process as a priority, and customer-focused metrics.
Studied the most important factors for a successful adoption of ITIL. The most important factors are management support, competence and training, information and communication, stakeholder involvement and ability to change organizational culture.
A summary of the benefits of ITSM frameworks found in literature sources is displayed in Table 2. In Table 3, a summary of the challenges faced when implementing ITSM frameworks is shown.
Table 2
Summary of benefits of the implementation of ITSM frameworks
Lack of internal skills/knowledge relating to ITIL
×
×
×
×
×
Lack of funding/Cost of adoption
×
×
Organizational/Cultural resistance to change
×
×
×
Maintaining momentum/Progress stagnates
(Plan objectives)
×
×
To comprehend at which level of adherence or maturity companies are in when adopting ITSM frameworks, numerous researchers including Cater-Steel et al. (2007) and Marrone et al. (2010) have used the maturity model. The maturity model levels presented in these studies are based on the model from CobiT and Capability Maturity Model Integration (CMMI). These levels are intended as profiles of IT processes, and companies would identify these levels as a description of their current state.
Until now, there has been no research which has involved the benefits, challenges and implementation of processes as well as their relation to the maturity of the ITIL implementation. Apart from that, the methodology of a large-scale survey for various countries has not been utilized.
3 Research Design
This section describes the levels of adoption, known as maturity model, as well as the propositions explored in this study. For the three questions listed in the introduction, a total of five propositions are described. These are shown below.
3.1 Maturity Model Levels
The maturity model is divided in levels which range from non-existent (0) to optimized (5). They were originally used by the CMM framework and later by CobiT. This study utilizes the same levels of maturity as those proposed in these frameworks. At the lowest level of the maturity model, the management processes are not applied at all. This level is known as non-existent (0) implementation. At the following level, named initial, processes are ad hoc and disorganized. Level 2 is referred to as repeatable, where the processes are intuitive, and provide, in most cases, deterministic and repeatable results. Level 3 is where processes are documented and provide standard procedures and clearly defined results/outputs. This level is known as defined. Level 4, known as managed, is based on critical success factors, where key performance indicators (KPIs) have been defined and are continually measured to quantify process performance. The highest level of maturity is known as optimized; this is where a continual improvement cycle has been implemented and is based on KPIs and internal audits. At this level the process is improved in effectiveness, efficiency and compliance. Companies would identify these levels as a description of their current state.
3.2 Implemented Processes and Maturity Level
The relation between the number of implemented processes and the levels of maturity is of interest for this research. To understand the effect of the total number of implemented processes on the maturity level of implementation, two possible propositions were initially suggested. Only one of the two propositions was then selected by the researchers. One possible proposition is that companies would select and implement processes which would, in their opinion, provide their companies with the biggest benefits, or processes that would help them deal with areas in which they are performing deficiently. Gammelgård et al. (2007) suggest that companies tend to adopt the processes that they require the most. Tuttle and Vandervelde (2007) argue that during the implementation of Business Process Management frameworks, not all processes are taken into consideration. Therefore, companies that have implemented only some of the processes would then report the maturity level of their ITIL implementation based on the maturity of those processes which they have implemented, rather than on the whole ITIL framework.
Conversely, another possible proposition is that the maturity of the implementation of ITIL is based on the full lifecycle model. Therefore, rather than concentrating on the maturity of their implemented processes, the perception of maturity level would be based on the completed ITIL model. Hence, the more processes of ITIL companies implement, the higher the companies’ maturity of the ITIL implementation. In this research, we support the latter. Since the number of processes varies depending on the version, each version is analyzed separately. The proposition is:
P1:
There is a positive relationship between implemented processes and perceived maturity of the ITIL implementation.
3.3 Perceived Challenges and Maturity Level
This research aims to understand which effect, if any, maturity levels have on the perceived challenges of implementation. The proposition of the effect of the perceived challenges on the level of maturity is based on the model of the learning curve effect. The learning curve, also known as the experience curve, is a phenomenon which was initially observed by Wright (1936). He observed that as the quantity of units manufactured doubles, the number of hours of direct labor required to produce an individual unit decreases at a uniform rate. Wright also argued that learning can occur for the production of any good or service. Applying this model to our proposition we can assume that the organization gains experience dealing with the challenges and becomes more efficient as it progresses in its learning, allowing for the perception of the challenges to decrease over time. Therefore, our proposition is formulated as follows:
P2:
There is a negative relationship between maturity levels of the ITIL implementation and perceived challenges of implementation.
3.4 Number of Realized Benefits and Maturity Levels
This question focuses on understanding the total number of realized benefits due to the implementation of ITIL for each company. Rather than concentrating on the individual benefits which were surveyed, the focus of this research is to understand the progression of the total number of realized benefits for the companies. Consequently, for each company, benefits which they realize are added up. The same approach is used for the number of benefits supported by metrics and the number of benefits acknowledged by the business.
We believe that the benefits provided by ITIL will be noticed by IT initially, and that metrics to back up the benefits will not be available at earlier stages. At later levels these benefits will be supported by metrics and may also be acknowledged by the business.
For this proposition, the Law of Diminishing Returns is taken into consideration. It suggests that the continued improvement efforts towards a particular project or goal would lead to a decline in effectiveness after a particular level of result has been accomplished (Drucker et al. 1998). In other words, after a certain level of standardization, increasing the standardization further provides few additional benefits. Consequently, the following proposition is suggested:
P3a:
There is a positive relationship between maturity levels of the ITIL implementation and perceived realized benefits.
We also expect that the number of realized benefits which are supported by metrics will become visible on the later levels of maturity, rather than on the initial levels of maturity. We understand that as at later levels of maturity companies are expected to use metrics in a regular manner. Particularly, the interest of this proposition is not to see if companies are using metrics per se, but rather if the metrics used were able to support the perception of the benefits that were attained. In other words, we are interested in understanding if IT executives were able to quantify the benefits of implementing ITIL with their use of metrics. Similarly, the business may recognize the benefits of the ITIL implementation on the later levels of adoption, possibly due to a better Business-IT alignment, which is a proposed benefit of ITIL. Therefore, the following propositions are suggested:
P3b:
There is a positive relationship between maturity levels of the ITIL implementation and usage of metrics to measure the realized benefits.
P3c:
There is a positive relationship between maturity levels of the ITIL implementation and acknowledgement by the business of the realized benefits.
4 Methodology
The online questionnaire was made available in the months of April and May 2009. An invitational email was sent to individuals who were on the mailing lists of Hornbill and the IT Service Management Forum (itSMF) in the United States of America and United Kingdom. Additionally, in an attempt to expand the findings of this research, the survey was announced in various internet groups and forums whose sole topic was ITIL. The target participants would be ITIL champions for their organization and would be broadly involved in the implementation of ITIL.
The structure of the questionnaire addressed many aspects of ITIL, its adoption, usage, implementation and maturity as well as effectiveness of processes and realized benefits. It also covered the topics of Business-IT alignment and service desk usage. The survey contained questions to which responses used Likert scales, nominal scales and open-ended questions.
First, those surveyed were asked to rate the perception of the maturity of their ITIL processes on a scale based on the CobiT and CMMI maturity.
Respondents were also requested to specify which version of ITIL they had implemented. Based on this, they were asked to select which processes they had implemented considering their ITIL version. All processes were listed in the survey.
Additionally, respondents were inquired about their perception of challenges of implementing ITIL. They were asked to rate challenges on a scale from 1 to 5, where 1 meant no challenge and 5 meant major challenge. The challenges that are studied, based on those gathered in our literature review and summarized in Table 3, are Lack of executive sponsorship, Business understanding of ITIL objectives, Lack of resources (time or people), Lack of internal knowledge/skills relating to ITIL, Lack of funding/costs of adoption, Organizational/cultural resistance to change, and Maintaining momentum/progress stagnates.
Finally, those surveyed had to select realized benefits that could be achieved due to the adoption of ITIL. The benefits listed on the survey are Improvements: of service quality, of customer satisfaction, due to having standardized processes, of interaction of IT with the rest of the business, of reduction in IT downtime, of return on IT spending (ROI), by applying best practice experience of others, in measuring the financial contribution of IT to the business, of call fix rate and of the morale of the IT staff. These benefits are those shown in Table 2. If respondents had realized one of these benefits, they were able to select whether these benefits were supported by metrics and whether the business had acknowledged the improvements.
5 Results
5.1 Respondents’ profile
More than 5,000 invitations were sent out to members of the itSMF UK and US and to the mailing list of Hornbill. Out of the 784 IT executives, who started the survey, 503 completed and submitted the survey. Twelve responses were identified as not valid and were excluded from the statistical analysis. Partially completed surveys were not used for the study. Four fifth of those who completed the survey were directly invited to participate in the survey, the others were recruited through the forum posting. The return rate of the survey invitations was of eight percent.
Respondents were asked about their companies’ industry, number of sites supported by IT, number of employees in the company as well as their title and their location. Table 4 shows the profile of the respondents. In the sample, around three fourths of those surveyed were from the technology, public, financial and banking sectors. Other industries include the professional, retail and manufacturing industries.
Table 4
Profile of responding organizations (n=491)
Nearly 70% of the respondents had ten or more sites supported by central IT, and close to 20% had two to five sites. Almost 45% of those interviewed worked in a company which had more than 10,000 employees. The job roles of those interviewed is also shown. Of those surveyed 33% were IT managers, while 23% were Process Specific Managers. Due to the usage of mailing lists and posting of invitations to the survey on English speaking websites, the majority of answers came from the United Kingdom (52%) and United States (36%) with various answers coming from Canada, India and Ireland.
Figure 1 reveals the implementation maturity of ITSM frameworks for all participants. As can be observed, more than half of those surveyed place their implementations to be either on Level 2 (Repeatable), with 32%, or Level 3 (Defined), with 25%, while a minority placed themselves at the extremes, Level 1 (Initial) with 13% and Level 5 (Optimized) with 11%.
Fig. 1
Maturity of ITIL implementation of respondents (n=491)
×
5.2 Empirical Results
An exploratory analysis was conducted for each variable to test for normality. Both, the Kolmogorov-Smirnov and the Shapiro-Wilk tests showed significance for versions of ITIL (p<0.001), all the variables describing challenges of ITIL’s adoption (p<0.001) and the realized benefits of ITIL (p<0.001). As the data was non-normal, the Kruskal-Wallis, a non-parametric one way analysis of variance, was used to study the data. If the data using the Kruskal-Wallis showed significant differences between the groups, the Mann-Whitney U test was applied.
Since the study was concerned with how the nominated variables were impacted as the ITIL implementation increases, caution was taken with the choice of test measures. The Mann-Whitney U tests inflate the Type I error rate, so care was taken in the choice of comparisons made.
The suggested comparisons are between the first and middle levels of ITIL implementation maturity, the first and last levels of ITIL implementation maturity, and between the middle and final levels of ITIL implementation maturity. Therefore, the following three tests were conducted:
Test 1: Level 1 (Initial) compared to Level 3 (Defined)
Test 2: Level 1 (Initial) compared to Level 5 (Optimized)
Test 3: Level 3 (Defined) compared to Level 5 (Optimized)
Since three tests were conducted, a Bonferroni correction was applied. Due to this correction, rather than using the critical level of significance of 0.05, all effects were reported at 0.0167 level of significance. All reported p values are using 2-tailed Monte Carlo p values with a confidence level of 99% and a number of samples of 10,000. This method was used because of the large sample size.
Additionally, to understand the trends in the data the Jonckheere-Terpstra test was used. Lastly, r was used to measure the strengths of a relationship between variables (Rosenthal, 1991 p. 19). Cohen suggests that the sizes of the effect are small (0.1), medium (0.3) or large (0.5). In the next sections, the following abbreviations are used: H corresponds to the Kruskal-Wallis statistic, U represents the Mann-Whitney U statistic, while J symbolizes the observed J–T statistic.
5.2.1 Implemented Processes and Maturity Level (P1)
The number of implemented processes for both versions of ITIL, version 2 (n=248) and version 3 (n=193), are studied to understand the effect on the maturity level of implementation. Figure 2 presents the mean and median of the number of implemented processes of ITIL version 2 or ITIL version 3 for each of the maturity levels.
Fig. 2
Descriptive statistics of processes implemented for version 2 and version 3
×
In general, the numbers of implemented processes significantly affect the maturity level of the implementation (Version 2 H(4)=99.03, p<0.001, Version 3 H(4)=82.108, p<0.001). Mann-Whitney U tests were used to follow up on the findings. Table 5 shows that the number of implemented processes is significant when comparing the Initial level (1) with the Defined level (3). The same occurs when comparing the Initial level (1) with the Optimized level (5). When comparing the Defined level (3) with the Optimized level (5) significance was also observed. Using Cohen’s benchmark, we observe that there is a medium to large change on the number of implemented processes as maturity increases.
Table 5
Mann-Whitney test results for ITIL versions and maturity
Level 1 compared with Level 3
Level 1 compared with Level 5
Level 3 compared with Level 5
U
p
r
U
p
r
U
p
r
ITIL Version 2
98.5
0.000a
−0.44
12.0
0.000a
−0.73
323.5
0.000a
−0.58
ITIL Version 3
75.5
0.000a
−0.68
83.0
0.000a
−0.61
345.0
0.013a
−0.30
aSignificance at 0.0167
With the help of Jonckheere’s test, a significant trend in the data can be observed. As the level of maturity goes up, the median of number of implemented processes increases. (Version 2: J=18001, z=10.49, r=.67, Version 3: J=11398, z=9.63, r=.69)
In conclusion, regardless of the version of ITIL implemented, as the level of maturity goes up, the number of implemented processes also ascends.
5.2.2 Perceived Challenges and Maturity Level (P2)
Respondents were asked to rate challenges on a scale from 1 to 5, where 1 meant no challenge and 5 meant major challenge. Figure 3 demonstrates the perceived ranking of the challenges. Of the challenges queried, the challenge that was perceived by those surveyed to be the least of challenges was Lack of Knowledge and Skills, very closely followed by Executive Sponsorship. A challenge which was highly rated by the respondents was the Lack of Resource. This challenge obtained the lowest 1 and 2 selections.
Fig. 3
Descriptive statistics on challenge rating (1=No Challenge, 5=Major Challenge)
×
Tables 6 and 7 show the means and medians for each challenge and maturity level, while Figs. 4 and 5 show a graph on the mean rating of the challenge perception for each of the challenges studied for companies on various perceived maturity levels. A downward trend can be observed. As the perceived maturity level increases, the perception of challenge of implementation decreases.
Fig. 4
Descriptive statistics on each maturity level for each challenge (Part I)
Fig. 5
Descriptive statistics on each maturity level for each challenge (Part II)
Table 6
Descriptive statistics on each maturity level for each challenge (Part I)
Maturity Level
N
Lack of executive sponsorship
Business understanding of ITIL objectives
Lack of resources (time or people)
M
SE
Mdn
M
SE
Mdn
M
SE
Mdn
1
62
3.26
.15
3
3.79
.13
4
4.15
.12
4
2
153
3.38
.11
3
3.57
.09
4
3.89
.08
4
3
123
3.01
.11
3
3.33
.09
3
3.74
.07
4
4
95
2.63
.12
2
2.99
.10
3
3.57
.11
4
5
58
2.34
.19
2
2.79
.17
3
3.22
.13
3
Table 7
Descriptive statistics on each maturity level for each challenge (Part II)
Maturity Level
N
Lack of internal knowledge/skills relating to ITIL
Lack of funding/costs of adoption
Organizations/cultural resistance to change
Maintaining momentum/progress stagnates
M
SE
Mdn
M
SE
Mdn
M
SE
Mdn
M
SE
Mdn
1
62
3.45
.15
4
3.52
.13
3
3.89
.12
4
3.63
.11
4
2
153
3.10
.08
4
3.41
.08
4
3.74
.09
4
3.50
.07
4
3
123
2.70
.08
3
3.35
.09
3
3.39
.10
3
3.45
.09
4
4
95
2.60
.09
3
3.20
.11
3
3.25
.11
3
3.13
.10
3
5
58
2.41
.14
2
2.88
.14
3
2.93
.16
3
3.10
.12
3
×
×
In general, challenges of adopting ITIL are significantly affected by the implementation maturity of ITIL: Lack of executive sponsorship (H(4)=37.75, p<0.001), Business understanding of ITIL objectives (H(4)=42.19, p<0.001), Lack of resource, time or people (H(4)=32.39, p<0.001), Lack of internal knowledge and skills relating to ITIL (H(4)=42.86, p<0.001), Lack of funding/cost of adoption (H(4)=14.21, p<0.01), Organization/culture resistance to change (H(4)=33.41, p<0.001), Maintaining momentum/progress stagnates (H(4)=18.88, p<0.005). Mann-Whitney tests were also used to follow up on this finding.
Table 8 shows the results from the selective comparisons. It can be observed that between levels Initial (1) and Defined (3) there was significance for Business understanding of ITIL objectives, Lack of resource, time or people, Lack of internal knowledge and skills relating to ITIL, and Organization/culture resistance to change. Based on Cohen’s benchmark, a small to medium change can be seen on the lowering of the perception of challenges as maturity increases. However, no significance could be shown for Lack of executive sponsorship, Lack of funding/cost of adoption and Maintaining momentum/progress stagnates.
Table 8
Mann-Whitney test results for perception of challenges and maturity levels
Challenges
Level 1 compared with Level 3
Level 1 compared with Level 5
Level 3 compared with Level 5
U
p
r
U
p
r
U
p
r
Lack of executive sponsorship
3375.0
0.189
−0.10
1104.5
0.000a
−0.34
2492.0
0.001a
−0.25
Business understanding of ITIL objectives
2736.5
0.001a
−0.24
993.0
0.000a
−0.40
2634.0
0.003a
−0.22
Lack of resources (time or people)
2756.5
0.001a
−0.24
894.0
0.000a
−0.45
2496.5
0.001a
−0.26
Lack of internal knowledge/skills relating to ITIL
2461.0
0.000a
−0.30
977.5
0.000a
−0.40
2884.5
0.031
−0.16
Lack of funding/costs of adoption
3433.0
0.254
−0.09
1234.0
0.002a
−0.28
2724.5
0.008a
−0.20
Organizations/cultural resistance to change
2825.0
0.003a
−0.22
1017.5
0.000a
−0.39
2785.0
0.014a
−0.18
Maintaining momentum/progress stagnates
3419.5
0.225
−0.09
1245.0
0.003a
−0.28
2869.0
0.027a
−0.17
aSignificance at 0.0167
When comparing the Initial (1) with Optimized (5) significance can be observed for all variables with a medium effect size.
Lastly, comparing the Defined (3) with the Optimized (5) maturity level, significance can be observed for Lack of executive sponsorship, Business understanding of ITIL objectives, Lack of resource, time or people, Lack of funding/cost of adoption, Organization/culture resistance to change. Marginal significance can be observed for Lack of internal knowledge and skills relating to ITIL and Maintaining momentum/progress stagnates.
Jonckheere’s test revealed a significant trend in the data. As the level of maturity goes up, the median of challenges decreases (Lack of executive sponsorship J=36500, z=−5.9, r=−.26, Business understanding of ITIL objectives J=35298, z=−6.6, r=−.30 Lack of resource, time or people J=37332, z=−5.5, r=−.25, Lack of internal knowledge and skills relating to ITIL J=35233, z=−6.7, r=−.30, Lack of funding/cost of adoption J=40536, z=−3.6, r=−.16, Organization/culture resistance to change J=36699, z=−5.8, r=−.26, Maintaining momentum/progress stagnates J=39816, z=−4.0, r=−.18).
We can conclude that as the maturity of implementation increases there is a reduction on the perception of difficulty when facing the challenges studied in this research. The challenge of Maintaining momentum/progress stagnates is the only factor that has no significance when examining the Initial (1) and Defined (3) level, and only marginal significance when comparing the Defined (3) and Optimized (5) level. However, when examining the Initial (1) and Optimized (5) level significance does exist. When examining the variables Lack of executive sponsorship and Lack of funding/cost of adoption one can observe that at the lower levels of implementation one can detect no significance. However, when comparing the Defined (3) with the Optimized (5) level significance can be observed.
5.2.3 Number of Realized Benefits and Maturity Levels (P3)
Figure 6 provides detail of the benefits received by those who implemented an ITSM framework. Overall, the figure would appear to suggest that half of those surveyed would perceive a service quality improvement and also an increase in the usage of standardized processes. Three benefits which were only seen by close to one tenth of those surveyed include improvement of Return on Investment (ROI), IT staff morale and an improvement on measuring of the financial contribution of IT.
Fig. 6
Percentages of companies that have realized a benefit
×
When conducting the Kruskal-Wallis test, the number of realized benefits is significantly affected by the level of implementation maturity (H(4)=134.49, p<0.001 for the total number of realized benefits, H(4)=139.37, p<0.001 for realized benefits backed by metrics, H(4)=91.64, p<0.001 for realized benefits acknowledged by the business). The mean and median of the benefits for each of the different maturity levels is displayed on Table 9.
Table 9
Descriptive statistics of realized benefits (n=491)
Maturity Level
N
Total realized benefits
Total realized benefits backed by metrics
Total realized benefits acknowledged by business
M
SE
Mdn
M
SE
Mdn
M
SE
Mdn
1
62
1.03
0.19
0
0.48
0.15
0
0.44
0.14
0
2
153
2.46
0.14
2
1.04
0.1
1
0.75
0.1
0
3
123
4.06
0.19
4
2.49
0.17
2
1.71
0.16
1
4
95
4.13
0.21
4
2.65
0.19
2
2.23
0.22
2
5
58
4.89
0.37
5
3.89
0.37
4
3.34
0.41
3
Figure 7 plots the perceived number of realized benefits for companies at various maturity levels of implementation. In this figure an upward trend can be observed. Companies at higher levels of perceived implementation maturity believe to realize a larger number of benefits.
Fig. 7
Descriptive statistics of realized benefits (n=491)
×
As shown in Table 10, the number of realized benefits is significantly higher when comparing the Initial (1) with the Defined (3) level of maturity of implementation. Based on Cohen’s benchmark, there is a large change on the number of realized benefits as maturity increases. One can also observe this when comparing the Initial (1) with the Optimized (5) level. Finally, when comparing the Defined (3) with Optimized (5) level no significance can be determined. Yet, significance can be observed when examining the number of realized benefits that have been backed by metrics as well as when observing the number of realized benefits that have been acknowledged by the business.
Table 10
Mann-Whitney test results for realized benefits and maturity levels
… per company
Level 1 compared with Level 3
Level 1 compared with Level 5
Level 3 compared with Level 5
U
p
r
U
p
r
U
p
r
Total realized benefits
980.5
0.000a
−0.61
434.0
0.000a
−0.67
2956.5
0.060
−0.14
Total realized benefits backed by metrics
1209.5
0.000a
−0.57
478.5
0.000a
−0.67
2529.5
0.002a
−0.24
Total realized benefits acknowledged by business
2099.0
0.000a
−0.40
658.0
0.000a
−0.59
2511.5
0.001a
−0.24
aSignificance at 0.0167
To corroborate with the findings, an additional analysis is conducted for each of the benefits listed. In this analysis, the individual benefits of implementation are evaluated at the different levels of maturity. Each variable is dichotomous (realized benefit or not realized benefit). The Kruskal-Wallis test shows that all of the benefits realized are significantly affected by the level of implementation maturity (Service Quality H(4)=92.08, p<0.001, Customer Satisfaction H(4)=44.41, p<0.001, IT resource downtime H(4)=46.41, p<0.001, Benefits from the usage of a best practice H(4)=42.05,p<0.001, Business-IT alignment H(4)=25.45, p<0.001, Call fix rate H(4)=34.65, p<0.001, Standardized Process H(4)=68.20, p<0.001, IT morale H(4)=19.61, p<0.001, Return on Investment H(4)=26.40, p<0.001,
Measurement of the financial contribution H(4)=18.93, p<0.001). Mann-Whitney tests were used to follow up on the findings.
Table 11 provides the results from this test. When comparing the companies on the Initial (1) level with those on the Defined (3), all variables had a significant change with the exception of the Improvement of IT morale, Return on Investment, and Financial Contribution. When comparing the Defined (3) with the Optimized (5) level, only Return on Investment was significant, and all other variables had no significant change. This confirms and expands the findings of the previous analysis (P3a). Finally, when contrasting the benefits received on the Initial (1) level with the Optimized (5) level significance can be observed for all variables.
Table 11
Mann-Whitney test results for each benefits and maturity levels
Improvements in
Level 1 compared with Level 3
Level 1 compared with Level 5
Level 3 compared with Level 5
U
p
r
U
p
r
U
p
r
Service quality
1544.0
0.000a
−0.57
720.0
0.000a
−0.59
3551.0
0.560
−0.05
Customer satisfaction
2536.5
0.000a
−0.32
908.0
0.000a
−0.49
2994.5
0.027
−0.15
Reduction in IT downtime
2416.0
0.000a
−0.37
1046.0
0.000a
−0.47
3382.0
0.301
−0.05
Best practice
2352.5
0.000a
−0.37
978.0
0.000a
−0.48
3306.5
0.218
−0.07
Business-IT alignment
2909.0
0.001a
−0.24
1220.0
0.000a
−0.34
3266.0
0.178
−0.08
Call fix rate
2601.5
0.000a
−0.33
1137.0
0.000a
−0.41
3389.0
0.311
−0.05
Standardized process
1856.0
0.000a
−0.48
852.0
0.000a
−0.53
3521.0
0.494
−0.01
IT staff morale
3346.0
0.018
−0.16
1449.0
0.003a
−0.26
3311.5
0.176
−0.08
ROI
3502.5
0.034
−0.15
1300.0
0.000a
−0.38
2869.5
0.001a
−0.24
Financial contribution
3533.0
0.073
−0.13
1422.0
0.001a
−0.30
3083.0
0.018
−0.17
aSignificance at 0.0167
Jonckheere’s test shows a significant trend in the data, as the level of maturity goes up, the median of number of realized benefits increases (J=66553, z=11.44, r=.52), the median of number of realized benefits backed by metrics increases (J=67164, z=11.69, r=.54), and finally, the median of number of realized benefits acknowledged by business increases (J=62730, z=9.73, r=.44).
We can conclude that as the level of maturity increases, so does the number of realized benefits. However, there appears to be a stall in the number of realized benefits between the Defined (3) and the Optimized (5) level. In contrast, on the later stages companies are able to show the benefits of the implementation of ITIL through the metrics used and are also showing the realized benefits to the business.
6 Discussion
The results of the current study confirm P1. It states that as more processes of ITIL are implemented, the perceived maturity of the ITIL implementation increases. It can be observed that there is a positive direct influence between the number of implemented ITIL processes and the maturity level of the ITIL implementation. This result may also give an insight on how ITIL adopters are implementing the ITIL processes, which is on increasing implementation of processes rather than of implementing all processes at once. While IT executives may hand pick the processes that they implement, they understand the framework as a whole and perceive the maturity of their implementation to be based on the maturity from the whole framework perspective.
In general, the second proposition P2 is also confirmed. This proposition states that the perception of the listed challenges of implementation decreases as the maturity levels of implementation increase. This can be due to the fact that as adopters overcome the initial challenges of implementation, experience is gained, and future challenges are perceived to be less complex than those in the earlier levels. As well, it follows the pattern of the learning curve model, where at the initial levels there is a difficulty in undergoing the implementation, yet, as experience is gained, these difficulties decrease.
It can also indicate that these challenges are reduced because the benefits of ITIL are made evident to the business and those involved in the project of implementation. As explained by Huber (1991), organization learning occurs to a larger extent when the knowledge obtained is recognized to be useful. In other words, the organization is more likely to learn if there are benefits presented to the individuals and to the organization. In the case of this study, the reason why the challenge perception decreases over time may also be due to the benefits shown by the implementation, which may encourage the organization and individuals to learn and to implement further processes.
When looking at the listed challenges specifically, challenges such as Lack of executive sponsorship, Lack of funding, and Maintaining momentum/project stagnates show no significance when comparing the maturity levels of implementation Initial (1) and Defined (3). However, when the maturity levels of implementation Defined (3) and Optimized (5) are compared, these challenges decrease. This may be due to the fact that at the earlier levels, the business has yet to realize benefits and skepticism exists. Conversely, in the later stages of implementation, once the business has acknowledged benefits of the ITIL implementation it is likely to support the further implementation of ITIL.
When comparing the maturity levels of implementation Defined (3) and Optimized (5), marginal significance can be observed in the challenges such as the Lack of internal knowledge and skills and Maintaining momentum/project stagnates. In the case of Lack of internal knowledge and skills, this could be due to the fact that acquiring personnel that have specialized knowledge in this field may be difficult, or that the training programs, being new, are not often available. Maintaining momentum/project stagnates is the only challenge that was not proven statistically significant in two of the three comparisons, the two being the comparison between Initial (1) and the Defined (3) level as well as the comparison of Defined (3) and Optimized (5) level. This indicates that, throughout the project, this challenge must be the focus of IT Managers and may be a critical success factor for the implementation. As well, this factor can be considered to be independent of the ITIL implementation and may be a factor attributed to those executing the implementation of ITIL.
Finally, results from P3a, P3b and P3c also showed to be statistically significant. The fact that there are benefits due to the adoption of ITIL agrees with the results from individual case studies on the effectiveness of ITIL presented by Potgieter et al. (2005) and Spremic et al. (2008). As reported by Hochstein et al. (2005), “Quick wins” are critical success factors when implementing ITIL. Thus, some companies may be attempting to realize benefits on the lower levels of implementation.
As it can be observed from the results of P3a, as the maturity level increases the number of realized benefits increases. However, there is no significance when comparing the Defined (3) with the Optimized (5) level of the ITIL implementation. Therefore, looking only at this result the Law of Diminishing Returns for the studied benefits of ITIL appears to apply. When we look at the individual benefits of implementation, none of the variables studies showed a change except for Return on Investment. This indicates that during the earlier stages companies will receive most of the direct benefit from the ITIL implementation. It could also indicate that most of the variables studied apply only to companies in the earlier stages, and that other benefits could exist.
Considering the results of P3b and P3c we cannot be certain that the Law of Diminishing Returns applies. As revealed from the findings of P3b, improvements backed by metrics are seen on the later levels of implementation. As well, benefits of the ITIL implementation, which are acknowledged by the business (P3c), are also demonstrated at the later levels of implementation.
The contribution to research is that it delivers insight into the perception of effectiveness of ITIL, the implementation of processes and the evolution of challenges. Additionally, this research opens the path for future research. In practice, the findings can serve as a guideline for those IT managers considering or who have already adopted ITIL. For those considering the adoption and having doubts on the benefits of ITIL, the results show that due to the implementation of ITIL companies do receive several benefits. For those IT departments that have adopted ITIL and ponder on the idea of expanding their implementation of ITIL, they are provided with the understanding of the evolution of benefits realizable, and how their perception of challenges of implementation is affected as they continue implementing ITIL.
7 Conclusion
Based on the results gained in the previous sections, it has been shown that as the maturity of the ITIL implementation increases the number of implemented processes also increases. In addition to this, as the maturity increases, the challenges of implementation decrease. This is explained using the learning curve as well as insights from organizational learning. Results conclude that as the maturity of ITIL increases, so does the number of realized benefits. Marginal returns can be observed after the implementation reaches the Defined (3) level. Yet, in later levels of implementation further returns of the ITIL implementation can be seen. In these later levels, there is an increase in the usage of metrics to measure the benefits of the implementation as well as in the business acknowledging the benefits provided by IT.
Limitations of this study are that the study concentrates only on the United States and United Kingdom, and that it over-samples the larger enterprises. This research aimed only at surveying IT executives, and only their views are included in this study. Another limitation is that empirical studies are dependent on the quality of data provided by the respondents. Additionally, the paper uses a perceived maturity which is based on a single measurement. As well, the results are based only on the challenges and benefits that were listed on the survey.
Since this research is targeted towards IT experts, further studies will be conducted to understand the views of the business in respect to the challenges and benefits of the ITIL implementation. A comparison of views, the IT and the business view, on these topics is relevant. As well, studies will be conducted to understand how Business-IT alignment is affected by the implementation of ITIL. This could be done by conducting a survey and using a proven method of measuring Business-IT alignment and measuring ITIL maturity. Future work should complete a longitudinal study of the impact of ITIL.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
WI – WIRTSCHAFTSINFORMATIK – ist das Kommunikations-, Präsentations- und Diskussionsforum für alle Wirtschaftsinformatiker im deutschsprachigen Raum. Über 30 Herausgeber garantieren das hohe redaktionelle Niveau und den praktischen Nutzen für den Leser.
BISE (Business & Information Systems Engineering) is an international scholarly and double-blind peer-reviewed journal that publishes scientific research on the effective and efficient design and utilization of information systems by individuals, groups, enterprises, and society for the improvement of social welfare.
Texte auf dem Stand der wissenschaftlichen Forschung, für Praktiker verständlich aufbereitet. Diese Idee ist die Basis von „Wirtschaftsinformatik & Management“ kurz WuM. So soll der Wissenstransfer von Universität zu Unternehmen gefördert werden.
Which of the following statements best describes your IT organization?
We have not adopted ITIL (Level 0)
We are new to ITIL and have just started to implement processes (Level 1)
We have a relatively low level of ITIL process maturity. Some processes are documented and these are generally understood, but errors are likely (Level 2)
We have a medium level of ITIL process maturity. Processes are documented monitored for compliance (Level 3)
We have a reasonably high level of ITIL process maturity. Our processes are documented, and measured according to established metrics (Level 4)
We have a very high level of ITIL process maturity. Our processes are documented, understood, backed by metrics and continually reviewed for improvement (Level 5)
(2)
Which version of ITIL (if any) are you using?
ITIL V2
ITIL V3, upgraded from V2
ITIL V3
Have not adopted ITIL
(The following two questions are only for respondents who answered ITIL V2 on question 2)
(3)
Which of the following ITIL V2 Service Support processes have you implemented?
Incident Management
Problem Management
Change Management
Release Management
Configuration Management
(4)
Which of the following ITIL V2 Service Delivery processes have you implemented?
Availability Management
Capacity Management
Financial Management
Service Level Management
IT Service Continuity Management
(The following five questions are only for respondents who answered ITIL V3 or ITIL V3, upgraded from V2 on question 2)
(5)
Which Service Strategy process have you implemented?
Strategy Generation
Service Portfolio Management
Demand Management
Financial Management
(6)
Which Service Design process have you implemented?
Service Catalogue Management
Service Level Management
Availability Management
Capacity Management
IT Service Continuity Management
Information Security Management
Supplier Management
(7)
Which Service Transition process have you implemented?
Transition Planning and Support
Change Management
Service Asset & Configuration Management
Release & Deployment Management
Service Validation & Testing
Evaluation
Knowledge Management
(8)
Which Service Operation process have you implemented?
Incident Management
Problem Management
Request Fulfillment
Event Management
Access Management
(9)
Which Continual Service Improvement process have you implemented?
Service Level Management – Seven Level improvement process
Service Measurement
Service Reporting
(10)
On a scale of 1–5, where 1=No Challenge and 5=Major Challenge, how would you rate the following barriers to ITIL implementation in your organization?
Lack of Executive sponsorship
Business understanding of ITIL objectives
Lack of resources (time or people)
Lack of internal knowledge/skills relating to ITIL
Lack of funding/cost of adoption
Organization/cultural resistance to change
Maintaining momentum/progress stagnates
(11)
Owing to the ITIL implementation, have you had an improvement in the following areas?
Service Quality
Customer satisfaction
Standardized process adoption across all of IT
Interaction of IT with rest of business
Reduction in IT downtime
Return on investments in IT
Benefited from best practice experience of others
Financial contribution of IT to the business
Call fix rate
Morale of IT staff
(Follow-up question, only for the selected elements of the previous question)
(12)
Have the improvements in the areas been noted by the business?
