Keeping Your Maturity Assessment Alive

Organizational capabilities can be defined as organizational entities that represent complex bundles of skills, accumulated knowledge, and systems that manifest in organizational processes (Kwon 2021). When deployed purposefully, capabilities enable organizations to perform certain activities to achieve particular goals and outcomes and serve as the fundamental basis of economic success (Kwon 2021). Maturity models have been used extensively to (i) assess the capabilities of an organization in a certain discipline, (ii) to provide a basis for benchmarking against competitors, and (iii) to guide an organization in the acquisition of the capabilities that it needs to improve in that discipline (Serral et al. 2020). In this context, maturity is a specific process that entails the explicit definition, management, measurement, and control of the evolutionary growth of an entity, such as - in this special case, capabilities (Kerpedzhiev et al. 2021; Paulk et al. 1993). Therefore, maturity implies evolutionary progress from an initial state to a final and more advanced one (Mettler 2011). A maturity model is generally structured as a sequence of distinct levels (Pöppelbuß and Röglinger 2011) that follow a path from an initial state of maturity to a potential final state of maturity (Becker et al. 2009). Those models are usually conceptualized as matrices, with maturity levels on one axis and capabilities on the other, while capabilities are mostly arranged along specific dimensions (Lasrado et al. 2015). Maturity levels are phases of development that are arranged sequentially from the lowest to the highest. Maturity models are particularly important for identifying the strengths and weaknesses of organizations by reference to an underlying phenomenon and for the collection of benchmarking information by the organization (Khoshgoftar and Osman 2009). Maturity models, as diagnostic or benchmarking tools, enable the identification of appropriate actions for creating or improving capabilities and, therefore, for reaching higher maturity levels (Kohlegger et al. 2009).

One of the first models was the Capability Maturity Model (CMM), which was designed to assess the maturity of software development processes (Paulk et al. 1993). Many models that followed the CMM were loosely based on it but lacked a comparable scheme. The capability maturity model integration (CMMI) project was initiated to create a standardized framework model (CMMI Product Team 2010). Although many models followed the basic outlines of CMMI, most of their authors did not disclose their research methods or the underlying design decisions (Mettler 2011).

Research on maturity models can be viewed from two perspectives, representing cycles that include specific and sometimes overlapping activities. The developer perspective is directed at providing suitable and rigorously designed models, whereas the user perspective is oriented toward their effective and appropriate application (Mettler 2011; Proença et al. 2020). Both perspectives entail the use of different methods and frameworks for the creation, selection, and application of models. Figure 1 shows the two cycles and the associated generic activities.

Due to the popularity of maturity models, a potential user is confronted with various heterogenous maturity assessment methods that cover different phases of the application cycle. These include established methods like SCAMPI or ISO/IEC TS 33030, which are the de facto standard for a wide range of models and are updated and refined regularly by their managing organizations. Both methods cover maturity assessment activities and address, among others, the analysis of assessment results. Furthermore, numerous proprietary assessment methods have been developed. Those methods are tailored to particular maturity models and are not applicable generally. The existing maturity assessment methods can be classified along the dimensions of generality and scope. The term generality describes the degree of generic applicability to different maturity models. Methods can be highly specific, with concrete questionnaires, or relatively generic, with structures that are free of specific references to individual models. The scope of a method has to do with how the user is supported in all phases of the application cycle. Along both classification dimensions, the methods can be clustered into two disjunct areas (see Fig. 2). Area 1 comprises methods that have been developed for a specific maturity model and cannot be applied easily to others.

This area includes limited and highly specific questionnaires that support the user during the collection of evidence and information about capabilities (Akdil et al. 2018; Schumacher et al. 2016) or methods that also describe activities that support the user within subsequent actions (Adyrbai et al. 2021; Rosemann and de Bruin 2005). Area 2 comprises methods that are relatively generic and often more comprehensive. They are designed for well-established maturity model frameworks or sets of related models within a specific domain. The most popular methods of Area 2 are SCAMPI and the ISO/IEC TS 33030. SCAMPI was originally designed for evaluating organizations against the CMMI process model, but the procedure can also be used for a limited number of related process models, such as ISO/IEC 12207 or DIN EN ISO 9001. The ISO/IEC TS 33030 is a revised version of the ISO/IEC 15504, the former Software Process Improvement and Capability Determination (SPICE), which guides users in the assessment of organizational processes and capabilities. Another contribution to Area 2 is a framework defined by Mettler (2011), which describes activities that are relevant for users as well as important decision parameters that they must consider. The 360 Digital Maturity Assessment (360DMA) is a less generic method. It is mainly applied to a range of models for digitalization but also outlines general principles that may fit any maturity model application (Colli et al. 2019). Finally, the Maturity Model Architect (MMArch) by Proença and Borbinha (2018) supports the execution of maturity assessments by using enterprise architecture models, ontologies, and description logics.

Despite a large number of existing maturity assessment methods, there are some inadequacies and weaknesses that call for further research. Some of these weaknesses have already been identified in the literature, while others can be isolated by investigating and analyzing the most relevant methods and the activities they include.

Given the existing inadequacies and weaknesses, we aimed to develop a novel maturity assessment method combining generality, comprehensiveness, and continuity. In line with existing research on maturity models, we positioned our contribution within the DSR paradigm. This decision enabled us to adopt established principles and draw on methodological guidance for the development of the artifact and its evaluation (Peffers et al. 2007). We relied on the well-established process model of Peffers et al. (2007), which is based on the methodology of Hevner et al. (2004) and provides a detailed development process for conducting DSR on information systems. The process model translates the guidelines and DSR principles of Hevner et al. (2004) into an easily applicable process. This translation enables DSR endeavors to proceed in a straightforward manner. Based on the inadequacies and weaknesses of the existing maturity assessment methods, which we described in Sect. 2.4, we defined four distinct objectives. Subsequently, we executed two development phases that included semi-structured interviews as formative evaluations. These interviews helped us to estimate and evaluate the compliance of the artifact with the design objectives (Stefanou 2001). Once the two development phases had concluded, we conducted two summative evaluations to produce empirically based feedback.

First, the artifact was assessed by practitioners from different domains to elicit interpretations and feedback from different practical contexts (Venable et al. 2016). Second, an extensive case study of a real-world setting was conducted over six months. All evaluations were conducted as part of a comprehensive evaluation strategy in line with the FEDS (Venable et al. 2016). Figure 3 overviews the development and evaluation phases.

Four concrete design objectives could be deduced from the RQ and from the inadequacies and weaknesses of the existing maturity assessment methods that we identified. Those objectives were used for orientation and guidance during the design and development phases. The first design objective (DO1) is that the design and the structure of the artifact must be understandable and easy to apply for practitioners. The second design objective (DO2) reflects the need for principles and activities that enable a continuous maturity assessment. The artifact must refer to activities that entail the tracking of capabilities and thus create a basis for an iterative assessment. The third and fourth design objectives refer directly to the classification of the artifact within the set of existing assessment methods, as described in Sect. 2.3. First, the artifact should be applicable to the largest possible number of maturity models and organizations without having specific links to existing models, that is, it should possess a high degree of generality (DO3). To that end, it is useful to define a generic assessment method that makes use of relevant information when applied at specific organizations. What information is relevant depends on the actual use case. Furthermore, the artifact should provide comprehensive activities that support users during all phases of the maturity assessment (DO4).

In the first development phase, we aimed to create an empirical basis, gain insights, and synthesize findings and best practices from past and current research on maturity assessment methods. To identify an appropriate selection of literature, we performed two structured literature reviews (SLR) to investigate both areas that we outlined in Sect. 2.3. These SLRs helped us to create a theoretical foundation for the development of our artifact (Sturm and Sunyaev 2019). The first SLR concerns the methods of Area 1 and thus investigates maturity models that are accompanied by proprietary assessment methods. The second SLR focuses on the assessment methods of Area 2.

We performed both SLRs according to the structured method of vom Brocke et al. (2009) and considered the most relevant journals and conference proceedings in the research domain by querying the databases ACM Direct Library, AISeL, IEEE Xplore, ScienceDirect, Scopus, and Springer Link. A detailed overview of the SLRs, including the search strings, the eligibility criteria, and all analyzed articles can be found in Appendix 1 (available online via http://​link.​springer.​com). During the SLRs, we found 45 eligible articles in Area 1 and eight articles in Area 2, which we analyzed in detail. Subsequently, we extracted the relevant data by using grounded theory, a qualitative research method that seeks to develop a theory that is grounded in data that is systematically gathered and analyzed (Urquhart et al. 2010). We aimed to identify patterns, common attributes and principles, and best practices from existing maturity assessment methods that can be used as a basis for an artifact. Accordingly, we applied the methods of open and axial coding, as proposed by Strauss (1997). In the first round of coding, each of the authors analyzed 20 publications from the sample. Open coding, an interpretative method, was used to disaggregate all assessment methods into their constituent parts analytically. The goal was to develop substantiated codes that would enable those parts to be described, named, and classified. The breakdown of each assessment method into separate activities is an example of this open coding approach. Codes were then assigned to the activities. After this first round of coding, we collated, compared, and contrasted the codes that we had identified. Eventually, we harmonized the individual interpretations of the main codes. In the second round, we applied the method of axial coding to connect the formulated codes to each other. During this process, we organized the codes from the previous round into categories. This categorization enabled the creation of phases within the maturity assessment methods that comprise similar activities. After a second discussion, the results were harmonized again. In round three, the remaining publications were coded with the findings of rounds 1 and 2 to test them against data. Subsequently, we clarified and resolved any remaining coding differences. Following inductive reasoning, as suggested by Hempel (1966), we extensively discussed the created codes and categories to identify best practices and the fundamental principles of maturity assessment. Table 1 shows the identified categories and codes, which are translated into phases and activities that are essential for maturity assessment methods. The table also overviews the descriptive statistics within the underlying articles. In total, we derived five generic phases, which cover all activities that form part of the investigated methods. A central finding that emerged from the coding is that few methods address the selection of models. Moreover, most proprietary methods in Area 1 only cover the preparation, assessment execution, and reporting phases. They do not provide support for adaptations or critical reflection on assessment results. Figure 4 shows how the identified phases can be mapped onto common maturity assessment methods of Area 1. After the SLRs and coding activities, we conducted seven semi-structured interviews with researchers and practitioners. This included two professors, one postdoctoral researcher, and four management consultants completing their doctoral degrees. Those interviews would serve as a formative evaluation episode EV1.

All experts possessed considerable expertise in the field of maturity models, whereas the consultants had already been involved in their practical selection, implementation, and monitoring at different organizations. The experts were presented with the RQ, the design objectives, and the created findings on maturity assessments. Against this background, they were asked which fundamental phases, activities, and indispensable aspects of maturity assessments should be considered for artifact creation. Moreover, they were asked to indicate what extensions would be necessary to achieve the design objectives. Appendix 2 shows the details and the results of the interviews.

The analysis of existing maturity assessment methods made it possible to overview the status quo and to identify fundamental phases, activities, and principles. Furthermore, the results of the formative evaluation EV1 provided us with expert knowledge on the necessary and potential extensions that would make the design objectives easier to attain. This expert knowledge formed the basis of the second development phase, in which we created the final CMAM by extending and advancing existing knowledge about assessment methods. To that end, we followed the method of abductive reasoning, a creative process that enables the introduction of new concepts and ideas (Peirce et al. 1998). Abduction can extend and create knowledge because researchers imagine and analyze all possible theoretical accounts of a given problem or a set of data and then form hypotheses until they arrive at the most plausible interpretation (Charmaz 2008). We built the CMAM iteratively by selecting the most useful phases and activities from the first development phase and extended it by incorporating information from the formative evaluation EV1. After each iteration, we performed semi-structured interviews (EV2) with the expert panel to obtain additional feedback. After three iterations and 21 interviews, we finished the procedure because the experts did not identify further extensions or new issues.

The first CMAM phase lays the foundations for all subsequent ones by having the users select and prepare an appropriate maturity model and formulate basic assessment definitions. For a suitable model to be selected, several decision parameters must be considered. They depend on business needs and situational factors. These factors and needs can include the origin of the model (i.e., academia or practice), its reliability (i.e., evaluation), its accessibility (i.e., cost of use, if any), its practicality (i.e., whether it is problem-specific or more general), and its design mutability (i.e., the convertibility of the model and the ease with which it may be integrated into the existing organizational model base) (Mettler 2011). Furthermore, the assessment approach is highly relevant. For example, much depends on whether the organization can perform all activities as self-assessments or if external support or the retention of certified experts is necessary. The latter can be relevant to assessments that are performed as part of contractor evaluations (Paulk et al. 1993). In addition, the user should become aware of the added value of the continuous application and of the resources and capacities that are required. This awareness depends strongly on the anticipated assessment goals, such as benchmarking against competitors or creating knowledge about internal capabilities (Serral et al. 2020). Thus, creating awareness necessitates a conclusive resolution of uncertainties or ambiguities in the assessment goals (SCAMPI Upgrade Team 2006). If the selected maturity model cannot be adopted because of the organizational situation and environment, goal-oriented adaptation and adjustment are possible. However, these should not change the fundamental structure of the model or contradict its objectives. Another important prerequisite to enabling an effective CMAM application is the creation of awareness about the relevance and importance of the model. This exercise should also highlight the need for continuity (Stoiber and Schönig 2022). All involved users must understand that continuous assessment is relevant and crucial to arriving at accurate and objective insights into organizational capabilities. Moreover, for the CMAM to be embedded into an organizational routine sustainably, all stakeholders must guarantee their commitment and long-term dedication (Colli et al. 2019).

Once an appropriate maturity model has been selected and all activities in Phase 1 have been performed, assessment details must be specified to enable the introduction of a systematic and iterative organizational routine. To that end, an assessment specification is created. It includes all vital building blocks, such as responsibilities, a data setup, and conditions for maintaining or terminating the CMAM. In this sense, the assessment specification describes objects, parameters, and characteristics of the real-life application of the maturity model. To facilitate the enumeration of substantiated assessment details, the CMAM provides a metamodel that can be used to instantiate the assessment specification for any underlying maturity model. Figure 6 shows the metamodel, which is presented as a Process Data Diagram (PDD), an approach that includes standards of the Unified Modeling Language (UML) (van de Weerd and Brinkkemper 2008). The PDD has already been used in related research and is sufficiently expressive for creating an appropriate model (van Steenbergen et al. 2010). The process view on the left-hand side of the diagram is based on a UML activity diagram, and the deliverables view on the right-hand side is based on a UML class diagram. The user performs the process on the left side to specify the classes and attributes on the right side. These classes represent all decision parameters and details that are relevant to the subsequent phases of the CMAM. Due to its representation as a class diagram, the metamodel can be implemented in different ways, for example, as a manual routine or as software that includes a database for defined and collected data.

In the third phase of the CMAM, the predefined assessment specification is used to track all relevant capabilities according to the formulated schedules. The product owner is responsible for either enabling the collection of quantitative data from the data sources and translating it into capabilities or for gathering capability information through qualitative methods, such as questionnaires and focus groups. The tracking results are then used to define the current maturity level. Tracking is also related to the RQ in that it enables the creation of a method for the continuous assessment of maturity through the constant monitoring of organizational capabilities. The information that is obtained thus can be used to determine maturity levels. Stakeholders obtain a more detailed and recent representation of the state of affairs, and the needs of many maturity model users are met, especially in highly dynamic environments (Englbrecht 2021; Stoiber and Schönig 2022).

In the fourth phase of the CMAM, the results of the maturity assessment are reported to the stakeholders and other focal groups. While it is not described in detail in many existing methods, reporting is an important activity in the SCAMPI, ISO/IEC TS 33030, and 360DMA, as well as in other methods. An appropriate debriefing includes a presentation of results and a comparison with prior assessments, which enable learning and the formulation of action plans. The results must be analyzed critically, and it is necessary to decide whether advances in maturity should be coupled or uncoupled from the regular target system of the organization and if improvement-related activities can be conducted ad hoc or whether specific project initiatives are necessary (Mettler 2011).

In the last phase, the user performs activities to ensure a valid CMAM representation, including an actual and correct assessment specification. Since organizational changes may necessitate the adjustment or adaptation of the specification, the user can decide to return to Phase 2 and rerun the assessment specification instantiation. The necessity of this operation depends on the maintenance conditions from the assessment specification. These conditions could include changes in organizational structures, stakeholders, or information systems for data collection. In some cases, the termination of the CMAM might be appropriate or necessary. The termination conditions may include, among others, the achievement of maturity goals, significant changes within the organization, or the introduction of a new and more suitable maturity model. In general, this phase ensures that the CMAM is validated and updated continuously, and it eventually results in the rationally justified termination of the assessment. If no maintenance or termination condition is met within a given iteration, the user is guided back to Phase 3 to track capabilities and to keep the maturity assessment alive.

Accurate evaluation is a central and critical part of DSR (March and Smith 1995), and its aim is to assess the utility that an artifact contributes to its environment (relevance cycle) and the knowledge that it adds to the knowledge base (rigor cycle) (Hevner et al. 2004). Therefore, we followed the FEDS of Venable et al. (2016), which complements the process model of Peffers et al. (2007) and extends it by introducing detailed evaluation principles. The FEDS complements and details the generic evaluation phase of Peffers et al. (2007) by introducing tools to create an overarching evaluation strategy. In this regard, it supports the (i) explication of evaluation goals, (ii) the development of an appropriate evaluation strategy, (iii) the determination of evaluation properties, and (iv) the design of evaluation episodes. The main goal of the evaluation was to support the achievement of the design objectives (ex-ante) and, eventually, to measure and assess the degree of attainment (ex-post). These two goals mean that a combination of formative and summative evaluations is required. We designed four evaluation episodes, two of which are formative (EV1 and EV2) and two of which are summative (EV3 and EV4). Table 2 presents an overview of the evaluation episodes, including the three guiding questions of evaluation in DSR, namely “why?”, “how?” and “what?” (Prat et al. 2015). The formative evaluations EV1 and EV2 were used to produce empirically based interpretations that provided a basis for improving the characteristics and the performance of the CMAM (Wiliam and Black 1996). For the summative evaluations EV3 and EV4, we defined four evaluation criteria that allowed us to conclude the attainment of the evaluation goals.

First, we chose the criterion of operational feasibility, which concerns the degree to which managers, employees, and other stakeholders might support the proposed artifact effectively, operate it, and integrate it into their daily practices (Mark et al. 2007). This process is essential because only feasible artifacts can be applied and maintained by organizations. Secondly, we evaluated the generality of the CMAM to ensure that the artifact can be used at any organization and for any maturity model. Third, the criterion of applicability was selected. Fourth, we evaluated the effectiveness of the CMAM, which we defined as the degree to which the artifact achieves its goal in real-life situations (Prat et al. 2015).

To assess the operational feasibility and generality of the CMAM, we conducted expert surveys at six organizations. To cover different scenarios and to collect heterogeneous feedback, we selected organizations from different industry sectors, of different sizes, and with varying experiences of maturity models. The organizations in question included market-leading businesses in the chemical industry, plant engineering, the manufacturing sector, and financial services. At least one interviewee at each organization was responsible for applying a maturity model at a specific business unit. In Step 1 of the interviews, the underlying RQ, the design objectives, and the CMAM, including all its phases and activities, were presented in detail. In Step 2, we proceeded with initial questions that were aimed at gathering information about the position of the interviewee within the organization, their experience with maturity models, and their awareness of the RQ. Subsequently, in Step 3, we administered a questionnaire. It contained 12 statements, and the interviewees were asked to indicate their agreement or disagreement with each. The questionnaire used the well-established Likert scale (Likert 1932), which has interviewees record their level of agreement or disagreement with a statement on a symmetric agree-disagree scale. The statements were formulated in a way that enabled us to draw direct conclusions about the two formulated evaluation criteria. In the final step, Step 4, we conducted a semi-structured interview that helped “to confirm what is already known whilst at the same time providing the opportunity for learning” (Kundisch et al. 2021; Recker 2013). Appendix 3 provides a comprehensive overview of the survey details and the results. The survey showed that all organizations were aware of the relevance of continuous maturity assessment and would implement principles and methods that support it. They pointed out that the phases and activities of the CMAM could be adopted as organizational routines within different departments and for different maturity models. The survey results also demonstrated that managers would support the adoption of the CMAM and that they saw the considerable potential benefits of its application.

After the first summative evaluation yielded positive feedback on the operational feasibility and the generality of the CMAM, the artifact was evaluated further in an industrial case study. The case study had the distinct objective of testing the hypothesis that the CMAM is an effective and applicable method for continuously assessing organizational capabilities and maturity.

This article proposes the CMAM, a comprehensive method for the continuous tracking of capabilities and the assessment of maturity that can be applied to any model and at any organization. To develop it, we analyzed the existing literature on maturity models, identified inadequacies and weaknesses, and formulated a central RQ. Drawing on the four design objectives that we defined, we extended and improved existing assessment methods on the dimensions of generality, comprehensiveness, and continuity.

Given the novel aspects of the CMAM, the implications of our research are twofold. As far as theory is concerned, the present paper analyzed existing assessment methods, investigated weaknesses and research gaps, and described a novel DSR artifact. It developed the first systematic analysis of existing assessment methods against the background of comprehensiveness, generality, and continuity. In this way, the analysis of the literature and the existing maturity models identified new perspectives and addressed existing research gaps concretely. The inclusion of continuity as an essential aspect of maturity assessments reflects a new perspective on the manner in which maturity must be assessed within rapidly changing business environments. The CMAM is a new theoretical artifact that advances and extends the existing knowledge of maturity assessments. It, therefore, has new facets that can be important to future research on the application cycles of maturity models.

From a practical and managerial standpoint, the article described a new method that can be adopted by organizations of all kinds and applied to any maturity model. The applicability and effectiveness of that method were validated by a case study, which proved that the artifact is ready for extensive practical use. In particular, the artifact will likely benefit organizations that use maturity models without proprietary assessment methods. The CMAM can be implemented as an organizational routine enabling up-to-date mapping of organizational capabilities. As shown by the study, this type of implementation makes it possible to identify improvements or deteriorations more rapidly, which means that plans for improvement processes and target setting may become more effective.

Although it is rigorous, the presented work is not without limitations. Those limitations are related to DSR, which allows various artifacts to be developed in line with different preconditions and aims at the identification of suitable rather than optimal solutions (Hevner et al. 2004). First, the selection of existing assessment methods within the SLRs and the choice of experts for the formative evaluations influenced the design and the development of the CMAM. We performed structured SLRs in line with an established method to collect all articles that may have been relevant (vom Brocke et al. 2009). While selecting the expert panel, we identified researchers and practitioners from different domains who possessed advanced expertise in the design of maturity models. Another limitation arose from the generic design of the CMAM, which does not provide concrete references, tools, or questionnaires for assessing capabilities in specific fields. However, organizations can easily embed existing questionnaires into the CMAM or create quantitative data-acquisition methods internally. The CMAM is not intended to be too specific. Therefore, the CMAM need not replace proprietary assessment methods; they can be included in superordinate frameworks. Finally, the two summative evaluation episodes were limited to six organizations, and the CMAM was only implemented at one organization. However, despite this limitation, we deem the results of the evaluation to be generalizable because the CMAM is sufficiently generic to be applicable in different scenarios.