Process Mining Success Factors and Their Interrelationships

CSF studies initially gained significant attention after Rockart (1979) highlighted their relevance in influencing the information needs of top executives (Williams and Ramaprasad 1996). CSFs are defined as “the limited number of areas in which results, if they are satisfactory, will ensure successful competitive performance for the organisation” (Rockart 1979). Since then, this concept has been adopted in diverse project-related contexts.

Despite the proliferation of CSF studies, they have been criticised (Fortune and White 2006) for providing mere lists of factors and lacking a deeper contextual understanding of how these factors may vary in importance over time (Bandara et al. 2021). For instance, in the process modelling domain, a mere list of factors without contextual understanding is considered ineffective in predicting success or designing interventions that enable process modelling success (Bandara et al. 2021). Fortune and White (2006) also argue that CSF studies in the project management domain often do not account for factor interrelationships, although these are “at least as important as the individual factors” (Fortune and White 2006). Thus, there is a clear push for CSF research to go beyond lists of factors and to provide deeper insights.

CSF studies have been conducted in related domains such as BPM and data mining (e.g., Alibabaei et al. 2009 and Sim 2003). Alibabaei et al. (2009) propose a holistic BPM success factors framework with nine CSFs and related sub-constructs and how they achieve success. The Big Data Analytics (BDA) framework by Grover et al. (2018) provides a detailed analysis of moderating factors, capabilities, and value realisation potentials for transforming BDA investments into value. Most CSF studies in BPM and data mining hardly explore CSF interrelationships, though this is a commonly criticised aspect of CSF studies (Bandara et al. 2021). While insights from related domains are valuable, context specificity is essential for a CSF study to be beneficial (Bandara et al. 2021), which points our attention to PM CSF studies.

As the PM field progresses towards maturity (Emamjome et al. 2019), the need to carefully examine the value proposition of PM has been highlighted (e.g., vom Brocke et al. 2021a). Hence, effort has been made by the PM community to understand issues relating to the enhancing and inhibiting factors for PM adoption (Syed et al. 2020), PM opportunities and challenges (Martin et al. 2021), and the common analysis strategies and challenges of PM analysts in PM projects (Zerbato et al. 2022). Other recent studies have also investigated how organisations can generate business value from PM features (Badakhshan et al. 2022) and the direct and indirect impacts of PM on organisations (Mamudu et al. 2023).

However, to the best of our knowledge, limited research (e.g., Mans et al. 2013 and Decker 2019) deeply explores PM CSF. The business process mining success model by Mans et al. (2013) is the first and most cited study on PM CSF. It describes how six factors categorised under “Project specific” (Management support, Project management and Resource availability), “Process mining related” (Process miner expertise, Process mining approach) and “IS related” (data and event log quality) are key “influential forces that contribute directly or indirectly to the success of process mining”. Using empirical evidence from four case studies, Mans et al. (2013) describe how the identified CSFs translate into three “criteria to evaluate the success of a project”: model quality, process impacts and project efficiency. Being the first comprehensive PM CSF study, the Mans et al. (2013) model could be considered a bedrock for similar studies in the PM domain. However, like most CSF models, it does not explore the nature and extent to which the identified PM CSF interrelate with regards to PM success.

Decker (2019) also identifies success factors for PM. With the overall objective of investigating how process mining can support business process optimisation, their paper undertakes a qualitative study of existing literature to identify success factors and areas of impact for PM. It proposes five PM CSFs (Data issues, Process improvement, Know-how, Management support and Organisational culture) and four PM impact factors (Automatic discovery, Transparency, Analytical capability and Achieving strategic objectives). The initial findings are validated their using semi-structured interviews with eight individuals who occupy diverse roles and have various levels of process mining experience.

Some other publications identify PM CSFs. Nevertheless, many of these studies do not set out initially to investigate success factors in PM, but only discuss these factors as prerequisites for achieving other intended outcomes. For instance, Syed et al. (2020) identify four enabling factors for process mining success at the early stages of PM adoption within an organisation: actionable insights, confidence in process mining, perceived benefits, and training and development. However, this study is based on a single case organisation and is specifically focused on the PM adoption stage, thus is potentially limited in its generalisability and broader applicability. Geyer-Klingeberg et al. (2018) also identify speed, efficiency and compliance as key success factors in organisations for strategically achieving automation and digital transformation, using PM as an enabler.

Industry reports such as the Deloitte Global IT and business executives survey⁴ has identified 19 PM CSFs. The five key factors reported are the need for a cross-departmental alignment between IT and business, good data quality and transformation, clear targets and the value hypothesis, the availability of dedicated resources towards PM, and the need for leadership commitment. However, as the respondents were all IT and business executives, the results only explained CSFs from a high-level organisational perspective with little insight into specific PM project contexts.

In summary, existing CSF literature in PM, at best, provides a list of factors. While some studies in the literature attempt to contextualise, they focus on a single case study organisation at the PM adoption stage; others explain these factors only from a high-level perspective. Some have proposed CSF interrelationships and their criticality level but without empirical validation. Considering significant maturity in PM tools, techniques and nature of current PM initiatives (Emamjome et al. 2019), validating PM CSFs and identifying their interrelationships is warranted. We aim to address this gap with our validated PM CSF model, first evidenced by secondary data from 62 published case reports, then supported with primary evidence from two real-life case studies and five expert interviews.

In each of Phases 2, 3 and 4, coding and analysis were conducted following the same standard approach—consisting of three rounds of coding and analysis. NVivo 12—a qualitative data analysis tool was used across phases 2–4 and their coding rounds. NVivo provided a solid base to managing the data gathered across the multiple phases and its functionalities such advanced queries also assisted with pattern identification.

A coding rulebook (following DeCuir-Gunby et al. 2011), was developed to ensure a formalised approach was followed (see Part B.2 of Appendix B). Coder corroborations also played a critical role across all rounds of coding, and ensured that a credible and high-quality coding process was followed. Coder corroborations were detailed reviews done by multiple members of the research team within each coding round. The overall objective of these corroboration sessions was to ensure conceptual clarity and parsimony of the CSFs, related sub-factors, their descriptions and the identified interrelationships.

In Round one, the aim was to extract the relevant data from the original data sources. Thus. we worked directly with the reports/interviews/documents (as relevant within each phase) and used an open-coding approach (Saldaña 2013). We inductively extracted all direct and indirect content regarding elements that contributed to the PM initiative’s success, going through the case reports and/or interviews line-by-line. These derived the first-level of codes- each specific to a micro-theme. In addition to creating open codes, we also captured any evidence of interrelationships (- between emerging (sub-)/themes) as indicated in the data points and coded them as ‘Relationship’ nodes⁶ in the NVivo qualitative analysis tool.⁷ Further explanations for identified interrelationships were captured in Memos⁸ during the coding process. After the inductive extraction of low-level codes by a primary coder in Round one, open-codes were discussed and critically reviewed with three secondary coders for alignment to the area of interest (i.e. PM CSFs).

In round two, the aim was to synthesise the open codes extracted from round one, and we used a hybrid approach- moving between deductive and inductive coding (Swain 2018) to achieve this. The PM CSF model resulting from the previous phase (e.g., the a-priori model from Phase 1 was used in Phase 2, and the model resulting from Phase 2 was used in Phase 3 etc.,) was used as the initial coding classification scheme where relevant open-codes were re-coded under the PM CSF and sub-factors. Open-codes that did not fit within the PM CSF model were inductively grouped to form new (sub-)factors. Using the latest version of the model as basis for the subsequent phases enabled the evolution of the model with cumulative evidence across the phases (See Appendix A).

The mapping of open-codes to the pre-identified CSFs were exposed to a detailed analysis where the resulting (sub-)factors and content coded within them, were critically analysed and refined to obtain conceptual clarity and parsimony. This involved reviewing each open-code to confirm mappings and sub-factors. Detailed coder corroboration took placed here after the (sub-)factor extraction phase where three coders reviewed the coded content of the first coder—aimed to arrive at consensus on the mapping of lower-level themes to resulting higher-level themes.

In the third round, the focus was on the identification and exploration of CSF interrelationships. This was done in two ways: (a) by revisiting the identified ‘relationships’ captured in round one coding, and (b) by complementing this method with NVivo’s matrix intersection⁹ and “near” search queries. The third round of corroboration reviewed the evolving CSF model and its interrelationships took place, during which three coders reviewed the evidence supporting each relationship to confirm (a) the existence of the relationship and (b) the nature of the relationship.

Table 5 presents the final set of validated PM CSFs, their sub-factors and descriptions. As mentioned above (Table 1)—the supporting evidence for the CSFs came from the PM case report analysis (Phase 2) and case study data (Phase 3). While Table 5 presents a summary of the supporting data points, Part C of Appendix B provides example data extracts for each CSF. They are explained in detail below- injecting evidence from the rich empirical insights obtained from the two real-life case studies.

We augment prior explanations of PM CSFs with empirical evidence from the two case studies conducted.

a. Stakeholder support and involvement

One CSF identified is the level of involvement of key stakeholders from the onset and throughout the PM initiative. In Case 1, such involvement came in the form of a mandate from top management, governance mechanisms such as setting up quarterly business reviews or business unit heads making the needed budgetary provisions for PM initiatives to thrive. R2-C1 expressed this as having “very good stakeholders who were able to push this, and together with the new Process Mining Tool, the Execution Management Tool that was part of our success.” Management support was often expressed by defining strategic goals and objectives. Aligning PM initiatives to these strategic goals and objectives enabled management to provide the needed support for PM success. R1-C1 reported how top management mandated the establishment of a process excellence team to address process improvement objectives—“process improvement was a core part of this global strategy that they announced, and what they want to do here is, first and foremost, to build a team: an operation excellence or process excellence team that brings the strategy into action”. In Case 2, R2-C2 explained how the interest of Top Management in “understanding further about how higher degree research journeys actually take place in order to see potential improvements which can be made with the system” led to the management support of a PM initiative on HDR journeys as a means to “understanding where potential improvements can be made in the journeys of the Higher Degree Research students”. The involvement of external stakeholders that influenced the business process being mined was only identified in Case 1. To achieve an integrated O2C and Purchase-2-Pay (P2P) process, there was the need to involve line managers from order management and supply chain to identify and resolve any allocation issues in the P2P process and ensure better delivery time estimates. R2-C1 reported how such collaboration between Order Management and Supply Chain enabled “the sales people know if they create a sales order, or if they sell something to the customer, they know how long it could take to deliver.” The involvement of subject matter experts (SMEs) was critical for providing the needed domain expertise for PM success. SMEs identified from both cases included business unit heads, business analysts and other individuals who had gained knowledge, experience and expertise about the process being mined and the organisational policies. They offered “a very strong collaboration between business units” to support “in deriving requirements, designing the solution, implementing it together with (implementation partner), testing it, and then rolling it out to the users in the Order-to-Cash process.” R1-C1. They were also instrumental in providing insights into understanding the process and facilitating a good understanding of key questions to be answered. R1-C2 confirmed that “occasionally we would need some SME support. When we were trying to pull up certain records and going “look these ones seem a bit odd”. And it was trying to work out “is our interpretation of the data wrong or is there an issue in the data itself?”” In Case 1, SMEs such as business heads oversaw the provision of budget for running PM initiatives. Furthermore, the buy-in of user groups was essential for the success of PM initiatives because they were instrumental in using PM tools in their operational activities. User groups also provided feedback on the usability of the PM tool by seeking ways in which the tool could be incorporated into their day-to-day operations. R3-C1 stated that they “have 250 users for this app, it’s basically the whole back office managing open orders in Europe”. R2-C1 also affirmed they did not have to convince user groups to run PM “because they loved it from the beginning; it was very helpful for us”.

b. Information availability

The extent to which right information resources were available for PM was considered essential for PM success. Such resources were often in the form of event data from business systems, detailed workflows, benchmarking and KPI information and privacy regulations. Such information was usually stored in business systems or in spreadsheet files. In Case 1, R1-C1 confirmed that information resources available for PM were stored in SAP systems “where they entered the customer orders and then edited the customer orders until they got shipped to the customer, and they had an Excel file where they tracked the status of the orders.” R1-C2 also indicated that for Case 2, there were “a couple of different data sources that we used. One was research master, which was the system we were trying to replace and the key data for that was around milestones in the system” R2-C2 added that “we also got individual data sets for each of the eForms which are lodged by students. And that we had seven imports for which we had to analyse the data.” The availability of contextual information such as process model documentation, meta-data (e.g., data on access controls, technical documentation on copyright information) and other organisational policies, and regulatory and compliance requirements enabled a clear understanding of the business process. It ensured that PM was executed within the right context. R3-C1 indicated that “we do need to understand how SAP books stuff, and how purchase orders flow through the system, how sales orders flow through and where they are connected.” R2-C2 explained how members with contextual knowledge were useful members of the team—“there were other people involved in the team who are not that familiar with how the system works from inside, but they are very well aware of the processes, policies, rules and guidelines who were involved in the team.”

c. Technical expertise

A crucial factor for executing PM projects effectively was the availability of technical expertise. Such expertise whether provided in-house or externally, came from data scientists, analysts and process modelling experts who had the ability to extract and integrate data from multiple sources, use PM tools to extract insights from event logs and build automated processes and applications. Three types of technical expertise were identified. First, process mining expertise—the competence of applying PM tools to extract insights from available event data was a critical skillset. In Case 1, PM analysts were technical counterparts for data engineers. They were critical for setting up the “technical things which are needed for process mining, threading the connection, testing the connection, do the SAP transports and so on.” R2-C1. They were also responsible for creating meaning out of the extracted data. In addition, R1-C2 confirmed that PM was able to take event data of the HDR process being mined “and actually turn that into what the new processes should look like in the implementation.” Second, for data extraction expertise, initial data engineering expertise facilitated a successful PM implementation. R1-C1 confirmed that the data analytics team within Case 1’s IT “own everything that has to do with data analytics, so they also own the BI solutions and stuff, so they are working together with them (process mining team) to make sure that they look at the right data, that they validate the data.” R1-C2 who was in charge of data extraction explained how he “and a couple of other technical experts who are dealing with reporting or, you know, building the reports in the system data migration, working towards integrations” to get the data out in a meaningful way for the process analysts. R2-C1 also explained his role in “creating new activities, setting up the whole data model … basically the whole ETL pipeline, from extracting transformation and loading for Order-to-Cash” as a precursor to the start of the PM activity. Thirdly, process analyst expertise—extensive knowledge in traditional process modelling techniques was critical, as it provided the competence to build new process models in-house. Case 1 had a well-setup cloud-based solution for process modelling and documentation. R1-C1 explained that “this is the solution where you draft, design, and store your business processes, like, you know, BPM models? This is what they do in BIC Cloud. Also owned by the team that’s owning Celonis, so they have a strong knowledge in business process management, and now also in process mining”. R2-C2 also confirmed that based on PM initiative questions to be answered they “tried to develop models; to develop models we used WoPeD (workflow and petri net designer)”.

d Team configuration

Considering the nature of PM initiatives in the case reports (see Sect. 3.2), team configuration was initially captured as a sub-factor of technical expertise. However, empirical evidence showed that ensuring a balanced configuration of technical and non-technical experts to embark on PM initiatives was critical to the success of PM initiatives. Team configuration has, therefore, been validated as a CSF for PM. From these two cases, the team configuration for PM initiatives consisted of internal and external stakeholders such as subject matter experts, change managers, external consultants, and implementation partners. Such PM teams were either established units such as a Centre of Excellence (CoE), Process Excellence Centre (PEC), Process Mining Insights (PMI), Business Process Leadership (BPL) or Process Mining Consulting (PMC) team or ad-hoc units composed of a temporary group of experts with the needed knowledge and expertise to execute PM initiatives. In Case 1, it was reported that consultants could also play a role in PM teams by providing tool support and access to implementation partners for PM implementation and training purposes.

e. Structured process mining approach

Both cases applied unique approaches to PM. However, the key stages in their approach resonated with the PM² Framework (van Eck et al. 2015) adopted in the re-specified PM CSF model in Sect. 4.1. As both organisations considered PM more iterative and discovery-based, a key component in both cases was the co-design approach which required constant stakeholder collaboration and interaction throughout the PM initiative. Planning was a critical first step in the PM approaches analysed. It involved identifying process improvement goals, opportunities or problems/questions to be addressed. Some goals and objectives from the empirical case studies were “to fully understand the process and derive inefficiencies out of those.” (R1-C1). R3-C1 confirmed that this goal stemmed from “a real problem with the Order-to-Cash Process; customers said, ‘We are not satisfied with your performance’”. R2-C2 also stated that in Case 2, the goal to be achieved was “clarity in high degree research journeys and process mining was an enabler for that one goal”. Extraction involved the specific activities of identifying data sources and taking relevant event data for PM. Some extraction techniques applied in Case 1 were setting up data connections, data integration and data loads, while extraction activities in Case 2 mainly consisted of extracting and migrating data from multiple data tables. Steps taken regarding data processing indicated that the nature of the process to be mined influenced the form of event logs generated. R3-C1 expressed: “we are actually now cutting our data model into pieces. So, we started to do data modelling work, and we create a document centred data model.” R1-C2 also explained the time-consuming nature of data processing as a result of the need “to sort of bring together multiple pieces of data and hope that it worked, and when we looked at it from a data quality perspective, there was a lot of issues in there.”

The mining and analysis stage often began with the automated discovery of as-is process (process discovery). This could be achieved by using a single PM tool (as reported in Case 1) or multiple tools (as reported in Case 2). “We used multiple different plug-ins in ProM. We used Disco as well to cross check the results and validate them. We also used WoPeD which was used to draw Petri Nets and process models.” (R2-C2). Other analysis techniques for identifying bottlenecks and process inefficiencies, pattern analysis and drill downs were undertaken. R3-C1 stated that in extracting insights to understand the O2C process, they “had the beautiful analysis showing the typical stuff like undesired activities, then the analytics or throughput times”. R2-C2 confirmed that analysis focused on addressing stakeholder interest “to understand what are the possible indicators, risk indicators of withdrawing from a PhD journey that we can find from data.”

Evaluation focused on comparing analysis results to improvement ideas to achieve project goals. R2-C1 noted that “the customer wants to have a very transparent delivery date, and we saw that in process mining that they don’t have this currently. The question then was why (Case 1) is not able to promise a good date for the customer, and the reason was … is still, the supply chain.” R3-C2 stated that “in terms of the outcomes for the project, the data we could pull in kind of demonstrated how long operationally, it took us to do things. So once the student submits, let’s say a leave request or an extension request, it would take X number of days with the Supervisor, X number of days with the faculty, X number of days of us and the gate. So that gave us a benchmark.” Process improvement and support involved the actions taken to adjust business processes based on new insights gained. R3-C1 expressed “so, there you had a project which started with the combination of RPA and process mining, the analytics behind it, some insights, and then a project that actually turned this into change actions and improving the process.” R2-C2 reported it thus: “we found deviations, we actually saw that there were I guess around 300 students who were actually doing annual progress report before the confirmation seminar. So that was an interesting finding, which again resulted in a new intervention in the new workflow.”

f. Data and event log quality

In both secondary and empirical evidence, there was acknowledgement of the significance of ensuring data and event log quality as pre-requisites for PM success. Hence, deliberate steps were taken to ensure the quality of event data. On actions taken towards data pre-processing, respondents from Case 1 reported only minor data quality issues on their data from the ERP system. R2-C1 stated that “so, when cleaning the data, we haven’t had any big problems with weird data. We just had problems with price values…there have been a lot of problems of how the prices of sales orders, have been calculated, and we created a new calculation schema”. On the other hand, Case 2 encountered major data quality issues such as noise, missing data and data inconsistency. R1-C2 recounted: “the other one would just be generally the noise because you ended up with lots of things going on. Sometimes the records would get messy if you had a student who had, you know, done a masters and then done a PhD because you might have some of the relics of when they did their masters in there as well, that the way that data was held was sort of a single record of a student, and normally it would clear out previous studies.”

For event log quality considerations, it was seen that data governance and ownership provisions were critical for event log quality and a pre-requisite for valuable process insights during PM. R3-C1 reported: “I think it’s actually a key enabler for a good process mining, that people know where the data is coming from, who owns it, and who defines the quality standard that you don’t have to take care of it.” In Case 2, however, although policies regulations for guiding the process being mined pertained, respondents confirmed that “in terms of provision, there was no one who provided us some systems or policies in order to rectify the data quality.” (R2-C2).

g. Tool capabilities

The key features and capabilities of PM tools were essential for executing PM projects. The ability to provide automated process discovery and visualisation or process models was a popular feature in both case scenarios. R1-C1 indicated that “there are certain ways of how you can apply process mining to a business process, and the most common, probably, is really to first and foremost get the 100% transparency on your as-is process”. R3-C2 also confirmed that “there was a presentation by the project team where basically students were made into little dots and you could see the entire journey they did and how many times they went in under review or whatever. And so I think in terms of an engagement exercise that was a really fantastic way of taking this really, you know, abstract information and being able to demonstrate in an animation, kind of the journey the student was running”.

Process benchmarking was another popular PM tool feature. Case 1 reported using PM to measure process performance metrics, throughput time, and identify process variants. R1-C1 reported the ability of the Celonis PM tool to “measure the process performance based on performance KPI’s like, you know, throughput time, and how many process variants do we have, and stuff like that”. R2-C2 also reported how they performed “comparative analysis where we divided the long into cohorts, and we had a cohort which took comparatively less time, which is around 3, 3½ years, and a cohort which takes more than 3½ years and tried to compare their processes.”

For the conformance checking functionality, R2-C1 reported that “there was also a conformance checker, that could create a BPM model, and compare the BPM model to the Process Mining Model; that was helpful”. R2-C2 also confirmed that “conformance was another thing that we did. So, according to the policies at (Case 2), the student should not be doing an annual progress report before confirmation. If they do an annual progress report before confirmation, it’s just wastage of time for the student as well as resources for the university as well as time for the supervisor”. In both cases, pattern analysis was conducted using the PM tool. R2-C1 reported how pattern analysis was used to identify undesired activities within the process, which were mostly change activities that needed to be rectified manually. R2-C2 also explained how pattern analysis was used to assess risk indicators (such as leave) for possible PhD student dropouts.

The PM tool’s filtering feature was found to be very useful for segregating problem areas for a detailed check. R2-C1 reported on the ability to “filter on such problems in the whole analysis, and then we have more dashboards, just still filtered on such problems”. R2-C2 also indicated: “we wanted to filter things out and we used SQL Server for that. We wanted to check some records in detail and we used SQL Server for that”. The drill down capabilities of the PM tool was essential to explore different levels of granularity of a process and identify problem spots, especially in Case 1. For instance, R2-C1 reported how drill downs were created on the master data of the O2C process to understand the root causes for sales prices increasing by 20%.

While PM tool users in Case 1 reported benefits from the integration capabilities of their PM tools, Case 2 users reported no direct outcomes as multiple PM tools were used in a “purpose-built” manner. R1-C1 indicated the ability of their PM tool to integrate with SAP systems and other BI tools. “We have a new technology inside Celonis where you can connect your Celonis to a BI system, so we are also merging this now together…. we also built a very strong integration with the SAP system so whenever an order manager is changing an order in Celonis, it gets written back to the SAP system.” With analytical scalability features, initial findings indicated the ability of organisations to analyse a single process or end-to-end (e2e) process at various levels of detail and even at high data frequencies. Discussing their experience, R3-C1 indicated that “we moved from pilot to European license to global license, so the scaling works, to some extent, and we are able to offer, now, our subsidiary in the US, to basically repeat, to some extent, what we did in Europe”. However, there was no report of analytical scalability in Case 2. In addition to the above, other PM tool capabilities were identified from the empirical evidence.

h. Project management

The ability of organisations to define the specificity of a PM initiative, boundary conditions and its relationship to other projects was a critical success factor for PM. As these initiatives were often executed within a larger organisational project context, a clearly defined project scope, governance mechanism and budget were essential for their success. In these two cases, project scope was often a specific defined aspect of a larger ongoing program. For instance, the PM initiative in Case 1 as reported by R1-C1 “was all based around their goals that were derived out of this global strategy. So, they really broke it down, narrowed it down, and said, ‘this is what we need to achieve, we need to achieve employee efficiency, we need to achieve customer satisfaction, order to cash process, we need to achieve cost savings, improve our margins,’ and so on, and so we definitely discussed very closely of how Celonis can help them reach their goals as part of this global strategy.” R2-C2 also indicated that “at the top level we have the digital transformation, then digital transformation can be divided into multiple goals. Alright, one of those multiple goals was clarity in higher degree journeys, right? And to understand the key decision points.”

In terms of governance, it was essential to have the necessary structures, review mechanisms and capabilities in place to ensure its success. R1-C1 reported “the governance needed some improvement, and by governance, I really mean the way how they set up the program management, how they set up the responsibilities within this whole organisation, how they set up the communication between business, IT, management, and us.” R1-C2 also opined that “process mining is good to do, but if you don’t have that structure of who your process owners are, and the people then feeling empowered to actually make the decision changes based on that, then it can end up just being a report, not actually something that’s leading to positive change. And I’m not sure if there was enough of that structure in place to ensure that the results of this were well utilised.”

The allocation of cost and budget by business unit heads was critical for PM success. R3-C1 reported that “there is some responsibility that comes with this, yeah, and we spend single digit millions on this, and you can always ask the question if that’s worth the investment, yeah? And it should be asked every year and every time.” R3-C2 also indicated: “So I think that it’s about the value in what the actual service is, and then also finding the funding to actually engage with the project team to deliver on this.”

i Change management

Having a well-defined and highly efficient change management approach was critical to accommodate the high rate of continuous change that PM brings. Both Case 1 and Case 2 had their own established change management team and procedures. Representatives from the change management team were involved in the PM initiative from the onset. Their role, however, actively kicked in after change initiatives had been proposed. The role of change managers was to facilitate organisational readiness for change initiatives likely to arise from PM. R2-C1 reported that “creating an app is not by itself doing any optimisation; it’s also change management. So, we change, together with a Change Manager, the whole order management process, and based on the new tool we created”. (R1-C2) confirmed the significance of change management activities as an ongoing monitoring mechanism—“it’s important to have an ongoing monitoring set up because then you can actually track change over time”.

j. Training

The CSF interrelationships were originally identified in Phase 2 via the secondary case report analysis. We explored, re-specified and validated these using empirical evidence from two case studies (Phase 3) and five expert interviews (Phase 4). Note that interrelationships were sought for across Phases 2–4 of the study and the primary aim of the expert interviews was to confirm these interrelationships.

The approach used for interrelationship identification and validation was outlined as ‘Round 3 coding’ presented in Sect. 3.2. Supporting evidence for CSF interrelationships, are presented in Part D of Appendix B. In this qualitative exploration, we decided to consider and report on CSF interrelationships that were supported by at least two forms of empirical data (i.e., case study data and interviews).

Figure 1 presents a visual summary of the CSF interrelationships identified through this. Note that there were multiple types of relationships: direct, indirect and bilateral. Following Frazier et al, (2004), we define ‘direct’ relationships as those where one factor influences another. Indirect relationships are those relationships between two CSFs which are influenced by moderating or mediating variables (moderating variables alter the strength or direction of a predictor—outcome relationship while mediating relationships are the mechanism by which a predictor influences an outcome). Bilateral relationships capture a two-way influence.

Each of these are described below. We first present those relationships that were initially identified from the PM case report analysis (Phase 2), and then present those that were initially identified from the case studies (Phase 3).

The bilateral relationship between technical expertise and stakeholder support and involvement [P1] was confirmed in both case studies. Such interaction was critical in determining relevant questions to be answered to meet PM objectives and the nature of PM analysis to be conducted to provide the needed insights. R2-C2, the PM analyst in Case 2 confirmed that “you need to understand what are the key questions of interest. And those key questions were not possible to articulate without interactions with the stakeholders”. The constant communication of PM insights from technical experts to stakeholders was also essential. Ex2 and Ex5 also explained the critical role of technical experts in extracting raw data for analysing "the right part of the process so that it makes sense" (Ex2) and the role of SMEs in explaining KPIs and metrics for an accurate interpretation of PM results.

It was confirmed that a structured process mining approach moderates technical expertise and stakeholder support and involvement [P1a]. Stakeholder support was critical at most stages of the PM process. By following a structured PM approach, interaction between stakeholders and technical experts was enhanced during the planning stage. Project questions were discussed to ensure that the right questions were being addressed by the technical team. Technical experts also discussed intermediate results with stakeholders during the mining and analysis stage “so that if we are going in the wrong direction or if this is some information that the stakeholders don’t require, then we change our approach and we start answering the questions that are more important for the stakeholders.” (R2-C2). Stakeholders were also involved in the final stage of evaluation and process improvement. Ex3 also confirmed how as part of the PM approach, initial results and recommendations were presented to stakeholders at a workshop for validation prior to proceeding with process changes and enhancements.

Technical expertise influences the impact of data and event log quality [P2]. The expertise of data engineers and data scientists was critical in identifying and resolving data quality issues. R1-C2 recounted the significance of engaging data experts “you needed to have somebody who actually knew what all of the individual fields were and how it all fit together in the broader picture.” (R1-C2). Such expertise was essential for ensuring meaningful data extractions for PM initiatives “because of the way that the data was structured and had a lot of issues with it”. Data experts and PM analysts identified and addressed data quality issues such as missing data, noise, granularity issues, and synonymous labels. Ex5 reported the relevance of data specialists to understand, manipulate and refine system data when extracting event logs “because there is always different data standards across different systems” (Ex5).

Technical expertise directly influences the ability to maximise tool capabilities[P3]. In Case 1, R1-C1 explained how technical experts were able to integrate other platforms into their PM tools within a day as integration expertise was built over time. Where technical experts had a vast level of knowledge and expertise on PM and other complementary tools, they were able to provide a vast array of insights and outcomes using available tools. “We tried to understand performance statistics which involves time and all for that, we used an appropriate plug-in from ProM or we used Disco at times. We wanted to filter things out and we used SQL Server for that. We wanted to check some records in detail and we used SQL Server for that.” (R2-C2). Ex5 expressed the significant role of technical experts in integrating core PM tools with other systems and databases.

It was confirmed that technical expertise enables the overall change management [P4] of PM initiatives. R1-C1 recounted how in Case 1, technical experts were involved in creating and managing software, data or system-related change requests to facilitate the implementation of system changes. In case 2, process improvement initiatives from PM analysts started discussions around “how to progressively introduce it and ensure that it becomes a part of the normal routine of staff and students” (R2-C2). Ex3 also reports how technical experts were instrumental in proposing change recommendations from PM results.

Information availability can directly enhance or inhibit data and event log quality[P5]. The form in which event data was made available and the degree of accessibility directly influenced data quality. It was reported in Case 1 that access to data from source systems rather than the data lake reduced the complexity of obtaining such data and its quality issues. Access to data through a custodian also enhanced the quality as they often worked with the PM team “to make sure that they look at the right data, that they validate the data” (R3-C1). Ex1 also confirmed how the lack of data availability could impact on data and event log quality, thereby negatively impacting analysis results.

Tool capabilities mediates stakeholder support and involvement and change management [P6]. Case 1 reported how specific tool capabilities were essential for deriving insights which led to the implementation of change initiatives. For instance, the process discovery feature was essential for deriving HDR student journey process models which were later streamlined for the new system. Comparative analysis capabilities were also used to analyse journeys across faculties “to find the differences for a particular process across the faculties” which “led to a case of standardisation for running of those processes across those faculties” (R2-C2). Through conformance checking, process deviations were identified which led to the standardisation of the HDR journey process across faculties, and the introduction of health checks reminder emails. Ex5 also reported how the graphical features of their PM tool have helped process owners justify change initiatives, “helped made an influence and flipped the bank in working in a very different way”.

It was confirmed from Case 1 that training directly influences stakeholder support and involvement [P7] as operative users were required to undergo some training before using the PM tool. “Before they get a user, they need to do small training. So, we have a governance for enablement here to make sure that when they start working with Celonis, that they understand how the solution works.” (R1-C1). Documents and live demos were also used to as part of the training process. Ex1 reported how training user groups was critical for curtailing instances where they “will either have wrong results, or will have problems in the end because they can't handle any extraction questions.”

Technical expertise directly influences team configuration [P8]. R3-C1 reported that “having a skilled implementation team is absolutely critical. You need your data engineers and data scientists.” The absence of technical experts within the PM team posed many data extraction and analysis challenges. “I spent the last couple of weeks with the SAP Basis Team to sort out the real-time extraction challenge; that shouldn’t be me doing this. I should have a person who’s closer to the SAP Basis Team to just sort this out and get back to me and say, ‘We fixed it.” (R3-C1). R3-C2 further explained how the absence of technical experts within the ad-hoc PM team for Case 2 inhibited the ability of the team to extract insights from available data via PM: “we identified that within the project team itself we didn't really know how to pull this information.” (R3-C2). Ex2 also expressed the essence of ensuring the presence of a well-balanced set of technical skillsets for data extraction, data transformation and analysis within the PM team for attaining the needed analysis results. “me and my implementation team usually cover the weakness rather than the team comes in and they've got all parts of the technical expertise that you would need. So sometimes they're very good at the initial abstraction and initial transformations, but not so good at then thinking about what the business KPIs that will need to then model in the solution to be able to show what that impact could be. Other times it's the other way round.”

Team configuration moderates technical expertise and stakeholder support and involvement [P1b]. The presence of established or ad-hoc units moderated how technical experts such as data engineers and PM analysts interacted with other stakeholders for the success of PM initiatives. R3-C1 stated it thus: “we had this Data Engineering Team who developed the analysis, and then we needed to validate it with a business again…data engineer needs to be very well informed about what’s happening in the business, and that close connection of business domain knowledge, data analytics, process knowledge with the data engineer, it only means you need to have an integrated team. So, the collaboration with domain experts, data engineering, analysts, is absolutely necessary.” Ex4 reported that configuring a PM team was scenario dependent because “the balance depends on the case, but I think establishing a team leads to a more multidimensional view on everything…. you have someone in charge of data cleaning, you have the data scientist and then you have the process analyst or the process mining analyst. And then you have say the SME or the domain expert, maybe users, maybe a champion and maybe a consultant also coming in to help with this or that or you know project manager, things like that”.

Stakeholder support and involvement influences team configuration[P9]. Stakeholders provided the mandate for setting up CoEs, engaging ad-hoc teams or consultants. This created the enabling environment for such teams to operate in, ensuring successful PM initiatives. “Process improvement was a core part of this global strategy that they announced, and what they want to do here is, first and foremost, to build a team … an operation excellence or process excellence team that brings the strategy into action at (Case 1)” (R1-C1). Ex4 reported how the absence of domain expertise could lead to missing “a lot of vital knowledge to make the analysis work.”

Team configuration moderates between stakeholder support and involvement and project management[P10a]. Stakeholders were also seen to provide their support by mandating the setup of CoE/Process Excellence Teams or involving consultants (Team Configuration) to facilitate the execution of Projects. By giving the CoE a mandate to operate and augmenting the level of expertise, Case 1 was able to expand its program. “So, we really created a lighthouse (project), which then led to a visibility on the board level that they said, ‘Wow, what the Navy Seal team is doing there, it’s really working well. Let’s increase the team, let’s increase the mandate that they have, let’s really make it a big program.’” (R1-C1). Ex4 and Ex5 also reported how team configuration heightened PM expectations, leading to increased stakeholder support and involvement in project cost and budgeting decisions.

Empirical data also confirmed the influence of team configuration on training [P11]. R2-C1 confirmed how “a lot of trainings were done, a lot of documentation for the sales reps, how to use (Case 1) in the daily business” by consultants or members of the PM team. For instance, they created illustrative dashboards to enable users to learn how to create dashboards using a PM tool. R1-C2 also reported how the ad-hoc PM team “at various points we would have training support as well within the team” specifically to train users who would be using the new systems going forward. Ex5 recounted the role of consultants in training users to “to make sure that we get it working correctly to get the benefit out of it” (Ex5).

Team configuration influenced various aspects of the structured process mining approach [P12]. Ad-hoc teams, established units or consultants were directly involved at the planning stage in co-designing solutions and creating implementation plans for PM initiatives to address questions and problems raised by stakeholders. PM team members were also involved in configuring and testing data connections for PM tools and facilitating process improvement initiatives. R1-C1 reported “we have the Process Excellence team, they are really responsible for all of the (PM tool) process mining and process improvement stuff” (R1-C1). Ex4 also reported the influence of a team-based approach to PM as essential for obtaining “higher quality results in general of the analysis of the insights” (Ex4).

It is worth noting that the mediating role of tool capabilities on structured process mining approach and project management and direct relationship between structured process mining approach and data and event log quality are not reported as they are only supported by secondary and case study data and not by evidence from expert interviews (see Part D of Appendix 2). Also, three CSF interrelationships initially identified from the secondary data, could not be empirically validated. i.e., (i) a direct relationship between technical expertise and training, (ii) structured process mining approach moderating technical expertise and training and (iii) a direct relationship between technical expertise and project management. Empirical evidence indicated that although technical experts were directly involved in training, this was conducted from a team configuration perspective. Ad-hoc units, established units, or consultants were directly involved in undertaking training activities. This relationship was replaced with [P11] as identified in Fig. 1. There was no empirical evidence to support the influence of technical expertise on project management. However, the moderating role of team configuration on stakeholder support and involvement and project management [P10a] implied that technical expertise indirectly influences project management via team configuration.