Business process improvement with the AB-BPM methodology
Introduction
Various lifecycle approaches to Business Process Management (BPM) have a common assumption that a process is incrementally improved in the redesign phase [1, Ch. 1]. While this assumption is hardly questioned in BPM research, there is evidence from the field of AB testing that improvement concepts often do not lead to actual improvements. For instance, work on business improvement ideas found that 75 percent did not lead to improvement: half of them had no impact while approximately a quarter turned out to be even harmful [2]. The results are comparable to that of a study of the Microsoft website, in which only one third of the ideas observed had a positive impact, while the remaining had no or negative impact [3]. The same study also observed that customer preferences were difficult to anticipate before deployment, and that customer research did not predict customer behaviour accurately.
If incremental process improvement can only be achieved in a fraction of the cases, there is a need to rapidly validate the assumed benefits. Unfortunately, there are currently two major challenges for such an immediate validation. The first one is methodological. Classical BPM lifecycle approaches build on a labour-intensive analysis of the current process, which leads to the deployment of a redesigned version. This new version is monitored in operation, and if it does not meet performance objectives, it is made subject to analysis again. All this takes time. The second challenge is architectural. Contemporary Business Process Management Systems (BPMSs) enable quick deployment of process improvements, but they do not offer support for validating improvement assumptions. A performance comparison between the old and the new version may be biased since contextual factors might have changed at the same time. How a rapid validation of improvement assumptions can be integrated in the BPM lifecycle and in BPMSs is an open research question.
We address this question by extending the business process lifecycle and providing techniques for these extensions. Our AB-BPM methodology integrates business process execution concepts with the idea of AB testing from DevOps, and supports the design of AB tests with simulation. The methodology and supporting techniques as a whole provide support for validating improvement assumptions inherent in new process versions.
AB testing compares two versions of a deployed product (e.g., a Web page) by observing users’ responses to versions A and B, and determines which one performs better [4]. We implement this technique in such a way that two versions (A and B) of a process are operational in parallel and any new process instance is routed to one of them. Through a series of experiments and observations, we have developed an instance routing algorithm, LTAvgR, which is adapted to the context of executing business processes. The routing decision is guided by the observed performance metrics of each version at runtime.
To manage the risks of exposing even a few customers to clearly inferior versions during AB tests, we propose a technique to simulate new versions of business processes beforehand, using the execution logs and performance data of the old version. For the purpose of this simulation, we devise a data structure, the Transition Simulation Tree (TST), which summarizes decisions and performance metrics available in the event log of a process. The TST allows the simulator to extrapolate historical observations from the existing version of a process to the new version, with minimal assumptions about how the process is implemented. The results of this simulation can be used for preliminary analysis of potential improvements, e.g., to rule out performing AB testing with a clearly inferior new version. They can also help in the designing rewards and configuring parameters of LTAvgR.
In an earlier version of this work [5], we proposed the AB-BPM approach. In this paper, we expand on this idea and show how it fits into a methodology that provides validation support for process improvement assumption. We also introduce a simulation technique that complements the AB testing approach.
The remainder of this paper starts with a discussion of the requirements and prior work in Section 2. Section 3 describes the lifecycle, techniques, and the framework that facilitate the AB-BPM methodology. In Section 4, we evaluate our AB testing and simulation approach. In Section 5, we discuss the strengths ans weaknesses of our approach, and finally draw conclusions in Section 6.
Section snippets
Background
This section discusses the background of our research. Section 2.1 identifies requirements for rapid validation of process improvements. Section 2.2 discusses in how far these requirements are addressed by prior research and outlines the general idea of AB-BPM.
AB-BPM approach and methodology
In this section, we present the AB-BPM methodology and the technical solutions that enable it. The first of these solutions is simulation, for which we discuss how we extract decisions and metrics from the event log of a process and use that to simulate new versions. Then we discuss the mapping of the instance routing problem to algorithms from the literature. Based on an experiment, we choose one algorithm and adapt it to the context of business processes. Finally, we present our high-level
Evaluation
In this section, we present the methodology and the outcomes of our evaluation of the proposed approach. We assess the AB-BPM methodology in terms of simulating a new process version using the log of the original version, as well as AB tests using the LTAvgR algorithm and find the best performing version. We use three datasets: synthetic data (helicopter licencingprocess, HL), three versions of a real-world loan approval process from a Dutch bank (Bank), and five versions of a real-world
Discussion
In the following, we discuss relative strengths and limitations of the techniques for simulation and AB testing proposed in this paper, as well as the evaluation. We also highlight opportunities for future work.
Conclusion
Business process improvement ideas do not necessarily manifest in actual improvements. In this paper we proposed the AB-BPM methodology and framework which can rapidly validate process improvement efforts, ensure fair comparison, and make process level adjustments in production environments. The methodology is supported by the framework, which includes our simulation and AB testing techniques, and the LTAvgR algorithm.
Our simulation technique extracts metrics and decision probabilities from the
Acknowledgements
The work of Claudio Di Ciccio has received funding from the EU H2020 programme under the MSCA-RISE agreement 645751 (RISE_BPM). We thank CASA for providing the information used in Section 4.1.
References (48)
- et al.
Best practices in business process redesign: an overview and qualitative evaluation of successful redesign heuristics
Omega
(2005) - et al.
An integrated approach based on execution measures for the continuous improvement of business processes realized by services
Inf. Softw. Technol.
(2014) - et al.
Workflow simulation for operational decision support
Data Knowl. Eng.
(2009) - et al.
Prediction of business process durations using non-Markovian stochastic petri nets
Inf. Syst.
(2015) - et al.
Fundamentals of Business Process Management
(2013) Breakthrough Business Results With MVT: A Fast, Cost-Free “Secret Weapon” for Boosting Sales, Cutting Expenses, and Improving Any Business Process
(2005)- et al.
Controlled experiments on the web: survey and practical guide
Data Min. Knowl. Discov.
(2009) - T. Crook, B. Frasca, R. Kohavi, R. Longbotham, Seven pitfalls to avoid when running controlled experiments on the web,...
- et al.
AB-BPM: Performance-driven instance routing for business process improvement
- et al.
A conceptual framework for understanding business processes and business process modelling
Inf. Syst. J.
(2000)
Reengineering the Corporation: A Manifesto for Business Revolution
Process Innovation: Reengineering Work Through Information Technology
The implementation of business process reengineering
J. Manage. Inf. Syst.
Business process change: A study of methodologies, techniques, and tools
MIS Q.
Investigating the success of operational business process management systems
Inf. Tech. Mgmt.
Work system theory: Overview of core concepts, extensions, and challenges for the future
J. AIS
Managing Quality: The Strategic and Competitive Edge
A statistical process control approach to business activity monitoring
IIE Trans.
Toyota Production System: Beyond Large-Scale Production
Similarities and differences between TQM, six sigma and lean
TQM Mag.
Cause, effect, efficiency and soft systems models
J. Oper. Res. Soc.
The role of communication and management support in a lean manufacturing implementation
Manag. Decis.
Cited by (35)
A/B testing: A systematic literature review
2024, Journal of Systems and SoftwareThe biggest business process management problems to solve before we die
2023, Computers in IndustryProcess science in action: A literature review on process mining in business management
2021, Technological Forecasting and Social ChangeCitation Excerpt :The scientific community has investigated PM mostly from a Computer Science standpoint mainly by developing new algorithms (Günther and van der Aalst, 2007; Leemans et al., 2014), improving the existing algorithms (Li et al., 2007; Weijters and Ribeiro, 2011) or checking for their scalability (Leemans et al., 2018) and focusing on pre-processing steps (Günther and van der Aalst, 2006; Song et al., 2009). From a Business Management point of view, PM appears to support decision making (e.g. Satyal et al., 2019). Nonetheless, the exploitation of PM for making better operational, tactical, and strategic decisions has been moved into the background.
Prototyping a business process improvement plan. An evidence-based approach
2021, Information SystemsCitation Excerpt :Their core claim is that process innovation and improved performance eventually require variable process behaviors from process workers with sufficient autonomy to enable continuous adaptation. In [55], authors argue about the value of having a rapid validation of process improvement efforts. To this end, they assume a redesigned version of the model and exploit ideas from DevOps (AB testing — testing the previous and the redesigned model simultaneously) to choose suitable improvements.
Every apprentice needs a master: Feedback-based effectiveness improvements for process model matching
2021, Information SystemsCitation Excerpt :Correspondences might also be revealed in the context of process mining, e.g., when low level events are aggregated into more abstract higher level events [66], or by comparing event logs underlying process models [67]. Further, process similarity search and process querying discover functional relationships between processes stemming from an overlap in their purposes, e.g., two models might be different variants or versions of the same process [68]. Finally, topical relationships between domain terms and models or model elements can be the result of element tagging or might be inferred from keyword search.