1 Introduction
2 Theoretical background and design objectives
2.1 Business process management and capability development
2.2 Project portfolio selection and scheduling
2.3 Value-based management
3 Research method and evaluation strategy
4 Design specification
4.1 Conceptual architecture
4.2 Objective function
4.3 Project types and performance effects
4.4 Integrating the performance effects into the objective function
4.5 Interactions and domain-specific constraints
Interactions among projects | ||
Local mutual exclusiveness |
\(LocMutEx(s,s^{\prime})\)
| Either project s or s′ can be implemented in the same period. According to assumption (A.2), all process-level projects referring to the same process are locally mutually exclusive |
Global mutual exclusiveness |
\(GloMutEx(s,s')\)
| Either project s or s′ can be implemented in the same project roadmap |
Local mutual dependency |
\(LocMutDep(s,s^{\prime})\)
| If project s or s′ is included in a project roadmap, the other project must be included as well. The implementation of both projects must start in the same period |
Global mutual dependency |
\(GloMutDep(s,s^{\prime})\)
| If project s or s′ is included in a project roadmap, the other project must be included as well |
Predecessor/successor |
\(PreSuc(s,s^{\prime})\)
| If included in a project roadmap, project s′ must be implemented after project s has been finished |
Project-specific constraints | ||
Earliest beginning |
\(Earliest(s, y)\)
| If included in a project roadmap, the implementation of project s must start in period y at the latest |
Latest completion |
\(Latest(s,y)\)
| If included in a project roadmap, the implementation of project s must be finished in period y at the latest |
Mandatory project |
\(Mandatory(s)\)
| Project s must be included in each project roadmap |
Process-specific constraints | ||
Critical quality boundary |
\(QualMin(x,i,y)\)
| There is a critical quality boundary x, which process i must not fall short of in period y. This constraint applies particularly to support processes where the number of instances is invariant regarding quality |
Critical time boundary |
\(TimeMax(x,i,y)\)
| There is a critical time boundary x, which process i must not exceed of in period y. This constraint applies particularly to support processes where the number of instances is invariant regarding time |
Period-specific constraints | ||
Periodic process-level budget |
\(BudPro(x,i,y)\)
| In period y, there is a budget x regarding process i, which the investment outflows of the currently running process-level project must not exceed |
Periodic BPM-level budget |
\(BudBPM(x,y)\)
| In period y, there is a budget x, which the investment outflows of all currently running BPM-level projects must not exceed |
Overall periodic budget |
\(Budget(x,y)\)
| In period y, there is a budget x, which the investment outflows of all currently running projects must not exceed |
Number of projects |
\(NumProj(x,y)\)
| In period y, the number of all currently running projects must not exceed x (e.g., due a given number of project managers) |
5 Evaluation
5.1 Validation of the design specification (EVAL2)
5.1.1 Feature comparison and competing artifacts
Characteristics of our planning model | Bandara et al. (2015) | Darmani and Hanafizadeh (2013) | Forstner et al. (2014) | Linhart et al. (2015) | Ohlsson et al. (2014) | Shrestha et al. (2015) | |
---|---|---|---|---|---|---|---|
Summary | Supports the selection and scheduling of BPM- and process-level projects to develop organization’s BPM capability and improve individual processes in an integrated way. Projects are compiled into project roadmaps, which are assessed via their value contribution. Our planning model takes a multi-process, multi-project, and multi-period perspective | Supports the prioritization of process improvement projects with the business value scoring (BVS) model. The BVS is a multi-dimensional, multi-level, multi-stakeholder approach in assessment. It integrates the assessment results into a single indicator to capture the business value of improvement projects | Supports the selection of processes and best practice candidates for business process reengineering. The method aims to achieve lower risk and higher probability of success for process improvement projects | Supports decisions on how to determine the optimal increase/decrease of process capability levels. The model focuses on a single core process with multiple related capability areas, which include management and support processes. The concept of projects is captured implicitly via increases/decreases of capability levels | Supports the selection and scheduling of process improvement projects along established industrialization strategies, accounting for process characteristics that reflect how work is performed and organized. Projects are compiled into improvement roadmaps, which are assessed via their value contribution | Supports the categorization of business processes and the prioritization of improvement initiatives. Central artifacts are the process assessment heatmap and the process categorization map | Supports the selection of processes for improvement in IT service management. The process selection method balances business and IT service management objectives and builds on a decision support system to recommend which processes should be considered for improvement |
(O.1a) | Our planning model considers BPM- and process-level projects. These project types help develop operational capabilities (processes) and BPM as a particular dynamic capability | The focus is on an organization’s individual processes. BPM is not considered | The focus is on determining best practices for selected strategic processes. BPM is not considered | Projects directly affect capability areas. The core process is affected transitively. BPM is not considered as the model builds on process maturity models | Projects can affect process performance or multiple characteristics that reflect how work is performed and organized. BPM is not considered | Projects can affect processes in terms of differentiation, formality, and value network governance such as indicated in the process categorization map. BPM is not considered | The method yields a process selection matrix without a focus on projects. It focuses on single processes. BPM is not considered |
(O.1b) | Process-level projects affect individual processes. BPM-level projects affect all processes under investigation and/or facilitate the implementation of process-level projects in the future | Projects affect a distinct process. There are no projects that affect multiple processes | Projects affect a distinct process. There are no projects that affect multiple processes | Projects affect a distinct process. There are no projects that affect multiple processes | Projects affect a distinct process. There are no projects that affect multiple processes | Projects affect single processes. There are no projects that affect multiple processes | The focus is on individual processes. Projects are not considered |
(O.2) | Our planning model accounts for the time, quality, and cost dimensions of process performance as well as for the trade-offs among these dimensions. The cost perspective is analyzed in great detail according to the VBM paradigm | The BVS gives a high-level overview of how to calculate the business value of improvement projects. It includes the six dimensions reputation, clients, business processes, financial opportunity, regulation and compliance, and human resources | Process performance is not quantified via performance indicators. 19 factors and 44 indicators are defined to determine the perceived degree of change in relation to corporate strategy | Process performance is measured in terms of the risk-adjusted expected NPV in line with the VBM paradigm. No operational performance indicators are considered | Process performance is operationalized in terms of time, quality, and costs, catering for trade-offs. For each dimension, several performance indicators are used. The cost perspective is analyzed in great detail according to the VBM paradigm | Process performance is assessed qualitatively via different color regimes in the process assessment heatmap. It covers five perspectives (i.e., positioning, relating, preparing, implementing, proving), which relate to de Bruin and Rosemann (2007) BPM capability framework | A perceived service gap is derived based on the SERVQUAL model. To do so, business drivers in the context of IT services are rated qualitatively |
(O.3a) | Projects can affect the performance of individual or of all processes. They can also influence the investment outflows of future projects. Project effects on process performance can be absolute or relative | Each project is estimated based on the expected outcomes with respect to the dimensions mentioned above | Project effects are defined and evaluated based on the perceived degree of change, i.e., the difference between the weighted value of the conditions before and after project implementation | Effects of individual projects are measured via increases/decreases of capability levels | Projects can affect the performance of an individual processes and further characteristics that reflect how work is performed and organized. Thereby, projects can transitively (but not directly) affect the investment outflows of future projects | Projects can affect processes in terms of differentiation, formality, and value network governance such as indicated in the process categorization map | The focus is on processes, not on projects. Thus, a perceived service gap is determined. No performance effects of projects are included |
(O.3b) | Our planning model considers deterministic, scheduling, and intra- as well as inter-temporal interactions among projects | No interactions among projects are considered | No interactions among projects are considered | Interactions are considered implicitly via the lifecycle logic of process maturity models. There are strict predecessor/successor interactions regarding single capability areas. No further interactions are considered | The approach considers deterministic and scheduling interactions. Inter-temporal interactions are only modeled implicitly. Intra-temporal interactions are neglected due to the focus on an individual process | No interactions among projects are considered | No interactions among projects are considered |
(O.3c) | The planning model accounts for general interactions among projects and for BPM-specific interactions | No domain-specific constraints are considered | No domain-specific constraints are considered | No domain-specific constraints are considered | Domain-specific constraints are only modeled implicitly | No domain-specific constraints are considered | No domain-specific constraints are considered |
(O.4a) | Our planning model ranks project roadmaps according to their value contribution, measured in terms of the project roadmaps’ NPV | The BVS aggregates qualitative estimations to a single indicator reflecting the business value of an improvement project | Projects effects are determined using non-monetary measures. The model maximizes the weighted perceived degree of change using fuzzy numbers | The planning model considers the risk-adjusted expected NPV of increasing/decreasing capability levels | Project roadmaps are ranked in line with their value contribution, measured in terms of the project roadmaps’ NPV | Project effects are assessed qualitatively by positioning processes within the process categorization map. Qualitative effects are not integrated into a single numeric value | The process selection matrix builds on strategic business drivers and a service gap perception. No cash flows or other monetary performance indicators are included |
(O.4b) | Long-term effects are considered via the NPV. Different periods in time are considered explicitly due to inter-temporal interactions among projects | Long-term effects are not considered | Long-term effects are not considered | Long-term effects are considered via the NPV. There is no distinction between different periods in time | Long-term effects are considered via the NPV. Different periods in time are considered explicitly. Inter-temporal project interactions are only modeled implicitly | Long-term effects are not considered | Long-term effects are not considered |
(O.4c) | Our planning model accounts for the decision-makers’ risk attitude using a risk-adjusted interest rate | The decision-makers’ risk attitude is not covered explicitly | The approach aims at maximizing the weighted perceived degree of change considering the risk of differing scenarios and the change tolerance of the company | Risk is considered using the risk-adjusted expected NPV. Expected value and risk are considered explicitly via the certainty equivalent method | The decision-makers’ risk attitude is captured using a risk-adjusted interest rate | Risk is not considered | No risk attitude is included. However, in the determination of the business drivers with the balance score card, it is possible to weight dimensions differently |
5.1.2 Expert interviews with organizational stakeholders
PRODUCT | SERVICE | |
---|---|---|
Processes | For many support processes, it was impossible to unambiguously determine the number of instances because of the high level of abstraction used for process modeling | The number of instances of most processes is driven by quality and time. Some processes are only driven by quality, others only by time |
Process quality was consistently measured in terms of maturity levels | The performance indicators used to operationalize quality and time strongly depend on the process at hand | |
The company must continuously invest to keep up with its customers’ increasing quality expectations (degeneration effects) | ||
Projects | There are BPM-level projects without positive effects that must be implemented before any other BPM-level project | There are process-level projects (pioneer projects) without positive effects that must be implemented before any other process-level project related to the process in focus |
The implementation of a project takes between 3 months and 1 year. | ||
Process-level projects and BPM-level projects are often implemented simultaneously (e.g., process modeling training and process analysis projects) | The implementation of a project takes either one or two periods according to the company’s PPS cycle. Longer projects are not allowed | |
Only one process-level project can be implemented per process and period | ||
Interactions and constraints | There is a global budget based on which BPM-level projects are funded and several (department-) specific budgets are used to fund process-level projects | There are many regulatory projects per period. These projects must be finished in a predetermined period at the latest |
To comply with the industry’s quality management standards, selected support and all core processes must not violate predetermined quality boundaries. There is no such boundary for time | There are sequences of BPM-level and process-level projects that reach up to five periods in the future | |
There is one budget for process-level projects and another budget for BPM-level projects |
5.2 Prototype construction (EVAL3)
-
The robustness check calculates how strongly the value contribution of the optimal roadmap is affected by variations in the input parameters. To do so, the robustness check compares the value contributions of the 50,000 best project roadmaps with that of the optimal project roadmap. For each of these roadmaps, different value contributions are calculated by varying all project-related input parameters ceteris paribus in the range from −2 to +2 % (in 1 % steps). Finally, the robustness is reported as the fraction of parameter variations where the originally optimal roadmap still ranks higher than the competing 50,000 roadmaps.
-
The robustness analysis enables more specific analyses than the robustness check by varying a selected parameter of a single process, project, or from the general setting in a range between −10 and +10 % ceteris paribus. Besides the effects on the value contribution, the robustness analysis shows for the selected parameter setting which roadmaps have a higher value contribution than the originally optimal roadmap.
-
The project success analysis helps identify which parameters of a distinct project most strongly influence the value contribution of the entire roadmap. Therefore, all project parameters are modified in a given range.
-
The roadmap comparison compares two different roadmaps, a functionality that is based on the visualization provided by the general scenario analysis section (Fig. 4). For example, trends in quality and time or periodic cash flows can be compared automatically.
5.3 Validation of applicability and usefulness (EVAL4)
5.3.1 Case based on real-world data
Process | Demand logic | Price and billing | Constraints | Degeneration |
---|---|---|---|---|
(I) | Driven by quality and time | Pay per execution | – | – |
(II) | Constant | Fixed price per account |
\(QualMin\,\,(80\,\% ,II,all)\)
| Quality |
(III) | Constant | No price, as process is integrated in core process |
\(TimeMax\,\,(60\, { \hbox{min} },III,all)\)
| Time |
(IV) | Constant | No price, as process is integrated in core process |
\(QualMin\,\,(70\,\% ,IV,all)\)
| Quality |
Project | Description/effects | Affected process | Interactions/constraints |
---|---|---|---|
(1) |
Process standardization
| (I) |
\(PreSuc\,\,\left( {s_{1} ,s_{2} } \right)\)
|
Increases quality and reduces operating outflows | |||
(2) |
Process automation
| (I) |
\(PreSuc\,\,\left( {s_{1} ,s_{2} } \right)\)
|
Reduces time, increases quality, and reduces operating outflows | |||
(3) |
Implementation of new regulatory requirements
| (II) |
\(Latest\,\,\left( {s_{3} , 3} \right),\)
\(Madatory\,\,(s_{3} )\)
|
No effects on process performance | |||
(4) |
Improving the IT infrastructure
| (II) | – |
Reduces fixed outflows | |||
(5) |
Time improvement
| (III) | – |
Reduces time | |||
(6) |
Quality improvement
| (IV) | – |
Increases quality |
Project | Description/effects | Interactions/constraints |
---|---|---|
(7) |
Training in BPR methods
|
\(LocMutEx\,\,(s_{7} ,s_{8} )\)
|
Indirect effect on operational capabilities as such training allows implementing future process-level projects more easily | ||
(8) |
Development of a process performance measurement system
|
\(LocMutEx\,\,(s_{7} ,s_{8} )\)
|
Direct effects on operational capabilities reduce operating outflows of all processes under investigation | ||
(9) |
Training in Six Sigma
| – |
Combination of direct and indirect effects. Indirect effects affect future process-level projects, direct effects reduce operating outflows of all processes |
Optimal project roadmap/value contribution | Description | ||
---|---|---|---|
(A) General case | Opt. | Project roadmap: \((\{ 1,\,9\} ,\,\{ 2,\,4\} ,\,\{ 3\} \,,\{ 6\} ,\{ \} )\)
NPV: 2.50 million EUR Robustness: 100 % | General case About 240,000 project roadmaps meet the interactions and constraints The interactions and constraints reduce the potential project roadmaps as follows:
\(LocMutEx\,\,(s_{7} ,s_{8} )\): 180,000
\(PreSuc\,\,(s_{1} ,s_{2} )\): 1,290,000
\(Latest\,\,(s_{3} , 3) {\text{and }}Mandatory\,\,(s_{3} )\): 650,000
\(Budget\,\,(750,000,ALL)\): 150,000
\(QualMin\,\,(70\,\% ,IV,ALL)\): 190,000 |
Pess. | Project roadmap: \((\{ 1,\,9\} ,\,\{ 2\} ,\,\{ 3\} \,,\{ 6\} ,\{ \} )\)
NPV: 1.20 million EUR Robustness: 90.8 % | ||
(B) Overall budget | Opt. | Project roadmap: \((\{ 1,\,7\} ,\,\{ 2\} ,\,\{ 3\} \,,\{ 6\} ,\{ \} )\)
NPV: 2.23 million EUR Robustness: 98.2 % | Overall budget is reduced by one-third About 40,000 project roadmaps meet the interactions and constraints About 480,000 project roadmaps violate the constraint: \(Budget\,\,(500,000,ALL)\)
|
Pess. | Project roadmap: \((\{ 4,\,9\} ,\,\{ 1\} ,\,\{ 3\} \,,\{ 6\} ,\{ \} )\)
NPV: 1.09 million EUR Robustness: 84.1 % | ||
(C) Latest finish | Opt. | Project roadmap: \((\{ 3,\,9\} ,\,\{ 1,\,4\} ,\,\{ 2\} \,,\{ 6\} ,\{ \} )\)
NPV: 1.92 million EUR Robustness: 100 % | Project (3) must be already finished period 1 About 80,000 project roadmaps meet the interactions and constraints About 1,000,000 project roadmaps violate the constraints \(Latest\,\,(s_{3} , 1)\)and \(Mandatory\,\,(s_{3} )\)
|
Pess. | Project roadmap: \((\{ 3,\,9\} ,\,\{ 1\} ,\,\{ \} \,,\{ 6\} ,\{ \} )\)
NPV: 1.02 million EUR Robustness: 93.4 % | ||
(D) Critical quality boundary | Opt. | Project roadmap: \((\{ 1,\,9\} ,\,\{ 2,\,6\} ,\,\{ 3\} \,,\{ \} ,\{ \} )\)
NPV: 2.37 million EUR Robustness: 100 % | Minimum quality of process (IV) is increased About 120,000 project roadmaps meet the interactions and constraints About 410,000 project roadmaps violate the constraint \(QualMin\,\,(80\,{\text{\%}},{\text{IV}},ALL)\)
|
Pess. | Project roadmap: \((\{ 1,\,9\} ,\,\{ 2,\,6\} ,\,\{ 3\} ,\,\{ \} ,\,\{ \} )\)
NPV: 1.19 million EUR Robustness: 90.8 % |
5.3.2 Discussion Against Evaluation Criteria
Criterion | Characteristics of the planning model and the software prototype |
---|---|
Applicability (model and instantiation) | The case based on real-world data, which we presented in Sect. 5.3.1, illustrated that the planning model is applicable in naturalistic settings. As the planning model’s calculation logic is complex and the number of possible project roadmaps heavily grows with the number of considered processes, projects, and planning periods, the planning model could not be applied without the software prototype. The expert interviews revealed that the planning model particularly fits organizations that aspire a well-developed BPM capability and are willing to invest accordingly. For instance, the planning model is oversized for PRODUCT, while it perfectly fits SERVICE. Organizations that plan to apply the planning model also require some areas of their BPM capability to be developed beforehand, including process metrics and enterprise process architecture Another issue with impact on applicability is that the planning model requires collecting and estimating input data regarding processes, projects, interactions, and constraints. According to the interviews, SERVICE disposed of most input data and only had to estimate project effects. PRODUCT’s experts indicated that the required data can also be collected in non-automated environments. To cope with estimation inaccuracies, which are inevitable in naturalistic settings, the software prototype implements robustness check and analysis functionality, as discussed in Sect. 5.2. Applying the planning model should not be an one-off initiative. Rather, the planning model should be applied repeatedly. A knowledge base should be built to institutionalize data collection routines and collect best practices |
Impact on the artifact environment and users (model and instantiation) | The planning model impacts how users think about how to develop their organization’s BPM capability and to improve individual processes in an integrated manner. On the one hand, the planning model’s formal design specification provides insights into central constructs and mechanisms of integrated BPM capability development and process improvement. On the other, the prototype’s visualization and analysis functionality helps users understand the situation and possibilities for action in their organizations. The experts from SERVICE and PRODUCT agreed that the planning model enhances the organizations’ process decision-making capabilities |
Fidelity with the real-world phenomena (model) | Based on the covered process and project types, interactions, and constraints as well as performance dimensions, the planning model can handle many different constellations that occur in naturalistic settings. This has been confirmed by the experts from PRODUCT and SERVICE |
Internal and external consistency (model) | The planning model is internally consistent as it has been designed deductively and as its components are modular such that side effects cannot occur. Further, the planning model’s design specification is available in terms of mathematical formulae, a property that facilitates checking internal consistency. As for external consistency, the planning model does not contradict accepted knowledge from other disciplines, such as BPM, PPS, or VBM. Rather, the planning model was built based on knowledge from these disciplines as justificatory knowledge. These disciplines also served as foundation for deriving our design objectives |
Effectiveness and efficiency (instantiation) | The experts we interviewed, particularly those from SERVICE based on whose data we applied the planning model, agreed that the software prototype can be effectively used to plan the development of an organization’s BPM capability and the improvement of individual processes in an integrated manner. As for efficiency, we conducted performance tests with the prototype on regular work stations such as used in business environments. The prototype efficiently processes industry-scale problems as long as the number of planning periods, which is the most influential driver of problem complexity, is not too large. As the number of planning periods is rather small in naturalistic settings (i.e., between 2 and 8 according to our experiences), this limitation does not heavily restrict the prototype’s efficiency. For example, the case presented in Sect. 5.3.1 required 26 s to determine admissible project roadmaps and to calculate the corresponding value contributions. The robustness check of the optimal project roadmap took about 3 min, being limited to the best 50,000 project roadmaps. Another driver of the problem complexity is the amount of available projects, which increases the amount of admissible project roadmaps over-proportionally. To reduce this complexity, it is important to include only those projects that already passed the first three stages of Archer and Ghasemzadeh’s (1999) PPS process and to consider all the known constraints in the prototype, as these considerably reduce the amount of admissible project roadmaps |