Evaluation Methods in Process-Aware Information Systems Research with a Perspective on Human Orientation

We perceive PAIS as a type of information system “that manages and executes operational processes involving people, applications, and/or information sources on the basis of process models” (Dumas et al. 2005). PAIS in practice have been considered as drivers for companies’ benefits such as (cf. Vanderfeesten and Reijers 2005): the specification, execution and control of business processes, easy coordination of work, higher quality of services, efficient execution of work, and flexible processes. The support of their processes through PAIS pervades nearly any application area or industry such as manufacturing (Schulte et al. 2012),¹ health care (Lenz and Reichert 2007), and finance (cf. Rabobank processes of BPI’14 challenge, http://​www.​win.​tue.​nl/​bpi/​2014/​challenge). Although PAIS are meant to assist human workers (cf. Klein et al. 2000), such systems have received criticism as well (cf. Vanderfeesten and Reijers 2005): particularly work psychologists and potential (future) users fear that workflow systems could lead to mechanical, monotonous, and rigid office work (Kabicher-Fuchs et al. 2013) and to the perception that system users are exchangeable resources. Several papers have addressed different aspects of PAIS that seem to be particularly relevant for the human orientation of such systems and for successful human computer interaction.

In the following, it is claimed that human orientation in PAIS constitutes an important challenge and is of increasing interest for the research community. This claim is supported by the broader literature on software systems and information systems (IS) (Gediga and Hamborg 2001; Irani 2002), by PAIS specific literature (Kabicher-Fuchs et al. 2012, 2013; Kabicher-Fuchs and Rinderle-Ma 2012), and by the recent advent of related topics in IS conferences.² Starting from this, a further claim is that research on human orientation requires special evaluation methods and artifacts. As stated in Song and Letch (2012) “[t]he study of human factors in evaluation also consists with the shift from traditional evaluation to understanding-driven stream”. Another study (Serafeimidis and Smithson 2003) claims that “[t]he organizational and subjective nature of IS evaluation brings into the foreground the human actor, a stakeholder, of an evaluation exercise”. Following both claims, the motivation for the study at hand is to investigate evaluation methods and artifacts that have been used in PAIS with specific focus on human orientation. We understand human orientation in PAIS in a rather general sense, i.e., as an umbrella term for humans playing a role in a PAIS (cf. Vanderfeesten and Reijers 2005; Kabicher-Fuchs et al. 2012).

In scientific literature, several analyses of research in the context of PAIS, and more generally of IS, have been conducted (see, e.g., Glass et al. 2002; Hevner et al. 2004; Ramesh et al. 2004; Wilde and Hess 2007; Houy et al. 2010). Such analyses create a firm foundation for advancing knowledge and for an informed understanding of existing research and gaps where further work and research is needed. However, according to the analysis in Song and Letch (2012), in the most recent time period considered in the study (i.e., 2006–2010), only one construct/measurement validation was conducted in IS. Specifically, a systematic analysis of evaluation methods as applied in PAIS in the context of human orientation is entirely missing.

In this work, we investigate which evaluation methods and artifacts have been used in PAIS research with focus on human orientation in general and graphical presentations (visualization) but also on security topics, since security can fail due to misunderstandings, wrong communication or false assumptions between users and systems. In order to illustrate the terms ‘artifact’ and ‘evaluation method’ in this context, take the paper Visual change tracking for business process models (Kabicher et al. 2011) as an example. Here, artifacts to be evaluated are graphical descriptions of process models, and questionnaires are used as evaluation method for this empirical study. Research in the IS field is characterized by the research paradigms ‘behavioral science’ and ‘design science’ (cf. Houy et al. 2010). Whereas Houy et al. (2010) analyzed empirical research following the behavioral and design science research paradigm in business process management (BPM), we have in our review concentrated on studies which investigated human oriented factors in the field of PAIS by following a design science research approach.

This study reviews scientific contributions that focus on human orientation in general (e.g., user, function, and task allocation), visualization (e.g., graphical representation of process models, and task lists), and security (e.g., access control, and privileges) in PAIS. We categorize the identified PAIS artifacts into theoretical and executable artifacts and classify the evaluation methods according to the methodical framework presented by Gediga and Hamborg (2001) into the categories Behavior-based, Opinion-based, and Predictive evaluation methods. The paper centers on the three following research questions: Which evaluation methods have been used so far to examine PAIS artifacts that are in direct contact with users? Which evaluation methods are of future interest? Can these evaluation methods be classified into the categories mentioned above?

We proceed based on the following methodology. In a first step, a literature review is conducted in order to identify which artifacts and evaluation methods have so far been applied in scientific contributions addressing human orientation of PAIS. Moreover, the relationship between the identified artifacts and evaluation methods is analyzed. The literature review yields an overview of the ‘as-is’ state concerning evaluation methods used with respect to human orientation in PAIS.

After the literature review, an expert survey shall provide insights into experts’ awareness concerning artifacts, evaluation methods, and their relationships, that have been used so far in research on human oriented aspects of PAIS. Further on, the expert survey shall also highlight the experts’ forecasts concerning artifacts, evaluation methods, and their relationships, that will be of future interest in the field of human orientation of PAIS.

Finally, a focus group supplements the study with further experts’ opinions concerning artifacts, evaluation methods, and their relationships used so far and with future potential to support and investigate human orientation of PAIS.

Although we gained valuable insights from the expert survey, we also identified misunderstandings about the meanings of questions or contradictory responses. Therefore, we decided to conduct a focus group session (cf. Stewart et al. 2007) which is a good method to analyze the results from different points of view in a short period of time. In particular, the focus group session gave us the possibility to discuss and verify the results, from the literature review (RQ1) and, from the expert survey.

Both, the expert survey and the focus group are intended to shed light on the ‘to-be’ state of evaluation methods used with respect to human orientation in PAIS. We decided to conduct a focus group session additionally to the expert survey to discuss and verify the results of the expert survey, since we identified misunderstandings concerning the meanings of questions or contradictory responses within the expert survey. Of particular interest is the comparison of ‘as-is’ and ‘to-be’ state as it might reveal blank spots in the evaluation landscape. The objectives of the study are threefold. First of all, it identifies and categorizes artifacts that have been developed in the context of human orientation of PAIS. In addition, evaluation methods are identified which have been used to evaluate artifacts that have been developed in the context of human orientation of PAIS. These evaluation methods are also evaluated according to the theoretical methodical framework as proposed by Gediga and Hamborg (2001).

This work presents an analysis of artifacts and evaluation methods which have been used in PAIS research with a focus on aspects that are particularly relevant for humans working with the system such as PAIS users and employees. Based on the findings of the literature review, expert survey and the focus group, we highlight ten widely used evaluation methods and interdisciplinary evaluation methods (e.g., combination human orientation and security, human orientation and visualization, as well as security and visualization). Furthermore, we discuss the trend towards human orientation and give recommendations for using evaluation methods in PAIS as well as an overview of evaluation methods and their use in human orientation, security, and visualization.

This article is structured as follows: Sect. 2 introduces the basic definition and concepts followed by an overview on related work and the classification applied in the remainder of the paper. Section 3 describes the overarching methodology used in the paper. This is followed by a literature review in Sect. 4, an expert survey in Sect. 5, and a focus group in Sect. 6. The results of the literature review, of the expert survey, and of the focus group are summarized in Sect. 7. The discussion provided in Sect. 8 includes ten widely used evaluation methods, interdisciplinary evaluation methods, recommendations, and comments on lessons learned. It furthermore discusses limitations of the paper. Section 9 concludes the paper.

Figure 1 illustrates the main concepts Artifact, Type of Investigation, Strategy, and Evaluation Method as used in the PAIS research papers analyzed in this article (presented in detail in the literature review in Sect. 4). The type of investigation determines the strategy and artifact selected for in the evaluation. The strategy and artifact, in turn, determine the evaluation method.

Which kind of investigation is chosen (e.g., case study or user evaluation) depends strongly on the purpose and goal of the investigation itself, for instance to determine usability or security aspects (e.g., if the goal is to observe how users will work with the system, user evaluation is a promising way). According to the type of investigation, different evaluation methods can be used (e.g., thinking aloud) and can be applied for all items (i.e., artifacts) that are produced during the development process. For example, if the purpose of the user evaluation is to detect how users will interact with the interface of a system – in this case the interface is the artifact –, evaluation methods, such as observation or thinking aloud, are effective methods to obtain information about their behavior. In the computer science domain, it is necessary to distinguish between executable artifacts (e.g., (high-fidelity) prototype) and theoretical artifacts (i.e., non-executable artifacts) such as designs or concepts. Depending on the type of artifacts, different evaluation methods are more or less adequate. For example, methods for the analysis of performances are only relevant for executable artifacts. Based on Gediga and Hamborg (2001), the evaluation methods can be classified according to their strategic direction (see Strategy in Fig. 1): behavior-based, opinion-based, and predictive. The decision for this classification is described in Sect. 2.3. For example a behavior-based evaluation method concentrates on the analysis of users’ activities during their interaction with the investigated artifacts (e.g., observation methods) whereas an opinion-based evaluation method identifies users’ views on the investigated artifacts (e.g., interviews). The predictive evaluation method has its focus on the investigation of the context, feasibility as well as performances of the artifacts (e.g., functionality tests). For instance, behavior-based evaluation methods are promising for the above-mentioned example of evaluating the interface of a system in order to obtain information how users will interact with the system.

The analysis of the papers for our literature review was particularly focused on Artifact, Strategy, and Evaluation Method. In contrast to Type of Investigation, which can include a combination of several different evaluation methods, Evaluation Method was considered as an elementary step to evaluate PAIS artifacts. Although evaluation methods have repeatedly been summarized in Information Systems, Business Informatics, or Software Engineering, there seems to be a lack of contributions illustrating research and use of evaluation methods in the context of PAIS contributions. The goal of this article is to identify these elementary evaluation methods and to categorize them according to their strategic direction.

As stated in the introduction, a scientific analysis of evaluation methods applied in PAIS with respect to human orientation has been lacking until now. In the past years, PAIS research has centered on artifact-related evaluation methods. For example, evaluation methods for process mining and data mining are investigated in Rozinat et al. (2007). Often evaluation methods are reviewed as part of the development of evaluation frameworks with focus on different aspects of PAIS. For instance, an evaluation framework with the goal to assess a business process modeling tool in every phase of the development process is displayed in Effinger et al. (2011). Evaluation is used as a criterion for the construction of business process reference models in Fettke et al. (2006). Lastly, a literature review by Leitner and Rinderle-Ma (2014) states that research evaluation is a challenge in the context of security in PAIS.

Furthermore, related literature can be found in the Software Engineering domain. A variety of assessments of evaluation strategies exist in Software Engineering (e.g., Gediga and Hamborg 2001; Glass et al. 2002) and for specific domains such as adaptive systems (cf. van Velsen et al. 2008) or visualization (cf. Shneiderman and Plaisant 2006; Kriglstein et al. 2014; Pohl 2012). Furthermore, Moody (2003) describes an evaluation framework with a set of metrics for assessing the quality of data models. However, none of these models center specifically on PAIS.

As stated in the previous section, evaluation methods in PAIS have only been selectively looked into so far. A systematic analysis is missing. In order to investigate evaluation methods in PAIS, we examined classifications from related domains, specifically from Software Engineering (e.g., Gediga and Hamborg 2001; Glass et al. 2002). For example, Glass et al. (2002) classify evaluative approaches into four categories: Deductive, Interpretive, Critical, and Other. However, this approach does not meet our requirements as the Other category is not distinctive and cannot be easily identified. Gediga and Hamborg (2001) suggest categories based on the purpose, i.e., Behavior-based, Opinion-based, and Predictive. As this categorization is well suited when evaluating user behavior, authorizations or human skills, we adopted it in this paper. In addition, it does not include a category “Other” that remains undefined. Instead, it provides a categorization that is highly distinctive. The three categories of evaluation methods are described in the following (adapted from Gediga and Hamborg 2001):This categorization centers on the purpose of evaluation methods, i.e., for which aim they are applied, and can be assigned to more than one category. For example, if the aim of using a focus group is to ask a group of people about their perceptions and opinions, then this method is classified as belonging to the category Opinion-based. However, if the focus group method is used to analyze the investigated artifacts in order to evaluate their utility and feasibility for a group of people, then the category is Predictive. In the next sections, we will utilize this categorization in order to classify the investigated evaluation methods.

This section describes the procedure of the systematic literature review based on guidelines defined in Kitchenham (2004) and Kitchenham and Charters (2007). The review can be divided into four phases, i.e., literature search, literature selection, data extraction and synthesis, and categorization, all of which are described in the following.

Table 2 displays all evaluation methods and artifacts concerning human orientation in general, security, and visualization in PAIS. As can be seen from the table, each evaluation method is categorized in Behavior-based, Opinion-based, or Predictive. A complete list of the literature found regarding the type of artifacts and evaluation methods is given in the Appendix (available online via http://​link.​springer.​com/​).

In addition, Fig. 3 shows a graph visualizing the connections (edges) between the different categories of evaluation methods (orange nodes) and the investigated artifact types (green nodes) based on the findings of the literature review. Size and numbers of nodes reflect the number of found artifacts/evaluation methods. The thickness and the number of edges displays the number of connections between evaluation methods and artifacts. It must be pointed out that for some artifacts more than one method was applied. For example, Fig. 3 shows that different evaluation methods from the category Predictive were used as combination for the evaluation of the same artifact (e.g., for the evaluation of a prototype the application and performance measures were used as predictive evaluation methods). Table 3 presents the number of the different evaluation methods and the investigated artifacts types for human orientation in general, security, and visualization respectively in detail.

Most of the selected literature contributions applied evaluation methods from the category Predictive (cf. Fig. 3). Especially used were evaluation methods from this category which do not consider users in their evaluation. For example, it was observed that implementation as evaluation method was applied in order to verify the theoretical artifacts (found in 21 of 59 publications in the area of human orientation, security: 37/67, and visualization: 112/151). Furthermore, applications include cases, examples, scenarios, and use cases (found in 36/59 papers of human orientation, security: 54/67, and visualization: 79/151) and were often used for the inspection of artifacts in order to ensure that the requirements are fulfilled.

In comparison with the predictive evaluation methods, behavior-based and opinion-based evaluation methods were applied sparingly (human orientation: 28/59 publications, security: 6/67, and visualization: 27/151). However, a closer look shows that the number of literature contributions which include evaluation studies with users has increased in the last five years (cf. Fig. 4).

The literature review provided us with a comprehensive overview of researchers who were working in the field of human orientation in general, security, and visualization in PAIS. We contacted this pool of researchers by email and asked them to take part in a survey concerning evaluation methods. They were selected because they are actively publishing in the field and together cover all three areas of human orientation, security, and visualization. In total, we received a positive response from 14 researchers (4 researchers with expertise in visualization, 6 researchers with expertise in security, and 4 researches with expertise in human orientation in PAIS). For each of these 14 participants, we created an account to provide them with access to the survey.

The expert survey consisted of two rounds. In the first round, the participants were asked to define their level of knowledge, to name at least three typical artifacts for human orientation, security, or visualization in PAIS, to specify at least two typical evaluation methods for these artifacts, and to find prospective evaluation methods. For the second round, we used the defined artifacts and evaluation methods from the first round and categorized the evaluation methods with the previously specified classification: Behavior-based, Opinion-based, and Predictive. In the second round, 12 participants rated the classification of all evaluation methods found in the first round, specified the relevance of evaluation methods with regard to theoretical and executable artifacts, and defined missing evaluation methods that were not previously mentioned.

RQ1: Table 4 displays examples of typical evaluation methods and artifacts found in the first round of the expert survey. Figure 5 shows the connection between the typical evaluation methods with regard to the theoretical and executable artifacts. It can be seen from the figure that most of the named typical evaluation methods for the theoretical artifacts belong to the category Predictive. This corresponds with the findings from the literature review. In contrast to the literature review, the opinion-based evaluation methods were often listed for theoretical artifacts. On the other hand, the experts named evaluation methods from the category Predictive and from Behavior-based for the executable artifacts. Security experts mentioned evaluation methods from the category Predictive most. Experts in human orientation named methods from the category Opinion-based most, and evaluation methods from the category Behavior-based were most stated by the visualization experts.

RQ2: Table 5 shows future evaluation methods identified by the participants. As can be seen from Table 5, prospective evaluation methods that were not mentioned as typical methods are listed in boldface (cf. Table 4). Eye tracking, questionnaire, and walkthrough were mentioned methods by all participants; each participant could mention several different evaluation methods.

Similar to the typical evaluation methods, the experts frequently mentioned evaluation methods from the category Predictive for theoretical artifacts, and behavior-based and predictive evaluation methods for executable artifacts. Whereas for security experts the most frequently mentioned evaluation methods were also from the category Predictive, experts in human orientation noted more behavior-based evaluation methods.

RQ3: In the second round, the participants supported our categorization of behavioral-based and opinion-based evaluation methods. Also in the case of predictive evaluation methods, the majority supported the classification. The focus group was the only method that was specified as opinion-based by six participants. The participants stated that focus group can include elements of the category Predictive but also of the category Opinion-based, because it can combine an inspection with a group discussion with focus on different viewpoints and opinions about artifacts. The definitions given by the participants in the first round of the expert survey already showed that their opinions were divided in regard to the focus group: “sessions with groups of users to collect information about requirements, current usage, current problems, etc.” and “a focus group is built in order to evaluate the proposed knowledge map against the background of underlying human-oriented processes”, while others define it as ”[...] opinions on visual representations” and state that “questions are asked in an interactive group setting where participants are free to talk with other group members”.

The results let us conclude that the participants were able to apply the used classification to the collected evaluation methods.

Since we decided on a face-to-face focus group (which allows the exchange of visual and nonverbal cues to enhance communication), we were restricted to inviting people from the local area. Further criteria for recruiting the participants were (a) that they were familiar with the topic, and (b) that they had the time and interest to attend a focus group session with a duration of about one hour. In addition, we selected experts who had not taken part in the expert survey in order to avoid that participants felt the need to defend the results that we gained from the expert survey. Based on the literature review and expert survey results we observed a trend toward Behavior-based and Opinion-based evaluation methods. Since these are well-known methods in Human Computer Interaction, it was important for us to have at least one participant with expertise in this field to identify further methods known in Human Computer Interaction but which have not been adopted in PAIS so far. A further criterion was that the participants covered the key concerns of business process management defined in van der Aalst (2012) and had a comprehensive knowledge of evaluation methods in computer science. Finally, we selected four senior researchers with expertise in human orientation, security, and/or visualization in the context of PAIS. One of these experts also had additional expertise in Human Computer Interaction.

The focus group was conducted in a university meeting room and took about one hour. The session was guided by two skilled moderators and one note taker who helped the moderators. The focus group session consisted of two steps: First, the participants filled out a questionnaire in which they had to (1) define their level of knowledge, (2) grade the relevance of typical and prospective evaluation methods for theoretical and executable artifacts in PAIS found in the first round of the expert survey, and (3) find future evaluation methods. In the second step, the participants discussed the relevance of evaluation methods and possible future directions for theoretical and executable artifacts.

RQ3: The participants stated that for rating the relevance additional information, such as which artifacts the methods relate to or a category scheme (e.g., theoretical, technical, and human related evaluation), is missing. A classification of methods would further support the relevance rating of evaluation methods (in the questionnaire). We proposed our category scheme (Behavior-based, Opinion-based, and Predictive), and the participants agreed to it.

In this article, we investigated and examined artifacts as well as evaluation methods in the area of human orientation in general, security, and visualization in PAIS. A complete list is shown in Table 6 in the previous section. This list can be used as reference to evaluate theoretical and executable artifacts. In the following, we will describe four results derived from the literature review, expert survey, and focus group.

Based on the results, we propose the following three general recommendations. These recommendations provide researchers with an overview of aspects which they should consider in their investigations.

We observed that experts in the expert survey and in the focus group referred to evaluation methods on different abstraction levels (e.g., review versus model checking), to different theories, and types of evaluation (e.g., grounded theory or usability evaluation). A possible reason is that often it is difficult to draw a clear boundary between the different granularity of evaluation methods.

Furthermore, multiple definitions of evaluation methods exist in human orientation, security, and visualization in PAIS. For example, case studies were mentioned as evaluation methods by experts in the expert survey and in the focus group. However, according to Yin (2003), a case study is a strategy and includes methods like interviews and participant observation for data collection. Hence, a framework specifically for PAIS that describes different evaluation strategies including artifacts and evaluation methods would be helpful as a common basis.

A reason for the different definitions and interpretations between experts could be that the participants came from different domains (e.g., human orientation, security and visualization in PAIS). Nevertheless, this diversity of experts had the benefit of collecting typical evaluation methods for PAIS from different viewpoints (e.g., systems engineering methods, human computer interaction methods, and social scientific methods). Moreover, the usage of interdisciplinary evaluation methods was perceived as gaining importance for future research.

However, not only the definitions of evaluation methods varied but also the meaning and its context differed between the fields human orientation, security, and visualization. For example, in process mining (cf. van der Aalst 2011) a log file analysis typically consists of the examination of (process) event logs which represent process execution histories. However, in the Human Computer Interaction domain the users’ activities (e.g., mouse clicks) are logged. Therefore, a taxonomy to provide a common understanding and contextual meaning would support the understanding of common practices and should be combined with the above mentioned framework for the different evaluation strategies.

In the literature review, the classification of the publications was performed based on (1) the content of the publication, and (2) the textual definition. By analyzing the content, we ensured that the misuse of definitions (e.g., a use case instead of a scenario) would not alter our results. During the review, we discovered that the studies are often not fully described in publications. For this reason, we skimmed the text headings and captions of figures to identify artifacts and evaluation methods used in the publications. Often, we had to fully read the paper. Furthermore, we assessed the main idea behind each publication and identified artifacts and evaluation methods based on the course of actions.

Furthermore, the assignment of the artifacts to be theoretical or executable artifacts as, e.g., shown in Table 2 was vividly discussed among the authors. We noticed that in our study the experts specified an algorithm as theoretical and as executable artifact. Hence, in Table 2, an algorithm is assigned to a theoretical and an executable artifact. Another example is the prototype. Most publications use various names for this such as prototype, (prototypical) implementation or proof of concept. In visualization, a prototype may also refer to a paper mockup as a theoretical artifact. However, in human orientation and security, a prototype always refers to an executable artifact. We acknowledge that these ambiguities exist in research. Here, we dealt with this challenge by carefully reading each publication to determine which method was actually used.

In this study, it was not possible to identify which of the evaluation methods named by the experts are more or less relevant. One reason could be that the choice of evaluation method depends strongly on which artifact is going to evaluated and on the aim of the evaluation. For example, if the aim of the evaluation is to find out how users interact with the system, the information about the time a user needs to complete predefined tasks might not give enough insight into user behavior. But a combination of logging users’ activities with the system, observation, and thinking aloud may be more suitable to assess users’ behavior. This means that the usefulness and applicability of each evaluation method depends on the investigated artifact.

Since the number of options and applications was extremely large it was not possible to generalize the results. However, an evaluation of the different evaluation methods in regard to their specific application (i.e., theoretical and executable artifacts) in different situations is essential for research in PAIS and thus subject to future work. In order to minimize the different options, a further possible direction for future work is to concentrate on a single category of the evaluation methods and compare these methods by means of experiments.

The aim of this paper was to assess how research conducts the evaluation of theoretical and executable artifacts for human orientation in general, in security, and in visualization in PAIS. For this purpose, we provided a list of these artifacts and which evaluation methods are typically used to conduct an evaluation. This collection of artifacts and evaluation methods may serve as a basis for researchers and practitioners who wish to investigate, e.g., theoretical artifacts when selecting typical evaluation methods. Furthermore, researchers and practitioners may use this collection to discover unfamiliar, interdisciplinary evaluation methods. For example in the area of security, research has neglected the evaluation of security modeling extensions (cf. Leitner et al. 2013). This paper may call attention to alternatives of how to evaluate users’ preferences or understanding (e.g., observation and interview as evaluation methods).

In addition, the classification of evaluation methods can be used as a guideline for categorizing the evaluation methods researchers utilize. This paper provides an allocation of artifacts and evaluation methods in PAIS. This may serve as a basis and can be extended by adding new evaluation methods and artifacts that are not listed in this paper.

Furthermore, practitioners may use this paper for reassessing evaluation methods and to become acquainted with unfamiliar evaluation methods. This might lead to an improvement of software, for example, by using new evaluation methods in the software development.