Modeling difficulties in creating conceptual data models

Conceptual modeling is a recurring activity during software engineering pursuing purposeful reconstructions of statements about a domain of discourse using a modeling language, e. g., for data or process modeling [18, 29, 33, 34, 37, 80]. Thence, conceptual modeling is a learning task faced by most students of computer science, software engineering and related programs as it is mandated by curricula standards, e. g., by the joint standard curriculum for Information Systems of the Association for Computing Machinery (ACM) and the Association for Information Systems (AIS) [2]. As an activity, conceptual modeling—e. g., when constructing a data model as entity–relationship diagram [23]—involves an intricate array of cognitive processes and performed actions including goal setting, abstracting, conceptualizing, associating and contextualizing, interpreting and sense-making, evaluating and judging, anticipating and envisioning and thinking ahead, drawing and visualizing and, in group settings, communicating, discussing and agreeing [57, 58]—marking the actual act and process of conceptual modeling. Creating a conceptual model is, hence, construed as a complex task involving codified and tacit knowledge [54, 74], a task that requires mastering theoretical foundations, modeling languages, modeling methods and modeling tools, applying them to practical problems and, along the way, critically thinking and reflecting upon the application domain in terms of its technical languages [35, 62].

Despite its relevance, the process of conceptual modeling has for long received limited attention in conceptual modeling research, with human factors and cognitive aspects having received only little attention [73]. Only recently, processes of conceptual modeling have seen increasing interest from researchers (e. g., [13, 25, 39, 52, 64, 84]). How conceptual data modeling is performed by modelers, how modeling processes proceed, which modeling difficulties modelers encounter and why and how to overcome these difficulties has been subject to studies on the cognitive processes and performed actions constituting conceptual modeling (e. g., [6, 11, 21, 68, 74]). However, at large, surprisingly little is known about the actual act of conceptual data modeling, about the reasoning of modelers and their deliberations (e. g., about modeling decisions), and whether different (idealized) types of modelers can be identified, e. g., by identifying patterns of modelers’ modeling difficulties, and whether these modeler types benefit from modeling assistance tailored to overcome their specific difficulties.

In the presented research, we integrate complementary modes of observation to study modeling difficulties in sixteen modeling processes of learning modelers following a mixed methods research design [26, 75]. We pursue the research objective of identifying and classifying modeling difficulties these modelers face while constructing a conceptual data model based on a natural language task description using a modeling software tool. The present research focuses on a learning context in which learners of data modeling are expected to model a universe of discourse as described in a textual description of a modeling tasks—a common learning scenario, e.g., in higher education courses on data modeling. Hence, the main focus is on modelers’ capability of conceptualizing a domain—the modeling dialogue between modelers and domain experts is excluded from the present study (cf. e.g., [38, 39]). In our pursuit of the research objective, we operate on the basic assumption that modelers’ individual modeling processes demand study from multiple complementary perspectives to account for the richness and complexity of the task and individual process of conceptual modeling—following Berger and Luckmann’s inspiring insight that “the object of thought becomes progressively clearer with this accumulation of different perspectives on it” [14, p. 10]. We use the concept of cognitive breakdown [11, 47] to identify modeling difficulties in verbal protocols (think aloud protocols, see [30]) and complement difficulty identification by visually inspecting recordings of modeler–tool interactions as well as video recordings of individuals’ modeling processes. We then complement difficulty identification by surveying the individuals about performing the modeling task. Here, we approach difficulty identification in two studies: An exploratory study with eight non-experienced modelers identifying five types of modeling difficulties relating to different aspects of constructing conceptual data models, i.e., entity types, relationship types, attributes and cardinalities (reported in [57]), and a follow-up study with eight medium-experienced modelers extending the findings by three further types of modeling difficulties.

Viewed as a process, conceptual modeling is a cognitively challenging undertaking bound to natural language and to (an) artificial modeling language. It is for this reason that we follow a mixed methods research design with multimodal observations of individual modeling processes to account for the intricacies of the process of conceptual modeling: Specifically, analyzing think aloud protocols has shown promising results for understanding cognitive processes of subjects working on problem-solving tasks in general and on modeling tasks in particular [6, 68], and we consider verbalization of thoughts as the best available means of expression for achieving insights into modelers’ reasoning as our spoken language provides a rich and flexible tool to express our thinking. However, to ask subjects to think aloud is a second-best approach, warranted only because it is not possible to directly access and capture cognitive processes and, thus, modeler reasoning while modeling. Modelers may have difficulties verbalizing their reasoning while modeling [15] on principle accounts (because verbalizing own thoughts can be difficult) or on modeling-related accounts (e. g., because of the difficulty of finding the right words to express oneself). Nonetheless, among all possible alternative modes of observation, think aloud verbalization promises the richest insight into non-directly observable cognitive processes of individual modeling processes. However, modeling difficulties will not always be observable from verbal protocols alone but from interactions of modelers with the software tool, with pen and paper or simply from modeler movements, e. g., erratic changes between looking at the graphical editor on screen and the modeling task provided on paper. Hence, multimodality of observations is assumed to provide a more complete picture of the phenomenon under investigation [75, 76]: Complementing different modes of observation is a research strategy common to mixed methods research designs [26]. We purposefully complement these different modes of observation on modeling processes to allow us to identify a wide range of modeling difficulties by enabling us to recognize and identify cognitive breakdowns more precisely than through a single mode of observation, and to allow us to obtain insights into the causes for cognitive breakdown.

The (meta) objective above the primary research objective of identifying and classifying modeling difficulties modelers face is to inform design science research on developing (tool) assistance for conceptual modelers at different stages of their learning and mastering of conceptual modeling aimed at mitigating modeling difficulties: By identifying modeling difficulties and by developing a taxonomy of such difficulties [36, 48] over the course of multiple studies, the present research aims to contribute to establishing a theoretical foundation for developing assistance for conceptual modelers tailored to help them to overcome their specific modeling difficulties.

After introducing the theoretical background (Sect. 2), related work is described (Sect. 3). Next, the mixed methods research design with the multimodal observation setup and the study conduct of the follow-up study—in comparison with the exploratory study reported in [57]—is explained in Sect. 4. Section 5 presents the findings, followed by a discussion of the findings and future research directions (Sect. 6). The paper closes with a discussion of limitations (Sect. 7) and a conclusion (Sect. 8).

Prior research has conceptualized conceptual modeling as ill-structured problem solving [6, 52, 68]: A modeling task (e. g., a data or business process modeling task) does not imply a clear path to a conceptual model (e. g., a data or process model)—similar to ill-structured problems where a problem representation does not imply a clear path to a solution of the problem [47, 55]. Rather, a modeling task starts from a problem representation in textual form (using natural language) and/or graphical and other visual forms and requires the application of modeling concepts of a modeling language to create a conceptual model by purposefully reconstructing the problem representation by means of modeling concepts of the chosen modeling language. The aspired artifact as result of this problem-solving process is the conceptual model.

A cognitive view on conceptual modeling processes as problem-solving processes allows us to better understand modeling processes in terms of modeling difficulties and underlying cognitive processes: Cognitive Load Theory (CLT) models cognitive resources of humans and how these resources are used in problem solving and learning [69]. At its core, CLT differentiates between a limited working memory and a comparatively unlimited long-term memory [50]. Following CLT, humans are assumed to possess a limited cognitive capacity in performing complex tasks as the capacity of the working memory, i.e., humans’ cognitive resources, is limited at a given time. Three types of cognitive load can be distinguished: (1) intrinsic load, i.e., the inherent difficulty determined by the complexity of the problem solving task; (2) extraneous load, i.e., the extra load originating from the problem representation; and (3) germane load, i.e., the load needed to relate information with long-term memory [13, 50, 52]. Hence, for conceptual modeling, a modeling process is influenced by modeling task characteristics referring to intrinsic load, characteristics of the task’s representation including tool and modeling language characteristics referring to extraneous load as well as the modeler’s individual cognition that refers to germane load [19, 52, 66].

If the overall cognitive load on a subject performing a problem-solving task overstrains the subject’s cognitive resources, difficulties are likely to occur—potentially leading to a cognitive overload [69]. Problem solving research stipulates three general “processes” or “components” involved in problem solving [42]: search, recognition and inference with search and recognition aimed at handling information of rather low complexity. As conceptual modeling usually requires several complex cognitive processes referring to making inferences—beyond search and recognition which imply handling information of rather low complexity [52]—creating conceptual models constitutes a complex problem-solving task that is assumed to lead to cognitive difficulties [3, 11, 20, 52].

To identify the difficulties which modelers experience while constructing a conceptual data model, we rely on the concept of cognitive breakdown [11, 47]. Following problem-solving research [47] and prior work on cognitive difficulties in problem-solving processes [11, 78], we conceptualize a cognitive breakdown as a difficulty a modeler experiences when constructing a conceptual model based on a natural language description [11]—a situation when the overall cognitive load overstrains a subject’s cognitive resources: ”when a line of thought fails” [20]. In their foundational work [47], Newell and Simon suggest that a cognitive breakdown during problem solving leads a subject to return to the problem presentation or, if unable to overcome the difficulty, to abandon the problem. Hence, a cognitive breakdown manifests itself in a modeler explicitly verbalizing a difficulty while modeling or in interrupting or terminating a modeling activity, e. g., a modeling activity which is not completed but instead the modeler switches to another activity [11].

This section provides an overview of main strands of related research, i.e., related prior work investigating individual (data) modeling processes as well as prior research focusing on difficulties and errors in conceptual (data) modeling.

Early contributions investigate data modeling processes: Batra et al. [7, 8] report a laboratory study that compares conceptual data models constructed by students using the relational model and the extended entity–relationship (EER) model. The data models are evaluated in terms of correctness with regard to modeling entity types and relationship types, compared to a solution developed by the researchers. Regarding modeling difficulties, the study suggests that difficulties do not primarily occur in modeling entity types but in modeling relationship types, leading to the conclusion that the complexity in data modeling is mainly related to relationship types. Specifically, the study indicates that the number of modeling difficulties in creating a relationship type increases with the degree of a relationship type. In their study on similarities and differences between non-experienced and experienced modelers [6], Batra and Davis derive a process model of conceptual data modeling from analyzing verbal protocols which distinguishes three distinct levels of abstraction, e. g., the enterprise level, the recognition level and the representation level, as well as the iterations between the levels. The process model is then used to identify similarities and difference between expert and novice modelers. It is concluded that experts focus on developing a holistic comprehension of the problem, whereas novices are largely unable to integrate parts of the problem description resulting in more errors in their models. A further study by Srinivasan and Te’eni targets the behavior of modelers while data modeling [68]. Considering conceptual data modeling processes as problem-solving processes, the study reports on two laboratory studies using think aloud protocols. The research design focuses on the problem representation and problem-solving heuristics, i.e., strategies for controlling cognitive activities, applied by the modelers to overcome cognitive limitations. First, a cognitive model of data modeling is developed including problem representation, cognitive activities, heuristics and constraints on the effectiveness and efficiency of the cognitive activities as well as their interdependencies. Analyzing the verbal protocols based on the cognitive model leads to insights into how individuals use heuristics to control complexity in their modeling processes suggesting that modelers would benefit from support in moving across levels of abstraction. Studying resulting data models, Shanks investigates differences between expert and novice modelers along several dimensions [65]. Evaluating the conceptual data models involving experienced reviewers in terms of correctness, completeness, innovation, flexibility, understandability and overall quality leads to the insight that the data models constructed by expert modelers are more correct, complete, innovative, flexible and better understood in comparison with data models constructed by novice modelers. Research by Hoppenbrouwers et al. builds on communication theory and identifies modeling strategies exhibited by individuals based on linguistics analyses—viewing conceptual modeling as a dialogue and coining the term “modeling dialogue” [38, 39]. Further work investigates cognitive mechanisms of conceptual business process modeling with a focus on collaborative modeling [82‐84], proposing relational reasoning and abstraction as key cognitive processes in modeling and suggesting a method for analyzing collaborative process modeling behavior, aimed at generating insights into psychological mechanisms of modeling skills and related cognitive processes [84]. Another stream of related research investigating the process of process modeling identifies distinct modeling styles [52] as well as cognitive process modeling techniques applied by modelers [24]—resulting in the so-called Structured Process Modeling Theory (SPMT) aimed at explaining how the probability of an occurrence of cognitive overload in process modeling processes can be reduced [24].

Complementing this prior research on individual modeling processes, research has investigated the frequency of certain error types in conceptual modeling for a long time. In an early work, Batra and Antony investigate errors of novice data modelers in two laboratory experiments [5]. The study evaluates errors in novices’ data models complemented with analyzing the modeling processes to achieve insights into why the errors have been committed. The errors committed by the novice modelers are classified in three categories referring to the cognitive dimension of the errors: (i) literal translation, i. e., a modeler “mechanically translating” a natural language sentence into a relationship type, (ii) anchoring, i. e., a modeler staying with an initial—but incorrect—starting point (anchored to an initial assumption) and (iii) incomplete knowledge, i. e., a modeler failing in the attempt to improving a model that is based on an initial incorrect assumption. In the conducted experiment, the three categories of errors are observed with approximately equal frequency. Please note that the study by Batra and Antony only considers errors relating to relationship types. As main causes of the errors, the complexity of the modeling task in terms of the number of possible relationship types increasing at a combinatorial rate with the number of entity types, misapplication of modeling heuristics and a lack of knowledge about database design are identified—leading to suggestions for supporting novices in data modeling with immediate feedback in supportive tools. A further related study by Leung and Bolloju investigates typical errors committed by novice systems analysts in creating domain models using the UML [43]. Based on Lindland et al.’s framework relating to quality in conceptual models and modeling processes [44], the errors are distinguished into different categories related to syntactic, semantic and pragmatic quality. In analyzing the class diagrams, the most common errors are observed in determining cardinalities for associations—relating to the semantic quality—and the presence of unexpected features (i. e., creating unnecessary entities or attributes)—relating to pragmatic quality. In [41], typical error types in creating class diagrams in a specific notation based on the UML made by novice learners are identified by Kayama et al. Four categories of errors are detected, i. e., syntactic errors, errors related to attributes, related to associations and related to classes. As a result, it is observed that errors related to associations constitute the most common type of errors made by novice learners with these errors referring to the associations’ labels and cardinalities. A recent study by Bogdanova and Snoeck investigates frequent errors in students creating domain models using UML class diagram [16]. Consistent with the findings of previous work [41, 43], the study finds that errors relating to modeling classes are less numerous than errors in modeling associations. Regarding classes, especially wrongly named classes and missing classes are observed while the most common error regarding associations is observed in determining multiplicities. A further study by Bogdanova and Snoeck [17] presents insights into novices’ errors in creating class diagrams in the MERODE modeling notation [17]. Again, in line with previous research, errors relating to modeling associations are identified as the most frequently occurring errors besides missing classes. Moreover, it is concluded that the most common errors relate to the novice modelers misinterpreting requirements during requirements analysis.

Different from earlier work, the objective of the present research is to identify and classify modeling difficulties in data modeling processes following a cognitive view on conceptual modeling using the notion of cognitive breakdown. Furthermore, the present research design stands out from extant studies by following a mixed methods research approach that combines multiple perspectives on modeling processes.

In this section, we first report on the modeling processes in the follow-up study in terms of peculiarities, domain knowledge of the participants and observations regarding verbalization skills. Subsequently, the types of modeling difficulties inducing cognitive breakdowns in the modeling processes of both studies are introduced and exemplified, refining and extending the results presented in [57].

Integrating complementary modes of observation of individual data modeling processes of 16 learning modelers and an analysis using the concept of cognitive breakdowns leads us to identify a wide variety of types of modeling difficulties these subjects faced while modeling. In an exploratory study with eight modelers with little to no modeling experience, we identify five types of modeling difficulties, complemented with three further types of difficulties encountered in a follow-up study with eight medium-experienced modelers. In the following, we discuss our findings in both studies and fruitful paths for future research, particularly implications for design science research on developing modeling assistance for data modelers at different stages of their learning and mastering of conceptual modeling.

Our findings suggest that the majority of difficulties encountered by the participants in the modeling processes relate to modeling relationship types: In the exploratory study, difficulties of the types Develop label for relationship type, Determine cardinalities and Decide between entity type and relationship type constitute the most frequent types of difficulties by far. Findings of the follow-up study confirm that difficulties are mainly experienced in modeling relationship types (difficulties of the types Develop label for relationship type, Determine cardinalities, Decide between entity type and relationship type, Establish relationship type) with the type Determine cardinalities being the most frequent one. Combining the findings of both studies (see Table 5), it appears that cognitive load overstraining a modeler’s cognitive resources can be observed most frequently in modeling relationship types: More specifically, cognitive overload leading to the most frequent and severe difficulties for the observed modelers occurs in deciding whether to model a relationship type, how to label a relationship type and which cardinalities are sensible for a relationship type.

These observations are in line with prior work on cognitive complexity in data modeling [4, 7, 8] suggesting that modeling problems are not experienced mainly in modeling entity types and attributes, but in modeling relationship types—originating in an extensive load on the modeler’s working memory due to the combinatorial rate with which the number of possible relationship types increases with the number of entity types in a model. Moreover, our findings are consistent with results from prior research investigating the frequency of certain error types in domain modeling: Kayama et al. also identify errors related to associations as the most common type of errors made by novice learners, specifically referring to the associations’ labels and cardinalities [41]. The present observations, in addition, tie in with the recent work by Bogdanova and Snoeck that finds errors relating to modeling associations are the most common types of errors [16, 17] with determining multiplicities to be a specifically frequent error [16]. Furthermore, while our observations confirm Leung and Bolloju’s insight that common errors relate to modeling cardinalities for associations, their study does not identify developing labels to be a frequent error [43]. However, the prior work on error types mainly focuses on domain modeling with the UML [16, 41, 43] and MERODE [17] while the present work targets conceptual data modeling with a variant of the ER model.

Altogether, the present research reinforces the finding that beginning and medium-experienced modelers especially face problems in modeling relationship types when creating a conceptual data model. Our results further specify this finding to the extent that the majority of relationship type-related difficulties evolve around determining cardinalities and developing sensible labels for relationship types. Specifically the difficulty of devising sensible labels for relationship types refers to the core of conceptual (data) modeling and, hence, requires careful attention: Meaningful and purposeful labels for model elements are a prerequisite for comprehensible and usable conceptual models—conveying semantics important to sensible interpretation of a conceptual model by human viewers [10, 27, 45]. Different labels bring up different cognitive association, and thus, it makes a significant difference how a model element—in this case, a relationship type—is labeled. Targeting this type of modeling difficulties requires to assist modelers in developing labels that are sensible with respect to a natural language description of a universe of discourse and its technical terminology, to the chosen modeling objectives and to the model’s intended application context, with the aim to support “identifier comprehension” [8]. While the medium-experienced modelers have more experience in conceptual data modeling and specific prior experience in working on a modeling task on lending items (cf. Sect. 4), the difficulties relating to cardinalities and labels for relationship types remain important in the follow-up study. This observation indicates that the difficulties are not easy to solve and, furthermore, cannot be attributed to modeling experience as a modeler characteristic referring to germane load for the observed modelers. Hence, a fruitful path for future research lies in exploring further modeler characteristics—beside modeling experience—as well as modeling task characteristics referring to intrinsic load and characteristics of the modeling task’s representation referring to extraneous load on the modelers. For example, besides varying modeling tasks and task representations, further studies could expand the surveying of participants, including more detailed questions on the modeler’s studies of conceptual modeling and domain knowledge—in order to shed light on varying modeler characteristics besides modeling experience.

In the follow-up study, the second and third most frequent type of difficulties encountered by the medium-experienced modelers relate to modeling entity types and attributes, i.e., difficulties of the types Choose data type of attribute and Decide between attribute and entity type. Hence, we conclude that—although the faced difficulties primarily relate to modeling relationship types—the cognitive load induced by modeling entity types and attributes also has the potential to overstrain the cognitive resources of medium-experienced modelers. This is surprising as it is contrary to prior work stating that difficulties in data modeling are not mainly experienced in modeling entity types [4, 73]. In addition, it is remarkable that only a small number of difficulties encountered in the follow-up study relates to the additional modeling concepts of generalization hierarchies (Specify generalization hierarchies with three occurrences) and recursive relationship types (with one occurrence related to developing role designators as part of Develop label for relationship type). Although these modeling concepts are generally acknowledged as more advanced [28], the majority of difficulties still relate to the basic modeling concepts of entity types, attributes and (non-recursive) relationship types.

For the meta-objective of informing future design science research on developing (tool) assistance for conceptual modelers at different stages of their learning and mastering of conceptual modeling aimed at mitigating modeling difficulties, the present findings constitute a first step: First, current insights into modeling difficulties faced by beginning and medium-experienced modelers during model creation provide a starting point for the design, implementation and evaluation of tailored tool assistance for modelers that deliberately targets modelers’ difficulties while conceptual data modeling. The majority of modeling difficulties the observed modelers encounter during conceptual data modeling relate to modeling relationship types, particularly, (1) determining cardinalities, (2) developing sensible labels for relationship types and (3) deciding whether to reify a relationship type or not. Hence, design science research on modeling tool assistance that targets devising meaningful labels for model elements and deciding on sensible cardinalities for relationship types promises to result in assistance that deliberately benefits modelers—opening a path for innovative, original contributions to research on modeling tools for end users [35, 51]. A first step in this direction is the Natural Language Processing-based Automated Assistant proposed in [71] that provides beginning data modelers with modeling-time assistance: Based on the insight that devising meaningful and purposeful labels has proven a particular challenge for data modelers, the Automated Assistant provides suggestions on identifying and signifying entity types, relationship types and attributes with meaningful and expedient labels and encourages modelers to rethink their choices of model element labels—and, hence, purposefully supports modelers in mitigating their modeling difficulties. Second, categorizing modeling difficulties contributes to the theoretical foundation of conceptual modeling. The identified types of modeling difficulties serve as a basis for developing a taxonomy of modeling difficulties over the course of multiple studies, in the sense of a classification or taxonomic theory (following, e. g., [36, 48]). Starting from the present results, further studies into modeling difficulties encountered by participants are suggested: Studying modeling processes of modelers with varying characteristics regarding modeling experience and knowledge have to build on, refine and extend the classification system by adding emerging types of difficulties on the basis of characteristics of the actual difficulties observed in modeling processes. In addition, a fruitful path for future research lies in complementing data collection and analysis: To contribute to obtaining a more complete understanding of modeling processes, recording and analyzing eye movement data offers a further promising mode of observation [13, 81]. Further, findings from analyzing the constructed data model as results of the modeling processes offer the opportunity to extend current insights into modeling difficulties. For extending the data analysis strategy to analyze the data models, the conceptual modeling quality framework suggested in [46] provides a starting point. Further, an additional in-depth analysis of cognitive breakdowns in modeling processes with regard to the different types of cognitive load (see Sect. 2) promises to result in rich insights into cognitive processes during modeling.

The observed differences among subjects in the length of the modeling process in both studies is in line with earlier work on prior modeling knowledge and modeling experience of conceptual data modeling (e. g., [6, p. 94]). It is remarkable that the outlier regarding prior modeling experience in the follow-up study (P10) does not exhibit a modeling process that is recognizably different from those of the other participants in terms of length or number of modeling difficulties encountered—in contrast to the outlier P8 who constructs a straightforward solution to the modeling task in a remarkably short modeling process without facing any difficulties in the exploratory study. However, these observations are consistent with practical experience being discussed as only one aspect of being an experienced modeler besides theoretical knowledge and training (e. g., [6, p. 87]; [74, p. 50]). A potential path for future research, hence, lies in further exploring how practical experience in conceptual modeling, knowledge of theoretical foundations of conceptual modeling and modeling training interact in achieving a more advanced level of modeling expertise.

The present research is subject to limitations with respect to the focus on modelers’ capability of conceptualizing a domain. As it excludes the modeling dialogue between modelers and domain experts, the present work does not claim to address the entire process of conceptual data modeling. In addition, the modeling processes analyzed in the present work assume an individual approach to conceptual data modeling. While this aligns with the focus on a learning context, a fruitful path for future research lies in extending the research to a setting in which modeling dialogues between modelers and domain experts are observed. Recording and analyzing think aloud protocols in such a setting promises to shed light on modeling difficulties experienced in a modeling dialogue.

Principle limitations relate to analyzing think aloud protocols. Modelers may have difficulties verbalizing their thoughts while modeling on principle accounts [15] or on modeling-related accounts—resulting in some verbal protocols not being as complete as others [67]. Generally, it is assumed that thinking aloud does not interfere with thought processes—but as the modeling task includes a visual, non-verbal perceptual component, thinking aloud may slow down the thought processes and/or the modeling performance [30]. We observe that participants with good verbalization skills tend to explicitly verbalize difficulties they encounter, while cognitive breakdowns in modeling processes of participants exhibiting problems in verbalizing thoughts primarily manifest themselves in interrupting or terminating a modeling activity (cf. [11]). Differences in verbalization skills have long been discussed (e. g., [31]), and thus, we included a think aloud training in the data collection procedures of both the exploratory and the follow-up study. Furthermore, all participants in the follow-up study model and think aloud in English, not their first language. However, the participants study in a context characterized by the use of English.

We recruited bachelor and master students from two universities as participants. In line with the educational context in which the present research is embedded and following the objective to achieve insights into modeling difficulties beginning and medium-experienced modelers face while performing a data modeling task, we consider students to be participants that can provide in-depth information about the phenomenon under investigation (e.g., [26]). Furthermore, recruiting students as study participants has a long tradition in related research—starting with, e.g., [6, 8] and continuing with, e.g., [16, 41]—and is acknowledged as viable choice in conducting studies in our field [32]. Beginning practitioners of conceptual data modeling are not explicitly included in the study sample, although several participants reported work experience outside their studies. To achieve further insights into the possible impact of germane load on modeling processes and modeling difficulties, we intend to complement the present studies with further follow-up studies observing subjects including practitioners with various backgrounds, e.g., regarding prior experience and knowledge of conceptual data modeling as well as their first language.

It is important to note that the present work does not investigate the impact of the problem representation—i.e., characteristics of the modeling task’s representation including tool and modeling language referring to the extraneous load on the modelers—on the modeling processes and modeling difficulties encountered by the observed modelers. With regard to the modeling tool, the data analysis differentiated between cognitive breakdowns induced by modeling difficulties and difficulties with the modeling tool. In coding the audiovisual protocols on cognitive breakdowns, we coded for non-task related issues that include difficulties in operating the modeling tool (see Table 1). We carefully examined if certain breakdowns can be attributed to hesitations on how to use the modeling tool. Furthermore, the modeling task and its representation as natural language description are kept constant in the exploratory study, i.e., the natural language description from the library domain, as well as in the follow-up study, i.e., the natural language car rental task. Hence, comparing modeling processes using different modeling tools and modeling on paper as well as varying modeling tasks representations opens a fruitful path for future research into the impact of extraneous load on tool-supported modeling processes.

Taking a cognitive view on conceptual modeling processes, we observe data modeling processes of eight learning data modelers with little to no modeling experience in an exploratory study, complemented with observing data modeling processes of eight participants with some experience in conceptual data modeling in a follow-up study. Analyzing over six hours of audiovisual protocols of the modeling processes combined with analyzing modeler–tool interactions and surveying modelers leads us to identify eight types of modeling difficulties these subjects encountered while modeling. Combining complementary observation modes and a corresponding integrated data analysis, we identify and classify eight types of modeling difficulties—relating to modeling entity types, generalization hierarchies, relationship types, attributes with their data types and cardinalities.

Our findings indicate that modeling difficulties of beginning and medium-experienced data modelers primarily refer to modeling relationship types, with a majority of difficulties specifically relating to determining cardinalities and devising sensible labels for relationship types—insights that confirm and extend results from prior research performed by different researchers, with different groups of subjects and varying experimental setups (e. g., [4, 16, 17, 41, 43, 73]. As to the meta-objective of informing design science research on developing (tool) assistance for conceptual modelers, our findings (1) provide a starting point for designing, implementing and evaluating tailored tool assistance for modelers—that deliberately targets modelers’ difficulties while conceptual data modeling, e. g., supporting modelers in deciding on adequate cardinalities or in devising sensible labels for relationship types (e. g., [71]), and (2) constitute a first step toward developing a taxonomy of modeling difficulties over the course of multiple studies (e. g., [36, 48]). In addition, (3) the present insights suggest further research efforts into how practical experience in conceptual modeling, knowledge of theoretical foundations of conceptual modeling and modeling training interact with respect to modeling expertise.