Mapping validity and validation in modelling for interdisciplinary research

Many of today’s global issues such as climate change, food security and social injustice, call for joint input of researchers from different scientific disciplines. The complexity of these issues has promoted the development of interdisciplinary research and complexity approaches, or as Fattore and Grassi (2015, p. 1551) put it “complexity is interdisciplinarity”.

Interdisciplinary research is (Aboelela et al. 2007, p. 341): “any study or group of studies undertaken by scholars from two or more distinct scientific disciplines. The research is based upon a conceptual model that links or integrates theoretical frameworks from those disciplines, uses study design and methodology that is not limited to any one field, and requires the use of perspectives and skills of the involved disciplines throughout multiple phases of the research process”. To make explicit what constitute distinct scientific disciplines, we use Kagan’s (2009) distinction of three distinct scientific cultures (2009): natural sciences, social sciences and humanities. In the Complex Adaptive Systems (CAS) context, the label ‘interdisciplinary’ refers to a team of people from at least the natural and the social sciences. We leave open whether the team includes researchers from the humanities.

One of the complexity approaches is modelling of CAS, which are composed of decision makers and their environment. Decision makers have goals, can interact, learn, and adapt within the environment. The system-level behaviour of a CAS emerges from interactions amongst decision makers and between individual decision makers and the environment. Examples of CAS are energy markets (North et al. 2002, with businesses, and electricity consumers as decision makers, within an environment composed of the power market and regulating administration), or land-use systems (Termansen et al. 2019, with farmers and land managers as decision makers, and the land and its vegetation as environment).

Keeping track of individual decision makers, their social interactions, their interactions with their physical environment and interactions between the social and physical aspects of the complex system is typically not feasible. Consequently, modelling is not only an important tool in the study of complex systems at the system level, but also for the social sciences that study social actors and support decision makers. Modelling a CAS may serve many purposes but three are unambiguously related to an empirical reality and will be considered in this paper: explanation, description, and prediction of new (i.e. not yet known) behaviours (Edmonds et al. 2019). For each of these purposes the modelling of a CAS builds on, and contributes to, different disciplines including the social sciences.

Hox (2017) suggested that social science research methodology could contribute to computational social sciences by exploring different kinds of validity. Because of the social sciences’ potential for the study of CAS, and vice versa, we take this suggestion to interdisciplinary computational sciences as represented by CAS modelling. After all, validity is an important quality criterion for all research and validation is an established criterion in the assessment of models. Since CAS modelling relies on input from the social sciences, validity of this input is important for the modelling process and conclusions. Because a common understanding and terminology presents one of the major challenges in interdisciplinary research (Fischer et al. 2011), the question rises how different definitions and uses of the concepts validity and validation relate to one another, and what a common terminology suitable for interdisciplinary research using CAS modelling, could look like. A common terminology would reduce confusion and facilitate debate about validity and validation of CAS models, but would also be useful in other interdisciplinary modelling approaches.

In the next section we describe the modelling process and adapt it to explicitly address the interdisciplinary nature of CAS. In the Methods section, we describe how we reviewed the use of the words validity and validation in research methodology and modelling textbooks, and how we synthesized both concepts with the adapted modelling cycle into the ‘map of validity and validation in CAS modelling’. The fourth section contains the results of this narrative review. In section five we present the synthesis: the map of validity and validation in CAS modelling. We use a CAS paper from the literature to show the full map from the validity of research results used as input in the modelling process, through model validation to the validity of the built model in section six. We conclude with a discussion of our work and suggestions to further explore and exploit the value of social sciences research methodology in the study of CAS.

Five iterative steps are commonly distinguished in the modelling process: the first three go from reality to the conceptual model (the conceptual step), from the conceptual model to model code (the programming step), and, from the code back to the conceptual model (the code verification step). The fourth step is running the model, the fifth step the model validation (Augusiak et al. 2014).

The Methodology for Interdisciplinary Research (MIR) framework (Tobi and Kampen 2018) distinguishes four phases. The first is the conceptual design in which the what and why of the modules or sub-studies and the theoretical framework are decided on; followed by the technical design in which the how of the research is explicated. The operationalization of concepts connects the conceptual and technical phase as the theoretical framework chosen in the conceptual design informs decisions on measurements in the technical design. The data is collected and analysed in the execution phase, and, finally, in the integration phase the different modules of the interdisciplinary projects are appropriately combined and reported.

Combining the five modelling steps and the MIR framework into the adapted modelling cycle, we distinguish four different phases (see Fig. 1). In the first, the conceptual design phase, the interdisciplinary team members, together, decide on the research objectives and the aims of the CAS modelling. Theoretical frameworks are chosen to guide decisions on the system’s boundaries, components, relations and their operationalizations. This usually includes social as well as ecological or technological components.

The technical design (phase 2) consists of two distinct parts. The first one is the joint design of the different distinguished modules of the study (Tobi and Kampen 2018). For the modelling module this includes the type of model, how the components and relations distinguished during the conceptual phase are included in the model, how different parameter values will be chosen (e.g., by direct measurements by members of the interdisciplinary team, from literature or experts, or through calibration Hill 1998), and how the model and the data it generates will be analysed and validated. The second part of the technical design, the CAS model construction, is the coding of the model and the code verification. In this phase, the modellers may consult with the other team members when faced with issues not anticipated that call upon content matter knowledge.

The execution (phase 3) generally starts with running the model and doing the planned analyses. In sensitivity analysis the effects of changes and uncertainties in model parameters on the model output are investigated (Iooss and Lemaître 2015). Depending on the modelling aims, sensitivity analysis can test under what (if any) parameter settings the model generates the behaviour that it was aimed to explain, or it can reveal ranges of model predictions and their likelihood. If the aim is description, sensitivity analysis is less relevant (Edmonds et al. 2019).

The role of validation depends on the modelling aims too. When the aim is explanation, validation investigates whether the model presents an adequate explanation for the behaviour to be explained (Grimm and Railsback 2012). If the aim is description, validation will assess whether the model adequately represents all relevant elements to be described (Edmonds et al. 2019). For prediction, validation will be aimed at investigating whether model predictions match reality by comparing them against new data (Guerini and Moneta 2017). Concluding, validation yields information on what the model tells about the system, related to the aims of the research. Consequently, the interpretation of this validation is always an interdisciplinary team effort.

The fourth phase (integration) involves, where appropriate, a synthesis of different research modules, and reporting. Different publications may address different modules and although the leading authors may differ, all publications require interdisciplinary input.

Since the system-level behaviour of CAS originates from interactions within the system, studying the within-system processes is essential for theory building and understanding CASs. Consequently, we focus on the most commonly used modelling approaches for representing these processes: system dynamics models (Phelan 2015; Schieritz and Milling 2003) and agent-based models (ABM) (Filatova et al. 2013). System dynamics models describe feedbacks between system components on an aggregated level and simulate the resulting development of the system over time. For example, Hsiao et al. (2012) have studied the development of professional baseball in Taiwan by describing feedbacks between several variables, including the number of fans, team income, player quality, and team image. They used their model to predict the development of these variables over time. ABMs are composed of interacting agents (e.g., people, animals, companies, etc.), with individual characteristics and behavioural rules. For example, Holtz and Pahl-Wostl (2012) have developed an ABM to better understand how farmers’ characteristics influence land-use changes in Upper Guadiana, Spain. The farmers are modelled as agents that are influenced by social factors (e.g., policies, risk aversion, knowledge of crop-irrigation technology), ecological and technological factors (e.g., possible crops, irrigation technologies), and relations amongst and between those components (e.g., respecting water law, crop-irrigation combinations, profit).

This section provides details on how the inventory of validity and validation terminology and its synthesis with the adapted modelling cycle were made (Fig. 1).

In this synthesis we present the `map of validity and validation in CAS modelling’ (Fig. 2). Validity as we have seen it in the research methodology books, is a quality characteristic of the model inputs. This forms the first layer in the map. The second layer depicts the modelling cycle (Fig. 1). The third layer “model validation” depicts model validation processes, and is based on the review of validation in the modelling textbooks. The bottom layer is about model validity. In the following three sections, we focus on how the layers inform one another to conclude with three classes of validity of models for interdisciplinary research.

Our aim with this illustration is to show how Fig. 2 facilitates discussion and assessment of validation and validity of CAS models. We selected a study on Philippine fishery system (Libre et al. 2015) for several reasons. First, the study has a clear interdisciplinary character, using inputs from biology, economics, and sociology. Second, the study describes the entire process of model development, including preliminary data gathering. Thirdly, the study used primary and secondary data from various sources as model inputs, which makes it a suitable example of what this input data implies for the validation and validity of the model. Table 2 summarises all the steps in the process of data collection and model development, as described by Libre et al. (2015). No additional data was gathered, beyond what was reported in Libre et al. (2015). In the following subsections, we use the map of Fig. 2 to discuss what the description of Libre et al. (2015) implies for the model validation and validity. Each of the following subsections corresponds to one layer in Fig. 2.

Libre et al (2015) recognized that models that inform policy and represent fishers as perfectly rational economic agents show several problems. Therefore, their aim is (p. 251) “to view fisheries as CAS to evaluate the effects of social factors and bounded rationality on macro-level outcomes of fisheries, including the number of vessels, total fishing days (the measure of fishing effort), fish stock, and industry profit”.

We identified three basic validities of primary studies from research methodology (measurement validity, internal validity, and generalizability) and two types of validation from computer modelling (structural validation and behavioural validation). Interdependencies between validities and validation of the conceptual model, the model inputs, the modelling process, and, the model as research instrument were identified and illustrated. Different aspects of validity and different processes for validation were disentangled, resulting in a map with four layers (Fig. 2) to facilitate communication in modelling for interdisciplinary research. Furthermore, we applied the map to an example to show how the map can help discuss and understand validity and validation in the modelling process and the model’s assessment more precisely. For this example, model validity was difficult to establish, partly because the validity of model inputs was difficult to establish and not only because additional research is needed. This illustrates that the map facilitates communication about validity, validation, and interdependencies in interdisciplinary CAS modelling.

In the scientific literature, validity and validation is discussed along disciplinary lines, like economics (Fagiolo et al. 2019; Windrum et al. 2007), land-use and planning (Brown et al. 2005), and, ecology (Augusiak et al. 2014). There appears to be no bridge, which confirms the importance of the present study. In economics, validation is discussed as the process of matching model output against real-world data, and is usually discussed alongside methods for calibration and sensitivity analysis (Fagiolo et al. 2019; Windrum et al. 2007). In land-use science, Brown et al. (2005) distinguished between validation of the modelled processes, and their final outcomes which is akin to what we call assessment of predictive validity and internal validity of the model. In ecology, Augusiak et al. (2014) proposed ‘evaludation’, as the merger of model validation and model evaluation, which became the foundation of the modelling documentation framework TRACE (Grimm et al. 2014; Schulze et al. 2017). We see the merits of ‘evaludation’ and the TRACE framework, but neither distinguishes between the validity of model input, validation of the model, and, model validity, like this paper does. Also within ecology, Getz et al. (2018) distinguished the modelling process from the model as a product, but did not go into details with respect to what is (cross-) validation of models and validity of models.

Interdisciplinary CAS models rely on inputs from various sources and the quality of a model inevitably depends on the quality of these inputs. The validity of interdisciplinary models is therefore vulnerable to their inputs regardless of the disciplinary backgrounds. This emphasizes the importance of validity as we know it from research methodology to all disciplines that aim to contribute to CAS models. If the quality of these models was to be assessed purely based on validation outcomes ignoring validity of inputs this could easily lead to inappropriate model structures, assumptions and outcomes. While validity of models will always rely on inputs, our proposed map (Fig. 2) makes this reliance explicit by showing how validity of model inputs impacts on different steps in the modelling process and, ultimately, on the validity of the resulting model. By explicating terminology, our study can help researchers using interdisciplinary models to communicate about the validity of models, the validation, and their assessment.

We hope that our map will support social scientists to get more involved in CAS, after all socio-technical and socio-ecological systems refer to social agents and processes which are generally not the expertise of, say, engineers and ecologists. While we focus specifically on CAS models, we expect much of Fig. 2 to be generalizable to interdisciplinary modelling applications in general. We think Fig. 2 also useful for modelling within the social sciences, in particular in modelling social systems where individual agents are nested in (sub)groups and societies, and inputs from multiple social sciences (psychology, anthropology and sociology) are needed. In addition, a better understanding of CAS modelling can help the social sciences to better understand human–environment interactions, e.g. how behaviours like physical distancing may be influenced by lay-out and COVID-19 warnings in public spaces.

The first weakness of the present paper lies in the stereotyping necessary for general statements and in the selection of textbooks. The textbooks are by no means representative of all research methodology and modelling textbooks, but they do reflect heterogeneity in both focus and audience. We investigated both ABM and system dynamics modelling, because both techniques can help understand the behaviour of system’s components but we did not investigate big data and machine learning techniques. The inductive reasoning enabled by these is insufficient to describe and explain how the components of the system operate. As interdisciplinary research aims to understand socio-technical and socio-ecological components and interactions, it cannot rely on inductive reasoning, but needs transparent models to build and test theory. Furthermore, without describing the interactions within the complex system, it is difficult to assess the generalizability of the model which hampers the utility of black-box models for informing policy. We presented one illustration (Libre et al, 2015) that modelled a specific real-world system. Many models are more abstract. For example, the Schelling model (Schelling 1971) suggests a mechanism that might cause segregation, but does not model a specific real-world case. Some of the steps in Fig. 2 may not apply to abstract models, since these rely less on real-world data as input.

We noticed that producing knowledge during the modelling process, through verification, validation or sensitivity analysis, may be seen as research through designing as we know it from the interdisciplinary field of landscape architecture (Lenzholzer et al. 2017). Indeed, research through designing does justice to the creative component of modelling in interdisciplinary research. The potential of this analogy needs further investigation. In the present paper we have omitted reliability, precision, responsiveness and accuracy. All these concepts relate to measurement properties, either of the instruments used to generate model inputs, the results of the modelling or the properties of the model as a product. Another opportunity for future research would be to incorporate these concepts in the presented map.

Concluding, this paper provided a framework that linked validity to validation in the context of interdisciplinary CAS modelling. The proposed terminology allows to describe how interdisciplinary modelling research relies on the validity of primary research used for input, how the quality of the modelling depends on structural and behavioural validation, and, how the assessment of the validity of the model is informed by these types of validation plus research with independent data. We hope that the framework will help to assess and improve the validity, and consequently, utility of models for interdisciplinary CAS research, and for informing policy advice related to CAS.