Trustworthy agent-based simulation: the case for domain-specific modelling languages

Although there are partial solutions to simulation engineering (cf. Sect. 6), most neglect the important aspect of fitness-for-purpose, and its connotations for principled complex-systems simulation. The importance of establishing and recording aspects that lead users or developers to trust the engineering and the resultant simulation has been captured in, for example, the CoSMoS Process [44]. CoSMoS is a ”lifecyle model” (see Fig. 1) of the whole simulation process, from domain exploration and identification of an appropriate simulation focus, to interpretation of results. CoSMoS is taken as the context for our vision of simulation engineering.

No techniques or work flows are mandated by CoSMoS, but in projects that have used the approach the modelling has been informal—often using ad hoc variants of UML. As a result, whilst developers have used programming environments, none has taken up CoSMoS’s suggestion of model-driven engineering. Some of the problems resulting from the lack of a coherent development approach are:These issues mean that simulation development continues to be a specialist activity, non-viable for many complex scenarios that would benefit substantially from computer simulation. Our vision is of well-founded, tool-supported notations, supporting CoSMoS-like development that encompass domain and scope exploration, software design and implementation, as well as experimental design and the recording of rationale. The later aspect, for which CoSMoS recommends argumentation techniques needs underpinning with traceability to development and experimentation designs. Our vision also needs to support the interpretation of simulation results into their real-world context, again through linkages across the family of models facilitating model management.

The vision underpins principled simulation with automation, with the potential to remove many of these problems and opportunities for error from the simulation development workflow. The envisaged basis for automation is a family of DSMLs to express features of the domain and simulation, as well as queries on the development. A DSML approach would allow domain experts to directly interact with simulation models and experiment specifications, while supporting the automatic translation of models and specifications into executable simulations.

Crucially, our envisaged family of DSMLs includes languages for expressing expected behaviours and fit-for-purpose arguments, integrated at a fundamental level with executable simulations, as well as languages for specifying appropriate simulation experiments and experimental protocols, with appropriate validation and sensitivity analysis to allow robust conclusions to be drawn. Based on our experience, the vision is expressed for agent-based simulations, but we believe that it would generalise to other forms of simulations.

This paper is an extension of [52], adding the following contributions: In the remainder of this paper, we first present a motivating example and highlight some challenges (Sect. 2). Section 3 presents an overview of our vision for a family of DSMLs for agent-based simulation. In Sect. 4 we introduce four user scenarios, which we discuss in detail in Sect. 5 applying them to the challenges identified in Sect. 2. Finally, we discuss other efforts for systematic engineering of agent-based simulations in Sect. 6 and conclude in Sect. 7.

Automated generation of executable simulations (Simulation platforms in CoSMoS and Fig. 2) enables separation of concerns: domain experts can focus on expressing their mental model of the domain, whilst software engineers can focus on simulation implementation. Further benefits arising from our vision of a family of DSMLs include the following:

We illustrate how our vision contributes to these user scenarios using artefacts from the PPSim simulator development [1, 2]. It is worth noting that, while some of these artefacts are diagrammatic models, they do not originally use any formally defined DSMLs. We will show the original artefacts and then show prototypical reimplementations of part of the artefacts to demonstrate the benefits DSMLs and a model-driven approach would offer.

It is useful, at this point, to give some more detail of the PPSim simulation activity. In the original PPSim development, the domain understanding was captured diagrammatically—an example from [1] can be seen in Fig. 3.

The agreed simulation purpose focused on an exploration of the mechanisms and actors in the formation of Peyer’s patches—clusters of cells that are ultimately part of the immune system, as formed in a neonatal mouse gut. Experimentation using the simulator established unexpected responses to chemicals that were subsequently tested in vivo. A key part of the simulation activity was to simulate the chemical stimuli and cell behaviours implicated in how cells bind together to form these clusters.

It is useful to give an intuition for what (simulated) binding and clustering entails. The basic context is that cells, of various sizes and properties, move through a tube (representing part of the gut), and may interact both with the surface of the tube and with other cells. Interactions are typically mediated by the state of the environment in the tube, modelling dispersing chemicals, flowing fluids, etc. Cell binding occurs when two cells—or a cell and the tube wall—are determined to have made contact: in reality, cells have receptors on their surface, and chemical bonds are created or broken. Interactions lead to changes of state in the cells: the new state may, for instance, cause one or both cells to emit a chemical, or respond differently to a chemical, or to differentiate to a state that enables different capabilities. Some of the cell states are labelled (biologically and in simulation) as “bound”, and it is the persistence of a group of bound cells that forms a cluster. Binding is not a once-and-forever action; the bind may be strengthened or weakened by interactions with and around the bound cells.⁵ Note that here, as in much of the natural world, terms such as “bound” and “cluster” are labels applied by scientists to what they observe: within the context and purpose of the simulation, there is no exact definition of when a cell is bound or a group of cells becomes a cluster.

In PPSim, the simulation is designed to cover a period of a few (specific) days in the development of an embryonic mouse gut. Unlike the real mouse “system”, the simulation of the behaviours starts with one type of cell already bound to the wall of the gut (the behaviours preceding this state are not relevant to the simulation purpose). The simulation ends at the point in simulated time that corresponds to the observed real-world appearance of significant cell clusters. Two similar but distinct types of cell are simulated moving through the tube/gut, with appropriate simulated behaviours, depending on the current state of each of the cells. The simulator design is thus a set of state changes, which model the known biological interactions that result in cell binding, including chemical expression and attraction (chemotaxis) by cells. Clusters can be identified visually (as they are in real-world experiments), but cell states include a variable that indicates whether a specific cell is bound or not, and a cell behaviour can probabilistically break a bind, mediated by the simulated chemical environment. Binding has potential self-reinforcement through both chemicals emitted by bound cells, and by further contacts from cells attracted by chemotaxis on such chemicals.

Whilst ongoing binding and recruitment of further cells is aspecific, the first bind relevant to the formation of a cluster is critical: it takes place between a mobile cell and a cell that is already bound to the tube/gut wall (i.e. the cells that exist in the initial state of the simulation). Both cells must be in the right state to take the first step towards binding once the initial, ephemeral, contact is made.

The PPSim simulation goal is simply to model the behaviours from which clustering of cells emerges, over a short period of time. This strict purpose and focus makes the simulation computationally feasible by limiting the range of behaviours, cell states, environmental chemicals, and so on, that need to be simulated. In reality, beyond the simulated time-window, the cluster itself eventually assumes identity and behaviours (becomes a Peyer’s patch), but this is out of scope for this simulation.

Again, it is worth noting that the point at which bound cells become a cluster is not well defined. A scientist can examine a section of gut and count the clusters, identifying each intuitively, perhaps based on their understanding of the subsequent phases of development where a cluster assumes the form and behaviour of a Peyer’s patch. In the simulation, cell states include variables that record contact and binding, to enable appropriate probabilistic behaviours. Since all clusters start from a single cell bound to the cell wall (of a type different to the mobile cells), the simulator can “count clusters” by setting a threshold value for the number of cells that are in contact and bound to the originating cell which is part of the initial state of the simulator. Terms like stable binding can be defined in biological terms, but assume a different sort of meaning computationally.

Next, we discuss each user scenario in turn.

We begin by discussing user scenario (US1), concerning the creation and maintenance of fitness-for-purpose arguments. The rationale for the PPSim model can be expressed in argument diagrams syntactically based on GSN (cf. Fig. 4). However, these models are essentially structured text, providing no explicit link to any of the concrete artefacts comprising the overall simulation model. The argument in Fig. 4 is only valid for a particular version of the simulation model and, in fact, relies on the results from a particular set of runs of the simulation. It also references data from relevant scientific publications of wetlab experiments (“[14]” in the figure), but this link is not explicitly encoded and quickly becomes difficult to track. It is possible to use hyperlinks within GSN, rather than text references such as “[14]” that have no inherent meaning, but this does not guarantee traceability over time.

Similarly, the Peyer’s-patch domain can be represented using a diagram like the one shown in Fig. 3. To date, this form of diagram does not have a formalised notation, but it could be presented in a DSML. The “Observables” at the top of the diagram correspond to domain model queries in Fig. 2 while the “Hypotheses” below correspond to the domain model capturing the hypothesised and to-be-simulated mechanisms.

Building on these observations, Fig. 5 shows an initial meta-model for a fitness-for-purpose DSML. The right part shows the standard GSN concepts of claims, justifications, definitions, solutions, and strategies, each allowing for a textual description.⁶

The Domain-Specific Domain Modelling Language for PPSim would define a Simulation Model concept that represents a container for all domain-model concepts. A specific instance of this concept can be referenced directly from the fitness-for-purpose argument model (see the Simulation Model concept in Fig. 5). Similarly, the Domain-Specific Model-Query Language for PPSim would, inter alia, define a concept of Query, with any specific observable being an instance of this concept—also making this explicitly referencable from the fitness-for-purpose argument model.

As we can see, a fitness-for-purpose argument model formalised on the basis of the meta-models above explicitly references all the specific artefacts making up the computational model. This enables the use of standard software versioning tools, such as Git [8], for tracking consistent sets of artefacts—and thus complete fitness-for-purpose arguments—as models and simulations change incrementally and iteratively. Such tracking can be done in different ways: Combined with support for citable code (e.g., the GitHub–Zenodo integration⁸), this completely supports user scenario (US1).

In the PPSim development, Fig. 3 summarises extensive dialogue between scientists and simulation developers, and expresses the abstraction level and potentially-simulatable scenarios, based on observations (data, visualisable emergent behaviours/structures) from real-world experimentation. The next phase of PPSim development, leading to the platform development, was to deduce behaviour diagrams. The simulator code was deduced from the diagrammatic design (and a lot of supporting text) [1]. Focusing on the step from state machine diagrams to code, Polack [38] explores the undocumented manual derivation, and how this might be replaced by transformation, noting the challenges of tracability and documentation of a manual complex system simulation development. This work also identified the way in which some of the non-diagrammatic information was used in manual design and coding, and showed how a systematic translation of the design resulted in often-subtle differences in the derived code structure.

In our vision, we need to establish how these manual derivation can become a series of model transformations. To improve the reproducibility of the simulation experiments and enable a clearer—and maintainable—argument for fitness for purpose, it would be beneficial to capture the translation as a model transformation between the domain model and the simulation platform. This would first require formalisation of the key domain concepts (the Glossary [44, pp. 127f]) as a meta-model so that the transformation into an agent model and, eventually, a platform model can be specified using an existing transformation language. In defining the transformation specification, the need for additional information may become evident; just as additional information became relevant when manually generating the simulation implementation before. Different to the manual approach, this would trigger an incremental extension of the domain-modelling meta-model, ensuring that all relevant information is explicitly captured in a structured artefact.

A similar effect occurs in relation to the “Observables”—Fig. 3—remember that these would be captured in the domain-specific query modelling language. For example, consider the observable ‘small cell clusters around LTo cell’. To express this in a formal model, we need to be able to have concepts such as ‘cell cluster’ or ‘LTo cell’ in our domain-modelling language. An agreed (possibly parameterised) definition of what it means for cells to form a cluster, and the rationale for this, could also be defined, with appropriate transformations into computation, so that a change in the agreed cluster definition could be easily translated into a change in the simulator platform code. By focusing attention away from manual traceability issues, this would allow more effective consideration of any side-effects of the change in definition (e.g. required changes in visualisation or experimental design).

In the PPSim example, binding of cells is captured statically via a reference between objects representing cells and dynamically via a set of operations that encode the conditions under which cells bind or unbind. Having an explicitly specified transformation between the concept of ‘cell cluster’ and these computation-oriented concepts makes the transformation explicitly inspectable and subject to discussion as well as referencable from a fitness-for-purpose argument. Our proposed transformation approach, linking models, queries over models, and fitness for purpose, would enable, for instance, differentiation in the simulator between “small” and “large” clusters, or between cell-type composition of clusters, without manual adjustment of the codebase. We could then generalise from this simulation instance, for instance to provide a more general expression language as part of the domain-specific model-query language to allow domain experts to capture different classifications of cell clusters that interest them, enabling new simulation experimentation and hypothesis creation without manual reworking of the simulator.

Another benefit of the model-based approach is that it enables separation of concerns and contribution of expertise from different backgrounds. In the example, assessing whether simulation results are aligned with real-world experimental data requires an appropriate domain model, but also requires significant statistical expertise to ensure the simulation results are interpreted correctly. Authors of fitness-for-purpose arguments may not have expertise in all of these areas. However, the model-driven approach means that some expertise that is required across domains can be encoded into the DSML. This is related to the notion of generic arguments proposed in [44, pp. 200ff] to capture recurring argument structures in simulation engineering. An MDE-based approach can go one step further by (partially) automating the evaluation of these argument structures.

As an illustrative example, the left part of the meta-model in Fig. 5 shows how a particular type of claim can be defined, which would capture an assertion of a statistically significant similarity between data from two separate sources (labelled left and right in the figure). Such a statistical similarity claim would still come with a textual explanation, but would further include explicit, computer-processable information about the kind of statistical test employed as well as the specific data sources compared. A data source has two components: the actual source of the data (in the meta-model this can be a CSV data file—for example, taken from a wetlab experiment or a published paper—or a simulation model) and a query that extracts the specific information required from the data source. We are using the Query concept from the model-query language, so such a query could be one of the observables from Fig. 3.⁹ A Simulation Model data source references a specific domain model and indicates that the data to be compared against comes from a simulation of that domain model.

Figure 6 shows the output of a prototype diagrammatic editor that supports generation of and reasoning about the fitness argument for part of the simulation. The editor is built using Eclipse Sirius [46], based on the meta-model from Fig. 5. The diagram shows a subset of the argument from Fig. 4, focusing on part of the statistical claims. The claim towards the bottom of the diagram (indicated by the orange arrow) is an instance of StatisticalSimilarityClaim from the meta-model. It references two data sources: a CSV file taken from a wetlab experiment, from which we extract the data in the first column, and a simulation model, from which we extract information about the patch behaviour. Here, neither the justification on the left, nor the solution entity at the bottom was specified by the user of the tool: the editor added both automatically as soon as the user inserted the claim that the statistical evaluation uses a Mann–Whitney test. The editor generates the argument, and tests it against the identified data sources, using a Mann–Whitney test (cf. Fig. 7). In this illustration, the claim is found not to hold, and the editor thus coloured the offending solution in red to highlight where the fitness-for-purpose argument breaks. This linkage of the argument and the experiment at the DSML level means that the argument, the data, or the model can be changed, and the argument updated automatically: for instance, a different data set (either the real world or simulation data) can be provided, and the fitness for purpose claim re-evaluated. Combined with standard software version-control mechanisms, this approach can satisfy the requirements of user scenario (US1).

The fitness-for-purpose argument in Fig. 6 simply references the simulation model, but does not specify anything about how this model is to be executed. Knowledge about how to execute a simulation is already encoded in the translation process that takes a domain model and produces a simulation implementation. The execution scripts thus generated can also include statistical analysis—for example using the Spartan tool [3] and its aleatory analysis to determine the number of simulation runs required to allow robust conclusions to be drawn.

The fitness validation shown in Fig. 6 requires running simulation experiments, which can be an expensive task. Validation of fitness-for-purpose argument models, therefore, needs to be triggered by the user. This also allows the experiment executions to be automatically scheduled so as to minimise the computational impact, similar to how modern software build tools automatically schedule the most efficient execution of only required build tasks. All of this execution infrastructure is provided as explicit code and transformation specifications, all of which are explicitly referenced from the fitness-for-purpose argument model, enabling full traceability.

So far, model-driven approaches to describing the domain under study have primarily focused on developing agent languages and (semi-)automated transformations into platform models. For example, the INGENIAS project [13] introduces an agent-modelling language to support replication by enabling the automated translation to different simulation platforms. MAIA [16] is a similar, slightly more recent approach. Here, there is some support for modelling variables of interest and extracting visualisations from the simulation logs. However, this is not connected to hypotheses, simulation experiments, or rationale for developing the simulation in the first place.

The OCOPOMO project [42] is, to the best of our knowledge, the first approach that partially addresses the need for achieving traceability from the original domain understanding and research context to the simulation implementation and final data, by allowing the inclusion of hyperlinks to the original data. However, models remain at the agent-language level, thus requiring a mental shift for domain experts to understand the models and relate them to their domain expertise. Other examples of model-driven approaches to agent-based modelling and simulation can be found in  [21] (possibly the first such approach) and [41], a more recent approach that provides a more domain-specific visual syntax for its agent language.

In [37], Parunak shows an interesting example of a domain-specific modelling approach for an agent-based model in the defence domain. Although this does not explicitly use language-engineering and model-driven engineering technologies, it is perhaps the closest work in intention and conceptualisation to our vision: high-level domain-specific languages (encoded in a variety of technologies, including spreadsheets) are made available to domain experts to express their domain conceptualisation and are automatically translated into an agent-based model that is automatically simulated and results extracted. While Parunak provides examples of domain-modelling DSMLs, there is no support for explicitly capturing a fitness-for-purpose argument over the agent-based model.

Muñoz et al. [34] describe the use of UML and some bespoke modelling languages for capturing the structure and behaviour in an agent-based model of autonomically driving vehicles and pedestrians, developed for purposes of training and testing autonomic-driving AI systems. This work proposes some domain-modelling DSMLs—for example, the authors provide a DSML for specifying different levels of uncertainty of agents about road conditions and other agents based on levels of visibility, etc. At the same time, however, the approach requires some very low-level ABM features to be modelled, too. In particular, the user is required to produce a UML model that captures the simulation-implementation classes down to the level of tick() methods. Thus, the approach has aspects that are more low-level than the agent-modelling approaches in [13, 16, 21, 41, 42] mixed with approaches that are at a similar abstraction level to the languages introduced in [37] and proposed in our vision. Explicit models of fitness-for-purpose arguments are again not considered in [34].

Barat et al. [7] describe OrgML, a domain-specific modelling language for describing organisational structures, levers and outcomes and translating these into agent-based models and simulations. This is similar to the left column in Fig. 2 in that OrgML is a domain-specific domain-modelling language and the underlying actor language is an agent modelling language.

In the wider simulation community, it has been recognised that the definition of simulation experiments follows similar patterns across domains and can be captured through a domain-specific language. SESSL [12] is one such language, which allows the specification of simulation experiments and the processing of results—for example for visualisation—in an internal DSL embedded in Scala. SESSL focuses, in particular, on providing a unifying mechanism for specifying simulation experiments that can be executed on different simulation platforms. However, the specification remains at what in our vision we refer to as the platform-specific level: domain-specific or agent-specific concepts are not used in the DSL.

ESS [30] is an external DSL for the specification of simulation experiments. Here, the focus is less on supporting different simulation ‘back ends’ and more on the support for different statistical analyses. However, as with SESSL, ESS also remains at what we would refer to as the platform-specific level and does not provide support to domain stakeholders.

There has been extensive research interest in the validation of agent-based models, and some of this work has started to explore approaches for ensuring the statistical soundness of the conclusions drawn from simulations. For example, Spartan [5] is a toolkit supporting statistical analysis of simulation runs to alleviate aleatory uncertainty and undertake sensitivity analysis, focusing on numerical output data. Similarly, \(\hbox {MC}^2\hbox {MABS}\) [20] provides support for drawing statistically sound conclusions based on temporal-logic queries about patterns of agent behaviour. However, neither of these approaches are currently integrated with the initial domain model; they remain at the platform-specific level. As a result, they require extensive manual translation and integration effort from domain experts and software engineers to be applied to a new simulation context.

ProMoBox [31] is an interesting example of a domain-specific query language outside the simulation world. The authors observe that domain experts struggle to work with linear temporal logic expressions and relating them to their domain concepts. To address this challenge, they provide a mechanisms for generating domain-specific query languages by lifting the structure of LTL expressions to the domain level and replacing predicates with patterns in the original domain language. This is achieved via a domain-specific meta-modelling language [53].

We are not aware of many works on results modelling. Many simulation tools allow the specification of what parameters and attributes should be monitored and exported as results of a simulation run. SESSL [12] discussed in more detail above, includes concepts for describing what data to extract and how to visualise it, in a manner that is independent of specific simulation tools.

Arguing for the validity of experiments and simulations is important and has been studied from different perspectives.