Uncertainty-Wise Cyber-Physical System test modeling

A CPS is defined in [7] as: “A set of heterogeneous physical units (e.g., sensors, control modules) communicating via heterogeneous networks (using networking equipment) and potentially interacting with applications deployed on cloud infrastructures and/or humans to achieve a common goal” and is conceptually shown in Fig. 2. Uncertainty can occur at the following three logical levels [7] (Fig. 2): (1) Applicationlevel, due to events/data originating from an application (one or more software components) of a physical unit of the CPS; (2) Infrastructurelevel, due to data transmission via information network enabled through networking infrastructure and/or cloud infrastructure; (3) Integrationlevel, due to either interactions of applications across the physical units at the application level, or interactions of physical units across the application and infrastructure levels. Notice that we chose the definition of CPS from [7] as it was defined in the context of our project and was further used to define the three levels of uncertainties in CPS that are modeled in this paper and conforms to the well-known definition in [1].

In our previous work [7], to understand uncertainty in CPSs, we developed a conceptual model called U-Model to define uncertainty and associated concepts, and their relationships at a conceptual level. Some of the U-Model concepts were extended for supporting MBT of all the three levels of CPS under uncertainty (Sect. 2.1). U-Model was developed based on an extensive review of existing literature on uncertainty from several disciplines including philosophy, healthcare, and physics, and two industrial CPS case studies from the two industrial partners of the U-Test EU project. In this paper, we implement U-Model as UncerTum to support the construction of test ready models with uncertainty. Details of U-Model are given in [7], and part of U-Model is provided in “Appendix A” for the purpose of keeping this paper self-contained.

UML Testing Profile (UTP) [20, 21] is a standard at Object Management Group (OMG) for enabling MBT. With UTP, the expected behavior of a system under test can be modeled, from where test cases can be derived. UTP can be also used to directly model test cases. Transformations from models specified with UTP to executable test cases can be performed using specialized test generators. Since UTP is defined as a UML profile, it is often applied to UML sequence, activity diagrams, and state machines for describing behaviors of a system under test or test cases. The key purpose is to introduce testing-related concepts [e.g., Test Case, Test Data, and Test Design Model and Model Libraries such as various types of test case Verdict (pass, fail)] to UML models for the purpose of enabling automated generation of test cases. UTP V.2 [21] is the latest revision of the UTP profile, which is conceptually composed of five packages of concepts: Test Analysis and Design, Test Architecture, Test Behavior, Test Data, and Test Evaluations. Various numbers of stereotypes have been defined for some concepts of these packages. Similar to other modeling notations, it is never been an objective to exhaustively apply all the stereotypes when using UTP V.2 to annotate UML models with testing concepts [21]. Which stereotypes to apply and how to apply them are however problem/purpose specific and should be defined by users of the profile. More information about the UTP V.2 standardization and the team can be found in [22, 23].

To enable MBT of CPSs under uncertainty, we rely on UTP V.2 to model the testing aspect of test ready models. In our context, only a subset of UTP V.2 was used.

The Belief part of UUP is one of the key components of UUP since we focus on subjective uncertainty (“lack of knowledge”), where a test modeler(s) (BeliefAgent(s), see “Appendix A”) creates test ready models of a CPS based on her/his/their assumptions (Belief, see “Appendix A”) about the expected behavior of the CPS and its operating environment. The Belief part of UUP thus defines concrete concepts to model Belief of test modelers with UML. As shown in Fig. 13, it implements the high-level concepts defined in U-Model:BeliefModel provided in “Appendix A.1” (the mapping is provided in Table 2, and further discussion is provided in Sect. 9.2.1). As shown in Fig. 13 and Table 2, five stereotypes are defined, among which «BeliefElement»  is the key stereotype associated with various UML metaclasses. This stereotype is used to signify which UML model elements are representing belief of belief agent(s). Generally speaking, any model element may represent a belief, but in the context of our work, we only extend UML metaclasses that are required to support MBT. For example, a StateMachine (subtype of metaclass Behavior) itself can be a belief element (i.e., expected state-based behavior of a CPS and its operating environment), such that «BeliefElement »  can be applied on it to characterize it with additional information such as to which extent a test modeler (stereotyped with «BeliefAgent») is confident about the state machine (i.e., beliefDegree of BeliefStatement), all uncertainties (i.e., Uncertainty) associated with the state machine, and their Measurements. In our context, we extend UML state machines; however, it is worth mentioning that «BeliefElement»  can be used, for example, with activity and sequence diagrams if needed. We intentionally kept the profile generic (e.g., by making «BeliefElement»  extend the UML metaclass Behavior) such that different MBT techniques based on different diagrams can be defined when needed.

In case that there is some evidence, e.g., existing data to support the belief, «Evidence»  can be used. It is defined to capture any evidence for supporting the definition of a measurement for an uncertainty. The stereotype extends UML metaclass Element, implying that any UML model element (e.g., Class) can be used to specify evidence, e.g., as a ValueSpecification. Each uncertainty is also associated with a set of indeterminacy sources (definition in “Appendix A”), which can be explicitly specified using «IndeterminacySource»  and classified with enumeration IndeterminacyNature (Fig. 13) as defined in “Appendix A.”

The profile also implements OCL constraints defined in U-Model. For example, as shown in Fig. 13, each beliefDegree (an instance of Measurement) of a «BeliefStatement»  must have exactly one measure associated with it, which can be specified in three different ways: a «Measure»  (via attribute measure of Measurement), DataType (via measureInDT) or Class (via measureInDTViaClass). This OCL constraint is given:

When we look at the running example, the belief agent (Fig. 10) is Man_Simula (stereotyped with «BeliefAgent») who defines three state machines: one for the alarm, one for the sensors, and one for the security system itself. As shown in Table 1, «BeliefElement»  is applied on the IntrusionOccurred transition from Normal to IntrusionDetected (Fig. 7 B.1). The belief agent of this belief element is specified as class Man_Simula (stereotyped with «BeliefAgent»  shown in Fig. 10). The belief degree of this belief element is specified as a value specification «VeryLikely»  and measured as Probability_7Scale. The belief element has one occurrence uncertainty, which is associated to «BeliefElement, Cause»  notifyIntrusion of «IndeterminacySource»  Sensor (Table 1).

The second key component of UUP is about the implementation of concepts related to Uncertainty («BeliefElement »  composed of Uncertainty in Fig. 14) and is presented in Fig. 14. In addition, each Uncertainty may have associated measurements that are captured in the Measurement part as shown in Fig. 15. In Fig. 14, the key element is Uncertainty, which is characterized with a list of attributes such as kind (typed with enumeration UncertaintyKind) indicating a particular type of uncertainties. All of the attributes (except for kind and field) are optional. For example, an uncertainty might or might not have an indeterminacy source (i.e., indeterminacySource as defined in “Appendix A”).

The U-Model concepts of Effect, Pattern, Lifetime, and Risk (“Appendix A”) can be specified with UUP in difference ways. For example, one can specify the effect (i.e., the result of an uncertainty, as defined in “Appendix A”) of an uncertainty simply as a string (attribute effect of Uncertainty). One can also create a UML model element and stereotype it with «Effect»,  and the uncertainty can then be associated with it (i.e., referredEffect). More details regarding the possible alternatives can be found in Sect. 7.

«IndeterminacySource», «BeliefStatement», Uncertainty, «Effect», and «Risk»  can be further elaborated with Measurement. A measurement can be specified in different ways: (1) as a string (attribute measurement of class Measurement), (2) as a value specification (measurementInVS), (3) as a package stereotyped with a subtype of «Measurement», and (4) a constraint stereotyped with «MeasurementConstraint». «Measurement»  is further classified into five subtypes, corresponding to the five types of elements to be measured.

«Measure»  is defined to classify different types of measures and provide users an option to denote classes and data types with concrete measure types such as «EffectMeasure». Such a stereotyped class or data type is linked via Measurement to «IndeterminacySource», «Effect», Uncertainty, «Risk»  or «BeliefStatement».

A set of OCL constraints has been implemented in UUP. One of the examples is that Element with applied «Effect»  should be referred at least once via the “referredEffect” attribute of the Uncertainty instance:

For the running example, «BeliefElement, Effect»  ActivatedAlarm is associated with Uncertainty of «BeliefStatement»  IntrusionOccurred via the “referredEffect” attribute (Table 1).

We define a small CPS Testing Levels profile with the three stereotypes (Fig. 16) to denote which model element belongs to which level of the three: «Application», “Infrastructure,” and «Integration» . This enables a test generator to use different mechanisms (if used) to control and access elements from different levels. For example, infrastructure-level variables may be accessed with different tools/APIs as compared to application-level variables. All the three stereotypes extend the UML metaclass Element, as one can apply them to a class, a state, a state machine, and many other model elements.

Note that this profile is defined for enabling MBT of CPS under uncertainty from the three logical levels, and we have no intention to break down CPS according to their system architectures by denoting physical units, sensors, network, etc. For example, class Sensor in Fig. 3 is stereotyped with «IndeterminacySource»  and «InfrastructureElement», meaning that sensors are infrastructure elements. As shown in Fig. 4, the composite structure of the system describes the interactions between the infrastructure elements (Alarm and Sensors) and the application-level elements: portSensor, portAlarm, and portSecurity, which are typed by three interfaces (i.e., ISensor, IAlarm, and ISecuritySystem) as shown in Fig. 3. This is the reason that the composite structure is stereotyped as «IntegrationElement».

We define three measure packages (Probability, Ambiguity, and Vagueness) to facilitate modeling with different uncertainty measures (Figs. 17, 18, 19; Table 3). Notice that in U-Model (Appendix A), these three concepts were defined only at a very high level; no breakdown of Probability, Ambiguity, and Vagueness was provided in U-Model. In this paper, we largely extended and implemented the detailed measures belonging to these three categories/packages, based on various theories such as Fuzzy Set and Probability Theory.

In the Ambiguity library, we define the data types corresponding to the relevant Ambiguity measures published in the literature (Fig. 17). Since these measures are well known, we do not provide further details in this paper; however, interested readers may consult the provided references listed in Table 3 for more details and the technical report corresponding to this paper [25]. The concepts of the fuzzy set theory [26] are defined in the Vagueness library (Fig. 18), and Table 3 lists the relevant references.

Various data types related to the probability are defined in the Probability library (Fig. 19). All the modeled probability distributions are well known, and thus we do not present further details in this paper. Some details of these distributions are provided in the technical report corresponding to this paper [25]. The other data types such as Percentage, Probability, Probability_Interval, and different qualitative scales of probability (e.g., Probability_4Scale) are from basic statistics and thus are not further explained.

For example, as shown in Fig. 9, the IndeterminacyDegree of Sensor_IntrusionSensed, which is used to measure the occurrence of successful sensing intrusion of Sensor, the self-transition of SensorActivated (Fig. 6), is expressed by BeliefInterval [27], which is composed of belief (97%) and plausibility (99%), which are predefined in the Ambiguity library (Fig. 17). Further details are provided in the technical report corresponding to this paper [25] for references.

This section presents Pattern Library shown in Fig. 20 to support modeling various known patterns, in which an uncertainty may occur. All the patterns except for Transient andPersistentPattern are imported from MARTE [11]. Details of these patterns can be consulted from the MARTE specification and the technical report corresponding to this paper [25]. The definition of Transient is adopted from [7], i.e., “Transient is the situation whereby an uncertainty does not last long”. Transient inherits from IrregularPattern. The newly introduced attributes are minDuration and maxDuration describing the duration for which the uncertainty lasts. The definition of PersistentPattern is adopted from [7], i.e., “the uncertainty that lasts forever”. The definition of “forever” varies. For example, an uncertainty may exist permanently until appropriate actions are taken to deal with the uncertainty. On the other hand, an uncertainty may not be able to resolve and stays forever. The duration attribute of PersistentPattern is set to “forever” to indicate that the uncertainty with this pattern stays forever until resolved. In the context of testing, “forever” may be the duration for which a test case is being executed on a CPS.

The modeling methodology is naturally organized from the viewpoints of the three types of stakeholders: Application Modeler, Infrastructure Modeler, and Integration Modeler, as shown in Fig. 21. For activities performed by each type of modelers, we distinguish them by tagging each of them (in their names) using “AP”, “IF”, and “IT”, respectively.

As shown in Fig. 21, all modelers are recommended to start from creating a package (i.e., AP1, IF1, and IT1), which is used to group and contain model elements for each respective level (DD1 in Table 4). Next, application and infrastructure modelers apply the UUP notations to model system behaviors of the application and infrastructure levels, respectively (i.e., AP2 and IF2). These two structured activities are further elaborated in Sects. 7.2 and 7.3. When these two activities are finished, integration modelers take their results as inputs and perform IT2: Model Integration Behavior. Details of this structured activity are further discussed in Sect. 7.4.

The application-level modeling activities (Fig. 22) include four sequential steps: creating application-level class diagrams (AP2.1, DD2 in Table 4), creating application-level state machines (AP2.2, DD5 in Table 4), applying CPS testing levels profile (AP2.3), and applying the UUP notations on the created class and state machines (AP2.4).

A class diagram (DD2) created for the application level captures application-level state variables (attributes), whose values either can be accessed directly or with dedicated APIs. We also model operations representing APIs to send stimuli to the CPS being tested. Also, it is important to mention that such a class diagram usually needs to specify Signal, which is a Classifier for specifying communication of send requests across different state machines. In addition, a class in the class diagram may receive signals from other classes (even across levels) that are modeled as signal reception (DD3/DD4 in Table 4). When creating a class diagram for the application level, for each class, each of its attributes captures an observable system attribute, which may be typed by a DataType in the UUP’s Model Libraries (Sect. [25]) or MARTE_Library [11]. An attribute may represent a physical observation on a device (e.g., battery status on an X4 device). Each operation of a class in a class diagram represents either an API of the application software or an action physically performed by an operator (e.g., switching on or off of an X4 device). Each signal reception represents the stimulus that can be received from a different state machine.

In a state machine (DD5-DD8 in Table 4), each state is precisely defined with an OCL constraint specifying its state invariants (DD6 in Table 4). Such an OCL constraint is constructed, based on one or more attributes of one or more classes of an application-level class diagram. Each transition in a state machine should have its trigger defined as a call event corresponding to an API or a physical action defined in the class diagrams of the application level and has its guard condition modeled as an OCL constraint on the input parameters of the transition’s trigger (DD6/DD7 in Table 4).

Next, application modelers need to apply UUP on state machines (AP2.4) to specify uncertainties and apply the UTP profile to add testing information (e.g., indicating TestItem). The application of UUP is the same for the three levels, and thus we only describe it under the integration-level modeling section (Sect. 7.4).

For the infrastructure level, a similar modeling procedure as the one defined for the application level should be followed to derive class diagrams and state machines, apply UUP and UTP (further details in Sect. 7.4), as shown in Fig. 23. One difference is that attributes of infrastructure-level class diagrams should capture observable infrastructure attributes. For example, an attribute (isIntrusionOccurred in Fig. 3) can reflect the occurrence of intrusion sensed by Sensor. Operations of infrastructure-level class diagrams represent APIs for manipulating infrastructure-level components. Regarding state machines, they should be consistent with the infrastructure-level class diagrams. In other words, states should have their invariants defined as OCL constraints based on the attributes defined in the infrastructure-level class diagrams, and transitions having their triggers defined as call events or time/change events (DD5–DD7 in Table 4).

Recall that, activity IT2 is started after class diagrams and state machines created for the application and infrastructure levels. As shown in Fig. 24, the IT2 activity starts from creating integration-level class diagrams (IT2.1) and state machines (IT2.2) and applying the CPS testing levels profile (IT2.3), followed by applying UUP and UTP.

There are different ways of defining model elements for the integration level. One way is to refine the created application and infrastructure-level state machines by directly introducing new model elements to them. For example, a state in the application level can send a Signal to the infrastructure level and vice versa. Transitions of a state machine in the application (infrastructure) level should capture triggers of type Signal Reception and effects containing Signals from the infrastructure (application) level. Another way is to keep application and infrastructure-level state machines untouched by applying a specific modeling methodology (e.g., Aspect Oriented Modeling methodologies) to specify crosscutting behaviors separately. In addition, one should also benefit from advanced features of UML state machines (e.g., concurrent state machines, parallel regions) to for example refer to existing state machines defined in the application and infrastructure levels.

Since test ready models can be created in several different ways, we propose a set of options to restrict the way, in which test modelers apply UUP. Notice that our test generators will only be able to generate test cases when one of these options is followed. The same modeling decisions (D12, D13 in Table 4) are taken for several concepts in UUP including Belief Agent, Evidence, Measurement, Lifetime, Cause, Pattern, Effect, IndeterminacySource, andRisk. All these can be simply modeled as String values. This option is informal since a test modeler is allowed to provide any String value. Second, a more formal way is to model these concepts as Fig. 10 (e.g., a Ph.D. student at Simula class for a particular BeliefAgent) with possible attributes and operations inside a dedicated package (e.g., for all BeliefAgents for a CPS under test). Followed by this, we recommend applying dedicated stereotypes (e.g., «BeliefAgent») either on classes, package, or both. The justification of these design decisions is summarized in Table 4.

Recall that the activity of applying UUP is invoked at all the three levels. We tag each type of the activities of the activity diagrams from Figs. 25, 26, 27, 28, 29, 30, 31, 32, and 33 with S, C, and A to represent structured activities, call behavior, and normal activity nodes (standard semantics as in UML). Note that these activity diagrams are developed to explain the step-wise procedure to create test ready models and themselves are not part of the test ready models. As shown in Fig. 25, applying UUP starts from applying «BeliefElement»  on any UUP allowed state machine model element. Then a modeler can specify values for the “from” and “duration” attributes of the stereotype, model belief agents, model belief degree, and/or model uncertainties (Fig. 25).

As shown in Fig. 26, there are two ways (D12, D13) to model belief agents (S1.1 and S1.2). A modeler can specify belief agents simply as one or more strings via the “beliefAgent” attribute of «BeliefElement»  (S1.1). She/he can also create a package to organize all the belief agents (S1.2). In this case, each belief agent can be modeled as a class in the package and the package is stereotyped with «BeliefAgent» . Alternatively, one can model each belief agent as a class and stereotype it with «BeliefAgent». The other option is to model each belief agent as a class and stereotype it with «BeliefAgent»  and also stereotype the package with «BeliefAgent» . When choosing to apply options 2, 3, and 4, one needs to link a created belief agent package to the agent attribute of «BeliefElement»  (S2). For example, we modeled the belief agent, Man_Simula, using Option 3 as shown in Fig. 10.

Modeling BeliefDegree is presented in Sect. 7.5.1, and modeling uncertainties are discussed in Sect. 7.5.2.

In this section, we describe the guidelines (in Fig. 35) to convert test ready models that were created based on the guidelines in the last section into executable models to facilitate validation.

As shown in Fig. 35, (1) in the CD1 activity, a test modeler can optionally specify UAL code on the model elements of classes (e.g., specifying default values for attributes and implementing bodies of operations). For example, the UAL code of the timeout attribute of SecuritySystem (Fig. 3) is false, i.e., its default value. (2) As shown in the CSD2 activity in Fig. 35, a test modeler should create a composite structure diagram (DD10, DD11 in Table 4) to model the internal structure of the Classifier (e.g., a physical unit) and interactions with other associated Classifiers (other physical units) or the operating environment of the CPS. For example, the portSecurity port of SecuritySystem (Fig. 4) specifies an interaction point, through which SecuritySystem can communicate with its surrounding environment or with Alarm or Sensor. The provided interface of the portSecurity is ISecuritySystem, which enables the reception of the IntrusionOccurred signal and other SignalReceptions in Fig. 3. Two connectors between portSecurity and portSensor (Fig. 4) are created to enable two-way communications between SecuritySystem and Sensor. (3) As shown in SM3 in Fig. 35, a test modeler can specify UAL code on the effect and entry/do/exit activity of a state in a state machine to implement a specific activity, especially the ones that involve sending signals across state machines. For example, the effect of the A transition in Fig. 6 is implemented with UAL as portSensor.send(new IntrusionOccurred(this.ID)). Since portSensor and portSecurity are connected (Fig. 4) and provided interface ISecuritySystem of portSecurity has the capability to receive the IntrusionOccurred signal (in Fig. 3), the B.1 / B.2 transition (in Fig. 7) can be triggered when SecuritySystem receives the IntrusionOccurred signal through portSecurity.

This section represents our recommendations (Table 5) to fix test ready models, once these are executed and problems are observed. For example, one observed problem is that the IntrusionOccurred signal event cannot be triggered (Fig. 7) even when it was sent out (O4, Table 5). One possible reason is that the IntrusionOccurredSignalReception in the ISecuritySystem interface of SecuritySystem is missing (SA7).

As previously discussed, the project has two official CPS case study providers. First, the first one is from the healthcare domain, which is about GeoSports (GS) provided by Future Position X (FPX) [9] Sweden. This case study includes attaching devices to Bandy¹ players that record various measurements (e.g., heartbeat, speed, location) periodically. These measurements are communicated during a Bandy game via a receiver station to the sprint system, where coaches can monitor them at runtime. In addition, these measurements can also be used offline for analyses, for example, aimed at improving the performance of an individual player or a team. To test this CPS in a laboratory setting without real players, Nordic Med Test (NMT) [15] provides a test infrastructure to execute test cases. The second case study is about Automated Warehouse (AW) provided by ULMA Handling Systems [10], Spain. ULMA develops automated handling systems for worldwide warehouses of different natures such as Food and Beverages, Industrial, Textile, and Storage. Each handling facility (e.g., cranes, conveyors, sorting systems, picking systems, rolling tables, lifts, and intermediate storage) forms a physical unit, and together they are deployed to one handling system application (e.g., Storage). A handling system cloud supervision system (HSCS) generally interacts with diverse types of physical units, network equipment, and cloud services. Application-specific processes in HSCS are executed spanning clouds and CPS requiring different configurations. This case study implements several key industrial scenarios, i.e., introducing a large number of pallets to the warehouse, transferring the items by Stacker Crane. To test these scenarios, ULMA [10], and IK4-Ikerlan [16] developed and provided relevant simulators and emulators. Further details on the case studies can be consulted in [54].

In addition, we used a real-world case study of embedded Videoconferencing System (VCS) developed by Cisco Systems, Norway. Simula has been collaborating with Cisco since 2008. As part of our long-term collaboration under the umbrella of Certus Center [55], we have access to real VCS systems. We created test ready models for one of the real CPSs ourselves without involving Cisco, based on the previous work [18] of the second author of this paper. The fourth case study is a modified version of the SafeHome case study provided in [19]. This case study implements various security and safety features in smart homes including intrusion detection, fire detection, and flooding.

The development and validation procedure of UncerTum and test ready models is summarized in Fig. 36, which involves four stakeholders: (1) Simula Researchers (including the first three authors of this paper) play the key role of developing UncerTum and creating test ready models; (2) Use Case Providers (i.e., FPX and ULMA) provided uncertainty test requirements and real operational data from previous Bandy games in the case of GS, and manually checked the conformance of the developed test ready models to their corresponding uncertainty test requirements; (3) Test Bed Providers (NMT and ULMA/IK4-Ikerlan) provide physical and software infrastructures (including test APIs) to automate the execution of test cases and manually checked that the test ready models conform to the provided implementation of the test APIs; (4) Tool Vendor is responsible to integrate UncerTum and the proposed test case strategies to facilitate test case generation and execution. Please note that the UncerTum methodology reported in this paper is fully developed by Simula Research Laboratory, which is generic and therefore can be applied to test CPS at the three levels. Notice that it is also possible to develop different modeling methodologies than the one proposed in this paper, e.g., one such instance is reported in [56] for the application level by one of the our project partners. Such modeling methodologies can be potentially compared when needed in the future.

The development of UncerTum took place incrementally (Activities A1 and A2 in Fig. 36). First, UncerTum (A1) was developed by researchers based on U-Model and MARTE, in parallel to creating the initial test ready models (B1) for VCS, SafeHome, GS, and AW with this initial version of UncerTum. For GS and AW, uncertainty test requirements were provided by FPX and ULMA; for VCS, we had some requirements available to us from our previous work [18]; SafeHome is from the literature. Based on our experience of creating these test ready models, we further refined UncerTum (A2) and as a result UncerTum V.1 was developed. This was in turn used to further refine the initial test ready models (B2). At this point, both versions of the test ready models and UncerTum were refined once again by researchers. As a result, Test Ready Model V.1 and UncerTum V.2 were produced (Fig. 36).

UncerTum V2 and Test Ready Models V1 were then used in the modeling technique workshop (two days) conducted with the industrial use case providers (FPX and ULMA), test bed providers (ULMA/IK4-Ikerlan and NMT), tool vendor (Easy Global Market (EGM))[14], and two other research partners who focused on their own modeling methodologies and models. During the workshop, UncerTum and test ready models were presented to the participants of the workshop and their feedback was collected. In addition, the test API documentation was also presented. Based on the feedback and test APIs, a plan was devised to further refine the test ready models after the workshop. The key output of the workshop from our side was UncerTum (V.3), which is presented in this paper. Based on the feedback and test API documentation, we refined the test ready models (i.e., Test Ready Models V2 in Fig. 36) after the workshop. In parallel, the test bed providers started to develop the test infrastructures to enable the execution of test cases, which is not in the scope of this paper.

To further refine the test ready models, another two workshops were conducted: one for AW and one for GS arranged by the respective industrial partners. The first workshop took place at IK4-Ikerlan [16], where Simula researchers and ULMA participated and the workshop lasted for three days. During the workshop, detailed uncertainty test requirements, test ready models, and detailed implementation of test execution were discussed. The second workshop lasted for two days and took place at the NMT’s site in Sweden. FPX, EGM, and other research partners participated. Similar discussion as with ULMA took place with FPX/NMT. In addition, EGM presented their tool (CertifyIt) and their plans to integrate UncerTum and further implementation of test execution APIs. The outputs of these workshops were Test Ready Models V3 as shown in Fig. 36. Finally, we validated Test Ready Models V3 using IBM RSA Simulation Toolkit (see Sect. 9.2.3 for results).

Descriptive statistics of the test ready models developed for the four case studies are provided in Table 6. For each case study, (1) the number of modeled UML diagrams is presented in the first row, (2) the second, third, and fourth rows represent the number of application, infrastructure, and integration-level elements, respectively, (3) the last row shows the number of uncertainties and indeterminacy sources modeled for each case study. Notice that these statistics provide an indication of the complexity and scale of the developed test ready models.

Based on the results presented in Sect. 9.2, we conclude our findings as follows: (1) With UncerTum, we were able to model all the identified uncertainties in the four case studies. Such modeling suggests that UncerTum is sufficiently complete to create test ready models of CPS with explicit consideration of various types of uncertainties to support testing of CPS in the presence of such uncertainties; (2) in terms of estimating the effort required to apply the UUP stereotypes and Model Libraries, we conclude that we need to apply them to on average 16.25% of model elements (Table 8). When estimating effort in terms of time, we observed that we needed on average additional 18.5% of time to apply UUP (Table 9); (3) with our model execution-based model validation, we managed to identify and fix in total 264 problems across the four case studies (Table 10) which are necessary before test case generation as otherwise generated test cases would have been incorrect.

In terms of evaluation, we would like to highlight the fact this section reported a preliminary evaluation of UncerTum from various perspectives. A more thorough evaluation would require conducting surveys and questionnaires from the participants from our industrial partners to solicit their views about the modeling methodology in terms of, for example, understandability and usability. We plan to conduct such evaluation at the end of our project when the complete results have been transferred to the industry partners with the participants who are not the co-authors of this paper in order to obtain unbiased feedback about UncerTum.

We would also like to mention that UncerTum cannot be used to model detailed continuous behaviors of a CPS, to support, for example, analyses during the system design and analysis phase or to generate code. UncerTum only supports test modeling for enabling the generation of executable test cases. Such types of models are less detailed as compared to models used for code generation or models for design time analyses. This is due to the fact that testing is always concerned with sending a stimulus to the system and observing whether the system transits to a correct state because of the stimulus according to the expected behavior specified in a test ready model, developed for the system.