8. Verification and Validation for CIPRNet

Within the GM-VV, M&S systems are considered to be systems of systems that have a lifecycle and are subject to system engineering practices. Moreover, models and simulations are considered to be part of a larger system in which they are used. From this perspective, M&S is a systems engineering specialization. V&V is an intrinsic part of the systems engineering process [6‐9]. Therefore, the GM-VV considers the V&V of M&S as a specialization of systems engineering V&V. Hence, the GM-VV can be integrated with, complement or extend the V&V processes within such existing systems engineering methodologies or standards.

The basic premise of the GM-VV is that models and simulations are always developed and employed to fulfil the specific needs of their end users (e.g., trainers, analysts, decision makers). Modelling and simulation is thus considered to be a problem solving process that transforms a simple statement of an end user’s need into an M&S-based solution for the problem implied in the need. The GM-VV assumes that V&V always takes place within such larger context. This context is abstracted by means of defining four interrelated worlds (Fig. 5). Together, these four worlds define a generic lifecycle and process view of M&S-based problem solving. A view that serves as a common basis, in which V&V for M&S (e.g., concepts, principles, processes, products, techniques) can be understood, developed or applied.

These four worlds can be described as follows:

When the M&S problem solving process described by the four-worlds view is properly executed, the resulting solution should satisfy the originally identified needs with a minimal level of (use) risk in the Real World.

The M&S system, M&S requirements, M&S results and other development artefacts (e.g., conceptual model, software design, code) are thus always directed toward contributing to and satisfying the Real World operational needs. The degree of success of such M&S in satisfying these needs depends on how well they are specified, designed, developed, integrated, tested, used, and supported. These M&S activities require the contribution of individuals or organizations that have a vested interest in the success of the M&S asset, either directly or indirectly. An individual or organization with such interest is referred to in GM-VV as a stakeholder. Stakeholders can play one or more roles in each of the four worlds such as M&S user/sponsor, supplier, project manager, software developer, operator, customer, or subject matter expert (SME). Depending upon their role, stakeholders may hold different responsibilities in the M&S life-cycle processes, activities or tasks.

Within the four-world context, stakeholders exist who are responsible for making acceptance decisions on the use of M&S. Within the GM-VV, these stakeholders are referred as V&V User/Sponsor. In this context the V&V User/Sponsor could be an M&S User/Sponsor, Accreditation Authority or any other domain specific role that uses the outcomes of the V&V. V&V Users/Sponsors face the problem of having to make a judgment on the development and suitability of the M&S system or results for an intended use. The key issue here is that it is not possible to demonstrate with absolute certainty that the M&S system or results will meet the Real World needs prior to its actual use. Consequently, there is always a probability that the M&S-based solution is not successful when used (i.e., fails). Such a failure would result in an undesirable impact (i.e., a risk) on the operational environment. Therefore, an M&S system or result is only acceptable to the V&V User/Sponsor if he or she has sufficient confidence that the use of an M&S system or result satisfies the Real World needs without posing unacceptable risks (e.g., costs, liabilities). This M&S acceptability is something relative to different V&V Users/Sponsors: what is acceptable to one V&V User/Sponsor may not be acceptable for another. The V&V User/Sponsor’s decision-making process therefore requires appropriate evidence-based arguments to justify his or her acceptance decision.

The basic premise of GM-VV is that V&V are performed to collect, generate, maintain and reason with a body of evidence in support of the V&V Users/Sponsors acceptance decision-making process. Here, validation is referred to as the process that establishes the V&V User/Sponsor’s confidence as to whether or not they have built or procured the right M&S system or result for the intended use (i.e., M&S validity). In other words “Did we build the right M&S system?”. To ensure that the M&S system or results at delivery can be demonstrated to be valid, it is necessary to ensure that the M&S system is built and employed in the right manner. Here verification is referred to as the process of establishing V&V User/Sponsors confidence in whether the evolving M&S system or result is built right (i.e., M&S correctness). In other words “Did we build the M&S system right?”. The GM-VV considers V&V as a specific problem domain of M&S with its own needs, objectives and issues. This domain is referred to as the V&V World (Fig. 6).

The V&V world groups the products, processes and organizational aspects that are needed to develop an acceptance recommendation that can be used by the V&V User/Sponsor in his or her acceptance decision procedure(s). This recommendation included in a V&V report is the key deliverable of a V&V effort and contains evidence-based arguments regarding the acceptability of an M&S system or results. Here the GM-VV premise is that the acceptance decision itself is always the responsibility of the V&V User/Sponsor and decision procedure(s) may involve trade-off aspects beyond the V&V effort scope.

The development of an acceptance recommendation in the V&V world is driven by the V&V needs that are traceable to the V&V User/Sponsor’s acceptance decision or procedure(s) needs (e.g., budget, responsibilities, risks, liabilities). Therefore, the extent, rigor and timeframe of a V&V effort depend on these needs. Depending on these needs, the V&V effort could span the whole or specific M&S lifecycle phase of the four worlds; could focus on one specific or multiple (intermediate) M&S products; and should match the development paradigm that was used (e.g., waterfall, spiral). Each case may require a separate acceptance recommendation with its own scope and development timeline. Moreover, the way the V&V effort interacts with the four M&S-based problem worlds also varies from case to case. These mutual dependencies are depicted in Fig. 6 with bidirectional arrows that interface the V&V world with each of the four M&S-based problem solving worlds. Two classical types of V&V that can be identified based on the time frame of their execution are [6, 10‐12]:

The GM-VV supports both V&V time frames but is not limited to these distinct types. A V&V effort can be post hoc, concurrent, iterative, recursive or even be a recurrent effort in the case where legacy M&S products are updated or reused for a different intended-use.

The objective of a V&V effort is to develop evidence upon which an acceptance recommendation is based. This V&V objective is articulated as an acceptance goal. This high-level goal should be translated into a set of concrete and assessable acceptability criteria for the M&S system or result(s). Relevant and convincing evidence should then be collected or generated to assess the satisfaction of these criteria. When it is convincingly demonstrated to what extent the M&S system or result(s) does or does not satisfy all these acceptability criteria, a claim can be made on whether or not the M&S system or result(s) is acceptable for its intended use (i.e., acceptance claim).

The GM-VV identifies three types of M&S properties for which acceptability criteria could be set (Fig. 7):

These three types of M&S properties include but not limited to capability, accuracy, usability and fidelity [13, 14]. To make an acceptance decision, the V&V User/Sponsor needs to know whether the M&S system or results are (un)acceptable, as well as the evidential value of this acceptance claim (i.e., strength). The required evidential strength to establish sufficient trust in such a claim depends on the use risks and the V&V User/Sponsor responsibilities (i.e., liability). The convincing force that can be placed on such a claim depends on the quality of the whole V&V effort. For this purpose, the GM-VV identifies quality properties that can be associated with identifying and defining the acceptability criteria; and developing convincing evidence for demonstrating their satisfaction (Fig. 7).

Typical examples of V&V quality properties are the completeness, correctness, consistency, unambiguous and relevance of the acceptability criteria or their supporting items of evidence. In the process of collecting or generating evidence, quality properties could comprise independence of applied V&V techniques or persons, knowledge gaps and uncertainties of referent data for the simuland [15], skill level of V&V personnel, and reliability and repeatability of V&V techniques. Relevance and warrants for any assumption made in a V&V effort could also be addressed in the form of quality properties.

The defined acceptability criteria, the collected evidence and assessment of the satisfaction of these criteria are the basis for developing the arguments underlying the acceptance claim. This acceptance claim provides the V&V User/Sponsor with a recommendation regarding the acceptability of the M&S system or result for the intended use. In practice, an acceptance recommendation is not necessarily just a yes or no claim, in the sense that an M&S system or results can be accepted only if it meets all of the acceptability criteria and cannot be accepted if it does not. Meeting all the acceptability criteria means the claim can be made that the M&S system or result should be accepted to support the intended use without limitations. In case not all acceptability criteria are met, alternative weaker acceptance claims with underlying arguments can be constructed. Such alternative acceptance claims could, for example, provide recommendations regarding conditions or restrictions under which the M&S system or result can still be used (i.e., limit the domain of use); or on modifications that, when implemented, will lead to an unconditionally acceptable M&S system or results for the intended use. Another rationale for alternative acceptance claims is when convincing or sufficient evidence is lacking (e.g., access to data prohibited, or referent system unavailable for testing). In any case, an acceptance recommendation always requires well-structured supporting arguments and evidence for the V&V User/Sponsor to make the right acceptance decision. Depending on the identified M&S use risk, the V&V User/Sponsor can also decide not to take any actions when not all acceptability criteria are met by the M&S system. In that case, the V&V User/Sponsor simply accepts the risks associated with the M&S system use.

Developing an acceptance recommendation that meets the V&V User/Sponsor needs usually involves the identification and definition of many interdependent acceptability criteria, particularly for large-scale and complex M&S systems or for M&S-based solutions used in safety–critical, real-world environments. Demonstrating the satisfaction of acceptability criteria requires evidence. Collecting the appropriate evidence is not always simple and straightforward, or even not always possible due to various practical constraints (e.g., safety, security, costs, schedule). In many cases, the collected evidence comprises a large set of individual items or pieces of evidence that may be provided in different forms or formats, and may originate from various sources (e.g., historical, experimental data, SME opinion). Moreover, the strength of each item of evidence may vary and the total set of collected evidence may even contain contradicting items of evidence (i.e., counter evidence). The quality of this effort determines the value of an acceptance recommendation for the V&V User/Sponsor. Therefore, the arguments underlying an acceptance recommendation should be developed in a structured manner using a format where the reasoning is traceable, reproducible and explicit. Alternative approaches to implement such reasoning exist and may be incorporated within the GM-VV technical framework to tailor it the specific needs of an M&S organization or domain. An example of such an approach is the V&V goal-claim network approach (Fig. 8). A V&V goal-claim network is an information and argumentation structure rooted in both goal-oriented requirements engineering and claim-argument-evidence safety engineering principles [16‐19].

Figure 8 provides an abstract illustration of a V&V goal-claim network. The left part of the goal-claim network is used to derive the acceptability criteria from the acceptance goal; and deriving solutions for collecting evidence to demonstrate that the M&S asset satisfies these criteria as indicated by the top-down arrows. The acceptance goal reflects the V&V needs and scope (e.g., simuland, intended use). Evidence solutions include the specification of tests/experiments, referent for the simuland (e.g., expected results, observed real data), methods for comparing and evaluating the test/experimental results against the referent. Collectively, they specify the design of the V&V experimental frame used to assess the M&S system and its results. When implemented, the experimental frame produces the actual V&V results. After a quality assessment (e.g., for errors, reliability, strength), these results can be used as the items of evidence in the right part of the goal-claim network. These items of evidence support the arguments that underpin the acceptability claims. An acceptability claim states whether a related acceptability criterion has been met or not. Acceptability claims provide the arguments for assessing whether or to what extent the M&S system and its results are acceptable for the intended use. This assessment, as indicated by the bottom-up arrows in Fig. 8, results in an acceptance claim inside the V&V goal-claim network. As such a V&V goal-claim network encapsulates, structures and consolidates all underlying evidence and argumentation necessary for developing an appropriate and defensible acceptance recommendation. The circular arrows in Fig. 8 represent the iterative nature of developing a V&V goal-claim network during planning, execution and assessment phases of a V&V effort.

In order to facilitate efficient and high quality V&V, the V&V effort inside the V&V world should be executed in a controlled and organized way. The basic premise of the GM-VV is that the acceptance recommendation for an M&S asset is developed and delivered by means of a managed project. Moreover, GM-VV assumes that V&V is conducted by a person, a team of people or a dedicated organization with assigned responsibilities, obligations and functions. Therefore, GM-VV identifies three organizational levels at which V&V efforts can be considered. In order of the lowest to the highest organizational level these levels are:

The core GM-VV concept on the V&V project level is the concept of a managed project. A V&V project can be viewed as a unique process comprised of coordinated and controlled activities that address: V&V effort planning in terms like cost, timescales and milestones; measuring and checking progress against this planning; and selecting and taking corrective actions when needed. A V&V project could be a separate project alongside the M&S project of which the M&S asset is part, or be an integral part of this M&S project itself (e.g., subproject, work package). A separate V&V project is particularly relevant in the case when a level of independence must be established between the M&S development and V&V team/organization. On the V&V project level, GM-VV also provides derived concepts such as a V&V plan and report to manage the technical V&V work.

For CIPRNet all three levels are important. For CI it is important to have a good set of tools and techniques to do the technical V&V activities. Since with the application of M&S systems for serious CI application there is always M&S Use Risk involved, for each project run by the to be established EISAC (European Infrastructures Simulation and Analysis Centre), V&V activities should be executed. A project approach is suited for that. Doing V&V from within EISAC means that EISAC should have support for the V&V activities at the highest level: the enterprise level.

The core GM-VV concept on the V&V enterprise level is the concept of an enterprise entity. A V&V enterprise entity can be viewed as an organization that: establishes the processes and lifecycle models to be used by V&V projects; initiates or defers V&V projects; provides resources required (e.g., financial, human, equipment); retains reusable knowledge and information from current V&V projects; and leverages such knowledge and information from previous V&V projects. The V&V enterprise provides the environment in which V&V projects are conducted. GM-VV defines two types of enterprise entities:

A V&V agreement is arranged between a V&V client and V&V supplier to provide products and/or services that meet the V&V client’s needs. Both these V&V entities could be organizations (e.g., companies) separate from the organization that develops or acquires M&S or it could be different units (e.g., department, division, group) within a single M&S supplier or client organization. Typically, a separate V&V supplier is an organization that has the provision of independent V&V products and services to external V&V clients as its core business. Though depending on their business model, an M&S supplier or client organization could have their own V&V supplier entity that may provide V&V services and products to internal and external V&V clients alike.

An independent V&V (IV&V) authority is often described as an organization or a person that is employed to conduct V&V, independent of the developer’s team or organization [6, 10, 12]. The need for IV&V is mostly driven by:

In practice however, it is highly incumbent upon the V&V User/Sponsor acceptance decision needs and complexity of the M&S system as to which parts and to what extent V&V should be conducted in an independent manner. Therefore, the GM-VV adopts a sliding scale of independence for V&V [15], which can be selected accordingly to match the V&V needs. The justification and selection of a proper level of independence is supported within GM-VV through the use of the V&V argumentation network. Within this sliding scale for independent V&V, certification and accreditation can be located in the right part of the scale (Fig. 9).

V&V of M&S is an information and knowledge intensive effort. In particular, during the V&V of large scale, distributed or complex M&S applications, care must be taken to preserve or reuse information and knowledge. Therefore, GM-VV applies the memory concept on both the V&V project and enterprise levels. A memory is viewed as a combination of an information and knowledge repository and a community of practice [20]. The repository is a physical place where information, knowledge objects, and artefacts are stored. The community of practice is composed of the people who interact with those objects to learn, understand context and make decisions.

The V&V project memory provides the means to manage information and knowledge produced and used during the lifetime of an individual V&V project. V&V is often an iterative and recurrent process linked to an M&S system’s life-cycle, hence V&V products for an M&S system may have different configurations. Therefore, a V&V project memory may also retain records on possible different V&V product configurations. The V&V enterprise memory retains the total body of information and knowledge from past and current V&V projects to sustain and support the cost-effective execution of future V&V projects. Such reusable information could be, for example, M&S technology or domain specific recommended practices, acceptability criteria, V&V goal-claim network design patterns, V&V tools and techniques, or policies and standards. On a more strategic level, a V&V enterprise memory could retain information and knowledge on V&V project costs and maturity as well.

As described above it is advised to use a decomposition of the top goal into smaller and smaller goals up to the point that a test can be devised that is within resources and is likely to deliver suitable evidence. During the balancing tailoring during the execution of the V&V work priorities need to be determined. These priorities together with the resources available are used to decide which goals will be further expanded and which will be left undeveloped. The basis for that decision and thus for the prioritization is risk. What is needed is to determine the contribution of a goal to the overall M&S use-risk. If a goal has a high contribution of risk it must be taken into consideration in the V&V work. If it has a very low contribution it can. In that case it should be explicitly be recorded that that goal is not used in the rest of the V&V work such that at the end a feeling for how complete the V&V work is can be obtained.

An evidence solution for a goal with a (relatively) high contribution to the overall risk should likely result in a high confidence in the evidence. For a goal with a low contribution to the M&S use-risk risk it may be sufficient to have evidence that contains some uncertainty, i.e. if the evidence is just an indication that the goal is met it may already suffice.

To find the contribution to the overall M&S use risk for a node it is necessary to make a risk decomposition in the same way as the decomposition of the Acceptance Goal, see Fig. 8. In practice it is difficult to make an exact risk decomposition, therefore it is advised to use a somewhat simpler approach as indicated in Fig. 11. The red stands for high contribution to the overall M&S use risk, orange for medium contribution and green for low contribution. During the decomposition nodes with a low contribution to the overall use risk may be left undeveloped. At the bottom of Fig. 11 the contribution to the risk is an indication of how convincing the evidence should be which is important for specifying which type of tests are required.

If after evidence collection it turns out not all goals are met, the contribution to the overall risk may be used during the acceptance decision to decide what to do. If it concerns a node with low contribution to the overall M&S use risk, it may be decided to leave things as is and accept the small risk. If it is goal with a medium or high contribution to risk it can be decided to either change the M&S system such that the identified problems are corrected, or the purpose for which the user intends to use the M&S System should be made smaller such that the current state of the M&S System will be fit for purpose.