An experimental scrutiny of visual design modelling: VCL up against UML+OCL

The paper’s contributions are as follows:

The remainder of this paper starts with some background (Section 2) focussed on VCL and diagrammatic description. This is followed by the experiment’s scope (Section 3), design (Section 4) and materials (Section 5). The paper then talks about the pursued statistical analysis (Section 6), the experiment’s participants (Section 7) and the results of the controlled experiment (Section 8). Finally, it discusses threats to the validity of the paper’s results (Section 9), presents related work (Section 10) and draws its conclusions (Section 11).

When solving problems, humans use both internal representations, stored in their brains, and external representations, recorded on paper or some other medium (Larkin and Simon 1987). For cognitive scientists, the form of external representations matters to the agents’ performance in various cognitive tasks, such as problem-solving or decision-making (Larkin and Simon 1987; Zhang 1997); form determines what information can be perceived, what cognitive processes can be activated and what can be discovered. According to Zhang (1997), “external representations are not simply inputs and stimuli to the internal mind; rather, they are so intrinsic to many cognitive tasks that they guide, constrain and even determine cognitive behaviour”.

Humans favour pictorial representations (Pinker and Freedle 1990; Larkin and Simon 1987; Golkasian 1996; Goolkasian 2000; Harel 1988; Chen 2004). Larkin and Simon’s study (1987) show that text and diagrams containing the same information are not necessarily equivalent with respect to the processing required to extract information, highlighting that diagrams facilitate perceptual inferences — some are called free rides because they are easily perceptible (Shimojima 1996). Subsequent studies (Golkasian 1996; Goolkasian 2000) have replicated the findings of Larkin and Simon (1987), highlighting a picture advantage. However, the superiority of pictures should not be taken for granted (Larkin and Simon 1987; Petre 1995).

The practical relevance of diagrams is endorsed by their ubiquity in engineering (Ferguson 1977; 1992). Visual thinking is seen as intrinsic to engineering; thinking, designing and communicating with pictures are recognised as essential engineering activities. Like in traditional engineering, diagrams became an integral part of software engineering (Moody 2009). Diagrams in all forms and shapes, from formal representations to informal and ephemeral sketches, constitute a prominent means of software engineering expression. Unlike traditional engineering, however, software diagrams are not tied to the physical shape of any designed artefact, which entailed greater freedom regarding diagrammatic shapes and forms (Blackwell et al. 2001).

Unsurprisingly, software engineering visual languages have been advocated for decades as means to facilitate human communication and problem solving (Harel 1988). Visual languages, such as Harel’s statecharts (1987), UML and SysML, are widely taught. The UML development is seen as a pivotal in software engineering, providing a common unifying language that the field had never had before. Although diagrams favour perceptual inferences, this does not entail cognitive effectiveness (Larkin and Simon 1987; Petre 1995) as diagrams need to be designed to exploit the benefits of visual representations (Larkin and Simon 1987; Moody 2009). This is where mainstream visual languages lag behind; the UML, for instance, is criticised for breaching many rules of visual language design (Moody 2009; Moody and van Hillegersberg 2009).

VCL follows the approach to modelling of UML and many of UML’s predecessors (Rumbaugh et al. 1991; Booch 1994; Wieringa 1998) based on an object oriented (OO) style of description, which sees a software system as a collection of data with associated behaviour inter-operatable from the environment. Another VCL pillar is discrete mathematics and set theory, the foundation of languages such as Z (Spivey 1992; Woodcock and Davies 1996; ISO 2002) and B (Abrial 1996).

VCL is introduced here with an example of a secure bank whose complete VCL model is given in Amálio (2011). A banking system manages customers, accounts and transactions, and needs to be made secure. Figures 1 and 2 present a few model excerpts that illustrate VCL.

VCL organises models around inter-dependent packages using ideas from aspect orientation (Amálio et al. 2010). Figures 1 and 2 present diagrams of two different packages. Package diagrams (PDs) define packages (represented as clouds) and their dependencies to other packages. PD of Fig. 1a says that package Bank imports sets from CommonTypes. Package CommonTypes provides definitions common across the model; Bank focuses on those concerns specific to banking; other packages in Amálio (2011) address security concerns and the modular weaving of security and banking.

Structural diagrams (SDs), VCL’s UML class diagram equivalent, express domains of discourse in state space (data or conceptual) models by representing entities of interest as sets (round contours). SD of Bank (Fig. 1b) has class sets for banking entities, namely, customers, accounts and transactions, and uses value sets from SD of CommonTypes (Fig. 1c — e.g. CT::Name. Objects and values are depicted as rectangles. Sets for dates and times of Fig. 1c use VCL’s predicate language to define the novel derived sets. Set Month, for instance, is defined as the natural numbers (set Nat) from 1 to 12 in a graphical depiction of a set comprehension — Month = {n : Nat | n ≥ 1 ∧ n ≤ 12}. The arrows emanating from Nat (predicate edges) refer to the source and are combined through conjunction.

State spaces are subject to constraints (or invariants) that must be respected in all states of the system. Unlike UML, VCL identifies invariants in the data model — a SD defines a state space made up of state structures and their constraints — as assertions (named elongated hexagons) defined in assertion diagrams (ADs). In Fig. 1b, assertions HasCurrentBefSavings, SavingsArePositive and CorporateHaveNoSavings, defined in ADs of Fig. 1b, e and f, embody relevant invariants: customers must hold a current account prior to opening a savings account (HasCurrentBefSavings), savings accounts must be positive (SavingsArePositive) and corporate customers must not hold savings accounts (CorporateHaveNoSavings).

Whilst SDs focus on static aspects, behaviour diagrams (BDs) concentrate on dynamics (or behaviour). BDs provide a map over the constituent units of a package’s behaviour defined in separate diagrams. BDs units of Bank (Fig. 2a) include the operations available to the environment, namely, create customer, open account, deposit, withdraw, delete account, view an account’s balance, get accounts in debt, and get accounts of a customer.

In BDs, operations that modify state are represented as contracts (double-lined elongated hexagons); those that observe (or enquire) state as assertions (single-lined hexagons). Global operations (visible to the environment) stand alone; local operations (invisible to the environment) are placed inside the contours of their class sets. Local modifier operations can either create new class objects (constructors, symbol \(\mathbb {N}\)), update existing objects (symbol \(\mathbb {U}\)) or delete class objects (symbol \(\mathbb {D}\)). For instance, the global OpenAccount does all that is involved in opening actual bank accounts and the New operation inside Account, a sort of sub-operation, creates new account objects only. Modifier operations are defined in contract diagrams (CDs); operations CreateCustomer, Customer.New, AccWithdraw and Account.Withdraw of Fig. 2a are defined in CDs of Fig. 2b, c, d and e, respectively. Observe operations are defined in ADs; ADs of Fig. 2h and g define the local Account.GetBalance and the global AccGetBalance of Fig. 2a; AD of Fig. 2f defines GetAccountGivenAccNo imported in Fig. 2e and g.

ADs and CDs comprise a box with an identifier, followed by compartments for declarations (top) and predicate (bottom) which may be further split in two. ADs have one predicate compartment as only one set of states is being referred to; CDs have two predicate compartments corresponding to the states of pre- and post-condition (to the left and right, respectively). Declaration compartments include relevant variables (either internal or of inputs and outputs), and imported assertions and contracts. Predicate compartments are made-up of graphical formulas read from top to bottom and combined using conjunction. Two kinds of formulas are supported: logic- and set-based. Logic formulas, read from left to right, resemble their textual counter-parts. Set formulas start from some inner graphical expression and grow outwards.

AD SavingsArePositive (Fig. 1e) expresses a local invariant of Account. It has no declarations; the predicate contains a pictorial propositional logic formula made-up of individual atomic statements involving predicate edges that are with an implication to say that if the account’s type is savings then its balance must not be negative — aType = savings ⇒ balance ≥ 0.

Predicate of AD CorporateHaveNoSavings (Fig. 1f) expresses a set formula. Relation Holds defined in SD of Fig. 1b, which denotes a set of pairs, is restricted to those pairs with corporate customers and savings accounts (a sort of filtering); the restricted relation is then required to be empty — hence, corporate customers must not hold savings accounts. In Fig. 1f the restrictions are performed using domain restriction (symbol \(\lhd \)) and range restriction (\(\rhd \)) using edge modifiers (represented as double arrows and denoting functions) and the outer shading indicates that the restricted set or relation must be empty. The resulting formula is: \(\{o : Customer | o.cType = corporate\}\lhd Holds \rhd \{o : Account | o.aType = savings\} = \emptyset \).

AD HasCurrentBefSavings (Fig. 1d) says that the set of customers with current accounts is a subset of customers with savings accounts — hence, a customer must hold a current account prior to holding a savings account. This involves two internal set variables (included in the declarations compartment) to represent customers with current accounts (custsCurr) and customers with savings accounts (custsSav), which are defined in the predicate in a similar way: relation Holds is range-restricted using edge modifiers (symbol \(\rhd \)) to accounts that are either current or savings, and the actual sets are obtained from these restricted relations using the domain relational operator (symbol \(\leftarrow \)); finally, the bottom-most formula says, using enclosure (or insideness), that custsSav must be a subset of custsCurr. This results in the following formulas:

AD GetAccountGivenAccNo (Fig. 2f) fetches the Account object corresponding to aNo? into output a! — a! ∈{o : Account|o.accNo = aNo?}. AD Account.GetBalance (Fig. 2h), which stores the account’s balance in the output bal!, is imported in global AccGetBalance (Fig. 2g), which fetches the account object corresponding to the aNo? input (via imported GetAccountGivenAccNo) and calls Account.GetBalance on this object.

CD Customer.New (Fig. 2c), a constructor, says how a new Customer object (c!) should be initialised in the post-condition; using predicate edges, the after-state (represented in bold) of custNo is set non-deterministically, and those of name, addr and cType are set to the corresponding inputs. CD Account.Withdraw (Fig. 2e) declares an amount natural number input and says in the post-condition that the new account’s balance (bold line around rectangle) is the old balance minus the requested amount. These two local operations are brought into the global context (of a system or package) through CreateCustomer (CD in Fig. 2b) and AccWithdraw (CD in Fig. 2d). CD of CreateCustomer declares inputs required from the environment and imports Customer.New; CD of AccWithdraw declares aNo? input for account from which money is to be withdrawn, and imported assertion GetAccountGivenAccNo of (Fig. 2f).

VCL’s ADs and CDs, illustrated in the VCL model excerpts of Figs. 1 and 2, are major novelties of VCL, giving VCL a strongly graphical character.

VCL’s design follows theories of visual notation design, namely, physics of notations (PoN) (Moody 2009) and cognitive dimensions of notations (CDN) (Green 1989; Green and Petre 1996; Blackwell et al. 2001). The following exemplifies how these theories are applied in VCL using Figs. 1 and 2:

VCL embodies ideas of both graphical and formal modelling. A major influence is Amálio’s PhD thesis (2007) which proposes UML+Z (Amálio et al. 2006; Amálio et al. 2004), a modelling approach combining UML with the formal language Z. VCL’s Z semantics (Amálio 2007; Amálio et al. 2005; Amálio 2019) was borrowed from this work. Another inspiration is the work lead by Kelsen on diagrammatic expression of behaviour in the language EP (Kelsen 2006; Kelsen and Ma 2008; Amálio et al. 2011).

VCL emphasises rigour, formality and modularity. Early works developed the concept with diagrams built using drawing tools (Amálio and Kelsen 2010a; Amálio et al. 2010; Amálio and Kelsen 2010b; Amálio et al. 2010). A major influence was the development of VCL’s aspect-oriented modelling approach (Amálio et al. 2010). Once VCL had been developed as a concept, we embarked upon the construction of VCL’s tool, the Visual Contract Builder (VCB),³ which made VCL more tangible and firmly defined (Amálio et al. 2011; Amálio and Glodt 2015). The tool paved the way to the use of VCL for coursework and student projects (Leemans and Amálio 2012a; Tobias et al. 2012), brought the language to life, and made the experiment presented here possible. Empirical results on modelling tools, emerging from a spin-off survey of the experiment presented here, highlighted that both VCL and its tool were being positively received by users (Amálio and Glodt 2015).

VCL is an experimental language embodying ideas on visual and formal modelling. It was designed to not drastically deviate from UML; it keeps the same OO foundation which has proved suitable for software design. We wanted to improve the graphics, precision and connection to mathematical modelling. VCL SDs, for instance, introduced just a few novelties with respect to class diagrams to exploit the idea of having more self-contained and closely integrated diagrams focussed on data modelling, but avoiding drastic divergences. The statecharts (Harel 1987) notation was inspirational; although developed independently from UML or any of its main predecessors, it ended-up incorporated in the standard. Likewise, we hope that VCL ideas can be incorporated into such standards if they prove to be useful.

VCL’s major novelty, its capacity to express predicates visually, makes it a prominent representative of graphical design languages. There are other languages with formal basis and capable of expressing predicates visually, such as augmented constraint diagrams (ACDs) (Fish et al. 2005) and Visual OCL (VOCL) (Bottoni et al. 2001; Ehrig and Winkelmann 2006); however, VCL’s most salient difference lies in its superior tool support. In a comparative study focussed on practical application (Tobias et al. 2012), VCL outperformed ACD and VOCL. VCL’s VCB tool outperformed the UML tool Papyrus in the comparison of tools used in the controlled experiment presented here that focussed on usability (Amálio et al. 2011). VCL is the only visual design language expressing visual predicates that tackles modelling in the large through its modelling primitives inspired by aspect-orientation (Amálio et al. 2010). In terms of usability, VCL appears to outperform both ACD and VOCL in what respects the use of colour for visual expressivity. VCL was applied to case studies proposed as challenges by research communities, such as the large car-crash crisis management system (Amálio et al. 2010) and its variant the Barbados car-crash management system (Amálio 2012; Mussbacher et al. 2012), and a cardiac pacemaker (Leemans and Amálio 2012a, ??b).

VCL is being used in student projects at undergraduate and masters level with education being a modest success. It has been used by many students to design systems and applications. Further developments of VCL could focus on: (a) VCL and its tool, aiming towards code generation, (ii) empirical experimentation, following from the results presented here, and (iii) VCL’s formal foundations.

The experiment’s objective is as follows:

Evaluate the effectiveness of VCL on user performance using a set of tasks associated with constructing and using design models by comparing VCL against UML and its satellite textual language OCL.

To accomplish this, two perspectives are considered: modellers and end-users. As said above, the experiment gauges effectiveness, which is meant here to be the degree to which something is successful or adequate in producing a result or accomplishing a purpose.⁴ This involves evaluating performance, which means how effective are modellers or end users in accomplishing tasks.

To fulfil its objective the experiment investigates the following: (i) modelling, which assesses performance in building design models and perceptions emerging from doing so; (ii) problem comprehension, which evaluates the problem comprehension gained from modelling; (iii) model usage, which assesses the performance of end-users in tasks related to the usage of models, namely defect detection and model comprehension; (iv) usefulness, ease of use, usability and overall appraisal, which assess perceptions emerging from experiment experiences.

The experiment seeks answers to the following research questions (RQs):

The dependent variables hold measures to assess the different RQs; they are sampled according to independent variables the most determinant of which being the notation (either VCL or UML). RQ1 to RQ3 are measured both objectively and subjectively. RQ4 to RQ6 are measured subjectively. All RQs are analysed quantitatively; RQ6 is analysed qualitatively also.

Table 1 summarises the dependent variables; it comprises columns for variable’s category and RQ, variable definition and description. Variables’ names follow a convention: abbreviation of category (e.g. Co = completeness; Ac = accuracy) followed by abbreviation of measured modelling aspect — either state space (S), invariants (I) or operations (O). For example, CoS is completeness of state space. We use two types of variables: (i) proportion (obtained quantity is divided by maximum quantity, yielding a continuous number between 0 and 1); and (ii) nominal or categorical (possible values drawn from a bounded discrete set).

The sequel gives further details on how the different RQs are assessed.

There is one major independent variable for used notation — it has two treatments: VCL and UML. The hypotheses are formulated from independent and dependent variables (Table 1). Each dependent variable has corresponding null, \({H^{0}_{i}}\) (no difference between the notations), and alternative, \({H^{a}_{i}}\) (there is a difference between the notations), hypotheses. For example, completeness of state space gives \({H^{0}_{1}}: CoS (VCL) = CoS (UML)\) and \({H^{a}_{1}}: CoS (VCL) \neq CoS (UML)\); 11 out of 31 hypotheses follow this template. Subjective dependent variables are three-valued, either VCL, UML or NP (no preference); the underlying hypotheses cater to this; for example, perceived modelling of state space gives \({H^{0}_{7}}: PMS (VCL) = PMS (UML) = PMS (NP)\) and \({H^{a}_{7}}: PMS (VCL) \neq PMS (UML) \neq PMS (NP)\); 19 out of 31 hypotheses follow this pattern. Hypothesis H₃₁ is formulated based on the values positive, negative and neutral; the first two are co-related to VCL and UML respectively; this gives: \(H^{0}_{31}: Appr (Positive) = Appr (Negative) = Appr (Neutral)\) and \(H^{a}_{31}: Appr (Positive) \neq Appr (Negative) \neq Appr (Neutral)\).

Participants were recruited from the following institutions: (a) Faculty of Sciences of the University of Lisbon, Portugal (FCUL); (b) Faculty of Science and Technology of the New University of Lisbon, Portugal (FCT/UNL); (c) University of Luxembourg (UL); and (d) University of York, UK (UY). Computer science students were rewarded with €50 vouchers for taking part in the experiment and were required to have completed or be in the process of completing a course on UML-based software design.

The experiment’s case studies, detailed in Amálio et al. (2013), are as follows:

The experiment’s tasks feed the dependent variables of Table 1. Each session required participants to work as both modellers and end-users, using either VCL or UML+OCL in either one of the two case studies. A session lasted two hours and comprised the following sequence of tasks (summarised in Table 2):

Participants worked on each of the two systems, using VCL or UML+OCL. They were prevented from collaborating; the work was monitored as tasks were performed in a classroom. Although aware of the experiment’s overall goal, participants were unaware of experimental hypotheses or dependent variables.

The experiment follows a crossover or within-subjects design — all participants are subject to the different treatments. It involves one main factor (or treatment), the design notation (either VCL or UML+OCL), and a secondary case study factor (either UL or FB), yielding four treatment combinations of a 2² factorial design. The rationale is: (a) maximise number of data points to increase statistical power, (b) remove or mitigate bias emerging from differences in case-study complexity or individual ability, (c) at the cost of possible carry-over effects (Greenwald 1976).

Participants were split into two groups. The group-assignment was mainly based on availability due to the experiment’s voluntary nature. However, an effort was made to scatter the high ability individuals evenly among groups based on the results of an ability questionnaire; this is elaborated further in Section 7.

Table 3 outlines the experiment’s four rounds. In each round, each group is given a different treatment. All participants were subject to the four treatment combinations.

The experiment relied on modelling tools. This enhances language usability, contributes to the quality of resulting models and ensures a connection with modern-day reality, which relies heavily on software tools. The experiment uses two Eclipse-based tools: Visual Contract Builder (VCB) (Amálio and Glodt 2015; Amálio et al. 2011)⁵ and Papyrus.⁶ VCB is the only VCL tool. Papyrus was chosen because it is Eclipse-based, which ensures a certain degree of similarity.

The training covered the examined notations, focussed on the more challenging tasks of modelling of state space, invariants and operations, and relied on the participant’s university training on UML-based design. It consisted of a live and assisted modelling of a system from a requirements description using the notations under study and their supporting tools mimicking the experiment’s modelling tasks. No training was given on the experiment’s model usage tasks (problem comprehension, defect detection and model comprehension) due to time reasons — if participants were able to model, then they would be able to undertake the easier model usage tasks.

The training lasted 4 to 6 hours with two mandatory hours per notation. Table 4 summarises the training provided at the different experiment replications. Each participant had at least 4 hours of training (2 VCL + 2 UML) with variations across the replications. At FCUL no extra training was given. Some FCT/UNL and Luxembourg participants were given one hour extra of training. York participants received six hours of training (3 VCL + 3 UML).

All tasks of an experiment session use either one of the two case studies, UL or FB (Section 4.1), exemplars of systems used in the teaching of software design.

At the start of a session, participants were given a requirements narrative (given in (Amálio et al. 2013)). For the task on modelling state space and invariants, participants had to complete a given incomplete model (called initial model) — Fig. 3a gives UL’s initial VCL model. For the modelling of operations, participants had to model two operations on another given model (intermediate model with a complete state space and incomplete dynamics).

The given models aimed to get the most out of the short modelling tasks. They acted as ice-breakers, countering against the blank page effect, and contributing to the task’s fluidity and engagement to provide meaningful experiment data.

For defect detection (DD), participants were given models with seeded defects, which they had to identify by completing an online form. Figure 3b gives the SD of UL’s DD VCL model and Fig. 3c gives the UML counterpart; all faulty models are given in full in Amálio et al. (2013). Defects were seeded in the solution models of both case studies; for example, one error in Fig. 3b is that authors do not have names, whereas in Fig. 3c class Author is missing altogether. Table 5 gives the frequency distribution of seeded defects across the different modelling aspects being analysed (state space, invariants and operations). A χ² goodness of fit test confirmed that the distributions of notations and case study are evenly distributed (see p-values of goodness of fit in Table 5) to avoid bias — no significant differences were found.

For model comprehension, subjects were given a complete case study model. Two operations of these models for the UL case study are given in Fig. 4.

After modelling, participants had to complete a questionnaire with 12 multiple-choice questions having one correct answer, to assess comprehension of modelled problem. A sample question is given in Fig. 5.

Model comprehension questionnaires were accompanied by a complete model of the system. They contained 12 multiple-choice questions having a unique correct answer. A sample question of a model comprehension questionnaire for the university library case study is given in Fig. 6. Whilst the problem comprehension questionnaire someone’s evaluates understanding of the requirements of the modelled problem, the model comprehension questionnaire was focussed on the understanding of what is modelled and what a model actually says. Note how diagrams of Fig. 4 give the answer to question of Fig. 6: when a Copy is recalled, its status changes from onloan to recalled.

The ability questionnaire assesses the capabilities of participants prior to the experiment. It questioned participants on whether they had completed or were in the process of completing a higher education degree in computer science, and what was their exposure to computer programming, discrete mathematics, visual modelling (using either object-oriented or structured methods), and formal modelling. Figure 7 gives two sample questions of this questionnaire.

The debriefing survey gauges perceptions resulting from individual experiment experiences. This includes perceived performance in tasks of modelling, problem comprehension, defect detection and model comprehension, as well as perceived usefulness, ease of use, usability, preferred notation, and positives and negatives of the two notations under study. In addition, the debriefing survey provides supplementary information meant to support and explain the quantitative results by providing qualitative insight. A sample question is given in Fig. 8.

The analysis emphasises measures of central tendency. For continuous variables, we calculate means (point estimates) whose precision is assessed through 95% CIs calculated using the following formula (Cumming 2012): Above: M is the sample’s mean, SE the standard error, SD the standard deviation, and N the size of the sample; t_.95(N − 1) is critical value of the t distribution with N − 1 degrees of freedom and corresponding to the 95% range.⁷

Categorical variables are studied with proportions, derived from frequency distributions, which, like means, are estimates whose precision can be assessed with CIs. For proportion CIs, we use the more robust approach of Newcombe et al (Newcombe 1998; Newcombe and Altman 2000), as recommended by Cumming (Cumming 2012) as it provides good approximations even when N is small and P is close or equal to 0 or 1; this gives the formula⁸: Where P is the estimated proportion, x is the observed frequency for the property of interest, and N is the total number of observations — P = x/N.

For a dependent variable V (Table 1), null and alternative hypotheses are formulated using either continuous or categorical templates: Hypotheses are tested by estimating probabilities called p-values. Given data D, and some null hypothesis H₀, a p-value is a conditional probability estimate, P(D|H₀) (Cohen 1994) — the likelihood of the given observations assuming that the null hypothesis is true. Rejecting H₀ means that it is unlikely because our observations deem P(D|H₀) to be unlikely — hence, we accept the alternative hypothesis. Null hypotheses are rejected based on three levels of statistical significance, depending on whether the p-value is below α = .05 (*), α = .01 (**) or α = .001 (***).

Continuous hypotheses are tested using the non-parametric Wilcoxon test (hereafter referred to as W), a robust test as it does not assume that the sample population is normally distributed. Categorical variables are tested using the χ² test. Calculated p-values are two-tailed.

To counter against the problems of multiple testing and to increase the reliability of NHST, the analysis uses the false discovery rate (Benjamini and Hochberg 1995) and the method of Benjamini and Yekutieli (BY) (Benjamini and Yekutieli 2001), a more relaxed alternative to the family-wise error rate and the conservative procedure of Bonferroni (Holm 1979), to calculate adjusted p-values. Null hypotheses are rejected based on adjusted p-values only.

We use as ES measurement the raw (or unstandardised) mean difference. We consider two cases: paired and unpaired data.

Given the experiment’s within-subjects design, most of our mean difference calculations fall under the paired case. The formulas are as follows (Grissom and Kim 2005; Pfister and Janczyk 2013; Franz and Loftus 2012; Cumming 2012): Above: M_PD = M_VCL − M_UML; SD_PD = standard deviation of paired differences.

The formula for the unpaired case is as follows (Cumming 2012): Above, we assume groups A and B, and M_UD = M_B − M_A.

Raw mean differences provide an intuitive measurement of an effect, but they lack uniformity, making it difficult to compare effects when the scales differ. To cater for uniformity, we use the ES Cohen’s d (Cohen 1988), which is appropriate for continuous variables and means. To estimate this, we use the standard deviation average (or pooled standard deviation) as the standardiser (denominator) (Cumming 2012): There are slightly different ways of calculating the Cohen’s d, which vary depending on the formula used for the denominator. The formula above, based on the standard deviation average, fits the paired design being followed (Cumming 2012). CIs for this ES are calculated using approaches based on non-central t distributions (Cumming and Finch 2001; Kelley 2007).

For categorical data, we use the ES measurement Cohen’s h (Cohen 1988), based on the arcsine transformation, which is appropriate for differences between proportions. The formula for h and the SE to calculate CIs (from Cohen (1988)) is:

The paper depicts its results using the following graph types:

The sequel adopts the following symbols to convey statistical significance and ES:

Table 6 summarises the outcomes of hypotheses testing. First column (Hyp) gives the dichotomous outcome: either null (no difference) or alternative (a significant difference in favour of either approach) hypothesis — highlighted using shading (green in colour version). Columns VCL_M/P and UML_M/P give either a mean (M) or a proportion (P) of VCL and UML, respectively. Column VCL − UML gives difference between either means or proportions. Column p/q gives NHST probabilities (p-values) with appraisals of significance (as per Section 6.5); raw p-values (first value), denoted as p, are calculated using either Wilcoxon (W) or χ² tests; if p is significant, we obtain the adjusted p-value, denoted as q, using the false discovery rate (Benjamini and Hochberg 1995) and the method of Benjamini and Yekutieli (BY) (Benjamini and Yekutieli 2001) to counter against multiple testing issues. Finally, column es (effect size) provides measures of magnitude, using either Cohen’s d or h and levels of magnitude (as per Section 6.5). Null hypotheses are rejected based on the adjusted p-value q.

The experiment’s results are given in full in Amálio et al. (2013) with an accompanying detailed analysis. The next sections present an abridged analysis with relevant results.

Modelling is evaluated objectively based on completeness (how much is modelled) and accuracy (quality of what is modelled), supplemented with subjective measures. We present the data using plots of means and CIs, focusing on the results for overall, each case study, and each experiment venue.

The tasks of problem comprehension, defect detection and model comprehension resulted in objective and subjective performance measures.

Variables U and EoU aggregate perception scores (proportion) emerging from statements evaluated on Likert scale from 1 (strongly agree) to 5 (strongly disagree).⁹ The results, corresponding to hypotheses H_16,17 of Table 6 and portrayed in the plots of Fig. 16, are as follows:

The usability results, corresponding to hypothesis H_18..25 of Table 6, are portrayed in Fig. 17, which contains a table of frequencies (Fig. 17a), a histogram (Fig. 17b), a plot of proportions and CIs (Fig. 17c), and a forest plot of ESs (Fig. 17d). They are as follows:

The usability analysis above together with Fig. 17d are consistent with the EoU results of the previous section. The spin-off study that compared the experiment’s tools (Amálio and Glodt 2015) highlighted that VCL underperformed UML+OCL in the writing criteria, albeit non-significantly; this suggests that, in certain circumstances, graphical editors are less convenient than their textual counter-parts (Amálio and Glodt 2015).

Participants’ overall perception was appraised with respect to preferred notation, and positive and negative aspects.

Figure 20 depicts an analysis of learning effects typical of crossover designs (Greenwald 1976). Figure 20a contrasts first and second attempts at the different experiment tasks for the orders V-U (VCL followed by UML) and U-V (UML followed by VCL). For four (out of nine) measures, namely, completeness of state space (CoS), accuracy of state space (AcS), accuracy of invariants (AcI) and program comprehension (PC), there is a performance improvement on the second attempt (a learning effect) independent of the used notation. For completeness of invariants (CoI), there are improvements only when UML is used in the second attempt. For four measures, completeness of operations (CoO), accuracy of operations (AcO), defect detection (DD) and model comprehension (MC), there are improvements only when VCL is used in the second attempt.

Figure 20b pictures the means and CIs of the paired differences of both V-U and U-V for each task measure — PD = V_VCL − V_UML. Figure 20c gives the calculated p-values and ESs of PD(V-U) and PD(U-V). We observe a significant effect for completeness of state space (CoS), accuracy of state space (AcS) and accuracy of invariants (AcI). The remaining differences are not significant; ESs tend to be small, being medium only for model comprehension (MC). Overall, there is a general tendency for the paired difference to be higher when VCL is used on the second attempt. This suggests that due to VCL’s novelty, participants’ VCL proficiency grows as subjects learn by doing the experiment’s tasks.

Conclusion validity is concerned with the relation between treatment and outcome, and the statistical results. Following from the debate on null-hypothesis significance testing (NHST) (Cohen 1994; Cumming 2013; Nuzzo 2014) and recommendations of high-ranked social science journals, the analysis supplemented NHST with levels of uncertainty, plausibility and magnitude; measures of central tendency (means and proportions) were accompanied by confidence intervals (CIS) and effect sizes (ESs). As highlighted in Section 6.2, null hypotheses are rejected based on adjusted p-values only, calculated using the false discovery rate (Benjamini and Hochberg 1995) and the method of Benjamini and Yekutieli (Benjamini and Yekutieli 2001).

The statistical analysis strived for robustness. We used non-parametric tests and avoided breaching tests’ assumptions. Continuous variables were analysed using the non-parametric Wilcoxon test as it is not dependent on normality assumptions; Wilcoxon calculated p-values are consistent with parametric t-tests and the robust trimmed means test (Yuen 1974) suggested by Wilcox and Keselman (Wilcox and Keselman 2003). The χ² test, used to assess categorical variables, is robust with respect to the distribution of the data, but all categorical variables were found to be normal by Pearson’s test. All statistical testing results are given in Amálio et al. (2013); here we provide Wilcoxon, χ² and BY p-values only.

For ESs, we complemented the classical Cohen d with the more robust alternatives of Algina et al (Algina et al. 2005) and Wilcox and Tian (Wilcox and Tian 2011). All calculated ESs, given in full in Amálio et al. (2013), were found to be consistent; here we focussed on Cohen’s d and h.

Construct validity concerns how measurements are affected by factors related to experimental settings. It relates to training, case studies, measurement instruments, including the defects seeded, and the different questionnaires. Table 7 presents statements drawn from the debriefing survey, accompanied by levels of significance and effect, that supports the analysis that follows. Likewise for Fig. 21, which depicts performance hindering factors.

The training concentrated on the challenging modelling tasks and relied on participants’ prior exposure to software modelling (Section 4.5). It tried to: refresh or enhance design skills and familiarise participants with the modelling tools within the time available. Many participants felt that (Table 7, training): the training was insufficient (T1, Table 7), more training would have improved performance (T5), training was a major performance-hindering factor (Fig. 21, LTr), they lacked knowledge and experience (Fig. 21, LKE), they felt prepared with UML (T2), but not with VCL or OCL (T3 and T4, respectively). This suggests the following: (a) the experiment’s tasks were challenging because many participants were unhappy with their performance; (b) participants’ prior exposure to UML (confirmed by the results of T2 in comparison to T3 and T4) could have biased the results in favour of UML, but the results are positive to VCL, hence: (i) although seen as insufficient, the training seems to have worked, and (ii) VCL accommodates UML-trained practitioners; (c) training was insufficient for the modelling of invariants and operations as most participants lacked prior training.

The experiment’s case studies (CSs), exemplars of real-world software systems similar to CSs used in the teaching of software design, do not favour one notation over the other, and are from familiar application domains (a university library and a flight booking system). They laid the foundation for tasks that are both feasible and challenging, but not overwhelmingly complex. Participants recognised many of these characteristics in the CSs, as evidenced by Table 7 (case studies); most subjects felt that: (a) they had enough time to read the use case narratives (CS1), which (b) they understood (CS2), (c) the case studies were fairly easy to understand (CS4 and CS5). Many subjects felt that: the tasks were carried with difficulties (CS3 in Table 7 and DTCS in Fig. 21). A few subjects lacked a clear understanding of the tasks (Fig. 21, LUT). York participants (N = 7) found the CSs to be realistic (CS7, CS8). Overall, this suggests that: (i) the tasks were challenging, (ii) many participants lacked prior knowledge for such a challenging experiment, (iii) although the training worked somehow, it was not able to fill the knowledge gaps of most subjects.

Lack of time (Fig. 21, LTi, 24 out of 43) was a performance obstacle. Limited time and CS’s reasonable complexity were countered with starting models to be completed within the allotted times. Despite this, the tasks on modelling of operations and invariants, and problem comprehension, were not entirely satisfactory. Modelling of invariants had to be done together with the state space within 30 minutes, which proved short; modelling of operations was too difficult for a 35-minute task. Task’s short duration, lack of training and inherent difficulty, explain the low results obtained in these tasks.

One of the authors marked the models constructed by the participants. To minimise bias and maximise objectivity, a systematic, repeatable and divide-and-conquer scoring approach, detailed in Amálio et al. (2013), was pursued. Both completeness and accuracy relied on a requirements-based marking breakdown made up of individual items that are fairly objective and with little room for subjectivity. The completeness scoring scheme focused on the modelling required by each requirement; the accuracy scoring scheme consisted of test-cases, expressed as snapshots or object diagrams based on an approach outlined in (Amálio et al. 2004; Amálio 2007). Marking both completeness and accuracy involved going through each item of the marking scheme.

The seeded defects, chosen to avoid both favouring any of the notations and being obvious, were scattered evenly across state space, invariants and operations (see Section 5.1). Defect detection (DD) was designed to be intuitive and challenging, which was acknowledged by the participants (DD results in Table 7); an overwhelming majority of them understood the task (DD1, in Table 7) and many found it challenging because the errors were not obvious (DD2), which is endorsed by DD’s modest final scores: .27 for VCL and .19 for UML+OCL.

The model comprehension questions were selected to be of a certain level of complexity and to ensure a reasonable level of coverage. They were articulated to avoid bias in favour of one treatment over the other; the multiple-choice questions eased marking. The debriefing survey was designed to capture the perceptions of subjects with respect to the experiment experience. It contained questions that targeted experiment hypothesis related to subjective assessments. To avoid bias, and to increase the reliability of the collected data, we followed guidelines of questionnaire design (Oppenheim 1996).

The modelling tools could bias or confound the results. We used tools built atop the same platform, Eclipse, to ensure a certain degree of similarity. Papyrus is a major UML Eclipse-based tool with a significantly larger user-base than VCL’s tool, VCB. Despite some criticisms, both tools were seen as being stable (Table 7, modelling tools). A spin-off survey, tied to the experiment presented here (Amálio and Glodt 2015), compared these two tools with a focus on usability; the results, favourable to VCB, concluded that participants could not always discern between tool and language. Nevertheless, VCB is the implementation of VCL’s language and its underlying ideas. The fact that VCB was competitive with Papyrus, a major UML tool, testifies to the quality of both VCL and VCB. Certain VCB features may be superior to Papyrus, but that per-se does not account for VCL’s positive results presented here and in Amálio and Glodt (2015). Both tools, built on top of Eclipse’s modelling infrastructure, have considerable room for improvement.

Internal validity threats are related to external factors that affect the outcomes of the experiment, but are not a consequence of the studied treatment. Table 8 presents statements drawn from the debriefing survey, accompanied by levels of significance and effect, that aids the analysis presented here.

The experiment’s volunteer basis is a possible internal threat. Usually, experiments resort to blocking to counter against variability in the capability of subjects. In our setting, this was, by and large, infeasible; assignment to a group was mainly decided on the basis of a subject’s availability. Section 7 highlights a slight and non-significant imbalance in the proficiency scores of both experimental groups.

Information exchange did not affect the experiment: (i) all participants were working in parallel on the same case study, (ii) group sessions would run separately, but on the same day (morning or afternoon) and (iii) participants were busy. There was hardly any opportunity or motivation for exchanging information.

The experiment’s crossover design suffers from known carry-over effects (Greenwald 1976). Here, the most relevant carry-over effect is the learning accrued from carrying out tasks on the same system using the two languages; modelling using one language may give rise to a learning effect leading to an improved performance when modelling with the next language. To balance the experiment and counter against such learning effects, the order of the two notations with respect to the two case studies was permuted for each group — a group starting with VCL on university library (UL) would start with UML+OCL on flight booking (FB), and vice-versa. We followed a 2² factorial design (case study is a factor) with the case study tasks being undertaken in parallel to: (a) avoid case study difficulty as a confounding effect,¹⁰ (b) counter against exchanges of information, and (c) have more experiment practice (a total of 4 rounds) because software design is challenging, the training was somehow insufficient (see discussion on training in Section 9.2 above) and any practice improvements would only lead to more interesting results and better founded opinions for the debriefing survey. To counter against further carry-over effects, participants were not given feedback on their performance. The analysis of order of notation (Section 8.7) detected a few significant learning effects with large or medium effect sizes, however, this does not affect the paper’s main results: modelling of operations, defect detection and model comprehension.

Lack of engagement was refuted by the debriefing survey (Table 7): participants liked the experiment (E1), finding it interesting (E2), somehow challenging (E3) and positive from a learning perspective (E4). One of the options to the question “please indicate the reasons for your lack of motivation if you felt somehow demotivated” was “I do not really see the need for software design modelling in software engineering”, which was selected by only one participant (out of 43).

External validity, concerned with the generalisability of the results, is a recurrent issue in software engineering research (Glass 1994). Given that we want to extrapolate our results to software engineering practice, a question that emerges refers to the experiment’s degree of realism (Sjøberg et al. 2002). In software engineering controlled experiments there is often an issue of scalability. Due to time constraints, size of case studies and tasks are reduced, and this may break the connection to industrial realities which usually deal with larger problems. We alleviated this issue with case studies that are neither too small nor trivial, and are realistic. This was acknowledged by many participants (see Section 9.2, above). To ensure the feasibility of the modelling tasks within allotted times, participants were given partial models that they were required to complete.

Another problem is the degree of representativeness with respect to software engineering practitioners (Falessi et al. 2017). The participant cohort was culturally diverse; we ran the experiment in universities of three European countries, involving participants from across the globe. Participants were students who may be perceived as unrepresentative of industrial professionals; however, many postgraduate participants had previous industrial experiences, and many were about to embark on new professional careers in industry. The experiment results attest to the cohort’s degree of representativeness; a few individuals performed remarkably high, others noticeably low. Collectively, the average results on state space modelling and other model usage and comprehension tasks show that participants were able to undertake these required tasks, even though they were challenging. The low results on the modelling of invariants and operations are more due to the experiment’s time constraints, the inherent difficulty of these tasks and the limited time available for training, rather than the cohort’s overall ability. We see our cohort as a reasonably competent workforce, which is reflective of industrial settings applying design modelling. Empirical studies failed to find significant differences in performance between students and professionals (Höst et al. 2000; Arisholm and Sjøberg 2004; Salman et al. 2015); a recent study found out that professionals appear to perform better in tasks they are accustomed to, but there is no difference when it comes to using new approaches or technologies (Salman et al. 2015).

A criticism of software engineering controlled experiments concerns pen-and-paper tasks which are not seen as reflecting modern-day realities. The experiment presented here used Eclipse tools, a popular platform for modern-day software development that we see as reflective of current practice.

We see the results presented here as generalisable to general graphical modelling, even though only two languages are examined. This is because VCL is a suitable representative of a largely visual software design language (Section 2.4) and UML+OCL is a suitable baseline.