1 Introduction
1.1 Contribution of this article
1.2 Research method
1.3 Outline
2 Background
2.1 Roles of testing in design and development
Source | Definition(s) |
---|---|
IEEE standard glossary (IEEE 1990) | Testing: An activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component |
Verification: The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase | |
Validation: The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements | |
ISO 9000 Quality management system (Hoyle 2009) | Testing: (Definition not found) |
Verification: a process to ensure through the provision of objective evidence that specified requirements has been fulfilled | |
Validation: a process to confirm that resulting product is capable of fulfilling the requirements for the specified application or intended use where known | |
Telecommunication community (Engel 2010) | Testing: (1) Physical measurements taken to verify conclusions obtained from mathematical modelling and analysis. (2) Physical measurements taken for developing mathematical models |
Verification: (1) Comparing an activity, a process, or a product with the corresponding requirements or specifications. (2) the process of comparing two levels of an information system specification for proper correspondence (e.g., security policy model with top-level specification, top-level specification with source code or source code with object code) | |
Validation: (1) Test to determine whether an implemented system fulfils its requirements. (2) The checking of data for correctness or for compliance with applicable standards, rules and conventions | |
Modelling and simulation community Balci (1998) | Testing: ascertaining whether inaccuracies or errors exist in a model. The model is subjected to test data or test cases to determine if it functions properly. “Test failed” implies a problem with the model, not the test. A test is devised and testing is conducted to perform validation, verification or both. Some tests are intended to judge the accuracy of model transformation from one form into another (verification) |
Verification: Substantiating that the model is transformed from one form into another, as intended, with sufficient accuracy. Model verification deals with building the model right | |
Validation: Model validation is substantiating that the model, within its domain of applicability, behaves with satisfactory accuracy consistent with the modelling and simulation objectives. Model validation deals with building the right model | |
Systems Engineering/Engineering design (Shabi et al. 2017) | Testing: “Operating or activating a realized product or system under specified conditions and observing or recording the exhibited behaviour” |
Verification: “Evaluat[ing] a realized product and prov[ing] its compliance with engineering requirements” | |
Validation: “Evaluating a product against specified (or unspecified) customer requirements to determine whether the product satisfies its stakeholders” |
As well as V&V, testing also has other roles in a development process:Testing: “an activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component” (IEEE 1990)
-
Testing may be undertaken for learning with respect to the design, e.g., to understand or refine new technology, or to explore the design space. Similarly, user testing can be integrated into the design process to gain stakeholder feedback on an emerging design, which can be helpful in situations where needs are difficult to express and/or formalise as technical requirements.
-
Testing may generate information and knowledge about the test apparatus and testing method, thereby contributing to insights about how future tests can be most usefully conducted.
-
Testing may reveal information about models or assumptions used in the design process, thereby helping to improve them.
2.2 Factors that influence how testing is done
2.2.1 Design complexity
2.2.2 Product architecture
2.2.3 Degree of novelty
2.2.4 Timing of testing
2.2.5 Susceptibility to design change
3 Testing in design and development process models
3.1 Mathematical and simulation models of testing
3.1.1 Models of testing focused on research insight
3.1.2 Models of testing focused on support for practitioners
3.2 Testing in procedural models of design and development processes
3.2.1 Testing in structured sequential models
3.2.2 Testing in iterative models
Issue | Selected publications (e.g.) |
---|---|
Relationship to quality and V&V | |
Relationship to prototyping | |
Relationship to design complexity | |
Relationship to modularity | |
Relationship to design novelty |
Song and Montoya-Weiss (1998) |
Relationship to change propagation |
Shankar et al. (2016) |
Optimising strategy/timing/integration of testing in the engineering design and development process |
3.3 Summary and critique
-
Testing happens throughout a development process, not just after the respective design activities.
-
Different types of tests are used at different points for different purposes.
-
Test results may lead to redesign or design changes.
-
Testing strategy changes according to the design context.
4 Observations on testing in industry practice
4.1 Case study method
Code | Role |
---|---|
DE1 | End-to-End Process Architect |
DE2 | Verification and Validation Manager |
DE3 | Validation Team Leader—Tier4 Interim/Final |
DE4 | Development Engineer |
DE5 | QFD and FMEA Manager |
DE6 | Master FMEA Owner and Test Planning |
DE7 | Business Transformation Manager |
DE8 | Core Engine Mechanical System Team Leader |
FL9 | Test and Validation Leader |
FL10 | Mathematical Modelling and Simulation Engineer |
TC11 | Design Manager |
TC12 | Senior Project Engineer |
TC13 | Lead Engineer—Product Development |
# | Date | Duration (min) | Participant(s) |
---|---|---|---|
1 | 28 Feb 2011 | 127 | DE1 |
2 | 19 May 2011 | 159 | DE2, DE3 |
3 | 13 Jul 2011 | 76 | DE2, DE3, DE4 |
4 | 13 Jul 2011 | 65 | DE3, DE4 |
5 | 23 Aug 2011 | 135 | DE2, DE3, DE5 |
6 | 15 Dec 2011 | 39 | DE1, DE2, DE3 |
7 | 15 Dec 2011 | 148 | DE1 |
8 | 03 Feb 2012 | 105 | DE2, DE3 |
9 | 03 Feb 2012 | 40 | DE3 |
10 | 27 Mar 2012 | 82 | DE2, DE3 |
11 | 27 Mar 2012 | 75 | DE1 |
12 | 04 May 2012 | 142 | DE3, DE6 |
13 | 17 Sep 2012 | 119 | DE1, DE7 |
14 | 16 Jan 2013 | 52 | DE1 |
15 | 18 Mar 2013 | 117 | DE1 |
16 | 09 Oct 2013 | 160 | DE3, DE8 |
17 | 18 Oct 2013 | 151 | DE1 |
18 | 21 Feb 2014 | 86 | DE1 |
19 | - | - | FL9 |
20 | 02 Jul 2013 | 75 | FL9, FL10 |
21 | 09 Sep 2015 | 40 | TC11 |
22 | 15 Oct 2015 | 90 | TC11, TC12 |
23 | 09 Nov 2015 | 180 | TC11, TC12, TC13 |
24 | 18 Nov 2015 | 120 | TC11, TC12 |
-
In the diesel engine design and manufacturing company, eighteen semi-structured individual and group interviews were carried out at the company premises from February 2011 to February 2014. Eight engineers including a senior engineer, a development engineer, a business manager, a verification and validation manager and a validation team leader were interviewed. The researcher’s understanding of the company context was informed by a previous series of interviews in the same company, focusing on system architecture (Wyatt et al. 2009).
-
In the forklift truck design and manufacturing company, two semi-structured interviews were carried out at the company premises with (a) the test and validation leader and (b) a mathematical modelling and simulation engineer. This study was carried out between 2013 and 2014. In addition, several informal discussions took place at the Open University with the mathematical modelling and simulation engineer. These were not recorded and are not shown in Table 4.
-
In the company that designs and manufactures turbocharging systems, four semi-structured interviews involving a senior project engineer, design manager and CAE and validation manager were undertaken in 2015. Some of these interviews took place at the company premises, while others took place at Huddersfield University.
4.2 Testing and design in the case study companies
4.2.1 Testing and design in the diesel engine manufacturing company
-
The System/Concept Demonstration (SD) stage is primarily concerned with demonstrating that the technology can deliver the required performance. Alternative concepts are analysed and evaluated. Combinations of old and new parts are built into a new product prototype called an MULE. This is tested to verify the performance of new parts. The product specifications evolve during this phase as design decisions are made. As more new parts become available, old parts are replaced in the MULE and testing continues. It is assumed that by GW2, the concept will be selected, the components will be specified and a complete prototype will be built with at least some production parts, and will be ready to be tested for Design Verification (DV).
-
The Design Verification (DV) stage is primarily intended to help develop optimal performance and verify hardware at the optimised performance. The aim is to ensure that design outputs meet the given requirements under different use conditions. At this stage, testing focuses on the single chosen design, involving analysis and testing of stress, strength, heat transfer and thermodynamics, etc. The design is thereby verified before committing to expensive production tooling.
-
The Product Validation (PV) stage focuses on the effect of production variability on performance and dealing with any remaining hardware changes. In this stage, hardware testing is limited to late design changes and emissions conformance testing. Comprehensive testing for reliability and durability also occurs and the product is validated. The mandatory tests required for regulatory compliance usually occur during this stage.
4.2.2 Testing and design in the forklift truck manufacturing company
4.2.3 Testing and design in the turbocharger manufacturing company
4.3 Insights common to the three case studies
4.3.1 Positioning of testing in the design process
We don’t simply test quality when everything is finished. We include it in our planning from the very start (TC12).
4.3.2 Managing complexity in testing
The baseline product definition is physically tested and that information is fairly adequate for the simulation to run for multiple variables for a longer time to find the optimum setup. Then a physical test is required to validate the product as well as simulated results (DE1).
4.3.3 Suppliers and testing
4.3.4 Incremental and radical innovation and testing
Incremental design changes can be managed with standard testing or validated simulations, whereas step changes may require a new test plan. For example, if the company is designing a cylinder block which is a scaled version of a previous product, critically stressed areas would be already known. Thus it might be possible to assess the risk accurately through simulation, whereas a new cylinder block will be physically tested (DE2).
4.3.5 Design changes and testing
4.3.6 Timing and planning of testing
5 Depicting testing in the incremental development of complex products
5.1 A descriptive model of interactions between design and test
-
Performance testing assesses how well the emerging product can satisfy requirements including legislative requirements. Performance testing focuses on characteristics such as speed, efficiency, vibration, and noise.
-
Mechanical testing is mainly undertaken to ensure the reliability and durability of the emerging product. Reliability tests ensure the product’s ability to perform the specified function without failing over a period of time. Durability is a particular aspect of reliability—durability tests assure that “the product will work—given proper maintenance, for a given duration”.
5.2 CAE compared to physical testing
This is particularly evident in the way that CAE simulations are used as virtual testing to complement and in some instances to replace lengthy and complex physical tests. For example, the modelling and simulation engineer interviewed in the forklift truck manufacturer commented:CAE is becoming increasingly important to the companies to minimise the effort and expense involved in product development (DE3).
The lead engineer interviewed in the Turbocharger manufacturer stressed that while the virtual tests performed in simulations are taken into account, physical tests are still vital for validation:There might be 20–30 variations of one product. We can’t build and test all of those, we may build three or four variants. If we can validate CAE or FEA against those physical trucks we have built, then we can sign-off the entire range (FL10).
To summarise, CAE analysis and virtual testing have an important role in reducing the number of physical tests as well as increasing the scope of the scenarios that can be covered (see also Becker et al. 2005). Even so, physical tests remain very important in practice. Their long duration, complexity and cost present a significant constraint on how design and test activities can be integrated in product development processes.Despite today’s advanced computer technology and detailed calculation programs, it is testing which finally decides on the quality of the new aerodynamic components (TC13).
5.3 Lengthy physical tests
5.4 Overlapping between testing and design
5.5 Observations on macro-level process structure
5.6 A macro-level depiction of testing in incremental engineering design
6 Discussion
6.1 Importance of testing as a topic for research
Physical tests are expensive in their own right, in terms of the resources and time required to carried out the test. The cost of needing to repeat tests is also a significant contributor to the costs of design iteration. Optimising testing and its positioning in the design process is, therefore, potentially a means to releasing resources for other design activities. For all these reasons, we believe testing is an important topic for research.To develop the Tier4 engines can cost R&D alone an excess of over [...] million, I would break it down to design and engineering is probably 15%, the material is probably around 30%, and actually testing around performance is the rest—around 55%. Therefore, most of the money in R&D is goes into testing for performance and durability (DE1).
6.2 Recap of contributions
Sect. | Insight(s) |
---|---|
Increase in design complexity increases testing complexity (Novak and Eppinger 2001) | |
Whole-system testing before deployment may not always be possible (Luna et al. 2013) | |
Test designs must be optimised when comprehensive testing is not practicable (Thomke and Bell 2001) | |
Modular architecture may help reduce the costs of testing (Loch et al. 2001) | |
Commonality reduces number of parts to be tested in a new project (Pineda and Kilicay-Ergin 2010) | |
Incremental projects allow reuse of test plans, enhancing V&V quality (Pineda and Kilicay-Ergin 2010) | |
Testing occurs throughout a project using different methods as per context (Shabi and Reich 2012) | |
High-fidelity tests are not possible until sufficient design information is available (Thomke and Bell 2001) | |
Significant tests are often planned late in a project for large scale, complex designs (Reinertsen 1997) | |
Early tests to learn about design space can reduce the likelihood of problems later (Kennedy 2008) | |
Frequent design changes and requirement changes reduce V&V quality (Pineda and Kilicay-Ergin 2010) | |
Change propagation must be considered when revising a test plan following a change (Shankar et al. 2016) | |
Decisions on test frequency should balance benefits of revealing flaws vs. test costs (Ha and Porteus 1995) | |
Increased delays between design and release of test results drives rework and churn (Yassine et al. 2003) | |
Parallel testing of alternatives is useful if cost of tests is low or duration of tests is high (Loch et al. 2001) | |
Frequent testing is most useful if tests support each other in uncovering flaws (Thomke and Bell 2001) | |
Overlapping testing with downstream design risks wasting effort on flawed design (Qian et al. 2010) | |
VVT methods should be matched to requirements, considering acceptable risk & cost (Shabi and Reich 2012) | |
Tests should be focused on critical interactions in a complex system of systems (Luna et al. 2013) | |
Tests can be prioritised using an FMEA-style prioritisation number (Shankar et al. 2016) | |
Macro-level models of design and development processes do not emphasise the role and complexities of testing. | |
In the case studies, testing occurs throughout the process and is a major driver of the development process | |
In the case studies, companies bundle multiple extreme adverse scenarios to reduce physical test costs | |
In the case studies, physical tests are done on baseline cases and used to calibrate virtual tests for other cases | |
In the case studies, companies use suppliers for most part-level testing, but may investigate specific concerns | |
In the case studies, “radically new” parts/subsystems involve testing to experiment & demonstrate technology | |
In the case studies, “incremental” parts/subsystems involve testing focused on V&V | |
In the case studies, design changes lead to retesting and revising test plans. Retests may be in a different mode | |
In the case studies, physical testing cannot typically be accelerated in case of delays | |
In the case studies, physical test facilities are shared across projects so physical tests must be planned early | |
In the case studies, physical test facilities can be bottlenecks and can cause delays to propagate across projects | |
In the case studies, information is progressively released from design for procurement of prototypes for test | |
In the case studies, two important types of testing are performance testing and reliability/durability testing | |
In the case studies, while virtual testing is important, physical testing is still required for final V&V | |
In the case studies, the long durations of physical tests pose significant constraints on design processes | |
In the case studies, reliability and durability tests are often the most time consuming | |
In the case studies, the frequencies of design–build–test iterations are higher in earlier process stages | |
In the case studies, issues identified through physical testing are often addressed by rework in the next stage |