Research methods in engineering design: a synthesis of recent studies using a systematic literature review

Doctoral studies have a long tradition in higher education systems (Bogle 2018). Doctoral studies are highly relevant because they are considered as a key for technical development and industrial excellence in developed countries. Normally, a PhD diploma is compulsory for pursuing and it is highly valued for getting involved in research projects in companies. The goal of doctoral programs is to provide postgraduates with competences for the generation of knowledge in a given domain. The means to generate knowledge depends on the area, being research methods and techniques potentially different, and evolving in parallel with the development of the domain. In young domains such as Engineering Design, the discussion about which research procedures and paradigms should be employed is still open.

Simon (1996), in his book The Science of Design,  defined design as a search for an optimum in a space of alternatives that take into account the specifications and restrictions of a given problem. Hatchuel (2001) highlighted limitations of Simon’s position discussing that designing cannot be reduced to taking decisions among a bounded set because the number of concepts related to the problem and the possible number of decisions to be taken could be expandable and uncountable, not only due to human creativity but also to social interaction. (Subrahmanian et al. 2020) place Simon and Hatchuel’s approaches in a historical timeline that describes different models of how designing is understood, evidencing the challenges for research design as a discipline that defines a common language that includes the impact of context and users in designing, in addition to the problems.. Probably due to the youth of design as a research discipline, or due to its socio-technical nature, it does not yet have a consolidated research methods and techniques. Blessing and Chakrabarti (2009) proposed the DRM (Design Research Methodology) motivated by “frustration about the lack of a common terminology, benchmarked research method and a common research methodology in design”. Through the analysis of recent research papers, this work has the aim to confirm how these visions about research in engineering design are projected in current state-of-the-art publications.

Since the work of Blessing and Chakrabarti, there have been some relevant proposals that have shed light on different aspects of the global design research landscape. Koskinen et al. (2011) proposed the term ‘constructive design research’ and presented alternatives to integrate research within the practice of design. Joost et al. (2016) used the term ‘design as research’ in a volume that compiled discourses of experts about questions on design research and its relationship with other disciplines. Vaughan (2017) presented a survey that collected different points of view related to doctoral education in the opinions of design graduates about practice-based research design. Redström (2017) presented an essay about how to develop theory -knowledge- by practice, experimentation and making -design. These works are a multi-faceted compendium of practical experiences and visions of experts on how to perform activities related to research in the domain of design. Although many examples and discussions presented in the cited books focus on the topic of research through/by design, rather than on research in engineering design, all of them agree on the relevance of research into the design due to the increasing number of PhD programs that could benefit from background knowledge about this topic. In this paper, we present an alternative approach to shed light on the relations between research and design: instead of collecting the personal visions of experts, we summarise and classify research papers on research in Engineering Design in terms of aims and contributions, methods and approaches, data collection techniques, and research instruments used for the collection of data. To this end, we have carried out a systematic review of the literature on research in engineering design. The overarching research question (RQ) that drives the review is: What is the current landscape of research methods in engineering design?

Access to doctoral studies normally requires candidates to have a Master’s degree in which they have taken courses about research methodologies. Doctoral studies normally culminate with the defense of a PhD thesis in which postgraduates have to show their capabilities to generate knowledge in a specific field. Submitting a PhD thesis that includes activities previously reviewed in scientific journals is generally considered as a quality warranty of the research performed by the student. Although publishing journal papers is not the only way to assess the excellence of the research work performed in a PhD thesis, the quasi-exponential increase of scientific publications we are witnessing (Tenopir and King 2014) indicates that it is probably becoming a universal standard for rating the quality of research. Therefore, being aware of the kind of works published in scientific journals related to engineering design could be of outstanding importance for scholars who have to configure the contents of the courses related to research methodologies in this field, as well as for PhD supervisors and students to focalize efforts for being more productive in terms of publications. The analysis of scientific papers about research in engineering design performed presented in this paper aims to contribute to this aim.

There are many possible ways to analyse, categorise or classify research works because there are many dimensions of analysis. Creswell (2009) presents a classical distinction between (1) quantitative, (2) qualitative and (3) mixed-methods (combining qualitative and quantitative research methods). For quantitative methods experimental designs, non-experimental design are distinguished. For qualitative, narrative research, ethnographic research, phenomenological research, grounded theory and case study research are distinguished. For mixed-methods, sequential, concurrent and transformative methods are distinguished. Blessing and Chakrabarti (2009) identified the following ones: (1) paradigm, that includes empiricism (Randolph 2003; Solomon 2007) and ethno-methodology (Atkinson 1988), methodologies, theories, views and assumptions (Kothari 2004); (2) aim, research questions and hypotheses; (3) nature of the study, including observational vs interventional (Thiese 2014), comparative vs non-comparative; (4) units of analysis; (5) data collection methods including recordings, interview, questionnaires (De Leeuw 2008); (6) role of the researcher (Fink 2000); (7) time constraints, duration and continuation of the research process; (8) observed processes including layout drawing, prototype or product; (9) setting referring to laboratory or field research (Paluck and Cialdini 2014); (10) tasks including type and complexity and nature; (11) number of cases, case size and participants (Diggle et al. 2011); (12) object of analysis distinguishing objects, companies, projects, documents… (13) coding and analysis, analysis and (14) verification methods (Brewer and Crano 2014); or (15) findings, that is, statement models or conclusions resulting from the study. Reich and Subrahmanian (2021) use the PSI framework (Problem, Social and Institutional space) to analyse and categorise research design works focussing on dimensions related to the problem being addressed concerning (1) disciplinary, (2) structural complexity and (3) knowledge availability; dimensions related with who is included in designing concerning (4) the perspective required to formulate the problem, (5) the inclusion of participants in the design process and the (6) capabilities of the design team; and finally dimensions related with how designing is executed taking into account (7) the ties or connections between actors, (8) the accessibility to knowledge and (9) the institutional complexity (Reich and Subrahmanian 2020). The dimensions presented by Blessing and Chakrabari have the ambition to classify different aspects to be taken into account when research in engineering design works are tackled. The dimensions proposed by Reich and Subrahmian are complementary and arise when they analyse the factors influencing success in engineering design projects. When analysing papers, some of the details related to some of the listed dimensions could be missing in the descriptions (timing, success validation etc.) so that we had to devise alternative proposals.

Our analysis pivots around the division between empirical qualitative, quantitative research and mixed-methods proposed by (Creswell 2009). This classification was complemented with analytical research methods, as specified by (Adrion 1993), cited by (Glass 1995) (defined in Sect. 2.2). From this germinal division, data-collection methods, strategies, and contributions of the studies are reported in cross-analysis tables. We aim to identify the main goals and results pursued or obtained by researchers (dimensions 2 and 15 of Blessing and Chakrabarti 2009), the strategies of enquiry and methodologies they follow (dimensions 1, 3, 9, 10 of Blessing and Chakrabarti 2009), and which data sources and instruments are most (and least) commonly used (rest of dimensions of Blessing and Chakrabarti 2009) in the domain of engineering design.

The structure of the document is the following: First, we present the review method and the categories used to classify the papers. We then present the quantitative results of the number of papers in each of the categories and the cross relations of the different classes, shedding light on the relative weight of each of the qualitative and quantitative approaches and the most frequent data-collection methods used. Next, we discuss the usefulness of the obtained results for academics and professionals interested in research design and the paper ends with the conclusions. Complementary material is provided with a brief description of each of the analysed papers.

We follow Kitchenham et al. (2009) as a guideline for performing the systematic review. The nature of the research question did not suit a usual search in the databases, as we were interested in analysing the approaches to research published in the field of engineering design. For this reason, we focused on identifying papers published in relevant journals in the field. The data sources are journal papers in the field of engineering design.

A simple search in the Journal of Citation Reports using the term “Design” as a key search title criterion, generates a list of 99 journals indexed in different categories. Only 80 are indexed in 2020, the rest of them in previous years. As we aimed to high-impact journals reporting research in engineering design, we focused on the journals indexed in SCIE (Science Citation Index Expanded) related to Science and Technology, discarding the 22 journals indexed in ESCI (Emerging Sources Citation Index), the 10 indexed in AHCI (Arts and Humanities Citation Index) and the 5 indexed in SSCI (Social Sciences Citation Index). Among the 43 remaining journals indexed in SCIE, 13 of them correspond to categories related to Chemistry and Biology (for example Anti-Cancer Drug Design or Molecular System Design & Engineering) 11 of them to Computer Science or Electrics (for example Design Codes and Cryptography or Computer Aid Design); 3 with Mathematics (for example Journal of Combinatorial Design) and 2 with Building (Architectural Engineering and Design Management or Structural Design of Tall and Special Building). Closer to engineering design are the 14 remaining journals: 4 indexed in Mechanics Journal of Mechanical Design, Mechanics Based Design of Structures and Machines, Journal of Advanced Mechanical Design Systems for Manufacturing and Journal of Strain Analysis for Engineering Design), 4 related to Materials (Materials & Design, Proceedings of the Institution of Mechanical Engineers, International Journal of Mechanics and Materials in Design and Road Materials and Pavement Design); and 2 related with vehicle design (Journal of Ship Production and Design, and International Journal of Vehicle Design). In spite of being closer to the topic of research in engineering design, we discarded these journals for being too specific. The remaining 4 journals were: (i) Design Studies (DS), (ii) the International Journal of Design (IJD), (iii) the Journal of Engineering Design (JED) and (iv) Research on Engineering Design (RED). Table 1 shows that these journals share the category denominated “Engineering Multidisciplinary”. In this category, there are 6 journals that have the term “Design” in the title, the four selected plus International Journal of Technology and Design Education (also indexed in SSCI), Artificial Intelligence for Engineering Design Analysis and Manufacturing (also indexed in Computer Science) that were discarded for being specialized in education and in artificial intelligence with applications in engineering design, respectively, and therefore, out of the focus of our research.

Each of the selected journals declare in their presentation their aims and audience: RED focuses on design theory and methodology, DS focuses on design processes, JED focuses on different aspects of the design of engineered products and systems, and IJD publishes research papers in all fields of design. The audience of DS, JEC and IJD is broader than the one of RED, which focuses on mechanical, civil, architectural, and manufacturing engineering. Overall, the four journals constitute a rich and representative sample that includes works of diverse nature, applying a variety of research methods and approaches to different problems in the context of research in engineering design.

Sample selection in systematic literature reviews must be structured, comprehensive, and transparent (Hiebl 2021). To comply with these three requirements, we established a recent and limited temporal window and applied random selection to select the sample. We collected 17 papers from each journal, as 17 is the number of papers available in one of the journals under analysis (IJD) and we chose to use the same number of papers per journal to avoid bias (i.e., giving more importance to one journal than another) in the study. For the journals with more than 17 papers in the period of analysis, random selection was applied. We focused on papers published between November 2018 and November 2019, which was the most recent available time window when this work was started.

This methodology led to a final total of 68 papers. We followed a collaborative team-coding approach (Saldaña 2021). Papers were selected and assigned randomly to a pair of reviewers. Each reviewer coded two papers every two weeks. Disagreements and new code proposals were resolved in periodic meetings involving the four researchers/authors. The first author of this paper played the role of “codebook editor” (MacQueen and Guest 2008), updating the code list after the meetings and he used the data from the analysis to build the final tables and present the resulting themes derived from the study.

With the aim of answering the general question of this review, RQ:, “What is the current landscape of research methods in engineering design?”, we focused on the following more specific sub-questions:

To answer these questions, we followed an anticipated data condensation approach (Miles et al. 2020). We defined four overarching topics corresponding to the research sub-questions: aims and contributions of the research; research approach; data collection techniques; and instruments for the collection of data. For each topic, we defined a set of categories, based on our revision of engineering design methods (see Sect. 2). During the iterative coding work, emerging categories were included when required. The new categories were used to re-codify all the works. This combination of deductive and inductive coding enabled us to derive new meanings from the data.

In the rest of this section, we present the categories that were identified in the analysis under each topic. Appendix shows complementary information with representative examples of the categories.

In this paper, we have presented a systematic review of recent literature on research methods and instruments used in a one-year period of research papers in the field of engineering design. By taking this approach, we offer a "fixed image" of recent research in the area and point to some gaps and challenges in the field.

The review shows that there is no single methodological approach accepted as the standard in the field; and that there is a large variety of goals, approaches, data collection methods and instruments to collect them. In spite of this variety, we have observed a certain preference towards qualitative methods, which can be justified by the increasing consideration of engineering design as a complex process affecting humans and their contexts.

We think that this paper contributes to research in engineering design by providing initial evidence for researchers about the kind of work that are expected by high-impact scientific journals in this domain. Additionally, academics can find in this paper a list of topics (methodologies, data-collection procedures, instruments, etc.…) that must be part of the programme of courses on research in engineering design.

The tables included in this Appendix have aim to present the knowledge generated in this paper in the form of a coding scheme, that can be used as an instrument to describe the taxonomy of research aims (Table 5), approaches (Table 6), data collection techniques (Table 7), and instruments (Table 8) in engineering design.