Information gathering activities in engineering design projects play an important role in the identification and definition of stakeholder needs and requirements. However, few studies have explored how students gather information from stakeholders and domain experts in capstone design settings. In this study, we analyzed audio recordings from 19 information gathering meetings submitted by six capstone design teams to investigate how student designers gathered information during these meetings. Our findings include 22 information gathering behaviors that student design teams exhibited during their meetings, half of which were more similar to recommended best practices for information gathering and half of which were less similar. Our findings, including the list of behaviors and associated examples, may be used to guide student designers in employing effective information gathering approaches.
Hinweise
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Information gathering activities play an important role in engineering design projects. These projects often start out with “ill-defined” problems, and designers rarely begin with all of the necessary information they need to develop effective solutions (Buchanan 1992; Goel and Pirolli 1992). For instance, designers often must gather additional information to develop a more comprehensive understanding of the stakeholder needs that may be driving their design problem (Coleman et al. 2016; Zenios et al. 2010). Moreover, designers may need to gather additional information to identify the full range of stakeholder requirements that must be met for a solution to be successful (Bursic and Atman 1997; Dieter and Schmidt 2013; Sutcliffe and Sawyer 2013). There are many other types of information that designers may also want to gather depending on the stage of their design project; in each case, this additional information can help designers make effective design decisions as they develop their solution concepts.
One way that designers can gather additional information about their design problem is by conducting information gathering meetings with project stakeholders or domain experts (IDEO 2015; Rosenthal and Capper 2006; Wooten and Rowley 1995). However, previous studies of student designers suggest that they may struggle to conduct effective information gathering meetings, for instance due to difficulties formulating effective interview questions or adopting stakeholder language (Bano et al. 2019; Luck 2007; Mohedas et al. 2014). These challenges with gathering information may negatively affect student designers’ abilities to identify relevant stakeholder requirements and/or deliver solutions that address stakeholder needs (Bursic and Atman 1997; Loweth et al. 2019; Mohedas et al. 2015). While several studies have explored how information gathering meetings, particularly with stakeholders, may impact student design processes (Hess and Fila 2016; Mohedas et al. 2015; van Rijn et al. 2011), few studies thus far have provided detailed descriptions of the different ways that students in capstone contexts may attempt to gather information as part of their information gathering meetings. To address this research gap, our study analyzed recordings of information gathering meetings that student capstone design teams conducted with project stakeholders and domain experts to understand how students gathered information as part of their projects. The in-depth descriptions presented in our study highlight approaches to gathering information that students may already be implementing effectively, as well as gaps in their approaches that future pedagogy may address.
Anzeige
Background
Recommended practices for gathering information
There are several practices that designers may employ to gather information effectively from stakeholders or domain experts. For example, designers may conduct “deep dive” interviews that explore stakeholder or domain expert knowledge or experiences (IDEO 2015; Kouprie and Sleeswijk Visser 2009; Wooten and Rowley 1995). A key feature of these “deep dive” interviews is the use of open-ended questions that, rather than confirming the designer’s prior notions about stakeholder needs and requirements, elicit stories and invite the interviewee to provide surprising information (IDEO 2015; Kouprie and Sleeswijk Visser 2009; Rosenthal and Capper 2006).
Designers may also employ practices that help stakeholders or domain experts communicate their ideas. For instance, the use of prototypes or other visual representations as “boundary objects” can provide individuals with an additional non-verbal means of expressing themselves that may reduce ambiguity (Deininger et al. 2017; Ewenstein and Whyte 2009; Stappers et al. 2009). In addition, including stakeholders as design team participants can in some cases enable stakeholders to communicate their own design ideas using the designer’s language (Luck 2018; Østergaard et al. 2018).
Furthermore, designers may strive to develop a mutual language with stakeholders or domain experts that is mutually comprehensible despite different backgrounds or experiences (Bucciarelli 2002; Kleinsmann et al. 2007). This mutual language may enable designers and stakeholders or domain experts to engage in “co-inquiry” where they combine their respective disciplinary knowledge to generate new, equally-accessible knowledge about the design problem (Adams et al. 2018; Lehoux et al. 2011). Together, participatory techniques and development of a mutual language can help stakeholders and domain experts contribute relevant project information that may not have been explicitly requested by the designers as part of a planned protocol (Adams et al. 2018; Luck 2018).
Student designer approaches to gathering information
While there are several practices that designers may employ to gather information from stakeholders or domain experts, it is unclear how and to what extent student designers are employing these practices as part of their curricular design experiences. Previous studies have mainly discussed whether conducting information gathering meetings with stakeholders helped student designers identify relevant stakeholder requirements and/or develop solution concepts that addressed stakeholder needs (Hess and Fila 2016; Mohedas et al. 2015; van Rijn et al. 2011). However, these studies did not specify if meetings with stakeholders were helpful because students leveraged effective information gathering practices or despite students employing ineffective ones. This distinction is important because other studies (e.g., Bano et al. 2019; Luck 2007; Mohedas et al. 2014) have highlighted student challenges with gathering information from stakeholders but did not describe how often these challenges occurred over time or what the consequences of these challenges might have been for the information that students gathered.
Anzeige
This knowledge gap may exist because many previous studies of student information gathering activities have focused on student designers’ approaches as a whole over the course of their curricular design projects (Coleman et al. 2016; Lai et al. 2010; Mohedas et al. 2014). For example, Mohedas et al. (2014) interviewed capstone design teams about their experiences gathering information from stakeholders but did not collect data on the content of student meetings to compare with student perceptions of their meetings. By comparison, Hess and Fila (2016) and Modehas et al. (2015) both collected data on student meetings with stakeholders; however, these two studies mainly examined how information gathering meetings influenced student decisions related to stakeholder requirements and/or solution concepts and did not describe the content of these meetings in depth. One study that did provide detailed examples of student information gathering meeting behaviors was Luck’s (2007) description of a recently-graduated architecture student’s interactions with stakeholders in a focus group setting. However, this example is limited to a specific design context that may not necessarily translate to other types of design projects. In addition, recent work by Mohedas et al. (2016) and by Bano et al. (2019) has documented how student designers gather information from stakeholders in controlled settings involving research-based, simulated design tasks. However, more data is needed to understand how student designers gather information in curricular design contexts, such as capstone projects, and how student information gathering approaches may impact their design processes.
Research Design
The goal of our study was to describe the information gathering behaviors that capstone design students exhibited during their information gathering meetings with project stakeholders and domain experts. Our study was guided by the following research questions:
1.
What types of information gathering behaviors do student designers exhibit in meetings with stakeholders and domain experts? What are the characteristics of these behaviors?
2.
In what ways are these information gathering behaviors similar to recommended best practices for gathering information?
We used a qualitative research approach to explore the different ways that students gathered information related to their capstone projects during their meetings with stakeholders and domain experts. Qualitative research methods are ideal for developing deep contextual understandings of human interactions (Borrego et al. 2009; Creswell and Plano Clark 2018; Leydens et al. 2004; Maxwell 2013) and are commonly used for this purpose in design research (Adams et al. 2018; Luck 2007; Stempfle and Badke-Schaub 2002). The methods we employed in this study facilitated our identification of specific information gathering behaviors that students exhibited during their meetings, as well as the relevant details that defined these behaviors.
Context and participants
The context of our study was a single-semester senior-level capstone design course at a large Midwestern university. This capstone course spanned several different design stages including problem definition, concept generation and selection, design iteration and prototyping, and verification and validation, thus allowing us to observe student information gathering behaviors across multiple different project stages. Participants included 24 students from six student design teams enrolled in the capstone course, which is an appropriate sample of teams given the in-depth research methods leveraged and is larger than other similar studies of design team communication and information gathering involving one to three teams (Safin et al. 2019; Stappers et al. 2009; Stempfle and Badke-Schaub 2002). Participants worked in teams of three to five undergraduate students majoring in mechanical engineering, with each team developing a prototype to address a different and unique design problem. While all participants had completed the required mechanical design course sequence, some participants also discussed exposure to other design experiences such as internships, co-curricular projects, and design electives. For all but one of the participants, their capstone design course represented their first experience conducting information gathering meetings, particularly with stakeholders, to inform their design projects. Both the composition of the six teams and their project foci are included in Table 1.
Table 1
Capstone team project focus and composition
Team
Type of project
Sex of team members
Race/Ethnicity of team members
A
Developing assistive device
1 Female, 2 Male
1 Asian, 1 Hispanic, 1 White
B
Developing assistive device
1 Female, 4 Male
3 Asian, 2 White
C
Developing assistive device
1 Female, 4 Male
2 Asian, 3 White
D
Modifying university space
1 Female, 3 Male
4 White
E
Developing measurement tool
3 Male
3 White
F
Modifying university space
4 Male
4 White
Data collection
Participants were initially invited to participate in our study as part of a project-selection survey during the first week of the semester. After the capstone instructor assigned teams to projects, we sent formal invitation emails to teams that had expressed interest in participating.
We collected several different types of data from participants, including (1) recordings of information gathering meetings, (2) semi-structured researcher interviews with participants, (3) participant notes from stakeholder/domain expert meetings, and (4) agendas that participants used to prepare for their meetings. The goal of data collection was to develop rich descriptions of student information gathering behaviors that could facilitate comparison of these behaviors across teams and that could enable us to identify aspects of these behaviors that may be transferrable to other design contexts (Borrego et al. 2009; Leydens et al. 2004).
Recordings of information gathering meetings
All teams were asked to submit recordings of information gathering “interviews” that they conducted with stakeholders or domain experts over the semester, as well as interview agendas, protocols, or notes. We originally used the term “interview” in our instructions to participants, although we elaborated that these “interviews” included all types of information gathering engagements that teams conducted as part of their projects. During semi-structured researcher interviews, we found that participants consistently preferred the term “meeting” over “interview” in describing their information gathering engagements with stakeholders or domain experts. We thus adopted our participants’ terminology in referring to these engagements as “information gathering meetings.”
Participants obtained verbal consent from stakeholders and domain experts before recording and clarified that no personally identifiable information from these individuals would appear in our study. Each participant was compensated $10 per recording submitted by their team, with a maximum possible compensation of $100. We initially recruited eight teams to participate in our study, but two teams had non-disclosure agreements with their primary stakeholders and thus were unable to submit recordings. The remaining six teams submitted recordings of 19 meetings representing over 14 h of audio data. Fourteen of these meetings involved just stakeholders, four meetings involved just domain experts (typically university professors or outside consultants), and one meeting involved both stakeholders and domain experts.
Semi-structured researcher interviews
Teams were also asked to complete three semi-structured interviews with a member of our research team. In keeping with rigorous qualitative methodology, we used these interviews to verify that our interpretations of the information gathering meetings submitted by participants aligned with participant interpretations of their meetings (Borrego et al. 2009; Creswell and Plano Clark 2018; Leydens et al. 2004; Maxwell 2013). Participants were compensated $25 per interview.
Figure 1 depicts the timing of the three researcher interviews. The first interview occurred before teams conducted their first information gathering meeting and sought to understand the background and previous design experiences of each participant. The second interview occurred a week after teams conducted their first information gathering meeting and explored team perceptions of this meeting. The third interview occurred at the end of the semester; during this interview, teams reflected on their meeting experiences holistically.
Fig. 1
Data collection timeline
×
Interview protocols were developed for each interview to ensure comparability across participant responses (Leydens et al. 2004; Maxwell 2013). Protocols were organized around open-ended questions designed to elicit stories and examples from participants while also allowing the interviewer space to opportunistically probe responses for greater depth. We also piloted each protocol with undergraduate students who had previously worked on similar design projects; these pilot interviews in turn informed further iterations on our interview questions. In total, our study collected 20 h of audio data from interviews to supplement the 14 h of audio data collected from stakeholder meetings.
Data analysis
Our data analysis proceeded through four steps, as outlined in Fig. 2. These steps are described in greater detail in the following sub-sections.
Fig. 2
Data analysis process
×
Step 1: Identification of information gathering interactions between students and stakeholders or domain experts
All recordings of student information gathering meetings and researcher interviews were transcribed and checked for accuracy by two members of our research team. Transcripts of student information gathering meetings were then inductively coded by these two researchers to identify and define specific ways that students interacted with stakeholders or domain experts when gathering information during meetings. In this context, inductive coding involved reading through meeting transcripts, highlighting unique interactions where students gathered information, and defining these interactions descriptively rather than according to a pre-defined set of codes (Creswell and Plano Clark 2018; Miles et al. 2014; Patton 2015). Each identified interaction consisted of a series of questions asked or statements made by the student team as they gathered information. Our initial round of analysis resulted in an inventory of information gathering interactions identified across the six teams and was completed using NVivo 12, a qualitative coding software. An example selection of initial identified interaction codes is shown in Table 2.
Table 2
Examples of identified information gathering interactions from student information gathering meetings
Identified interaction
Definition of interaction
Example
Repeat response
Students paraphrase stakeholder’s or domain expert’s earlier response when asking a clarification question
“So to discuss the sources of [the issue], you said it’s mostly [this source], and of course it sounds like there’s a lot of interactions [contributing to that source].”
Validate contribution
Students validate stakeholder’s or domain expert’s contribution to the meeting
“[Your] questions are really helpful. There are things we may not have thought of.”
Ask closed-ended question
Students ask a closed-ended question when transitioning to a new discussion topic
“Okay, and is there any similar device that exists that tries to serve that same purpose, or is it a pretty un-tackled problem right now trying to really quantify [this metric]?”
Step 2: Grouping of identified interactions into behaviors that were more similar to recommended best practices for information gathering
We next grouped together similar identified interactions and defined these groupings as distinct information gathering behaviors. These information gathering behaviors captured the overarching ways that teams in our study gathered information during their meetings. We based our initial list of information gathering behaviors on a list of interview best practices developed by Mohedas (2016)/Mohedas et al. (2016) to facilitate comparison between the behaviors that teams exhibited in their meetings and recommended best practices for information gathering from the literature. An example of our initial grouping process based on the list from Mohedas (2016)/Mohedas et al. (2016) is shown in Table 3.
Table 3
Example of grouping identified interactions into information gathering behaviors
Information gathering behavior
Definition of behavior
Identified interactions that were grouped into behavior
Definition of interaction
Example
Avoid Misinterpretations [from Mohedas (2016)/ Mohedas et al. (2016)]
Students repeat and clarify stakeholder’s or domain expert’s responses to make sure that accurate information is being collected
Repeat response
Students paraphrase stakeholder/domain expert’s earlier response when asking a clarification question
“So to discuss the sources of [the issue], you said it’s mostly [this source], and of course it sounds like there’s a lot of interactions [contributing to that source].”
Check interpretation
Students double check that they made a correct inference from stakeholder’s description
“And what kind of tables? Are they circular tables or are they long tables?”
We encountered two issues during this initial grouping process. First, the list of best practices described in Mohedas (2016)/Mohedas et al. (2016) was compiled primarily from literature sources (e.g., Rosenthal and Capper 2006; Wooten and Rowley 1995) with a specific focus on informational interviews for identifying stakeholder needs and/or requirements. However, informational interviews are not the only context during which designers might gather information from stakeholders or domain experts; as noted in our “Data Collection” section, “meeting” was the preferred term that many of the participants in our study used when discussing their engagements. Thus, as we grouped our identified interactions, we revised and expanded our initial list of information gathering behaviors to account for additional practices related to exploring stakeholder experiences in depth in meeting or co-design contexts (Coleman et al. 2016; IDEO 2015), employing prototypes to gather information (Deininger et al. 2017; Ewenstein and Whyte 2009; Stappers et al. 2009), and developing a mutual language with stakeholders or domain experts to improve communication (Adams et al. 2018; Bucciarelli 2002; Kleinsmann et al. 2007).
The second issue that we encountered while grouping identified interactions into distinct information gathering behaviors was that many identified interactions resembled recommended best practices, but few matched the literature descriptions of these best practices exactly. We also observed noticeable variety in how teams exhibited certain types of interactions. For instance, one of our identified interaction codes was “ask open-ended questions” to invite deep responses from stakeholders. However, the depth of response that may be elicited from open-ended questions may vary substantially depending on the question. To use two examples from our data, “Could you tell us a little bit about how we might interface with [this stakeholder], and what we could go back and forth with them about?” is an open-ended question that is soliciting details on how to contact an individual as well as a suggested list of discussion topics. By comparison, “How do we verify that doing something like that with a [prototype] here is scalable, for example?” is soliciting a process and also asking, in this case a domain expert, to think more critically about the design problem and what it may mean from their experience to “verify” a prototype. To account for the observed variation in how teams exhibited each behavior in our data, we thus titled our list of behaviors that resembled information gathering best practices as behaviors that were more similar to recommended best practices.
We ultimately grouped identified interactions into 11 information gathering behaviors that were more similar to recommended best practices. We also classified these 11 behaviors into structural, exploratory, and collaborative categories based upon similarities that we saw across the behaviors. These similarities were based upon the types of information that each behavior seemed to elicit during meetings as well as the types of meeting situations where students exhibited each behavior. Our categorization is described in greater depth in our “Findings” section.
Step 3: Grouping of remaining identified interactions into behaviors that were less similar to recommended best practices for information gathering
Our 11 information gathering behaviors that were more similar to recommended best practices did not capture all identified interactions from our initial round of coding. Our remaining identified interactions all diverged from recommended practices in specific ways. As a result, we grouped remaining identified interactions into 11 additional behaviors that we categorized as behaviors that were less similar to recommended best practices since each behavior in this category contrasted strongly with one of our behaviors that was more similar to recommended best practices. For example, the behavior elicit shallow responses (shown in Table 4) encompassed several remaining identified interactions that all principally diverged from the behavior encourage deep thinking. This second grouping process accounted for all remaining identified interactions from our initial round of coding.
Table 4
Example of grouping remaining identified interactions into information gathering behaviors that were less similar to recommended best practices
Information gathering behavior that was less similar to recommended best practices
Definition of behavior
Remaining identified interactions that were grouped into behavior
Definition of interaction
Example
Elicit Shallow Responses
Students ask questions that implicitly constrain stakeholder or domain expert responses
Ask a closed-ended question
Students ask a closed-ended question when transitioning to a new discussion topic
“Okay, and is there any similar device that exists that tries to serve that same purpose, or is it a pretty un-tackled problem right now?”
Ask multiple questions
Students ask multiple questions at once without first giving the stakeholder or domain expert the opportunity to answer
“What it’s like to go in the [space] for a [stakeholder]? Can you tell us what that’s like? Do they bring their own [materials]?”
Step 4: Refinement of final list of information gathering behaviors
Lastly, the two original coders reviewed the meeting transcripts again to identify any information gathering interactions that had been missed during the initial round of inductive coding and that aligned with one of the 22 information gathering behaviors from our final list. After this second transcript review, we discussed remaining interactions that did not cleanly align with one of the defined behaviors, as well as interactions that had been grouped differently by the two researchers. Discussing these discrepancies helped us clarify and iterate on our definitions of the 22 information gathering behaviors observed in this study; these definitions were further validated through comparison to participant descriptions obtained through researcher interviews of how teams gathered information during their meetings.
Findings
Table 5 shows the full list of 22 information gathering behaviors that student teams exhibited in their meetings with stakeholders and domain experts. Since student structural information gathering behaviors mainly involved practices for meeting organization and basic clarification rather than in-depth information gathering, we only describe student exploratory and collaborative behaviors in depth in the following sub-sections.
Table 5
List of behaviors that students exhibited during their information gathering meetings with stakeholders and domain experts
More similar to recommended best practices
Less similar to recommended best practices
Behavior
Definition
Behavior
Definition
Structural Behaviors
Build Rapport with the Stakeholder or Domain Expert
Students express their appreciation of the stakeholder’s or domain expert’s contributions and seek to help the individual feel comfortable during the meeting
Damage Rapport
Students express judgment of the stakeholder’s or domain expert’s contributions or otherwise cause the stakeholder or domain expert to feel uncomfortable during the meeting
Avoid Misinterpretations
Students repeat and clarify the stakeholder’s or domain expert’s responses to make sure that accurate information is being collected
Muddle Information Received from the Stakeholder or Domain Expert
Students use imprecise language and/or allow technical difficulties to decrease the clarity of the stakeholder’s or domain expert’s responses and thus insert ambiguities into collected information
Guide Meeting Direction while Inviting Stakeholder or Domain Expert Input
Students clarify the purpose of the meeting and consistently guide the meeting direction while also inviting the stakeholder or domain expert to suggest topics of interest
Cede Guidance of Meeting
Students surrender to the stakeholder or domain expert the position of guiding the meeting direction and/or exhibit uncertainty as to who should be guiding the meeting at a given moment
Exploratory Behaviors
Encourage Deep Thinking
Students ask questions that encourage the stakeholder or domain expert to move beyond superficial responses and provide detailed knowledge on a given subject
Elicit Shallow Responses
Students ask questions that implicitly constrain stakeholder or domain expert responses
Flexibly & Opportunistically Probe Responses
Students employ spontaneous probes, as indicated by vocal cues indicating surprise or curiosity, to dive deeper into the stakeholder’s or domain expert’s experiences or knowledge
Rigidly Adhere to Structure
Students resist departing from the pre-determined topics of the meeting
Verify the Conclusions Drawn from Meetings
Students check that their conclusions drawn from the meeting match with the stakeholder’s or domain expert’s own perceptions
Lead the Stakeholder or Domain Expert to Conclusion
Students indicate a suggested or preferred answer when asking questions or soliciting feedback and thus influence the stakeholder’s or domain expert’s response
Delve into Stakeholder or Domain Expert Experiences
Students evoke specific ideas or experiences of the stakeholder or domain expert to better understand how the individual thinks and feels about the design problem
Conflate Student and Stakeholder or Domain Expert Experiences
Students suggest that the stakeholder or domain expert’s experiences likely resemble their own and do not explore the individual’s experiences in greater depth
Collaborative Behaviors
Use a Co-Creative Meeting Strategy
Students establish space within the meeting for the stakeholder or domain expert to make project decisions or give design feedback
Use a Student-Centered Meeting Strategy
Students control the goals of the meeting and project, making decisions and informing the stakeholder or domain expert of those decisions rather than soliciting input on those decisions
Develop Mutual Understanding with the Stakeholder or Domain Expert
Students leverage language and/or design representations that help them to communicate across disciplinary barriers and develop mutual understanding about the design project
Assume Stakeholder’s or Domain Expert’s Understanding
Students embed assumptions about the stakeholder’s or domain expert’s understanding of the design project in their questions or language
Introduce Relevant Information
Students provide relevant knowledge about the design project to build a repository of shared information between the design team and the stakeholder or domain expert
Introduce Unclear Information
Students provide information about the design project but do not clearly explain the meaning of the information and/or clarify that the information is likely inaccurate
Explore Differences Between Perspectives
Students explore the nuances of the stakeholder’s or domain expert’s point of view by presenting the differing perspective of another stakeholder or domain expert not present at the meeting
Place Own Perspective Above Others’
Students describe the perspectives of other stakeholders or domain experts not present at the meeting but dismiss these other perspectives as irrelevant to the project
Exploratory information gathering behaviors
Exploratory information gathering behaviors represented ways that students tried to obtain deeper insights about stakeholder or domain expert perspectives and experiences. Exploratory behaviors that were more similar to recommended best practices (encourage deep thinking, flexibly & opportunistically probe responses, verify the conclusions drawn from meetings, and delve into stakeholder or domain expert experiences) helped students explore the responses of others in greater depth to uncover new and surprising insights about their design problem. By comparison, exploratory behaviors that were less similar to recommended best practices (elicit shallow responses, rigidly adhere to structure, lead the stakeholder or domain expert to conclusion, and conflate student and stakeholder or domain expert experiences) constrained the range and potential depth of stakeholder or domain expert responses.
The following example of the delve into stakeholder or domain expert experiences behavior demonstrates how exploratory behaviors that were more similar to recommended best practices helped students explore stakeholder or domain expert perspectives and uncover new aspects of their design problem. This example comes from a meeting conducted by Team E, who was working with their primary stakeholders to develop a safety measurement tool. In this case, Team E was meeting with an engineer who had built a previous iteration of the tool to learn more about how the current design might be improved:
Team E:
So you’re speaking to the subcommittee [who sets measurement standards]…If there were to be a time when a [measurement] standard was put into there, what would go into that process?
Engineer:
Well first we have to say, “Okay there’s a possibility of injury,” either real or perception… We discuss if there’s a need for a [measurement] standard for the [product] to prevent [injury]. Then we do studies to see what the issue is and how we can prevent it, what kind of tests we need to do to ensure that does not happen.
Team E:
Sure. So if you determined that a specific [deformation] was dangerous, would the subcommittee also recommend a specific device and method for [measurement] as part of those standards?
Engineer:
…If there is [data] and a method to saying, okay, the [product deformation] caused that injury, and then there’s a method that you guys perform and suggest, the subcommittee will take that over and do the testing and validate that what you guys have come up with is correct. And they’ll go through the motion of getting that approved and adding that to the [measurement] standard.
Team E’s initial open-ended question invited the engineer to speak about their experience with implementing a measurement standard. Team E also probed the engineer’s response to learn more about how this measurement standard might relate to the tool they were developing. This exploration by Team E thus led to the discovery of important new contextual information about how their measurement tool might be used in practice.
Conversely, the following example demonstrates how exploratory behaviors that were less similar to recommended best practices constrained the range of potential stakeholder or domain expert responses. This excerpt, featuring Team A, represents both the elicit shallow responses and the conflate student and stakeholder or domain expert experiences behaviors. Team A was working on an assistive device for a young person with a disability. Their primary point of contact was a volunteer who worked for the non-profit that was funding Team A’s project and who knew the team’s user personally. In this exchange, Team A was meeting with the volunteer for the first time and was trying to clarify their user requirements:
Team A:
What is [the user’s] age, ‘cause I know if you want it to be adjustable I imagine if she is between 12 and 14 she will probably continue to grow.
Volunteer:
So right now she’s 16 or 17 but our goal is for it to be adjustable for future [users] too. I mean, all of our kids are under 19. Generally if they have a diagnosis that they need to use [this device] they’re generally smaller ‘cause generally they’re kids who are in wheelchairs… who aren’t overweight necessarily or large.
Team A:
This might be a weird number to ask but do you know on average do they get to be super tall kids or not really? So we can know how much…
Volunteer:
It just kind of depends, generally if the kid’s over, I’m trying to think ‘cause I’m… so generally, it’s hard to gauge ‘cause they’re not standing, I’d have to look and see how tall [the user] is but she’d probably be my height if she were standing and if they’re much taller than [that], we probably wouldn’t utilize the [device]… does that make sense?
Team A:
That answers my question perfectly… We can test this on me…
Team A was looking for two specific pieces of information, the user’s age and height, that they needed to develop their user requirements. However, Team A worded their questions in a way that suggested that they already knew this information (i.e., elicited shallow responses). Rather than providing an open response that might have led to surprising insights for the team, the volunteer instead commented on Team A’s perspective and provided minor additional details that did not significantly challenge Team A’s prior conceptions about their user or push the team to think more deeply about how their user’s experience may differ from their own. As a result, Team A concluded that they could test the prototype on one of their team members rather than involving their user (i.e., conflated student and stakeholder or domain expert experiences). It is thus unclear if Team A’s eventual solution accounted for their user’s unique capabilities or preferences.
Collaborative information gathering behaviors
Collaborative information gathering behaviors represented ways that students tried to facilitate the participation of stakeholders or domain experts during information gathering meetings. Collaborative behaviors that were more similar to recommended best practices (use a co-creative meeting strategy, develop mutual understanding with the stakeholder or domain expert, introduce relevant knowledge, and explore differences between perspectives) bridged differences in understanding between students and stakeholders or domain experts that resulted from differences in domain background, past experiences, or knowledge about the team’s design problem. These behaviors also helped stakeholders and domain experts contribute relevant information without explicit prompting from the student team. In contrast, collaborative behaviors that were less similar to recommended best practices (use a student-centered meeting strategy, assume stakeholder’s or domain expert’s understanding, introduce unclear information, and place own perspective above others’) made it more difficult for stakeholders or domain experts to discern what information might be most relevant to provide to the design team and/or articulate their own understanding of the design project.
The following example of the develop mutual understanding with the stakeholder or domain expert behavior demonstrates how collaborative behaviors that were more similar to recommended best practices helped students solicit more informed responses from stakeholders or domain experts. This excerpt comes from a meeting conducted by Team C, who was building an assistive device for a young person with a disability (“the user”). During this meeting, Team C wanted to solicit design feedback from both their user and their user’s caregivers. Team C thus opened this meeting with:
Team C:
Since we last met with you…We’ve put together a couple of our preliminary ideas… put together the design requirements from the feedback you gave us. Based on that, we built a couple of [functional] prototypes we brought to show you today… they’re the rough sketch of what we’re thinking. We want to get your feedback on them, and then we’re going to take one of those… and try to build that full-scale… [The] final design will be a lot more fleshed out, but these [prototypes] illustrate the design ideas we’re looking at.
Team C:
…Again, the same idea with this [second prototype] as with the last one. We’d have it on a swivel, so you can choose the direction. One of the things that we’re very interested in is if you rigidly bound the swivel to a handle-
Caregiver:
When he drags his left hand, because it’s higher functioning… if he’s trying to drive and control a switch at the same time, that would be difficult unless there’s something on it close to his joystick so he could put it in stop. Are you thinking of a static [action]?
Team C:
It’s totally controlled by his right hand. We would be putting some sort of lever here-ish, and then as you pulled it this way that would turn the frame to the direction you wanted to go. Then, we can put a button on the end of it, or a trigger, and when you push the button you get [the action].
Caregiver:
Yeah. I think he could do something in this plane right here. No problem.
Team C:
Which [prototype] do you like?
User:
Something like this [second prototype]. Seems more understandable to me…
Caregiver:
I think it seems more like a natural [action].
Team C:
Sure. We can work around this one.
This example highlights several different ways that Team C helped their user and associated caregivers understand the design project in order to solicit more in-depth feedback on their current ideas. First, Team C updated their user and associated caregivers on their progress since the last meeting and how they had used the user’s input from this previous meeting. Team C thus drew a connection from their last meeting to their current one, demonstrating the value of the user’s and caregivers’ contributions and clarifying their role in the design project. Team C then described how the purpose of the current meeting was to solicit feedback on a couple of solution concepts that the team had “prototyped.” Since their user and associated caregivers were unfamiliar with the terminology of “prototypes,” Team C clarified that their prototypes were objects meant to roughly illustrate their current design ideas. This clarification helped Team C’s user and their associated caregivers understand both what a prototype was and what sort of feedback might be most useful; they were thus able to justify their preferred solution concept with specific reference to how each prototype functioned.
Conversely, the following example of the assume stakeholder’s or domain expert’s understanding behavior demonstrates how collaborative behaviors that were less similar to recommended best practices made it more difficult for stakeholders or domain experts to provide relevant responses. This excerpt features Team D, who was modifying a building associated with the university. During this meeting, Team D hoped to clarify the goals of their project with their project sponsor. They did so by asking questions such as the following:
Team D:
Okay.So, with that in mind, we’d like to know by which criteria the project will be judged. Is there a preferred method you have by which the [issue] should be quantified?…
Sponsor:
So, first of all, I think quantifying it is a great thing. Obviously we don’t have the real time opportunity during the semester to have [the space] full of people… But I would assume… and again, I have no background at all in engineering… I would assume you could model, kind of, expectations of [the issue]. And then, based on what you recommend, say, “This will absorb x percentage.” I don’t know what that looks like… But I think that’s probably the best way to judge it… But also, taking into account [other stakeholder] feedback and the aesthetics with [other stakeholders].
Team D:
So, I think with the project description, we’re mainly doing a prototype. Is a prototype something we’ll be installing, and then we’ll be able to kind of understand the impact of it?Because we won’t be doing the full on project, correct?
Sponsor:
Yeah. Again, that’s something beyond my scope of knowledge, as far as what a prototype might do… But, over the summer, if there’s equipment we need to test we can definitely do that…
Here, Team D asked several questions using technical language around quantification and prototyping that was inaccessible to their project sponsor. The project sponsor thus qualified each of his responses by pointing out that he did not have a technical background before attempting to answer Team D’s questions in vague terms. Placing an emphasis on quantification also meant that Team D missed an opportunity to probe deeper into other potential project criteria, such as aesthetics, about which their project sponsor could have provided a more informed response.
Discussion
Student information gathering behaviors in context
We identified 22 information gathering behaviors that students exhibited when meeting with stakeholders and domain experts to inform their capstone projects. We categorized these behaviors in two primary ways. First, we classified behaviors as either structural, exploratory, or collaborative based upon similarities that we saw across the behaviors in terms of the types of information elicited and the types of meeting situations during which students demonstrated each behavior. Second, we defined behaviors as being either more similar to recommended best practices or less similar to these best practices. These two types of categorization highlight unique aspects of capstone student information gathering approaches.
Student information gathering meetings with stakeholders and domain experts in a capstone design context exhibit characteristics of many different types of engagements, such as informational interviews and collaborative project meetings, that have traditionally been discussed separately in the literature. For instance, many of the behaviors that we categorized as exploratory corresponded primarily to recommended best practices (e.g., using open-ended questions to elicit detailed descriptions of experiences) for conducting effective information gathering interviews (IDEO 2015; Kouprie and Sleeswijk Visser 2009; Wooten and Rowley 1995). By comparison, many of the behaviors that we categorized as collaborative corresponded primarily to recommended best practices (e.g., developing a mutual language) for collaborating effectively with stakeholders as fellow design project participants (Adams et al. 2018; Kleinsmann et al. 2007; Lehoux et al. 2011). Almost all meeting recordings submitted by participants included both exploratory and collaborative behaviors as students employed various strategies to understand stakeholder or domain expert perspectives and solicit relevant design feedback. The categorization scheme described in this study thus reflects the composite nature of student information gathering meetings in a capstone design context, as also discussed in Mohedas et al. (in press), and highlights the various types of best practices that students may need knowledge of to conduct these meetings effectively.
In addition, our 11 pairings of behaviors that were less similar to recommended best practices with those that were more similar to recommended best practices may represent student information gathering “learning progressions” towards ideal pedagogical outcomes (Crismond and Adams 2012). The collection of behaviors that were less similar to recommended best practices represent “low anchors” describing the baseline knowledge or skills that student designers may possess related to gathering information from stakeholders or domain experts. Many of these “low anchor” behaviors also resemble previous descriptions of different ways that student designers may struggle to conduct effective information gathering meetings (Bano et al. 2019; Luck 2007; Mohedas et al. 2014). Conversely, the collection of behaviors that were more similar to recommended best practices represent “high anchors,” or specific learning gains and approaches that we would hope students exhibit when gathering information. All teams in this study demonstrated both “high anchor” and “low anchor” behaviors to some extent, thus highlighting information gathering best practices that students seemed to be applying successfully as well as specific knowledge gaps.
Impact of different student information gathering behaviors
We did not directly assess how different behaviors impacted the quality or content of information that students gathered from stakeholders and domain experts. However, the in-depth descriptions of student behaviors discussed in this study provide some indication of expected impacts, particularly the likelihood of students eliciting unknown knowns or unknown unknowns from stakeholders and domain experts. Unknown knowns represent relevant information that stakeholders or domain experts may possess but do not immediately articulate; the information may exist in the form of tacit knowledge or may be suppressed for political, social, or emotional reasons (Sutcliffe and Sawyer 2013). Unknown unknowns, by comparison, represent relevant information that is unknown and inexpressible for both the student team and the stakeholder or domain expert; this gap may exist due to a collective lack of knowledge or due to inadequate problem exploration by the student team (Sutcliffe and Sawyer 2013). Failure to uncover unknown knowns and unknown unknowns may limit students’ understanding of stakeholder needs and/or lead students to develop stakeholder requirements that fail to reflect crucial aspects of these needs (Bursic and Atman 1997; Loweth et al. 2019; Sutcliffe and Sawyer 2013).
While there is no way to guarantee that students will uncover unknown knowns or unknown unknowns, information gathering behaviors that are more similar to recommended best practices may increase the likelihood that this information is uncovered. For instance, exploratory behaviors may help students uncover unknown knowns by diving deep into stakeholder perspectives and use contexts (IDEO 2015; Kouprie and Sleeswijk Visser 2009; Wooten and Rowley 1995). Collaborative behaviors may help stakeholders or domain experts articulate unknown knowns of their own initiative (Adams et al. 2018; Luck 2018; Østergaard et al. 2018). Both types of behaviors may also help students discover unknown unknowns as they collect additional information about their design problem. By comparison, the exploratory behaviors exhibited by teams in this study that were less similar to recommended best practices may constrain the range of stakeholder or domain expert responses. The collaborative behaviors that were less similar to recommended best practices may make it difficult for stakeholders or domain experts to express themselves and contribute relevant information. Both outcomes may decrease the likelihood of uncovering unknown knowns or unknown unknowns.
Limitations
One limitation of our study was the relative lack of diversity across our participants, with 83% identifying as male and 71% identifying as White. A more diverse group of participants might have interacted with stakeholders or domain experts in ways that were different from the interactions that we observed in our data, which would have potentially led us to define our information gathering behaviors differently. Future work could study how a more diverse group of students interacts with stakeholders or domain experts when gathering information.
In addition, we did not measure the outcomes of student information gathering behaviors relative to each team’s design process or product. While we could identify how certain behaviors may have impacted the immediate conversation, broader implications of these behaviors for each team’s project were less clear. Future work might explore how project outcomes, such as user satisfaction, may relate to the information gathering behaviors exhibited by a given team.
Implications for design education and practice
Our findings point to several implications for design pedagogy and practice. Student designers could use the list of 22 information gathering behaviors identified in this study as a tool to improve their information gathering processes. This list indicates 11 ideal information gathering behaviors that students should aim to exhibit in their meetings with stakeholders and domain experts, as well as 11 corresponding behaviors that are less similar to recommended best practices and that students may be unintentionally exhibiting instead. When reflecting on their information gathering meetings, student designers could use this list to identify the information gathering behaviors that they are exhibiting most frequently. This facilitated reflection could help students understand what they are already doing well when gathering information, identify specific areas where they might improve their process, and determine new ways to approach gathering information.
Design instructors could also use the findings from our study to develop targeted pedagogy related to conducting effective information gathering meetings. For example, as also noted in Mohedas et al. (in press), the composite nature of student information gathering meetings in a capstone design context means that capstone students would likely benefit from instruction that covers both exploratory and collaborative information gathering practices. Such instruction might describe effective methods for soliciting deep information (e.g., IDEO 2015; Kouprie and Sleeswijk Visser 2009) and also for facilitating stakeholder or domain expert design participation (e.g., Adams et al. 2018; Østergaard et al. 2018). In addition, our in-depth descriptions of student information gathering behaviors that were less similar to recommended best practices highlight specific student struggles and/or knowledge gaps that may be transferrable to other design contexts and that design instructors could address. For instance, many of these behaviors may have resulted from difficulties that participants experienced while planning out their information gathering meetings. Student designers may thus benefit from additional tools and support that can help them develop well-structured open-ended questions and identify multiple potential follow-up questions that will enable them to gather more comprehensive information from stakeholders and domain experts.
Conclusions
We identified and described 22 information gathering behaviors that student designers exhibited when meeting with stakeholders and/or domain experts. We defined these behaviors in terms of 11 behaviors that were more similar to recommended best practices for gathering information and 11 behaviors that were less similar to these best practices. Each pair of behaviors represented preferred ways that students might gather information, as well as less ideal practices that students may exhibit instead. In addition, we also classified student information gathering behaviors into three categories, structural, exploratory, and collaborative, based upon similarities in the types of information elicited by each behavior and the types of meeting situations during which students demonstrated each behavior. These categories highlight the composite nature of student information gathering meetings as exhibiting characteristics of both informational interviews and collaborative project meetings. In conclusion, student designers might use the list of behaviors described in our study to help them reflect on and improve their information gathering approaches to be more in line with recommended best practices. Design instructors might use the in-depth case examples presented in this paper to develop targeted pedagogy related to gathering information from stakeholders and domain experts. The findings from our study can thus help student designers gather information more intentionally and effectively as part of their design projects, and as a result develop a deeper understanding of the stakeholder needs that may be driving their design problem and the stakeholder requirements that must be met for their solution to be successful.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 1611687. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The research team would like to express their gratitude to Jiangqiong Liu for her work verifying and revising the transcripts of information gathering meetings and researcher interviews.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
