We will now present the results used to design the prototyping model aspects presented in the previous section. In this section, we report on the demographics of our systematic map, the results from the focus group and from the interviews of our multi-case study. The model was initially designed based on previous literature, then validated and adjusted through the empirical data of the focus group and the multi-case study.
6.1 Demographics of Map (RQ2 and RQ3)
Our systematic map consists of thirty-three primary studies published between 1988 and 2018, see Table
4. The majority of these (17) were within software engineering (in general), 11 within human factors (HF) & design, and 5 within requirements engineering (RE), see Table
5. The number of articles appear to have increased somewhat over the years from on average 1 to 2 articles a year, except around the year 2000. The increase is mainly within the areas of HF & Design and RE. The research type for each paper was classified according to the categories by Wieringa et al. (Wieringa et al.
2006): (1)
Evaluation research investigates a problem or technique in practice and provides new knowledge of causal or logical relationships, (2)
Solution proposals present a solution without a full-blown validation, (3)
Validation research presents a solution proposal validated outside of industrial practice. (4)
Philosophical papers sketch new theories or frameworks, (5)
Experience papers describe personal experiences and may contain anecdotal evidence.
Table 4
Number of articles in each step of the selection process
LUBsearch | 2.713 | 84 | 30 | 19 |
ACM | 1.958 | 62 | 24 | 14 |
Total | 4.671 | 146 | 54 | 33 |
Table 5
Number of articles published per 5-year period for each main discipline
1988–1992 | 4 | 1 | | 5 |
1993–1997 | 5 | | | 5 |
1998–2002 | | | 1 | 1 |
2003–2007 | 2 | | 4 | 6 |
2008–2012 | 2 | 3 | 3 | 8 |
2013–2018 | 4 | 1 | 3 | 8 |
Total | 17 | 5 | 11 | 33 |
Around 40% of the articles (14 of 33) are based on
in-vivo research and empirical data from industry (
Evaluation), see Table
6. Another 40% (12 of 33) propose solutions based on in vitro (only) validation or theoretical reasoning (
Solution proposal and
Validation type). Our systematic map indicates an increase of in-vivo research concerning prototyping over the past 10–15 years.
Table 6
Types of research of articles in our systematic map.
Evaluation | 1 | 1 | | 3 | 4 | 5 | 14 |
Experience | 1 | | 1 | | | 1 | 3 |
Philosophical | 2 | | | 2 | 1 | | 5 |
Solution proposal | 1 | 3 | | | 3 | 2 | 9 |
Validation | | 1 | | 1 | | | 2 |
6.2 Focus group results: Validation of Initial Version of Model2
Practitioners discussed the use of prototyping to perform requirements-related tasks such as elicitation, specification, and validation at the focus group by reflecting on scenarios based on our prototyping aspects model. In general, the participants agreed to the scenarios, that they correspond to a typical way of working and to their prototyping practices, which they see as a good way of working with agile software development. There was some disagreement about prototyping in early project stages. Several participants said that prototyping is time consuming and difficult, and rarely used during early phases and at the first meetings with customers. Instead, these participants preferred stakeholder interviews and competitor analysis to understand the problem domain. In contrast, two participants said that early prototyping, e.g., through sketching, can be useful in understanding the problem. Another participant said that early prototypes can be beneficial in loosely defined projects and facilitate discussions about user expectations. They had used simple prototypes in the form of sketches, which helped designers and users understand what they were discussing and assisted in framing the role of the system.
6.2.1 Purpose
The participants described several purposes for prototyping. Its value in
exploration & learning was described as “
the more you work with design [of the prototype] the more you know what works for you”.
3 Several participants described the value of prototyping for internal use and the importance of “
building understanding before you start thinking about solutions.” One participant described creating personal prototypes to get an overview of the future system, essentially working as a
specification. Another participant said that prototypes are more useful than a formal specification to
communicate requirements. Two participants described the importance of exposing developers to the prototype as a way of
testing the technical feasibility and, if possible, adjust the requirements to “
the easiest way … to implement the solution”. Another participant described the importance of communicating and involving developers by testing prototypes and discussing requirements, although some developers just want information on “
the components I need to build.”
6.2.2 Prototype scope
The focus group provided validation for this aspect of our model. They recognised the relevance of considering breadth and depth of functionality, and refinement of all our identified facets (of the first version of our model) except for data richness, which was not mentioned during the focus group. The participants described that in early project stages, they prefer to use simpler prototypes, either paper sketches with shallow functionality and low refinement or a prototyping tool that supports producing prototypes with shallow functionality and mid to high degree of visual refinement. These simpler prototypes provide “a way to understand the problem” and to “organize and explore … [the solution] space.” Another participant implied that feedback on visual style could be avoided by using prototypes with a low degree of visual and interactive refinement such as wireframes that encourage feedback on functionality. In contrast, for more complex solutions “you get value out of being able to click around” and a high degree of interactive refinement is preferred. Some participants said that prototyping must involve interactivity and that they “wouldn’t do a good job … [without] interactive prototypes”. This illustrates the need for a common definition of prototyping. The fact that prototype scope was spontaneously discussed by our participants demonstrates that our model can provide support for this.
6.2.3 Prototype use
The participants confirmed all four review methods included in the first version of our model, i.e., internal use, demonstrations, scenario-based, and free testing. Several participants described using prototyping to test new ideas with colleagues and that they first show the prototype “within the team and not with users”. This was especially the case for early prototypes that one participant rarely showed to customers, despite experiencing challenges in discussing with users without any visuals. When prototypes are shown to external stakeholders it is often as a demonstration that takes place after first having discussed their problem and current situation. One participant said that users are only occasionally involved in interacting with prototypes while this is done “all the time” within the project. Instead, users are involved in beta testing “all the time”, which some, but not all, of the participants see as prototyping. One participant described the use of scenario testing a few times a year, e.g., with new employees or students.
Several participants mentioned the importance of structuring feedback sessions to “get feedback on the right thing”, i.e. relevant facets. One participant said that it is easier to get feedback on what is good rather than what needs improving. Another participant described that “talking with just one person isn’t enough” since stakeholders may have preconceived solution ideas. One participant suggested using solution-agnostic questions to encourage users to consider what problems the system should solve and to discuss requirements at domain and product level.
6.2.4 Exploration strategy
Our participants validated the four exploration strategies included in the first version model and described that the strategy is varied throughout development stages and for different prototyping purposes. One participant described the importance of creating several parallel prototyping variants to avoid getting stuck in a single solution too soon. She said that using multiple prototypes helps users’ express the direction the system should take, and thereby involves them in determining product scope and requirements. Similarly, another participant described that they “throw a wide net with many different options” in initial project stages. In addition, several participants said that building several variants of a prototype is useful for exploring alternative solutions and demonstrating these to stakeholders to “collect ideas about how to proceed and what path to choose” and thereby optimise the solution. Furthermore, one participant described that in a later stage, an incremental and flexible exploration strategy is needed to “fit in [customer requests] with the concept you have in mind”.
6.3 Multi-case study results: broader validation and adjustments (RQ4)4
We validated the prototyping aspects model through our multi-case study with eleven startups. The model was presented to practitioners in the interviews and used in the analysis of the interview data to categorize the prototyping instances described by our interviewees. In total, we identified forty-three prototyping instances among our startups, see Table
7. Prototyping in these startups ranges from using simple sketches through mock-ups, to using source code to explore and to obtain feedback on early software versions. We have categorised the identified prototyping instances using our prototyping aspects model. In this section, we describe and motivate changes to our model based on our empirical data. The first and the revised (second) version of the model are shown side-by-side in a table in Section
5. We recommend that the reader consults that table while reading this section. Initial results of the prototyping practices of startup Companies A-D are available in a separate publication (Bjarnason
2021).
Table 7
Overview of the 43 prototyping instances (Nn) described by case companies (N = A-K), categorized by our prototyping aspects model. Described lacks of a facet are denoted with –. Blank cells indicate lack of information in the empirical data
A1 | x | | | | | | | | s | | - |
A2 | x | x | | | | x | | | m | h | l |
A3 | x | | x | | | x | | | W | h | h |
A4 | | x | | | x | | x | | W | h | h |
B1 | | | | | | x | | | s | | - |
B2 | | | | | | x | | | m | | - |
B3 | | | | | | x | x | | W | | lh |
C1 | | x | | | | | | | W | h | h |
C2 | x | x | | | x | | x | | m | h | - |
C3 | | x | x | | | | | | W | h | h |
D1 | x | | | | | | | | m | | - |
D2 | | x | | | | | | | s | | |
D3 | | x | x | | | x | | | W | m | h |
D4 | | x | | | | | | | s | | - |
E1 | x | | | | | | x | | m | - | h |
E2 | | | x | | | x | | | W | h | h |
F1 | | | | | | | | x | m | | - |
F2 | x | x | | | | | | x | m | h | - |
F3 | | | | | | | | x | W | h | h |
F4 | | x | | | | | | | W | h | h |
G1 | x | | | | x | | | | s | | - |
G2 | x | x | | | x | | | | m | h | |
G3 | | | | | | x | | | W | | |
H1 | | x | | | x | | | | v | | m |
H2 | | | x | | | | | | W | h | h |
H3 | | x | | | | x | | | W | | |
I1 | | | | | | x | | | vsm | | - |
I2 | x | | | | | | | | vsm | | |
I3 | x | | | | | | | | i | - | - |
I4 | | | | | | x | | | m | h | l |
I5 | | | | | | | x | | W | l | h |
J1 | x | | | | | | x | | i | - | - |
J2 | x | | | | | | x | | W | l | l |
J3 | x | | | | x | | | | W | | |
J4 | x | | x | | | | | x | o | h | - |
J5 | x | | | | x | | | | i | h | - |
J6 | | x | | | | | | | v | | |
J7 | x | x | | | | | x | x | W | l | m |
K1 | x | x | x | | | | | | W | h | m |
K2 | | x | x | | x | | | | W | h | h |
K3 | | x | | | x | | x | | W | h | h |
K4 | | x | | | x | | | | o | h | h |
K5 | | x | | | x | | | | o | h | - |
A1 | h | - | | if | yn | | | | p | |
A2 | h | m | m | ie | n | s | m | | bp | |
A3 | h | h | | ie | y | f | a | | b | |
A4 | l | l | | i | | s | | | bpf | |
B1 | l | l | l | e | n | | m | | b | |
B2 | m | - | l | e | n | | m | | bp | |
B3 | h | lh | -h | ie | y | f | a | | pf | |
C1 | h | h | h | e | yn | | ma | | b | |
C2 | m | - | | ie | yn | | m | | bp | |
C3 | | h | | ie | y | f | | | pf | |
D1 | m | | | i | n | | | | p | |
D2 | - | - | - | i | y | | | | f | |
D3 | h | h | | ie | y | f | | | bp | |
D4 | m | | | i | y | | | | p | |
E1 | l | l | | i | y | | | s | bo | |
E2 | h | h | | i | y | | | | f | s |
F1 | m | - | | ie | y | s | m | p | f | |
F2 | m | m | | ie | yn | s | m | | pf | |
F3 | h | h | | e | y | f | a | | p | |
F4 | h | h | | e | n | | m | | b | |
G1 | l | - | | i | y | | m | | f | |
G2 | m | m | m | ie | yn | s | m | | bp | |
G3 | lm | | | e | y | f | | | bp | |
H1 | m | - | m | e | n | s | | s | b | |
H2 | - | - | h | i | y | | | s | b | |
H3 | | | h | e | yn | f | | s | o | |
I1 | h | - | | ie | yn | | | | b | |
I2 | h | | | ie | yn | | | | b | |
I3 | - | - | - | e | n | | | | b | |
I4 | mh | l | | ie | y | f | | | bf | |
I5 | | | | i | y | | | | p | |
J1 | - | - | | e | n | | | | bp | |
J2 | l | l | | i | y | | | | f | |
J3 | | | | if | y | s | | | p | |
J4 | h | - | h | if | n | | m | | bp | |
J5 | - | - | m | e | n | | | | b | |
J6 | | | | e | n | | | | b | |
J7 | h | l | | i | y | | | | o | |
K1 | m | m | | i | y | | | | pf | |
K2 | h | h | | ie | y | f | | | f | |
K3 | h | h | | ie | y | f | | | bo | |
K4 | h | h | | e | n | s | | | bp | |
K5 | l | - | | e | n | s | | | b | |
6.3.1 Purpose
This aspect mostly worked well in discussing and categorising how the startups use prototyping, and only two minor revisions are made to this aspect, namely for the purposes Communication and Validation.
One of the main purposes of prototyping is “as a tool for selling products to customers” (Company B) and is thus used in communicating “about sales… to make them understand that there is a need” (Company A). For this reason, Company A had invested “quite a lot of work” in building a high-fidelity interactive mockup (prototyping instance A2) which “is fake, but quite good… communicates that we know and builds trust… Based on that we get investors.” (Company A). For these reasons, we adjust the detailing of the aspect of Purpose in our model to highlight that communication includes sales and marketing, thus Communication & Alignment is changed to Communication: Sales, Alignment.
For
Validation & Testing, the boundaries between
Market viability and
Business viability were hard to determine for the prototyping instances, and the difference between these two is unclear. Instead, we replace these with the terms
Problem–Solution fit and
Product-Market fit (Osterwalder et al.
2014). This will more clearly separate between prototyping to ensure that the product matches the needs of users and customers by provides a solution that addressing their problem, and prototyping to validate that customers are willing to pay for your product and thus the existence of a viable market and business opportunities. Different prototyping instances may be used for each dimension. For example, the mock-up C2 was used to “explore what is required [by the market]”, i.e. product-market fit, while the beta release C3 “to 2–3 [customers] is used to tune [the implementation] further” (Company C) and thus improve the problem–solution fit. Product-Market Fit is especially important for startups that “
must produce a prototype to be able to confirm that you have buyers, and thus revenue… [before] incurring more costs without knowing that it is sellable” (Company A).
Furthermore, while we observe that prototyping is not directly used within our startups for the purpose of improving quality, we retain the purpose of Quality Improvement in our model. The lack of support for this prototyping option in our case study is likely due to the nature of early startups, rather than lack of relevance in general of this purpose. In startups, the main focus is on establishing a viable business model with a product solution that matches customer needs and problems, and developing initial product version to realise their solution ideas. Thus, prototyping is primarily used for sales & marketing (Communication), internal exploration of the solution domain, feasibility testing of technical aspects of the solution design, and to validate the product-solution and product-market fit of their business model.
When discussing and analysing the prototyping in the startups it became clear that the media used to represent a prototype, e.g. paper or PowerPoint sketches, computer-generated mock-ups, or source code software, was an important aspect to consider. A similar aspect, related to paper prototyping, was discussed by the authors during our original design of the model, but discarded at that point in time since the aspect of prototype scope appeared to be sufficient. However, we reintroduce this aspect in our model since the multi-case study indicates that the choice of prototype media affects the costs and benefits of prototyping. Our interviewees described that they “have chosen to produce a mock-up due to cost and time aspects” (Company A). We believe that this is an important aspect of prototyping that can further support practitioners in making informed decisions about the kind of prototype media to use. In particular for early stages of product development, we want to highlight the cheaper and easier kinds of prototype media such as videos and interviewing, which some of our startups find very useful. For example, in prototyping instances H1, J6, I1 and I2, videos are used for sales & marketing (communication) and for validating product-market fit. One interviewee described videos as a way “to present products far before they are ready” (Company I). This interviewee also described prototyping instance I3 and interviews as a means of prototyping “even without anything to show” by “talking to people about how they solve this problem today” (Company I). Similarly, the now growing startup Company J used videos in social media channels for sales & marketing purposes; “funny videos that became a bit viral” (Company J). This startup also uses interviewing for exploring the problem domain. In the early prototyping instance J1, they “asked people [users]… [and] that became an important signal” that their solution idea was viable, and later used a similar approach in prototyping instance J5 to validate their revenue model.
6.3.3 Prototype scope
When discussing the scope of a prototype, the degree of refinement of the visual appearance, interactive behaviour, and functionality worked well. For example, one interviewee described that “we do not focus that much on interactivity at the beginning” (Company F). In addition, five interviewees expressed the importance of realistic data, e.g. to capture behaviour around “errors and bad data” (Company I). Also, “it is really important to use that [realistic data] otherwise your [customers] probably get stuck on that” (Company F) and that when “the data is fairly realistic… [it] can be shown to customer without any problems” (Company G). Another startup that markets advanced AI algorithms described that they “need accurate data to back up [our algorithm]”, and thus demonstrate their solution by using “actual [customer] data” (Company H).
The aggregated dimension of broad vs narrow functional scope was harder to convey, though we saw examples of both broad and narrow prototypes covering many vs a few features. For this reason, we revise this aspect to include Breadth as a stand-alone dimension, alongside the dimensions that can be refined, i.e. functionality, visual appearance, interactive & haptic behaviour, and data realism. We believe this provides a clearer and more easily comprehensible model for describing the scope of a prototype.
6.3.4 Prototype use
This is the aspect where the previous structure was less aligned with how the interviewees described their use of prototypes. The dimension of usage environment worked well, even though many of our startups only tested their prototypes in meeting settings. In part, this is due to challenges in accessing the actual usage environment. For example, for Company F where testing in a live environment is “difficult since our product targets clinics… varies a lot how open they are with this.” Other reasons for this, are likely cost and awareness of when testing in a live environment is important. One interviewee believed that “[environment] is an important aspect of mobile solutions, to test them in their correct context” (Company I).
We find that the (previous) dimension of review method is not fine-grained enough to categories the identified prototyping instances. Instead, we replace this dimension in our model with the three dimensions reviewers (who receives or gives feedback on the prototype), prototype interaction (directly with prototype or not, e.g. when demoing), and review approach (scenario-based or free). This corresponds better to how the interviewees describe their use of prototypes.
All interviewees described with whom prototypes were used, i.e. Reviewers, and often the same prototype is used with multiple kinds of reviewers. For example, the sketches of prototyping instance A1 were “tried out on ourselves and on people in our proximity, then we started developing” (Company A). The later more refined “mockup [A2] then becomes a blueprint for the product to be built by developers” (Company A). Thus, reviewers are often internal within the startups and development team, as well as, external such as customers, funders, or product users. In addition, some startups described using family and friends (so called FFF, Family, Friends, and Foes) for obtaining feedback on prototypes. For example, for Company J, the prototyping instance J3 used “FFF when we have made something new. What do they think?” and for J4 they “grabbed people in the corridors and ask them what they think… [and then] realised that we had missed several things”(Company J).
There were also variations in how these reviewers could interact with the prototype. When demoing, e.g. of sketches or early mock-ups, there is no reviewer interaction with the prototype. There are also examples of choosing not to allow reviewer interaction with the prototype since “[customer] awareness is too low” (Company A) and “the market is still not that aware of their needs…to have the level of understanding to be active [in giving constructive feedback on premature mock-ups]” (Company E). For more refined prototypes w.r.t. functionality, visual appearance, and interactivity, the users may be encouraged to try the prototypes out for themselves. For example, the fully functioning product prototype of B3 was available to the general public and “customers get to use our stuff… [allowing the startup to] follow up how they are used in the field” (Company B). Similarly, users are encouraged to interact directly with the mock-up of prototyping instance F2 “to see reactions…. how they reason, how they select and react to things” (Company F), which provides the startup with rich feedback. We have added a dimension of Interactivity to our model to cover this dimension.
Finally, while scenario testing was included in our initial version of the model, our interviewees primarily described the use of scenarios as an approach that could be applied both when demoing a prototype, or when allowing users to interact directly with the prototype. For example, when demonstrating the paper mock-ups of prototyping instance F1, the startup would “try to get them [customers] to think about what they do then [in that scenario] (Company F). Similarly, a startup that provides a technical solution uses scenarios to help non-techie customers “see the value [in the solution] …[through] presenting these cases to them… and avoid just clicking us through” (Company G). A similar approach is seen for prototyping instances K4 and K5, where the startup “goes through a simple scenario, explains the challenges, and shows the solution” (Company K). Thus, we added Review approach as a dimension with the options scenario-based or free, where free, covers both free testing, e.g. for beta releases, and demos without any clear scenarios.
6.3.5 Exploration strategy
This final aspect was the hardest to discuss with our interviewees, and the one with the least number of responses with the exception of parallel exploration. Using multiple prototype versions in parallel was a clear and understandable strategy, though most of our startups stated that due to resource constraints they tend to focus on one version at a time. As one interviewee said: “we go with one [option] first since we are a small team” (Company H). Another interviewee mentioned that “we always try to sketch different [options]” (Company F) though without giving any concrete example of this (and therefor not seen in any of the reported prototyping instances.) A third interviewee said: “focus one thing at a time and learn about. But, in some cases [parallel exploration is useful] … e.g. for a pricing model” (Company I). Thus, we have modified the aspect of exploration strategy to include three facets: single or parallel exploration, iteration focus (business, product, feature or optimisation level), and iteration size. The two facets related to prototyping iterations were indicated by a few of our interviewees. For example, one interviewed technical lead advocated the “need to iterate slowly… [to ensure] what the stakeholder originally wished for” (Company D). Initially, startups often iterate at the level of the product, as was described by the interviewee from Company E: “we have an idea and want to see if it works.” And, then move towards “more gradual development … and more structured feature growth from the very basic experimental ideas up to the ready-for-market products” (Company E).
The dimension of iteration size replaces the other three exploration options (of the initial version of our model), namely point-based, optimisation, and flexible exploration. The difference between these previous strategies mainly connects to the size of the change between prototype versions. Prototyping with an optimisation strategy, and thus with small changes between versions, could be conveyed through giving the Purpose of Quality improvement. Similarly, the difference between point-based and flexible could also be deferred to the size of changes and connected to the stage and maturity of a product or an idea. For example, in the early stages of a startup prototyping could be used for broad exploration of a suitable solution approach, in which case flexible exploration appears suitable. As a startup and their solution approach matures, they are more likely to want to explore more fine-grained variations, e.g. in what features or what user-interface design to implement, in which case point-based exploration, or even optimisation exploration is more relevant.