Data-driven prototyping via natural-language-based GUI retrieval

To retrieve relevant GUIs for NL-based search queries, we first require a suitable GUI repository. Recent data-driven design research constructed and released several GUI datasets appropriate for our approach by automatically crawling a vast amount of Android applications from Google Play such as ReDraw (Moran et al. 2018), ERICA (Deka et al. 2016), Rico (Deka et al. 2017) and Enrico (Leiva et al. 2020). ReDraw collects GUIs by automatically exploring 5416 Android applications comprising a total of 14,382 unique GUI screenshots and GUI hierarchy data using GUI testing automation techniques to simulate user input. Screenshots of the GUIs are captured either with third-party frameworks or platform-dependent utilities. In contrast, ERICA solely relies on manual human-based GUI exploration and is primarily developed for capturing user interaction traces. ERICA comprises around 18,600 unique GUIs harvested from 2400 applications. Rico, an extension of the ERICA dataset, mines GUI screenshots, GUI hierarchy data, application meta data and interaction traces with both human-based and automatic exploration techniques and constitutes the largest design dataset of the discussed ones with 72,219 GUIs collected from 9772 unique Android applications (from 27 different application categories). Rico employs a custom Android service to access detailed information about all GUI components such as text, absolute position on the screen, visibility indicators, resource identifiers and the activity name, among others. For our prototyping approach, we employ Rico based on multiple important reasons: (i) the large amount of GUIs available for retrieval, (ii) the coverage of many diverse application domains and (iii) the comprehensive textual information provided particularly valuable for NL-based GUI retrieval.

For text preprocessing of both the GUI text representations and the NL input query, we apply a standard preprocessing pipeline. First, we lowercase the text and apply tokenization. Tokens are then excluded by the following multiple filters. We discard stopwords and words comprising numeric or non-ASCII characters. Subsequently, we describe the employed baseline ranking models, the AQE techniques and the semantic BERT-based Learning-To-Rank models.

Since Rico provides only GUI screenshots which are not directly reusable for rapid GUI prototyping and editing, we propose a simple yet effective algorithm to transform them into partly editable GUI screens. The automatic derivation of editable GUIs from the GUI repository is integrated into the graphical editor of RaWi, which will be described in more detail in the subsequent section. In particular, we can exploit the GUI hierarchy data and semantic labels for GUI components encompassed in Rico to extract them accordingly from the accompanied GUI screenshots. Therefore, (1) we initially obtain the original GUI screenshot provided by Rico and (2) crop each contained GUI component from the screenshot based on their absolute position. We applied this procedure for both individual GUI components and additionally identified layout components. Third, (3) for three GUI component types including labels, buttons and text-input we extract multiple style properties in our current prototype to provide more detailed editing capabilities. For the remaining components, we currently simply reuse their image crop in the editable GUI screens. For each layout group, we compute the background color as the top-ranked RGB color in their histogram. For labels, we identify the font color by first obtaining the background color as before and then computing a normalized color distance to the background color for all colors in the histogram. The first top-ranked color in the histogram that exceeds a predefined distance threshold is selected as the font color being a simple yet effective heuristic. In addition, we estimate the font size by comparing the bounding boxes from the hierarchy data and an instantiated bounding box containing the text. Since Rico often provides only rough bounding boxes for the labels, we compute refined bounding boxes using Tesseract-OCR (Smith 2007). For buttons and text-inputs we similarly extract the background and font colors. We enrich the Rico dataset with the identified style properties for enabling proper GUI component instantiation in the editor later. Finally, (4) the cropped GUI components are placed on their exact GUI screen positions. Subsequently, we describe our prototypical implementation.

The presented GUI retrieval and GUI screen derivation components are integrated in a prototype that we implemented as a web-based rapid prototyping editor using HTML5 and JavaScript. Our current prototype provides a GUI search view, a graphical prototyping editor and an interactive and dynamically updated preview of the built application as shown in Fig. 6. Subsequently, we provide more details about the features of our current implementation. Our current interactive prototype is publicly available at RaWi Prototype (2022).

In addition to measuring the GUI prototyping productivity, we assessed the perceived usefulness of our GUI prototyping approach by the study participants. Using a five-point Likert scale, we asked participants (a) how much our approach supported them during the prototyping process, (b) how relevant the retrieved GUIs were on average, (c) how much the GUI retrieval results supported the users during the prototyping and (d) how much the GUI search results helped in better visualizing how the GUI could be prototyped. Moreover, we computed the System Usability Scale (SUS) (Brooke 1996) which is a reliable and well-known measurement for system usability. To gain further insights, we asked the participants for free-form feedback to potentially discover interesting ideas for improvements of our GUI prototyping approach.

Table 1 shows the evaluation results of the different baseline ranking models, AQE techniques and BERT-based LTR models for the P@k and \({\textit{NDCG}}@k\) metrics, whereas Table 2 shows the evaluation results for the \({\textit{AP}}\), \({\textit{MRR}}\) and \({\textit{HITS}}@k\) metrics. Considering the three baseline ranking models, we can observe that BM25 outperforms the other two models to a large extent across most of the examined metrics. In particular, the \({\textit{MRR}}\) of the BM25 model indicates that the first relevant GUI appears at rank 1.92 on average over the gold standard. Moreover, the \({\textit{HITS}}@5\) indicates that in 69% of the queries the BM25 model ranked at least one relevant GUI among the top-5. Thus, BM25 seems to be an effective baseline model for NL-based GUI retrieval. Comparing the plain TF-IDF and nBOW models, we can observe that especially the P@k and \({\textit{HITS}}@k\) as well as the \({\textit{NDCG}}@k\) for small k are higher for the nBOW model, indicating that pretrained word embeddings can provide additional benefit for GUI retrieval from NL searches.

In addition, both tables show the results of the four different AQE methods based on the PRF-KLD score using the previously best performing baseline ranking model BM25 as the base ranker. The results show that the text-segment-wise PRF (s) method obtains the highest scores among most of the considered metrics and clearly outperforms the BM25 base model across all the ranking metrics. Both weighted PRF-KLD variants can outperform the PRF (s) method for some individual metrics and mainly perform better for the binary metrics P@k and \({\textit{HITS}}@k\) compared to the BM25 base ranker. However, both are outperformed on the \({\textit{NDCG}}@k\) metric that takes into account the entire relevancy scale and additionally requires to rank GUIs with medium relevance high. Overall, the results indicate that AQE techniques in the form of PRF-KLD can improve the performance for NL-based GUI retrieval compared to the baseline BM25 ranking model on average, especially the variant that takes the GUI-specific text segments into account. Additionally, we provide a more detailed per-query analysis of the AQE evaluation results in our accompanying material (RaWi Repository 2022). For example, the \({\textit{MRR}}\) and P@5 metrics were improved or at least as good in 69% and 77% of the queries with an average improvement of .19 and .07 compared to the base model BM25 by applying the PRF (s) method. In particular, for queries that have many relevant GUIs in the GUI repository and the initial retrieval results with the base model BM25 provides relevant and cohesive GUIs, the AQE method can improve the results. For instance, for the query “a list of songs” meaningful expansion words such as artist, bookmark, and play can be extracted, which strengthen the query.

Considering the results of the trained BERT-LTR models, we can observe that all of the three models significantly outperform both the baseline ranking and AQE models. In our experiments, the pairwise (BERT-LTR (2)) and pointwise (BERT-LTR (3)) model appear to mainly perform best among the examined models. In addition, the pretrained Sentence-BERT baseline similarly outperforms the standard baseline ranking and AQE models, however, is significantly outperformed by the trained BERT-LTR models across all metrics. Regarding the \({\textit{NDCG}}@k\) metrics, the BERT-LTR models outperform the Sentence-BERT baseline to a large extent, indicating that the models learned to rank GUIs with both medium (\({\textbf {R}}={\textbf {1}}\)) and high relevance (\({\textbf {R}}={\textbf {2}}\)) higher in the overall ranking. Overall, the results indicate that the pretrained and fine-tuned BERT language model helps to better represent the semantics of the query and GUI documents and that, in combination with a LTR model trained specifically on the task of GUI ranking, it can be effectively applied to significantly improve the NL-based GUI ranking performance compared to the examined traditional IR methods and the pretrained Sentence-BERT baseline. In particular, these specifically trained semantic models are better in bridging the gap between the language used in NL queries and the GUI text representations. Figure 10 shows two example queries taken from the gold standard with their respective top-5 GUIs as ranked by the trained BERT-LTR (2) model. Each GUI is annotated with its ground truth relevance. For the queries we can observe that only a single GUI is annotated with a low relevance (due to an information overlay), whereas all other GUIs appear to have a medium to high relevance for the queries, which shows the ranking strength of the BERT-LTR.

In Fig. 11 we show the evaluation results of our controlled experiment. Each of the four boxplots (a)–(d) shows one of the four #GUI-comp metrics as a proxy for productivity as explained earlier. For the basic #GUI-comp count shown in Fig. 11a, we can observe that our approach outperforms the traditional approach to a large extent at each time step t. Naturally, the counts in both approaches increase with increasing t, however, the differences between the two approaches become smaller with increasing t. Participants are often able to find suitable starting GUI screens via adequate searches in RaWi and thus often begin with higher counts followed by smaller adjustments (adding and removal of components). In contrast, in the traditional approach it was often more difficult for participants to find an appropriate GUI at the beginning, however, repetitive component groups could be copied quickly after initial creation which led to a strong increase in counts over time t. When working with RaWi, participants were challenged to extract good queries on the basis of the given NL requirements and select relevant starting GUIs. Here, participants were not always able to find optimal starting GUIs, however, productivity improvements typically could still be observed in comparison to the traditional prototyping approach since RaWi simplified the task in general. For the #GUI-comp-div counts that take into account the content diversity shown in Fig. 11b, we can observe a similar behaviour as in (a), however, the differences between the two approaches are apparently larger. GUIs retrieved with RaWi typically display already diverse and realistic data since they are derived from GUI screenshots of real applications. In contrast, participants often copied repetitive component groups and often were not able to provide diverse data in their GUI prototypes with the traditional approach. For the #GUI-comp-neg counts that measure the components available not requested shown in Fig. 11c, we can observe that in both approaches the counts decrease over time t due to the removal of components available in the initial GUI screens that were not required, however, at the same time the GUI prototypes created with RaWi tend to contain more unrequested components. In RaWi, participants frequently found suitable screens in the beginning, yet often these GUIs included many additional components for other specific requirements that were not relevant for the requested prototype. However, we can observe a similar phenomenon in the traditional approach were participants that started with unsuitable GUI screens obtained high counts indicated by the outliers. In particular, one participant used a GUI with many unrequested components and did not manage to remove them up to minute 7. In addition, users of RaWi tend to focus on adding more requested functionality not yet contained in the starting GUIs and neglect removing unrequested components initially. This could potentially be due to the fact that the unrequested components in RaWi were possibly often perceived as less wrong (e.g., a star rating for each of multiple search results) compared to unrequested features in the templates in the traditional approach, and therefore potentially neglected initially more. For the adjusted #GUI-comp counts that represents the difference between the original count in (a) and (c) shown in Fig. 11d, we still can observe that our approach has higher counts at time t, however, the differences become smaller when punishing the prototypes for containing unrequested components. In addition to the differences observed in the boxplots, the Wilcoxon signed-rank test results indicate that the counts as a proxy for productivity using our approach are statistically significant larger at all considered time steps t (p-value < 0.05) and all considered counts (a), (b) and (d). Moreover, Cliffs \(\delta\) indicates a large difference between counts of the approaches also in the cases where the difference appears smaller in the boxplots. Overall, the results indicate that our approach can improve the prototyping productivity compared to a traditional GUI prototyping approach and therefore provides a better applicability for interactive GUI prototyping.

Figure 12 shows the evaluation results of the perceived usefulness of our GUI prototyping approach. The first question (a) regarding the support during the prototyping process received a high score (Mean: 3.73/SD: 0.80). The second question (b) regarding the perceived relevancy of the search results received a similarly high score (Mean: 3.63/SD: 0.68). The third question (c) regarding the support of the results received an even higher score (Mean: 4.21/SD: 0.78). Finally, the fourth question (d) regarding the support of the search results for better visualization received the highest score (Mean: 4.47/SD: 0.61). Overall, we achieved a SUS of 81.57 which indicates that our approach has a high usability compared to the average SUS of 68. Participants found our approach very easy to understand and use, even for inexperienced and novice users in GUI prototyping. In general, the participants reported further that they liked the large choice of available GUI screens to create a first GUI prototype quickly. Concerning improvements of the approach, participants mainly reported that the editor functionalities should be extended to include more editing options.

Guigle (Bernal-Cárdenas et al. 2019) similarly exploits automatically crawled Android apps from Google Play (Moran et al. 2018) to index multiple parts from the crawled GUI hierarchy data such as the app name, text and type from GUI components, the screen color and employs a basic Boolean query language for relevant GUI retrieval. In contrast, our approach RaWi enables users to specify simple NL-based searches to retrieve relevant GUIs and employs a more sophisticated ranking mechanism including BERT-based LTR models. In addition, RaWi enables users to directly employ retrieved GUIs to rapidly create GUI prototypes including the derivation of partly editable screens whereas Guigle stops after GUI image retrieval. However, a direct comparison of the retrieval performance is not possible due to the unavailability of the implementation and the employment of a different, much smaller GUI dataset in Guigle. As stated in their paper, Guigle is built on top of the open-source retrieval framework Lucence (Lucene 2011). The Lucene framework employs TF-IDF and BM25 retrieval methods, which are included in our evaluation experiments as well-known standard IR baselines. Therefore, although implementation details such as preprocessing and retrieval function parameters are not specified in Guigle, the presented IR baselines can be considered a good proxy to the retrieval methods employed in their approach. To support further research and enable a direct comparison between approaches in the first place, we publicly provide the first and high-quality gold standard for NL-based GUI retrieval on a state-of-the-art GUI repository in this work. Prior research already introduced NL-based GUI retrieval (Kolthoff et al. 2020, 2021), however, we (i) considerably extend the retrieval techniques using BERT-based LTR models, (ii) conduct a novel and more comprehensive retrieval performance evaluation on the basis of a newly created and contributed high-quality gold standard and (iii) provide an extensive user study for evaluating the GUI prototyping productivity improvements in practical GUI prototyping settings. Gallery D.C. (Chen et al. 2019) crawls a large number of real-world applications and automatically extracts GUI components such as various types of buttons from GUI screenshots and provides a multi-modal search supporting multiple dimensions such as width, height, main color and text for them. In addition, there is a plethora of GUI retrieval approaches that employ visual inputs such as screenshots, hand-drawn sketches or basic wireframes to retrieve similar GUIs in contrast to our approach that focuses on NL input. For example, Chen et al. (2020) trains an image autoencoder to find relevant GUIs employing a basic wireframe of the GUI as their input. Similarly, VINS (Bunian et al. 2021) expects visual input either as a basic wireframe prototype or fully designed GUI image to retrieve similar GUIs using a multi-modal embedding network. Swire (Huang et al. 2019) also employs a visual embedding approach to find relevant GUI images on the basis of hand-drawn sketches provided by the user. GUIFetch (Behrang et al. 2018) requires an entire Android application sketch as input to retrieve similar apps from GitHub. d.tour (Ritchie et al. 2011) uses stylistic keywords and stylistic similarity to find similar website designs. Using a web design as input and structure matching, FaceOff retrieves web GUI components fitting the design. Another approach proposes a neural translator for transforming GUI images to GUI skeletons (Chen et al. 2018). Screen2Vec (Li et al. 2021) proposes a technique for learning multi-modal (textual content, visual design and layout patterns) GUI embeddings, which are useful for many GUI-related downstream tasks (e.g., retrieving similar GUIs using a GUI as the query). Due to the difference of the problem, these GUI embeddings cannot be directly reused in our approach for comparison. The NL query embeddings (e.g., based on BERT) and Screen2Vec embeddings represent two separate embedding spaces that require alignment in order to enable NL-based GUI retrieval with Screen2Vec GUI embeddings. However, we included the same text-only Sentence-BERT baseline in our experiments, also used in their approach. The Screen2Words (Wang et al. 2021) approach proposes an Encoder–Decoder architecture for translating GUIs into short text descriptions. The GUI encoder could potentially be employed to obtain GUI embeddings, however, these embeddings similarly cannot be directly employed for retrieval based on NL queries as described earlier. Considering general-purpose search engines (e.g., Google), a direct comparison with our approach is not meaningful since these engines cannot be evaluated directly on our gold standard. Especially due to the huge differences in the employed datasets (specialized Rico GUI dataset vs. general Google Image dataset), it would be unclear whether differences between the retrieval performances are due to differences in the algorithms or due to the differences in the datasets. For example, since Google uses a general image dataset, many image results would be obtained that are not related to GUIs, compared to the Rico dataset specializing in Android GUI screenshots. To make the comparison between these approaches meaningful, the algorithm would need to be evaluated directly on the new gold standard.

Apart from GUI search approaches, retrieving code through NL queries has been studied extensively in research before (McMillan et al. 2011; Cambronero et al. 2019; Gu et al. 2018; Lv et al. 2015; Zhang et al. 2016). These approaches range from the application of traditional IR algorithms to specifically developed semantic Deep Learning architectures to find relevant program code for a given NL query. Recently, the CodeSearchNet challenge (Husain et al. 2019) released a dataset of NL queries and expert relevance annotations of respective functions from various programming languages to provide a source for systematic evaluation of code search approaches. In contrast, we created the first GUI retrieval gold standard in this work, to foster further research on retrieval methods specifically for NL-based GUI search.

In addition, many traditional prototyping approaches exist and are employed by many designers, developers and analysts in their everyday work such as Balsamiq (Faranello 2012), Sketch (Sketch 2019), Figma (Figma 2016) and Mockplus (Mockplus 2014), among others. These approaches enable users typically to create prototypes in a graphical editor on the basis of a small number of hand-crafted templates and basic GUI components supporting low-fidelity or high-fidelity prototyping. Other approaches such as UISKEI (Segura et al. 2012) and SketchiXML (Coyette et al. 2006) attempt to automatically identify GUI components from hand-drawn sketches and generate reusable GUI representations. Recent work on GUI prototyping assistance such as GUIComp (Lee et al. 2020) support novice users during the prototyping process via the recommendation of similar GUIs, provide various complexity metrics and visual attention maps based on the design currently created by the user. In contrast to these approaches, RaWi enables users to quickly retrieve matching GUIs based on simple NL queries from a large-scale GUI repository and automatically provides partly editable GUI screens to reuse for requirements elicitation via interactive GUI prototyping.