Intelligent Crowdsourced Testing | springerprofessional.de

Springer Professional

nach oben

2022 | Buch

Kapitel lesen Erstes Kapitel lesen

Intelligent Crowdsourced Testing

verfasst von: Prof. Qing Wang, Prof. Zhenyu Chen, Assoc. Prof. Junjie Wang, Dr. Yang Feng

Verlag: Springer Nature Singapore

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Einloggen, um Zugang zu erhalten

Über dieses Buch

In an article for Wired Magazine in 2006, Jeff Howe defined crowdsourcing as an idea for outsourcing a task that is traditionally performed by a single employee to a large group of people in the form of an open call. Since then, by modifying crowdsourcing into different forms, some of the most successful new companies on the market have used this idea to make people’s lives easier and better. On the other hand, software testing has long been recognized as a time-consuming and expensive activity. Mobile application testing is especially difficult, largely due to compatibility issues: a mobile application must work on devices with different operating systems (e.g. iOS, Android), manufacturers (e.g. Huawei, Samsung) and keypad types (e.g. virtual keypad, hard keypad). One cannot be 100% sure that, just because a tested application works well on one device, it will run smoothly on all others.

Crowdsourced testing is an emerging paradigm that can improve the cost-effectiveness of software testing and accelerate the process, especially for mobile applications. It entrusts testing tasks to online crowdworkers whose diverse testing devices/contexts, experience, and skill sets can significantly contribute to more reliable, cost-effective and efficient testing results. It has already been adopted by many software organizations, including Google, Facebook, Amazon and Microsoft.

This book provides an intelligent overview of crowdsourced testing research and practice. It employs machine learning, data mining, and deep learning techniques to process the data generated during the crowdsourced testing process, to facilitate the management of crowdsourced testing, and to improve the quality of crowdsourced testing.

Anzeige

Inhaltsverzeichnis

Frontmatter

Preliminary of Crowdsourced Testing

Frontmatter

Chapter 1. Introduction

Abstract

The Internet is a decisive technology of the information Age, as the electrical engine was the vector of technological transformation of the Industrial Age. This global network of computer networks, largely based nowadays on platforms of wireless communication, provides ubiquitous capacity of multimodal, interactive communication in chosen time, transcending space.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 2. Preliminaries

Abstract

Figure 2.1 presents the overall procedure of crowdsourced testing. The project manager provides a test task for crowdsourced testing, including the software under test and test requirements. The crowdsourced testing task is usually in the format of open call, so a large number of crowd workers can sign in to perform the task based on its test requirements, and are required to submit crowdsourced test reports. The project manager then inspects these submitted test reports, confirm whether it is a bug, debug and fix it. Note that not every test report involves a bug, and different reports might describe the same bug (i.e., duplicate reports).

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 3. Book Structure

Abstract

Despite the successful application of crowdsourced testing as we introduced in previous chapters, there are still challenges faced to fully take advantage of this new testing paradigm. For example, previous chapters mentioned there are large number of duplicate reports submitted by diversified crowd workers who conduct the testing in parallel. It is extremely valuable to automatically detect those duplicate reports and aggregate them together for a summarized view, which can facilitate the crowdsourced testing managers in their following bug resolving activities. Another example is that the previous industrial case study has also revealed that some crowd workers can detect bugs in a crowdsourced testing task, while others may fail to detect bugs in the same task. If there is automatic support for recommending a set of crowd workers to a specific crowdsourced testing task, more bugs can be detected, and the bug detection activity would be more efficient.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Supporting Technology for Crowdsourced Testing Workers

Frontmatter

Chapter 4. Characterization of Crowd Worker

Abstract

This chapter presents how we characterize the crowd workers so as to conduct the task recommendation and worker recommendation to facilitate the crowdsourced testing practice. We propose a data-driven crowd worker characterization which characterizes the crowd workers automatically mining from the historical crowdsourced testing repositories. We first conduct explorations about the crowd worker’s characteristics from mining the historical repositories to motive the characterization in Sect. 4.1, then present the characterization with activeness, preference, expertise, and device in Sect. 4.2.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 5. Task Recommendation for Crowd Worker

Abstract

A wealth of previous literature has shown the inequalities gap between task requesters’ and workers’ decision support provided in crowdsourcing platforms. On the one hand, many platforms allow requesters to assess worker performance data and support the gauging of qualification criterion in order to control the quality of crowd submissions. On the other hand, workers are usually provided with very limited support throughout the task selection and completion processes. In particular, most workers must manually browse through a long list of open tasks before they determine which ones to sign up. Manual task selection not only is time consuming, but also tends to be sub optimal due to subjective, ad hoc worker behaviors.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Supporting Technology for Crowdsourced Testing Tasks

Frontmatter

Chapter 6. Crowd Worker Recommendation for Testing Task

Abstract

One challenging problem with crowdsouced testing is optimizing crowd workers’ participation. Crowd resources, while cheap, are not free. Hence, when scaling up crowdsouced testing, it is necessary to maximize the information gain from every member of the crowds. Also, not all crowd workers are equally skilled at finding bugs. Inappropriate workers may miss bugs, or report duplicated bugs, while hiring them requires nontrivial budgets.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 7. Crowdsourced Testing Task Management

Abstract

Trade-offs such as “how much testing is enough” are critical yet challenging project decisions in software engineering. Insufficient testing can lead to unsatisfying software quality, while excessive testing can result in potential schedule delays and low cost-effectiveness. This is especially true for crowdsourced testing given the complexity of mobile applications and unpredictability of distributed crowdsourced testing process.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Supporting Technology for Crowdsourced Testing Results

Frontmatter

Chapter 8. Classification of Crowdsourced Testing Reports

Abstract

In order to attract workers, testing tasks are often financially compensated, especially for these failed reports. Under this context, workers may submit thousands of test reports. However, these test reports often have many false positives, i.e., a test report marked as failed that actually involves correct behavior or behavior that was considered outside of the studied software system. Project managers or testers need to manually verify whether a submitted test report reveals fault—-true fault. However, such process is tedious and cumbersome, given the high volume workload. Our observation on industrial crowdsourced testing platforms shows that, approximately 100 projects are delivered per month from a typical platform, and more than 1000 test reports are submitted per day on average.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 9. Duplicate Detection of Crowdsourced Testing Reports

Abstract

The benefit of crowdsourced testing must be carefully assessed with respect to the cost of the technique. At first place, crowdsourced testing is a scalable testing method under which large software systems can be tested with appropriate results. This is particular true when the testing is related with the feedback on GUI systems, or subjective opinions about different features.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 10. Prioritization of Crowdsourced Testing Reports

Abstract

In Chaps. 8 and 9, we have mentioned that it is often time-consuming and tedious to manually inspect all received test reports. Besides automatic classification of test reports introduced in Chap. 8 and duplicate detection of test reports introducted in Chap. 9, this chapter seeks for other alternatives for managing these test reports, i.e., to prioritize test reports for manual inspection.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 11. Summarization of Crowdsourced Testing Reports

Abstract

However, while all of these techniques are built on the assumption that duplicate reports are harmful to software maintenance and aim at filtering out this information, Zimmermann et al. and Bettenburg et al. empirically found that duplicate reports are helpful for report comprehension and debugging.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 12. Quality Assessment of Crowdsourced Testing Cases

Abstract

While the crowdsourced testing often calls for a large number of end-users, who may have no knowledge on programming, to conduct the black-box testing tasks. It can also gathers a group of professional testers to write test code for white-box testing. To efficiently process the crowdsourced testing reports that consist of textual descriptions and screenshots, we have discussed various crowdsourced test report processing techniques in the previous chapters. However, it is difficult to apply these techniques on test cases which are often implemented in various coding styles.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Conclusions and Future Perspectives

Frontmatter

Chapter 13. Conclusions

Abstract

Software testing is an important, yet often overlooked, part of the software development lifecycle. Traditionally, software testing was conducted by dedicated quality assurance teams with formally trained testers. Although these quality assurance teams are reliable, the high cost and delayed responses made them hard to scale and non-flexible for rapid update needs for the software industry today.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Chapter 14. Perspectives

Abstract

The growing ubiquity of data-driven learning models in algorithmic decision-making has recently boosted concerns about the issues of fairness and bias. Friedman defined that a computer system is biased “if it systematically and unfairly discriminates against certain individuals or groups of individuals in favor of others”. For example, job recommenders can target women with lower-paying jobs than equally-qualified men. News recommenders can favor particular political ideologies over others. And even ad recommenders can exhibit racial discrimination.

Qing Wang, Zhenyu Chen, Junjie Wang, Yang Feng

Backmatter

Titel: Intelligent Crowdsourced Testing
verfasst von: Prof. Qing Wang
Prof. Zhenyu Chen
Assoc. Prof. Junjie Wang
Dr. Yang Feng
Copyright-Jahr: 2022
Verlag: Springer Nature Singapore
Electronic ISBN: 978-981-16-9643-5
Print ISBN: 978-981-16-9642-8
DOI: https://doi.org/10.1007/978-981-16-9643-5

Premium Partner

Weitere Formate
Fachgebiete
Bücher
Zeitschriften
Themenseiten
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024