Crowdsourcing

Large scale software systems with increasing complexity, variability and uncertainty, brings about grand challenges for traditional software engineering. Recently, crowdsourcing practices in the domain of software development such as Apple App Storeand TopCoder have exhibited a promising and viable solution to the issues. The use of a crowd for developing software is predicted to take its place alongside established methodologies, such as agile, global software development, service-oriented computing, and the traditional waterfall. In this chapter, we propose a conceptual framework for the emerging crowdsourcing development methodology. We define the fundamental principles, software architecture, development process, and maturity model of the methodology for crowdworkforce motivation, coordination and governance.

Today’s software systems build on open source software. Thus, we need to understand how to successfully create, nurture, and mature the software development communities of these open source projects. In this article, we review and discuss best practices of the open source volunteering and recruitment process that successful project leaders are using to lead their projects to success. We combine the perspective of the volunteer, looking at a project, with the perspective of a project leader, looking to find additional volunteers for the project. We identify a five-stage process consisting of a connecting, understanding, engaging, performing, and leading stage. The underlying best practices, when applied, significantly increase the chance of an open source project being successful.

Crowdsourcing is emerging as a compelling technique for the cost-effective creation of software, with tools such as ODesk and TopCoder supporting large scale distributed development. From the point of view of the commissioners of software, there are many advantages to crowdsourcing work—as well as cost, it can be a more scalable process, as there is the possibility of selecting from a large pool of expertise. From the point of view of workers, there is a different set of benefits, including choice of when and how to work, providing a means to build a portfolio, and a lower level of commitment to any particular employer. The crowdsourcing of software development—in common with some other activities such as design—represents an alternative to existing mechanisms that require skilled workers. However, if crowdsourcing were to replace traditional employment for a significant proportion of software developers, the reduced levels of commitment between workers and commissioners could prove problematic for workers over time. In this paper, explore three areas of interest: (i) trust and reputation development; (ii) team selection and team building; (iii) contextualisation of the work carried out. By drawing together work in these areas from the point of view of workers rather than commissioners, we highlight some of the incipient issues with the growth of crowdsourced labour. We also explore ways in which crowdsourcing of software development—and other skilled practices—differers from microtasking.

The term “social machines” denotes a class of systems where humans and machines interact so that computational infrastructure supports human creativity. Flagship examples such as Wikipedia and Ushahidi demonstrate how computational coordination can enhance information sharing and aggregation, while the Zooniverse family of projects show how social machines can produce scientific knowledge. These socio-technical systems cannot easily be analysed in purely computational or purely sociological terms, and they cannot be reduced to Turing machines. Social machines are used in the creation of software, from software crowdsourcing projects such as TopCoder and oDesk, to distributed development platforms such at GitHub and Bitbucket. Hence, social machines are increasingly used to create the software infrastructure for new social machine. However, social machine development is a more complex process than software development, as the community must be “programmed” as well as the machines. This leads to development in the context evolving and unknown requirements, and having to deal with more sociological concepts than formal systems designers usually work with. We hence model the process using two coupled social machines: the target social machine, with whatever purposes the creators envisions, and the development social machine which is used to create it. As an example, oDesk can form part of a development social machine which might be used to create a target social machine, e.g. “the next Facebook”. In this chapter, we describe a formalism for social machines, consisting of i) a community of humans and their “social software” interacting with ii) a collection of computational resources and their associated state, protocols and ability to analyse data and make inferences. We draw on the ideas of ‘desire lines’ and ‘play-in’ to argue that top down design of social machines is impossible, that we hence need to leverage computational support in creating complex systems in an iterative, dynamic and emergent manner, and that our formalism provides a possible blueprint for how to do this.

The advent and successful practices of software crowdsourcing needs to investigate its in-depth essence and seek effective technologies to support its activities and satisfy increasing requirements. We highlight crowdsourcing participants consist of a multi-agent system and software crowdsourcing is a multi-agent problem-solving process. This paper discusses the characteristics and challenges of software crowdsourcing in contrast to traditional software development, and present a general analysis framework based on multi-agent system to examine the organization and behaviours of software crowdsourcing. Several software crowdsourcing models performed on typical platforms like Topcode, uTest are established and their organization and coordination are discussed. We have developed a service-based multi-agent system platform called AutoServicethat provides some fundamentalcapabilities like autonomy, monitoring, flexible interaction and organization, and can serve as an infrastructure to support software crowdsourcing models and tackle its challenges. A software crowdsourcing prototype is developed and some scenarios are exemplified to illustrate our approach.

Crowdsourcing systems of the future (e.g., Social Compute Units—SCUs, collective adaptive systems) need to support complex collaborative processes, such as software development. This presupposes deploying ad-hoc assembled teams of human and machine services that actively collaborate and communicate among each other, exchanging different artifacts and jointly processing them. Major challenges in such environments (e.g., team formation, adaptability, runtime management of data-flow and collaboration patterns) can be somewhat alleviated by delegating the responsibility and the know-how needed for these duties to the participating crowd members, while indirectly controlling and stimulating them through appropriate incentive mechanisms. Existing process-centric collaboration modeling approaches (e.g., workflows) are incapable of encoding such incentive mechanisms. Therefore, in this paper we analyze different interaction aspects that incentive mechanisms cover and formulate them as requirements for future systems to support. We then propose an artifact-centric approach for modeling incentives in rich crowdsourcing environments that meets these requirements.

Crowdsourcing has demonstrated its capability of supporting various software development activities including development and testing as it can be seen by several successful crowdsourcing platforms such as TopCoder and uTest. However, to crowd source large-scale and complex software development and testing tasks, there are several optimization challenges to be addressed such as division of tasks, searching a set of registrants, and assignment of tasks to registrants.Since in crowdsourcing a task can be assigned to registrants geographically distributed with various backgrounds, the quality of final task deliverables is a key issue. As the first step to improve the quality, we propose a systematic and automated approach to optimize the assignment of registrants in a crowdsourcing platform to a crowdsourcing task. The objective is to find the best fit of a group of registrants to the defined task. A few examples of factors forming the optimization problem include budget defined by the task submitter and pay expectation from a registrant, skills required by a task, skills of a registrant, task delivering deadline, and availability of a registrant. We first collected a set of commonly seen factors that have impact on the perfect matching between tasks submitted and a virtual team that consists of a selected set of registrants. We then formulated the optimization objective as a fitness functionłthe heuristics used by search algorithms (e.g., Genetic Algorithms) to find an optimal solution. We empirically evaluated a set of well-known search algorithms in software engineering, along with the proposedfitness function, to identify the best solution for our optimization problem. Results of our experiments are very positive in terms of solving optimization problems in a crowdsourcing context.

Crowdsourcing markets, such as Amazon’s Mechanical Turk, are designed for easy distribution of micro-tasks to an on-demand scalable workforce. Improving the quality of the submitted results is still one of the main challenges for quality control management in these markets. Although beneficial effects of synchronous collaboration on the quality of work are well-established in other domains, interaction and collaboration mechanisms are not yet supported by most crowdsourcing platforms, and thus, not considered as a means of ensuring high-quality processing of tasks. In this paper, we address this challenge and present a new method that extends majority vote, one of the most widely used quality assurance mechanisms, enabling workers to interact and communicate during task execution. We illustrate how to apply this method to the basic scenarios of task execution and present the enabling technology for the proposed real-time collaborative extension. We summarize its positive impacts on the quality of results and discuss its limitations.

Recently software crowdsourcing has become an emerging development paradigm in software ecosystems. This paper first introduces a software crowdsourcing framework in the context of software ecosystems. The framework includes a game-theoretical model for peer software production to describe the competitive nature of software crowdsourcing. The analysis of this model indicates that if the only reward is the prize, only superior developers will participate in the software crowdsourcing. This explains the phenomenon that while software crowdsourcing is open for anyone to compete, but only few will engage in competition. This is validated by a large historical data collected from a popular software crowdsourcing website over a 10-years period. Further, we perform a case study on a NASA software crowdsourcing project to take a closer examination at how community developers participate in different types of tasks through the software crowdsourcing process.

Software development is either creation activities that rely on developers creativity and talents, or manufacturing activities that follow the engineering processes. Engineering processes need to include creation activities to address tasks such as requirement elicitation and bug finding. On the other hand, by exploiting the crowd wisdom, open-source development has been demonstrated to be a suitable environment for software creation. However, it also has several limitations, such as guaranteeing the progress and quality of production process. This paper introduces a software development platform and ecosystem that combines the strengths of the two models. First, we propose the Trustworthy Software Model as a basis to support such a hybrid development ecosystem. The core of this model contains a novel lifecycle model, an evidence model and an evolution model. Second, based on the model, we propose the Trustworthy Software Development and Evolution Service Model. It integrates crowd collaboration, resource sharing, runtime monitoring, and trustworthiness analysis into an unified framework. Based on this integrated model, we designed and implemented TRUSTIE, which distinguishes itself from other software crowdsourcing platforms by providing the software collaborative development service and the resource sharing service with the general support of trustworthiness-analysis tools. TRUSTIE enables crowd-oriented collaboration among internal development teams and the external crowds by combining the software creation and software manufacturing in one ecosystem.

Software crowdsourcing is becoming an increasingly viable model for creating production software addressing every aspect of the software development lifecycle. However, as software development processes become yet more complex requiring dedicated systems for development, testing, and deployment, software crowdsourcing projects must also acquire considerable infrastructure in order to facilitate development. We propose the use of an infrastructure crowdsourcing model, termed a Social Cloud, to facilitate a user-contributed cloud fabric on which software development services and systems can be hosted. Social Clouds are motivated by the needs of individuals or groups for specific resources or capabilities that can be made available by connected peers. Social Clouds leverage lessons learned through volunteer computing and crowdsourcing projects such as the willingness of individuals to make their resources available and offer their expertise altruistically for “good causes” or in exchange for other resources or payment. In this chapter we present the Social Cloud model and describe how it can be used to crowdsource software infrastructure.

With the rapid growth of Web Services in the past decade, the issue of QoS-aware Web service recommendation is becoming more and more critical. Web service QoS is highly relevant to the corresponding invocation context like invocation time and location. Therefore, it is of paramount importance to collect the QoS data with different invocation context. We have crawled over 30,000 Web services distributed across Internet. In this work, we propose to use crowdsourcing to collect the required QoS data. This is achieved through two approaches. On the one hand, we deploy a generic Web service invocation client to 343 Planet-Lab nodes and these nodes serve as simulated users distributing worldwide. The Web service invocation client is scheduled to invoke target Web services from time to time. On the other hand, we design and develop a mobile crowdsourced Web service tesing framework on Android platform, with which a user can easily invoke selected Web services. With the above two approaches, the observed service invocation data, e.g. response time, will be collected in this way. Then we design a TemporalQoS-Aware Web Service Recommendation Framework to predict missing QoS value under various temporal context. Further, we formalize this problem as a generalized tensor factorization model and propose a Non-negative Tensor Factorization (NTF) algorithm which is able to deal with the triadic relations of user-service-time model. Extensive experiments are conducted based on collected Crowdsourced testing data. The comprehensive experimental analysis shows that our approach achieves better prediction accuracy than other approaches.

In the vast galaxy of crowdsourcing activities, crowd-sensing consists in using users’ cellphones for collecting large sets of data. In this chapter, we present the APISENSE distributed crowd-sensing platform. In particular, APISENSE provides a participative environment to easily deploy sensing experiments in the wild. Beyond the scientific contributions of this platform, the technical originality of APISENSE lies in its Cloud orientation, which is built on top of the soCloud distributed multi-cloud platform, and the remote deployment of scripts within the mobile devices of the participants. We validate this solution by reporting on various crowd-sensing experiments we deployed using Android smartphones and comparing our solution to existing crowd-sensing platforms.