Crowdcloud: a crowdsourced system for cloud infrastructure

Crowdsourcing is usually defined as the practice of collecting and aggregating needed services, information, or other kind of resources provided by the general public [32]. Crowdsourcing as a practice has been utilised in numerous domains of study, such as business, computer science, and medicine. Crowdsourcing has also been utilised in several commercial and non-commercial platforms, such as Amazon Mechanical Turk [40] and Threadless [8], and has been structured in several forms, such as micro tasking [44] and crowdfunding [50].

Furthermore, when applied in complex tasks, crowdsourcing can leverage the so called wisdom of the crowd [62], e.g., crowdsourcing for the purpose of applying the wisdom of the crowd within enterprises [30]. The notion in this particular application of crowdsourcing is that apart from being an efficient and cost effective method of pooling resources, crowdsourcing can also provide qualitative benefits in task execution, i.e., by employing the collective intelligence of the crowd, ordinary people can often outperform individual experts.

In spite of the common understanding, crowdsourcing has been employed as a method for many years in various forms. One example is the provision of juries in several judicial systems. Another example that is widely used in the literature comes from 1906, when statistician Francis Galton observed that the aggregated result from the estimations of the weight of an ox from 800 people was accurate within the 1% error margin [21]. The latter example nicely demonstrates the mechanism that underpins crowdsourcing; individual contributions can be seen as sampling points of a probability distribution with the true answer as its mean value. This interpretation characterises crowdsourcing methodologies as relying on statistical sampling and therefore provides hints towards a wise crowd (e.g., homogeneous coverage of the sampling space, stochastic independence of trials, etc.).

In his book [62], Surowiecki has identified the four characteristics a crowd needs to have in order to be “wise”. These characteristics are:Later, we will refer to these characteristics when characterising the system requirements for Crowdsourced Systems.

During the past few years, smart phones and other truly portable devices (such as tablets, smart watches and smart glasses) have evolved into sophisticated multi-sensory computing platforms, thus fuelling the rise of MCSs. In [22], an overview is provided of the current state of applications that are based on MCS. The main challenges recognised refer to resource limitations, such as available energy, bandwidth and computational power, privacy issues that may arise due to the correlation of sensor data with individuals, and the lack of a unifying architecture that would optimise the cross-application usage of sensors on a particular device or even on a set of correlated devices (e.g., if they are located in the same geographical area).

In [55], authors use the notion of Participatory Sensing (PS) to describe such systems. They consider the problem of efficient data acquisition methods for multiple PS applications while taking into consideration issues such as resource constraints, user privacy, data reliability, and uncontrolled mobility. They evaluate heuristic algorithms that seek to maximise the total social welfare via simulations that are based on mobility datasets consisting of both real-life and artificial data traces.

In a previous work, we identified the basic design issues of MCS and investigated some characteristic challenges [5]. In particular, the core elements of an MCS were defined—the task, the server, and the textitcrowd—along with the functions governing their interactions. For a given type of task, and a finite budget, the server makes offers to the agents of the crowd based on some incentive policy. Then, each individual of the crowd makes a decision on whether to contribute to the execution of the task based on its own utility function. From this formulation, interesting results are extracted on the heuristics the server can follow in order to increase the efficiency of the system subject to the available budget.

In a more applied work presented in [6], an IoT testbed architecture for smart buildings was presented that enables the seamless and scalable integration of crowdsourced resources, such as smart phones and tablets. The purpose of this integration is two-fold; first, the embedded sensory capabilities of the resources provided by the crowd are combined with the sensing capabilities of the building for efficient smart actuations. Second, the system is able to interact with its users in a direct, personal way both for incentivising them to provide sensory data from their devices and for receiving feedback on their preferences and experienced comfort. This work is among the very first demonstrating the use of crowdsourcing in order to opportunistically augment the infrastructure of an ICT system. It also highlights the dual nature of crowdsourcing where the crowd not only contributes to the system but also ameliorates it, thus receiving services of higher quality. The same principles are also followed in [19], although in a different context. Here, the focus is on employing crowdsourcing as a powerful tool not only for conducting research, but also for driving research via the co-design of experiments for problems proposed by the crowd.

This brings us to the focal point of this study, i.e., the notion of crowdsourced systems and the application of crowdsourcing in the domain of cloud computing. The stimulating advantages of crowdsourcing and the extensive capabilities of cloud computing, plus the existence of similar characteristics (e.g., reducing costs and increasing diversity), facilitate a solid ground for the unification of the two practices. Such a unification has already been noticed and utilised in some cloud projects such as SETI@home [4] and BOINC [2]. However, these cloud projects belong to corporations and organisations, i.e., the crowd resources have been utilised not by other crowd members but by organisations. For example, SETI@home belongs to Berkeley and Microsoft Azure belongs to Microsoft. As a result, we believe that there is still a lack of a comprehensive cloud infrastructure that can actually be stemmed from the crowd, be organised by the crowd, and be utilised for the crowd. While Torrent clients and similar peer-to-peer platforms do exist, they are mainly used for file sharing and not for sharing other cloud resources such as computing power, cloud storage, and software on-demand services.

The cloud marketplace, as the online storefront for providing cloud services by cloud service providers, is not a recently devised concept [7]. However, the cloud market is still mainly dominated by a handful of mostly private (and sometimes public) cloud providers, which are both willing and capable of investing in their cloud service provision and the required infrastructure [7]. Examples of well-known cloud marketplaces include the Amazon AWS cloud marketplace, the Oracle cloud marketplace, the Microsoft Azure cloud marketplace, and the Salesforce AppExchange cloud marketplace [47].

With the move from Cloud 1.0 to Cloud 2.0, which adds the new Web 2.0 social networking functionalities to the cloud marketplace [52], more emphasis has been put on the role of new players in this marketplace. For example, the Open Cloud Exchange (OCX) has been proposed as a public cloud marketplace where several stakeholders participate in implementing and operating the cloud, as opposed to only one cloud provider [7]. Intercloud is another attempt to support and utilise the scaling of applications across multiple vendor clouds in the cloud marketplace [9], which we will discuss later as well. Other similar cloud marketplaces and their characteristics have been extensively researched and proposed, e.g., in [14, 48].

Most of these newer paradigms of cloud computing still have two very distinct players: cloud service providers and cloud service clients. In other words, cloud service clients usually cannot be cloud service providers at the same time. On the other hand, more recent attempts to unify cloud service providers and clients, such as Social Cloud [11], downplay the role of traditional cloud service providers and instead rely heavily on social networking components and services, but limit cloud service provision to those clients with whom a social networking link has already been established in the process. Such a limitation prevents Social Cloud from being a truly open marketplace to everyone, regardless of their social networking status. Consequently, a cloud paradigm that could benefit from the principles and fundamentals of crowdsourcing could be an advance in the cloud marketplace.

The high adoption rates of truly portable smart devices (e.g., smart phones and smart watches) by the general public, as well as the emergence of Do-It-Yourself computing platforms [49] (e.g., Raspberry Pis and Arduinos) have paved the way for new paradigms of computing and networking with strong distributed and ad-hoc characteristics. Both of these classes of devices are highly affordable and are supported by appropriate development tools that enable the public to use them in developing applications and systems with relative ease. When such applications and systems are designed to be open, then crowdsourcing methods can be employed for them to grow and scale.

One real-life example of such a project is Safecast [57], which was developed after the devastating earthquake and tsunami which struck eastern Japan in 2011, and the subsequent meltdown of the Fukushima Daiichi Nuclear Power Plant. Safecast enabled citizens to build their own DIY radiation monitoring sensor kits by employing open source and open data methodologies. Essentially, the crowd was enabled to monitor, collect, and openly share information on environmental radiation and other pollutants that was then aggregated and visualised in radiation maps.

The same rationale spawned FLOAT [20], in which citizens of Beijing were able to generate their own data on air quality in the city using sensor kits attached on kites. In this case, the design of the kits was also crowdsourced, thus carrying strong elements of co-creation as well.

Apart from such ad-hoc examples, that emerged somehow spontaneously, Mobile Crowdsensing Systems (MCSs) [22] have been thoroughly studied in the research community. The main technological enablers for MCS are smart phones; truly portable and personal devices equipped with a variety of sensors which are able to support several communication interfaces. MCS seek to exploit these characteristics by orchestrating the collaborative operation of multiple smart phones towards performing a task. Tasks in the context of MCS mainly focus on collecting sensory data and therefore crowdsensing systems can be regarded as distributed sensing infrastructures whose sensing points are crowdsourced.

Crowdsourced systems are systems whose constituent infrastructure is pooled or augmented via crowdsourcing methods. A clear distinction needs to be made between crowdsourced systems and crowdsourcing systems, also known as crowdsourcing platforms. Crowdsourcing systems or platforms act as tools enabling or facilitating the execution of a task via crowdsourcing. For instance, they may act as the gateway to the crowd (e.g., a web service individuals use) or provide supporting mechanisms such as a directory of active contributors or aggregation mechanisms. On the other hand, crowdsourced systems are themselves created or heavily rely on infrastructure that is crowdsourced. For instance, in [6] a smart building equipped with ambient luminance sensors is able to opportunistically augment its sensing infrastructure (and therefore improve the quality of service to the end users) by employing the embedded sensory capabilities of the smart phones of the end users. This also demonstrates the advantages of crowdsourcing in developing scalable systems in a cost effective way.

While a crowdsourced system may be developed with the aim of performing a particular task (such as in the case of Safecast and FLOAT), in the general case a crowdsourced system should be application agnostic. In order to characterise the “canonical” crowdsourced system—and therefore define the corresponding system paradigm—we revisit the prerequisites of the “wise” crowd as those are specified by Surowiecki (see Sect. 2.1).In light of the above discussion, Fig. 1 depicts the high-level architecture of a canonical crowdsourced system.

In general, a crowdsourced system has the purpose of performing a task and its architecture consists of three layers; the Crowd layer, the Crowdsourcer layer and the Task Execution layer. The base of a crowdsourced system is the crowd that contributes to the system by providing the infrastructure that is necessary for performing the task; these are referred to as crowdsourced resources. The crowdsourced resources are then employed by the crowdsourced system in order to provide data and/or services. On the top of the architecture lies the crowdsourcer, who is the key beneficiary of the operation of the crowdsourced system. The crowdsourcer issues the task to be executed and may also provide additional specifications related to the task, such as specifications for the incentive mechanisms to be employed or the available budget. Note that the crowdsourcer is identified as the issuer of the task; a role that may or may not be assumed by an individual of the crowd, depending on the context of operation of the system.

The Task Execution layer lies in between the Crowd layer and the Crowdsourcer layer, providing the necessary corresponding abstraction mechanisms. The Resources Gateway module provides connectivity between each individual resource and the Task Execution layer in such a way that the heterogeneity of the crowd is hidden from the upper layers. The Task Manager module provides the interface between the Task Execution layer (and subsequently the Crowd layer) and the Crowdsourcer layer. It enables the crowdsourcer to issue the task to the crowd while remaining agnostic of the complexity and the potential diversity of the underlying mechanisms. Towards facilitating the task execution, this layer also provides core management mechanisms. The Resource Manager module supports the curation of the crowdsourced resources and therefore of the crowd. The mechanisms embedded in this module may include (but not be limited to) maintaining a resource directory, implementing the incentive mechanisms, maintaining the corresponding ledgers for managing the available budget for incentives, and so on. Similarly, the Data Manager module supports the curation of the collected data by providing services related to normalising the received raw data according to a data model, mining the data and semantically annotating them, data aggregation services, etc.

In this work, we only provide a high level presentation of the architecture for a canonical crowdsourced system. There exist several other aspects that need to be addressed, such as trust and privacy issues, preserving anonymity for the crowd, the efficient use of incentive mechanisms for efficient task execution, the use of open interfaces and open data in crowdsourced systems, and so on. These will be elaborated in our future work. In the following section, we will compare the notion of crowdsourced systems to other already-existing paradigms.

As already mentioned, the paradigm of crowdsourced systems is underpinned by the high acceptance rates of DIY computer platforms [49], i.e., affordable hardware platforms that can be used by amateurs and the general public to develop small ICT projects. Certain DIY computer platforms, such as some flavours of the Raspberry Pi, while being affordable are also characterised by significant computational resources and communication capabilities. As depicted in Fig. 1, the paradigm of crowdsourced systems provisions the federation of several such devices towards a distributed system that provides services and infrastructure to a set of users.

While this may seem similar to or a special case of cloud-based paradigms, such as the Intercloud [26], crowdsourced systems carry unique characteristics and pose distinct challenges that justify their identification as a new paradigm. In particular, crowdsourced systems differ in at least the following ways.

Infrastructure and services provision and management in cloud-based systems the cloud provider provisions and provides access to the ICT infrastructure and the services that run on top of it. This means that cloud systems are centralised systems in the sense that they are centrally managed. This, of course, does not preclude the adoption of distributed architectures in the way the hardware and the services are deployed and managed, but the management and the provision of those is carried out centrally by the cloud provider (e.g., consider the Amazon Web Services). Cloud paradigms like Intercloud [26] consider the federation of multiple individual clouds in the context of distributed architectures. Such federations allow individual clouds to share and exchange data, and to distribute loads among them, overall providing improved services to their end users in terms of availability, scalability and elasticity. Crowdsourced systems radically differ as their infrastructure is not provided by a centralised provider but is pooled from the crowd consisting of independent participants. This means that the infrastructure of a crowdsourced system is not centrally managed and therefore it is characterised by heterogeneity and uncertainty regarding its availability. Furthermore, a crowdsourced system is not a federation of individual, stand-alone clouds but a system whose constituent infrastructure is crowdsourced by contributing individuals. One way to regard it is that such systems follow similar organisation principles to cooperative initiatives where the provider is also a consumer, i.e., a “prosumer” [56].

The human factor as mentioned above, crowdsourced systems pool their infrastructure via means of crowdsourcing from the general public. One direct implication is finding the answer to why someone should contribute to the crowdsourced system. Therefore, a crowdsourced system should incorporate an incentive mechanism, either implicitly (e.g., the contributors benefit from the operation of the system itself) or explicitly (e.g., monetary incentives via micro-payments). Another important aspect is the fact that DIY platforms [49] (e.g., the Raspberry Pi)—that are key enablers of crowdsourced systems—are commonly used in small scale projects by amateurs and are deployed in sensitive premises such as homes and work environments. This means that aspects such as anonymity, privacy, and trust lie in the core of crowdsourced systems. Overall, due to their nature, the human factor and the challenges it poses are more profound in crowdsourced systems than in cloud-based systems.

Crowdsourcing systems and crowdsourced systems both rely on contributions from the general public (i.e., the crowd). However, in spite of the similarity of the terms, crowdsourced systems clearly differentiate from crowdsourcing systems. Crowdsourcing systems typically refer to digital platforms that collect and consolidate input from the crowd. In this case the input may come in several forms; a recommendation or a review, a vote, a video, sharing of location, or sensory data in the context of a crowdsensing application. A crowdsourcing platform may or may not be centralised or distributed in terms of system architecture. Most often, however, the platform is deployed and centrally managed. On the other hand, in crowdsourced systems the general public contributes to the system itself. Individuals do not only provide input to the system but also provide the means for collecting, processing, and curating data and information.

As already mentioned, one of the key enabling technologies for the paradigm of crowdsourced systems is the DIY computer platforms such as Raspberry Pi and Arduino. Such platforms are also regarded as key enablers for the IoTs, mainly due to their small size and their use in small automation projects. IoT as a Service is a system paradigm in which smart devices are interconnected with a remote cloud infrastructure via which their functionalities (e.g., sensor measurements) are made available [10]. Here, although the IoT devices may be deployed in several different areas (thus allowing such systems to be characterised as distributed), the core of the system remains centralised—at least from a management perspective—as it is based on a cloud provider. This is a fundamentally different approach to crowdsourced systems, where the components of the system itself are crowdsourced. Also, the scope of a crowdsourced system is much broader as it is not restricted in IoT applications.

Crowdcloud [28] refers to the provision of computing services at different levels of IaaS, PaaS, and SaaS by the crowd and for the crowd. The crowd, in this definition, can include both individuals and organisations. Crowdcloud acts like an online free market where every individual and every organisation can supply their resources or demand for other crowd members’ resources, following the regulations of the free market. The idea of crowdcloud applies several features of crowdsourcing such as largeness, diversity, and incentives provision [29] in the cloud. Crowdcloud also builds upon the notion of crowdsourced systems in terms of architecture and application. Crowdcloud lets the crowd provide their idle resources to other individuals or organisations in the crowd, and also request their required resources from other crowd members. This can happen at the infrastructure level (i.e., IaaS), e.g., by providing or asking for CPU power and storage space, at the platform level (i.e., PaaS), e.g., by providing or asking for runtime libraries and web servers, or at the software level (i.e., SaaS), e.g., by providing or asking for email applications and on-demand software systems.

Ordinary cloud services, such as Amazon EC2 or Google Drive, are cloud services which are provided by organisations for other organisations or people. These cloud services come with a legally binding contract between the cloud service provider and the cloud service client and are mostly costly for other organisations, but they are usually free or inexpensive for individuals to use. In some cases where provided cloud services are free of charge, organisations usually compensate for the costs by introducing advertisements along with their free cloud services. Furthermore, these services are generally managed in a centralised way, and this has instigated issues related to data control and privacy [74] as well as legal issues [17]. Crowdcloud, on the other hand, is fully decentralised, provides cloud services from the crowd to the crowd, and can be contract-free.

Crowdcloud bears some similarities with a few concepts in the literature, such as cloudsourcing, volunteer cloud computing, and social cloud. However, the differences between crowdcloud and these concepts render crowdcloud as a novel idea and make crowdcloud stand out as an entirely free market model for cloud service provision. These differences will be discussed in detail in the following paragraphs.

Cloudsourcing refers to outsourcing various elements of a business or organisation IT infrastructure to other companies or organisations which provide such services in the cloud [23, 41]. Therefore, the first difference between cloudsourcing and crowdcloud lies both in the cloud service providers and cloud service clients. In cloudsourcing, organisations provide some cloud services to other organisations. In crowdcloud, however, the crowd provide some cloud services to the crowd. The second difference between the two is that cloudsourcing is centralised while crowdcloud is not. The last difference between cloudsourcing and crowdcloud is that cloudsourcing is always based on a contract between two organisations, while crowdcloud may or may not be contract-based.

Volunteer cloud computing, also known as peer-to-peer computing or global computing, refers to the use of computers volunteered by the general public for distributed scientific computations [3, 15]. SETI@home and BOINC are examples of volunteer cloud computing. In this case, and apart from its aforementioned purpose, two main differences exist between volunteer cloud computing and crowdcloud. The first difference is that in volunteer cloud computing, the crowd provides a service, such as CPU power or storage, solely for organisations (and normally for research purposes) and not for other people. The second difference is that volunteer cloud computing, unlike ordinary cloud services, is not based on a contract and people have no obligations whatsoever to provide cloud services or keep providing cloud services to their beneficiaries. Crowdcloud, however, can be both contract-free and contract-based.

Social cloud is probably the closest in meaning and application to crowdcloud. Social cloud refers to a framework for sharing resources and services based on relationships amongst the members of a social network [12]. The notion of social cloud implies three ideas that form the differences between social cloud and crowdcloud. The first difference is that social cloud depends on a social network and relationships amongst the members of that social network. This limits the cloud service provision to socially connected members within the social network. Crowdcloud, on the other hand, is not necessarily a social network, and can exist independently as an online free market for cloud services. This provides a wider range of services to acquaintances and non-acquaintances alike. The second difference is that social cloud works solely on social contracts, while crowdcloud can work on legally binding contracts or be contract-free. Finally, social cloud explicitly limits the use of each individual’s resources to other individuals. Crowdcloud, on the other hand, is open to both individuals and organisations, for-profit or non-profit, for the use of resources.

The differences between crowdcloud and other similar cloud services are shown in Table 1. These differences include who the service providers and service clients are, how these platforms are managed, and whether a contract is needed between cloud service providers and cloud service clients for the use of cloud services.

The proposed architecture for crowdcloud is presented in this section along with a short description of its constituents. It is depicted in Fig. 2.

As Fig. 2 illustrates, crowdcloud has the following three constituents: the cloud, the crowd, and the crowdcloud platform. The crowdcloud platform utilises three distinct modules which interact with each other. These modules are explained below:

In this section, some of the characteristics and advantages of crowdcloud are presented. In the same fashion, some challenges that the implementation of crowdcloud can introduce are discussed and possible solutions to avoid or mitigate them are presented. It should be noted that crowdcloud, as a novel concept, will need more theoretical research to be conducted before any implementation attempts are made in order to guarantee a quality service which addresses all benefits and possible challenges accordingly.