A privacy-preserving exception handling approach for dynamic mobile crowdsourcing applications

The ability values of candidate workers play an important role in worker selection for mobile crowdsourcing applications. In the outsourcing platform Amazon Mechanical Turk, the candidate workers are evaluated, compared, and selected for an outsourcing task based on their ability values, e.g., worker reputation and worker skills. In [10], a self-organized outsourcing toolkit, i.e., Crowdlet is proposed to employ a set of workers to execute the crowdsourcing task from the requester. In Crowdlet, a candidate worker is selected or not depends on the worker’s service quality levels which are influenced by the skills, arrival time, and rewards of the worker. While the reputation values and mobile device performance are often neglected. In [11], the authors suggest an online crowdsourcing framework which is response for allocating appropriate workers to crowdsourcing tasks based on the important information from both workers and requesters. The final goal is to maximize the utility function defined in the crowdsourcing framework. While the utility function only depends on the state of network. In [12], a behavior-aware worker selection strategy is suggested to improve crowdsourcing performances, which is often self-adaptive. Typically, the strategy calculates the reputation value of a candidate worker based on the worker’s historical service quality data. However, the reputation model only considers the skills of workers without considering other elements such as device capability. In [13], a budget-aware crowdsourcing model evaluates and generates the final crowdsourcing results based on the workers’ information, such as his/her willingness, his/her reliability in executing the task. Therefore, high-quality crowdsourcing results can be expected within a certain budget. However, the decision-making basis of crowdsourcing is still not comprehensive enough.

An online auction mechanism is put forward in [14] to satisfy the dynamic crowdsourcing scenarios where each candidate worker can arrive at or leave the crowdsourcing platform at will. The auction mechanism can be utilized to accommodate the dynamic arrivals of both candidate workers and crowdsourcing tasks. However, it is often hard to get the optimal crowdsourcing performance through auction. In [15], an incentive mechanism is brought forth to evaluate and rank the candidate workers for a certain crowdsourcing task based on the workers’ context information, e.g., location and time. However, in this work, the proposed mechanism does not consider the ability values of workers. In [16], the authors put forward an online double auction strategy between candidate workers and task requesters. The proposal can guarantee to achieve better crowdsourcing results than the simple one-side interaction between workers and requesters. An online learning method is brought forth in [17] to produce the optimal pricing strategy and incentive mechanism, whose final goal is regret minimization. In [18], two online auction mechanisms are suggested to accommodate the dynamic arrival of candidate workers; the final optimization goal is to recruit enough workers to maximize the defined utility value before deadline constraint. In [19], the candidate workers are encouraged to publish their configuration files to the crowdsourcing platform; this way, the platform can reduce the budget while selecting enough workers to execute a crowdsourcing task. However, the abovementioned auction strategies in [17‐19] are often not efficient enough to satisfy the quick response requirements from the task requesters in mobile crowdsourcing applications. Moreover, additional auction burdens are possible for both the candidate workers and the task requesters.

Through the above analyses, a conclusion can be drawn that existing mobile crowdsourcing approaches seldom consider the probable exceptions (as well as the resulted privacy leakage in similar worker search and replacement) caused by worker unavailability in the dynamic crowdsourcing environment. Considering this drawback, a privacy-preserving exception handling approach for dynamic mobile crowdsourcing is proposed in this paper, to smooth the normal execution of crowdsourcing process and improve the crowdsourcing robustness. The details of our exception handling approach will be introduced in Section 4.

As Section 3 introduces, if worker_i (∈WORKER) has ever executed the crowdsourcing task task_j (∈TASK), then the value (indicating whether a worker has ever executed a task) for worker_i and task_j (denoted by v_i,j in this paper) is equal to 1; else, v_i,j = 0. Afterwards, an m*n matrix is obtained as in Eq. (1), where each entry is a Boolean value. Furthermore, the ith row of the matrix, i.e., V_i = (v_i,1, …, v_i,n) represents the historical task execution records of worker_i. As we introduce in Section 1, V_i should be protected by the crowdsourcing platform CP as V_i often contains certain sensitive information of worker_i.

Next, we introduce how to transform the sensitive V_i (1 ≤ i ≤ m) data into the less sensitive hash value H(V_i) through the Simhash technique. Here, we set parameter r = \( \left\lceil {\mathit{\log}}_2^n\right\rceil \). Thus each crowdsourcing task can be depicted by an r-dimensional vector. We demonstrate the calculation process with the following example. Assume there are 100 tasks and worker_i has ever executed ten tasks (i.e., task₁, …, task₁₀) in the past, then the ten tasks can be denoted by the following seven-dimensional vectors, respectively.

Afterwards, for the above ten seven-dimensional vectors, we replace the element “0” by “− 1” (for example, task₁₀ is transformed from (0, 0, 0, 1, 0, 1, 0) to (− 1, − 1, − 1, 1, − 1, 1, − 1)) and then count the sum of each column of the matrix constituted by the above ten seven-dimensional vectors. Afterwards, we derive a seven-dimensional sum vector W as presented in (2). In (2), the negative values are substituted by “0”, the positive values are substituted by “1”. Afterwards, we obtain the new vector W′ in (3). In our suggested ExH_Simhash approach, W′ is regarded as the index value of worker_i, i.e., H(V_i) = W′ holds. Thus through the above process, we transform the original 100-dimensional vector for worker_i, i.e., V_i into a seven-dimensional worker index H(V_i). Therefore, the data scale is reduced significantly. Besides, the index values H(V_i) employed for similar worker search are less sensitive; therefore, the workers’ private information is secure.

Next, we utilize the worker indices derived in step 1 to search for the similar workers of the unavailable worker worker_excep. Concretely, as formulated in step 1, H(V_excep) = (p₁, …, p_r) and H(V_i) = (q₁, …, q_r). Thus their Hamming distance d(H(V_excep), H(V_i)) can be calculated by (4), where “x” means XOR. According to the nature of Simhash [9], if condition d(H(V_excep), H(V_i)) < 3, we can conclude that the tasks executed by worker_i are approximately equal to the tasks executed by worker_excep; namely, worker_i and worker_excep are similar with high probability. Here, we utilize WORKER_X to denote the similar worker set of the unavailable worker_excep; namely, the equation in (5) holds.

In step 2, we have obtained the similar workers of the unavailable worker_excep, i.e., WORKER_X. The set WORKER_X may contain multiple elements and each element is a qualified worker who can replace the unavailable worker_excep. Next, we evaluate and select an optimal candidate worker from set WORKER_X.

Concretely, for each worker_x in set WORKER_X, we calculate his/her Hamming distance with worker_excep, i.e., d(H(V_excep), H(V_x)). According to the Simhash theory, a smaller Hamming distance d(H(V_excep), H(V_x)) often means a larger similarity between worker_x and worker_excep. Therefore, we select the optimal (i.e., the most similar) worker, denoted by worker_optimal, from set WORKER_X according to the Hamming distance; in other words, the equation in (6) holds. Specifically, if multiple workers have the minimal Hamming distance, then the crowdsourcing platform CP selects a worker randomly and utilizes the worker to replace the unavailable worker_excep. This way, the exception caused by the unavailable worker is handled successfully.

We conduct a set of experiments based on the WS-DREAM [20] dataset to test the performances of ExH_Simhash. We compare the suggested ExH_Simhash approach with three approaches, i.e., UCF (user-based CF), ICF (item-based CF), and P-UIPCC [21]. To evaluate the performances of different exception handling approaches, time cost and substitution equivalence are compared, respectively. If the ever-executed task sets of workers u and v are similar, then we can conclude that v is an ideal alternative for u when u becomes unavailable. Motivated by this hypothesis, a new criterion “substitution equivalence” (∈[0, 1], the larger the better) is defined as in (7), where A and B denote the set of tasks executed by worker_optimal and worker_excep, respectively (Fig. 2).

Our experiments are deployed on a PC with 2.40 GHz processor and 4.0 GB RAM. The operation system is Windows 7 and the programming language is JAVA. Experiments are executed ten times repeated and their average values are registered.

Next, three experiment profiles are designed, deployed, and tested, respectively. Please note that m and n represent the size of worker set and the size of crowdsourcing task set, respectively.

However, there are still several potential shortcomings in the experiments.