More for less: adaptive labeling payments in online labor markets

Prior work did not consider how to determine and continuously adapt the payments offered to labelers before they undertook a task, so as to improve the model’s predictive performance cost-effectively. Existing work about online labor markets in particular discussed a variety of mechanisms to improve the work quality, such as ways to screen workers and improve task design, and methods for acquiring multiple labels for the same instance so as to increase the probability of obtaining a correct label (e.g., Kazai 2011; Downs et al. 2010; Lee et al. 2013; Paolacci et al. 2010).

One stream of research focuses on repeated acquisition of multiple labels for the same instance, assuming that the payment per label is fixed and pre-determined (e.g., Ipeirotis et al. 2014, Dai et al. 2013, Lin et al. 2012, Zhang et al. 2016). Such repeated acquisitions can be used, for example, to infer the most likely label from multiple noisy ones. In contrast to the study we present here, these works did not address the problem of determining the payment per label to offer workers. Within this stream of research, some studies suggest methods to infer the likely label so as to learn better models using data instances that undergo repeated labeling (Dalvi et al. 2013; Kumar and Lease 2011; Raykar et al. 2010; Rodrigues et al. 2013; Zhang et al. 2015; Zhou et al. 2012). Some repeated labeling methods have also been applied together with active learning methods, in order to reduce the number of instances for which multiple labels are acquired for a pre-determined payment (Ipeirotis et al. 2014; Karger et al. 2011, 2014; Sheng et al. 2008; Wauthier and Jordan 2011; Lin et al. 2016). However, none of these methods identify which payment per label is advantageous to offer labelers and assume that the payment and subsequent quality of each label is predetermined and remains fixed.

Our work differs from this stream of research in several important ways. As noted earlier, the key difference is that prior work did not aim to identify advantageous payment per label to be offered to crowd workers so as to cost-effectively yield a given predictive performance. Some prior work proposed means to assess work quality retrospectively, after workers have performed the task (e.g., Raykar et al. 2010; Wang et al. 2017). In this work, we aim to identify advantageous payments offered to workers per label before they decide whether or not to take on the task. We also do not assume the availability of individual worker-performance history. Therefore, the method we propose is suitable for recommending payments in popular crowdsourcing labor-market settings such as Amazon Mechanical Turk, where employers continuously encounter new workers, and where tasks and the corresponding payment are offered to all workers who meet certain criteria (e.g., workers are from a certain country). We also consider settings where the work quality that can be obtained for a given payment may not remain the same indefinitely, but rather that market conditions, and consequently the labeling quality that can be obtained for different payment levels, may vary over time. To remain cost effective over time, our approach continuously adapts the payment so as to yield cost-effective improvements in model performance. Finally, in contrast to prior work that aimed to improve label quality (e.g., Wang et al. 2017), we aim to cost-effectively improve the predictive performance of the model induced from the acquired labels. Indeed, as we discuss in more detail below, the same quality of labels may have different implications for model performance across modeling tasks, induction algorithms, and across different points along the learning curve. Hence, our approach proposes to assess directly and in a data-driven manner the cost-effectiveness of different label acquisitions toward the model’s performance. Consequently, in some settings our approach may recommend buying fewer labels of higher quality and in other settings it may recommend buying more labels for the same or lower quality.

The ALP problem we consider here is also related to active learning (e.g., Lewis and Gale 1994; Abe and Mamitsuka 1998; Saar-Tsechansky and Provost 2004; Kong and Saar-Tsechansky 2014). Active learning methods typically assume that labels for unlabeled instances can be acquired at some fixed, pre-determined cost, and that the acquired labels are also correct; consequently, active learning methods aim to identify the training instances for which to acquire labels, so as to produce the best model performance for any given number of acquisitions. Therefore, while ALP aims to determine how much to offer workers per label to produce cost-effective models, active learning addresses a complementary problem, where the payment per label is assumed to be predetermined and fixed, and thus the goal is to identify the instances for which to acquire labels. Importantly for the challenge we address here, most active learning methods also assume that the acquired labels are correct, thus, active learning methods do not typically consider the challenge of inferring the benefit of alternative acquisitions, when the labels acquired are noisy.

Our study is perhaps most closely related to work by Yang and Carbonell (2012), but they make assumptions that do not hold in real online labor markets. Specifically, Yang and Carbonell (2012) do not consider settings which we consider here where: new crowd workers with no prior history are encountered continuously, workers labeling quality for different payments are unknown a priori and can change over time, and where the tasks are offered to all workers on the market (or to all worker who meet a certain criteria); rather, they assume theoretical settings in which labelers’ label quality for a given price is predetermined and known to the requester.

In what follows, we discuss the desired properties of an Adaptive Labeling Payment approach in our setting, followed by our proposed ALP approach from which we derive a specific ALP method.

In this section, we outline the ALP problem and develop our approach to address it.

We evaluate our method’s performance, compared to alternatives under a variety of simulated settings, corresponding to different labeling tasks and trade-offs between payment and quality that have been reported in prior work on crowdsourcing labor markets. In addition, because different trade-offs may arise at different times, we also evaluate our approach’s performance under conditions in which the prevailing trade-off between payment and quality changes over time. Across multiple settings, we aim to identify whether a given method yields a robust performance. Specifically, a robust ALP method that can be relied on in practice ought to yield consistent performance across settings, and should not fail miserably in some settings.

Towards the evaluations we simulate different trade-offs between payment and quality identified in prior works. These include “Asymptotic”, “Concave”, and “Fixed” trade-offs illustrated in Fig. 2a–c. In Fig. 2 and throughout the paper, quality is characterized by the likelihood of error. Specifically, Kazai (2011) and later Kazai et al. (2013) have found an “asymptotic” trade-off between payment and quality, illustrated in Fig. 2a. A “concave” trade-off, illustrated in Fig. 2b, where quality improves initially with increasing payments, but then degrades beyond a certain payment level, was reported in several prior works, including by Kazai (2011) and Kazai et al. (2013), as well as by Feng et al. (2009). Kazai et al. (2013) conducted analyses to understand the possible causes for this trade-off; they have identified that while individual workers tend to produce better quality with a higher payment, higher payment levels also draw increasingly unethical and opportunistic workers, which can reduce the overall work quality. In fact, the two opposing effects may also explain the asymptotic trade-off function observed by Kazai (2011), and by Kazai et al. (2013). The third trade-off we consider here is a “Fixed” trade-off, documented by Mason and Watts (2010) and illustrated in Fig. 2c, where different payments yield the same quality.

Because different trade-offs arise in different market settings that can vary over time, it is also useful to examine the robustness of an ALP approach when the underlying trade-off between payment and quality shifts from one trade-off to another. In the evaluations reported below, we report alp-mtr’s performance when a given trade-off switches to another after 50% of the acquisition budget has been used.

In the simulation studies we report below, once the payment to be offered to labelers for the next batch of labels is selected, the probability q that a label acquired at that payment is correct is determined by the prevailing trade-off. Thus, the correct label for an instance is assigned at probability q, and the label is reversed at probability \( (1 - q \)). We also followed prior work for the payment range and the number of payment alternatives from which payment methods can select. Specifically, prior studies considered 2–4 payment levels, ranging from a minimum payment of $0.01–$0.03 to a maximum payment of $0.1–$0.25 (e.g., Feng et al. 2009; Kazai 2011; Kazai et al. 2013; Mason and Watts 2010; Rogstadius et al. 2011); thus, in the evaluations that follow, we considered three payment alternatives within this range of $0.02, $0.14, and $0.25, reflecting low, mid, and high payment levels.

For each of the trade-offs outlined above, as well as for settings where the trade-off changes over time, we report experimental results for labeling tasks corresponding to three publicly available datasets. The first two, Pen Digits and SPAM (Lichman 2013), were selected as they reflect typical labeling tasks that are easy for humans to produce. The third dataset, Mushroom (Lichman 2013), is used for additional robustness and had been used in prior simulation-based research on labeling via crowdsourcing markets (Ipeirotis et al. 2014). Table 2 provides details regarding the different datasets. In “Appendix F” we also provides empirical results that show the number of instances required to reach various levels of model performance (measured by AUC) as a function of different levels of labeling quality.

As machine learning becomes integral to the routine operations of firms and the products and services they provide, the immediacy and accessibility of online labor markets present unprecedented opportunities for on-demand labeling to be brought to bear on machine-learning tasks. Yet for these opportunities to materialize, it is important to devise solutions for the novel challenges these markets present. Given the different trade-offs that can arise between labeling payment and work quality under different labeling tasks and market conditions, in this paper we first formulate the problem of Adaptive Labeling Payments (ALP), then develop an approach to address it, and study extensively the performance of our proposed approach. Specifically, we develop an ALP approach, alp-mtr, designed to determine and continuously adapt the payment offered to crowd workers before they undertake a labeling task, so as to improve the performance of a model induced from the data cost-effectively. Our ALP approach estimates the effect on induction from omitting training instances, previously acquired at different payments; our approach also incorporates elements that benefit this estimation, particularly in the presence of noisy labels and changing prevailing trade-offs between payment and quality.

We empirically evaluated the performance of alp-mtr relative to that of a robust alternative under a variety of market scenarios, reflecting different labeling tasks and trade-offs between labeling payments and quality documented in prior work, including settings where the trade-off changes over time. Our results show that alp-mtr yields robust performance across settings and that it offers meaningful and substantial cost savings, with an average savings of 33.4% across settings. We also demonstrate that the design elements of alp-mtr, which aim to improve its estimations of the expected benefits from alternative payments in the presence of noisy labels, and its adaptation to changing market conditions, indeed benefit alp-mtr’s performance. Given our method’s consistently robust benefits under different settings, it can be considered a benchmark for evaluating future mechanisms to determine labeling payments.

The practical implications of this research are important for enabling a growing reliance on instance labeling via crowdsourcing labor markets. Our ALP approach yields both robust performance and meaningful cost savings. Because crowdsourcing for labeling tasks is becoming increasingly popular in academic research as well, academic efforts can similarly benefit from our method’s efficiencies.

Our proposed approach offers a wealth of opportunities on which future research can build, both to improve as well as to extend our work. One interesting direction would be an adaptation of our ALP approach to regression predictive tasks, in addition to the classification task that we consider here. As prescribed by our ALP approach, the effect of target (dependent variable) values acquired at different payments may, in principle, be evaluated by estimating the effect on the regression model’s performance after omitting instances previously acquired at the corresponding payment, as the estimations can benefit from the same elements included in alp-mtr to improve the estimation under noisy labels and changing market conditions. Similarly, while in this work we considered an important and popular measure of performance (namely, Area Under the ROC Curve) to evaluate a classification model’s performance, the framework we develop could accommodate other performance measures as well.

Our approach is designed to adapt to changing market conditions. However, inherent to all predictive modeling approaches is the assumption that the environment remains stable for some time so that the learned patterns can be exploited. Different solutions may be required in chaotic settings, where market conditions change significantly and very frequently.

Any data-driven-learning-based approach incurres costs for learning. In our context, a practical reference to assess these costs corresponds to the non-data-driven and robust, uniform, alternative. A useful question to consider is whether our approach remains beneficial even if only a small number (i.e., a few hundred) of labels are acquired on the market. Indeed, alp-mtr includes an initialization phase, during which it acquires an initial set of labeled instances, and from which it begins learning; importantly, these initial label acquisitions are uninformed by an estimation of the expected benefits and may thus be rather costly. However, during the initialization phase, alp-mtr selects payments in the same manner by which payments are selected by the robust uniform alternative. This is also demonstrated in our empirical results, where alp-mtr does not incur higher costs relative to a non-data-driven alternative early on. Nevertheless, it would be beneficial for future work to further improve alp-mtr’s performance as well as reduce the cost of (shorten) the initialization phase.

alp-mtr was designed for and evaluated in challenging settings where all the labels are potentially noisy and no gold standard labels are available. However, with trivial modifications, alp-mtr can be adjusted to operate in more favorable conditions when correct labels (“gold standard”) are available for some instances. To improve performance, such gold standard data can be used towards alp-mtr’s evaluation of \( {\text{EPI}}_{{{\text{c}}_{\text{k}} }} \) (estimation of expected performance improvement for a certain payment level) as an internal holdout set, instead of applying cross-validation over noisy instances. Future work can also suggest how to best use gold standard data, including how to partition such data for training and alp-mtr’s internal holdout sets.

Recall that alp-mtr adapts the payment per label only in consecutive acquisition phases, and does not simultaneously offer different payments for the same task. Nevertheless, it is possible that adaptive labeling methods may introduce “second-order effects”—i.e., that the mere change in payment, rather than the payment level, offered at different times might impact worker behavior. Prior work (Chen and Horton 2016) has shown that in different settings than those we consider here, and where the same worker who works continuously on a task is faced with wage cuts, the worker is more likely to discontinue taking additional tasks. (Chen and Horton did not find evidence that this impacts the work quality, unless workers are solicited to continue working in an inadequate manner). Importantly, our approach’s performance may not be effected by an individual worker’s decision to discontinue work due to changes in pay. This is because payments are revised and offered to the entire market in consecutive phases, and payments are not revised for a given worker. Furthermore, because the market is often large (as in real markets such as Amazon Mechanical Turk), the potential effect of such adaptation may not be substantial. Nevertheless, it is possible that this or similar effects may arise in different market conditions. It would be therefore beneficial for future work to model and measure any effects from revising the payment offered on the market over consecutive phases. Lastly, because our approach corresponds to a sequential process, where labeling tasks are offered sequentially on the market, to alleviate such effects it may be beneficial to consider revising the framing of the labeling task in subsequent phases.

Our approach does not consider the properties of individual instances, such as instance level properties that might yield differential difficulty for labelers to label them correctly. If a given instance is less likely than other instances to be labeled correctly, it may be beneficial for future work to develop methods that identify advantageous payments for labeling individual instances, so as to cost-effectively improve model performance.

Another possible venue for extending the ALP problem is to adapt it to domains in which it is possible to compartmentalize the feature sets into different subsets, each of which can be used to produce different models, and for which a different trade-off between labeling payment and quality might arise. For example, in text classification tasks, some of the workers may be presented just with the text headline and receive a lower payment, whereas, other workers may be requested to classify an entire news article and receive a higher level of payment.

Finally, it would be interesting for future research to explore different ways of extending our approach. In particular, it would be beneficial for future work to explore the possibility of removing instances labeled at certain payments if this could improve the model’s generalization performance. Similarly, it would be interesting to explore an extension of our approach that studies possible effective stopping criteria for acquisitions at a given time. Such stopping criteria could be based, for example, on the EPI measure outlined in Eq. 1 to assess whether acquiring labels at a given payment might undermine induction.