Connecting algorithmic fairness to quality dimensions in machine learning in official statistics and survey production

We discuss the use of ML in NSOs in their roles as producers and analysts of data.⁵ Most if not all of these ML applications fall under either supervised learning or unsupervised learning.⁶ In supervised learning, the goal is to learn the functional relationship \(f\) between inputs or features \(X_{1},\ldots,X_{p}\) and the outcome or label \(Y\), both of which are contained in the training data. In the supervised ML paradigm, illuminated further in Sect. 2.2, the focus is on prediction, i.e., the ability to predict the outcome from the feature values for new data points (i.e., not used for training the model): \(\hat{y}=f(x_{1},\ldots,x_{p})\).⁷ In other words, the rationale for supervised ML is deployment: being able to use the learned model \(\hat{f}\) to predict the outcome for new units for which the outcome is unknown. This is in contrast with the traditional inferential statistics approach: there, the goal is the estimation of (population) parameters \(\theta\), typically in order to answer substantive questions about the world which were translated into statistical parameters, \(\mathbb{E}(y|x_{1},\ldots,x_{p})=f(x_{1},\ldots,x_{p};\theta)\). Additionally, compared to prediction, population-based inferential statistics is less focused on the individual (Breiman 2001).

Supervised learning tasks model either a qualitative outcome (called classification) or a quantitative outcome (called regression in the ML world, regardless of whether statistical regression or other methods are used). Classification predominates in the theoretical literatures (including that on fairness, see Sect. 3), in ML applications in general, and also in ML applications within NSOs (e.g., Beck et al. 2018a, Chap. 4).

Unsupervised learning comprises a very heterogeneous set of methods and tasks: clustering, dimensionality reduction, and outlier/anomaly detection, but also latent variables, archetypes, association rule learning, and more (Molnar 2022, Chap. 9). In contrast to supervised learning, unsupervised tasks lack an outcome variable \(Y\) in the data; only \(X_{1},\ldots,X_{p}\) are available. This also means that measuring performance is much more difficult. The common mission in the diverse collection of unsupervised tasks is to “find hidden patterns” (ibid) or to “discover interesting things” (James et al. 2021, Chap. 12) about the units, about the features, or, more generally, about the data. It is possible that the output of unsupervised models is the endpoint of the data analysis: e.g., one may be satisfied to learn how many ‘groups’ a clustering algorithm has detected or whether particular units are placed in the same cluster. Often, however, unsupervised methods are applied to pre-process or transform the data before they are fed into other, typically supervised models (James et al. 2021, Chap. 12): e.g., a high-dimensional set of variables can be reduced to a few, more manageable, perhaps more interpretable set of ‘principal components’ which are then employed as features in a prediction model (Bach et al. 2022). For some applications, both supervised and unsupervised learning may be useful, depending on the particular situation, goals, and available data: e.g., in the identification of the same units in two disparate data sources, one may or may not have gold-standard information about true matches via a unique identifier.

We believe it is important to reflect upon how ML proceeds, how and why ML is successful, and what the field has brought to data analyses in general. Only based on this explicitly articulated understanding does it make sense to discuss how ML relates to the quality dimensions of NSOs and to fairness. ML-based data analysis is characterized by two somewhat separate aspects. First, the ML mindset, paradigm, or approach which, along with its procedural contributions, we discuss in the next two paragraphs. Second, the increased use of ML methods (or model classes) in a stricter sense, which is discussed in the subsequent paragraph. Some model classes clearly fall under traditional statistical methods⁸ and others are considered machine learning in a stricter sense (e.g., decision trees, random forests, and neural networks). Thus, while there is no sharp, universally agreed-upon boundary separating the two, the notion of ML methods is still useful. The following discussion of why and how ML succeeds and the contributions it has brought shall be informative in several ways: to help decide whether a ML mindset and ML-based methods fit a particular application in official statistics and as necessary background information for our discussion of ML in official statistics.

With ML procedures we refer to four practices, discussed in the next paragraph, that, empirically, are most associated with sound ML-based data analysis, but that work largely independently from whether the considered model classes are statistical or ML. At the heart of the ML mindset is the rigorous evaluation of performance on a particular task.⁹ In supervised learning, evaluation is about how well a model is able to predict on new data, i.e., the expected out-of-sample prediction error (generalization error). Thus, the ML mindset is one of competition, with the best-performing model being chosen. One typically considers several model classes (e.g., linear regression, LASSO, decision trees, random forests, and XGBoost) and several models within each class. The latter correspond to different selected features and ‘parameters’ (e.g., the features and splits in a decision tree, respectively) and different tuning or hyperparameters (e.g., the depth in a decision tree or the regularization penalty in LASSO; Hastie et al. 2009, Chaps. 3.4 and 9).

First, for both, model selection and assessment of the final model, unbiased estimation of a model’s generalization error is paramount. The main procedural building block helping to ensure this is data splitting into two parts: the training data, which are only used for training the model, and the evaluation data, which are only used to evaluate the predictive performance of the trained model.¹⁰ The error on the training data is a systematically over-optimistic measure. In contrast, the error on the separate, fresh evaluation data provides a valid estimate of the generalization error which also guards against overfitting (i.e., fitting too closely to the observed data, thus fitting partly to random noise inherent in the training observations). A second procedural contribution from supervised ML is that information that, in reality, would not be available at the time of the prediction may typically not be used during model training and previous steps (see Ghani and Schierholz 2020, Chap. 7.8.1 and Guts 2020): Data leakage occurs when any information from the supposedly separate, unseen evaluation data is used in some form, hurting the freshness of the evaluation data. Target leakage is about using the values of the outcome \(Y\).¹¹ Both open the door for over-optimistic performance evaluations. Leakage can sneak in very subtly, as when the information is used for pre-processing the data, e.g., in feature engineering or imputation of missing data. Third, the centrality of performance comparisons in the ML approach brought focus to the issue of metrics used for model evaluation and during model training (i.e., the loss function to be optimized).¹² This is not unrecognized in traditional statistics, but the choice of metrics is more active and task-driven in the ML approach. In classification in particular, false negatives may be much more important relative to false positives for one application than for another. Recognizing the trade-off between the two error rates and choosing a task-suitable metric is an improvement over always employing overall accuracy. Fourth, as the ML approach involves the consideration of several models, one natural procedural extension was to combine several models, sometimes from different model classes, into one ensemble (via, e.g., bagging, boosting, stacking, or simpler methods such as averaging or majority vote; see Hastie et al. 2009, Chap. 8). The intuition for improved predictive performance is two-fold: for different data points a different model (class) may be closest to the truth and an ensemble of models is more stable than any one single model would be.

We now turn to ML methods or model classes and three of their reputed benefits, particularly relative to traditional statistical methods. First, flexibility, which in supervised learning is about the functional forms of the relationships between the features and the outcome as well as about interactions among the features. This actually entails two components:The former is afforded by ML methods that can be quite complex; however, typically the more flexible, the more data are required (James et al. 2021, Chap. 2.1.2). A further reason for (a) lies in the ML-based approach ‘trying out’ many ML model classes. For a different true functional form, a different model class is the most natural fit: e.g., trees are most suitable for step functions.¹³ However, to compare the whole basket of ML model classes with just one statistical model class and conclude that statistics (every statistical method) is less flexible than ML (every ML method) is not fair. It also not accurate: in particular, Generalized Additive Models (James et al. 2021, Chap. 7) are a statistical model class that is able to automatically adapt to non-linear relationships and can accommodate interactions. In comparison studies, particularly for small and medium sample sizes, simpler and traditional statistical methods are often not inferior (Christodoulou et al. 2019; Grinsztajn et al. 2022).¹⁴ Thus, even for performance reasons alone, simple methods and traditional statistical model classes should always be among those tried out; we will address other rationales such as interpretability in Sect. 4. Second, automatic feature selection is built into some ML model classes: e.g., in trees, at each split, only one variable is chosen. In high-dimensional settings, feature selection helps to stabilize the model estimation and, especially for traditional model classes, is even necessary when the number of predictors exceeds the number of observations (“\(p> N\)”, James et al. 2021, Chap. 6). However, traditional statistics is not without methods for feature selection (Hastie et al. 2009, Chaps. 3.2ff.): e.g., subset selection procedures or, more modern, via regularization such as the LASSO. Third, traditional statistics is geared towards what the ML culture calls structured or tabular data: e.g., for survey data represented in a matrix format, each row corresponds to exactly one respondent and each column corresponds to one survey question. It is undeniable that ML has made great progress regarding un- and semi-structured data such as (a collection of) images, audio or video data, texts, or even multimodal combinations thereof. In particular, Deep Learning is able to process unstructured data end-to-end: the raw input data are fed into the network – no feature engineering needed on the part of the data analyst (Molnar 2022, Chap. 11).

We conclude with two remarks. First, when the ML mindset fits an application, e.g., when prediction is the focus, then the procedural and methodological lessons discussed above are also relevant when traditional statistical model classes are used. Thus, much of the rest of this paper is not just relevant to the use of ML models. It is true, however, that more complex model classes have more potential for overfitting, i.e., over-adapting to their training data, so adhering to good practices tends to be more important (James et al. 2021, Chap. 2.1.2). Second, for a quantitative outcome, the expected squared out-of-sample prediction error (ESPE) at a point \(x_{0}\) in the feature space can be decomposed (Hastie et al. 2009, Chap. 7.3): \(E((Y-\hat{f}(x_{0}))^{2}|x_{0})=\text{Var}(Y|x_{0})+\text{Bias}(\hat{f}(x_{0}))^{2}+\text{Var}(\hat{f}(x_{0}))\). The first term, \(\text{Var}(Y|x_{0})=E(Y-f(x_{0})|x_{0})^{2}\), is the conditional variance of the outcome around its true conditional mean \(f(x_{0})\): for given features, it cannot be reduced and is independent of choices made by the data analyst. The second term depicts the squared bias of \(\hat{f}\), i.e., the expected squared deviation of the learned model from the (unknown) true conditional mean or true model \(f(x_{0})\). It is typically monotonously decreasing in model complexity, quickly at first and then leveling off (James et al. 2021, Chap. 2.2.2). The greater the (allowed) complexity, the greater the set of possible models, and thus the smaller bias, i.e., the distance of the best model in the consideration set to the true model (see Hastie et al. 2009, Chap. 7.3). The third term, \(\text{Var}(\hat{f}(x_{0}))=E(\hat{f}(x_{0})-E(\hat{f}(x_{0})))^{2}\), is the variance of \(\hat{f}\), denoting the variation in the learned model when learned on different training data sets (same population, same sample size). In general, this estimation uncertainty is monotonously increasing in model complexity due to higher susceptibility to small perturbations in the data (James et al. 2021, Chap. 2.2.2). This is why, typically, the more flexible a model class, the more training observations are needed (James et al. 2021, Chap. 2.1.2). Note that the implied bias-variance trade-off is due to the focus of the supervised ML approach on prediction error: any model (class), ML or not, trained to minimize ESPE will exhibit at least a small amount of bias as long as it results in a larger decrease in variance. This results in a U-shaped relation of model complexity and ESPE (at least for the typical, ‘under-parameterized’ models, see Belkin et al. 2019). ML-based data analysis operates on both sides of this trade-off: flexible model classes and trying out different models mean more complexity (low bias, high variance) whereas feature selection (more bias, less variance) reduces model complexity.

The primary drivers are NSOs striving for improved products and processes (according to the quality dimensions, see Sect. 4), new products, new applications, new data, and an interest in new methods. These are not isolated aspects: e.g., some new products (e.g., data about very dynamic sectors or companies) make only sense when released very frequently or timely and some new data are too voluminous or unstructured for existing methods (based on traditional statistics or human work). We consider two of these drivers in more detail.

First, desired improvements include producing data and statistics more cheaply, releasing them more frequently, more timely, or on a more granular level, making them more accurate, or lowering response burden. Partial automation is seen as a vehicle for such improvements: e.g., algorithms can handle the easy cases, allowing staff to focus on cases that are hard to classify (Coronado and Juárez 2020) or important or influential (Dumpert 2020, p. 2), or to contribute to other activities (Coronado and Juárez 2020). Alternatively, algorithmic assistance can take on the form of providing a model’s most likely outcomes for a given data point as suggestions in, e.g., human coding tasks (Measure 2020, p. 7 and Sthamer 2020b, Chap. 7).

Second, new data are considered to complement and, in part, to replace some of the traditional main data sources of NSOs – censuses, surveys, registers, and administrative data. We would like to remind that survey data have already been more than just the responses to the survey items: e.g., respondents and interviewers can provide samples (soil, saliva, blood, etc.) and measurements (Groves et al. 2009, Chap. 2.2.2), information from digital devices can be used (Keusch et al. 2024), and paradata about the data collection process are captured (Kreuter 2013; Schenk and Reuß 2024). How surveys will evolve in the era of, in particular, Big Data has received increasing attention since the second part of the 2010s (e.g., Baker 2017 and the BigSurv (https://www.bigsurv.org) conferences, see Hill et al. 2019): where they can replace survey data (Couper 2017, p. 134f.), where and how the two can complement one another (ibid; Japec et al. 2015, p. 873), and what can be learned methodologically from each other (Hill et al. 2021). The community appears to agree that surveys are here to stay: in contrast to most other data, surveys can be designed to give the desired breadth, level of detail, and fitness for a specific use, and to control the various error sources better. New data types may be collected in conjunction with a survey¹⁵ or without it. Given user consent, wearables, apps, and sensors are emerging sources (Keusch et al. 2024), as are data donation and (screen) tracking (Ohme et al. 2024). Instead of single values, these data exhibit complex measurement series.

Another important new data source is images – so far mostly aerial images and other kinds of remote sensing (Coronado and Juárez 2020, p. 4 and 9): in particular, there have been vast improvements in the frequency and availability, level of detail, and costs of satellite images. The volume is too much to handle for classical processes (i.e., involving traditional statistics or human work), making ML approaches a virtual necessity. Another reason is that in some cases multiple spectra or sensing technologies, going beyond wavelengths that humans can perceive, can be combined for the same object.

Textual data are a further new avenue (Text Classification Theme Group 2022). They range from open-text responses in surveys, to traffic, coroners’, or police reports, to complaint filings, building permits, and other legal documents. Some of these are acquired via web scraping, as is information from company websites, online shops, news reports, job ads, and social media posts.

We close with a note on the compatibility of the combination of ML and traditional statistics and of data processing and eventual data analysis. We use the imputation of missing data during processing as an example. If the eventual, ‘downstream’ data analysis is traditional, it is well known that multiple imputation, rather than single imputation, is needed to properly quantify the uncertainty of parameter estimates (Little and Rubin 2019, Chap. 4f.). For that, multiple draws from the posterior predictive distribution are needed. A statistical imputation model provides such a distribution while ML models, even ensembles, typically do not. Conversely, suppose the eventual data analysis follows the supervised ML paradigm, i.e., prediction. If the downstream analyst is not interested in quantifying the uncertainty of the predictions, multiple imputation is not needed. However, a traditional statistical imputation model would be estimated on the whole data set while the downstream ML-user is only permitted to use the training data portion, but not the evaluation data, for learning how best to process the data (and for model training). In other words, this statistics-ML combination is likely to produce data leakage.¹⁹ The downstream ML-user may also be affected by target leakage when the outcome variable was used in the imputation model (which a downstream statistics-user typically would not view as problematic). These are two examples in which using ML in data processing and traditional statistics in the eventual data analysis, or vice versa, can lead to problems.

Following controversial applications of machine learning in high-stakes settings (Angwin et al. 2016; Buolamwini and Gebru 2018; Allhutter et al. 2020), fairness concerns have sparked a multi-faceted and multi-disciplinary research field centered around the social impacts of algorithmic decision-making (ADM). Research on fairness in machine learning (fair ML; see Mehrabi et al. 2021; Mitchell et al. 2021; Makhlouf et al. 2020; Caton and Haas 2024 for overviews) is thus typically focused on prediction models as part of larger socio-technical systems which may allocate access to positions, treatments or, more generally, valuable resources. The scope of fair ML, however, extends beyond ADM applications and includes fairness implications of the use of ML in other contexts, such as in data processing and survey production (Rodolfa et al. 2020).

A key concept in the fair ML literature is the notion of protected attributes. Protected attributes are inherent or ascribed characteristics of individuals (such as ethnic origin, gender, age, or religion – i.e., characteristics over which the the individual typically has no control), for which they can (or should) not be made responsible, but which nonetheless may be the grounds for differential treatment of individuals in the real world due to prejudice and discrimination. In a narrow sense, protected attributes may be defined based on anti-discrimination legislation (such as the Equal Credit Opportunity Act in the U.S., Mehrabi et al. 2021), but the eventual set of attributes that should be considered in a given application may be context-specific. Note that the adaptation of U.S.-centric concepts such as ‘race’ to other contexts is also non-trivial. The implication of introducing protected attributes is now not to ignore these features in the ML pipeline, but rather to faithfully acknowledge heterogeneity in data and to build subgroup-aware models that incorporate moral considerations on how to account for and resolve societal biases in a given context. To approach conceptions of fairness in machine learning, an initial, higher-level requirement could be based on the adaption of the disparate impact doctrine to data modeling – prevent outcomes or practices that have disproportionately adverse impacts on members of protected groups (Barocas and Selbst 2016). There are various pathways through which this principle may be violated in machine learning practice, the most prominent one being (different types of) biases in data (Mehrabi et al. 2021). Historical bias may be present in any data that result from social processes: administrative labor market records capture historical discrimination on the labor market, educational attainment histories are reflective of social biases in the education system, and (geospatial) records of criminal incidents are in part affected by decisions on which areas should be patrolled. Historical bias can easily be learned by and incorporated into ML models if data that reflect social processes is used for model training. Model training may, however, also be affected by measurement bias. In supervised learning, the outcome variable that is observed in the data may be a biased proxy for the actual outcome of interest such that social biases sneak into the model in the model specification step (Obermeyer et al. 2019): e.g., arrests are a biased proxy for criminal activity when, conditional on the same behavior, arrest probability is higher for some individuals or groups. Lastly, representation bias refers to deficits in the composition of the training data. Such deficits may refer to the (mis)representation of specific social subgroups in absolute or relative terms, or to the match between the data that is available for model training and the eventual target population more generally. We caution that very different meanings and haphazard usage of the term ‘representativity’ in the ML/AI community have been documented (Clemmensen and Kjærsgaard 2023), sometimes strongly diverging from the statistical notion. Regardless of these causes for biases in the data, there can be feedback loops: when (biased) predictions influence real-world outcomes, they may maintain or worsen biases in the next round of data, thereby sustaining or perpetuating biases in the predictions (Perdomo et al. 2020).

The fair ML literature notes that disparate impact may also be caused by other factors. This includes data pre-processing and modeling decisions along the ML pipeline which may operate next to or in interaction with existing data biases (Gerdon et al. 2022; Rodolfa et al. 2020). Examples include the compilation and matching of information about subpopulations in the training data preparation step, the encoding of (correlates of) protected attributes and their use in model training, as well as decisions on how model outputs are eventually used downstream, e.g., for classification purposes.

In light of the various ways machine learning models may be affected by social biases, an abundance of fairness notions has been proposed in the literature which formalize different fairness conceptions and often imply corresponding fairness metrics to quantify adherence to a given notion in practice. Fairness notions typically focus on binary classification tasks and have been formulated on the group, subgroup, or individual level. Group fairness notions compare members of protected groups to their non-protected counterparts with respect to different prediction-based quantities. Given protected attribute \(A\) and predicted outcome \(\hat{Y}\), independence-based group fairness notions require the predictions to be independent of group membership: \(\hat{Y}\perp A\). The separation criterion additionally considers the observed outcome \(Y\) and requires independence conditionally on the true label: \(\hat{Y}\perp A\mid Y\). Sufficiency-based notions, in contrast, condition on the predictions: \(Y\perp A\mid\hat{Y}\) (Barocas et al. 2023; Makhlouf et al. 2020). Next to group fairness, subgroup fairness aims to provide stronger fairness guarantees by imposing fairness constraints on large collections of subgroups that may be defined by intersections of many (protected and non-protected) attributes (Hebert-Johnson et al. 2018; Kim et al. 2019; Kearns et al. 2018). Finally, individual fairness formulates requirements on the individual level, e.g., by mapping distances between individuals to distances in predictions (i.e., similar individuals should receive similar predictions; Dwork et al. 2012) or by drawing on causal reasoning (Kusner et al. 2018; Kilbertus et al. 2017).

While group-based fairness metrics are rather straightforward to compute and evaluate in practice, a central result of the fair ML literature has been that complying with multiple group-based fairness notions simultaneously along the dimensions discussed above is difficult. Except for highly stylized cases, a prediction model cannot fulfill independence, separation, and sufficiency criteria at the same time (Chouldechova 2016). Requesting group fairness thus comes with trade-offs, and considerations on which (group) fairness notion should be prioritized might be highly context-specific.

Valid criticisms of the fair ML literature must be acknowledged: e.g., some fairness notions may suggest changing a prediction model so as to provide worse predictions (for some groups) in order for some equality constraint to become satisfied. However, as discussed by Kuppler et al. (2022), this is an artifact of considering only ADM systems in which the decision is a function of solely the model’s prediction \(\hat{Y}\), ignoring the protected attribute \(A\), other features used to predict the outcome \(W\), and further information such as the accuracy of \(\hat{Y}\) given \(A\) and \(W\). Such systems exist, but the flaw is in their construction, not in the consideration of the fairness of algorithms per se. This is solved by splitting up the prediction task (i.e., building a good model) and the decision-making task: then, too, fairness notions for the former and justice notions for the latter can be cleanly separated. As suggested by Kuppler et al. (2022), accuracy-based or, conversely, error-based fairness metrics may be the most natural: For classification tasks, one can ask how rates of overall errors, false positives, false negatives, 1‑precision, or 1‑recall, as well as miscalibration²⁰ differ across groups or subgroups. For regression tasks, bias and variance can be looked at. This is all the more important for the work of NSOs: they do not engage in ADM, they cannot know during data production what justice principles (and other goals) downstream users of the data will have, and accuracy is already one of the main quality dimensions they consider (see Sect. 4.6). Furthermore, error fairness translates more easily to data that are not about humans but about, e.g., companies – a large part of NSOs’ work. We will therefore concentrate on error-based fairness notions later on.

The catalog of fairness notions that have been proposed in the literature highlights that fairness can be conceptualized in various (and conflicting) ways. Given a fairness metric, additional parameters might need to be set to formalize the range of values deemed acceptable. Thus, technical measures which quantify whether a prediction model satisfies some fairness constraint do not substitute for human judgment and reflection. In contrast, fair ML implies moral reasoning and raises questions of distributive justice (Kuppler et al. 2022; Heidari et al. 2019; Loi et al. 2021; Binns 2018; Lee et al. 2020; Gajane and Pechenizkiy 2018): How should (different types of) prediction errors be distributed across social groups in a given context? Given fair predictions, which downstream allocation of resources do we perceive as just?

Committing to fairness in building and implementing machine learning systems thus requires developers and stakeholders to explicitly specify their goals. This inevitably includes engaging with various normative questions such as which attributes should be considered sensitive, which fairness concept should be prioritized, and how exactly deviations from ‘optimal fairness’ should be defined and potentially addressed (Bothmann et al. 2022). Some guidelines have been proposed to help navigate the fairness field: Makhlouf et al. (2020) and Saleiro et al. (2019), for example, structure fairness notions based on a set of selection criteria. Such templates can point out critical decision points and help in guiding discussions among stakeholders, but nonetheless require normative input and context-specific weightings of interests. This implies that NSOs may need to critically engage with downstream users, and reflect on whether the same product can meet heterogeneous needs in different contexts.

Recent research has started to focus on the human component in fair ML by studying human perceptions of algorithmic fairness. This line of work focuses on how design aspects of ADM systems or characteristics of the human evaluators affect individual fairness perceptions, or how algorithmic decisions are perceived in comparison to human decision-making (see the review by Starke et al. 2022). Studies that investigate which type of input data (Kern et al. 2022) or attributes (Grgic-Hlaca et al. 2018) are perceived as sensitive in a given context or which types of prediction errors are evaluated as particularly problematic (Srivastava et al. 2019) by the general public may provide valuable input to tackle the normative dilemmas mentioned above.

Finally, characteristics of the individual decision-maker, the algorithm, and the context in which it is applied can affect “algorithm aversion” or “algorithm appreciation”, i.e., the individual’s under- or over-reliance on the algorithm’s results (Burton et al. 2020; Jussupow et al. 2020; Hou and Jung 2021). While NSOs are typically not the place for ADM, the data they produce may very well be frequently employed for such purposes, e.g., by governmental bodies. Thus, the data and how they are produced as well as what information (documentation, metadata, etc.) is released can influence aversion to such downstream algorithms. The same can be said for the fairness perceptions discussed in the previous paragraph. In addition, the internal high-level decisions of whether and how to implement ML algorithms in a NSO’s processes are likely affected by the very same characteristics, as are attitudes by other stakeholders (staff, recipients of statistics, data users, etc.).

We begin with explainability and interpretability which we address in somewhat more detail than the other quality dimensions.²¹ As is widespread, we treat interpretability and explainability as synonyms (e.g., Miller 2017, Chap. 2.1.5 and Molnar 2020, Chap. 3.0). Interpretability has a dual role: it is a desirable property of a model and denotes a set of tools that can help to investigate other desirable properties. As this dimension is not explicitly part of pre-ML data quality frameworks (Yung et al. 2022, p. 4), we give an introduction to the field of Interpretable Machine Learning (IML)²² – on a high level, without delving into specific methods, and to the extent useful for our later discussion. We refer the interested reader to Molnar (2020)’s excellent and accessible book on the subject. While the field has created a broad set of concepts and methods, it is still developing. We will also touch upon the fact that IML is not a monolith and the various concepts and methods are sometimes competing (Lipton 2018). Thus, when IML methodology is put into action, practitioners need to be aware of the (general or situation-specific) limitations of these methods and how to use them properly (Molnar et al. 2022; König 2023).

Cost-effectiveness is about the relationship between the (quality of the) outputs and the incurred costs Yung et al. (2022, p. 5). Costs may be (quasi-)fixed or ongoing. Important categories include: the necessary equipment, data, and skills must be acquired; the data must be processed, a model must be trained and evaluated; equipment, skills, and models must be monitored, maintained, and updated.²⁸ We refer to Yung et al. (2022, Chap. 6) for more details, but want to highlight some aspects.

Standardization of processes is one important tool to manage cost-effectiveness and other quality dimensions (e.g., Eurostat 2017, Indicator 10.4 and Destatis 2021). Automation, driven by ML (or statistical) models, is a promising avenue in this regard. One anticipated benefit is that automated processes may entail higher (quasi-)fixed costs of setting up – e.g., for equipment, knowledge acquisition, and the training, selecting, and evaluating of models – but once they are implemented, the marginal cost per additional unit – e.g., the cost to generate a prediction for an additional data point – is very low: i.e., such processes are highly scalable.

We consider two cost aspects in more detail. First, the work does not stop with training a model: it must be evaluated on more than its performance – namely its interpretability and its fairness – and it must be continuously monitored after deployment for model drift or decay (declining performance and other changes in behavior) and, when needed, re-trained; this binds manpower, computational resources, and may necessitate further data and data processing work (Choi et al. 2022, Chaps. 1 and 4). Note that these requirements are largely independent of whether the chosen model for automation is ML or statistical: changes in real-world mechanisms or in data collection affect them both. Choosing models that are more stable (see Sect. 5.4) may thus provide financial relief via a decreased need for re-training. Second, data are not only at the core of model performance but also an important consider cost consideration. ML models and the ML paradigm typically exhibit high demands regarding computational resources and data volume,²⁹ affecting costs, timeliness, and the uncertainty (Sect. 4.6) of the resulting output. The flexibility of ML is one of its advantages, but also increases data requirements: the less ‘known’ structure and other types of relevant expertise are used, e.g., to create and transform features, the more the method must learn on its own – Deep Learning on unstructured data is the prime example. Also, the training data for supervised editing and imputation models need to be carefully labeled, requiring perhaps more time than the simple editing and imputation itself would, and, if anything changes, existing training data may need to be re-relabeled (Sthamer 2020a, p. 6).

Published cost studies are rare in general and findings for one setting or organization may not translate directly to another (see Groves et al. 2009, Chap. 5.3.6 on survey costs). ML and automated solutions have not shown to be always more cost-effective for NSOs’ applications, but there are positive examples (Sthamer 2020b, p. 13). In addition, switching from one process to another is not cost-free. So far, in applications such as editing and imputation, no single ML method clearly dominates and a lot of work may be required, especially to yield more than marginal benefits, and a full range of model classes must be prepared and considered for each application (Dumpert 2020, p. 7).

Unsurprisingly, automation is also being pursued in the ML world. First, automated machine learning (AutoML; see Tornede et al. 2022; Weerts et al. 2023) is concerned with automating the whole pipeline, from data pre-processing and feature engineering to model training, hyperparameter optimization, and model selection. Second, monitoring of a deployed model for drift (Choi et al. 2022) can also be automated. Yet, complete automation is not the goal of NSOs, and expertise and skills are still needed to implement and monitor these even more automated systems. This is also evident in Deep Learning: being able to process (raw) data end-to-end, it may not require (costly) feature engineering, but choosing the proper types of neural networks, optimizing its building blocks, and choosing the best (hyper)parameters does require expertise and some work (e.g., Coronado and Juárez 2020, p. 7 and James et al. 2021, Chap. 10).

Timeliness can be broadly seen as “the time between a need […] and the release of the information to meet that need”; particularly for information covering a certain point or period of time, this is often conceptualized as the time between that reference point or period and when the information is made available (Yung et al. 2022, Chap. 5).³⁰ Economic indicators such as GDP and inflation are examples of time-sensitive information: the more delayed they are released, the less relevant and valuable they are to decision-makers. For the work of NSOs, once should consider time for data collection and acquisition, for data processing, and for data analysis. NSOs have processes for these three tasks and Yung et al. (2022, Chap. 5.2) differentiate between the time needed for the development of a process, i.e., from conceptualization to implementation, and the time needed for the process to run. These processes can be sequential so that one cannot begin before the previous one is finished: bottlenecks, such as editing and imputation, may thus particularly benefit from improved, model-based processes (Dumpert 2020, p. 5).

A different aspect of timeliness is the ability of a model to be used in (near) real-time, particularly to assist with data collection. In surveys, models may be employed to predict the likelihood of break-offs or of poor answering behavior and intervene accordingly (e.g., Mittereder 2019, Chap. 6). They may also be used to evaluate data accuracy³¹ as interviewers or respondents on the spot should be more able to correct errors than data processing staff can do later on. Sophisticated models that would need constant re-training during ongoing data collection might be too slow to be implemented.

First, in the ML community, adversarial robustness is about the ability to withstand attacks (Varshney et al. 2022, Chap. 11): Adversaries may either target the modeling phase, poisoning the training data by injecting additional data or by modifying data in order to change the trained model’s behavior (e.g., generally lower accuracy or a different prediction for specific, targeted points in the feature space). Or they may target the deployment phase: e.g., to be able to evade the model’s ‘intentions’, “gaming the system” (Molnar 2020, Chap. 3.1), or they may try to extract the trained model’s parameters or use the model’s outputs to reverse-engineer valuable information about the training data (perhaps even the whole data set). While ‘attacks’ might sound overt, documented examples include how slight, imperceptible-to-humans changes to some of an image’s pixels can change classification drastically. When NSOs collect their own data, there are typically very few, if any units that have the capacity to modify the training data meaningfully (very large companies might be a counterexample). This is different when there is reliance on outside data and data providers. While not used by NSOs themselves for decision-making, governmental institutions and others may base their decisions on data or results provided by NSOs, making the topic of evasion not completely irrelevant.

Second, robustness in the ML community can also be in regard to data shifts and model drift, i.e., to changes between training and deployment (Quiñonero-Candela et al. 2008; Moreno-Torres et al. 2012, Chap. 7). This an aspect of transportability, i.e., the question of whether a model trained on one data set also holds, without bias, for a different set of circumstances: In the supervised ML paradigm, the main concern is whether the target population (deployment data) and the training population (training data) exhibit the same distribution or whether there is a shift. Traditional inferential statistics is mostly worried about whether estimates from the data generalize to a ‘general population’, i.e., whether results possess external validity.³² Transportability is a more general concept than these two concerns: any change in the circumstances, environment, or context may pose a threat to the validity of a model outside its training data. Among these, the Total Survey Error (TSE) framework (see Groves et al. 2009 and Sect. 4.6) highlights changes to the data collection protocols and data processing procedures.

Third, traditional statisticians might be inclined to think of robust statistics: the “insensitivity to small deviations from the assumptions” of models in actual, finite data (Huber and Ronchetti 2009, Chap. 1.1) – in particular, the robustness of the results to the presence of outliers and otherwise extreme data points (possibly the result of gross errors) in the data (see also Hampel et al. 1986, Chap. 1.1). Ensembles are a ML answer to a lack of robustness of individual learners: this is the advantage of, e.g., random forests over a single tree (James et al. 2021, p. 340). Ensembles can be specified according to fixed rules such as averaging or majority vote. It is also possible to learn how best to weight an ensemble’s components. Clustering is another application that is often prone to instability. It can also demonstrate another way to employ ensemble thinking: different cluster models can be compared regarding their conformity and a robust combined model can be created that only contains results that are common to many or even all of the models (Hornik 2005).

Robustness does not only concern the model itself but also the assessment of its accuracy (Sect. 4.6) and model interpretations (Molnar 2020, Chap. 3.5; Dutta et al. 2022). Conversely, IML methods can uncover which features are important in a model. These may then be more closely monitored for signs of drift. Particularly for the adversarial and drift notions of robustness, features that exhibit a causal effect on the outcome are often preferred over mere correlative features (e.g., Molnar 2020, Chap. 3.1): causal relationships may be much more stable over time and much harder to manipulate or game. A model’s most important features can be found with IML and investigated in this regard. Similarly, if the most important features are just proxies, we might improve future data collections to come closer to the actual variables of interest, increasing statistical robustness by reducing measurement error: e.g., survey questions may be tweaked or alternative data sources considered. Furthermore, markedly worse than spurious features are shortcut learning and Clever Hans effects: i.e., when the training data contain signals about \(Y\) that in deployment will not be present or exhibit a very different relationship (Bellamy et al. 2022). Examples in medical image analysis for disease prediction include the type of imaging device, image timing, watermarks, or, worst, circles or arrows pointing to tumors that were of course not present on the disease-free among the training images (e.g., Chen et al. 2019, p. 104f.).

Within and across scientific disciplines, a number of contrasting definitions and terminologies exist in this area (Barba 2018; Plesser et al. 2018). They all describe under which conditions the results from a new data analysis must be the same or qualitatively similar to those from a first data analysis (Goodman et al. 2016): Methods reproducibility is defined as obtaining identical results, using the same data and the same methods. By leaving the era of ‘point-and-click adventures’ and instead archiving code and data and making them accessible, scientific communities approach what is increasingly seen as the bare minimum for credible empirical work. The importance of metadata and proper documentation has also been emphasized (Choi et al. 2022, Chap. 5). Results reproducibility is about finding the same results using different data but the same methods. We consider inferential reproducibility as whether (qualitatively) the same results are produced with the same data but different methods.³³

Reproducibility is a big part of Open Science and “Open Science is just good science in a digital age” (Seibold et al. 2023), regardless of the type of data analysis.³⁴ Thus, this important quality dimension is not tied directly to ML. It is, however, true that more sophisticated model classes are more likely to contain stochastic elements; also, data splitting in the supervised ML approach introduces randomness. If uncontrolled, these random elements are a threat to methods reproducibility. Yet, this is not restricted to ML: e.g., traditional Gaussian clustering methods may use random initialization values. Versioning, referencing, and archiving of code and data are relatively straightforward for a traditional data analysis world where everything is in-house. However, new data sources may be non-static (e.g., even past social media platform data are frequently changed retroactively, see West et al. 2023) or too big. A similar argument pertains to large, pre-trained models (e.g., foundation models) trained and provided by an outside organization and employed by a NSO possibly after some fine-tuning. Any process containing human decision-making is more difficult to archive and to reproduce than an automated one, although very strict, documented guidelines may help.

While we agree with Yung et al. (2022, p. 20) that NSOs often cannot easily collect new data – especially if they are to be collected in precisely the same manner – we, however, do not believe that this makes results reproducibility generally unachievable or irrelevant to NSOs. In fact, whether new data sources permit the same results (but more cheaply or timely, see Sect. 2.3) is directly coupled with questions of results reproducibility. Also, recall that the ML paradigm is typically not about just analyzing one data set to answer questions: rather, the rationale is the deployment of the trained model on new data – often more than once or even continuously. How well the model holds up is about results reproducibility and the reason for monitoring for model drift.

Finally, we note that ‘results’ in the above-mentioned reproducibility definitions are understood to refer to outputs of data analyses. NSOs as producers of data that are used, often in multiple ways, by end users inside and outside the organization, may also consider the reproducibility of data processing (or data production more generally). This is also true for additional information they release: e.g., IML methods may contain stochastic elements.

The many conceptions and measures of accuracy share a common notion (Yung et al. 2022, Chap. 3): accuracy is about the closeness of ‘what one has’ to the truth or, when ‘truth’ is not an adequate concept, to what one intended.³⁵ Inspired by NSOs’ dual role as producers of data and of statistical outputs, we think of three kinds of objects of interest ‘which one has’: the data, estimates of some population parameters (in traditional statistics), or predictions (in the supervised ML paradigm). In particular for the data and the predictions, one can take an individual view (a specific data point or a local prediction \(\hat{y}|x_{0}\)) or an aggregate view (differences in the distribution of the data relative to that of the target population or quantifying a model’s overall performance with one accuracy number).

Statisticians tend to think of the deviations from the truth/the intention as either systematic (bias) or random (variance). Survey methodologists often employ the Total Survey Error (TSE) framework to conceptualize the different sources for such errors (Groves et al. 2009, Chap. 2): errors of measurement (i.e., deviations of the values in the cells of the data matrix from the truth), errors of representation (i.e., differences in the composition of the analyzed data relative to the target population), and errors occurring during data analysis – although the latter are often not explicitly considered in TSE-based operations. Whether data analysis employs ML or not, the TSE framework remains a powerful tool for planning data collection and considering data quality (Puts et al. 2022). Yet, we suggest that the extensions of the TSE to newer data sources (e.g., Big Data, see Amaya et al. 2020) or to data sources more general than surveys (West et al. 2023) may provide additional value.

Accuracy metrics are at the core of the training, evaluation, and selection of models in the supervised ML paradigm (see Sect. 2.2). For the selection in particular, two types of comparisons exist. First, the relative comparison of models, i.e., against each other, is used for model selection. If NSOs wish to test new procedures involving ML against existing procedures, both must be evaluated on equal ground: ideally, on the same unseen evaluation data and in a manner identical to what the actual implementation in practice would look like. This is also true for ML-assisted procedures, e.g., combining a model’s results and human work. Second, there is also an absolute comparison of a (chosen) model’s accuracy: how well does it perform? Does it achieve a required minimum standard? On the issue of absolute comparisons, we would like to highlight a common problem in the discussion of (binary) classifiers: often, a high overall accuracy (i.e., the proportion of correct predictions \(\hat{y}=y\)) such as 0.91 is touted as evidence for a great model, implying that the suitable reference point might be 0.50 or 0. However, the correct reference point is the frequency of the majority class – and this piece of information is often missing from performance discussions. To see why it is crucial, consider an imbalanced classification problem in which the majority class occurs with a frequency of 0.9. The simple model containing only a constant and hence always predicting the majority class thus has an accuracy of 0.9.³⁶ Suddenly, the added predictive ability of \(0.01=0.91-0.9\) achieved by the selected model and its features is recognized to be only tiny.³⁷ Alternatively, as in Sect. 5, one may use balanced accuracy, i.e., the unweighted average of the true positive rate and the true negative rate, for which the constant classifier always achieves a value of 0.5.

Accurate results hinge on accurate, i.e., high-quality training data (e.g., Coronado and Juárez 2020, p. 8). Deviations might come in (or be remedied) at any level described by the TSE framework: i.e., on the measurement side, the choice and design of constructs and of proxy variables, measurement instruments, human annotations of the collected raw data, etc., and, on the representation side, the choice of a population frame (coverage), sampling schemes and sample size, strategies about how to combat nonresponse, decisions about which units to exclude during data (pre-)processing, and so on. Their consequences depend on the type of data analysis. If prediction is the ultimate goal, then the error mechanisms in the training data should be as close as possible to those in the deployment data. This is another example of transportability (see Sect. 4.4). If, however, the data are analyzed with traditional statistics, any information about the error components and mechanisms is helpful for deriving unbiased estimates via, mostly, measurement error models or mixed (hierarchical, multi-level) models. In survey data, the contributions at different levels are acknowledged, e.g., via fixed or random effects on the level of respondents, interviewers, and items (e.g., Couper and Kreuter 2013). Yet, similar information about data processing is often not released: e.g., who annotated a particular data point – a ML model or a human (and if so, a pseudo-id for the particular annotator). We must acknowledge that research on how to optimize guidelines, instructions, and other characteristics for annotation tasks is still nascent (Beck et al. 2022; but see, e.g., Fort 2016 on annotating texts).

Accuracy not only concerns the outputs (estimates or predictions) but, especially in the ML paradigm, also the performance evaluations. Adherence to good practices documented in Sect. 2.2 is key, but violations are not necessarily obvious. In particular, any type of initial, exploratory data analysis (influencing feature engineering) and kind of data processing should only be done on the training data, not the whole data including the evaluation data, in order to prevent data leakage and overoptimistic performance evaluations. Target leakage should also be avoided – but this is difficult when data processors do not know the eventual data analysis, i.e., they do not know which variables are outcomes.

Particularly for results of data analysis released by NSOs, uncertainty assessments are required (Yung et al. 2022, p. 13). While traditional statistical methods come with ‘self-assessed’ uncertainty quantifications, some ML model classes do not (e.g., support vector machines) while others do (e.g., random forests). For classification tasks, the predicted class probabilities are an uncertainty measure; however, ML model classes typically do not quantify how uncertain these probabilities themselves are.³⁸ One way to express uncertainty is, instead of point predictions, to output prediction sets (for multi-class outcomes, e.g., many image classification tasks) or prediction intervals (for continuous outcomes): conformal prediction (e.g., Angelopoulos and Bates 2022) is a technique to turn predicted probabilities or scores into such sets or intervals – with desirable properties even when the underlying model is not perfect.³⁹

Note that the ML performance comparisons typically do not involve the uncertainty inherent to having to estimate the models’ accuracy. This is unsatisfactory for NSOs: e.g., changing an existing process to a new, ML-based procedure is not cost-free and an organization wants to have some level of confidence about what procedure it should choose.

Finally, we should note that robustness and/or reproducibility without accuracy are typically of little value. (For instance, one could imagine a procedure that outputs the same value every time, regardless of the training data that was used. This is a very robust, precise, and repeatable procedure, but its predictions would not reflect reality in any way.) This re-emphasizes the importance of considering accuracy, as implied by the trade-offs implicit in the bias-variance decomposition (see Sect. 2.2) and in the duality of validity and reliability (see footnote 34).

Explainability and fairness can be viewed as strongly intertwined processes throughout the ML pipeline. At the development stage, IML methods can help understand whether and how a model inherits societal biases. To this end, initial steps may include investigating the role and importance of (correlates of) protected or sensitive attributes and studying whether ‘legitimate’ features are utilized in different ways for social subgroups. At the deployment stage, the (perceived) degree of interpretability may shape fairness perceptions of the eventual ‘user’ of the algorithm, and their reliance on the model’s outputs. If IML methods are used in either the production or deployment stage, another consideration is the degree to which the IML methods’ fidelity varies by group:⁴⁰ if the explanations are not able to correctly reflect the models’ decisions similarly across the feature space, any conclusions that are drawn about the models’ functioning can be differently accurate across subgroups.

In practice, a first step towards merging model interpretation and fairness considerations may include the use of protected attributes as grouping variables to structure the application of IML techniques. Fig. 2, for example, shows two surrogate decision trees based on the same random forest model which predicts long-term unemployment of job-seekers. In Fig. 2a, only predictions for German job seekers are used to build the surrogate tree, whereas the tree in Fig. 2b is based on LTU predictions for non-German job seekers. Note that citizenship was not used as a predictor for the original random forest. In both surrogate trees, the duration of previous unemployment benefit receipt episodes (LHG dur) plays a major role in predicting (future) LTU, with longer receipt histories being associated with higher LTU risk. However, in the surrogate tree for German job seekers older age appears as an additional risk factor. This may indicate that the random forest learned different effect patterns for both subgroups – a finding which seems reasonable from a performance optimization perspective, but which also points to fairness implications in practice when groups defined by protected attributes are being scored on different grounds: it may be the case that predictions for one group stronger rely on features that are less reliable, exhibit (stronger) measurement error, or are proxy variables for (other) protected attributes such that structural differences map to differences in prediction quality. In any case, putting an emphasis on protected groups in the model interpretation process can be helpful to understand if a model may behave differently for important subgroups in downstream applications.

Bild vergrößern

We note some further connections between explainability and fairness. Adversarial attacks are facilitated by an understanding of the model and of the data. Unfortunately, well-intentioned opportunities to probe a model offer such a gateway for attackers. Unchecked access to a model, particularly with IML tools but also under the guise of fairness evaluations, is more threatening than presenting some aggregate evaluation results. Of course, attacks can also occur on privacy (Pawelczyk et al. 2023). Very large models are able to memorize examples (Belkin et al. 2019; James et al. 2021, Chap. 10.8): allowing unchecked access to such models can then have similar consequences as just releasing the original training data would. Members of minority groups \(A\), because of their smaller size, may be both, easier to re-identify (because of the small group size) and more likely or vulnerable to suffer negative consequences from re-identification. We are not aware of privacy metrics used as fairness metrics (but see Wachter et al. 2017 arguing for privacy-preserving IML).

Individual-level explanations match well with individual-level fairness notions. Algorithmic recourse is the notion of giving explanations and recommendations (how to achieve a desired prediction) to the individual, particularly in the form of counterfactual explanations (Verma et al. 2022; Karimi et al. 2021). While giving explanations is often desirable, particularly in the context of ADM, the extent to which there exist legal rights to explanations for individuals or legal mandates for organizations gets commonly overestimated (Doshi-Velez et al. 2019; Wachter et al. 2017). NSOs may also care about individual data points: during data analysis, some may be uncovered as influential to the estimates of a statistical model, and during data processing, some may be flagged as outliers; these might cause performance, robustness, and fairness problems. If such data points were produced by a ML model, e.g., in data imputation, IML-based evaluation of these imputations can help to solve the just-mentioned problems.

In most parts, the costs of adopting ML at NSOs as presented in Yung et al. (2022) are seen to reflect technical needs such as IT infrastructure, maintenance, and staff training. We want to re-emphasize quality assurance and control as a critical component not only as a means to monitor machine learning models with respect to, e.g., fluctuations in (subgroup) performance, but also as a safety measure: Humans may (need to) overwrite the models’ output if the uncertainty exceeds a pre-specified threshold (Bhatt et al. 2020). Introducing a ‘reject option‘ in supervised learning models, i.e., forwarding difficult cases to humans for classification, can increase error-fairness (Kaiser et al. 2022), but by definition comes at the cost of additional manual work. Assessing the need and degree of human oversight thus should be factored into the cost-benefit analysis of high-stakes ML applications at NSOs. Furthermore, fairness cannot be fully automated (Weerts et al. 2023).

Some of the new data sources might be cheaper to acquire than traditional data sources, but the savings might not hold up when the additional work to clean up elevated fairness problems is figured in. This is particularly true for ‘found data’ (Groves 2011; Japec et al. 2015) over which NSOs and other stakeholders have little to no discretion in design. Even large sample sizes cannot make up for errors of measurement and representation: If a key variable is subject to differential measurement error or if a protected group is missing it does not matter how many observations are in a data set.⁴¹ Likewise, to give an extreme example, if the training data do not contain any women at all, it does not matter how much you increase the sample size – a model trained solely on men will tend to yield poor predictions for women in domains in which the two are markedly different.

A similar argument pertains to data processing: e.g., it might be better to use a medium size survey – perhaps one that combines survey responses with other data types – than to use a record linkage model that introduces fairness problems. In data analysis, simpler models have lower demands for data volume and computational resources, for both training and prediction, implying cost and time savings. In addition, higher interpretability may lead to better discovery and removal of fairness problems. Sophisticated, more flexible models might provide more accuracy and thus could be fairer by being more likely to discover model heterogeneity. Thus, both should always be included among the set of models considered for selection.

Adding fairness to the discussion and evaluation of quality dimensions should not be perceived as an additional burden to NSOs. As we try to argue and illustrate throughout this section, fairness considerations can be integrated into existing evaluation procedures in practice and can be viewed as an additional safeguard to ensure that the improvement in timeliness that may be achieved through ML-based automation does not come at the cost of disparate impact downstream. At the same time, the quality dimensions of the QF4SA framework each can benefit from a fairness perspective as it enriches the evaluation of algorithms by highlighting the critical role of (social) subgroups.

An interesting development that implicitly links fairness to timeliness is the work on fairness-aware automated machine learning (Weerts et al. 2023). In this line of research, methods are being proposed that can improve timeliness and cost by automating parts of the machine learning pipeline, while the resulting output is also required to fulfill some fairness constraints. While it is important to recognize the limits of such an approach – the authors agree that fairness cannot be fully automated –, fairness-aware AutoML can still expand the methodological toolkit of NSOs.

Timeliness, cost of data collection, and overall sample size are reasons to try to predict, e.g., response propensity in surveys. It has been recognized that focusing recruitment efforts on units with a high predicted propensity to participate is tempting on the aforementioned dimensions but widens the potential for representation bias when response propensity also depends on the outcome variable \(Y\) for the eventual data analysis. From a fairness perspective, it is important to note that hard-to-survey or hard-to-reach subpopulations (Tourangeau et al. 2014; Willis et al. 2014) may often coincide with groups for which we worry about discrimination and biased outputs (e.g., Keusch et al. 2021). Note that even if there is no relation between outcome and response propensity in a group, if the group is small in the collected data, the power to detect model heterogeneity is diminished and fairness evaluations become more statistically uncertain.

The lack of robustness and stability can be connected to fairness concerns in multiple ways. On the organizational level, model decay or drift (i.e., deteriorating model accuracy over time) can be a reason for (potentially selective) skepticism towards algorithmic solutions (Choi et al. 2022, p. 2). In downstream applications, model drift can affect different parts of the target population in different ways. That is, differential error (see footnote 41) across subgroups may surface or may be amplified due to shifts in the data to which the model is applied. It may also be harder to detect model drift that occurs mainly or first in (small) protected groups. Also, one type of drift or drift indicator, namely the emergence of new categories in a categorical feature or outcome variable, might itself be directly about the existence and recognition of protected groups. To our knowledge, monitoring of models and decisions about the need to re-train to date consider global performance measures (e.g., Choi et al. 2022). From the fairness perspective, we suggest that (sub)group measures must also be monitored: this will not only inform when error-fairness drops below a pre-specified threshold, but might also inform about the causes and possible countermeasures. We further argue that careful monitoring is also needed even if models are re-trained on a regular basis, as new biases may be picked up along the way.

Furthermore, the robustness of a model within a group and the (epistemic) uncertainty of a model’s predictions for a group have, to our knowledge, not been seen as fairness criteria so far. We suggest that these desirable global model properties should be also investigated as individual or (sub)group fairness notions and metrics in the algorithmic fairness literature. This pertains to models that are used in data collection, data processing, and data analysis.

One way to robustify models against drifts (Varshney et al. 2022, Chap. 9.4.3) is to employ causal models: e.g., causal relationships rooted in physics or biology are assumed to be more stable over time than spurious correlations.⁴² Causal features and causal fairness notions are also being discussed (e.g, Makhlouf et al. 2022; Plecko and Bareinboim 2022). Yet, while employing causal features may be attractive in some settings, for images or other unstructured data types analyzed with Deep Learning, traditional features (in a potentially causal sense) are hardly involved.

Monitoring fairness metrics can be particularly important in a deployment context that includes data sources that capture complex, natural processes. In Fig. 3 we hold the model design constant, that is, random forest models for predicting long-term unemployment are used with the same hyper-parameter settings and features, but we repeatedly train and test models with data that change over time. Specifically, we use labor market records from 2010–2016 and train one random forest model for each year, and evaluate the respective model with data from the next year. The bold black line shows the difference in overall model performance (balanced accuracy) as we move from one year to the next. From this point of view, we might conclude that we can safely apply our random forest modeling schema over time without any major disruptions. However, assessing fairness metrics points to a different conclusion: a considerable increase in false negative rate (FNR) differences between non-German and German job seekers (dashed red line) can be observed when training and evaluating models with more recent data. This is accompanied by increasing parity differences in the models’ predictions (dashed green line), which over-amplify the true differences in base rates as observed in the data (dotted gray line). As such changes over time can have considerable implications in practice, requiring robustness assessments to also consider subgroup-specific (fairness) metrics appears advisable.

Bild vergrößern

From a fairness perspective, (inferential) reproducibility raises questions as to how strongly design decisions in the machine learning pipeline affect outcomes not just overall, but also separately for sensitive subgroups of the target population. Fairness-relevant decision points may not only include the machine learning model itself (e.g., the model type and hyperparameter settings), but also more subtle aspects that include implicit decisions in data pre-processing steps (e.g., NSOs may employ a standard procedure for imputing missing values, while different imputation strategies can affect fairness measures in different ways; Caton et al. 2022). In practice, the implications of non-reproducibility may again be assessed by structuring model evaluations by protected attributes, paired with a grid of design decisions that is centered around the intended deployment setup.

A strong susceptibility to design decisions is of particular concern if the model outputs are further used downstream, either as an input to further analysis or to directly inform actions. Fig. 4 focuses on the effects of different hyperparameter settings on the classifications of random forest models predicting long-term unemployment. Four forests were trained that differ in the number of trees and the minimum size of the trees’ terminal nodes and are then used to predict LTU, using the same classification threshold (top 25%). The Jaccard similarities, denoting the overlap (between 0 and 1) between the LTU predictions of the different random forest models are plotted, separately for German (Fig. 4a) and non-German (Fig. 4b) job seekers. Considering the modest changes that were made in the random forests’ setup, we observe non-trivial differences between the lists of job seekers that are predicted as being at high risk of LTU by each model. While this generally holds for both German and non-German job seekers, the lowest agreement in predictions is recorded in Fig. 4b (between RF 2 and 4). Assessing the susceptibility of outcomes to small changes in the modeling design with a focus on protected groups thus may allow to identify variation that can challenge both overall reproducibility and the consistency of outcomes for societal subgroups.

Bild vergrößern

As stressed in Sect. 4.5, methods reproducibility is increasingly viewed as a minimum standard. If the root causes of a model’s discovered fairness problems are investigated upstream, the respective models that were used, e.g., in data processing must be reproducible or the search for problems and solutions may be futile.

In the three aforementioned definitions of reproducibility (see Sect. 4.5), one criterion was whether the methods are kept the same. As described in Sect. 2.2, for supervised ML problems, typically many supervised ML model classes are trained and only one model is selected, typically based on performance. In that sense, some exploration of the role of methods is more built-in for ML than for traditional statistics – although it would not be accurate to suggest that ML practice is anywhere close to a multiverse analysis approach, in which all decisions that can be made by an analyst are evaluated (Steegen et al. 2016). Simson et al. (2024) propose to look at all these choices – including those that affect how the raw data are fed into model training – and how they affect fairness.

Fairness interacts with accuracy (and with the human component) at multiple stages of the production process of NSOs. In the context of data processing and preparation, we note that one of the purported benefits of automation is an increase in consistency: e.g., in annotation tasks, even subject matter experts can disagree (Sthamer 2020b, p. 12) and, over time, an annotator might become tired or less motivated (inter- and intra-annotator reliability, respectively).⁴³ Meanwhile, a model will ‘decide’ the same way every time – but it is trained to reproduce the patterns contained in the training data, including those made by human annotators. First, consider a single annotator. She or he is or feels required to provide a label even for difficult-to-decide cases. Absent any other option, the annotator might resort to the marginal distribution of \(Y|A\) (or their subjective notion thereof), even when \(Y\) is independent of \(A\) given \(X\). The trained model will then learn that \(A\) (or proxies of \(A\)) are predictive of the provided labels (even though it is not predictive of the true \(Y\)). Solutions may include letting annotators express uncertainties instead of forcing a choice; this may actually fit well with the aforementioned reject-option models (Gruber et al. 2024; Bhatt et al. 2020). Also, information about \(A\) may be better hidden from annotators, although it can be difficult to do so in, e.g., image-based classification tasks. A second mechanism of how annotations can induce biased models predictions is a non-random allocation of observations to a set of inherently heterogeneous annotators: e.g., if units from \(A=a\) are mostly processed by an annotator with a low general propensity to label \(Y=1\) and, conversely, units from \(A\neq a\) are mostly processed by an annotator with a high general propensity for \(Y=1\), again \(A\) becomes predictive of the labels even though when it is not related to the true values \(Y\). This is an example of how biases can be introduced during data processing even when there is no overt discrimination. A similar argument pertains to data collection, e.g., the allocation of interviewers might cause measurement or representation errors. We see two options here: NSOs can use stratified randomization to allocate data points to annotators and they can release annotator IDs because, conditional on the annotator, the spurious relationship of \(A\) and the labels vanishes.

Aggregation of data is one of the core tasks of NSOs: both in terms of data analysis, which may be simple descriptive statistics, and in terms of producing (aggregate) data for release. For the former, ML may seem hardly helpful for the estimation of population parameters: Other than (short) trees, ML model classes hardly possess parameters that correspond to interpretable, meaningful population characteristics. Also, systematically biased estimation, due to the bias-variance trade-off caused by training supervised ML models to minimize the expected prediction error, is problematic. However, ML can be useful when a parameter of interest is not identical across all subpopulations. For up to a medium number of pre-identified subpopulations, multiple testing correction can be employed to limit the error of falsely claiming heterogeneity (GCSILab 2023, Chap. 4.1). If there are, however, many subpopulations to investigate, as is the case with intersectional fairness, or if there are no pre-specified hypotheses at all, ML can help to discover heterogeneity: data splitting is then the procedure that guards against false discovery (GCSILab 2023, Chap. 4.2). If interpretable heterogeneity is the goal, trees for univariate statistics or, for more complex analyses, causal trees (Athey and Imbens 2016) appear to be most suitable.⁴⁴ We suggest that NSOs use such methodology for more fair reporting of the results of data analysis: subgroups for which the parameter deviates more than a pre-specified, meaningful amount from the global average should be identified and reported along with the global average. Aside from fairness, this approach can also be used to determine whether the global parameter value is meaningful and worth reporting at all: from Simpson’s paradox, it is well known that the global value may be completely different than the value in all subpopulations, e.g., taking on a different sign.

For error-based fairness notions, the same methodology can be used to find subpopulations that suffer from more errors. Similarly, if there is gold-standard evaluation data for a data production process whether based on human work, ML, traditional statistics, or a combination thereof, such as editing and imputation, NSOs can find groups for which their process performs more poorly compared to others or to an absolute threshold. If such heterogeneity is found, it may help – but will not replace subject matter and data knowledge – in fixing deficiencies in data collection (e.g., improving survey questions to yield less measurement error for \(A\)) and data processing systems. To our knowledge, using heterogeneity-finding ML machinery has not been explicitly suggested in the fair ML literature for either fair reporting of data analysis or finding biases in the data (production process), potentially with the exception of Zahn et al. (2023).

Self-assessed confidence measures by ML models may be used to decide whether something should be labeled by the model or be referred to a human expert (e.g., Text Classification Theme Group 2022 for text classification). However, not every ML model class yields self-assessed uncertainty measures and for those that do, there is no guarantee that they are accurate on average. Moreover, a model’s overconfidence may not be the same for every group but could be worse for some (protected) groups or individuals. Some uncertainty measures also do not recognize epistemic uncertainty (i.e., uncertainty related to the correct specification of the structure of a model class and to the estimation of model parameters due to sampling uncertainty; Gruber et al. 2023) which may be greater for small minorities \(A\). Uncertainty evaluations thus must also use actual evaluation data and cannot solely rely on models’ self-assessments.

In a supervised learning context, accuracy as a quality dimension can be naturally extended to capture fairness concerns by requiring accurate predictions not just overall, but also for subgroups which may be defined by protected attributes or other features that are viewed as substantively relevant in a given application (Kim et al. 2019; Hebert-Johnson et al. 2018). Based on our long-term unemployment prediction example, Fig. 5 shows balanced accuracy scores of a random forest predicting LTU computed for 48 subgroups in the test set. Specifically, subgroups of job seekers were defined by intersections of the attributes citizenship, gender, age group, and region. While the model achieves an overall balanced accuracy score of 0.667, considerable variation in subgroup performance can be observed. Accuracy ranges between 0.417 and 0.8, indicating that prediction performance is no better, and sometimes worse, than random guessing for some demographic subgroups. The strongest variation in scores can be observed for non-German job seekers (upper half of Fig. 5), i.e., for subsets of the minority group in our example. While this finding may in part be driven by small sample sizes in some cells (although all but three cells include more than 50 observations), it highlights the utility of assessing subgroup accuracy as a means to provide pointers for further model investigation.

Bild vergrößern