Assembled Bias: Beyond Transparent Algorithmic Bias

Based on empirical studies, this acceptance of the automated decision-making is not uncritical, with initial trust affected negatively by subsequent mistakes detected (Hildago et al., 2021).

For the purposes of this paper, we take machine learning to be any technique that creates new features, as we discuss in Sect. 3.2.

Here by explicit and implicit, following Johnson (2021), we are not necessarily tracking a distinction in the possessor’s awareness, but rather the distinction between being explicitly represented versus not.

See Bonnefon et al. (2020) for a discussion of how the COMPAS case illustrates that programmers face an ethical trade-off with machine learning between equal predictive power versus equal false positive and false negative rates for different groups. For lengthier treatments of racial bias with algorithmic decision-making, see Ferguson (2017) and Safiya (2018). In July 2021, it was reported the some citizens in Pasco, FL had received letters from the Pasco Sheriff’s office that they had been selected for policing surveillance on the basis of “an unbiased, evidence-based risk assessment designed to identify prolific offenders in our community” (Blest, 2021).

Our enhanced recognition of this kind of algorithmic bias is aided, in part, by the use of AI audits, as will be discussed shortly below.

See Johnson (2021) for a discussion of the natural kind of implicit bias inclusive of both algorithmic bias and human implicit bias.

For a proposed framework for algorithmic auditing as applied to machine learning in hiring, see Wilson et al. (2021). For a socio-technological approach to hiring algorithms and auditing, see Sloane et al., (2021). For a discussion on the efficacy of available AI audit for commercial facial detection AI, see Raji and Buolamwini (2019). For a description and available representative toolkit, see Bellamy et al. (2019).

For a discussion of right to explanation and ML decision-making, see Vredenburgh (manuscript). For a discussion of transparency of AI systems and explainable AI frameworks, see, for instance, Creel (2020), Gunning et al. (2019), and Zednik (2019). Rudin (2019) argues that programmers ought to favor the creation of inherently interpretable models over explainable black box models for high-stakes socially embedded algorithmic decision-making.

Gandy, for instance, in a 2020 interview with Logic magazine presses that members of algorithmically defined groups could be discriminated against without their knowledge, stating “But what I’m trying to get us to pay attention to is the other groups that we have been assigned membership, and through which we experience discrimination, but which we know nothing about.” Later in the interview, he notes that “[s]o yes, we need to address the variety of identities that people are aware of, including the many new ones being created. But we also need to address those that we don’t have names for yet, the ones that are being generated by algorithmic manipulation.” Creel and Hellman (2021), meanwhile, critically assess why is it morally wrong, when it is, that individuals are systematically excluded on the basis of algorithmic classification when the ML model classifies outside of protected class groups. Likewise, Farrell and Fourcade (manuscript) highlight that the classifiers, or categories, that machine learning algorithms use to sort individuals are distinct from human bias in that such categories are emergent and dynamic. Our contention of a novel algorithmic bias both in source and content reinforces these worries for the use of ML-driven decision-making in social contexts. We thank an anonymous reviewer for drawing our attention to these prior relevant discussions of the moral ramifications of algorithmically defined groups.

For an updated review of discussion of the interpretability problem for machine learning and assessment methods, see Linardatos et al. (2021).

Elsewhere, Buckner discusses the nature of this process of feature formation for deep convolutional neural networks as feature learning and feature abstraction (Buckner, 2018, 2019) or feature detection (Buckner, 2020). It should be stressed that new feature formation distinguishes neural networks from other ML programs by the manner these new features are generated, and that new feature formation is an integral aspect of machine learning more generally. Here we take a plethora of cross-cutting approaches and techniques to fall under the ML umbrella, including, but not limited to, methods of classification, regression, clustering, visualization, and projection, and methods such as decision trees, random forests, gradient boosting, and support vector machines.

Ilyas et al. (2019) asks whether highly predictive features, which they refer to as nonrobust, are part of the objective world. Here by nonrobust, they mean not comprehensible to humans, akin to not interpretable (Doshi-Velez and Kim, 2017). It is this aspect of the created features that we will argue makes the emerging novel bias, composed of these features, as so epistemically pernicious. By using the term feature creation we are not taking a stand on the issue of whether such features are part of the objective world or not.

The use of visualization tools to make ML representations at the hidden layer(s) more perspicuous has been used by, among others, Olah et al. (2017) and has been reviewed elsewhere (Zhang and Zhu, 2018). Olah uses a circuits approach to yield more interpretable visualizations of artificial neural networks that are utilized to model visual processing in biological systems. Further, Olah claims, albeit cautiously, that biological neural systems may learn similar features to those learned by artificial neural networks for this task domain. We will not take a stand in this work on whether similar learned features are shared across artificial and biological neural circuits for, say, vision at a subpersonal level of processing. For more informative work on this issue, see, for example, Yamins and DiCarlo (2016). Yamins and DiCarlo (2016) discuss how hierarchical convolutional neural networks aid in understanding sensory cortical processing, such as in the ventral visual stream, in humans. Rather, our assumption that humans have superior low-dimensional spatial reasoning skills is restricted to person-level processing. We acknowledge that in claiming superior reasoning in this domain relative to machine learning, the human visual system can error (e.g., the phenomenon of pareidolia—seeing everyday objects in random patterns) (Bies et al., 2016).

For a good non-practitioner introduction to convolutional neural networks, see Choi et al. (2020).

For a recent review of the literature on adversarial examples in machine learning, see Wiyatno et al. (2019).

For an accessible recent primer on recursive neural networks, see Salehinejad et al. (2017). We use a recursive neural network to aid in exposition, enabling us to cleanly progress from higher zooms to lower zooms and see how the zooms relate. However, importantly, assembled bias is not endemic to machine learning with recursive neural networks or with neural nets specifically but, rather, occurs with machine learning more broadly. For more on the notion of ‘zoom’ operative, see footnote 18.

This is an approximate description of the process generally. Offering a more technical explanation would take this discussion into the domain of mathematics or computer science, but fortunately we need not do so here, as we can lean on our intuition for certain tasks that humans are better at than machines, such as recognizing a picture of a spider. See, for instance, Dara and Tumma (2018) for a survey of feature extraction for deep learning from a more technical approach.

We use the term ‘zoom’ to lean on the intuition we gain from our spider example. In this context, zoom explicitly refers to the number of pixels being considered (the literal zoom on the spider picture). The features for lower zooms (bigger box, more pixels) of the spider are assembled from the features of higher zooms. Hence, the general ML analogue of higher and lower zooms are earlier features (the initial features given to the ML, and features assembled from them directly) and features assembled from these earlier features, respectively, and need not correspond to images.

The dynamic nature of ML representations in feature creation echoes Farrell and Fourcade (manuscript)’s concern that classifiers that ML algorithms use to sort individuals are emergent and dynamic.

For early discussions of intersectionality, see Crenshaw (1989). For a recent overview of intersectional theory, see Crenshaw (2017); Collins and Bilge (2020).

We qualify this statement with “does not necessarily,” as an algorithmic decision over social data can be plagued with the compounding of assembled bias on top of algorithmic bias stemming from, say, underrepresentation and proxy features. On its own, though, assembled bias does not exaggerate the initial features themselves (hence the lack of images of darker pixelled spiders or differently sized spiders generated from BigGAN). As argued in Sect. 3.3, assembled bias exaggerates assembled features in the ML higher-dimensional feature space and so the representation of the population itself.

We won’t take a stand here on the nature of wrongness of social discrimination, but for candidate proposals, see, for example, Alexander (1992), Arneson (2006), and Hellman (2008).

In real-life examples, such ML applications to credit scoring may well be compounded by algorithmic bias in the training set and proxy bias.

For recent coverage of cases of women and of residents of particular post codes being given smaller lines of credit, see Telford (2019) and Ziffer (2022). While protected class information is not the culprit (Campbell, 2021), some have offered proxy problem explanations of real-life cases (O’Sullivan, 2021). Our working example abstracts away from protected class concerns and any proxies to an illustrative plausible case of assembled bias. Our contention above is that exaggeration of the social population and privilege in credit lending can crop up with machine learning on social data even apart from sampling and proxy issues.

See O’Sullivan (2021) for a discussion of this point with regard to credit scoring and potential gender discrimination in assigned credit lines.

See Mittelstadt (2017) for a discussion of the emergence of algorithmically assembled groups more broadly and the implications for privacy.