Studying bias in visual features through the lens of optimal transport

Throughout our analysis, we work with a feature extractor \(F: {\mathcal {I}} \rightarrow \mathbb {R}^d\), where \({\mathcal {I}}\) is an image dataset of which the samples are converted to d-dimensional feature spaces. In these datasets, we consider a sensitive binary attribute S that assumes values among \(\{s_0, s_1\}\) for each data sample. We also denote the sample distributions of images sharing sensitive attribute value \(S=s_0\) and \(S=s_1\) as \(\mu\) and \(\nu\) respectively (e.g., women and men). That is:where \({\mathcal {I}}_s=\{x \in {\mathcal {I}}\; |\; x\, \textsf { has}\, \textsf {sensitive}\, \textsf {attribute}\, \textsf {value } S=s\}\) and \(\delta _x\) is the Dirac distribution centered at x, which lets us move from integral definitions to discrete computations.

Given two probability measures \(\mu\) and \(\nu\) over two spaces X and Y and a (lower semi-continuous) cost function \(c:X\times Y \rightarrow \mathbb {R}\), OT theory (Villani 2003, 2008) searches for the minimal cost of transferring one measure into the other. In this work, the probability measures correspond to the distribution of samples with different sensitive attribute values, for example men and women, in the same image feature space (in that case \(X=Y\)).

As an intuitive formulation for the OT theory (Villani 2003), one can imagine a set of mines and factories, and the respective distributions of coal production and demand. In this setting, solving an OT problem would mean finding the most cost-effective transport map of which mine should provide coal to which factory. This translates to the following objective, known as Monge’s problem:where \(\nu=T_{\#}\mu\) and \(T_{\#}\) is the push-forward of \(\mu\) along the function \(T:X\rightarrow Y\).

The condition of T being a function can be too restrictive in that it transfers the entire production of each mine to only one factory, thus creating a potentially insolvable Monge’s problem (Villani 2008). To address this theoretical concern, the more general Kantorovich’s problem aims to find the coupling between two measures \(\mu\) and \(\nu\) that allows the masses allocated on individual elements to be split. This is formulated per the following objective:where \(\Pi (\mu ,\nu )\) denotes set of couplings between \(\mu\) and \(\nu\).

When the sets are the same d-dimensional Euclidean spaces \(X=Y=\mathbb {R}^d\), one popular cost function is the quadratic Euclidean distance \(c(x,y)=\tfrac{1}{2}\Vert x-y \Vert _2^2\). Under these circumstances, the square root of the solution to Equation 2 is a metric in the space of measures (with finite second moments) over \(\mathbb {R}^d\) and is usually referred to as the 2-Wasserstein distance \(W_2(\mu ,\nu )\). This property is the one that guarantees that Wasserstein distance is suitable for checking demographic parity (Sects. 2.3 and 3.2).

Solving the OT problem in the space of couplings is usually difficult, but it is equivalent to the tractable dual form (Villani 2003):where \(W_2^2\) is the quadratic Wasserstein distance, \(\Phi _c\) is the space of function pairs \((f, g) \in L^1(d\mu )\times L^1(d\nu )\) such that \(f(x) + g(y) \le \frac{1}{2}\Vert x - y \Vert _2^2\) for \(d\mu\)-almost all \(x\in X\) and \(d\nu\)-almost all \(y\in Y\). An intuitive interpretation (Villani 2003) of the dual form is that, instead of minimising the transportation cost of coal from the mines to factories, it maximises the profit of an external company to which the transportation problem is outsourced. The values of f(x) and g(y) respectively define the company’s price for loading the coal at a mine x and unloading it at the factory y and are known as Kantorovich potentials.

We state two results (Villani 2003) that are useful in the remainder of this work. Given that the probability measures \(\mu\) and \(\nu\) over \(\mathbb {R}^d\) that exhibit finite second-order moments the following hold true:

Fairness definitions can be categorised (Mehrabi et al. 2021) as either group, subgroup, or individual fairness. The first two aim to balance properties of interest, such as the percentage of positive predictions or misclassification rates, between groups and random subgroups respectively. Individual fairness (Dwork et al. 2012) aims to treat similar individuals similarly.

In this work, we capture distributional differences between two groups of images, which is a type of group fairness. In particular, this consideration is a form of demographic parity (Kamiran et al. 2010; Kamiran and Calders 2012; Kamishima et al. 2012; Ntoutsi et al. 2020), which stipulates that both groups should exhibit similar statistical properties. Demographic parity typically involves checking for proportional representation of each group in outputs of AI systems. In this work’s terminology, demographic parity can be written as:where, for this definition, \(F:X\rightarrow \{0,1\}\) is a binary classifier. In this case, this is equivalent to the otherwise stronger condition⁴Contrary to other types of group fairness, such as disparate mistreatment that aims to create similar error rates between groups (Zafar et al. 2017), demographic parity can also be quantified on reference or training data by measuring the representation of groups in desired prediction outcomes instead. As such, it has been used to quantify raw dataset biases as well (Sattigeri et al. 2019; Ding et al. 2021).

In this work we have a similar goal of quantifying CV dataset bias, although we are looking to do so regardless of downstream tasks in which datasets are used. Thus, we adopt Equation 5 as a fairness definition with the only change that \(F:X\rightarrow \mathbb {R}^d\) is an embedding into a high dimensional Euclidean space. The metric properties of the Wasserstein distance ensure that it is a good way to measure demographic parity in feature spaces.

OT theory has recently been used as fairness constraints to enforce variations of demographic parity in bias mitigation tasks (Jiang et al. 2019; Gordaliza et al. 2019; Zehlike et al. 2020; Chiappa et al. 2020; Chiappa and Pacchiano 2021), to measure and explain bias in classification or (1-dimensional) regression (Miroshnikov et al. 2022) or generally look at the problem of bias through the lens of OT (Kwegyir-Aggrey et al. 2021).

Most OT bias works intend to understand and mitigate model bias and are not suited to our data mining and bias discovery perspective. The approach closest to ours is the one of Kwegyir-Aggrey et al. (2021); it introduces both an individual and a global measure of bias, where individual bias refers to individual sample bias and is defined as:where the cost function c is the distance \(d(x_0, y)\). Later, in Proposition 1, we show that the above rankings and Kantorovich potentials are equivalent when the cost function c is the square distance. This approach also sums all individual bias scores to procure a global measure of bias, which is a coarser measure than \(W_2^2\) in that it does not take the transport map into account. Finally, this approach was originally devised to measure bias in the prediction space of tabular data, instead of the feature space of vision data.

Fabbrizzi et al. (2022) found that bias in visual data can be roughly divided into three categories: selection bias, framing bias, and label bias. Selection bias refers to “disparities or associations created as a result of the process by which subjects are included in a visual dataset”. Framing bias comprises “associations or disparities that convey different messages and/or can be traced back to the way in which the visual content has been composed”. Finally, label bias comprises “errors in the labelling of visual data, either with respect to some ground truth, or due to use of poorly defined or inappropriate semantic categories”.

Previous findings indicate that the presence of visual data biases can indeed be detected early. Characteristically, Torralba and Efros (2011) showed that benchmark datasets for object detection tend to exhibit heavy selection biases. For example, out of popular vision datasets that include depictions of cars, ImageNet (Deng et al. 2009) exhibits a strong preference for racing cars, while Caltech101 (Li et al. 2022) for side views of cars. These biases were found by training an SVM (Cortes and Vapnik 1995) to distinguish the source of images. The detection accuracy of the model serves as a global indication of how biased the datasets were. The same method can check for bias within a dataset, by training the SVM to distinguish between groups of samples.

The confidence of SVM predictions can help us draw conclusions for individual samples, where high confidences indicate more biased images, i.e., which exhibit characteristics intrinsic to datasets or groups of samples. While this method was introduced as a toy experiment, it remains popular in the literature (Tommasi et al. 2015; Panda et al. 2018; Kärkkäinen and Joo 2021; Wang et al. 2022). In Sect. 5.1, we employ a variation of SVM-based biased image detection as a baseline against which we compare our approach.

The dual formulation of the Kantorovich’s problem in Equation 3 lets us compute \(W_2^2\) by approximating the Kantorovich potentials f and g with deep neural networks (Makkuva et al. 2020). We also follow this technique, because it works around the expensive computational demands of solving OT problems for high-dimensional data (Cuturi 2013) by considering the following reformulation of the Kantorovich dual problem:

This reformulation enables approximation of \(W_2^2\) by substituting \(\texttt {CVX}(\mu )\) and \(\texttt {CVX}(\nu )\) with the set \(\texttt {ICNN}(\mathbb {R})\) of scalar-valued Input Convex Neural Networks introduced by Amos et al. (2016). A byproduct of solving this problem is the pair of convex functions \(\phi\) and \(\psi\), which can be used to express the Kantorovich potentials as \(f(x) = \frac{1}{2}\Vert x \Vert _2^2 - \phi (x)\) and \(g(y) = \frac{1}{2}\Vert y \Vert _2^2 - \psi (y)\) for f and g as in Kantorovich duality of Equation 3. Hence, we easily compute the value of f for image features and rank images according to the produced potentials.

The push-forward formulation of demographic parity at the end of Sect. 2.3 is also applicable to high-dimensional systems, such as image feature vectors outputted by feature extractors. For any (Borel) subset of such feature spaces, the formulation imposes that the probability of sampling the feature vector of an image from such a subset does not depend on the demographic attribute. Note that this property is stronger than comparing distributions through their average. A visual demonstration can be found in Figure 2.

As the Wasserstein distance is a metric on the space of probability measures with finite second moments, we have that \(F_{\#}\mu = F_{\#} \nu\) iff \(W_2(F_{\#}\mu , F_{\#}\nu ) = 0\). Thus, we propose using \(W_2^2\) as a measure of bias:

While we used \(W_2^2\) to assess bias at the dataset level, we also explore the contribution of images with features \(x_i \in {\mathcal {I}}\) drawn from \(\mu\) by capturing how far they are from the whole distribution \(\nu\). To do this, we look at the interpretation of the dual Kantorovich problem in Equation 3; Kantorovich potentials \(f(x_i)\) compute the prices for moving \(x_i\) into their correspondent image sampled from \(\nu\) to maximise profit. Intuitively, the greater the distance the higher the price, at least on average. We theoretically express this property below:

Given the above result, we propose using the potential function f to rank the images in X according to how costly they are to be transported into the distribution \(\nu\). This can drive qualitative bias analysis that inspects higher-ranking images for common characteristics. This is fundamental to understanding whether the differences in the distributions of two protected attributes’ values are due to discriminatory characteristics or not.

The approximation method reviewed in Sect. 3.1 works under the hypothesis of Brenier’s theorem, which stipulates that at least one of the two probability distributions is absolutely continuous. Since we end up with two sample feature distributions, we adjusted the algorithm’s original form presented by Makkuva et al. (2020) to ensure that the theorem’s requirement is met. The adjustment consists of sampling from the kernel density estimation of one of the two distributions instead of directly parsing raw data.

The Mapper algorithm (Singh et al. 2007) is a method for reducing high-dimensional data to a graph, called the nerve complex, that provides a high-level understanding of the topological structure of the data. We want to exploit Mapper’s properties to get a fine-grained bias analysis of the data.

In particular, assume a dataset D lying in a topological space X and a continuous function \(f:X\rightarrow \mathbb {R}\) that quantifies a certain property of the data to be studied. Given a covering \({\mathcal {U}}\) of \(\mathbb {R}\), Mapper stipulates that a covering of X is obtained from the pre-image \({\mathcal {V}} = \{ V \subseteq f^{-1}({\mathcal {U}}) \text { connected components}\}\). Thus, it defines the nerve complex graph \(N({\mathcal {V}})\) of the covering by considering as nodes the connected components of the sets \({\mathcal {V}} = f^{-1}({\mathcal {U}})\) and forming edges between \(V_i\) and \(V_j\) only if \(V_i \cap V_j \ne \emptyset\).

To mimic this graph construction process on a dataset D, whose discrete data sample nature prevents the identification of connected components, Mapper performs a clustering step on the pre-image space of each \(U_i \in {\mathcal {U}}\) along \(f|_D:D\subseteq X \rightarrow \mathbb {R}\). Thus, the clusters play the role of connected components.

Mapper works with any clustering algorithm. This way, we obtain a clustering of D and the relative nerve complex \(N({\mathcal {V}}|_D)\). Depending on the filter function f, the clustering captures different information that we can use for exploratory data analysis. This makes Mapper a suitable choice for our framework as it allows clustering the data using the Kantorovich potentials introduced in Sect. 2.2 as a filter function.

Our study is conducted on the well-known benchmark CelebA dataset (Liu et al. 2015). This contains 202,600 face images, each of which is supplied with 40 different attributes. The dataset is derived from the benchmark CelebFaces+ (Sun et al. 2014) by annotating the latter with the help of a professional data annotation company. The annotation includes information about the gender of subjects depicted in each image.

We follow the pipeline outlined in Figure 1, that aims to detect bias after feature extraction, but before training any system. In principle, our OT results could be applied to any kind of feature spaces, as long as these are endowed with a metric. However, the adopted approximation method of Makkuva et al. (2020) (Sect. 3.1) is tailored to Euclidean spaces only and, thereon, deep-neural network features.

Our approach admits any (Euclidean) feature extractor and we explore three popular ones: a) ResNet (He et al. 2016) is a well-known convolutional neural network architecture used for a variety of vision tasks (e.g., object classification, object recognition, segmentation). We use its PyTorch implementation⁵ with the default weights pre-trained on ImageNet (Deng et al. 2009) as a feature extractor by cutting out its fully connected output layer. b) Autoencoder features trained with a CNN-based encoder and decoder to minimise the reconstruction loss of CelebA dataset. c) FaceNet (Schroff et al. 2015) is an embedding framework used for facial recognition and face images clustering. We used a Pytorch implementation⁶ with default weights pre-trained on VGG Faces 2 (Cao et al. 2018).

Clustering methods in the last step of our pipeline are often affected negatively by high-dimensionality. Here, we support that step by preemptively performing dimensionality reduction. This step is not always necessary, and may (should) be omitted for exploration of low-dimensional features, where it risks impacting the image ranking induced by Kantorovich potentials. Hence, it is important to select a dimensionality reduction approach (e.g., among popular ones) that mostly preserves ranking order. We show what this selection entails in Sect. 5.4, which leads us to reduce the extracted 512-dimensional features to 3 dimensions with PCA. This reduction mostly preserves image ranks produced by the next step while also creating clustering-friendly image representations.

To score -and ultimately rank- images based on their contribution to demographic disparity, we approximate \(W_2^2\) in the way described in Sect. 3.1, which produces Kantorovich potentials as a computational byproduct. We obtain the following approximations to serve as bias scores for different feature extractors: 20.97 for ResNet, 29.88 for the autoencoder, and 0.06 for FaceNet. If embedding spaces are the same (e.g., are all \(\mathbb {R}^{512}\)), such scores become comparable. If some feature extractor shrinks the space (for example, FaceNet features are normalised), we are going to obtain smaller bias scores, without this necessarily implying lesser bias.

As stipulated in Sect. 3.2, we retain the Kantorovich potentials during computations and use them to rank women in the dataset. Then, in Figure 3, we show the images of women with the top and bottom 18 potentials for each of the feature extraction methods. The most common attribute among the top images for ResNet18 and autoencoder is having long hair, while women in the bottom images either have their hair tied up or shorter hair, or wear some sort of headgear. This indicates that hair length could be a discriminative factor when correlated with predictive tasks and should therefore be kept in mind when using the first two types of features to produce new systems. To make matters worse, the ranking for the autoencoder feature seems to also be correlated with skin colour and this warrants additional investigation. On the contrary, a visual inspection of the most extreme ranking for FaceNet features does not highlight apparent prospective sources of discrimination for these.

We finally conduct a finer-grained analysis throughout the available data. To do this, we follow the methodology suggested in Sect. 3.4 and apply the Mapper algorithm using the Kantorovich potential functions as filters; the resulting nerve complex guides our bias investigation. To run the algorithm, we enlist K-Means as its clustering component and select a number of clusters for each of its layers by applying the elbow technique on the clustering inertia measure (Cui et al. 2020). This retains a homogeneous level of Kantorovich potentials within each cluster.

Figure 4 shows the nerve graph for ResNet18 features produced by Mapper. We uniformly sample images from the clusters that compose the shortest path between two clusters in the first and last layers. As hypothesised from the ranking inspection, women in the lower-layer clusters (purple/blue nodes in the nerve graph) have their hair covered and at intermediate layers have progressively longer and less covered hair, until the end-result is more pronounced in the top layers (yellow nodes). This particular path visually corroborates the previous step’s hypothesis that hair length and obfuscation is a characteristic captured by ResNet18, and hence should be kept in mind as a potential source of bias.

Similar behaviour is exhibited by the autoencoder’s features, where once again hair length increases as we move from the lower layer to the higher ones. The full exploration can be found in Appendix C. Previous work (Balakrishnan et al. 2020) points out that the hair length characteristic might be correlated with skin colour and would turn an apparently innocuous bias into something potentially discriminatory. As for the FaceNet features, the Mapper analysis does not provide any additional interesting information about possible biases.

To address potential confirmation biases during the above investigation, we also set up a complementary user study. This aims to check whether other people can also discover prospective biases in CelebA images by following our approach compared to randomised dataset exploration. The study gathered responses by 25 participants from the authors’ research groups, which viewed image batches and were asked to answer questions pertaining to bias insights they revealed about the data. 13 of the participants were randomly assigned as a control group and were shown uniformly sampled images, whereas the rest were shown batches produced with our methodology. Further details on the questions and their analysis can be found in Appendix D. Overall, we found that participants in the experiment group would be able to find a variety of biases in the batches we showed them and, in particular, to find a different representation of women between the images ranking high with respect to the Kantorovich potentials and those ranking low. On the contrary, the participants in the control group were prone to find biases that are very prominent in CelebA (e.g., majority of white women) and when asked to compare two batches they would be less confident in describing one of them as more biased. Generally, we draw the conclusion that, while there is some possibilities to fall into some confirmation bias, our method is useful to formulate hypotheses about possible biases in the data.

Goal. The notion of statistical parity led us to select \(W_2^2\) as a measure of disparity. We investigate whether this choice can quantify dataset bias as well as alternatives, like the popular SVM-based approach of Torralba and Efros (2011) and the measures proposed by Kwegyir-Aggrey et al. (2021).

Torralba and Efros (2011) measure how separable two datasets or groups of samples are. Analogously to our approach, they set SVM accuracy as a global measure of bias, and classification confidence for each image as a bias contribution score. To make a fair comparison between this approach and our use of \(W_2^2\) , we replace accuracy (which suffers from the limitation of being bounded in [0, 1]) with the average quadratic distance of points from the SVM’s decision boundary \({\mathcal {B}}\):This way, the distance \(d(p, {\mathcal {B}})\) replaces the (lack of) classifier confidence.

We also consider a variation of this methodology that replaces the SVM with a fully connected neural network. Computing the distance from the latter’s decision boundary is not as straightforward, and we approximate it by repurposing the LIME algorithm (Ribeiro et al. 2016):

Kwegyir-Aggrey et al. (2021) instead propose a linear individual bias measure that depends on averaging over pairwise sample distances, as shown in Equation 6. To make this comparable to our approach, we create an adjusted variation that uses the same squared distance formulation as \(W_2^2\) , i.e., \(c(x_0,y) = \frac{1}{2}d(x_0, y)^2\). As a global bias measure, this approach uses the sum of individual biases, which depends on the number of samples. We again make this comparable to our approach by averaging individual biases.

Setup. To show that there are cases in which \(W_2^2\) is more expressive than model-based baselines in quantifying bias, we crafted two non-linearly-separable synthetic datasets, for which SVMs tend to exhibit diminishing learning power. According to Proposition 1 the approach of Kwegyir-Aggrey et al. (2021) would create the same ranking as the Kantorovich potentials, but the global bias measure would be coarser due to averaging all transportation costs instead of taking into account the OT map.

We create the datasets by uniformly sampling points from two concentric circles in a two-dimensional feature space, where each circle corresponds to a different sensitive attribute value. In the first dataset, the circles are perfectly concentric with radii 10 and 7.5. In the second case, we added points to the inner circle sampled from a Gaussian distribution centred in \((-4,4)\) and covariance matrix \(0.5\cdot I_2\). In these settings, the distance from SVM decision boundaries does not provide any information about the two distributions; but even a more versatile learning model, like a neural network, could leave some biases undetected.

Results. For the first synthetic dataset, we compare a) \(W_2^2\) , b) the measure proposed by Kwegyir-Aggrey et al. (2021), and the quadratic distance from the decision boundary of both a c) linear SVM and d) our variation with a 10-layer neural network classifier. For these approaches, we respectively obtained dataset bias scores of 1.98, 82.0, 21.05, and 1.22.

For the second synthetic dataset, we compute the distance from the decision boundary of a deep neural network with the Kantorovich potentials on the dataset’s outer circle. Figure 5 demonstrates that, even for a perfectly accurate decision boundary, the distance is evenly distributed across the outer circle and can therefore be a misleading indication of individual bias. On the other hand, Kantorovich potentials exhibit higher values on the side opposite to the Gaussian blob, and therefore correctly identify that points on that side differ more from the inner distribution than those near the blob. We also verify that the result for Kwegyir-Aggrey et al. (2021) is nearly-identical to the Kantorovich potentials, up to the goodness of our approximation.

Goal. We now investigate the ability of \(W_2^2\) to capture different degrees of data bias under the same feature extractors.

Setup. We experiment with two datasets: CelebA and Biased MNIST (Shrestha et al. 2022). We construct splits of CelebA with different degrees of selection bias, that is, we sample four sub-datasets ensuring that respectively 90%, 60%, 30%, and 10% of the samples have a positive value for a specified binary attribute (e.g, wearing ties, hats, or eyeglasses). \(W_2^2\) is computed on the 12 datasets that combine different attributes and selection biases, as well as for a -this time- uniformly sampled dataset of the same size of 9K images (in the latter, 7% of people wore ties, 7% wore glasses, and 4% wore hats). In Biased MNIST, we compare its training splits (50K images) for available bias levels (0.1, 0.5, 0.75, 0.9, 0.95, and 0.99) with the dataset’s test split (10K images), which is unbiased. For both datasets, we experiment with the ResNet18 feature extractor.

Results. Figure 6 shows that \(W_2^2\) monotonically reflects the degree of artificially injected bias in most cases, except for the smiling attribute.