AI-based healthcare: a new dawn or apartheid revisited?

Wong Chut King, a Chinese labourer, was found dead in the basement of a hotel in San Francisco’s Chinatown in March 1900 (Risse 2012). After an ‘egg-shaped blackish lump’ was discovered in his groin, King would prove to be the first confirmed victim of San Francisco’s Bubonic Plague outbreak at the turn of the twentieth century (Risse 2012: 76). Unbeknownst to most at the time, King’s death would instigate a public health and political crisis, with infighting tearing the city apart and an intensification of Sinophobia (Skubik 2002).

At the end of the nineteenth century, San Francisco was home to California’s largest Chinese population and traditional Gold Rush arguments remained: Chinatown was filthy, and its bad smells bred disease (Risse 2012). Germ theory was in its infancy and the plague was considered an “Oriental disease” (Morton Todd 1909). This scapegoating was not new; Malaria, Smallpox and Leprosy were all placed at their door (Trauner 1978). Racial prejudices justified such claims; the phrase ‘yellow peril’, coined at the time, demonstrated the supposed threat that Asian immigrants represented for European-American civilisation (Risse 2012). In the 1870s, these racist arguments broadened as Chinatown grew. It was argued that their cheap labour undermined wage rates, their poor sanitation endangered the nation and they were ‘inferior in organic structure, in vital force, and in the constitutional conditions of full development’ (Trauner 1978: 70–72). The result was the Chinese Exclusion Act of 1882, which dictated policy for the next sixty years. Popular racism enabled the maltreatment of the Chinese and explains how politicians could disregard the plague in Chinatown.

Multiple quarantines were enacted over the period, along with a proposed vaccination programme (Skubik 2002). The vaccine used dead bacteria but rumours that it was poisonous left Chinatown on the verge of riot (Skubik 2002). Meanwhile, California’s Governor, along with San Francisco’s business community, chose to deny the existence of the plague and suppress facts to protect their interests (Skubik 2002). Many anti-Chinese groups considered using the plague as an excuse to destroy Chinatown (Risse 2012). As one scurrilous commentator put it: “The only way to get rid of that menace [plague] is to eradicate Chinatown from the city… clear the foul spot from San Francisco and give the debris to the flames” (The San Francisco Call 1900). This was arguably only ruled out due to fears over disrupting trading links with China (Risse 2012).

In 1900, the disease was mainly spreading among the rodent population (Skubik 2002). Real action was only taken in 1903, once the disease had become highly contagious and fatal and with a new Governor in office (Skubik 2002). The epidemic was declared over in summer 1904 after 122 deaths, most, but not all, of whom were Chinese (Skubik 2002). The lack of genuine medical help offered, compared to the speed with which Chinatown was blamed, is symbolic of the period’s prejudices. Those dealing with the epidemic were driven by their own interests; only when the impact was felt outside Chinatown, was action taken.

So, how is this tale from history relevant to the rapid growth of Artificial Intelligence (AI) today? The connection is bias. In San Francisco in 1900, prejudice was present in multiple forms, demonstrated by numerous characters and its impact had life and death consequences for those on the receiving end. Also, this case study offers the first example, of many for this piece, of how such prejudice can trigger a negative feedback loop. That is, as the plague worsened, Sinophobia intensified, further limiting the response and allowing the loop to repeat. Unfortunately, over 100 years on, similar biases and feedback loops are emerging, only this time our technological creations have the potential to be biased. On various scales and in areas from recruitment to policing, bias could emerge and if unchecked could guide decisions worldwide.

This argument for the existence of potential ‘bias’ in AI systems is not without foundation. Humanity’s capacity for prejudice in society, our thinking or actions is evident throughout history, whether that bias be unconscious or deliberate. O’Neil (2016: 203–204) takes this further, suggesting that ‘injustice, whether based in greed or prejudice [or both], has been with us forever’. O’Neil acknowledges that human decision making can often be flawed but can also evolve, unlike automated systems which ‘stay stuck in time until engineers dive in to change them’ (O’Neil 2016: 203–204). This inference that humanity’s biases can sometimes be replicated in our technological creations also appears in Whitaker, Colombo and Rand’s (2018: 10) findings: ‘the distributed collective intelligence of machines is also a social endeavour, and it is potentially susceptible to prejudicial phenomena as seen in the human population’. Sismondo (2010: 9) agrees regarding this social side of technology, arguing that ‘people act in the context of available technology’.

So, given this idea that technology influences social conduct, which in turn determines the technology humanity uses; it seems unlikely that even the most sophisticated AI-based technologies will be able to avoid the ‘forever’ phenomenon that is bias. Yet, as Plant (1997: 46) states, resources that ‘were once restricted to those with the right face, accent, race, sex’ are now available to all. With that in mind, this paper works to draw attention to various case studies where the potential for bias has become a reality and has served to subvert AI’s beneficial potential.

This paper chooses to delve deeper into this wider suggestion of AI’s ability to embody various biases. To do this, it will focus on an area of the most pressing importance should such biases emerge: healthcare. Specifically, a toy model imagining an AI in control of a vaccination programme in the event of an epidemic will form the basis of this article. From this basis, the paper has an overarching narrative which is split into three sections (called stories) for structural reasons. While the scenarios discussed are strongly intertwined, they are tackled separately here to demonstrate the multiplicity of the dangers. The first proposes that this fear of the algorithm is an overreaction. It will proffer that the outcomes can be the same, no matter if the algorithm is altered, hence, there is nothing to worry about? In contrast, the second story observes how seemingly insignificant changes to the code and alterations to the datasets used can hugely alter the outcomes for individuals. Finally, the third story alters the dataset again to demonstrate the consequences when supposedly ‘clear’ instructions are written into AI systems and yet when the programme is allowed to run freely, deviation from these instructions can occur. Then, a discussion will draw together all these ideas from the toy model and reference real-life situations, including referring back to the example of San Francisco’s Chinatown, to further enhance this paper’s arguments.

The toy stochastic network modelling approach used here is sufficient to highlight the issues raised within the scope of the paper, where the focus is on the interactions and feedbacks between data, different actions and potential outcomes. The target of this paper is not to offer novel AI tools for the healthcare, but rather to draw attention to and stir a debate regarding the potential outcomes, should the feedbacks discussed here become a reality in AI systems. It is appreciated that in the ‘AI in Healthcare’ context in reality, there are many more complex models already in the field. A deeper and more comprehensive investigation into the intricate and sensitive topic of AI in healthcare scenarios in the ‘real-world’ would draw on a wide range of models. For example, Brailsford et al. (2009) analysed frequency of use, domains of application and the level of implementation of a wide range of research modelling approaches in the healthcare remit. In fact, they note that ‘simulation methods are prominent in planning and system/resource utilisation’ which enhances the legitimacy of this paper’s choice of modelling technique to effectively meet its target (Brailsford et al. 2009: 139). Brailsford et al. (2009) demonstrate the sheer range of potential methods to be used but regarding the earlier suggestion of more complex alternative models, Weng et al. (2017) offer an excellent overview of some of the machine-learning models being used to help improve cardiovascular risk prediction. They looked at four different machine learning algorithms compared to an established algorithm that predicts future risk based on well-established factors such as cholesterol or smoking (Weng et al. 2017). On the other hand, the machine learning approaches can exploit ‘big data’ and the neural network model performed the best in their study (Weng et al. 2017). Therefore, the two mentioned studies demonstrate the wider ranging and more complex nature of models currently at work in the ‘AI in healthcare’ field. However, they also evidence that for the scope of this paper, the simple stochastic network models used are ideal as suggestions of what could happen, and the use of features—even on this paper’s smaller scale—is clearly a well-founded technique in this field.

Skubik’s work on the bubonic plague in San Francisco’s Chinatown, in particular, highlights that it was assumed at the time that only Chinese people required vaccination for this ‘oriental disease’ (Skubik 2002). Hence, it can be assumed that assigning an experimental, and potentially dangerous, medicine was based on prejudice. This work will take this suggestion of bias guiding healthcare decisions further, by imagining that issuing vaccinations fell to an AI system. This paper will examine how that might be decided and look into some of the implications of that automation.

Age is often used to determine who receives a vaccine. For example, over sixty-fives are one of the groups specified for the free flu vaccine in the UK (NHS UK 2019).¹ Similarly, Gupta et al. (2014) state that ‘top-ranked influential spreaders need to be identified for targeted immunization’ and that traditionally ‘nodes with high centrality are considered to be the influential nodes’. Although they admit that many measures can be used for this ‘centrality’ parameter, they note that ‘degree centrality’, where each node is given an ‘importance score based simply on the number of links held by each node’ (Disney 2020), is a popular choice (Gupta et al. 2014). More generally, in any network or population of people, every individual has a vast number of different attributes, including age and weight but also race and religion, or any number of medical conditions. These attributes could be linked to susceptibility and, like age for the flu vaccine, be used to classify or, in AI parlance, cluster individuals for the purposes of determining a vaccination policy Fig. 1. Although, populations are not an unstructured, homogeneous wholes in which disease spreads out. They are heterogenous, structured by social connections, be they physical or virtual via social media. For this toy model, two types of parameter will be used as metaphors for all the various attributes of individuals in a population. The first will be referred to as health parameters, which for this toy model will be summarised by age and weight. The second are network parameters, which will be simplified to the edge degree of each node in the population (number of connections an individual has to others in the population).

In large, high-dimensional networks of populations with multiple and varying datapoints describing individual features, determining clusters is beyond human action but well within the scope of powerful AI systems. Such AI-based decision systems are already in development in the healthcare context, particularly in the context of Acute Kidney Injury (Argyropoulos et al. 2019). Google DeepMind’s initially developed a machine capable of playing the ancient Chinese game of Go (DeepMind 2019). Now, by use of a neural network, they are connecting around 9000 data-points and historic incidents of kidney injury for a dataset of American veterans in an attempt to predict acute kidney injury before it happens (Powles 2019). Admittedly, at present, the accuracy is only around fifty-five percent, but the example proves the extent to which social connections and features are constantly being captured, processed and used to make decisions (Powles 2019). If you doubt this statement, no one knew when their data had been sold to Cambridge Analytica and even now the scandal has been unearthed, it is unlikely that you know whether you were a victim (Amer and Noujaim 2019). So, how might this clustering and AI-based vaccination lead to biases in future healthcare?

The bias may have been written into all the simulations in this paper due to the fact that decisions over vaccinations are based on using weight as a feature in the dataset. The inputting of a biased dataset, rings comparable, if intrinsically different, alarm bells to those of San Francisco in 1900. Those health officials may have used health records to identify who ‘needed’ the vaccine. The programmer of the toy model above holds a similar role and uses records of age, weight and edge degree to determine vaccine receipt. In both these cases, the prejudice is written in, be it via a prejudiced dataset or prejudiced thoughts inherent to the time period. However, where this toy model deviates from the San Francisco case study is where bias emerges separate to this inherent feature. The further biases that emerged in both story two, against the individual, and in story three, against particularly the youngest cluster, could not have been predicted. In the simulations in story three, bias emerged on top of what was dictated, simply by the algorithm’s requirement to follow the initial instructions to the end, despite it no longer being the best option for the population. The feedback loop that emerged between clustering, exclusion and vaccination proved impossible in story three for the algorithm to escape from. Importantly, even if the initial prejudice that was written in here was completely unintended, the emergent bias would remain an outcome from the AI system. The idea of intended or unintended (emergent) bias is not distinguishable to an AI.

As previously stated, it is unlikely that such a simplistic model would be utilised in the vaccination context. However, similar such models are already in action; as an example, consider crime prediction software in America, specifically PredPol. This system looks at crime in one area and uses historical patterns to predict when and where it might occur next (O’Neill 2016). On the surface, this seems highly effective; the model is blind to race and ethnicity since it simply targets geography (O’Neill 2016). Unfortunately, this desire can be evaded, as Nixon’s notorious “dog-whistle” words in his Southern Strategy apparently showed (D’Souza 2018). As in this case, geography can be an accurate proxy for race because impoverished neighbourhoods are primarily occupied by ethnic minorities (O’Neill 2016). This becomes more critical when nuisance crimes are included in the model, which are prevalent in impoverished neighbourhoods (O’Neill 2016). As with the idealised vaccination model, a negative feedback loop might well be created: policing creates new data, justifying more policing (O’Neill 2016). Evidently, there is an important difference between the toy vaccination model and Predpol; the dataset Predpol utilised was not intentionally biased, but both examples prove that care needs to be taken when programming AI systems. If not, there is a real danger of unconscious biases being fed into them or emerging. Importantly, this is not a definite outcome; prejudice does not have to be inherent as in San Francisco’s Chinatown. As Leonhard (2016: 75) accurately states, the ‘technology is neither good nor bad; it simply is. We must—now and here—decide and agree which exact use is evil or not’.

Predpol is not the only example of unintentional biases guiding the decisions of AI systems. Examples of this phenomenon are commonplace. Even the big technology giants are not immune. In 2018, it came to light that Amazon had to scrap its AI recruiting tool because it was biased against women (Dastin 2018). Their model was trained on applications submitted to the company over the past ten years, which reflected the male dominance of the industry (Dastin 2018). The AI system was trained on the data of the past resumés, whilst the new resumés became the so-called test data. It was from the training data that the AI picked up the bias against women, it then transferred this knowledge onto the test data. Therefore, the system learnt that male candidates were preferable and penalised the resumés that included words such as ‘women’s’ and downgraded graduates of two all-female colleges (Dastin 2018). This obviously had not been Amazon’s intention but their lack of consideration of the potential consequences from using the biased dataset, resulted in a biased recruitment tool. By simply changing the dataset used, the outcome could have been very different for the women who were rejected purely based on their gender, since the system would not have learnt to exclude them. However, as the toy vaccination model proved, changing the dataset may well improve the outcomes for one individual or group but equally likely, it could cause other individuals or groups to be even worse off. If Amazon are having such difficulties, then it is unsurprising that others are too, with stories of ‘biased AI’ frequently in the news.

One upsetting example, that combines all the elements of this article; the prejudiced backdrop of decisions, the impact on individuals and the problems in clustering people into defined groups, is apartheid in South Africa. In the 1950s, those in power ‘sought to divide the population into four basic groups: Europeans, Asiatics, Persons of mixed race or [black people], and “natives” or “pure-blooded individuals of the Bantu race”’ (Bowker and Star 2000: 197). However, an entire population would never conform to such ‘simple’ groupings, making the process completely inconsistent (Bowker and Star 2000). Despite the problems, a person’s racial classification could be challenged at any time, even if you merely associated with someone of the ‘wrong group’ (Bowker and Star 2000). Also, these cluster groups were not just administrative, they determined where a person could live and work (Bowker and Star 2000). In effect this process ‘separated families, disrupted biographies, and damaged individuals beyond repair’ (Bowker and Star 2000: 218). While this may be an extreme example of the consequences of classification, it does demonstrate the dangers of trying to make individuals fit into groups. As Bowker and Star (2000: 224–225) state, ‘there can be tremendous torque of individual biographies’. The toy model in this paper demonstrates these damaging consequences for individuals from over-simplifying a population into a set number of groups. Particularly, its outcomes in stories two and three show how algorithms will not remove this potential for damaging ‘torques’ should care not be taken when determining datasets and making ‘innocent changes’.

Unfortunately, these problems with classifying individuals are already arising in the technological world and they are relatively unchecked. Bearing in mind these historic problems with classification, it seems unlikely that the introduction of AI will immediately fix the problems. Once again, choices regarding datasets are wreaking havoc in multiple areas. For example, Buolamwini (2016) first encountered exclusion problems with generic facial recognition software with social robots. The robot could not classify her face as such unless she wore a white mask (Buolamwini 2016). Such facial recognition software are often trained on similar datasets, but if these are not sufficiently diverse, then any face that deviates from what the AI has learnt to be a face will not be detected as such (Buolamwini 2016).

Not being classified at all by this kind of software is not the only problem. In 2015 Google’s Photos service labelled a black software developer and his friend as ‘gorillas’ and since then, their main solution has been deleting gorilla from the service’s options (Simonite 2018). These problems in facial recognition are awful for the people affected and embarrassing for the companies involved. However, the consequences become scarier still when considering some of the uses for these technologies. For example, Google are branching out into autonomous vehicles, which use image recognition software to classify the world around them. The potential consequences of that hypothetical scenario are best left to your imagination.

Despite this paper’s simulations being idealised, the way it maps onto real-life examples of negative feedback loops is important to change how we manage these technologies. The model’s consequences may be exaggerated but the consequences detailed in the case studies above were very real. As already stated, it is extremely unlikely that such a simplistic modelling technique would be used in reality in healthcare. However, it has been used here to highlight a feedback loop between datasets and outcomes which are not idealised. Such feedback loops do exist and are already having negative impacts on certain groups’ lives as seen in the examples offered. All of the points raised by the toy model; feedback loops, dataset choices and small changes in code are all tied together and impacting on each other to have sometimes disastrous consequences. Clearly, people do not fit neatly into groups and this toy model has shown that by simply changing the dataset and seeding, the outcome for an individual can be completely different. The apparent human need to classify things, and in this case, people, is visible throughout our history (Bowker and Star 2000). Equally visible, however, are the problems that come with this desire to sort people and there will always be anomalies. If this is forgotten, then the true impact of oversimplifying these decisions over datasets and variables, such as the seed, cannot be truly appreciated and may not be prioritised. As AI branches out into more areas of life the focus needs to be on protecting those members of society who could potentially suffer at the hands of a poorly considered AI system. It is vital we take heed from the examples that have gone before and consider this ‘toy model’s’ worst case scenario outlook and work to avoid it.

There is a consensus that the original intentions of AI are generally benign, making the term “bias AI” a little misleading. No programme will ever start out biased, they are like children, they have to be taught and they learn what their parents or, in this case, programmers and data tell them. Buolamwini (2016) calls this the ‘Coded Gaze’, when the views that are embedded into systems are propagated by those who have the power to code the systems. ‘Whoever codes the system embeds her [their] views. Limited views create limited systems’ (Buolamwini 2016). As demonstrated in the many examples in this article, this embedding could be intentional or unconscious, but the consequences are the same. These consequences not only impact groups of our society as a whole but also can have nuanced impacts on individuals within the population.

There is much more scope for further work in this area that can build on the simple stochastic network models used in this paper. Alternative approaches/more complex examples have been noted in the introduction. These could be used to develop the many ways in which this paper’s work, models and conclusions could be extended and taken more into ‘reality’. Models in this paper have focussed on the impact AI could have on healthcare today, one of the most current and certainly topical dilemmas in the field. A report from the NHS in the UK recognises the problems with data as discussed in this research but equally, they recognise the opportunity that AI presents for improvement and therefore to save lives (Harwich and Laycock 2018). But to blame AI and present it as ‘bad’ is to paint too simple a picture, the bias that is appearing in systems in all areas is often inherited from the data or even the programmer, whether it is deliberate or unconscious is another question. Equally, learning from the past will prevent making the same mistakes again. Unless care is taken, prejudices such as those that governed both San Francisco’s Chinatown in 1900 and South Africa under apartheid will continue to emerge, only now they will emerge through AI systems. Stopping and considering the datasets we are using, what the consequences of a seemingly harmless decision could be, will enable us to get the best out of this bright, new technology.