Concordance as evidence in the Watson for Oncology decision-support system

During the past several years, IBM has been developing, among others, the Watson for Oncology platform (WFO), which is an artificial intelligence cognitive computing system (see IBM 2018). Such systems are more generally called medical decision-support systems. These systems are designed to support doctors in making decisions on which treatment option is best suited for their patients based on the latest medical evidence that is available (He et al. 2019; Char et al. 2018). The system relies on natural language processing and machine learning to provide treatment recommendations.

Machine learning platforms are not new and have been operating for years within search engines, such as Google, as well as financial markets and many other everyday services (Carlson 2018; Mittelstadt 2016; Sharon 2016; Buchanan 2015). Algorithmic decision-making is becoming increasingly omnipresent in our everyday lives (Zarsky 2015). According to its proponents, one of the benefits of algorithmic decision-making is that decisions become more objective (cf. Lepri et al. 2018; D’Agostino and Durante 2018). Machine learning-based medical decision-support systems can help to identify new research findings that physicians may not have time to find out for themselves among the rapidly expanding medical literature, as well as free up time for patients themselves. Machine learning platforms can also help to speed up the identification of treatment options, help to reduce errors, provide cost-efficiency, help provide standardized care, as well as support oncologists through uncertainty and risk. Some applications have even received FDA approval for diagnosis, such as the IDx-DR device, which is used to diagnose diabetic retinopathy (Healio 2018).

Within the medical field, machine learning platforms are expected to revolutionize the way in which we study and diagnose disease (Hinton 2018; Thrall et al. 2018; Syeda-Mahmood 2018). Domingos (2015, xi) has suggested that machine learning is different from other computational approaches in that machine learning platforms are able to “figure it out on their own, by making inferences from data.” The treatment options offered by the WFO platform have been developed between IBM and Oncologists at Memorial Sloan Kettering Cancer Center in New York, which is a world-leading center for cancer treatment. Numerous efforts are underway globally to test its concordance with tumor board decisions (cf. Xu et al. 2019; Kim et al. 2019a, b; Liu et al. 2018). Concordance refers to the level to which the treatment options offered by the platform agree with the treatment options that are chosen by the oncologists at a certain hospital. It is also the metric IBM uses to advertise its platform on its website (cf. Somashekhar 2017).

Our paper is based on observations of a pilot project conducted between IBM and the Department of Oncology at Rigshospitalet in 2017, which is Denmark’s leading hospital. We use the Danish experience and compare it with other pilot projects IBM has conducted around the world and that IBM itself highlights on its own Watson for Oncology webpage as success stories. We critically examine the practice of using concordance levels between tumor boards and the WFO decision-support system as a form of evidence, although we recognize the potential of the WFO platform to aid oncologists in their everyday work. We address a challenge related to the epistemic authority between oncologists on tumor boards and the Watson Oncology platform by arguing that the use of concordance levels as a form of evidence is problematic since it does not address the fundamental metric of outcome of specific treatment options for different patients. Furthermore, the WFO platform does not make visible the scoring criteria it uses to rank results from cancer clinical trials. Although WFO supports its recommendations with the literature that makes the treatment claims, we see this as masking a more fundamental problem regarding evidence: how does the WFO platform score or weight different clinical studies with regard to quality and local applicability of results? We, therefore, criticize the assumption that all data and studies that algorithms draw on are equal in quality or value (cf. Jaton 2017). This tension raises the substantive question of how to evaluate the quality of data algorithms draw evidence from and how should machine learning decision-support systems be evaluated in general.

Studies suggest that cancer trials contain significant levels of bias. Ledford (2019), for example, argues that ethnicity is a major concern for geneticists with regard to studying cancer across populations, but is often overlooked in Western clinical trials for cancer. A recent study on cancer drugs approved by the European Medicines Agency (EMA) argued that almost all studies between 2014 and 2016 were at risk of bias (Naci et al. 2019). Others have suggested that trials conducted by the pharmaceutical industry, as opposed to academic investigators alone, differ vastly with regard to their goals, whereby studies conducted by the pharmaceutical industry may not have the best interests of the patient as a concern, but rather focus on drug approval (Piccart et al. 2007). Furthermore, Devasenapathy et al. (2009) suggest that clinical trials conducted in developing countries may suffer from inadequate research infrastructure, as well as expertise to conduct robust studies. The value judgements regarding which studies ought to be weighted and for what populations are at the heart of the problem of concordance as a form of evidence in the WFO platform since WFO does not provide scoring information regarding which studies are deemed more reliable and how it accounts for bias. As such, inexperienced oncologists may lack the training and understanding of how WFO operates to critically examine the evidence that WFO provides.

As machine learning platforms begin to proliferate, we suggest that several questions emerge within the medical field. First, how should the recommendations made by WFO be evaluated in relation to physician expertise of treatment options on a global scale? Second, what can be said about the multiple levels of concordance between different tumor boards around the world in relation to Watson Oncology? Third, are there better alternatives for evaluating the effectiveness of machine learning platforms for cancer treatment other than concordance?

The pursuit of validation through global concordance levels is further complicated by discussions surrounding value-flexibility in the development of machine learning platforms (Hodgkin 2016). McDougall (2018), for example, has suggested that many artificial intelligence systems that are developed to support medical decision-making may mask fixed and covert value judgements, which erode the important role of shared decision-making in medical treatment (for a debate on this see also Di Nucci 2019; McDougall 2019). McDougall suggests that in the medical field, for example, patient perspectives are rarely considered in developing treatment option recommendations, which suggests a reintroduction of medical paternalism to patient treatment practices. Although platforms such as Watson may provide exciting new opportunities to help oncologists make decisions about possible treatment options at a global level, there is a risk that such platforms also introduce values and practices, which are not locally shared by physicians and patients alike. Following Mckinlay’s assertion that “If we expect computational systems to provide us with something as complex as explanation or evidence, it seems appropriate that any epistemic assumptions we bring to bear upon such systems be subject to careful and rigorous analysis” (Mckinlay 2017, 463). The use of concordance as a form of evidence represents a type of “algorithmic culture” (Striphas 2015), which we consider problematic since it lacks reliable and comparable reference or metric through which patient outcomes can be evaluated.

This paper is based on 14 semi-structured qualitative interviews conducted by AUTHOR in 2017 and 2018 in Denmark. The interviews were conducted with policy-makers, hospital administrators, IBM representatives, as well as doctors who were involved in piloting or developing a broad range of machine learning platforms in their respective fields. Written informed consent was sought from the interviewees before the interview. The interviews were transcribed and thematically analyzed for themes regarding the testing use and possible implementation of decision-support systems in medical care. In addition to interviews, we have collected and analyzed scientific publications, reports, media articles, as well as online documents published by IBM related to its oncology platform. Some of the countries where WFO has been piloted include India, China and South Korea. Our analysis has focused on the way in which the notion of concordance is used as a form of evidence to generate credentials through agreement between oncologists and the machine learn ing platform.

The WFO platform seeks to address an issue that some have perceived to be a problem within medical treatment. According to Somashekhar (2017) the system “processes structured and unstructured data from medical literature, treatment guidelines, medical records, imaging, lab and pathology reports, and the expertise of memorial Sloan Kettering experts to formulate therapeutic recommendations.” The algorithm then suggests what treatment options the physician should use in treating the patient. The programming for the algorithm accounts for numerous different data sources that can be weighted differently. These data sources include different bio-markers derived from the patient (sex, age, type of tumor, etc.), published findings in journals, national treatment standards, etc. The system then comes up with a score regarding which treatment options are best suited for the patient.

This process might mask, however, bias and statistical errors, which are built into the system, as well as the way in which results have been reported from other pilot studies. In a recent blog post Coeira (2018), for example, provides a critical analysis of a study published by Somashekhar et al. (2018)—a study in which oncologists in an Indian hospital report high concordance rates with Watson for Oncology. In the critical review, Coeira points out numerous limitations of the study. These limitations included how the WFO study drew on a sample population which excluded many patients, failing to describe well enough what data were entered into the system and what came out of it, why concordance rates were calculated twice allowing for oncologists to change their preferred treatment options, as well as not sufficiently recognizing that the outcome of the study reflects a process and not a clinical outcome. As Coeira notes:

“So far we have got to the point where the papers’ results are not that WFO had a 93% concordance with human experts, but rather that, when humans from a single institution read cancer cases from that institution, and extract data specifically in the way that WFO needs it, and also when a certain group of breast cancers are excluded, then concordance is 93%. That is quite a list of caveats already” Coeira 2018).

In addition to the statistical criticism laid out by Coeira, the WFO raises a number of other concerns regarding the usefulness of using concordance as a form of evidence. First, the platform has thus far been developed on US-based treatment guidelines (Kim et al. 2019a, b) and does not take into account differences in incidence or risk of certain kinds of cancers in different populations (Liu et al. 2018). Yet, different countries, such as Denmark, base their treatments on their own national guidelines that have been established by local expert panels. These nationally based expert panels also discuss, debate, and interpret the latest findings in international studies. Consequently, expert panels form value judgements on different studies based on their perceptions regarding the quality of the evidence and rigor of the methods used. Using the Sloan Kettering treatment options, the IBM programmers and developers are creating a de facto ‘ground truth’ (Jaton 2017) against which other treatment options are measured (Kim et al. 2019a, b). This ‘ground truth’ is a type of bias that is introduced into the program (Gianfrancesco et al. 2018), since it is not compared or validated against other treatment standards. In and of itself, this approach may not be problematic, so long as the treatment standards used in the US can be proven the best treatment standards in the world. This variance between guidelines reflects value judgements that medical experts make in each local setting in relation to the preferred treatment options and evidence, which they see as trustworthy and relevant. IBM could address this bias using population-based studies, as well as long-term prospective studies, for example, or explaining the logic behind their scoring system. The Nordic countries, for example, are some of the only countries in the world, which maintain cancer registries for the whole population (Tupasela et al. 2020). These data can be linked to personal health and treatment data, which provide a far more robust data set, albeit limited in terms of its applicability to other populations (Pukkala et al. 2018, 2009).

Second, the global market for cancer treatments is not unitary. Not all medications and treatments are available or have been approved in a uniform manner in different markets. Therefore, there may be treatment options in the US that are not available in different parts of the world. This may also work the other way, there may be treatment options available elsewhere that have not been approved in the US yet.

Third, in reading the pilot studies that have been run in different countries, it becomes evident that IBM regularly releases new versions of its software. Liu et al. (2018), for example, report that they have used version 17.1 of the WFO software; while, Kim et al. (2019a, b) have tested version 18.9. In trying to compare these results, it is difficult to ascertain how IBM has changed its program between different versions. The WFO website does not provide information about version history, which essentially generates a black box regarding the scoring system. Many of the published studies do not even report the version number, making it even more difficult to compare different iterations of the WFO algorithm.

Fourth, in reading the published reports regarding WFO, one must note how different oncology teams use different sets of data points, which they entered into WFO. The Danish oncologists reported that they provided 17 data points, whereas Kim et al. (2019a, b) provided 13. Although not specifically a major problem, we see this as another situation in which opacity is introduced into the scoring system.

The opportunities and benefits that Watson may offer lie in its ability to cover a great deal of literature and offer a service for countries where there may not be enough resources for local oncology panels to evaluate the latest findings. Its weakness, however, lies in the black boxing of epistemic value judgements regarding which studies can be considered of good quality and how they are weighted in the algorithmic decision process.

The use of concordance levels as the only form of evidence in marketing is a problematic metric for validity of treatment options. The challenge regarding the WFO platform is that oncologists do not know what value judgements IBM codes into their algorithm regarding the scoring of different cancer studies. We have shown that clinical trials for cancer drugs contain many biases, which IBM does not make transparent in the platform. IBM could easily address this problem by making the scoring and ranking criteria more transparent. This would align the WFO platform with recent policy calls for transparency, diversity, non-discrimination as well as fairness in developing AI (European Union 2019). Still, it should be nonetheless emphasized that the opacity issue and the algorithmic bias issue are related but independent from each other, in such a way that more transparency does not necessarily guarantee less bias.

Although Watson is able to perform many important tasks relating to the collection and screening of a large amount of medical literature regarding cancer treatment options, it is still unable to evaluate the role of value judgements made by different oncology guidelines and clinical trials regarding the best treatment options. These value judgements include what studies are considered as valid in evaluating treatment options, which treatment options are made available through insurance and national health insurance schemes, as well as whether patient perspectives are included in selecting preferred treatment options. In revisiting our original questions, the use of concordance as a form of evidence has the following problems:

First, the Watson for Oncology platform is unable to provide criteria or explanation for the differences in expert judgment in different contexts. Consequently, it is difficult to evaluate why specific treatment options are ranked higher than others. Second, low concordance rates do not necessarily mean that patients in a given hospital are receiving poor treatment. All that can be said about it are that protocol choices differ between Watson and a particular set of oncologists. As we show with our scenarios, differences in concordance can have multiple explanations. Third, a better alternative would be to draw evidence from high-quality and validated outcomes. Such studies would include population studies, as well as prospective cohort studies. Although, Watson does this partially by considering the outcomes of studies, the platform needs to take more into account differences in value judgements and quality criteria that different oncologists and national boards for treatment options take into consideration. As we note, the quality and reliability of cancer clinical trials vary greatly, with many studies containing biases (Copur 2019). Despite providing evidence through literature, WFO does not reveal its scoring system and how it accounts for possible bias in studies. Such functionality could be added to Watson and provide interesting new insight for oncologists around the world regarding differences in treatment options that oncologists choose and for what reason. As IBM is able to pilot and test its platform around the world in different contexts, it is also able to amass a large amount of data regarding different iterations of its algorithm; that is to say different versions that may prioritize one form of evidence over another. Making this information available would make evaluation of the platform easier and more transparent. Fourth, our example highlights how all data sets and studies are not of equal value. Instead, researchers, as well as oncologists, use numerous different quality criteria to evaluate the validity of oncology studies. With the WFO platform, oncologists do not know what the scoring criteria are regarding which studies are more relevant to others.

Following McDougall (2018), considerations for bias are not accounted for enough in developing machine learning platforms for medical decision-support systems (see also Ledford 2019; Devasenapathy et al. 2009). Many of the people listed in the early findings’ reports have also disclosed funding and ownership relations to IBM. For such platforms to gain broader acceptance and validity, the platform needs to provide more transparency regarding the scoring criteria it uses, as well as the changes that different versions bring with it. Programmers and developers also need to consider value-flexibility in developing systems, which prioritize treatment options.

The platform does provide some interesting new opportunities, however. Through its piloting projects around the world, IBM could develop a repository of data regarding the different treatment guidelines and applications used in different countries, as well as the values that lie behind such treatment guidelines. As such, IBM is in a unique position to gather data that could hypothetically be used later to compare different guidelines. To evaluate and develop evidence as to the best treatment guidelines, however, IBM would need to trace the outcomes of the patients over many years to see which guidelines generated the best survival and re-occurrence outcomes among the patient populations. It is imperative that the suggestions made by Watson can be critically examined in light of competing treatment guidelines and data quality to ensure that the suggestions that it is making are the best possible options available globally, not just in the US and by a small set of oncologists.

Finally, we would like to conclude by addressing a broader problem which the Watson for Oncology platform appears to point towards. Many of the machine learning platforms that are developed to support medical decision-making seem to reproduce expert knowledge, which is already known. This, one could argue, is not machine learning, but rather a form of automation (Wajcman 2017). This raises the question of which set of values ought to be relied on as guiding decision-making? This suggests that decision-support systems within medicine might always have to rely on locally rooted value judgements, regardless of whether IBM is able to offer high concordance rates between the oncologists and its platform. We suggest that further research on how value judgements are made in these different contexts will help to provide valuable insight into the quality of care that patients receive in different regions of the world.