Cost-sensitive learning for imbalanced medical data: a review

Class imbalance is a prevalent phenomenon observed in many real-world datasets, where the distribution of instances across different classes is significantly skewed. While the technical definition of class imbalance encompasses any dataset with unequal class distributions, the term typically refers to datasets that exhibit substantial and sometimes extreme imbalances (He and Garcia 2009). This imbalance has garnered considerable attention from researchers and practitioners due to its widespread occurrence in various classification problems, such as anomaly detection (Zhou et al. 2021), face recognition (Huang et al. 2020), medical diagnosis (Mazurowski et al. 2008), and more. In such scenarios, the minority class, often referred to as the positive class, represents the concept of interest and is characterised by its low frequency. On the other hand, the majority class, also known as the negative class, constitutes the class with higher representation. The scarcity of instances belonging to the minority class can stem from their inherent exceptional or rare nature, or it may result from the high cost of acquiring data for these particular examples (López et al. 2013). Consequently, accurately identifying and classifying instances from the minority class becomes crucial, as it often carries significant implications in practical applications.

While class imbalance is commonly associated with binary classification tasks, it is crucial to note that imbalanced distributions can extend to multi-class (Wang and Yao 2012) and multi-label (Lankireddy et al. 2022) settings. Within such contexts, the presence of multiple minority classes introduces additional complexities, amplifying the challenge at hand (López et al. 2013; Lankireddy et al. 2022). Moreover, it is essential to acknowledge that class imbalance is not limited to classification tasks alone but can also manifest in segmentation tasks, specifically in medical imaging (Rezaei et al. 2019). In such scenarios, a significant disparity emerges between the number of pixels or voxels representing regions of interest and those corresponding to the background class or other classes (Nasalwai et al. 2021).

The degree of class imbalance in a dataset can be quantified using the Imbalance Ratio (IR), which is calculated as the ratio of the number of examples in the majority class \(N_{maj}\) to the number of examples in the minority class \(N_{min}\):The IR metric quantifies the severity of class imbalance, providing insights into dataset composition and aiding in devising effective strategies to address imbalanced datasets. An annotation such as 1:100 can be used to represent an IR of 100, indicating that the majority class is approximately 100 times more prevalent than the minority class.

Mitigating class imbalance is crucial due to several key reasons. Primarily, biased learning poses a significant challenge. Standard learning algorithms are designed to work optimally on balanced datasets, where the numbers of instances in each class are roughly equal. When applied to imbalanced datasets, these algorithms tend to be biased towards the majority class, leading to suboptimal classification models and frequent misclassification of minority class instances (Fernández et al. 2018). Consequently, the minority class instances are frequently misclassified, reducing overall performance. Moreover, the significance of the minority class in various real-world scenarios cannot be overstated. Misclassifying samples from the minority class can have severe ramifications, such as missed opportunities, erroneous diagnoses, or potential risks. Furthermore, the natural occurrence of imbalanced datasets due to the inherent characteristics of the problem domain adds another layer of complexity. For instance, rare medical conditions often have limited data availability compared to more prevalent cases. To address these challenges, it is imperative to develop effective strategies that specifically target class imbalance. By doing so, classification performance can be significantly improved, ensuring accurate identification of minority class instances and enabling the extraction of valuable insights from the available imbalanced datasets.

Many strategies have been proposed in the literature to address the class imbalance challenge. These strategies can be classified into four groups (Fernández et al. 2018; Haixiang et al. 2017):For a closer examination of these strategies, Table 1 provides a breakdown of their strengths and weaknesses, which have been carefully curated from prior reviews and discussions on addressing class imbalance. These strategies enhance model performance and effectively tackle class imbalance in various contexts. Data-level strategies offer a promising starting point due to their ease of implementation, straightforwardness, flexibility, and versatility, as they remain independent of the underlying algorithm. However, they are not without their trade-offs. Oversampling techniques risk overfitting and often demand extended training times, while undersampling methods can potentially discard informative samples from the majority class. In contrast, algorithm-level strategies introduce targeted solutions to class imbalance without altering the underlying data distribution. They exhibit an advantage in that they are less likely to affect training time, yet they demand an in-depth understanding of the algorithm in use and are inherently algorithm-specific, potentially compromising their flexibility and ease of implementation. CSL stands out for its computational efficiency and data distribution preservation while addressing class imbalance. Nonetheless, it presents challenges in setting appropriate misclassification costs, often requiring careful optimisation and facing potential overfitting to the minority class during cost tuning. Note that a detailed exposition of the strengths and weaknesses associated with CSL is provided in Subsection 6.1. Ensemble-based strategies, leveraging multiple classifiers, offer superior predictive performance and improved resilience to noise, thereby enhancing generalizability. However, they come with the trade-off of reduced model interpretability and increased computational complexity. Considering these diverse strategies, researchers and practitioners can select the most suitable solution based on the specific problem’s requirements and constraints, as each strategy offers a unique blend of advantages and considerations to guide their decision-making process.

This section introduces the core concepts of CSL and presents an illustrative example of cost-sensitive logistic regression applied to a cervical cancer diagnosis dataset.

This study aims to provide an overview and a structured understanding of the existing literature on using CSL for imbalanced medical data. To this end, nine RQs were identified and are presented along with their rationales in Table 6.

The search is conducted in five digital libraries: PubMed, ScienceDirect, IEEE Xplore, SpringerLink, and Google Scholar from January 2010 until December 2022. These libraries were chosen based on their extensive coverage of peer-reviewed publications in the fields of medicine and health sciences, as well as computer science and engineering.

The search string was formulated based on the principal terms from the RQs and the PICO (Population, Intervention, Comparison, and Outcomes) framework (Kitchenham and Charters 2007). Note that the third and fourth letters of PICO were not included in the search string formulation since neither empirical comparison nor measurable outcomes were considered in this study. Additionally, the search string was expanded to include alternative spellings and synonyms of the derived terms to ensure a comprehensive search.

The main search terms were initially linked with their substitutes using the Boolean operator “OR” and were joined using “AND” afterwards. Table 7 exhibits the complete search string, where the “scope” column demonstrates the main terms and the “search terms” column displays the related keywords.

The Inclusion Criteria (IC) and Exclusion Criteria (EC) utilised to identify the relevant papers are presented in Table 8. The systematic selection process involved a multi-tiered approach, as outlined below: One author conducted the initial examination of the papers, and the remaining authors evaluated the final selection. Any disagreements during this process were resolved through constructive discussions in meetings, ultimately leading to a consensus on the final set of included studies.

While Quality Assessment (QA) remains optional in the guidelines followed, it is still recommended (Petersen et al. 2015) as it can help minimise the risk of bias, guide the interpretation of findings, and determine the strength of inferences (Kitchenham and Charters 2007), thereby improving the overall validity of the study.

Each paper was evaluated by two authors based on the checklist in Table 9 to ensure that the selected studies are of sufficient quality and provide reliable and valid evidence to address the RQs. The checklist comprised six QA criteria, but only QA5 and QA6 were deemed relevant for reviews and theoretical papers. The decision to exclude the first four criteria was based on their applicability to empirical studies only, and their use for reviews and theoretical papers may have compromised their eligibility for inclusion in this study.

In this phase, a data extraction form was developed to retrieve relevant information from the selected papers, addressing the RQs in Table 6. The structure of the form is detailed in Table 10. One author conducted the task meticulously, while the other two authors rigorously reviewed the extracted data to ensure its accuracy and objectivity.

Data synthesis aims to summarise and synthesise the extracted data pertaining to each RQ. The vote-counting method is utilised to aid in result interpretation, followed by a narrative synthesis to comprehensively report and discuss the outcomes for each question. Additionally, visual representations in the form of charts and tables are incorporated to enhance the clarity and presentation of the findings.

Figure 1 displays the number of articles at each stage of the selection process. Initially, 49325 candidate papers were identified, from which 49124 studies were discarded according to the IC and EC. 28 studies that did not fulfil the QA criteria were later excluded. Eventually, 173 papers were retained to answer the RQs. The list of selected papers and their extracted data can be obtained through an email request to the authors.

Figure 2 shows the number of selected studies per publication channel from January 2010 to December 2022. Three main channels were identified: journals, conferences, and workshops. Out of the 173 selected studies, the majority, precisely 69.9% (121 papers), were published in journals, 27.2% (47 papers) were published in conference proceedings, and only 2.9% (five papers) were published in workshops. Table 11 outlines the publication sources that have published more than two papers. The findings indicate that Computer Methods and Programs in Biomedicine was the most commonly targeted journal venue, while the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) emerged as the most frequently occurring source for conference papers. Chronologically speaking, conference papers were the dominant publication type in 2012 and 2013. However, the trend shifted in 2014 as the journal publication frequency surpassed that of conference papers in subsequent years. A key observation is that the gap between the two types of publications became increasingly pronounced from 2020 onwards. The analysis further revealed a growing trend of publications, particularly since 2020, when the count peaked significantly. Notably, no study was published in 2010, and only one workshop paper was published in 2011.

The dearth of published papers in 2010–2011 and the dominance of conference papers until 2013 suggest that CSL research in the medical field was in its early stages. However, as the field progressed, researchers began to prioritise top-tier journals due to their strict review processes and higher publication standards, resulting in more rigorous research. This shift towards journal publications started in 2014 when the number of journal articles surpassed conference papers and continued to widen in subsequent years. This trend indicates a maturing field and researchers increasingly meeting the demanding standards of high-quality journals.

The growing interest and abundance of publications on CSL can be attributed to several key factors. Firstly, the development of high-throughput technologies has resulted in massive amounts of medical data (Johnson et al. 2018), including clinical data, electronic health records, and data from wearable devices. These advancements in data collection have created an urgent need for novel methods to analyse and leverage this data for improved medical outcomes. Secondly, the inherent imbalanced nature of this collected data poses a critical challenge that impacts the accuracy and reliability of ML models in medical applications. Thirdly, the significant advances in CSL algorithms (Khan et al. 2018) and their success in other fields (Sahin et al. 2013) have encouraged researchers to apply these techniques in the medical domain, where they are much needed. Additionally, the advances in Deep Learning (DL) techniques have been a significant catalyst for progress in medical data analysis (Esteva et al. 2019). Finally, the increasing availability of public datasets and tools for analysing medical data has facilitated the dissemination and replication of research findings. As a result, the research community has become more aware of the importance of addressing the imbalance problem, leading to a surge in publications on this topic, particularly in recent years.

Besides, the findings revealed diverse publication sources covering various disciplines such as medicine, medical informatics, computer science, and artificial intelligence. This diversity reflects the interdisciplinary nature of the research topic, requiring a multi-faceted approach that draws on expertise from different fields.

After scrutinising the selected studies, four distinct research types were identified: Evaluation Research (ER), Validation Research (VR), Solution Proposal (SP), and reviews. No other research types were observed. Of the studied literature, 147 papers (85%) were found to be both SP and ER, introducing new or improved cost-sensitive methods and testing them on medical data. ER was the second most frequent type, comprising 23 studies (13.3%), whereas VR was the focus of only two papers (1.2%) published in the years 2018 and 2021 (Wang et al. 2018a; Aldraimli et al. 2021). Notably, one paper (0.6%) in 2021 stood out as a combined review and ER effort, initially surveying existing methods before conducting performance benchmarking (Rahman et al. 2021b).

The evolution over time of the two most frequent research types, ER and SP, is displayed in Fig. 3. It is apparent from the line chart that the number of SP with ER was consistently higher than that of ER alone. The number of papers proposing new cost-sensitive techniques or enhancing existing ones rose from 2011 to 2013, peaking at six papers per year before falling in 2014 and levelling off at three papers until 2016. After that, SP+ER studies surged significantly, especially after 2020, with the highest number of papers (52) published in 2022. Conversely, the trend for studies evaluating existing solutions followed a different pattern. They first appeared in 2012 with one paper, peaked at three papers in 2013, and then declined to zero papers in 2016, where they remained until 2019, except for one paper published in 2017. In 2020, five ER studies resurfaced, increasing slightly to six studies in 2021 and then falling to four studies in 2022.

The analysis revealed that all the papers proposing new CSL methods also conducted experimental evaluations to demonstrate their effectiveness. This is a noteworthy point, as it indicates that researchers are not simply proposing theoretical solutions but are also committed to demonstrating the practical value of their work.

Furthermore, the dominance of SP+ER papers in the literature suggests that researchers primarily focus on proposing new methods for CSL rather than evaluating existing solutions. While this could be attributed to the complexity of the CSL problem and the unique challenges posed by imbalanced medical data, it also indicates the significant investment and interest in advancing the state-of-the-art in this area. The surge of SP+ER studies after 2020 suggests an increasing awareness of the importance of effective CSL methods in medical applications. This trend is further fueled by advancements in ML, the availability of larger datasets, and increasing computational resources, enabling researchers to develop more effective CSL techniques and conduct more extensive and complex experiments.

Out of all the selected papers, only two focused on validating cost-sensitive methods, which may be attributed to the challenges associated with conducting validation studies in hospital settings. Such studies require close collaboration with medical professionals and access to sensitive patient data. Nevertheless, the limited number of validation studies suggests a need for further research to demonstrate and validate the practical value of CSL in real-world medical settings. Besides, the fact that there is only one review paper in the literature points to the necessity for more synthesis and critical evaluation of existing CSL methods in medicine.

Overall, the findings suggest a promising outlook for the future of CSL for medical data but also underscore the need for continued validation and rigorous evaluation of the developed techniques.

The selected studies were evaluated using three empirical methods: Case Study (CS), Historical-Based evaluation (HBE), and survey. Figure 4 illustrates the distribution of research and empirical types. It can be observed from the bubble plot that HBE was the most prevalent empirical type, with 117 papers (65.4%) using publicly available medical datasets to assess their models. Of these papers, the majority (105) proposed and evaluated novel or improved solutions, 11 studies evaluated existing ones, and only one study was dedicated to reviewing and evaluating previously suggested methods. The CS empirical type ranked second, with 60 papers (33.5%) using real-world datasets from hospitals or healthcare units. Among these papers, 46 were classified as SP studies, 12 as ER studies, and two as VR studies. By contrast, survey-based evaluations were relatively uncommon, with only two SP studies (1.1%) employing this method. It is worth noting that six papers used both public and real-life datasets and were hence double-counted in HBE and CS empirical types. Besides, 12 CS papers used real-world data from their previous works.

The prevalence of HBE studies indicates that many scholars rely on existing, publicly available datasets to assess their models. This practice partly owes to the abundance and ease of accessibility of historical data. However, it also stems from the challenges associated with obtaining real-world medical data, including ethical and legal considerations (Mello et al. 2018). HBE is considered a cost-effective and accessible way to evaluate models on a large scale and benchmark novel solutions against existing literature. Nonetheless, it is essential to recognise that HBE may not accurately reflect the complexities and nuances of real-world medical data. Therefore, researchers may need to supplement their HBE findings with other empirical methods, such as CS.

A less yet significant number of papers used CS with real-world data collected from healthcare units or hospitals. This indicates that researchers are keenly interested in testing their models in practical settings to ensure their applicability. CS provides a more detailed and nuanced understanding of how models perform in specific contexts and can prove valuable in validating or fine-tuning models developed using publicly available datasets. However, it is worth acknowledging that CS is typically more resource-intensive than HBE and requires collaboration with medical professionals and institutions.

Combining public and real-life medical datasets demonstrates an understanding of the strengths and limitations of each. Four papers merged these two types of data and employed the CS type for tasks such as collecting healthy controls (Wang et al. 2018b), gathering additional negative samples (Calderon-Ramirez et al. 2021), or testing solutions after training on historical data (Pranto et al. 2020). While this approach may enhance models’ generalizability and practical applicability, it may also introduce complexity by requiring additional preprocessing to ensure data comparability and consistency, as highlighted by Wang et al. (2018b). Therefore, researchers using this approach should be transparent about their methodology. The remaining two studies (Hu et al. 2021; Xu et al. 2020) employed the datasets separately to assess the generalizability of their methods. By testing their models on datasets with diverse characteristics and features, researchers can assess how well their models perform in various settings and contexts, contributing to their research’s overall reliability and validity.

The relatively limited number of survey-based evaluations may be attributed to the inherent challenges associated with effectively designing and implementing surveys and the multiple sources of bias (Cunningham et al. 2015) that may arise. These sources of bias include, among others: non-response bias, which occurs when patients who choose not to participate in the survey are systematically different from those who do participate; social desirability bias, which stems from respondents providing answers they perceive to be socially desirable rather than truthful; recall bias, which arises from patients inaccurately recalling past events or experiences, such as the duration and timing of symptoms or treatments; and instrument bias, which can occur if the survey instrument itself is flawed or biased, such as when a question is phrased confusingly, potentially distorting the accuracy of responses.

The 173 selected studies collectively explored 21 distinct medical disciplines. Interestingly, 17 papers addressed more than one discipline, either by investigating a topic at the intersection of two medical sub-fields (e.g., (Sung et al. 2021)) or by testing their methods on a diverse range of disciplines (e.g., (Gan et al. 2020)). Figure 5 showcases the distribution of studies per medical sub-field, focusing solely on sub-fields addressed by at least 2% of the selected papers.

The findings revealed that oncology emerged as the discipline garnering the highest degree of attention, accounting for 31.2% (54 papers) of the selected studies. As per the World Health Organization (WHO), cancer is a leading cause of mortality globally, accounting for approximately 10 million deaths in 2020 alone (World Health Organization 2022). The significance of accurate and timely diagnosis and treatment is paramount, and ML techniques hold great promise in this regard. However, cancer is a highly heterogeneous disease that can manifest differently in each patient. Additionally, patients often present with complex medical histories and comorbidities, which can complicate diagnosis and treatment. These factors can contribute to imbalanced medical data, making CSL an attractive approach to address these challenges and improve cancer care.

Cardiology and neurology received significant focus in subsequent order, constituting 15% (26 papers) and 12.7% (22 papers) of the investigated literature, respectively. CSL has demonstrated significant benefits in addressing cardiovascular and neurological diseases, widely recognised as significant health concerns. This finding is in line with the WHO’s report (2021), which identifies cardiovascular diseases as the primary cause of mortality globally, responsible for 17.9 million deaths in 2019. Additionally, the WHO acknowledges that neurological disorders such as stroke, Alzheimer’s disease, and other dementias are among the leading causes of disability and death worldwide (World Health Organization 2016). Given the high mortality rate associated with these diseases, accurate predictions are imperative. However, data imbalance can lead to biased models that fail to capture important patterns in the data. By adopting CSL, researchers aim to improve prediction accuracy and contribute to preserving human life.

Infectious diseases occupy the fourth position, representing 8.7% (15 papers) of the total studies. Notable attention has been dedicated to researching this sub-field since 2020. This trend is not surprising, considering the urgency and global impact of the COVID-19 pandemic, which first emerged in 2019 and has since garnered substantial research attention. Additionally, imbalanced data is a common issue in COVID-19 studies due to various factors such as differences in testing availability and criteria, variations in reporting standards, differences in demographics, healthcare infrastructure, and compliance with public health measures. Besides, there may be a publication bias towards COVID-19 studies due to the pandemic’s global impact, and funding agencies may have prioritised research on this topic. Lastly, data availability may have contributed to the popularity of COVID-19 as a research subject.

Other medical sub-fields, such as ophthalmology, endocrinology, and hepatology, were investigated by 11 papers (6.8%) each, demonstrating the relevance of cost-sensitive methods in these domains. Galdran and colleagues (2020) highlighted the value of cost-sensitive classifiers in addressing two critical challenges in diabetic retinopathy grading. These classifiers can effectively model the complex structure of a heterogeneous label space and are also advantageous in addressing severely class-imbalanced scenarios. Fan et al. (2022) pointed out the inadequacy of conventional models in considering the imbalanced distribution of diabetic datasets and the varying misclassification costs across distinct patient categories. In a previous study by Yang et al. (2021), the predictive accuracy of traditional ML methods and cost-sensitive models were compared for predicting hepatic encephalopathy in cirrhotic patients. The study’s results demonstrated the superiority of cost-sensitive models, underscoring their high suitability and potential for future prognosis studies.

Pulmonology was featured in 8 articles (4.6%), and nephrology, dermatology, and medical and health services were each investigated by six studies (3.5%). On the other hand, emergency medicine (2.9%), radiology (2.9%), and obstetrics & gynecology (2.9%) received relatively little attention, as did orthopaedics, which was addressed by only 2.3% of the selected studies (four papers).

Disciplines that received the least amount of attention in the selected studies were classified as "other," which included geriatric psychiatry and neonatology, each addressed by two papers (1.2%), as well as intensive care, radiomics, urology, and podiatry, which were each the focus of only one study (0.6%). This may be explained by factors such as limited data availability and researchers prioritising other research areas deemed more crucial and pertinent to patient care.

Upon rigorous analysis, it was observed that all six predefined medical tasks, namely screening, diagnosis, prognosis, management, monitoring, and treatment, were covered in the selected literature. Notably, a small subset of seven papers delved into multiple medical tasks, owing to their utilisation of diverse datasets associated with distinct objectives.

The distribution of studies per medical task is graphically presented in Fig. 6, providing a clear overview of the prevalence of each task within the selected studies. The findings unveiled diagnosis as the most extensively explored medical task, dominating the literature with an overwhelming majority of 66.5% (115 papers). This dominance can be attributed to a multitude of factors. Foremost, diagnosis is the keystone of patient care and treatment decisions, guiding healthcare professionals in determining appropriate therapeutic interventions. Accurate and timely diagnosis allows for identifying the most suitable treatment strategies (Mirbabaie et al. 2021), thereby increasing survival prospects and enhancing patient well-being. Recognising this fundamental role, researchers and practitioners invest significant efforts in developing effective and accurate diagnostic models and algorithms. Moreover, the availability of diverse and well-annotated datasets specifically designed for diagnostic purposes contributes to the preponderance of diagnosis-focused studies. These datasets serve as invaluable resources for training and evaluating diagnostic algorithms, encompassing various medical conditions and their associated diagnostic information. Furthermore, the relatively lower frequency of certain medical conditions within the population results in a scarcity of positive instances. This inherent imbalance within diagnostic datasets further accentuates the significance of exploring and advancing diagnostic methodologies. Additionally, the diagnostic process itself can introduce imbalances in datasets due to the requirement of invasive procedures or expensive tests to confirm certain cases. As a result, researchers are actively exploring CSL techniques tailored to address the challenges posed by imbalanced diagnostic datasets.

Prognosis ranked second in terms of research focus, accounting for 13.3% (23 papers) of the selected studies. The presence of a substantial body of research focused on prognostic prediction underscores its significance in medical research. The study of prognosis holds paramount importance in understanding and predicting the future course of various medical conditions (Moons et al. 2009). It provides healthcare professionals with vital insights into potential outcomes, recovery rates, disease progression, and possible complications. This information enables informed decision-making regarding treatment options, care plans, and patient management strategies. Furthermore, the emphasis on prognosis aligns with the contemporary shift towards precision medicine and patient-centred care (König et al. 2017). By customising interventions based on individual predictions, healthcare providers can improve patient outcomes and enhance the overall quality of care.

Screening was represented by 9.8% (17 papers) of the selected studies, highlighting the importance of early detection in the medical domain. Researchers have recognised the importance of developing effective techniques and models to identify individuals at risk or needing further diagnostic evaluation. Furthermore, the presence of 14 papers (8.1%) dedicated to treatment highlights the efforts invested in optimising therapeutic interventions and evaluating their effectiveness. These papers explore various treatment modalities, including pharmacological interventions (e.g. Wu et al. 2022), surgical procedures (e.g. Dorado-Moreno et al. 2017) and non-pharmacological approaches (e.g. Aldraimli et al. 2022).

On the other hand, management was featured in a limited number of seven papers (4%). The relatively scant attention given to this particular medical task may be attributed to various factors. One potential explanation is that management often involves intricate and multi-faceted strategies, which necessitate a combination of clinical expertise, patient engagement, and healthcare system considerations, which may be challenging to capture solely through data-driven approaches. Additionally, the focus on other medical tasks, namely diagnosis, prognosis, screening, and treatment in the selected studies, reflects the immediate priorities in medical research, where there is substantial stress on improving diagnostic accuracy, predicting clinical results, and optimising treatment interventions. However, despite the scarcity of selected papers on management, it remains a critical aspect of healthcare research. Effective management strategies can significantly impact long-term patient outcomes, quality of care, and resource allocation (Esfandiari et al. 2014). The paucity of publications highlights the need for future investigations and multidisciplinary collaborations to address the complexities of managing medical conditions.

The monitoring task exhibited the lowest representation within selected research, with a modest inclusion of only five papers (2.9%). This disparity in attention can be attributed to the immediate impact that other medical tasks hold, which often overshadows the perception of monitoring as a complementary aspect of care rather than a primary focus. Moreover, the findings may be elucidated by considering data and research resources availability. Monitoring requires longitudinal data collection and continuous observation of patients (Esfandiari et al. 2014), which can be challenging and resource-intensive. Researchers may face constraints when seeking access to large-scale, high-quality monitoring data, resulting in fewer studies in this area. Nonetheless, the limited number of papers on monitoring does not diminish its importance in healthcare. Monitoring assumes a crucial function in evaluating treatment efficacy, identifying early signs of complications, and ensuring patient safety (Khan et al. 2016). Moving forward, future research needs to consider the significance of monitoring in providing comprehensive patient care. Furthermore, researchers should explore innovative approaches to address the challenges encountered in monitoring within medical research.

CSL techniques can be broadly classified into two categories: direct approaches and meta-learning approaches (Fernández et al. 2018; Liu et al. 2021; Johnson and Khoshgoftaar 2019; Ling and Sheng 2008; Sheng and Ling 2006). The former category modifies the learning algorithms by incorporating misclassification costs during the model training phase (Fernández et al. 2018; Feng et al. 2020). Conversely, the latter category does not alter the learning algorithms per se (Liu et al. 2021). Instead, meta-learning adjusts the training data (preprocessing) or the model’s outputs (postprocessing) to ensure cost sensitivity. Popular preprocessing techniques include instance weighting based on a cost matrix and MetaCost (Fernández et al. 2018), which relabels the training data according to misclassification costs. Postprocessing techniques, meanwhile, often involve adjusting the decision thresholds based on the predefined costs (Fernández et al. 2018; Liu et al. 2021).

This study seeks to categorise the selected papers according to the cost-sensitive approaches they have employed, with the goal of obtaining a thorough understanding of the distribution and prevalence of these approaches within the medical literature. Figure 7 illustrates the distribution of cost-sensitive approaches used in the selected studies.

Direct approaches account for the largest share of papers, representing 76% (133 papers) of the qualified studies, indicating a clear focus on integrating cost-sensitive considerations directly into the learning process. Some researchers modified the objective function of the model to minimise the expected cost of misclassification. For example, Al-Sawwa and Ludwig (2019) introduced a new objective function within their cost-sensitive centroid-based differential evolution classification algorithm. This function involves two key steps: allocating misclassification costs to each class label and evaluating the fitness of individual vectors from the population. Initially, instances are assigned to the closest centroid based on the Euclidean distance. Subsequently, the misclassification cost is computed by summing over the misclassified instances. Other works incorporated the cost matrix directly into the loss function. For instance, in (Naceur et al. 2020), the authors used a weighted cross-entropy loss function in their Convolutional Neural Network (CNN) model for brain tumor segmentation. Furthermore, researchers have explored the fusion of multiple loss functions to tackle diverse problems effectively. Notably, one study fused the Focal loss with the Dice loss (Wang et al. 2022), while another investigation integrated the Dice loss with the weighted cross-entropy (Taghanaki et al. 2019). The ease of implementation is the primary factor contributing to this trend since most ML libraries offer readily available implementations (Sterner et al. 2021). Moreover, certain packages provide the flexibility to apply custom loss functions directly to the algorithm, allowing users to employ cost-sensitive loss functions tailored to their specific applications.

A considerable share of the selected studies (16.6%) adopted meta-learning approaches. Precisely, preprocessing was applied in 24 papers (13.7%), and postprocessing was employed in 5 papers (2.9%). Preprocessing was carried out using weighting or MetaCost. For instance, Wang et al. (2013) implemented cost-sensitive logistic regression and decision trees by weighting training instances based on the total cost associated with each class in the provided cost matrix. In another study, Afzal et al. (2013) employed MetaCost to integrate cost sensitivity into four ML models. Preprocessing techniques are adopted by researchers as they alter the training data instead of the underlying algorithm (Fernández et al. 2018), rendering them a suitable approach for different types of classifiers. On the other hand, postprocessing relied on thresholding. In a study conducted by Zhao et al. (2018), empirical thresholding was employed to iteratively adjust the decision threshold, aiming to select the classifier with minimal total cost. Alternatively, Liu et al. (2021) determined the threshold as per Eq. 3, leveraging prior knowledge about misclassification costs when developing a multi-label ECG classifier. This method demonstrated superior performance compared to commonly used thresholding techniques, including rank-based thresholding, proportion-based thresholding, and fixed thresholding. Thresholding is less frequently used in the selected studies due to the computational challenge of tuning multiple thresholds (equivalent to the number of labels considered), particularly in multi-label classification (Liu et al. 2021).

Note that direct and preprocessing approaches were utilised together in two papers, resulting in double counting in these categories. Additionally, 13 articles (7.4%) did not provide information on the cost-sensitive approach they adopted and were thus categorised as “unspecified”. Incomplete reporting may hinder the reproducibility and comparability of results and the identification of effective methods for dealing with imbalanced medical data. Given the importance of transparency in medical research, future studies should provide a clear and detailed description of the implemented cost-sensitive techniques, including any modifications made to the model, to allow for better understanding, comparison and replication of findings.

CSL, as applied to imbalanced medical data, presents a unique framework that brings both advantages and limitations to the table. Understanding these strengths and weaknesses is crucial to making informed decisions and driving advancements in the field. Dedicated tables have been prepared to provide a comprehensive overview of these aspects.

Table 12 presents the strengths and weaknesses of CSL in addressing class-imbalanced medical datasets. CSL techniques offer a multitude of advantages when dealing with such challenging scenarios. Firstly, they efficiently mitigate class imbalance, leading to more balanced predictions and improved performance across all classes. Notably, an overwhelming majority of the selected studies (97.7%, 169 papers) reported enhanced performance compared to cost-insensitive methods, state-of-the-art models, or other balancing techniques. Moreover, CSL explicitly considers the unequal misclassification costs in cost-sensitive problems, particularly inherent in medical decision-making (Liu et al. 2021; Siddiqui et al. 2020; Wang and Cheng 2021), ensuring that the model’s predictions align with the critical consequences of FP and FN. Importantly, these techniques achieve these benefits without altering the underlying data distribution, conserving the integrity and representativeness of the dataset. This preservation of the original data structure allows for full utilisation of all available data, in contrast to resampling techniques. Additionally, CSL techniques exhibit computational efficiency, enabling their application to large-scale medical datasets without excessive resource requirements. This finding aligns with the broader consensus from other reviews (Haixiang et al. 2017; Kaur et al. 2019; Tarekegn et al. 2021). Lastly, they prove particularly effective in handling severely class-imbalanced scenarios where conventional learning algorithms struggle to provide accurate predictions. This notable characteristic has also been highlighted in the survey conducted by Leevy et al. (2018), reaffirming the effectiveness of CSL in addressing highly imbalanced data. Together, these advantages position CSL as a valuable tool in tackling class imbalance and enhancing the reliability and applicability of ML models in challenging medical contexts.

Building upon the strengths and benefits of CSL, it is essential to also acknowledge the associated limitations and challenges that require careful consideration. One significant concern arises from the unknown nature of misclassification cost values, a challenge echoed in previous review studies (He and Garcia 2009; Haixiang et al. 2017; Kaur et al. 2019; Tarekegn et al. 2021; Leevy et al. 2018; Johnson and Khoshgoftaar 2019; Elrahman and Abraham 2013; Sun et al. 2011; Feng et al. 2020). Accurately defining the costs of misclassifying different classes can be intricate and challenging. The design of the cost matrix often requires expert judgment and domain-specific knowledge (Fernando and Tsokos 2022), making it a delicate and complex task, especially considering that such expertise and knowledge may not always be readily available or accessible.

One strategy to address the issue of unknown costs is to conduct a thorough search for the optimal cost setup (Nunes et al. 2013). This approach entails exploring various combinations or configurations of cost values and assessing their impact on the performance of the cost-sensitive model. Techniques such as cross-validation or grid search can be employed to identify the best cost setup by iteratively testing and evaluating different cost values using predefined performance metrics. Additionally, the literature proposes specific cost assignment strategies. For example, a potential solution suggested in previous research (Haixiang et al. 2017) consists of setting the misclassification cost of the majority class to 1 and equating the penalty for the minority class to the IR. This approach has been adopted by several selected studies (e.g., (Fan et al. 2022; Wang et al. 2013; Liu et al. 2019; Wang et al. 2020c; Ashfaq et al. 2019; Hashemi et al. 2018), while other studies have proposed alternative cost formulas (e.g., (Zieba et al. 2014; Calderon-Ramirez et al. 2021; Roy et al. 2022; Yao et al. 2022; Zieba 2014).

In a noteworthy contribution, a study (Gan et al. 2020) highlighted a particular concern about the prevalent use of fixed misclassification costs in most CSL methods. In response to this constraint, researchers have investigated dynamic weight assignments during the training process. For instance, Focal loss (e.g., (Galdran et al. 2020; Li et al. 2022b; Lu et al. 2021; Naseem et al. 2020; Shirokikh et al. 2020; Lee et al. 2023)) adapts the costs based on the varying difficulty or significance of individual samples. Another study (Liu et al. 2019) incorporates an online-learning step to dynamically reweight each batch of the training set based on its validation performance.

Another limitation that warrants attention is the risk of overfitting the under-represented classes. When misclassification costs are inadequately defined and heavily weighted towards the minority classes, CSL methods can exhibit excessive adaptation to these classes (Sun et al. 2011), which may lead to overfitting (Elrahman and Abraham 2013) and reduced generalization performance. Therefore, thoughtful attention should be paid to defining the costs, ensuring their appropriateness, and mitigating the risks associated with excessive adaptation.

Transitioning to the analysis of CSL approaches, it is essential to acknowledge their inheritance of the broader advantages and disadvantages of CSL. Additionally, they exhibit their own specific strengths and weaknesses, which are succinctly outlined in Table 13 for reference. It should be noted that some of these particular strengths and weaknesses are derived from existing reviews beyond the scope of the selected papers in this study.

Direct approaches offer both advantages and disadvantages. On the positive side, these approaches benefit from the availability of readily implemented solutions in many ML libraries. This accessibility allows researchers and practitioners to apply CSL techniques easily without extensive coding efforts. However, it is important to note that direct modifications in the learning algorithm, such as modifying the Gini index to account for misclassification costs in decision trees (Barot and Jethva 2021b, a), require a deep understanding of the underlying algorithms. This requirement means that researchers and practitioners must possess comprehensive knowledge of the specific algorithms being utilised. Another limitation of direct approaches is their potentially reduced versatility compared to other CSL approaches. By directly modifying the learning algorithm, these approaches are often customised for specific ML models, constraining their applicability across a broader range of models.

Preprocessing approaches, particularly weighting, offer several advantages in the context of CSL. Firstly, weighting is a simple technique that is easy to implement and interpret, as it involves adjusting the weights assigned to training instances. Moreover, it exhibits high flexibility by not necessitating any alterations to the underlying learning algorithm, ensuring adaptability to various ML models. As such, weighting emerges as a versatile approach for incorporating cost sensitivity.

Similarly, one key strength of MetaCost as a preprocessing approach is its flexibility, which allows for adapting classifiers to cost-sensitive scenarios without altering the underlying learning algorithm, operating at the instance level. This preserves compatibility with different ML models. Nevertheless, it is crucial to consider the computational implications of MetaCost, as it introduces additional steps during the training phase, such as data relabelling, which may result in increased computational complexity and longer training times.

In the realm of postprocessing approaches, particularly thresholding, a set of distinct advantages and limitations emerges. Notably, thresholding techniques exhibit a remarkable level of flexibility, enabling the adjustment of the classification model to various cost definitions without the need for retraining (Liu et al. 2021). This flexibility empowers researchers to adaptably fine-tune the model’s behaviour to align with specific cost-sensitive requirements. Furthermore, it is worth highlighting that thresholding does not require modifications to the underlying learning algorithm. However, it is important to acknowledge the limitations associated with thresholding techniques. One such limitation lies in the division between the training and the subsequent cost-sensitive evaluation phases. During the initial training, where cost information is unavailable, the classifier is driven by error minimisation rather than cost optimisation (Fernández et al. 2018). This implies that the estimation of cost parameters is initialised using a cost-insensitive method, which may introduce inherent biases into the outcomes. The problem is effectively addressed by incorporating an ROC-based criterion into classifier training, as performance for both classes can be evaluated at once (Fernández et al. 2018). Another challenge is the tuning of thresholds, as the number of thresholds that need to be adjusted is typically no less than the number of considered labels. This process can be time-consuming and may require careful fine-tuning to achieve optimal cost-sensitive performance.

Expanding upon our exploration of the strengths and weaknesses of CSL approaches, it is crucial to consider the valuable insights derived from other studies in the field. Recent research (Vanderschueren et al. 2022) categorises CSL approaches based on the stage at which misclassification costs are incorporated into two classes: cost-sensitive training of models and cost-sensitive decision-making. The former category encompasses techniques applied before or during model training to construct a classifier, notably direct approaches and meta-learning preprocessing methods. Conversely, the latter category concerns thresholding techniques used after training to inform decision-making processes. Regarding performance, models utilising thresholding may yield superior overall predictive accuracy; however, those adopting cost-sensitive training excel in their ability to make high-quality decisions, focusing on accurate predictions as they impact decision outcomes. Notably, the study, conducted on nine datasets from various application areas, revealed that training a cost-insensitive model and subsequently introducing misclassification costs during the test phase through thresholding can be conceptually straightforward and effective. Nevertheless, it was highlighted that, under specific conditions, cost-sensitive training may emerge as the optimal choice. For instance, in cases of model misspecification, adopting a cost-sensitive objective function may outperform thresholding. Furthermore, the investigation indicated that combining cost-sensitive training and thresholding may not consistently enhance performance. In light of these results, we hold the view that exploring the combination of CSL approaches presents a promising avenue that merits further experimentation and investigation.

In a previous comparison of instance weighting and thresholding (Zhao 2008), it was observed that instance weighting is computationally more demanding, particularly in scenarios with uncertain cost settings. Unlike thresholding, which adjusts the decision threshold, instance weighting requires retraining the classifier when changing cost settings. An interesting observation from this study was that when instance weighting is applied and the classifier is insensitive to the cost ratio, the thresholding technique suffices. However, when the classifier is highly sensitive to the cost ratio under instance weighting, it becomes crucial to incorporate misclassification costs during training. These findings underscore the importance of conducting more extensive evaluations and comparisons of CSL approaches to validate these observations, as we emphasise the need for further research in this direction.

Building on these insights, it becomes evident that the choice of a CSL approach in medical applications, especially in cases characterised by imbalanced class distributions, has significant implications for model performance. Researchers and practitioners should commence by evaluating the computational resources at their disposal. Preprocessing techniques such as weighting offer an expedient and versatile avenue for implementation, requiring no modifications to the underlying algorithms. However, direct approaches may offer tailored solutions for individuals with a deep understanding of algorithms, emphasising the need to balance computational efficiency and model customisation. Moreover, the tuning task when dealing with thresholding necessitates anticipation. Additionally, staying abreast of the most recent advancements in CSL is crucial, as new techniques may yield enhanced results. Regardless of the chosen CSL approach, thorough validation on the specific medical dataset at hand remains non-negotiable. This validation ensures the alignment of the selected approach with the data’s inherent characteristics and the fulfilment of particular research objectives. Consequently, well-informed decisions can be made to enhance model performance across various medical disciplines and tasks.

Examining the implemented methods in the selected research necessitates a thorough assessment of their strengths and weaknesses. Table 14 compares various proposed methods to facilitate this evaluation, encompassing their main tasks, data types, employed CSL techniques, weighting formulas, and the reported advantages and limitations for each method.

The information provided in Table 14 illuminates several key trends and findings. Firstly, it is evident that the proposed solutions encompass a wide range of models, spanning from traditional ML to DL architectures. This diversity highlights the versatility and adaptability of CSL across different modelling paradigms. Furthermore, the application of CSL is not limited to specific types of data. The selected studies showcase the use of CSL for various data types, including numerical data, categorical data, images, time series, and textual data. This broad utilisation of CSL underscores its effectiveness in addressing cost-sensitive challenges across diverse data modalities.

A common thread among the presented works is the consistent achievement of enhanced performance by leveraging CSL techniques, effectively addressing the challenges posed by class-imbalanced medical datasets. Moreover, many methods exhibit reduced computational time indicating their efficiency and practicality in real-world applications. However, one work (Zhenya and Zhang 2021) employing ensemble learning deviated from this trend, possibly due to the inherent complexity of the ensemble approach. Additionally, the robustness of the proposed methods is apparent, with several studies highlighting their ability to deliver reliable and stable results. This robustness enhances the trustworthiness of the proposed solutions in practical healthcare settings. Notably, one study (Ashfaq et al. 2019) even emphasised the potential cost savings associated with CSL, further underscoring the practical benefits of these approaches.

The papers also consistently underscore the significance of validation and generalizability in medical applications. Acknowledging the importance of validating the proposed models with diverse datasets and across different clinical sites reflects a commitment to ensuring the reliability and applicability of the methods in real-world scenarios. Furthermore, interpretability surfaces as a critical consideration in developing cost-sensitive solutions. One study (Liu et al. 2021) highlighted the challenge of interpretability as a potential limitation, recognising the need to balance model complexity and interpretability to facilitate transparency and understanding in clinical decision-making processes. Moreover, the studies shed light on the challenges related to the specificity and sensitivity trade-off, a common concern in CSL. Achieving an optimal balance between these measures is crucial for attaining accurate predictions while minimising false alarms.

It is also noteworthy that several works compared CSL with alternative strategies such as resampling and ensemble learning. The outcomes of these comparisons varied, with some studies showcasing the superior performance of CSL, while others found alternative strategies to be more effective. These findings highlight the importance of carefully selecting the most suitable strategy based on the specific characteristics of the dataset and the learning task at hand.

The medical datasets used in the selected studies are imbalanced and thus perfectly suitable to assess the performance of the developed cost-sensitive methods and evaluate their effectiveness. A total of 196 datasets were identified across the 173 selected papers. Note that 52 papers (30%) employed multiple datasets to evaluate their methods. Table 15 presents the most common datasets used in at least four studies, along with their sources, data types, number of instances, attributes, classes and papers. All the presented datasets are publicly available. The MIT-BIH Arrhythmia database was the most commonly used in seven selected studies, followed by COVID-19 Chest X-ray, Wisconsin Diagnostic Breast Cancer, and Pima Indians Diabetes datasets, each employed in six papers. The ISIC 2019 dataset was used in five papers, while Thyroid Disease, HAM10000, ILPD, and BUPA Liver Disorders datasets were each used in four papers. Of the 187 remaining datasets, 70% were publicly available, and 30% were acquired from hospitals and healthcare units.

Based on the findings, the MIT-BIH Arrhythmia Database (Moody and Mark 1980) was the most commonly used dataset, owing to the numerous selected studies in cardiology and its reputation as a well-known benchmark dataset. The MIT-BIH Arrhythmia Database comprises 48 ECG recordings, each with a duration of 30 min and a sampling frequency of 360 Hz (Shi et al. 2019). All heartbeats within the recordings are expertly annotated and categorised into one of 15 heartbeat types. The AAMI standard provides a standardised approach for labelling these arrhythmias to ensure consistency and comparability across different studies. According to this standard, heartbeats are recommended to be grouped into five main classes (Shi et al. 2019): Normal (N), Supraventricular ectopic beat (S), Ventricular ectopic beat (V), Fusion of ventricular and normal beat (F), and Unknown beat (Q). Although most studies adhere to this standard (Li et al. 2022b; Zhao et al. 2023; Wang et al. 2019; Han et al. 2022), some may choose alternative classification schemes (Lu et al. 2021). The extensive use of this database has resulted in numerous published studies, providing ample opportunities for comparisons with previous results. The reported IRs of the database differ based on the class distribution employed for the classification task. One such study, conducted by Zubair and Yoon (2022), reported an IR of approximately 9:1, with normal beats accounting for 89.5% of the dataset and abnormal beats accounting for the remaining 10.5%. This severe class imbalance makes the MIT-BIH Arrhythmia Database an exceedingly challenging dataset for cost-insensitive models.

The COVID-19 Chest X-ray dataset (Cohen et al. 2020a), assembled by Cohen and colleagues (2020b) in February 2020 from publicly available sources, has gained widespread recognition as a reference dataset for developing and evaluating DL algorithms to detect COVID-19 from chest X-ray images. Comprising five distinct types of pneumonia cases, including COVID-19, SARS, Streptococcus spp., Pneumocystis spp., and ARDS, this dataset has been extensively utilised as a starting point for exploring various DL techniques, particularly during the surge of research amidst the COVID-19 pandemic. Its public availability and recognition have made it a convenient choice for researchers to compare their findings with other studies in the field. Moreover, the dataset’s acknowledged class imbalance, with a higher number of COVID-19-positive cases compared to other respiratory diseases, makes it a relevant and challenging testbed for evaluating the performance of CSL algorithms. Consequently, the COVID-19 Chest X-ray dataset is frequently employed in the selected studies for these compelling reasons. Nevertheless, it is noteworthy that the dataset has certain limitations, as it may not provide a fully comprehensive or representative sample of the general population. This is likely why the six studies that employed the dataset incorporated additional datasets to supplement their findings. In September 2020, the dataset was updated with 679 frontal chest X-ray images from 412 individuals in 26 countries (Cohen et al. 2020c).

The Wisconsin Diagnostic Breast Cancer dataset (Wolberg et al. 1995) has been extensively employed in numerous studies, primarily due to the prominence of oncology in the selected research. Of the 54 studies conducted in this sub-field, 38.9% focused on breast cancer, making it a prevalent research topic. Moreover, the dataset’s imbalanced distribution, with a disproportionate number of malignant cases (212) compared to benign cases (357), makes it a relevant resource for research on CSL in the medical field. The Pima Indians Diabetes dataset (National Institute of Diabetes and Digestive and Kidney Diseases 1990) has garnered equal attention in the selected research papers owing to its pertinence in the diagnosis of diabetes, a critical healthcare concern that affects a large population worldwide. Moreover, the dataset suffers from a significant class imbalance between the presence (34.9%) and absence (65.1%) of diabetes cases, rendering it an appropriate choice for evaluating the performance of cost-sensitive models.

In the selected research, using multiple datasets to evaluate model performance was commonplace. This approach is highly beneficial as it enables the assessment of model generalizability, facilitates a more comprehensive evaluation, and establishes benchmarks for the field. Datasets with varying difficulty levels, IRs, and feature spaces can expose the strengths and limitations of models. Using multiple datasets mitigates the risk of models overfitting to one particular dataset, ensuring models are more readily applicable to new and unseen data.

Data represents a fundamental aspect of CSL in the medical field, and comprehending the used data types is paramount. The analysis of the selected papers revealed a diverse range of attribute types. Of the 173 studies analysed, 78 papers (45.1%) utilised images as the primary data type, while numerical and categorical data were combined in 51 papers (29.5%). Additionally, time series (24 papers, 13.9%), textual data (seven papers, 4%), and numeric data (four papers, 2.3%) were also employed. Less prevalent combinations included images with numerical and categorical data (seven papers, 4%), time series with text (one paper, 0.6%), time series with numerical data (one paper, 0.6%), time series with numerical and categorical data (one paper, 0.6%), and images with numerical data (one paper, 0.6%). For a more granular perspective, the most prominent types of medical imagery data included Computed Tomography (CT) scans (19 papers), Magnetic Resonance Imaging (MRI) (16 papers), Chest X-ray (13 papers), dermoscopic images (10 papers), fundus photographs (eight papers) and thermograms (four papers). Regarding time series data, the most commonly used types were Electrocardiogram (ECG) records (nine papers), Electroencephalogram (EEG) records (five papers), and Cardiotocography (CTG) records (two papers).

Regardless of the data types, medical data is inherently prone to imbalance due to the nature of medical conditions and patient populations. The analysis revealed a broad spectrum of data types among the selected studies. This diversity in data types can be justified by the fact that different medical applications require different types of data for accurate outcomes. For instance, medical imaging techniques, such as CT scans, MRI, and Chest X-rays, are essential for detecting anomalies in anatomical structures. In contrast, time-series data, such as ECG and EEG records, are necessary to monitor physiological function changes over time.

The dominance of images as the primary data type can be attributed to the increasing use of medical imaging techniques in clinical practice and research. With the advancements in imaging technology, clinicians can now obtain high-quality images that provide detailed information about the structure and function of various organs and tissues. Additionally, numerical data can capture continuous measurements such as blood pressure, heart rate, and body temperature. In contrast, categorical data can capture non-continuous measurements such as gender, age, and medical history. Combining these two data types can help achieve a more comprehensive understanding of a patient’s health status. Moreover, the use of time series data highlights the importance of temporal information in medical applications. Time series data can capture changes in a patient’s health status throughout time, aiding in the detection and prediction of medical conditions.

Among the selected research, the presence of text data was also observed. Textual data can capture unstructured information like clinical notes, medical reports, and patient history. Such data can provide valuable insights into the subjective nature of a patient’s medical condition, including their symptoms, emotions, and experiences, which may not be accurately captured by numerical or categorical data alone. Additionally, textual data can be leveraged to identify patterns and relationships between medical conditions and patient characteristics, paving the way for developing personalised treatment plans.

Traditional performance metrics, such as accuracy, precision, sensitivity, and F1 score, provide insights into the overall predictive performance of cost-sensitive algorithms. These metrics evaluate the model’s ability to correctly classify instances without explicitly considering the cost associated with misclassifications. To quantify these metrics, a fundamental tool called a confusion matrix is employed.

The confusion matrix, showcased in Table 16, provides a detailed breakdown of the model’s predictions and the actual class labels. It summarises the counts of True Positives (TP), True Negatives (TN), FP, and FN. TP corresponds to the accurately predicted positive instances, while TN represents the accurately predicted negative instances. Conversely, FP and FN denote instances that were erroneously classified as positive or negative, respectively.

Cost-related metrics thoroughly evaluate classifier performance by explicitly incorporating the costs associated with different misclassifications. These metrics go beyond traditional metrics and take into account the real-world impact of classification errors. By considering the consequences of misclassification, cost-related metrics offer a more nuanced and practical assessment of a classifier’s effectiveness, especially in domains where misclassification costs are high, such as medicine.

Figure 8 presents the most commonly used performance metrics in the selected studies. Among these metrics, sensitivity was the most frequently used, appearing in 139 papers. Accuracy and specificity followed, being utilised in 100 and 91 studies, respectively. AUC and precision were employed in 77 papers each, while the ROC curve was utilised in 54 papers. G-mean was observed in 43 papers. The Dice score and balanced accuracy exhibited similar usage, being used in 14 studies each. On the other hand, FPR and MC were employed in nine studies each. Less common metrics included, among others, AUPRC and FNR, which were utilised in five and four papers, respectively.

In addition to the MC metric, other cost-related metrics were also relatively underutilised. Specifically, the CWA metric was identified in only one paper, while cost curves and the weighted Kappa score were not employed. This finding highlights the need for increased emphasis on incorporating and exploring cost-related metrics in CSL research.

It is also noteworthy that the vast majority of the selected studies incorporated several metrics in their performance evaluation. Combining multiple evaluation metrics is a widely adopted practice in ML research, particularly when dealing with imbalanced datasets. This approach ensures a comprehensive assessment of the model’s performance. Additionally, different metrics can capture distinct aspects of model performance. By integrating multiple metrics, researchers can obtain a more nuanced understanding of the model’s strengths and limitations, allowing them to make informed decisions about which model best suits a given problem.