Regulatory Considerations on the use of Machine Learning based tools in Clinical Trials

The availability of huge amount of data, together with an extraordinary computational power, is promoting a wide use of AI and at the same time it is raising ethical and social concerns that must be addressed to optimize the benefits and to prevent risks. The importance of the social role played by AI is described in the definition released by the Organisation for Economic Co-operation and Development [1], that verbatim quote: “AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy”. The use of AI and more in general of digital technologies is causing a global socio-economic change that also affects medicine and healthcare and this can be verified by the growing interest and by the number of scientific publications [2] of researchers and players attracted by AI and digitalization. By simply performing a Google Scholar review article search [3], we can retrieve more than 26,000 articles with a searching criteria “artificial intelligence healthcare” and more than 62,000 articles with a searching criteria “artificial intelligence medicine”. Health digitalization is a broad definition encompassing various technologies, ranging from apps for smartphones used to reveal skin cancer [4], or telemedicine often used during Covid-19 pandemic as measures to mitigate risks [5], to digital therapies [6] that are being developed and that impact on the behavior, like the first game-based digital therapeutic to improve attention function in children with attention deficit hyperactivity disorder (ADHD) approved in the United States (US). Digitalization can be found to be involved also in the clinical development processes of medicines where the use of ML, a discipline encompassed by AI and based on the use of mathematical modelling to analyze data, is providing important insights in term of prediction or classification by adapting the performance of the algorithm as the availability of data increases, obtaining important and significant gain of resources. The importance of digital technologies and AI is shown also by the increasing number of ML-based tools approved by the food & drug administration (FDA) [7] and by the fact that it is considered as a strategic goal in the european medicines agency (EMA) regulatory science to 2025 strategic reflection paper [8]. Regulatory Agencies (RAs) together with political institutions and scientific organizations are working and discussing about new paradigms, that are bringing along concerns, but with the aim to foster scientific research, and to accelerate the access of patients to therapeutic opportunities, still ensuring strong regulatory requirements. For instance, the international medical device regulators forum (IMDRF) [9] has released a framework for risk categorization based on the importance of the information provided and on the seriousness of the clinical condition, while FDA is giving various substantial contributions, for instance with the promotion of good machine learning practice (GMLP) [10], referred to data management and selection, training and tuning for building reliable software. The EMA and the heads of medicines agencies (HMA) have published two reports [11, 12] focused on the regulatory validity of big data, with the definition of various steps such as data standardization and evidence generation. The need to test the ability of ML methods to identify data that may support with the interpretation of healthcare data together with real-world data (RWD), in a clinical trial setting, is clearly identified by EMA in the recent regulatory science research needs publication [13]. The application of ML in CTs may widely vary, from patient recruitment to study design, to the definition of endpoints or to perform a more accurate diagnosis; in any case the assessment of these technologies is impacting the activities of RAs involved in the authorization of clinical studies [14]. From a regulatory point of view, the lack of dedicated guidelines and harmonized approaches brings uncertainty among applicants and RAs [15], making it difficult to frame these tools, that sometimes according to the stated intended use, may be used within a trial for instance in the selection of patients to be enrolled, with the aim to just save resources in time consuming processes, and so they may not meet the definition of medical device. However, ML tool software should in principle be considered as a medical device, providing information which are even used by physicians for decision-making purposes with a therapeutic or diagnostic aim, and therefore e.g. in the EU, these are regulated under the Medical Device Regulation (EU) 2017/745 (MDR) [16] or the In Vitro Diagnostic Medical Devices Regulation (EU) 2017/746 (IVDR) [17]. Although the regulatory assessment of a medical device may be performed by a different office from the clinical trials one, or by a different RA in some Member States (MSs) in EU, the interaction between RAs, offices and assessors is mandatory. It is crucial to share data and information, and to ensure the compliance with fundamental principles such as the protection of rights, safety, dignity and well-being of subjects, and the generation of reliable and robust data in accordance to requirements set out in the Regulation (EU) 536/2014 [18] in EU or, in case of other countries not in the European Economic Area (EEA), in compliance with those principles described in the International Council of Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) [19], Guideline for Good Clinical Practice [20] and the Declaration of Helsinki [21]. There is a clear need to standardize the regulatory approach to the assessment of ML tools in CTs to support a prompt regulatory acknowledgement and speed-up the incorporation of innovation into the CTs assessment and authorization process. We propose a step forward in such a process by discussing the requirements for a trustworthy AI and their relationship with the required key regulatory points and characteristics that need to be addressed for CTs that use ML-based technologies, focusing mainly on adaptive algorithms, as considered potentially more critical in a CT setting. However, the same approach and principles may apply to the use of any ML/AI. We also analyze how issues identified during an assessment process could potentially impact both on crucial regulatory requirements and the requirements for a trustworthy AI, highlighting critical areas of interaction.

To identify the key points that need to be addressed in a CT setting involving the use of AI or ML systems, we took as a reference the first supporting guide available in the overall EU regulatory landscape specifically focusing on the request for authorisation and assessment of clinical trials involving the use of AI/ML [22]. Even if the guide may reflect one Competent Authority (CA) perspective, it is the only one that to the best of our knowledge lists and describes the regulatory information that should be submitted to the CA to request the authorization of a CT impacted by ML-based tools. The Ethics Guidelines for Trustworthy AI publication by the European Commission defines instead the key requirements for a trustworthy AI [23]. The standard assessment process of a CT is oriented in order to identify to what extent some key regulatory points are impacted, focusing on those primary areas of interaction with the requirements for a trustworthy AI as well as potential challenges that may be further elaborated and extrapolated to implement dedicated policies able to support the regulatory assessment and decision-making process in a real CT setting.

A list of 33 potential issues was identified, reported in Table 2.

Potential issues were linked to the requirement for trustworthy AI and impacted key points. For a given potential issue identified during the assessment, one or multiple key points as well as one or multiple requirements for trustworthy AI were associated using the issue categorization form (Table 4). The absolute number of issues impacting each single area after linking them both to the key point impacted and to the requirements for trustworthy AI in CTs, is reported in Table 3.

The highest number of potential issues were identified in the fields of technical robustness and safety of the ML tool and in relation to the data. However also the level of evidence, data accountability and transparency are greatly impacted. Results were further elaborated to map them in terms of interactions between the key points and the requirements for trustworthy AI, highlighting the most impacted issue combination areas as reported in Fig. 3.

The use of ML-based tools or AI is an opportunity as far as it is trustworthy. Fulfilling this requirement in the context of a CT setting means to be able to provide a contribution to the safety and efficacy profile of the study, whose evaluation is the main task of RAs [27]. For this reason, the assessor at the CTO should know the tools employed into the clinical study, regardless of whether it meets the regulatory definition of medical device or not. In the assessment process, the key information submitted to RAs should be evaluated taking into consideration risks and benefits and how they may affect the main requirements for trustworthy AI. The findings resulting from the application of the assessment process, collected in the issue categorization form, contain some limitations like the potential lack of a strong evidence because of the methodological bias. Our considerations are extrapolated from the experience in the assessment process of assessors and, we could not report detailed data of a specific study, providing their interpretation and explaining the reasons of a regulatory decision. It is also important to consider that the assessment of the CT is performed by one team of professionals only however, even if this is the regular process, it could be also useful to share and compare our insights with an enlarged group of assessors facilitating the identification of inferential correlations. Sharing similar projects with other RAs would be a desirable aim to reach harmonization in the assessment. Another limitation is the awareness that the issues reported in Table 2 cannot be considered as an exhaustive list, since additional information may emerge during the assessment of a real protocol, and because multiple types of CTs (complex trials, different therapeutic areas, different patient populations etc.) and designs could be approached. Although these limits, the results in Fig. 3 can provide significant suggestions on how to complement the assessment process currently followed for any CT application at the CTO and could support the output of the actual regulatory authorisation process by helping to focus on those most impacted issue combination areas, highlighting those spheres of potential greatest risk. In addition, this is valuable information that could be used to further elaborate the intrinsic value of the results retrieved, as detailed in the following subsections, in light of an extrapolation exercise to cover other study designs and ML tools. This method could be even used to support the drafting of a dedicated guideline on the assessment of CTs impacted by ML or AI tools.

The most impacted areas in terms of interaction, combining key regulatory points and the principles for a trustworthy AI, after the identification of potential issues during the assessment, are those related to the technical robustness and safety of the ML tool when connected to the level of evidence generated and the data used. However also the transparency of the algorithm and of the data are additional areas of greatest impact. There are also other combined areas highlighting how training and validation datasets, the algorithm and the intended use can directly impact on the technical robustness and safety of the tool. Stakeholders and the level of evidence are connected to human agency and oversight, and data management is impacting accountability. Important information is also that the performance metrics of the tool and the healthcare and clinical setting do not seem to be very critical information, at least in terms of quantitative number of potential issues. However, the independence from a clinical setting could be considered as a point in favor of the potential extrapolability of the used method. Even if there are also additional areas of interaction that may have a minor impact, a qualitative analysis of each issue should always be completed, and all issues should be further explored during the assessment process.

Digital health technologies are triggering a paradigm shift and consequently RAs should implement new methods and approaches to complement the assessment of the safety and efficacy profile of medicinal products developed using data-driven tools, where data used play a main role. Data can both inform the adaptive algorithms, able to optimize their performance over time, or can be used in locked algorithms that do not update themselves in presence of new data and generate always the same outputs. In any case, the advantages in optimizing the performance with the use of learning algorithms should be balanced with various potential biases whose evaluation is an emerging issue on which various health institutions and international organizations are currently working on. Although the important efforts and significant results obtained so far, when a ML-based tool is proposed in a CT setting, additional efforts should be made to achieve a global harmonization of the assessment process. Stemming from our assessment, we propose a concrete starting point providing regulatory considerations following a bottom-up approach, moving from the point of view of the assessors that already had on their desks and assessed CTs impacted by ML methods, to link the key regulatory information to general principles for a trustworthy AI. The regulatory authorisation of CTs that use ML-based tools is a challenging task for all RAs that need to identify new methods of assessment and paradigms. The initial contributes of this paper should be further explored by enlarging the pool of assessors and by extending the feedback collection to a multistakeholders’ platform including among others, additional RAs, Ethic Committees, sponsors of CTs and patients. Our insights show the interaction between key regulatory points impacted and the requirements for trustworthy AI, as designed by the potential issues identified during the assessment, highlighting the most impacted issue combination areas. There is a clear evidence of the importance of the data used, with its connected level of evidence, directly impacting the technical robustness and safety of the ML tool. The cruciality of the data and the algorithm transparency also highlights elements to take into considerations in the regulatory assessment process. Other areas of mutual involvement are those relating to the intended use, algorithm, training and validation datasets, technical robustness and safety. Further areas of interaction may have additional intrinsic value depending on the specific CT design and setting, therefore even if specific areas of attention are clearly indicated, none of the key regulatory points or requirements for trustworthy AI should be excluded during the assessment of a CT that foresee a ML-based tool.