Use case cards: a use case reporting framework inspired by the European AI Act

An AI model is a mathematical construct that generates an inference, or prediction, based on input data (Estevez et al., 2022). It can either be the result from training based on a machine learning algorithm, or the outcome of other approaches based on symbolic or knowledge-based AI methods (OECD, 2023a). Popular examples of AI models include object detectors, language/image generation models or content search algorithms. While some AI models are designed for specific purposes (a.k.a. narrow AI models), many others are conceived as general-purpose AI models that can be eventually adapted and deployed in multiple application areas (Gutierrez et al., 2023). The level of generality varies, and models can range from being versatile to highly specialised based on their intended use case. For instance, an object detector can be embedded in car’s software system to recognise vehicles, road signs and pedestrians (Gupta et al., 2021), or be used for automatic people counting during a demonstration for surveillance purposes (Sánchez,Hupont,Tabik and Herrera, 2020). Similarly, the same large language model can be adapted to function as a chatbot system for e-commerce (Zhou et al., 2023) or in the medical domain (Li et al., 2023).

An AI system is typically built by combining one or more AI models. The compromise agreement of the AI Act (Council of the EU, 2023) aligns the definition of AI system with the approach recently proposed by the OECD (OECD, 2023b)¹. Bringing one or more AI models to a real-world application is not immediate, as it implies the effort of integrating them in a functional system (i.e., an AI system), including the necessary infrastructure, user interfaces, data pipelines, and other components required for the application to operate effectively in a production environment (Hupont et al., 2022). Further, it is important to consider in the process the use cases or variety of scenarios where the resulting system can be deployed. Use cases illustrate how users can utilise the AI system to accomplish their goals and therefore provide a key user-centric perspective on its functionality.

The EU AI Act supports precisely this human-centric approach, putting the concept of intended purpose at the centre of regulation. The definition provided for the intended purpose is the same across the three versions of the AI Act, that is:

According to the proposed regulation, the system’s intended purpose determines its risk profile which can be, from highest to lowest (European Commission, 2023): (1) unacceptable risk, covering harmful uses of AI or uses that contradict European values; (2) high-risk, covering uses identified through a list of high-risk application areas that may create an adverse impact on safety and fundamental rights; (3) limited risk, covering uses that pose risks of manipulation and are subject to a set of transparency obligations (e.g. systems that interact with humans such as conversational agents, are used to detect emotions or generate or manipulate content such as deep fakes); and (4) minimal risk, covering all other AI systems. Figure 1 illustrates this risk level approach. It is important to note that the risk categorisation is also consistent across the three version of the AI Act, although the specific use cases included in each risk level slightly differ among them.

The AI Act establishes a set of harmonised rules that associate use cases with risk levels, which in turn imply different legal requirements. Of particular significance are AI systems classified as high-risk, which are further subjected to conformity obligations. The rules to categorise an AI system as high risk are provided in Article 6 in all the three versions of the AI Act. Despite minor nuances between the three legal texts, we have the following two options. First, an AI system intended to be used as a safety component of a product, or that is itself a product, covered by Union harmonisation legislation listed in Annex II (e.g. machinery, toys, medical devices regulations) if it requires undergoing a third-party conformity assessment. Second, AI systems falling under one or more of the critical areas and use cases referred to in Annex III (e.g. remote biometric identification systems, AI systems used to prioritise the dispatch of emergency services, those used as polygraphs by law enforcement). Therefore, the risk level depends on a series of key information elements that are essential to document its intended purpose. We have compiled them in the list presented in Table 1. As can be observed, the system should be put into context by providing information on the operational, geographical, behavioural and functional contexts of use that are foreseen; who the users and impacted stakeholders will be; and what the system’s inputs and outputs are. In addition, it is equally important to clearly specify the intended use of the system as well as its foreseeable potential misuses. Lastly, three elements are particularly important when it comes to identifying an AI system’s risk level. The first one is the type of product, which is linked with Annex II and the possible need for a third-party conformity assessment. The second element is to determine whether the AI system is a safety component or not. The third one is the application area, which is linked with Annex III.

Having all these information elements adequately covered in a unique use case documentation methodology would be a valuable tool both for policy makers and AI system providers to better navigate the AI Act and properly assess the risk level of AI systems as well as tailoring the different requirements. However, current AI documentation approaches fail to provide full coverage as we will see in the next section.

In recent years, key academic, government and industry players have proposed methodologies aimed at defining documentation approaches that increase transparency and trust in AI. Table 2 summarises the most popular ones, and analyses the extent to which they cover the use case-related information needs identified in the previous section. Note that the table exclusively considers documentation methodologies focusing on AI models, systems or services. For instance, it does not include works tackling only dataset documentation such as Datasheets for Datasets (Gebru,Morgenstern,Vecchione,Vaughan,Wallach,Iii and Crawford, 2021), The Dataset Nutrition Label (Chmielinski et al., 2022) or Data Cards (Pushkarna et al., 2022).

Firstly, the table shows the importance the AI community places on documentation, as big tech (Google, IBM, Microsoft, Meta) and high stakes institutions such as the Organisation for Economic Co-operation and Development (OECD) are behind most adopted methodologies. For instance, Google’s Model cards (Mitchell et al., 2019) can now be automatically generated from the widely used TensorFlow framework,² which is strongly fostering its adoption by AI practitioners.

Nevertheless, as anticipated in the Introduction, the majority of methodologies have a strong technical focus. They are generally conceived as tools for AI developers and providers to demonstrate AI models’ performance and accuracy. Most recently proposed methodologies, including the Framework for the classification of AI systems by the OECD (OECD, 2022), AI usage cards (Wahle et al., 2023) and System cards (Meta, 2023), are expanding to cater to other audiences such as policy-makers and end-users. Even though some methodologies do explicitly ask about the intended use of AI the system (e.g. “What is the intended use of the service output?" in Arnold et al. (2019), “Intended Use" section in Mitchell et al. (2019) and “Task(s) of the system” in OECD (2022)), they do so in very broad terms and provided examples lack sufficient details to address complex legal concerns. Moreover, none of these methodologies are based on a formal standard or specification. In summary, to date there is no unified and comprehensive AI documentation approach focusing exclusively on use cases and covering information elements such as type of product, safety component and application area. Our proposed use case cards aim to bridge this gap.

Among use case modelling methodologies, the one proposed in the Unified Modelling Language (UML) specification is the most popular in software engineering Koç et al.(2021). It has the advantage of being an official standard with over 25 years of use, supported by a strong community (Object Management Group, 2017). Further, it is user-friendly, offering a highly intuitive and visual way of modelling use cases by means of diagrams and a set of simple graphic elements (Fig. 2).

UML use cases capture what a system is supposed to do without delving into technical details (e.g. concrete implementation details, algorithm architectures). Instead, they focus on the context of use, the main actors using the system, and actor-actor and actor-system interactions. A use case is triggered by an actor, which might be an individual or group, who is referred to as the primary actor. The use case describes the various sets of interactions that can occur among the different actors while the primary actor is pursuing a goal. A use case is considered successfully completed when the associated goal is reached. Use case descriptions also include possible extensions to this sequence, e.g., alternative sequences that may also satisfy the goal, as well as sequences that may lead to failure in completing the goal.

Once the use case has been modelled in a diagrammatic form (Fig. 2-right), the next step is to describe it in a brief and structured written format. Although the UML standard does not impose this step, it is commonly carried out in the form of a tablet. The most widely-used layout is the one proposed in Cockburn (2001) and shown in Fig. 2-left.

The information elements related to use cases under the AI Act (c.f. Table 1) were found to closely correspond with those of the software use case documentation under UML, e.g.: context of use and scope \(\longleftrightarrow\) intended purpose; primary actor \(\longleftrightarrow\) user; stakeholders and interests \(\longleftrightarrow\) stakeholders; open issues \(\longleftrightarrow\) foreseeable misuses; and main course \(\longleftrightarrow\) inputs/outputs. For this reason, we decided to ground our proposed use case cards in UML. The process of transforming classic UML use case diagrams into use case cards was carried out in a co-design workshop with stakeholders which is detailed further in Sect. 4.1. In the next sections, we focus on presenting the final use case card design and explaining how to fill it.

The designed use case card template is shown in Fig. 3. It is composed of two main parts: a canvas for visual modelling (right) and an associated table for written descriptions (left). Both are very close to the UML standard, with a few additional information elements inspired by European AI policies as follows. The canvas contains the following visual elements:As later discussed in the validation process (Sect. 4.2), a common issue for stakeholders who are not familiar with UML methodology is understanding the distinction between a system (in our case, an AI system) and a use case. The system perspective considers the AI system as a whole and helps in understanding its components (both AI and non-AI) and their relationships. Use cases, on the other hand, represent the specific interactions that actors have with the system and the functionalities the system provides them. By distinguishing systems from use cases, UML provides a modular and flexible modelling approach, allowing to focus on different aspects of the system at different levels of abstraction and granularity. Also note that for a system to be considered AI system in a use case card it has to content at least one AI use case.

The table layout has some changes with respect to the one proposed in (Cockburn, 2001). First, the intended purpose of the system encompasses three fields. Two of them already appeared in the original table, namely, context of use and scope. Both are to be filled with a short text description; we recommend a maximum of 100 words. Remaining field is Sustainable Development Goals (SDGs) and its values should by picked from the official United Nations’ list presented in Appendix A’s Fig. 12. Note that the purpose of this field is stating the SDGs to which the use case contributes (i.e. has a positive impact).

In addition, three new fields have been added as they are essential to determine the use case’s risk level –and thus the one of the AI system containing it– according to the AI Act. Their description can be found in Table 1 and below we comment on their possible values:Remaining fields correspond one-to-one with those in the original table. The only change appears in the description of the open issues field where we have emphasised the need to include foreseeable misuses of the system.

This application is inspired by real products in the market, including Microsoft’s Seeing AI App (Microsoft, 2023), Cloudsight’s TapTapSee (Cloudsight, 2023) and Google’s Lookout (Google, 2023). It is a complex application in computational terms, as it combines AI algorithms of different nature: object and person detection, optical character recognition (OCR), face recognition, text and synthetic voice generation. There are also data use and data privacy issues to be carefully addressed, e.g., regarding the management of captured facial images or the possibility of using extracted scene information for purposes other that assisting visually impaired people, such as targeted marketing.

We propose the use case card presented in Fig. 4. First, we focus on the visual modelling side. The key questions to ask are what is the AI system, which are the use cases within it we want to document and the main actors involved. The AI system can be easily identified as the scene narrator application. This system may have multiple use cases, ranging from classic software functionalities (e.g. installing the app, user registration, user logging, manage settings) to the more complex AI-based functionalities related to the scene narration part (i.e. object/person detection, OCR, face recognition, etc.). We decide to include within the system’s boundary only the uses cases directly linked to the scene narration functionality for the sake of clarity. Then, we reflect on a simplified interaction pipeline for the person with visual impairment to get a scene description, which is: opening the app on the smartphone \(\rightarrow\) taking a picture of the scene \(\rightarrow\) the system computes the scene description \(\rightarrow\) the person listens to the audio narration.

Within this pipeline, we realise that the whole AI core is contained under the computation of scene description phase. We therefore decide to introduce a describe scene use case as the principal one, which includes all AI-based functionalities (those with blue background colour). By modelling describe scene as the main use case with “include” dependencies to other AI functionalities, we simplify the documentation process to a single UML table.³ We additionally decide to show some non-AI use cases in the diagram to provide a complete and self-contained overview of the pipeline, namely: take scene photo and register familiar person. The register familiar person use case is particularly interesting, as it shows that certain persons (e.g. family, friends, caregivers) might be registered in the platform by the user, and thus subject to identification through face recognition. The last point to define in the diagram are the actors involved. The main actor is clearly the person with visual impairments as they are the one triggering the scene narration process. However, the modelling process has also identified other relevant actors, namely the (unknown) surrounding persons that might appear in the scene and the familiar faces that might eventually be present. Note that surrounding persons are a generalisation of familiar persons, and that the identify people use case “extends” the detect people one.

After the visual modelling exercise, we proceed to complete the table associated to the main use case describe scene. The context of use field provides an overview of pre-conditions and conditions of normal usage (e.g. the app is already installed in the smartphone, the primary actor wears goggles, s/he has already registered some familiar faces in the system), while scope delineates the specific functionality of the use case. This use case has a strong positive social impact, allowing for a better inclusion and social life for the visually impaired, and therefore contributes to two SDGs: good health and well-being and reduce inequalities. The use case is part of a software product and may not be considered a safety component, as it is meant to assist but not to fulfil a safety function. Interestingly, it has two application areas. The first one is social assistance, and the second one is remote biometric identification systems as it includes face recognition to identify familiar people. This is particularly important as the former is not considered a high-risk application area under the AI Act, while the latter does. Therefore, if the system’s provider prefers to bring the application to the market as a low-risk one, the face recognition functionality should be removed. The following fields are relatively straightforward to document, as they merely describe the main actors and course of actions within the use case. In our example, the main course field contains as steps the calls to the different AI algorithms. Extensions tackle problems that may arise, e.g. if the taken picture has poor quality, which are simply addressed with the failure protection mechanism of asking the person to retake the shot. Last, but of extreme importance, the open issues field allows the provider to clearly state that the application is conceived for ethical use. It stresses that the system is not intended for use by people who are not visually impaired, clarifies that data privacy is adequately treated (the provider does not keep a copy of taken scene images) and that under no circumstances will the provider engage in any marketing activities with the extracted information.

Through this example, we have shown that use case cards is a powerful, standardised methodology to document AI use cases. Beyond the goal of documentation, the process of filling in a use case card fosters reflections of the utmost importance about an AI system, such as its risk level, foreseeable misuses and failure protection mechanisms to put in place. Appendix B provides four additional use case cards involving different types of AI systems with varying levels of complexity, to provide the reader with a variety of illustrative examples.

Co-design, co-creation or participatory design refers to an approach where stakeholders come together, as equals, to conceptually develop solutions that respond to certain matters of concern (Zamenopoulos and Alexiou, 2018). As such, the co-design method aims to develop a solution “with” the target individuals/groups rather than “for” them. There has been an increasing trend in recent years towards greater inclusion of stakeholders in designing and carrying out research through the adoption of co-design methods (Nesbitt,Beleigoli,Du,Tirimacco and Clark, 2022). Given the multidisciplinary nature of our work, involving both policy and technical matters, we decided to take advantage of this methodology in this first design phase.

The co-design phase involved six participants. Two of them were EC policy experts with legal background, and having high involvement and proficiency in the AI Act. Three are EC scientific officers with proficiency in AI and medium-to-high knowledge on UML. It is important to note that, although these three experts have primarily a technical profile, they are involved in a daily basis in digital policy issues, including scientific advice related to the AI Act. Finally, we invited an external expert with high expertise in AI, and a proficiency background in UX and UML.

The three scientific officers and the external UML/UX expert prepared a three-hour tutorial on UML to kick off the first day. The tutorial started with the presentation of the UML standard (Object Management Group, 2017), with particular emphasis on the use case modelling part. Then, three exemplar AI use cases modelled in classic UML format (c.f. Figure 2) were presented for illustrative purposes: an affective music recommender, a driver monitoring system and a smart-shooting camera system.

After the tutorial, the six participants engaged in a guided discussion covering the following key points:Results can be summarised as follows. First, participants unanimously agreed on the high overlap between UML’s information elements and those required to document use cases under the AI Act (c.f. Table 1). Therefore the standard was considered fit for purpose. Participants however identified missing fields essential in the context of the AI Act and that should be added to the UML table, namely: (i) the type of product to which the AI system belongs; (ii) its application area(s); and (iii) whether the use case is a safety component of a product.

Participants raised important additional points. They mentioned different uses of the methodology, including the creation of a public repository of AI use cases, useful in the context of the registration process mentioned in Article 51 and Annex VIII–part II of the AI Act. This repository would be a valuable and usable tool to help companies –and more particularly SMEs, with more limited legal resources– identify the risk level of their AI systems: “use case cards would give companies a hook to go through the AI Act”. Authorities would also benefit from such repository, allowing them to “have a better overview of the landscape of existing AI systems” and “engage with companies to articulate bordercases”. Although not an explicit information requirement under the AI Act, given its human-centric nature, it was deemed interesting to include the link of each use case to Sustainable Development Goals (SDGs) which “would help keep track of AI-for-good applications”.

During the second workshop day, participants proceeded to the design of use case cards according to the findings identified the previous day. They first added the four missing fields (i.e. “type of product”, “application area(s)”, “is it a safety component?” and “SDGs”) to the UML table, and agreed on its final layout (e.g. colours, order/position of the different fields). Then they developed the list of products types (c.f. Appendix A’s Table 4) and application areas (c.f. Appendix A’s Table 5), carefully considering AI Act’s Annex II and III, respectively. Finally, participants concluded with a practical exercise, where they converted the three UML use cases in the tutorial to the new use case card format. They additionally implemented two new use case cards: the scene narrator one (presented in the previous section) and a student proctoring one. The new use case cards can be found in Appendix B. This final exercise allowed us to confirm the ease of use and implementation of the methodology, whose adaptation is “straightforward with respect to traditional UML” as confirmed by the UML/UX expert.

Once the first solid version of the use case cards was available, we conducted a questionnaire-based study to validate two main aspects. On one hand, those components referring to the clarity and complexity of the proposed approach, such as its learning curve, its level of detail and granularity, the importance of the visual components with respect to the table, as well as open questions regarding possible missing or unnecessary fields. On the other hand, those elements related to the level of contextualisation with respect to the AI Act, risk level assessment, requirements, etc. A summary of the questions is provided in Table 3. As can be seen, 9 questions were designed to have a possible answer aligned with a 5-point Likert scale, 2 questions allowed for a yes/no answer plus an elaboration if the answer was yes, and 2 questions were designed as completely open questions.

The online survey included an introduction with the description of the project, the main goals and procedure. Then a brief introduction of the main components of use cases modelled with UML was provided, followed by a short description of the proposed structure for the use case cards. After some demographic questions, the participants were provided with three exemplar use case cards. The first one corresponds to the scene narrator system previously presented in Sect. 3.3 (Fig. 4). Remaining two correspond to the driver attention monitoring system and the student proctoring tool presented in Appendix  B (Figs. 15 and 16, respectively).

We involved 11 participants (5 female, 5 male, 1 prefer not to say), 7 of whom had a technical background (computer scientists/engineers), with the rest having varied profiles including 1 legal expert, 1 social scientist and 1 mathematician. All of them had experience in trustworthy AI, science for policy, and the AI Act, as well as varying degrees of knowledge of UML. More specifically, their knowledge about the AI Act was self-assessed between “low” and “very high”, with mean \(M1=3.27\) (question 1, Fig. 6-left), whereas their knowledge about UML was self-assessed between “none” and high, with mean \(M2=2.36\) (question 2, Fig. 6-right). Since the use case cards are intended to be used in the context of the AI Act, it is coherent to validate them with participants with some knowledge of the AI Act. However, in principle, it is not strictly necessary to have knowledge of UML, so validation should incorporate participants with little or no knowledge of UML.

Figure 7 shows the histograms of answers for the questions related to the intrinsic features of the method. The difficulty to understand the three exemplar use case cards was assessed as “somewhat easy” (\(M3=4.09\)), the level of detail as “adequate” (\(M4=3.00\)), the importance of the UML diagram (the canvas) between “moderately important” and “important” (\(M5=3.45\)), and the learning curve at the midpoint between “moderately appropriate” and “quite appropriate” (\(M6=3.55\)). Regarding the question on missing fields (OQ1), 6 participants answered “no” and 5 “yes”. The suggestions provided by those who answered “yes” can be seen in Fig. 8. Most of them can be easily integrated into the “Open issues” field of the table. Other suggestions such as “more explicit contextualisation with the AI Act” or “other relevant EU policies” could be considered in future versions. And as for the question on possible dispensable fields (OQ2), \(73\%\) of the participants answered “no”, and \(27\%\) “yes”. As depicted in Fig. 9, there were three concerns, one referring to the type of product, another focusing on the Sustainable Development Goals (SDGs), and one comment on the UML diagram. First, it is important to note that the type of product has to be considered together with the specific area. Otherwise, we cannot obtain a detailed classification. On the other hand, we believe that asking about the SDGs can have positive effects on AI systems providers, as a way for them to consider whether or not their systems contribute to sustainable development. Finally, the importance of the UML diagram has been positively assessed by most of the participants in question 5.

Concerning the alignment of use case cards with the AI Act, the feedback from the participants is also very positive. For example, regarding the level of contextualisation with the AI Act (question 7, Fig. 10 left), the mean answer is between “somewhat” and “to a great extent”, with \(M7=4.18\). Regarding its utility to assess the risk-level (question 8, Fig. 10 right) the answers are between “very little” and “to a great extent”, with a mean value very close to “somewhat” (\(M8=3.82\)). And the general feedback from question 9 (Fig. 11) is mostly positive towards an agreement on its appropriateness to different AI Act specific aspects.

From the participants’ answers to open question 3, we highlight the following suggestions for other potential uses:Some of these answers echo our goal of proposing a methodology for documenting use cases for AI systems that is easy to understand by a non-expert audience. Other answers also point in the direction of a possible standard that could help with documentation needs, risk mitigation or conformity assessment.

However, there are also some issues raised by some participants in the last open question. In almost all cases, the feedback obtained refers in one way or another to a limited expertise on UML for documenting use cases. For example, some participants did not clearly understood the difference between the “AI system” and the “use cases”, including some confusion about the type of dependencies between the use cases. This issue is highly correlated with the lack of previous knowledge on UML. Difficulties in learning and using UML are well-known issues in the research and industry communities (Siau & P P., 2006)⁴. However, the benefits of UML have been empirically validated in multiple studies (Chaudron,Heijstek and Nugroho, 2012). While we recognise the potential initial difficulties of a wider audience in interpreting the UML canvas, we do not expect a major impact for AI providers, as UML is a de facto industry standard for modelling software systems. Moreover, as most of the participants emphasised, the table is the main element of the proposed approach, and its clarity has been validated regardless of prior knowledge about UML.