Bringing Trustworthy Robo-Advisers into the Insurance Distribution Process: An Accountability Perspective

First of all, accountability can be roughly split into three dimensions: ex ante, ex post and procedural dimension.

From the ex ante perspective, accountability means that the law must explicitly clarify the responsibility and duty of relevant parties. The ex ante dimension of accountability provides certainty to all relevant parties that the measures that they must comply with. In other words, they are informed of the possible consequences if they fail to do so. These predefined rules serve with a deterrence function, incentivizing all relevant parties to behave properly.

From the ex post perspective, accountability means that the law must clearly determine which party shall bear the liability when harmful consequences occurred. The ex post dimension of accountability discloses the specific parties that shall undertake the accident costs of harmful consequences. In addition, the ex post dimension of accountability is crucial for compensation, making sure that victims could recover their loss fully and effectively.

In practice, whether or not a person is accountable is not self-evident. The claimant needs to take measures to recognize the accountable persons and prove that they are accountable by meeting other essential requirements, such as fault, defect, causal link, damage, etc. Therefore, there is a third perspective when conceptualizing accountability, which is the procedural one. The procedural dimension of accountability aims to provide the necessary degree of transparency in support of the first two dimensions. The meaning of transparency composes multiple facets, including traceability, answerability, explainability, etc. Therefore, the procedural dimension of accountability is important to facilitate trustworthy AI, since it addresses crucial practical issues, such as who to which extent contributed to damage. ¹³

Accountable person is an indispensable angle when decoupling the concept of accountability, as it is the target of relevant mechanisms. Accountability must be assigned to specific persons beforehand. In the context of AI, in principle, all parties including natural persons and legal persons along the supply chain may be accountable under certain conditions.

Manufacturers determine the technical specifications of an AI system. Accountability shall cover manufacturers, so that they will be properly incentivized when designing and manufacturing AI systems. ¹⁴ They will also be obliged to take necessary conformity assessment to ensure that the product has meet all legal requirements before being placed on the market.

After an AI system is put into circulation, importers and various distributors take the role in making the AI system available on the market and spreading the AI system to a broader range of regions. Although their activity does not directly determine the function or performance of an AI system, they can facilitate the process of distribution, which thereby indirectly influences the scale of accidents. In this digital age, online platforms take a crucial role in facilitating the distribution of AI-enabled applications in various manners, such as cloud services and software as a service (SaaS). Therefore, it is necessary to hold online platforms accountable as well. ¹⁵

After circulation, the next stop for an AI system is to be deployed in a certain context. In reality, depending on the concrete application of an AI system, the typical deployers may include educational institutes, healthcare organization, public authorities and many others. It is noteworthy that although the whole operation and monitoring of an AI system is normally under the umbrella of deployers, they need to assign specific natural persons (i.e., user) to operate the AI system. In practice, users can be teachers, doctors and civil servants, who take the concrete control over an AI system. To ensure that AI systems are deployed for intended purposes and operated as expected, deployers including the assigned users shall also be considered as accountable persons.

Users are indeed not the end point in the spectrum of accountable persons. The use of AI can influence a variety of parties (e.g., consumers, students and patients). In principle, these parties are often not considered as accountable parties, since they are impacted persons. ¹⁶ However, in a bilateral accident, where the activity of impacted persons can also influence the likelihood and severity of damage, impacted persons shall also be motivated to behave properly. ¹⁷ From the accountability perspective, this normally means that impacted persons shall also take necessary care and their fault may reduce or even disallow the liability of others.

Accountability must be secured throughout the lifecycle of an AI system. As the HLEG Ethical Guidelines points out, accountability shall cover the stages ‘both before and after their development, deployment and use’. From this perspective, four facets are distinguished when designing mechanisms for accountability.

The first stage is ‘before development’, which covers the stage from design to manufacturing. This is the stage that sets up the architecture and technical specification of an AI system.

The next stage covers the activities between development and deployment. In this stage, an AI system is ready to be or has already been placed on the market. By the end of this stage, the AI system is physically under the control of a deployer, even if the manufacturers may still maintain some control (e.g., through software updates). ¹⁸ In this regard, accountability should cover both manufacturers as well as deployers.

The third stage spans from deployment to use. Normally, this refers to the assignment of human oversight to the appropriate people within the deployer’s organization. This person shall in principle be technically and knowledgably competent in terms of operating the AI system under the instructions of use provided by the manufacturer.

The last stage starts from the moment when an AI system is put into use. It specifically refers to the interaction between impacted persons and the AI system. This process also includes the necessary updates provided by the upstream parties.

To summarize, the lifetime of an AI system can be split into four concrete stages. The discussion in this paragraph explains that varying activities featured in each stage. This also indicates the necessity of developing diverse mechanisms for different stages, which is the central topic of the next part.

The last, but not the least, dimension of understanding accountable AI lies in the mechanisms that are adopted for the purpose of implementing accountability. Traditionally, a variety of mechanisms have been introduced prior to the appearance of AI systems. Risk regulation provides the requirement that a product must meet before they are allowed to be placed on the market. It also lays down the obligations that parties must comply with. Besides risk regulation, liability is also a crucial mechanism in terms of allocation liability and compensation.

In recent years, new mechanisms are developed in order to catch up the challenges posed by new technologies. For example, meta-regulation (e.g., self-regulation, impact assessment and standardization) provides the essential flexibility to strike accountability and other social goals, such innovation. ¹⁹ In addition, the law starts to require manufacturers to embed ethical values (e.g., accountability and transparency) into the designing and manufacturing process of a product. This ‘by design’ insight is considered an important approach toward accountability issue. ²⁰ Education, e.g., digital literacy, is another arising mechanism that is important for properly materializing accountability. ²¹ In the era of AI, literacy means that accountable persons (e.g., users) shall be sufficiently educated to the extent that they understand how to properly comply with the obligations.

In the EU, sector-specific regulations are the main mechanisms to address the risk occurred in particular activities. In the sphere of insurance distribution, Insurance Distribution Directive (IDD) ²² provides harmonized requirements that Member States must transform when regulating the activities of insurance distribution. As insurance technologies (e.g., robo-advisers) are constantly introduced to insurance distribution, the questions arise are (1) whether such innovative applications fall under the regulatory framework of the IDD, and (2) whether the IDD is the right mechanism to regulate the risk of these insurance technologies.

For the first question, as literature explained, the application of robo-advisers will fall under the IDD without any doubt, since the provision of insurance advice regardless its form (by humans or by machines) is clearly a sort of insurance distribution. ²³ Nevertheless, the regulatory framework provided by the IDD is largely principle-based, meaning that concrete regulatory rules specifically designed to address the challenges caused by robo-advisers are missing.

Then it comes to the second question, i.e., whether the IDD is the right mechanism to introduce new rules for robo-advisers. Regarding this issue, authorities at the EU level (e.g., EIOPA) contends that the regulation for AI-related applications (including robo-advisers) in the insurance sector shall still be sector-specific. ²⁴ This indicates the confidence of insurance authorities to rely on existing insurance regulations to resolve the challenges raised by emerging technological challenges. Nevertheless, the EU ultimately determined to introduce a horizontal approach to regulate the risk of all AI applications.

Therefore, in the aftermath of the AI Act, robo-advisers should comply with the IDD as well as the horizontal AI Act. They are complementary each other, which has significant implications for insurance distributors, as they need to not only respect the obligations as deployers of AI system laid down in the AI Act but also follow the obligations within the IDD as insurance distributors.

In April 2021, the European Commission introduced its proposal for a comprehensive regulatory framework specifically for AI applications. ²⁵ After three years discussion, the final version of the EU AI Act came into force as of 1 August 2024. ²⁶ The EU AI Act followed a so-called risk-based approach that has been widely adopted in product safety regulations. ²⁷ Depending on the risk-profiles of their applications, all AI systems are distinguished into the four groups which are subject to different regulatory requirements.

First of all, some AI applications that result in serious fundamental rights concerns are considered with unacceptable risks. They are banned according to the AI Act. ²⁸ What is more, a wide range of AI systems are categorized as high-risk AI applications, as they may pose substantial risks to safety and fundamental rights. High-risk AI systems are in principle allowed for circulation, but they must firstly meet specific mandatory safety requirements (laid down by the AI Act as well as other sector-specific regulations, if there is any) and have them verified through necessary conformity assessment procedures. ²⁹ In addition, the AI Act laid down obligations for the operators of high-risk AI systems. ³⁰ Moreover, the AI Act set up transparency obligations for the providers or deployers of certain AI applications. ³¹ The objective is requiring their operators to take proper transparency measures to ensure that the impacted persons can make an informed decision. Last but not least, for all other AI applications, the AI Act does not provide any additional requirements or obligations for them or their operators. They can be placed directly on the market, once they meet the mandatory requirements from relevant sector-specific regulations. Besides the regulatory measures toward various AI systems, the AI Act also extends to regulating General Purpose AI models by obliging their operators to respect certain obligations. ³²

To summarize, the AI Act provides a comprehensive regulatory framework for all kinds of AI applications. Insurance technologies, such as robo-advisers, are thereby not isolated from this framework.

As analyzed, robo-advisers are in nature developed to take a role as professional insurance advisers. They may engage in a variety of tasks, including communication, risk-differentiation and advice provision. Therefore, robo-advisers may fall under different risk profiles depending on their concrete practices. ³³

First and foremost, robo-advisers may generate unacceptable risks, if they are designed with subliminal and deceptive techniques and has the potential to result in harmful consequences. For robo-advisers that are used for supporting insurance distribution, it is highly likely that they more or less employ manipulative techniques in its design for the purpose of persuading customers to make decisions that ultimately benefit insurance distributors. Manipulative techniques, as the AI Act interprets, will only be prohibited if they (are likely to) cause significant harms. ³⁴ The concept of significant harm under the AI Act is complementary to unfair commercial practices, meaning that any commercial practices, irrespective of AI-driven or not, shall be prohibited under all circumstances, as long as it generates economic or financial harms to consumers. Therefore, the standard of ‘significant harm’ is very low, which is mostly linked to the disadvantage of the economic benefits of consumers. Robo-advisers have the potential to lead consumers to make wrong decisions and caused economic losses by purchasing a coverage that they do not need. If this falls under the scope of significant harm defined by the AI Act, then the application of rob-advisers would be substantially limited in terms of recommending insurance products to customers. In the aftermath of the AI Act, two clarifications are important: firstly, the distinction between manipulative techniques and others should be explicitly interpreted; secondly, the meaning of significant harm should be clarified, especially in the context of financial sectors.

Secondly, robo-advisers with a function of using biometric information to infer certain protected attributes, may be prohibited as well. For instance, if a robo-adviser embeds facial recognition technology (FRT), uses the data collected from FRT to infer the race of their customers and then categorizes the risk-level of customers based on race, such a robo-adviser will be prohibited. ³⁵ According to the AI Act, the ‘certain protected attributes’ that are prohibited for inference are limited to race, political opinions, trade union membership, religious or philosophical beliefs, sex life and sexual orientation. Therefore, if a robo-advisers uses biometric data to infer protected attributes other than the aforementioned ones, they will not be prohibited. Instead, they may fall under the scope of high-risk AI applications and be subject to a different framework of accountability.

Thirdly, robo-advisers for insurance distribution may be categorized as a high-risk AI application. To begin with, biometric-based robo-advisers may be defined with a high risk. In particular, in the case that a robo-adviser is used to categorize the risk of insureds based on their biometric information or it is able to infer their emotional status, it will be considered as a high-risk application. ³⁶ In addition, robo-advisors that are used for assessing and pricing the risk of natural persons in the case of life and health insurance are also defined as with a high-risk. ³⁷ Accordingly, from an accountability perspective, additional regulatory requirements and obligations shall be fulfilled before a high-risk robo-adviser is allowed for circulation.

Fourthly, if robo-advisers are developed to facilitate the communication between insurance distributors and customers (e.g., chatbots), the AI Act will also apply to them by exerting a specific transparency obligation onto their providers. In this regard, providers should make sure that robo-advisers are developed in a manner that impacted natural persons are sufficiently informed that they are interacting with AI rather than a human. ³⁸

To summarize, the discussion in this part is intended to clarify the multiple facets of robo-advisers. Depending on the concrete techniques that they employ and the purposes that they intend to apply for, robo-advisers may fall under different risk categories outlined by the AI Act. As a result, they themselves and their operators shall also be subject to particular requirements and obligations. In essence, the risk-based approach laid down by the AI Act is a pertinent reflection of the different components of accountability: different mechanisms (e.g., regulatory requirement and obligation) shall be developed to relevant accountable persons (e.g., providers and deployers) throughout the lifetime of robo-advisers depending on their risk profiles.

Having mapped out robo-advisers under the risk-based approach introduced by the AI Act, it is clear that high-risk applications are the main target under the new regulatory framework. As aforementioned, robo-advisers that are used for insurance distribution purposes are considered with a high-risk in several scenarios, if, e.g., they employ biometric systems as well as they are used for assessing and pricing the risk of natural persons in the case of life or health insurance. From the accountability perspective, this means that the operators of robo-advisers are subject to new obligations beyond the existing insurance regulations (e.g., the IDD). Therefore, it is of great importance to unveil the specific regulatory requirements and obligations for high-risk robo-advisers.

The AI Act established new essential legal requirements for high-risk AI systems. According to Section 2 of Chapter III, the requirements that a high-risk AI system must meet include risk management system, data and data governance, technical documentation, record-keeping, transparency and provision of information to deployers, human oversight and accuracy. In practice, the proof of meeting such legal requirements must be supported on the technical side. This requires providers of high-risk AI systems to use reliable technical specifications (e.g., standards) to explain how their products have complied with the legal requirements. ³⁹ Furthermore, they need to follow the necessary conformity assessment procedures to verify their compliance.

The demonstration of compliance in line with harmonized standards can significantly reduce the effort of verifying conformity, because it will lead to a presumption of meeting relevant legal requirements. Nevertheless, as harmonized standards are instruments that the Commission requires European standardization organizations to draft, in reality it may take a long time before they are established. This is also the challenge for standardizing AI products. ⁴⁰ At the moment, the European standardization organizations are busy with developing harmonized standards regarding all legal requirements set out in the AI Act. ⁴¹ It, however, is unknown whether such standards match the expectation within the insurance sector.

The AI Act also sets up obligations respectively for providers and deployers of high-risk AI systems. Noticeably, the obligations for deployers will have significant impact on insurance distributors from the accountability perspective. According to Article 26, insurance companies that adopt robo-advisers with high risks shall respect a wide range of obligations. First of all, insurance companies shall assign human oversight to competent natural persons within the organization to operate and supervise robo-advisers. Such persons must also be provided with necessary training in support of their supervision. What is more, to the extent that the input data of a high-risk robo-adviser is under the control of insurance companies, they must ensure that the data is representative for the purpose that high-risk robo-adviser is intended to use. Moreover, insurance company is the ultimate party to monitor the proper operation of robo-advisers. Their operation shall follow the instructions of use and they shall notify providers and competent authorities in the cases where robo-advisers present risk to safety and fundamental risks and where a serious accident resulted. Last but not least, insurance companies must make sure the logs are automatically kept when logs are under their control.

Besides the aforementioned obligations for deployers, the AI Act also obliges in Article 27 certain deployers of high-risk AI systems to conduct a so-called fundamental rights impact assessment (FRIA). ⁴² It is noteworthy that the FRIA does not apply to every deployer of a high-risk AI system. Only two types of deployers are covered: public or private bodies providing public services and certain financial service providers. According to Article 27, deployers of AI systems that are used for assessing and pricing the risk of nature persons in the case of life or health insurance will be subject to FRIA. Therefore, if a robo-adviser holds such a capability, their deployers must be prepared to conduct the FRIA.

The discussion so far unveils how the AI Act utilizes different types of mechanisms to necessitate the accountability of AI systems. The mechanism contains not only traditional legal requirements and obligations but also emerging meta-regulations, such as standards and risk assessment. Noticeably, the AI Act also stresses the role of AI literacy in promoting trustworthy AI.

On this basis, Article 4 of the AI Act further clarifies that providers and deployers of AI systems shall take measures to make sure that their staff are employed with a sufficient level of literacy in terms of operating the AI system. To achieve this objective, all relevant methods, such as education and professional training, shall be offered. AI literacy is crucial for accountability, because it prepares the necessary level of knowledge and techniques to help staff to better understand their responsibility and obligation in the course of operating and monitoring an AI system.

Unlike other mechanisms, AI literacy in the AI Act is very much principle-based. This means that the Member States shall further develop their own AI literacy rules. Also, industries and companies can introduce concrete guidelines to implement AI literacy. When implementing the AI Act, insurance authorities shall work closely with insurance industry to set out sufficient guidance on AI literacy in the insurance sector.

According to the 1985 product liability directive (i.e., 1985 PLD ⁴⁵), producers shall be liable for the damage caused by the defect of their products. ⁴⁶ With regard to the liability for the providers of robo-advisers, it is not surprising that people may question whether providers could be comparable to the producers of normal products. From a positive legal analysis, however, there are some barriers to justify the applicability of product liability for the providers of robo-advisers.

First of all, a robo-adviser may not be qualified as the ‘product’ defined in the 1985 PLD. Product in the 1985 PLD refers to movable products. ⁴⁷ The CJEU further clarified that movable goods should refer to tangible goods. ⁴⁸ Robo-advisers per se, however, are standalone AI applications, meaning that they look nowhere like the traditional products.

Secondly, the damage caused by robo-advisers may not be compensable under product liability. As 1985 PLD clearly listed, the interests protected by product liability include a person’s life, physical integrity as well as their property. ⁴⁹ Other interests, such as mental health as well as other fundamental rights (e.g., fairness, non-discrimination, personal data) are thereby not protected by product liability. Robo-advisers, however, are remote from causing damage to the aforementioned protected interests. The main consequences of using robo-advisers are pure economic losses or non-material damage resulting from violating fundamental rights. Therefore, robo-advisers are not well covered by the existing product liability framework, if we delve into the damage caused by them.

Thirdly, as other AI applications, there are procedural obstacles preventing victims from smoothly claiming the liability of providers. When establishing the liability of providers, victims must prove the existence of defectiveness and that there is a causal link between the defectiveness and the damage. The proof of defectiveness and causation, however, is not an easy task for victims, because they may lack the necessary information as well as the capability to do so. ⁵⁰

The discussion in this part so far disclosed the substantive and procedural obstacles of holding the providers of robo-advisers liable under product liability. On 12 March 2024, the European Parliament adopted a proposal to revise 1985 PLD. ⁵¹ The revised PLD (rPLD) was finally published in the Official Journal on 18 November 2024. ⁵² The rPLD adapted product liability rules to the new challenges caused by digitalization. In the aftermath of rPLD, the question remains: to what extent can the providers of robo-advisers be held liable within the rPLD?

First of all, the rPLD significantly extends the scope of product to intangible goods, such as software, AI and 3D printing. ⁵³ By doing so, robo-advisers will be clearly covered by the rPLD.

What is more, the rPLD provides some concrete rules to help victims to prove defectiveness and causal link. Victims can request the defendant to disclose relevant evidence in the case that they have ‘presented facts and evidence sufficient to support the plausibility of the claim for compensation’. ⁵⁴ When certain conditions are met, defectiveness and causal link can also be presumed. ⁵⁵

The challenging issue of holding the provider of robo-advisers under the rPLD lies in the scope of compensable damage. Although the rPLD extends the scope of personal injury to ‘medically recognized damage to psychological health’, the scope of protected interests is not significantly expanded, e.g., to other fundamental rights. ⁵⁶ Therefore, in the end, whether the damage caused by the defect of a robo-adviser can be held accountable under the rPLD is very largely determined by the type of damage caused by it. ⁵⁷ If robo-advisers caused one of the following damage, i.e., death, personal injury (including medically recognized damage to psychological health) and property, then their providers might be held liable given the successful demonstration of other factors.

To summarize, there is a large chance that the damage caused by robo-advisers may not be recoverable under the product liability regime. In order to be compensated, the harm must fall under the scope damage defined by the rPLD and it must be caused by the defectiveness of the robo-advisors. If these conditions are not met, for example, the damage was caused by the fault of manufacturers rather than defectiveness of the robo-advisor, the rPLD may not apply. As a result, holding providers accountable would be less applicable via product liability regime. In this scenario, fault-based rules are an important complementary regime for holding the providers of robo-advisers accountable. ⁵⁸

Besides providers, the activity of deployers can also significantly contribute to harmful consequences due to their carelessness in operation and supervision. Holding deployers liable will incentivize them to take proper precautionary measures and reduce accidents. Traditionally, deployers are subject to fault-based liability, meaning that they will be liable where they fail to take the due level of care as expected. The failure to take due care level can be verified from both subjective and objective perspective: from the subjective perspective, they are negligent in triggering harmful consequences through their activities; from the objective perspective, it means that they failed to comply with regulatory requirements.

Several elements need to be successfully demonstrated when relying on fault-based liability to hold deployers of robo-advisers accountable. First and foremost, the claimant must prove that the deployer had a fault. As aforementioned, the proof of fault can be established from both subjective and objective dimensions. One visible benchmark of the fault of deployers are the failure of complying with the obligations of deployers listed in the Article 26 and 27 in the AI Act. If an insurance distributor failed to comply with one or these several obligations (e.g., proper assignment of human oversight and data governance), it can be an important source of demonstrating their fault. Secondly, the claimant must prove the damage. The concept of damage can be defined very differently from one Member States to another. ⁵⁹ The claimant needs to check the extent to which the damage suffered by them is compensable in a given legal regime. Thirdly, the claimant must prove the causal link between the fault of the deployer and the damage.

The elements of fault liability are not self-evident in a given case. Claimants shall take the burden of proof toward these elements. However, in reality they are very likely to encounter difficulties in establishing these elements. The products and services empowered by AI and digital technologies will even worsen this situation, considering the opacity of its operation. Since 2022, the European Commission started with a proposal to set up the liability of AI systems (i.e. AI Liability Directive, AILD). ⁶⁰ Considering the huge divergence of the Member States in terms of the substantive issues, such as concept of fault, causation and damage, the AILD proposal does not intend to provide a harmonized liability regime at the EU level at the moment. Alternatively, the main purpose of the AILD is requiring the Member States to ease their procedures to facilitate the claims that are based on non-contractual fault liability. ⁶¹ In specific, the AILD requests national courts to require defendants to disclose the evidence to facilitate their claims. ⁶² In addition, the causal link between fault and damage shall be presumed when certain conditions are met. ⁶³ Therefore, the AILD aims to set up minimal harmonized rules for the Member States. ⁶⁴ The Member States can of course introduce or maintain their higher requirements in their national laws.

The proposal for the AILD, however, was criticized concerning its clarity and added-value. ⁶⁵ The Member States like France also questioned the necessity of this directive. Under this background, the AILD was withdrawn by the Commission in February 2025. After the withdrawal, it is expected that in the following years national-based rules will play a crucial role in reshaping the liability rules of AI.

To summarize, the liability for the deployer of robo-advisers is very largely shaped by national laws. The recent proposal for AILD shows the determination of the EU authorities to establish harmonized rules. Nevertheless, although it only aimed for a minimal harmonization and only targeted procedural issues rather than substantive ones, the AILD was ultimately withdrawn due to strong objection from the Member States. There is still a long way to explore the liability mechanism in the future.