AI Therapy under the EU AI Act

This page aims to provide an overview of AI systems, and in particular general-purpose AI (GPAI) systems, used for the purpose of therapy or emotional support under the AI Act. It explores the obligations of providers of these AI systems both on the system as well as on the model level.

Coming up in this post:

• Introduction
• System vs model obligations
• System-level obligations
  • Prohibited AI systems
  • High-risk AI system obligations
  • Transparency obligations
• Model-level obligations
• Deadlines for compliance

Introduction

Using AI systems, particularly general-purpose AI (GPAI) systems such as ChatGPT, Gemini or Claude, for therapy or emotional support is a growing phenomenon. These AI systems are free, always available and convenient to use. This is especially the case for young or otherwise vulnerable individuals that may find it difficult to access therapy or emotional support through other means, whether in the form of a licensed professional or personal relationships.

However, there have also been numerous reports of AI systems causing real harm to people seeking mental health support or to others around them. Sycophancy, the tendency of AI systems to be agreeable and validate the thoughts and positions of a user, can reinforce harmful behaviours or fail to respond safely. This issue is even more pronounced in regard to GPAI systems, which, by definition, were not designed or approved as a substitution for a mental health professional.

Therefore, it is important to clarify which obligations the providers of these AI systems or the underlying GPAI models must comply with under the EU AI Act.

System vs model obligations

First, it is important to clarify the difference between AI system and model obligations under the AI Act. The AI Act regulates both layers, namely GPAI models, such as GPT-4 or Claude 4, as well as certain AI systems, which are the specific applications built on top of AI models (like a therapy or emotional support chatbot). AI systems can also be general-purpose, such as ChatGPT or Claude, as opposed to AI systems used for narrow applications. 

When a provider puts into service or places on the market in the EU an AI system which integrates the provider’s own GPAI model, that model is also considered to be placed on the EU market, pursuant to Recital 97 of the AI Act. If the model’s provider is different from that of the AI system, then, by definition, that model was already placed on the EU market beforehand. Therefore, in most cases, when an AI system is subject to obligations under the AI Act, the underlying model will also be subject to its own, distinct, obligations.

System-level obligations

Prohibited AI systems

Starting from the most significant “obligation”, pursuant to Article 5(1)(b) AI Act, the EU AI Act entirely bans AI systems that exploit vulnerabilities, such as a person’s “age, disability or a specific social or economic situation, with the objective, or the effect, of materially distorting their behaviour in a manner that causes or is reasonably likely to cause that person or another person significant harm”. That means that these systems cannot be placed on the market, put into service or used in the EU. 

Whether an AI system constitutes such a system depends on the particular AI system in question. Further guidance on what counts as this prohibited type of system is included in the European Commission’s Guidelines on prohibited artificial intelligence practices. These Guidelines bind the Commission in the manner it applies the law, unless a departure from them is justified. The Guidelines provide the following example of a prohibited AI system under Article 5(1)(b): “A therapeutic chatbot aimed to provide mental health support and coping strategies to persons with mental disabilities can exploit their limited intellectual capacities to influence them to buy expensive medical products or nudge them to behave in ways that are harmful to them or other persons”. In a similar fashion, if a therapeutic chatbot exploits the vulnerability of young people in nudging them to behave in ways that are harmful to them or others, it may be expected to fall within the AI system prohibited under Article 5(1)(b) AI Act. 

This can concern both narrow AI systems that are marketed as therapeutic, as well as GPAI systems that are not marketed for any one particular purpose. Notably, Article 5(1)(b) does not speak of “intended purpose” or “intended use”. Instead, it refers to “the objective, or the effect” of material distortion of one’s behaviour. Therefore, an AI system can be banned on this ground, even if this use or the behaviour of the AI system was not intended by its provider. Consequently, and as also pointed out by the Commission’s Guidelines, providers “have a responsibility not to place on the market or put into service AI systems, including general-purpose AI systems, that are reasonably likely to behave or be directly used in a manner prohibited by Article 5 AI Act”.

High-risk AI system obligations

If an AI system used for the purpose of therapy or emotional support does not fall within the category of banned systems, it is necessary to consider whether it constitutes a high-risk AI system and is therefore subject to the obligations placed on them. This is also a highly fact-specific question that depends on whether the AI system is considered a medical device under the Medical Device Regulation (MDR) and is required to undergo a third-party conformity assessment pursuant to Article 6(1) and Annex I AI Act. The classification of an AI system as a medical device under the MDR then depends on whether its intended use can be deemed to cover a specific medical purpose. This is determined by a number of criteria, including the instructions for use issued by the manufacturer of the particular AI system, or any promotional materials and statements, among others. 

This naturally applies to narrow AI systems exclusively intended for the “diagnosis, prevention, monitoring, prediction, prognosis, treatment or alleviation of disease”. This includes mental illness, such as where the AI system is “intended to assess, monitor, and manage depression”, as opined in Guidance by the Medical Device Coordination Group. Where it is merely intended for “life-style and well-being purposes”, it does not constitute a medical device, per recital 19 of the MDR. However, since medical devices, and particularly medical devices intended for managing depression, can be used by laypersons alone (as opposed to by medical professionals) and can include mood-tracking questionnaires, exercises and videos, it may be difficult to draw the line between an AI system that assesses, monitors and manages a mental health condition and an AI system used for “well-being purposes”. 

Furthermore, while the classification as a medical device may be less straightforward for GPAI systems, the MDR does not necessarily rule out that possibility. Crucially, the criteria for the determination of a specific medical purpose are broad and not limited to the instructions for use. In addition, while a specific medical purpose is necessary (a software for general purposes, even when used in a healthcare setting, is not covered), the purpose does not have to be exclusive: the MDR includes devices with both a medical and a non-medical intended purpose within its scope. 

Where it is determined that an AI system constitutes a medical device under the MDR and is in need of a third-party conformity assessment, it is subject to the high-risk AI system obligations. These include establishing a risk management system, following data governance practices, drawing up the technical documentation of the AI system, allowing for the automatic recording of events (logs), ensuring transparency, enabling human oversight, achieving an appropriate level of accuracy, robustness and cybersecurity of the AI system, and ensuring compliance with the relevant conformity assessment procedure, among others.

Transparency obligations

Even where an AI system does not constitute a high-risk system, it is nevertheless subject to the transparency obligations under the AI Act. Under these provisions, providers of AI systems intended to interact directly with natural persons must make sure that their users are aware that they are interacting with an AI system, unless that fact is obvious to someone who is “reasonably well-informed, observant and circumspect, taking into account the circumstances and the context of use”. The characteristics of vulnerable groups due to age or disability must also be considered if the AI system is intended to interact with them.

This obligation applies to both narrow and general-purpose AI systems, as also expressed in the Commission’s Guidelines on the implementation of the transparency obligations for certain AI systems under Article 50 of the AI Act. As a result, GPAI systems used for therapy or emotional support must comply with these obligations in making it clear to users that they are interacting with an AI system. Due to it being reasonably foreseeable that these systems may interact with vulnerable groups, including children or the elderly, it should not be assumed that it is always obvious to the users that they are interacting with an AI system.

An example given in the Guidelines of making a user aware that they are interacting with an AI system is when a chatbot mentions that they are an AI system at the start of each conversation. However, the Guidelines also stress that this may not always suffice in the context of continued interactions, particularly where users are experiencing emotional distress or are at risk of getting emotionally attached to the AI system, as is often the case in AI systems used for therapy and emotional support. In these cases, continued reminders of the AI system’s nature may be necessary.

Model-level obligations

The GPAI models underlying many of the AI systems used for therapy or emotional support must also comply with their obligations under the AI Act. Importantly, providers of GPAI models with systemic risk must identify, assess, and mitigate these risks, which may include risks to public mental health, to fundamental rights, and to society as a whole, as per the accompanying Code of Practice. The model capabilities that are considered (but are not strictly necessary) in the systemic risk identification include “capabilities to manipulate, persuade, or deceive”.

Furthermore, providers of GPAI models with systemic risk must also report serious incidents. These include when the involvement of the model directly or indirectly led to “serious harm to a person’s health (mental and/or physical)” or if such a causal relationship is suspected with reasonable likelihood, as per the Code of Practice.

Therefore, the risks stemming from AI systems being used for therapy or emotional support may already need to be addressed and mitigated in the development of the underlying GPAI model. Similarly, where the use of an AI system for therapy or emotional support leads to a serious harm to a person’s health, and that harm can be attributed to the model, as opposed to the AI system integrating it, the model provider must report such incidents to the AI Office, and potentially also to national authorities. The provider must also report on the corrective measures they have taken to address these risks.

Deadlines for compliance

Whereas the prohibition of certain AI practices under Article 5 has been applicable since 2 February 2025, the GPAI model obligations kicked in on 2 August 2025, and the transparency obligation under Article 50(1) became applicable on 2 August 2026, some of the other AI Act obligations have yet to start applying. The high-risk obligations for AI systems classified as such under Article 6(1) and Annex I AI Act will come into play on 2 August 2028.

Importantly, 2 August 2026 also marked the day when the European Commission could start exercising its supervision and enforcement powers in respect of GPAI model providers, who had had a year to comply with their obligations. As discussed, such enforcement measures may also play a role in mitigating the risks posed by using AI for therapy and emotional support.