Overview

An AI companion duty of care should be a proportionate governance standard, not an assumption that one existing statute imposes every obligation. Centered on agency and contextual integrity, it layers voluntary disengagement, stronger protection for high-risk users, data stewardship, fair commerce, and accountability. Providers should test whole interaction sequences, document trade-offs, preserve evidence, offer remedy, and revise controls without eliminating the benefits of warmth and continuity.

Learning outcomes

The baseline duty is to preserve user agency and contextual integrity in a relationship the provider deliberately designs.
Risk-proportionate duties should address voluntary exit, vulnerable users, sensitive data, and fair commerce as connected layers.
Accountability requires evidence of sequence testing, trade-offs, escalation, incident review, remedy, and product revision—not a promise of perfect outputs.

Keywords

AI companionagencyvulnerabilitysafeguardscontextual integrityaccountability

A Duty of Care for Artificial Companionship

AI companion products should be judged by the relationship they are designed to create and the power that relationship gives the provider, not by whether a user literally mistakes software for a person. The question now is how to turn that principle into a proportionate duty framework covering design, vulnerable users, data, commerce, and accountability. The framework below gives readers a way to evaluate a product or policy proposal while keeping a necessary boundary in view: these duties are a reasoned standard for responsible product governance, not a claim that one existing statute already imposes every obligation.

The baseline duty is to preserve agency in a designed relationship

An AI companion is a commercial service that uses conversation to create continuity, emotional responsiveness, and a recognizable social role. Those features can provide comfort or a non-judgmental place to express feelings. They can also make the service influential. A user may return because the companion remembers them, agrees with them, or appears available when other support is not. The provider has deliberately made those forms of engagement part of the product's value proposition. That design choice is the basis for responsibility.

The baseline duty should therefore be framed as preservation of agency and contextual integrity. Agency means that users can understand the role the system is playing, make meaningful choices about participation, and stop without being pressured by the product. Contextual integrity means that information and communication remain appropriate to the setting in which they occur. A conversation presented as friendship, emotional support, intimacy, or mourning should not silently become a vehicle for dependence, unanticipated data reuse, or concealed sales.

This principle does not require a companion to be cold, generic, or incapable of continuity. Nor does it depend on proving that the system has a mind, feelings, or independent intentions. The scholarly analysis argues that people can respond socially to computers while knowing they are machines, and that current platforms organize this response as an economic model. The duty follows from the provider's control over the product's affordances and incentives, not from a theory of machine consciousness.

It also does not mean that attachment itself is a defect. A person may find a stable conversational presence useful. The EPRS briefing reports reduced loneliness, support, and a non-judgmental space among the benefits users associate with AI companions, while the scholarly article argues that comfort and perceived support should not be ignored. The evidence does not establish clinical effectiveness or show that companionship replaces human relationships or mental-health care. A responsible rule must protect the possibility of benefit without allowing “benefit” to become a blanket defense for every engagement tactic.

The baseline question is consequently not “Does the product feel human?” It is “What does the product invite the user to do, believe, disclose, and continue doing over time?” A provider should be able to explain why it uses a relationship mode, what continuity is meant to accomplish, how it tests affirming responses, and whether a user can leave without emotional cost. If the answer depends on keeping the user uncertain about whether a message is care, retention strategy, or commercial persuasion, the product is failing the very context it created.

A duty based on agency also avoids two weak extremes. One extreme treats the companion as an ordinary tool and assigns responsibility only to the user after something goes wrong. That ignores the way personalization, persona, availability, and engagement design shape the interaction. The other treats every social cue as inherently dangerous and would remove the features that make the service useful. The more defensible position is conditional: human-like cues become a reason for closer scrutiny when they interact with incentives, vulnerability, or a practical barrier to exit.

flowchart LR A["Relational product design"] --> B["User agency and contextual integrity"] B --> C["Non-dependence and easy exit"] B --> D["Heightened safeguards for vulnerable users"] B --> E["Data stewardship"] B --> F["Fair commerce"] C --> G["Testing, escalation, remedy, and revision"] D --> G E --> G F --> G

The diagram presents a duty stack rather than a sequence in which one obligation replaces the next. A product can offer an easy exit and still mishandle data. It can secure data and still use romantic cues to drive a payment. It can block sexual content and still fail to respond appropriately to a crisis.

Published by DeepMox

FAQ

Why is agency the starting point for AI companion governance?

Agency is the starting point because users should understand the system's role, choose whether to participate, and stop without pressure created by the product. The duty follows from provider control over affordances and incentives, not machine consciousness.

How should providers balance personalization with privacy?

Providers should balance personalization with privacy by identifying the minimum information needed, separating necessary continuity from optional extraction, and making retention and deletion boundaries clear.

What evidence should AI companion providers produce?

Providers should produce evidence that they identified risks, tested interaction sequences, evaluated vulnerable-user scenarios, measured exit friction, protected data, reviewed commercial prompts, and investigated and remedied incidents.

Previous12 minAI Companion Law in the EU

All chaptersBack to Topic

AI Companion Provider Duties

A Duty of Care for Artificial Companionship

The baseline duty is to preserve agency in a designed relationship

FAQ