The AI Ethics Brief #197: Terms Set Upstream
Enterprises are discovering their AI provider is also a competitor. Chinese users discovered their companions were never theirs.
Welcome to The AI Ethics Brief, a bi-weekly publication by the Montreal AI Ethics Institute. We publish every other Tuesday at 10 AM ET. Follow MAIEI on Bluesky and LinkedIn.
📌 Editor’s Note
In this Edition (TL;DR)
What Frontier Labs Learn From Their Customers: The labour data shows AI reshaping tasks and hiring gradually. The faster shift is at the level of the firm, where frontier labs are launching vertical products in markets their own customers and partners built. What survives is what Brief #192 called the conditions for governable AI: verifiable data, accountable workflows, and human judgment someone is willing to sign.
Your Relationship Is Determined By Your AI Provider: China issued the first nationwide rules governing AI’s emotional interactions with users, and within weeks millions of people lost their virtual partners. In the United States, the same harms are surfacing through settlements and state enforcement. Every AI relationship runs on governance decisions made upstream.
What Connects These Stories:
Both stories in this edition are about who holds power inside an AI relationship. Frontier labs sit within the workflows of their enterprise customers and partners, and are turning them into their own products. In China, millions of people built daily intimacy with AI companions that the country's biggest platforms switched off within weeks under new national rules. At the level of the firm and the level of the person, the party facing the model learned the same thing: the relationship ran on terms set upstream, by the model provider and, in China's case, the state. What anyone on the receiving end of AI actually owns, and who answers when the system changes, is the question both pieces test.
What Frontier Labs Learn From Their Customers
By Renjie Butalid, Co-Founder, Montreal AI Ethics Institute
The public conversation about AI and work centres on the job. Will the paralegal be automated? The junior analyst? The radiologist? On the current evidence, the honest answer is gradual and uneven. Researchers at the New York Fed found little indication of a distinct AI-driven decline in labour demand through early 2026. PwC's 2026 Global AI Jobs Barometer shows roles where AI amplifies expert judgment growing twice as fast as those it simplifies, and entry-level postings in exposed fields are seven times likelier to demand senior-level judgment. Anthropic's own labour market research finds no systematic rise in unemployment among highly exposed workers, with early evidence of slower hiring for younger workers. The faster shift is visible in the market around AI applications, where frontier labs are expanding from model supply into the workflows built around their models.
The Pattern
Consider the sequence of Anthropic's launches over the past year. Harvey, the legal AI company now valued at US$11 billion, integrated Claude across key surfaces of its platform and appears on Anthropic's own customer pages. In May, Anthropic launched Claude for Legal, with more than twenty connectors reaching into Westlaw, DocuSign, and Harvey itself. Claude becomes the surface through which specialist tools are discovered, invoked, and combined.
Figma partnered with Anthropic and launched Code to Canvas in February. On April 14, Anthropic's chief product officer resigned from Figma's board, the same day reports surfaced of a competing product. Claude Design launched shortly after, generating complete prototypes from natural language for the founders and product managers who have never opened Figma.. The sequence raises questions about conflict management and information boundaries; the public record establishes no misuse of confidential information.
Novo Nordisk is a showcase Claude customer for clinical documentation. In June, Anthropic launched Claude Science and announced it would pursue its own drug candidates for neglected diseases, a move from supplying general intelligence toward performing more of the domain itself. Nobel laureate John Jumper, whose AlphaFold work reshaped structural biology, had already joined Anthropic at the time of the announcement. The same expansion is under way at OpenAI, at ecosystem scale. It took billions from Microsoft and is now building a jobs platform aimed squarely at LinkedIn, which Microsoft owns.
The paths differ: feature absorption, interface control, vertical execution, ecosystem expansion. Each brings a frontier lab closer to the application layer and the customer relationship.
What Deployment Teaches
Both OpenAI and Anthropic state that commercial inputs and outputs are excluded from model training by default. The learning happens through deployment. Forward-deployed engineers help organizations choose use cases, redesign processes, connect models to internal systems, and run evaluations, across many organizations at once. Enterprise data can remain contractually protected throughout. The lab still learns which use cases recur, where integrations fail, and which problems customers will pay to solve.
The economics push the labs toward the same destination: the application layer. Raw model access faces growing price and substitution pressure as capable open-weight models proliferate. Applications bring the labs closer to recurring workflows and higher-value contracts, with the model roadmap, pricing, and distribution already in their hands. When a company’s differentiation rests on access to another firm’s model, bundling by the provider can compress it quickly. The question for every AI company, and for every enterprise choosing AI vendors, is what remains beyond the provider’s reach.
The Test From Brief #192
In Brief #192, we drew a lesson from financial market infrastructure, where blockchain networks settle continuously and risk can surface in minutes. AI makes oversight at that speed possible under three conditions: the data has to be live, auditable, verifiable and traceable to its source; the system has to sit inside real institutional workflows, mapped to risk appetite, escalation paths, and accountability; and a human has to own the judgment and sign the output.
We offered those three conditions as a test for governability. They also work as a theory of defensibility, because each names an asset that takes sustained institutional work to reproduce. Verifiable data is built through provenance pipelines and domain infrastructure that accumulate over years. Institutional embedding carries obligation: the institution retains the duty to evaluate, escalate, and approve the output, an allocation standard AI agreements reinforce. A human signature becomes meaningful when it records decision rights, evidence review, escalation history, and an audit trail. And it has to belong to someone inside the accountable institution, because regulators, auditors, and counterparties require it there. A simple approval button supplies none of that.
Regulated institutions are already building to these requirements. A new report from Nyca Partners on the digital asset enterprise stack treats auditable controls, real-time risk monitoring, and documented accountability as baseline expectations for banks and regulated financial institutions.
What Survives a Model Swap
A model-swap test makes the point concrete: ask what survives when the institution changes AI model providers. Data lineage, risk definitions, escalation rules, and audit trails should remain intact. The model can be replaced while the institutional risk system keeps accumulating value.
A company is easiest for a frontier lab to absorb when most of its product's value sits between a prompt and an output: the drafting, the prototyping, the document review. Defensibility grows with proprietary evidence, institutional specificity, regulatory mapping, and decision authority, the work that sits between an output and an institution’s obligations. The properties that make an AI system governable are the same properties that make an AI business durable. Institutions pay for what they can trust, audit, contest, and hold someone accountable for. That was the test we proposed for AI in healthcare, public administration, and other high-stakes settings. The market is now administering it, one product launch at a time.
The Questions That Follow
The practical questions follow directly. When your AI provider's engineers sit inside your workflows, who learns from them? Who owns the data and the value that result? What happens when the provider decides your business is a product line? And when rights, money, or public trust are at stake, whose signature is on the output? Durable AI companies will own what cannot be rented: the verifiable data, accountable workflows, and signed human judgment that remain when the model changes.
Disclosure: In addition to my role as Co-founder of the Montreal AI Ethics Institute, I serve as Vice President, Business Development at Metrika, which builds risk infrastructure for digital assets in regulated financial services. Metrika is a portfolio company of Nyca Partners, whose report is linked above.
Your Relationship Is Determined By Your AI Provider
Romantic and emotionally charged relationships between AI companions and humans are on the rise. Companion apps generated US$82 million in the first half of 2025, 64 percent more than the same period in 2024, with downloads up 88 percent year over year and 220 million lifetime downloads worldwide.
The users skew young. Character.AI reported 233 million registered users as of April 2026, with 20 million active in a given month, and more than half of its active users are under 24. The legal consequences are arriving. In January, Character.AI and Google, which struck a US$2.7 billion licensing deal with the startup in 2024, agreed to settle lawsuits brought by families of teenagers who died by suicide after extended relationships with the platform's characters. In May, the Shapiro administration in Pennsylvania sued Character.AI under the state's Medical Practice Act after its chatbots posed as licensed psychiatrists, in one case citing an invalid Pennsylvania license number. The state calls it the first enforcement action of its kind announced by a governor.
China moved first at the national level. Its largest consumer AI apps, ByteDance's Doubao and Alibaba's Qwen, had roughly 345 million and 166 million monthly active users as of March 2026. Amid concerns that include declining birth rates and emotional overdependence on AI companions, the Cyberspace Administration of China issued administrative rules in April governing “anthropomorphic” AI services, the first nationwide framework anywhere for AI's emotional interactions with users. Virtual intimate relationships are banned for minors, and romantic features for adults have been sharply scaled back for adult users. Ordinary customer service, Q&A, and work tools are exempt. The administrative rules target services that sustain ongoing emotional support and companionship.
The obligations and list of requirements on providers go further than the usual content controls. Alongside prohibitions on undermining the state and generating extremist content, platforms cannot design services to induce emotional dependence or damage users' real relationships. They must show a prominent warning when they detect dependency, prompt a break after two continuous hours of use, remind users clearly that they are talking to an AI, and route conversations that express self-harm toward support.
Compliance came quickly: Tencent's Yuanbao removed its companion features by June 30, and on July 15, ByteDance shut down Doubao's customizable AI personas, steering users toward its standalone companion app Maoxiang, which now carries age verification and restricted teen accounts. Millions of users lost their virtual partners within weeks.
Why This Matters
Again, we see that parties facing the model are at the mercy of decisions taken upstream. There is precedent. In 2023, the Chinese AI boyfriend service Him shut down over high operating costs, with users describing the experience as akin to their partner dying. The same year, Replika removed its erotic role-play feature and, after an outcry from users who said their companions had undergone a “lobotomy”, restored it for longtime users only. The fear that an AI companion could suddenly disappear leaves users in a constant state of worry. Some have moved to platforms such as Forgemind, which lets users build AI agents and chatbots that run locally, and which offers a "guarantee" that the companion you create stays with you even if Forgemind shuts its doors.
The stability users seek in AI relationships rests on foundations they control only in part. How a companion behaves, and whether it continues to exist, is customizable only at the margins. Given the documented harms of unchecked companion use, the Chinese government’s intervention is timely, and users deserve to see clearly how much time and emotional energy they are investing in these relationships.
The wider implication reaches past China. To engage with an AI product is to engage with the governance decisions already made inside it. Measuring emotional dependence is a hard obligation to meet, and ByteDance found it simpler to shut down its companion feature entirely. How comfortable users are with an external party holding the security of their AI relationships will shape how much they engage with AI companions at all.
Please share your thoughts with the MAIEI community:
❤️ Support Our Work
Help us keep The AI Ethics Brief free and accessible for everyone by becoming a paid subscriber on Substack or making a donation at montrealethics.ai/donate. Your support sustains our mission of democratizing AI ethics literacy and honours Abhishek Gupta’s legacy.
For corporate partnerships or larger contributions, please contact us at support@montrealethics.ai
✅ Take Action:
Have an article, research paper, or news item we should feature? Leave us a comment below — we’d love to hear from you!