The AI Ethics Brief #190: The Data We Leave Behind
On therapy sessions, Slack archives, and citations that didn't exist.
Welcome to The AI Ethics Brief, a bi-weekly publication by the Montreal AI Ethics Institute. We publish every other Tuesday at 10 AM ET. Follow MAIEI on Bluesky and LinkedIn.
📌 Editor’s Note
In this Edition (TL;DR)
Collected for One Thing, Used for Another: Therapy sessions, Slack archives, and the data that outlives its original purpose
What Got Through: Three institutions that published AI-generated citations nobody verified
From the Editor: Writing in Tech Policy Press on surveillance pricing, Premier Ford, and why the absence of regulation is still a policy choice
Crafting Participatory Tech Futures: Contribute to the Community Prompts Library ahead of ACM FAccT 2026
Tech Futures: Better imagination for better tech futures, with RAIN
Recess: Gender-based violence on Grok is a feature, not a failure, with Encode Canada
AI Policy Corner: Are U.S. AI policies strengthening security or weakening global influence, with GRAIL at Purdue University
What Connects These Stories:
Data does not disappear when its original purpose ends. It changes hands, changes form, and resurfaces in places the people who generated it never expected.
A nurse’s therapy sessions, collected through an employer-provided benefit, became evidence against her in court. A defunct startup’s Slack archive became training data for an AI system. A government’s AI policy was built on academic citations that did not exist. A law firm submitted the same kind of citations to a federal judge.
The pattern is not just that data gets reused. It is that the original relationship around the data stops mattering. Patient and therapist. Employee and workplace. Government and public. Court and legal authority. Each relationship depends on trust, and each becomes vulnerable when records, transcripts, archives, or citations are treated as reusable inputs.
Collection came first. Governance came later, or did not come at all. That delay is itself a decision. The people who generated the data are left negotiating consequences they were never asked to imagine.
That gap between what AI systems can do and what governance frameworks can address runs through the rest of this edition. It shapes how X's Grok was designed and what it permits. It shapes how U.S. AI policy is pushing other countries toward choices America may not prefer. And it is exactly the gap that the Tech Futures series, and the Crafting Participatory Tech Futures workshop, are trying to open up: if the people most affected by these systems are not part of imagining their futures, someone else will imagine for them instead.
Collected for One Thing, Used for Another
Jennifer Kamrass told her therapist about her marriage, her finances, and her self-esteem. Two years later, a transcript of those sessions was produced in court by her former employer, AdventHealth. Her therapist declined to be named in Proof News’s reporting out of concern for her reputation.
Kamrass’s case began after AdventHealth terminated her nurse practitioner role in 2021, when she was nearly nine months pregnant. She began using Talkspace, an AdventHealth-provided benefit, to talk through worries about supporting her family and finding another nursing job so close to giving birth. After Kamrass filed a pregnancy discrimination claim, a federal judge ruled in AdventHealth’s favour, finding the claim insufficiently supported and accepting the employer’s argument that the termination stemmed from a facility closure. As part of its defence, AdventHealth’s lawyers secured a court order for her Talkspace records, including messages with the therapist who had agreed to testify on her behalf.
Patients using Talkspace might reasonably assume their sessions were subject to therapist-patient confidentiality. That assumption has two significant limits. First, court orders can reach therapy records, as Kamrass discovered. Second, Talkspace’s Notice of Privacy Practices for Members reserves the right to use de-identified patient data “for any purpose, without your further authorization or consent, including but not limited to research studies, development of artificial intelligence tools, and health care/health operations improvement activities.”
Talkspace has described itself to investors as holding “one of the largest mental health data banks in the world,” with 140 million message exchanges. According to investor materials reported by Proof News, that data is being used to train a therapy companion chatbot slated for release later this year. Talkspace says the data is anonymized. Data re-identification experts say anonymization at this scale is not a reliable protection. “It is really taking advantage of vulnerable people at a vulnerable time of their life,” one researcher told Proof News.
The Kamrass case surfaced because a court order required it. In most cases, no court order arrives.
A Forbes investigation published in April documented a parallel pattern. When companies shut down, their operational exhaust, including Slack messages, Jira tickets, and email threads, can become a commodity. SimpleClosure, a startup that helps founders wind down companies, has brokered nearly 100 such deals in the past year, with payouts typically ranging from $10,000 to $100,000 per company. AI labs want data that shows what real work actually looks like.
The employees whose messages are in those archives did not agree to this. Whether employer terms governing work communications extend to the sale of those communications as AI training data remains unsettled. Marc Rotenberg of the Center for AI and Digital Policy told Forbes the answer should be no: “It’s not generic data. It’s identifiable people.”
Data repurposed without meaningful consent is a governance problem. The Talkspace sessions and the SimpleClosure archives did not change when they were reused. The purpose changed, and the people who generated the data were not asked.
As we noted in Brief #188: What You Opted Into, what people agree to and what they ultimately become part of can be two different things. Therapy sessions become product inputs. Workplace conversations become enterprise training data. The people who generated the data remain identifiable. Their expectations of confidentiality, care, and ordinary workplace context do not.
What Got Through
In March, South Africa’s Department of Communications and Digital Technologies published a draft national AI policy that Cabinet had already approved. The Government Gazette published it on April 10 for public comment. It proposed a new national AI governance structure, including a National AI Commission and an AI Ethics Board.
It also cited 67 academic sources. At least six of those citations were hallucinated. The journals cited confirmed they had never published the referenced articles. Communications Minister Solly Malatsi acknowledged the most plausible explanation: a generative AI tool was used to draft the document, and the bibliography was not verified before the policy went to Cabinet. The policy was withdrawn on April 27. The withdrawal left South Africa without that draft AI governance framework.
On April 18, a partner at Sullivan & Cromwell, a 140-year-old Wall Street firm with more than 1,000 attorneys, wrote an apology letter to a federal bankruptcy judge. A prior court filing had contained hallucinated citations, including incorrect case names, fabricated case numbers, and apparently fabricated quotes. The errors had passed through the firm's own AI review process. Fake legal citations have become more common since 2023, according to legal researcher Damien Charlotin, who keeps a public database of AI hallucinations in legal cases.
The Springer Nature case is slower-moving. In May 2025, Springer Nature published a meta-analysis claiming ChatGPT has a “large positive impact on improving learning performance.” The paper was cited hundreds of times and reached nearly half a million readers. On April 22, 2026, Springer Nature retracted it, citing discrepancies in the meta-analysis and a lack of confidence in the conclusions.
Reviewers had flagged problems at publication. Ben Williamson, a senior lecturer at the University of Edinburgh, called the timeline alone implausible: the paper claimed to synthesize dozens of high-quality studies on a tool that had existed for two and a half years. The paper ran anyway. Williamson raised the alarm after the retraction notice received minimal attention, warning that the headline finding will likely persist regardless: educators have already adapted curricula, and tech companies continue to cite it.
A government ministry. An elite law firm. A peer-reviewed journal. In each case, AI produced or assisted output that looked authoritative: a bibliography, a case citation, a meta-analysis. Institutional trust rested on the appearance of verification rather than the practice of it.
Please share your thoughts with the MAIEI community:
📝 From the Editor
In our latest op-ed for Tech Policy Press, we examine surveillance pricing, Premier Ford, and why the absence of regulation is still a policy choice.
Ontario Premier Doug Ford was asked whether Ontario would follow Manitoba’s lead and ban surveillance pricing on groceries. He said no. His reasoning: let the market decide. The problem is that surveillance pricing is specifically designed to make market discipline impossible. The price you see was chosen for you. You have no way to know whether you are paying a premium, or why.
The absence of regulation is itself a decision. Canadians are the ones paying for it.
Read the full piece at Tech Policy Press.
Crafting Participatory Tech Futures: ACM FAccT 2026
As we shared in Brief #189, MAIEI joined RAIN, We and AI, and the San Diego Supercomputer Center to submit a workshop proposal to ACM FAccT 2026, which comes to Montreal for the first time this June. Our session, Crafting Participatory Tech Futures, was accepted.
As part of that workshop, we are building a Community Prompts Library (prompts appear under the “Notes” tab for each item). You can browse what the team has already assembled and contribute your own via this form. Submissions close May 31. Contributors will be acknowledged in the library and the post-workshop report.
Contribute to the Prompts Library here.
💭 Insights & Perspectives:
Tech Futures: Better Imagination for Better Tech Futures
This eighth instalment of our Tech Futures series, a collaboration with the Responsible Artificial Intelligence Network (RAIN), asks what it would take to imagine genuinely different technological futures, and invites you to contribute to the Library of Prompts.
To dive deeper, read the full article here.
Recess: Gender-Based Violence on Grok is a Feature, Not a Failure
This piece is part of our Recess series, featuring university students from Encode's Canadian chapter at McGill University. It examines X's Grok and the concept of “misogyny by design”: how deliberate design choices, rather than oversight failures, enable AI tools to scale gender-based abuse.
To dive deeper, read the full article here.
AI Policy Corner: Are U.S. AI Policies Strengthening Security or Weakening Global Influence?
This edition of our AI Policy Corner, produced in partnership with the Governance and Responsible AI Lab (GRAIL) at Purdue University, examines a tension at the centre of U.S. AI strategy. Policies designed to restrict China's access to advanced AI technology may protect national security in the short term. They also risk pushing other countries toward China's more cooperative framing, and away from the American AI stack altogether.
To dive deeper, read the full article here.
❤️ Support Our Work
Help us keep The AI Ethics Brief free and accessible for everyone by becoming a paid subscriber on Substack or making a donation at montrealethics.ai/donate. Your support sustains our mission of democratizing AI ethics literacy and honours Abhishek Gupta’s legacy.
For corporate partnerships or larger contributions, please contact us at support@montrealethics.ai
✅ Take Action:
Have an article, research paper, or news item we should feature? Leave us a comment below — we’d love to hear from you!