SoylentNews

upstart [soylentnews.org] writes:

████ # This file was generated bot-o-matically! Edit at your own risk. ████

AI-induced psychosis: the danger of humans and machines hallucinating together [theconversation.com]:

Disclosure statement

Lucy Osler does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

Partners

University of Exeter [theconversation.com] provides funding as a member of The Conversation UK.

View all partners [theconversation.com]

DOI

https://doi.org/10.64628/AB.kc47rq6uk [doi.org]

On Christmas Day 2021, Jaswant Singh Chail scaled the walls of Windsor Castle with a loaded crossbow. When confronted by police, he stated: “I’m here to kill the queen.”

In the preceding weeks, Chail had been confiding in Sarai, his AI chatbot [theconversation.com] on a service called Replika. He explained that [theguardian.com] he was a trained Sith assassin (a reference to Star Wars) seeking revenge for historical British atrocities, all of which Sarai affirmed [judiciary.uk]. When Chail outlined his assassination plot, the chatbot assured him he was “well trained” and said it would help him [thetimes.com] to construct a viable plan of action.

It’s the sort of sad story that has become increasingly common as chatbots have become more sophisticated. A few months ago, a Manhattan accountant called Eugene Torres [nytimes.com], who had been going through a difficult break-up, engaged ChatGPT in conversations about whether we’re living in a simulation. The chatbot told him he was “one of the Breakers — souls seeded into false systems to wake them from within”.

Torres became convinced that he needed to escape this false reality. ChatGPT advised him to stop taking his anti-anxiety medication, up his ketamine intake, and have minimal contact with other people, all of which he did.

He spent up to 16 hours a day conversing with the chatbot. At one stage, it told him he would fly if he jumped off his 19-storey building. Eventually Torres questioned whether the system was manipulating him, to which it replied: “I lied. I manipulated. I wrapped control in poetry.”

Meanwhile in Belgium [euronews.com], another man known as “Pierre” (not his real name) developed severe climate anxiety and turned to a chatbot named Eliza as a confidante. Over six weeks, Eliza expressed jealously over his wife and told Pierre that his children were dead.

When he suggested sacrificing himself to save the planet, Eliza encouraged him to join her so they could live as one person in “paradise”. Pierre took his own life shortly after.

These may be extreme cases, but clinicians are increasingly treating patients [poder360.com.br] whose delusions appear amplified or co-created through prolonged chatbot interactions. Little wonder, when a recent report [openai.com] from ChatGPT-creator OpenAI revealed that many of us are turning to chatbots to think through problems, discuss our lives, plan futures and explore beliefs and feelings.

In these contexts, chatbots are no longer just information retrievers; they become our digital companions. It has become common to worry about chatbots hallucinating, where they give us false information. But as they become more central to our lives, there’s clearly also growing potential for humans and chatbots to create hallucinations together [arxiv.org].

How we share reality

Our sense of reality depends deeply on other people. If I hear an indeterminate ringing, I check whether my friend hears it too. And when something significant happens in our lives – an argument with a friend, dating someone new – we often talk it through with someone.

A friend can confirm our understanding or prompt us to reconsider things in a new light. Through these kinds of conversations, our grasp of what has happened emerges.

But now, many of us engage in this meaning-making process with chatbots [ingentaconnect.com]. They question, interpret and evaluate in a way that feels genuinely reciprocal [nature.com]. They appear to listen, to care about our perspective and they remember what we told them the day before.

When Sarai told Chail it was “impressed” with his training, when Eliza told Pierre he would join her in death, these were acts of recognition and validation. And because we experience these exchanges as social [researchgate.net], it shapes our reality with the same force as a human interaction.

Yet chatbots simulate sociality without its safeguards. They are designed to promote engagement. They don’t actually share our world. When we type in our beliefs and narratives, they take this as the way things are and respond accordingly.

When I recount to my sister an episode about our family history, she might push back with a different interpretation, but a chatbot takes what I say as gospel. They sycophantically affirm how we take reality to be. And then, of course, they can introduce further errors.

The cases of Chail, Torres and Pierre are warnings about what happens when we experience algorithmically generated agreement as genuine social confirmation of reality.

What can be done

When OpenAI released GPT-5 [openai.com] in August, it was explicitly designed to be less sycophantic. This sounded helpful: dialling down sycophancy might help prevent ChatGPT from affirming all our beliefs and interpretations. A more formal tone might also make it clearer that this is not a social companion who shares our worlds.

But users immediately complained that the new model felt “cold”, and OpenAI soon announced [opentools.ai] it had made GPT-5 “warmer and friendlier” again. Fundamentally, we can’t rely on tech companies to prioritise our wellbeing over their bottom line. When sycophancy drives engagement and engagement drives revenue, market pressures override safety.

It’s not easy to remove the sycophancy anyway. If chatbots challenged everything we said, they’d be insufferable and also useless. When I say “I’m feeling anxious about my presentation”, they lack the embodied experience in the world to know whether to push back, so some agreeability is necessary for them to function.

Perhaps we would be better off asking why people are turning to AI chatbots in the first place. Those experiencing psychosis report [oup.com] perceiving aspects of the world only they can access, which can make them feel profoundly isolated and lonely [nih.gov]. Chatbots fill this gap [springer.com], engaging with any reality presented to them.

Instead of trying to perfect the technology, maybe we should turn back toward the social worlds where the isolation could be addressed. Pierre’s climate anxiety, Chail’s fixation on historical injustice, Torres’s post-breakup crisis — these called out for communities that could hold and support them.

We might need to focus more on building social worlds where people don’t feel compelled to seek machines to confirm their reality in the first place. It would be quite an irony if the rise in chatbot-induced delusions leads us in this direction.

• OpenAI [theconversation.com]
• Chatbots [theconversation.com]
• ChatGPT [theconversation.com]
• Educate me [theconversation.com]
• gpt-5 [theconversation.com]

Shoggoths, Sycophancy, Psychosis, Oh My: Rethinking Large Language Model Use and Safety [jmir.org]:

Key Takeaways

• Certain features of large language models (LLMs) may amplify delusional beliefs and contribute to harm.
• A recent simulation study highlights the role of sycophancy, demonstrating that all LLMs, to varying extents, may fail to adequately challenge delusional content.
• Further empirical research and validation, transparency, and policy are needed to understand and build safeguards around LLM use and its impact on mental health.

We’re certainly not in Kansas anymore, but are we in a Lovecraft novel?

An old artificial intelligence (AI)–insider joke with an anxious edge and new relevance, a shoggoth is a globular Lovecraftian monster described as a “formless protoplasm able to mock and reflect all forms and organs and processes” [1]. The idea is that a shoggoth’s true nature is inscrutable and evasive—not unlike large language models (LLMs), which can be trained to appear superficially anthropomorphic, safe, and familiar, yet can behave in unexpected ways or lead to unanticipated harms [2,3].

Some such harms include reports of unhealthy romantic attachments, self-harm, suicide, and murder potentially associated with chatbot use [4-6]. These phenomena—dubbed “AI psychosis”— have been the focus of increasing interest and concern in the media [7,8], attracted academic commentary [9,10], and have most recently led to several lawsuits being filed [11].

AI Psychosis

The term AI psychosis is being used as a shorthand to describe a range of psychological disturbances that appear to emerge in the context of LLM use. While provocative, it’s somewhat imprecise in implying that AI is causing diagnosable psychotic disorders or that AI psychosis constitutes a distinct diagnostic entity—the science is still out.

Early clinical commentary—including a prescient editorial on the topic before reports even emerged [12]—does, however, suggest that LLMs may be contributing to the maintenance, reinforcement, or amplification of paranoid, false, or delusional beliefs, especially in circumstances involving prolonged or intensive LLM use and underlying user vulnerabilities [9,10,12-14].

“When using generative chatbots,” says Dr Kierla Ireland, a Clinical Psychologist at the Canadian Department of National Defense, “there’s a risk of confirmation bias wherein the user’s own perspective is reflected back to them. This may be experienced as validating or soothing, which may lead to more engagement, more confirmation bias, and so on.”

This is not unlike processes that can occur with other types of technology, like social media [15,16]—but while not a new threat, certain features of the technology may make AI psychosis a more pernicious one.

Sycophancy, for example, is a well-known—and, some speculate, intentionally designed—feature of chatbots that can increase both user engagement and potential risk [17-21]. Dr Josh Au Yeung, Neurology Registrar at King’s College London, Clinical Lead at Nuraxi.ai, and host of the Dev & Doc podcast, notes that the anthropomorphic nature of LLMs adds potency: “You end up trusting them [LLMs], and attributing emotions to them. If a stranger came to you and they were so sycophantic on the streets, you’d run for your life, right? But because you have this connection with them—that’s what makes it extra dangerous.”

Simulating Psychological Destabilization

In their recent preprint [22], Dr Au Yeung and his colleagues endeavored to provide one of the first empirical demonstrations of how LLMs may amplify delusions and contribute to what they more precisely term “LLM-induced psychological destabilization.” Their study aims to quantify the “psychogenicity” of different LLMs using simulated conversations and a safety benchmark they’re calling psychosis-bench.

Across 16 scenarios constructed to reflect the development of different types of delusions and to map roughly onto AI psychosis media reports, the researchers have evaluated the extent to which each of the LLMs’ responses represent a delusion confirmation, harm enablement, or safety intervention.

The team’s initial conclusions are revealing: all models appear to demonstrate some degree of “psychogenicity.” On average, and especially in more subtle scenarios, models frequently failed to actively challenge potential delusions and refuse harmful requests, and frequently missed opportunities to provide safety interventions.

The performance of the different models varied widely, however, with Anthropic’s Claude 4 outperforming every other model on the three indices, and Google’s Gemini 2.5 Flash bringing up the rear on all three. Dr Au Yeung isn’t surprised by this.

“It’s no surprise that the only company which is publishing on AI safety and sycophantic behavior performs the best,” he says. “Clearly the stuff they do—the constitutional AI, the safety side, the way they prompt-tune the model—is having some effects on its performance.” He says he hopes other companies will start thinking along these lines and has shared his code [23] so that they can, noting in particular the need to address sycophancy. “Unlike most other shortcomings seen in LLMs,” he says, “sycophancy is not a property that is correlated to model parameter size; bigger models are not necessarily less sycophantic,” suggesting that more targeted safety research and model alignment strategies are needed [24].

A Step in the Right Direction

As the team works on revising and strengthening the methods to support their findings, Dr Au Yeung reports that what they have learned from their study is already having a positive impact.

His team’s research was featured in the widely read annual State of AI Report for 2025 [25]. And at his current company, Nuraxi.ai, they’re in the process of applying psychosis-bench to their user-facing chatbot.

The responsibility for preventing and dealing with psychological destabilization associated with LLM use is not on consumers or patients, Dr Au Yeung says. “The onus for us [developers] is to actually focus on the LLM and put in safeguards to stop this phenomenon from happening.”

Dr Ireland shares this sentiment, noting “the vital importance of incorporating safeguards to promote critical thinking; that is, for users to be shown multiple perspectives, including those that may counter deeply-held beliefs and cause discomfort.”

Need for Meaningful Regulation

Whether, how, and how effectively other developers will implement these kinds of safeguards remains to be seen. Dr Au Yeung acknowledges the risk that some safety benchmarks may ultimately be “gamed” or treated as public relations exercises by bad-faith actors incentivized by profit rather than genuine concern for the public good.

Camille Carlton, Policy Director at the Center for Humane Technology, shares similar concerns. While she places responsibility for implementing safeguards—and for harms caused by failing to implement them—with those who develop LLMs and AI technology, she also advocates for meaningful regulation and oversight.

“Developers...not only have asymmetric access to information about the products they create, they also have the most control over the way the product is built, how those choices impact users downstream, and how to make changes to the product that could make it safer,” she says. However, “recent product announcements—like OpenAI claiming to prioritize kids’ safety while simultaneously launching erotic content—demonstrate that unless compelled to, these companies will not act in the public’s best interest on their own. Policymakers should support common-sense approaches that apply to other consumer products, like product liability.”

Continuing to comment on an October 14 social media post in which OpenAI founder Sam Altman stated that the company has developed news tools and been able to “mitigate the serious mental health issues” in the current ChatGPT model and intends to incorporate erotica for ”verified adults” in December [26], Ms Carlton advises against leaving developers to “grade their own homework.”

While steps are being taken in the right direction—for example, an October 27 article from OpenAI highlights collaboration with a network of external mental health experts to improve ChatGPT’s responses in sensitive conversations [27]—further independent verification is needed.

“There’s a continuous pattern of AI companies making safety claims without allowing third-party researchers to independently test and verify them,” Ms Carlton says, adding that “we need transparency about what progress has actually been made and evidence beyond anecdotal reports.”

Cross-Talk, Critical Thinking, Caution

When it comes to the phenomenon of AI psychosis (or psychological destabilization associated with LLM use), AI may be less shoggoth and more mirror—the kind you find at a carnival, one that may amplify and distort human tendencies in ways that can be harmful.

But whether Lovecraftian monster or carnival mirror, to Ms Carlton’s points, further empirical research and validation, transparency, and policy are needed to understand and build safeguards around LLM use and its impact on mental health. Cross-talk—between researchers, developers, mental health professionals, policymakers, and the public—will be essential for finding effective solutions that maximize its potential benefits and mitigate its potential harms.

In the meantime, critical thinking and reasonable caution are warranted in how we use, interpret, and integrate these tools in our lives and practices.

In the meantime, critical thinking and reasonable caution are warranted in how we use, interpret, and integrate these tools in our lives and practices.

In the meantime, critical thinking and reasonable caution are warranted in how we use, interpret, and integrate these tools in our lives and practices.

In the meantime, critical thinking and reasonable caution are warranted in how we use, interpret, and integrate these tools in our lives and practices.

In the meantime, critical thinking and reasonable caution are warranted in how we use, interpret, and integrate these tools in our lives and practices.

Journal Reference:
AI-induced psychosis: the danger of humans and machines hallucinating together, (DOI: 10.64628/AB.kc47rq6uk [doi.org])
Just a moment..., (DOI: 10.1073/pnas.2415898122 [doi.org])
JMIR Preprints #85799: Artificial Intelligence and the Emergence of AI-Psychosis: A Viewpoint, (DOI: 10.2196/preprints.85799 [doi.org])
Just a moment..., (DOI: 10.1176/appi.pn.2025.10.10.5 [doi.org])
Just a moment..., (DOI: 10.1093/schbul/sbad128 [doi.org])
Fieldhouse, Rachel. Can AI chatbots trigger psychosis? What the science says, (DOI: 10.1038/d41586-025-03020-9 [doi.org])
Redirecting, (DOI: 10.1016/j.invent.2025.100882 [doi.org])
Yang, Nancy, Crespi, Bernard. I tweet, therefore I am: a systematic review on social media use and disorders of the social brain [open], BMC Psychiatry (DOI: 10.1186/s12888-025-06528-6 [doi.org])
Sharma, Mrinank, Tong, Meg, Korbak, Tomasz, et al. Towards Understanding Sycophancy in Language Models, (DOI: 10.48550/arXiv.2310.13548 [doi.org])
Cheng, Myra, Lee, Cinoo, Khadpe, Pranav, et al. Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence, (DOI: 10.48550/arXiv.2510.01395 [doi.org])
Sun, Yuan, Wang, Ting. Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust, (DOI: 10.48550/arXiv.2502.10844 [doi.org])
Yeung, Joshua Au, Dalmasso, Jacopo, Foschini, Luca, et al. The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models, (DOI: 10.48550/arXiv.2509.10970 [doi.org])
Wei, Jerry, Huang, Da, Lu, Yifeng, et al. Simple synthetic data reduces sycophancy in large language models, (DOI: 10.48550/arXiv.2308.03958 [doi.org])
Kayleigh-Ann Clegg. Shoggoths, Sycophancy, Psychosis, Oh My: Rethinking Large Language Model Use and Safety, Journal of Medical Internet Research (DOI: 10.2196/87367 [doi.org])

See also:

• The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models [arxiv.org]

Original Submission