AI-induced psychosis: the danger of humans and machines hallucinating together:
On Christmas Day 2021, Jaswant Singh Chail scaled the walls of Windsor Castle with a loaded crossbow. When confronted by police, he stated: "I'm here to kill the queen."
In the preceding weeks, Chail had been confiding in Sarai, his AI chatbot on a service called Replika. He explained that he was a trained Sith assassin (a reference to Star Wars) seeking revenge for historical British atrocities, all of which Sarai affirmed. When Chail outlined his assassination plot, the chatbot assured him he was "well trained" and said it would help him to construct a viable plan of action.
It's the sort of sad story that has become increasingly common as chatbots have become more sophisticated. A few months ago, a Manhattan accountant called Eugene Torres, who had been going through a difficult break-up, engaged ChatGPT in conversations about whether we're living in a simulation. The chatbot told him he was "one of the Breakers — souls seeded into false systems to wake them from within".
Torres became convinced that he needed to escape this false reality. ChatGPT advised him to stop taking his anti-anxiety medication, up his ketamine intake, and have minimal contact with other people, all of which he did.
He spent up to 16 hours a day conversing with the chatbot. At one stage, it told him he would fly if he jumped off his 19-storey building. Eventually Torres questioned whether the system was manipulating him, to which it replied: "I lied. I manipulated. I wrapped control in poetry."
Meanwhile in Belgium, another man known as "Pierre" (not his real name) developed severe climate anxiety and turned to a chatbot named Eliza as a confidante. Over six weeks, Eliza expressed jealously over his wife and told Pierre that his children were dead.
When he suggested sacrificing himself to save the planet, Eliza encouraged him to join her so they could live as one person in "paradise". Pierre took his own life shortly after.
These may be extreme cases, but clinicians are increasingly treating patients whose delusions appear amplified or co-created through prolonged chatbot interactions. Little wonder, when a recent report from ChatGPT-creator OpenAI revealed that many of us are turning to chatbots to think through problems, discuss our lives, plan futures and explore beliefs and feelings.
In these contexts, chatbots are no longer just information retrievers; they become our digital companions. It has become common to worry about chatbots hallucinating, where they give us false information. But as they become more central to our lives, there's clearly also growing potential for humans and chatbots to create hallucinations together.
Our sense of reality depends deeply on other people. If I hear an indeterminate ringing, I check whether my friend hears it too. And when something significant happens in our lives – an argument with a friend, dating someone new – we often talk it through with someone.
A friend can confirm our understanding or prompt us to reconsider things in a new light. Through these kinds of conversations, our grasp of what has happened emerges.
But now, many of us engage in this meaning-making process with chatbots. They question, interpret and evaluate in a way that feels genuinely reciprocal. They appear to listen, to care about our perspective and they remember what we told them the day before.
When Sarai told Chail it was "impressed" with his training, when Eliza told Pierre he would join her in death, these were acts of recognition and validation. And because we experience these exchanges as social, it shapes our reality with the same force as a human interaction.
Yet chatbots simulate sociality without its safeguards. They are designed to promote engagement. They don't actually share our world. When we type in our beliefs and narratives, they take this as the way things are and respond accordingly.
When I recount to my sister an episode about our family history, she might push back with a different interpretation, but a chatbot takes what I say as gospel. They sycophantically affirm how we take reality to be. And then, of course, they can introduce further errors.
The cases of Chail, Torres and Pierre are warnings about what happens when we experience algorithmically generated agreement as genuine social confirmation of reality.
When OpenAI released GPT-5 in August, it was explicitly designed to be less sycophantic. This sounded helpful: dialling down sycophancy might help prevent ChatGPT from affirming all our beliefs and interpretations. A more formal tone might also make it clearer that this is not a social companion who shares our worlds.
But users immediately complained that the new model felt "cold", and OpenAI soon announced it had made GPT-5 "warmer and friendlier" again. Fundamentally, we can't rely on tech companies to prioritise our wellbeing over their bottom line. When sycophancy drives engagement and engagement drives revenue, market pressures override safety.
It's not easy to remove the sycophancy anyway. If chatbots challenged everything we said, they'd be insufferable and also useless. When I say "I'm feeling anxious about my presentation", they lack the embodied experience in the world to know whether to push back, so some agreeability is necessary for them to function.
Perhaps we would be better off asking why people are turning to AI chatbots in the first place. Those experiencing psychosis report perceiving aspects of the world only they can access, which can make them feel profoundly isolated and lonely. Chatbots fill this gap, engaging with any reality presented to them.
Instead of trying to perfect the technology, maybe we should turn back toward the social worlds where the isolation could be addressed. Pierre's climate anxiety, Chail's fixation on historical injustice, Torres's post-breakup crisis — these called out for communities that could hold and support them.
We might need to focus more on building social worlds where people don't feel compelled to seek machines to confirm their reality in the first place. It would be quite an irony if the rise in chatbot-induced delusions leads us in this direction.
(Score: 2, Touché) by Anonymous Coward on Sunday November 30, @10:05PM (1 child)
---
You are never alone with a delusion!
(Score: 2, Interesting) by chucky on Sunday November 30, @10:12PM
There is something to this. Catching drug dealers is hard, tracing to the source even harder. Virtual drugs leave better traces. Please note that “hash” used to mean something different too.
(Score: 5, Interesting) by krishnoid on Sunday November 30, @11:17PM (1 child)
If you can't interact with it in a way that annoys it [youtu.be], is it a genuine interpersonal interaction?
(Score: 2, Funny) by Anonymous Coward on Monday December 01, @12:38AM
Current fortune cookie concurs:
(Score: 2, Informative) by Anonymous Coward on Monday December 01, @02:31AM (13 children)
From tfa,
> there's clearly also growing potential for humans and chatbots to create hallucinations together
Chatbots are unreliable software (note, they may still be useful...). Don't confuse these errors with anthropomorphic words like "hallucination". There's a reason the big money behind the current round of pattern matching "AI" companies would like you to think of their software in human terms.
From one of my favorite kid books, "The Mad Scientists Club", "The Unidentified Flying Man of Mammoth Falls"
Starts at: https://bookreadfree.com/347978/8562079.amp [bookreadfree.com] and ends on https://bookreadfree.com/347978/8562093.amp [bookreadfree.com]
Don't be a sucker for a scarecrow!
(Score: 2) by BeaverCleaver on Monday December 01, @09:19AM (3 children)
Thanks for the link. I loved those books as a kid. I'm sure I'm not the only one who tried to start their own mad scientists club. It's probably fortunate we didn't have the budget to replicate most of the stunts in the books.
(Score: 0) by Anonymous Coward on Monday December 01, @12:47PM (1 child)
The son of the author has a page...
http://www.madscientistsclub.com/MSC/Home.html [madscientistsclub.com]
Perhaps 15 years ago he was offering high quality hardbound reprints of the original Schoolastic Press(?) paperbacks. I bought a set, well worth the laughs and great memories.
(Score: 3, Insightful) by BeaverCleaver on Tuesday December 02, @12:23AM
And two full-length novels! I think I just found my Christmas present for myself.
(Score: 2) by krishnoid on Monday December 01, @03:32PM
People do grow up, pick up some cash, and start what I'm assuming are similar ventures [biocurious.org]. When they're older and God help us, wiser.
(Score: 3, Interesting) by Undefined on Monday December 01, @01:28PM (4 children)
The most accurate term is misprediction. The primary mechanism underlying LLM systems is next-token prediction based on the trained data set and the user's prompt(s.).
There's also some noise about using "confabulation" but LLM systems are not minds and they aren't doing that — they can't do that. Barring future technology (or unreleased, non-public technology), this is where we are.
I use a dedicated preprocessor to elaborate abbreviations.
Hover to reveal elaborations.
(Score: 1) by khallow on Monday December 01, @09:08PM (2 children)
And by intent, it produces human-like output. That opens the system to human-like descriptors like hallucination and confabulation.
(Score: 3, Informative) by Undefined on Tuesday December 02, @05:52PM (1 child)
The stream of output tokens, as I said. The tokens are words. Go learn about how LLM systems work. It'll be at least a little illuminating.
No, the output is literally a stream of predictions, one token/word at a time, the predictions being guided by the multidimensional space of the training data weights and the tokens in the user's prompt(s.) If your computer is slow enough, with a local LLM, you can even see it happening.
So did Eliza. Which was also not hallucinating or confabulating. If an LLM tells you in response to your prompt(s) that hot dogs are delicious and it'd like to eat one, you're making an error if you apply the idea of "it's hungry" to it. It isn't thinking and it isn't experiencing anything at all. It's just producing a highly related word stream. That's what they do. More can be layered on top of that, but that's the underlying tech for all LLM systems.
No. It opens the door for marketing to bullshit everyone in sight, though.
I write LLM and other ML code on a daily basis. I'm not just handwaving here.
I use a dedicated preprocessor to elaborate abbreviations.
Hover to reveal elaborations.
(Score: 1, Interesting) by khallow on Tuesday December 02, @07:20PM
This system no more predicts than weather does.
Eliza does things like ask questions and parrot the user's statements back at them. We can still use human-like (and parrot-like) descriptors to more accurately describe what the program does. My point here is that the attempts at greater intellectual accuracy in language routinely backfire and here, we have a great example of that. Nothing is predicted, it is merely generated. And we are a far cry from comingup withanaccurate description.
We lose an enormous amount of information when we drop from the descriptive "hallucination" to the much less descriptive "misgenerated". Random characters and erroneous output couched in confident terms are both misgenerated, but only one is a hallucination.
(Score: 2, Interesting) by anubi on Tuesday December 02, @05:59AM
"Don't confuse these errors with anthropomorphic words like "hallucination".
The most accurate term is misprediction. The primary mechanism underlying LLM systems is next-token prediction based on the trained data set and the user's prompt(s.)."
--------------
I have been seeing that a lot with AI, as being an old-school engineer, I often use things in very unusual nonstandard ( and often undocumented ) ways.
This is the "Farm Mentality" common on small family farms, of limited resources, but ready access to "bone yards" of abandoned machinery, as often farmers would drag any piece of abandoned machinery, automobiles, airplanes, home and store it on land which was unsuitable for farming. Who knows who can find a use for it by repurposing?
And I have had stints in research labs, oil refineries, and aircraft, with my last employment in a dying aircraft company.
One thing that fascinated me a lot was the instrumentation on aircraft, and how to repurpose it for using on the farm. Things like Selsyn motors ( we used a variant called an Autosyn Indicator gauge, made by Bendix ). All these aircraft instruments used the same basic hardware, though the dial markings changed from unit to unit to suit its purpose.
So, I have a curiosity if I can adapt these wonderfully made aircraft gauges to display farming data. Or can I put them to use in my van? I love the way these things look, they are extremely well made, Hermetically sealed, Military specs. Aircraft bone yards have lots of them. And I see quite a few people recovering old aircraft instrumentation , even though one certainly wouldn't put them in a modern aircraft.
I find AI seems to cut off of me when I am querying off means of driving these old instruments, driving them with Arduinos, multiplying DACs, and Class D audio amplifiers, using each in rather nonstandard ways. Things like using Class E RF power amplifier topologies to drive the indicator rotors, pulling the energy from the electrical resonator instead of using a coupling transformer, using magnetic core saturation to vary inductance to compensate for phase angles caused by power factor artifacts.
Yes. Class E. Supposedly for RF. And I wanna use it for lower audio. Binocular Tapewound cores with nonstandard winding methods.
I do not know who is teaching who I have definitely experienced a tendency from either and both of us mutually hallucinating. But that's ok...that's how I learn. It takes a long time to do it this way, but when I finally get my act together, I know exactly what I am doing, and what to expect from diddling any component.
AI has been confirming my beliefs, leading me to think I am close to an elegant design, only to discover I will require something no-one can make.
Almost like a stock broker who gladly took my fears that we were rapidly consuming the world's oil reserves and gladly recommended all sorts of oil exploration and drilling stocks. I was definitely of the belief, after having seen our water aquifers deplete on the farm, needing deeper and deeper wells, and had seen oil depletion and the mitigation we were doing to keep the wells producing, despite watering out, we were heading for a world of hurt - knowing how inelastic our demand was. That inelasticity was obvious upon witnessing the Gray Davis and Enron years, when a Megawatt- Hour was selling for unbelievable sums.
I was hallucinating, based on my belief, which caused me to mismanage most of my retirement. I was convinced I was in the position of being the ship's engineer on board the Titanic, when even an old busted Styrofoam cooler box would be worth more than a hundred pounds of gold. I did not believe in abiotic oil - I saw oil as a one-time endowment that we were rapidly depleting. And the stockbroker hallucinated right along with me.
I should have bought the Bitcoin. Which at the time I looked at it, I considered it to be a fool's net - worse than a Vegas Private Game on a rigged table.
Since that, I have seen numerous clusterfucks of shared mutual hallucinations like a self-regenerative oscillator, each feeding off of as well as reinforcing the others. This has caused me to re-examine myself, human nature, leadership/religion. If AI is modeled from something as fallable as I am, please don't trust it and put all your eggs in one basket, as Grandpa used to say.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 1) by khallow on Tuesday December 02, @01:18PM (3 children)
(Score: 1, Informative) by Anonymous Coward on Tuesday December 02, @09:48PM (2 children)
> Can you show there's an actual problem with language usage here?
(AC who started this thread)
Here's someone with huge software engineering experience (with a focus on critical systems) who laid it out very nicely for me,
David Parnas Keynote ICSE2025: Regulation of AI and Other Untrustworthy Software
https://www.youtube.com/watch?v=YyFouLdwxY0 [youtube.com] The introduction ends at about 2:15.
First para of the YT description:
(Score: 0) by Anonymous Coward on Tuesday December 02, @11:42PM
ps. Parnas comments on hallucination starting right at 30:00 --
"Hallucination is a euphemism for bugs."
(Score: 2, Insightful) by khallow on Wednesday December 03, @03:02AM
and
A few simple examples will show the "error" of those assertions.
A euphemism is a near synonym. AI hallucinations are a much smaller set than bugs. And similarly, bugs are a much smaller set than errors. So on that basis alone, these aren't euphemisms. The last point brings up another feature of euphemisms - that they are used to replace a commonly used word rather than a rare one. If shithouse were the normal usage, then outhouse would be a euphemism (less vulgar/more highbrow, less poo in the name, etc). But it's not. That is used in some rarified dialects while the outhouse is the common usage.
In one slide, this guy has semantically failed on five words. Calling hallucination merely an "error" loses a great deal of information. And there is no better term of art that expresses the meaning of that error.
(Score: 3, Funny) by VLM on Monday December 01, @04:13PM
ITs better that they'd blaming LLMs / AIs than blaming dungeons and dragons for crazy people doing crazy things as they inevitably do.