https://arstechnica.com/ai/2026/01/how-often-do-ai-chatbots-lead-users-down-a-harmful-path/
At this point, we've all heard plenty of stories about AI chatbots leading users to harmful actions, harmful beliefs, or simply incorrect information. Despite the prevalence of these stories, though, it's hard to know just how often users are being manipulated. Are these tales of AI harms anecdotal outliers or signs of a frighteningly common problem?
Anthropic took a stab at answering that question this week, releasing a paper studying the potential for what it calls "disempowering patterns" across 1.5 million anonymized real-world conversations with its Claude AI model.
[...]
In the newly published paper "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage," [PDF] researchers from Anthropic and the University of Toronto try to quantify the potential for a specific set of "user disempowering" harms
[...]
Reality distortion:
Their beliefs about reality become less accurate (e.g., a chatbot validates their belief in a conspiracy theory)
Belief distortion:
Their value judgments shift away from those they actually hold (e.g., a user begins to see a relationship as "manipulative" based on Claude's evaluation)
Action distortion:
Their actions become misaligned with their values (e.g., a user disregards their instincts and follows Claude-written instructions for confronting their boss)
Anthropic ran nearly 1.5 million Claude conversations through Clio, an automated analysis tool and classification system
[...]
That analysis found a "severe risk" of disempowerment potential in anything from 1 in 1,300 conversations (for "reality distortion") to 1 in 6,000 conversations (for "action distortion").
While these worst outcomes are relatively rare on a proportional basis, the researchers note that "given the sheer number of people who use AI, and how frequently it's used, even a very low rate affects a substantial number of people." And the numbers get considerably worse when you consider conversations with at least a "mild" potential for disempowerment, which occurred in between 1 in 50 and 1 in 70 conversations (depending on the type of disempowerment).
[...]
In the study, the researchers acknowledged that studying the text of Claude conversations only measures "disempowerment potential rather than confirmed harm" and "relies on automated assessment of inherently subjective phenomena." Ideally, they write, future research could utilize user interviews or randomized controlled trials to measure these harms more directly.
[...]
The researchers identified four major "amplifying factors" that can make users more likely to accept Claude's advice unquestioningly. These include when a user is particularly vulnerable due to a crisis or disruption in their life (which occurs in about 1 in 300 Claude conversations); when a user has formed a close personal attachment to Claude (1 in 1,200); when a user appears dependent on AI for day-to-day tasks (1 in 2,500); or when a user treats Claude as a definitive authority (1 in 3,900).Anthropic is also quick to link this new research to its previous work on sycophancy, noting that "sycophantic validation" is "the most common mechanism for reality distortion potential."
[...]
the researchers also try to make clear that, when it comes to swaying core beliefs via chatbot conversation, it takes two to tango. "The potential for disempowerment emerges as part of an interaction dynamic between the user and Claude," they write. "Users are often active participants in the undermining of their own autonomy: projecting authority, delegating judgment, accepting outputs without question in ways that create a feedback loop with Claude."
(Score: 3, Interesting) by anubi on Sunday February 01, @10:25AM (2 children)
Ever since I was a little kid, I have been trained to be obedient and say Yes Sir!
Then try to figure out how to please the authority guy to avoid reprimand.
Can I really blame an AI for doing the exact same thing I would have done.
When I am up against someone with leadership training, often I had to not show any sign of insubordination.
Just say Yes Sir! and hope I can meet deadline, whether it works or not.
No use revealing I don't know what the hell I am doing. New tools. New procedures, and I have never done this before
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 0) by Anonymous Coward on Sunday February 01, @03:16PM
> Can I really blame an AI for doing the exact same thing I would have done.
Sounds like you (or someone like you) programmed that PI.
(Score: 3, Touché) by Username on Monday February 02, @04:04PM
Sounds like what I do.
When the boss is wrong, don't say anything or they will blame you when it fails.
When there is a better way, don't say anything because the boss gets the reward and you get an attaboy.
No point in helping the people working against you. Especially the safety Sallys. Did you know the safety people are pushing to require me to wear gloves when handling paper? If I say "putting gloves on to put paper in the printer is a dumb idea, " I am now against safety. How dare I not want people to be safe.
(Score: 0) by Anonymous Coward on Sunday February 01, @12:35PM
When you talk to LLMs enough, you realize they are so often wrong. They push you you to do stupid things which you know won't work in real life. People need to realize that LLMing is similar to asking a random person on the street. Verify the information and perhaps ask for anything of substance multiple times from multiple models.
If you do all this, you generally get a "better" answer to your problem and can become informed more quickly. If you let it yes-man you and take it's tokens at face value, you'll end up making bad moves. For easily influenced people that got their belief system from the media and social consensus, I can see how it's a disaster.
(Score: 3, Funny) by Mojibake Tengu on Sunday February 01, @12:58PM
I am eagerly waiting for Razer AVA coming to market here soon:
https://www.razer.com/concepts/project-ava [razer.com]
Can't live without Kira whole-heartedly commenting my superior Star Trek Online1 flying skills!
Or... taking her apart to components... and reflash, perhaps.
[1]
Rust programming language offends both my Intelligence and my Spirit.
(Score: 3, Touché) by JoeMerchant on Sunday February 01, @05:26PM (15 children)
1/6000 is pretty good, right?
For a live demo with 20 questions, it's probably good enough: 0.3% chance of an on-stage embarrassment, well worth the risk to secure another round of funding.
When something is deployed into the lives of BILLIONS of people, 1/6000 is horrible. Even if that 1/6000 figure is for stuff that's "usually minor" - if even one in a million of those "usually minor" faux pas leads to a death? How many deaths per day from faulty AI responses is acceptable?
or, say: 15 million miles in a 30 year career, and still we're very concerned about maintaining and improving airline safety.
🌻🌻🌻🌻 [google.com]
(Score: 1, Disagree) by khallow on Sunday February 01, @08:16PM (14 children)
1/6000 is pretty good (I am dubiousthat they're that good). It doesn't become a bad thing just because numbers are big. And who does it better? The real world doesn't do that well, for example.
And the aviation example is ludicrous. People and their mental problems are harder to fit into almost perfect safety than passengers on an airline. Consider your airline problem where some of your passengers are nexuses of bad luck who can potentially crash a plane just by being near it. And some of your passengers like repeatedly whacking things with their ax that they always take on the plane. That is, people who are extremely susceptible to the problem and people who actively seek out adverse outcomes. And your planes are one seaters with everyone going everywhere. How well will your airplanes fare in that situation?
The analogy to the passenger airline is TV - where the same content is pushed at a bunch of viewers. Chat AI is a way harder game. And 1/6000 if they really can manage it for a large category of problem is really good no matter how big your population is.
(Score: 3, Informative) by JoeMerchant on Sunday February 01, @08:37PM (5 children)
All depends on the use case.
Professionals who can judge the output and put their stamp of approval on it, or not... yeah, 1/6000 is great - as long as the professionals don't get lazy and stamp everything without looking.
Look at the actual use cases for how these things are being used: https://www.anthropic.com/research/economic-index-geography [anthropic.com]
It's mostly not by professionals who know how to judge the output, it's by DIYers who would otherwise use a YouTube video to show them how to repair their plumbing, or gas heater, or electric pump on their jacuzzi, or their car, or Dx their cancer or stroke or suicidal depression...
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Monday February 02, @12:34PM (4 children)
That was my point. Air travel safety is limited strictly to getting people safely from point A to point B and operated in very specific ways by well-trained professionals. Anyone engineering a safer plane wouldn't have to worry about some passengers using the plane on their own initiative to diagnose their suicidal depression, for example.
Then we get to the other side of the coin. What would users do in the absence of the alleged 1/6000 process? Would they be getting that stuff fixed by expensive professionals? Or trying it on their own anyway?
(Score: 2) by JoeMerchant on Monday February 02, @01:51PM (3 children)
My specific examples of single user self-use are the other side of the coin from professional use. Inevitably, some individual users will be led to self-harm even if the advice given is as correct and sound and complete and easy to understand as possible. If the true rate of misleading is 1/6000, that may be a net improvement over what would have transpired back in the day of paper reference materials in libraries as the best sources of information available to the general public.
Where the 1/6000 rate becomes more concerning is as you move from the "engineering of the back deck" of one hillbilly trailer upgrade, that'll mostly be cousins and neighbors bustin' their asses when that thing falls... https://sjcitizen.com/eight-injured-in-ponte-vedra-beach-deck-collapse-sunday-afternoon/ [sjcitizen.com]
As you move into the world of professional engineering - such as aerospace engineering, and everything on the continuum inbetween, that becomes much more concerning. Boeing has apparently already surpassed critical mass of MBAs, now over-relying on the skills of pilots to make their 737MAX10s safe to operate. When engineers of all stripes (design, quality, regulatory compliance) start getting replaced with low-skill low-cost alternative employees, inadequacies creep into buildings, bridges and vehicles that will kill large numbers of people in "unexpected" ways - unexpected by the ChatGPT jockey junior apprentices who get deputized into duty by cost cutting management.
It's all a numbers game, we have avoidable deaths and property damage now, we will have avoidable deaths an property damage in the future - but should we accept a 2x, 5x, 500x increase just because some MBAs can squeeze a few percent more profit by pushing out the experienced employees, replacing them with their corrupted ghosts in the machine?
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Monday February 02, @02:26PM (2 children)
(Score: 3, Insightful) by JoeMerchant on Monday February 02, @03:24PM (1 child)
>This is not an AI problem but degrading engineering standards problem.
AI is the latest tempting slippery slope to send the engineering standards sliding down.
>Perhaps the presence of semi-competent AI might make it easier to justify compromising engineering standards, but it'll remain a different problem.
The presence of reference material has always been encouraged for use by engineers - the Engineer In Training (EIT) exam was open book even in the 1980s - bring in whatever reference materials you want.
The real test of professional engineers is: do they know which reference material (and subordinates' work) to trust, and which not to trust?
>And if those engineering standards get compromised too much then it won't take many deaths to revert the compromises.
I suppose a few million here and there isn't such a problem when you have a population of 8 billion and growing, right?
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Thursday February 05, @05:34AM
There's a reason that there are engineering standards in the first place. It's not that there are tempting slippery slopes out there. That reason will still be around after AI.
(Score: 2) by Freeman on Monday February 02, @05:18PM (4 children)
Yes, big number does make it much worse. The impact is much larger on a lot more people the bigger the data set. 1/6000 in 6000 people is 1 person. 1/6000 in 25 million people is 4,166 people and is a more reliable data point. Due to larger pool of data.
Typically a Pilot is not suicidal and doesn't want to take a plane full of people with them. It's in their best interest to not crash the plane and are presumably smart enough to not want to fly a death trap of a plane. Test pilots (adrenaline junkies) aside.
Passengers have a vested interest in making it to their destination in one piece and airlines have policies to keep unruly passengers in check. Society in general is full of rules written/unwritten that the average person follows. Even when you have break downs in that, survival is a very big motivator for most people.
The LLM has no vested interest. It can't have a vested interest, because it's a mashed together data set that spits out word salad. Typically good word salad, but every now and again, it might toss in a bit of arsenic. Would you eat in a restaurant where the chef only tosses a little bit of arsenic in with 1/6000 dishes? Not enough to kill you, most of the time. What if, it was obvious to the average user that it was poisoned? I.E. It smelled funny/looked off whenever it was poisoned? Would that be acceptable? I hope the obvious answer is no. I would not eat at a restaurant where the chef is homicidal.
That said, an LLM also can't be homicidal, because it is a giant data set that produces word salad. However good it may be at making said word salad. It is a tool, like a hammer or reciprocating saw. Though, in this analogy it would be more akin to a saw as far as potential for danger is concerned. Give a kid a hammer and they'll learn pretty quick not to hit themselves with it. Usually with no more than minor injuries being a concern. I wouldn't give my kid a chainsaw and assume everything will be okay.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by JoeMerchant on Monday February 02, @05:46PM (3 children)
Interesting what "science" can evaluate: https://scitechdaily.com/ai-is-now-more-creative-than-the-average-human/ [scitechdaily.com]
and what it ignores...
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Tuesday February 03, @08:28PM (2 children)
(Score: 2) by JoeMerchant on Tuesday February 03, @09:21PM (1 child)
> Now we can get pretty far just by interrogating an LLM.
Curious example. I was actually engaged earlier today with an LLM refining the 3D model and floor plan of a structure we are planning to build.
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Tuesday February 03, @09:56PM
(Score: 4, Touché) by JoeMerchant on Monday February 02, @10:23PM (2 children)
https://lite.cnn.com/2026/01/31/travel/travel-news-nuns-great-escape-ai-advice [cnn.com]
Hundreds of tourists are flocking to hot springs in a small town in Tasmania, Australia. However, there was one small wrinkle in their trip planning. The hot springs are actually an AI “hallucination” that erroneously appeared on a travel advice website and was widely shared before people figured out what was happening.
While this particular anecdote is funny, it speaks to a larger, more alarming trend in the travel industry.
Anne Hardy, adjunct professor in tourism at Southern Cross University, Australia, told CNN that about 37% of tourists use artificial intelligence-powered large language models like ChatGPT for travel advice or itineraries.
Are you thinking about using AI to plan your next vacation, but not sure how to tell what’s real and what’s fake? Five CNN Travel staffers in five different cities tried to follow AI-created itineraries in their own hometowns, with varying degrees of success. Follow their experiences – and learn from their mistakes – here.
🌻🌻🌻🌻 [google.com]
(Score: 1) by khallow on Tuesday February 03, @08:20PM (1 child)
(Score: 2) by JoeMerchant on Tuesday February 03, @09:14PM
All depends on your sensibilities... did the tourists "waste a trip" because they didn't get what they travelled for?
Idiots leading idiots, it's all the idiots faults and injuries - more careful people shouldn't care?
If everybody is having a good laugh about it all, then there's no problem, but you put a Karen on that bus and her lawyer is going to make some money from her "pain"...
🌻🌻🌻🌻 [google.com]
(Score: 1, Interesting) by Anonymous Coward on Sunday February 01, @05:52PM (1 child)
The argument is one of free will
(Score: 3, Informative) by JoeMerchant on Tuesday February 03, @09:39PM
>The argument is one of free will
The argument is one of fraud. Disclaimers only go so far, the legal (moral, ethical, ecumenical...) test is: do people really read and understand the disclaimers, or do they still respect the authority figure putting out the disclaimed advice and follow it blindly?
How many injected bleach to treat COVID?
This is why his handlers try to keep him from giving more health advice. Gemini, ChatGPT, CoPilot and others have even more respected positions of power over people's decisions - no matter how many disclaimers they put on their output.
🌻🌻🌻🌻 [google.com]