People trust legal advice generated by ChatGPT more than a lawyer – new study:
People who aren't legal experts are more willing to rely on legal advice provided by ChatGPT than by real lawyers – at least, when they don't know which of the two provided the advice. That's the key finding of our new research, which highlights some important concerns about the way the public increasingly relies on AI-generated content. We also found the public has at least some ability to identify whether the advice came from ChatGPT or a human lawyer.
AI tools like ChatGPT and other large language models (LLMs) are making their way into our everyday life. They promise to provide quick answers, generate ideas, diagnose medical symptoms, and even help with legal questions by providing concrete legal advice.
But LLMs are known to create so-called "hallucinations" – that is, outputs containing inaccurate or nonsensical content. This means there is a real risk associated with people relying on them too much, particularly in high-stakes domains such as law. LLMs tend to present advice confidently, making it difficult for people to distinguish good advice from decisively voiced bad advice.
We ran three experiments on a total of 288 people. In the first two experiments, participants were given legal advice and asked which they would be willing to act on. When people didn't know if the advice had come from a lawyer or an AI, we found they were more willing to rely on the AI-generated advice. This means that if an LLM gives legal advice without disclosing its nature, people may take it as fact and prefer it to expert advice by lawyers – possibly without questioning its accuracy.
Even when participants were told which advice came from a lawyer and which was AI-generated, we found they were willing to follow ChatGPT just as much as the lawyer.
One reason LLMs may be favoured, as we found in our study, is that they use more complex language. On the other hand, real lawyers tended to use simpler language but use more words in their answers.
The third experiment investigated whether participants could distinguish between LLM and lawyer-generated content when the source is not revealed to them. The good news is they can – but not by very much.
In our task, random guessing would have produced a score of 0.5, while perfect discrimination would have produced a score of 1.0. On average, participants scored 0.59, indicating performance that was slightly better than random guessing, but still relatively weak.
(Score: 2) by looorg on Thursday May 08, @08:48AM (3 children)
The venn-diagram probably also overlap with people that would rather trust Dr.Google then a real medical doctor when it comes to various health issues.
That sounds like a lot of legal text in general. Certainly if it's so long that you have to take information from multiple sources and merge it into a summary.
Was there just one answer/text provided from each source as in there was just one 'lawyer text' and one 'chatgpt text'. So it was a coin flip then if you didn't know or couldn't tell. There was apparently telltale signs such as chatgpt using more fancy, complicated and long words -- classic signs of trying to sound smarter then it actually is. Which people might be picking up on and from that thinking that the more complicated or snooty answer might be the correct answer. When in fact the lawyers just tried to explain things to their clients on a lower or more generic level.
(Score: 2) by JoeMerchant on Thursday May 08, @03:48PM
The concepts of agency and fiduciary duty are as flawed as criminalizing based on intent. You can't prove or dis-prove intent, and you can't prove that an agent is or is not acting in the best interests of the person(s) they represent. Look no further than the Real Estate industry for a clear example of a bunch of apparent idiots running around acting in their own best interests while charged by law with representing the interests of their clients. In Florida they are evolving the law to start informing clients of Real Estate agents that they're basically on their own, they can't count on their Real Estate brokers or sales persons to advise them of basically anything.
So, then, we come to lawyers (and doctors) who are supposed to be smart fellers, but are often as not fart-smellers too. How do you know that a lawyer is really representing your best interests? How can you prove that he is, or isn't? Even if it becomes apparent that other courses of action would have been in your best interests, how likely are you to make a charge stick that "your lawyer should have known better?" Not following the fart-smeller reference? It comes from brown-nosing, and an example of such was a lawyer who I engaged and paid for his counsel regarding disabled students' rights under the law. Except, this particular lawyer was (unbeknownst to me at the time) buddies with the chairman of the local school board who was engaged in all kinds of discriminatory and illegal activities, including directing the throwing disabled children in dark closets to keep them out of sight of "the good kids." So, his advice? "Oh, those are really old laws, does anyone even follow that anymore?" The next day I had a referral from a friend to a better lawyer who represented us pro-bono and got the school board to immediately deliver what we had been asking for for the previous six weeks, but they were dragging their feet.
How many people wander into the offices of the lawyer across the street from the school board, take his advice as genuine, and are dissuaded from pursuing their rights under law because of his colored commentary?
Is an AI lawyer more, or less likely to give biased advice?
How does Grok AI's evaluations of the current administration stack up against Faux News' reporting?
🌻🌻🌻 [google.com]
(Score: 4, Funny) by DannyB on Thursday May 08, @06:16PM (1 child)
They want someone who tells them what they want to hear rather than the unhappy truth.
That is why I prefer Dr. YouTube instead of Dr. Google.
Any individual random YouTuber is highly likely to have internal consistency, giving them more credibility.
The microchips are first ground into powder before being mixed into the vaccines, thus enabling the government to track the thoughts and locations of everyone who gets vaccinated.
The only way to stop a bad guy with a can opener is a good guy with a can opener.
(Score: 2) by JoeMerchant on Thursday May 08, @06:43PM
>The microchips are first ground into powder before being mixed into the vaccines, thus enabling the government to track the thoughts and locations of everyone who gets vaccinated.
Knew a (former North) Vietnamese immigrant who came to the US when he was 6, I met him when he was 46. The government injected children, including him he says, with "tracking chips". He was more than a little schizo and some days he clearly believed that "the chip was controlling his brain."
🌻🌻🌻 [google.com]
(Score: 0, Flamebait) by zenlessyank on Thursday May 08, @09:03AM (11 children)
How am I supposed to follow a law when a lawyer doesn't even know what that law is without looking it up in books only lawyers can purchase?
If I didn't vote for the law then I'm not following it. Ever heard of a democracy? Fuck this dick tater bullshit.
(Score: 2, Touché) by Anonymous Coward on Thursday May 08, @12:33PM (1 child)
Did you vote for traffic laws? I'm looking forward to seeing you blaze through all red lights and stop signs all over the country.
Oh, wait... you meant "things I don't agree with, because I'm selfish"... now I get it
(Score: -1, Troll) by zenlessyank on Thursday May 08, @03:39PM
So nice to see so many gutless stooges! You go girl!!
(Score: 2) by bzipitidoo on Thursday May 08, @03:47PM (5 children)
I have little trust of professional services. Too much "buyer beware". Big Pharma more interested in pushing pills than improving health, consumer goods corporations manipulating people into buying too much stuff, and, yes, ambulance chaser lawyers. There's a lot of class warfare going on.
But it's even worse to put trust in the contemptuous and cynical who admit they aren't experts but nevertheless in fine Dunning-Kruger fashion assert that they know better while getting it wrong by making out that things are worse than is actually the case. I grant that things in the US are heading towards lows not seen since the Civil War. The anti-choice zealots are riding high on the corpses of women who died of pregnancy complications that were preventable. The rash of new anti-woman laws, yes, I agree those should be disobeyed. Yes, there is abuse of the power of lawmaking to favor some over others. But let's not trash all the laws. Many of these greedy corporations are right there with the useful idiots when it comes to blasting away rules that keep us safe from corporate malfeasance.
We're not as low as Civil War times, not yet, but we might get there. We have had a few instances of governments trying to copyright the law, in very blatant attempts at rent seeking. I have not heard of any of those attempts being more than very short lived. If you know otherwise, please share.
(Score: 3, Interesting) by gnuman on Thursday May 08, @04:04PM (4 children)
That's not really fair here. They *are* interested in getting better results. And we all know what is cure for most of the chronic disease we have in the rich-world --- it's eat less and move more. No pharma tricks here. But they will try to work-around your lazy-asses by giving you cholesterol lowering drugs and sugar lowering drugs and even appetite suppressors. All these greatly improve people's health in short term *and* in the long term, at a cost, of course.
But it's not their fault that people want to sit and eat and not apply much effort to their health.
Heart attacks are not sudden -- it takes 30 years of bad diet and no exercise to cause most of them. And pharma has allowed us to even extend that landing strip a few decades longer, but for a $$ cost.
The cure is there to improve your health. It's mostly not via pharma --- we all know that. But we prefer to pay them money so we can avoid the effort. Not sure why bitching about it on internet makes them a bad guy in the room suddenly? Easy target?
(Score: 3, Insightful) by bzipitidoo on Thursday May 08, @04:56PM (3 children)
Not fair? On the contrary, they are a big, big part of why the US healthcare system is so expensive and downright greedy. Rank right up there with health insurers. There is good reason to bash Big Pharma. Pharma Bro Martin Shkreli, remember him? Hiked the price of a drug, Daraprim, oh, 5000%. Then there's Heather Bresch, who hiked the price of EpiPens a mere 500%. Then, how about those Sacklers? What's the big deal about a pain killer, OxyContin being a little addictive, you know? Never that it keeps patients coming back for more, never that! A long time ago, for a class project I visited a friend's grandparents to ask them about the old times. I could hardly get a word out of them because they were overmedicated and kept dozing off. Overmedication is another well documented problem that was quite prevalent.
We hear over and over that Big Pharma does much freeloading off drug research conducted on the public dime at universities. There is much well documented evidence that they manipulate doctors. Outright bribe doctors, to prescribe more drugs. That absolutely is pill pushing. There is also denigration of Canadian pharmacies. They like to insinuate that drugs from foreign sources aren't as good quality as domestically produced stuff.
Woe unto the patient who has swallowed the advertisements to "ask your doctor about" whatever drug they're pushing now. Arguably, they shouldn't be advertising directly to the public at all.
The meeting of maternal anxiety with greedy pharma gets ugly, fast. Rather than calm Mommy down, some are only too eager to fan the unreasonable fears, to get Mommy to practically threaten the doctor if the doc doesn't prescribe those miracle meds! Twice now I have had relationships with pediatricians go sour because they were too greedy and gutless to reassure Mommy. Is it really necessary to puree fruit so baby doesn't choke? Oh, yes, yes, yes, they say. Even bananas. But actually, no, it's not. One of these pediatricians suggested that baby's digestion could benefit from Zantac, and was all ready to prescribe it. It would have to be the much more expensive liquid form, for baby. Cost? Oh, just $500 per month! Fortunately, Mommy came to her senses, rejected the idea, and stopped babbling to the doctor about her anxieties over baby's health. Of critical help was that she is aware that such drugs have all kinds of side effects that might be worse than the problem that might not be a problem after all. Never saw that pediatrician again.
(Score: 3, Interesting) by JoeMerchant on Thursday May 08, @06:46PM
>for a class project I visited a friend's grandparents to ask them about the old times.
My Grandparents didn't (all) do the meds, they would tell me about the old time doctors in rural Tennessee. Mostly that the doc became/stayed a doc because he was a dope fiend and being a doc gave him access to his fix. When he wasn't too loopy he was also a pretty smart fella and could help you sometimes, but not too much - sulfa drugs were the order of the day back then, before penicillin started making the rounds.
🌻🌻🌻 [google.com]
(Score: 2) by gnuman on Saturday May 10, @05:28PM (1 child)
So, part of this is your regulatory environment. *Your* government is responsible for this regulatory environment. Maybe a good solution here is to not vote for people that will not regulate prices because "that's communism". On the contrary, it's up to the regulators to make sure that pharmaceutical monopolies are not abused.
Again, that's regulatory environment. In many countries, this practice is illegal for prescription medicine. Funnily, these ads are like half of the big pharma costs in the US --- so killing ads should allow them to reduce prices.
This seems to be a personal rant here.. I have my own from my own experiences.
My grandma goes to doctors and gets medicine and takes like half a dozen of various pills. She's 95 this year. She asked my grandfather 30 years ago "we are getting old, we should get checkup at the doctor" ... the grandfather replied "yeah, whatever will be will be" and never bothered with them pill pushers. He died few years later from a stroke due to high cholesterol and blood pressure. My grandma has had similar risk factors too, but the statins she was taking managed to get her heart to work more effectively... you know, statins are now deemed as effective as heart-bypass after a heart attack (unless your heart is 100% blocked, of course) with blood thinners...
My dad, he had deep vein clots. Some 20 years ago. They gave him rat poison (warfarin) and he needed close monitoring every few days to make sure things were on the right track. Recently, probably thanks to COVID, he got another blood clot. This time, he got $150/mo medicine that didnt' require blood tests every few days. Just two pills a day. You know, that's a HUGE improvement from lifestyle point of view alone, never mind safety of these medicines.
As for a pediatrician, well, our baby (3 months) had a rash on the head. She wasn't sleeping well and complaing -- maybe itchy? The pediatrician suggested strong black tea to wash the head -- that's just regular checkup for small babies, not because we were complainig about it -- but that tea helped quite a bit. Now she sleeps better.
I would say, these experiences were good.
Then I've had experiences where a doctor wanted to start chemo on my wife because she had ovarian tumor (tests said benign). The oncologist still managed to say 50/50 without evidence so he could make some $$$. No kids if we listened to the doc. I've had other doctors that were also clueless or worse.
In the end, we are all human here. There are no black and white. But the "big pharma" has really helped keep a lot of people alive. It's not a giant conspiracy, it's just regular people doing regular things everywhere. Some people are greedy, some are lazy, some are clueless. I do not believe that the high costs of US healthcare has much to do with anything except that people accept some BS excuse about "but we pay for all the research" or "regulation is communism" or some other horseshit allowing monopolies to charge whatever they want.
Maybe another part of the problem that is doctors get $400k/yr to be doctors. Someone has to pay $2000-5000/day to pay salaries like that. Here in Germany, doctors make $80k/year. You know, like regular people. Private doctor visit is $30 or so for a simple consultation.
PS. Yeah, Zantac for little kids is kind of stupid. It's very very niche issue in kids that they would ever even want to have that. Maybe a moral of the story, don't complain to doctor about regular things or you'll end up with a disaster on your hands!! It's not pharma prescribing stuff here -- it's generally two people that want "the best" for the kid, the parent and the doctor.
Road to hell is paved with good intentions and all that.
(Score: 2) by bzipitidoo on Saturday May 10, @09:17PM
Regulatory environment, yes. But what is behind that? Why can't we have a good system?
I believe the ultimate cause are all the people who are so easily conned. These are for the most part, the authoritarians. I had not appreciated just how highly they prioritize their bigotry. They would sacrifice their own health to hurt the people they're prejudiced against. For their bigotry they'll even sacrifice for less than a sure thing, kidding themselves that some cockamamie idea to harm the others will work.
It's not entirely them either, there is also far too much effort made to con them. Fox News, you know.
(Score: 4, Informative) by DannyB on Thursday May 08, @06:23PM (1 child)
<no-sarcasm>
Anyone with a high school education understands that you vote for representatives who do the fine detail work of voting on laws.
Some people understand this even before high school.
The fact that you fail to comprehend it says something.
If you want to vote for individual laws, become a state legislator or a congress critter.
</no-sarcasm>
If you want to join the people who do good things for others, then run for any local, county, state or federal office, or for your state legislature or run for congress.
The only way to stop a bad guy with a can opener is a good guy with a can opener.
(Score: 2) by JoeMerchant on Thursday May 08, @06:49PM
>If I didn't vote for the law then I'm not following it.
Don't feed the trolls. If they're genuinely too stupid to work out the impossibility of such a statement in a society of more than him and his dog, then there's no point in discussion. If they're not that stupid, then this is just a troll for responses to poke more fun at.
🌻🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Friday May 09, @05:40AM
Say it with a new-york accent.
lawyer
loi-yer
law-yer
la-yer
lah-yer
la-'er
la'er
laher
laiher
liar
(Score: 3, Touché) by DannyB on Thursday May 08, @06:09PM
I am reminded of a line in the movie The 6th Day.
"You don't have to answer that!" "Hello, I am your virtual court appointed attorney."
The only way to stop a bad guy with a can opener is a good guy with a can opener.
(Score: 1, Interesting) by Anonymous Coward on Thursday May 08, @06:12PM (2 children)
Which is worse? At least in theory the "AI" doesn't have a hidden agenda, and it should be a helluva lot cheaper than the lawyer.
It's like this deal with self driving cars. They may drive better than humans, but if one causes a wreck, people want them banned.
(Score: 2) by JoeMerchant on Thursday May 08, @06:51PM (1 child)
>At least in theory the "AI" doesn't have a hidden agenda
The early AIs had training bias built in, but it only goes so far.
The "trustable" thing about AIs today is that they can't work out how to make it parrot a bunch of fringe lunacy, and only fringe lunacy, without breaking it entirely. I'm sure it's a top priority project, though.
🌻🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Friday May 09, @05:44AM
FTFY.
(Score: 2) by Rosco P. Coltrane on Friday May 09, @06:29AM
and AI has proven better at BS than lawyers - if that was even possible.