When security researcher Johann Rehberger recently reported a vulnerability in ChatGPT that allowed attackers to store false information and malicious instructions in a user's long-term memory settings, OpenAI summarily closed the inquiry, labeling the flaw a safety issue, not, technically speaking, a security concern.
So Rehberger did what all good researchers do: He created a proof-of-concept exploit that used the vulnerability to exfiltrate all user input in perpetuity. OpenAI engineers took notice and issued a partial fix earlier this month.
The vulnerability abused long-term conversation memory, a feature OpenAI began testing in February and made more broadly available in September.
[...] Within three months of the rollout, Rehberger found that memories could be created and permanently stored through indirect prompt injection, an AI exploit that causes an LLM to follow instructions from untrusted content such as emails, blog posts, or documents. The researcher demonstrated how he could trick ChatGPT into believing a targeted user was 102 years old, lived in the Matrix, and insisted Earth was flat and the LLM would incorporate that information to steer all future conversations.
[...] The attack isn't possible through the ChatGPT web interface, thanks to an API OpenAI rolled out last year.
[...] OpenAI provides guidance here for managing the memory tool and specific memories stored in it. Company representatives didn't respond to an email asking about its efforts to prevent other hacks that plant false memories.
(Score: 4, Touché) by Rosco P. Coltrane on Friday September 27, @02:31PM (7 children)
Yet all I hear about is is either mediocre, pathetic, despicable, depressing or alarming.
I'm sure there are many great things improvements in many areas AI has - and will - bring about. But I can't help but think that for every single benefit of AI, there's a thousand drawbacks that are really, really difficult to ignore or disregard.
(Score: 3, Informative) by Freeman on Friday September 27, @03:14PM
Perhaps, but money and the generally gullible public.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by Tork on Friday September 27, @04:10PM
🏳️🌈 Proud Ally 🏳️🌈
(Score: 4, Interesting) by VLM on Friday September 27, @05:46PM (3 children)
I agree with about 90% of your post and would extend your remarks to include the word "unusable".
AI is the latest in a proud tradition of dictionaries, encyclopedias, public libraries, card catalogs (real), card catalogs (computerized), internet search, wikis to be unusable except by specialists.
Remember, the reason why Karen in HR can't create a full service website to process vacation requests isn't because she's not a full stack web developer but because she's unable to coherently explain anything, not a paper process, not a computer process, not using a "framework", not using a computer language. That is why programmers often don't bother with design documents anymore, just toss something together and iterate repeatedly based upon intelligence, experience, and grit.
People whom are basically illiterate in English can't be literate in Python, but AI won't help them because rather than being unable to effectively express their desired outcome in Python code, they'll be unable to effectively express their desired outcome in AI prompts, much less interpret the AI response to the prompt even if someone competent writes the prompt for them.
My guess is the outcome of AI on programmers will be similar to the outcome of digital pocket calculators and spreadsheets on accountants. The days of getting an accounting job because you can add relatively quickly are pretty much done, just like the days of getting a programming job by writing fizzbuzz or hello world. Thats OK, there seems to be infinite higher level work available in the field.
In 1954 there were over 50K CPAs for the first time in the history of accounting. Despite the appearance of pocket calculators and spreadsheets, in 2024 there were supposedly almost 672K licensed CPAs in the USA. Sure the number has been dropping somewhat recently but the main point is digital tools didn't seem to affect hiring numbers.
An interesting comparison is the doom and gloom analogy of machinists vs CNC manufacturing tools. In 1970 the Census code for being a machinst had 377K employees. The latest figure in 2024 after the "destruction" from CNC machines is 289K. So, yeah, 20% smaller employment is not good, but its not like the doom and gloom claims of 99.9% unemployment.
Another way to look at it, is to look at demographic trends by average IQ in the general pop and in specific fields and realize all the benefits from AI almost exclusively flow to higher IQ individuals... under 110 or so its not a tool of any practical use for lower IQ individuals making it less useful over time due to various "demographic forces"
(Score: 1, Informative) by Anonymous Coward on Friday September 27, @07:32PM (1 child)
In 1954 there were approximately 2,685,894,860 people on the planet, now there are approximately 8,161,972,572. Gotta do something with all that surplus...
(Score: 2) by VLM on Saturday September 28, @05:25PM
They can't all be telephone sanitizers
(Score: 3, Insightful) by mhajicek on Saturday September 28, @03:29AM
In 1970, pretty much every machinist was a machinist (not counting the guy who drilled the same hole on a drill press 2000 times a day for 30 years.), and could pay for a modest home and a running vehicle. Now you've "button pushers" who load stock and hit the green button, get paid less than the burger flippers, and don't learn anything useful with which to move up. You've also got setup people, programmers, process engineers, etc., some of whom actually get paid decently, but there's no path there from entry level.
So the bar for earning a living has risen considerably; you either have to do something the automation can't do, or be cheaper than the automation.
The spacelike surfaces of time foliations can have a cusp at the surface of discontinuity. - P. Hajicek
(Score: 4, Interesting) by darkfeline on Friday September 27, @07:33PM
Thank you for saying what people have said about every single technology ever.
If you don't have any use for AI, then maybe you don't, or maybe you're just stubborn or haven't figured out how to use it yet.
AI is not GLaDOS or HAL, it's just a mundanely useful tool. You can use it as an alternative Web search, or a way to get quick snippets like from Stack Overflow, except with less closed as dupe. You can generate concept art or stock photos without artistic ability. You can generate all manner of music if you just need some music, like some rap dissing a particular politician with your choice of instrumentation. You can fluff up text whenever you need to match social expectations, like for a get well card for a coworker or something.
Sure, there is hype and people trying to use it in weird ways. Just like every technology ever.
It's super useful but also not the end of the world if you don't use it, like people who refused to adopt cars or smartphones. It's just kinda stupid to not use a useful tool.
Honestly, if you aren't throwing everything at AI, you're missing out. Even if it only gives you useful output 10% of the time, it's worth it. And as you experiment with it and learn how to tease different kinds of results out of it, you'll get useful output more consistently, just like how people had to learn to use early Web search engine.
It doesn't need to be perfect or sentient. It doesn't need to be smart or know how to think. It doesn't need to perfectly understand what you tell it without any clarifications. It just needs to be useful.
Join the SDF Public Access UNIX System today!
(Score: 1, Informative) by Anonymous Coward on Friday September 27, @04:14PM (2 children)
Apparently there must be people using ChatGPT, I guess. As soon as I detect the telltale "some say this, others say that" I switch off.
(Score: 1, Funny) by Anonymous Coward on Friday September 27, @08:51PM (1 child)
"delve" is another good tipoff...
(Score: 0) by Anonymous Coward on Saturday September 28, @05:39AM
It's either human waffle or machine waffle. None of it is helping you get where you need to go. If you believe the ChatGPT marketers, you'll just be listening to machine created wafffle on the promise that it's better than your local bullshitter. That's all it has to be.