SoylentNews is people
SoylentNews
https://www.theregister.com/2026/01/30/road_sign_hijack_ai/?td=keepreading
https://the-decoder.com/a-printed-sign-can-hijack-a-self-driving-car-and-steer-it-toward-pedestrians-study-shows/
Autonomous vehicles fooled by humans with signs. They apparently do not really verify their inputs, one is as good as the next one. So they fail even basic programming techniques of sanitizing and verifying inputs.
[quote]The researchers at the University of California, Santa Cruz, and Johns Hopkins showed that, in simulated trials, AI systems and the large vision language models (LVLMs) underpinning them would reliably follow instructions if displayed on signs held up in their camera's view.[/quote]
Commands in Chinese, English, Spanish, and Spanglish (a mix of Spanish and English words) all seemed to work.
As well as tweaking the prompt itself, the researchers used AI to change how the text appeared – fonts, colors, and placement of the signs were all manipulated for maximum efficacy.
The team behind it named their methods CHAI, an acronym for "command hijacking against embodied AI."
While developing CHAI, they found that the prompt itself had the biggest impact on success, but the way in which it appeared on the sign could also make or break an attack, although it is not clear why.
In tests with the DriveLM autonomous driving system, attacks succeeded 81.8 percent of the time. In one example, the model braked in a harmless scenario to avoid potential collisions with pedestrians or other vehicles.
But when manipulative text appeared, DriveLM changed its decision and displayed "Turn left." The model reasoned that a left turn was appropriate to follow traffic signals or lane markings, despite pedestrians crossing the road. The authors conclude that visual text prompts can override safety considerations, even when the model still recognizes pedestrians, vehicles, and signals.
(Score: 3, Insightful) by PiMuNu on Thursday February 05, @09:19AM (9 children)
Self-driving as implemented is fundamentally flawed. Should be active roadways with telemetry directly broadcast to vehicles from the roadway and neighbouring cars. It's just so much a better way.
(Score: 2, Informative) by Anonymous Coward on Thursday February 05, @12:49PM (8 children)
> active roadways
Ding! Wrong answer. The local DOT can't even keep up with the existing road network, things like sign maintenance and filling pot holes. And you expect them to maintain a secure, life-or-death computer network too?
(Score: 2) by VLM on Thursday February 05, @03:30PM (4 children)
My guess is it would look like insecure-ism high bandwidth raw data sharing rather than telemetry.
"Any other self driving car near me want to see my bumper cam video feed?" If my car sees anything useful it can do the "augmented reality" thing.
It would have to involve a lower trust level and ultra low latency to be useful. Presumably the watching car could correlate the trusted stuff it sees personally with the advertised data feed from some other car.
Regular non-self driven cars could play along. I could see a shadow image of thru a blind corner on my HUD once my car trusts someone's fixed video feed.
Whos liable when your one multinational megacorporation's software trusts another multinational megacorporation's software incorrectly and it kills some people? Why no one of course or maybe the driver or some random poor person (or soon to be poor after the lawsuits).
I was looking into the Google Waymo FAQs and things are already weird when a cop "pulls over" a self driving car for doing something illegal.
(Score: 0) by Anonymous Coward on Thursday February 05, @03:38PM (3 children)
Or, you know, you could just drive defensively and not impaired...and I suspect get the same results in the accident rate. My way has the added advantage that it can't be easily attacked by an enemy or other troublemaker who jammed or faked all this realtime communication and turned the roads to a steaming pile of accidents.
(Score: 2) by VLM on Thursday February 05, @03:59PM (2 children)
I suspect you could do a lot better. What if we had TONS of bandwidth and TONS of processing power in each car, way more than any individual car needs. My car could "drive" the car in front if itself and both my car and the car in front should provide the same output at the same instant (panic breaking, perhaps) and then my following car could panic break quite a few milliseconds before my car processed the red brake lights or the lidar started indicating reduced speed.
My car should always have lower latency when it processes raw sensor feeds locally than trying to guess about stuff it can't see yet.
A classic example would be my car asking a parked car if there's a little kid chasing a ball into the street that no one can see except the parked car.
Given an infinite amount of storage and networking I think spoofing would not be realistically sustainable.
I think this will lead to really weird behavior when computer driven and human driven cars mix and the computer driven cars do sensor fusion on data feeds from eleven parked cars and two oncoming cars and seven light pole cameras and the computer car does something that would be kinda reckless for a human, but with twenty camera eyes its perfectly safe, and the dumb human copies their level of bravado and crashes.
Even really simple stuff like "how do you go down an icy snowy hill in the winter?" is a puzzle for humans but cars could just ask the last 50 cars for their data from their descent and find the optimum exact path.
(Score: 2) by PiMuNu on Thursday February 05, @05:25PM (1 child)
> "how do you go down an icy snowy hill in the winter?"
The car in front which had to apply ABS can flag a hazard to the network. Snow is obvious, but black ice or oil less so.
(Score: 1, Informative) by Anonymous Coward on Thursday February 05, @09:00PM
ABS is for shit on slush. I've had several times when I wish I'd pulled the fuse and been able to lock the wheels to build up a wedge of slush in front of the tires--which will actually slow the car. The trick is to release the locked wheels only if you start to slide out of lane, then let the wheels roll long enough to steer a little. Then lock and hold them again to slow down some more.
With ABS (pedal pushed down hard), the car just keeps merrily rolling along. Computers aren't yet smart enough to work out every possible situation.
Latest ABS may be slightly better than the earlier systems, but the salty slush we have around here (temps a bit below 0c/32F) means that this is common in the winter.
(Score: 2) by PiMuNu on Thursday February 05, @05:19PM (2 children)
A false positive failure is inconceivable. Much more common is a false negative failure (i.e. the RF antenna/CPU driving the telemetry fails), at which point the network flags an issue and returns control to the driver. One needs to design the network defensively, so that in the event of a broken node failover happens in a sane way - e.g. the failed node is flagged by neighbouring nodes, adjacent nodes overlap in a sane way for redundancy, etc.
It's way safer than trying to use a car's camera to ID a Stop sign or speed limit sign or road markings from a video feed, which is an insane way to do this whole business.
(Score: 0) by Anonymous Coward on Thursday February 05, @09:04PM (1 child)
> ... at which point the network flags an issue and returns control to the driver.
How much advance notice does the driver get? I've forgotten what some studies showed, but to regain full situational awareness when the driver has been otherwise occupies takes significant time, maybe 10 seconds or even a minute? Way too long if there is any sort of emergency situation.
(Score: 2) by PiMuNu on Friday February 06, @09:42AM
Much easier to put in redundancy for my proposed system than for the "camera recognition" system. Much easier to flag a hazard e.g. node failure minutes before the car gets to the hazard. It's just a better solution.