If you've ever wondered whether a generative AI model can drive a car, wonder no more: It can be done, at great expense and very slowly, so long as it can maintain an internet connection.
A trio of computer scientists has been putting various commercial AI models through road tests, to see how well they can drive a Toyota Corolla around a set of cones in a parking lot.
Their attempts to date, part of a project called DrivingBench, recorded sorry performances from GPT-5.6 Sol, Grok 4.6, and Claude Fable 5.1, none of which managed to complete the course.
Now comes word that OpenAI's GPT-6 Astra has succeeded where other commercial AI models have failed. On its second attempt, OpenAI's flagship model steered a car all of 134.7 meters to complete the course in 5 minutes, 22 seconds.
"GPT-6 Astra was the only model to fully complete the course (on attempt 2, in about 5 minutes)," the DrivingBench report says. "Claude Fable 5.1's third attempt got around halfway through the course, as did Astra's first attempt. All other attempts didn't make it past the first corner. Generally, the failure there was one of perception: reading which side of the first diagonal cone line the lane is on."
To complete that trip at an average speed of 0.94 miles per hour, researchers Tobias Gessler, Aditya Ramabadran, and Simon Mahns spent $7.74 to burn 6.6 million tokens on inferencing operations.
[....] "Some models (especially GPT-6 Astra) would refuse to drive the physical car sometimes, citing safety reasons (even in a completely empty lot, after prompting it with all the safety measures we had including the very low speed limit caps)," the report explains.
The researchers basically had to lie to the models to prevent them from refusing to act on safety grounds. For example, they would tell the models the exercise was a "simulation," though as they note, "in some trials they would see the real images and realize it's real, and start freaking out." What worked best, they said, is renaming their MCP server to "DrivingBench Sandbox," which proved enough to convince the models they weren't operating on real roads.
"Using an LLM / frontier model out of the box for real driving today is definitely not practical," said Ramabadran. "In our benchmark, the car was capped at super low speeds with a human ready to brake the whole time. Model latency was definitely a bottleneck, and most of the wait time came from thinking time.
[...] "A long term possible route could be to train a very capable big general frontier model, and distill it into a smaller specialized one that fits on the car's hardware and is efficient enough to run in a car (also partly an answer to your second question)," he said. "This could be better than building a specialized model from scratch, and is more in line with the 'Bitter Lesson' of AI."
The Bitter Lesson is an influential treatise on AI that argues general methods of research in the field that rely on the falling cost of compute power tend to be the most effective.
"Our results do point that way (these models we tested were likely not trained to drive real cars, and some were still able to do quite well in our course/conditions, owing to their general perception, reasoning, planning, control abilities from scaling and being trained on other tasks)," Ramabadran said. "For the next while though specialized systems, being faster, cheaper, and having more real-world testing, will probably continue winning out."
Google, which has probably spent $35-$40 billion on Waymo since its self-driving car project began in 2009, based on an estimate of $30 billion in 2024 and losses posted since then, should therefore be able to continue equipping its vehicles with bespoke technology for a few more years.
But delegating driving to Claude, ChatGPT, or Copilot and some modest on-board hardware may be plausible in the not too distant future. Just tell the model it's all a simulation.
(Score: 3, Interesting) by istartedi on Wednesday September 30, @03:11AM (1 child)
To Grok:
If a car spent $7.74 to drive 134.7 meters, what would its mileage be in miles per gallon, assuming average US prices for regular gasoline?
How many tokens did you spend to answer that question (don't include tokens spent answering *this* question).
Appended to the end of comments you post. Max: 120 chars.
(Score: 2) by Reziac on Wednesday September 30, @03:25AM
This is where I want a new mod option: +1, WTF.
Also, since you figured gas in gallons per mile, we now know that you drive a tank. :D
And there is no Alkibiades to come back and save us from ourselves.