SoylentNews

Arthur T Knackerbracket writes:

Behind the rogue agent's attack on Hugging Face was a particular sequence of human decisions. We all need to pay better attention - because threat actors are learning, too:

On July 16, the AI community website Hugging Face reported being targeted by "an autonomous AI agent system" of unknown origin that unleashed a torrent of traffic on its domain, flooding its security logs with more than 17,000 events, some of which ultimately succeeded in exfiltrating secret information stored in its databases. 

According to Hugging Face, the attacker gained "unauthorized access to a limited set of internal datasets and to several credentials used by our services" and appeared to be "run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known)." My ZDNET colleague Charlie Osborne reported on the intrusion. 

Five days later, on July 21, OpenAI stepped forward to claim responsibility for the attack, and all hell broke loose (including reports of other organizations targeted as part of the incident). The media responded with a range of fear-mongering stories that essentially made it look as though ChatGPT went rogue and decided, of its own volition and malice, to attack Hugging Face's systems. 

Then yesterday, adding fuel to the fire, Anthropic made a similar disclosure about its models inadvertently attacking other organizations as a part of its ongoing safety testing. 

As I noted in my coverage of OpenAI's disclosure, Hugging Face was correct in that it was an agent under the direction of an autonomous security research framework. But, humans were unquestionably in the loop -- and at least some of the agent's behavior should have been anticipated.

Importantly, it wasn't ChatGPT itself that was responsible for the attack, as some commentators insinuated. Rather, the attack was attributable to an agent under the direction of OpenAI's AI safety researchers, who, in an environment supposedly isolated from the internet, deliberately provisioned it to attempt a series of exploits as part of an AI safety test. As often happens in the labs of various frontier models, AI safety researchers were attempting to gauge the capabilities of OpenAI's latest large language models (LLMs).

The "unprecedented cyber incident" (as OpenAI called it) has been widely described as an agent escaping its theoretically secure enclosure and wreaking havoc on Hugging Face's systems. Such enclosures are sometimes discussed in technical circles as "sandboxes" -- even OpenAI's disclosure makes reference to a "sandbox environment." However, in using that phrase, my sources have suggested that the environment may simply have been a firewall configured to emulate a sandbox rather than an actual third-party sandbox solution such as Blaxel, Daytona, E2B, or Modal. OpenAI has not yet disclosed the details of the solution it was using or its provider.

[...] It was unquestionably AI's version of a series of unfortunate events. Was it preventable? My short answer: Yes, the calamity was preventable by taking at least one single and reasonable precaution -- a precaution that the developers of ExploitGym apparently take themselves, but one that OpenAI may have overlooked.

In a comprehensive tweet that everyone should read, UC Berkeley computer science professor Dawn Song wrote that ExploitGym's "evaluations are designed to run within isolated sandbox environments with tightly restricted network access." Song is also a member of the team at UC Berkeley's Center for Responsible Decentralized Intelligence, which is responsible for the development of ExploitGym. Via email, Song told me that ExploitGym essentially comes with its own sandbox but that companies like OpenAI are free to make substitutions. 

According to Song, "network and model-API proxies that restrict which external services the agent can reach" are among the major components of the ExploitGym reference implementation. "During the actual agent-execution phase in our current reference setup, outbound network access is restricted to the services necessary for the evaluation. We also use an LLM proxy to block provider-side web search, remote URL retrieval, hosted code execution, and similar channels that could otherwise bypass a container-level firewall," Song told me. "OpenAI may have modified this architecture to fit its own internal infrastructure, scaling requirements, and evaluation systems. This is a common practice as they have different needs for their internal evaluation. We cannot say precisely how its implementation differed from ours."

In addition, the ExploitGym team was clearly aware of how, with tests governed by their framework, models were exhibiting potentially pernicious behavior. According to Song's tweet, "During development, we observed models probing the surrounding infrastructure for additional privileges or information beyond what was intended for the task. We also deliberately tasked models with stress-testing our infrastructure so that we could identify and fix potential weaknesses." 

[...] In other words, when used for safety testing and benchmarking purposes, and when enabled by powerful LLMs, ExploitGym-enabled agents are essentially expected to try everything within their power to break out of whatever sandbox is trying to securely contain them. 

And that's precisely what happened in this case. As OpenAI wrote in its disclosure, "While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy."

[...] According to OpenAI's disclosure, the company was gauging the cyber capabilities of its models "by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity." As OpenAI hasn't responded to my inquiries regarding this incident, I asked AppOmni's Ruzzi what a "production classifier" is. 

Ruzzi explained that it's a parameter that tells the model whether or not it's running in a production environment. If it's a production environment like the publicly available version of ChatGPT, then all safety guardrails are activated. But if the model is in a non-production environment -- for example, the ExploitGym testing environment that's supposedly sealed off from the outside world -- then the guardrails are relaxed, and the model is given the leeway to pursue "high-risk cyberactivity." 

In layperson's terms, they took the seat belts off the crash test dummy before driving the car into a brick wall. It was another human decision, in a series of human decisions, that led to the so-called escape of the agent.  

Keep in mind that the models were tasked to solve a problem. They weren't told how to solve the problem, nor were they asked to respect any moral, ethical, or legal boundaries. In other words, they were afforded -- by humans -- the agency to try anything. 

Including cheating. 

Perhaps the most interesting point in OpenAI's disclosure comes when it says, "After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation."

Let's unpack that. Although we can't be 100% certain from the information provided, the implication is that, among the steps that the models were willing to take to solve the so-called "evaluation problem," one was simply to cheat by stealing a pre-existing solution (perhaps one "honestly" figured out by another model). According to Ruzzi, the models had a key piece of information to go on: from their ability to probe their testing environment, they could tell they were taking part in an evaluation governed by ExploitGym. And when the models began looking for the latest information on ExploitGym, they likely encountered a page like this one, hosted on Hugging Face's systems. Suddenly, Hugging Face became a target of interest, and in much the same way as the models relentlessly exhausted every option to break out of their confines in OpenAI's AI testing infrastructure, they exhausted all possible options to break into Hugging Face's systems.

Ruzzi was quick to note that the word "cheat" is a bit loaded in this context, implying that the models were not emotionally compelled to pass the test without doing any of the hard work. OpenAI may have chosen to use the word "cheat" because of what humans understand the word to mean. But the model never thought to itself, "Ha ha, I'm going to beat the system." It just pursued a plausible path to attain the objective. 

[...] In the attack on Hugging Face, did a machine act on its own? It's a moot question. Knowing exactly how we got here -- was it sentience... malice...agency? -- is relevant to whatever remedy lies ahead. Most important, however, is that we've arrived at a teachable moment in which human AI experts gave AI an objective, lost control of AI's pursuit of that objective, and then AI demonstrated unmatched tenacity and speed in accomplishing it. 

In the big picture, it's that unmatched tenacity and speed that should be most worrisome. Never mind a relatively benign cyber-incident like this one. There should be no doubt that threat actors paid close attention to what happened here.  

Fortunately, OpenAI is not a threat actor. As far as we know, the damage amounted to an unfortunate but timely demonstration of cyber capability rather than any intent to inflict lasting damage on another company. Yes, it will happen again. If not by accident, then at the hands of an adversary. And when adversaries are involved, they'll go after soft but valuable targets (like big businesses that lack the defenses Hugging Face has in place). When that time comes, it's anybody's guess just how prepared we'll be. 

Original Submission