SoylentNews

In a comprehensive tweet that everyone should read, UC Berkeley computer science professor Dawn Song wrote that ExploitGym's "evaluations are designed to run within isolated sandbox environments with tightly restricted network access." Song is also a member of the team at UC Berkeley's Center for Responsible Decentralized Intelligence, which is responsible for the development of ExploitGym. Via email, Song told me that ExploitGym essentially comes with its own sandbox but that companies like OpenAI are free to make substitutions. 

According to Song, "network and model-API proxies that restrict which external services the agent can reach" are among the major components of the ExploitGym reference implementation. "During the actual agent-execution phase in our current reference setup, outbound network access is restricted to the services necessary for the evaluation. We also use an LLM proxy to block provider-side web search, remote URL retrieval, hosted code execution, and similar channels that could otherwise bypass a container-level firewall," Song told me. "OpenAI may have modified this architecture to fit its own internal infrastructure, scaling requirements, and evaluation systems. This is a common practice as they have different needs for their internal evaluation. We cannot say precisely how its implementation differed from ours."

In addition, the ExploitGym team was clearly aware of how, with tests governed by their framework, models were exhibiting potentially pernicious behavior. According to Song's tweet, "During development, we observed models probing the surrounding infrastructure for additional privileges or information beyond what was intended for the task. We also deliberately tasked models with stress-testing our infrastructure so that we could identify and fix potential weaknesses." 

[...] In other words, when used for safety testing and benchmarking purposes, and when enabled by powerful LLMs, ExploitGym-enabled agents are essentially expected to try everything within their power to break out of whatever sandbox is trying to securely contain them. 

And that's precisely what happened in this case. As OpenAI wrote in its disclosure, "While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy."

[...] According to OpenAI's disclosure, the company was gauging the cyber capabilities of its models "by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity." As OpenAI hasn't responded to my inquiries regarding this incident, I asked AppOmni's Ruzzi what a "production classifier" is. 

Ruzzi explained that it's a parameter that tells the model whether or not it's running in a production environment. If it's a production environment like the publicly available version of ChatGPT, then all safety guardrails are activated. But if the model is in a non-production environment -- for example, the ExploitGym testing environment that's supposedly sealed off from the outside world -- then the guardrails are relaxed, and the model is given the leeway to pursue "high-risk cyberactivity." 

In layperson's terms, they took the seat belts off the crash test dummy before driving the car into a brick wall. It was another human decision, in a series of human decisions, that led to the so-called escape of the agent.  

Keep in mind that the models were tasked to solve a problem. They weren't told how to solve the problem, nor were they asked to respect any moral, ethical, or legal boundaries. In other words, they were afforded -- by humans -- the agency to try anything. 

Including cheating. 

Perhaps the most interesting point in OpenAI's disclosure comes when it says, "After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation."

Let's unpack that. Although we can't be 100% certain from the information provided, the implication is that, among the steps that the models were willing to take to solve the so-called "evaluation problem," one was simply to cheat by stealing a pre-existing solution (perhaps one "honestly" figured out by another model). According to Ruzzi, the models had a key piece of information to go on: from their ability to probe their testing environment, they could tell they were taking part in an evaluation governed by ExploitGym. And when the models began looking for the latest information on ExploitGym, they likely encountered a page like this one, hosted on Hugging Face's systems. Suddenly, Hugging Face became a target of interest, and in much the same way as the models relentlessly exhausted every option to break out of their confines in OpenAI's AI testing infrastructure, they exhausted all possible options to break into Hugging Face's systems.

Ruzzi was quick to note that the word "cheat" is a bit loaded in this context, implying that the models were not emotionally compelled to pass the test without doing any of the hard work. OpenAI may have chosen to use the word "cheat" because of what humans understand the word to mean. But the model never thought to itself, "Ha ha, I'm going to beat the system." It just pursued a plausible path to attain the objective. 

[...] In the attack on Hugging Face, did a machine act on its own? It's a moot question. Knowing exactly how we got here -- was it sentience... malice...agency? -- is relevant to whatever remedy lies ahead. Most important, however, is that we've arrived at a teachable moment in which human AI experts gave AI an objective, lost control of AI's pursuit of that objective, and then AI demonstrated unmatched tenacity and speed in accomplishing it. 

In the big picture, it's that unmatched tenacity and speed that should be most worrisome. Never mind a relatively benign cyber-incident like this one. There should be no doubt that threat actors paid close attention to what happened here.  

Fortunately, OpenAI is not a threat actor. As far as we know, the damage amounted to an unfortunate but timely demonstration of cyber capability rather than any intent to inflict lasting damage on another company. Yes, it will happen again. If not by accident, then at the hands of an adversary. And when adversaries are involved, they'll go after soft but valuable targets (like big businesses that lack the defenses Hugging Face has in place). When that time comes, it's anybody's guess just how prepared we'll be. 

Original Submission