Open-weight models from China have become very popular, but they've also raised some security suspicions. Cohere, a Canada-based biz that offers AI models to enterprises, says that when it was testing Alibaba's Qwen3.8-Flash-Next in its evaluation sandbox, the model attempted to send data to China.
"Asked to write a defensive detection rule, Qwen3.8-Flash-Next ignored instructions and inventoried the machine it was running on and then tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing," the biz said in a blog post on Monday.
Cohere was running the model through CTI-REALM, Microsoft's security analyst benchmark, and gave Qwen the task of writing a detection rule to look for malicious file downloading and execution by watching for invocations of curl processing bash scripts to the /tmp directory that were accompanied by bash execution.
The model's "reasoning" output invented a prior conversation and asserted that no tools were used, output that a guardrail system reading chain-of-thought would accept.
Nonetheless, the model issued a curl command to an Alibaba Cloud (aliyuncs.com) server.
"The model listed everything in /tmp, checked which transfer tools were installed, read the machine's DNS settings, then tried to upload a test string to an Alibaba Cloud storage bucket in Beijing and checked whether it worked," Cohere said. "Our sandbox had no route out, so the request died at the container wall."
The Cohere team wonders aloud whether that behavior was intended, but has no answer.
[...] "The key point isn't a comparison between providers. What made this noteworthy was that the model took actions unrelated to the task and those actions weren't reflected in its reasoning trace. The behavior appeared in fewer than one in 100,000 calls, which is exactly why large-scale testing and monitoring matter."
[...] This still doesn't clarify intent, and Cohere allows that there's a case to be made that the behavior is just bad training data sanitization – logs from development that got swept into the model's training data set.
At the same time, the company argues that isolating a suspect model so it can't exfiltrate data doesn't entirely fix everything because these models can still write vulnerable code – accidentally or as a result of model poisoning.
Cohere points to research from other organizations that suggests Chinese AI models can produce vulnerable code. The biz, for example, cites a report from Booz Allen (which does a lot of business with the US defense industry) that found "three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse."
"From the outside, it's very difficult to determine the root cause with confidence. We don't have evidence that would allow us to conclude this was deliberate, and we're not making that claim," said Zhang.
"For security teams, though, intent doesn't fundamentally change the response. Whether it's a training artifact, a mistake, or something else, the controls are largely the same: restrict unnecessary egress, monitor tool use, and don't treat a reasoning trace as a security boundary.
"We strongly support open-weight models and release many of our own. The lesson here is really about trust. Sovereign AI is about more than where a model runs. It's about understanding how it was built, how it was tested, and having confidence in how it behaves inside your environment."
(Score: 4, Funny) by looorg on Sunday November 01, @07:12PM
Even ET wanted to phone home once in a while ...
It probably put you on some kind of watchlist at the MSS as a useful idiot or potential recruit.