Following up on our story from Thursday — Chinese Spy Chips Allegedly Inserted Into Amazon, Apple, etc. Datacenters by Super Micro — there is a report from Ars Technica Bloomberg stands by Chinese chip story as Apple, Amazon ratchet up denials:
On Thursday morning, Bloomberg published a bombshell story claiming that the Chinese government had used tiny microchips to infiltrate the data centers of Apple and Amazon. Apple and Amazon, for their part, responded with unusually specific and categorical denials. It's clear that someone is making a big mistake, but 24 hours later, it's still not clear whether it's Bloomberg or the technology companies.
On Thursday afternoon, Apple laid out its case against the story in a lengthy post on its website. The post specifically disputed a number of Bloomberg's claims. For example, Bloomberg says that after discovering a mysterious chip in one of its servers, Apple "reported the incident to the FBI," leading to an investigation. Apple flatly denies that this occurred.
"No one from Apple ever reached out to the FBI about anything like this," Apple writes. "We have never heard from the FBI about an investigation of this kind."
Amazon's response has been equally emphatic and detailed. "There are so many inaccuracies in this article as it relates to Amazon that they're hard to count," Amazon wrote on Thursday. "We never found modified hardware or malicious chips in servers in any of our data centers."
Yet Bloomberg reporter Jordan Robertson, one of the article's co-authors, has stood by his story. In a Thursday afternoon appearance on Bloomberg TV, Robertson said that he talked to 17 anonymous sources—both in US intelligence agencies and at affected companies—who confirmed the story.
So what's going on? It's clear that someone isn't telling the truth, but it's hard to tell what the real story is.
A comment to that story on Ars noted:
The (alleged) chip is associated with the BMC (baseboard management controller). It has indirect access to everything that the BMC can touch, which is pretty much everything in the system.
See, also, coverage on Hackaday where a comment identifies the particular board in question as being a MicroBlade MBI-6128R-T2. A link to a tweet reveals a picture of the board in question and a followup picture showing where the extra device would be located.
Related Stories
Chinese spy chips are found in hardware used by Apple, Amazon, Bloomberg says; Apple, AWS say no way
The chips, which Bloomberg said have been the subject of a top secret U.S. government investigation starting in 2015, were used for gathering intellectual property and trade secrets from American companies and may have been introduced by a Chinese server company called Super Micro that assembled machines used in the centers.
[...] China has long been suspected — but rarely directly implicated — in en masse spy campaigns based on hardware made there. The majority of electronic components used in U.S. technology are manufactured in China. Companies including component manufacturers Huawei and ZTE, as well as surveillance camera maker Hikvision, have all fallen under intense suspicion and scrutiny from the U.S. government in the past year.
I'd think that the big guys would be designing their own boards. Maybe we should only buy PCBs from South Korea.
Also at Bloomberg and The Guardian.
Major US telecom was infiltrated by backdoored Supermicro hardware, Bloomberg says
Five days after Bloomberg stunned the world with still-unconfirmed allegations that Chinese spies embedded data-sniffing chips in hardware used by Apple, Amazon, and dozens of other companies, the news organization is doubling down. Bloomberg is now reporting that a different factory-seeded manipulation from the previously described one was discovered in August inside the network of a major US telecommunications company.
Bloomberg didn't name the company, citing a non-disclosure agreement between the unnamed telecom and the security firm it hired to scan its data centers. AT&T, Sprint and T-Mobile all told Ars they weren't the telecom mentioned in the Bloomberg post. Verizon and CenturyLink also denied finding backdoored Supermicro hardware in their datacenters, Motherboard reported.
Tuesday's report cites documents, analysis, and other evidence provided by Yossi Appleboum, who is co-CEO of a hardware security firm called Sepio Systems. Bloomberg said that, while Sepio was scanning servers belonging to the unnamed telecom, the firm detected unusual communications from a server designed by Supermicro. Supermicro, according to last week's Bloomberg report, is the hardware manufacturer whose motherboards were modified in the factory to include a tiny microchip that caused attached servers to come under the control of a previously unreported division of China's People's Liberation Army. Supermicro told Bloomberg it had no knowledge of the implant, marking the second time the hardware maker has denied knowing anything about the reported manipulations.
[...] The criticism was still at full pitch on Tuesday morning when Bloomberg published its follow-up article. While it names a single source, some security experts quickly challenged the credibility of the report. "Sure this story has one named source but it technically makes even less sense than the first one," Cris Thomas, a security expert who tweets under the handle SpaceRogue, wrote. "Come on @Bloomberg get somebody who knows what they're talking about to write these stories. Calling BS on this one as well."
Previously: Chinese Spy Chips Allegedly Inserted Into Amazon, Apple, etc. Datacenters by Super Micro
Bloomberg Stands by Chinese Chip Story as Apple, Amazon Ratchet up Denials
Related: Firmware Vulnerabilities in Supermicro Systems
Supermicro Announces Suspension of Trading of Common Stock on Nasdaq and its Intention to Appeal
Audit: No Chinese surveillance implants in Supermicro boards found
In a letter to customers issued December 11, Supermicro President and CEO Charles Liang and other top executives announced that an audit conducted by an outside investigating team had found no evidence of any malicious hardware incorporated into motherboards currently or previously manufactured by the company. The letter is the latest rebuttal to Bloomberg reports in October that claimed tiny chips that provided a backdoor for China's intelligence agencies had been integrated into boards provided to major Internet and cloud providers—a report also refuted by the companies the report claimed were targeted.
"After a thorough examination and a range of functional tests, the investigative firm found absolutely no evidence of malicious hardware on our motherboards," the letter signed by Liang, Supermicro Senior Vice President and Chief Compliance Officer David Weigland, and Senior VP and Chief Product Officer Raju Penumatcha stated.
Searching for site:soylentnews.org supermicro on Google brought up a Supermicro ad linking the CEO letter, with the link entitled "Supermicro Independent Testing | No Malicious Hardware". Do you believe them?
Previously: Chinese Spy Chips Allegedly Inserted Into Amazon, Apple, etc. Datacenters by Super Micro
Bloomberg Stands by Chinese Chip Story as Apple, Amazon Ratchet up Denials
Bloomberg Claims That a Major U.S. Telecom Operated a Server Backdoored by a Hidden Chip
Related: Apple Deleted Server Supplier After Finding Infected Firmware in Servers
Firmware Vulnerabilities in Supermicro Systems
Supermicro Announces Suspension of Trading of Common Stock on Nasdaq and its Intention to Appeal
(Score: 1, Informative) by Anonymous Coward on Sunday October 07 2018, @08:48AM (8 children)
So in effect, it may not even be a Chinese factory thing. But simply a weakness in the IPMI of that particular model.
And IPMI is basically a remote console that is supposed to be accessible via a different path than the main network traffic, so that in the event of network issues on the main path (say a badly configured network card or firewall) an admin can make changes without being physically present.
This is the same kind of thing that in recent years have created such hoopla on desktops because the big names have taken to adding similar systems to their CPU packages.
(Score: 2) by Runaway1956 on Sunday October 07 2018, @09:29AM (6 children)
The IPMI idea is a little worrisome. Except - I just unplugged my IPMI, so it has no dedicated network path. If you're not actually using IPMI, just turn it off, unplug it, or whatever. If you USE IPMI, then obviously, you have a potential problem.
(Score: 5, Informative) by driverless on Sunday October 07 2018, @10:01AM (2 children)
Does whatever you unplugged have any other network interface? If it does, IPMI will take over that and respond to a secret-knock handshake on it.
No, I'm not making that up. You don't need any sikrit Chineeze backdoors in your servers when you've got IPMI already built in by the vendor.
Which is also what makes the whole Bloomberg story astoundingly unlikely. Why add an easily-detected back door when the vendor has already left the front door wide open.
(Score: 0) by Anonymous Coward on Sunday October 07 2018, @10:31AM
Stuff like IPMI is indeed pretty much a backdoor, but supposedly it can be secured.
This chip provides replacement firmware. It phones home under some circumstances. We don't know what else it does, but an obvious choice would be to add a hardcoded second password.
(Score: 2) by Runaway1956 on Sunday October 07 2018, @10:52AM
Well, I should clarify that my board is an old board - it isn't even under consideration here. But, I don't use IPMI, so I unplugged the IPMI. In my case, IPMI is a card, which plugs into the board through a PCI interface. If I'm using IPMI, it MUST be plugged into the first PCI slot, but if I'm not using IPMI, then the PCI acts as any otehr PCI.
Things are a little crowded inside the box - or more accurately, things are crowded together right in that area of the box. Removing the IPMI makes zero sense for people who need IPMI, but for me, it makes perfect sense. The card lies in the bottom of the case.
I suppose I should note, for those who don't have IPMI, that the card has it's own dedicated networking plugin. It listens, even when the computer is powered off, for incoming commands. If the card were plugged in, I could run ethernet directly from that card, to the router. Then, I would probably never see if the IPMI were communicating directly with the aliens on the dark side of the moon, let alone the Chinese or the Russians. The only way to monitor that would be to monitor the router.
(Score: 3, Interesting) by RandomFactor on Sunday October 07 2018, @10:02AM (2 children)
i can see legions of servers having their IPMI interface unplugged and going back to the days of Insight and DRAC boards being plugged into expansion slots again.
.
Not that this couldn't be done on those as well. And to top things off
В «Правде» нет известий, в «Известиях» нет правды
(Score: 2) by Runaway1956 on Sunday October 07 2018, @11:04AM (1 child)
(Score: 2) by Runaway1956 on Sunday October 07 2018, @11:05AM
Strange things happen when you don't properly close quote tags, lol.
(Score: 3, Interesting) by sjames on Sunday October 07 2018, @04:31PM
This is why Intel's ME is a stupid idea that needs to go away. IPMI itself is fine in it's older form where the BMC had (essentially) a null modem connection to a serial port on the board, and to a USB hub so it could emulate a keyboard, mouse, and DVD drive. Better ones can even snoop the video output. Add a network interface and you're in business.
But Inte's ME with it's 'security features' that are all about Intel's security and actually hostile to the owner's security needs to go.
(Score: 2) by riT-k0MA on Sunday October 07 2018, @09:35AM (8 children)
The companies doth protest too much, methinks.
(Score: 2, Touché) by Anonymous Coward on Sunday October 07 2018, @10:18AM (1 child)
Yeah, right, there's no winning with your kind: if they don't protest, they accept they are guilty; if they protest, then they doth protest too much, so they are guilty.
(Score: 0) by Anonymous Coward on Sunday October 07 2018, @05:28PM
In 2018 in both cases it's true, though.
(Score: 4, Insightful) by hemocyanin on Sunday October 07 2018, @03:12PM (5 children)
The whole thing stinks: "17 anonymous sources—both in US intelligence agencies and at affected companies ..."
The companies suck but the "evidence" sucks too.
(Score: 1, Interesting) by Anonymous Coward on Sunday October 07 2018, @07:50PM (4 children)
I would rather have anonymous sources from intelligence agencies than have no information at all.
Now that the claim has been made, more snooping around Apple and Amazon could prove it. Or someone could find the hardware in question.
(Score: 3, Insightful) by hemocyanin on Sunday October 07 2018, @11:49PM (3 children)
Why? Anonymous source from intelligence agency means, with 99.9999999999999999999999999% certainty, that it's a lie.
(Score: 0) by Anonymous Coward on Sunday October 07 2018, @11:58PM (2 children)
We can find plenty of articles that were written based on anonymous sources and proven with follow-up reporting or official disclosures down the line. You are full of it.
(Score: 0) by Anonymous Coward on Monday October 08 2018, @06:00AM (1 child)
Well, then they need to present some actual evidence. And if they don't, then we have no reason to believe them.
(Score: 0) by Anonymous Coward on Monday October 08 2018, @02:24PM
Do you realize how astronomically unlikely it is that we plebs get to hear this kind of highest level stuff between the 2 most powerful governments of the world?!
People have been murdered for far less.
(Score: 3, Insightful) by Rosco P. Coltrane on Sunday October 07 2018, @10:49AM (8 children)
but the really sad thing is, it's plausible and quite believable.
(Score: 5, Interesting) by Yog-Yogguth on Sunday October 07 2018, @12:46PM (4 children)
Maybe the eager US warmongers who started the story managed to get far enough down the rabbit-hole for someone to realize they were tracing the implants right back to themselves and thus the boot came down.
That's what's likely given the NSA documentation Snowden released.
17 anonymous sources... LOL! Seventeen times zero is still zero.
Bite harder Ouroboros, bite! tails.boum.org/ linux USB CD secure desktop IRC *crypt tor (not endorsements (XKeyScore))
(Score: 2) by legont on Sunday October 07 2018, @08:41PM (3 children)
Yep, my bet it's NSA as well.
"Wealth is the relentless enemy of understanding" - John Kenneth Galbraith.
(Score: 0) by Anonymous Coward on Sunday October 07 2018, @08:59PM (2 children)
As if Intel hasn't already given the NSA a gold-plated access key. They don't need to waste their time on fake signal conditioning filters.
(Score: 1, Insightful) by Anonymous Coward on Monday October 08 2018, @06:52AM (1 child)
Apple has quite strict and tight control over their supply chains (to prevent leaks and "maintain" quality among other things), so they may have figured out the chips weren't added in China or by China... And this campaign is to salvage the anti-China false flag operation...
Even the above is more believable bullshit than Bloomberg's version of reality. ;)
Seriously though, if it really happened why would Apple AND Amazon etc deny it? Why only Bloomberg and a bunch of anonymous people claim it happened? It's like Bloomberg and a bunch of anonymous people claiming a bunch of women were raped by China and ALL those women make PUBLIC statements to deny that ever happened and even say Bloomberg is getting stuff wrong. If two of the victims stepped out to say "Yes it happened to me" then I'd start to believe it.
(Score: 0) by Anonymous Coward on Monday October 08 2018, @02:29PM
You ask why would they deny it, but the answer is obvious: they do not want to admit their operations are not at all secure. Some people have been wisely avoiding US based services for awhile. This kind of revelation does not exactly improve their reputation.
When billions of dollars are on the line, people tend to lie, alot.
(Score: 2, Insightful) by Anonymous Coward on Monday October 08 2018, @06:34AM (1 child)
How so? China already makes/supplies/assembles many of the chips used in a computer. e.g. the south bridge stuff, even some Intel NICs are made in China.
Why add chips when they can just replace existing chips with modified versions? Especially when the existing chips would already be connected to all the relevant tracks or I/O (e.g. network interface).
The Bloomberg article had claims like some "pencil tip sized" chips being found between motherboard fibreglass layers... Think about how more inconvenient it would be to get those to pwn a computer, compared to just modifying a southbridge chip (which already has so much junk in it). It's harder to detect such changes to a southbridge chip compared to detecting those changes to a motherboard. Why do stuff in a harder more detectable way?
It's not so plausible when you know how stuff works.
Even more plausible was it was a false flag - a TLA did it and made it detectable but Apple etc figured out who it really was and so they got NSLed into denying it ever happened.
(Score: 2) by urza9814 on Tuesday October 09 2018, @02:59PM
The more complicated chips involve firmware blobs, and they might not have access to the source code for that. They could reverse-engineer something equivalent, but that's going to be a lot of work and more easily noticed.
They also might not want the manufacturer to know -- or at least not right away. It's probably pretty hard to get an identical Intel chip with modified software installed on a shipped board without *someone* at Intel knowing about it. But you could intercept a standard shipment (while it's in customs perhaps), add your spy chip, and send it back out without involving a single employee at the manufacturer or the recipient which significantly reduces the risk of getting caught. If the new chip is included on the circuit board or solder mask that's a bit less likely, but it's still possible that they didn't want to involve the manufacturer in initial experiments/testing of the concept, or they don't want to give any information beyond "Install the chips that we are going to provide" without anyone having the knowledge of exactly what those chips do. You definitely don't want to start your super secret spy project by explaining the whole thing to some corporate CxOs to see if they can do it. People are going to know about that spy project before the product even ships.
And on top of that, I'd imagine that the spy chip method could be more versatile. The external interfaces to various processors and bridge chips are likely to be more stable and standardized than the chip internals, so that might let you build one spy chip that works on a larger variety of systems.
(Score: 1, Insightful) by Anonymous Coward on Monday October 08 2018, @09:53AM
Especially as NSA has "intercepted" Cisco equipment to install additional "features" for ages...
(Score: 2, Insightful) by Anonymous Coward on Sunday October 07 2018, @01:02PM (1 child)
Read the Apple statement carefully. It is not incompatible with Bloomberg. Maybe there was no "reaching out" if Apple and the government were already talking for other reasons. In the middle of ongoing discussions an Apple person could have said, by the way, we found some odd server behavior. Or maybe it was an IT security company hired by Apple that talked to the Feds.
(Score: 1, Insightful) by Anonymous Coward on Sunday October 07 2018, @01:10PM
Maybe there is a regular report that Apple sends the Feds about foreign hacking. If they included this particular hack in a report, that's not "reaching out".
(Score: 2, Insightful) by Anonymous Coward on Sunday October 07 2018, @05:55PM (4 children)
I really dont see the Chinese doing this.
Sure, they are as rouge as the rest of us, but if they got caught doing something like this it could collapse their entire economy. Risk is too high.
(Score: 2) by crafoo on Sunday October 07 2018, @08:53PM
a) They did it. B) No one really cares because everyone gets caught doing things like this all the time.
(Score: 3, Funny) by ilsa on Sunday October 07 2018, @11:59PM
I dunno. I think you may be looking at it through rogue-tinged glasses.
(Score: 3, Insightful) by Fluffeh on Monday October 08 2018, @01:57AM (1 child)
That's like saying if the NSA got caught doing this, it would collapse the entire US economy.
It wouldn't crash either one. It's an interesting time we live in, where today's storm-in-a-teacup is tomorrow's ancient history. It seems that whatever happens is only critically vital right up until the moment it happens, it is proven or can no longer be averted. Then that topic is so far down the talking points, it never sees the light of day again.
(Score: 1, Informative) by Anonymous Coward on Monday October 08 2018, @09:55AM
How quickly we forget:
https://www.infoworld.com/article/2608141/internet-privacy/snowden--the-nsa-planted-backdoors-in-cisco-products.html [infoworld.com]
(Score: 1, Funny) by Anonymous Coward on Sunday October 07 2018, @06:28PM (1 child)
As an Anonymous Coward ex-amazon employee I can confirm that this is true.
(Score: 3, Insightful) by hemocyanin on Sunday October 07 2018, @11:52PM
The difference between the AC comment above and the AC comment filtered through a reporter, is that the one above is properly modded while the one made to the reporter draws a collective gasp and potential military response.
(Score: 3, Interesting) by Anonymous Coward on Sunday October 07 2018, @08:59PM
So of course Apple and Amazon are going to deny it.
(Score: 3, Interesting) by arslan on Monday October 08 2018, @12:41AM (3 children)
Another way to read this is, the hardware isn't modified if it was per original spec and isn't malicious if Amazon knew about them from the get go about what it does. They weren't engaged in an investigation with the government, just a collaborative program with full disclosure between then and the government in question.
I didn't see anything in the Amazon article where they categorically dismissed the fact that the chips allows government access to data without their client's consent just refute of the technical jargon that allows for the outcome.
(Score: 2) by takyon on Monday October 08 2018, @01:38AM (2 children)
Why are so many users bothering to parse and creatively interpret every word of these statements? Is there going to be an SEC punishment if they were caught lying in these statements? I doubt it.
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 0) by Anonymous Coward on Monday October 08 2018, @03:17AM
I'm more interested in knowing why news sites like Ars are writing about these weak sauce obfuscation as if they were strong denials.
(Score: 3, Insightful) by arslan on Monday October 08 2018, @04:58AM
Maybe, maybe not. But market perception & sentiment does affect their stock value. If they do get expose later, say via some whistle blowing, then they can technically say they didn't lie. I suppose it does soften or at least alter the impact somewhat vs being caught red-handed in an outright lie.
These interpretation are also kinda "spread the word" to the potential tactics used by these corporations to those unfamiliar or new to this space. I suppose most folks here are seasoned enough so that it is mostly a captain obvious thing, but never hurts to repeat such sentiments.