SoylentNews is people
SoylentNews
Toshiba & WD NAND Production Hit By Power Outage: 6 Exabytes Lost
Toshiba Memory and Western Digital on Friday disclosed that an unexpected power outage in the Yokkaichi province in Japan on June 15 affected the manufacturing facilities that are jointly operated. Right now, production facilities are partially halted and they are expected to resume operations only by mid-July.
Western Digital says that the 13-minute power outage impacted wafers that were processed, the facilities, and production equipment. The company indicates that the incident will reduce its NAND flash wafer supply in Q3 by approximately 6 EB (exabytes), which is believed to be about a half of the company's quarterly supply of NAND. Toshiba does not disclose the impact the outage will have on its NAND wafer supply in the coming months, but confirms that the fabs are partially suspended at the moment. Keeping in mind that Toshiba generally uses more capacity of the fabs than WD, the impact on its supply could be significantly higher than 6 EB with some estimating that it could be as high as ~9 EB.
Both companies are assessing the damage at the moment, so the financial harm of the incident is unclear. Not even counting potential damage to production tools and other equipment used at the fabs, 6 EB of NAND cost a lot of money. Furthermore, analysts from TrendForce believe that a consequence of the outage will be some loss of confidence from clients of both companies, which will have a financial impact as well.
1 exabyte = 1 million terabytes.
Related: TSMC Fab 14 B hit by Massive Wafer Defection due to Chemical Contamination, 16/12nm Production Line
TSMC Contamination Issue Expected to Result in $550 Million in Lost Revenue
(Score: 3, Funny) by krishnoid on Friday June 28 2019, @09:11PM (26 children)
Once again emphasizing the importance of having offsite backups.
(Score: 3, Interesting) by Rupert Pupnick on Friday June 28 2019, @09:21PM (9 children)
Also illustrating the surprising (to me) vulnerability of bleeding edge commercial manufacturing technology to a garden variety event. Is there some kind of UPS system big enough to prevent this, and what would it cost? Also will be interesting to see what this does to pricing.
(Score: 0) by Anonymous Coward on Friday June 28 2019, @09:31PM (1 child)
Yes, there are arbitrarily large back up systems, for example, you can buy a gas turbine generator. Probably lots of suppliers, but I happened to remember this name, they have single units up to a megawatt, like this one,
https://www.capstoneturbine.com/products/c1000s [capstoneturbine.com]
It can be configured as standby/backup.
(Score: 3, Informative) by HiThere on Friday June 28 2019, @10:45PM
Those take too long to come on-line. If you want to go for something like that, you also need a huge ballast to stabilize voltage and current while it's coming on-line.
This is probably more about eliminating voltage fluctuations than maintaining flow of current, since they're talking about equipment damage.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 1) by RandomFactor on Friday June 28 2019, @09:49PM
American Superconductor comes to mind. Their Grid Power assurance section's description reads particularly on-point for this.
В «Правде» нет известий, в «Известиях» нет правды
(Score: 1) by liberza on Friday June 28 2019, @10:20PM (5 children)
The equipment & facilities draw massive amounts of power (robots, furnaces, building-wide air filtering / fans, process exhaust, pumps of all varieties, ion implanters, plasma etchers, big-ass lasers, chem & gas waste treatment equipment... the list goes on). Not saying it can't be done but probably not as easy as it might seem at first glance.
(Score: 3, Insightful) by pipedwho on Saturday June 29 2019, @12:12AM
You don’t need a redundant power system for everything, just the things with catastrophic failure modes. This is similar to a city not having a reliable data backup strategy.
(Score: 2) by toddestan on Saturday June 29 2019, @05:42AM (3 children)
A huge battery backup system like what Elon Musk built in South Australia seems like just what they need to prevent this kind of thing from happening. It'll probably be eye-watering expensive, but also dirt cheap compared to losing half a quarter's worth of output.
(Score: 1, Insightful) by Anonymous Coward on Saturday June 29 2019, @09:26AM (2 children)
Generators aren't that expensive. Datacentres have massive power requirements too, but they essentially ALL have battery then generator backup.
This isn't anything special. It's bizarre that someone, anyone with such sensitive processes would ever, ever not have generator backup. And the thing is, it's not even like a datacentre...
In a datacentre, you still need all the switches, servers, etc on. The biggest power draw? Must keep running! Including air conditioning, or servers will literally DIE!
I bet in a FAB like this, there are many steps that can be stopped. It isn't like you need power for everything, only the processes that will cause a plant to, you know, absurdly shut down for months because of a few hours without power.
Offices can shut down, people can go home, AC can be stopped to all but critical areas, on and on, just absurd.
I don't know what really happened, but I assume the next financial update will include 'how we fixed this boneheaded decision', otherwise Toshiba = downward stock price...
(Score: 0) by Anonymous Coward on Saturday June 29 2019, @04:22PM (1 child)
Toshiba most recently bounced back from near bankruptcy due to investments in nuclear power. Quite possible that an MBA exec told the plant managers that backup power systems weren't justifiable due to cost.
(Score: 2) by Rupert Pupnick on Saturday June 29 2019, @05:47PM
Where the bean counters are in charge (like Boeing), beware!
(Score: 3, Interesting) by archfeld on Friday June 28 2019, @09:45PM (13 children)
Ummm did you even read the article. Data was not lost but rather the chips in process of being manufactured were exposed and thus rendered unusable. That being said 6 - 9 exabytes of wafers is a huge almost unimaginable amount of potential data capacity, and makes the lack of a backup power supply/UPS system even more questionable.
1 exabyte = 1 million terabytes
"Wafer thin mint ?"
https://www.youtube.com/watch?v=aczPDGC3f8U [youtube.com]
For the NSA : Explosives, guns, assassination, conspiracy, primers, detonators, initiators, main charge, nuclear charge
(Score: 2) by krishnoid on Friday June 28 2019, @10:21PM
[Slow clap]
(Score: 2) by edIII on Friday June 28 2019, @10:51PM (6 children)
It makes no sense. TFS says 1/8th of yearly production for WD is gone. Even more so for Toshiba. Yet, how does 13 minutes affect over 1 month of production? That implies a very long series of steps in automation that takes weeks before final product, or? It's just plain strange.
Another poster pointed out their power requirements could be a lot higher than your average data center. Even though, to mitigate 1/8th of your yearly supply being gone in 13 minutes it seems economically prudent to have N+1 or N+2 power throughout the entire factory. Something in particular was very damaging during the 13 minutes of outage, and should've had N+1 on it at the very least.
It seems that whoever designed the facility, or whatever committee, failed to understand just how bad it could be.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 2) by krishnoid on Friday June 28 2019, @11:20PM
It said it affected already-processed wafers and the facilities and equipment, so I could see 1/8th of the yearly production gone. I wonder if something actually blew up and threw pieces around, er, an eighth of the facility.
(Score: 5, Informative) by Snotnose on Friday June 28 2019, @11:23PM (3 children)
I can guess (warning: I'm pulling this out of my (educated) ass). Wafers are very fragile. At one company we occasionally got rewards of an 8" wafer of circuits that failed QA. Once in a while someone would take it out of it's frame, and it would snap in 2. Think of a thin potato chip in dip, where the dip is gravity.
First you have the immediate damage of things just stopping. I can see inertia alone breaking wafers. Lost stuff + cleanup costs.
Then you have environmental considerations. No more air filtering == more useless wafers. Lost stuff + cleanup costs.
Then you have wafers in various parts of production. Sit too long in an acid batch and lost wafers. Plus, all together now, + cleanup costs.
Sounds like the power outage busted some machines. Takes time to replace them babies.
About those cleanup costs. Do you know how clean a cleanroom has to be? Once such a room is contaminated, how long do you think it will take to clean it? Especially when you have stuff like wafer dust in places dust should never be. Do you think these machines are designed for easy cleaning? I doubt it's as simple as "1) drop the magazine 2) clear the chamber 3) press this button 4) turn this doohicky 5) remove slide assembly 6) remove barrel and follower spring". I suspect it's more like "1) fly someone in from a long way away 2) make a separate cleanroom 3) tear this complicated machine down to it's skeleton 4) clean everything 5) Put it all back together 6) put it back where it belongs".
Actually, I suspect it's more of "1) put the machine on Craigslist 2) Order a new one".
Yeah, I can see this easily halting production for a month.
PS. Why yes, I do own a Baretta M2. Why do you ask?
When the dust settled America realized it was saved by a porn star.
(Score: 3, Insightful) by ChrisMaple on Saturday June 29 2019, @04:05AM (2 children)
I've only handled a wafer once. It was about 1 mm thick and not particularly fragile. Although there are special processes that require grinding off the back side of the wafer to get thickness down to about 0.1 mm, my understanding is that such processes are uncommon and that 1 mm is about standard.
Once a wafer is fully processed it's fairly resistant to the environment, although you wouldn't want to leave it out in the rain.
If I recall correctly, AMI used to shut down their fab for a week once each year to clean and calibrate. That sounds like a reasonable industry standard to me, and thus purging the equipment shouldn't be too huge a deal.
My best guess for why the loss is so large is that they're writing off everything in process including a large stock of raw wafers, just so they can say "everything you buy from us is guaranteed not to have been affected by the blackout." Perhaps everything is insured and they're trying to maximize the insurance payout. Maybe this is publicity so they can more effectively sue the power company.
(Score: 4, Informative) by toddestan on Saturday June 29 2019, @06:17AM (1 child)
I used to work on test equipment that was used with silicon wafers at facilities like these. The wafers they tested were extremely valuable. They wouldn't tell us how expensive they were, of course, but if something went wrong they wanted the our equipment to do everything it could to not damage the wafer, even if it meant our equipment purposely damaging/destroying itself if need be. We sold these machines for over a million dollars by the way.
The wafers we handled were much thinner than 1mm thick, 100-200um is more common. They are silicon, so they aren't particularly fragile - they can easily support their own weight so you don't have to worry about gravity breaking them from just picking one up and handling it. Much like a piece of glass it's fairly durable, but also very brittle - you don't want to drop one. Contamination though is a big problem - you don't want to get any dust or foreign material on them, and fingerprints are a total disaster. They are also incredibly flat and smooth. The robots (robotic arms) they use to move them around grip onto the bottom of them using vacuum to hold the wafers onto the arm. Since it's so flat, they can get a really good grip on the wafer this way, and thus the robotic arms whip the wafers around at absolutely frightening speeds. If there's any contamination at all so that robotic arm doesn't get a good seal, they could potentially have a missile on their hands.
Incidentally, since they are so flat and smooth and relatively lightweight, they slide really easy on a table. Almost like air hockey, except using a regular old table. You wouldn't play air hockey with anything important, but we had a few scrap wafers where something went wrong and we got use them for development purposes.
If I had to guess, most of the losses revolve around some of the photo and chemical processes they use to process the wafers that are either very time sensitive and/or temperature sensitive.
(Score: 1) by LAV8.ORg on Sunday June 30 2019, @01:45AM
Wafers are readily available on ebay for a few bucks. The demand for them is limited, as you might imagine, since they're of little use in that unfinished state. I reckon the few big reasons for buying would be: curiosity, art, and industrial espionage, but even the sum of demand for those reasons is low relative to the amount that somehow fall out of the production line (presumably most are lost during shipment). I wound up with a boat (purpose built tupperware) with a handful and couldn't sell for the lot for $50. From what little I could glean they were destined for some medical equipment, so yeah, a value ca. megabucks to the mfgr. seems possible. And yet I doubt they'd have any interest in buying them back for more than a few bucks due to the cost/complexity of not being lock step to production, potential contamination, etc. FWIW the sample I have is at the ~1mm thick side.
(Score: 0) by Anonymous Coward on Saturday June 29 2019, @04:59AM
> ... how does 13 minutes affect over 1 month of production? That implies a very long series of steps in automation that takes weeks before final product, or?
A close friend worked in a fab for many years, left a few years ago. At that time the pipeline for one wafer was up to about 6 weeks -- many, many process steps, creating (and partially tearing down) many layers on top of the base silicon. Some steps are resisting followed by etching, others depositing or ion implanting, some in vacuum, others in ionized chambers, etc.
Since complexity follows something like Moore's Law, I could imagine a modern fab with a couple of months of process steps for their most complex wafers.
(Score: 2) by RS3 on Saturday June 29 2019, @02:19AM (4 children)
You must be new to tech blogs / social media.
(Score: 2) by archfeld on Saturday June 29 2019, @03:09AM (3 children)
LOL not really I have a 3 digit ID at the green site which shall not be named, but I do get your drift...
Eat well, Shit strong, and Live long...
For the NSA : Explosives, guns, assassination, conspiracy, primers, detonators, initiators, main charge, nuclear charge
(Score: 2) by RS3 on Saturday June 29 2019, @03:39AM (2 children)
Yeah, sorry, it's a long-running thing, as you know, that nobody reads TFA. Why bother? Everyone else will glean out the facts and comment on them, right? :^|
Wait what? 3-digit ID? That's pretty impressive. I was reading it early on but was (and am still) somewhat hesitant to "register" with websites, but I do have a just barely into the 6-digits ID at green. Culture there seems very different in the past few months. I don't keep up, but I wonder if they've culled the trolls.
Yes, live long and prosper.
(Score: 2) by archfeld on Saturday June 29 2019, @09:13PM (1 child)
I have a friend who pointed me to /. very early on because I was having issues installing Slackware on a PC. Back then the /. 'community' was home to some of the developers and a whole host of very sharp people. These days those that 'know' don't go anywhere near places like /.
Cheers...
For the NSA : Explosives, guns, assassination, conspiracy, primers, detonators, initiators, main charge, nuclear charge
(Score: 2) by RS3 on Saturday June 29 2019, @10:28PM
Oh yeah, I've been around too. SW, HW, systems, admin, etc. /. is really much better now. I think they cleaned house. But I like this site more for many reasons.
I got into Linux around 1994 and still have some of the early CDs. I had an SLS distro, which iirc morphed into Slackware, which is still my personal favorite. Although, I'm having many problems with 14.2, so 14.1 still runs (ain't broke...) I have not deployed it on a live server, and I think the only reason is that if someone else takes over the operation (small hosting company), I don't want them cursing me or bugging me with stupid questions (where is cpanel!?, or plesk, or chef, or puppet, or rpm, or yum, or ...) so I went with CentOS but that will come to an end soon as the hardware is old, 32-bit only, runs very well (ain't broke...) That said, if a cheap enough much better newer server came along, we could justify it probably...
Do you remember what problem you were having with Slackware?
There were many years / Slackware versions that I shied away- kept old installs running. Originally, iirc, there was very good package management that kept track of dependencies. I realize that got too messy.
Very most likely I'll switch to Alpine very soon. Not super thrilled with apk, and Alpine isn't the best for GUI, but otherwise Alpine is just awesome as a server.
(Score: 0) by Anonymous Coward on Friday June 28 2019, @10:19PM
Yeah, all that porn.
(Score: 0) by Anonymous Coward on Sunday June 30 2019, @07:30AM
In this case, having POWER backup failover systems IN PLACE.
I can see the electric supply CEO carpet crawling in apology...
(Score: 1, Funny) by Anonymous Coward on Friday June 28 2019, @11:13PM (1 child)
That's 6 to 9 EB of analytics about you that won't get stored.
(Score: 0) by Anonymous Coward on Saturday June 29 2019, @04:50PM
fingerprints on the NAND are BAD. fingerprints IN the NAND are GOooooOD!
(Score: 1, Touché) by Anonymous Coward on Friday June 28 2019, @11:19PM
Beancounter: Hmmm prices are getting too low, we aren't getting enough margins.
Manager: Let's call a flood at the factory.
Beancounter: That trick was already used.
Manager: Let's call a blackout at the factory.
Beancounter: That trick was used by Samsung.
Manager: Let's join the club.
Blackout at Samsung NAND factory destroys chunk of global supply [theregister.co.uk]
(Score: 2) by Bot on Saturday June 29 2019, @06:48AM
jap managers should tour a bit more around here in Italy, where surprises abound (oh! no bus today! oh! no garbage collection today! oh! lil' nino bumped into a hole in the street, the hole won! oh! somebody misplaced the defibrillator!). That would reduce the slight evaluation mistakes, oppa Fukushima style.
Account abandoned.
(Score: 1, Interesting) by Anonymous Coward on Saturday June 29 2019, @04:57PM
i suppose lots of this is sheets of silicon crystals? are there also like .. big ones?
maybe they're gonna make a firesell? are biggish silicon crystals useable for something else? is guess they reflect the light pretty nice? so maybe futuristic wall paneling?
maybe purify water? dunno, so asking.
i'd sure pay transport cost if they have a big one they've grown but not cut into sheets yet:
https://en.wikipedia.org/wiki/File:Leo_Tie_Rodsedit.jpg [wikipedia.org]