The Register reports that Wells Fargo Bank experienced an issue in a data center Thursday morning, and things have not been the same since:
Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers.
Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being rejected by cash machines and stores.
"We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance," the bank said in a statement. "We're working to restore services as soon as possible. We apologize for the inconvenience."
[...] According to fire chief Tim Boehlke, of Lake Johanna Fire Department, the downtime kicked off at a Wells Fargo data center in Shoreview, Minnesota. We're told a fire suppression system was activated at around 0500 local time (1100 UTC), forcing a power shutdown, and switching off the facilities' servers. When the fire department got there, though, they found no evidence of a blaze.
Fire suppression systems tend to be rather punishing to data center machines, particularly their hard drives, as sysadmins in this Reddit thread on the outage point out. It could take a while to restore power and undo the effects of the suppression system.
[...] Whether the backups failed, or the shutdown caused a cascading fault that affected other data centers and took out the rest of the bank's online presence, isn't known at this time. It is baffling that a single incident has set off such a chain reaction that it knocked out the entire internet-facing infrastructure of the bank.
Also at SF Gate.
(Score: 1, Insightful) by Anonymous Coward on Friday February 08 2019, @12:22PM (4 children)
Because, you know, data centres these days, they never go down.
Why waste money on a hot, or even a cold, remote failover site?
It's not like the place is going to burn down or flood under water.
Just don't build it in Australia.
(Score: 0) by Anonymous Coward on Friday February 08 2019, @12:38PM
My guess is that they used the same Sys Adm as HRC and IRS.
(Score: 3, Touché) by c0lo on Friday February 08 2019, @12:47PM (2 children)
Actually, from the ris... eeerrr... public image management perspective, Australia is the best choice; with the internet here, it will never cross customers' mind your site went down, they'll blame it on NBN [theguardian.com]
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Funny) by DannyB on Friday February 08 2019, @03:09PM (1 child)
Isn't natural cooling now a factor in selection of data center location?
Thus the Australian outback would be a great place to put a data center? Or Death Valley CA.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug into other computer. Right-click paste.
(Score: 2) by bob_super on Saturday February 09 2019, @01:23AM
Well, the roof's solar panels would love the Nevada sun, so I suggest a floating/sunk data center on Lake Mead
(Score: 4, Insightful) by c0lo on Friday February 08 2019, @12:40PM (7 children)
Baffling? Not at all.
It's called a "lean, mean, efficient capitalistic machine", where the most common free association between "redundancy" and "engineering" is no longer reliability engineering [wikipedia.org] but the surge of the share prices whenever a company sacks some of its engineers.
For those that have the perverse pleasure of displaying engineering traits (not the El Reg journos, seemingly), see cascading failure [wikipedia.org]
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by JoeMerchant on Friday February 08 2019, @02:48PM (5 children)
All too predictable and reliable result of Wall Street valuation methods that look more closely at present operating costs than consistency and reliability.
I wonder, if CEO pay extended into the future - say 10 years after the CEO steps down, based on performance metrics, how differently would they behave? Not extra pay, but instead of pay today for last quarter's performance, get 10% of their compensation in the year worked, and an additional 10% every year after based on metrics.
🌻🌻 [google.com]
(Score: 2) by DannyB on Friday February 08 2019, @03:12PM
Many years ago there was a Dilbert about surging share prices as a result of firing engineers. The PHB said they could fire another batch of you every time they needed a boost in the share price. "It's like printing money!" he exclaimed.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug into other computer. Right-click paste.
(Score: 0) by Anonymous Coward on Friday February 08 2019, @06:19PM (3 children)
The CEO will never agree to that. Money now cost more than money later.
(Score: 2) by JoeMerchant on Friday February 08 2019, @09:04PM (2 children)
They will if it's a condition of hiring.
If I were playing the CEO game, I'd love to have 10 years of residual income after I resigned my duties.
🌻🌻 [google.com]
(Score: 2) by bob_super on Saturday February 09 2019, @01:25AM (1 child)
If they don't like it, keep asking and you'll soon find a very competent person who will.
And you probably will get better results from that kind of person.
(Score: 2) by JoeMerchant on Saturday February 09 2019, @03:11AM
I'd like to think so, but so often the CEO selection game is more about contacts than it is about competence. Both are important to success.
🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Friday February 08 2019, @04:13PM
Or, the smoke was a distraction from the agent planting malware in the system.
(Score: 3, Insightful) by Gaaark on Friday February 08 2019, @01:05PM
Did they not watch Mr. Robot?
No remote backup site?
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 3, Insightful) by nobu_the_bard on Friday February 08 2019, @02:36PM
On one hand, I'd rather use a different banking service than Wells Fargo.
On the other hand, I have no guarantee other large banks I can reasonably choose from won't have the same problems.
I'm hoping when some of the smaller local guys finally get some apps setup (to allow me to do deposits without driving 30 minutes to a branch) I will be able to finally change banks away from Wells Fargo. At least my business matters to the smaller guys more, so I get treated better as a client, even if their services aren't necessarily better.
(Score: 3, Insightful) by backsplatter on Friday February 08 2019, @02:52PM
Of course they have backup data centers. There is one in Tempe.
I would more guess that this is a complex interconnected network with some systems automatically switching data centers.
Some processes that, although they work fine for DR testing, maybe aren't quite as automated as certain departments assume.
Going with a major bank you have a whole set of different platforms, file transmission protocols, some hardcoded IP addresses, processes that were setup by employees long gone, etc.
Along with all these fancy front ends that everyone loves to put on everything.
(Score: 4, Touché) by All Your Lawn Are Belong To Us on Friday February 08 2019, @02:59PM (5 children)
After all, it's Backed Up and Redundant!
All I'm waiting for is to learn if they were outsourced because I'd love to put this in both my "Why You Don't Outsource" file as well as "Why You Don't Cloud" file.
This sig for rent.
(Score: 5, Funny) by DannyB on Friday February 08 2019, @03:14PM (2 children)
Outsource your cloud infrastructure into another cloud. Taken to its logical conclusion, there can be unlimited clouds and no hardware at all! Managers: think of the cost savings!
To transfer files: right-click on file, pick Copy. Unplug mouse, plug into other computer. Right-click paste.
(Score: 2) by realDonaldTrump on Friday February 08 2019, @05:30PM
Many people are saying the future of finance is Quantum Fog & Cloud A.I, Block Chain & 6G I.O.T. Smart!!
(Score: 2) by bob_super on Saturday February 09 2019, @01:28AM
> Outsource your cloud infrastructure into another cloud. Taken to its logical conclusion, there can be unlimited clouds and no hardware at all!
https://xkcd.com/908/ [xkcd.com]
(Score: 2) by darkfeline on Friday February 08 2019, @11:12PM (1 child)
You jest, but they wouldn't have had this particular problem if they were using the cloud.
The cloud is someone else's server, yes, but if you're using a big provider like AWS et all, it's someone else's global, redundant, professionally managed datacenter network. A single datacenter or two going down (N+2 redundancy) would just be a blip in the fleet-wise performance graphs.
Meanwhile, a company not specializing in cloud services probably doesn't maintain a global, redundant datacenter network.
Join the SDF Public Access UNIX System today!
(Score: 2) by All Your Lawn Are Belong To Us on Friday February 08 2019, @11:58PM
Maybe. And I'm willing to accept that if true, if it was an internally managed datacenter at fault so be it.
OTOH, hell yes they could have had this problem with a cloud installation. From what I read in TFA this may have been an incorrect configuration taking the whole system out. That can happen in the cloud just as easily as it can in a managed datacenter, if not moreso. Not to mention financial networks needing far better failover than just redundancies making everything just a little more complex. And that troubleshooting might actually require knowing *which* box is malfunctioning.
If there's anyplace that will go for reduction of costs by using cloud services you can bet it will be a bank. If it is allowed to.
This sig for rent.
(Score: 2) by DannyB on Friday February 08 2019, @04:54PM (1 child)
Consider the existing headline:
My suggestion is one additional word:
Wells Fargo Data Center Smells Smoke; Bank Website, App, Burn Down
To transfer files: right-click on file, pick Copy. Unplug mouse, plug into other computer. Right-click paste.
(Score: 2) by istartedi on Saturday February 09 2019, @04:33AM
To be followed by:
Where There's Smoke, There's Firing. Wells Fargo seeks to replace key technology staff.
Appended to the end of comments you post. Max: 120 chars.