Stories
Slash Boxes
Comments

SoylentNews is people

posted by mrpg on Friday February 08 2019, @11:40AM   Printer-friendly
from the emergency-failover-failed dept.

The Register reports that Wells Fargo Bank experienced an issue in a data center Thursday morning, and things have not been the same since:

Wells Fargo customers have been unable to access their online bank accounts for more than seven hours today – after smoke knackered one of its data centers.

Starting around 6am Pacific Time (2pm UTC), the American bank's online portal and its mobile application have been totally unusable. There are also reports of cards being rejected by cash machines and stores.

"We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance," the bank said in a statement. "We're working to restore services as soon as possible. We apologize for the inconvenience."

[...] According to fire chief Tim Boehlke, of Lake Johanna Fire Department, the downtime kicked off at a Wells Fargo data center in Shoreview, Minnesota. We're told a fire suppression system was activated at around 0500 local time (1100 UTC), forcing a power shutdown, and switching off the facilities' servers. When the fire department got there, though, they found no evidence of a blaze.

Fire suppression systems tend to be rather punishing to data center machines, particularly their hard drives, as sysadmins in this Reddit thread on the outage point out. It could take a while to restore power and undo the effects of the suppression system.

[...] Whether the backups failed, or the shutdown caused a cascading fault that affected other data centers and took out the rest of the bank's online presence, isn't known at this time. It is baffling that a single incident has set off such a chain reaction that it knocked out the entire internet-facing infrastructure of the bank.

Also at SF Gate.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Touché) by c0lo on Friday February 08 2019, @12:47PM (2 children)

    by c0lo (156) Subscriber Badge on Friday February 08 2019, @12:47PM (#798272) Journal

    Just don't build it in Australia.

    Actually, from the ris... eeerrr... public image management perspective, Australia is the best choice; with the internet here, it will never cross customers' mind your site went down, they'll blame it on NBN [theguardian.com]

    --
    https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
    Starting Score:    1  point
    Moderation   +1  
       Touché=1, Total=1
    Extra 'Touché' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   3  
  • (Score: 3, Funny) by DannyB on Friday February 08 2019, @03:09PM (1 child)

    by DannyB (5839) Subscriber Badge on Friday February 08 2019, @03:09PM (#798318) Journal

    Isn't natural cooling now a factor in selection of data center location?

    Thus the Australian outback would be a great place to put a data center? Or Death Valley CA.

    --
    People today are educated enough to repeat what they are taught but not to question what they are taught.
    • (Score: 2) by bob_super on Saturday February 09 2019, @01:23AM

      by bob_super (1357) on Saturday February 09 2019, @01:23AM (#798633)

      Well, the roof's solar panels would love the Nevada sun, so I suggest a floating/sunk data center on Lake Mead