SoylentNews is people
SoylentNews
Cringley speculates like hell:
Delta Airlines last night suffered a major power outage at its data center in Atlanta that led to a systemwide shutdown of its computer network, stranding airliners and canceling flights all over the world. You already know that. What you may not know, however, is the likely role in the crisis of IT outsourcing and offshoring.
Do any Soylentils have inside/better information?
This discussion has been archived.
No new comments can be posted.
What Happened When Delta Went Down Yesterday?
|
Log In/Create an Account
| Top
| 73 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
They have been at a great feast of languages, and stolen the scraps.
-- William Shakespeare, "Love's Labour's Lost"
(Score: 5, Funny) by Dunbal on Tuesday August 09 2016, @03:38PM
Windows 10 automatic upgrade.
(Score: 5, Touché) by Anonymous Coward on Tuesday August 09 2016, @03:42PM
Couldn't be, it came back up.
(Score: 2) by SrLnclt on Tuesday August 09 2016, @04:17PM
The Windows 10 free automatic upgrade time has now passed [microsoft.com].
Is the Windows 10 free upgrade offer still available?
No, the Windows 10 free upgrade offer ended on July 29, 2016.
Windows 10 will continue to be available for purchase, either on a device or as a full version of the software. A discounted "upgrade" version of Windows 10 will not be for sale.
However, automatic updates for those with Windows 10 may be another story...
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @04:30PM
What would be hilarious is if that was still the cause of the downtime. For instance, what if the computer had scheduled their upgrade time for August, after the free period passed. Then when the system tried to update, after downloading everything and rebooting into Windows 10, it refused to validate because it was no longer in the free period and required payment. :D
Yeah, I doubt that was the cause too. I'm sure we would have definitely heard about it by now if that was the case. XD
(Score: 2) by darnkitten on Wednesday August 10 2016, @04:48PM
Out here in the country, not just the Win10 upgrade, but the updates bork people's systems--about a third of the Win10 computers I'm aware of have had some sort of major failure during the planned updates, resulting in anything from loss of access to the Edge browser, being unable to use (or even find) the search function, up to complete failure requiring entire system reinstall.
As far as I can tell, (aside from normal bugs in the updates), the update processes were interrupted by either the power or the network failures that commonly occur around here.
Also, on one lady's notebook, Windows tried to use her cell data plan to download the update, even though it theoretically wasn't allowed to do so, and crashed when it ran up against her data cap.
A good portion of the local computer guy's business is fixing bad Win10 upgrades/updates or rolling back to more robust versions of Windows.
-
Microsoft apparently didn't realize that some people operate their computers under rural conditions...
(Score: 2) by davester666 on Wednesday August 10 2016, @07:01AM
Delta still gets the free update to Win10 because they need the accessibility support.
(Score: 3, Interesting) by Snotnose on Tuesday August 09 2016, @03:43PM
Outsourcing? [cringely.com]
Or ancient software. The news this morning said that Delta's system had been written by an airline that Delta bought in 1982. I'm sure they've added all sorts of updates over the years, but if the architecture is 30 years old then yeah, I can see how that might be a problem. Especially considering how much air travel has changed in the last 10-15 years.
It was a once in a lifetime experience. Which means I'll never do it again.
(Score: -1, Offtopic) by Anonymous Coward on Tuesday August 09 2016, @04:06PM
Don't worry brother! Obama gonna CHANGE things back to the way they been in 1989, that number! Another summer! FIGHT THE POWER!
(Score: 3, Insightful) by Anonymous Coward on Tuesday August 09 2016, @04:24PM
It doesn't matter how old the software is. The fact is that Delta had no disaster recovery plan at all and no off-site passive standby ready to take over. That's just negligence or incompetence for a major corporation with IT infrastructure that is critical to their business. Honestly, I hope this hurts them hard enough that they go out of business. This complete lack of planning or foresight for such a critical piece of their business makes you wonder what other corners they're cutting elsewhere: like in plane maintenance.
(Score: -1, Flamebait) by Anonymous Coward on Tuesday August 09 2016, @04:31PM
Critical IT infrastructure? Delta isn't an IT company. America isn't even an IT country. IT is Indian Technology.
(Score: 2) by maxwell demon on Tuesday August 09 2016, @08:47PM
The failure of that infrastructure caused them not being able to do their core business correctly. Yes, I'd call that critical infrastructure.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 3, Interesting) by bob_super on Tuesday August 09 2016, @04:50PM
When United merged with Continental in 2010, they looked at both IT systems, and saved a few millions by adopting the most ancient and underpowered one (from Continental, I think).
I had a friend working for United in ORD, they had two months of pure hell, because the system just couldn't handle tens of thousands of novices making mistakes just as its load doubled.
Nothing like getting yelled at all day by justifiably tired and angry customers, while the people providing the inappropriate tools celebrate their bonuses... I was actually surprised at the lack of spontaneous combustion of C-suite-owned cars and buildings.
(Score: 3, Interesting) by frojack on Tuesday August 09 2016, @06:17PM
Since they were merging, and both systems were already in-hand and sunk costs, how could there one be cheaper than the other? Perhaps the newer one was bug ridden and full of maintenance headaches and caused all sorts of down time.
I've seen large scale accounting systems get newly developed replacements at the cost of years of work and multiple millions of dollars, and even after 5 years of operations couldn't manage the task the old system did with ease. Newer is not automatically better.
No, you are mistaken. I've always had this sig.
(Score: 2) by bob_super on Tuesday August 09 2016, @07:54PM
True, but his manager-of-users viewpoint was that the one they had was stable enough and had more productivity features, and the company chatter was that the one cheapest to double the load was selected despite being ancestral.
It might have been impossible to double United's, or he may have been biased by his habit, but the end result was both sides of the company (and the customers, but who cares) having a horrible merger experience.
(Score: 3, Interesting) by JoeMerchant on Tuesday August 09 2016, @06:24PM
I was actually surprised at the lack of spontaneous combustion of C-suite-owned cars and buildings.
Don't be, when you're "in the system" close enough to access C-suite-owned cars and buildings, you're already in "lackey mode" where you're prime motivation is to ingratiate yourself to those guys so they cut you in on a tiny slice of their pie.
Or, you're at the other end of the pay-scale where you need this damn job in order to pay the past-due rent, so bombing the CEO's car might not be the best way to stay employed, or get re-employed at any of your crappy options.
🌻🌻🌻🌻 [google.com]
(Score: 2) by deadstick on Tuesday August 09 2016, @08:13PM
A fundamental principle of oppressive regimes, economic as well as political. If you want to trust people, let them dip their beaks to a depth appropriate to their level.
(Score: 3, Funny) by krishnoid on Tuesday August 09 2016, @07:11PM
When United merged with Continental in 2010, they looked at both IT systems, and saved a few millions by adopting the most ancient and underpowered one (from Continental, I think).
... pure hell, because the system just couldn't handle tens of thousands of novices making mistakes just as its load doubled.
Too bad your friend didn't just print up a sign with that information on it, and add "If you want to complain, please call our M&A department at ..." . I bet a bunch of the customers would have been at least a little understanding.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @05:12PM
(Score: 3, Insightful) by sjames on Tuesday August 09 2016, @05:30PM
Beyond that, Delta, like many businesses out there has no resiliency in their system at all. The whole damned operation worldwide runs through a single point of failure. Further, if that single point fails, even with ticket, passenger, and plane all in the same place at the same time they somehow cannot put the passenger on the plane. Why should any of that require more than a local server? Sure, I can see how there would be problems with scheduling future flights, but anyone with a ticket to fly should be able to fly.
(Score: 3, Insightful) by frojack on Tuesday August 09 2016, @06:12PM
Local server?
Passengers have the tickets in hand. One or two gate agents is all you need to check tear ticket stubs and board the plane.
A few gorillas to load the baggage.
A pilot to file a flight plan, and a call for a pushback tug.
Done.
Sure its not sustainable beyond a couple days. But it shouldn't turn to shit the instant power fails 2000 miles away.
No, you are mistaken. I've always had this sig.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @07:32PM
You never saw Airport 77, did you? Old lady made her own boarding passes with a felt-tip pen. Without central server to verify lady was authorized to fly, she got lots of free flights.
TSA rules require central passenger verification anyway.
(Score: 2) by edIII on Tuesday August 09 2016, @08:31PM
No security. You could get 100 terrorists on the planes simply by creating fake tickets and then taking out the power in a single building someplace. Which brings up another point, how does the TSA check the paper tickets for authenticity? More than just the airline company is accessing airline systems, and the emphasis these days is not so much logistics but security.
All of that could be accomplished with a local caching/authentication server. If we really wanted to create a system capable of these things we could. The problem is management and budgeting, not availability of solutions and technology.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 2) by sjames on Tuesday August 09 2016, @08:59PM
That's why I said local server. When the ticket is created, the server at the departing airport gets a record of the ticket. The ticket itself gets a barcode containing the record and a signature to make it VERY hard to forge. The signature must verify and it must match the already downloaded record.
In addition, the local server can then inform the central control that the passenger was actually boarded once things return to normal.
As for management, call it a private cloud app and they'll be pissing themselves with excitement to get it done.
(Score: 2) by maxwell demon on Tuesday August 09 2016, @08:51PM
Sorry, but the zoo was not willing to give the gorillas to the airport.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by frojack on Tuesday August 09 2016, @06:02PM
Agreed. Old Software is not the problem.
Not having an offisite backup MAY be a problem, but it is a secondary problem, not a causal event.
The problem seems to be they were running on shore power only, and had zero, or inadequate on site backup power.
Its also possible they didn't have control of their uplink to the network, no redundant links and left that in the hands of some telco with equally inadequate backup.
I doubt Cringley.
No, you are mistaken. I've always had this sig.
(Score: 1, Informative) by Anonymous Coward on Tuesday August 09 2016, @06:04PM
At least planes get inspected every so often. But from what I read in the USA Today, only part of the system failed, the backup came up, and promptly locked up. Because of the half broken nature, the offsites wouldn't fail over. This just screams to me of improper testing of failure modes.
(Score: 3, Informative) by DannyB on Tuesday August 09 2016, @06:19PM
What you call negligence and incompetence, executives call bigger bonuses for cost saving. Job well done!
For some odd reason all scientific instruments searching for intelligent life are pointed away from Earth.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @06:38PM
> no off-site passive standby ready to take over.
Its hard to tell, but reading between the lines of the reporting in the popular press, they did have a standby but something went wrong with it too.
(Score: 2) by VLM on Tuesday August 09 2016, @08:06PM
It doesn't matter how old the software is. The fact is that Delta had no disaster recovery plan at all
I started my career at a major financial services company a long time ago and their disaster recovery plan was based on having the same code at many sites, which works perfectly when the problem is a hurricane or earthquake (neither of which were issues at any of the sites, intentionally, but I digress) and fails miserably when the problem is the code itself.
Of course we had two devs and a test system to go with our dual prods and disaster recovery schemes. Back when having a test or dev meant literally buying five mainframes instead of just two.
Anyway if there's an old bug that barfs on 8/8/16 for whatever reason, all the hardware DR plans in the world won't help.
(Score: 2, Insightful) by Anonymous Coward on Tuesday August 09 2016, @04:55PM
> Or ancient software. The news this morning said that Delta's system had been written by an airline that Delta bought in 1982. I'm sure they've added all sorts of updates over the years, but if the architecture is 30
> years old then yeah, I can see how that might be a problem. Especially considering how much air travel has changed in the last 10-15 years.
Yeah, 30 year old code must be bad. They should have scrapped it and rewritten it from scratch. That worked so well for netscape/systemd/etc
(Score: 1) by fustakrakich on Wednesday August 10 2016, @04:33AM
Netscape [seamonkey-project.org] still works for me.
As far as system redundancy is concerned, It has to be made cheaper than lawsuits and insurance/tax write offs for it to happen.
La politica e i criminali sono la stessa cosa..
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @03:58PM
Peon: "Boss, the batteries are 8 years old, and I don't know whether they can hold up in an outage."
Boss: "How much is it going to cost to replace?"
Peon: "Around $XXX,XXX."
Boss: "There just isn't any money in the budget right now."
(Score: 4, Insightful) by pkrasimirov on Tuesday August 09 2016, @04:09PM
*battery failure*
Boss: "Why the f*** didn't you prevent that?! You had it coming for a long time!"
Peon: "But... but... budget?"
Boss: "Is it any cheaper to have this outage?! You're fired!"
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @04:20PM
Another unemployed peon who can't afford food will be living off Soylent and Tang.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @06:47PM
Another unemployed peon who can't afford food will be living off Soylent and Tang.
If he can afford that Soylent crap then he can afford real food.
(Score: 2) by maxwell demon on Tuesday August 09 2016, @09:02PM
Well, that's why you document the first situation. Because then the second situation becomes:
*battery failure*
Boss: "Why the f*** didn't you prevent that?! You had it coming for a long time!"
Peon: "Well, as this document proves, I already warned about the problem and suggested an appropriate action, which you denied for budget reasons."
Boss: "Err … well … so the blame has to go to that dumb guy that didn't give our department a large enough budget."
The Tao of math: The numbers you can count are not the real numbers.
(Score: 5, Informative) by Anonymous Coward on Tuesday August 09 2016, @05:53PM
This is what I wrote yesterday on the other site.
Having had my hands in designing a few of these sorts of 'never go down' systems it is not 'easy' either as slapping a few bits of cisco/hp/dell kit together and calling it a day.
First off you need a minimum of 2x the floor space in a min 2 different very different geographic locations.
Second you need a min 2x the hardware at both locations. Oh and make sure you DCs can have fail over power and separate power systems.
You need 2x the number of people running it. 1 set for each location. Support 'can' remote in to each location, preferably onsite but remote location 'can' work but wears out your support staff.
Next you need to design the system to be able to handle what we called 'split brain'. Where half your data is out of the wrong data stores and the system is cross pointing to the wrong data centers. That takes time and proper design of the software, hardware, and network infrastructure.
Your software and QA guys need to have their own sets to play with to make sure. Preferably two sets each. Does not have to be geo redundant. Just virtual redundant to simulate.
Oh and your external network better be able to handle it. So that means playing with the proper providers of ISP networks, AND managing them and holding them accountable to fuckups. Plan on them not being up to task and you have to be redundant.
Dont forget your upgrade plans. How will you fail between systems while you upgrade in place (both hardware, firmware, and software). Oh and your QA should be testing that plan as well.
Also to your end customers and employees? It looks totally transparent. So you better have a decent network guys and load balancer guys.
Also be sure to *TEST* your fail over systems. Do they actually fail over? Do they actually come back up? Do they actually get the right data from the right place? It is not just enough to have them. You want to make sure all of your plans actually work. You can even do it during the day, instead of 3AM on a sunday in a massive conference call. Oh and does the system fail properly while people are using it?
Last of all you *need* and *want* to make sure your VP and up management is 100% on board. If they are not *none* of the junk above matters. Dont bother and find another job as it will never be funded correctly.
It takes a fairly seasoned hand build systems like this. Your fresh off the plane (hehe) h1b probably will not cut it no mater how much smoke the temp agency blows up your ass. You can in place train them. But expect it to take time. Last time I did this it took about 2 months to put the hardware together (and it was a smallish system of 40 or so racks). It took another year and a half to work out all the bugs and procedures. Who gets called when. Who replaces what when. How is software upgraded. How does QA sign off on it etc.
Some of the newer techs like docker, vmware, nosql can help mitigate some of these issues. But not all of them. You need to test them and find the holes. So you can either mitigate them or minimize them.
Also you can outsource but remember they dont 'own' it. You do. All they care about is getting the contract complete. That does not mean a working viable system.
That is the sort of system you want to build for a thing like this. Your customers and your fellow employees *expect* it to 'just work'.
I feel for the dudes at that atlanta data center. Return to service is just the first step. "does not happen again" is the next step and that takes a lot of humility and fortitude to make it happen. It also means not hiding things that are wrong and being a right bastard to 'fix it'.
(Score: 2) by Thexalon on Tuesday August 09 2016, @07:13PM
That presumes, of course, that they really care. If they don't really care, they'll find an unpopular junior admin, blame the whole thing on that one person, fire them, and tell upper management that the root cause was that hapless schlemazel and is now fixed.
"Think of how stupid the average person is. Then realize half of 'em are stupider than that." - George Carlin
(Score: 3, Insightful) by darkfeline on Wednesday August 10 2016, @04:53AM
I guarantee you Delta can afford to pay for it. Take a chunk out of the CEO's umbrella.
Join the SDF Public Access UNIX System today!
(Score: 1) by redneckmother on Wednesday August 10 2016, @05:13AM
Sorry - bad moderation... showed up as "spam", but intended as "insightful". Dunno why that happened... Admins?
Mas cerveza por favor.
(Score: 2) by The Mighty Buzzard on Wednesday August 10 2016, @10:09AM
Hit End or Page Down while the moderation dropdown has focus and it goes to the bottom of the list, Spam. Taken care of.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Wednesday August 10 2016, @12:19PM
Maybe you could add another dashed line below "spam"? It should be a bit more difficult to accidentally pick :)
(Score: 2) by The Mighty Buzzard on Wednesday August 10 2016, @12:29PM
Yup, that is indeed the plan.
My rights don't end where your fear begins.
(Score: 3, Informative) by Anonymous Coward on Tuesday August 09 2016, @03:59PM
You already know that.
No I don't know and I don't care. I made a vow to boycott air travel until the TSA is eliminated, and Jetsetting assholes like you are keeping Delta in business instead. I want to see every airline go bankrupt. I want to see the airline industry demand change from the fascist regime. You're just a complacent piece of shit. Fuck you to hell, supporter of state terrorism.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @04:41PM
>>" I made a vow to boycott air travel until the TSA is eliminated"
That'll show'em, sparky.
You forgot to add: "WAKE UP SHEEPLE!!!"
Can you tell us something about FEMA camps, too?
(Score: 5, Insightful) by Nerdfest on Tuesday August 09 2016, @04:44PM
I think everybody would be a lot happier if they refused to knuckle under the fear-mongering and security theatre, but that's just me apparently.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @05:05PM
People don't like to think. You're a statistical aberration.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @07:10PM
Says the person using the phrase "statistical aberration" ;)
(Score: 3, Touché) by maxwell demon on Tuesday August 09 2016, @09:03PM
Well, I guess this whole site is a statistical anomaly. :-)
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Wednesday August 10 2016, @02:08AM
I like you
(Score: 0) by Anonymous Coward on Wednesday August 10 2016, @10:40AM
If it was a Fascist government you'd have a say in it. In Fascism the many corporations (which are groups of people, not only companies, so there is one corporation for the aviation industry, another for IT workers, etc) would discuss and resolve the issue. With a Republic you get no say except on election day.
(Score: 4, Informative) by arulatas on Tuesday August 09 2016, @04:29PM
The person responsible for outsourcing has been sacked and the empty post will now be filled by an outsourced consultant.
----- 10 turns around
(Score: 2) by WillR on Tuesday August 09 2016, @05:02PM
But I'm sure it looks like a cost savings on paper.
(Score: 1, Funny) by Anonymous Coward on Tuesday August 09 2016, @05:32PM
Someone will do the needfull.
(Score: 2) by TheGratefulNet on Wednesday August 10 2016, @12:20AM
completed at great expense, at the last minute, and in an entitrely different style.
(those who were going to do the sacking, have just been sacked)
"It is now safe to switch off your computer."
(Score: 4, Insightful) by Fnord666 on Tuesday August 09 2016, @04:32PM
Delta Airlines was an IBM outsourcing customer, they may still be today, I don’t know. They haven’t returned my call.
Now I don’t know if any of this applies to Delta Airlines because they are too busy to answer a question from little old me, but I’m sure the answers will appear in coming days
cringely - adverb - To use false modesty in an attempt to inflate one's own self importance.
(Score: 3, Insightful) by Snotnose on Tuesday August 09 2016, @04:41PM
Yeah, read his article on why Softbank bought ARM. Guy has no clue about how Arm's market works.
It was a once in a lifetime experience. Which means I'll never do it again.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @07:37PM
"cringely" is the answer to the question "How does he write?"
(Score: 3, Insightful) by JeanCroix on Wednesday August 10 2016, @02:30PM
(Score: 2) by Nerdfest on Tuesday August 09 2016, @04:41PM
Not much of a speculation; this has the stench of IBM and its crufty mainframe environments all over it. Probably not hardware failure (except perhaps power supply), but if you're ever looking for an OS environment that encourages human error, IBM probably has you covered on their z/OS machines.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @04:44PM
Most if not all new buildings are build with ground fault interrupt on the main.
1) high raise in concord failed twice in a month. First was a rebuild and saw through a wall cut the ground and hot together. Tripped the whole building. Second was a electrician checking to see if the line was hot by comparing to ground.
2) major Corp data center in Florida. Again electrician was checking the line was hot before working on it. Tool out 5 main frames plus 20 more minis. With the open indexes. Took a while to come on line again.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @05:35PM
Just how sensitive is a ground fault interrupter? Hard to believe that a decent voltmeter (with high input impedance, >10meg ohms) would draw enough current to cause a ground fault?? Or do electricians normally use a low impedance VOM??
(Score: 2) by Scruffy Beard 2 on Tuesday August 09 2016, @05:56PM
According to this page (PDF) [nema.org] the standard trip current is about 6mA.
120V/6mA=20kOhms.
But a whole-building supply may be a 600V, raising that to 100kOhms (within an order of magnitude of 1MOhm).
Now why a whole-building should trip on 6mA, I don't know.
(Score: 3, Informative) by Scruffy Beard 2 on Tuesday August 09 2016, @06:08PM
Whole-building devices may be more likely to trip at 500mA.
Residual-current device [wikipedia.org]
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @08:34PM
Those sorts of devices can indicate phantom voltages.
For a Go/No-Go test, an electrician will tend to use a device that pulls a bit of current. [google.com]
-- OriginalOwner_ [soylentnews.org]
(Score: 1) by jrmcferren on Tuesday August 09 2016, @10:42PM
The ground fault relay is probably factory set to 50 amps or so. These aren't your standard GFCI circuits, these are designed to trip to prevent a nasty arcing fault from occurring. When installed correctly the relay is set to take in account the tripping curve of the largest downstream breaker (both time and current) with the idea that a fault in a branch circuit will trip ONLY it's circuit breaker and the ground fault relay only trips IF there is a fault in the switchgear itself. In this case there was likely either the incorrect product installed (a low threshold device that operates more like a GFCI) or there was an accident and the relay was not coordinated with the downstream breakers. Remember, a ground fault should only ever activate one overcurrent (fuse or breaker) device if everything is coordinated correctly.
(Score: 1, Interesting) by Anonymous Coward on Tuesday August 09 2016, @09:44PM
Transfer switch the thing connects the internal supply (UPS generally) to either the grid or generator. That is what when out, hence "power problem".
Work in Memphis, where we had that problem one day. The transfer switch was located outside of the build on a 6ft "power pole". Conduit ran to grid transformer, generator, and into the building. Was located there so it could be tripped manually by security guards (just in case), with getting access to equipment.
Karma and lightning is bitch. The lightning bolt hit the transfer switch frying it. Generator also cooked. UPS, too. Grid transformer was good, but no way to power the reconnect quickly. The main frames internal dual "UPS" gave it 5 minutes to shutdown before those main primary batteries went dead. 2x 30A 3-phase, to dual 300 pound, top of cabinet mounted. Actually comes with an "engine puller" device to get them up there and take it back down. The main cabinet had dual 500VDC buses (one from each UPS) that each card/planer connected to., each with dual voltage power regulators. Think Googles 48V cabinets were cool.
Hell on main frame, it had dual battery packs (looks like 6 AA) and memory on each disk controller card, just in case to handle failure of the actual card or one memory module. You could pull the memory & battery (keeping the memory "fresh") and plug into another card and continue, keeping the raid-5 fully in sync.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @05:38PM
this happened awhile back
http://news.delta.com/delta-and-travelport-agree-transfer-passenger-service-flight-operations-software-responsibility [delta.com]
they have "in-house" stuff i guess
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @06:19PM
"What Happened When Delta Went Down Yesterday?"
Man, I knew airline companies were cocksuckers, but a whole day?
*ba-dum-tsh*
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @11:38PM
I thought this was an article about a famous singer, her new boyfriend and a public event
Obviously I was wrong. This is by no means stopping my imagination.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @08:29PM
OK, pop quiz. Let's say you're the Executive Assistant to the C_O. You're presented with two plans:
1) Spend $4,000,000 on achieving complete redundancy in your IT systems, and you suspect there will be another $2,000,000 or so in cost overruns.
2) Spend $400,000 per year on an insurance premium for IT system loss events, and analysis shows a system loss event will cost your company $600,000 in uncovered costs.
Which do you recommend your boss choose and why?
Let's do one more:
1) You can wait for a critical element of your infrastructure to die, costing several hours in downtime plus parts to fix it. You take it on the chin on that day but two weeks from that nobody will remember it.
2) You can upgrade your systems now, costing at least three times the downtime costs of plan one in implementation fees plus the added hardware to achieve it. No downtime but no better response than what you had before.
The world is more than just what IT thinks it needs....
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @10:42PM
One solution is called "investing in the future" while the other one is finacially sound even if it might mean that in one hundred years time the same question will have as a financial sound answer "the cave" to the question of " which do you chose: house or cave?" ^_^
(Score: 0) by Anonymous Coward on Wednesday August 10 2016, @06:40PM
True, and I most certainly rigged the numbers to give rigged answers, too! :) But I've heard nothing but, "How could they do that! Stupid Management! [or Stupid I.T.!]" The truth is simply almost certainly more nuanced than that. (Especially in an industry with some actually pretty razor-thin profit margins.)