SoylentNews is people
SoylentNews
from the details-matter-even-really-little-details dept.
The root cause of the collapse of Baltimore's Francis Scott Key Bridge when hit by container ship Dali has been identified. It was the wrong placement, by a few millimeters, of the label on one wire. As usual, the National Transportation Safety Board has taken their time and done a detailed investigation--summarized in this short video
https://www.youtube.com/watch?app=desktop&v=bu7PJoxaMZg
tl;dr - the wire was not completely inserted into a terminal block, due to the wire label wrapped over the ferrule. Over time the connection became intermittent and eventually shut off power on the ship...after which it drifted into the bridge. Of course there were additional contributing problems as well.
The YT video comments include some more interesting details.
[Ed. note: For those not inclined to watch the YouTube video, the narrative summary of the video is listed in the spoiler below.]
1. The Dali electrical system distributes power and control signals throughout the vessel.
2. The control circuits contain hundreds of terminal blocks that organize thousands of wires.
3. The wires on the Dali were terminated with metal sleeves called ferrules that allowed for easier assembly into the terminal blocks.
4. Each wire was identified with a labeling band.
5. This image shows several terminal blocks on the Dali with wires connected.
6. To assemble a wire into a terminal block, a tool inserted into a side port opens a spring clamp, which allows the wire's ferrule to slide into place.
7. Removing the tool closes the spring clamp, securing the ferrule firmly against the terminal block's internal conductor bar.
8. Labeling bands identify wires and are typically positioned on the wire insulation.
9. However, many labeling bands on the Dali wires were placed partially on the ferrules, which increased the ferrules' overall circumference.
10. As a result, during vessel construction, some of the ferrules could not be fully inserted in the terminal blocks, including the ferrule on wire 1 from Terminal Block 381.
11. On that wire, the labeling band prevented full insertion of the ferrule, so the spring clamp gripped only the ferrule's tip, resulting in an inadequate connection.
12. Due to this unstable connection, over time the ferrule on wire 1 slipped out of the spring clamp to rest atop the spring clamp face, resulting in a precarious electrical connection.
13. When a gap occurred between the ferrule and the spring clamp face, the electrical circuit was interrupted, leading to a blackout on the Dali.
(Score: 5, Insightful) by Runaway1956 on Monday December 01, @09:40PM (22 children)
The NTSB animation makes everything make sense. And, it highlights why craftsmen should always teach the RIGHT WAY of doing everything.
Thank God and technology, there isn't much spaghetti wiring left in service today. But I remember well looking into a spaghetti collander of a cabinet, and finding wires with no labels at all! The only way to figure out what you were looking at, was to trace each wire back to it's origin, then look at the drawings to see what THAT terminal was supposed to be!
So, in the shipyard, electricians were learning to label wires incorrectly, then the labels interfered with the assembly of the wires to the boards. Wow. And, not one supervisor, or inspector, ever caught it. Or, if they caught it, it was deemed not worth the effort of correcting.
How many other ships are affected with the same potential fault?
I'm going to buy my defensive radar from Temu, just like Venezuela!
(Score: 1, Funny) by Anonymous Coward on Monday December 01, @09:57PM
> How many other ships are affected with the same potential fault?
<sarcasm>I'll keep a lookout for more bridges being knocked down...</sarcasm>
(Score: 2, Interesting) by Anonymous Coward on Monday December 01, @10:07PM
That shipyard in Ulsan S. Korea is just down the road from Hyundai HQ. It's a serious factory town, also there (when I visited c.1995) was Hyundai Precision (big machine tools) and several other Hyundai branches. When it came time to check out of our hotel, the clerk just had us confirm "company pays"...so they probably owned the hotel too.
One of many possible scenarios: Maybe that day the electricians were on loan from one of the car assembly lines...where the wire harnesses are all pre-assembled with color coded wires, not individual numbered wire labels?
(Score: 5, Touché) by JoeMerchant on Monday December 01, @11:08PM (7 children)
Seems to me on a ship that size, some aircraft systems' double redundancy would be in order. A single fault like that should not send that much mass adrift in ways that risks dozens of lives and tens of millions in property.
Of course, I have an 40 year old sailboat that's full of single fault failure points like that, but it only weighs 10,000 lbs and when we've had problems (transmission linkage coming apart while backing out into the marina fairway was a fun one), you can kinda grab some rope and throw it to a helpful bystander to get things under control, not so much on a serious freighter that barely fits through the channel in the first place.
🌻🌻🌻 [google.com]
(Score: 2) by VLM on Tuesday December 02, @02:37PM (6 children)
https://www.ntsb.gov/news/press-releases/Pages/NR20251118.aspx [ntsb.gov]
My immediate guess is it was a normally closed e-stop circuit. So any break anywhere and down it goes.
"normally" you can use a double pole switch and use the other pole to sense at a PLC or whatever which of the probably many estop switches is being pressed and remove whatevers leaning on the switch or jammed it or whatever before the UPS batteries shut down. However that'll never work because the break in the circuit is an open wire not a open switch...
(Score: 2) by JoeMerchant on Tuesday December 02, @04:22PM (5 children)
The problem is: conflicting definitions of "safe."
Old joke among helicopter engineers: "The only way to make this thing safe is to make it so heavy it will never fly."
When you've got 200,000 tons moving at six knots, running with a little fire in the engine bay is a whole lot safer than killing all ability to maneuver the ship at a bad moment.
🌻🌻🌻 [google.com]
(Score: 2) by VLM on Tuesday December 02, @05:16PM (4 children)
Ah its worse than a fire where they'd have some control, they seem to have lost control entirely. I found this:
https://www.ntsb.gov/investigations/Documents/Board%20Summary%20Contact%20of%20Containership%20Dali%20with%20Francis%20Scott%20Key%20Bridge.pdf [ntsb.gov]
"When Wire 1 electrically disconnected, one of the high-voltage breakers
connecting the high-voltage bus to its step-down transformer—a mechanism that
lowered voltage carried from the main, high-voltage, electrical bus before transferring
it to the low-voltage bus—opened."
Ouch. They call it a breaker not a contactor (maybe a marine thing?) so I assume its a NC safety circuit that opened but who knows.
It was intermittent and they were running the breaker in manual mode not automatic so someone had to turn it on by hand, which "makes sense" for a safety shut down switch, I guess, although the report ripped on them for having an emergency shutdown switch (assuming thats what it was)
So the HV breaker was feeding a LV transformer causing a LV blackout, the LV was powering the main engine coolant pump and the steering gear coolant (hydraulic cooler?)
The engine had an automatic shutdown when the coolant flow was interrupted.
They were running the generators fuel supply using a maintenance cleaning pump instead of the main and backup fuel pumps (why?) so maint is not safety critical so they had precisely zero fuel pumps available. I guess if you have a triply redundant system some dumbass will try to run everything off the third backup until it fails too.
It was their third blackout in a couple days and they're just like "oh well what could possibly happen?" FAFO and they hit a bridge.
They had a separate backup generator that wouldn't start because some damper was moving too slowly for the firmware to be happy so it shut down.
"Mostly" the piles of safety equipment caused the accident. If they had no emergency electrical shutdown, sure they'd electrocute people and have fires once in a while, but the power would be on. If the engine didn't auto-shutdown when it got toasty, they'd occasionally burn out engines but at least they'd have power. If they didn't have triple redundant fuel systems they wouldn't do something dumbass like not maintain it at all until all three failed at once. If they didn't have a backup generator that worried too much about stuck cooling damper doors, they'd have power.
(Score: 2) by VLM on Tuesday December 02, @08:00PM
I read between the lines some more and I think maybe they linked the circuit breakers for the HV and LV sides of the transformer. So, if either side pops, they both pop. But then they both should have popped. Hmm.
This would make sense because if they popped the HV side and powered up the LV off a plain old generator, then there would be out of phase HV power on both sides of the HV circuit breaker. Probably makes sense to "de-energize the entire transformer" rather than just one winding. Less exciting that way.
More data will be released sooner or later, probably.
(Score: 2) by JoeMerchant on Tuesday December 02, @08:04PM (2 children)
>They were running the generators fuel supply using a maintenance cleaning pump instead of the main and backup fuel pumps (why?)
That's the ingenuity the owners pay them for. Can you make it through next year before submitting any new parts orders? If you do there's a 5% bonus in your paycheck!
> I guess if you have a triply redundant system some dumbass will try to run everything off the third backup until it fails too.
This is why aircraft get annual inspections. Ships above a certain size should, also, but do not AFAIK. Were I harbormaster somewhere with a big fancy commuter bridge over the channel, I do believe I would require safety inspections before letting those scows in my harbor, but then they'd probably replace me with a more "business friendly" harbor master pretty quick.
> they'd occasionally burn out engines but at least they'd have power.
At least until the engine burns out. That one I get. What I don't get is allowing these things entry to ports with no seaworthiness checks whatsoever. The Dali can displace 149,000 tons (according to some sources) and cruise at 22 knots, that's 9.5x10^9 Joules of kinetic energy. If a Kh-47M2 Kinzhal 4300kg hypersonic missile manages to strike at Mach 10 (as I understand things they don't always hit at maximum speed), that's only 2.3x10^10 Joules of kinetic energy - equivalent to 5 tons of TNT - so the ship ramming out of control is only equivalent to 2 tons of TNT so it's "safe enough?"
>If they didn't have triple redundant fuel systems they wouldn't do something dumbass like not maintain it at all until all three failed at once.
As captain of a vessel with more destructive potential than a Mach 6 hypersonic missile, I wouldn't be leaving port without redundant fuel systems operational. That's like driving your car down the freeway with no brakes and a steering linkage that can let go at any moment.
>If they didn't have a backup generator that worried too much about stuck cooling damper doors, they'd have power.
By the time you've MacGyver'ed it that far, you already should be stopped waiting for proper repairs.
🌻🌻🌻 [google.com]
(Score: 2) by VLM on Tuesday December 02, @08:38PM (1 child)
I was thinking about that while eating lunch and I bet steering works at idle. Sure it'll burn out / boil over / warp the heads at full throttle but dropping to idle instead of shutting off altogether might be a safer response to overheating.
Might be a situation where the engine manufacturers guarantee is at financial risk unless the electronics shuts down the engine, vs the risk of knocking over a $5B bridge taking 6 years to replace.
That is also an amazing part of the story, inflation adjusted they built the first bridge for $0.5B equivalent half a century ago, and the replacement is supposed to cost $5B inflation adjusted now. Ten times the cost. Well maybe it'll be ten times as strong, who knows.
(Score: 2) by JoeMerchant on Tuesday December 02, @08:59PM
> Ten times the cost. Well maybe it'll be ten times as strong, who knows.
It'll probably have more effective bumpers against future accidents. That's one thing they did for the Sunshine Skyway after it got clipped.
🌻🌻🌻 [google.com]
(Score: 4, Funny) by krishnoid on Tuesday December 02, @12:13AM (10 children)
Dear God.
I worked at a company that produced physical products that had to comply with federal regulations (remember those?), and one of the documents indicated that you should report non-compliant "signage" (or something similar). I was confused and feeling a little ornery that day, so I called and got ahold of a person and asked how "signage" could be defective or a regulatory compliance issue. She replied that a label could be in the wrong place, the wrong size or the wrong size font.
I thought, "Ok, I guess that makes sense, but it seems a little like overkill."
(Score: 3, Informative) by Anonymous Coward on Tuesday December 02, @01:09AM (9 children)
Where I work we make communication equipment mostly for military, and some satellite communication companies. We're quite strict with everything, including labeling. I don't mean to be mean, but some of the people who will install and use our equipment are not the smartest nor highest trained people. Mistakes could very well cost lives. So yeah, labels and signage could be very critical. Sometimes it seems almost silly when it seems like there's no way someone could mess things up. But our equipment might be going in with many other kinds of equipment, with possibly the same connectors, and mixing them up would cause major failure of many systems. Hopefully people rest a bit easier knowing that somewhere someone is doing things right and well.
(Score: 3, Insightful) by krishnoid on Tuesday December 02, @02:21AM (3 children)
The (U.S., and probably others) military is sometimes the best option for children neglected/underprovided for by family, society, and the educational system. If "not the smartest nor highest trained" are otherwise staring down the barrel of a dead-end job (or a life of crime) since they won't be going to college, they'll find themselves in an environment within a domain of the more complex machinery we have in the country -- coincidentally, which is designed around killing people and damaging harder targets.
I'd say unambiguous labeling gives the loving, caring drill sergeants something to point to while they're yelling at a recruit as to why they screwed up on something that could kill people. If "not the smartest nor highest trained" recipients can get the sense that someone didn't really care when writing up the signage [schlockmercenary.com], they'll also get the impression that materiel suppliers don't care either, which I suspect propagates an attitude (around weaponry and in combat zones) that could eventually kill people.
(Score: 4, Insightful) by JoeMerchant on Tuesday December 02, @03:18AM
> could eventually kill people
I suspect it's more a case of: numerous GAO studies concluding that expensive injuries and deaths could have been prevented with better / less sloppy labeling and that the cost of such labeling, being less than the cost of disability benefits for 50+ years, is a worthwhile investment.
🌻🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Tuesday December 02, @03:27AM
OP here. Great insight, and you're inspiring me to think about military personnel morale, and doing anything I can to inspire them to realize how important their work is, and to take pride in doing good work.
(Score: 0) by Anonymous Coward on Tuesday December 02, @09:53AM
On a related note: https://maritime.org/doc/neets/mod04.pdf [maritime.org]
See also: https://hackaday.com/2017/06/09/electronics-education-courtesy-of-the-us-navy/ [hackaday.com]
https://electriciantraining.tpub.com/14176/css/Rattail-Joint-47.htm [tpub.com]
(Score: 3, Interesting) by PiMuNu on Tuesday December 02, @09:25AM
Many situations where incorrect signage is very dangerous. Repurposing bottles comes to mind as a way to quickly kill people.
(Score: 4, Insightful) by VLM on Tuesday December 02, @02:41PM (3 children)
My personal experience (and I'm not offended) is everybody's smart, until they're awake 24 hours or standing outside in the snow or just made a 15 mile march or they've been doing this for the last 12 hours without a break. Thats when people get pretty sloppy. Labeling helps. Like how paperwork has a standardized explanation and format so you can do it in your sleep, because sometimes you are half asleep...
(Score: 3, Interesting) by hendrikboom on Wednesday December 03, @03:05AM (2 children)
My wife was a doctor.
One night she was on call.
That means the hospital can call her at ungodly hours to advise on the treatment of a patient.
One morning when she got to the hospital she inquired about the status of a patient.
She was informed that the treatment she had recommended in the middle of the night was working.
She had no memory of being consulted during the night.
She concluded that despite being sleep, she must have taken the call and advised the right treatment despite never having woken up.
Scary? Or an indication of extraordinary competence?
(Score: 2) by RS3 on Wednesday December 03, @04:23AM
A little of both maybe? But I do all kinds of things every day that I don't remember, so I wouldn't give it much worry. Now where'd I put those darn keys...
(Score: 0) by Anonymous Coward on Thursday December 04, @02:58AM
> She had no memory of being consulted during the night.
Some similarity to something that happened to me 40 years ago?
I was in a minor accident, slightly concussed (lost memory for a short time) and was transported to the local hospital ER. Took awhile to get into a room at which point I was seen by a very attentive young intern (not yet an MD). He got me some x-rays and kept me overnight for observation. There was also a night nurse, an older tough cookie who (correctly) kept waking me up every half hour, based on the concussion.
A few weeks later I got a bill from an MD's office, along with the MD's report on his examination of me and a totally fabricated "transcription" of his conversation with me.
(Score: 4, Informative) by driverless on Tuesday December 02, @01:52AM
It was actually a bit more than that though, they were using a flushing pump as a service pump to get fuel to the generators and a bunch of other naughty things. For people really wanting their nerd fix, there's a discussion on an EE forum [eevblog.com] that goes into lots of technical detail.