Boeing acknowledges "gaps" in its Starliner software testing:
On Friday, during a detailed, 75-minute briefing with reporters, a key Boeing spaceflight official sought to be as clear as possible about the company's troubles with its Starliner spacecraft.
After an uncrewed test flight in December of the spacecraft, Boeing "learned some hard lessons," said John Mulholland, a vice president who manages the company's commercial crew program. The December mission landed safely but suffered two serious software problems. Now, Mulholland said, Boeing will work hard to rebuild trust between the company and the vehicle's customer, NASA. During the last decade, NASA has paid Boeing a total of $4.8 billion to develop a safe capsule to fly US astronauts to and from the International Space Station.
At the outset of the briefing, Mulholland sought to provide information about the vehicle's performance, including its life support systems, heat shield, guidance, and navigation. He noted that there were relatively few issues discovered. However, when he invited questions from reporters, the focus quickly turned to software. In particular, Mulholland was asked several times how the company made decisions on procedures for testing flight software before the mission—which led to the two mistakes.
He struggled to answer those questions, but the Boeing VP said the reason was not financial. "It was definitely not a matter of cost," Mulholland said. "Cost has never been in any way a key factor in how we need to test and verify our systems."
The first software error occurred when the spacecraft captured the wrong "mission elapsed time" from its Atlas V launch vehicle—it was supposed to pick up this time during the terminal phase of the countdown, but instead it grabbed data 11 hours off of the correct time. This led to a delayed push to reach orbit and caused the vehicle's thrusters to expend too much fuel. As a result, Starliner did not dock with the International Space Station.
The second error, caught and fixed just a few hours before the vehicle returned to Earth through the atmosphere, was due to a software mapping error that would have caused thrusters on Starliner's service module to fire in the wrong manner. Specifically, after the service module separated from the capsule, it would not have performed a burn to put the vehicle into a disposal burn. Instead, Starliner's thrusters would have fired such that the service module and crew capsule could have collided.
NASA and Boeing have been conducting a joint assessment of these software problems, and they're expected to report their findings in a week, on March 6. But on Friday, Mulholland was prepared to discuss two issues with Boeing's software verification that the company intends to fix.
First of all, he acknowledged the company did not run integrated, end-to-end tests for the whole mission. For example, instead of running a software test that encompassed the roughly 48-hour period from launch through docking to the station, Boeing broke the test into chunks. The first chunk ran from launch through the point at which Starliner separated from the second stage of the Atlas V booster. Unfortunately for Boeing engineers, the mission elapsed timing error occurred just after this point in time. "If we would have run the integrated test through the first orbital insertion burn time frame, we would have seen that we missed the burn," Mulholland said.
Related Stories
Boeing to Launch Starliner Spacecraft for Second go at Reaching the ISS After First Mission Failed:
On Monday, Boeing announced it will take a second shot at sending an uncrewed Starliner to the station as part of NASA's Commercial Crew Program. The program aims to launch astronauts from US soil for the first time since the end of the space shuttle era in 2011.
[...] "We have chosen to refly our Orbital Flight Test to demonstrate the quality of the Starliner system," Boeing in a brief statement. "Flying another uncrewed flight will allow us to complete all flight test objectives and evaluate the performance of the second Starliner vehicle at no cost to the taxpayer."
Boeing and NASA have not yet revealed a date for the launch. Starliner must pass its uncrewed flight tests before NASA uses it to send astronauts to the ISS.
Do not cry too much for Boeing as they are the prime contractor for the SLS (Space launch System) which is currently funded to the tune of over $1 billion per year.
Previously:
(2020-03-07) Boeing Hit With 61 Safety Fixes for Astronaut Capsule
(2020-03-01) Boeing Acknowledges "Gaps" in its Starliner Software Testing
(2020-02-07) NASA Safety Panel Calls for Reviews after Second Starliner Software Problem
(2019-12-24) Boeing Starliner Lands Safely in the Desert After Failing to Reach Correct Orbit
(2019-12-23) Boeing's Failed Starliner Mission Strains 'Reliability' Pitch
(2019-12-20) Starliner Fails to Make Journey to ISS
(2019-11-19) Boeing Provides Damage Control After Inspector General's Report on Commercial Crew Program
(2019-11-06) Boeing Performs Starliner Pad Abort Test. Declares Success Though 1 of 3 Parachutes Fails to Deploy.
(2019-09-03) Boeing Readies "Astronaut" for Likely October Test Launch
(2018-04-07) Boeing Crewed Test Flight to the ISS May be Upgraded to a Full Mission
Click to search SoylentNews for more Starliner stories.
Independent reviewers offer 80 suggestions to make Starliner safer
Following the failed test flight of Boeing's Starliner spacecraft in December, NASA on Monday released the findings of an investigation into the root causes of the launch's failure and the culture that led to them.
Over the course of its review, an independent team identified 80 "recommendations" for NASA and Boeing to address before the Starliner spacecraft launches again. In addition to calling for better oversight and documentation, these recommendations stress the need for greater hardware and software integration testing. Notably, the review team called for an end-to-end test prior to each flight using the maximum amount of flight hardware available.
This is significant, because before the December test flight, Boeing did not run an integrated software test that encompassed the roughly 48-hour period from launch through docking to the station. Instead, Boeing broke the test into chunks. The first chunk ran from launch through the point at which Starliner separated from the second stage of the Atlas V booster.
Previously: Boeing's Failed Starliner Mission Strains 'Reliability' Pitch
Boeing Starliner Lands Safely in the Desert After Failing to Reach Correct Orbit
NASA Safety Panel Calls for Reviews after Second Starliner Software Problem
Boeing Acknowledges "Gaps" in its Starliner Software Testing
Boeing Hit With 61 Safety Fixes for Astronaut Capsule
Boeing to Launch Starliner Spacecraft for Second Go at Reaching the ISS after First Mission Failed
(Score: 5, Funny) by Runaway1956 on Sunday March 01 2020, @05:55PM (6 children)
"As we fall toward a fiery death, you will be comforted to know that Boeing has acknoledged that there are some gaps in the software testing. These gaps are expected to be filled in sometime after we all die."
“Take me to the Brig. I want to see the “real Marines”. – Major General Chesty Puller, USMC
(Score: 2) by janrinok on Sunday March 01 2020, @06:47PM (1 child)
[nostyle RIP 06 May 2025]
(Score: 0) by Anonymous Coward on Sunday March 01 2020, @09:54PM
Sort of it only hurts when I laugh?
(Score: 5, Insightful) by JoeMerchant on Sunday March 01 2020, @10:47PM (2 children)
Missed the intentional one, replaced by a much larger unintentional one.
They say it's not about money, but it is about schedules. It's true that you can scale up the project teams, double the headcounts, and not make any more positive progress on complex systems development.
Everyone continually praises the Apollo flight computers and how much they did with so little, but... that's really a saving grace: they did so little that multiple people could fully understand the system from end to end, when something went weird there were people on hand who could tell you, in a moment, what was going on.
Ever since software moved past the 8 bit, 64K memory space model, I feel like it has gotten out of hand, layers upon layers upon layers with some layers that literally nobody on the planet understands anymore.
🌻🌻🌻 [google.com]
(Score: 1, Insightful) by Anonymous Coward on Monday March 02 2020, @09:01AM (1 child)
Like systemd?
(Score: 2) by JoeMerchant on Tuesday March 03 2020, @12:55AM
systemd is an easy target, but... it is not without its merits. If nothing else, forcing SysVinit and friends to clean up their acts and compete with systemd in the boot time and other performance domains.
A couple of years ago, systemd was an option in Raspbian, and switching to it reduced boot times by ~50%, now it is standard. When a more "open and friendly" init system can deliver the goods, I welcome the demise of systemd. Meanwhile, I use my init system about 100x more than I tweak it, and my customers NEVER tweak it - so open user friendliness is of zero importance to them.
🌻🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Monday March 02 2020, @07:59PM
Please, let this become a meme.
(Score: 2, Interesting) by Anonymous Coward on Sunday March 01 2020, @06:16PM
> did not run integrated, end-to-end tests for the whole mission.
Reminds me of the Formula SAE team (students build small race cars) that never ran a full "22 km endurance" run in their testing. Their car failed because the pickup in the gas tank didn't get all the fuel out--tank was large enough, plenty of gas, but engine starved for fuel and quit.
Probably a 15 minute fix, but they never ran the right test to find the problem...until competition.
(Score: 5, Insightful) by Snotnose on Sunday March 01 2020, @06:31PM (4 children)
Typically these are broken up into a handful of modules with specs indicating how everybody talks to each other, but otherwise each module is it's own fiefdom. Every once in a while you schedule an integration test, where all the modules are put together and tested to ensure everybody can talk to everybody else. These tend to be large and resource intensive, and run for at least a day (hopefully). These also always show several places were Things Don't Work As Expected (tm).
The fact that Boeing didn't do this is mind boggling.
Every time a Christian defends Trump an angel loses it's lunch.
(Score: 2) by Beryllium Sphere (r) on Sunday March 01 2020, @11:17PM (3 children)
Boeing used to have a core competency as a system integrator, too, so integration testing should have come naturally.
(Score: 1) by Ethanol-fueled on Monday March 02 2020, @12:26AM (1 child)
I wonder what the reasons for their bad code are? I bet they're brown, stinky, wear ties with blue-jeans, and make the shareholders more money while the company literally crashes and burns.
(Score: 1, Informative) by Anonymous Coward on Monday March 02 2020, @08:03PM
No, Boeing's management is all home-grown USian and predominantly white devil.
(Score: 2) by DannyB on Monday March 02 2020, @04:07PM
System integration testing would cost more. And Boeing is trying achieve this for only double the price that SpaceX bid.
Remember a few years ago when SpaceX bid about half of Boeing? Yet the Dragon2 may be the first to fly humans. Or human flies.
Q. How much did Santa's sled cost?
A. Nothing. It was on the house.
(Score: 4, Insightful) by fustakrakich on Sunday March 01 2020, @06:33PM (9 children)
Yeah yeah pull the other one...
La politica e i criminali sono la stessa cosa..
(Score: 3, Insightful) by knarf on Sunday March 01 2020, @07:37PM (7 children)
Well, if it isn't cost then it has to be competence, or rather the lack of it.
(Score: 3, Insightful) by choose another one on Sunday March 01 2020, @07:59PM (5 children)
...and competence as we all know, is not related to cost at all particularly in software.
This is why you can hire perfectly competent software developers in, say, India, at $peanuts per hour and they are clearly competent as they have ISO 11001, CMM level 11, minimum of 3 degrees and a PhD each and all have ten years proven experience in (for example) Rust and Go.
Just be sure that when writing the spec you can specify exactly how "the software must not crash the plane or spacecraft", otherwise it will, and it will be argued that it is written to spec because you didn't specify that it shouldn't do what it did.
As you can tell, I've never ever had to work with cheap offshore developers at all...
(Score: 5, Funny) by Runaway1956 on Monday March 02 2020, @02:33AM (4 children)
So, uhhh, Ford mechanics?
“Take me to the Brig. I want to see the “real Marines”. – Major General Chesty Puller, USMC
(Score: 2) by MostCynical on Monday March 02 2020, @03:14AM (3 children)
No, they only know how to plug in the diagnostic computer and print off the error codes.
You're thinking of the people who write the system that runs the car, and gps, and radio, and ignition, and cruise control....etc..
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 3, Touché) by Runaway1956 on Monday March 02 2020, @03:29AM (2 children)
Oh, please, tell me that you weren't just "WHOOOSHED". Rust and Go. Somewhat synonymous with "Found On Road Dead". LMAO
“Take me to the Brig. I want to see the “real Marines”. – Major General Chesty Puller, USMC
(Score: 2) by MostCynical on Monday March 02 2020, @03:57AM
Alas, yes. "Too close to home" territory..
Current project supplier arguing about printing being in scope or not, and about how hard it is to change tab order on forms..
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 2) by fido_dogstoyevsky on Monday March 02 2020, @04:54AM
And Fix Or Repair Daily. Or Fscked On Race Day.
It's NOT a conspiracy... it's a plot.
(Score: 3, Interesting) by Beryllium Sphere (r) on Sunday March 01 2020, @11:22PM
I wonder if this was one of the dumping ground projects.
Boeing used to make sacrifice projects, usually sold to customers who didn't know the aerospace game, where they dumped the least competent employees.
Not that the other projects were necessarily full of competent people. Some highlights from my time there were the engineer who told me that friction decreases with increasing speed and the person who looked at a drawing of a hydrazine thruster and told me it was a nitrogen thruster.
(Score: 5, Insightful) by driverless on Sunday March 01 2020, @11:25PM
It's actually true, cost has never been an issue for Boeing's spaceflight program because they know that their sweet sugar daddy, Uncle Washington, will keep feeding them money no matter what happens.
That's one massive thing the up-and-coming spaceflight players have that Boeing doesn't: accountability. They have to show a return on investment and value for money, while Boeing just have to show an ability to bat their eyelashes at their sugar daddy.
(Score: 3, Interesting) by pvanhoof on Sunday March 01 2020, @08:31PM (6 children)
Last few months, Boeing has shown utter incompetence in Q&A. Instead of "Gaps" should Boeing's HQ report that they have fired certain top people in their Q&A division.
Not report that they have found gaps.
(Score: 2) by PiMuNu on Sunday March 01 2020, @09:29PM (5 children)
> should Boeing's HQ report that they have fired certain top people in their Q&A division.
Maybe a start, but they need to figure out what the QA process they have been following is and where the flaws are in that process. Firing people sounds great but rarely fixes the problem; the job is rather to hire people!
(Score: 0) by Anonymous Coward on Sunday March 01 2020, @10:21PM (3 children)
Six Sigma?
(Score: 3, Funny) by driverless on Sunday March 01 2020, @11:28PM
More like six delta where Boeing is concerned.
(Score: 3, Insightful) by Runaway1956 on Monday March 02 2020, @02:38AM (1 child)
Five S, Sigma Five, Six Sigma - that stuff keeps coming 'round and 'round - and every time it does, there's a whole new crop of young little idiots who think they've reinvented sex or something. Sometimes, I wish they would all choke to death on their slimy little sigmas.
“Take me to the Brig. I want to see the “real Marines”. – Major General Chesty Puller, USMC
(Score: 5, Funny) by DannyB on Monday March 02 2020, @04:13PM
But Six Sigma is so good!
Repost: [soylentnews.org]
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Rowing from a Six Sigma Lean perspective [blogspot.com]
A Japanese company and an American company decided to have a canoe race on the Mississippi River. Both teams practiced long and hard to reach their peak performance before the race.
On the big day, the Japanese won by a mile. The Americans, very discouraged and depressed, decided to investigate the reason for the crushing defeat. A management team made up of senior management and internal Lost Race Analysts was formed to investigate and recommend appropriate action. Their conclusion was the Japanese had 8 people rowing and 1 person steering, while the American team had 8 people steering and 1 person rowing.
To validate their conclusions, the American management hired a consulting company and paid them a large amount of money for a second opinion. The consultant advised that too many people were steering the boat, while not enough people were rowing.
Taking pride in quick action and to prevent another loss to the Japanese, the rowing team's management structure was totally reorganized to 4 steering supervisors, 3 area steering superintendents and 1 assistant superintendent steering manager. The American HR team devised an innovative incentive that would give the 1 person rowing the boat greater rewards for working harder. It was called the "Six Sigma Lean - Pay for Rowing Performance - Total Quality Program", with meetings, dinners and free pens for the rower. There was discussion of getting new paddles, canoes and other equipment, extra vacation days for practices and bonuses.
The next year the Japanese won by two miles. Humiliated, the American management team laid off the rower for poor performance, halted development of a new canoe, sold the paddles, and canceled all capital investments for new equipment. The money saved was distributed to the Senior Executives as bonuses and the next year's racing team was outsourced to India.
Q. How much did Santa's sled cost?
A. Nothing. It was on the house.
(Score: 2) by pvanhoof on Monday March 02 2020, @12:15AM
Sure, replacing incompetent people with competent people sounds good.
(Score: 1, Insightful) by Anonymous Coward on Monday March 02 2020, @02:50AM (1 child)
Usually a discussion of s/w testing involves how many corner cases were tried outside the main, expected execution path.
An assumed prerequsite is that you of course tried the main, expected path.
It looks like they missed that not so fine point.
This seems incompatible with the idea of a company that charges big bucks because they know how to do things right.
More worrysome is that the govt folks watching them didn't trust but verify.
Does nobody have a deep enough bench to understand what is happening before a problem makes it obvious?
(Score: 2) by DannyB on Monday March 02 2020, @04:18PM
What you describe sounds like it would cost more money? And Boeing only bid about double what SpaceX did for SpaceX Dragon 2 which may soon fly humans.
Q. How much did Santa's sled cost?
A. Nothing. It was on the house.