SoylentNews is people
SoylentNews
Submission Preview
Once Considered Lost, ESA and NASA's SOHO Came Back From the Brink of Death to Work Even Better Than
████ # This file was generated bot-o-matically! Edit at your own risk. ████
Once considered lost, ESA and NASA's SOHO came back from the brink of death to work even better than it did before [theregister.com]:
Space Extenders Welcome to the final episode in The Register'sseries on engineering longevity in space [theregister.com]. We conclude with the joint ESA and NASA project, more than 24 years into a two-year mission: the Solar and Heliospheric Observatory (SOHO) [esa.int].
Launched [esa.int] atop an Atlas II-AS from Cape Canaveral Air Force Station on 2 December 1995 at 08:08 UT, SOHO's prime scientific objectives are to investigate the outer layer and study the interior of the Sun as well as observe the solar wind.
The spacecraft itself consists of two modules, a service module providing power, thermal control, pointing and communication, and a 610kg payload module housing SOHO's 12 instruments (supplied by US and European institutions).
SOHO orbits the First Lagrangian Point (L1), lurking approximately 1.5 million kilometres from Earth and enjoys an uninterrupted view of the Sun. Its primary mission was supposed to last for two years but is now approaching the quarter-century mark.
However, it is not its extraordinary longevity for which SOHO is famous. It is for an almost mission-ending incident in 1998, the recovery from which cemented the probe's reputation as one of ESA's luckiest spacecraft, as well as one of its most long-lived.
The Register spoke to Project Scientist for SOHO Bernhard Fleck [esa.int], his NASA equivalent Joseph Gurman, and Spacecraft Control Engineer Ton van Overbeek about the mission, recovery and future for the veteran spacecraft.
The luckiest spacecraft on Earth?
SOHO's good fortune began before launch as engineers identified potential problems with the spacecraft's reaction wheels, used for pointing purposes. Fleck told us that SOHO used the same type of wheels as ESA's XMM-Newton spacecraft and, four months before SOHO's launch, tests had shown some vibration.
Fleck recalled: "Guenter Brueckner [harvard.edu] was the PI [principle investigator] of LASCO [SOHO's Large Angle and Spectrometric Coronagraph] and at that time kind of the spokesperson of the PIs. At one of the reviews he stood up and protested against management: 'You cannot fly these wheels, and if it causes a delay of the mission, so be it.'"
The team took the wheels apart and did indeed find issues. After a refurbishment and relubrication, the wheels have since performed admirably. "These are the best wheels you can dream of," laughed Fleck, knocking on wood, "there is no sign of change in 25 years, we've never had to lubricate them, they just work."
SOHO's good luck continued at launch, which, according to Fleck, should have been on 23 November 1995 [esa.int]. A flaw was discovered in a precision regulator, which throttled the power on the booster. The component was switched out and retested.
"One hour before the launch," he told us, "under pressure, one of these diaphragms (in the Atlas II-AS) had broken and the launch was scrubbed. This diaphragm was under pressure already for several hours. If it had held another hour and we had launched... we would have blown up!"
As it transpired, when SOHO eventually launched, the accuracy with which it was placed into its halo orbit meant the team found itself with a substantial amount of fuel left in SOHO's tank. It would prove very useful, as things turned out.
A further bullet was dodged with another decision, prior to launch, to switch out one of SOHO's tape recorders with a solid-state device. Tape was the norm, but since solid-state was now a possibility the team hedged their bets and fitted one.
"This solid-state recorder," Fleck told us, "is operated with exactly the same interface to the spacecraft as a tape recorder. A tape recorder goes bi-directionally, so you record and when you playback you just wind in the other direction." The solid-state recorder had to appear to work the same way.
While not hugely relevant during the early years of the mission, the Solid State Recorder (SSR) would prove its worth from 2003, when SOHO began experiencing problems with its High Gain Antenna (HGA).
It is, however, the incident [nasa.gov] five years prior to the HGA issue that the SOHO mission is famous for, and the mistakes that almost resulted in the loss of a perfectly functional spacecraft.
Of gyros and momentum management
Back in 1998, SOHO enjoyed three functioning gyroscopes. The trio were not used for much of the mission, normally seeing action for thruster-based activities, such as momentum management, Emergency Sun Reacquisition (ESR), and Initial Sun Acquisition (ISA).
The gyros required periodic calibrations to ensure the attitude control system of SOHO was pointing the spacecraft correctly. In addition, once every two months, SOHO's reaction wheels needed to be spun down as they approached their speed design limits. Without the wheels in action, the attitude of the spacecraft was controlled with thrusters.
A seemingly routine calibration of the roll gyros and a normal momentum management manoeuvre, coupled with a ground team working with new procedures, led to a loss of contact with SOHO on 25 June 1998 and, were it not for some impressive boffinry, the end of the mission.
Gyros tend to be a life-limited resource for spacecraft so mission plans are on the conservative side regarding their use. For SOHO, gyro A, used for roll rate sensing and ESR using thrusters, and connected to the Fault Detection Electronics (FDE), was normally spun down after calibration in order to preserve its life. Gyro B was also connected to the FDE and was used to spot excessive roll rates. Gyro C was hooked up to the Attitude Control Unit (ACU) and was used for roll attitude sensing when the thrusters were in action
ESR is a safety net in the event of something bad happening. It is hardwired, part of the FDE rather than the ACU, and uses thrusters to control the spacecraft. Gyro A should spin up automatically if the ESR mode is tripped.
Five hours to catastrophe
Following a routine calibration of all three gyros, gyro A was spun down as normal. However, the team were using an update command sequence to compress the timeline (and make more time for science) which followed the gyro work immediately by momentum management. A step was missed in the sequence and the function to spin up the gyro needed for ESR was not enabled.
"This omission resulted in the removal of the functionality of the normal safe mode and ultimately caused the catastrophic sequence of events," according to the final report [nasa.gov] into the incident.
It got worse. After the momentum management manoeuvre, gyro B was erroneously left in its high-gain setting (thanks to a change in command procedures), meaning that a roll rate 20 times greater than reality was reported. An ESR was triggered, the fifth since the mission began, at 23:16 UTC on 24 June 1998.
As per design, gyro A replaced gyro C as the roll gyro used for ESR thruster-based control. Gyro B continued in its fault detection role, and the team noted the error in its gain and corrected it. Alas, nobody spotted that gyro A was still despun.
The spacecraft kicked off Initial Sun Acquisition (ISA) as part of the ESR process, but used the still despun gyro A. The result was thruster firings that tripped the fault detection system, which was using gyro B, in less than a minute. SOHO fell into its sixth ESR at 02:35 UTC on 25 June 1998.
Although still pointing the correct direction, and thermally safe, SOHO was in serious trouble even if the controllers had yet to realise it.
The spacecraft was depending on a deactivated gyro for roll control in ESR and ISA mode and the roll rate was not looking good. The ground control team, however, had not grasped this. Instead, the team reckoned that gyro B, which was performing fault detection duties, must be at fault since its output (correctly showing the anomalous roll rate) disagreed with the despun gyro A, which was showing zero.
So, the decision was taken to command off gyro B.
Things swiftly went south. Gyro A was despun. Gyro B and its fault detection were inactive. The ground team instructed SOHO to move to ISA mode, which resulted in more firing of the roll thrusters as the spacecraft attempted to null the attitude error associated with Gyro A until eventually the Sun-pointing error exceeded the five-degree limit.
The seventh and final ESR event occurred at 04:38 UTC on 25 June 1998 and all telemetry was lost shortly after at 04:43 UTC.
Had the team realised that gyro A had been despun, the mishap could have been prevented.
"The incorrect diagnosis of a gyro B fault and the subsequent ground response to this diagnosis ultimately resulted in loss of attitude control, subsequent loss of telemetry, and loss of power and thermal control," observed the report [nasa.gov] into the incident.
Hindsight is a wonderful thing. SOHO appeared lost and the mission, just past the nominal two-year point, was likely over.
Fleck was there as events unfolded. "We had the first ESR: we recovered from it like we did before – we had already gone through several ESRs. It was nothing really new. We had a second ESR during the recovery from the first.
"That should have told us something, looking back," he mused. "To run into repeated ESRs. Something is not quite right.
"It took us 10 or 15 minutes to realise what had happened," he added. One can only imagine the sensation as the awful realisation dawned.
Fleck went on: "When you realise you made a blunder that lost a billion-dollar mission, a flagship mission of ESA and NASA... I saw people age in those 15 minutes by 10 years."
Fleck recalled the first meeting with NASA and ESA the following morning, where the situation was assessed. He recalled a pair of senior NASA engineers simply saying "forget it" before leaving the room. The spacecraft was surely lost.
Joe Gurman, Fleck's NASA counterpart, told us: "There was certainly a feeling among NASA management and people here familiar with similar incidents on earlier missions that we had at best a ghost of a chance at recovering SOHO."
Both Gurman and Fleck paid tribute to Dr Roger Bonnet [esa.int], ESA's Director of Scientific Programmes from 1983 to 2001, who championed the recovery efforts.
"He was like a true leader," recalled Fleck. "'You bring me back my mission; you tell me what you need to do it and I will make it available.'"
Bonnet's support proved essential for the team's morale and he ensured extra ESA resources were made available to rescue SOHO.
Gurman agreed, telling us that as well rallying the troops with an insistence that the mission would be recovered, "[Bonnet] backed that determination with a major commitment of really top-notch ESA and contractor engineers."
NASA would eventually commit engineering time as well, but as Gurman observed: "It was really ESA that took the lead."
Recovery
For those following at the time (including this hack), the recovery of SOHO was a gripping saga documented by newfangled blogging technology.
The Flight Operations Team continued to uplink commands via NASA's Deep Space Network in the hope of getting a response from the spacecraft, but were met with silence. Engineers speculated that SOHO's solar panels might be edge-on to the Sun so not generating power, and as the spacecraft's orbit progressed, the Sun might illuminate the arrays and bring the probe back to life.
But still SOHO remained silent.
Gurman recalled asking a radio astronomer friend if it would be possible to use a large radio telescope to hunt for SOHO in and around where the spacecraft was expected to be. He'd read work had been done in the field to detect Near Earth Objects. His friend pondered the question, before replying, "No, I don't think so," and gave Gurman some reasons why it wouldn't work.
"Fortunately for us all," laughed Gurman, "a friend in Colorado was far more persistent."
It transpired that the only real problem (other than nabbing time on the scarce resources) was that it would not be possible for something like the 305-metre Arecibo radio telescope to act as transmitter and receiver. The team hit, according to Gurman, on "a very old idea back from World War Two: the bistatic radar."
This required a transmitter (Arecibo) and a 70-metre dish from NASA's Deep Space Network to receive.
Locating SOHO was critical. One doomsday scenario speculated that a thruster was stuck on, and the spacecraft had diverged from its halo orbit. But if it could be found, there was a chance it might respond.
On 23 July, scientists had a crack at locating SOHO [nasa.gov] by transmitting a signal from the Arecibo radio telescope to where the spacecraft was expected to be, using the 70-metre DSN dish at Goldstone as a receiver.
It took a little more than an hour to find the observatory. The spacecraft turned out to be slowly rotating close to where the team had expected to find it.
After more than a month of silence, very brief contact was re-established. The stricken spacecraft answered commands sent through the DSN station at Canberra on 3 August with bursts of carrier signals, lasting from 2 to 10 seconds.
The team swiftly commanded SOHO to divert what power was available from the solar arrays into charging one of the onboard batteries. By 8 August, sufficient power was available to fire up the telemetry for insight into the state of the spacecraft.
The results were not good. The fuel was frozen, but the team persevered. After fully charging both batteries, the fuel tank was thawed, a process that took 275 hours (including interruptions to recharge the batteries). On 30 August, the team began heating the first of four fuel pipes, using them to connect the tank to the thrusters.
While thawing and recharging continued, the team went through the procedures to recover SOHO's attitude, verifying and rehearsing them until on 16 September the gang was ready to hit Go. Sun pointing was achieved at 18:30 UTC, following a despin of the spacecraft and a planned ESR.
The recovery of the lost spacecraft was completed, and SOHO returned to "normal" mode at 19:52 UTC on 25 September. The instruments [nasa.gov] that formed the payload were then reactivated, starting with SUMER [mps.mpg.de] on 5 October and ending with CELIAS [uni-kiel.de] on 24 October.
Astonishingly, and despite the extremes of heat and cold to which they had been subjected, all 12 instruments performed as well as they had prior to the mishap. "Some even better," according to an ESA report [esa.int] [PDF].
Going gyroless
While SOHO's payload had survived its thermal adventure, the gyros themselves were not so lucky. "One of the things that did get too cold because it was on the inside of the spacecraft were the... I guess the best thing to call them is 'aviation heritage gyroscopes'," said Gurman.
The bane of many a spacecraft over the years, and a factor in the near-scrapping of SOHO, losing the gyros meant stabilisation would wither even after the engineers' Herculean efforts. The last gyro failed in December 1998 and development to find a way of operating the mission without gyros [harvard.edu] was put into high gear.
The Register spoke to SOHO spacecraft control engineer Ton van Overbeek, who, after modestly pointing out that "a lot of people worked on it", explained how the mission was saved yet again.
With the gyros no longer available, attention turned to the spacecraft's star tracker as an alternative roll measurement. Although SOHO was power positive, every time ESR mode was tripped, 7kg of fuel per week was used, according to Fleck. Controllers could also manually operate the thrusters.
However, while the spacecraft had more than 200kg of fuel in the tank, thanks to the accuracy of its orbit insertion, the propellant would be exhausted in six months unless something was done.
Fleck summarised the situation: "So, half a year at max to come up with a solution which has never been done before, to develop software for a three-axis stabilised spacecraft, which was designed for gyros, but without gyroscopes. And the deadline is non-negotiable."
The engineers came through once again.
By the 19 April 1999 design review at ESTEC, an entirely new Attitude and Orbit Control Subsystem (AOCS) mode had been developed for the Central On Board Software (COBS). The flight manual was rewritten, and the NASA controllers trained in the new way of operation. Between 24 September and 4 October 1999, the updates were installed and commissioned. A formal spacecraft recertification review was successfully conducted on 13 April 2000 at NASA's Goddard Spaceflight Center.
"It is based," said van Overbeek, "on the fact that over a day or so the total angular momentum of the spacecraft remains fixed in inertial space (moves only 1 degree per day due to orbit around the Sun).
"We use the angular momentum component perpendicular (transverse) to the roll axis. The momentum vector in the spacecraft can be constructed from the wheel speed measurements. The angular change in the transverse component is the amount the spacecraft has rolled.
"This is done while the wheels are controlling the spacecraft. Pitch and yaw are measured with the Fine Pointing Sun Sensor. So, the wheels are used both as sensor and actuator.
"Using this, the accuracy achieved with 180-degree rolls is an order of magnitude better than when using the gyro. On a 180-degree roll we are usually off by only as little as 40-80 arcseconds."
In the gyroless mode, the thrusters are fired in short pulses with relatively long periods between.
Perversely, the early failure of the gyros contributed to the longevity of the SOHO mission. Fleck observed that had the devices lasted a few more years, then the effort to develop the new mode might not have been deemed worthwhile. "The gyros would not have lasted 20 years. After six or eight years, nobody would have invested in gyroless..."
Following the success of SOHO's gyroless operation, one has to wonder why, other than when releasing the spacecraft from its booster, use gyros at all? Reflecting on NASA's Solar Terrestrial Relations Observatory (STEREO) mission, Gurman remarked: "I think conservatism.
"It was a design that had been used since the late '40s in aircraft. And simply the fact that we've always done it this way; therefore it's 'low risk'. And I hope that people think differently in the future about designing the attitude control systems for solar spacecraft."
Keeping the data flowing and dealing with flaking sensors
SOHO continued burning through its lives (to use a particularly tortured feline metaphor) in 2003 after its High Gain Antenna (HGA) got stuck in one axis. Despite temperature cycling and moving the motor back and forth, the HGA remained stuck in the east-west axis, potentially limiting the science data that could be received from the spacecraft.
To work within the new limits of the HGA position, the team parked the antenna in the best position for coverage over half a halo orbit, and now rotate the spacecraft through 180 degrees to get coverage for the other half. Unfortunately, there remained a two to three-week period between the halves where there would be no HGA coverage. These were dubbed "keyhole periods [esa.int]".
The ever-creative team then worked out that when the HGA was temporarily out of action, the Low Gain Antenna (LGA) could be used, in conjunction with a 70m DSN dish on Earth, to achieve the high-gain data rates needed to dump data from the spacecraft.
The DSN is a resource in demand, and while the keyhole periods only occur twice per orbit (or approximately once every three months) there was still a chance important science data might be lost due to availability of the dishes. SOHO, however, possessed a pair of recorders, one tape and one solid-state (added to the mission late in the day).
While the tape recorder, by its nature, is very much a sequential device and cannot be started or stopped without a slight delay, the solid-state recorder (SSR) offered some intriguing possibilities for engineers. The instantaneous start-and-stop ability meant that science data could be streamed to the SSR from the payload when high-gain connectivity was not available, before eventually being sent back to Earth.
Van Overbeek paid particular tribute to ESA's contractor, Saab-Ericsson Space, during development of the patch from April to September 2003. The patch was finally uploaded in September 2004 and has been used to keep data flowing during keyhole periods since.
Also in 2004 came the problem with the Fine Sun Pointing Attitude Anomaly Detector (FSPAAD) on which suspicion fell as SOHO tripped into a number of seemingly spurious ESR events. The FSPAAD triggers when the spacecraft points away from the Sun by more than five degrees. A Coarse Sun Pointing Attitude Anomaly Detector (CSPAAD) exists to warn when things reach 25 degrees.
After a normal ESR recovery operation, the team noted that the FSPAAD triggered another ESR event. All the signs were that the spacecraft was pointing the right way, so the gang turned to the sensor itself, which consisted of a hole and a solar cell. In a triumph of simplicity, if there was no sunlight through the hole, the spacecraft must be pointing away from the Sun.
However, the team speculated that a flake of something must be over the hole used by the FSPAAD.
"What we think has happened," said Fleck, "is that some of the MLI [Multi-Layer Thermal Insulation] around the spacecraft had become flaky and is shadowing this sun sensor." Indeed, SOHO's LASCO instrument imaged a piece of MLI [esa.int] back in 2005.
The result was a disabling of the sensors, with their functions replaced by the Sun signal from the Fine Pointing Sun Sensor (FPSS).
The future
SOHO's mission continues even as the spacecraft passes the 9,000-day mark.
The science return from the spacecraft over multiple solar cycles has surpassed all expectations despite the team gradually reducing in size from its original 24/7 operations in the 1990s to something considerably more automated today.
It has also enjoyed an additional career as a comet hunter, notching up its 4,000th comet (nicknamed SOHO-4000) [esa.int] in June this year. Thanks to its location, its instruments – the Large Angle and Spectrometric Coronagraph Experiment (LASCO) – and long lifetime, SOHO has inadvertently also become "the greatest comet finder in human history", according to ESA.
But all good things must come to an end.
"The one consumable that will eventually end the life of SOHO," reflected Fleck, "is the solar arrays... We have power at least until the end of 2026," he said, before laughing that each study performed on the hardware had always added five more years.
Degradation aside, what is most likely to finally end SOHO's mission is the launch of new spacecraft with newer instruments able to do SOHO's work. "I expect sometime in 2025," said Fleck ruefully, "the mission will come to an end."
As we've seen during this series, ESA has a long history of keeping spacecraft running and performing useful science long past their expected end date and the thought of the plug being pulled on SOHO while it remains functional seems wrong.
SOHO is, however, different.
"I don't think it is up to ESA," said Fleck. "ESA cannot run the spacecraft without NASA." And, at some point, NASA will want it removed from its bottom line, particularly once other missions with similar capabilities are up and running. An extension already in process will see SOHO operate until 2022. The next will take things to 2025 and Fleck does not expect anything beyond that.
Even if SOHO's mission had ended a decade or so earlier, the data returned by the spacecraft would have revolutionised solar science. Nearly 25 years of operations has afforded scientists with an unprecedented insight into our nearest star.
Fleck has spent nearly 28 years with SOHO and became emotional when discussing the eventual deactivation and safe disposal of the spacecraft, which will consist of a final uploaded command sequence.
"I dread the moment when I will go in and hit the return key..."
And that will be that.
Further reading
The loss and recovery of the SOHO mission in 1998 is well documented, and we'd recommend the final report [nasa.gov] of the SOHO Mission Interruption Joint NASA/ESA board as a jumping-off point. The development of SOHO's gyroless operation and the solution to 2003's High Gain Antenna problem are described in Ton van Overbeek's "SOHO, What Happened After 1999? [harvard.edu]" article.
Both Ton van Overbeek and Joseph Gurman are now retired, although Bernhard Fleck remains enthusiastically at the helm of SOHO, and we are grateful for their time in describing the highs and lows of this extraordinarily long-lived mission.
Like van Overbeek, Gurman and Fleck, we must also pay tribute to the talented engineers and scientists at ESA, NASA and the spacecraft contractors who created and maintained a spacecraft that continues to exceed all reasonable expectations. ®
Get ourTech Resources [theregister.com]
