Station teams scramble to resolve computer glitch
BY WILLIAM HARWOOD
STORY WRITTEN FOR CBS NEWS "SPACE PLACE" & USED WITH PERMISSION
Posted: June 14, 2007
A major computer failure in the Russian segment of the international space station, possibly triggered by the addition of new U.S. solar arrays earlier this week, has shut down critical systems and left the outpost dependent on the shuttle Atlantis for any major changes in orientation. The space station has plenty of supplies on board and the combined 10-member crew is in no immediate danger. But the space station cannot safely operate without the Russian computers and the problem must be resolved before Atlantis departs next week.
Russian engineers will "begin in ernest tomorrow morning to try techniques to recover these computers," said Mike Suffredini, manager of the space station program at the Johnson Space Center in Houston. "I fully expect us to be able to do this. I'm not thinking this is something we will not recover from.
"But in the meantime, being NASA and the fact that we try to do everything we can to protect every option, we are looking at options to extend the time that the shuttle guys stay (docked) with us just in case we'd like an extra day or so. We can't extend it much, but we're certainly looking at that. We're also looking at a number of different options for how we'd conserve (propellant) if we decided we needed more attitude control help from our shuttle friends."
Space station assembly began in November 1998 and it has been continuously manned by rotating U.S. and Russian cosmonauts and astronauts since November 2000. Atlantis ferried a fresh U.S. astronaut to the station - Clay Anderson - to replace Sunita Williams, who was launched to the station last December aboard the shuttle Discovery. Shortly after Atlantis docked, Anderson officially joined Expedition 15 commander Fyodor Yurchikhin and flight engineer Oleg Kotov. Williams, in turn, joined Atlantis' six-member crew.
The Russian segment of the space station utilizes three guidance, navigation and control computers to maintain the lab's orientation and make major orbit adjustments by firing rocket thrusters in the Zvezda command module. Three command-and-control computers also are available to operate a variety of systems like the Russian Elektron oxygen generator and the Vozdukh carbon dioxide scrubber.
For normal operations, the station relies on U.S. control moment gyroscopes, or CMGs, which can make small adjustments in orientation without the need for fuel-consuming rocket firings. Controlling the station's orientation, or attitude, is critical for ensuring its solar arrays can stay lined up on the sun and to prevent sensitive systems from getting too hot or too cold.
The Russian system features "three of those (guidance computers), they're all redundant to each other so any one of the computers can do the job," Suffredini said. "And then we have (three) command-and-control-type computers, called 'lanes,' any one of which can do the command-and-control function. And then we have computers on the U.S. segment that talk to these computers."
When problems occur that might take down the guidance computers, the system is programmed to reboot itself and in the past, that procedure successfully resolved a variety of problems. This time around, it didn't.
On Monday, the Atlantis astronauts attached a new 36,000-pound solar array truss segment featuring two new solar blankets capable of generating some 20 kilowatts of additional useable power. On Tuesday, a computer crash occurred that ultimately left the Russian segment with a single guidance computer and a single command-and-control machine. That, in and of itself, was not crippling.
But Wednesday morning, as the Atlantis astronauts were beginning work to retract a different solar array wing, "we lost both of those computers," Suffredini said. "Currently we're in that configuration. The guidance, navigation and control computers and command-and-control computers in the service module are not functioning. Our Russian colleagues tried a number of techniques to try to recover the computers and were not successful."
Unlike NASA, the Russian space agency does not have its own communications satellites. To troubleshoot the computer problem, Russian engineers had to wait until the station was within line-of-site of Russian ground stations to downlink telemetry, beginning early Thursday.
The combined shuttle-station crews have plenty of oxygen, food and water, along with carbon dioxide removal capability. Even so, "this is a very serious situation," Russian flight controllers told Yurchikhin early today.
"Where does that leave us? The guidance, navigation and control computers allow the system to control the attitude and provide propulsion, if necessary, with Russian thrusters," Suffredini said. "Without those computers, we can't get attitude control from the Russian system. That's not a problem for us, the CMGs (U.S. control moment gyroscopes), in fact, are controlling the attitude and they've done a very fine job. ... If they saturate (get overloaded), we can hand over to the shuttle system and they can take care of the attitude control function."
"That's not an urgent situation but clearly we need to get this resolved before our shuttle friends leave," Suffredini said. "We have plenty of resources and plenty of time to sort this out. We have in the history of the program often had these computers go down, one or two lanes. It's not uncommon to be on one lane of each system and wait for the last one to finally give up and then the whole system reboots itself and starts back up. Of course, what's unique is when the system went to reboot itself it wasn't able to do that."
Engineers are trying to identify changes in the station's configuration or the space environment that might have triggered the crashes, including space radiation and the electrical environment around the station.
"The one that folks are starting to look at and scratching their heads a bit about is we did, of course, add another power source in the form of the S3/S4 (solar array) truss, so we're off looking at is there anything about that power source that may make it slightly different for reasons that aren't clear ... that may have affected the ability of these computers to operate correctly," Suffredini said. "And then, of course, on the Russian segment there are a number of things our Russian colleagues will look at as well to see if they can figure it out."
Suffredini said engineers have not ruled out a software glitch, but no software changes were made over the past few days and all six computers "didn't just all have a hardware failure. It appears to me that something has changed in the environment, either something in the (space) environment or the source of power to these computers is different coming from S3/S4 for reasons we do not understand."
"Our Russian colleagues believe it's the power source," he said. "That's the latest theory they have. They've suggested that perhaps we could stop feeding them power and let them just use their own internal power to bring up their computers and see if that solves that particular problem. They cannot run (completely) on their own power because of all the systems to handle the entire crew. We have to feed them a certain amount of power to keep everything running that we need to run. So eventually we have to work through that.
"Another option we're working through is just stop feeding them power from the S3 truss, which is maybe the first thing we'll try. We're still sorting through our plan."
In the meantime, shuttle planners are assessing possible power conservation steps that would permit Atlantis to remain docked an extra day if necessary. As of late Wednesday, the shuttle had enough hydrogen and oxygen for its electricity producing fuel cells for an additional 18 hours of docked activity. By deferring the planned transfer of 40 pounds of oxygen to the station, the shuttle crew will likely be able to get an extra docked day if necessary.
"We have to have propulsive attitude control even with CMGs," Suffredini said. "The CMGs do a great job but occasionally they get saturated and when they get saturated, we need propulsive attitude control in order to recover. The other need for propulsion is to make sure we can do debris avoidance maneuvers, to adjust our orbit for dockings, there's a number of reasons why we need propulsive attitude control. That's a requirement of the ISS. We have to have that capability."
In a worst-case scenario, he said, the astronauts would have to abandon the station and come back to Earth. The Expedition 15 crew would come home in a Russian Soyuz capsule while the shuttle crew would depart aboard Atlantis.
But that's strictly worst case, Suffredini said, adding he was confident the Russians would resolve the problem in short order.