Launch on hold for troubleshooting
BY WILLIAM HARWOOD
STORY WRITTEN FOR CBS NEWS "SPACE PLACE" & USED WITH PERMISSION
Posted: July 13, 2005
Shuttle engineers are trying pinpoint why one of four critical hydrogen fuel sensors failed a test late in the shuttle Discovery's countdown Wednesday, forcing NASA managers to scrub the agency's long-awaited return to flight.
"Unexplained anomalies are the worst ones," said deputy shuttle program manager Wayne Hale. "What you'd really like to do is find the problem and fix it. So when you have a problem that kind of comes and goes and you can't put your finger on it, that's a tough issue."
Hale was referring to recent ECO sensor system problems collectively even though Wednesday's test indicated an actual failure, not an intermittent glitch. In any case, launch director Michael Leinbach had no choice. NASA flight rules forbid a launch if all four fuel depletion sensors are not operating normally.
"The vehicle, the ECO sensors, for some reason did not behave today, so
we are going to have to scrub this launch attempt," Leinbach radioed
Discovery's crew at 1:32 p.m. "I appreciate all we have been through
together, but this one is not going to result in a launch attempt today."
Because the box had been disassembled, a unit from the shuttle Atlantis* was installed aboard Discovery for a second tanking test to troubleshoot the ECO sensor issue and other, unrelated problems. This time around, the sensors worked normally but the point sensor box malfunctioned later, during additional troubleshooting to understand what happened during the first test in April.
NASA managers had already decided to haul Discovery off the launch pad and to attach it to a fresh tank and a new set of boosters, in part to ensure that fuel sensors in the original tank were not at fault. Engineers replaced virtually all of the electrical cabling in the ECO sensor system and installed yet another point sensor box, this one from Endeavour.
The Endeavour box, serial number 111, included eight transistors that were known to be from a suspect lot. Engineers did not expect any problems, however, because the unit passed a battery of tests. A third tanking test was not ordered because engineers believed they had replaced everything that could have been at fault.
"We have a new tank that's different from the tank we used in the (April) tanking test, we had changed out every wire, cable, connector, electronics box associated with these sensors. We had done a lot of tests on the boxes and the various components, we felt like we had a good system.
"Today, we had another anomaly. So we're going to sit back and think about what the cause of that problem was and how we're going to rectify it."
The fuel depletion sensors are a critical safeguard against potentially catastrophic failures. Here is a bit more background from a reference manual written by shuttle-builder Rockwell International:
"They're there to protect us in case we run out of gas," Hale said. "Now we don't plan to run out of gas. As a matter of fact, we launch with some fairly comfortable propellant reserves in the external tank to allow for dispersions that might happen within the launch phase. So you have to have something go wrong to really need these sensors. That's the first thing you really need to understand.
"There are four of them and the flight software waits until it gets close to MECO ... close to the time when you are ready to be inserted into orbital velocity, you're almost there, and then starts looking at these sensors. There are four of them and if two of them go dry, that indicates to the computer it's time to shut down the main engines to provide a safety margin on the engines."
To cause the main engines to run the hydrogen tank dry, three of the four ECO sensors would have to fail "wet," that is, send indications they were immersed in cryogenic liquid hydrogen when in fact they were not. Likewise, three of the four would have to fail "dry" to trigger a premature shutdown.
During a test Wednesday to verify the sensor system was working properly aboard Discovery, three sensors responded normally and one appeared stuck on the wet setting. But that's all it took, Hale said, because "we have a very clear and unambiguous criteria that says all four of those sensors must work to provide us the kind of redundancy and reliability that's necessary for safe flight.
"So when one of those indictors started acting up today, we decided it was time to quit."
Asked if NASA might amend the flight rules to get Discovery off the ground, Hale said: "The answer to that is no. Every time we have reviewed the rules, we've come to the same conclusion. They're in place for good reason."
Steve Poulos, a senior shuttle manager at the Johnson Space Center, said engineers planned to re-examine the system from top to bottom and "hopefully by tomorrow (Thursday) evening, we'll have some better insight into what the potential concern is. I wouldn't guarantee at this point we'll know exactly what we have. But we'll certainly have more information tomorrow."
As it turns out, all of the transistors from the suspect lot were on a circuit board used to test the operation of the sensor in question. But Poulos refused to speculate about a possible connection, saying he needed more data before drawing any conclusions.
It took three hours or so to drain the propellants from Discovery's tank Wednesday and Leinbach said it would take another 21 to complete a long "safing" process. If all goes well, the launch pad will be re-opened for normal work around 3 p.m. Thursday.
"We're virtually certain we'll go into the aft of the orbiter and at least look at the box," he said. "Anything beyond that is fairly invasive. If we have to start thinking about getting into the external tank, that's an extremely large job to do and one we've said in the past we'd prefer to do back in the Vehicle Assembly Building. It's not out of the question to go into the aft of the hydrogen tank at the pad, we've never done it at the launch pad and we'd have to take a hard look at that."
Given the troubleshooting so far, Poulos said "there is a potential problem that might exist in the sensor box. We do not think it has (anything to do) with the power supply cards within the box. But there could be a concern in the signal conditioning system itself. And then from there, it could be an open circuit for this particular sensor somewhere beyond the box into the external tank and then ultimately to the sensor itself."