Atlantis to be fueled again for more troubleshooting
BY WILLIAM HARWOOD
STORY WRITTEN FOR CBS NEWS "SPACE PLACE" & USED WITH PERMISSION
Posted: December 11, 2007
Engineers are drawing up plans to load the shuttle Atlantis' external tank with supercold liquid hydrogen next week in a critical test to pinpoint the source of elusive, intermittent electrical problems in low-level fuel sensors that derailed two launch attempts.
Shuttle Program Manager Wayne Hale said today engineers will tap into the engine cutoff - ECO - sensor circuitry near a control unit in the shuttle's aft engine compartment to hook up test instrumentation that should help locate any bad wiring or connectors in the 100 feet or so of cabling between the box and the sensors at the base of the external tank.
The tanking test is tentatively planned for next Tuesday.
"We think that we have some instrumentation we can put in an appropriate place in these circuits to observe what happens as we fill the tank," Hale told reporters during an afternoon teleconference. "If the erroneous condition repeats, which is what we think will happen, we can capture the location in the circuit of that opening with the use of some equipment and a technique known as time domain reflectivity, TDR.
"We have a high degree of confidence of pinpointing the location of where we're having our problems and once we know the location ... we'll be able to concentrate our go-forward efforts, presumably put together a fix and go fly. And again, our go-fly date is no earlier than Jan. 2. It could definitely be a little bit later than that, depending on the troubleshooting and the repair work involved."
The holiday work schedule also might play into the recovery plan.
"I am very concerned about team fatigue," Hale said. "We have worked our teams very hard the last several months and so we are in discussions about taking some days off on or about Christmas day, Christmas Eve, potentially the weekend before, to allow for crew rest.
"When we talk about no earlier than Jan. 2, part of the discussion is not only how quickly we can troubleshoot and fix this problem, but what is prudent to allow our folks to have a few days with their families and rest and recuperate. So I expect we'll be making that decision shortly after the tanking test."
The ECO sensors are part of a backup system intended to make sure the shuttle's main engines don't inadvertently suck a tank dry after some other problem - a leak, for example, or an improper hydrogen-oxygen mixture ratio - used up propellant at faster than normal rates. An engine running out of hydrogen during normal operation would have catastrophic consequences.
Early in the shuttle program, the ECO sensor launch commit criteria required three of four sensors to be operational for a countdown to proceed. That later was changed to a four-of-four requirement because of concern about lack of redundancy in part of the cicuitry. After modifications, the LCC was amended to three of four and that was the rule in place going into Atlantis' initial launch attempt.
The electrical resistance of platinum wires in the ECO sensors changes depending on whether they are exposed to cryogenic temperatures or not. A control unit called a point sensor box in an aft avionics bay monitors that resistance to determine whether a given sensor is wet - submerged in hydrogen - or dry, indicating the tank is nearly empty. That information, in turn, is passed along to the shuttle's flight computers.
Because of past problems with ECO sensors, engineers now send commands to simulate wet or dry conditions during the pre-launch tanking process. It was during a so-called "sim dry" test, when the ECO sensors were, in reality, submerged in hydrogen that problems cropped up for Atlantis.
During fueling for a Dec. 6 launch attempt, ECO sensors Nos. 3 and 4 "failed wet" about 35 minutes after they were submerged in liquid hydrogen at minus 423 degrees Fahrenheit. After the launch was called off, another sensor that shows when the tank is 5 percent full failed wet. After the tank was drained, ECO sensor No. 1 failed wet as well.
As the temperature in the tank increased following the hydrogen off load, all the sensors eventually returned to normal operation.
Because of past problems with the ECO sensors, new instrumentation is in place that lets engineers monitor voltage levels in the circuitry in real time. That provides an independent way to determine whether a sensor is working or not and some engineers suggested relying on that instrumentation to permit a launch even if one ore more sensors failed to work properly the second time around.
In an email to astronaut Bill McArthur, manager of safety and mission assurance for the shuttle program, Hale questioned whether the ECO sensor system has ever been truly reliable.
But NASA's Mission Management Team, acting on a proposal from the crew office, agreed to make a second launch attempt on Dec. 9 only if all four ECO sensors operated normally. During the second fueling, ECO sensor No. 3 failed wet about 24 minutes after submersion in hydrogen. No other problems were detected.
Hale told reporters today any talk of developing operational work arounds for dealing with ECO sensor anomalies was, for the time being at least, off the table.
"We have tabled that discussion," he said. "Our point is to try to fix this system. If we fix it, I would presume we would go back to the previously existing launch commit criteria (requiring three of four ECO sensors). If we come to a place where we're less than completely confident in our fix, then of course we would readdress the launch commit criteria."
Hale said there is about 100 feet of wiring between the point sensor box in the aft avionics bay and the ECO sensors at the base of the tank. The circuitry includes several connectors and wire splices.
Before next week's fueling test, engineers will climb into the aft engine compartment and tap into the wires running from the point sensor box to all for ECO sensors and the 5 percent sensor that failed wet during the first detanking.
"We're going to take five wires and cut them, install jumper cables to lead to the equipment that's outside of the orbiter," Hale said. "We're going to make that wire cut in the orbiter aft compartment close to the point sensor box in the avionics bay. We'll run the jumpers to the TDR equipment, which will be out on the mobile launch pad surface.
"We actually have to have people present to run that equipment. We can watch the instrumentation remotely but to physically switch from one sensor circuit to another on the TDR equipment, people have to be present. So we will send a red crew out during the stable replenish time frame.
"If the problem occurs during the fill as it has in the past, it would remain with us through the stable replenish time frame. We don't have people out on the launch pad during fill and drain operations because of the hazards involved. But we will send a red crew out when we're in the stable replenish mode."
Among candidate problem areas: Broken or damaged wires; a recessed connector pin/socket; contamination in a connector; wire splice damage; and a combination of problems leading to a tolerance "stack up" in a feed-through connector. Engineers are hopeful whatever is wrong can be fixed at the launch pad, avoiding a time-consuming roll back to the Vehicle Assembly Building for more extensive repairs.
In addition to the tanking test, Hale said engineers will subject ECO sensor cables and harnesses to cryogenic conditions in a laboratory setting to collect additional insight into how they behave when the system is cooled to such low temperatures.