The shedding of foam insulation from space shuttle external tanks was never considered a safety-of-flight issue, experts told the Columbia Accident Investigation Board this week. But tank engineers have worried for more than 20 years about potentially catastrophic impacts and a top level program requirement held that any shedding of large, potentially dangerous pieces of debris was forbidden.

Even so, despite years of work to eliminate foam shedding, the fleetwide average for foam impact damage resulting in tile blemishes more than one-inch across runs about 30 per mission. So instead of meeting the still existing design goal of no shedding, shuttle program managers somehow came to accept a certain level of impact-related tile damage as a normal occurrence.

In hindsight, the reasoning that allowed NASA to continue launching space shuttles even after a large piece of foam debris broke away from the shuttle Atlantis last October is even more difficult to understand given the critical nature of the shuttle's fragile heat-shield tiles and other insulation. The thermal protection system, or TPS, is one of three systems that have no backup and are not considered fail safe.

"The shuttle was designed with the philosophy that you should not have a system in which you suffer a failure and you lose your vehicle or your crew," said James Halsell, a senior shuttle commander and former launch integration manager at the Kennedy Space Center. "It needs to be fail safe. Furthermore, there was a high operational desire to be fail operational, that is, suffer a failure and still complete the mission.

"The basic requirements are the vehicle and all of its subsystems will be fail safe. From the very beginning, there were three systems which it was acknowledged we could not achieve that desired goal. The thermal protection system was one, it was recognized as being criticality 1, that is, if it doesn't work you're going to lose the vehicle and/or the crew. And we don't have a backup system to it."

Columbia's left wing was struck by a large piece of external tank foam debris 82 seconds after launch Jan. 16. The debris broke away from an aerodynamic ramp in front of the left bipod attachment strut, one of two struts that hold the nose of the orbiter to the tank. The impact is believed to have damaged the wing's leading edge enough to permit a catastrophic plume of super-heated air to burn its way into the wing during re-entry Feb. 1, triggering the shuttle's destruction.

Investigators now believe the fatal breach occurred just outboard of leading edge panel No. 6 - the panel in the center of the foam debris impact "footprint" - somewhere at or near panels 7 through 9. Ongoing wind tunnel testing shows unusual temperature increases seen on Columbia's aft left rocket pod can be explained, for example, by a breach at or near panel 9. Whether the breach occurred because of damaged leading edge panel or because of damage to a so-called carrier panel is not yet clear.

In any case, during a flight by Atlantis late last year - mission STS-112 - a large piece of foam broke away from the bipod ramp area and hit one of the shuttle's solid-fuel boosters. That impact did not cause any major damage, but NASA managers ordered engineers to explore redesign options to minimize or eliminate such bipod ramp foam shedding. But they did not declare the shedding an "in-flight anomaly," or IFA, with a constraint to flight. In other words, the shuttle program could continue flying while the redesign was implemented.

The reasoning behind that decision was that foam shedding did not represent a "safety of flight" concern. It was viewed as a maintenance issue only, i.e., the impact could cause re-entry heating damage requiring tile repairs after a flight that could delay processing for a shuttle's next mission.

How that conclusion might have evolved into a mindset is the subject of intense scrutiny by the Columbia Accident Investigation Board. For board member Sally Ride, the first American woman in space and a member of the presidential commission that investigated the 1986 Challenger disaster, the nature of that evolution is of more than passing interest.

"We're actually trying to understand that," she said Tuesday at a post-hearing news conference. "There is the guiding principle that nothing should hit the orbiter. And it's written in the documentation. The other thing that's written in the documentation is that the TPS (thermal protection system) is criticality 1. That's on the one side and on the other side is that through the history of the hundred flights or so, there's been debris impacting the orbiter on virtually every flight.

"I can tell you, tile damage was a very big concern in the early stages of the shuttle program, a huge concern," she said. "Everyone was worried about the main engines on ascent and the tile on re-entry. Those were the two leading issues in the early days of the program. The early flights came back with some tile damage, but in almost every case it was a turn-around issue.

"As time went on, people, I think, got used to tile damage from debris off the external tank, got used to repairing that between flights, got used to thinking of it as a turnaround issue and I think we saw that in the way that problems were handled after STS-87, STS-112 and during 107. And that is just the sort of thing that we're trying to get at.

"Of course, during the Rogers Commission, one of the things that came out early on was that the (solid-fuel booster) O-rings (blamed for the mishap) were not a problem for the first time on that flight, on 51L (Challenger). They'd been a problem on not just one, not just two, not just three, but several shuttle flights before the Challenger accident. It was almost the case that if you, you know, the famous discussion of Richard Feynman, that you survived it the first time so suddenly it becomes more normal. And it happened enough and now it's a normal occurrence. I think that we're trying to understand whether that same thinking crept in with the foam off the tank."

Richard Blomberg, president of Dunlap and Associates Inc. and former chairman of NASA's independent Aeropace Safety and Advisory Panel, said in the end, the Columbia disaster will be blamed on human error or oversight. But he said NASA could not be faulted for not trying as hard as humanly possible to prevent such oversights from happening.

"I'm a human factors person and I'm the first one to tell you humans are perhaps the mosty fallable part of any system," he told the CAIB. "We design the systems, we operate the systems, we make the decisions to go. So somewhere in whatever you're going to find for Columbia, humans failed. But the question I would want asked is did we fail through malice, did we fail through neglect or did we fail through ignorance? If we failed through ignorance, let's learn from it, let's increase our vigilence and make the system better and keep that closed loop going. That's all we can do with any vehicle.

"I am sure that whatever caused the accident escaped a process at some point," he added. "It had to have, because it flew. So at some point in the process, somebody missed it and it may have been my panel. We may have been staring it in the face and missed it. But it wasn't for lack of trying, I'm convinced, on the part of all concerned because as I said in my opening remarks, I just have never seen a system more safety conscious and people more dedicated to safety. That's not a hundred percent assurance, it just says their heart is in the right place."

For their part, tank experts have known foam shedding was a potentially serious problem for years. And they have been aware all along that the tank isn't supposed to shed any foam in the first place. But they, like everyone else, grew to accept the reality of foam impacts as an unavoidable aspect of flight.

"The program level requirement is that we shall release no debris that is harmful to the orbiter," said Lee Foster, a senior engineer at the Marshall Space Flight Center with extensive experience in external tank issues. "So it's a very subjective thing and while we have been working hand and glove with the system over the years, you know, we've worked with them on debris teams and all that, again, everything was judged as a maintenance item and not a safety of flight issue. I'm not going to say that was right or wrong in the past, but that's the way it happened."

In a hearing Monday, Air Force Maj. Gen. Kenneth Hess asked Foster and Scott Sparks, a senior tank engineer at Marshall, "did you ever think that it was possible to pop a big enough piece of foam of this external tank to severely damage the shuttle itself?"

"The answer is yes," Foster said. "We have large areas where we have closeout materials that we know are hard to spray. So yeah, we are always worried there's going to be a big piece that comes out that would throw us over that maintenance item line."

Said Sparks: "I agree with Lee, we watch (ascent) very closely. Because we know that material could come off and cause some damage. We understand that's a potential and we understand it does require a lot of focus on that material to make it not do that."

So why, in the wake of mission STS-112, didn't NASA managers view the bipod ramp foam shedding as a safety of flight issue?

"It looks to me like something hitting the thermal protection system or damaging the thermal protection system, it's a crit 1 system and therefore damaging, anything that hits the TPS ought to be an IFA, looks to me like," said CAIB chairman Harold Gehman. ""How in the world ... does the system determine there's no safety of flight issue?"

Halsell, who until recently oversaw launch processing and provided the final "go" for flight for the agency's mission management team, said engineers and managers believed the STS-107 foam shedding event was a random occurrence and not evidence of a generic problem.

"The nature of the rationale that was presented in that forum was that the external tank (project) had gone back, even at that point in time, before they had responded to the follow on action, and they had vigorously tried to understand did we do something different with the tank where we had this problem as compared to all the other tanks that had flown successfully?

"And what came out of that was they felt comfortable there was no generic new issue that they could identify ... that they had no generic issue that indicted follow on future tanks that we were going to go fly. ... There was no elevated level of concern that anything liberated from that location would have impacted the orbiter. And what all this added up to was the conclusion that we had not moved up and to the right on the risk matrix with respect to the previously accepted hazard."

Sparks and Foster testified that despite more than two decades of work to eliminate foam shedding, the shuttle's external tank continues to release enough debris during every ascent to cause an average of 30 or so "hits" on the shuttle's belly that cause blemishes larger than one inch across. In at least five cases, foam broke away from the bipod ramp area.

The majority of these shedding events occur in the intertank region of the external fuel tank, the section that separates the ET's liquid oxygen and hydrogen tanks. The intertank features ribbed structural members, or stringers, and other components used to attach the solid-fuel boosters and the bipod assembly struts that hold the nose of the orbiter in place.

The ribbing, bolts in the flange area where the top of the hydrogen tank connects to the intertank, the bipod components and other components must be insulated by hand with sprayed-on foam. How that foam is applied and how it bonds with the complex geometry of the underlying structure is believed to play a role in debris shedding. Air inside voids in the foam, for example, can liquefy once the tank is loaded with supercold propellants. During launch, such liquid air can warm up and explosively evaporate, blowing off pieces of insulation.

"You've got a foam that has to be applied over a certain type of underlying structure and making that so that it is free from shedding seems to be, over the last 20 years, a tough thing to do," observed board member Scott Hubbard.

"Yes sir. And generally, you've really got to go back to the beginning as far as the design of the tank," said Sparks. "I'm not so sure the TPS (thermal protection system) processors were in the same room when they designed the tank.

"Because it was designed structurally to be optimized. It's not designed for the TPS to be processed on there. If you were to redesign completely a tank, you would make the exterior a bit smoother, you know, you'd have those people in the same room."

In the meantime, despite ongoing work to eliminate shedding, the current impact average evolved into a baseline of sorts that is accepted as a routine occurance.

"This is a very impressive list of all the things that have been done over the past 22 years to address the shedding of external tank debris," Hubbard said. "Nevertheless ... the line is pretty much a flat line there, whether it's 10 or 15 or 20 or whatever. So do you see any way to drive that line down to zero or near zero?"

"We're always trying to improve the product," Sparks said. "But we don't want to change the product unless we're justifiably sure that's going to improve the product. ... There have been several improvements that I think the program has been proactive in pursuing. But indeed, there's still a level (of debris) and generally they're coming from those closeouts in that intertank region that seem to be problematic. So we try to improve our processing to the extent possible but thus far, it's staying in that (average) range."

Hubbard followed up, saying "I guess if I had a problem that in over 20 years, the average stayed essentially constant, it seems to me that that might argue something about the basic chemistry, or basic properties of the thing you're dealing with, the foam itself. Do you see the foam as being difficult to control in a very precise manner?"

"No, I don't, Mr. Hubbard," Sparks replied. "Really, what I'm seeing ... is that it's an issue of trying to process that material the best you can. If I had to take a guesstimate as far as the location where we're shedding the most debris, it would be in that hydrogen-intertank-flange area. That's just a hard area to close out, there's a lot of bolts there and when you're spraying that material a lot of potential for shadowing of that foam and possibly having some voids behind that. We've always attributed that to the reason why we're losing some of that material from that area."

As for STS-112, Sparks said "the position was that it was a random occurrence of faulty processing and that nothing had changed in this system to indicate that was a systemic issue as far as processing or material, they had gone and done their homework as far as that goes."

In earlier testimony, Halsell strongly defended NASA's post-Challenger management practices, saying problems are thoroughly discussed before each flight and that engineers are encouraged to weigh in with contrary opinions or to challenge assumptions.

"I know that after Challenger it was recognized that these processes were not as disciplined and rigorous as they should be and what I hope to tell you today, what I hope comes out, is that following the Challenger disaster we went back and did rigorously enforce that discipline," he said. "And the degree to which we fell short in the Columbia accident, that's why we're here today and that's what we want to find out."

Halsell defended NASA's waiver process, whereby components can be cleared for flight even if they don't meet specifications. A study was conducted in 2000, he said, to find out "how many are out there, are they all still valid, how often do we review this situation so that we're not guilty of unknowingly accumulating waivers, to what degree are we confident that we have good rationale for retaining waivers in place? And what we found out from that review is that we do have a good process in place. There's an annual review of the waivers to make sure it's still appropriate, it's still applicable, it's still necessary."

Halsell also defended NASA's flight readiness approval process. Board member Steven Wallace pointed out that by the time a mission's flight readiness review is held a few weeks before launch, most major issues already have been addressed by lower-level engineers and managers.

"This is sort of a recurring message, the work is kind of done before these meetings," Wallace said. "I'm curious, is it fair to say these meetings then don't get scheduled until the work is done, or is it unusual that things get stopped at these meetings because you know, does the meeting become sort of a sign off formality?"

Halsell said when he first moved into management, "I perceived some of the same flavor that you're talking about, that is, the important work was being done and being done exceptionally well, so well in fact that when we got to some of these milestone reviews, it appeared to me that all of the hard issues had been discussed, all of the hard decisions and tradeoffs had been made. So I questioned the value to our senior management of these level of reviews.

"But after being in the job for a longer period of time and having discussed the situation with a number of my project managers, they had a different point of view. They didn't disagree with the fact that the way we do business is such that most of these problems - not always - but most of them had been flattened out by the time of the formal review.

But it's because of the presence of these formal reviews and the fact that you know that senior NASA management, the people you answer to and the people who are ultimately responsible for the safety of the upcoming mission, because you know they're going to be there to hear that story, it drives all that outstanding work that happens before. So from the point of view of the projects and elements, they did not want to change or consider any dramatic changes to the form or to the agenda of any of these reviews because from their perspective, they were driving the kind of reaction within the system that was healthy and needed."

Halsell also went to great lengths to defend the analysis following the foam impact during the launch of Atlantis on mission STS-112.

"It's well known that we did liberate a piece of foam on STS-112," he said. "The process by which we went through understanding what had happened, how that related to our previously accepted hazards ... and what was the appropriate course of action from that point on, all followed the processes we had in place to try to ensure the right decision and right tradeoffs and risks got made.

"For example, the in flight anomaly situation for STS-112, that did come to a program requirements change board, it was decided there that an in-flight anomaly designation was not required for this particular item because the previously accepted and documented hazards, and if I remember correctly there were two integrated hazards which were violated, or which were called into question by this particular instance, two of them dealing with the external tank liberating foam and creating a hazard to some other vehicle component, there was nothing about that particular instance which invalidated the rationale for the previously accepted risk.

"In other words, we didn't move up into the right on the risk matrix according to what we knew at that point in time," Halsell said. "So the action that was levied at that program requirements change board was to the external tank project, to go back and fully understand what had happened, why it had happened and what we were going to do to keep it from happening in the future. Also another action was levied to bring that item forward at the flight readiness review to make sure it was discussed fully prior to STS-113. So using that as my example, I would say that that's an example of how the process worked properly and the item was brought forward to the flight readiness review and it was discussed at some considerable length there."

Asked who had the ultimate responsibility for accepting or rejecting such analyses, Halsell said "the short answer is that it's the space shuttle program manager's job to organize the appropriate response to any and all issues when it comes to making the final determination if we can recommend to the associate administrator that we're ready to go fly safely. So if Ron Dittemore was sitting here in front of me, he would say it's my in box because he's the one who controls the resources and the application of those resources."