SpaceX is replacing two engines on the Falcon 9 booster set to launch four astronauts next month on the company’s first operational Crew Dragon flight to the International Space Station, following an investigation that revealed a subcontractor failed to adequately clean out narrow vent ports on multiple engines across the company’s fleet, officials said Wednesday.
NASA and SpaceX officials said they are working toward a target launch date of Nov. 14 for the Crew Dragon mission, which will take off on top of a Falcon 9 rocket from pad 39A at the Kennedy Space Center in Florida.
The mission, designated Crew-1, will launch NASA commander Mike Hopkins, pilot Victor Glover, mission specialist Shannon Walker, and veteran Japanese astronaut Soichi Noguchi on a half-year expedition on the space station.
Managers delayed the launch of the Crew-1 mission from Oct. 31 to investigate an engine problem that triggered an automatic abort just before liftoff of a different Falcon 9 rocket Oct. 2 with a GPS navigation satellite for the U.S. military. Hans Koenigsmann, SpaceX’s vice president of build and flight reliability, said Wednesday that engineers identified a masking treatment inadvertently left behind in the engine as the cause of the Oct. 2 abort.
Engineers found two of the nine Merlin 1D engines on the Falcon 9 first stage for the Crew-1 mission have the same problem.
“We’re in the process of swapping out two engines on the first stage on the Crew-1 vehicle due to this problem, and that is in work at this time,” said Steve Stich, NASA’s commercial crew program manager, in a conference call with reporters Wednesday.
According to Koenigsmann, the Oct. 2 launch abort at Cape Canaveral occurred immediately after sensors detected an unusual pressure rise in gas generators on two Merlin engines.
“When we looked at the data, we saw that two of the engines attempted to start early, and the auto abort prevented that,” Koenigsmann said. “And by doing that, it prevented a possible hard start that could have been damaging to the engine hardware.”
The Merlin engines power up with help from an igniter fluid known as TEA-TEB — or triethylaluminium-triethylborane — that gives off a bright green flash at the start pf the ignition sequence.
“And then we have liquid oxygen, and we have kerosene, or RP-1 as its called,” Koenigsmann said. “And you need to introduce these liquids in the right order. If you do this in the wrong order, if you happen to throw in the liquid oxygen and the RP-1 and the igniter fluid, then what would happen is we call it a hard start.”
A hard start would “rattle” the engine in most cases, but could cause damage, Koenigsmann said. “So in general, you do not want that. You want a good startup.”
“If we see the pressure rise too early, then we know liquid is in there, and it shouldn’t be there,” he said. “And software in the engine controller then stops the whole process.”
Inspections at the launch pad uncovered no sign of a hardware problem on the Merlin engines after the Oct. 2 abort. After returning the Falcon 9 rocket to its hangar, technicians removed the engines and shipped them to SpaceX’s test facility in McGregor, Texas.
SpaceX was able to analyze the engines’ startup behavior in McGregor, and additional inspections found a blockage in a narrow line leading to a relief valve on the gas generator.
Koenigsmann said the vent port, which measures just one-sixteenth of an inch wide, was obstructed by a hardened masking lacquer. He said liquid lacquer — similar to red nail polish — is used by a third-party vendor that anodizes aluminum engine components for SpaceX.
The lacquer protects certain parts during the anodizing treatment process, but the vendor — which officials did not identify — is supposed remove the material before shipping the components to SpaceX for engine manufacturing.
The gas generator on each Merlin engine drives a turbopump feeding kerosene and liquid oxygen propellants into the main combustion chamber.
“It’s obviously a very important part,” Koenigsmann said. “That little red substance was blocking a relief valve that caused it to (ramp up pressure) a little bit earlier than it was supposed to.”
“Probably during the washing or the cleaning process some of that masking lacquer went into this vent hole and blocked it,” Koenigsmann said.
“How was this missed? The little borehole that goes to the vent valve is tiny. It’s 60 thousandths of an inch … When something goes in there and is hung in there, it’s possible you miss it,” Koenigsmann said. “You shouldn’t … but I can see how people overlook that and didn’t quite see that, even if it is red.”
Engineers at SpaceX’s McGregor test site also demonstrated that the engines performed normally after removing the blockage from the vent valve. Koenigsmann said the issue was “very subtle, but can have obviously some negative impact on the engine operation.”
“No question that rocketry is tough and requires a lot of attention to detail,” Koenigsmann said. “To me, this almost tells me that rockets are humbling me every day I work with them. It’s always a challenge, and it’s always difficult, and you have to be super diligent and on your toes to get this right.”
SpaceX engineers had not detected the problem before. Older boosters in SpaceX’s fleet of reusable Falcon 9 rockets seem to not exhibit early ignition data signature, but the rockets for the GPS and Crew-1 missions are brand new.
“What we’ve noticed is this is something that must have happened within the last couple of months, so it affects primarily engine build dates that are on new boosters and not on old boosters,” Koenigsmann said.
But the issue doesn’t affect every new engine on SpaceX’s rockets.
“There’s certainly the possibility that we had cases of this earlier, and they were basically so harmless that we completely missed them,” Koenigsmann said. “It’s also possible that little things changed. This is a process that’s done out of house at a special vendor, so it could be that person is now more generous with cleaning fluids or other things.”
He said engineers looking at data from previous engine testing “found similar early tendencies” on two engines on the Crew-1 booster, two engines on the rocket for the GPS mission, and one engine on another new Falcon 9 rocket scheduled for launch with a NASA-European oceanography satellite Nov. 10 from Vandenberg Air Force Base, California.
The two new engines being installed on the Crew-1 rocket have clean vent holes, Koenigsmann said.
Josh Finch, a NASA spokesperson, confirmed both engines being replaced on the Crew-1 rocket are on the first stage.
SpaceX is beefing up inspections on engine parts before assembling the Merlin powerplants at the company’s headquarters in Hawthorne, California.
“We have tons of valves that we build,” Koenigsmann said. “We have tons of regulators, and they have little vent paths inside the hardware, so we’re now looking at all of this and making sure that there’s no leftovers — all the valves are clean and good. We did a lot of that already in testing. This triggers another effort to really look at every passage and make sure that things are clear when you bolt them together.”
Koenigsmann said SpaceX is working with the vendor that uses the masking lacquer to correct the issue on future engine parts.
“At the end of the day, you improve your process, you improve your vigilance, and you have a better rocket on the launch pad for the next couple of launches.”
Since discovering the engine problem Oct. 2, SpaceX has successfully launched three missions with 180 of the company’s own Starlink internet satellites. None of the engines on those rockets — which were all reused — showed the early start data signature, so SpaceX elected to proceed with the Starlink missions.
“It’s an advantage when you fly a booster and it worked fine, then you know it’s flight-proven,” Koenigsmann said.
The nine Merlin 1D engines on the Falcon 9 rocket’s first stage each generate 190,000 pounds of thrust. The engines reignite in flight to slow the booster down a propulsive landing on SpaceX’s drone ship at sea, or at an onshore landing facility near the launch site.
A single Merlin Vacuum engine powers the Falcon 9’s upper stage into orbit.
NASA said Wednesday that efforts to resolve the issue with the Merlin engines should pave the way for launch of the Crew-1 mission Nov. 14 at 7:49 p.m. EST (0049 GMT on Nov. 15). If the launch happens then, the Crew-1 astronauts are scheduled to dock with the space station at 4:04 a.m. EST (0904 GMT) on Nov. 15, less than eight-and-a-half hours after liftoff.
Hopkins and his crewmates are expected to live and work on the space station for around six months. They will join NASA station commander Sergey Ryzhikov, Russian flight engineer Sergey Kud-Sverchkov, and NASA astronaut Kate Rubins on the international research outpost, raising the lab’s long-duration crew to seven people for the first time.
The first operational Crew Dragon flight follows a 64-day Crew Dragon demonstration mission with NASA astronauts Doug Hurley and Bob Behnken. Hurley and Behnken launched to the space station May 30 and returned to Earth on Aug. 4 with a splashdown in the Gulf of Mexico, marking the first flight of astronauts into orbit from a U.S. spaceport since the retirement of the space shuttle in 2011.
With the test flights now in the books, SpaceX’s Crew Dragon is set to begin a series of at least six regular crew rotation flights to the space station, ending NASA’s sole reliance on Russian Soyuz missions for crew transportation.
Steve Stich, NASA’s commercial crew program manager, said Wednesday that space agency managers have reviewed all the data from the Demo-2 test flight and cleared two issues detected on the Crew Dragon spacecraft after splashdown.
Inspections of the Demo-2 spacecraft, named Endeavour, revealed part of the capsule’s heat shield eroded more than expected during its scorching re-entry into the atmosphere. SpaceX and NASA officials said last month that teams noticed the erosion in an an area at the bottom of the capsule, where it connects to an unpressurized trunk that is jettisoned just before re-entry.
“There are some bolts that protrude out once that trunk separates on orbit and the vehicle comes back in,” Stich said in a Sept. 29 press conference. “So that area around these tension ties is a very complex flow field. Basically, on Demo-2, what happened is in this complex flow field there was some erosion in between tiles.’
Koenigsmann said the heat shield erosion was limited to a “small area,” and was “nothing to be concerned about.”
NASA officials said SpaceX engineers were introducing “some very localized upgrades” of the heat shield in the attach points between the Crew Dragon capsule and trunk.
The other issue noted on the Demo-2 mission’s descent involved an altitude sensor that tells the capsule’s parachutes when to deploy. The parachutes on the Demo-2 mission deployed at a slightly lower altitude than expected, but still within design parameters, officials said.
For the Crew-1 mission, SpaceX has changed an instrument that measures barometric pressure to determine altitude.
Stich said Thursday that a “clogged area” in vents in the Crew Dragon’s nose cone led to the early parachute deploy on the Demo-2 mission. He said NASA has cleared the heat shield and parachute deployment concerns leading up to the Crew-1 launch, pending final readiness reviews.
“We have, side-by-side with SpaceX, reviewed all the data from Demo-2 mission, all the (minor issues) and we dispositioned all those, including … the heat shield erosion around the tension ties, we dispositioned that. The clogged area in the vents up in the nose cone that led to the early drogue (parachute) deploy, we dispositioned that. So we’ve gone through all the Demo-2 data.”
Other changes to the spacecraft for the Crew-1 mission, named Resilience, include sturdier structural panels to handle higher winds and rougher seas at splashdown, upgraded solar arrays, and new flight software to enable the ship to dock with two different docking ports on the space station.
With the new target liftoff date of Nov. 14, the Crew-1 astronauts entered a “soft quarantine” last weekend to limit their health risks leading up to the launch. They will start NASA’s formal pre-launch quarantine protocol Saturday.
SpaceX plans to convene the company’s internal Crew-1 flight readiness review Thursday, followed by a flight readiness review led by NASA’s commercial crew program Friday.
Those reviews will give the go-ahead to load hydrazine and nitrogen tetroxide propellants into the Crew Dragon spacecraft at Cape Canaveral Air Force Station. Next week, the capsule will be transported to nearby Kennedy Space Center, where SpaceX ground teams will connect the spaceship with its Falcon 9 launcher.
The Crew-1 astronauts are scheduled to fly from their home base in Houston to the Florida spaceport Nov. 6.
SpaceX plans to perform a test-firing of the fully-assembled Falcon 9 rocket and Crew Dragon spacecraft Nov. 9 on pad 39A. The static fire test is a standard part of SpaceX launch campaigns, but it will give engineers another chance to verify the engines on the rocket are free of the defect that aborted the GPS launch attempt Oct. 2.
A broader flight readiness review involving officials from across NASA is planned for Nov. 9.
On Nov. 11, the Crew-1 astronauts will put on their SpaceX-made pressure suits and board the Crew Dragon spacecraft on pad 39A for a countdown rehearsal.
If all the pre-launch activities go according to plan, SpaceX and NASA will clear the Falcon 9 rocket and Crew Dragon spacecraft for blastoff Nov. 14.
Meanwhile, SpaceX is expected to launch at least one mission — and perhaps more — before the Crew-1 flight.
Another launch attempt for the military’s next GPS 3-series navigation satellite is planned as soon as Nov. 4 from pad 40 at Cape Canaveral Air Force Station. Stich said NASA would like to evaluate data from the GPS launch before the crew mission.
“One of the engines that we’re installing has a slight change that we would like to see fly on the GPS 3 mission first,” Stich said Wednesday.
The launch of the U.S.-European Sentinel-6 Michael Freilich oceanography satellite is currently scheduled no earlier than Nov. 10 from Vandenberg Air Force Base, California. Like the rockets for the GPS and Crew-1 missions, the Falcon 9 booster for the Sentinel-6 launch is undergoing an engine swap.
Email the author.
Follow Stephen Clark on Twitter: @StephenClark1.
