Spirit upgraded to 'serious' condition
Posted: January 24, 2004

With NASA's Opportunity rover on track for an early Sunday "bouncedown" on Mars, engineers said today they have identified the problem crippling the Spirit rover and hope to resume relatively normal science operations in three weeks or so.

Pete Theisinger, manager of the Mars Exploration Rover project. Photo: NASA/Bill Ingalls
"We made good progress overnight and the rover has been upgraded from critical to serious," said project manager Pete Theisinger. "We have a working hypothesis we are pursuing that is consistent with many of the observables and consistent with operations that we performed on the vehicle last night. It involves the flash memory on the vehicle and the software used to communicate with that memory."

Engineers do not yet know whether the problem involves the flash memory, which works much like the memory cards used in digital cameras, or the computer software that controls it. In the meantime, Spirit has been successfully instructed to bypass its flash memory, thereby eliminating the problem that was causing the computer to constantly reset itself, and the rover is now in a healthy, safe and commandable condition.

"This is a very good news," Theisinger said. "We've established an ability to communicate with the vehicle reliably, we've established that in fact we do have controllability of the vehicle, can establish a good power and thermal state, our working hypothesis is one that we can work around with significant measure if it turns out our working hypothesis is correct. So a good day for an Opportunity landing."

Opportunity is scheduled to slam into the atmosphere of Mars just before midnight and bounce to an airbag-cushioned landing at Meridiani Planum at 12:05:37 a.m. Sunday. Opportunity's navigation has been dead on and two final course corrections were deemed unnecessary. The rover is expected to touch down in a landing ellipse measuring 46 miles long by 4 miles wide - a bull's-eye considering the 283 million miles Opportunity has traveled since launch July 7.

Like Spirit, the Opportunity rover will search for geologic evidence showing how long water might have existed on the surface of the red planet in the distant past and whether life ever had a chance to evolve.

Spirit was sent to Gusev Crater, a 100-mile-wide impact basin where scientists believe a vast lake once existed. Opportunity will attack the problem from a different angle, landing in a region where orbital instruments have detected hematite, a mineral that typically forms in the presence of water.

Opportunity's entry, descent and landing will mirror Spirit's "six minutes of hell" - the same high-speed plunge into the atmosphere followed by deployment of a braking parachute at about 1,000 mph, retro rocket ignition just above the surface and a final free fall to a beach ball-style bouncing impact. Telemetry from the lander will be sent to Earth via a low-data-rate transmissions from the lander itself and from a UHF relay through the Mars Global Surveyor.

Here's the timeline:


10:34:46 p.m...Turn to entry orientation
11:44:46 p.m...Cruise stage separation
11:59:46 p.m...Atmospheric entry begins
				(altitude: 79.6 miles)
12:01:26 a.m...Peak heating (26.5 miles)
12:01:46 a.m...Maximum deceleration
				(6.3 Gs; 21.7 miles)
12:03:49 a.m...Parachute deployment (5.5 miles)
12:04:19 a.m...Lander/backshell separation
				(3.6 miles)
12:04:24 a.m...Radar altimeter turns on
				(3.4 miles)
12:05:07 a.m...Landing imager activated
				(1.2 miles)
12:05:31 a.m...Airbag inflation (518 feet)
12:05:31 a.m...Retro rocket ignition
				(397 feet)
12:05:34 a.m...Bridle cut; lander begins
				free fall (39 feet)
12:05:37 a.m...LANDING
12:13:42 a.m...Mars Global Surveyor sets
12:15:34 a.m...Roll stop; UHF radio off
12:19:36 a.m...Begin rover low-gain antenna
				tones (2.5 minutes)
12:25:08 a.m...Begin landing petal low-gain
				antenna tones (2.5 minutes)
12:27:53 a.m...Begin airbag retraction
01:13:28 a.m...Begin petal deployment
01:21:04 a.m...Realtime Global Surveyor
				landing data relay
01:25:59 a.m...Petal deployment complete
				(if on base petal)
01:26:25 a.m...Deploy primary solar array
01:26:43 a.m...Deploy secondary solar array
01:38:01 a.m...Begin deployment of
				pancam mast assembly
02:25:59 a.m...Eart sets
03:17:17 a.m...Last normal critical deploy
				completion time
03:38:46 a.m...Mars Odyssey rises
03:55:03 a.m...Mars Odyssey sets
03:57:42 a.m...Rover shutdown
04:05:00 a.m...Playback of Mars Odyssey
				data (possible pictures)
05:04:04 a.m...Sunset

In Spirit's descent, engineers were amazed to receive data almost continuously, from atmospheric entry to rollout on the surface, but that was as much a matter of luck and the lander's orientation as anything else. Engineers cautioned reporters today not to expect such lucky circumstances for Opportunity's landing. Depending on how the folded-up lander comes to rest - and whether one of its two antennas can "see" Earth - it's possible flight controllers won't hear from the spacecraft for nearly a full day.

But engineers are keeping their fingers crossed. If the lander unfolds normally and if Opportunity's solar panels deploy as planned, it's possible the first pictures from Meridiani could be beamed back to Earth via the Mars Odyssey orbiter beginning shortly after 5 a.m.

Flight controllers do not yet know whether the problem bedeviling Spirit is a random failure or a generic issue that could affect Opportunity. But Theisinger said it won't have any impact on this evening's landing.

"We will take whatever we learn from the Spirit diagnostics and apply it to Opportunity," he said. "But that is for the surface software. There is no real urgency to do that. ... The kinds of things that we were doing early on impact-to-egress - we would not anticipate a problem. Certainly it is not an issue with respect to EDL (entry, descent and landing).

"So we do have time to post-landing on Opportunity to find out where we are with Spirit, determine whether in face we have a lessons learned case or not, and then decide how to play that with Opportunity," he said. "I would expect that if we did have a lessons learned case, we would probably initially establish some operating rules for 'this is how you use it' which might limit what science could do in volume or other things until we could work out a software fix that we could upload. We have the capability to patch the software on the surface - that would not be an issue."

Spirit went on the blink Wednesday, suddenly acting erratically and refusing to transmit stored science and engineering data. Engineers later determined the rover's computer was constantly rebooting itself because of a problem that initially eluded investigators. But after studying limited telemetry from the rover, troubleshooters identified a problem with the computer's use of flash RAM.

The rovers are equipped with three types of memory: 128 megabytes of random access memory, or RAM, which holds real-time data that is wiped out when power is lost; 256 megabytes of flash RAM, which holds data with or without power; and programmable memory, which is used to store critical flight software.

"Flash RAM is just like the memory in a digital camera," Theisinger said. "It can also be read to and written from easily. But it has non-volatile characteristics - the information that is stored there stays overnight even if the vehicle is powered down.

"The vehicle normally uses the flash memory mostly for the storage and retrieval of engineering and the scientific telemetry. The software has to communicate with the flash memory - has to open up files, establish file directories, close files in that flash memory. That process has to work correctly.

"We are capable of operating the vehicle without going to the flash memory in what we call a 'cripple mode.' That is name we have chosen - you should not read too much into that. That basically tells the flight software that when it boots up, it should operate with its file directory out of the random access memory rather than the flash memory. That would avoid any issues that we might have with either the flash memory itself or the flight software that is used to write to it."

Theisinger provided a detailed description of how the engineering team zeroed in on the problem. On Thursday, he said, "we were attempting to shut down the vehicle because the batteries were becoming depleted. We had an apparent inability to shut down the vehicle last night, yesterday at the end of the Mars day, which is about the time of the press conference at 9 a.m. Pacific. That was, in fact, confirmed because later we had a UHF communications session with Odyssey where we got 73 megabits of data, mostly fill or garbage data, although we did get some fault data, some current, some 14 hours old.

"We did not know, but thought we might go into low-power overnight if the batteries were fully depleted. When it came up this morning, we looked for the 9:30 Mars time communications window and it was not there, indicating, we thought, that we had gone into low-power. That would cause the vehicle to come up at 11 o'clock and tell us that.

"We, just prior to 11 o'clock, sent a command to the vehicle that said 'go into this cripple mode.' That is only done at the next reset, so then we sent a command to the vehicle that said 'and reset' in order to do that. And we timed it so that when the 11 o'clock session would start, we would begin to get that session at 10 bits per second indicating we'd gone into low-power. When the commands reached the spacecraft light-time away, it would send us into cripple mode, reset the computer and we would come up in the mode. And in fact that is precisely what happened."

At that point, flight controllers sent commands ordering Spirit to carry out a one-hour communications session, which went off exactly as planned.

"The progressive set of resets that we were getting into every hour did not reoccur, leading us to conclude that our hypothesis is largely correct, that is there is something involved in the flight software that talks to the flash memory that's causing this difficulty," Theisinger said.

"Why that might cause us difficulty is because when the spacecraft first wakes up, it needs to communicate with the flash memory to establish a file structure and when it goes to sleep and shuts down, just like you shut down your computer at home, it needs to go out to that memory and close all of those files and clean everything up. If it is unable to do that, it would not complete those tasks appropriately and will basically reset itself and not shut down.

"In the midst of that 120-bits-per-second, one-hour session, we decided to shut down the vehicle in order to replenish the batteries. We commanded the shutdown just prior to the end of the communications session so that we would see the communications session terminate early if we were able to operate it correctly. That happened. And we sent two post-shutdown beeps, which we expect not to hear if it is asleep but we would hear if it was not asleep, and we did not get those, once again confirming that the vehicle to the best of our ability to determine is now sleeping on Mars.

"So we have a vehicle that is stable now in power and thermal," Theisinger said. "We have a working hypothesis that we have confirmed. The fault protection to the best of our estimation has worked as designed. It took us a lot to figure out what was going on, but we think everything has worked in the fault protection as we expected it to do."

But Spirit is not yet out of the woods. Flight controllers must develop an efficient way to operate Spirit in cripple mode, without flash memory, until they can determine exactly what happened to cause the problem in the first place. If the flash memory cannot be recovered - and it will take quite a while to figure that out - the team must develop new procedures to operate the entire mission with the RAM memory.

"The mission consequences of this are uncertain at the present time," Theisinger said. "But I think that we feel that we probably have more capability left in the vehicle that we can establish than the worst-case scenarios by quite a bit. We still see a couple of weeks to determine what had happened and to rebuild our confidence into what is working on the vehicle and to get back closer to routine operations. I think we are probably like three weeks away from driving.

"But this is a very good news. We've established an ability to communicate with the vehicle reliably, we've established that in fact we do have controllability of the vehicle, can establish a good power and thermal state, our working hypothesis is one that we can work around with significant measure if it turns out our working hypothesis is correct. So a good day for an Opportunity landing."

Spaceflight Now Plus
Video coverage for subscribers only:



Status quicklook

Check the status center for complete coverage.