After Hours

IT at NASA: How the support crew keeps them flying

Matthew Bordelon is an IT manager who supports end users who may be only a few hundred miles away, but they're orbiting the Earth at 17,000 mph. You can pick up a few tips from this NASA IT pro on how he keeps these very remote users flying high.

IT support staffers may sometimes feel they have to boldly go where no man has ever gone before, but they are not alone out there.

At NASA’s Johnson Space Center in Houston, for example, project manager Matthew Bordelon typically finds himself supporting the needs of people in remote locations—very remote locations.

Bordelon and his staff support space shuttle astronauts. By the end of the year, they expect to be supporting astronauts and cosmonauts on the International Space Station. So even though these remote users are sometimes only about 132 to 383 miles from Houston, that distance is straight up—and there is no quick and easy way to get there for on-site help.

In this article, we’ll focus on some of the issues NASA faces in supporting its high-flying “remote” users, and how you can adopt some of their solutions for your users on the ground. At NASA, they:
  • Provide 24/7 manned support.
  • Mirror end-user machines in support.
  • Train end-users in troubleshooting procedures.
  • Try to foresee problems and engineer solutions.

Up, up, and away
There is a huge amount of support for astronauts when they are rocketed into space, and IT support for the IBM ThinkPads the astronauts carry with them is only a small portion of that effort, Bordelon said.

During a typical two-week shuttle mission, the IT support is 24/7 for the crew and their spacecraft. During a flight, the front room of mission control is crowded with people who monitor the major systems on the shuttle.

“Then there are back rooms that provide all the data feeds to the front room guys to keep track of the system,” said Bordelon.

In the front room, there are two console positions that interact with the IT department. One is the payloads officer, who keeps track of all the payloads onboard. The other is the flight activities officer, who plans all the operations and manages all the power and time resources on the shuttle.

Isolated in space, every bit of energy and manpower on the spacecraft is orchestrated and scheduled, Bordelon said.
This is the first of a two-part series looking at support issues of NASA’s most remote users. Next week, we’ll focus on procurement and support issues and look into the future of personal computers on the U.S.space shuttle and the International Space Station.
The payloads and flight activities officers control the IT staff’s link with the space shuttle. Bordelon and his staff are there to troubleshoot problems and answer questions every moment of the flight.

“We have tried to squash all those bugs before they got up there,” Bordelon said.

Everything the astronauts have on their computers in space is duplicated by the IT staff on their Earth-bound twin systems. To be certified for flight, the computers have to be identical, from their BIOS to their operating system. Different programs may be on the computers, depending on the mission, but all of that is consistent with the machines IT support has on the ground.

There is a procedure for troubleshooting the problem, and as a last resort, software can be reinstalled in-flight.

One advantage NASA may have over your typical end user group is that astronauts get their computers when they are selected for a flight, about a year before they leave the planet. Reporting errors or problems is part of the standard malfunction procedures the astronauts learn during training.

Still, engineers are always trying to find ways to minimize the role of the end user in computer maintenance.

“We are getting to the point where we can remotely administer it from here on the ground,” Bordelon said.

It’s like a big flying Palm Pilot
Remote administration is not an uncommon form of network administration here on the big, blue planet, but then, most remote users aren’t circling Earth at 17,000 mph.

Meanwhile, the astronauts are happily working on projects, collecting data, and writing home to their spouses and kids via Microsoft Outlook, just like many other network users.

When it comes time to send their e-mail or exchange data with mission control, the astronauts depend on a constellation of satellites they fly by.

“It is not really as if they were connected to the Internet or anything, but we can uplink and downlink files because we have a network setup there,” Bordelon said. “We can make some changes that way.”

That uplink is scheduled based on the shuttle’s position in relation to the satellites.

“It is kind of like having a (personal digital assistant), basically, and every once in a while you link back up to your computer and it sends out all of the e-mail that you have gathered up until then and reloads you with more stuff,” Bordelon said. “In one sense of the word, we are using them sort of like PDAs are used with PCs.”

Testing and customization, keep your feet on the ground
You just don’t throw any old computer and peripherals up into space and expect them to work. At NASA, all equipment goes through a certification process, but because of budget constraints, much of the hardware and software that goes into space is purchased off the shelf, Bordelon said.

There are a few interesting differentiations from the ordinary.

“As far as safety is concerned, the hardware is coated to reduce flammability,” he said. The computers also have “locking power connectors so you do not accidentally unplug the thing.”

Gravity, or the lack of gravity, can cause a few problems for mechanisms that depend on gravity, such as most tape drives.

With a four-year purchasing cycle, the astronauts are using 760xe IBM ThinkPads that are too old to have built-in 10Base-T connectors, so the IT staff have had to build a “strain relief” device on the side for everything that hangs off the PCMCIA cards.

Another unique problem involves a phenomenon called the South Atlantic Anomaly, radiation bursts, which affect both astronauts and computers.

Astronauts report that when a spacecraft flies through the anomaly, they can close their eyes and see little particles penetrating their eyelid, hitting the optic nerve to produce bright flashes of light.

In a computer, these streams of radiation can cause “single event upsets” where the radiation will flip bits in a memory cell. Most laptops don’t come with error-correcting RAM memory.

“What we have to do in the process of certifying a laptop for space flight is take it to a facility where we can do radiation tests and aim a beam at it to see how susceptible the memory and processor might be to radiation hits, or these single event upsets,” Bordelon said.

It’s a NASA thing, testing before distributing the hardware and software. If it’s good enough for astronauts, maybe it’s good enough for your end users.
Next week, Bordelon outlines how you buy, set up, and support computers for 2,000 people in 16 nations and make it all come together for a shot into space. You will also learn about the future of personal computer use in the International Space Station and future space shuttle flights. Post your comments below or send us a note.

Editor's Picks