There’s a lot riding, literally, on the decisions Jim Foley
makes. An assistant engineer with Rahal Letterman Racing, Foley and his fellow
technicians are responsible for collecting critical metrics from the three
racecars the team fields in the Indy Racing League series, including the Honda-powered
racer piloted by rookie sensation Danica Patrick.

Race weekends present critical windows in which the team’s
engineers must collect racecar telemetry real-time, pour over gigabytes of data,
quickly make sense of the information, and recommend chassis, motor, wing, tire
and other adjustments. (For more on the technologies used by IndyCar teams,
read TechProGuild’s IndyCar IT: The technology that makes Danica Patrick go.) Just
one wrong decision, one corrupt or lost piece of data, and an incredibly
expensive race car could be destroyed. Worse, the driver could be injured.

Protecting against data loss and systems outages is critical
in most every business. But when safety, staggeringly expensive machinery and
sponsor’s hopes (and it’s sponsors that pay IndyCar team’s bills) stand in the
balance, having strong data protection and recovery practices in place are a
necessity. Here are lessons every business can take from Indycar race
engineers, who each weekend depend upon complex interconnected mobile networks
of onboard electronics systems, radio (both UHF and 802.11) transmitters,
servers, laptops, specialized software and customized macros to just finish a
race, much less compete for the victory.

Back up your data

Of course it’s a no brainer, but how often do you do it?
Weekly, monthly, daily? What about a few times a day?

In IndyCar racing, Saturdays typically present one or two
practice sessions and then it’s time to qualify. It’s imperative that teams
arrive prepared. There simply isn’t time to work through myriad changes. Once
Sunday arrives, teams can practice for maybe 45 minutes in race trim, and then
nationwide TV audiences tune in to witness how well the team performs.

How do team’s prepare? By meticulously backing up data.

“In this business, a lot of what we end up doing is due
to good bookkeeping,” said Foley.

That’s true in every business. But lose your data from a
test session and you’re out in the cold.

“If you just show up with the same thing you had last
year, well then, you’re a year behind,” added Foley.

Figure A
A member of the Rahal Letterman Racing team studies data from one of its
racecars at Kentucky Speedway.

Following test sessions, the team’s data acquisition
engineers offload data from the racecars’ onboard recorders right in the pits, then
head for team transports–really high-tech mobile offices on wheels–to begin
crunching the data. (View trackside images, including photographs of the
equipment teams use on race day, in TechProGuild’s
Trackside Technology Photo Gallery). Inside these trucks, servers form the
backbone of the local area networks. Backups are a critical part of the
process.

“We back up our information on each other’s machines,
so there are very rare instances, if any, where this chunk of information is
only on one machine,” said Foley.

The laptops are also backed up to the servers in each team’s
hauler.

“There’s a server machine in each of the trucks,”
said Rahal Letterman Racing’s IT Manager, Rob Trinkner. “It’s basically a
location to backup information on the truck. It gives them [the team’s
engineers] a location to do that. Typically, the most information we’re worried
about there is the data that is collected from the car because it’s
irretrievable, basically. If you’ve lost it, it’s gone.”

Eliminate single points of failure

IndyCar teams travel more than four months a year. The
series’ peripatetic nature complicates information technology efforts, which
must be broken down and reconstructed week after week.

As you’ve seen, engineers download racecar data – everything
from tire pressures and temperatures to chassis behavior to engine performance
and more is recorded – to laptops via Ethernet cables during practice and
testing, then back up those laptops to other notebooks or servers in the garage
area. Those backups help eliminate one point of failure.

But during a race, the car’s onboard electronics systems
send critical data back to the pits wirelessly. Having access to these metrics
is critical during a race, as car setups, fuel mileage calculations and other
vital decisions are based on the real-time information returned and recorded.
This data is so sensitive, and would provide such significant competitive
advantages to other teams if revealed, that it’s encrypted. The pit-side server
provides a back up role.

“One of the reasons we run the server as well as the
laptops is that I’ve created a dual system, so that, if my laptop locks up or
goes into a reboot or anything like that during a race, that server will have
the encryption key and still be getting information or vice versa,” said
Foley.

Having such redundancy proves helpful in critical
environments, whether your business is delivering health care, maintaining an
e-commerce system or racing cars.

“If Jim’s machine has some kind of a problem and he
needs to reboot, in a lot of race instances when you’re only turning 18- or
16-second laps,” added Trinkner, “you can be way behind if you’re
waiting on a machine to reboot for five minutes. They still need that
information streaming in.”

Efforts to eliminate single points of failure don’t end
there, though, for Rahal Letterman Racing. And neither should they prove simple
in your organization. Audit your systems. Determine which servers, PCs and
applications are critical, and ensure you have contingency plans in place to
bring redundant systems online immediately. Depending on the nature of your
business, deploying a new server from a backup image may take too long. You may
need a secondary system already running online.

Again, “the server acts as a redundant system,”
added Trinkner. “There are other people within the company that keep a
copy of Jim’s, what we call, setup for the software. So that, if his machine is
run over by a truck, he can grab somebody else’s laptop and put it in place.”

Figure B
“>
Rahal Letterman Racing engineers study data trackside.

Think it’s unlikely laptops will get run over by a truck?
You might be right, but vehicles do prove hazardous.

Prepare critical system replacements

The dependence upon mobile systems introduces additional
challenges. Whether you support traveling engineers, salespeople in the field
or another class of road warriors, keeping pre-imaged systems ready for instant
deployment can ensure critical systems remain operational in the event of a
hardware failure or otherwise-compromised PC.

Supporting IndyCar engineers at the track “is no
different than any other business in that you’ve got people carrying laptops
and things happen,” said Trinkner. “People lose laptops. People drop
laptops. Information in that device gets destroyed and you have to have some
way, something, to fall back on.”

Every time a team transport departs from Rahal Letterman
Racing headquarters in Hilliard, Ohio, extra units make the trip.

“We keep spare chasses and hard drives on the trucks,”
added Foley. “So that way, when one [fails], if you have a chassis go out,
we’ve got another one on the truck.”

And fail they do. Trinkner said a system was lost at
Kentucky Speedway, where the team’s Patrick won the pole and teammate Vitor
Meira finished second.

Data acquisition engineers often place their laptops on a
racecar’s radiator ducts, a convenient flat surface above the air inlet, when
testing the engine for leaks or conducting basic checks.

“If you’ve ever been around a racecar when it’s
running, there’s a lot of vibration, heat – it’s a volatile environment,”
said Trinkner. “To have a laptop basically sitting on the side of the
racecar isn’t necessarily the smartest thing to do, but we do it a lot.”

The vibration alone can cripple a hard drive, not to mention
proximity to methanol (which burns on contact), frenzied mechanics and their
tools, dust from desert race tracks, stray wheels and other hazards.

“We just had an incident of that last week,” said
Trinkner. Someone fired up a motor and vibrated the hard drive of an engineer’s
laptop into oblivion.

While your organization’s laptops may not have to battle
chemicals or the concussions from a 700-horsepower motor, they must withstand
inadvertent drops, being tossed in rental cars and being knocked about
airports. When supporting critical mobile efforts, such as at trade shows,
conventions, presentations and other key events, consider sending a second,
preconfigured backup unit. Should a primary unit fail, having a second machine ready
could prevent a wasted trip or lost business.

Regularly upgrade critical systems

The average laptop service life for an IndyCar engineer is
one season. Teams upgrade equipment frequently, not only in an attempt to
ensure they’re using the most current hardware, but to help eliminate failures.

“I’m a very conservative person,” added Trinkner. “I
don’t like to make a lot of changes during the season. Typically, I will get a
new model laptop myself, set it up and do a bunch of testing with it, basically
to see if it passes my specs.”

Just as swapping out new systems can introduce hiccups –
such as driver issues or software incompatibilities – on a race team, so too
can it wreak havoc in your business. Budgets are always an issue. But regularly
replacing crucial PCs or servers (and relegating the replaced units to other,
less important tasks) is a small price to pay to ensure critical systems don’t
fail at the most inopportune times.

Remembering Murphy

Sooner or later, all hard drives fail. Laptops, particularly
those exposed to excessive travel, can become unstable. Servers and other
components, over time, deteriorate.

Take the lead from these IndyCar engineers. Even though your
organization’s circumstances likely differ, following these four methods is one
way to help mitigate IT disasters and prevent data loss.