You think you have IT problems? Try maintaining security, data integrity, and reliability in the go-fast world of Indy Racing. Erik Eckel shows the issues that face an IT professional in the pits when supporting a racecar doing 220 mph.
There's a lot riding, literally, on the decisions Jim Foley makes. An assistant engineer with Rahal Letterman Racing, Foley and his fellow technicians are responsible for collecting critical metrics from the three racecars the team fields in the Indy Racing League series, including the Honda-powered racer piloted by rookie sensation Danica Patrick.
Race weekends present critical windows in which the team's engineers must collect racecar telemetry real-time, pour over gigabytes of data, quickly make sense of the information, and recommend chassis, motor, wing, tire and other adjustments. (For more on the technologies used by IndyCar teams, read TechProGuild's IndyCar IT: The technology that makes Danica Patrick go.) Just one wrong decision, one corrupt or lost piece of data, and an incredibly expensive race car could be destroyed. Worse, the driver could be injured.
Protecting against data loss and systems outages is critical in most every business. But when safety, staggeringly expensive machinery and sponsor's hopes (and it's sponsors that pay IndyCar team's bills) stand in the balance, having strong data protection and recovery practices in place are a necessity. Here are lessons every business can take from Indycar race engineers, who each weekend depend upon complex interconnected mobile networks of onboard electronics systems, radio (both UHF and 802.11) transmitters, servers, laptops, specialized software and customized macros to just finish a race, much less compete for the victory.
Back up your data
Of course it's a no brainer, but how often do you do it? Weekly, monthly, daily? What about a few times a day?
In IndyCar racing, Saturdays typically present one or two practice sessions and then it's time to qualify. It's imperative that teams arrive prepared. There simply isn't time to work through myriad changes. Once Sunday arrives, teams can practice for maybe 45 minutes in race trim, and then nationwide TV audiences tune in to witness how well the team performs.
How do team's prepare? By meticulously backing up data.
"In this business, a lot of what we end up doing is due to good bookkeeping," said Foley.
That's true in every business. But lose your data from a test session and you're out in the cold.
"If you just show up with the same thing you had last year, well then, you're a year behind," added Foley.
|A member of the Rahal Letterman Racing team studies data from one of its racecars at Kentucky Speedway.|
Following test sessions, the team's data acquisition engineers offload data from the racecars' onboard recorders right in the pits, then head for team transports—really high-tech mobile offices on wheels—to begin crunching the data. (View trackside images, including photographs of the equipment teams use on race day, in TechProGuild's Trackside Technology Photo Gallery). Inside these trucks, servers form the backbone of the local area networks. Backups are a critical part of the process.
"We back up our information on each other's machines, so there are very rare instances, if any, where this chunk of information is only on one machine," said Foley.
The laptops are also backed up to the servers in each team's hauler.
"There's a server machine in each of the trucks," said Rahal Letterman Racing's IT Manager, Rob Trinkner. "It's basically a location to backup information on the truck. It gives them [the team's engineers] a location to do that. Typically, the most information we're worried about there is the data that is collected from the car because it's irretrievable, basically. If you've lost it, it's gone."
Eliminate single points of failure
IndyCar teams travel more than four months a year. The series' peripatetic nature complicates information technology efforts, which must be broken down and reconstructed week after week.
As you've seen, engineers download racecar data - everything from tire pressures and temperatures to chassis behavior to engine performance and more is recorded - to laptops via Ethernet cables during practice and testing, then back up those laptops to other notebooks or servers in the garage area. Those backups help eliminate one point of failure.
But during a race, the car's onboard electronics systems send critical data back to the pits wirelessly. Having access to these metrics is critical during a race, as car setups, fuel mileage calculations and other vital decisions are based on the real-time information returned and recorded. This data is so sensitive, and would provide such significant competitive advantages to other teams if revealed, that it's encrypted. The pit-side server provides a back up role.
"One of the reasons we run the server as well as the laptops is that I've created a dual system, so that, if my laptop locks up or goes into a reboot or anything like that during a race, that server will have the encryption key and still be getting information or vice versa," said Foley.
Having such redundancy proves helpful in critical environments, whether your business is delivering health care, maintaining an e-commerce system or racing cars.
"If Jim's machine has some kind of a problem and he needs to reboot, in a lot of race instances when you're only turning 18- or 16-second laps," added Trinkner, "you can be way behind if you're waiting on a machine to reboot for five minutes. They still need that information streaming in."
Efforts to eliminate single points of failure don't end there, though, for Rahal Letterman Racing. And neither should they prove simple in your organization. Audit your systems. Determine which servers, PCs and applications are critical, and ensure you have contingency plans in place to bring redundant systems online immediately. Depending on the nature of your business, deploying a new server from a backup image may take too long. You may need a secondary system already running online.
Again, "the server acts as a redundant system," added Trinkner. "There are other people within the company that keep a copy of Jim's, what we call, setup for the software. So that, if his machine is run over by a truck, he can grab somebody else's laptop and put it in place."
|Rahal Letterman Racing engineers study data trackside.|
Think it's unlikely laptops will get run over by a truck? You might be right, but vehicles do prove hazardous.
Prepare critical system replacements
The dependence upon mobile systems introduces additional challenges. Whether you support traveling engineers, salespeople in the field or another class of road warriors, keeping pre-imaged systems ready for instant deployment can ensure critical systems remain operational in the event of a hardware failure or otherwise-compromised PC.
Supporting IndyCar engineers at the track "is no different than any other business in that you've got people carrying laptops and things happen," said Trinkner. "People lose laptops. People drop laptops. Information in that device gets destroyed and you have to have some way, something, to fall back on."
Every time a team transport departs from Rahal Letterman Racing headquarters in Hilliard, Ohio, extra units make the trip.
"We keep spare chasses and hard drives on the trucks," added Foley. "So that way, when one [fails], if you have a chassis go out, we've got another one on the truck."
And fail they do. Trinkner said a system was lost at Kentucky Speedway, where the team's Patrick won the pole and teammate Vitor Meira finished second.
Data acquisition engineers often place their laptops on a racecar's radiator ducts, a convenient flat surface above the air inlet, when testing the engine for leaks or conducting basic checks.
"If you've ever been around a racecar when it's running, there's a lot of vibration, heat – it's a volatile environment," said Trinkner. "To have a laptop basically sitting on the side of the racecar isn't necessarily the smartest thing to do, but we do it a lot."
The vibration alone can cripple a hard drive, not to mention proximity to methanol (which burns on contact), frenzied mechanics and their tools, dust from desert race tracks, stray wheels and other hazards.
"We just had an incident of that last week," said Trinkner. Someone fired up a motor and vibrated the hard drive of an engineer's laptop into oblivion.
While your organization's laptops may not have to battle chemicals or the concussions from a 700-horsepower motor, they must withstand inadvertent drops, being tossed in rental cars and being knocked about airports. When supporting critical mobile efforts, such as at trade shows, conventions, presentations and other key events, consider sending a second, preconfigured backup unit. Should a primary unit fail, having a second machine ready could prevent a wasted trip or lost business.
Regularly upgrade critical systems
The average laptop service life for an IndyCar engineer is one season. Teams upgrade equipment frequently, not only in an attempt to ensure they're using the most current hardware, but to help eliminate failures.
"I'm a very conservative person," added Trinkner. "I don't like to make a lot of changes during the season. Typically, I will get a new model laptop myself, set it up and do a bunch of testing with it, basically to see if it passes my specs."
Just as swapping out new systems can introduce hiccups – such as driver issues or software incompatibilities – on a race team, so too can it wreak havoc in your business. Budgets are always an issue. But regularly replacing crucial PCs or servers (and relegating the replaced units to other, less important tasks) is a small price to pay to ensure critical systems don't fail at the most inopportune times.
Sooner or later, all hard drives fail. Laptops, particularly those exposed to excessive travel, can become unstable. Servers and other components, over time, deteriorate.
Take the lead from these IndyCar engineers. Even though your organization's circumstances likely differ, following these four methods is one way to help mitigate IT disasters and prevent data loss.