Data Centers

The mystical world of data center fire suppression

Data center fire suppression is more than a few sprinkler heads and a cool-sounding alarm. Rather it is about risk, budget, and a little common sense coming together to meet business continuity outcomes.

I can hear you now… “Not another article on the boring topic of fire suppression!” Well, I promise if you ignore your growing angst and an urge to shift to another page, any other page, I will make it worth your time.

Like data centers (once called computer rooms or electronic data processing centers), the approach to containing a fire in what is now the nerve center of your organization has significantly changed. It isn’t enough to simply install a smoke alarm and a few sprinkler heads. This only works if you don’t mind having your business down for days or weeks after a fire.

In this article we’ll look at the business outcomes you should consider when writing a check for fire suppression services and how to achieve them.

Business outcomes

Before purchasing a fire suppression system, it’s important to understand what it is your trying to achieve.  Yes, you’re trying to put out future fires. But that is an incomplete requirement. The real business requirement is to put out the fire and get your business back up and running within hours.

The components of a business-friendly suppression system include:

  • A detection system that detects actual fires and not a leak in your cooling system (more on that later…)
  • An alarm that includes both a loud noise and flashing lights (a regulatory requirement in most jurisdictions)
  • Portable fire extinguishers placed in critical locations
  • Emergency power-off switch
  • Emergency suppression system delay or cancel (more on that when we talk about false alarms)
  • A suppression agent that:
    • doesn’t destroy your equipment; and
    • doesn’t take so long to clean up that you pass right by your maximum tolerable downtime threshold

Like all physical security controls, the number of dollars you spend and the system you select depend on the business risk you are willing to accept. With that in mind, let’s take a closer look at the elements of today’s data center fire suppression solutions.

Detection systems

You would think a detection system has a simple job: detect a fire. The problem arises when you begin to consider what a detection sensor sees as a fire.

A few years ago, our suppression system decided to dump Inergen in our data center on a Saturday night.  Since no one was present, we couldn’t hit the switch to delay the release.  If we had, we would have found that one of the sensors detected the gas emitted by a leak in the cooling system as a fire.  Since we had a contract with a data center cleaning and restoration company, we quickly called them; it took 24 hours to return the data center to normal.

The moral of the story? Make sure your detection system is tuned not to react to possible false alarms caused by other components in your data center. Another consideration is just as important. If your sensor detects smoke, it should also want to see heat before ruining your day.

Alarms

Alarms are pretty simple; they make a loud noise and people walk quietly to the exit. That’s fine if you’re using systems that cause no harm to humans. However, many types of suppression materials (covered later) are intended only for areas with no human presence.  So you should integrate the alarm into the suppression release process, allowing an adequate human evacuation period to elapse before putting out the fire.

The flashing light is not only a good idea when one or more employees have hearing problems. It is also useful when the alarm is trying to rise above the ambient decibels generated by hundreds of servers, storage devices, etc.

Portable fire extinguishers

First, the primary purpose of the portable fire extinguishers hanging on support posts in your data center is not to encourage your server engineers to bravely stand between a fire and that new enterprise server platform they just spent days getting to work right. Rather, they are intended to ensure people in the data center have a way out when fire is blocking the exits.

Second, make sure your fire extinguishers are placed properly (see Fire Extinguisher Placement) and contain the appropriate suppression agent.  For more information on this topic, see Fire Extinguishers or Online Safety Library:Fire Extinguishers.  These sites explain their use and types required for various combustible materials.

Emergency power off switch

This isn’t something I plan to spend a lot of time on. It’s important to take away that no large facility with scores of electrical or electronic devices should ever be without a big red button on the wall that, when pushed, immediately cuts all power. However, use sparingly. Hard power shutdowns are not computer friendly.  (Can you say backups?)

Selecting the right suppression agent

So we finally arrive at the one item in our list that can either allow quick recovery or weeks of finger pointing. Selecting the suppression agent to put out actual fires is a decision made with all critical business process stakeholders at the table. The discussion should include the following options:

  • Wet pipe – Wet pipe systems are basic water sprinklers installed in commercial buildings. Charged with water at all times, they release plain water in the presence of heat, smoke, or manual intervention. Although they work great in most areas of the business office, they have one big disadvantage when placed in the data center. THEY DUMP WATER ON YOUR EQUIPMENT. If you think a hard power shutdown is bad, try to recover 300 servers, several storage devices, and other critical infrastructure once hundreds or thousands of gallons of water is dumped on them.  If you opt to go with water, make sure your hot site contract is current and your team is well practiced.  Because putting a fire out like this constitutes a catastrophic event. And you might have trouble explaining why your data center is down for two weeks.
  • Pre-action or dry pipe – Pre-action systems work just like wet pipe solutions with one exception; the water is not kept in the pipes.  This system is marketed to businesses as safer than wet pipe, because water-charged pipes can accumulate moisture via condensation. This moisture can then drip down on critical equipment. However, the problems associated with putting out a fire with water still remains.
  • Gaseous agents – These types of systems provide immediate fire suppression with relatively short system and business process recovery times. Gaseous agents deny a data center fire access to two of the three elements necessary for combustion: heat and oxygen.
    • Agents such as HFC-227ea, FM-200, and HFC 125 remove heat from fire. This is what water does, but these agents won’t destroy your equipment.
    • Carbon dioxide and Inergen remove oxygen from the environment.
So gaseous agents are usually your best choice, if you can afford them. In addition to higher implementation costs than water systems, gas systems require annual maintenance and floor space for canisters storing the agent (see Figure A from Wikipedia.org). In this example, there isn’t much space used. However, in my last data center the Inergen canisters took up about 200 square feet in a room separate from the data center.  We affectionately called it the missile room.

Figure A

In addition to these traditional systems, some companies have released what they call fire prevention systems. These proactive approaches reduce the normal amount of oxygen in the data center, reducing the opportunity for a fire to really get started if sufficient heat and fuel present themselves. Before running out and buying one of these systems, check with local and federal safety regulators. Most are still on the fence, teetering between acceptance and their belief that these systems reduce oxygen to levels harmful to human health.

The final word

So as you can see, implementing fire suppression as a physical security control is not so easy. It requires knowledge of business risk acceptance, understanding of maximum tolerable downtime, and the right budget. And as always, it requires the cool, calm security manager who brings the right people to the conversation and enlightens them in how properly dealing with Prometheus’ gift sometimes requires a little wisdom.

About

Tom is a security researcher for the InfoSec Institute and an IT professional with over 30 years of experience. He has written three books, Just Enough Security, Microsoft Virtualization, and Enterprise Security: A Practitioner's Guide (to be publish...

8 comments
Thinman2
Thinman2

Hi, just a few comments. Inergen does not remove oxygen from a data center, it "reduces" the level of O2 below the point where combustion can occur. Clean agents like FM-200 and Novec operate by lowering temperature and interfering with the chemical process which is present during combustion. You will never be able to get away from dry pipe pre-action systems because there is a fire code to require it. IMO a gas system is not needed if you have a well designed detection system. The majority of fires/smoke in the data center are electrical in nature. Once the power is dropped the combustion will go out quickly. There is nothing else to support the combustion. Rather then use the word "fire" which implies flames, smoke, heat, combustion is a better word. What would burst into flames in a data center? Computer equipment has not been made out of wood for years, if you know what I mean.

minstrelmike
minstrelmike

I like how suppressing the fire is not the goal of your article. The goal is getting up and running quickly after suppressing the fire. Interesting comment on slashdot when talking of the oil rig fire on Deepwater Horizon. One guy was a Navy fireman and said half of their practice time was spent on de-watering, removing water that you've already sprayed into some cabins. Sinking the ship does put out the fire but it is not a 'solution.'

walter
walter

I believe that FM200 is the better agent to use. Inergen is stored at 200bar and and we've heard reports of it lifting tiles out raised floors and ear drums popping when the gas discharges. By comparison FM200 is stored at 30bar, I seem to recall reading that it is also the gas used in asthma pumps. If you do use FM200 make sure that the airconditioners are not linked to the emergency power off switch as the gas should preferably continue to circulate in the room to continue suppressing possible flare-ups.

seanferd
seanferd

Make me wonder what the detector is detecting as a proxy for fire, and how accurate and precise it is in detecting the proxy rather than other things which "mimic" the proxy as far as the detector is concerned. Home smoke alarms tend to activate in the presence of water aerosols (like from a hot shower or boiling water), which is incredibly stupid. People end up pulling out the batteries on non-hardwired alarms because of this. The reason is that they are using a bad proxy: any particulate matter in the air, with possible tuning for particulate size, and density of particulates.

spearson@8herons.com
spearson@8herons.com

Paint. On outsides of equipment. On walls of server room. Floor tiles. Wax/sealer on floor tiles or floor. Plastic parts of equipment, as many to most of them are not made of thermo-plastics. Insulation on wiring inside equipment, like the insulation on hard-drive cables. Non-conductive varnish coating covering copper traces on motherboards and other circuit boards. Some IT departments may have taken this into consideration and chosen materials wisely. But most places don't stop to think about the paint on the walls of the server room and on the equipment itself. Some of this stuff may be fire-retardant, but that does not mean it does not catch fire. Often times it means that it is fire-retardant as long as the heat does not exceed the limitations of the fire-retardant material. After which it continues to burn, although supposedly not as fast as non-fire retardant materials.

tatec3
tatec3

A good (i.e., low false positive rate, low false negative rate) detection system won't rely on just heat and smoke as the article implies, as it can take a lot of heat close to the detector (read: fully involved room, i.e., too little too late) to trigger a thermostatic heat detector (see below for other issues with infrared detectors). If using smoke detectors, you'll need both types, ionic (for flaming fires) and photoelectric (for smoldering fires with small numbers of large smoke particles; think: "burning cable insulation"), but the ionics suffer either from generating a high false positive rate or being detuned to the point of too many false negatives. Instead, modern top-of-the-line 'machine vision' systems designed to protect high-value equipment also use optics combined with software algorithms to improve detection of real fires while lowering the false positive rate. Here again, it's important not to cut corners and go with infrared detectors only (though that will get you significant reduction in false negatives), but rather to add ultraviolet detectors to see the flame plasmas, combining the two to reduce the false positive rate. The USAF learned this lesson the hard way, having a foam (AFFF) system installed in an F-15 hangar with automatic triggering by an infrared detection system. Can't remember the exact false positive circumstances in this case, either a plane taking off in full afterburner passing by the open hangar door, or a security policeman driving by in a cart while smoking a cigarette (both have triggered foam dumps with IR-only systems), but $1.5m later, they finally had the ruined cockpit avionics refurbed on the F-15 that was in the hangar with its canopy open when the AFFF system dumped its load. Cf. http://www.shaw.af.mil/shared/media/photodb/web/090116-F-5561D-027.jpg, http://web.me.com/maclarenj/Jet_Aviation_Foam/Photos.html, http://www.ityt.com/forums/f24/lots-of-bubbles-489.html and http://gardneredge.com/news/2010/04/02/2618-fire-suppression-foam-fills-chinook-hangar for similar cases of accidental AFFF discharge. Also keep in mind that smoke detectors' sensors can get coated or chemically 'poisoned' over time by all the other stuff in the air (e.g., normal off-gassing of vinyl and other plasticizers), especially in an enclosed, warm space, so they do need to be periodically replaced.

Editor's Picks