Disaster Recovery optimize

Ten years later: IT and life lessons from the South Tower

Bob Eisenhardt was an IT pro employed in the South Tower on 9/11. He explains what it took to recover -- both personally and in IT.

When the Twin Towers of the World Trade Center were attacked ten years ago, 2,753 people lost their lives, including employees of companies housed in the towers, 343 firefighters, 60 police officers, and eight private emergency medical technicians and paramedics. Another 184 people were killed in the attack on the Pentagon. The overwhelming majority of casualties were civilians, including nationals of over 70 countries.

TechRepublic wanted to commemorate that day in a meaningful way. We have asked Bob Eisenhardt, an IT pro and TechRepublic member who was employed in the South Tower, to talk about his own experience and what it took to recover and rebuild his IT shop in the aftermath.

On September 11, 2001 the world changed.  I was part of the IT team for Aon Group (a global insurance company) located in the 99 to 105 floors of the South tower.  TechRepublic has invited me to commemorate the tenth anniversary with these personal notes and memories.

Since this is an IT site, I will talk about the loss of and recovery of data in the immediate aftermath of the crashes. But I have to stress that, as it is with any disaster scenario, people are the most precious commodity.  Aon's Chairman, Patrick Ryan, demonstrated this when he headed a memorial service at St. Patrick's Cathedral, with Judy Collins singing "Amazing Grace."  For most of us, this was the first time we all came together again in a single room, and we were mostly still in shock.  Recovery would take a long time.

It's important that your staff know that in evacuations they should not worry about computers, data, retrieving their personal effects from their desks, or going back to get their car.  I learned this in the worst possible way: Steve Poulos, the system administrator for Aon Risk Services, returned to the 103rd floor to retrieve the Risk Service data tapes.  By doing so, he ran out of time and did not survive.

Get clear, get safe and go home as required.  You should have your staff participate in evacuation drills, taught to use stairs and not elevators, and to have on hand "go bags" (small, easy-to-carry container for basic supplies such as flashlights, small medicals, maps, food and money).  IT personnel may consider placing periodic backup data in their go bags if it fits snugly.

Another thing I learned on that day ten years ago was that simple common sense proved to be the ultimate skill.  "Down was good, up was not an option, speed was essential."  Thousands of people made snap decisions without knowing the consequences of their actions; survival often meant just following the basics.  (To quote Jean Reno's wonderful advice in the 1998 remake of Godzilla - "Running would be a good idea."  When time came, I ran!)

The technical details

Due to maintenance in the 103 floor data center on Sept. 8-9, the data tapes for Aon Consulting ran cleanly and were taken offsite on Monday morning.  We were lucky to have those backups as Tuesday was a very different day.

Technology also aided survivors tremendously in the days that followed the tragedy. In distant Chicago, Aon quickly developed a web bulletin board for survivors to post comments, which also enabled us to do head-counts.  With that and the Yahoo-finance message board, survivors and family members could reconnect, and have a space to express their mourning.

September 11 was traumatic to the extreme, and to this day I am truly amazed that the Aon team began restoration literally within hours after the event.  On September 12, Michael Edghill was arranging for our data tapes to be shipped to the South Carolina data center. Some of us went to work quickly, others took their time in emotional recovery but, eventually, we all returned. Our professional ethics required that we get to work as soon as possible, and we had it in abundance.

On September 24, we received in our temporary mid-town office over 900 Dell Optiplex desktop towers, large 21" monitors and 250 laptop systems. The distribution of this equipment required a huge multi-office team effort. We put GHOST imaging to the ultimate test on a bench running 10 systems at a clip. (The only unforeseen problem was that in the beginning we were too easily handing out free keyboards and mice to anyone who needed one, so we later just ran out of them entirely. No good deed goes unpunished). In this intense recovery environment, you worked fast and smart, ate many boxes of pizza, and made lifelong friends.

The 900 plus systems used in rebuilding the network. Photo: Bob Eisenhardt

Not everyone was using imaging systems. I was initially assigned to cover the dislocated Aon Consulting staff move to our Greenwich, CT site.  Here my primary job was to keep these emotionally stunned workers productive while staying out of the way of Larry Manno, manager of this location, who already had more than enough to do.  By fortunate circumstance, I carried a small laptop that performed truly heroic work for my co-workers. For a few days, I was virtually the only IT staffer who actually had a computer! Print server, email conduit, temporary loaner - this was a wonderful little laptop indeed.

For months prior to 9/11, we were forever complaining about the size of Lotus Notes email files.  And yet, after September 11, these same large email files contained a wealth of stored easy-to-access data that provided its own ad-hoc backup protocol!  I will never complain about large email files again.

We also learned that Dell does not build substandard equipment.  At Ground Zero, workers found an intact Dell Latitude CSX laptop belonging to Louis Sausa. It survived falling 101 floors amidst tons of collapsing building.  Now in the hands of the World Trade Center museum, it has not been turned on again, so it remains set to September 10, 2001.  I wonder if the BIOS knows what day today is.

For me, the essence of the day is now compressed into 102 minutes, from 8:45 am to 10:02 am.  Last year on 9/11, while visiting family in Atlanta, I walked away to a fence at a Soccer game, touched the ground at 9:59 am - the moment the South Tower collapsed.  The earth held the remains of many of my friends:  Donald Havlish, John Crowe, Laura Snick, Catherine Nardella, and Pamela Gaff, among others.  I took a deep breath and then a small hand took mine. My granddaughter, Nicole Skyer, age 7, came to take me back to the soccer game and another day.

Every morning, take a glance up at the sky and remember that you are alive to enjoy this day and that for all the good and bad it may bring, you are here to see it through.

I dedicate this piece to Steve Poulos.

See also:

ZDNet: How 9/11 changed my IT consulting career ZDNet: The day the planes hit (9/11 Diary) CBSNews: Holes remain in flight school scrutiny after 9/11 CBSNews: Patriotism's new face 10 years after 9/11
62 comments
wyattharris
wyattharris

I was teaching IT when 9/11 happened. For years after that when the topic of backups and disaster recovery came up I would point to the towers, what could happen and how being prepared for anything is not just empty words. I am glad to read the actual events of the aftermath. Thinking clearly during a crisis is a skill all it's own. PS. I've got my own little workhorse laptop. Gotta love 'em.

ccalcut
ccalcut

Thank you, Bob. I can only imagine how hard this was for you to revisit mentally and write about it.

fhrivers
fhrivers

God bless you, Bob, and your entire family.

morie123
morie123

know the 9/11 as the americans.

saberrattler
saberrattler

I was an Aon employee for 5 years in Chicago, back in the 90's. This brought back memories of all the good people I knew and worked with everyday. Thanks for the article, Bob. I hope you and the other surviors are all doing well today.

Ana Cantu
Ana Cantu

What a moving story. Thanks for sharing it with us, Bob. Though I now work for Dell, on 9/11/01, I was a journalist in L.A. and was awakened by a frantic phone call from my best friend in D.C. I went straight to the newsroom and helped with the extra we printed at noon. I remember looking at the photos from New York and Washington on the wire and being thankful that so many made it out safely. Those who died that day will always be missed.

I_remember
I_remember

Almost 3000 soul's whose only crime was to be at work on Tuesday morning. May they rest in peace and know we remember!

Professor8
Professor8

I was trying to debug a nasty memory management problem -- a conflict with a custom reference counting code that had been installed as a system trap and a co-routing sharing dynamically allocated memory using the standard system calls. Anyway, late, very frustrated, I got on usenet to "play". We argued economics (and the state of the economy; it'd been several months since we'd been able to make sufficient revenues to cover pay-roll) and ethics and criminology and movie-making and politics and over differing views of the various amendments in the Bill of Rights. They mentioned box-cutters (and others discussed blade-weapons vs. firearms-wielding defenders vs. martial arts and I quoted some criminologists who said firearms are the most effective means of defense), burning temperatures of jet fuel, fuel capacity of passenger jets, what would happen if a jet crashed into a building (I mentioned the concrete encasement of steel to protect it against fire heat, and it should have if it had been done correctly), passenger capacity of planes, how many people worked and shopped in the WTC and would likely be there at about 09:00, whether and how much of an emotional attachment Americans had to the WTC (none, I wrote; I'd be more upset if they did something to... other places I won't mention so as not to give the slime ideas). It was all in a scattering of different discussions so that their intentions were not in the least bit apparent before the fact, but obvious the next day. So, when I woke shortly after the first plane hit and heard the coverage on the radio, I knew exactly what was happening, and that it was too late to do a thing about it. Some of the terrorists, or people connected with them, were participating in some of those discussions, telling us exactly what they planned, but in little scraps. That was the most frustrating thing, at first, that morning. Then, as I got ready to drive into the office, I realized that, compared to their goals, it was a huge failure for the terrorists. You see, they'd estimated that their would be about 80K to 120K people working and shopping in and around the WTC, and it was apparent afer the fact that they were expecting to kill over 80K people and most likely close to 100K. So the most frustrating thing after the deeds were done was that the media turned that abject failure into a "win" for the bad guys. I mean, don't get me wrong; even if they had only 1 or 2 victims it would be bad. But the buildings held up better than they'd expected, and the people in them had acted in a more level-headed way than the terrorists expected, so that, in the scheme of things, nearly all of their intended victims survived. Other 9-11s: 0813 Charles the Great crowned Louis I emperor; 1297 Scots under William Wallace defeated British at Stirling Bridge; 1649 Oliver Cromwell and his "roundhead" protestants seized Drogheda, Ireland and killed 3K; 1697 prince Eugen van Savoye defeated larger Turkish force at Zenta; 1777 Battle of Brandywine, PA; 1857 Mountain Meadows Massacre; 1940 Hitler sent Nazi reinforcements into Romania.

jpclyons
jpclyons

Nice job, Bob. I made a video as part of an entry into the Tropfest/Tribeca Film Festival in 2007. It can be viewed on Vimeo at the link below. http://www.vimeo.com/9211202

tavent
tavent

thanks for helping us to remember some of the things that we did not directly experience, and speaking the names of those who were lost.

UrNotPayingAttention
UrNotPayingAttention

[i]"I took a deep breath and then a small hand took mine. My granddaughter, Nicole Skyer, age 7, came to take me back to the soccer game and another day."[/i] Best. Therapy. Ever.

wompai
wompai

And bless your little laptop, too!

BradTD
BradTD

Loved hearing about the technical details--all while never forgetting about the human impact. We will never forget that day. Well done.

jkameleon
jkameleon

... such commemorations are ... I don't know... a bit tasteless. Sorry, just had to say it.

alexisgarcia72
alexisgarcia72

I'm at Mexico and our headquarters in sept 11 was located in the tower: about 6 floors. Was a terrible day. Thanks God no lives was lost. In terms of IT, they had a terrible experience with the backup stuff. The offsite backup recovery failed so most of the data was lost. Thanks for sharing this. Good bless Steve Paolos, who leave his life for get the data safe, but lost his life. There is plenty of IT admins and professionals who works day and night and even during weekends and put all the efforts to give their best to the work.

Dale Cook
Dale Cook

Thanks for writing this and sharing.

ilya.shick
ilya.shick

"For me, the essence of the day is now compressed into 102 minutes, from 8:45 am to 10:02 am" Thanks

gvienna
gvienna

As I think back to that day I remember the sickening fear I had for all the people that lost their lives. Thank you for helping us remember the fallen and bring some perspective to our own lives.

realvarezm
realvarezm

Thanks for the advice on how to act when a disaster strikes. I know you heard this a lot but is the right thing to say, sorry for your losses and im happy to share the experience you wrote. Nothing happens without a reason and maybe this tragic event touch and make many people like oyu and make them better human beings and light for others who are dwelling in the shadows of anger, hate and sadness.

pinoysgotit
pinoysgotit

this is what i felt while reading this post - mixed feelings. i felt the need for you to explain how an IT guy coped with the tragedy and what IT guys like us has to know.. yet your heart wanted to tell more.. i believe you succeeded in both. definitely the best post i read here in techrepublic! http://www.pilipinas.it

onalethata
onalethata

"It???s important that your staff know that in evacuations they should not worry about computers, data, retrieving their personal effects from their desks, or going back to get their car." True A very touching commemoration... Thank you..

AreV
AreV

I have been avoiding all of the 9/11 recurring sadness but your approach using the familiar IT functions that most will not discuss made your post even more poignant for me. I am glad that I didn't click through as your duties painted an even more human picture. I am sorry for your losses, and glad you came into my life once, this morning.

john_black
john_black

I cannot begin to feel what you have been through and I am glad that you have had the opportunity to share both your personal and business challanges with fellow IT employees. Thanks again for letting us into your past world experience and allowing us to connect with that moment in time.

paradisewebdesigntx
paradisewebdesigntx

Great little article that must have been tough to write. I like that you kept it from getting too down with a line that I just had to giggle at "I wonder if the BIOS knows what day today is."

Englebert
Englebert

What a poignant description of the event. Thank you Bob for expressing yourself and passing a powerful message to us all

Mark.fancourt
Mark.fancourt

Thanks for the reminder that what we think is important often isn't. Companies continue and data can be rebuilt or simply gathered again. People can't. It puts into context disaster recovery and that whilst important it's not that important in the grand scheme of things.

SAStarling
SAStarling

...and the reminder about what is truly important. God bless you and my prayers go out to everyone who lost someone on that awful day.

peege3
peege3

I bet you were frustrated! Did you ever share that information with law enforcement?

Timwateru
Timwateru

Thanks for the message. This is reminds me of how thankful God is and for all the knowledge and wisdom he has bestowed upon us. For all those who had lost their lives. Rest in Peace. God bless us all.

daflamingo
daflamingo

It's never tasteless to remember those that died, whether for our country or as inocent victims. "Such commemorations" remind us to be respectful of others and for their loss. It reminds us to be thankful for our freedoms, our lives, and our nation, and especially for those that have sacrificed since 9/11 so we can again feel safe. And no, you didn't "just have to say it."

jkameleon
jkameleon

You lost your freedoms, and you only feel safe. You traded your liberty for the false sense of security.

JamesRL
JamesRL

Whether or not one agrees with the wars that followed, the victims with 9/11 were not responsible for the madness that drove the terrorists to committ the atrocities. Their innocence is in no way diminshed by what happened afterwards.

jkameleon
jkameleon

There is a huge difference between a freedom fighter and terrorist. Freedom fight is public and transparent, goal of a struggle are well defined. Significant part of population is involved. Terrorism is individual and secret. Ends are usually exactly the opposite of what the terrorists think they are. Namely, members of secret organization can never know for sure where their orders are coming from. Consequently, deception and machination is the rule. Before you start thinking about the future of terrorism, take a look at its past. What America is going through now, Europe already went through in the 1970s. There were number of terrorist groups on the rampage, Red Brigades, Red Army Fraction, Baader-Meinhof, you name it. Bombs going on left and right, panic, wanted posters offering huge rewards, innocent people gunned down by police. The whole thing culminated in Bologna train station bombing in 1980. Despite of being the worst terrorist attack on the European soil ever, it's not mentioned at all nowadays. The reason: It was a false flag. The MO was similar to recent London and Madrid bombings. It happened just before the elections, mass media were quick to blame a communists, and communists lost the elections. Later it turned out, that the attack was carried out by CIA/MI6/SISMI backed neofascists. Further investigation uncovered, that ALL of the 1970s terrorism was false flag. Not one terrorist group acted on its own, they were ALL directly cotrolled and manipulated by various western intelligence agencies. There was even a name for this: "Strategy of Tension". The Italian part od "Strategy of Tension" was called "Operation Gladio", which is well researched & documented. The following documentary is long, dry, boring, but it offers very good insight into the inner workings of terrorist organizations: http://www.youtube.com/watch?v=7fB6nViwJcM As a consequence of all this, European Parliament adopted the Resolution on Gladio ... http://en.wikisource.org/wiki/European_Parliament_resolution_on_Gladio ... which basically says: "If you want to get rid of terrorism, get your spooks in line!" This counter terrorism strategy worked quite well during the 1990s. Alas, we forgot this lesson of our history, and now we are doomed to repeat it- again.

Joanne Lowery
Joanne Lowery

As has happened all around the world, today's terrorist is tomorrows' freedom fighter, next decades politician and in twenty years time the head of state. You only need to look at Isreal during the British mandate to see the progression. Those classed as terrorists for murdering arabs, soldiers and guerilla based destruction back then are now the leaders of the Isreali state. If Isreal is too contraversial look at South Africa, Kenya, Chile, India etc. Their freedom fighters were all once classified as terrorists. We all need to be aware that the creed of "The enemy of my enemy is my friend" (Afganistan, Iraq, Pakistan etc) will bite you in the rear in the future.

jkameleon
jkameleon

War is. Terrorism is usually fabricated, a PR stunt. Bombs and patsies never come with return address. Killing people for a lie is a pretty outrageous idea, but it was done many times before, and willingly forgotten as well. People planning a war, which is going to kill millions, certainly won't care about a couple of thousand. Considering the neighbourhood I'm living in, I have a pretty strong feelings about war. I'm particularily disgusted by lies and propaganda leading to it. It's always the same.

sboverie
sboverie

In the 10 years since then, we have done things that we should not have done. For now, we should remember the events of 9/11 and know that we really are not protected from terrorists by the wide oceans that surround us.

jkameleon
jkameleon

My alleged cynicism doesn't matter. It doesn't matter how you look at things. What matters are people in power, people who make the decisions. They definitely lost sight of humanity, and they are 100 times more cynical than I ever hope to be cynical. They are the one who made the 9/11 victims an asset, not me.

JamesRL
JamesRL

The people lost in 9/11 will always be considered the innocent victims of terror to me. Time or historical events do not diminish that fact. I don't look at them cynically as political capital. They may be that to many cynical people, but they are also the friends and families of real live people. I used the term guilt because you used the term innocence....

jkameleon
jkameleon

> There are valid, rational and moral reasons to go to war. Real reasons for war are closely guarded military secret ever since the introduction of Bellum iustum. They are never stated publically. A bovis stercus called "Casus belli" is touted instead. Casus belli is either a blatant lie, gross exaggeration, or, in the best case, fabrication or provocation. Regardless of that, the duty of a true patriot is to pretend to believe it, and call crazy everybody who doesn't. Well, and, 9/11 is a casus belli. You can't argue with that. > 9/11 was not the reichstag fire. Looks like a duck, walks like a duck, quaks like a duck...

JamesRL
JamesRL

"*Everybody who talks about principles and moral in the context of politics and war is a crook and a liar. There is no moral in politics, and especially not in war. Therefore, there's nothing to condemn" Not even remotely true. There are valid, rational and moral reasons to go to war. If a country is attacked it has the right to defend itself. As for the conduct of soldiers and leaders in a war, there are war crimes and they can and should be punished. There is a difference between shooting at a soldier whose weapon is pointed at you, and shooting at an unarmed civilian. "Osama Bin Laden is irrelevant. First off, it's very questionable whether he was behind the attacks at all, and even if he was, it doesn't matter. The very moment 9/11 was used as a pretext for war and erosion of civil liberties, it became, for all practical purposes, a Reichstag fire. It doesn't matter who did it anymore. Not even historians care whether Marinus van der Lubbe was really the arsonist or not" 9/11 was not the reichstag fire. One could argue that at a certain point in the rise of the Nazi movement that their takeover of Germany was inevitable, if it hadn't been the Reichstag fire it would have been something else, and the end result would have been the same. But 9/11 was a large scale direct attack, a Pearl Harbour and it needed a response. There had been previous attacks, but they had not triggered the response that Al Qaeda was wanting. To not respond would invite more attacks and increase the loss of innocent life. The erosion of civil liberties is sad to be sure. But it wasn't a consquence of dealing with terror not the reason for war.

jkameleon
jkameleon

Once the 9/11 victims were utilized as a war pretext, they can no longer be cosidered dead people. They were dehumanized, they became political & propagandistic asset. It's wrong, I know, but that's how it is. If you have to blame someone, blame those who used them this way. As an asset, they have value. With every new war, with every new innocent collateral victim, their value diminishes. It's a depreciation of assets, like a car for example. The greater the mileage, the lower the value.

JamesRL
JamesRL

Seriously. As far as I am aware, none of the dead were actively planning wars in the middle east or Afghanistan when the acts of terror took their lives. Ansu is right, you are looking at this in a relativistic method. But the death of a father/mother/husband/wife/child/friend etc. is NOT realtive to those who suffer the lose and feel the grief. I don't dimiminish the deaths of those innocents killed by stray bombs in Afghanistan, be they innocent civilians or allied soldiers killed by friendly fire. Neither would I diminish the deaths of those who died on 9/11.

jkameleon
jkameleon

* Everybody who talks about principles and moral in the context of politics and war is a crook and a liar. There is no moral in politics, and especially not in war. Therefore, there's nothing to condemn. * Osama Bin Laden is irrelevant. First off, it's very questionable whether he was behind the attacks at all, and even if he was, it doesn't matter. The very moment 9/11 was used as a pretext for war and erosion of civil liberties, it became, for all practical purposes, a Reichstag fire. It doesn't matter who did it anymore. Not even historians care whether Marinus van der Lubbe was really the arsonist or not.

AnsuGisalas
AnsuGisalas

I am fully capable of condemning in my heart both the offender and the would-be avenger. Isolated matters do not become intermixed by way of the overt associations that politicians may desire to create. If we look at it with your glasses: That the wars that followed were planned and just awaited an excuse - that makes it a political lie that there ever was a connection between the despicable attacks on sept. 11 and the despicable haliburton-sponsored excursions that followed. I don't know about you, but I try my damndest never to let political lies influence me. I feel that Osama Bin Laden and his cronies are criminals against humanity, traitors against Islam and generally a waste of clean air. I also feel that the neo-friedmanesque industries that would instigate wars in order to increase their own profits are a travesty, criminals against humanity and traitors against the free democracies they sink their roots into. No difficulties consolidating those views whatsoever.

jkameleon
jkameleon

... their innocence IS diminished. Collateral damage is also innocent.