Data Centers

Is it time to get humans out of data centers?

Sun's former CEO Vinod Khosla believes humans working in data centers are bad business. The noted venture capitalist may be right, but for the wrong reason.

data-center-image-from-stockxchng.jpg
 Image: stock.xchng

Vinod Khosla, former Sun Microsystems CEO and now partner of Khosla Ventures, is known as a controversial figure, and he kept that reputation alive at this year's Gigaom Structure conference in San Francisco. While talking to Gigaom executive editor Tom Krazit, Khosla said, "It's ridiculous to have humans manage the level of complexity that we have humans manage inside the data center." Khosla added that humans are a big cost in IT, and that they should be removed from the equation.

Before you decide whether Khosla is right or not, let's look at what he is talking about.

Too complex?

It's hard to deny that a data center is complex. Every so often, I read about a worker in a data center accidently doing something that completely shuts down the facility. TechRepublic contributing writer Jack Wallen even compiled a list of 10 stupid things people do in their data centers.

A quote by Wallen seems appropriate for this discussion. He said, "We've all done it -- made that stupid mistake and hoped nobody saw it, prayed that it wouldn't have an adverse effect on the systems or the network. And it's usually okay, so long as the mistake didn't happen in the data center."

My favorite of Wallen's 10 mistakes is midnight massacre, where someone working an all-nighter in a data center is sucking down caffeine to stay awake and ends up doing more harm than good. It's a scenario that I am willing to bet most IT professionals understand.

Replace humans with automation

When talking with Krazit, Khosla gave credit to Google for driving automation in data centers. In his Data Center Knowledge article about Khosla's interview, author Yevgeniy Sverdlik states that Google is using tools including Omega to automate management of its global data-center infrastructure. Khosla also feels that Mesos, an Apache cluster manager, is approaching the same goal.

To make his point about Google inventing new automated hardware, Khosla said to Krazit, "I don't think Google talks to Cisco about getting its networking gear. I don't think they even look at those product lines."

Facebook, Rackspace, and other companies in the data center business are looking at ways to automate hardware management of their data centers using designs from the Open Compute Project.

Aligning automation with reality

There may be good reasons to believe automation will take over more tasks, but a reality assessment must be made to assure that automation improves the situation. Automating an already complex environment is not trivial.

Paul Venezia in an InfoWorld post speaks highly of efforts to automate the data center. He also urges caution -- what if something happens in an automated data center, and there is no one there to fix the problem? Venezia sounds like he's been in similar situations and prefers not to repeat them.

A different reason to keep humans out

I read another interesting blog post on the Data Center Knowledge site that noted humans might be forced out of the raised-floor portions of data centers sooner rather than later. New studies show that servers, which were once thought to perform best when the temperature is 68 to 72 degrees F, can handle temperatures that are approaching 95 degrees F. In the article, author Rich Miller writes:

Studies by Intel and Microsoft showed that most servers do fine with higher temperatures and outside air, easing fears about higher hardware failure rates. Dell recently said it would warranty its servers to operate in environments as warm as 45 degrees C (115 degrees F).

OSHA states that the Maximum Wet Bulb Globe Temperature (i.e., the measure of heat stress) in a work environment is:

  • Continuous: 86 degrees F / 30 degrees C
  • 25% work / 75% rest: 90 degrees F / 32 degrees C

If that is the case, there will be all sorts of number crunching to see where money can be saved, as energy used for cooling is a significant data-center expense.

Last thoughts

Khosla's comments may be timely, but for a different reason than he originally thought.

Finally, if you work in a data center, do not feel bad about Khosla stating that data centers are too complex for humans. Khosla says in this TechCrunch post that he wants to replace doctors with automated processes.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

12 comments
techrepublic
techrepublic

Humans are extremely flexible. Automating things to replace humans is a formidable task; self-driving cars comes to mind. I think many of these ideas will go the way of the "paperless" office.  Also, regardless of how sophisticated our robotic exploration of the solar system is, robots will never completely replace humans on the ground.

kquinnIDC
kquinnIDC

As an aside, I have to say the author may be on to something in "Last Thoughts" when you consider the quality and cost of healthcare today...

TRgscratch
TRgscratch

To be fair, we're also eliminating the people in venture capital via crowd-sourcing

pjboyles
pjboyles

Wrong conclusion driven by poor assumptions and a poorly stated problem.  Automation can only handle what you can foresee and is only as good as the person who programed it.


There will ALWAYS need to be "humans" running the data center.


What needs to be fixed is the culture where the human needs to pull the "all nighter" to do the work.  Appropriate staffing would alleviate this issue.  Unfortunately the penny counters are penny wise and pound foolish.  They need the old hand and the new kid.  Without this pairing knowledge doesn't get passed along and the new kid doesn't get the time to learn or the experience required to fix it when things go wrong.


The race to make IT a generic service like ordering a buffet just doesn't work in most instances.  Oh things can limp along for a while but then you have the eventual melt down that costs more than doing it right.  I see it in many outsourcing efforts every day.


Adding appropriate automation tools is simply a good practice.  The automation can prevent certain classes of mistakes. But it takes talented "humans" to implement, configure, manage, maintain, update for changing requirements and run that automation package.  And it takes humans to understand when the predefined automation choices are not going to work for the specific situation they face.  It takes talented humans to fix things when the automation breaks things.  Those talented humans need to be involved in the day to day operations so they know everything that happens and has the experience required to understand what they are seeing.  Without that the intimate level of knowledge, resolving issues when they happen will take much longer and are more likely to cause other issues later.


And "some guy" with a minimum of knowledge and capability paid a pittance to push the automation buttons is not who you want fixing the systems when problems happen.  Oh they can fill in for some of the day to day operations work, but not all of the work.


You need skilled, knowledgeable people who know what happens at that data center to run, manage, maintain and review everything that happens at the data center.  This level of expertise can see potential issues as they develop and hopefully prevent an outage. 


You need someone able to react quickly and correctly when those depending on the data center are screaming for blood because there is an outage.  They need to be able to quickly diagnose issues and resolve them which takes those intangibles gained from the intimate involvement in setting up, maintain, expanding, changing to meet new needs that just can't be gained in any other way.

bsalloum
bsalloum

"Humans are a big cost in IT." Yep, that's the nub of it. Just another cost in the equation, to be eliminated where possible. Unfortunately, we've been headed down this road for quite some time now, and not just in IT.

tkensc1
tkensc1

The same logic could be applied to eliminating humans from the role of CEO. I wonder how well that idea would be received?

pgit
pgit

When does "eliminating people" start to include the pine box?

Vallum Halo
Vallum Halo

@pjboyles Unfortunately the penny counters are penny wise and pound foolish.


The root cause for a lot of problems, as you state above. Accountants (and a lot of managers) fail to understand that these data centers are the MOST COMPLEX artifacts of humanity, and a lot can go wrong.


Anyone who has doubts about that, take a quick look at the data breach at Target,


This is why I worry about nuclear power plants.

Michael Kassner
Michael Kassner

@pgit


Good point. I suspect the raising of operating temperatures will have the biggest influence, just from the obvious savings. 

pgit
pgit

@Michael Kassner

I suppose you could run your enclosures (currently human-oriented buildings) within a few percent of the CPU temp safely, so long as you maintain a generally constant outward flow.

Or better yet, use the heat to generate electricity on site. So long as you maintain CPU temps whatever distance to reach a steam generator you could run turbines! With a closed-loop water supply there goes all the cooling you really need. All you'd need to do is run the CPU temps up a bit and/or lower the boiling point of whatever fluid you use for the necessary phase changes.

My stupid 6 year old core 2 duo runs within 45-50 degrees of the sea level boiling point of water regularly.

Editor's Picks