Data Centers

Reasons to keep data in your own data center

Andy Moon says there are still there are more reasons to keep most of your data local than there are to move it to the cloud.

In last week's piece about cloud storage, I discussed some of the reasons why enterprises aren't moving their data in bulk to the cloud. In the related discussion, TechRepublic members brought up some tidbits (particularly about bandwidth) that I hadn't considered because I live in the United States, and most of my cloud connections are right here in North America. Bandwidth isn't a concern for me with regards to how I interact with the cloud; I don't incur per-megabyte costs; and most of the cloud services I use (Gmail, Google Docs, etc.) don't have much latency because the hosts are right here in the States.

But that isn't the case in some parts of the world where bandwidth is not as fast or as cheap as it is in the United States. For instance, if an Australian company had to rely on cloud services based across the Pacific Ocean, latency could cause a lot of headaches. As such, there are still a lot of good reasons to keep your data in a local data center (the private "cloud," according to Oracle's Larry Ellison), and there are more technologies coming that will make this storage better and cheaper.

Seagate released a new version of its high-end hard drive called Savvio 10K.4, which will double the capacity of drives that go in the data center. Though you'll still want to use solid state or fiber channel arrays for tier 1 applications, the new Seagate offerings are well positioned for tier 2 apps. The capacity per platter has been increased to 200 GB, bringing a three platter drive to a whopping 600 GB.

Ultimately, there are many factors to consider, but price is the one that everyone sees on the bottom line. Data storage is not cheap, but it is getting less expensive per megabyte over time. However, no matter how much the $/MB ratio drops, it seems that we are seeing bigger and bigger data storage needs every day. At my last job, we put in a to-disk backup solution that was running out of space within 18 months.

Maybe what we need is a full week of spring data cleaning every year. When I clean out My Documents before winter break every year, I am able to get rid of massive numbers of files that are simply not relevant. Every time I do that, I wonder how much useless data is clogging up my servers just because nobody takes the time to delete files that don't need to be there any more.

Right now, there are more reasons to keep most of your data local than there are to move it to the cloud; but for bulk file storage, the local data center is a better choice. The cost per megabyte keeps going down; there are not as many bandwidth constraints; and at least it feels like we have a better handle on security when the data is local. These dynamics may change eventually, but for now, local storage is king.

TechRepublic's Servers and Storage newsletter, delivered on Monday and Wednesday, offers tips that will help you manage and optimize your data center. Automatically sign up today!

11 comments
The 'G-Man.'
The 'G-Man.'

Sound like 90% of other blogs around here. Mind you, I have a 50Mb line to my house.

Osiyo53
Osiyo53

Ummm, I pretty sure that is not what he was referring to. The bandwidth available to you at your house is only one piece of the equation. The real speed at which you can access and transfer data is also a matter of adding in all the other possible causes of bottle necking and delays between yourself and the source/repository of the data you wish to view or work with ... which are part of the path between yourself and that place. Add in the possible delays caused AT the site where that data actually resides. Such as what is the max bandwidth they have available, how many users of their services are connected at any one time, how large are the file streams being moved back and forth at any particular moment, etc. Specifically, the OP spoke about someone in Australia who might be using a storage service that was located elsewhere ... other than in Australia. No matter what that person's local connection speed was, they'd be limited by the slowest bottleneck between themselves and that data storage location. Now, I quite routinely visit certain Web sites located in Australia (I'm in North America). As there are web sites there maintained by folks in the same line of business I'm in, who have quite useful info I'm interested in. Etc. And there is always a quite noticeable delay upon initial connection before the data streams commence. That is also true of some other places. Even within the same relative geographic locale, one can run into bottlenecks, slowdowns, etc for any variety of reasons. Can be at your own end, at the destination, or most anywhere in between. Recently I had to explain this to my wife. Now, at home I have a much slower service than you evidently do. Originally, up until last year, a 5 mbps cable modem connection. Which was plenty speedy enough to suit me, for what I do at home that requires an internet connection. Then I came home one day to find the wife had called the service provider and upped it to 10 mbps. Okay, fine, if she felt she needed that, no problem for me. I was kinda curious why, what she might be doing, that required the increased speed. Its not as if we watch movies, listen to music, or make phone calls over that Net connection. But, WTH, it was only an added $10 (we're not billed by volume of data, only by speed of the connection). Then one morning I awoke and grabbed coffee and went into the den where we have his and her computers and desks set up and she was grumbling. I won't mention which online service she uses, as it really doesn't matter, but she has one she uses mostly frequently for email, general file storage, etc. This is all personal stuff. She writes and receives emails to and from friends, family and so forth. Plus they swap music, picture, and video files. Rather than download such and storing it on her own system locally, most of it she leaves on the service provider's system where she has made numerous folders in order to keep things somewhat organized. Anyway, she was trying to read emails with largish attachments. Specifically, in this case, a video made by our daughter of her youngest taking her first steps. The video was coming through, but sluggishly, etc. And the wife was trying to also forward it to a number of people. In any event, things weren't going well, and she was getting grumpy. Mumbled that maybe she had been better off signing up for an even faster service. I commented, "Patience, Dear. Let me check to see what is happening." She doesn't actually know anything about computers or networks except for using the apps she uses. So I dropped down to the command line on my system and did a little pinging and this and that. Then informed her it wasn't our connection speed that was the problem. The problem seemed to be at the other end, the service she was using for email, file storage, etc. I don't know exactly what their problem was, wasn't actually all that interested so as to investigate more. Line problems? Too many on line trying to use that service at that moment? Equipment failure? Don't know, don't really care. It happens. I reminded her, once again, that I'd told her before that anything she considered important to her, she should copy to local storage. And from there I'd take care of it as I backup our home machines routinely. On another occasion, I found her grumbling and upon checking determined the fault lay with our ISP. They'd had an equipment failure, and were in the midst of replacing the failed stuff, but in the meantime service was slow, sluggish, on and off again. Etc. Etc. Stuff happens. She's handicapped and is often on some game site, where folks not only play online games, but make friends, engage in online chatter, etc. Took me quite a while to get it through her head that when the times occurred, as they regularly do, when things got real sluggish that it really wasn't an issue with the speed of her own hardware or that of our network connection ... most times. It was a matter of the system providing the game services getting overloaded. As I showed her several times by the simple expedient of having her close down that game room and then opening up another provided by a different service and she'd find everything working fine there. She's now become more adept at figuring out for herself where the origination of such problems are. And, as well, has started to routinely go through all her saved emails, attachments, etc and mirroring them locally ... those she really wants to save ... so that I can back them up at home. She's also finally coming to the realization that MOST times, for what we do over the network, the 10 mbps is more than adequate. That more often than not any excessive delays and so forth originate elsewhere. Now, I'd suppose that if for entertainment purposes we listened to or watched streamed music, movies, or TV over the Internet connection we might want a faster connection but we don't do that. We have regular cable TV, for TV watching. And regular TV's. I have no interest in watching TV on my computer screen. If I'm on the computer, I'm doing other stuff, a Window with a TV show on it would just get in the way and be distracting. Same for movies. Neither of us watches just all that much TV or that many movies. And when we do we tend to prefer to get comfy in the recliners, look at a much larger screen, and to concentrate on the show. Or, why bother? Phone service, for ourselves, I keep separate. We have regular copper line service plus each has his/her own cell phones. My wife wondered why I insisted on keeping the copper line along with the cell phones. Nowadays she understands. Cell service had routine and regular interruptions caused by any number of factors. From local system overloading, to cell towers going down due to weather storm, or whatever. Copper lines go down to, but much less frequently. But the odds of BOTH being down simultaneously are minor. As I live in an area where major storms ... tornado, blizzard, ice storm, etc are just a fact of life ... I prefer to have a backup plan/ability.

l_e_cox
l_e_cox

I'm not an expert on legal, but saving storage space by deleting files may not be an option for everyone due to either legal or in-house policy requirements. I know one place I worked had to regularly add space because in-house policies required that no data be thrown away or archived. In the case of legal requirements preventing deletion of "old" on "unneeded" documents, that could possibly be handled by archiving them onto a cheaper storage medium. The trend, though, seems to be to keep as much data as possible actively accessible. This doesn't in any way invalidate the other arguments against out-sourcing server space. As things stand, I think it is a lot better for an organization to keep its data under its own roof.

paulenet
paulenet

Deleting "old" files is not always an "across the board" scenario, especially for companies that must adhere to strict HIPAA requirements. One of the previous companies I worked for was a healthcare company that definitely fell into this scenario. We had to maintain and protect customer PII data. However, the problem with uncontrolled data sprall or data growth was not actually with the PII data. Not only that, just taking snapshots of massive PII data, the actual storage requirements for such data is most often much smaller than most other data (of course, depends on how and in what format the PII data is stored.) Not surprisingly, since there are strict policies and procedures regarding to storage, backup, security, accessibility, and HIPAA compliance, the storage footprint for PII data is most often considerably smaller and more manageable than most other typical types of data. In fact, the storage problems most often are encountered with other data ranging from general office documents, project plans, PowerPoint slide decks and other graphics or multimedia files, software install bits, and of course, like the example I gave earlier, often-challenging storage management with source control systems for dev teams, such as SourceSafe, Team Foundation Server, and especially others such as PerfForce and ClearCase. All these different file storage needs, compliance requirements, backup and recovery needs, all can be a storage mishmash if not managed well. This really underscores the importance of having hired IT talent that understands the needs of the business, its end users, business compliance, availability requirements, as well as the general access trends of the different types of data. Like you pointed out, it is much better for businesses to keep data under their own roof. If no one in-house has an adequate understanding of the needs, requirements, and data availability, then clouds definitely will not help the situation. In fact, it will make it far worse. On the other hand, if internal engineers have a good grasp on the different types of data, storage footprint, etc., that a specific business typically uses, implements adequate file management policies and procedures for end users, then it is highly unlikely that when this scenario is combined with high-performance, highly-accessible, scalable, yet low cost storage SANs that are increasingly becoming available today, that clouds would be of any significant value anyway. This essentially means that regardless of the type of data a business maintains, it's storage needs, or how effective and efficient the data is managed (or not), the use of clouds will provide zero value. Clouds will actually be more costly, increase exposure risks, and experience significant performance and availability issues, which further decreases office productivity and increases operational costs. What is amazing to me is that there are some IT folks and business leaders that cannot get their head around these fundamental facts. I think of cloud computing sort of as a "SaaS Gone Wild". There are some external products and services that work great for businesses under a SaaS model. However, when someone gets the bright idea to offer SaaS-based solutions for company IT infrastructures and data storage, it is just asking for severe, costly, and highly-unecessary trouble for ANY company with IT needs. Therefore, it really doesn't matter how large or small a company's IT needs are, or how effective (or ineffective) IT architects or systems engineers are managing the IT infrastructure and it's intellectual property, as clouds easily can and should be completely avoided.

net.minder
net.minder

In our case, the data we store almost always belongs to our clients. The contracts we agree to with our clients almost always force us to be responsible and accountable for the security of the data while it is in our hands. So there's no question really - cloud storage is impossible because then we would be accountable for security breaches that a cloud provider would have. But aside from that, there are plenty of software products we use that can only store and access files on a mapped drive-letter or UNC path. Cloud products, and even SharePoint won't work. To combat the problem of unnecessary file storage, get a product that moves the least-recently-used files to a secondary storage array, replacing the file with a shortcut or a stub. Arkivio or Rainfinity can automatically migrate files that haven't been accessed in say, 6 months to "near-line" storage. So the important files can be close to the user in each of your offices. The rarely-used ones can be far away at your main datacentre, and there will just be a brief delay when opening them. The secondary storage array doesn't require frequent backups. Maybe none, if older data is not deemed crucial in your company. (e.g. old sales quote files are usually not keepers.) So yes, I agree. Private cloud systems are good, and a tiered approach can save money and effort. - Netminder

tbostwick
tbostwick

Great point - now how do we get that message out to 'Joe-user'? In another article from TR today, they alluded to BB's and a renaissance - I vote NO to "cloud" when it comes to phones and my data. Keep it local, accessible and on my own "secure" network and servers | Please and | thank you. What will happen when the day comes and every iPhone and Android user says 'the sky is falling'? - the phone is dead, and all those wonderful apps leave them stranded in some foreign Starbuck's somewhere. ..because it will happen, just a matter of when.

steven.taylor
steven.taylor

The spring cleaning is a great idea, and one I had thought of last spring, but never implemented. Even in a small company (less than 100) employees, my latest file server report shows about 57 GB in just duplicate files. Imagine what that would be in a company with 800 or 1,8000 employees. Just last year we had to add 2 TB to our backup system, but fortunately the cost has come down and with the new SATA drives, we were able to implement that for less than $500.

Andy J. Moon
Andy J. Moon

...has over 5005 users, some of whom are working with massive AutoCAD files, video editing, and other applications that store massive amounts of data. My last job, the entire full time staff and faculty added up to less than 200 and I was amazed at how quickly those people could fill up all storage space that was given to them.

paulenet
paulenet

Cloud Computing = Clown Computing Most serious businesses will not leverage cloud computing for the long haul. Most companies will fall back to more traditional data center solutions (if they haven't already), especially if they get a full grasp of their actual IT challenges and hire competent IT staff to manage them. A popular position in today's economic times is that companies feel that hardware resources and / or IT staff are too expensive. The irony that it proves out in most cases over the long haul to be the exact opposite. Companies should be implementing adequate hardware solutions and hiring IT talent. For the overhwhelming majority of companies, there simply is no cost-effective alternative to hiring good IT talent to manage issues in which some people currently hold the popular belief that, storage problems (or any other IT specific problems for that matter), are going to be solved by cloud computing. Cloud computing does not eliminate storage issues. Every organization that finds themselves continuously running short on disk space, especially after multiple storage expansion initiatives, definitely needs to evaluate their storage usage, and perhaps implement more appropriate policy changes to inforce periodic cleaning. A classic example of this is with source control systems, where people do not manage folders or versions well, and they end up storing EVERYTHING, including files that are irrelevant or no longer used. Over the course of 20 years in IT, nearly every IT shop I worked in either had inefficient data storage policies and / or end users with nightmarish storage habits, impacting the IT enterprise somewhere. Useless and / or redundant files are often found everywhere. None of these issues will be solved by cloud computing. Server performance drops dramatically, increase risks of drive corruption problems, user productivity typically drops (if not completely interrupted), backup processes become bogged down, and disaster recovery becomes highly impractical, undependable, if not an outright joke. To repeat, none of these issues will be solved by cloud computing. At the same time, cloud computing actually introduces numerous other problems. I don't care how "secure" cloud vendors say their environment is, it still does not change at least one thing: Your corporate data is stored in another company's data center. Unless you do not have any intellectual property or other sensitive data, there is absolutely no reason to use a cloud that trumps the security risks in using them. It just isn't worth it. Why do this when it is easily avoidable and moreover, completely unnecessary? Contrary to some cloud computing cheerleader's claims, storage costs ARE rapidly going down. Nevertheless, it doesn't matter if all the disks in the world are free if an organization does not know how to effectively manage their data. It is that simple. Not so long ago, most found it fairly expensive to upgrade to mainstream 500 Gig drives. Today, I am running a 2 TB SAN, and one 6 TB SAN, all consisting of 10 1TB enterprise drives in a SAS configuration, in a single, 5U host, and this is ONE of my servers AT HOME! My solution to data storage issues? It really is very simple: 1. Address your in-house data center challenges in-house, otherwise you are just sharing your problems, AND your important data and storage locations with multiple other companies. 2. Hire systems engineers or IT architects that have solid real-world storage management and storage expansion experience, and put them in positions that have adequate authority and autonomy to implement reliable, high-performance, and scalable storage solutions. Equally important, is for such individuals to understand the end users, have solid understand of the applications and resources the end users use (read: customers), and nature of their data storage requirements. 3. Implement storage polices that prevent unnecessary or inefficient data storage incidents from happening in the first place. Doing so will help IT organizations think and act with a clear head, allowing them to also realize that cloud computing is an overhyped fad at best, disconnected from serious business needs and realities, and will completely avoid them. The potential for storage issues are perpetual. Storage demands need to be continually managed, and storage usage must be continually monitored. IT shops that do not fully understand these issues will continue to experience costly, and avoidable problems. Just like with other IT resources, companies need to keep focus on maintaining high performance and high availability.

Andy J. Moon
Andy J. Moon

We are constantly having to upgrade and add to our storage capacity. One of these days, when virtual storage is more ubiquitous, we might even utilize a significant portion of the capacity we have, but right now we have some storage that is crammed to capacity while other servers sit on partitions with hundreds of free gigabytes. I go through my files on at least a yearly basis to clean up (I get time off every December, so I do it before the winter break). We are moving some data around so that we can stop relying on a server that is OOP and nobody ever seems to have the time to look through their data to make sure that everything they have stored is necessary. Of course, the fact that nobody wants to do things like this means that we have massive growth in the amounts of data we store even though I know that only a fraction of it is useful, relevant data. What has your experience been in this area?

ozchorlton
ozchorlton

Another thing, about Australia, besides the lag, is that connections, to the net, are usually changed per MB, (or you get an allowance, per month - then you pay per MB).