IT Employment

Ask something
Discussion Locked

Reasonable growth rate: data on Servers

0 helpful
22 replies

by nzwaneveld

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

What is a reasonable rate of data (normal office applications) on file servers?
How do you define 'reasonable'?

I am trying to determine if our rate of data growth on file servers is reasonable and whether the current data storage policy needs to be reviewed.

I am therefore looking for comments / benchmarks on what a reasonable rate of growth is and how one determines what is reasonable. I expect that a reasonable rate of growth will vary according to industry segment. I feel that it will be easier to distill a generic answer if the answers focus on the data used in normal office applications (i.e. word processing, spreadsheet and presentation).

I am intentionally posting this question as a discussion because I expect that there is no single correct answer and I also expect that many others are also concerned with rate of growth of data on their file servers.

22 replies

by soutlaw

I have found that Storage Area Networks are the best way to manage data storage. When you're not sure if you want to get rid of any of your data and your storage needs are constantly fluctuating, a SAN is the best way to handle the situation. Withsingle hard drives able to hold 180GB of data, the size of the SAN array has decreased as well. You also have data transfer rates of 160 mb/s so a large number of clients accessing the data will not affect performance. Check it out!

by mlmassociates

I agree that Storage Area Networks are the most flexible, scalable method of managing data storage. However, throwing DASD at your storage problem isn't always the answer. Storing large amounts of data brings with it various maintenance headaches such as backup/restore processes and general organization challenges. How many times have you gotten the call "I KNOW I stored this on the network, but I can't FIND it." This is particularly true when your users have the perception that space is unlimited. So, in answer to one of the original poster's questions - although we do have a SAN (with capacity up to 5 TB) we do NOT allow unlimited growth of file server data. We have policies in place about how long a file may remain on the file server without activity. Those timelines differ by department as needs change. For example, our Informatics department may need access to raw data files for up to two full years. However, our Operations department only needs access to certain image files for 3 months. Additionally, we have fairly strict e-mail storage policies with automated e-mail reminders when users get too close to their designated space limits. These, too, are adjusted by department.

The term 'reasonable' is driven by true business need. We have found that a combination of anecdotal evidence "we always need access to XYZ type of file" combined with some straight-ahead file access analysis "when was XYZ file really used?" gives the most realistic picture.

by glatiak

'Reasonable' depends upon an understanding of the business activities that resulted in storage utilization changes. In all likelihood, no one ever purges documents -- so the increased utilization is probably a direct result of ongoing email and other departmental activities. It may be useful to correlate the rate of change (first derivative) with business indicators -- sales volumes, customer service statistics, whatever is relevant. These factors are normally tightly coupled. You will want to understand what the historic value of existing stored objects is as well. You should know enough to predict future usage as well as understand the business impact of differing storage management approaches. By understanding how business activity relates to storage usage you will be able to use business forecasts to anticipate usage (to manage proactively). It is very difficult to determine 'reasonable' when the drivers for the storage-using activity are not included in the analysis.

by theviper

Your users are probably keeping many copies of the same document, or sending documents via email instead of sending just links to originals.
Document versioning and collaboration might help alleviate some of your issues.
Those mean training for and dependence on end users.
Ultimately, you'll probably still have to resort to setting storage limits based on volume, date, usage, etc.
Lots of luck.

by crypt01inguist

What I mean is, it is literally NOT their job to care how much they are generating in terms of file usage. It's an organizational problem, and specifically a problem for IT, and should be addressed as such.
After two years handling backups for amidwestern state government, and watching data volume double with no end in sight, I got wise and started analyzing WHAT was being stored, and not just how much. Like TheViper said, there were lots of copies of the same document, not just between users but in different folders belonging to the same user!
I started with the backup logs, but soon realized that they didn't offer the detail and search capabilities I needed. I looked through industry pubs and found several products that were aimed at this issue, allowing me to run searches by name/size/file extension/etc, and produce reports to demonstrate the problem to my boss. THAT was key - getting the boss to realize how bad the server space situation was, and why. I downloaded trial versions and gave them each a try.
A week of playing with the software let me go to the boss and show her that there were three issues we could address: Unauthorized files, Duplicate files, and Orphan files. The first, Unauthorized Files, consistedofthings like .mp3 collections and .mp3 servers running on our file servers, internet joke files and executables, and games. The second, Duplicate files, was the multiple copies of documents in shared and personal folders. The last, Orphan files, included files not accessed for months or (!)years, and also software that should have been loaded locally, but ended up on the server instead, such as clip art collections.

by crypt01inguist

We discussed the problem, and she then called in several people from other teams in the organization, including the Director(big boss). We drafted a policy before making any changes, and briefed the Help Desk, since they would get any calls if things went wrong.
What we decided was:
(1)Limit user action to getting their permission to make changes, rather than have them do it and create new problems.
(2)KEEP IT PROFESSIONAL - no blame, ever, even if their files are bringing down the server. Putting this in writing made helped the IT staff (mostly me) stay focused when taking to the users.
(3)Keep the problem simple - focus on servers, not desktops. In fact, in dealing with Unauthorized files we suggested that users should keep their files on their local machine. (The desktops were fairly new, with big harddrives, and the network was fast enough they could share files to their heart's content. In the end, most just pulled the files completely.)
(4)Handle users individually,not en masse. I ran reports, and shared them with my team, and we then called each user with excess files and discussed the issue one on one. This limited their embarassment, since no one knew who the space-pigs were. Further, in the case of Duplicate files, they sometimes had no idea that they had several versions of the same file, having done it over time. We were also able to explain how the shared file structure was supposed to work to several users who had never been trained.
(5)Keep good backups! We were backing up the servers anyway, but for all Orphan files we deleted we burned copies to CD, just in case. We also offered that as an option to our users, so long as they weren't burning software without a license (Windows NT, Adobe Acrobat, etc.)
Dealing with our 1000+ users took about a month, and we gained back between 15% and 25% of space used per file server. We were also able to modify our file str

by michael.baumann

I am currently working on a similar project and we need to reduce the amount of data stored and data growth on F&P serververs from something like 8 - 10 % per year to less then 5 %. We work in two areas.
1) business 2) technology
If you can not get data growth in the business area under control, you simply move the problem from this year to next year and the year after. There are several reasons for data growth and we were able to address two of them with our concept. We implement it at themoment and I can tell you next year it it works.

by oldefar

The issues of duplication and unauthorized files, along with the support issue of backups to workstations, are real challanges.

I believe the products from Connect Corporation (http://www.connect.com) are worth a look as you tackle the storage issues. Unfortunately, I can't speak from experience on this.

The product overviews indicate capabilities to limit copies and prior versions, specific file types, etc.

by jastormont

I work for a company that runs a large database and also run 11 servers in our main server room. We have just done a roll out of new Compazsystems and allready we are suffering from no space. One thing we did do with this roll out was to restrict users to a limit of 150 MB of space and this has helped us to maintain our storage needs. We are now looking at purchasing a new file server as our users require more space for thier work.

by nzwaneveld

I can understand restricting users to 150MB but is this a limit that is set by the business management or by an IT department. What if someone / or a department feels that this is an unreasonable limit?

by mckaytech

I think it's going to be hard to draw any generalizations because records retention requirements vary so widely. For example, if your organization becomes involved in litigation or some regulatory action, it is quite common for the opposing side toget a court order that, in its usual form, prohibits any destruction of information.

To give some round numbers from a government agency with about 1500 employees that provides services to about 250,000 individuals, the total data currently on-line is about 1.2 terabyte and growing by about 15% per year. A significant amount of that information must be available in perpetuity (although not necessarily on-line).

It seems to me the first step is to adopt and enforce a legally-reviewed records retention policy and see what that does to the growth rate.

The other is to look at the information being stored and determine the business need for it. In the weeks following the WTC tragedy, we found an awful lot of large files that contained pictures, presentations, etc. related to it on our network. We tried to maintain some sensitivity to a difficult time in our lives and treaded lightly but the fact remains that a significant amount of storage was occupied by data for which we could not establish a business need and eventually we will have to deal with it.

But the politics of storage can be fierce...

paul

by radiic

Whats the point of having a "DISCUSSION CENTER" if we can't discuss things. This is the perfect place for a Conversation like this.

As for the Conversation...Resonable growth of storage depends on document retention and the type of business your company does. I work for a bank and our rules for record retention are very strict and timelined. I know in the local government sector, even e-mails are subject to public disclosure. So every e-mail has to be saved (technically).

I would venture to guess that anything above 40% storage growth/YR. is not acceptable, and would require some digging like one of the posters above. Hell i did some diggin on one of my file servers and found some .mp3's that I didn't have ;)

Rad

by admin

one gigabyte per user for file storage. They are free to use removable media for extra, which many never get to. Much of our data goes into several database's which grow according to the number of employees/contracts we maintain and is added to the fee for service or cost of new employee's.

Many employees don't use the whole gig (or even close) so we guesstimate the actual use every so often, but since it's already budgeted in to the new employee cost, we have extra funds in this barrel.Basically, we just want to have people able to do their jobs, have a little fun, but not be filling our servers with over a gig of mp3's etc. (power users, let them enjoy, but not so much rope they hang :>)

So our growth is directly related and figured by number of employees and number of contracts.

# of employee's x 1 gig = HR database, Payroll Program, % of accounting program etc....

# of contracts x estimate per service = % of accounting program, direct billing programs etc.

by admin

# of employee's x 1 gig = Personal Space

# of employee's x 50mb additional space = HR database, Payroll Program, % of accounting program, e-mail storage etc.....

# of contracts x estimate per service = % of accounting program, direct billing programs etc.

by nzwaneveld

Doe you have a periodic review of the limit? As disc capacity becomes cheaper, and users seem to want to retain more information it seems apparent that this time next year a user may desire to have a 2GB limit. How is the limit determined? Is the net-cost of storage a consideration in the calculation or is it more a gut feeling in determining an acceptable limit of storage.

by admin

Yes, it was 100mb about 5 years ago.

Reviews are conducted as needed. Mainly when people start going over it gives us a chance to see what they are using the space for and if it is cost-effective and productive for the business. I'm sure it will be 5 or more gigs with future needs, but for now, 1 gig seems to be adequate and even generous for most people and lessens practices that would abuse the network. It has been weighted more financially in the past, but now although that is a consideration, it is weighted towards a reasonable amount of space before rampant abuse or negligence takes place.

by atsmar

I'm the IT person at my bank. We recently converted our core banking & document imaging systems in June. Right off the bat I had the vendor configure our systems to be able to store up to three years of data on-line. This was determined by the number of accounts we service, the size (asset-wise) of the bank and our rate of growth per year.

Past that we also installed a system that monitors volume activity on the server hard drive and then burns the volumes to CD automatically. In addition to that we purchased a 700 CD jukebox. It will take us
approximately 5 years to fill up the jukebox.
I feel we are more than prepared to deal with long term storage issues.

As far as our NT network goes we also configured the server to handle 3-5 years of data storage online. Again, that was determined by doing studies of daily/monthly usage.

by srimersing

You made a right judgement upon determining the storage capacity. I, too, have this storage problem, and everytime we hit this problem, we just increase the storage capacity which is not a best solution at all. I believe the main decision comes fromIT/IS key decision maker by looking at these areas:
1. Business needs of the latest available records online: If your company says it needs 3-years data online, the IT figures out the capacity. The rest of the data can be stored on backup media (ina proper structure which can be easily retrieved).. by this you can maintain percentage of storage to a minimal increase.
2. Technology availability: whether it can be expanded and fulfill required needs throughout the company's life. Do remember, technology always changes for the better
3. Disaster Recovery Plan: This will largely determine storage capacity. It might come up double of the existing business needs' storage.
4. Limitation of storage per user: come up with policies which have been agreed by Management Team. Set the standards and controlled them once approved.
5. Communication, knowledge and expertise: do not forget, knowledge is power. Application of various formula (as discussed in here) can be a storage solution. And this is what I'm gonna do to solve my storage problem. Thanx

by bism

Rationed access

A rule of thumb that I use is analyze the last two years growth 1999-2000, 2000-2001 and use the following.

1999
2000 growth = 5gb

2000
2001 growth = 7gb

The most recent growth rate is 7gb/ year and the increase in growth rates is 2gb. Add this together 7gb + 2gb and you have a safe estimate to use as an allowance for next years growth rate. I say allowance because I may go by a 30gb hard drive but only make available 7gb to start. This should let me have just shy of 3 years growth. For 2 years out I would plan on 9gb + 2gb + 1gb (last increase in growth / 2). As I said before this will get me just shy of three years growth, I wouldn't want to spend any money for growth any farther out. I wouldn't make all available at once, as someone else mentioned; people will use as much as you make available. As space becomes used review it and as people to review they're files to delete any unneeded files. Another process that I have used with success is after users review they're files they can submit files that they don't readily need but are afraid to delete. Put this on tape or other archive media.

Good luck
Mike

by dsimpson

Two things - have you considered the priority of your data? How are you storing your data today?

I would have to agree with the statement earlier that a SAN would be a big advantage to managing your data - provided it is the right one and you deploy policy based storage (C:\Documents and Settings\dsimpson\Local Settings\Temporary Internet Files\OLKF\Enterprise Hardware and Client Systems - The future of storage is virtually assured - ZDNet Tech Update.htm) . Second, there are a few simple, affordable software products out there to help with data management/storage management that will help: Check out CommVault and BMC. These products can help you determine which users are eating up the most storage, and how to set policies so their data can be prioritized for more efficient use of your storage resources.

As far as reasonable data growth, I have seen 18-25% overall annual growth with most companies.

This discussion is closed to new replies.