Why didn't the chicken cross the road? Because it didn’t have an invoice saying it could go anywhere. For some, it's just another variation on an old joke. But if you’re in the poultry business handling a highly perishable commodity, missing invoices are no laughing matter. And when you’re a $1.5 billion company like Livingston, CA-based Foster Farms, the situation can threaten to undermine your entire operation.
On a weekly basis, the IT folks at Foster Farms were running around trying to solve storage space issues that were causing their file servers to crash. "If you run out of space on one of our file servers, e-mail stops, printing stops, production stops," said Joseph Mann, senior system engineer for Foster Farms. As a fresh meat company, the ongoing cycle of outages was unacceptable. So why was space becoming such a problem?
Like a good crime scene investigator, Mann followed the clues to the smoking gun. In this case, he discovered, "we were running out of space on our servers because users were filling them up with junk." Running an analysis on users’ home folders, Mann found thousands of mp3, .jpg, and wave files clogging up file servers to the point where there wasn’t even enough disk space to spool files to printers. "If you can’t spool that raw file because there’s no space," explained Mann, "production goes down because you can’t print invoices to put chickens on trucks." According to Mann, printer stoppage in the main shipping department can cost Foster Farms upwards of $30,000 an hour.
The search for a solution
The company could have simply replaced servers and added drives to increase capacity. But, as Mann stated, "we’re in a huge budget crunch because there’s a glut of meat on the market. Prices are down. The economy is down." Absorbing the capital expense of new servers and drives just wasn’t an option.
So Mann searched for an inexpensive way to increase uptime without increasing his administrative workload. When he posed the problem to one of the Internet newsgroups he belonged to, his colleagues suggested imposing a storage quota on each user. Mann conducted an impromptu poll of about 400 people in the newsgroup to find out which vendor they’d recommend. According to Mann, nearly all of them suggested giving Quota Server from Tampa, FL-based Northern Parklife, Inc. a look. According to Mann's peers, Quota Server would give him the centralized control he needed to allocate storage space throughout his WAN at a price he could afford.
The criteria checklist
Mann’s wish list for the storage solution was pretty specific:
- No additional workload
- No agents installed on user machines
- Reliable support
- Small price tag
Northern’s Quota Server gave Mann the storage allocation control he needed and then some. "My biggest concern was to not add to our administrative workload," said Mann. With only five systems engineers servicing the entire billion-dollar-plus operation, the IT staff was already stretched pretty thin. With Quota Server, Mann could install the software in about 10 minutes and apply quotas to individual users in about five. "What absolutely sold me is that once I installed it, I didn’t need to monitor it. I didn’t need to administer it. I didn’t need to touch it. It’s a very, very hands-off product."
Because its operations are spread across several remote rural communities, and biocontamination protocols restrict IT’s access to many of the sites, Foster Farms needed an enterprise-wide storage management system that could be installed and managed at corporate headquarters. "There’s no way I can get to some of our more rural sites, and we don’t have the technology people back there to do it for me," explained Mann. With Quota Server, Mann sits at his laptop in Livingston and tells the software to install itself on whichever server on the WAN is next in the queue. Within minutes, depending on the bandwidth at that site, he’s ready to set quotas.
"Quota Server contains a great feature called Auto Dir Quotas that lets you set the top-level hierarchy in the directory, and it is inherited down to the individual personal folders," said Mann. The Storage Policy Wizard lets him quickly establish a set of user-focused rules—setting quote size, threshold levels, notification types and texts, even imposing file-blocking measures to prevent unwanted file types from being saved. Mann can then modify rules for individual folders as needed or turn off the restrictions on an individual folder altogether. Quota Server even produces reports on a regular cycle or on-demand to let Mann see who is reaching the limits of his or her quota.
The $1,500 per license price tag had Mann a little worried. "It wasn’t in the $30,000 to $50,000 range I was expecting," he said, which made him concerned about how reliable a product he might be getting and what kind of support he could expect. A rigorous trial run proved his concerns were unfounded.
Exceptional documentation and support
Northern didn’t skimp on documentation. It sent Mann a CD and a professionally printed manual securely housed in a hard-to-lose plastic case. As for the caliber of the documentation itself, "I think the documentation was one of the best surprises that I received," stated Mann.
He bragged about the support he’s received as well. "I just want to emphasize the support department because that’s a big factor in my world, especially when you have chicken ready to be processed." During his lab test, Mann intentionally broke the system and called Northern for help. The first time, the fix was trivial, but the second time he had broken the system a little too well. "I really screwed up," he said, "and they stuck with me and kept calling me back to ask how it was going. It was fantastic support. And I don’t think I was getting the support because I’m me [someone from a $1.5 billion company]. I talked to three other customers of theirs, and they all said the same thing: You call them and you always get someone who knows what he’s talking about."
Up and running in 10 mouse clicks
Support aside, Mann found the product extremely easy to install. "It was really trivial, something like 10 mouse clicks." In those few clicks, he set up a 250-MB quota per individual home folder for approximately 360 users on a single production server. To discourage overloading the server with junk, he also blocked mp3, .jpg .bmp, wave, .dll, and .exe files from being stored. As Mann put it, "Individual home folders are for storing business data, not running software." With Quota Server, Mann made sure that employees could save only Excel, Access, PowerPoint, and Word documents in their home folders.
Three levels of warning
Mann also set up a three-level notification system in Quota Server to warn users if they were exceeding their quotas. When they reach 90 percent of their quota, the system automatically e-mails users a Level One alert suggesting that they delete data older than 14 months and check whether they're saving data that should be stored in a group share folder rather than an individual home folder.
At 95 percent of their quota, users receive a Level Two e-mail that employs a bit of a scare tactic. It basically tells them that they won’t be able to save their documents if they exceed their allotted 250 MBs. At 100 percent of quota, users get a Level Three alert telling them they can’t open an existing file and add data to it. If a document is already open, the system will let them save any work in progress. But once they close that document, they’re basically restricted from starting anything new.
Today, if a Foster Farms employee needs more storage in his or her home folder, the employee calls Mann to request the additional allotment. Mann will add to the quota for a limited period of time (the expected duration of a major project, for instance) and then revert the individual to the original quota level once the need is over.
Dealing with the politics of control
Though controlling software seemed to fly in the face of Foster Farms’ family-oriented culture, Mann was surprised at how well users took to the idea. He believes acceptance went so smoothly because he was able to frame the new quota system in terms of how well it would keep the business running. "I let people see the reports for themselves and draw their own conclusions," said Mann. When they saw how more than 75,000 mp3 files in individual home folders were eating up hard drive space on their servers, users understood how their actions were contributing to server crashes.
"I sold users on why they needed this [quota management system], not why I need it," said Mann. "We’re getting paid to sell chickens and whatever it takes to do that process. This is how [this product] is going to help you. And it has." Mann insisted that if you take that kind of tact when educating users, you won’t have to fight them on the issue of putting in controlling software.
Keeping the vendor on its toes
Today, Foster Farms has all of its 20-plus servers under Quota Server control. "At this point, the project is all routine. All our engineers are able to easily install, manage, and configure the software," said Mann. While Mann likes the idea that Quota Server is a hands-off product, he says that any company buying it should keep retesting and reevaluating the product, even after it’s installed.
Mann cited his dissatisfaction with a few features of the 5.4 version of the product. He felt Quota Server’s Web interface was slow and didn’t work closely with Active Directory. "Whatever shortcomings they had," said Mann, "Northern fixed in the new version, version 6.0." According to Mann, the latest release uses more optimized Web code, runs on SQL, and integrates with Active Directory like it should.
Mann believes in being proactive with vendors. "If I don’t communicate with my vendor and tell them that something isn’t working for me, or this is what I’d like to see [in subsequent releases], they’re not going to improve." So his advice is to go back and reevaluate, even if the product and the company are great and the support is incredible.