Storage goes through the looking glass

Shayne Nelson believes storage has changed from something rather limited and expensive to something remarkably cheap and virtually without limit.

We are at an immense turning point in IT. Almost overnight, storage has gone through the looking glass, changing from something rather limited and expensive to something remarkably cheap and virtually without limit.

We've been waiting for IT to turn this corner since 1980 or so. In 1983, a 20 MB hard disk cost as much as a used Volkswagen, and PCs generally had around 16 MBs of RAM at most. Storage was scarce. Storage space on the chip and on disk had to be husbanded, squeezed, pruned, and reallocated immediately for maximum performance.

For 25 years, software designers faced the same challenge when it came to storage: More memory meant faster performance, but space for code in memory was always extremely limited by the hardware's physical constraints. Designers usually had to settle on a middle position between maximum performance and optimal code size. Applications whose code was too large to fit on the free space on the chip tended to run too slowly because of the massive paging the size of the code triggered.

The first disk drives I encountered in the late Sixties were the size of large washing machines. The drives weren't as noisy as washing machines, but a roomful of them (you needed a roomful to get enough storage for your IT operations) could just about drown out a conversation. IBM rented (never sold) the drives to customers, and so the size helped justify the hefty price. The disks were in a stack inside a kind of semi-transparent cylindrical cake carrier almost 20 inches in diameter.

Apple brought out the Mac in 1984, with an operating system at least 10 years ahead of Microsoft's MS/DOS, but a hard disk (which was optional) for the minuscule Mac cost as much as a new Volkswagen. Storage was priced beyond the consumer's reach, so most Macs never had a hard disk; this means the Macs ran at a tenth of the speed for which they were built.

Then a few years ago, drive technology turned a corner and took us through the looking glass to a place that's just the opposite, where there's almost infinite space to profit from in ways that will speed up your code. The new challenge to the entire IT industry is: Figure out how our software can be improved to take advantage of this suddenly unlimited disk space.

With unlimited disk storage and massive RAM, a totally new architecture is necessary. The basic structure of software and database has to be reconsidered, for example. For more than 40 years, almost all database systems have been using some form of data compression forced on the designer by the limited space available to him on hard disk and in memory. Now, suddenly, no form of data compression is needed for any operation on a computer, making that compression obsolete.

The new storage technology doesn't just bring freedom for the programmer, it also brings a new level of difficulty to the job of backup. No big change is without its price. As someone once pointed out, "Believing in progress is believing in getting something for nothing." Twenty years ago, many IT pros had most of their hard disk backed up at any time. Today, with these gigadrives packed with monster applications, you'll have a hard time finding anyone who has most of his hard drive space backed up in a recent copy somewhere.

The extra drive space can also be used to increase the frequency of backups; have you done that yet? Perhaps you used to keep seven generations of backup on tape. With megadrives, you can keep 100 generations just as easily or 700 for that matter.

The extra drive space can also be used to build historical depth into databases and data warehouses. Records can be flagged as deleted instead of being physically deleted, and historic copies retained when records are updated, meaning the user obtains the power to look at the history of the values in any field of the record. With that architecture in place, "Show me every address where this person has ever lived" becomes a matter of a single click. Most corporate databases currently can't even answer this kind of query because of lack of retention of past versions of records due to perennial storage scarcity.

This challenge of using the new wealth of space available to us is one we'll be looking at several more times in this blog. Share your thoughts on how this bonanza can improve operations, security, and even business intelligence.

TechRepublic's Servers and Storage newsletter, delivered on Monday and Wednesday, offers tips that will help you manage and optimize your data center. Automatically sign up today!


... is the rise of bloatware. I can still remember the days when the entire word processor, spreadsheet or accounting software would fit into a single 5.25" floppy disk.


should be 500GB hard drive 7200 RPM x 10 or 20 times = RAM's speed because people do not like to waiting for loading to open something bigger files and should be faster open like less 1 sec


Going back 40 years, yes, disk storage has increased tremendously in size. And, even in the last 10 years, that growth has been more than impressive. And, even more impressive than the amount of storage that can be accessed on just about any computer, is the price of that storage. The prices have come down to a level where even terabyte disks are easily affordable for the average computer /owner/user. But, there is still an area where storage hasn't progressed much, and that is in speed. Sure, compared to the 80s and 90s, storage access speeds have improved tremendously. But, in the world of computers where the CPUs and RAM have improved to the point where instructions are executed in billionths of a second, disk storage is still the biggest handicap for computers. A computer is only as fast as its slowest component. Besides disk storage, all other forms of storage are very limiting when it comes to the speed of access to the data. No matter how fast Intel or AMD or IBM make their computers, the access to the data is still the biggest determining factor when it comes to how fast an application performs.