Storage

Peter Cochrane's Blog: Outdated formats are depriving us of precious data

The ravages of time are threatening our stored files and photos...

Written as a passenger on the M3 heading to Southampton and dispatched to silicon.com from a free wi-fi service in my hotel.

Stored data just keeps growing and will probably last well into the future as storage technology improves. Unfortunately, the same cannot be said for operating systems and applications.

All my digital life I have been saving stuff, lots of stuff, in all manner of formats. Documents, presentations, animations, movies, graphics, photos, and more, stretching back decades all reside on my hard drives.

I try not to delete, destroy or corrupt anything. It is all there just in case. The only consolidation that has occurred is to get everything off discs and sticks and onto multiple hard drives.

Occasionally, I try to recover files, not because I remember any of them, but because they turn up in a search or I stumble on them while looking for something else. But I'm experiencing more problems with old data, with files that refuse to open, or more commonly, that partially open and contain significant format errors.

This morning I tried to open a document from 1987. Since that time the associated application has had numerous upgrades, but still remains popular today. But the operating system I used at the time has had its day and is now long gone. And while the file opened after a fashion, the formatting errors rendered it useless.

After about an hour of experimentation I got the file open in the correct format, and then updated to today's version and resaved it. This issue of old formats is not a new problem.

Throughout human history, languages have come and gone, important stuff has been recorded and lost, along with the definitions and frameworks that were well understood by the original scribes. In a few cases a decode, translation or some understanding is achieved without a Rosetta Stone.

We can only guess at what knowledge and wisdom have been lost as a result of being unable to decode the meaning.

Today, the problem is much worse because we have well over 100GB of information stored for every living soul on the planet. That data translates into more than 1,000 tonnes of stone, or clay, tablets for each human.

CD storage

Storage technology is advancing but formatting errors are an age-old problem
(Photo credit: Shutterstock)

Storage media in the form of CDs, DVDs and Flash now last for less than 20 years, while document formats have much shorter lives of between five and 10 years. The bad news is the technology of storage is accelerating and these format times may continue to shorten so the problem will become more acute.

And now the good news. A secret bunker in the Swiss Alps is receiving an upgrade as I type - digital DNA is being stored in a vault for the future. Over the past four years, 16 European libraries have contributed to this project costing $18.5m.

The plan is to seal the vault and then open it in 25 years to see what can and cannot be recovered in a useful form, and where the next set of risks lie.

And now the obvious question. Why haven't they put the collection and all the software online? I for one could use them now, and my guess is many companies, governments and individuals could too.

About Peter Cochrane

Peter Cochrane is an engineer, scientist, entrepreneur, futurist and consultant. He is the former CTO and head of research at BT, with a career in telecoms and IT spanning more than 40 years.

Editor's Picks

Free Newsletters, In your Inbox