Software

E-mail fiasco by leading Web host results in unrecoverable e-mails

Ever wondered why dirty, white shoes become black, but black shoes become white? Or for that matter, why incriminating e-mails deleted by criminals get recovered via computer forensics, but e-mails that you want to keep get trashed for good? Don't laugh, because this wasprecisely what happened in the United Kingdom.

Ever wondered why dirty, white shoes become black, but black shoes become white? Or for that matter, why incriminating e-mails deleted by criminals get recovered via computer forensics, but e-mails that you want to keep get trashed for good? Don't laugh, because this was precisely what happened in the United Kingdom.

According to The Register, "the U.K.'s number 1 Web host" - Fasthosts - has admitted that a botched update to a mail server permanently deleted some of its customers e-mails.

Fasthost's explanation of the fiasco to The Register:

Following a human error during an update to our mail server housekeeping processes, a number of Fasthosts Advanced POP3 mailboxes unfortunately incurred a failure resulting in a loss of stored e-mails. Although approximately 50 per cent of e-mails were able to be restored from our backup, we regret that any lost e-mails will not be recoverable.

Unfortunately for the other 50 per cent of e-mails, the backup processes ran soon after the error, and hence these e-mails were non-recoverable.

The problem has affected a small number of Fasthosts Advanced mailboxes, not Standard mailboxes or Exchange mailboxes.

How many megabytes of e-mail do you have? What's your backup strategy for e-mails, both old and new?

--------------------------------------------------------------------------------

Stay on top of the latest tech news

Get this news story and many more by subscribing to our free IT News Digest newsletter, delivered each weekday. Automatically sign up today!

About

Paul Mah is a writer and blogger who lives in Singapore, where he has worked for a number of years in various capacities within the IT industry. Paul enjoys tinkering with tech gadgets, smartphones, and networking devices.

25 comments
Meesha
Meesha

Best response is, don't use Exchange/Outlook - very poor architecture which does not allow for off-line, proper replications, native archiving, etc. The article didn't explain whether proper change management (ITIL) practices were in place. If best practices were in place, then their problem would have been a mute point since risks would have been identified, roll back plans, special back ups, etc. all best practices that would have prevented this. With the little bit of info in this article it's hard to determine if in fact this was not simply human error and bad practices.

a.ende
a.ende

Which day or days' emails were lost?

JCitizen
JCitizen

We were contracted to a non-profit that couldn't afford to externalize to a paid HIPPA compliant service. I wasn't on the Exchange 2003 end of the topology so I can't speak on their techniques out there; but we split each major remote office with their own server, with backups done at those servers. The pst files were allowed to be stored on the users local machine also. So email was stored and backed up at the main offices using optical disc; stored at each branch office using hard drives; and stored in each user's system unit. If the user lost email we could restore remotely from a main office but usually used the branch office back up; if a branch server failed we could pull it from optical disc at the main exchange site. If I remember we used the incremental back up strategy; but I don't remember what third party software if any that we used. I know it worked because we had several highranking users who were able to recover their files when the local machine failed.

jondfwmortgage
jondfwmortgage

No you don't understand. They had an implemented plan. They had a backup on two stone tablets they received from a burning feux ficus tree in the office from God...LOL

JohnMcGrew
JohnMcGrew

...that you need to personally take responsibility for its storage and backup. Relying on any other single party, even if they are "the biggest" is certain folly.

PhilippeV
PhilippeV

Relying also only on your own storage and backup is as much dangerous than relying only on someone else. (1) If the data is precious and its value comes effective when you suddnely have lost it, then the effective important point is the backup. But if you loose your own data after you thought you had taken all possible mesaures, chances are high that you'll loose as well you backup in case of acceident or disaster. (2) If you rely only on someone else, at least they may compensate you if you have paidfor a warranty. (3) So externalizing the backups is not a bad solution, as there's muchless risk that you'll loose both your localandremote backup at thesame time. Now the real point issecurity of your data if it has to remain secret. This does not change the fact you still need and external backup. And regarding privacy/secret of your data, your home storage is more probably muchmoreexposed to data theft than the storage at a third party site that have stronger data protection protection than at your own site. So the good strategy remains to externalize your data as much as possible, and use two indepedant remote backups (e.g. your ISP's mail server, and a mirror server were you forward copies using a secure link). You canstillhave a local backup by forwarding also your emails to your own site, to a storage you'll manage yourself, but that you will keep secure and closed to almost everybody even in your organization. Your employees will still get their individual emails by connecting to the primary mail server that allows them to perform daily operations; in case of disaster on your local central site, just have a scriptready so that they can at redirect to another site. We have similar strategies for local storages in applications and databases: backup is not much as useful as a jourlized mirroring system, each mirror having its own backup and recovery and thetools needed to recover from the source at another site. We call this "High Availability" and this is a general strategy. Nobody is completely protected from accidents or disasters, even if he is well prepared and trained.

JohnMcGrew
JohnMcGrew

...if your loss of data is irreplacable and results in the loss of your business. It happens every day. I seriously doubt that the ISP that flushed all those messages will be willing or able to compensate all those users for the damages they may suffer because of the loss. Externalizing is wise (do it myself) as long as it's not your only solutuion. But again, relying on a single avenue is folly. It's up to those responsible for the data to take responsibility for multiple forms of backup, and not relying upon a single solution.

Industrial Controller
Industrial Controller

I used to work for a company that wrote software for banks. You would not believe the banks that did no backup on their loan documentation.

Carl_Liu
Carl_Liu

For my personal e-mail, I now rely completely on Gmail. If they lose e-mail, or lose service for just an hour, it will be a major news that would probably affect their stock price. I used to store my e-mail on my computer, which has a habit of losing at least some of the data every 3 or 4 years, just in time to remind me of the importance of backup. My oldest e-mails and other stuff are on 3 1/2 disks, which are not easily readable these days. I even have some data stored on those tape reels about 1" wide and 12" in diameter. I have no idea how to get at their contents now. I had a need to get back some of the data I backed up about 3 years ago. Unfortunately, I used Microsoft backup utility to store it. It's unfortunate, because the system was Windows 3.1. I also have some data stored on Zip drives, not accessible any longer due to its archaic hardware and format. Sorry, I digressed. Anyway, backup is good, but backed up data is not the same as archive, which requires updating like your computer.

JCitizen
JCitizen

Zip tech is still kept current so you should be able to backtrack that as long as you don't get the click of death during the process. From what I've heard about Microsoft backup utility though, it is probably a lost cause.

KeReleaseSpinLock
KeReleaseSpinLock

My Rules of Backup. 1. Know where and what your important data is, otherwise don't bother going any further. Where is your pst file? Come on now - right now! Where is it! :-) Mine is D:\outlook\outlook.pst 2. 4 minimal monthly backups. Local machine, network storage, DVD (encrypted) - off site (also encrypted) eg Gmail. The latest PGP will encrypt your files for free. (Personal use anyhow). 3. Know how to restore your backups. Done. *true* peace of mind..... Have a nice backup.

asjeff
asjeff

I hope to hell that when you say you backup to D:\Outlook\Outlook.pst the "D" in question is a mapping to another system/Network drive or external harddrive and not a partition on your local computer!! How any times have I had to explain to users that storing your archives on your local machine means that if your machine cocks up you have no backup? At the very least your network storage should be backed up daily, even if it only an incremental!! Still, it's quite funny when I see the realisation dawn on people's faces after we have rebuilt their machines and they can't find their archive :)). (But that's just the sort of guy I am - we tell them often enough!!)

KeReleaseSpinLock
KeReleaseSpinLock

Did you look at step 2? :-) Network Storage. I have a win 2003 Server in the other room (HAL) is his name of course. :-) I dump my pst, docs and source code to that box. Source code is priceless for what I'm doing- so that gets encrypted and shipped off-shore daily. Yes, I recall a co-worker that lost several years of email once. He was a hi-tech guy too - could have easily done a nighty robocopy of his stuff. It was heartbreaking to watch his distress. Cheers asjeff. PS: I give encryped DVD backups to my family members to look after too, as an extra precaution. This adds a nice cheap geographical diversity. I recommend it.

JCitizen
JCitizen

Their files were automatically backed to the branch and main office anyway.

Jaqui
Jaqui

if any particular email is critically important shouldn't you, as the person who places that importance on it, have your own backup plan outside of the service providers backup sequence? if any fasthost clients lost important emails, then they are to blame for NOT making sure they had a backup in their possesion. Fasthost should have more than a day in their backups, but they are not responsible for storing important communications for thier clients, unless it is in the contract that they will do so.

Timbo Zimbabwe
Timbo Zimbabwe

"but they are not responsible for storing important communications for thier clients" But it was only ONE DAY of their client's emails. Certainly you don't think that the client should be admonished for this. I mean, come on.... one day of backups? That is hardly realistic for a professional hosting company.

paulmah
paulmah

Why not share with us your backup strategy for emails, both old and new.

drew.mcbee
drew.mcbee

We archive every inbound and outbound email, in addition to nightly tape on a monthly rotation, and rsync everynight to backup server. Uses a bit if disk, yes, but disk and bandwidth are really cheap when you consider the value of being able to sleep at night.

info
info

For a hosting company not to have regular backups(by that I mean daily and weekly stores)is not just corporate irresponsibility but also a cardinal sin in the IT workspace.

salvarad
salvarad

They must work for the Republican party, or trained by the Attorney Gen.'s office. LOL

wnfaknd
wnfaknd

I use DPM to backup our exchange server and it works better than veritas and anything else out there that I'm aware of. For personal email I just use Gmail.

drew.mcbee
drew.mcbee

Is that a anti-Linux icon I see? My God, man, have you lost your mortal soul? I don't think I've ever seen one of those, and hope I never do again! ;)

Genera-nation
Genera-nation

Nightly to local servers disk (quick access for a restore where server is OK) Nightly copy of this to remote disk Nightly to Tape, just a copy of local server disk backup files (week rotation with Fridays 1-5) Full image of e-mail server daily using Acronis

TechieRob
TechieRob

We run a program called 'Shadow Protect' which is similar to Ghost, but it can take a snapshot image without having to shutdown the pc. This program runs every half hour on our email server (incremental/differential) then at midnight (full image). The images are saved on a NAS drive then backed up on tape daily. The tapes are on a 7 day rotation with two sets of tapes. The redundant set of tapes are stored offsite in an EMP/ Environmentally protected storage container.

chiefywiggum
chiefywiggum

who overwrites their backups every day? no rotation, not even a week?

Editor's Picks