I get in and the receptionist asks me to look at an e-mail. She thinks that it may contain a virus. I make sure she’s got the latest Norton update and scan it. Not a problem.
Get caught up on this week's Admin Diary entries:Read Monday's installment Read Tuesday’s installment
I open one of my various e-mail newsletters and read about this new virus called TROJ_EXPZIPWMPAK. I send out an e-mail to the staff to let them know about it. I plan to spend my day making sure everyone has the update that Norton has posted the day before.
I get a reply to my virus-warning e-mail from one of the users. It reads just like the e-mail that contains the infected file. I call up the user it’s from and ask her if it’s some kind of joke. She replies, “What e-mail?”
I delete her e-mail and run my virus scanner. I run to her desk to unplug her from the network, hopefully keeping the virus off the network. Now I wonder how I’m going to get her the anti-virus update without re-connecting her to the network. I also need to see who may have been sent the virus from her machine, as well as who it came from.
I quickly connect to Symantec and download the update. While the update is running, I open Outlook. I note the sender and receivers of the virus, as well as their e-mail addresses. I delete the infected e-mail, including the copies in the Sent Items folder.
The update completes, Outlook closes, and I unplug the network cable. It has only gone out to a few people; none of them staff members. WHEW!
The update notices the virus, but says it can’t repair it because the file it’s infecting is in use. I go to the Symantec Web site and print out instructions for getting rid of the modifications the virus has made. The registry fix for it is quite simple, and it works. But I can’t get rid of the Explore.exe file; it’s still in use. I try to stop the process and then go into Windows NT Explorer, but it just restarts it.
My desktop support person, Bill, arrives, and I let him know what’s happening. I give him the e-mail addresses of those who sent the message, as well as those who received it. I ask him to send them an e-mail explaining that they have the virus and warning the additional recipients to delete the e-mail and not open the attachment. Julie, our main help desk person, says someone is missing some files. I’m hoping it’s not related.
I look at my watch and notice it’s getting close to lunchtime. I’m supposed to go out to lunch at noon with a couple of co-workers for my birthday. I’m hoping to eliminate the virus soon.
In the meantime, I start the virus scanning on the servers. On the user’s machine, I eventually figure out that by stopping the Explore process and switching to a command prompt, I can delete the Explore file.
That works, but Norton still detects the virus in some files located in the temp directory. I go in through the command prompt and get rid of those, too. I do another scan of the machine, and it appears to be gone. I restart it to make sure there’s nothing left. I let the user know that any .doc, .ppt, or .xls files that were on her hard drive are gone. There’s no way to get them back, as we don’t back up the files on the user’s machines. If they want to have anything backed up, it has to be on their server drives.
I talk to the two users who are missing files. They tell me they were in the middle of working on the files, and they locked up. They’ve already created new instances of the files. I’m hoping it was some coincidental occurrence and not a side effect of the virus. I start scanning the servers to see if they’re missing any files. I find one server with all of its Word documents, and our main file server with a few of its Word documents, all set to 0 bytes—the virus at work. The servers with the messed up files are virus free. I may still make lunch.
Delay of lunch. Maybe I can make it at 12:30. I decide it’s best to go ahead and restore the missing files, doing it before the users notice they’re gone. A couple of the files are the same ones the users had commented on earlier. Looks like it was no coincidence.
The files are finally restored, and everything is looking good. According to Norton, the virus only goes after drives that a user is connected to. I do some double-checking to make sure I’ve gotten everything, and I’m not missing anything on any other server that the user had mapped.
Lunch at last! One of my co-workers has already gone to lunch. She couldn’t wait any longer. Well, at least everything’s taken care of. I can go have lunch, relax, and then come back to start on the rest of the machines.
Nothing like a nice, long lunch. I didn’t get any calls while I was away, so it’s looking good.
The servers have finished scanning, and nothing was found.
All of a sudden I get a call from Sacramento. All their Word, Excel, and PowerPoint files have 0 bytes. I run to our main file server and disconnect any users from Sacramento who are connected. I check the Terminal Server that connects them to our btrieve database, and I disconnect them from there as well. Luckily, as this starts happening, our network-cabling guy shows up there and starts helping them out.
I try to explain to him what’s going on, and he doesn’t seem to get it at first. I want the IT folks at the site to go to everyone’s e-mail and see if they are sending anything out. I’m sure the virus has got to be up there. The user who had the virus earlier wasn’t connected to any of their shares, so it couldn’t have traversed that way. It’s only supposed to go through mapped drives.
A call comes in about a large amount of e-mails getting sent to a mailbox set up for our customer service department. There are thousands of them, and they can’t figure out where they’re coming from or why. I’m getting partial descriptions of what happened. I hear some general details, and I surmise what’s going on.
E-mail was sent to all of our customers via a distribution list on our Exchange Server. The distribution list was put on the To: field, instead of the BCC: field. One or more of the customers, in replying, hit Reply to All, instead of just Reply. This, in turn, sent out an e-mail to more than 10,000 customers, and if two people hit Reply to All, that’s tens of thousands of e-mails. The Exchange Server is taking a big hit, and it is slowing down minute by minute.
Sacramento decides to physically disconnect everyone from the network and then tackle one machine at a time. They’ll connect, install the update, and then disconnect. My scan on their server up there shows it has multiple copies of zipped_files.exe in different locations, but it’s not active.
I clear the infected files, run an update, and then trigger a virus scan. I check out our servers again to make sure no one up there is connected. My manager, Max, has taken the lead on the Exchange problems, leaving me free to deal with the virus.
The IT department meets to discuss the e-mail issue. My assumption is correct: Someone did hit Reply to All. We’re starting to get e-mails from customers being deluged with e-mails. People are barely able to work off the Exchange Server. I’m supposed to go out to dinner with my wife and son for my birthday. Things aren’t shaping up well, and I’m afraid it may become like lunch.
Things are slowing down a little bit, but no one can figure out where the virus is in Sacramento. It’s not showing up anywhere. I check their server again to see where the documents are missing. I notice a directory with some unaffected documents. Why aren’t they set to 0 bytes?
The virus scan hasn’t found any more traces of the virus. I decide to start restoring their lost data. I check the backup and see that the last successfully completed session was three days ago. This means that anything they’ve done over the last couple of days is gone. I tell Cynthia, and she says not to worry—they’ve been relatively slow, anyway. I ask her to load the tape from three days ago, and I attempt to run the restore. It keeps looking for the tape and can’t find it. I’ve got messages in the event viewer that suggest the driver for the tape backup isn’t functioning. I ask Cynthia if she’s got the tape properly loaded. It’s a five-tape unit, so she checks to make sure the right day is loaded. The right tape is loaded; I just can’t get it to go.
Max is still dealing with the e-mail, trying to find out more details on what happened. I let him know I’m not having any luck with the restore, and I think it would be wise for me to fly up to Sacramento to deal with both the virus and the restore. At first he’s not sure, but I tell him that with my laptop and the remote control program, I can do just as much for this site as if I stayed in the office. It would be more effective for me to be up north and work everything out from there. He agrees. I call up my travel agent and get the first flight to Sacramento from LAX for the next day. I check in with Sacramento one last time before heading out for the evening. They’re not finding any machines with the virus, so I let them know I’m on my way up in the morning. I also ask them to leave everyone off of the network. We’ll deal with it first thing in the morning.
I leave for my birthday dinner—two hours late, but I’m leaving. I call my wife and ask her to meet me at the restaurant. Not one of my favorite birthdays, but certainly a memorable one.
Read California Virus Fighter's Thursday entry. To share your thoughts about this diary entry, please post a comment below.