Software

Spam relay: Up close and personal

Recently, Michael Kassner got up close and personal with a compromised Exchange 2003 e-mail server. It certainly wasn't fun, but he learned a few surprising things about rootkits and their removal.

Last week, I received a frantic call from an on-site contact (contracted facility) who told me that e-mail seemed to be down. I'm sure I had a quizzical (maybe even scared) expression on my face because I wasn't sure what "seemed to be down" meant. After an abysmal attempt at calming the contact down, I remotely accessed the network, right away noticing that the Exchange server's response was very sluggish. Now I'm getting nervous; rebuilding Exchange servers isn't much fun on a good day and a whole lot worse when it's an unplanned activity.

Take a deep breath and think

Remote access wasn't cutting it, so I decided to go to the facility. During the drive, I had time to settle down. I even laughed at myself, as I was trying to remember how Exmerge works, ultimately realizing that Exmerge was out of the question. I'd be rebuilding Exchange using the same computer, dahh. After arriving, I logged on the Exchange server locally, with the event logs being my first stop. The application log was full of red Xs, all referring to NDRs. That's not good, because I check the event logs daily and normally they're clean.

Next step, Exchange System Manager

I then opened Exchange System Manager. At first glance everything looked good. Next, I checked the queue. Oh my, there were over 9,000 outgoing messages in the queue, all from the postmaster. That's also not good. I started to get a picture of what was going on, but how could it have happened? When I first set up the Exchange 2003 server five years ago I followed all the suggested guidelines. The server is up to date patch-wise and well-sheltered on the network.

I wanted to be sure about my hunch, so I went to DNSgoodies.com and used their Open Relay Check. Sure enough, the Exchange server was indeed acting as an open relay. Hmm, that doesn't make sense. So, I opened Exchange System Manager once again to check the Relay Restrictions configuration and found it was set up as shown below:

relay.JPG

According to Microsoft, this configuration should prevent the relay of e-mail by unauthorized users or servers. I started to have the sinking feeling that this Exchange server was compromised, but it didn't make any sense. The server is well-protected in the classical sense, as shown in a simple Visio diagram of the network:

visio.JPG

To further explain, the entire network is protected by Symantec Enterprise Edition anti-virus. Symantec System Center updates virus signatures daily and completely scans all networked computers nightly. As for notebooks, they are scanned immediately upon connection to the network. In addition, the network has IDScenter and Snort on the ISA server. Finally, all mobile devices (notebooks and smart phones) have software firewall applications, and the users are cautioned to not disable them.

Looking specifically at the Exchange server, it has Symantec's Enterprise Edition client as well as Mail Security for Microsoft Exchange installed. Both Windows Defender and Malicious Software Removal Tool were up to date and scanning the Exchange server files automatically.

Needless to say, the on-site contact didn't share my fascination with determining how the Exchange server got subverted, asking instead why couldn't I just get it working again. I made mention of how important it was to understand the how and why of the attack process. Otherwise any resolution of the problem potentially would be short lived.

That didn't cut it, and I was directed to get the e-mail flowing without delay. It kind of reminded me of Dune and the "spice of life" (one of my favorite movies), sorry, I digress. Having a computer compromised can mean many things, but in the case of e-mail servers it usually has to do with being rooted and under the control of a remote entity. Knowing that certainly didn't give me any confidence that I'd be able to resolve this with anything less than a complete rebuild of the Exchange server.

Can a rootkit be removed?

I doubted it, but I thought it best to try. The following steps are how I went about trying to restore the Exchange server to normalcy:

  • Ran a special scan using Windows Defender: No results
  • Ran Malicious Software Removal Tool: No results
  • Ran Windows Sysinternals Rootkit Revealer: No results
  • Ran Trend Micro Rootkit Buster: No results
  • Completely uninstalled AV client and reinstalled it locally. Next, I scanned the complete computer and guess what: Symantec found Backdoor.Rustock.B. This was the first indication of any malware. The AV client wasn't able to remove it completely as it re-established itself after each quarantine.
  • Went to Symantec's Web site, hoping to find removal instructions for Backdoor.Rustock.B. Luckily, I found the exact process on the Rustock Removal page.
  • Followed the steps and rebooted the server in normal mode.

The server appeared to be running at its normal pace. A check of Task Manager confirmed that. I next opened Exchange System Manager and checked the queue. To my relief, the queue was normal, as shown in the image below:

queue.JPG

I'm sorry, but I was in shock. I didn't expect to solve this issue with anything less than a total rebuild of the Exchange server. Almost immediately, user mailboxes started filling up, and sending e-mail was almost instantaneous. Being very skeptical, I closely monitored the server for several days; actually, I'm still monitoring it.

I'm very relieved but still perplexed as to how the infestation took place. I ran a complete scan on every computer on the network and that turned up nothing out of the ordinary. The firewall or IDS/IPS logs didn't offer any help either.

Lessons learned

This is the hard part: I'm not sure what I learned from this exercise. The network and Exchange server were using current best practices. Just to recap what that means, the following list is what most experts recommend to prevent rootkit infestation:

  • Systems patched: There's a WSUS server on the network, and it keeps every computer completely up to date. Secunia is running on the network as well, which informs me when other-than Windows applications or drivers need updating.
  • Never run under Admin privileges: Since an Exchange server was compromised, the only time anyone has access to it is when administrative activities are required. So that's a potential problem, except the server's sole purpose is Exchange.
  • Install security and AV tools: As I described earlier, the Exchange server had multiple applications running that supposedly would sound an alarm if any kind of malware was attempting to be installed.

I guess the real lesson that I learned (well re-learned) was that nothing is for sure. I was sure that this network would never get rooted. I was also sure of it being impossible to remove a polymorphic rootkit like Rustock. I was wrong on both counts.

As a small aside and to explain why I thought it would be impossible to remove Rustock, please check out this dialogue on the Windows Sysinternals forum. It's quite amazing to follow the Rustock lineage and the increasing tenacity displayed by each new version, especially when the authors are fairly sure Rustock.D is in the wild and undetectable as of yet.

Final thoughts

I consider myself very lucky. No doubt about it. I also know that we are playing some serious catch-up when it comes to keeping computer systems and networks secure from malware of this sophistication.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

194 comments
ScarF
ScarF

What I don't see into the published network diagram is an AntiSpam server. In my network I have a Linux server which relays all the incoming messages to the Exchange server after analyzing them for spam and checked for malware. Although not so well protected than your Exchange server - no ISA server and anything like Snort, mine didn't have any problem so far. We were blacklisted twice but only because of some malware on a couple of workstations which started to send emails through the Exchange server before locking it down. My fault for not closing the open relay after the original installation and configuration - it simply slipped out of my head.

BigAmcInroy
BigAmcInroy

Michael, I always read your articles. You do an excellent job with something most admins can't comprehend. Namely, humility, you don't act as if you know it all. I really appreciate that. I always learn something from your informative articles. Keep up the good work. Alan

The 'G-Man.'
The 'G-Man.'

Than take a daily / weekly image of the critical systems that take a long rebuild time and keep it safe using something like Acronis. When you need a reinstall of the entire server (-data, you keep that seperate) only takes around 20 minutes. I do this for Exchange, SQL & Domain C0ontroller

Cactus Pete
Cactus Pete

It is possible that any of your other internet connected servers were the first point of intrusion, perhaps through supportive software you have installed. (A webserver might have a log analyzer installed that has a vulnerability.) Once any server is compromised, your network is compromised. You might want to check them all. Also, check for activity through the firewall to workstations when people are not in the office.

d_g_l_s
d_g_l_s

no one accessed the server(s) using the admin account? Just wondering as someone might try to cover up for a serious mistake by either diverting attention from this or outright lying about it to protect their job. I'm not accusing anyone, but I do know that the human factor in all this is the weakest link :) I am of course speaking from personal experience, not that I want to lie or deceive but in the "sweating" stage of discovery and accusations one might say just about anything in the spur of the moment. Well, it will end up being the most simple item in the end as I've always found it to be. Just where I might not be looking or thinking and that's what the authors of the rootkit want to happen.

JohnMcGrew
JohnMcGrew

...I usually talk my small business clients into outsourcing their e-mail servers. The second you take your eye off them, this kind of thing always seems to pop up. As for root-kit removal: Once a compromise is discovered and removed, I always have this uneasy feeling that some part of it is still there, waiting to go active again once my back is turned. What I usually do is pull the drive and install on a clean "utility" PC that I keep on standby for this very purpose for a complete scan. Since it's the nature for root-kits to make themselves invisible to the OS when scanning itself, I just don't completely trust scans conducted under already compromised OSes.

casternj
casternj

Excellent Article which version of symantec are you using? if you had another product instead of symantec, do you think it wouldve caught this?

Cyclops116
Cyclops116

At my old job I put in a product called GWAVA from Beginfinite that ran on our Novell GroupWise gateway box, it integrated with multiple brands of antivirus software, Symantec Corp V.9 in our case it also had heuristic Spam filtering along with Blacklist and SURBL checks. Worked fairly well not one infected email got through in the 5 years I used it(could have been the Groupwise clients more than the scanning), but was it was ALOT of work to keep up! Had to get the users to submit the SPAM that got through to the server and run diagnostics on them monthly which updated the rules kind of a pain. Where I am now we use MXLogic, a third party filtering company and it is SOOO much easier on me and the server! Limited training for the users, they get an email with a list of emails that got blocked and they can click release, always allow or always block.

Michael Kassner
Michael Kassner

I'm starting to think that the way the system is setup may not be the best. The exchange server has Mail Security for Exchange on it and supposedly that has Anti-Spam capabilities. It does, but it may be something that is taking too many resources from the Exchange server. Do you have to update/maintain the Linux server often? That may be a issue that would favor having an off-site resource like Postini or Red Condor take that over.

grouper
grouper

What I've done in my network is I have a Barracuda spam filter that's first in my MX records and Exchange is second. All inbound mail must pass through the Barracuda. I have rules in my firewall so that Exchange ONLY accepts mail from the Barracuda. I also have port 25 blocked on my LAN as my LAN clients don't need access to port 25 and I've ran into an issue at my previous job where a compromised machine was sending TONS of mail messages out because we didn't have SMTP blocked. If the Barracuda were to ever die, I would just remove the firewall rule on port 25 and allow mail directly to exchange till it's up again. I'm running Exchange 2007/MS Forefront Security for Exchange, which is a pain to administer and install but has been working pretty well.

Michael Kassner
Michael Kassner

I see they have one specifically for Exchange. Is that what you use? The cost maybe a factor for this company, but I'll certainly pass the suggestion along.

Michael Kassner
Michael Kassner

That location has only the ISA server facing the Internet. I did run a complete scan on all the servers, but your comment made me realize that I didn't find the rootkit until I reloaded the AV software. Hmmm, I may have to rethink that.

Michael Kassner
Michael Kassner

I really understand your point. I try very hard to gain the trust of the users, so that they wouldn't be worried about telling me exactly what happened. It's so much easier to troubleshoot if you have the correct details. It's not perfect and there are some that will avoid the truth, as it may be embarrassing for them. So, it's possible. I'm not sure why, but I still feel that the attack vector was via the Internet.

ewieder
ewieder

What I have found to be quick and accurate removal of the AntiVirus XP and other rootkits is MalwareBytes. It has saved me countless hours of trying to remove these nasty root kits and other malware. I then run other AV products to verify the system.

Michael Kassner
Michael Kassner

The client is using version 10.2 I have mentioned to them about trying version 11. That appears to be a uniquely new approach. I think the issue is that it's quite a bump in the maintenance agreement costs. This is hard to answer, I suspect so, the only difference I see, is that I reinstalled the AV client. That seemed to make the difference. I'm not an AV expert by any means, but I'd like to think that all the top brands are up to the task.

lhAdmin
lhAdmin

I would also like to know if another antivirus product would have caught/prevented this. Also what part, if any, did user error play in this? Is it possible that contact or someone else accidentally allowed this rootkit to install itself? I know that can be hard to believe for someone whose job is to maintain those systems but I have seen several of our more IT savvy users recently get bamboozled by that annoying Antivirus2009 malware.

Michael Kassner
Michael Kassner

From your experience, would you consider out-sourcing the filtering to let's say Postini or Red Condor? Do you see any advantages to keeping it in-house?

ScarF
ScarF

We use Roaring Penguin's CANIT. It works on different distribution from which we selected Debian. It also come with ClamAV. The installation is extremely easy. Download the ISO, burn a CD, boot from the CD and follow the steps. The updating process is automated. Should I not be needed to check the queues from time to time, I may completely forget about this server. In 5 years we had to manually update the product once, only - meaning: backup 2 databases on external storage, reinstall the product using a new ISO, and restore the databases. In our configuration, the MX record points to the router's external IP. The router forwards to the Canit box. Only the Canit box is accepted by the Exchange server to receive emails. Port 25 is disabled on all the workstations since they don't need it. Of course, there are other security measures implemented on the Exchange server itself - since it is a Windows machine, but the headache is much reduced by the presence of Canit box between the sender and the Exchange server.

casternj
casternj

this is a software package I also looked into. you have to make sure the package is capable of doing its job while not causing a lock on the db's. The cost is something that also has me in a tight spot

Michael Kassner
Michael Kassner

I appreciate your sharing of that information. I'm not familiar with that vendor. It sounds like you have used it successfully for removing rootkits. Is it OK to ask what rootkits you had to deal with?

casternj
casternj

Our maintenance agreement didnt change if we go to 11. I was demoing 11 as we can install it and dont like the direction they are taking. Next year when our contract renews, Im thinking of giving symantec the boot and going to another solution. I have been trying NOD and Kepersky

stuoutlaw1
stuoutlaw1

Does anyone hav an easy way to remove that antivirus2009? My daughter goti t on my wife's machine and I ended up reformatting the damn thing because I couldn't find all of it apparently

Michael Kassner
Michael Kassner

Determining how the server got infected is turning out to be harder than the actual removal. I've been researching every avenue that I can think of. There were some guest users at the facility during that time frame. They're not allowed to connect to the internal network (there's a guest VLAN) but sometimes the guest accidentally or on-purpose plug their computers into a jack connected to the internal network. That may have been an avenue, but I'd of thought the AV would have caught it. The other avenue I'm looking into is that the rootkit gained access via a malicious web site. My problem is that I'm not a forensic expert by any means. I gain some solace though from reading that many sharper people than myself are getting infected and have no idea as to the attack vector.

dirtylaundry
dirtylaundry

lately 4 computers have been taken over by this AV2009 and phony registry scanner on lay people's computers - one client got it 3 times, I finally switched his laptop to Ubuntu successfully with no problems. I'd like to meet the people that create that gunk and throttle them to within their last breath...

casternj
casternj

the Antivirus 2009 should almost be considered phishing. If its considered phishing MS would add it to their database.

Michael Kassner
Michael Kassner

I see your point about the costs. I just have a hard time determining a realistic cost center for the expertise required to maintain an in-house system. A SPAM filter is great only if it's setup correctly and maintained. If not it can give a false sense of security. That aspect scares me, as I've had to deal with the fallout of that on more than a few occasions.

Cyclops116
Cyclops116

I want to bring it in house for security, I don't like the fact that our email goes through someone elses servers. Call me Parinoid! And cost, we're paying $2.50 a person a month, which was ok when we were 30 or 50 people but as we get larger almost 75 now and about to outgrow our Small Biz Server it's getting expensive and I can get a Barracuda for the about the cost of 1 months service.

Michael Kassner
Michael Kassner

I am going to ask you to clarify a statement of yours though. "As much as I like the third party scanning I will bring it in house eventually. " From your other posts, it seems to make sense, but I'd love to know the details of why you feel this way. Sorry, but I have clients that ask me this sort of question all the time and I'd love to have ammunition one way or the other.

Cyclops116
Cyclops116

At my old company they didn't have internet email or even High speed internet for that matter. I implemented outside email with the scanning solution from scratch when I upgraded Groupwise from V5.2 to 6. In hindsite there are probably better programs I could have chosen than GWAVA but I needed something that didn't need it's own box so.. I'm the first in house IT person for my current company so the offsite was the only way for them to go. As much as I like the third party scanning I will bring it in house eventually. When I do, I'm going to look at some sort of appliance like a Barracuda, I'm no Linux guy so the one rsarlet is talking about is out of the question for me. I think some firewalls like sonicwall have SPAM filters and web protection pieces you can add to them too. Surf Control used to have one too.

ScarF
ScarF

You are very welcome

Michael Kassner
Michael Kassner

That's exactly the sort of information I need. It allows me to offer more choices with a reasonable amount of assurance. I really appreciate the information.

ScarF
ScarF

Well,... :) In my case I am the only IT personnel employed by my company. I support 8 servers and 50+ users/workstations in two locations separated by 3000 km (2000 miles), and I don't feel overwhelmed by the work requirements. In the case of a company with no IT personnel at all the out-sourcing is the only strategy since some activities require a human operator - and, not for antispam only but for many others. On the other hand, Canit - which I know the best since I use it, doesn't require extensive IT knowledge. A normal user may be able to look in the queues and take actions with minimal training. The server itself is the forgotten kind covered in dust, the administration is done using a web application and everything is straight forward. Of course, the server may be administered remotely - through secured channels, by anyone. Need to mention here that the provider's support is excellent and the application comes with "readable" manuals.

Michael Kassner
Michael Kassner

I agree with your comments whole-heartedly. Yet there also seems like a certain due-diligence is required. Almost to a point where there needs to be a dedicated support person. What would you suggest for entities that do not have any internal IT support? Or small companies that have minimal IT departments that totally run ragged by the mundane details.

ScarF
ScarF

Cyclops116 emphasized very well the main advantage of having such a service in-house. The company's e-mails represent sensitive information, and according to my principle of opposing anything "cloud computing" - especially when it comes to the sensitive information or company data, it is preferable to keep it inside your network. In my case, using Canit, users have nothing to do but report to me in the rare event of a spam not trapped by the system. For me, it is a 5-minute task twice every day to check and clean the queues and, maybe, add some rules. This isn't a big load. The rules provided by Roaring Penguin are very good since it is rarely when a spam message reaches the user's mailbox, and there are very few false positives. I plan for the near future to purchase a content filtering solution and the Webwasher from Secure Computing, for better protecting the users who browse the Internet.

Cyclops116
Cyclops116

The only thing I see as an advantage is the control you have in house. Off site means that all of your emails are going through someone elses servers and they may be doing something with it with out my knowing, there's no way to tell if one of their employees is doing something on their end maybe they are reading your stuff. There was a case a month or 2 back about the guy that got mad at his company so put a script on to the exchange server to forward all emails to his gmail account, then he released them to the press, that kind of stuff worries me. If one of their employees gets a bug up his bu++... I assume they know what they are doing on their end and since I've managed a system myself and I don't want to do that anymore! So it's worth the $2.50 a month per person to me not to have to do it!

Michael Kassner
Michael Kassner

I really like it when I get all these comments. I learn about applications and hardware that I didn't even realize existed. Very cool.

Michael Kassner
Michael Kassner

Thanks G-Man, That's exactly what I wanted to know. That way it doesn't require a special version.

The 'G-Man.'
The 'G-Man.'

and use the option of VSS support to take a copy of the active databases while imaging meaning that they never go off line. Same for SQL server & Active Directoty.

casternj
casternj

I never deployed it or started testing it. just was reading some white notes

Michael Kassner
Michael Kassner

So you aren't using the specific image application for Exchange? Just the generic Acronis application for the various servers? I would rather do that, as it would be a much easier sell.

casternj
casternj

i read there is an exchange version but you should make sure it doesnt require the server to be put down.

Michael Kassner
Michael Kassner

Where you referencing the Exchange version? I had some experience with that once and it created some problems.

Michael Kassner
Michael Kassner

I will store that away in the gray matter, to be sure. The client I referred to in this article would be in serious trouble with that much demand for bandwidth.

casternj
casternj

Yes that was one detail I forgot that endpoint had various installs for the options you chose. You mention GFI MS, I recommend it and advise you to get multiple virus scan engines with it. Dont count on one.

Forum Surfer
Forum Surfer

Exactly the same thing with unmanaged/managed choices. I no longer work at the same company so I can't go and get screen shots for you. It defaulted to unmanaged just like 10 if I remember correctly. The unmanaged part is nice for laptops but if you are blissfully unaware like myself, you will have severely high internet bandwidth usage. Symantec did a great job in my experience, but it had its fair shair of odd quirks. I've since switched to a networking position at a different employer and I can %100 say that I don't miss playing with symantec or any other spam filters I experienced.

Michael Kassner
Michael Kassner

I had an associate that had the identical problem. I don't have experience with 11, sounds like I don't want to for other reasons. Is that check box the same thing as when you made the choice to be managed or un-managed when setting up 10?

Forum Surfer
Forum Surfer

Don't know if you've tried it, but somewhere buried in the console amongst symantec's "right click he11" is one lowly checkbox. It makes all the endpoint clients point to the symantec server on the local network for live updates. I had the same issue with huge chunks of bandwidth going to symantec's update servers. All the endpoint clients were going to the internet for updates. Once I found that check box, the internet bandwidth dropped to almost nothing for the updates. I felt like a complete tool after that and never really mentioned it, lol. I do keep all the mobile clients in a group that allows live update via the end client through the internet, but that's it.

JCitizen
JCitizen

I've seen NOD32 sense a virus in the middle of an install without even going to the scan phase and splat the malware in the windows environment just before completing a forced reboot! I am very impressed with it so far. Now the crud can't even get through my firewall despite the fact that I am using Comodo for that. NOD32 nails the malware before it can get through port 80, and it does so very quickly. ESET works faster than the malware can, it has been the first product to come out since 2006 that gives me true peace of mind on the security front. I still use a lot of other products (mostly free), in this mix though. I still say you can't substitute for defense in depth. Fortunately ESET works very well with almost any worthwhile product I've used, so far.

derekdcosta
derekdcosta

Recently upgraded from v10 to EndPoint and it's playing hell into our SBS R2 Server. The combined footprint of SMSE6 and Endpoint means having to add an extra 1Gb RAM to cope with the load. More alarming is the 2 apps clog our Internet bandwidth with MASSIVE traffic volumes(3gb last month) to and from the liveupdate servers. The Enpdoint Manager console also takes some serious getting used to. We got rooted AS we were rolling out the Endpoint solution.. it took 3 installations and 2 Release Candidate updates and a week of nail biting to get it right, meanwhile getting hosed by a user plugging in a flash disk on several PC's running SAV Client v10 prior to Client remote uprading. A local variant of W32.autorun was running amok on our network, unknown to anyone including SAV or Endpoint. Used a rootkit scanner developed locally (AVS2007)to clean out our LAN. Now testing GFI Mail Security to replace SMSE and will dump Endpoint as soon as I can source a product with a smaller footprint. Symantec products have served us well for over 6 years now, but this incident was too scary to be complacent any longer

Michael Kassner
Michael Kassner

Thanks for that advice. Symantec has been never intuitive on their web site. It difficult to follow their methodology of getting software downloaded. Alas, I don't think that company is going to want to start over though.

casternj
casternj

They have totally revamped it. The user interface is not intuitive. Once I started to work with it on a demo system I decided it was time to change to another provide. This and the engine slowing down pcs was my decision.

Michael Kassner
Michael Kassner

What are your exact thoughts of 11? I am just starting to look at it. It appears to be completely different. They talk about endpoints and it appears to almost be a AV application that uses cloud computing.

casternj
casternj

Just to add I noticed that symantecs engine really bogs down a system. NOD doesnt bog down a system like symantec

Cyclops116
Cyclops116

In all the tests Symantec is about 97% 98% just like everything else out there including McAfee, Trend, Nod32 and Kapersky. That being said Symantec has HUGE memory footprint and V11 is the biggest version. We have Symantec corp edition now but our subscription is coming up and I'm testing Nod32 now, there is no way I'm using V 11 it's just too big!! Don't care how well it works!

Michael Kassner
Michael Kassner

I'm researching for my next article on botnets and have been in conversation with IronPort as they have several leading experts in that field. So, I will have to chat with them about their products as well.

Dumphrey
Dumphrey

for the 2008 scamola crap.

JCitizen
JCitizen

This is why many techs suggests you turn off restore and expose your system files/folders before booting to safemode and running your antivirus/antispyware from there. I have defeated many a pest doing this with Spybot Search & Destroy also. If an AV/AS utility can't run in safemode somehow, someway, it isn't worth keeping; in my estimation. Of course you'd have to do this off hours. I suppose most enterprise can't afford a backup server.

stuoutlaw1
stuoutlaw1

I actually tried malwarebytes last night and avast, I have been using avg and spybot none of the three helped at all. A brand new install of avast did not even find anything but malwarebytes found 35 instances and removed it all in one shot

scav8tor
scav8tor

Used together these have worked for me in the past with the whole AntiVirus2008 and variations.

Editor's Picks