Broadband

Botnets: Web-hosting site closed down and spam drops 50 percent

Two contracted ISPs stopped providing Web hosting firm McColo with Internet access, which immediately quieted several major botnets responsible for almost half of all delivered spam. Learning how and why this was done is worth your time.

How does the total amount of spam drop by 50 percent in one fell swoop? To help explain, I refer you to Brian Krebs' (Washington Post) article "Host of Internet Spam Groups Is Cut Off," where he gives a high-level account of what caused the dramatic decrease in delivered spam. Krebs wrote a follow-up piece, "A Closer Look at McColo," that goes into specific details, and needless to say it's pretty amazing.

McColo reportedly just a conduit

Several sources have stated that McColo Corp is the major North American host for international firms that control millions of subverted computers. Reportedly, these botnets are used to deliver spam focused on selling pharmaceuticals, designer goods, fake security documents, and worse things.

Krebs even gets specific about the activity at McColo by quoting security expert Joe Stewart (whom I have a great deal of respect for) of SecureWorks in his second article:

"The upper right-hand section of the graphic highlights the numeric Internet addresses assigned to McColo that were used by some of the most active and notorious spam-spewing botnets--agglomerations of millions of hacked PCs that were collectively responsible for sending more than 75 percent of the world's spam on any given day. In the upper left corner of the flow chart are dozens of fake pharmacy domains that were hosted by McColo."

The above quote refers to the following diagram (courtesy of Brian Krebs and the Washington Post)

mccolo2.JPG

Details of McColo's involvement

Security experts aren't surprised, because they have known about McColo's involvement with botnets and spam for years. In fact, McColo has quite a solid reputation for reliably supporting command-and-control servers for several of the most prolific botnets in history. Once again, Brian Krebs brings this into perspective:

"Multiple security researchers have recently published data, naming McColo as the host for all of the top robot networks or "botnets," These include SecureWorks, FireEye and ThreatExpert.

Joe Stewart (SecureWorks) said that these known botnets: Mega-D, Srizbi, Pushdo, Rustock and Warezov, have their master servers hosted at McColo."

What happened?

In what I would consider unique circumstances, Global Crossing and Hurricane Electric, the two ISPs providing Internet access for McColo, took it upon themselves to sever all connections to the facility. What happened after that was dramatic to say the least. Check out the following graph (courtesy of SpamCop); I'll let you decide if this graph is more dramatic than those depicting Wall Street's performance over the past few months:

mccolo3.gif

What makes this situation rather unique is the response by the ISPs. It wasn't motivated by legal action, but due largely to Brian Krebs and other experts bringing it to the attention of businesses and the general public. Along with Krebs' articles, many credit Hostexploits.com's second annual Cyber Crime report (focused heavily on the activities at McColo) as incentive enough for the ISPs to shut down McColo. One of the ISPs, Global Crossing, declined to discuss the matter, but Krebs was able to get the following quote from Benny Ng, director of marketing for Hurricane Electric:

"We shut them down. We looked into it a bit, saw the size and scope of the problem the Washingtonpost.com was reporting and said 'Holy cow!' Within the hour we had terminated all of our connections to them."

No legal involvement

As I pointed out earlier, there's no legal activity being publicly acknowledged at this time. One can sense the lack of precedence, and Krebs makes mention of this fact as well:

"Also unclear is the extent to which McColo could be held legally responsible for the activities of the clients for whom it provides hosting services. There is no evidence that McColo has been charged with any crime, and these activities may not violate the law.

Mark Rasch, a former cyber crime prosecutor for the Justice Department and managing director of FTI Consulting in Washington, D.C., said Web hosting providers are generally not liable for illegal activity carried out on their networks, except in cases involving copyright violations and child pornography."

Simply amazed

This is my first encounter with what it actually means to shut down a command-and-control center for several high-volume spam botnets. It feels like a victory for the good guys. Yet it's actually just a drop in the bucket, when one looks at the overall picture. I'll let Nilesh Bhandari, product manager with IronPort explain:

"IronPort sees an average of about 190 billion spam e-mails each day. Then, at around 4:30 p.m. ET yesterday (when McColo was shut down), IronPort saw a huge decline in spam levels. For the 24 hour period ending Tuesday, the company tracked about 112 billion spam messages."

Gee, only 112 billion spam messages a day.

Final thoughts

Most experts agreed that this victory was going to be a short-lived one. Prophetically, as of Nov. 15, 2008, McColo was back on-line. Ironically, experts are divided about this. Some were concerned that shutting McColo down would force the bot-masters to locate the command-and-control servers at multiple hosting sites, making them harder to track. It sounds like the experts know about Sun Tzu and his quote "Keep your friends close and your enemies closer."

Finally, I'm still trying to comprehend the fact that shutting down one command-and-control facility eliminated 78 billion spam messages per day.

Need help keeping systems connected and running at high efficiency? Delivered Monday and Wednesday, TechRepublic's Network Administrator newsletter has the tips and tricks you need to better configure, support, and optimize your network. Automatically sign up today!

About

Information is my field...Writing is my passion...Coupling the two is my mission.

61 comments
PraveenBhalla
PraveenBhalla

You have a good site. Here a best service provider in India. Last week i have register 6domains from this service provider. It have a dedicated team to serve their clients. If you need any help to webhosting solutions and domain name registration. Log on to the website. http://www.tiainterweb.net/

harrylal
harrylal

Spamming has been around for a long time, even before the internet (e.g. junk mail). I have always thought that the way to control it, is to bill per each email sent. After all, the postal service has been doing it long before the internet. Junk mailers have gone to the internet because it's so cheap (and easy to abuse). Perhaps those who are in a position of control should consider a way for those who are currently abusing it, to pay for their mass electronic junk mail or at least pay more for it. I'm not against commerce, I am against abuse that borders on the criminal...

bigjude
bigjude

I've been wondering for several days what had happened to my SPAM. Seriously, with an email address showing at a website that is now a year old, I have been getting (as expected)a steadily increasing stream of SPAM. In the middle of last week it suddenly stopped. I would have received less than 10 pieces since then. The story demonstrates to me the extent to which a single decisive act of this type can impact on the life of an unknown individual like myself.

luenib
luenib

NOOOOOOOOOOOOOOOOOOO!!

rainmaker_68
rainmaker_68

dam great that means yahoo chat will be less bugged ..and i will have less worries when making remote connections.. not to say that there is not treats around the place. but 50% crap not to deal with is just fantastic

ben
ben

I have to ask some questions: 1. What if the experts were wrong, and the hosting company in question was a small business, effectively put out of business based on a news report? Benny Ng, director of marketing for Hurricane Electric, suggested it the washington post was his source, which of course has nevern been mistaken. But what if they were? 2. What provisions of the service agreements between bandwidth providers and the hosting service allowed for this action? Were those contracts honored? 3. Are legal contracts (such as service agreements) to be ignored when the "greater good" is involved? And if so, who should make that call? 4. The FCC recently has made significant rulings suggesting a particular bandwidth service provider (ComCast) can NOT filter internet traffic. Yet here is an extreme example of a provider doing just that. Does the ideal of "net neutrality" have limits? Again, if so, who decides what should be filtered? I fear there are greater things to fear than spam.

chris
chris

so how much money do these guys actually make from all this spam? Do people actually click the link and buy (or try to buy) stuff from these guys?

Dumphrey
Dumphrey

Ive been crazy busy here at work and haven't had time to peruse my usual news.

jmcgachey
jmcgachey

Amazing. As small company we were getting anywhere from 120,000 to 150,000 emails bounced off of our spam firewall daily. Last Wednesday it dropped by 100,000. I didn't think too much about it until the second day...same thing. I went to far as to email our ISP and ask them if they'd put something new in place but they were as puzzled as I. Thanks for the article...I finally got my answer.

Jaqui
Jaqui

When the fine print in internet access service agreements allows them to take anyone offline for being part of a botnet and for spamming? why?

Michael Kassner
Michael Kassner

I have heard of charging per email message before. I'm not sure if it would really resolve the issue or not. My reason for asking this is that most spam isn't sent from the entity that is producing it. A person with a subverted computer is and would get charged for sending all the spam. Is that right or not?

bigjude
bigjude

Half an hour after my previous post,I checked my email, and my Junk mail folder was chokka. So the SPAMMERS who used to send it to me, are obviously back on line.

Michael Kassner
Michael Kassner

I doubt that they even need financial help. The profits are staggering.

Michael Kassner
Michael Kassner

This is big business and as with any enterprise that is making money it's not that easy to stop. I was just at an economic seminar that had an IT flavor and they say the dark side of the Internet is going to get worse because of the poor economy. Spam makes millions and people are losing jobs, the math is easy.

bernalillo
bernalillo

I'd suggest that having a terms of use does not nessesarily qualify as filtering in the sense that the court cites.

Michael Kassner
Michael Kassner

The questions are good ones and I'd think the ISPs and McColo had lawyers make sure it was so. It is also my understanding that McColo keeps a real low profile and it's up and running again so, it was just a blip

samhill
samhill

The article says Benny Ng "looked into it a bit" and confirmed what washingtonpost.com reported. I'm sure it didn't take much "looking" to see that McColo was blatantly violating the terms of service. I haven't read Hurricane Electric's TOS agreement, but I'm sure it contains a provision that covers illegal activity, and Mr. NG was simply enforcing it. I don't think these actions had anything to do with ignoring terms of a contract for the "greater good". McColo breached their contract and suffered the consequences. To somehow equate these actions as an "extreme example" of filtering traffic is baffling to me. This isn't even remotely like what ComCast did. Bob Parsons over at GoDaddy.com is who you should be afraid of.

kgunnIT
kgunnIT

Believe it or not, people actually do click and open spam, and thinking it is real, go through with a purchase or releasing their personal information. I have to warn my coworkers all the time of spam, and that they shouldn't open every email they get or downloading attachments if they don't know what it is. It's a full-time job just trying to keep the network secure.

Michael Kassner
Michael Kassner

You were missed, but busy is most cases is better than not busy.

Michael Kassner
Michael Kassner

My initial problem was trying to figure out how much bandwidth McColo had in order to accomplish this. Then I woke up and realized it's our bandwidth that's getting used. All McColo had was command and control servers that have minimal bandwidth requirements.

csmith.kaze
csmith.kaze

Ours dropped by nearly 50,000 a day. Hopefully this will keep up. Our spam firwall has had a workout since we put it in (went from 30,000 to almost 100,000 over the past year.) back down to 30k~40k now. Not too shabby. Great article!

WindsorFox
WindsorFox

China and Russia don't play by the same rules.

Michael Kassner
Michael Kassner

The problem is that it's a colo with I suppose significant legitimate business. I also suspect that there are other ISPs willing to provide access at a price. The news that they were up is pretty recent, and I haven't heard much more than that. Their home Web site is still down, though.

mickey
mickey

under the terms of the Canned Spam Act you can sue up to $2000 per email after the first time you contact the sender to stop. The ISPs are knowingly allowing it. The 1st time someone sues successfully the ISP just watch the other ISPs fall inline. If they knew they could be held liable they would look at the situation differantly. Remember McColo were sending out 110 Billion emails a day. If the average person got only 10 SPAMs and made the contact on the 1st: thats 110,000,000,000 - 10% or 99,000,000,000 x $2000 = $198,000,000,000,000 in law suit payouts for just 1 days spam as stated in the law. How many ISPs would keep them in business with THAT president ?

Michael Kassner
Michael Kassner

It doesn't take long for the entities in charge of the botnets to find another hosting source. Once the command and control structure is back in place the bots come back on line. Which is what we are now seeing. The researchers themselves stated that this was going to happen and their work starts all over again. If you are interested to keep tabs on the amount of spam, SpamCops have some great graphs like the one in the article. I've linked the site below: http://www.spamcop.net/spamstats.shtml

anne.powel
anne.powel

The other article I just read stated that McColo had found a Swedish hosting site, but had been shut down there after only a short stint because the researchers had contacted them as well. Hopefully they can chase McColo around and either get them to stop or to behave as a colo truly should??

Michael Kassner
Michael Kassner

I've been trying really hard to get more details on this, but to no avail. So your comment is justified. Hopefully the facts will eventually surface.

mb.techrepublic
mb.techrepublic

Even if you don't want to say yourself here (or maybe you do), do you have any links pertaining to why we should be afraid of Bob Parsons? Michael - another good article - thanks.

Dr Dij
Dr Dij

is not always about buying something. They may just want to infect your computer to add to their botnets

chris
chris

do that many people really want to be "larger"?

sergiortc
sergiortc

As a micro web hosting company in Nicaragua we were stopping around 1000 spam and phishing messages a day (we manage +- 100 email accounts). We experienced a 75% decrease last week and I was wondering what was wrong. I even spent a few hours trying to detect a bug in our in-house spam handling application. Thanks for the article. Now I know the cause. A shame it seems it won't last.

Michael Kassner
Michael Kassner

This is why I consider the advent of the Internet a civilization-changing technology. I'm also an optimist, I see this whole process as a path the people of the world have to travel to ultimately come together as one community.

ben
ben

There are too many options and no way to cut it off. There have been many legal cases bumping heads with the nature of the internet, which crosses jurisdictional boundaries. A Judge in CA was POed when he ordered a page taken down, only to find mirrors of the site were still up...hosted The FCC has struggled with this also, and Congress failed to grasp the problem with the "communications descency act" of 1996, which of course did not stop sites hosted outside the US. In a way, the internet community presents us with the first practical global anarchy since the invention of governments.

Jaqui
Jaqui

I should send the url to the RCMP for the articles and reports. Under Canadian laws about child porn they could probably force service providers here to drop traffic from them. the laws about it up here are so broad in the definition that a stick figure sketch can be described as child porn, one done in MS paint even. They have zero tolerance for it. [ which is a good thing, really. Though it could also make a laughing stock of the Canadian Legal system. ]

Michael Kassner
Michael Kassner

The information that's available is not specific enough. From what I have been able to gather McColo was not the source of the email. That was the botnet members. This is the conundrum. Is the command and control the source of the email or is it the actual individual computer that's sending the traffic?

Michael Kassner
Michael Kassner

Hello, Anne Could you point me to that article please? The reason I ask is that I suspect it's the entities that were leasing servers at McColo that are moving around trying to find colos. McColo is just a colocation center that rents servers and storage. As such, it may be why they are not going to see any legal action against them. I haven't heard any information on those entities, hence my interest.

Michael Kassner
Michael Kassner

I just had lunch with a friend in marketing and found out that spam is a whole different cost center. It's the cheapest way to advertise known to man. Even still the return on cost doesn't have to approach what marketing types would consider normal. It started to make sense after I thought about it.

Michael Kassner
Michael Kassner

I try to get a clear picture of what's happening and what I think should happen and the results change daily. I just hope that it works out, I'd hate to lose this amazing tool we call the Internet.

Michael Kassner
Michael Kassner

I suspect that this is yet another example of the law not being able to keep up with technology. It may sound strange, but I think that's a good thing.

bernalillo
bernalillo

any jury properly presented by the defence with the evidence and it's interpretation would aquit. Poorly written law prehaps but not a danger to the accidental browser.

Michael Kassner
Michael Kassner

I agree with your premise and I'm by no means taking any sides. My concern is that sometimes the command and control traffic passes through VPNs, or uses other encryption means. That would make it difficult to determine what the traffic is. Also newer botnets are using HTML traffic that's very hard to distinguish from normal Web server/client traffic. Ultimately, I'm amazed at how researchers are able to track this traffic down with such precision.

ryumaou@hotmail.com
ryumaou@hotmail.com

I think it would be more like charging the Hertz employee as an accomplice if they knew the car was going to be used in a robbery and rented it to the crooks anyway. At this point, it's pretty obvious that this colo company is aware of the problem. It's quite likely that they even know which servers are the specific command-and-control servers, based on traffic analysis that I'm sure they do. So, based on that information, they're accomplices to whatever criminal activity is being coordinated by those servers, whether it's actually spam distribution or simple control over the machines that are doing the actual e-mailing. So, yeah, I think they should be shut down until they take responsibility for their customers whom they are allowing, even facilitating, to break the law.

Michael Kassner
Michael Kassner

It's kind of interesting and ultimately will drive how spam is taken care of, I suspect.

Jaqui
Jaqui

that the laws need to be toughened up, so that the spammers face ruinous fines and jail time for spamming. The issue is that the internet technologies weren't designed with the current issues in mind. not only does it make it harder to control the spam, it makes it harder to legislate. any law designed by non IT people to stop spam would most likely also hit legitimate uses also. [ The E.U.'s proposed criminalization of "hacker tools" being an excellent example, since the "hacker tools" are IT pros tools also. ] I'll look at the laws in both countries about it and give a more detailed response later on this week. Though the "legislative body" here in Canada that should be doing something about internet stuff won't touch it at all. The Canadian Radio and Telecommunication Commission [ CRTC ] refuses to try to put anything into effect to enable law enforcement.

Michael Kassner
Michael Kassner

I have a great deal of respect for your opinion, Jaqui. So what do you think of the rules in Canada and here ? What would you change?

basstrumpeter
basstrumpeter

You don't think that spam is against the public good? Or that a host is ignorant of the actions of a huge percentage/number of clients? The crime is stealing resources from millions of PCs throughout the world and the host to me is an accomplice.

Jaqui
Jaqui

little sisters is a gay book store here in Vancouver. :D yup, Canada has the tightest legislation for child porn in the world. It is over the top in many ways, but they don't charge someone who finds it and reports it, even though they know that person has it in the browser cache and therefore has it in their possession. The legal system chooses to ignore that, under those circumstances.

Timbo Zimbabwe
Timbo Zimbabwe

... and you have to prove (beyond a reasonable doubt) ownership of the illicit code. Just having it on the servers shows no criminality to the host, as it were, because they could be as unaware as anyone, legally, about what is residing on their servers.

Beoweolf
Beoweolf

I am all for shutting down Spam operations. However, I would expect (ask) that Laws have some actual, profound basis in favor of the public good, before declaring an activity, which is in fact is only a mildly odious, as "crime against Humanity". Somewhere between the "War on Drugs", MAD (Mothers against Drunk Driving) and mandatory seat belt and helmet laws - we have, as a global society, decided to legislate morality, choice and freewill.... with accompanying hefty fines and exorbitant "sin taxes" thrown in for good measure. Sometimes I wonder if the crime is in theever evolving enforcement rather than the crime itself. Seems to me, if there really is a crime, then no amount of money should be allowed to dissipate the infraction with an ?indulgence? fee.

keith
keith

That's a bit steep isn't it? I had a member of staff who was researching a product we sell do a search for Little Sister. You can imagine the sort of results he got. It looks like in Canada he would be in a lot of trouble. By the way a Little Sister is a sterilizing unit we sell to hospitals and Dental practices.

Jaqui
Jaqui

the Hosting company is in violation, if the content is on their machines. You are clicking links on sites and come across a child porn site, report it to the appropriate legal authorities yet are guilty of possessing child porn, because of your browser cache here. C.C.C. = Criminal Code of Canada. The anti spam laws are weak, but the child porn laws hold anyone involved even peripherally as guilty until proven innocent.

Michael Kassner
Michael Kassner

I think that's where the problem is here. Is it the colo or the firms that are using the colo's equipment who are responsible?

Editor's Picks