Broadband

IPv6 blacklisting: Will it be harder to fight spammers?

The reason IPv6 is needed may cripple blacklisting. FUD or for real? Michael Kassner investigates the potential for harm that IPv6 could pose for fighting spam.

Why is there so much negative karma surrounding IPv6? Case in point, last week, a colleague of mine called, "Have you read the Register yet?"

No, some of us have to actually work during the day, I replied, with more than a little sarcasm.

"Well, your highness; when you get a chance. Read it."

My friend knows which buttons to push. I had to find out what he was talking about. I opened my Register e-newsletter and right at the top: IPv6 intro creates spam-filtering nightmare.

More bad vibes

Wait a minute. That's only one source. After checking, other media outlets seem to agree: "New IPv6 protocol could complicate e-mail spam filtering" and "Will IPv6 render blacklisting obsolete?" Whoa, it is big. The fact that IPv6 has an available address space I can barely fathom may screw up blacklisting.

I had to get on this bandwagon. Jumping into action, I shot out emails to my special corps of experts asking them:

Will blacklisting become obsolete because of the obscenely-large address space created by IPv6?

Anup Ghosh (Invincea): IPv6 does create a much larger address space from which spam and malicious sites can park themselves. However, current techniques for blacklisting known spamming/malicious sites will likely be unaffected. Blacklisting techniques are largely agnostic to the size of the address space.

An IP address is added to a blacklist when the address is verified as a bad site. The issue with blacklisting is by the time a domain makes it to a blacklist, its IP address has already changed.

Giorgio Maone (NoScript): Blacklisting has always been the weakest form of protection in security, on principle. A much larger address space just makes this more evident. But, it's hardly news. For example, Mark Ranum, father of the firewall explains in this old editorial.

I believe statistics method to recognize spam, e.g. Bayesian filters, are the only really scalable solution, for now at least.

Joe Klein (IPv6 Security Researcher): The IPv6 spam-list problem has been solved for some time. Many of the non-IPv6 aware blacklist companies currently block a single IP address in IPv4. In IPv6, each home user receives a single /64 bit address, where the first 64 is the user's unique network and the last bits are the user's local network. All the black list needs to do is block the network or the first 64 bits of the address.

I have been teaching this information in my IPv6 hacking class for over a year.

Johannes Ullrich (SANS Internet Storm Center): Blacklisting will have to be thought over when it comes to IPv6. In IPv4, blacklists for the most part, list individual IP addresses. This will not work very well in IPv6.

In IPv6, the address is broken down into two parts: The first half is addressing the subnet. The second half is addressing the individual host (interface) on that subnet. A user could pick any address within that subnet and some operating systems will pick a random interface ID whenever they reboot to assist with privacy.

This "second half" is 64 bits long and allows for 4 Billion squared possible combination (the entire IPv4 internet only has 32 bits or 4 billion worth of addresses).

I think blacklisting will need to be able to block subnets. That way, it doesn't matter which IP address within the subnet the spammer uses. IPv6 luckily uses fixed subnet sizes (/64 being the smallest, and /42 typically assigned to organizations). This may lead to some collateral damage but it is probably the only way to make blacklists effective.

On the other hand: Blacklists haven't really been that terribly effective in the IPv4 world. Maybe we will finally think about more systematic spam fixes than blacklists.

Cameron Schmauch (EdgeWave): In my opinion blacklists and whitelists have been obsolete for several years. This is not so much because it's hard playing whack-a-mole with the spammers, but rather that the lists usually aren't maintained in such a way as to aggressively prevent False Positives (FP).

EdgeWave (Powered by Red Condor) hasn't relied on third-party blacklists for anything other than supplemental information. We do employ methods for sussing out IP blocks that are operated by spammers. Blocking on IP can be very effective and efficient, but it should not be the mainstay technology if keeping FPs to a minimum is your top priority.

In our records (which go back over half a decade), about 22% of our categorizations were due to matching IP rules (not necessarily origin only). Over the past year, that figure has plummeted to only 6%. The reason this statistic has dropped so dramatically is spammers are already good at usurping the resources and reputation of others to get their dirty work done.

You can't outright block mail from abused addresses if legitimate mail also uses those paths. You have to rely on more sophisticated techniques because a single dimension just doesn't provide enough information to make an absolute decision on message disposition in many cases.

Kassner: Since spam and malware filtering is a big part of EdgeWave's business, I decided to ask Cameron a few more questions.

Do we need to worry about it?

Short answer: I'm not worried about it. We have largely abandoned such lists. Much of the worry seems to be simply about the size of the new address space. However, from what I've seen reported on this issue, many tacitly assume that spammers will somehow be able to pop in and out of this address space at random.

Yet, practical limitations inherent in doing this could prove more difficult than is actually warranted for the spammers to get their messages delivered. Spammers, like all other things, follow the path of least resistance.

Moreover, this is an old trick. Various spammers already tread very lightly with respect to their address space holdings, sending lower volumes and using what I like to call "IP crop rotation" to avoid using the same addresses too often which would otherwise make them easy targets for blacklist maintainers.

The real Achilles's heel of spam is in its volume in combination with a specific intent over small time domains, not simply where it comes from.

The other thing is that IPv6 adoption will likely take many years. I don't see any problem with mail administrators "blocking first and asking questions later" with regard to inbound IPv6 connections. This could be in the form of aggressively gray listing IPv6 connections from unknown sources, or outright blocking until reputation has been established.

That's sort of what I imagine will come out of best-practices from the early adopters. In terms of our solution, we've believed that the best way to reliably filter unwanted mail without collateral damage is with multiple layers of defenses, behavior analysis, multi-scale feedback systems; and of course, human-in-the-loop real-time analysis. Anyone who is still using IP blacklists as a primary filtering solution is likely already having a bad time.

Do you see any other issues with IPv6?

I think its adoption will exacerbate short-comings in the DNS system, routing and miscellaneous security systems for example. But those problems are somewhat mundane.

The more interesting impact, I think, will be from the fact that everything will start to come online. The vacuum of such a large address space with the advent of embedded computing and cheap bandwidth is the far more interesting aspect of this whole transition.

Final thoughts

Well, this piece ended up in a different place than I thought. It seems IPv6 blacklisting will work in a fashion. More to the point, it's obsolete. My expert resources came through again. I can't thank them enough for their willingness to set the record straight.

Now, you will have to excuse me. I need to make sure someone reads this.

About

Information is my field...Writing is my passion...Coupling the two is my mission.

57 comments
macgvr
macgvr

One of the problems we face today is that the Internet was developed with the mindset of trusting everyone. Worked great when it was just arpanet. Given the lack of security in the original design it is not surprising that we are having trouble today. One of the supposed additional goals of IPV6 was to try to deal with that inherent, "trust everyone", problem. I get what one respondent said in that during the transition phase a lot of things are being done that compromise the security that IPV6 could supply. We are also faced with the problem of how much privacy do we give up in order to gain security. There are many out there who do not want any accountability to anyone, and not just spammers. However, in order to be able determine who is trustworthy and who isn't will require less anonymity and more accountability. Of course governments are going to love that since that is what they really want, as we have already seen in China, and what our American government would like to do. I agree that blacklists and such are not as effective as we would wish, sooo what do we do? We can go on as we are or we can change things such that spammers and other scam artists can't hide behind the anonymity of the Internet. And yes, I realize that even legitimate systems can be compromised but they could be identified and dealt with. The bottom line is that IPV6 is coming and avoiding it is not an option. The question I have is, can we use this new system to improve our experience when communicating on the Internet? Can we achieve a reasonable balance between personal privacy and accountability? Given the disparate views out there, achieving consensus will not be easy but I don't believe we have the option of doing nothing or of continuing to do what we are currently doing.

jgustafson
jgustafson

I have long advocated blocking all IP addresses and only allowing email through from trusted sources. An example of this is a business contact that you made at a conference you bring back their card and have someone responsible for adding them to the allowed list. This will eliminate the spam and unwanted email as well as saving you money for a subscription to spam services. If someone is trying to get email to us and we don???t know who they are than we don???t miss any email. If they are a legitimate company looking to do business with our company then let them call us or stop by and visit. No one trusts a new sender anyway at least in our case I have trained our staff to disregard email from someone you do not know. Less instances of viruses being spread if it is someone you know and less waste of time as well. If someone sends me an email and it gets bounced back then they call me and ask what the issue is. Or some coworker talks to me and says so and so talked to me and said their email is bouncing back. Then i take the proper steps to fix the issue. Any case the point is, do we really miss email from a source that has never sent email to us? How often do we do million dollar deals with someone that has sent us junk mail or spam as advertising? If we are good about adding the new business contact to our allowed list then business will go on as normal or perhaps better.

macgvr
macgvr

One of the things I heard early on is that ipv6 would have better security which was supposed to help deal with people, like spammers, who want to use someone Else's identity to do their dirty work. It was supposed to be harder to spoof someone's ipaddress and there were supposed to be other security features I've forgotten for the moment. Is this not true? Can anyone enlighten us?

Michael Kassner
Michael Kassner

If you have any questions, now is the time to ask them. A hearty thank you to Cameron for helping out.

charlot
charlot

Can we replace IPv6 "blacklisting" with IPv6 blocklisting. It's more accurate.

seanferd
seanferd

The address space is bigger. So... what does that imply, exactly, other than there being more addresses? That lists will just be that much more unwieldy for someone who wants to eyeball them for some reason? I notice that PhilippeV mentions the v4 to v6 tunneling schemes. While my assumption was that the original issue was taken with "straight" IPv6, I rather tend to agree that these are ugly beasts for which the sole purpose seems to be gumming up the output of ipconfig /all on Windows machines. (Yes, that is light sarcasm and not a serious analysis in the second half of the sentence.) Really, though, pick one - v6 or v4, and forget the protocols for in between. From El Reg: "The primary method for stopping the majority of spam used by email providers is to track bad IP addresses sending email and block them ??? a process known as IP blacklisting," explained Stuart Paton... It rather sounds to me that this is not even true. I know for a fact that some people rely on various *BLs, but is that really the primary method for ISPs and organizations with large networks? "As an example, the address space is so large that it would be easy for spammers to use a single IP address just once to send a single email," he said. OK. So why can spammers assign themselves so many IP addresses in any IPv(x) system? This doesn't sound like a problem with the size of the address space. Quoting Joe Klein from Michael's article: "In IPv6, each home user receives a single /64 bit address, where the first 64 is the user???s unique network and the last bits are the user???s local network. All the black list needs to do is block the network or the first 64 bits of the address." So it would seem that part of the problem is apparently in the way in which IPv6 addresses are assigned by... whom? Is this the law, as it were, or a choice of the upstream netblocks holder (ISP or whomever)? Regardless, the problem seems to have already been solvled before it has even occurred: block the spamming network where it meets the public network in IPv6 space. Regardless as to the address space in use, and considering that ISPs have these blacklists, it would seem that the real problem is that ISPs do not take action against spammers whose IP addresses are in the ISP's netblocks. Of course, botnet-mediated spam would require a bit more subtlety when dealing with a customer, but so what? Always interesting to see what ISPs will spend their time and money on, and what they won't, eh?

kevlar700
kevlar700

"I think its adoption will exacerbate short-comings in the DNS system, routing and miscellaneous security systems for example. But those problems are somewhat mundane." Or is ipv6 the problem. I use but don't rely on blacklisting but if someone wants to contact my mail server and won't use ipv4 they can use my ipv4 webform. As I've thought and found the OpenBSD project leader had already said, we need ipv5. Here's a small part of why, that I read recently after I had already decided ipv4 was better. "http://marc.info/?l=openbsd-misc&m=129666298029771&w=2"

PhilippeV
PhilippeV

The main problem is fully in the hands of ISPs, and not caused by IPv6 itself. ISPs have constantly pushed IPv6 promoters to adopt a "gradual" evolution from IPv4 to IPv6, but they are so late in their deployment that they are now promoting proxying solutions, i.e. protocol converters, between end-users (the most important source of pollution the Internet, as they are the least protected, and have to constantly manage their connection using tools that most of them definitely cannot manage themselves) and backbones. So now Ipv6 is starting with lots of protocol converters acting as supplementary proxies that are comlpletely out of control from the public Internet side: 6to4, Teredo, and now LSN (Large Scale NAT). With IPv4, at least NAT was deployed on each end-user connection, so that it was easy to isolate. But the situation will be extremely difficult with the proxying servers now deployed by ISPs worldwide (and notably by mobile internet providers), because it's almost impossible to isolate the pollution passing through a proxy server (6to4, Teredo or LSN), due to real lack of cooperation by ISPs, and their desire to maintain their customers captive in their local IPv4 networks that they are deploying to end-users. Let's just say that we don't want proxies/relays. IPv6 was build so that EACH end-user would have a dedicated IPv6 address block which is easily traceable and routable, and easy to isolate in case of problem on one connected source. We want a full native IPv6 deployment. The transition proxying technologies have never been designed to be used on large scales, and anyway they cannot even fulfill the need for an always increasing data throughput for many more applications. None of the relaying/proxying solutions correctly support the IPsec security suite. All of them suffer from stability problems, routing instabilities, easy injection of data, easy to exploit. With native IPv6 we would get AT LEAST the same level of protection offered in IPv4. In fact native IPv6 should be used as a solution against spam as it will really ease the management of routing tables, allowing fast identification of compromized or abusive end-user connections. All the security problems that are appearing in IPv6 are caused by the initial massive use of proxying/relaying protocols, exactly the same ones that are already causing problems in IPv4, except that the expected initial deployment of IPv6 will massively use these technologies. Transition technologies should have only been used for initial creation of the backbone in order to allow building enough peering interconnections between networks that make up the Internet, and a few, well identified users that have enough technical knowledge or monitoring tools to detect and resolve problems very fast in a collaborative way. NAT could persist for long for interconnecting private LANs owned by end-uers, as long as the NAT router remains ONLY within this private LAN. But for interconnecting them to the IPv6 internet, there's an aboslute need to have native IPv6 deployed up to the end-user's router. All solutions offered by ISPs on their own shared servers will cause unsolvable problems. But IPS still tend to propose this evolution because it creates captive situations where the ISP can decide what their customers can see and with whom to interact on the Internet, but using their own decision in a non-collaborative way. Dozens of years have been invested in isolating the "open-relays" in the IPv4 network. But 6to4, Teredo and LSN servers are creating a new situation where these open-relays will have a massive traffic which will not be manageable with the existing security tools. Give native IPv6 to everyone, and forget 6to4, Teredo and LSN deployed by ISPs (they should have only been used, just like NAT over IPv4, in the configuration of routers configured and managed in the private network of each responisable end-user, only as a facility of deployment for these end-users). ISPs have forgotten their role: offering the best interoperability and excellence of service. Not only their relaying technics are extremely bad for the security, but they still don't have any tool or collaboration platform to solve in a timely manner the many problems that exist in the Internet, because nobody else than them will be able to isolate the sources. Internet was born as a collaborative end-to-end network, where ISPs were only involved in managing the bandwidth, but not the data exchanged themselves. Unfortunately, the legal framework of ISP is dramatically not pushing them to do anything, because laws are still putting them out of legal suites due to the bad data sent by their clients. As all waht counts for them is to capute a maximum number of clients, they absolutely don't care about what kind of traffic they are pushing to other ISP networks (and anyway, they don't have the personnal and technical infrastructure to implement the collaborative framework needed to fight and isolate security issues). So, just to avoid massive problems, they are now implementing protocol filters, for example restriting connections to emails and HTTP(S), notably on their mobile 3G+ networks, and they are in fact creating lots of interoperability problems. They have effectively reduced the level of collaboration. End-users can no longer act directly, and just have to "hope" that their ISP (or worse another foreigh ISP) will act on their complaints (but these ISPs will not be able to do lots of things, because it becomes extremely difficult to describe the source of problems, and ISPs are hiding the identities of their connected clients). The only solution would be to use secure connections (i.e. banning classic SMTP through 6to4/Teredo/LSN relays/proxies, as well as unsecured HTTP), but the relaying technologies currently deployed by ISPs will not support this (it would be much more costly than deploying native IPv6 directly to end-users). In one word: blame the ISPs for refusing to deploy native IPv6 up to the home of their end-users. They have no excuse, because all these same ISPs are already connected to the IPv6 backbone. And buying large sets of IPv6 address blocks for all their customers is really not expensive (compared to IPv4), and allows an easy management of routes (with much more routing stability than in IPv4). So let's not criticize IPv6, just the "transition" technologies that ONLY the ISPs are pushing as a false long-term solution for everyone...

gearond
gearond

White listing, verified by exchanged keys and certificates, and non encrypted email secured with SHA1 hash is really the only way to truly block all span. And no one wants to do it. Kind of like global warming. The crap is going to hit the fan, and there's going to be a lot of whiners and losers. But no one is willing to do anything about it. So, let's just all suffer :-)

swade
swade

Our RBL has been INCREDIBLY effective in our case. This long-time domain is subject to withering amounts of spam (20,000 messages a day; 600 of which is generally not-spam), and 80% of it is instantly blocked by simply using an RBL. Based on my experience with RBL, I would imagine that IPv6 isn't going to make any real difference here but I would be curious if some of these experts would agree, or even discuss why RBL's might not be effective.

pgit
pgit

The more interesting impact, I think, will be from the fact that everything will start to come online. The vacuum of such a large address space with the advent of embedded computing and cheap bandwidth is the far more interesting aspect of this whole transition. That's my take on IPv6, the real concern is going to be privacy and control of one's "papers and effects." The day when even the ripening garbage barrel in the picnic area at the public park has an IP address, we'll all be fish in a barrel for kidnappers, pickpockets, thieves and other government officials. I see "legitimate" technology as being a bigger threat than spammers, going into the brave new world of a future we seem to be getting.

Michael Kassner
Michael Kassner

Are not that secure. Also, many people dismiss those that are out-of-date or have a problem. Interesting though, as it might be better than what exists now.

Michael Kassner
Michael Kassner

The interesting thing I found is that security enhancements available for IPv4 are not being used. IPv6 has even more available and the experts are concerned they will not be used either.

Michael Kassner
Michael Kassner

The only problem I see is total trust on email that gets through. I have seen many cases where a company's email server was pwned and email with malware links got through filters.

bboyd
bboyd

It might complicate port scanning but beyond that I don't see the other improvements as significant to security. Anyway, converters for back compatible IPV4 will make that moot until IP4 use decays.

jsklein
jsklein

IPv6 is no more or less secure the IPv4. In many cases it is less, do to: 1. Lack of IPv6 security tools, such as host/network firewalls, IDS/IPS, vulnerability scanners, log management, anti-malware, SIEM, etc, where the vendors are waiting for a need and high request from customers before they even start working on implementing the base line requirements. At present very view vendors are at parity with the IPv4 tools and no vendors can deal with IPv6 uniqueness. Remember the security industry uses the CDC method of designing features into their tools, that is, as soon as someone is compromised then you begin development on a product feature, even if the compromise is obvious. 2. ISP forcing tunnels on users with little or no notification of the risk or limitations. In general, the management of most companies, based on recommendations from well know consulting groups, to delay implementation of IPv6 until the last moment, to preserve profits for the company's. This as forced ISP's to do work around to implement minimal IPv6 features. See my slides from IPv6 Next Home.pdf located http://tinyurl.com/4d8a8xl , to understand the issues. 3. People attempting to apply IPv4ish architecture, policies, procedures and technology issues on IPv6. IPv4 is not equal to IPv6 and requires understanding of host interactions, local/remote network interactions. Currently there are many assumptions of how IPv6 operates, and many are wrong. A good example is the whole black list issue which started this discussion. Expect a bunch of announcements after the SANS - "Security Impact of IPv6 Summit" 2011 for http://tinyurl.com/4lrh9s2 Joe Klein #joeklein

Michael Kassner
Michael Kassner

I think that would be a good idea. But, I am so not the one who decides. I will pass your comment on and see what the SMEs think. Thanks for sharing that thought.

Michael Kassner
Michael Kassner

Is that with IPv6, spammers can consider IP addresses to be throw away. Use one for a spam run and never use it again. Black lists would not work in that scenario.

bboyd
bboyd

Complication will happen but it is still worth the effort to adopt now. Its no longer the early adoption phase and IPv5 is pretty much dead. I think the IPv4 backward compatibility is where the pain points will come. Wish people would explain their opinion when the down vote...

Michael Kassner
Michael Kassner

Has several flaws. I also doubt security will improve significantly if we stay on IPv4, especially with regards to DNS issues.

pgit
pgit

I have been having network problems with an ISP lately. It happens to be my ISP but the issue is really with clients relying on the 'web' for business. The last few weeks we've noticed, and I've heard a lot of complaints, about slow downs and DNS resolution failures becoming all too common. At the same time mail, instant messaging and even bit torrents are unaffected. It's only 'web' traffic. Even google comes up unresolvable sometimes. Reading your explanations above I have to wonder whether what we are observing here is resulting from the ISP's attempt at deploying IPv6 compatibility? I'll have to ask, should it get bad enough that I have to talk to a real live person. Meantime, is there any way I can poke and prod around and determine what kind of technologies the ISP is deploying vis IPv6? I've tried some of the on line tests for compatibility and get the result that my local segment of the ISP's networks are not IPv6 compliant. Am I to assume then that they are applying something as you describe? A conversion of some sort? I can see how there will be tremendous resistance to deploying true IPv6 down to the end user on that part of my ISP. I imagine wireless carriers are doing a better job at this.

Michael Kassner
Michael Kassner

That was the just of the article. Your other points are well taken and agree with Cameron's viewpoint. The problem is that many spam-blocking procedures were available for IPv4 and not used. So many experts feel the same will happen with IPv6.

cameron.schmauch
cameron.schmauch

Without addressing the practicality of what you mention (it falls under the category of: X would be solved only if *everyone* did Y), what do you do when a cert you trusted at some point is stolen or abused? What do you do when that friend you trusted suddenly stabs you in the back without warning? Identity theft takes on broad meaning in the digital frontier. Run away from anyone touting trust toys as any kind of general solution to security problem X.

Michael Kassner
Michael Kassner

I am interested in learning more about your statement "Crap is going to hit the fan" and what you mean by that. Please help me out.

cameron.schmauch
cameron.schmauch

The vast majority of spam is trivially easy to block accurately and efficiently. It's the stuff that remains after the large quantity of low hanging fruit that represents the perpetual thorn in the side. Everyone's mailstream is different. It might seem like my spam shouldn't be any different from your spam, but in practice, that's just not the case. RBLs may work well for you, but in general, they do not. You might not even have noticed the good mail you missed because you blocked it at the session. You might never have had to deal with getting off of a blacklist after suffering a loss of control over your IT resources. You probably don't suffer continuous attempts by scammers to dupe your users into divulging their webmail credentials. You probably don't have much of a spam egress problem. You probably aren't responsible for protecting hosts distributed all over the globe. You probably aren't going to be crippled if you cannot filter mail because the RBL you are using goes down under sustained attack, or simply for maintenance. The bottom line is simply that there are a lot of variables in play for such a simple protocol, and these are just some of the problems anti-spam companies face. RBLs will continue to exist IPv6 or no, it just boils down to whether it is the right tool for the job you face. Lots of people still use client-side "solutions" to deal with spam (which makes me shudder), but for some, it works. But when you are a company who's stated purpose is to solve the spam problem for your customers, RBLs just don't cut it.

Michael Kassner
Michael Kassner

I have not thought about RBLs for about 6 years now. You raise an interesting question. I will see what I can find out.

Michael Kassner
Michael Kassner

I like your phrase "legitimate technology." That has a great deal of significance.

melias
melias

This might be able to work in a military or even B2B environment, but what about B2C (Consumer) or government or law enforcement? If you block everything but IPs that you know, you could lose business, or alienate the public, getting you defeated in the next election. How many local-only ISPs are there? I use one in Cincinnati that is pretty much unknown on a national level, but it is operated by the local phone company, definately legitimate. If Amazon was to block me because they don't recognize my ISPs SMTP server, they lose business! And as you said, compromised ISPs or IPs are not addressed by this.

bboyd
bboyd

Joe can you post those link un-tiny? I can not peruse them with my filters here and would like to.

Michael Kassner
Michael Kassner

I don't use the voting thing. All comments are valuable to me.

kevlar700
kevlar700

Maybe you should elaborate, otherwise your part of the problem. I can guarantee you know less about security and code correctness (their main goal) than the authors of likely the most secure desktop OS and best firewall system on the planet. ipv4 should have had it's address space enlarged and that's all. It would certainly be better than the mess and bugs that are now happening because of ipv6. DNSSEC is another example of a bad and expensive process, but atleast I foresee DNSSEC being fixed someday.

pgit
pgit

Thanks for telling it like it is from the server side of the equation. You might never have had to deal with getting off of a blacklist after suffering a loss of control over your IT resources... The laundry list you supplied reads like a mitigation checklist... although of course the debate here is what exactly is the proper mitigation. There doesn't appear to be anything universal, as with any security a multi layered/multi faceted approach has naturally evolved. But included in the battle is very definitely the "client side," it would be foolish to leave clients totally out of the equation. I get the sense you're comment below is speaking to server admins and/or IT support services personnel: Lots of people still use client-side "solutions" to deal with spam (which makes me shudder), but for some, it works. It would be doubly foolish to rely entirely on a client side "solution" indeed. But I deal with a lot of individual home and small business users, who use gmail or a mailbox provided by their ISP. They pretty much have to waste an amount of time babysitting their software. (as mentioned by another poster above as being an undesirable necessity in some scenarios) What is one to do if they do not have access to the server side? Is it more effective to set a high bar on a filter then spend time retrieving the 'ham' from the 'spam' folder, or pressure the ISP or other mail provider to get their act together? Where should I, as not only a server admin in some instances but also support for single end-users be putting my eggs? What I have always considered the ideal on the email battlefield would to have 100% eyes-on monitoring of server traffic, put a human in charge of placing literally everything. If you have to hire a dozen people to keep an exchange server flowing, then so be it. The question is what do you want? Security? So it's obvious, at least to me, that cost trumps security as a first consideration. Then afterward pressure is put on clever folks such as you to come up with better solutions that don't involve more personnel. I just heard a fellow mention that 4 out of 5 workers at the troubled Japanese nuclear plant are make-work hires that have no legitimate function. They are just given "jobs" instead of a welfare check. Japanese industries will often make a position for an otherwise redundant employee, for numerous cultural reasons, but the result is a lot of elevators have "operators" in Japan. This nuclear worker had worked in the doomed Japanese plant, he said you'd see things like three people walking along with an empty cart, parading back and forth all day until it was time to punch out, all three of them with one hand on the push bar. With widespread non-productive laborers abound, Japan has still maintained a viable economic system, the country certainly hasn't fallen apart as a result of a lot of these 'make work' positions in just about every industry. Certainly most companies could afford to hire a few people to actually do something productive, maintaining security in an email system. Of course software has a huge roll in the task, but it would best be applied to augmenting a human, hands/eyes-on approach to the email problem. At least I think so. Or am I missing something? Won't having flesh and blood on the beat always be the best ultimate security measure? Is it ever better to accept a reduced bottom line in exchange for attaining a desirable outcome?

Michael Kassner
Michael Kassner

I automatically assumed that everyone's spam was similar to what I was receiving. No longer.

JCitizen
JCitizen

There are so many new technologies and services coming about; the spammers will be able to keep ahead of the game faster than the technicians can block it. Social networking comes to mind as just one example. I have many clients bamfoozled as to why they get attacked, spammed, and used as an attack/spam vector on FaceBook everyday.

Michael Kassner
Michael Kassner

I use spam filtering now, and it works well. The supplier told me they do not use blacklisting. And, they are not worried about IPv6. That's why I wrote this article. There seems to be some confusion as to what will work and what won't.

jsklein
jsklein

bboyd@ Sorry, the Android twitter client does not have a URL shortener. Thanks for reminding me. Joe Klein

bboyd
bboyd

Good reading.

pgit
pgit

But the 'powers that be' have other things in mind than just computers interconnecting. IPv6 will just about allow every grain of sand to have it's own address. To get to the point, I see the planet becoming a self-aware machine on which humans are regarded as parasites, first and foremost. Then it's guilty until proved innocent on everything from being "green" on down to diet, exercise and how you interact with others. There's no need for the magnitude of address space afforded by IPv6, unless the goal is to know the whereabouts and activities of everything and everyone at every moment. My 2 cents anyway.

Michael Kassner
Michael Kassner

I whole-heartedly agree with you. I am not as qualified with regards to security or code development. I also appreciate your comments and sentiment about IPv6.

pgit
pgit

I recommend throw-away email accounts as well, though usually through one of the free services or the ISP. I hadn't heard of spamgourmet, that looks like a much better way to handle the matter. Thanks for the tip! Glad to hear the eyes-on/hands-on approach is indeed the ideal. Nice to know there are people who specialize in analysis, too. I imagine I would not find such folks to be 'quirky' in nature, though as for their skill sets I defer to your judgment. Everything is best viewed through motivation. Why is someone doing what they are doing? You folks battling the mayhem out there are some of the most honorably motivated people I can think of, you have my utmost respect. Additionally I have to say your posts here have been an education exceeding any one semester I can recall from college. I for one really appreciate your taking the time to fill us in on what we're all facing and what you're up to. (and thanks again, Michael, I feel like I'm auditing one of your classes)

cameron.schmauch
cameron.schmauch

"But included in the battle is very definitely the "client side," it would be foolish to leave clients totally out of the equation." Client-side considerations shouldn't be entirely excluded from the conversation. My beef with spam protection at the client is that it is usually much less effective than server side protections, generally speaking. This is a big deal when you are talking about malicious email. It also makes a lot of sense to keep spam as far away from the end user as possible for numerous reasons (efficiency, management, bandwidth, processing, etc.). Last year I wrote extensively about single-click malware campaigns. If you are on a vulnerable system and you accidentally click one of the links in these campaigns, you will be compromised, and it won't necessarily be obvious that you just got pwned. Another related issue is that people are just curious sometimes, or simply aren't all that technically savvy. These users need additional protections and should not have a quarantine area for nasties on their local machine, especially if they are allowed to poke through it. Malicious spam campaigns often spoof popular brands that people recognize. If a user sees something they recognize in their quarantine (spam folder) they might think it's a false-positive, when actually, it's a virulent message riding on the coattails of good reputation ala brand recognition. It's also not unusual to receive virulent spam and scams from someone you know because their account has been compromised or their machine infected. Enabling the user to inspect these messages safely on a remote server reduces some of the risk when they go poking through quarantine. "But I deal with a lot of individual home and small business users, who use gmail or a mailbox provided by their ISP. They pretty much have to waste an amount of time babysitting their software. (as mentioned by another poster above as being an undesirable necessity in some scenarios) What is one to do if they do not have access to the server side? Is it more effective to set a high bar on a filter then spend time retrieving the 'ham' from the 'spam' folder, or pressure the ISP or other mail provider to get their act together? Where should I, as not only a server admin in some instances but also support for single end-users be putting my eggs?" Good questions. Personally, I use Gmail accounts which I find to be acceptable in terms of filter accuracy (considering it is a free service). I still have to hunt through my spam folder at least once a week on my main account though. That being said, (warning: shameless self promotion coming...) the company I work for services numerous ISPs and SMEs. All you need to use our service is that you control MX records for your domain. There are lots of Software As A Service offerings out there that simply proxy the email either through managed appliances or through cloud type services and combinations thereof. But when you just have a free email account through an ESP and their protections are insufficient, there's not much you can do short of switching to something else, or dealing with client-side protections. Another thing I find incredibly useful are disposable addresses -- check out spamgourmet.com sometime. I've been using their free service for about eight years. They are good people. Because of this service, I know exactly who sells my address and who's address book has been stolen. "What I have always considered the ideal on the email battlefield would to have 100% eyes-on monitoring of server traffic, put a human in charge of placing literally everything. If you have to hire a dozen people to keep an exchange server flowing, then so be it." Absolutely! This is one of our (EdgeWave's) core differentiators. We embrace human-in-the-loop real-time threat analysis. We strongly believe it is the only way to respond dynamically to novel challenges presented by adversaries on the net. It is also the only way to have superior accuracy IMNSHO. We have staff analyzing new threats 24/7/365 and as a managed service that stresses simplicity, we give you what you describe above. Let's face it, it's an odd bird that loves to analyze spam all day, anyway! Kidding aside, it's actually a pretty difficult job, requiring a quirky skill set. Even if budget were not a concern, it's difficult to find good spam fighters.

Michael Kassner
Michael Kassner

Some times the planets align and we appear to have a clue as you suggest. "Did you reboot" is not cutting it with users any more.

pgit
pgit

Probably because that's not my concern. I just have to keep the computers running smoothly, so I wouldn't encounter any issues resulting from identity theft, borrowing or whatever facebook does with your info. I'll have to start asking if anyone has had that kind of a problem. Meantime, the one office did get nailed with some of the drive by garbage facebook was spewing for a time. The machine picked up a nasty rootkit and was apparently part of a botnet but inactive at the time. I remember Sophos mentioned the exploit in one of their newsletters and it wasn't more than 48 hours later the client called me to report the problem. Nice to have a clue beforehand! If nothing else it makes you look smarter. ;)

Michael Kassner
Michael Kassner

For things like water leaks seem like a great idea. Not the other one though.

Michael Kassner
Michael Kassner

Facebook was more of a privacy concern than a malware drive-by site. Am I wrong?

bboyd
bboyd

and it was just male enlargement spam. Can't wait for that day.

pgit
pgit

All but one of the offices I service ban facebook on company hardware, though a few allow workers to access it it with smartphones, etc. The one place, oddly, has seen no mayhem resulting from facebook, I certainly can't say the same for a substantial chunk of the home users I hear from. I have to assume what has kept them safe are the noscript and adblock plus add ons on firefox, and maybe windows security essentials, all three of which are in place and updated diligently on all computers. The enforcement of this policy was helped by the owner of the company disabling noscript on her computer, which in the course of a week or two resulted in a compromised OS. (made the more pointed by the fact this machine was the only one receiving and storing data from machinery in the place) It's uncanny, really. These people are all over facebook, including farmville and the like, and to date just the one compromised machine, out of 20. It's just as likely they've just been lucky I suppose.

Michael Kassner
Michael Kassner

With you in that regard. If you read Cameron's comments, you will see that they do not use blacklisting.