Software

E-mail's swan song: How this affects programmers

Justin James says that the death of e-mail has led to some wonderful opportunities for programmers to make a buck without actually accomplishing much of anything.

Back in the day, e-mail was a killer app that was getting people onto the Internet in droves. Even people without computers ended up with Hotmail or Yahoo! Mail accounts and would check them whenever possible. People were thrilled that they could easily, conveniently, and cheaply get in touch with people over a long distance.

E-mail was going to revolutionize our world, but this is no longer the case. E-mail is hopelessly devalued to the point where I barely use it. The problem is the SMTP protocol. Most people seem to be fleeing standard SMTP/POP3 (or SMTP/IMAP, or SMTP/MAPI) e-mail when an alternative presents itself. SMTP e-mail is so bad that people will use any pile of garbage with zero features, as long as traditional e-mail's problems are not there.

The death of e-mail (while not being a causing factor) removes a roadblock towards the decline of traditional desktop computing. This decline will affect a huge portion of programmers, as the need to shift to new platforms, display sizes, and so on becomes necessary.

How did we get to this point?

The SMTP protocol was shortsighted; it is good enough for delivering the mail, but it does a lousy job of protecting mail. (If you want the precise technical details of SMTP, read RFC 2821.) To enumerate SMTP's shortcomings and the consequent results:

  • SMTP does not enforce any assurances that the server sending e-mail is authorized to act on behalf of the domain that the sender is from. There are add-ons for this like SenderID and the use of SPF tags in DNS, but they are hardly universal. Result: Spam.
  • There is no requirement that encryption of the SMTP connection must always be available. Result: E-mail is open to snooping, which increases the cost and hassle of using add-on e-mail encryption products.
  • SMTP pushes the full message across to the destination -- as opposed to a system like RSS (or most NNTP readers) where it only pushes a notice or message header across -- to be picked up at a later time. Result: Wasted bandwidth. Disclaimer: This guarantees delivery even if the sender is unavailable at the time of reading whether by accident or by design.
  • SMTP doesn't have any way of controlling or even monitoring the progress of the mail sent. Result: E-mail is not nearly as useful for business purposes as it should be.
  • SMTP does not offer any authentication, verification, or proof of identity. Result: E-mail is not trusted; spam.

We blame our e-mail clients, systems, and administrators, and we buy even more software to address many of these issues. The reality is, those e-mail clients, systems, and administrators have accurately and faithfully implemented the SMTP protocol. For them to add better, more efficient protections against the dangers of SMTP, it would render them out of spec.

SMTP has its roots in an era in which many servers placed long-distance phone calls to each other (sometimes waiting until the middle of the night when the call was cheaper) to establish connections to the next destination in the routing tables. E-mail delivery time was measured in hours and sometimes days. In other words, e-mail was fine for non-critical items. The SMTP standard, and many of the legacy systems for sending it (such as sendmail), are written to meet the demands of that environment. As a result, it is nearly worthless in today's environment and, in fact, it makes matters much worse due to the security and spam issues involved.

How programmers benefit from e-mail's death

"But wait... I use some sort of magic elixir algorithm/software/firewall that solves these problems!" you say. This proves my point. If SMTP, as a protocol, fit the modern world, you would not need it.

For programmers, this has led to some wonderful opportunities to make a buck without actually accomplishing much of anything; it is not hard to write rules-based filtering systems, and people will pay a lot of money for one that promises to solve the spam problems. The slow death of e-mail removes one of the major legs that the current computing environment stands upon. When the current paradigm falls, programmers will need to adapt to what takes its place.

The point is simple: Sending messages no longer requires the resources of a desktop PC (when I say desktop, I also mean laptop). So what happens when people no longer use PCs, or they only use PCs of necessity? Well, it depends on if they have found any other reason to stay on that platform. And for many users (particularly in the consumer space), the blossoming of Web applications means that they really do not need a PC per se -- they just need a device that can access the Web and preferably has a way of attaching a large display and a full-size mouse and keyboard.

I will continue to explore this theme in a slew of upcoming blog posts. In the meantime, what do you think about how the death of e-mail is affecting programming work?

J.Ja

About

Justin James is the Lead Architect for Conigent.

88 comments
DigitalFrog
DigitalFrog

SMTP's death. Email will live on in one form or another, what is needed is a brand new protocol that includes support for the problems plaguing SMTP. It's time to retire the protocol and start new, just like the analog vs. digital TV broadcasting.

rclark
rclark

1. Lots of people have PC's for their kids to use for homework/play games with. (Nancy Drew, Barbie Costume) 2.Lots of Baby Boomers that have left the workforce seem to spend an inordinate amount of time playing freecell. 3. Every digital device sold today seems to want to dump junk on your harddrive. 4. Email does timeshifting, and is embedded in the skillsets of a generation. So given these, I don't think email will go away any time soon. What ever replaced it would have to add security, remove complexity, and be cheaper and easier to use. Given those requirments, I think you'll agree that it will be a phased approach where what we end up with may not look/act like email, but we will still call it that.

deepsand
deepsand

What realistic alternative is there for communicating with a [b]specific individual[/b], at both a time and place of the choosing of both the sender and the recipient; i.e. a method that is [b]asynchronous[/b] with respect to both time and space? Merely having access to a web app will not serve to meet such requirements.

Justin James
Justin James

All good points. "1. Lots of people have PC's for their kids to use for homework/play games with. (Nancy Drew, Barbie Costume)" There are a TON of special purpose education computers out there now, like the Leapfrog stuff, that does the same stuff, without the attendent worries of letting your kids touch the PC (adult Web sites, MySpace stalkers, viruses, etc.). For older kids, I think you are right. However... a "smartphone" has the computing power to run Word and other student apps. Give them a docking station, throw out the PC. "2.Lots of Baby Boomers that have left the workforce seem to spend an inordinate amount of time playing freecell." Yes! That's my mother to a T, except she's of the Mahjong variety. It's amazing, they have a $800 PC to play a game that a TRS-80 could handle, with (until Vista) graphics reused from 1993 or so. How long will it take them to discover that their cell phone can play Freecell? "3. Every digital device sold today seems to want to dump junk on your harddrive." Yup, and most of it is actually not needed. Like my camera wants to install junk. No idea why, it gets detected as a USB storage device just fine. 95% of these devices don't *need* the junk, they just want to install it anyways. The first folks to get these things hooked together in a way that people feel OK without installing anything gets a prize. From what I hear, Apple is like that, but for whatever reason, people feel "weird" if they don't have to work to get a periperal hooked up. It's odd. "4. Email does timeshifting, and is embedded in the skillsets of a generation." I agree 100% that *email* is great. The problem, as I tried to make clear throughout the article, is specifically the SMTP protocol used to transport the email. Unless that protocol undergoes some radical changes, the problems that it allows to occur will continue to overwhelm traditional email. People would rather lose the benefits of *email* than continue to put up with the problems of *SMTP*. "What ever replaced it would have to add security, remove complexity, and be cheaper and easier to use. Given those requirments, I think you'll agree that it will be a phased approach where what we end up with may not look/act like email, but we will still call it that." I agree completely with this too. All of these devices operate seperate from the SMTP nets, more or less. Why? You think Verizon, Alltell, etc. what to deal with handling spam being sent to cell phones? No way. It would cost them a fortune, and people would just turn off their phones instead. Put a cell phone bareback on the SMTP system, it will blow up in an instant. I get over 100 spam emails a day. That would blow out my messaging plan 5 days into the biling cycle. TXT, IM, and "walled garden" systems like MySpace, Facebook, etc. will continue to expand and interoperate (look at the cell phones with MySpace access for proof) outside of SMTP. We will quite possibly call it "email", but it is not SMTP. And that's what matters. J.Ja

dave.amorde
dave.amorde

Email is too deeply entrenched to discard it completely, and the ability to create a subject line, CC and BCC copies, etc., make traditional IM unsuitable as a replacement. A simpler approach would be to add a security protocol layer over SMTP. If it was good enough to TCP, HTTP, etc., it's good enough of a solution for SMTP.

Justin James
Justin James

For personal use, most folks I know use MySpace and/or Facebook for this. Heck, I am forced to maintain a MySpace account solely for this reason. I know other people who more or less replaced email with TXT messaging. As Jaqui says, IM is out there (at least a few people I know will *never* reply to an email, but an IM gets an instant response). Yet others I have dealt with in the corporate world are so busy, face-to-face and phone calls are the only way to break through to them. At the end of the day, for a great many people, email has become the last resort, or "fallback option". J.Ja

Elmonk
Elmonk

for quite some time I've been dreaming of a soft migration path from SMTP based mail to a new protocol allowing co-existence. This would have to be based on a discovery phase where a sending agent could find out if a receiver is capable of handling the new protocol, in which case it would all go state-of-the-art; if not, SMTP would still be used. The sender could also define if they allow mail to go ye olde path. As time passes more and more mail agents would use the new protocol (-suite) and in a few years time we would get rid of SMTP altogether. What do the gurus think of this approach?

Jaqui
Jaqui

is supported by most instant message clients you know. you can send a message to an offline contact in them.

apotheon
apotheon

"[i]Give them a docking station, throw out the PC.[/i]" Docking stations for smartphones impose significant detrimental interface restrictions as compared with general-purpose laptops. Interface technology needs to advance significantly -- to the point where docking stations are entirely unnecessary -- before the full-featured general purpose PC (in the form of the laptop computer) can be replaced by something like a smartphone. "[i]How long will it take them to discover that their cell phone can play Freecell?[/i]" I had a cellphone that had both solitaire and tetris on it in 2001. People don't generally play Freecell on their computers because they specifically want to play solitaire, though -- they do so because Freecell happens to be in front of them. If you want to find out why people aren't giving up their desktop computers for smartphones, look somewhere other than the ability to play solitaire. "[i]Unless that protocol undergoes some radical changes, the problems that it allows to occur will continue to overwhelm traditional email.[/i]" [url=http://blogs.techrepublic.com.com/security/?p=393][b]SMTP isn't the reason we have spam.[/b][/url] "[i]Put a cell phone bareback on the SMTP system, it will blow up in an instant. I get over 100 spam emails a day.[/i]" The only reason SMS on cellphones doesn't get hammered by spammers as hard as SMTP in email is the pain in the butt interface of cellphones. There's little to no return on investment for SMS spam as long as cellphones continue to have physical interfaces that make instant gratification responses to spam unappetizing to recipients, and as long as text messages cannot carry hidden payloads (whether it be an image served from an arbitrarily remote location, quick click-through links in the message, or malware). The protocol is not the reason for the lesser volume of spam in cellphone text messaging.

Justin James
Justin James

I agree, the concept of email *in and of itself* is just fine. It is a protocol level problem. The issue with an "add on" layer to SMTP is that it needs to be added to the protocol itself. There have been countless attempts to have unofficial add ons to SMTP to fix it, starting with PGP encryption through SenderID and SPF records in DNS. Until they have the weight of "you are not SMTP compliant unless you support them" they will sadly continue to have zero traction. J.Ja

deepsand
deepsand

None of those "alternatives" serve both the needs of carrying capacity asynchronisity that I above noted. Absent such, they are, for serving as the mainstay for business purposes, DOA.

Omnifice
Omnifice

Email is one of my most important tools (and many of my friend's and relatives)! I always have my email client open and monitor many accounts. Then I check my free accounts (Gmail, Yahoo, etc.) on a less frequent basis. Everyone knows they can reach me using email...even if it takes a day or two for me to reply. Email is an excellent form of communication that is non-intrusive. YOU check/read it when YOU have time. It is much less intrusive than a phone call and you can cut right to the point. Phone calls waste time (IMHO), although there are times when it just has to be done over the phone. It is much less intrusive then IM/Chat. I despise IM unless I am actively engaged in a conversation or game that needs it. Email is important for many people to stay in touch. An aging great uncle of mine cannot get out much due to health. He loves email and stays in touch with everyone with it. Now, I know that every person is different...so IM/Texting may be their method of choice for communication, but not everyone has a cell phone on all the time (or wants to spend all that time trying to type on a tiny key pad in the case of a cell phone). Spam is not a huge issue if you configure your client. I don't use anything fancy...just the built in spam tools, but set up filters and it only takes a minute or so a day to delete the very small amount of crap in my many inboxes. One issue with email is that many people do not know how to put thoughts in text in an effective manner in order to be completely understood. That's another topic...but simply thinking will solve most of that problem...something many younger people seem to have a problem with. I think this is all a perspective/needs issue. Every person has their own needs and ways of doing things. Email is here to stay for quite a while...thank the digital gods! I'm open to the evolution of email and welcome more secure and efficient forms...but until there is a widely accepted alternative, I'll use what's available. There have been some good suggestions in these posts. Somebody with more talent that me should get crackin' on it! :) In the mean time...you can reach me through email. ;) Cheers!

deepsand
deepsand

SMTP-[b]AUTH[/b], defined in RFC 4954, would have added Access Control. While not a perfect solution, it would have served to as an improvement over unauthenticated SMTP. But, it never really gained any traction.

apotheon
apotheon

I've been dreaming of an article about opportunistic encryption, but I haven't written it yet. I keep ending up with more urgent articles to write, and/or easier articles to write. I'll get around to it, though. Basically, you're talking about opportunistic encryption and authentication with your reference to the new protocol being offered parallel to the old as available. Well, you aren't talking about that specifically, but Justin is when he (rightly) states that what's needed to make your migration path work is a strict superset of SMTP. Of course, there are some problems with that. 1. The major security issues Justin wants solved by the new protocol he posits are message encryption and authentication. Truly effective authentication requires encryption technology for digital signatures. Encryption systems should never be embedded in communication protocols because that approach would simply guarantee the early obsolescence of the protocol -- which obviates the point of an encryption protocol (reliability of compatibility) -- as [url=http://blogs.techrepublic.com.com/security/?p=393][b]I pointed out elsewhere[/b][/url]. 2. Both message encryption and authentication require realtime communication between endpoints of the security feature's path. For authentication, those are the sending client or the SMTP server and the POP/IMAP/MAPI server, which is convenient because it can usually be assumed that the two servers are both essentially 24/7 systems with the current email protocol. For message encryption, on the other hand, we're screwed, because the whole point of email is asynchronous communication, and the endpoints for message encryption are the sending client and the receiving client -- neither of which can, in general, be assumed to be connected at any given moment. The value of both authentication and message encryption is attached to the assumption that they will not be "turned off" in transit, which is effectively what a backward-compatible superset protocol that is asynchronously capable would have to allow. Non-starter. So . . . we're basically looking at a situation where migration to a new protocol would grant no real benefit over keeping the same protocol with add-on encryption and authentication features, and would probably impose significant detriment in the long run.

Justin James
Justin James

Creating a superset of SMTP (or a "side-by-side-set") is the only way to get something adopted. Even then, it is beyond difficult to make a change at this point. :( J.Ja

deepsand
deepsand

it too is lacking for security. And, it lacks the loading capacity of e-mail. By analogy, IM is more akin to a telephone call, whereas e-mail is similar to physical mail and package delivery.

rclark
rclark

I was unaware that this was being allowed. I guess I'm showing my age. I was referring to the burned in MAC on the NIC chip. In googling this I see that the industry has changed and it seems easy to spoof the once untouchable MAC address. I still question whether it would be effective on a handshake at the hardware connection layer, but with Sun and mobile, it doesn't look like there would be much to keep it from happening beside segment sequence analysis.

boxfiddler
boxfiddler

Would that be the 'smart user' category? ;)

Justin James
Justin James

"As long as the [tech] news is full of the latest hacks, stolen identities, exploits, security holes, etc... I think that smart users are going to stay away from 'going web only', as you put it." I would love to agree, and I wish I could agree. I think that far too many users do not fall into that category. Look at the number of people entrusting their email to services like Hotmail, photos to Flikr, and personal information to MySpace. For the consumer, they really do not do much (aside from students writing papers) other than email that uses any kind of "productivity software". In fact, as far as I can tell, students writing papers is the only thing other than games that consumers frequently do that they don't already do in Web apps. For the businesses, I think that the day when someone starts offering a "be your own provider" sales pitch instead of the current "we host it for you" model, this will be alleviated. I am truly shocked and surprised at how few Web app vendors let you install the software on your own servers. I've been saying the same thing as you on this for a while (years, in fact, just about my very first post EVER on TechRepublic was about this!). I just think that eventually vendors will wise up, let the customer host it themselves, and this is done with. J.Ja

Absolutely
Absolutely

Me: [i]More importantly, although some mail from as-yet-unknown senders is not "SPAM," I also don't mind, in many cases, missing it.[/i] I meant to say more about those messages, which I sign up to receive from Internet fora, legitimate business mailing lists, etc. Other than most passionate "causes" and items already purchased, I don't [b]need[/b] messages other than from my personal acquaintances. My assumption is that there are a very large number of users who would be perfectly happy with a whitelist-based filter which reliably delivers messages from all their personal acquaintances [unless the silly end-user types them incorrectly to /home/user/.procmailrc, which he sometimes does] even if that means they miss some "legitimate" unsolicited or even solicited commercial e-mail. Coupons, for example, might be welcome, but not a disaster to miss, either. Along with correct identification of SPAM, zero "false positives" seems to be the holy grail of SPAM filters. But the situation in business is very different than for home users, because of the profit motive. Although meeting new people is enjoyable, the quest to expand one's circle, even for the most ambitious social butterfly, is nothing compared to the drive to "grow one's business." As a result, precise identification of [b]legitimate[/b] contacts might be more trouble than it's worth for most businesses, but without similar growth to "customer base" the same approach is trivial outside of business. Gramma might miss a couple of Justin's brother's e-mails, but eventually I suspect he'd call and tell her his new address, or another relative would relay it. Eventually, updating contacts might even help convince him that new e-mail addresses are not "fun." You'll still want Bayesian algorithms at the office of course and without encryption they're just best guesses, but the shortcomings of whitelists are much less important out of the office than in it.

Absolutely
Absolutely

Justin James: [i]The problem is, *not all unsolicited email is "spam"*. For some users (quite possibly most users) that may be the case. But it can never become a universal constant, because it is essentially a whitelist, and whitelist (particularly for something as plastic as an email address) have huge maintenance issues with them, which go up even more when a system like PGP is layered on top of it. In other words, I cannot imagine my grandmother trying to maintain PGP keys for my brother who seems to have a new Hotmail account every week...[/i] More importantly, although some mail from as-yet-unknown senders is not "SPAM," I also don't mind, in many cases, missing it. But, if I don't get my tracking number from orders@amazon.com, that is a problem. That general problem also has a simple solution: businesses make their e-mail addresses easier to find on their websites, prominent on the website's tools that would cause a person to receive e-mail from that address. Just above, below, or to one side of any blank e-mail address form on any web page should be the address I would need to add to my [b]whitelist[/b], which is not "magic," cannot be marketed with the fanfare of "greylisting" software and requires maintenance ["huge" maintenance? I dunno about that.]. But whitelists are definitely not unworkable, and do not need to "become a universal constant" to be useful to people whose needs are suited to restricting the e-mails we receive to that degree. For "most people," or to be less vague, for home users with e-mail/browser appliances who do business online rarely, if ever, I would say it's probably the best solution. For myself, on my home network, it's certainly the best solution, by a very wide margin.

apotheon
apotheon

"[i]I submit that it is highly unlikely that a bot net herder or a person expecting to make 1500 a week sending emails is going to be able to fool a router into thinking his mac address is something other than it actually is.[/i]" You obviously overestimate the difficulty of spoofing a MAC address. I recommend you have a look at my newest article, [url=http://blogs.techrepublic.com.com/security/?p=395][b]How to change a MAC address[/b][/url]. "[i]They would have to have multiple versions of their virus, and that in turn would make finding them much easier.[/i]" I can only assume, from this statement, that you are unfamiliar with conditional constructs in most programming languages like the "if" statement. "[i]Right now, we can't even get to the point where we know what domain sent the stuff.[/i]" Sure we can. There are numerous means of doing this. Many just don't do it. "[i]So the future I see is I/O kiosks that you walk up to, put a coin into, and pair up with your device of choice.[/i]" Why bother? I have a laptop. I'm going to leave the Linux vs. Windows subject you brought up (I'm not sure why) alone here, for the moment. It's a much bigger debate than should be engaged as a mere tangent to the current discussion.

rclark
rclark

A majority (reportedly) of spam is sent by two types of people. Botnet herders and people who answer those (If you have a computer you can make 1500 a week from home) ads. I submit that it is highly unlikely that a bot net herder or a person expecting to make 1500 a week sending emails is going to be able to fool a router into thinking his mac address is something other than it actually is. Certainly, any thing done in hardware is possible in software, but unlikely to be available to either of these communities. Any time you go down to hardware level on anything, you limit the audience that will be able to run your code. So I don't see bot net herders doing so. That would limit their targets to a specific nic card. They would have to have multiple versions of their virus, and that in turn would make finding them much easier. But be that as it may. We must have some method of identifying who sent the email. Don't care how you do it as long as it is limited to one person or machine. Right now, we can't even get to the point where we know what domain sent the stuff. We can track back to the originating network, but that doesn't mean that the email originated there. PGP is not the answer. It's good for the mildly paranoid (guilty as charged), but is unworkable for the vast majority of users. And any scheme that has the population of the US/EU/PRC/RU/AU going down to a center with three forms of photo id's to get a certificate that will allow you to send emails is not an answer either. So keep thinking, but remember that most people use email to send pictures of their grandkids, gossip about their inlaws and coworkers, and occasionally to find out if the meeting is at 8 Central or Eastern time. I'm sure that if everyone wanted to solve the problem, the problem would get solved. The real problem is that people are making money off of the problem, and as long as that is true, the problem will engender much angst but little progress. As for the smart phone/ultra mobile/laptop debate. IMHO, you are all wrong and you are all right. With the advent of blue tooth and usb wireless, the technology exists to create a kiosk that will have a 19 inch screen ( touch or not), a full keyboard/trackball/touchpad/mouse whatever. With these two items, you have everything except storage and CPU. Both of those are available in the form of smartphones, ultra mobiles, laptops, etc... So the future I see is I/O kiosks that you walk up to, put a coin into, and pair up with your device of choice. They loan you their generic keyboard/monitor/network connection for a few cents. All the data is on your device and the connection is gone when you are. The keyboard/screen may not even actually exist. They may be software running in a processor on the kiosk that simulate a keyboard and monitor. With DSP and lasers, almost anything is possible. As for the whole Linux/Microsoft Office debate. Linux has come a long way and getting better by the day. Walmart and Sears are making huge progress in getting cheap desktops and laptops into the hands of consumers. Which is great for Linux, but does absolutely nothing for the vast majority of business users. Microsoft sells products because it has the standard office package. Huge investments in training and culture aside, Linux is just not counted as a player in that market. I've tested the office offerings and I think they are great. But I wouldn't recommend a client use them. A home user, absolutely, unless they had to send the results to a business. But the number one thing that Linux does not do is run all of the packages that windows does. Until that is true, Linux won't be a contender for the business desktop OR savvy home users either. It is unreasonable to expect that software companies will make windows, mac and linux versions of their software. Very few do both windows and mac, fewer still do both well. Mac has survived by carving out a niche where their fanboys can exist. It does one thing really well. Unfortunately, one trick pony is not enough. Most people are generalists, and windows is the generalist OS. Linux is getting there. It is no longer a one trick pony, but it's not yet a contender either. Will it be in 10 years? I don't know if windows will be a contender in 10 years, much less Linux. Linux may take it over. If so, I really don't know if it will survive long after that. We may have something much better by then, or not. Depends on what you guys think of.

boxfiddler
boxfiddler

"I think Microsoft Office is holding people back from going Web only more than anything else." Now, you guys are sharp and I have learned a lot from you. But I have to quibble with the above, and quibble pretty simply at that. As long as the [tech] news is full of the latest hacks, stolen identities, exploits, security holes, etc... I think that smart users are going to stay away from 'going web only', as you put it. As a primarily home user, the last place I want any of my (or my students) stuff is on someone else's server. I flat out do not trust anyone but me to keep my stuff safe. While I wonder about me now and then... I do trust me to do what I need to do to keep my stuff private. I can't say that for 'going web'. Virtually all of the people I know think the same. It has nothing to do with MS Office and everything to do with the perception that any server can be cracked. And that one or another of them regularly is. edit typo - at this hour don't be surprised if I missed another

apotheon
apotheon

[b]Absolutely:[/b] "[i]What about a PGP-signed replacement of 'MAC address'?[/i]" Yes, it would. Somewhere else, in either this discussion or [url=http://techrepublic.com.com/5208-6230-0.html?forumID=102&threadID=251095&messageID=2408333][b]that one[/b][/url], I made reference to use of digital signatures for this purpose, in fact. The reason a digital signature works is that it can only be encrypted using the signer's private PGP key and the message's content, and it can only be decrypted using the signer's public PGP key and the message's content. A MAC address, meanwhile, can simply be copied and sent along with the message. "[i]whether MAC address or flesh_name is used as seed of the pRNG[/i]" This doesn't work, because a MAC address is too easily discovered. The only way to get around the weakness of the MAC address as a key is to introduce some other factor into the process as well -- some other factor that either makes the key indecipherable (and thus useless) or is something like a private PGP key (thus making the MAC address redundant and pointless). (edit: By the way, I think you mean as an "encryption key", not as a "PRNG seed".) "[i]Most importantly, why do you propose that we all trust that 'the ISP would do packet inspection to see if it is SMTP traffic' instead of using PGP keys given to us in person by our friends & lovers & loved ones, to verify the content itself, as opposed to the attached 'sender' info [IP, MAC address], which we all know can be obfuscated by any 3d-rate script-kiddie?[/i]" Actually, it's easier than that. You can publish a public key on the Internet (thus the term "public key"), and anyone/everyone that wants to can get it from there. Because systems like PGP use public key cryptography, a secret key is only needed at one end for digital signatures: the sender's end. [b]Justin:[/b] "[i]Mandating that evrerything be PGP encrypted is great for making unsolicited email stupid simple to filter out.[/i]" Actually, Absolutely wasn't referring to encrypting everything -- he was talking about using digital signatures (which also leverages encryption technology, but more in a "cryptographic hash" sense than an "enciphered communique" sense). You are, in fact, advocating use of technology that is meant to achieve exactly the same end as digital signatures, but in a different way: technology that authenticates the sender. "[i]it can never become a universal constant, because it is essentially a whitelist[/i]" I'm going to remember you said that. "[i]I think the docking station handles that for the mostly non-mobile workers (programmers, sys admins, business users who occassionally need to go to a meeting).[/i]" Oops, you caught me in an act of sloppy phrasing. When I said "Anyone that does a lot of typing," I was referring to people who do so away from the office from time to time. "[i]More to the point though... go to an office and see how few people are typing (or even using the mouse) at any given moment in time. Surprising, isn't it?[/i]" These people, in many cases, [b]aren't working[/b] either. If you're going to suggest that everyone's going to stop using portable computers with reasonable interfaces for doing real work, you're basically suggesting that their bosses will all recognize that they never do any work anyway -- and will allow them to just carry around cellphones so they can "act busy" away from the office. "[i]'More and more people are using smartphones. Many of these people are not using computers for much of anything.' Yup.[/i]" I think you missed my point: For the most part, the only people who will be "giving up" their laptops in favor of smartphones are people who [b]didn't have laptops to begin with[/b]. "[i]I think Microsoft Office is holding people back from going Web only more than anything else. And the mobile devices have had versions of Office forever now. No need to have a full XP or Vista machine just for Word.[/i]" I think you may have a very myopic view of the general population of computer users. Many computer users are not limited to MS Office. Regardless of this, you still need to have a decent input interface and more than twenty characters' width of screen real estate to use an office suite without gouging out your own eyeballs in frustration. If people were going to give up MS Office for smaller devices that have crappy interfaces, they would have made the jump to the Palm within months of its introduction, years ago. Hell, my Handspring Visor back in 2002 had a far better interface for note-taking than any smartphone I've ever seen (including the iPhone). Heaven forfend anyone should try hammering through the inconveniences of trying to use an iPhone touchscreen keyboard about two and a half inches wide to post a question here at TechRepublic. "[i]Notice the way these systems are architected. They are Web apps. They're just replacing HTML with something else.[/i]" No, they aren't. They're desktop application frameworks that use JavaScript interface and a server for database CRUD. If that was going to replace general-purpose computers, we would never have left the realm of dumb terminals -- we would have just eventually had mobile terminals. These frameworks, in fact, incorporate more desktop system functionality into them than most people realize, and require proportional power of general purpose PCs. I speak of stuff like separable multitasking windowing systems, software management systems, and other mainstays of "thick client" computing. "[i]Not quite. The access to those resources is climbing. The usage stopped climbing, more or less, a few years ago.[/i]" Most of the systems of today wouldn't even run on resources such as those available just a few years ago. There isn't even enough resource to go around for running the installers of today on systems from a few years ago. "[i]The simple fact is, barring program malfunction (out of control loop, for exmaple), the average programmer is unable to write a program that uses more than 5% - 10% of the available CPU or RAM.[/i]" Funny -- I've seen moving a mouse pointer across the desktop of an MS Windows Vista system spike by 15% or so on the CPU meter "Gadget". Now . . . it may be that the average programmer can't use much of the system's resources [b]effectively[/b], but that doesn't mean (s)he isn't using them. "[i]This is a wording issue. I wasn't precise enough. I interchange 'replacement' and 'superceded' a number of times.[/i]" Either way, you're talking about replacing the current SMTP -- either with something "else" or something "improved". That's a matter of syntax, really -- not even a semantic issue. In other words, you're not talking about SMTP (as we know it) any longer. I am talking about SMTP, and how to secure it -- which is more easily done with technologies already implemented than your hypothetical SMTP++. "[i]The features that make SMTP unique useful to spammers (namely, the lack of enforced sender verification) do *not* make it uniquely useful to legit users.[/i]" Your phrasing is imprecise -- and you use the imprecision of your own phrasing as the basis for an argument. It is not just "features" of SMTP that you address: it is "features" of [b]unsecured[/b] SMTP. "[i]Any email-esque messaging system will have those good points you discussed.[/i]" . . . and will thus be enticing to would-be spammers. "[i]Heck, IMs have most of them at this point.[/i]" No, they don't -- because they: 1. do not run on platforms that are as easily compromised and added to spam botnets, and 2. do not run on platforms that are rich enough in functionality for the users to actually get "tricked" by spam into doing anything that would benefit the spammers, except in rare cases. "[i]Umm, the Internet *is* a WAN.[/i]" Only in the broadest, least meaningful sense of the term "WAN". "[i]The add ons have been around for decades, they still have not come even close to mainstream or even widespread adoption.[/i]" . . . despite the fact that their widespread adoption is [b]far more likely[/b] than the wholesale replacement of contemporary SMTP. What we need is to raise awareness and increase the visibility of the solutions that exist, not to just flail about creating new (and lower technical quality) solutions willy-nilly on the assumption that if we build it they will magically come. "[i]It is quite clear that the industry will not universally support add ons.[/i]" No -- it's clear that the industry has not universally support add-ons [b]yet[/b]. Ironically, you're using the single most devastating argument against your "replace SMTP" idea as an argument against using better solutions: if "the industry" won't support add-ons, it [b]certainly[/b] won't support complete [b]replacements[/b] with the same functionality as the original plus those of some of the add-ons. "[i]I think over the next few weeks, as more posts get put up regarding this overall theme, you'll see where I am headed with this, and you'll be more inclined to agree.[/i]" If so . . . I look forward to it. So far, though, I don't buy what you're selling -- the doom of email and mobile general-purpose computers.

Justin James
Justin James

"Anyone that does a lot of typing, pretty much." I think the docking station handles that for the mostly non-mobile workers (programmers, sys admins, business users who occassionally need to go to a meeting). More to the point though... go to an office and see how few people are typing (or even using the mouse) at any given moment in time. Surprising, isn't it? "How long do you think it'll take for NetBSD to run on these things?" Not long, most likely. :) "More and more people are using smartphones. Many of these people are not using computers for much of anything." Yup. And what I am looking at is that most people I've observed do not use a computer for much of anything, other than accessing Web apps and a local email client, and a few other local clients that can easily be made into Web apps. "People have been predicting the death of desktop apps in favor of web apps since 1993. I don't buy it." Personally, I don't want to buy it, but I do. Here's where you get to thank Bill Gates. I think Microsoft Office is holding people back from going Web only more than anything else. And the mobile devices have had versions of Office forever now. No need to have a full XP or Vista machine just for Word. "Notice what's actually happening: Everybody's rushing to the middle. Look at future plans for Firefox, the advent of Apollo, and the Silverlight marketing. These things in no way herald the end of the desktop app in favor of the web app. Instead, they herald the mixing of web and desktop in a new class of application that requires both web and desktop targeted application development." Notice the way these systems are architected. They are Web apps. They're just replacing HTML with something else. It's still JavaScript (mostly), XML, the HTTP protocol, and the J2EE and ASP.Net application servers on the backend. In a nutshell, they are attempting (as they have been for the last 10+ years!) to bring even the most rudimentary imitation of traditional client/server computing to the "RPC over HTTP" idea. As much as I really dislike that whole trend, the indistry keeps trying it again. And again. And again. Other than the cross platform implementation, I don't see how Silverlight is *not* ActiveX 2.0, using .Net CLR instead of COM/COM+/DCOM (whichever ActiveX used). I don't see how Adobe AIR is *not* "Flash, tightly integrated with a J2EE application server", but obviously I'm a minority, considering the buzz it's getting. "Those needs aren't dropping. They're still climbing." Not quite. The access to those resources is climbing. The usage stopped climbing, more or less, a few years ago. The simple fact is, barring program malfunction (out of control loop, for exmaple), the average programmer is unable to write a program that uses more than 5% - 10% of the available CPU or RAM. Period. Why? Because most apps, as we've discussed in the past, are essentially just database browsers, and most programmers don't understand much more than that. For those apps, these devices have more than ample horsepower. "I didn't say we shouldn't make it "more resistant". You did, by saying we should just replace it." This is a wording issue. I wasn't precise enough. I interchange "replacement" and "superceded" a number of times. At this point, "replace it" or "supercede" it (SMTP++), I really don't care, so long as in 10 years, our email is not carried by a protocol lacking the features I beleive need to be in that protocol. "That was my point -- the same things that make SMTP uniquely useful to those of us who are not spammers make it uniquely useful to those who are spammers." I agree. But the converse is NOT true. The features that make SMTP unique useful to spammers (namely, the lack of enforced sender verification) do *not* make it uniquely useful to legit users. Any email-esque messaging system will have those good points you discussed. Heck, IMs have most of them at this point. But the things that make SMTP super-easily spammed (those spambots could be doing IM or SMS spam if they chose to) stem mostly from things that do not affect the vast majority of end users. "This, alone, is by definition antithetical to the concept of a replacement for SMTP." I know... over the last few years, my "archive everything" philosophy has disappeared because so much of my life is now carried over SMS and within "walled gardens" (heck, why do I need to use TR's forum system and not NNTP?). All because I deal with too many people who no longer use email. It really hurts me to maintain a dozen or so routes, when 5 years ago, I *only* needed email. "Who's talking about a WAN? I'm talking about the Internet." Umm, the Internet *is* a WAN. "The first is a monumentally bad idea. I really don't think you'd like that idea any more than I would. The second is essentially doomed, for largely social reasons. In addition, too-close integration can cause obsolescence problems, as I've mentioned, making it a technically bad idea as well." I agree. I think as far as this goes, we are up the creek. No paddle. No canoe. No tins of baked beans. Uphill both ways in the snow. The add ons have been around for decades, they still have not come even close to mainstream or even widespread adoption. PGP, the granddaddy of them all, remains relegated to the super-privacy crowd. SPF is sort of big, only because it is free and easy to implement. It is quite clear that the industry will not universally support add ons. And you are sadly correct than an SMTP replacement or an SMTP++ will probably never fly either. It's a shame, honestly. I think over the next few weeks, as more posts get put up regarding this overall theme, you'll see where I am headed with this, and you'll be more inclined to agree. Certainly with regards to the underlying, fundamental premises. The conclusions I'm drawing are more pliable, of course. But I am starting to see an avalanche of "pretty smart people" simultaneously coming to the exact same conclusion, it's fairly creepy. J.Ja

Justin James
Justin James

"And software running on desktops that function[s] as an SMTP MTA is much more of a problem than software on the desktop that connects to the ISP's SMTP server." This was in reference to spambot software. The spambot software has two choices. It can either have its own SMTP MTA built into it (Type A). Or it can "hijack" another piece of software that has connectivity to an SMTP MTA (say, a virus that takes over Outlook), let's call this Type B. The statement which you question is meant to say that spam software of Type A is responsible for much more of the spam problem than spam software of Type B. To put it simply, Outlook (the primary victim of Type B attacks), while hardly super-secure, is now "good enough" secure so that it is easier to get a Type A virus on the PC. In the context that you read it in, your points are correct, I will add. :) Regarding the spoofing of "sender info", I never once said to trust the sender info. In fact, everything I've said is to distrust it, and verify it before accepting it. The email claims to be from "bob@foo.com"? Fine. Go ask "foo.com" if the SMTP server that is sending the message is allowed to send messages on behalf of foo.com. Mandating that evrerything be PGP encrypted is great for making unsolicited email stupid simple to filter out. The problem is, *not all unsolicited email is "spam"*. For some users (quite possibly most users) that may be the case. But it can never become a universal constant, because it is essentially a whitelist, and whitelist (particularly for something as plastic as an email address) have huge maintenance issues with them, which go up even more when a system like PGP is layered on top of it. In other words, I cannot imagine my grandmother trying to maintain PGP keys for my brother who seems to have a new Hotmail account every week... :) J.Ja

Absolutely
Absolutely

[i]rclark: "I think the best way to kill the spam is to stamp the email with the originating mac address." MAC addresses can be spoofed. This would not work very well.[/i] What about a PGP-signed replacement of "MAC address"? Wouldn't that [close enough] ensure that $foo_sender = meatsack_called_Absolutely_bar[tender]@techrepublic.com[.com]? Once sender & recipient have established means of authenticating one another, however awk, ie private keys, isn't $sender=OK guaranteed to so many nines as to be not worth counting, whether MAC address or flesh_name is used as seed of the pRNG that eventually creates the key? And, once that's accomplished, wouldn't false positives all fall into the category of 'badly/incompletely-described sender address[es] of [solicited] commercial e-mail,' aka, 'who gives a [mentionable] f***?' [b]apotheon v. Justin James[/b] [i]"Outbound SMTP connections do not necessarily need to use the same email server -- or even the same port number. They also do not necessarily need a separate SMTP server at all." Yes, well aware. That's why the ISP would do packet inspection to see if it is SMTP traffic. And software running on desktops that function[s] as an SMTP MTA [b]is much more of a problem than software on the desktop that connects to the ISP's SMTP server[/b].[/i] Absolutely: To whom, Justin, is that ?much more of a problem?? 1) Which operating systems offer ?an SMTP MTA?? [Answer=Unix derivatives: BSD & Linux] 2) Which operating systems offer ?software on the desktop that connects to the ISP's SMTP server? & nothing else? All; Windosed end-Luser operating systems are alone in offering 'NoAlternative'. Most importantly, why do you propose that we all trust that ?the ISP would do packet inspection to see if it is SMTP traffic? instead of using PGP keys given to us in person by our friends & lovers & loved ones, to verify the content itself, as opposed to the attached ?sender? info [IP, MAC address], which we all know can be obfuscated by any 3d-rate script-kiddie? Blecccchhhhhhhhhhhhhh, Justin, you know I'm not normally harder on you than anybody else, but this time around, in order to be honest, I have to come down HARD on you. The ?MTA? is unknown outside of *nix & essay, and whether ?positive ID? is determined by the ?client? or the ?server? is irrelevant. That identifier is either an attribute of the message itself, or it is worthless.

apotheon
apotheon

[b]Justin:[/b] "[i][working mobility] is hardly the sweeping sensation that advertisers make it out to be.[/i]" The fact it didn't happen instantly in no way means it won't happen gradually. Back when everyone thought we'd all end up working on the beach, I was still working at home. I got to the point of working from coffee shops, libraries, and so on, gradually -- not instantly, when advertisers said I should. Others may get there more gradually than I did, but many of them will get there. As such, docking stations are not the cure-all you seem to think they are. "[i]There will always be some workers for whom the ability to have a full OS with installable apps *anywhere* at *anytime* with a full screen/keyboard/mouse is needed, like maintenance technicians, salespeople, and so on.[/i]" Anyone that does a lot of typing, pretty much. "[i]It is that a 'smartphone' has an OS/app loadout mostly or entirely controlled by the vendor.[/i]" [url=http://www.openmoko.com/][b]OpenMoko[/b][/url] [url=http://www.openmoko.org/][b]OpenMoko (.org site)[/b][/url] How long do you think it'll take for NetBSD to run on these things? "[i]Remember, my prediction here is for a full 5 - 15 years out... hardly 'right around the corner'. It's a long term trend. Some people will always need laptops with a full (or nearly full) size keyboard, decent mousing system (stick, trackpad, etc.), and decent sized screen build in to the same unit. But if the trend for the average user keeps going the way its been going, they won't need the laptop nor the desktop for a lot of tasks, and the UMPC/smartphone + docking station combination gives them everything they need.[/i]" I think you're looking at the wrong trends. More and more people are using smartphones. Many of these people are not using computers for much of anything. These people, for the most part, weren't using laptops and desktops to begin with, though. The laptop and desktop market will continue to grow, even as the number of people using smartphones but not using computers also grows. Laptops and desktops are not "dying". edit: Check out [url=http://blogs.techrepublic.com.com/tech-news/?p=1925][b]Global shipments of PCs on the rise[/b][/url] for more evidence. "[i]General purpose CPUs have been overpowered for the typical user, and will be even more so as more and more users use fewer and fewer desktop apps and more and more Web apps.[/i]" People have been predicting the death of desktop apps in favor of web apps since 1993. I don't buy it. Notice what's [b]actually[/b] happening: Everybody's rushing to the middle. Look at future plans for Firefox, the advent of Apollo, and the Silverlight marketing. These things in no way herald the end of the desktop app in favor of the web app. Instead, they herald the mixing of web and desktop in a new class of application that requires [b]both[/b] web and desktop targeted application development. "[i]As the need for local CPU, RAM, and storage goes down, and the mobile devices get more resources, we will hit an inflection point (and I beleive we are there, plus/minus 1 year for most users) where the smartphones (and the UMPC's are definitely there) have enough resources to handle the needs of the typical user. At that point, we are looking at interface, not resource issues.[/i]" Those needs aren't dropping. They're still climbing. Even smartphones provide greater resources than many desktop systems of a decade ago. So-called "ultra-mobile" PCs provide greater resources than many current desktop systems. The difference is not a reduce need for resources -- it's the ability to pack more resources into smaller spaces. Couple that with a need for a good interface, and the smartphone isn't taking over. Ever. "[i]Why *not* make SMTP more resistant to spam and such?[/i]" I didn't say we shouldn't make it "more resistant". You did, by saying we should just [b]replace[/b] it. "[i]If SMTP-based email is hardened to the point where it is harder to invade/infest than other protocols, will those other protocols become just as bad (end result, not in terms of attempts) as SMTP email currently is? I doubt it.[/i]" They will if those other protocols become as useful to legitimate users as SMTP. That was my point -- the same things that make SMTP uniquely useful to those of us who are not spammers make it uniquely useful to those who [b]are[/b] spammers. "[i]Give the superset 5 - 7 years to be "optional" before becoming "mandatory" and the issue is solved.[/i]" 1. Again: you don't need to replace SMTP. SMTP already provides the functionality you need via add-on functionality. The superset already exists. Creating an integrated superset just causes backward compatibility issues, obsolescence issues, and dilution problems for adoption of a single standardized approach. 2. You can't make it "mandatory" without legislation -- which is a great way to [b]really[/b] screw things up. "[i]But [corporate-controlled protocols] are [replacing SMTP], for a huge number of people that have a choice.[/i]" . . . for relatively small, "walled garden" pseudo-communities. This, alone, is by definition antithetical to the concept of a replacement for SMTP. "[i]The communication between Outlook/pine/elm/Thunderbird/whatever to the outbound SMTP server's queue are *typically* *relatively* secure (not perfect by any means) since it is almost always within the LAN or encrypted VPN.[/i]" Incorrect. The majority of email users make use of email from home (or coffee shops, or whatever). It is, believe it or not, a minority that uses email from within a controlled corporate environment. Even corporate email is often handled via an off-site service that is not accessed via VPN. The world is changing. "[i]It is fairly rare to see someone connecting across the WAN to a SMTP server to send mail.[/i]" Who's talking about a WAN? I'm talking about the Internet. "[i]The connection *most likely* to have snoopers on it is between that outbound SMTP server and the recipient's mail server.[/i]" Even if that were true (which it isn't -- it's very difficult to snoop in that part of the journey of an email, because general Internet traffic tends to follow very unpredictable paths by contrast with communication between a home client and, say, an ISP's SMTP server), end-to-end (A to Z) message encryption would solve the problem you're trying to deal with by encrypting between point B and point Y. "[i]It's the lack of authentication. In a nutshell, I think that if either SenderID or SPF became mandatory parts of the SMTP protocol, I would have 80% - 90% of what I want right there.[/i]" There are only two ways to make it a "mandatory part": 1. legislation 2. replace SMTP The first is a monumentally bad idea. I really don't think you'd like that idea any more than I would. The second is essentially doomed, for largely social reasons. In addition, too-close integration can cause obsolescence problems, as I've mentioned, making it a technically bad idea as well. [b]rclark:[/b] "[i]I think the best way to kill the spam is to stamp the email with the originating mac address.[/i]" MAC addresses can be spoofed. This would not work very well.

rclark
rclark

I agree with the authentication issue. If we nail down where spam is coming from, which is not a difficult proposition, then we can shut it down. I think the best way to kill the spam is to stamp the email with the originating mac address. Once you do that, deniability goes out the window. The IP address tells us where it entered the internet, and the mac address says where it entered the network.

Justin James
Justin James

"Someone forgot to tell both me and my significant other that we're supposed to only go to a few places where we can keep docking stations, when working." That's why I said "most", not "all". Laptop sales surpassed desktop sales in terms of units a year or two ago, if I recall. If the whole "work from the coffee shop" or "work at the beach" thing were the case for most people, we'd see it. Enough people do it to jusitfy the cost of adding a WiFi access point to a lot of businesses (what, all for $50 to do it, sell and extra coffee a week and it pays off in a few months), but it is hardly the sweeping sensation that advertisers make it out to be. "Those are differences that matter, though -- and even UMPCs' interfaces are suboptimal enough that laptops with full-size keyboards won't be driven out of the market by them, let alone smartphones." I agree, I do not think that laptops will ever fully disappear. Nor do I think that desktop towers will ever fully disappear. There will always be some workers for whom the ability to have a full OS with installable apps *anywhere* at *anytime* with a full screen/keyboard/mouse is needed, like maintenance technicians, salespeople, and so on. That's actually the biggest difference between a "smartphone" and a "UMPC", in my book, once which we *both* managed to overlook, and has been nagging me all day. It is that a "smartphone" has an OS/app loadout mostly or entirely controlled by the vendor. A UMPC does not have that. I can take a UMPC, connect a USB CD-ROM drive, and install any OS that will work on the hardware, and once the OS is installed, install any software I want. With a smartphone, the OS choice is out of my hands, and in many cases, the software choices are limited to what the telco will amke available to me, particularly on phones not running Palm OS or Windows Mobile. This is a tradeoff. The locked down nature of the smartphones makes them a bit cheaper and reliable, in theory, since anything put on there has been (again, in theory) rigorously tested before being put on there. The smartphones are definitely cheaper than the UMPCs. For the first time in computing history, a specialized device (the smartphone) is just as capable and less expensive than the general computing equivalent. But that also speaks to the maturity of the smartphone market (building on the 10 year old PDA market) compared to the UMPC market (extremely new market). "They'll just serve as complements -- creating and filling, at the same time, a new niche that coexists with the current laptop niche." I agree. Remember, my prediction here is for a full 5 - 15 years out... hardly "right around the corner". It's a long term trend. Some people will always need laptops with a full (or nearly full) size keyboard, decent mousing system (stick, trackpad, etc.), and decent sized screen build in to the same unit. But if the trend for the average user keeps going the way its been going, they won't need the laptop nor the desktop for a lot of tasks, and the UMPC/smartphone + docking station combination gives them everything they need. "Add to that the proviso that these ultra mobile devices also have to be able to do everything a general purpose PC does (and if that doesn't qualify it as a general purpose PC, I don't know what does)." This is actually the foundation of my whole prediction. General purpose CPUs have been overpowered for the typical user, and will be even more so as more and more users use fewer and fewer desktop apps and more and more Web apps. As the need for local CPU, RAM, and storage goes down, and the mobile devices get more resources, we will hit an inflection point (and I beleive we are there, plus/minus 1 year for most users) where the smartphones (and the UMPC's are definitely there) have enough resources to handle the needs of the typical user. At that point, we are looking at interface, not resource issues. "The moment you do away with SMTP in favor of one of these other protocols, it becomes widespread enough that it becomes as good a target of spam as SMTP ever was. It's an economy of scale -- not an SMTP vulnerability contrasted with some magical lack of vulnerability of other protocols." Yes, I agree. The other protocols *will* come under attack once SMTP becomes a much harder target. Which brings up the following items: * Why *not* make SMTP more resistant to spam and such? Do we want it to be a honeypot, so the other protocols are left alone by the "bad guys"? Or do we want to move to a world where we can compute without that kind of junk? * The other protocols are indeed less attractive to the "bad guys" because they are not nearly as useful to spam through, as you have pointed out. Email, particularly SMTP-based email is uniquely useful to spam through. You don't have to wait for someone to be online, you don't have a single closed vendor "choke point" to get through. It's WIDE OPEN, and technically extremely easy to send a message through. You don't even need to attempt to be a legitimate user. If SMTP-based email is hardened to the point where it is harder to invade/infest than other protocols, will those other protocols become just as bad (end result, not in terms of attempts) as SMTP email currently is? I doubt it. * I still think that getting the SMTP protocol into the modern world, support authentication of sender ("are you, as an SMTP server, allowed to send a message on behalf of the person this message claims to be from?"), and add encryption (which is not related to the spam issue, but would serve to make email much better) can be *nothing but a good thing* for everyone but the spammers. The only negative is the backwards compatability issues. Give the superset 5 - 7 years to be "optional" before becoming "mandatory" and the issue is solved. "Maybe. I'm not convinced. Anyway, that control is the most important reason corporate-controlled protocols will never really replace SMTP." But they are, for a huge number of people that have a choice. Particlularly in the developing world, where cell phones & SMS are much more widespread than general purpose PCs with TCP/IP connectivity to the Internet. "I'm not sure what you're trying to prove with that statement." Just saying that it is quite possible to flag infected PCs at the ISP level. "Please refer to it by some other term, then, so I'll know which server you mean -- something like "POP/IMAP server", perhaps (since that's what it is)." I've started calling it "the destination mail server", that is more accurate (only a fraction of the time) than "POP/IMAP server"; technically (you often see this in Exchange environments), you can have an SMTP server in the DMZ catch incoming mail, process it, and then make it available to a POP/IMAP/MAPI server within the LAN. Although for most mail servers, the POP/IMAP server is also the SMTP server (both outbound and inbound), which is a bad idea from the security standpoint. ". . . but most of your discussion in that regard involves a break in the chain of encryption from end to end. This is the problem with trying to slip encryption in the back door for email protocols." Not really, according to the chain of messaging. The communication between Outlook/pine/elm/Thunderbird/whatever to the outbound SMTP server's queue are *typically* *relatively* secure (not perfect by any means) since it is almost always within the LAN or encrypted VPN. It is fairly rare to see someone connecting across the WAN to a SMTP server to send mail. Contrast that to the connection between that outbound SMTP queue and the recipient's mail server. That connection is almost always over the pure, raw, unencrypted Internet. Once it hits the recipient's mail server, it gets from their DMZ to their LAN (a very tightly controlled connection, hopefully!), and from there, sits within the LAN or encrypted VPN when it gets picked up. The connection *most likely* to have snoopers on it is between that outbound SMTP server and the recipient's mail server. At the end of the day, the encryption issue is a periperal discussion. Lack of encryption is not putting nails in the coffin. It's the lack of authentication. In a nutshell, I think that if either SenderID or SPF became mandatory parts of the SMTP protocol, I would have 80% - 90% of what I want right there. J.Ja

Justin James
Justin James

Amen to that. I had a much more in-depth reply written out, but decided otherwise... J.Ja

apotheon
apotheon

These regular moments of brokenness get pretty old.

apotheon
apotheon

"[i]Most people, though, when on a mission to do 'real work', almost invariably take their laptop to one of a few pre-chosen locations.[/i]" Someone forgot to tell both me and my significant other that we're supposed to only go to a few places where we can keep docking stations, when working. "[i]What's the difference between 'smartphone with WiFi adapter' and 'UMPC with cellular connectivity and Bluetooth adapter'? None at all, other than screen and keyboard size, and an inch or two, and often the OS.[/i]" Those are differences that [b]matter[/b], though -- and even UMPCs' interfaces are suboptimal enough that laptops with full-size keyboards won't be driven out of the market by them, let alone smartphones. "[i]In a year or two, the two concepts will be inseperable.[/i]" If so, they'll have converged on a point where the system's capabilities and interface are both just a little too far below standard to replace laptops with full-size keyboards. They'll just serve as complements -- creating and filling, at the same time, a new niche that coexists with the current laptop niche. "[i]'General purpose PCs', 'desktops', 'laptops', etc. mean the same thing for me. It does not mean 'x86/x64 CPUs running Windows'. It's the idea of a fully thick client that sits on a desk fairly immobile (even a laptop is immobile compared to a UMPC, cell phone, or smartphone), that only 'speaks' 80X.X networking protocols (WiFi, Ethernet).[/i]" In that case, we're back where we started -- the general purpose PC isn't going out until there are interfaces for ultra-mobile devices that are at least as good as those of general purpose PCs. Add to that the proviso that these ultra mobile devices also have to be able to do everything a general purpose PC does (and if that doesn't qualify it as a general purpose PC, I don't know what does). You seem to have this weird notion that something can only be general-purpose, personal, and a computer, all at the same time, if it is only usable while sitting on a desk (or equivalent piece of furniture). I, meanwhile, would still consider it a general-purpose PC if it could be used as I use this laptop every single day -- though I wouldn't consider using it that way unless it had a physical interface at least as good as the one built into the laptop. "[i]So yes, we are talking about the same devices, it's a terminology issue.[/i]" . . . which makes me wonder what you think makes these devices unqualified to be called "general purpose", "personal", and "computer" in the same breath. "[i]The barrier for entry is significantly higher for those vectors.[/i]" That's only because of how we use them -- which is to say "not much". The moment you do away with SMTP in favor of one of these other protocols, it becomes widespread enough that it becomes as good a target of spam as SMTP ever was. It's an economy of scale -- not an SMTP vulnerability contrasted with some magical lack of vulnerability of other protocols. "[i]Also, all of those other systems have something in common,which is that they are controlled by a few large corporate entities. It is trivial for them to clamp down on spammers.[/i]" Maybe. I'm not convinced. Anyway, that control is the most important reason corporate-controlled protocols will never really replace SMTP. "[i]Yes, well aware. That's why the ISP would do packet inspection to see if it is SMTP traffic. And software running on desktops that functions as an SMTP MTA is much more of a problem than software on the desktop that connects to the ISP's SMTP server.[/i]" I'm not sure what you're trying to prove with that statement. "[i]When I say 'SMTP server', that *includes the recipient server*.[/i]" Please refer to it by some other term, then, so I'll know which server you mean -- something like "POP/IMAP server", perhaps (since that's what it is). "[i]I really don't care too much about the ISP's SMTP server used for the customer's outbound queues. Most of the problem is when it gets delivered to the recipient.[/i]" That may be where the problem starts becoming more apparent, but it's not where the problem starts. The problem starts with the sender's computer. "[i]I care a LOT less about encryption (and authentication, for that matter) within my network to the outbound SMTP server than I do between that server and the destination.[/i]" . . . but most of your discussion in that regard involves a break in the chain of encryption from end to end. This is the problem with trying to slip encryption in the back door for email protocols.

Justin James
Justin James

"Docking stations aren't mobile. Laptops are." That's a good point. Most people, though, when on a mission to do "real work", almost invariably take their laptop to one of a few pre-chosen locations. The cost of the docking station would not make it prohibitive to have them already there much of the time. "I disagree. General purpose PCs will just become as compact and mobile as smartphones -- and may include cellphone capability. Actually, that's where we're headed now: a smartphone is a step toward a compact, ultra-mobile general-purpose PC. It's not just a replacement. Now . . . if you replaced "general-purpose PC" with something like "towers and laptops", I'd probably agree with you (and might be the first in line to buy the new device, retiring my laptop to a role like "firewall")." This is exactly what I said last week. You jumped into this in the middle. This is an ongoing topic for me last week. What's the difference between "smartphone with WiFi adapter" and "UMPC with cellular connectivity and Bluetooth adapter"? None at all, other than screen and keyboard size, and an inch or two, and often the OS. In a year or two, the two concepts will be inseperable. "General purpose PCs", "desktops", "laptops", etc. mean the same thing for me. It does not mean "x86/x64 CPUs running Windows". It's the idea of a fully thick client that sits on a desk fairly immobile (even a laptop is immobile compared to a UMPC, cell phone, or smartphone), that only "speaks" 80X.X networking protocols (WiFi, Ethernet). In other words, make it small enough to carry 24x7 comfortably, and give it cellular phone capability, and it isn't a "general purpose PC" or what I've just been calling a "desktop". The idea is that the concept of "desktop computing" is going to fade away for most users, as the apps migrate to the Web. As a result, they need less horsepower on the client machine. Furthermore, converging multiple devices (particularly the phone) saves big bucks and eases maintenance. So yes, we are talking about the same devices, it's a terminology issue. "No. We have spam via other communication media -- including IMs, SMS texting, and so on. Even VOIP can potentially be a vector for spam." The barrier for entry is significantly higher for those vectors. Look at your typical spam mail. It stinks, at a technical level. It would take a really good programmer a few weeks to write a much better spam writing "engine" that slips through the filters that what we see now. I am convinced that the people writing these applications are lousy programmers, essentially script kiddies with just enough skill to slide an SMTP sender on top of a hijack programmer. Also, all of those other systems have something in common,which is that they are controlled by a few large corporate entities. It is trivial for them to clamp down on spammers. I used to get spam on MSN Messenger all of the time. It ended quickly. Not hard for Microsoft to see and control that. No such luck with SMTP, the best we can do is form what amounts to a co-op to find and discover patterns and create filters (SpamAssassin), blacklists (SpamHaus and such), etc. In this case, "Big Brother" *is* our friend. "Adding to it is at least as effective, and from a technical standpoint is probably more effective in the long run." Fine. I can do with that too. As long s the existing standard, as it is written now, is not what we are using. "Outbound SMTP connections do not necessarily need to use the same email server -- or even the same port number. They also do not necessarily need a separate SMTP server at all." Yes, well aware. That's why the ISP would do packet inspection to see if it is SMTP traffic. And software running on desktops that functions as an SMTP MTA is much more of a problem than software on the desktop that connects to the ISP's SMTP server. "The two sides to stopping spam are preventing the client from running unauthorized code at all and rejecting it at the recipients' side. Protecting your mail client and SMTP server against unauthorized use of SMTP resources is only a very small part of this." When I say "SMTP server", that *includes the recipient server*. It is, after all, running SMTP! That is where my proposals work. I really don't care too much about the ISP's SMTP server used for the customer's outbound queues. Most of the problem is when it gets delivered to the recipient. #1, most spam doesn't use an outbound server in the tradtional sense, the spam bot *is* the outbound server (unless they are using a discovered open relay somewhere). #2, I care a LOT less about encryption (and authentication, for that matter) within my network to the outbound SMTP server than I do between that server and the destination. #3 it is a lot easier to hijack someone's PC and install software that does an MX lookup and opens an SMTP connection, than it is to hijack Outlook or another mail client. J.Ja

apotheon
apotheon

"[i]I don't see how, not if the docking station combines a USB hub, Ethernet port, and monitor output.[/i]" I thought it was obvious. I'll spell it out for you: [b]Docking stations aren't mobile. Laptops are.[/b] re: interfaces that make docking stations obsolete "[i]I think that the moment alternative interface items like this are available *and catch on*, general purpose PCs are headed for the sunset.[/i]" I disagree. General purpose PCs will just become as compact and mobile as smartphones -- and may include cellphone capability. Actually, that's where we're headed now: a smartphone [b]is a step toward a compact, ultra-mobile general-purpose PC[/b]. It's not just a replacement. Now . . . if you replaced "general-purpose PC" with something like "towers and laptops", I'd probably agree with you (and might be the first in line to buy the new device, retiring my laptop to a role like "firewall"). "[i]But SMTP is the reason why spam is *possible*.[/i]" No. We have spam via other communication media -- including IMs, SMS texting, and so on. Even VOIP can potentially be a vector for spam. Hell, junk mail you get from the post office is effectively hardcopy, meatspace spam. This is not a phenomenon that would cease to exist just because one vecotr/medium for it went away. "[i]But hey, if the SMTP protocol mandated that the recipent actually checked the sending server, and said, 'wow, this guy is sending on behalf of "bob.com", and "bob.com" doesn't have this host as a valid sender, let's reject it,' then most of your spam would be eliminated.[/i]" True, at least in the short term. This doesn't mean SMTP should be replaced. Replacing it isn't necessary for solving the problem, and doesn't actually provide any specific benefit in trying to solve the problem. Adding to it is at least as effective, and from a technical standpoint is probably [b]more[/b] effective in the long run. "[i]Nail it down like this, and sure, the home PC can have a virus, but all they can do is frustratredly try to open outbound SMTP connections, and get rejected every time.[/i]" Outbound SMTP connections do not necessarily need to use the same email server -- or even the same port number. They also do not necessarily need a separate SMTP server at all. The two sides to stopping spam are preventing the client from running unauthorized code [b]at all[/b] and rejecting it at the recipients' side. Protecting your mail client and SMTP server against unauthorized use of SMTP resources is only a very small part of this. "[i]Most cell phones have a Web browser in them, and the smartphones most assuredly do, as well as a message client that allows this.[/i]" . . . and the design of the interface [b]still[/b] sucks horribly, compared with a PC.

Justin James
Justin James

"Docking stations for smartphones impose significant detrimental interface restrictions as compared with general-purpose laptops." I don't see how, not if the docking station combines a USB hub, Ethernet port, and monitor output. This is definitely available today with off-the-shelf stuff and some custom firware, for under $100. "Interface technology needs to advance significantly -- to the point where docking stations are entirely unnecessary -- before the full-featured general purpose PC (in the form of the laptop computer) can be replaced by something like a smartphone." I think that the moment alternative interface items like this are available *and catch on*, general purpose PCs are headed for the sunset. The technology for this is getting better and better, but it has not turned the corner yet. Give it a few more years... remember, I am predicting 5 - 15 years out, not "2008". :) "People don't generally play Freecell on their computers because they specifically want to play solitaire, though -- they do so because Freecell happens to be in front of them. If you want to find out why people aren't giving up their desktop computers for smartphones, look somewhere other than the ability to play solitaire." I agree on both of these points. They happen to be on the PC, and happen to be bored and/or done with whatever they were doing, so they fire up Freecell. It's an "impulse game", for sure. Now Hearts, on the other hand... just kidding. "SMTP isn't the reason we have spam." I read the post, and it is really well done! That being said, my arguement here is not that SMTP is the *reason* we have spam. But SMTP is the reason why spam is *possible*. You are very correct that the home PC security problems lead to the vast majority of the spam we see. But hey, if the SMTP protocol mandated that the recipent actually checked the sending server, and said, "wow, this guy is sending on behalf of 'bob.com', and 'bob.com' doesn't have this host as a valid sender, let's reject it," then most of your spam would be eliminated. At that point, protecting those home PC falls into 2 categories: properly locking the keys to the SMTP server up by the programs that connect to an SMTP server like mail clients, to ensure that viruses can't get the credentials. And making sure that programs authorized to send spam are not taken over by things like macro viruses. I think item #1 is fairly easy, just follow the best practices for storing data like this. #2 has already been handled, I cannot recall the last time I heard of even Outlook being taken over by a macro virus or otherwise controlled like used to happen in the past. Nail it down like this, and sure, the home PC can have a virus, but all they can do is frustratredly try to open outbound SMTP connections, and get rejected every time. "There's little to no return on investment for SMS spam as long as cellphones continue to have physical interfaces that make instant gratification responses to spam unappetizing to recipients, and as long as text messages cannot carry hidden payloads (whether it be an image served from an arbitrarily remote location, quick click-through links in the message, or malware)." Most cell phones have a Web browser in them, and the smartphones most assuredly do, as well as a message client that allows this. I have no idea why I am not getting spam to my cell phone. But I know that if "jjames@whatever.com" got translated and sent to my cell phone, I'd throw out the phone in about 3 hours. Oddly enough, "XXXXXXXXXX@vtext.com" goes to my phone, but it doesn't get bombed. Either Verizon has some wicked filtering on their end (I suspect they do, I used to forward some emails to my phone and many of them never came through), or the spammers haven't picked up the address because it's never been posted or used anywhere on the Web. J.Ja

apotheon
apotheon

"[i]I too would prefer 100% encryption across the whole route. But if I had to choose only part of the route, it's the between server route that I choose.[/i]" If I had to choose only part of the route, from the perspective of the vast majority of email users, I'd choose between the client and the server -- because that's the point where real human beings authenticate themselves. Loss of control of my own email account is worse than loss of ability to cleanly differentiate between spam and non-spam via server authentication. "[i]Any add-on standard I am calling 'nonstandard' from the perspective of the SMTP protocol, sorry for not making that clear.[/i]" That neatly sidesteps the possibility of an encompassing standard -- which is the best answer to the problem.

Justin James
Justin James

I too would prefer 100% encryption across the whole route. But if I had to choose only part of the route, it's the between server route that I choose. Any add-on standard I am calling "nonstandard" from the perspective of the SMTP protocol, sorry for not making that clear. I know that many (most!) of them are indeed standards. But without something like an "SMTP++" as you called it, there is zero hope of seeing them appear everywhere, and an SMTP++ as you say has zero hope of appearing everywhere. Quite the Catch 22. J.Ja

apotheon
apotheon

Good idea! I'll add that compliance complaint functionality into my plans for eventual design of a browser that doesn't suck. Maybe I'll even get around to doing something with the design ideas some day.

apotheon
apotheon

"[i]I am not so concerned about the trip to my outbound SMTP server... that is sitting in the closet within my LAN, as it is for most non-mobile business users as well.[/i]" I know you're probably not saying anything about others' situations here, but just in case: Your situation is not everyone's. Others will need encryption for their authentication sessions with the SMTP server (and the POP/IMAP/MAPI server, too), in part because they will be communicating outside of their LANs. Furthermore, even within a LAN it is often a good idea to maintain authentication security, for a couple reasons: 1. Someone inside your network (depending on the network) might not be as benevolent as you think. 2. Someone outside your network might find a way in. "[i]It is the trip between my outbound server and the recipient's mail server that concerns me. That trip is not guaranteed to be safe at all. Sure, I could encrypt the message contents instead, but without some autonegotiated encryption at either the protocol level (within SMTP itself, or as a "carrier" protocol), or even at the message level, it is hard to encrypt unless it is within the corporate LAN where the IT department set up the same software for everyone.[/i]" If encryption isn't end-to-end (from the sender client to the recipient client), it's often worthless. Your case is not the norm, assuming it's as secure from your client system to the SMTP server as you think. If your only protection is from SMTP server to POP/IMAP/MAPI server, you have the problem of emails being in an unencrypted state on each of the servers -- which breaks the end-to-end security that we need. "[i]Regardless, email needs baked in encryption negotiation methods, not a slew of non-standard add ons.[/i]" The difference between standard and nonstandard is simply who is using it, and whether it's open and published. The fact it's an add-on doesn't make it necessarily "nonstandard". "[i]I think we definitely agree for the need for encryption of email from time to time, and that it needs to be standardized in a way for that to work, and work with minimal user intervention.[/i]" If there's a point of disagreement on that, it's in the fact that I believe email should be encrypted [b]all the time, every time[/b]. The two major hang-ups I have with your suggestions are: 1. adoption incentives 2. rigidity of the standard, ensuring obsolescence in the foreseeable future

Justin James
Justin James

Many times, email contains information that it probably shouldn't. I know, for example, to not put usernames and passwords, or credit card data into an email. I am not so concerned about the trip to my outbound SMTP server... that is sitting in the closet within my LAN, as it is for most non-mobile business users as well. It is the trip between my outbound server and the recipient's mail server that concerns me. That trip is not guaranteed to be safe at all. Sure, I could encrypt the message contents instead, but without some autonegotiated encryption at either the protocol level (within SMTP itself, or as a "carrier" protocol), or even at the message level, it is hard to encrypt unless it is within the corporate LAN where the IT department set up the same software for everyone. I prefer encryption to be at the SMTP level of things, just because that forces it to be available to everyone. I support if you make it part of the Internet Mail Message protocol, that would be just as good for my wants. And possibly better, now that I think about it. Regardless, email needs baked in encryption negotiation methods, not a slew of non-standard add ons. I think we definitely agree for the need for encryption of email from time to time, and that it needs to be standardized in a way for that to work, and work with minimal user intervention. We're both beleivers in standards, too. You understand this stuff better than I do, very little that I say about encryption should be taken as canonical truth, but I know enough about it to understand the mechanisms and such. I know that no matter what, the way we do things right now is wholly inadequete. J.Ja

Justin James
Justin James

That's why I say to make it part of the SMTP standard. Do it as a superset that is backwards compatible, and say, "this superset is optional for the next 5 years, then it is mandatory". No one leaves there mail server un-upgraded for 5 years, I hope. :) Look at HTML, all of the "obsolete" tags are still quite widely in use, because the browsers still interpreting them correctly. That just gave me an idea for a sweet browser add on... whenever it detects a page that is not standards compliant, it finds the "Contact Us" link, or looks up the domain owner in WHOIS and sends an email saying, "This page [URL here] is not compliant!" Annoy people into compliance... ;) J.Ja

deepsand
deepsand

It's part of the human condition to do no more than is required; and, organizations tend to behave in a like manner. I've witnessed several instances where an ASP automatically sends e-mails, triggered by certain transactions on behalf of their clients, to said clients' customers who particpated in the transaction, from their servers but under the clients' names, with the result that such e-mails bounced for not being SPF compliant. And, despite requests from their clients, said ASPs have declined to deal with SPF.

apotheon
apotheon

The "envelope" doesn't need to be encrypted between SMTP server and POP/IMAP/MAPI server. All that's needed there is verification of the source -- that it's coming from the source it claims. The content should be encrypted from true source to true destination -- from client to client. Authentication between origin client to SMTP server needs to be encrypted because there are usernames and passwords involved. The same is true between destination client and POP/IMAP/MAPI server. Between servers, we don't have that problem.

Justin James
Justin James

You are 100% correct here, and I misspoke. There are existing encrypted "carrier" protocols out there,in actual use, for SMTP. But you are also (unfortunately) right that they are not used as much as they should be. But you still are not getting end-to-end saftey here, because there is no mechanism for the SMTP server to be protected when it delivers the message to the recipient's server, because that "carrier" protocol can't be automatically negotiated. Why? Because it is outside of the SMTP protocol. So what is needed is either a universal encryption negotion system that can establish "carrier" protocol encryption, or the concept of encryption needs to be built into SMTP. Anything less will, at best, cover only the mail client to the outbound queue. J.Ja

apotheon
apotheon

"[i]SMTP does that only in the sense that it could not care less what is put into the messaging envelope.[/i]" Incorrect. It does so in another way: It doesn't care what is used to carry it from one point to another. This latter is the characteristic that could be used to reduce spam -- tunneling your outbound connection to the SMTP server through an encryption protocol. Being able to encrypt the contents of your email provides privacy, but not protection for email system resources -- and spam is an abuse of email system resources. "[i]Contrast that to say, SSH or HTTPS, where the entire connection is encrypted, included the headers and such.[/i]" SSH, SSL, and TLS can be used for secure email transactions, encrypting everything -- including the SMTP protocol transmissions themselves. SSL and TLS, at least, are already in widespread use toward this end. Even Outlook does some opportunistic encryption magic when it's talking to an Exchange server. Someone, somewhere, has probably created some kind of automatic SSH-protected connection process for connecting to his SMTP server when sending messages, too. What's needed is not a new protocol. What's needed is for mail clients to start arriving with opportunistic encryption functionality turned on by default.

Justin James
Justin James

... does not need to specify the encryption algorithm, it simply needs to provide a framework for that to be negotiated, automatically, by the systems. SMTP does that only in the sense that it could not care less what is put into the messaging envelope. So you have an encrypted message in an unencrypted envelope. Contrast that to say, SSH or HTTPS, where the entire connection is encrypted, included the headers and such. Is that overkill for email? Sure. But until something like that gets baked in at the protocol level, it will never become a system that can be counted on to be available. J.Ja

apotheon
apotheon

"[i]The issue with an "add on" layer to SMTP is that it needs to be added to the protocol itself. There have been countless attempts to have unofficial add ons to SMTP to fix it, starting with PGP encryption[/i]" [url=http://blogs.techrepublic.com.com/security/?p=393][b]Encryption systems should never be integrated directly with communication protocols.[/b][/url]

Absolutely
Absolutely

... & endurance. [i]Maybe I've been immersed in it too long, but most companies with more than a few employees that I've been in spent huge amounts of time worrying about "CYA" types of issues. Half of the decisions to outsource that I saw were based on the idea that it was a hedge against risk (legal, or career), as opposed to being a good idea in and of itself. In other words, if you pay a vendor to do something and it gets screwed up, at worst, you as a company and you as an employee get blamed for picking a bad vendor, and as long as you picked a vendor in the right portion of a Gartner "magic quadrant", you can't even get blamed for that. In other words, I have spent a LOT of time in a Dilbert-esque environment, and I've seen a lot of goofy decisions made based on thing that were irrelevant at the technical level. That colors a lot of my commentary since *my experience* has been that those kinds of environments are fairly typical and normal.[/i] I see. I have had similar experiences, and I know that getting accustomed to them and agreeing with them are very different. I hate to see talented programmers assigned to such unproductive work, but I have also noticed that it happens. http://www.circleid.com/posts/811611_david_ritz_court_spam/

Absolutely
Absolutely

Justin James: [i]I have a feeling that one day I will be able to monetize my spam. I have been theorizing that I can use spam volume as a predictor of consumer purchasing trends, for example. I can use it to test spam filters on. And so on. Its an oil well waiting to be tapped.[/i] It's certainly a unique pastime. I'll be cheering for you if it happens, but not holding my breath, in the meantime.

boxfiddler
boxfiddler

"While it would theoretically be possible, for example, for ones TelCo or ISP to archive everything, and produce such data on demand, I'm quite sure that I'd never trust them enough to use such a feature." Gotta go with you on that one, deepsand. I don't trust ANY large, profit-oriented, corporate business, which is what ISP's are. "If you want an audit trail, especially one that might need to stand up to legal scrutiny, I beleive that one maintain by a telco or an ISP would be significantly more "trustworthy" in the eyes of a court or a jury than one that your company maintains." Justin - I work for an educational institution. While not entirely certain as to all the legalities involved due to our need to conform to HIPAA (SP?), I THINK that about all that the courts would recognize from the ISP (who pretty much simply provides only external web access) without nailing us to the wall for violating regs would be time and date stamps, origin, recipient. NONE of our internal email ever gets to our 'ISP'. Thus rendering ISP time and date stamps meaningless to a large degree. The intricacies of all of this are fascinating. But I have noted within this thread at any rate what appears to be a failure to differentiate between internal and external email. Internally, email is my CYA tool of choice. WTH, externally as well... It's pretty much all I've got for now. Um... all caps here and there for emphasis, not yelling. I haven't figured out this HTML stuff that you folks use to such good effect just yet. As a side note, it is a sad state of affairs when a primary concern in any job/career is 'How do I cover my a**?"

Justin James
Justin James

I used SNARF at the last job I worked (I would get 50 - 150 emails a day there, all legit, most requiring responses), and it was a miracle worker for me. My personal spam handling is "do nothing". I just go through my email and delete the spam. My sorting settings are simply by date. Why? Because,quite simply, spam facinates me. I have literally over 100,000 spam messages back to 2000 or so in my perosnal archives. I beleive that it is probably one of the largest collections of spam emails sent to a single email address out there (I've had the same email address for around 10 years now), since I don't ever delete any of it. I think I may be able to find an archive of stuff that I thought I lost, too, which would get me back to 1996, which would make my day. I have a feeling that one day I will be able to monetize my spam. I have been theorizing that I can use spam volume as a predictor of consumer purchasing trends, for example. I can use it to test spam filters on. And so on. Its an oil well waiting to be tapped. I also am so scared of false positives at this point (nothing works well for me and what comes in, it seems like) that it is just safer to leave it alone. Finally, I find that despite the volume of spam I get, deleting it takes less than 5 minutes of my day, every day, less than the amount of time I spend going to the bathroom on any given day. Not a big deal. I know why people freak out over deleting it, but I really don't care. It's like a form of personal hygeiene to me... not a big deal to do. J.Ja

Justin James
Justin James

Maybe I've been immersed in it too long, but most companies with more than a few employees that I've been in spent huge amounts of time worrying about "CYA" types of issues. Half of the decisions to outsource that I saw were based on the idea that it was a hedge against risk (legal, or career), as opposed to being a good idea in and of itself. In other words, if you pay a vendor to do something and it gets screwed up, at worst, you as a company and you as an employee get blamed for picking a bad vendor, and as long as you picked a vendor in the right portion of a Gartner "magic quadrant", you can't even get blamed for that. In other words, I have spent a LOT of time in a Dilbert-esque environment, and I've seen a lot of goofy decisions made based on thing that were irrelevant at the technical level. That colors a lot of my commentary since *my experience* has been that those kinds of environments are fairly typical and normal. J.Ja

Absolutely
Absolutely

Justin James: [i]There have been so many incidents of things like "missing backups" and such, it actually makes a lot more sense (in terms of legal protection) to have a neutral, third-party company simply maintain an archive of every message (IM, email, SMS, etc.) that passed through their systems before heading to their customers (this is from the telco/ISP point of view) than for each of those customers to try to maintain their own archives.[/i] There have been what I'd describe as "a handful of" [b]highly-publicized[/b] incidents of that type. When I think of the number of computers with backup policies, and compare to the number of foul-ups, I'm not exactly overwhelmed with the sense that "missing backups" are a national security risk. I would agree that IT policies are probably, in many companies, not as well-implemented or carefully practiced as what's fondly recalled as some "normal" by people with more experience than I have. I blame the dot-com boom for enticing a lot of people with "get-rich-quick-scheme" mentalities into a field that, to put it mildly, does not suit their personalities. I mention that because I don't believe data that's not backed up [outside of the social sciences] is the crisis you're describing, and what problem there is, I think has little or nothing to do with your "neutral, third-party" analysis. Careless people are pretty randomly distributed through companies & industries. I don't see the third party being inherently more careful or trustworthy than in-house backups. Sure, anything you outsource may allow you to pass the buck in the worst-case scenario, but I think it presents a lot of possibilities to worsen other scenarios. If I saw a company so obsessed with CYA tactics that they didn't want direct control over their electronic records, I'd sell my shares ASAP,

Absolutely
Absolutely

But in the broad context of "E-mail's swan song" and "SMTP is broken," a better operating system can sort mail routed to you [b]and[/b] prevent your computer from being used to send messages as a spambot.

Omnifice
Omnifice

Justin has his circle of acquaintances, as do we all. I'm sure there are many variations in the percentages of who uses what within those circles...although I do find it strange that his circle differs so much from mine, apotheon's, and I'm sure others. Whatever...doesn't matter...that's his experience, and it's no more valid than me stating my circle has different results. It's all just personal experience. And...I'm not doing anything special. I use Thunderbird with no special plugins for filtering. I use the built in junk filter and simply add filters to sort the mail to folders as it comes in. Nothing special about that. What this does is leave what is about 99% junk in the default inbox, which takes a second or two to glance over and delete them all in one fell swoop. I have VERY few false positives in my junk folder...on ONE of my 20 or so accounts set up in Thunderbird. I recommend this same approach to friends and family...and it appears that it works for them as well. NONE of my friends or family are particularly tech savvy...especially family. Again, your mileage may vary...my personal experience. Yes, everyone should be careful what they do with their email addresses...but spam happens. Definitely use a throw-away address for miscellaneous sites that want one. For me (and MY circle of acquaintances), email is a great tool that does a decent job until something better comes along. Bottom line: It's not email's swan song. Cheers!

apotheon
apotheon

"[i]90% - 95% of email voolume is spam, according to everything I've read. If 90%+ of the links that were on Web pages went to SC 404's, you'd stop bothering to click links in Web pages, the Web as a concept would be 'broken'.[/i]" The numbers I've seen are closer to 85%, but that doesn't change your point. On the other hand . . . the two situations are not at all analogous. Bear with me -- this is about to get hopelessly meta: Your analogy (most emails being spam is like most links leading to missing pages) is equivalent to an analogy where one posits that most snailmail being junk is equivalent to most brick-and-mortar stores being out of business when you get there. Luckily, it's still possible to actually find a grocery store that's still in business most of the time, or I'd have had to resort to hunting and wouldn't have time or resources to write for TechRepublic. "[i]Sure, filters and such reduce that, but we all know how well they work... either they let too much through, or there are false positives.[/i]" That's the case with the most common sorts of filters, but there are filtering systems out there that don't suck. "[i]The very fact that people and organizations spend a fortune (as measured in time and money) combating the spam make it fairly clear that something has gone quite wrong.[/i]" Yeah -- two things: 1. Microsoft has arranged for the general computing market to be dominated by machines that aren't properly secured. 2. People are stupid when it comes to shopping for spam filtering solutions. "[i]But not even you have been able to show that the levels are spam and the rest of the problems that plaque email are not *enabled* (not 'caused', 'enabled', there is a WORLD of difference there) by the SMTP specification.[/i]" You made a positive assertion. It is not my logical responsibility to disprove it -- it is yours to prove it. You have failed to do so. Meanwhile, I have offered more compelling explanations for the enabling of spam, and have even taken a stab at disproving potential proofs of arguments you haven't even effectively made. "[i]The simple fact is, simply enforcing a system in which the originating SMTP server is checked against a master list (DNS server in an SPF-like situation being the most likely) of servers authorized to send mail on behalf of the domain that the email supposedly is "from" by the receiving mail server is enough to eliminate the spambots and open relays from sending email.[/i]" So far, this seems to be the case. You have yet to offer any ideas for enforcement that are not either akin to putting bullets in people's heads for failing to secure their computers against infection by spambots (ethically speaking) or less likely to succeed than simple propagation of add-on solutions to the authentication problem without molesting the SMTP specification itself. "[i]You have pointed out that if SMTP gets to be stricter, then the spammers will turn their attention elsewhere.[/i]" I don't recall saying that. See below for what I actually meant in the text of mine that you quoted. "[i]The shortcomings of SMTP make the cost of email spam precisely $0.[/i]" The fact that a computer can be infected by an automated, self-propagating spambot installation virus or worm, or by a clever trojan, is by no stretch of the imagination the fault of the SMTP spec. "[i]As you say, get a couple of Trojans running around, instant spam network. Verifying that the sender is authorized to send those emails significantly curttails that.[/i]" Not necessarily. What happens when unauthorized use of a PC makes use of otherwise legitimate channels of communication via the standards you propose to introduce and enforce? What happens is: Spam continues unabated, just under a slightly different technical model, until end-users secure their systems. Of course, that's what happens now, without your proposed solution. "[i]I've got a ton of experiential evidence.[/i]" The technical term is "anecdotal evidence" and, since others have opposite anecdotal evidence, I'm afraid I can't assume too much weight to your personal experience. "[i]Please remember, *you are not representative of computer users*.[/i]" Back atcha. "[i]You may or may not have bogus or one-use-only email addresses for usage on less-than-reputable Web sites. You may have tweaked your email client to do some sort of white/black/grey-listing based on your address book.[/i]" Actually, I haven't done either of these things. I delete spam by hand, without filters. You're right about the fact that I keep spam volume down by being careful with where I enter my email address, and I use a mail client that is very good at sorting and dealing with emails quickly (mutt, to be precise), but I don't do any automated filtering at all. "[i]And many, if not most of your friends, associates, and acquiantences do as well.[/i]" Many, for some definition of "many". Not most. Probably quite a bit fewer than you'd think. "[i]I bet that even your family's ability to keep their email clean is above average, simply because I am fairly certain from other threads that you help them with their computing; some of your knowledge is bound to rub off.[/i]" Somewhat -- but I don't base my impression of the world on the statistical outliers of my acquaintance. In fact, from what I've seen, the statistical outliers in terms of taking additional measures to keep their email clean are the people most likely to actually use networked textual communication media other than email. Those who are closer to the mainstream don't use IMs, VOIP, SMS messages, and so on. "[i]Let's get real... who can really measure global email usage?[/i]" Actually . . . email usage is probably one of the most accurately measurable statistics out there in the IT industry. All it takes is access to means of measuring the categories of email you wish to measure across enough mail servers to constitute a statistically significant sample. The only way that gets significantly off is if the server admins misreport or are simply incompetent. The key is that email across the Internet tends to travel via a twisting path. It's not simply point-to-point, and the majority of email touches a small minority of email servers in their journey. Unfortunately, this doesn't help correlate spam usage trends with trends in other messaging protocols. "[i]Because email alternatives have been booming in the last few years[/i]" Those alternatives will not [b]replace[/b] email for anyone that wouldn't ditch email [b]regardless of spam volume[/b], for the most part. The extent to which I, personally, use other alternatives to email (like this discussion at TR, for instance) is the extent to which email [b]itself[/b] is not as suitable to my needs as the alternative. "[i]So yes, you do have the option here of saying, 'well, without the numbers to back it up, you can't prove your point.'[/i]" That's not the sole factor I'm looking at, though -- there's also the fact that my own experience directly contradicts yours. It contradicts yours so thoroughly that, even ignoring the technical, social, and logical reasoning that has gone into my arguments, my first reaction to your observations is "What planet is he living on that he sees these trends so different from those visible to me and pretty much everyone I know?" Obviously, you must live in a [b]very[/b] different world than not just me, but pretty much everyone I know -- everyone with enough on the ball to observe what's going on around them, anyway. "[i]the 'broken-ness' of the SMTP system will drive the members of group #3 to the alternatives.[/i]" What I see as more likely is that people who only used email (regardless of protocol) because IM and SMS systems weren't widespread enough will end up mostly using IM and SMS systems [b]even if they end up more spam-hammered than email[/b], because what they want is brief realtime messaging ability -- not email. For those of us who want email, it isn't drying up any time soon just because of spam -- even if spam was the fault of SMTP (which it isn't). "[i]Meanwhile, the members of group #1 continue to have alternative systems (primarily smartphones/PDAs of the BlackBerry type of device) which often have an SMTP gateway somewhere, but rely on MAPI or IMAP within the corporate firewall for the vast majority of their communications. While still being email, they've sidestepped the spam issue by using a mobile messenger as IM for than email.[/i]" No, they haven't. They've sidestepped it by using firewalled mail servers that aren't subject to spam problems -- and would gain the same benefits for email communications as well. If you only communicate with others inside the same walls, you never need to open the door to unwanted visitors.

Justin James
Justin James

"In plain English, given sensible choice of operating system, the sorting of mail according to sender, subject, date/time received or any other *variable* deemed *pertinent* by the *recipient* are all trivial." I used a great Outlook add-on called SNARF from Microsoft Research, it worked wonders with this as well. No "sensible OS" required, either. One thing I liked it that it let me define self-writing rules in a way for determining message priority, based upon my past communications with the sender. For example, emails to my boss were considered critical, because it saw that I emailed him many times a day, and emails from the CIO were critical because I tended to respond to his emails quickly. But emails from other people got downgraded. Very neat app. The existence of these things is evidence that the way we approach messaging (these things are higher up the chain) simply is not adequete for many users' needs. J.Ja

Justin James
Justin James

#1... Apple's QuickTime installer just made my hitlist. It decided that IE needed to be shutdown to update the plugin, so it shut IE down... without notification or warning. Losing the stuff I had already typed but not posted. Hooray. "1. You keep talking about how it's broken, and failing to establish that in the minds of those who disagree with you (more than agree with you, it seems). Could we possibly settle that before proceeding as though your assumption is true?" Certainly. 90% - 95% of email voolume is spam, according to everything I've read. If 90%+ of the links that were on Web pages went to SC 404's, you'd stop bothering to click links in Web pages, the Web as a concept would be "broken". Or if 90%+ of a search engine's results were no good, you would stop using that search engine, it would be "broken". And so on. Some people defend email, and say that postal mail has about the same ratio of junk to ligit items. That's fair enough. They also conveniently forget that most people get only a few pieces of mail a day, and only a few pieces of legit mail per week or even per month; meanwhile, spam can easily reach 30 - 50 messages (unfiltered) per day per account, and most people have multiple accounts. Sure, filters and such reduce that, but we all know how well they work... either they let too much through, or there are false positives. The very fact that people and organizations spend a fortune (as measured in time and money) combating the spam make it fairly clear that something has gone quite wrong. I would also like to point out that you are the only one to respond with something truly meaningful on this. Every other responder has not at all addressed this topic at a technical level; at best, they show that email as a concept is a good one (which I wholeheartedly agree with!). But not even you have been able to show that the levels are spam and the rest of the problems that plaque email are not *enabled* (not "caused", "enabled", there is a WORLD of difference there) by the SMTP specification. The simple fact is, simply enforcing a system in which the originating SMTP server is checked against a master list (DNS server in an SPF-like situation being the most likely) of servers authorized to send mail on behalf of the domain that the email supposedly is "from" by the receiving mail server is enough to eliminate the spambots and open relays from sending email. Period. You have not addressed this. You have offered some great points regarding encryption. You have pointed out that if SMTP gets to be stricter, then the spammers will turn their attention elsewhere. In your post on the topic, you state: "Spam email would not be nearly as beneficial to spammers if they had to send their own email ? the reason it is cost effective is that spammers aren?t really doing the spamming." I agree completely. The shortcomings of SMTP make the cost of email spam precisely $0. As you say, get a couple of Trojans running around, instant spam network. Verifying that the sender is authorized to send those emails significantly curttails that. Adding into the SMTP protocol that a valid reverse pointer entry must exist as well puts the nail in the coffin. I know; my personal server was prohibited for ages from sending email to certain domains because they voluntarily prohibited mail from IPs without RPTR entries, and I had to jump through hoops to get that sorted out. In fact, I will go as far as to say that many domain owners have been quite succesful in dramatically reducing spam by simply checking RPTR entries and SPF entries. Sadly, *because these are not mandatory in the SMTP protocol*, they also prohibit mail from a significant number of legitimate servers. I would love to see someone give a response to this item. "2. Here's the biggie: What evidence do you have that people are abandoning email? I don't see it. I don't see any evidence of it. Email appears to be alive and well, in terms of people using it. Hell, other means of online communication often rely on email, in that people expect email notification when they get messages through other means (including RSS)." I've got a ton of experiential evidence. I walk in a wide variety of crowds, with some pretty varied people and demographics. I get to see a lot of how non-techies use tech, as a result. The abandonment of email seems to be a constant amongst the groups I'm associated with, *amongst those with a choice*. That means consumers. Business people still use email, but many of them are forced to, due to a variety of factors (need for an audit trail, "lowest common denominator" between customers and vendors, Internet filtering, needs to have a large archive of emails, restrictions on installing software, and so on). The business users have a different problem entirely, and it is cultural, not technical, in that the volume of *legitimate* email is overwhelming. But that is another issue entirely. The end result is the same. Please remember, *you are not representative of computer users*. You know how to keep your email "clean" by being careful where you enter your email address. You may or may not have bogus or one-use-only email addresses for usage on less-than-reputable Web sites. You may have tweaked your email client to do some sort of white/black/grey-listing based on your address book. You're a smart guy who knows the tools. And many, if not most of your friends, associates, and acquiantences do as well. In fact, I bet that even your family's ability to keep their email clean is above average, simply because I am fairly certain from other threads that you help them with their computing; some of your knowledge is bound to rub off. I am always really cautious of any kind of numbers on these things. Let's get real... who can really measure global email usage? And we all know the rubric about the types of lies out there. Even if the "90% - 95% of email is spam" number is wrong, it matches fairly well with my experience. More to the point, even if it has a wide margin of error, it is still a huge number showing a real problem. So why am I back on that percentage? Because it gets higher every single year. But again, at best that is giving us a snapshot, and who knows what the methodology behind it is? I've seen some organizations where people send messages via BlackBerry so quickly it might as well be IM, but it never leaves the firewall, it all stays on the corporate Exchange server. I have seen other groups of people who check their email once a month. I think I can summarize groups of email users roughly like so: #1 - Power users: These users understand the options in their mail client. They have multiple emails accounts and their client is set up to check them all. Their email is relatively spam free, and they experience a low number of false positives by the filters, since they have everything tweaked, white/black/grey-listed, etc. They know enough to maintain a throw-away account for things like signing up for software demos, which further reduces their email load. This user treats their primary account like gold, and has a large network of people who have this account on file. As such, this user is reluctant to ever change it. This user also typically only uses their work/business email for work/business. The last thing they want is for private communications to be tied to their employers systems, and they understand that messages through their employers systems are not private. #2 - Dedicated, ignorant emailers: This group is stuck on email. They use it, and do not explore the alternatives for a variety of reasons. Chances are, it took them a while just to learn email as an idea. They are not computer savvy, and even worse, they are not email savvy. They have 1 email account, quite possibly (probably, at this point) a free Web mail account at that. This user uses the same email address for everything, so it receives a TON of spam. If this user's email provider does not do any filtering, they are innundated. Chances are, someone else set up the account for them, and showed them how to use it ("click on the envelope icon, twice, really quickly to open your email..."), so there is a very high cost for them to even use a different account. #3 - Messaging agnostic: This group messages, they do not send email per se. Whatever is convenient for them is what they use. They are not "early adopters", but they are "frequent adopters". They bounce from IM to email to TXT to phone to "walled garden" without thinking about issues like maintaining a message archive, syncing contact lists, and so on. Messaging is a tool of convenience, and as soon as a particular method or venue becomes inconveneient, they move on. This group is often characterized by the frequent changing of email addresses, IM screen names, and sometimes even phone numbers. Teenagers fall into this group more often than not, and the group is mostly made up of people under 40. Overall, these users have zero desire or patience to learn how to do what they want to do, they just want to do it, which is often a self-sabotaging attitude. What I am personally seeing is that the third group is a LOT bigger than anyone ever thought it was. Until recently, they seemed to mostly fit into group #2 because they will never learn enough to be a power user, but they did not have too many options either. Because email alternatives have been booming in the last few years, the folks who were not particularly tied to email now have a chance to either leave it entirely, or heavily supplement it with other methods. Personally, I am user #1. But because of the people I associate with, I am finding that non-SMTP based messaging is how I contact people. It's mostly split between MySpace (ugh) and TXT at the moment; a year or so ago it was almost all IM. It's a moving target. SMTP-based email remains a lowest common denominator, and quite probably always will. But between the relative merits of other systems and the severe spam/virus/phishing situation with traditional email, the folks in group #3 are leaving fairly rapidly. Again, all I can talk about here is my personal observations, since no one will have good numbers on this at a fine-grained level with a methodology that makes sense. Overall, in the course of my posts, you will see very little reference to numbers. Some call that a weakness of mine, I suppose. Personally, it is because so much of what I write is strictly based on my own personal experience. In my time around this industry, it's become pretty clear that no matter what the topic, someone can dig up some number from a fairly credible source with a fairly credible measurement methodology which shows something different from another group of numbers with a similar level of quality. Heck, just look at discussions regarding browser market share. No matter where the numbers come from, someone is going to find some pretty good reasons why they are not the canonical truth. So yes, you do have the option here of saying, "well, without the numbers to back it up, you can't prove your point." That door is wide open, and it is better to point it out than to even bother to try to hide that fact. I trust that my history has shown that my experience is usually, if not almost always, extremely representative. Also adding it, of course, that this is part of my overall market prediction, which is a long-term target (5 - 15 years). But I definitely think that, based on what I've been observing, email usage has turned a corner, and group #3 will continue to represent a bigger portion of users while at the same time the "broken-ness" of the SMTP system will drive the members of group #3 to the alternatives. Meanwhile, the members of group #1 continue to have alternative systems (primarily smartphones/PDAs of the BlackBerry type of device) which often have an SMTP gateway somewhere, but rely on MAPI or IMAP within the corporate firewall for the vast majority of their communications. While still being email, they've sidestepped the spam issue by using a mobile messenger as IM for than email. J.Ja

Absolutely
Absolutely

In fact, after reading & implementing some of Vincent Danen's procmail recipes http://blogs.techrepublic.com.com/opensource/?p=145&tag=nl.e138 http://blogs.techrepublic.com.com/opensource/?p=148&tag=nl.e550 [hosted by Jack Wallen, copyright WTF, TR is so confusing these days!] I'm durn near ready to start using ee-lectronic mail and the en-tar Intarweb agin! In plain English, given sensible choice of operating system, the sorting of mail according to sender, subject, date/time received or any other *variable* deemed *pertinent* by the *recipient* are all trivial. Unfortunately, the "sensible choice of operating system" is not yet "given" in the analysis of many [most?] end-[L]users. :p

Omnifice
Omnifice

...can't seem to post a deeper level... I keep hearing "everybody I know is abandoning email". Well, ok, everybody YOU know maybe. This just goes back to one of my original points that we all run in different circles...not really a good way to measure the amount of people abandoning email. Cheers!

apotheon
apotheon

"[i]Unfortunately, the broken nature of SMTP is leading to the abandonment of email as a communication system.[/i]" 1. You keep talking about how it's broken, and failing to establish that in the minds of those who disagree with you (more than agree with you, it seems). Could we possibly settle that before proceeding as though your assumption is true? 2. Here's the biggie: What evidence do you have that people are abandoning email? I don't see it. I don't see any evidence of it. Email appears to be alive and well, in terms of people using it. Hell, other means of online communication often rely on email, in that people expect email notification when they get messages through other means (including RSS).

Justin James
Justin James

Yes, that is correct. Unfortunately, the broken nature of SMTP is leading to the abandonment of email as a communication system. Most of the "non-techies" I know have already fled email wherever possible. They still use it for work where they are forcced to, and they might check it on occassion for personal reasons, but at this point, they get so much spam and such, they threw their hands up and gave up with it, esepcially for the folks without PCs and who do not work with them (those who cannot afford PCs, or choose not to buy them, or who live in very rural areas, working jobs that don't have a desk). Those people rarely have a chance to use a PC, but they *always* have a cell phone with them. J.Ja

Justin James
Justin James

If you want an audit trail, especially one that might need to stand up to legal scrutiny, I beleive that one maintain by a telco or an ISP would be significantly more "trustworthy" in the eyes of a court or a jury than one that your company maintains. There have been so many incidents of things like "missing backups" and such, it actually makes a lot more sense (in terms of legal protection) to have a neutral, third-party company simply maintain an archive of every message (IM, email, SMS, etc.) that passed through their systems before heading to their customers (this is from the telco/ISP point of view) than for each of those customers to try to maintain their own archives. In fact, I could very much see that as one of those "managed service" add ons that ISPs like to offer. I think many companies would jump at the chance to be releived of the necessity of archiving that stuff in case it needs to be audited. J.Ja

deepsand
deepsand

That's yet another strike against most of the mentioned "alternatives." While it would theoretically be possible, for example, for ones TelCo or ISP to archive [b]everything[/b], and produce such data on demand, I'm quite sure that I'd never trust them enough to use such a feature.

deepsand
deepsand

2 very distinctly different things. As for the benefits of e-mail that the mentioned alternatives lack, most of those alternatives will, by virtue of those design factors which make them different from e-mail, never possess.

apotheon
apotheon

Hopefully your authenticated SMTP is encrypted.

boxfiddler
boxfiddler

Thanks for the response. I now know a tad more than I did before I asked! :) edit a freaking typo

Justin James
Justin James

Doesn't help too much. It helps with the open relay issue, and lets you have the SMTP server public (for example, for mobile users to access). But it doesn't prevent the problem in which many spambots act as their own SMTP server. J.Ja

Omnifice
Omnifice

Yes, SMTP really needs an overhaul. No argument there. But, I've never been that concerned with privacy or spam or other annoyances because it hasn't been a huge issue for me...and I practically live with my email client open. If something needs more protection...PGP or such. We must just travel in different circles. Almost everyone I know uses SMTP. I guess that's just more "to each his/her own". The real problem isn't the great apps and tools out there. The problem is that there are malicious SOBs that have to misuse and cause problems in the first place. If everyone played nice, there wouldn't be as many issues. Ahhhh, Utopia. :) Cheers!

Omnifice
Omnifice

The "cover my butt" feature has really covered my butt many times in the past. I love it when someone I work with points a finger and cries that the reason their end of things isn't complete is because of me not sending an email...then I magically reproduce the email complete with timestamps. :) Cheers!

boxfiddler
boxfiddler

to agree with the poster above me (sorry whoever you are - my brain is old and tends to leak out my ears at night while I sleep). I am curious. One of my ISP's uses authenticated SMTP. Does that make any difference as regards your concerns? (outgoing server settings: smtpauth.blahblah.netorcom) As a mildly educated user it appears to me that this is a step in the right direction?

Justin James
Justin James

... to entrust to SMTP, is my point. Right now, I would say that out of the dozens of people I know, maybe 2 of them use SMTP based email as the communication method of preference. Most of them either went back to the phone, or forward to TXT or IMs, or maybe a lateral move to the MySpace "walled garden". All of these items provide the same level (or close to it) of timeshifting and other benefits of email (almost all sadly lacking file attachments and proper archiving, though). The point here, is that the *email* concept is great. But the SMTP protocol is allowing so much garbage in that the benefits are being choked by the problems. The fact that people have to add garbage on at the client level is proof positive of the broken nature of the SMTP system. J.Ja

boxfiddler
boxfiddler

and one of the most important tools for my students. Online coursework demands it to some degree. I am a part-time faculty member and getting responses to student questions in a timely manner is best achieved via email. I'm not in the office enough to respond quickly to phone messages. Email also serves to provide verification that I am doing my job, that students have in fact done theirs, and to prove that when a student is negligent to point of failing I have nagged that student to death for his/her work. Can't do without email - my 'cover my butt' tool of choice!

apotheon
apotheon

The option you missed is the one I'd advocate: Keep the SMTP standard. Make sure you have standards for add-ons. Create complete messaging standards that leverage SMTP and add-on standards. This keeps the flexibility of the SMTP + add-ons architecture without breaking forward-compatibility or dooming the system to languish in obscurity from day one.

Justin James
Justin James

The sad thing is, I generally agree with that entire post. I know, it is self defeating. The "success" (note the sarcastic quote marks) of the HTML standards is a great example. Go back 15 years, you'll find people pitching a fit about sendmail's refusal to properly follow the SMTP standard for all sorts of minor things, and the disasters that resulted. This entire train of thought comes from the following principles: * Where one computing system meets another, there should be an industry standard governing the transaction. Standards enable inovation and growth by allowing programmers to only implement the details of the transaction once, and system adminstrators to only need 1 system to maintain/control/groom that type of transaction. * Multiple standards to control the same type of transaction (or "transaction genre") are wasteful and limit innovation, even if one of the standards is clearly technically superior to the other more popular, less technically capable standard (see: Betamax vs. VHS). * Those industry standards need to be of sufficient clarity and simplicity to be universally followed and fully implemented in all systems that require them; no partial implementations. * It is better to have a standard force a software upgrade, given a sufficient amount of time for that to occur, than for the standard to not meet current needs. In other words, the computing environment of yesterday should not confine, constrain, or define the capabilities of today's systems. Standards that do not change limit innovation and growth. To summarize... I think we, as an industry, are in a state of crisis regarding messaging. Look at the wasted resources spent trying to control the spam problems. It's rediculous. My personal beleif is two fold: * SMTP enables these problems. * These problems are driving, and will continue to do so until they are resolved, users away from using traditional, SMTP-based email wherever users have the option to choose alternatives. Is trying to "fix", "repair", "supercede", or simply "add on" to SMTP the "right" answer? Quite possibly not. I've highlighted in the original post where SMTP doesn't meet the curent needs. The current needs are met by a variety of add-ons. Unfortunately, none of those add ons carry the full weight of being mandatory (or even supported) for a system to be SMTP compliant. This leaves the following routes available: 1) Continue coming up with standardized add ons that are not part of the SMTP or mail message specs. 2) Continue with the status quo. 3) Try to replace SMTP with an alternative protocol. 4) Try to supplement SMTP; an SMTP++ as it were. #1 has proven to be a non-starter pretty much, and cause chaos. Like mail servers bouncing mail because they insist on SPF records. #2 is not acceptable; I am fairly sure we can all agree on that. #3 is probably never going to happen. SMTP has too much momentum, much like HTTP does at this point, or Microsoft Windows, or Microsoft Office... you're looking at literally decades to change that trend. #4 is left as the least unlikely to work alternative. Are there problems with it? Sure, you've pointed out a zillion. But right now, we're still in the exploratory mode here. Personally, if it looks like I may or may not be flip-flopping on the proposed remedies here, it is because I am to an extent. I am pretty open to suggestions here on what can be done to remedy the situation. But the fact remains, "email" as a concept is far too useful to let it go away, but the status quo is unsustainable. J.Ja

apotheon
apotheon

"[i]Relatively few people use certificates because they are expensive and difficult to obtain. Let domains self-issue their own certs and set up their DNS server to authorize that server to issue certs, and all of a sudden, everyone can have certs for free that are trustworthy.[/i]" The certificates themselves are pretty much useless. Certificates are a form of public key, in essence -- but designed so that you're accepting a "public key" from a supposedly "trusted" third party. The certificate almost guarantees that the certifying authority genuinely gave the website you're visiting its seal of approval -- which assumes that you trust the certifying authority's judgment. It's really just a form of misdirection of trust. Rather than deciding for yourself whether you trust the website you're visiting, you're letting someone else decide who you trust. Ultimately, all you need is a genuine "this is who I'm talking to", and to decide for yourself who you are trusting. Notice that SSH, for instance, provides this without using "certificates". There is some technical, if dubious, benefit to the certificate system: you get your certificates from somewhere other than the website itself. This in theory means that you have out-of-band confirmation that the certificate hasn't been hijacked. Unfortunately, when dealing with commercial certifying authorities, you're dealing with someone that has a vested financial interest in granting certificates -- which very effectively counterbalances the limited benefit of the out-of-band confirmation. Furthermore, the false security of some "authority" saying a website is trustworthy tends to lull people into a sense of complacency that stops them from determining for themselves whether they trust the website in question. Until there's a certificate issuing service that doesn't operate under a conflict of interest, and until certificates are issued in a manner that doesn't impart that sense of complacency on the end-user, certificates are roughly useless (or worse) in general, whereas explicit key exchanges serve exactly the purpose for which they were designed: providing authentication (verifying identity of the website), after which the end user can decide on authorization (level of trust). "[i]We've had add on encryption and signing of email for decades now. I have yet to receive a PGP encrypted email, and I have like 7 GB of email (and that's only back to 2000 or so!), tens of thousands of emails, sitting on my system.[/i]" Is your public key available publicly -- and in a very easily accessed location? "[i]When they send the message, they can request that the encryption either be mandatory ('don't send it on if the other end does not support encryption') or optional ('encrypt it if possible, but send it anyway if encryption is not possible').[/i]" This requires a certain amount of security awareness on the part of the end user. While this is not in and of itself a problem, it is a problem for driving increased adoption of encryption -- because the best you'll get is a bunch of "use it if it's available, and don't bother me either way" as the default -- or worse, "don't bother me with this crap at all, I don't have anything to hide". Ultimately, there needs to be a turning-point where the default becomes "encrypt, unless I specifically say not to", but it won't happen without some better plans for how to solve the problem. It's a problem that needs solving, though. I just don't think your approach is sufficient to move things in that direction. It is, however, sufficient to make life difficult enough for people using a new protocol so that most will never move away from comfortable old SMTP. That's why the key seems, to me, to be in application development and server deployment defaults. Here's the last nail in the coffin for you: If you change the RFC standard used for email, major players in the commercial market for email services will *ignore* that standard. In other words, the problem of increasing email security is not a technical problem, for the most part. The technical solutions already exist, and creating an RFC standard for SMTP++ won't improve on that. There's no economic and sociological solution to the problem in your "change the standard" approach, however. For that, we need to find a way to upgrade all the applications to versions that default to security awareness enhancing settings and all the server deployment defaults to support email security features.

Justin James
Justin James

"In other words, certifying authorities are the actual scam, and certificates are the gimmick used to rope you in." I agree completely. I think that the official nameserver for a domain should be able to designate an issuing CA for each domain (or host in the domain), so self-signed certificates carry full weight, and we can dump the expensive and rediculous scheme of Verisign, Thawte, etc. Relatively few people use certificates because they are expensive and difficult to obtain. Let domains self-issue their own certs and set up their DNS server to authorize that server to issue certs, and all of a sudden, everyone can have certs for free that are trustworthy. That being said, while the existing scam system is indeed a scam, it is better than nothing. "Furthermore . . . you seem to have missed one of my points. A backward compatible encryption scheme, where it's "backward compatible" with a procedure that doesn't use encryption at all, is a problem. The problem is that encryption is only useful if it's used. The moment you stop using it, even temporarily, you can no longer trust that important information (such as authentication stuff) has not been leaked." Like I said in another message, you know more about this than I do. I see your point here, and it makes sense. I am not against message encryption, but whatever form it takes, it needs to be opportunistic, auto negotiated, and part of the standard. We've had add on encryption and signing of email for decades now. I have yet to receive a PGP encrypted email, and I have like 7 GB of email (and that's only back to 2000 or so!), tens of thousands of emails, sitting on my system. "Of course, you could do that, but then it's not a backward compatible superset. Then, it's just another option, and you're back to expecting people to suffer significant inconvenience for the Greater Good of spreading your alternative protocol gospel." Not entirely. Let that be a choice of the sender. When they send the message, they can request that the encryption either be mandatory ("don't send it on if the other end does not support encryption") or optional ("encrypt it if possible, but send it anyway if encryption is not possible"). We have the exact same choices when setting up VPN connections and such, and they are sane, sensible choices. It's essentially a QoS thing. J.Ja

apotheon
apotheon

"[i]probably starting with the usual certificate system[/i]" Certificates are essentialy a scam or a gimmick, depending on how they're being used. Certificates are poppycock, except in the very broad sense of a "certificate" as a form of digital signature. In other words, certifying authorities are the actual scam, and certificates are the gimmick used to rope you in. "[i]That server now has the message and decrypts it[/i]" Result: Untrusted encryption. It is decrypted in transit. The fact it is encrypted again in no way guarantees end-to-end privacy. Furthermore . . . you seem to have missed one of my points. A backward compatible encryption scheme, where it's "backward compatible" with a procedure that doesn't use encryption at all, is a problem. The problem is that encryption is only useful if it's used. The moment you stop using it, even temporarily, you can no longer trust that important information (such as authentication stuff) has not been leaked. I'm not saying you can't implement an encryption procedure that encrypts or not without based on the capabilities of the other end. I'm saying that for some purposes you would have to effectively break the end-to-end privacy that encryption is supposed to provide, and in others any case of a message being sent unencrypted obviates the benefits of the cases where encryption is used. "[i]This gives the sender the choice of re-sending without encryption[/i]" Of course, you could do that, but then [b]it's not a backward compatible superset[/b]. Then, it's just another option, and you're back to expecting people to suffer significant inconvenience for the Greater Good of spreading your alternative protocol gospel. (edit: fixed italics)

Justin James
Justin James

"Truly effective authentication requires encryption technology for digital signatures. Encryption systems should never be embedded in communication protocols because that approach would simply guarantee the early obsolescence of the protocol -- which obviates the point of an encryption protocol (reliability of compatibility) -- as I pointed out elsewhere." I agree. The system needs to act as a framework to allow and support encryption (of the entire communique, not just the envelope contents like PGP does) and authentication (probably starting with the usual certificate system, but allow others as well). "The value of both authentication and message encryption is attached to the assumption that they will not be "turned off" in transit, which is effectively what a backward-compatible superset protocol that is asynchronously capable would have to allow. Non-starter." I don't think this is a non-starter. Imagine this scenario: User sends email to their outbound SMTP server. The client and the server negotiate authentication (a pre-exchanged certificate, let's say) and the client requests encryption of content (similar to SSL, let's say). That server now has the message and decrypts it, and then store it temporarily, either encrypted or in clear, depending upon the server's policy for data at rest. Now, the MTA on that server looks up the destination's MX record, and initiates delivery to that system. It tries to negotiate encryption, but the other end does not support it. The outbound server cancels the transmission, and sends a bounce back to the sender saying that delivery was attempted, but the receipient's system does not support encryption. This gives the sender the choice of re-sending without encryption, or delivering the message another way, without jeopardizing the safety of the contents in transmission. If this is done as a superset to SMTP, we're in good shape on the adoption end. Where every single one of the add ons fall down, is that they are not SMTP supersets, and are not part of the SMTP standard. There is zero motivation for all SMTP vendors to support it when it isn't standard. That's the clincher. Ger Exchange, sendmail, qmail, AOL, Hotmail, GMail, and Yahoo! Mail on board, and this will be done in a year. :) J.Ja

deepsand
deepsand

would be an assumed essential, that any acceptable alternative would provide at least a capacity equivalent to that of e-mail.

Jaqui
Jaqui

saying it's a good alternative, just that it has the feature you mentioned as important. heck, if we want to talk about security with any form of messaging then there isn't a good solution available. :D

Editor's Picks