Software Development

Why encryption that doesn't trust the user isn't trustworthy


In the words of Wikipedia's article on pseudorandomness at the time of this writing, "a pseudorandom process is a process that appears to be random but is not." In programming, the term "pseudorandom" is most often used to refer to number generators that produce numbers with the statistical appearance of randomness (that is, the chance of any given number being generated is roughly the same as if the number generation were truly random), but an algorithm is being used to generate the numbers such that numbers can be duplicated by someone else using the same algorithm (and the same parameters).

Since numbers of varying degrees of randomness are so important to certain types of security procedures, most notably those involving encryption, the ability of some unauthorized individual to duplicate your number generator's results could constitute a severe security vulnerability. If your network's security relies on encryption -- and many do, especially if they're wireless networks -- a poor pseudorandom number generator behind the encryption scheme is something that might keep you up at night, and with good reason.

This doesn't mean you should only use systems that employ "true" random number generators. The only reason encryption systems tend to use pseudorandom number generators is that, at the current level of technology, there are no "true" software-based random number generators. You just need a generator with a sufficiently high period (the number of generated numbers that occur before the pattern repeats itself) and the introduction of some factor to the generation of pseudorandom numbers that actually is random.

In most cases, this random input is used to generate a "seed". That seed is used as the start point for generating a pseudorandom number, and it becomes significantly difficult to duplicate the operation of the pseudorandom number generator without having the same seed -- effectively impossible, in fact. The importance of your number generator's period is then to keep the pseudorandom numbers generated from repeating themselves quickly enough that a brute force attack will work. Encryption systems have been developed over the years that actually rely on sharing the seed between parties or nodes that need to be able to encrypt and decrypt each others' missives.

Other encryption systems have been developed that had so short a period they were worse than useless -- because "useless" would be something that just didn't work, but "worse than useless" is something that makes you feel secure, so that within its "protection" you say things you wouldn't say in public, but someone else is listening in without your knowledge. An example of such a system was the World War II Japanese cipher codenamed Purple by the United States cryptanalysts who repeatedly cracked the encryption keys used by that system.

There are other matters to keep in mind when developing a pseudorandom number generator algorithm (PRNG), too. For instance, the difficulty of determining previous generated numbers when all you have cracked is the current state of the PRNG should be quite high, at least in cases where the number generator is used as part of an encryption system. In systems where the number generator algorithm does not make it sufficiently difficult, a malicious security cracker can collect encrypted messages, such as your credit card numbers sent over SSL connections to e-commerce Websites, and work on determining the current state of the PRNG. Once the security cracker has that information, he might then attempt to trace previously generated numbers from that point, which would lead to the ability to recreate encryption keys used on those stored encrypted messages, and decipher them at leisure.

Some encryption systems protect you against that eventuality, where cracking the current key does not substantively help you crack past or future keys. This protection is called by some cryptographers "perfect forward secrecy". One example of an encryption system that provides perfect forward secrecy is the OTR Messaging library, used in encryption plugins for applications like the Pidgin multiprotocol IM client (the core functionality of which is coincidentally provided by a library called "libpurple").

Some encryption systems do not provide perfect forward secrecy. In fact, some systems that people have been trusting with their private communications and data for years provide quite deeply flawed forward secrecy. One example of an encryption system that utterly fails in the perfect forward secrecy arena is the Microsoft Windows PRNG, which was reverse engineered by Israeli researchers who demonstrated how easy it is to determine past and future encryption keys produced by the MS Windows integrated encryption systems if you know the current state of the PRNG.

Encryption is one of those technology areas where anyone that knows anything about the field these days knows that there is no such thing as real security through obscurity. Trying to protect your security by hiding your procedures is a losing proposition, because the technology is easily reverse engineered to determine what those procedures are. Thus, your best bet for developing and maintaining secure encryption systems is security through visibility: develop a system that does not depend on the inner workings of the system itself remaining secret to work properly. All it takes is a single leak, or a single case of someone successfully reverse engineering the procedures embodied in your algorithms, to destroy your entire illusion of secrecy.

It is in part for such reasons that open source encryption systems like OTR and OpenSSH have remained useful, strong protection for your privacy. When you open up your process to peer review, it can be fixed and improved based on the feedback you receive. When you do not, the only people who will be reviewing it are the people who want to circumvent the protections you have put in place -- and it is not in their best interest to give you feedback at all.

Unfortunately for organizations whose software development and business model are based on secrecy for their procedures and processes, you have to trust your user base with those procedures and processes to get the kind of feedback you need to improve them. In matters of security, as in other matters, trust is a two-way street. This is not a new principle of encryption security. Auguste Kerckhoffs elaborated on this principle in the 19th century, with what came to be known as Kerckhoffs' Principle -- that a cryptosystem should be secure even if everything about the system, except the key, is public knowledge. It was later reformulated by Claude Shannon, the father of information theory, to produce what is generally known as Shannon's Maxim, a rather more alarming statement that has (almost) exactly the same implications:

The enemy knows the system.

With that assumption in mind, you can dispense with the bread and circuses of "security through obscurity" and focus on the matter of trust. Would you rather trust your friends and customers, who are inclined to help you keep your encryption system secure, or would you rather keep them in the dark without any reasonable expectation that the "enemy" will be likewise hampered?

I know which group I would rather have reviewing my security technologies. Which would you prefer?

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

25 comments
Justin James
Justin James

Chad - I understand your point here completely and agree with it to an extent (I think the quality of the eyes on the code is much more valuable than the number of eyes on the code). But the PRNG in Windows is now a really bad example of it. FreeBSD just released a patch for its PRNG which (from what I can tell) is doing exactly wht Microsoft's was with the same consequences. In fact, I would not be surprised if Microsoft's code came from BSD like their TCP/IP stack used to, or if news of the Microsoft vulnerability prompted the FreeBSD team to examine their code more closely. J.Ja

apotheon
apotheon like.author.displayName 1 Like

The FreeBSD team didn't spend any time trying to downplay the danger of problems with a PRNG, and just fixed it as soon as a problem was detected. Finding bugs doesn't mean your software sucks -- it means you're searching for them, and finding them, before the "bad guys" do. The problem isn't that MS Windows' PRNG had issues, but rather that the only way to find and fix it is to reverse-engineer it, tell everyone about it, and hope Microsoft bothers to get off its corporate butt to do something about it. Microsoft's immediate response wasn't to fix the problem. It was to try to pretend there wasn't anything wrong with it.

Justin James
Justin James

I agree completely. Microsoft's repsonse to these things is typically pretty "deer in headlights" and reminds me of, say, the phone company when you tell them that your phone is out, and they pretend it is not their fault. But that was not what I was responding to... I was responding specifically to your statements in the original that "security through obscurity" does not work, and that the best way to have secure encryption (and crypto) code is to have it open source. I was pointing out that since FreeBSD's PRNG had what appears to be the exact same problem at the same time, the PRNG situation is not a good example. After all, FreeBSD is open source, so it is clear that open source certainly is not *always* more secure; in this case, it is *just as* secure. Another interesting note here. BSD code tends to get copied in a LOT of different places, due to the permissiveness of the BSD license. I wonder how many other items now have this bad code in them? That is one side effect of open source code that is extremely unpleasant. :( J.Ja

Justin James
Justin James

Yes, this is all making sense now... your second "edit" block is *exactly* what my understanding was. J.Ja

apotheon
apotheon

"[i]Nowhere do you mention the way people respond to these things. It talks *specifically* and *only* about the fact that open code can be reviewed by others.[/i]" What the heck do you think the [b]point[/b] of open source development is, as a factor in improving the security of a system, if [b]not[/b] "the way people respond to these things"? (edit: See the section in the article where I said "When you open up your process to peer review, it can be fixed and improved based on the feedback you receive." That is central to the point I was making -- the point you just said I didn't actually address in the article.) "[i]My poiunt is specifically and only that the PRNG situation is a bad example, because in this specific instance, the peer reviewed code was just as flawed as the closed code.[/i]" My point is specifically and only that the PRNG situation is not a bad example because security is [b]not just about producing something perfect from the very beginning[/b], since that's effectively [b]impossible[/b], but rather that security has a lot to do with [b]improvement[/b] after initial implementation. "[i]maybe the original was not quite as clear as you think it is[/i]" The article assumes certain understanding on the part of the reader. There's always some assumption of that nature when writing such an article, starting with the assumption that the reader understands the language in which the article is written. Unfortunately, one must choose the assumptions of readers' knowledge necessary to form a basis for the subject matter to be addressed -- otherwise one would never get the chance to address that subject, spending the entire article trying to cover the material not being assumed as part of the reader's foundational knowledge. I rather suspect that I'm assuming some piece of knowledge that just isn't actually there, in this case. I just wish I knew what was missing so I could address it. (edit: Okay, I think I've figured it out after reading the article again several times. You seem to think that my mention of Microsoft's PRNG vulnerability was meant to imply that it would not have had that vulnerability if it was open sourced. That wasn't what I intended to convey. Rather, it was simply an example of a failure of forward secrecy -- that's it. No more, and no less, in that context. In other words, somewhere between what I wrote and what you understood, there arose an assumption of more meaning in what I said about the PRNG than was intended.)

Justin James
Justin James

Please re-read your original post (not in the forums). Nowhere do you mention the way people respond to these things. It talks *specifically* and *only* about the fact that open code can be reviewed by others. My poiunt is specifically and only that the PRNG situation is a bad example, because in this specific instance, the peer reviewed code was just as flawed as the closed code. Just a simple suggestion to go use another example, because this particular example no longer supports the point as written originally. Maybe I am misreading the original, or maybe the original was not quite as clear as you think it is, I am not sure, but the point as you talk about it here is not the takeaway that I got from reading it. This happens on my blog too, where I know what I meant to say, and it seems like one or two people keep missing it. :) J.Ja

apotheon
apotheon

"[i]I was responding specifically to your statements in the original that 'security through obscurity' does not work, and that the best way to have secure encryption (and crypto) code is to have it open source. I was pointing out that since FreeBSD's PRNG had what appears to be the exact same problem at the same time, the PRNG situation is not a good example.[/i]" My point wasn't that open source crypto is necessarily perfect and invulnerable, by definition. That would be a terrible point, since it's obviously false. My point was that open source crypto algorithms and implementations allow for easier examination and testing by the "good guys", leading to more rapid advancement. You seem to be saying that the fact that both FreeBSD's PRNG and MS Windows' PRNG had issues discovered around the same time (so did the Linux kernel's PRNG, for that matter) somehow disputes my point. That couldn't be farther from the truth. My point was about how the open source community deals with issues that arise, and the fact that Microsoft may well have known about problems with its PRNG and been sitting on it for a long time without doing anything -- and in fact tried to do so even [b]after[/b] the vulnerability was discovered and made public. That sort of duplicitous dealing is pretty much [b]impossible[/b] with open source software.

Old-Fart-IV
Old-Fart-IV like.author.displayName 1 Like

Very excellent presentation of a complex concept for some people. I'm excited to see this discussion When I was in the Military working with "Crypto", vulnerabilities were discussed only on a "need to know" basis. Good to see this discussion out in the open.

alanavella
alanavella

Good fact that everybody must know, however in the practice is not so easy to "deploy" this knowledge.

apotheon
apotheon

"[i]in the practice is not so easy to 'deploy' this knowledge.[/i]" How so?

R1scFactor
R1scFactor

Good article. Poor source. Despite the perception of your credibility, citing sources such as wikipedia looks lazy, uneducated, and just plain lame. Points lost for anyone using wikipedia as a source. Boo!

alaniane
alaniane like.author.displayName 1 Like

Why is wikipedia a poor source? Is the 1898 edition of Webster's Unabridged dictionary a better source? This is an article for businesses not a college term paper. What may make a poor source for one does not make a poor source for another.

medullaoblongata
medullaoblongata like.author.displayName like.author.displayName 2 Like

You know, the only Wikipedia source in the article is to a definition of a term. How is that "lazy, uneducated, and just plain lame"? What would be the difference if the author quoted the exact same definition from some other source? It's not like the definition of "Pseudorandomness" is fluid and therefore changing all the time. There's nothing wrong with using it as a source for a definition to a term. It gives you more description than you would get by referencing a dictionary instead. Also, it doesn't look like the author based the entire article on one Wikipedia entry, since he points to various different sources. I think you're being too dismissive about the value of Wikipedia in such circumstances and one shouldn't immediately consider someone lazy or uneducated if they include it for added information within an article.

apotheon
apotheon

I'd forgotten how fashionable it has become to act like using Wikipedia was the Internet equivalent of living in a double-wide trailer.

mb.techrepublic
mb.techrepublic

Good article, but I do have a naive question: in the use of encryption for, say, software licensing, how would you avoid the issue of storing keys needed for decryption of a licence or other such encrypted data? Mark

apotheon
apotheon

It seems to me that what you're asking is how to protect encryption meant to keep the end user from being able to access content for copying, while still allowing the end user to access it for viewing. In other words, it strikes me as likely that you're talking about DRM. If that's not what you mean, please correct my mistaken impression so I (or someone, if others beat me to it) can answer the question you actually intended to ask. I'll restate your question using the term "DRM": "[i]in the use of encryption for, say, DRM, how would you avoid the issue of storing keys needed for decryption of a licence or other such encrypted data?[/i]" The short answer is that you can't effectively do that. A number of corporations are trying to both provide end users with keys for DRM so they can access the content for viewing and prevent end users from accessing the keys for DRM so they cannot copy it, at the same time, but ultimately what these corporations are trying to do is akin to having their cake and eating it. As the recent shenanigans involving repeated creation and cracking of the AACS key used to "protect" the HD-DVD format has proven, it just isn't working. In fact, one of the cracked AACS keys was located at [url]hddvdkey.com[/url] last I checked. The long answer is the very next article in TechRepublic's IT Security weblog after this one, with the title [url=http://blogs.techrepublic.com.com/security/?p=363][b]Radiohead knows more than Microsoft about security[/b][/url].

divineprime
divineprime like.author.displayName 1 Like

It was very informative, and objective. You'd make a great chess player, with that logic.

apotheon
apotheon

Chess is a closed system with an ultimate solution. While we haven't quite gotten to the point where it's easily solvable, technologically, that point is looming ever larger in the foreseeable future. That's part of the reason chess never really caught my interest. In fact, we're so close to solving the chess problem that becoming a good player is more a matter of memorization than of developing strategic skills. That's why I prefer Go. I'm not all that great at Go, because I don't play it much, but I find it much more enjoyable. Thanks for the compliments, in any case. I'm glad you found the article worthwhile.

TonytheTiger
TonytheTiger

[i]n fact, we're so close to solving the chess problem that becoming a good player is more a matter of memorization than of developing strategic skills.[/i] Isn't that how one develops strategic skill?

apotheon
apotheon

Memorization is useful for exercising strategic skill effectively, but employing strategic solutions in and of itself is at least as often a matter of creation as of "best practices" in most cases.

Michael Kassner
Michael Kassner

I only know enough to be dangerous when it comes to encryption, so I appreciated your well-written post. I do have one question though, if you have "perfect forward secrecy" does that also mean that if private keys were compromised, prior messages would still be secure?

normhaga
normhaga like.author.displayName 1 Like

It does explain encryption almost identically to that which was covered at the U of Utah in CS2420 (CSII). I would like to see this covered in greater depth.

apotheon
apotheon

. . . and thank you for the kind words. "[i]I do have one question though, if you have 'perfect forward secrecy' does that also mean that if private keys were compromised, prior messages would still be secure?[/i]" That's exactly what it means. In each "conversation" (for lack of a better term that comes to mind right at 7:30 in the morning), the users' public and/or private keys are used to calculate a "session key". That session key is a one-time throw-away key that is used only to encrypt the current conversation based on circumstances that will never be true again. With a cryptographic key protocol that does not provide perfect forward secrecy, a compromised private key can be used to calculate previous session keys. The term "perfect forward secrecy", however, refers to a system where that is not true, and a compromised private key means your previous session keys can't be calculated. In either case, if your private key is compromised, you should throw it away and generate a new one -- but the damage is much greater if you do not have perfect forward secrecy.

Penguin_me
Penguin_me like.author.displayName 1 Like

Just want to say, Thank you and Congratulations for writing a well-balanced and informative article on encryption, while avoiding the scaremongering that seems to be commonplace now.