At the time of this writing, the Heartbleed threat was discovered exactly six weeks ago. I discussed last month how it represents a flaw in certain versions of a product called OpenSSL, which protects various kinds of servers by utilizing encryption. This bug involved the improper handling of certain types of communication-related traffic that could permit unauthorized individuals to view data in memory. It applied to both public (internet-facing) and private (internal only) systems with no one operating system to blame (a refreshing change).
My article recommended updating OpenSSL to versions beyond 1.0.1 through 1.0.1f (even downgrading is preferable to running the impacted versions). I also advised readers to replace certificates (using new keys) on systems with vulnerable versions of OpenSSL and to change the passwords involved after that.
How have things fared since then? Has the tech community responded by taking the threat seriously or has the security atmosphere become a case of crying "Wolf!" so often that admins have been slow to mobilize on this?
Well, it's a bit of both, it seems. Based on the number of emails I've received from various service providers at my organization, this is indeed being taken seriously by many organizations. A little TOO seriously, as it turns out in some cases. Liam Tung reported on ZDNet earlier this month that a security researcher named Yngve Nysaeter Petterson discovered that 2500 web servers not vulnerable to Heartbleed were erroneously patched with the buggy OpenSSL version... thereby making them vulnerable. Of course, those servers were then added to the list of systems that need to be fixed.
The good news is that as of ten days ago 375,000 out of 500,000 servers which were checked did indeed get the correct patch, but 2.33% of servers still showed the problem OpenSSL version, and patching may be grinding to a halt, leaving many systems still unprotected (318,000 was one estimate from a few days ago). Certificates have not been found to be replaced as rapidly as patches have been applied, and worse, some 30,000 sites - 5% of the vulnerable ones - replaced their certificates while reusing the same private key which does not mitigate the threat. Until the compromised certificates are fully replaced, simply patching OpenSSL doesn't finish the job.
Netcraft services has reported that "only 14% of affected websites have done all three required steps after patching the Heartbleed bug - they have replaced their SSL certificates, revoked the old ones, and made sure to use a different private key."
Site operators and system administrators aren't holding all the blame however; MarketWatch.com recently issued a scathing critique ("The idiotic thing Americans did after Heartbleed") indicating that "nearly half of users who heard about the Heartbleed Internet bug still haven't changed any passwords, and about 12% say just the thought of changing passwords is 'too overwhelming."
Changing passwords is much less overwhelming than dealing with stolen account data or worse, especially if you log your passwords in a central encrypted database like Password Safe or KeePass - either of which can generate strong passwords for you. You can check to see which popular sites were impacted and patched and also determine if you should change your password on these sites thanks to a handy infographic provided by www.thesecurityblogger.com. You can also test sites yourself for the Heartbleed vulnerability via this online test.
Expanding the scope (and keeping the eggs separate)
In the wake of Heartbleed there has been some discussion regarding the need to scrap OpenSSL entirely. Passwords have been impugned in favor of biometrics and password management has also come under fire, with one blogger, Matt Ramsay of Centrify.com, suggesting the broader use of single-signon technologies (namely SAML, or Security Assertion Markup Language). "Secure SAML-based Single Sign-On means users enter passwords less frequently - perhaps just once a day - so keyboard loggers and other forms of attack, both on the client as well as server end, (namely Heartbleed) become less effective - or at least vastly more difficult to exploit on a large scale," Ramsay stated.
Whatever changes come about in encryption or passwords/password management to respond to threats like Heartbleed may not be immediately forthcoming, so it's important for system administrators to gain as wide an understanding as possible of the concepts of secured network communication. This will help them understand and predict where these pain points can and will occur.
One of the comments upon my original Heartbleed article brought up some interesting points along these lines. The user "sysop-dr" stated: "I wish people would also talk about if the systems involved also are running Perfect Forward Secrecy (PFS) [from Wikipedia: "which ensures that a session key derived from a set of long-term keys cannot be compromised if one of the long-term keys is compromised in the future"] and how this in some cases mediates this bug. Google for instance uses PFS and Twitter has implemented it. Others are also implementing it now. It won't prevent people getting the private key but because the private key is not the only thing involved in making session keys having the private key for a server is pretty much useless. Some sites are using PFS and don't know it. By setting Apache to use SSLSuite of ALL and then not the less secure ones the ALL includes things like TLS_DHE_RSA_WITH_AES_128_CBC_SHA which if you notice the DHE means that PFS is included. And you have to use a browser that uses PFS which is all recent versions of Chrome, Firefox and the last 2 versions of IE. PFS means that someone having captured a lot of traffic to a site cannot decrypt it now that they can get the private key from the server. I started looking at this because Google says that you don't need to change your Google passwords. Now we know why. I encourage everyone to look at PFS (search the net please, or look in Wikipedia) and if you can implement it on your systems do so as soon as you can. If you have discovered you have the Heartbleed bug and are freaking out, check if your Apache setup is using PFS already because you are using All SSL suites. Point your browser at an encrypted page on your server. Check the encryption details to see if the letters DHE show up in your technical details or connection details under encryption. I get DHE_RSA in Chrome and TLS_DHE_RSA_WITH_AES_128_CBC_SHA in Firefox for my server. In that case where you get something similar your users are safe and while you should still patch your OpenSSL install you are free from the worst implications of this bug because of your using PFS."
You can find out more about Perfect Forward Secrecy and how it applies to Heartbleed here.
Another element to safeguard against future threats involves deepening layers of security. I spoke recently with Jonathan Goldsmith of Fogpad, an encryption platform for Google Docs. Mr. Goldsmith stated: "The Heartbleed incident highlights the need for multiple layers of security on the web. The prime reason why Heartbleed is such a severe bug is that most sites use TLS as the only measure for protecting secure data. While the cryptography itself behind TLS is secure, bugs in software like OpenSSL show that there is much more to data security than a single layer of encryption over the data transport protocol like HTTPS. Increasingly, sensitive data is best protected by multiple layers of security, so the data remains secure even if any single layer of encryption is compromised by something like Heartbleed. One possible extra layer of protection on the web? Client-side encryption (we can't access decryption keys even under subpoena), so sensitive data is double-encrypted in transport and at-rest. Fogpad helps users securely store files in Google Drive by encrypting them client-side before they're saved, preventing cyber-attacks and providing compliance with data security regulations like HIPAA."
Finally, methodologies themselves will always play a critical role in the field of security-based system administration. Dominic Wellington of BMC.com presented last week his "Six Steps to Fix Any Configuration Problem - Not Just Heartbleed" which discussed the importance of automated discovery, management agents, audits, remediation details and documentation. Mr. Wellington wrote: "The democratization of IT means that the owners of those systems may well not have the skills to harden them on their own, so IT will need a mechanism to distribute, validate and enforce secure configurations. Because of the rate of change of virtualized environments, this process has to be automated to keep up with the current state of IT."
It's a hopeful sign that a cavalry of sorts is amassing to help prevent future incidents. Many large organizations such as IBM, Amazon Web Services, Dell, Facebook and Google offered over $3.5 million to "help maintain under-funded open source software projects which are 'essential to the global computing infrastructure' and OpenSSL will be first in line for these funds.
However, nothing can ever be entirely secure or predictable. Those of us who work in IT all know there will be another Heartbleed, much the same as we know there will be server crashes, internet outages, browser flaws, email disasters and all of the other things which chaos brings about. We're maintaining order in an inherently disordered world, after all. However, concepts such as redundancy, threat awareness, proper monitoring and in-depth discussions with the right people can help to reduce and mitigate these events before and after they occur.
Typically system administrators have had to rely on security experts to assist them in preparing for and responding to threats, since the two roles are too complex to generally be performed by one individual. The partnership hasn't always been amicable, much the same as police officers and internal affairs have butted heads in various cop movies, since system administrators may feel the security staffer is overly bossy and paranoid while the security guru feels the system administrator is a work-shirker who doesn't take threats seriously. Nevertheless, everyone is on the same side with the same end goal. As we go forward I recommend system administrators work in partnership with their security teams and gain as much insight as possible into how things currently work as well as what's coming over the next hill. That's one of the many reasons I usually chat about technology over beers on a weekly basis with my security expert.
Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.