Last year, Microsoft quietly announced a security competition called BlueHat, back in August of 2011. It didn’t garner much media attention since it was fairly obscure and aimed at security professionals — the types that work in academia, research firms, and so on. Now, they announced the three finalists last week, and the results are interesting to look at, because not only is the contest itself a new event in the software industry, but the results may be a hint of what is to come in future versions of Windows, and other Microsoft software. While a lot is still in the early stages and speculative, there is a common thread, and that can give us some clues as to what the ultimate benefit of this contest will be for Microsoft, IT pros, and eventually common users.

First of all, the competition itself is fairly attractive for those with the know-how to participate. The first prize is worth $200,000, which is far more than any company gives for offensive security. Typically, software makers offer bounties to hackers or other security researchers when they find a bug or an exploit that could lead bad guys to take advantage of their software. The bounty system is well established by now, where a hacker can make easy money by giving the exploit to the company, which would dissuade them from releasing it in the wild, or exploiting it themselves. All the large companies like Google, Apple and Adobe offer such a system. But with the BlueHat competition, Microsoft called it the first “defensive security” system. From its blog post, the company says that while most industry players stick to offensive security, Microsoft thinks that, in the long run, a defensive approach will work better.

So, after the table was set, the contest launched to the public. In all, Microsoft received 20 entries, which is a fairly small number, but we have to remember that the bar was quite high to enter. Here, we’re talking about submitting brand new proposals to make Windows and other Microsoft products fundamentally harder to attack from a security standpoint. In their words, they were looking for runtime mitigation technologies designed to prevent the exploitation of memory safety vulnerabilities. One interesting note that the post mentions is that some of the best entries happened to be those submitted at the very last minute, even seconds before the deadline. But let’s take a look at who the finalists are. All three of them submitted new proposals that will earn them the various prize monies, and their proposals are listed on the finalist’s page.

  • Jared DeMott, security researcher: This novel defensive lowers the effect of address space disclosures and mitigates known return-oriented programming (ROP) exploits.”
  • Ivan Fratric, security researcher: “ROPGuard is a system that can detect and prevent the currently used forms of ROP attacks at runtime.”
  • Vasilis Pappas, Ph.D. student: “This proposed technique is called kBouncer, an efficient and fully transparent ROP mitigation technique.”

So while the details quickly get highly technical, it’s not hard to see a pattern here. It seems like the top people in the security community agree that the way to solve one of the most problematic issues in software security is to have ways to deal with return-oriented programming (ROP) attacks.

Attacks come in many forms, from buffer overflows to brute force attacks, but Microsoft, like every other major software maker out there, has added a lot of low-level systems to prevent a lot of those malwares and viruses from working in the first place. Something like DEP, or data execution prevention, is a huge deal that was added to the Windows core a few years ago. By itself, it can prevent code from being executed in user memory, in places where only data should reside, and not binary programs. Ironically, this is when ROP started to become so popular, because it’s a way to bypass DEP, among other things.

Basically, ROP attacks allow execution of code in the presence of non executable memory segments, and without the need to sign code either. It’s a way to get malware to be executed on computers without the user knowing it. So the best way to deal with these types of attacks right now, everyone agrees, is to deal with ROP.

So right now, what does this mean for you and me? For one thing, it’s clear that Microsoft hasn’t figured out how to deal with all the malware out there, and that’s why they created the contest and offered such a generous prize. If one of those contest entries works, and manages to remove ROP attacks completely, we could see the landscape of Windows malware change drastically in the near future, with many of the attack vectors used becoming completely useless. Then, it could also lead other software companies to start dealing with defensive security as well as offensive bounties. This could be a great opportunity for security pros to get recognition and focus on pre-emptive strategies to combat future threats.