Tech Tip: Harden software with StackGuard

By Jonathan Yarden

Securing applications and an OS from exploits isn't a simple task these days. OS security generally depends more on application layer security rather than on the OS core itself.

Current methodology for keeping OSs and applications secure is seriously lacking. Many people consider applying updates and rebooting systems a normal procedure—that shouldn't be necessary.

Over the past few years, some members of the commercial software industry may have realized that they aren't doing a good job of addressing security. This has given the free software movement room to seriously embarrass commercial software companies when it comes to OS and application security. Linux, other free UNIX-like OSs, and a host of free applications have taught Microsoft and commercial UNIX vendors alike valuable lessons about the cost of reliability.

How StackGuard may help protect against buffer overflows

Buffer overflows become exploits by tricking the CPU into running code of the attacker's choice. The attacker usually wants to run arbitrary commands, such as a remote command shell, rather than crash the machine.

Buffer overflows compromise the microprocessor stack, and this is sometimes called "stack smashing." It's exhausting to try to find and correct current applications that can be exploited in this manner; it's impossible to ensure that millions of lines of new software are going to be bug-free and totally reliable. Ultimately, the fix for stack smashing is protecting the microprocessor call return stack.

The Defense Advanced Research Projects Agency (DARPA) funded research for how to better control software failures caused by buffer overflows onto the call return stack. This would limit the exploitation of OS and application flaws. (Controlled failure of Internet applications by preventing buffer overflows would likely have prevented many exploit-driven worms.)

The proof of concept work was done using free software, but the concepts are what are important. Changes were made to the free GCC "C" compiler to protect the stack, using a "canary" value that is checked before and after a function call. Like a canary in a coal mine, this compiler change allows a running program to literally monitor itself for buffer overflows onto the call return stack, which are common with intrusions.

So if the canary value is incorrect or dies (so to speak), then the stack has been altered. The program detects this, logs an error, and then self-destructs because the process stops itself from being subverted.

Software compiled with StackGuard is highly resistant to remote buffer overflows, but not totally resistant, since nothing really can be. The Immunix research group that developed StackGuard also produced a Linux variation, in which many of the applications have been "hardened" by compiling them with StackGuard. Even Microsoft has adopted the concepts of StackGuard and has added a similar feature called the GS flag to its Visual C++ and .NET compilers.

Some security researchers claim that this stack protection feature is itself vulnerable. While this method of stack protection may not be perfect, it's better than nothing. Sure, there's always a way to break software if you're determined. However, the concept of software that monitors itself for problems is definitely a step in the right direction.

Jonathan Yarden is the senior UNIX system administrator, network security manager, and senior software architect for a regional ISP.

Editor's Picks