IT Employment

Basics of stack-smashing attacks and defenses against them

Understanding the basics of stack-smashing attacks can teach admins what OSes are best protected against them and developers how to protect their programs from stack buffer overflow vulnerabilities.

A memory structure used in many programming languages to store state -- variable values, for instance -- is known as the "stack." The most well-known languages to rely heavily on the stack are probably C and C++. Forth is a language that has gained some fame specifically for its focus on stack-based programming. Program control flow is also managed by the stack.

Variables whose memory is allocated on the stack need to be carefully managed so that data stored in them will not exceed the stack space that has been allocated. If it does, that additional data can overwrite other data stored on the stack and cause problems for other variables and program control flow.

If a malicious security cracker is able to intentionally exceed the stack space allocated to a variable, he or she can use malformed data to actually affect program control flow in a deliberate way. This sort of security compromise is known as a "stack-smashing attack" and, depending on the software whose vulnerability to the attack is exploited and that program's execution environment, might even lead to a root compromise of the OS itself.

Memory management vulnerabilities such as stack-smashing attacks are extremely dangerous. The stack-smashing attack is in fact a type of buffer overflow attack, and may sometimes be called a stack buffer overflow attack. Many vulnerabilities can only affect the specific parts of the system the vulnerable software was meant to affect, but memory management vulnerabilities can often "break out" of the intended limits on the software to affect other parts of the system, turning an apparently innocuous piece of software into a terrible threat to the rest of the operating environment.

A number of software solutions meant to protect against, or detect, stack-smashing vulnerabilities are available. Among the ways to protect your system against stack-smashing attacks are non-executable stacks and stack canaries. Each of these includes several subtypes.

  • Non-Executable Stack: Both hardware- and software-based non-executable stack protections exist. They work by differentiating between executable stacks and non-executable stacks, so that data not intended to be executed can be stored in non-executable stack. Then, if a malicious security cracker (or a bug in your program) overwrites the end of a variable, the operating system at least won't try to execute the malformed data. A number of operating systems implement variations on non-executable stack management, including FreeBSD PG_NX, Microsoft Windows Data Execution Prevention (not the same thing as "Software DEP"), OpenBSD W^X, and Red Hat Linux Exec Shield.
  • Stack Canaries: Stack canaries include random canaries, random XOR canaries, and terminator canaries. In essence, a canary is a designated piece of data used to validate stored data so that, when malformed data appears, the difference can be identified and hopefully an attempted vulnerability exploit can be defanged. In theory, this requires a would-be attacker to access the stack indirectly to avoid overwriting the "canary", though implementations may vary in their effectiveness.

An entire book could be written about this subject. Further investigation of this topic is left as an exercise for the reader.

About

Chad Perrin is an IT consultant, developer, and freelance professional writer. He holds both Microsoft and CompTIA certifications and is a graduate of two IT industry trade schools.

15 comments
christianbski
christianbski

I created a program back in college using C++ intended to do just that, break the stack. Windows was smart enough to know to allocate more memory and once memory was full it created virtual memory off the disk, it never crashed. I would imagine something more complicated than using memory would crash a stack.

AlexMotto
AlexMotto

Better is to implement AAI(Advanced ArtificialIntelli- gence)algorithms to manage "your memory".... In that context,would be you,and "not your computer"with an imediate response... agree ??... One CHIP could(dynamically)control the others... Parallel Programming...and with some hardware integration...QUANTUM COMPUTING...!!

Photogenic Memory
Photogenic Memory

The exploit in mention involves some serious programming understanding. it's amazing how much can be manipulating by changing memory addresses. Scary stuff. Me thinks I should bone up on some programming skills. Yep yep.

Sterling chip Camden
Sterling chip Camden

I've seen stack-smashing (intentional and otherwise) even in languages that don't provide pointers, if they don't validate array indexing.

apotheon
apotheon

Some stacks can "grow" in the middle, and some cannot (and they all have limits to how much they can grow), pretty much regardless of your OS. You managed to pick one that can, evidently. Of course, that is not enough to protect against stack-smashing attacks, in and of itself. The stack can only grow to accommodate more data if it "knows" it has to grow. Application design choices can result in stack allocation failing to take additional data "size" into account, for instance. In the general case, protecting against stack-smashing attacks is a matter of making sure you never forget anything you need to do to avoid leaving the stack vulnerable.

apotheon
apotheon

If you're inclined to learn them, programming skills can be extremely useful. I'd encourage your (new?) interest in the subject.

Aakash Shah
Aakash Shah

Hello Chad! In your article, you say 'Microsoft Windows Data Execution Prevention (not the same thing as ?Software DEP"'. Can you clarify the difference? I was under the impression that they are the same. Or, are you referring to "Microsoft Windows Data Execution Prevention" as "Hardware DEP"? Thanks!

apotheon
apotheon

I hope I didn't give the impression that languages without pointers (or an equivalent bit-shifting data management facility) are immune to this problem. Hell, I think PHP has run afoul of a number of absurdly bad stack-smashing vulnerabilities over the last decade.

apotheon
apotheon

Microsoft's "Software DEP" is also known as "SafeSEH", for "Safe Structured Exception Handling". It's basically a means of dealing with errors produced by application code, which essentially has nothing to do with NX protection. Microsoft probably calls it Software DEP sometimes because of the fact that it comes into play in its implementation of data execution prevention: when DEP encounters code it isn't supposed to execute, it doesn't exactly do The Right Thing the way NX bit support on most other OSes does. Instead of just ignoring the code as something that doesn't get executed, Microsoft's DEP raises a type of error called an "exception"; it is said to "throw an exception". Exceptions are error types that can be handled in a non-fatal way within a program. Thus, the application developer decides how an exception will be handled -- and when DEP throws an exception, SafeSEH is what handles it. By contrast, an NX bit implementation on (for instance) OpenBSD won't throw an exception, but will instead just not allow any (malicious) code that has leaked into non-executable memory to execute or to be treated like code in any way at all.

seanferd
seanferd

:^0 You wouldn't catch me outside my home in that state of mind. :0

Aakash Shah
Aakash Shah

A new article (when you have time) clarifying all of this would be much appreciated! I look forward to reading it :)

apotheon
apotheon

I think, to make sure I don't miss anything you actually want answered, I should probably start over and write something a little more formal about these subjects than just a forum comment. As such, I'll look into writing an article about the subject of Microsoft's DEP, Software DEP and SafeSEH, and SEHOP in the near future. No promises -- I haven't even considered the possibility of writing such an article until just a few moments ago, and I don't like to make promises about this sort of thing without taking some time to research a bit to see if I can simultaneously come up with enough for an article and keep it succinct enough for a single article. Hopefully, if I get something like that written and submitted to TR, it'll answer your questions. I'll aim for getting it done some time this month, if I decide to tackle that series of subjects. Part of the problem with answering this in forum discussion is that much of the confusion over the matter is the way Microsoft chooses terms for marketing value rather than technical accuracy. Sorting all that out requires a somewhat more rigorous, structured approach than typical forum discussions support.

Aakash Shah
Aakash Shah

Thanks for the information apotheon! MS introduced a feature called SEHOP starting from Vista: http://blogs.technet.com/srd/archive/2009/02/02/preventing-the-exploitation-of-seh-overwrites-with-sehop.aspx http://support.microsoft.com/kb/956607 According to the articles, it appears that the new protection SEHOP uses the /SafeSEH option, and appears to be different than DEP (or at least that's the impression that I get). If you have information about SEHOP, can you clarify this? Thanks!

Editor's Picks