A memory structure used in many programming languages to store state — variable values, for instance — is known as the “stack.” The most well-known languages to rely heavily on the stack are probably C and C++. Forth is a language that has gained some fame specifically for its focus on stack-based programming. Program control flow is also managed by the stack.

Variables whose memory is allocated on the stack need to be carefully managed so that data stored in them will not exceed the stack space that has been allocated. If it does, that additional data can overwrite other data stored on the stack and cause problems for other variables and program control flow.

If a malicious security cracker is able to intentionally exceed the stack space allocated to a variable, he or she can use malformed data to actually affect program control flow in a deliberate way. This sort of security compromise is known as a “stack-smashing attack” and, depending on the software whose vulnerability to the attack is exploited and that program’s execution environment, might even lead to a root compromise of the OS itself.

Memory management vulnerabilities such as stack-smashing attacks are extremely dangerous. The stack-smashing attack is in fact a type of buffer overflow attack, and may sometimes be called a stack buffer overflow attack. Many vulnerabilities can only affect the specific parts of the system the vulnerable software was meant to affect, but memory management vulnerabilities can often “break out” of the intended limits on the software to affect other parts of the system, turning an apparently innocuous piece of software into a terrible threat to the rest of the operating environment.

A number of software solutions meant to protect against, or detect, stack-smashing vulnerabilities are available. Among the ways to protect your system against stack-smashing attacks are non-executable stacks and stack canaries. Each of these includes several subtypes.

  • Non-Executable Stack: Both hardware- and software-based non-executable stack protections exist. They work by differentiating between executable stacks and non-executable stacks, so that data not intended to be executed can be stored in non-executable stack. Then, if a malicious security cracker (or a bug in your program) overwrites the end of a variable, the operating system at least won’t try to execute the malformed data. A number of operating systems implement variations on non-executable stack management, including FreeBSD PG_NX, Microsoft Windows Data Execution Prevention (not the same thing as “Software DEP”), OpenBSD W^X, and Red Hat Linux Exec Shield.
  • Stack Canaries: Stack canaries include random canaries, random XOR canaries, and terminator canaries. In essence, a canary is a designated piece of data used to validate stored data so that, when malformed data appears, the difference can be identified and hopefully an attempted vulnerability exploit can be defanged. In theory, this requires a would-be attacker to access the stack indirectly to avoid overwriting the “canary”, though implementations may vary in their effectiveness.

An entire book could be written about this subject. Further investigation of this topic is left as an exercise for the reader.