On the Infeasibility of Modeling Polymorphic Shellcode for Signature Detection
Polymorphic malcode remains one of the most troubling threats for information security and intrusion defense systems. The ability for malcode to be automatically transformed into to a semantically equivalent variant frustrates attempts to construct a single, simple, easily verifiable representation. The authors present a quantitative analysis of the strengths and limitations of shellcode polymorphism and consider the impact of this analysis on the current practices in intrusion detection. The examination focuses on the nature of shellcode decoding routines, and the empirical evidence they have gathered illustrates the main result: that the challenge of modeling the class of self-modifying code is likely intractable even when the size of the instruction sequence (i.e., the decoder) is relatively small.