Malware development vs. malware analysis is a dangerous cyclical arms race–a digital form of cat and mouse where security analysts attempt to reverse engineer malcode created by developers who go to great lengths to make reverse engineering their malware nearly impossible. This is clear from a 2015 research paper written by Christoph Csallner and Shabnam Aboughadareh, security researchers at the University of Texas in Arlington, along with Mehdi Aarmi, security researcher at Purdue University.

Csallner in his recent The Conversation post Inside the fight against malware attacks writes, “Today, as many as 80% of malware authors include elements in their attacks that specifically try to defeat malware-protection software.”

SEE: Download: 10 ways to minimize fileless malware infections (TechRepublic)

Kernel-mode, user-mode, and mixed-mode malware

The attack elements mentioned by Csallner can be grouped into three types: kernel-mode malware, user-mode malware, and mixed-mode malware.

“Some malware programs [kernel-mode malware] operate on a very low technological level, working directly with specific areas of the computer’s memory and hard-drive system, even changing how the computer works–so users can no longer trust the machines to do what is expected of them,” writes Csallner in The Conversation article. “Other malicious software works at a higher level [user-level malware], more like normal software that interacts with the operating system rather than the computer’s hardware directly.”

In their 2014 paper Mixed-Mode Malware and Its Analysis, Csallner, Aboughadareh, and Aarmi explain that the most advanced malware attack–mixed-mode malware–uses both kernel-mode and user-mode malware (Figure A), typically in the following order:

  • First phase (kernel-mode): Malware component modifies a part of the OS kernel, i.e., kernel code, kernel data, or both.
  • Second phase (user-mode): Malware component executes the main malicious payload.

Figure A

Enter SEMU

When it comes to kernel-mode, user-mode, or mixed-mode, distinguishing all three is vital. Csallner writes that most analysis tools focus on either kernel-mode or user-mode malware but not mixed-mode malware, adding, “So, they can’t catch everything, and–even the malware they do detect–can’t show every action the malware takes.”

SEE: Computer Hacking Forensic Investigation & Penetration Testing Bundle (TechRepublic Academy)

The research team’s proposed solution is SEMU, as discussed in their paper Automatic Profiling of Evasive Mixed-Mode Malware with SEMU. Besides sensing the three modes, SEMU is capable of:

  • Constructing a reverse-engineered model of the guest operating system;
  • Capturing a snapshot of the guest (malware-free) operating system state;
  • Monitoring malware activities and logging them; and
  • Analyzing the log to match system calls from user-mode with operations invoked by malware in kernel-mode.

Figure B is SEMU’s architecture and its main execution phases: Pre-malware-execution phase (left) and malware execution and post-execution log-analysis phases (right).

Figure B

Put simply, SEMU sitting outside the virtual machine containing the operating system can detect all malware operations, and create a detailed log of any abnormal activity. “That comprehensive log–recording events at the lowest levels of the virtual machine’s operating system–is the key to SEMU’s success,” writes Csallner, “because it allows human analysts to track where and how malware manipulates aspects of the operating system.”

SEE: Malware Protection Policy (Tech Pro Research)

In Csallner’s Conversation article, he notes that, when tested against other malware analysis tools, SEMU is the only publicly available tool that can consistently detect all the activity needed to understand how the installed malware works, including when malware:

  • Reads files;
  • Changes memory or file data; and
  • Sends information out over a network connection.

Comparative testing with other malware analysis tools

The research team compared SEMU to two popular malware analysis tools TEMU and Ether asking (detailed graphs available here):

  • Is SEMU execution time competitive with closely related approaches that are publicly available, i.e., TEMU and Ether? Answer: SEMU’s overhead is similar to that of TEMU and Ether.
  • Can SEMU detect evasive mixed-mode malware that evades TEMU and Ether? Answer: SEMU was the only tool able to log all the events that are necessary for analyzing these attacks. (Mixed-mode malware: DKOM, DKSM, and hooking, by manipulating OS objects or data structures such as KTHREAD, EPROCESS, DRIVER_OBJECT, and SSDT.)

Also, something important to beleaguered malware analysts, SEMU reduces the amount of manual effort required for them to understand what the malware program is trying to accomplish.

Csallner concludes, “By merging close examination of computer activity with detailed logging, and running in a safe environment where the malware couldn’t tamper with its monitoring, SEMU shows a direction for future analysis methods.”