You have determined that your network has been breached. There are two standard approaches on what to do next:

  • Close the holes as fast as possible and put in safeguards from future attacks.
  • Identify the perpetrator and prepare for prosecution.

Most organizations decide to close the holes as quickly as possible because the probability of actually catching an intruder is very low. But if you can identify the hacker and opt to prosecute, you must gather as much information about the attack as possible. Data such as the hacker’s location, the domain and IP address from which the hacking took place, the name of the hacker, and what specific damage the hacker inflicted are all necessary for prosecution. One method of gaining this information is by using tried-and-true UNIX networking tools usually employed in incident-response forensics.

UNIX forensic tools
In forensic analysis, you cannot use any tools that are currently installed on the hacked system, because it is possible that those tools could have been replaced with Trojan programs. For example, the ps program that displays the process table could have been replaced with a Trojan ps program that displays everything except the process of a running hacker daemon. Whatever tools you decide to use for analyzing evidence should all be freshly installed. Key items you’ll want to look at and retain for analysis are:

  • A list of open ports and services
  • Aberrant packet behavior
  • Accurate dates, timestamps, and images of evidence
  • Suspicious IP addresses
  • Geographic locations of suspect IP addresses

Various software tools and UNIX commands can help you gather this information. Some of the UNIX tools that come bundled with most UNIX operating systems or that are freely available on the Internet and are worth familiarizing yourself with include:

  • netstat
    This tool prints a listing of network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
  • tcpdump
    This tool prints out the headers of packets on a network interface that match a Boolean expression.
  • dig
    This tool sends domain name query packets to name servers.
  • traceroute
    This tool prints the route that packets take to network hosts.
  • find
    This tool searches for files in a directory hierarchy.
  • dd
    This tool converts and copies a file.
  • grep, egrep, and awk
    These tools are used to process text.

Use netstat to determine open ports and services
Netstat is the tool to use to determine what ports and services are currently open. When you execute the command netstat -an, you’ll see a listing of all the connections along with their listening ports and the network addresses associated with these ports. The output will look something like this:
TCP 128.88.41.2:1025  140.216.41.2:80  CLOSE_WAIT
TCP 128.88.41.2:2180  140.216.41.2:80  CLOSE_WAIT
TCP 128.88.41.2:1188  140.216.41.2:80  CLOSE_WAIT

Look for patterns such as similar source ports used to connect to different sockets. (A socket is an IP address and port together e.g., 206.208.163.15:80.) In the above example, three connections (now closed) were used to connect to the Web server port, all from different source ports. If you discover a server on a particular port that is not normally in use, it’s possible that a hacker (with root level access) installed it for malicious purposes.

Use tcpdump to look for aberrant packet behavior
Use the tcpdump tool to trace packets and print out packet header information. When you execute the command tcpdump > outputfile on a shared network that is not switched and dumping to an output file, you’ll see a timestamp, a source socket, a destination socket, a TCP flag, a sequence number and offset, and a maximum segment size. The output will look something like Listing A.

One thing that should tip you off to a problem in this example is that 295.128.16.0 is not a valid Internet address, because 295 is not in the range of acceptable octets. Another indicator that something could be awry is that port 4950 is a known hacker port for ICQ exploits or attempts to log into port 23 (telnet) from clients that shouldn’t need to use telnet. The TCP flag indicated is S, which, if there were a lot of these flags in sequence, could indicate a SYN flood attack. You’ll want to look at the packet sizes and see if they look typical for the traffic on your network. In short, you should look for aberrant behavior.

Keep in mind that your tcpdump output is valid only for the time during which you executed the command, and any aberrant activity that has already ended will not be captured. For that reason, tcpdump is particularly useful when you have observed suspicious behavior on a live system.

Use dig to uncover suspicious IP addresses
The dig utility, a replacement for the older nslookup, is a good tool to use to look up suspicious IP addresses discovered through netstat, tcpdump, or other commands. To use this command, insert the IP address or hostname after the dig command, like so: dig 140.216.41.2.

The dig tool takes the IP address and returns the hostname, or, if you put the hostname after the dig command, it returns the IP address. If you need to map an IP address to find the fully qualified domain name, you can use the dig command to find out who owns it. You can then go to the American Registry for Internet Numbers (ARIN) or the WHOIS server and find contact, registration, and ownership information regarding the domain in question.

Use traceroute to find geographic physical locations
The traceroute tool can help you figure out the route a packet follows to get from one place to another. Most administrators use traceroute to find out the physical geographic location of a system.

If you decide to involve law enforcement, you’ll likely need to know the actual location where crime originated. If the hacker isn’t using relaying, false identities, or anonymizers, you can use the traceroute command in conjunction with the information garnered from the WHOIS database to find his or her physical location.

You run traceroute by listing the hostname, preferably the fully qualified domain name, after the command, like this: traceroute company.com.

The output will list the name and IP address of the destination, and will list all hops along the way, similar to the output shown in Listing B. The output is in the form hostname |IP address | probe1 | probe2 | probe3. Each probe is a timed attempt to get the proper response. Each probe also returns the time it took to get that response.

The traceroute tool is most useful if you execute it while suspicious activity is in progress, or shortly thereafter. The goal is to find the route through which the traffic between your compromised host and the suspect host traveled. The route determined from your traceroute could be different from the route an intruder used, depending on Internet traffic conditions, but usually it will be accurate. If you can determine the ISP of the suspected host, the ISP, with proper encouragement from law enforcement, can help identify the owner.

Note, however, that if there are routers or firewalls along the route that block time to live (TTL) packets, or ICMP message types, traceroute will not be able to report any findings from these nodes.

Find
There are many ways in which you can use find in incident-response forensics. For example, if you suspect that an intruder broke in by exploiting SUID or SGID files, you can use find to locate all these files (as shown in Listing C) by looking for any log file that references the SUID (2000) or SGID (4000) permission bits.

An even better use is to find all these files in advance and dump the output into a log file. You can then check this list by running new find commands from time to time and using the diff command to compare the two log files. By doing this, you’ll see if anyone, possibly an intruder, installed new SUID or SGID files. You would use diff to examine two log files by issuing this command: diff suid_gid_logfile1 suid_gidlogfile2.

Typically, the SUID and SGID file list shouldn’t change. If you see new additions, examine any new files from the SUID and SGID list and determine whether or not they should be there. You can also use find to look for world-writeable files, because these files are also often exploited. To look for all the world-writeable files, run find as shown in Listing D.

This list shouldn’t change much after your system is in production, so I recommend saving this log file and using diff to determine whether any changes occur. Running diff on the log file outputs (e.g., diff worldwrite_logfile1 worldwrite_logfile2) will tell you if new, world-writeable log files have been installed after the system has gone into production.

You can also build these commands into shell scripts and set up cron to run them on a nightly basis to check to see if new SUID, SGID, or world-writeable files have mysteriously appeared.

Use dd to preserve the evidence
The dd utility, a UNIX command for dumping data, has been a part of almost every UNIX-based operating system for as long as I can remember. Use the dd utility when you want to preserve evidence and create a forensic disk image, without changing timestamps or anything else. First, figure out where you’re going to dump the image; a writeable CD or another disk are the most likely places.

Various syntaxes for dd exist, but a typical command-line version of it looks like the example in Listing E.

The above command would tell your system to copy crime scene partition /dev/hd1 to a disk image called evidence.img. In the command, the bs switch specifies the block size of the files being transferred, and the count tells the system how many blocks to transfer. If you’re transferring the evidence image to a CD (CDs can hold 660 MB), set the count to something less than 660, to be sure you don’t accidentally overwrite information to the CD. You can then take the image and burn it on the CD. CDs might be preferable over hard drives, since they’re easier to send in the mail and not susceptible to magnetic interference.

If you have a 6-GB drive, you’ll need to run through a series of dd commands that chops the evidence into a series of images that will fit on a set of CDs. If you’re counting by 600-MB blocks, tell dd to skip the 600 block you already copied, and copy the next bunch of 600 blocks. To do that, use the skip switch and increment it by 600 each time, as shown in Listing F.

Grep, egrep, awk find 0 UID and GID accounts other than root
You’ll want to look for root accounts that might not look like root accounts. Often, intruders set up new accounts with root access that aren’t associated with the root name. To find these accounts, use the command shown in Listing G.

From the output of the command in Listing F, look for non-root accounts that have 0 UID privileges. These accounts will give you clues about where some of the damages might have been inflicted. Once you locate these 0 UID and 0 GID accounts, you can poke around in their directories for hacker rootkits and other malicious tools.

What next?
Figuring out who cracked your network is only part of the battle. After you have gathered all the information you can find, you must report the activity to the right institution, such as the FedCIRC, a government-run organization dedicated to computer security-related issues affecting the civilian agencies and departments of the federal government. Other groups include the Department of Energy’s Computer Incident Advisory Capability (CIAC) and the Federal Bureau Of Investigation National Computer Crime Squad.

When you find you have been cracked and you’re able to gather the necessary information, report the criminal. The more of these cracks that are reported, the better the chances that something can be done to stop them.