The hair stands up on the back of your neck, and you feel the first bead of sweat roll down the side of your face: You’ve been hacked. The adrenaline starts to flow and you’re ready to jump into action. But what do you do first?

In the first installment of this series, we talked about the steps you should take within the first few minutes of discovering that someone may have compromised your system. Now, we’ll focus on actions you should take during the first hour. Last time, we wrapped up the initial steps by disconnecting the network from the Internet. In this article, we’ll see what you need to do to patch all vulnerabilities and get back online.

First reactions

To learn what to do immediately after you detect an attack, see “You’ve been hacked: What to do in the first five minutes.”

Image the system to preserve a record
If you’re hoping to identify the individual who caused the problem or perform more diagnosis to determine the exact source of the attack, you’ll want to image the system or systems that were compromised. Imaging a system, using a package such as Symantec Ghost, creates a file you can use to re-create the system as it was when you discovered the attack. You can copy the image to another hard drive to preserve a record of the problem and then use it to help pinpoint the attacker or specific vulnerabilities. Once you’ve created the image, you can restore the production system to its operational state.

It’s best to image to another set of hard drives and use them for your further investigations, because the imaging process copies only data that is marked as being used in the file allocation tables. As a result, recently deleted files that can be recovered will not be copied to the image. Ultimately, few go to the lengths required to recover files that have already been deleted, but some do.

Evaluate systems to detect tampering
Before reconnecting systems to the Internet, you have to determine whether they have been compromised. This is perhaps the most difficult part of being hacked because it requires a critical look at the status of the systems and the logs that may have been generated.

The first step is to review every security account on the machine and all of the connected systems. In a typical network environment, this means both the local machine accounts and the network accounts coming from Novell Directory Services (NDS) or Active Directory. It also means reviewing database accounts and verifying that they have not been tampered with. This includes making sure that none of the disabled accounts that you have in your system has been activated. You’re looking for any account that shouldn’t be there or can’t be explained. If you find one, it should be disabled until you can determine its reason for being on the network.

The next step is to review group memberships to ensure that no new users have been given more group memberships and therefore more privileges than they should have. Pay particular attention to the administrators group, since this group membership essentially allows users to do anything they want to the system. The easiest way to approach this is to go to each group and review its memberships. If a user account has a membership it shouldn’t have, you should remove the membership and flag the account for further evaluation.

You’ll also need to review the access control lists (ACLs) for the system. This is probably best done with a tool such as SomarSoft, which allows you to capture all the ACLs from the system and output them to an easily readable format. You’re looking for entries that appear to give more access than is appropriate.

On the logs side of the evaluation process, you should first review the existing logs for abnormal patterns of usage, excessive errors, or anything that just doesn’t look right. You should also turn up logging to its maximum levels so that you can track further attempted attacks. This step may have performance implications and may require that the logs be expanded. This is a normal recourse when you’ve been hacked. But if the logging level is so high that you’ll never be able to review all the logs, you should reduce the level so that the logs are manageable.

If you do find log entries that you aren’t comfortable with, the system should be isolated until the source of those entries can be identified or until you can provide detailed observation of the system. Log entries can indicate successful and unsuccessful attempts to use security, but most frequently, they’re set to capture only information pertaining to failed attempts. It’s quite possible that you’ll quit seeing failure events in the log because the hacker has gained access.

An important step is to verify that no Trojan horse programs are loaded and running on the systems. These are typically identified by antivirus programs, but they aren’t always detected by every scan engine. If you want to check your systems with an alternate antivirus engine, try Trend Micro’s online scanner.

Rebuild the compromised systems
After compromised systems have been discovered, your next challenge is to figure out what to do with them. You will have determined which systems are affected and reviewed security settings to ensure that the intruder hasn’t created another opening. However, dealing with compromised systems is difficult because the best way to handle them is impractical for most environments.

Ideally, a compromised system would be rebuilt from scratch or restored from a backup that was made before the compromise happened. Rebuilding from scratch is exceedingly time consuming. However, determining which backup to use for a restore can be difficult because it may not be possible to pinpoint the exact time of the compromise. As a result, you must guess which backup is not compromised and then reevaluate it after the restoration is done. If the compromise is still evident after the restore, you must use an even earlier backup.

Rebuilding compromised systems is painful but necessary to complete before the systems are returned to service. Because of the urgency and the pain of a proper restoration, many organizations choose to patch a system, close vulnerabilities, and either hope that the system has been cleaned or come back to the system later.

Patch vulnerabilities
Before you reconnect the network to the Internet, you need to patch all the vulnerabilities you can. In most cases, it won’t be clear what vulnerability was exploited to gain access to your network, so the goal is to patch them all.

In some environments, you’ll need to have patches approved before they can be implemented. But in many other environments, patches can be installed as needed. In either case, you should load all the patches you can. In particular, you should apply security patches to all systems, making sure that your firewall is up to date, external systems are appropriately tightened, and that all the internal systems are patched.

If you skip this step thinking you’ll get to it later, you may find that your network is compromised again shortly after you’ve cleaned everything up. The amount of effort required to really clean up from a security breach makes it imperative that you do everything that can be done to prevent further compromise.

Reconnect your systems
The physical process of reconnecting the systems to the Internet sounds trivial: You plug things back in and they start working. Of course, it’s what happens as soon as the systems are logged back in that is important.

As soon as the devices are reconnected, they’ll start making outbound connections through the firewall. You need to monitor those connections to see where they’re going. Most people discover that a lot of applications and utilities on their workstations are talking to the outside world.

Every connection should be investigated to see whether it’s valid. This is particularly true for connections that aren’t destined for port 80 (the HTTP port). To investigate the destination IP address, you can use nslookup. Type NSLOOKUP at a command prompt. Next, type SET TYPE=PTR and press [Enter]. Finally, take the IP address, reverse it, and append in-addr.arpa. After you enter this whole thing, you can look for an answer. For example, if the outside address is 216.37.52.229, you would type 229.52.37.216.in-addr.arpa. If you don’t get the answer you want, start dropping off the end of the IP address until you do. For example, our next command would be 52.37.216.in-addr.arpa.

Cleaning up pays off
When a network has been compromised, a lot of work is required to ensure that it doesn’t happen again. But despite all the effort, it’s not a good idea to cut corners. No cost can be placed on the problems that can result if you don’t completely clean up compromised systems and monitor the results after they are reconnected.

More to come

Look for the final installment—what to do in the first week after an attack—next Monday.