As organizations move to a global infrastructure, with workers scattered all over the world, we’re increasingly called upon to provide connectivity between various locations and disparate operating systems. At the same time, we have to protect the systems involved, as well as the data being transferred.

One of the most popular methods of setting up a secure connection between different platforms is to use IP Security Protocol (IPSec). IPSec provides cryptographic services at the IP layer supporting data origin authentication, integrity, and confidentiality. The use of IPSec is transparent to users and network applications, making it an attractive way to improve the security of existing network services. In this article, I’ll show you how to establish and configure an IPSec connection between Solaris and Linux machines.

The Linux implementation of IPSec is FreeS/WAN; Solaris provides its own implementation in Solaris 8. Because both implementations follow the standard protocol, it’s not that difficult to get them to communicate with each other. Let’s walk through the process of configuring a connection between the two systems. For this example, the Solaris machine is a Sparc20, running Solaris 8, and the Linux system is a laptop, running a heavily modified copy of Mandrake Linux.

Solaris setup
Solaris 8 includes native support for IPSec, but due to export restrictions, the CDs don’t include all the software necessary to implement it. To use IPSec, you must download and install the Solaris 8 Supplemental Encryption Packages.

If you don’t already have a user ID for the Sun Download Center, you’ll be asked to provide some information to register for downloads. Look under Individual Solaris Sparc Downloads and download the Solaris 8 Supplemental Encryption Packages Utilities package.

Once downloaded, uncompress and install the packages as shown in Listing A.

Select all of the available packages for installation and after installing the packages, reboot the system.

Now, create the security associations in /etc/inet/ipseckey, as shown in Listing B.

Make sure that the file is readable only by the root user:
chmod 0600 /etc/inet/ipseckey

What you are entering here are the two IP addresses that will be participating in the IPSec tunnel, as well as the encryption method and keys.

Next, load the Security Association database:
ipseckey -f /etc/inet/ipseckey

You can see the database contents with ipseckeydump.

Now, edit /etc/inet/ipsecinit.conf to set the security policy for the port traffic that’s to be protected. Entries can define ports, IP addresses, or subnets, as shown in Listing C.

That’s it on the Solaris side. If you want, reboot the system and make sure that the settings are loaded by running ipseckey dump and ipsecconf.

Linux configuration
For Linux, unless your distribution already includes it, you’ll need to download the FreeS/WAN package. The package includes kernel modules that will have to be built against your kernel source, as well as the utility programs that set up the IPSec connection. The FreeS/WAN documents do a pretty good job of outlining this procedure. Source and documentation are available from FreeS/WAN. You can quickly grab the latest stable release from here:\*

Again, if your distribution does not provide FreeS/WAN, you’ll need to compile your kernel with the FreeS/WAN patches to enable IPSec functionality. Provided that your kernel source is in /usr/src/linux, and the configuration corresponds to the actual kernel you’re running, the FreeS/WAN folks have made this fairly easy, as shown in Listing D.

This takes care of all the patching, kernel configuration, and installation for you. Alternatively, you can build/install individual pieces or create your own RPMs using some of the other build options.

The configuration on the Linux side is in /etc/ipsec.conf, as shown in Listing E.

The Solaris implementation limits us to a manual keying mode, rather than auto keying. The first stanza consists of basic configuration options, fairly common to any setup, with the additional entry to manually start our Sparc connection.

The conn entry outlines the two IP addresses that are participating in the connection, as well as the same encryption entries we defined on the Solaris side. Most Linux IPSec implementations come with init scripts, so to start the implementation you run:
/etc/rc.d/init.d/ipsec start

Listing F shows how to see the loaded configuration.

Note the routing entries, with the normal eth0 entries and the additional ipsec0 routes.

It’s assumed that you were able to make net connections between the two machines before enabling IPSec. If things stop working between the two machines all of a sudden, it’s probably due to typos in your configuration files or failure to enable ipseckey on the Solaris side. You can troubleshoot what’s going on while trying to connect by looking at the system logs.

tail -f /var/adm/messages

tail -f /var/log.messages

So how do we know that the connection is really encrypted? It’s pretty easy to use snoop on the Solaris side to monitor the connection. Listing G provides a look at a Telnet session, which is notorious for passing usernames and passwords in clear text. As you can see, no clear text is visible while snooping the interface.

Wrapping up
That does it for setting up a secure connection between a Solaris and Linux system. Although the two systems in our example were on the same network, it is possible to configure a secure connection to tunnel through the Internet or some other untrusted network to ensure security. Both machines may still participate in normal, insecure connections with other clients, but all traffic between them is encrypted. The advantage over ssh, or other encrypted connection types, is that IPSec can encrypt all traffic between the clients rather than being limited to a certain type of protocol.