Data Management

Enhance MRTG's network graphing with the Round Robin Database Tool

A good network monitoring tool is worth its weight in IP addresses. Unfortunately, most tools can't be customized to fit your needs. With the RRDtool, you can monitor darn near anything. Scott Lowe will help you get it up and running.


The Round Robin Database Tool (RRDtool), created by Tobias Oetiker, the author of the Multi Router Traffic Grapher (MRTG), is a system to store and display data. As stated on the RRDtool Web page, it can be thought of as a replacement for MRTG’s graphing and logging features. You can think of this utility as a replacement on steroids.

MRTG
Need more information on MRTG? Don’t forget to take a look at Scott Lowe’s "Graph and watch your network with MRTG 2.9.17."

MRTG’s original database design is rather inefficient because it allows the database to grow larger and larger over time, resulting in a loss of performance and an increase in storage requirements. RRDtool uses a much more efficient method of storing data. A specific number of data points are allotted at the time that the database is created. When these fill up, one is reused, thus saving space in the database. While this makes it sound as if data will be lost once the database fills up, that’s not the case. For example, those of you using MRTG are familiar with the daily and weekly graphs. If you use RRDtool, once the daily data points are used up, an average is taken and used for the weekly data point, and then the daily data points are reused.

In this Daily Drill Down, I’ll discuss RRDtool as it pertains to replacing the graphing component of the MRTG network-monitoring package. For my example, I’ll download RRDtool and another utility, routers.cgi, that will act as the front-end component to the new system. When I’m done, I’ll have the following:
  • ·        Data collection by MRTG (stored in an RRD database)
  • ·        Graphs generated from this RRD database by RRDtool
  • ·        A front end for a Web browser using routers.cgi

Get it and install it
You can get RRDtool from the RRDtool Web site. Download the tar’d and zipped file called rrdtool-1.0.33.tar.gz. This is version 1.0.33, which is the latest stable release (as of this writing); version 1.1 is available for a sneak preview via CVS.

CVS
Interested in giving CVS a go? Take a look at Vincent Danen’s series on CVS: "Source Code Management: Installing CVS," "Checking it out: CVS!," and "More control with CVS."

If you’re going to follow along with this example, before you begin, make sure that you have at least Perl 5.004 installed, or RRDtool won’t function.

Distribution?
I’m running on a Red Hat 7.1 installation on standard hardware, with a mostly default installation.

I’m going to go through the installation of RRDtool very quickly because it’s very straightforward and like almost any other source distribution installation. To properly follow these installation instructions, you should save the gzipped tarball into the /usr/local directory of your server and make sure that you’re a root user. Once this is done, run the following commands:
cd /usr/local
gunzip –dc rrdtool-1.0.33.tar.gz | tar xvf –
cd rrdtool-1.0.33
./configure –prefix=/usr/local/rrdtool
make
make install


Note that the make process takes quite some time to complete. Once the above steps are complete, you’ll have a fully functioning copy of RRDtool in /usr/local/rrdtool.

Integrating RRDtool with MRTG
While RRDtool can be used to graph any time-sensitive data that you deem fit, I’m going to concentrate on how RRDtool can enhance the use of MRTG and make use of the very well-written and comprehensive front end, routers.cgi.

You’ll see some major changes by the time you’re finished with this Daily Drill Down. First of all, MRTG will no longer generate any graphs whatsoever. None. MRTG will become nothing more than a data-collection engine. RRDtool will handle all the graphing work, and the presentation will be done by yet another program.

What good is this? First, the overhead of having graphs created at each and every MRTG update interval is removed. Graphs will only be created when they’re requested, saving CPU cycles. Second, you’ll end up using the round robin database format, which is much more efficient than the native MRTG database format. And you can use any front end you wish. If you get through this Daily Drill Down and don’t like what you see, you can create your own front end or find a different one.

Note
A default MRTG installation uses the rateup tool to create the graphs that you see.

I’ll assume that you have a working copy of MRTG installed and configured and that you have a target that is being actively monitored. For this example, I’ll be monitoring a Zyxel DSL router. If you don’t have MRTG installed and configured or you’re not monitoring a specific target, please read my previous article on that subject, "Graph and watch your network with MRTG 2.9.17," and get to this point before continuing.

Let’s put all the RRD configuration files in a config directory under MRTG:
mkdir /usr/local/mrtg/config

The next step in this process is to copy the configuration file that you wish to convert from the MRTG rateup tool to RRDtool. My MRTG configuration file for my DSL router is /home/mrtg/zyxel.cfg, which I’ll copy to /home/mrtg/rrdzyxel.cfg with this command:
cp /home/mrtg/zyxel.cfg /usr/local/mrtg/config/rrdzyxel.cfg

Now, I need to modify this configuration file so that it knows where to find RRDtool and certain libraries, and I must tell it which log format to use.

At the top of my /usr/local/mrtg/config/rrdzyxel.cfg file, I’ll put the following statements (I use pico for a text editor):
PathAdd: /usr/local/rrdtool/bin/
LibAdd: /usr/local/rrdtool/lib/perl/
LogFormat: rrdtool


Once this is done, I need some way to actually view the graphs. As stated, it offers an improved database format and a lot of graphing options, but RRDtool doesn’t generate graphs unless a front end asks it to. Luckily, folks out there have done a huge amount of work and created some very good scripts to do this for you. For this Daily Drill Down, I’ll download a set of scripts from Steve’s Web Applications. Table A lists files that you can download from this site. I’ll concentrate on the routers scripts, but the others are also quite useful.

Table A
Filename Version Description
http://www.cheshire.demon.co.uk/pub/routers-v1.30.tar 1.30 A set of CGI scripts for monitoring routers with MRTG and RRDtool – written in Perl
http://www.cheshire.demon.co.uk/pub/servers-v1.0.tar 1.0 A set of CGI scripts and agents that are used to monitor servers with MRTG and RRDtool
http://www.cheshire.demon.co.uk/pub/generic-v0.13.tar 0.13 beta This set of CGI scripts is designed to replace the 14all set of CGI scripts, which was the first front end developed for RRDtool.

When you download these files, save them to /usr/local. Next, create a directory called rrdscripts in /usr/local:
mkdir /usr/local/rrdscripts

Switch to this directory and expand the routers-v1.30.tar file that was downloaded in the previous step:
cd /usr/local/rrdscripts
tar –xvf /usr/local/routers-v1.30.tar


This set of scripts has a number of dependencies. If you’re closely following the instructions in this article and the article on installing MRTG, the MRTG/RRDtool parameters should be fine. If you’re running on a fairly recent, standard installation of UNIX or Linux, the built-in installation script should work fine, as well. However, if you’re running something unusual, you may need to make adjustments to your installation to make everything work. Refer to the documentation that came with the program for details if you run into trouble.

To install this set of scripts, do the following:
cd /usr/local/rrdscripts/routers-v1.30
perl install.pl


Run it, and you’ll see this.

When you press [Enter], you’ll see this.

In step 0, no user input is required because the install script will automatically detect what your OS is and where you have your Web server installed. Step 1 asks for the location of your Web server’s document directory. In my installation of Apache, it’s /usr/local/apache/htdocs. Along the same lines, in step 2, you’re asked for your Web server’s cgi-bin directory.

In the MRTG installation that I walked through in my previous article, I used the MRTG user’s home directory as the location for scripts (step 3). This directory is /home/mrtg. Earlier in this article, I copied my zyxel.cfg configuration file to rrdzyxel.cfg. I’ll continue to use this naming convention, so I’ll tell the RRDtool installer that any configuration file starting with rrd is fair game (step 4).

In step 5, I plan to keep my RRD files under the Apache directory to prevent permission problems. Where is Perl? Step 6 shows that I’m configuring it for /usr/bin/perl on my system. Step 7 allows me to tell the system where I would like to have the configuration file for this program stored.

The following section will ask a rather confusing question.

Since this is very poorly documented and described, we’ll use mixed; it’s the default, and I don’t have a better reason than that.
INSTALLING SOFTWARE

We’re almost done.
Perl is     : /usr/bin/perl
MRTG files  : /usr/local/mrtg/config/*.cfg


** ALL COMPLETE **

And finally, a bit of installation information is shown:
The routers cgi files are now installed and ready to use!

Let’s see what we have
You should wait 10 or 15 minutes to allow MRTG and RRDtool to actually gather some statistics. Also, if you’re converting from an MRTG installation that has a large amount of data, you may need to wait while RRDtool converts it.

Using a java-enabled browser that allows cookies, browse to http://<your_server>/cgi-bin/routers.cgi (where your_server is the actual address of the server being used). When you get there, you’ll get the introduction screen for the Router Monitor, as shown in Figure A.

Figure A
The Router Monitor initial screen is showing very little traffic.


On the left side of the screen, you’ll see an Options button and a list of configuration files under Routers. For this example, I only have one file, rrdzyxel.cfg.

If you click on Options, you’ll be presented with a whole host of options with different ways to look at your data. You’ll notice that the graphs are slightly different than the ones produced with the MRTG rateup tool. By default, the RRDtool utility puts a red line at the top of the initial graphs indicating the maximum speed for that interface.

Let’s look at a specific difference. Want to see a graph that has a line for each and every interface in your router, along with maximum, average, and current statistics, but only for outgoing data? Okay. Click on Options | Outgoing and take a look at the results.

Figure B
Graphing router traffic for all ports


In Figure B, you can see some traffic for two ports on my DSL router and statistics for these ports. The remaining ports aren’t active.

We’re finished with the first configuration file. Now it’s time to do the rest. I recommend copying all of your configuration files to a new location and then rerunning the install.pl script that came with the routers.cgi package, so that they’re all converted and available at once. Don’t forget to modify the copied configuration files before you rerun the install.pl scripts. Each one needs to have the three lines at the top of the script modified to add path information and library location and to be told to use RRD rather than rateup.

An actual use of RRDtool from the Web
RRDtool can measure just about anything. While graphs in MRTG can be slightly customized, that’s nothing compared to what can be done with RRDtool. As an example of just how customized a graph in RRDtool can be, take a look at Figure C, which is from the RRDtool sample image gallery showing line usage at an ISP. Included in this graph are lines for the points at which the ISP imposes additional fees. The user of this graph created it to be able to verify and estimate the bill from his ISP. The graph was created with a fairly complex script, which I’ll also show you.

Figure C
A typical ISP billing graph from RRDtool


As you can see, the graph has been heavily customized to fit the needs of this user and contains quite a lot of information in a surprisingly easy-to-read format. This script generated the graph. It’s available in the public domain and was written by Alex van den Bogaerdt, who has also authored at least two tutorials on the use of RRDtool.

Conclusion
Switching from using the rateup tool to RRDtool databases in MRTG serves a number of purposes, including better speed, better graphs, and the use of a number of tools, such as the routers.cgi tool, that create a full monitoring environment.

I like the RRDtool approach because it makes the network-monitoring tool more componentized. I can use any utility I want to get the data and anything that I want to view it.

If you look around the Net a bit, you’ll also see that RRDtool is used by folks to track the space shuttle, look at the weather, and more. It’s useful for much more than gathering network statistics.

I recommend that you look around the Web for other uses of RRDtool or learn to create your own RRDtool applications.

I hope that this Daily Drill Down was helpful in exposing you to RRDtool and how it can be used to interact with MRTG.

Need more?
Would you like to see more content like this? E-mail Jack Wallen, Jr. and let him know.

 

Editor's Picks

Free Newsletters, In your Inbox