Linux

Create a local Fedora mirror system and get blazing-fast updates

Vincent Danen shows you how to set up a local mirror for Fedora systems that will get you faster updates and conserve bandwidth.

One of the things that I like most about Fedora, is how simple it makes managing infrastructure. While many Linux distributions have multiple mirrors, if you run a lot of instances of that distribution, it's nice to have a local mirror. For primary mirrors, a lot of vendors have some "magic goo" to make things work for them, but if you're an end user wanting to mirror the bits locally, and use them, you're often left to your own devices.

Fedora makes it easy to run a public mirror, and a local private one, and gives you the means to do so easily, without clumsy reconfiguration of all your local systems. Using the Fedora Mirror Manager, you can set up a private mirror easily and have it transparently, and without any reconfiguration necessary, be seen and used by local Fedora systems.

You must have a FAS (Fedora Account System) account in order to use Mirror Manager, which is a free account that you can create. You should also subscribe to the mirror-list mailing list to be notified of new releases to properly manage your mirror. Once you have an FAS account set up, log into the Mirror Manager using your FAS account.

Here you will need to create a new Site, which we will call School. The Site Name will be 'School', and the password can be whatever you like. If you have a URL for your organization, specify it in the Organization URL field. Since this is a private mirror only to be used internally, select the Private checkbox and then save the site.

Next, create a Host. The Host Name should be the FQDN of the internal server providing the mirror; perhaps mirror.school.org. Set the Country code (i.e., US if in the USA, CA for Canada, etc.) and again make sure the Private checkbox is enabled. Save the Host. Once it is saved, add a new Site-local Netblock. If you are behind a firewall or have one (or more) public IP addresses, this is where they will be listed. For instance, if your public IP address is 1.2.3.4, you would use the Netblock 1.2.3.4/32.

Once this is saved, still under the Host setting, add a new Category. This will tell the Mirror Manager what categories of software this host carries. Examples include Fedora Linux, Fedora EPEL, and so on. If you intend on carrying everything recent, use Fedora Linux. You will also need to add a URL serving the content definition, such as http://mirror.school.org/Fedora.

Now, on the host system, install the mirrormanager and mirrormanager-client packages. The mirror host will also need to be a Fedora system.

# yum install mirrormanager mirrormanager-client

If Apache is not installed, install it. This example will set up an HTTP-based mirror, which is probably the easiest to set up. Edit /etc/httpd/conf/httpd.conf, and at the end add something similar to the following:

<VirtualHost mirror.school.org>
  DocumentRoot /srv/mirror
  KeepAlive On
  KeepAliveTimeout 2
  MaxKeepAliveRequests 100
  AddType application/octet-stream .iso
  AddType application/octet-stream .rpm
  <Directory "/srv/mirror">
    Options +Indexes +FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

Suit to taste, of course. If mirror.school.org were the primary hostname of the system, and you served no other HTTP content, you could forego the use of the VirtualHost directives and plug it directly into Apache's primary host configuration.

Enable and start Apache, and visit http://mirror.school.org/ and make sure it is listing content.

# chkconfig httpd on; service httpd start

Next, edit /etc/mirrormanager-client/report_mirror.conf. Many of the fields you need to change here must reflect what you defined in the Mirror Manager web UI. The file is heavily commented, so it is easy enough to configure. Building on our example above, we would have, in the content categories, the following:

 [Fedora Linux]
enabled=1
path=/srv/mirror/Fedora

Once this is done, we can create the script to create our mirror. The mirror will take a lot of space, and will take quite a while to set up initially. Create /srv/mirror/sync with the following contents (suit to taste; in the below snippet I am mirroring Fedora 13 and 14 only, as well EPEL4 and EPEL5):

#!/bin/sh
fdest="/srv/mirror/Fedora"
edest="/srv/mirror/EPEL"
lock=".rsync_updates.lock"
options="${1}"
if [ -f ${lock} ]; then
    echo "Updates via rsync already running."
    exit 0
fi
for version in 13 14; do
    if [ -d ${fdest}/releases/${version}/Everything ]; then
        echo "Synchronizing Fedora ${version}"
        pushd ${fdest}/releases/${version} >/dev/null 2>&1
            rsync -avH rsync://fedora.mirror.iweb.ca:/fedora/releases/${version}/Everything . --exclude-from=/srv/mirror/fedora-excludes.txt ${options} \
                --numeric-ids --delete --delete-after --delay-updates
        popd >/dev/null 2>&1
        if [ "${version}" == "14" ]; then
        echo "Synchronizing Fedora updates for version ${version}"
        pushd ${fdest}/updates/${version} >/dev/null 2>&1
            rsync -avH rsync://fedora.mirror.iweb.ca:/fedora/updates/${version}/ . --exclude-from=/srv/mirror/fedora-excludes.txt ${options} \
                --numeric-ids --delete --delete-after --delay-updates
        popd >/dev/null 2>&1
        fi
    else
        echo "Target directory ${fdest}/${releases}/${version}/ not present."
    fi
done
echo "Synchronizing EPEL"
pushd ${edest} >/dev/null 2>&1
    rsync -avH rsync://linux.mirrors.es.net/fedora-epel/ . --exclude-from=/srv/mirror/epel-excludes.txt ${options} \
        --numeric-ids --delete --delete-after --delay-updates
popd >/dev/null 2>&1
/usr/bin/report_mirror
/bin/rm -f ${lock}

Symlink the script in /etc/cron.daily/ to have the script run every day and you should be up and running:

# cd /etc/cron.daily/
# ln -s /srv/mirror/sync mirror-sync

Be sure that the report_mirror command is run after every rsync so that Mirror Manager knows what you have and that it is up-to-date. Also note that it may take a bit of time initially before your local mirror shows up in the mirror list. A quick and easy way to see whether the mirror is up and running is to load up the mirror XML data file with curl and look for a private mirror:

% curl -s "https://mirrors.fedoraproject.org/metalink?repo=fedora-14&arch=i386" | grep private
        <url protocol="http" type="http" location="CA" preference="100" mm0:private="True">http://mirror.school.org/Fedora/releases/14/Everything/i386/os/repodata/repomd.xml</url>

If you get output similar to the above, your local Fedora boxes should start using your mirror. You may have to clean the caches on the client Fedora boxes first (I had to do yum clean all; yum makecache; yum update before I saw the updates being pulled from my mirror).

Having your own local Fedora repository is great if you manage a lot of Fedora machines in your local network. You can fine-tune what you want; if you only want to host a local mirror of EPEL for local RHEL or CentOS machines, you can do that too. Having this setup will allow for blazing fast updates compared to downloading from the public mirrors, and can also conserve on bandwidth. Having this in place will also make pre-upgrades of Fedora systems run much faster as well.

Get the PDF version of this tip here.

About

Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.

0 comments