Cloud

Increase download speed with Aria2 utility

Vincent Danen recommends the multisource, multiprotocol, and cross-platform Aria2 utility to drastically increase download speeds.

Traditional file downloads are done via an FTP client, a Web browser, or a file-sharing program such as BitTorrent. These means of downloading files will probably never change, but the tools to do the downloading, and how they download, will.

One program that changes how files are downloaded is Aria2. Aria2 is a lightweight multiprotocol and multisource download utility that offers some very compelling reasons for its use over traditional "single thread" downloading such as with an FTP or HTTP client.

Being a multiprotocol client, Aria2 can download via HTTP, via FTP, and via BitTorrent. What makes Aria2 unique is that it can download the same file at the same time utilizing different protocols. It can also open multiple connections to the same, or different, URLs to download the file even faster. By default, Aria2 uses five connections to download a file.

To download a file, use:

$ aria2c http://somehost.com/file.zip

If the same file is available from two different places, you can maximize your bandwidth and download it from both, even using different protocols:

$ aria2c http://somehost.com/file.zip ftp://otherhost.com/pub/file.zip

To compare speeds, I used curl to download a bzip2 compressed ISO image that weighs in at 200MB:

$ time curl -OL ftp://ftp.ibiblio.org:21//pub/linux/distributions/annvix/releases/ISO/
annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  200M  100  200M    0     0   108k      0  0:31:27  0:31:27 --:--:--  104k
curl: (28) FTP response timeout
curl -OL   0.57s user 2.04s system 0% cpu 31:27.27 total

With Aria2, I downloaded the same file from three different places together:

$ time aria2c ftp://ftp.ibiblio.org:21//pub/linux/distributions/annvix/releases/ISO/

annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2 http://gd.tuwien.ac.at/opsys/linux/annvix/

releases/ISO/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2 http://ftp.cc.uoc.gr/mirrors/

linux/annvix/releases/ISO/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
 *** Download Progress Summary as of Thu May 13 17:28:12 2010 ***
=============================================================================================
[#1 SIZE:88.2MiB/200.4MiB(43%) CN:5 SPD:2.0MiBs ETA:55s]
FILE: /home/user/tmp/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
---------------------------------------------------------------------------------------------
 *** Download Progress Summary as of Thu May 13 17:29:12 2010 ***
=============================================================================================
[#1 SIZE:167.6MiB/200.4MiB(83%) CN:5 SPD:559.9KiBs ETA:59s]
FILE: /home/user/tmp/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
----------------------------------------------------------------------------------------------
[#1 SIZE:200.3MiB/200.4MiB(99%) CN:5 SPD:192.3KiBs]
2010-05-13 17:29:57.277148 NOTICE - Download complete: /home/user/tmp/
annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK|   1.2MiB/s|/home/user/tmp/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
Status Legend:
 (OK):download completed.
aria2c     0.53s user 4.62s system 3% cpu 2:45.08 total

What an amazing difference in speed! To verify that both files downloaded are identical, I ran md5sum against both files:

$ md5sum o/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
a7d1e84533a27daf8fe794256ca1f928  o/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
a7d1e84533a27daf8fe794256ca1f928  annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
% ls -al o/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
-rw-r--r-- 1 user user 210219877 May 13 17:29 annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2
-rw-r--r-- 1 user user 210219877 May 13 17:25 o/annvix-netinstall-x86_64-3.0-RELEASE-r2.iso.tar.bz2

The files are indeed identical.

What if you are at work when the new release of your favourite Linux distribution becomes available? Instead of using BitTorrent at work which may go against company usage policies, if you are able to SSH to your home system, you can launch Aria2 under screen and begin the download from there. If you want to download a number of files at once, you can have Aria2 download them sequentially or in parallel; to download files in parallel use:

$ aria2c -Z http://somehost.com/file.zip linuxdistro.torrent

Aria2 allows you to specify the maximum upload rate when downloading torrents. This can be done with the "-u" option, such as "-u40K" would only allow an upload speed of 40KB/s. In fact, Aria2 is a full-featured BitTorrent client with a lot of features such as support for seeding files, downloading files with encryption, and DHT (Distributed Hash Table) support.

While some might balk at the use of a command-line client to download files, perhaps cringing at the remembrance of old-school FTP clients, Aria2 is easy to use and easy to script.

The speed gains of Aria2 are incredible. If BitTorrent isn't your thing, imagine finding five different mirrors of your favourite Linux distribution and passing the URLs to Aria2. Possibly the longest part of that would be finding the URLs and pasting it to the command-line, rather than the actual downloading.

Get the PDF version of this tip here.

About

Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.

4 comments
Orodreth
Orodreth

Thanks for the information. I never looked into Aria2 as a download client but now I'm eager to try it.

Ajax4Hire
Ajax4Hire

Yet Another downloader. curl, wget, ftp, http now aria2. taxes the limit of my ability to remember. It may be great but is it worth it?

vdanen
vdanen

ftp and http are protocols, not download programs (well, I suppose someone might have named their programs that way, but...) wget and curl do a single download from a single source (or recursive downloading of a single source). Aria2 is unique in how it grabs from multiple sources/protocols and (successfully!) munges that data together. It's really quite slick and the download speeds should indicate that, yeah, it's worth it. =) At least worth looking into, if nothing else.

Editor's Picks