Testing web
services and even web applications at a granular level is cumbersome process, especially when you need to examine the nitty-gritty details of
the data going back and forth. There are a number of great tools available (Chrome,
IE,
and Firefox
browser extensions) with the cURL command-line (and programmable) option being an
elder statesman. cURL provides a simple way to
transfer data via various protocols. 

Awaiting
your command

Like many
great utilities, cURL has its roots in the UNIX world, but it has been ported
to most operating systems. The download page provides packages for most flavors, with the download wizard providing
a guide as well. The source code is available to compile it yourself. I have used
it on OS X, but for this article, the Windows option is used.

I used the download wizard to install it on my
Windows 7 development machine. You choose curl executable, select the
appropriate operating system from the dropdown list, and then follow the prompts for
your environment. The Windows installation places the cURL directory in the
system path (C:\Program Files\cURL\bin) so it may be used from a standard
command line without any additional setup. Figure A shows it run (with the help
switch) from a Windows command prompt.

Figure A

 

The cURL utility run with the help switch from Windows command prompt.

So many
options, so little time

cURL allows you to send and receive data via a variety of
protocols. The HTTP and HTTPS protocols are the focus of the examples in this
article, though the cURL man page lists a wide range of protocols supported by the utility. Using the
command line includes the command name followed by any switches and an address.
As an example, the following command returns only information (the –I option)
about the TechRepublic.com site using HTTP 1.1:

curl
—http1.1 -I http://www.techrepublic.com

It returns
the following data:

HTTP/1.1 200
OK

Content-Type:
text/html; charset=UTF-8

Server:
nginx

X-Tx-Id:
bd242125-8cbf-471c-a8a6-8e369558908c

Expires:
Thu, 13 Feb 2014 03:02:17 GMT

Cache-Control:
max-age=0, no-cache, no-store

Pragma:
no-cache

Date: Thu,
13 Feb 2014 03:02:17 GMT

Connection:
keep-alive

Set-Cookie:
fly_edition=us; expires=Thu, 20-Feb-2014 03:02:17 GMT

Set-Cookie:
fly_geo={“countryCode”: “us”}; expires=Thu, 20-Feb-2014
03:02:17 GMT

Set-Cookie:
fly_device=desktop; expires=Thu, 20-Feb-2014 03:02:17 GMT

You can pass
parameters to the request with the d command line switch as well. The following
example returns the results of searching for myself on the TechRepublic.com
site. Note, it uses the verbose (v) option as well, which basically means
everything is displayed.

curl –v –d
“q=tony+patton” http://www.techrepublic.com

These
interesting examples are only the tip of the cURL iceberg. When
using web calls like the examples, you can utilize a number of options, including using any HTTP method (PUT, GET, POST) along with defining the
content type, character set, and so forth. The following command sets the
content type to xml and passes one parameter to a local ASP.NET web service on
my development machine.

Curl –H ‘Content-Type: application/xml; charset=utf-8’ –d ‘value=test’ http://localhost/service.asmx

As you can
see in Figure A, there are a variety of options available in cURL.

Beyond the
command line

The command
line options covered so far are powerful; however, you are not restricted to this
interface when using cURL. You can use the command line in your scripts – UNIX
shell, PowerShell, etc. The cURL functionality may be used in your applications
as well. At the core of cURL is the libcurl library (written in C).

The libcurl
library has been ported to almost any programming language imaginable. The bindings section of the cURL site provides a complete list of what is available. A few examples include
.NET, Python, and Perl. It is a standard part of PHP. These libraries allow you to
utilize cURL functionality within your code. The libcurl library is very
popular–examples include the Tornado web server used by Facebook,
Yahoo.com site, and its shipping with the Apple iOS. Since it is open source,
you can contribute to its evolution if
you desire.

Testing web
services and applications is just one use of cURL; it can also be used to transfer data via a variety of protocols and for scraping and crawling
activities. With its simple syntax and programming language offerings, cURL is
ideal for screen scraping activities. It would be interesting to see how many
Internet
bots use the libcurl library.

Another tool
at your disposal

I
have only used cURL for interacting with websites and services. The ability to quickly and
easily call services and return only the header and response information is
vital when debugging. I know this functionality is readily available in other
tools such as
Fiddler, but the command line interface is clean and simple, and some of us
just prefer the command line.

The amount of information on the web about cURL (and libcurl) is a bit overwhelming, but the breadth means you will likely find
something to fit your needs. Its mailing lists are also excellent resources.