Flex the power of cURL for web dev testing and data transfers

cURL allows you to test web services and applications, transfer data via various protocols, and scrape and crawl screens. Tony Patton describes how to use this great open source utility.


Image: iStock/nico_blue

Testing web services and even web applications at a granular level is cumbersome process, especially when you need to examine the nitty-gritty details of the data going back and forth. There are a number of great tools available (Chrome, IE, and Firefox browser extensions) with the cURL command-line (and programmable) option being an elder statesman. cURL provides a simple way to transfer data via various protocols. 

Awaiting your command

Like many great utilities, cURL has its roots in the UNIX world, but it has been ported to most operating systems. The download page provides packages for most flavors, with the download wizard providing a guide as well. The source code is available to compile it yourself. I have used it on OS X, but for this article, the Windows option is used.

I used the download wizard to install it on my Windows 7 development machine. You choose curl executable, select the appropriate operating system from the dropdown list, and then follow the prompts for your environment. The Windows installation places the cURL directory in the system path (C:\Program Files\cURL\bin) so it may be used from a standard command line without any additional setup. Figure A shows it run (with the help switch) from a Windows command prompt.

Figure A


The cURL utility run with the help switch from Windows command prompt.

So many options, so little time

cURL allows you to send and receive data via a variety of protocols. The HTTP and HTTPS protocols are the focus of the examples in this article, though the cURL man page lists a wide range of protocols supported by the utility. Using the command line includes the command name followed by any switches and an address. As an example, the following command returns only information (the –I option) about the TechRepublic.com site using HTTP 1.1:

curl ---http1.1 -I http://www.techrepublic.com

It returns the following data:

HTTP/1.1 200 OK

Content-Type: text/html; charset=UTF-8

Server: nginx

X-Tx-Id: bd242125-8cbf-471c-a8a6-8e369558908c

Expires: Thu, 13 Feb 2014 03:02:17 GMT

Cache-Control: max-age=0, no-cache, no-store

Pragma: no-cache

Date: Thu, 13 Feb 2014 03:02:17 GMT

Connection: keep-alive

Set-Cookie: fly_edition=us; expires=Thu, 20-Feb-2014 03:02:17 GMT

Set-Cookie: fly_geo={"countryCode": "us"}; expires=Thu, 20-Feb-2014 03:02:17 GMT

Set-Cookie: fly_device=desktop; expires=Thu, 20-Feb-2014 03:02:17 GMT

You can pass parameters to the request with the d command line switch as well. The following example returns the results of searching for myself on the TechRepublic.com site. Note, it uses the verbose (v) option as well, which basically means everything is displayed.

curl –v –d “q=tony+patton” http://www.techrepublic.com

These interesting examples are only the tip of the cURL iceberg. When using web calls like the examples, you can utilize a number of options, including using any HTTP method (PUT, GET, POST) along with defining the content type, character set, and so forth. The following command sets the content type to xml and passes one parameter to a local ASP.NET web service on my development machine.

Curl –H 'Content-Type: application/xml; charset=utf-8' –d 'value=test' http://localhost/service.asmx

As you can see in Figure A, there are a variety of options available in cURL.

Beyond the command line

The command line options covered so far are powerful; however, you are not restricted to this interface when using cURL. You can use the command line in your scripts – UNIX shell, PowerShell, etc. The cURL functionality may be used in your applications as well. At the core of cURL is the libcurl library (written in C).

The libcurl library has been ported to almost any programming language imaginable. The bindings section of the cURL site provides a complete list of what is available. A few examples include .NET, Python, and Perl. It is a standard part of PHP. These libraries allow you to utilize cURL functionality within your code. The libcurl library is very popular--examples include the Tornado web server used by Facebook, Yahoo.com site, and its shipping with the Apple iOS. Since it is open source, you can contribute to its evolution if you desire.

Testing web services and applications is just one use of cURL; it can also be used to transfer data via a variety of protocols and for scraping and crawling activities. With its simple syntax and programming language offerings, cURL is ideal for screen scraping activities. It would be interesting to see how many Internet bots use the libcurl library.

Another tool at your disposal

I have only used cURL for interacting with websites and services. The ability to quickly and easily call services and return only the header and response information is vital when debugging. I know this functionality is readily available in other tools such as Fiddler, but the command line interface is clean and simple, and some of us just prefer the command line.

The amount of information on the web about cURL (and libcurl) is a bit overwhelming, but the breadth means you will likely find something to fit your needs. Its mailing lists are also excellent resources.



By Tony Patton

Tony Patton has worn many hats over his 15+ years in the IT industry while witnessing many technologies come and go. He currently focuses on .NET and Web Development while trying to grasp the many facets of supporting such technologies in a productio...