Software Development

Download files over the Web with .NET's WebClient class

The System.Net namespace includes the WebClient class for uploading and downloading files via HTTP. You can copy or read files with only a few lines of code. Tony Patton details how the WebClient class helped him solve a recent problem.

 

A recent project involved moving data from one Web site to another site while keeping the data synchronized. One requirement was to move image files from one site to the other. I suggested keeping the images on one site and using its URL on the second site, but the client wanted the actual files moved. I anticipated a large effort to move the files, but the .NET Framework provides a simple solution via the WebClient class contained in the System.Net namespace.

Downloading data

The size of the .NET Framework makes it difficult to know everything it offers without a little research. The WebClient class provides methods for sending and receiving data to and from local, intranet, or Internet resources. The resource is identified via its Web address. This article will focus on the methods available for downloading data.

WebClient class

The WebClient class is available regardless of the .NET Framework version. The WebClient class provides numerous methods for working with data. The following list is a review of the methods available for downloading data.

  • OpenRead: Resource data is returned as a Stream object.
  • OpenReadAsync: Resource data is returned asynchronously as a Stream object.
  • DownloadData: Resource data is returned as a Byte array.
  • DownloadDataAsync: Resource data is returned asynchronously as a Byte array.
  • DownloadFile: The resource is downloaded to a local file.
  • DownloadFileAsync: The resource is downloaded asynchronously to a local file.
  • DownloadString: A String resource is downloaded to a String object.
  • DownloadStringAsync: A String resource is downloaded asynchronously to a String object.

Some people are surprised that you can readily retrieve resources from a Web application. They tend to forget that you can easily access a site and save images and other resources locally (although their use may be impeded by copyright).

Copying a file

The WebClient methods in the list are simple and easy to use. The next Windows C# console application pulls the search button image from the TechRepublic.com home page and saves them to a local C drive. It checks to see if the file exists before copying it.

using System;

using System.Net;

namespace TechRepublicExamples {

class Program {

static void Main(string[] args) {

string imageAddress = String.Empty;

string filename = String.Empty;

try {

WebClient fileReader = new WebClient();

imageAddress = "http://b2b.cbsimg.net/images/200609/go_search_button.gif";

filename = imageAddress.Substring(imageAddress.LastIndexOf("/"), imageAddress.Length);

if (!(System.IO.File.Exists("c:\" + filename)))

fileReader.DownloadFile("", "c:\" + filename);

} catch (HttpListenerException ex) {

Console.WriteLine("Error accessing " + imageAddress + " - " + ex.Message);

} catch (Exception ex) {

Console.WriteLine("Error accessing " + imageAddress + " - " + ex.Message);

} } } }

The equivalent VB.NET follows:

Imports System.Net

Module Module1

Sub Main()

Dim imageAddress As String = String.Empty

Dim filename As String = String.Empty

Try

Dim fileReader As New WebClient()

imageAddress = http://b2b.cbsimg.net/images/200609/go_search_button.gif

filename = imageAddress.Substring(imageAddress.LastIndexOf("/") + 1)

If Not (System.IO.File.Exists("c:\" + filename)) Then

fileReader.DownloadFile(imageAddress, "c:\" + filename)

End If

Catch ex As HttpListenerException

Console.WriteLine("Error accessing " + imageAddress + " - " + ex.Message)

Catch ex As Exception

Console.WriteLine("Error accessing " + imageAddress + " - " + ex.Message)

End Try

End Sub

End Module

The previous example utilizes a try/catch block to handle any problems with downloading and copying the file locally. The System.Net namespace provides the following Exception classes:

  • HttpListenerException: Represents exceptions encountered when working with HTTP.
  • ProtocolViolationException: Represents exceptions encountered when working with network protocols.
  • WebException: Represents exceptions encountered while accessing the Internet through a pluggable protocol.

You can handle these exceptions accordingly when working with objects within the System.Net namespace.

Working with text files

While downloading an entire file may be good for images, you may wish to access a Web page as text. This is often called screen scraping. The OpenRead method of the WebClient class allows you to access the contents of a file via a Stream object. You may choose to save it to a file or work with its contents according to requirements.

The following example accesses the TechRepublic home page as text and displays each line one at a time. The code skips empty lines from the page. The code also introduces the approach to adding HTTP headers to the file request. By default, accessing files via the WebClient class does not include any HTTP headers; however, some sites may require headers so a request identifies itself. The Headers property of the WebClient class includes the Add method for adding header information to the request.

using System;

using System.Net;

using System.IO;

namespace TechRepublic {

class Program {

static void Main(string[] args) {

try {

WebClient fileReader = new WebClient();

fileReader.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");

Stream data = fileReader.OpenRead("http://www.techrepublic.com/");

StreamReader sr = new StreamReader(data);

string currentLine;

do {

currentLine = sr.ReadLine();

if (!(currentLine.Trim().Equals("")))

Console.WriteLine(currentLine);

}

while (currentLine != null);

data.Close();

} catch (WebException ex) {

Console.WriteLine("Error accessing site " + ex.Message);

} catch (Exception ex) {

Console.WriteLine("Error accessing site " + ex.Message);

} } } }

The equivalent VB.NET follows:

Imports System.Net

Imports System.IO

Module Module1

Sub Main()

Try

Dim fileReader As New WebClient()
fileReader.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)")

Dim data As Stream

data = fileReader.OpenRead("http://www.techrepublic.com/")

Dim sr As StreamReader

sr = New StreamReader(data)

Dim currentLine As String

currentLine = sr.ReadLine()

While Not (currentLine Is Nothing)

If Not (currentLine.Trim().Equals("")) Then

Console.WriteLine(currentLine)

End If

currentLine = sr.ReadLine()

End While

data.Close()

Catch ex As HttpListenerException

Console.WriteLine("Error accessing site " + ex.Message)

Catch ex As Exception

Console.WriteLine("Error accessing site " + ex.Message)

End Try

End Sub

End Module

It is worth noting that the other approach to accessing Web data involves a byte array. The following example code illustrates it being used to display the contents of a Web page. The Encoding class located in the System.Text namespace is used to set the encoding used by the page data via the Byte array object.

using System;

using System.Net;

using System.IO;

namespace TechRepublic {

class Program {

static void Main(string[] args){

try {

WebClient fileReader = new WebClient();

byte[] byteArray = null;

Uri addr = new Uri("http://www.techrepublic.com/");

byteArray = fileReader.DownloadData(addr);

string content = System.Text.Encoding.ASCII.GetString(byteArray);

Console.WriteLine(content);

} catch (Exception ex) {

Console.WriteLine("Error accessing " + imageAddress + " - " + ex.Message);

} } } }

The equivalent VB.NET follows.

Imports System.Net

Imports System.IO

Module Module1

Sub Main()

Try

Dim fileReader As New WebClient()

Dim byteArray As Byte()

Dim addr As Uri

Dim content As String

addr = New Uri("http://www.techrepublic.com/")

byteArray = fileReader.DownloadData(addr)

content = System.Text.Encoding.ASCII.GetString(byteArray)

Console.WriteLine(content)

Catch ex As HttpListenerException

Console.WriteLine("Error accessing site " + ex.Message)

Catch ex As Exception

Console.WriteLine("Error accessing site " + ex.Message)

End Try

End Sub

End Module

Working with Web data

The .NET Framework is massive; it has plenty of features to address almost any situation that may arise. The System.Net namespace includes the WebClient class for uploading and downloading files via HTTP. You can copy or read files with only a few lines of code.

What facets of the .NET Framework have you recently discovered or rediscovered while working on a project? Share you thoughts and experience with the Web Developer community.

Tony Patton began his professional career as an application developer earning Java, VB, Lotus, and XML certifications to bolster his knowledge.

---------------------------------------------------------------------------------------

Get weekly development tips in your inbox Keep your developer skills sharp by signing up for TechRepublic's free Web Developer newsletter, delivered each Tuesday. Automatically subscribe today!

About

Tony Patton has worn many hats over his 15+ years in the IT industry while witnessing many technologies come and go. He currently focuses on .NET and Web Development while trying to grasp the many facets of supporting such technologies in a productio...

3 comments
hushcat
hushcat

I am happy when reading your blog with updated information! thanks alot Plaster Coving

nikeeta
nikeeta

thank u very much... it is vary helpfull article.. there was one question that... when we use download data() of webclient , it returns the data in the form of bytes... then this bytes we do response.binarywrite() such that it opens dialog box to "save", "open" , "cancel" is there any way that we can trace programatically, that user has actully saved file using save button, not clicked on cancel button, is there any way to trace it programatically ? thank u.. any help on this will be appriciated...

Justin James
Justin James

... I just started using this class a little while ago for a project that I am working on. One thing that should be clarified, is that even if you use the DownloadFile() method, you can still access the HttpResponse object that was created to check things like the status code, content type, etc. J.Ja