Developer

Pre-fetching Web sites with Border Manager 3

Ron Nutter explores the pre-fetching feature of Border Manager 3 and explains how you can use it to download all or part of a Web site.


No matter how much bandwidth you have, there never seems to be enough. If you’re in a disaster-recovery situation, your backup connection is probably just a fraction of what you normally have. One way to control how much of your bandwidth is used for viewing Web sites is to use the pre-fetching feature of Border Manager 3. This feature allows you, at specified intervals, to go out and download part or all of a Web site. Then, when your users access a site that you have pre-fetched, they actually call it out of the Border Manager cache and don’t have to use your Internet connection. There’s only one thing that Border Manager can’t pre-fetch from a Web site: content pages that are dynamically created. (A dynamically created page will be different for each user, each time the page is accessed.) In this article, we’ll take you through the process of setting up the pre-fetching feature of Border Manager 3.

Deciding what you need to pre-fetch
By viewing the access control logs in NWAdmin, you can get a good feel for which sites your users are visiting and how often. By determining how often a particular site is visited, you can adjust how many sites you need to pre-fetch. Check the amount of information coming from the cache versus what is coming from the site, and you’ll know when you need to make adjustments. These adjustments will help control how much space on the cache volume is consumed by pre-fetched Web sites.

You can see how much information will be cached by doing an on-demand cache. Using Rconsole.exe, establish a remote console session to the server running Border Manager 3. Press [Alt][F1], select the Select A Screen To View option, and press [Enter]. Highlight the Proxy console screen title and press [Enter] again. Option 22 on the Proxy console screen will allow you to take a one-time capture of a Web site. Select 22 and press [Enter].

At this point, enter the URL for the Web site and press [Enter]. You’ll probably see a default value of http://www.novell.com. Be sure to precede the name of the Web site you are testing with http://, or the pre-fetch process will fail. The next question asks you how many links you want to follow. The default value of 1 is good to start with. If you use a value of 0, you will find that you get very little of the site. Take a site such as www.usatoday.com. If you find that users are continually going past the front page, you may need to increase the value to 2 or higher. As we mentioned earlier, look at your access control logs and other reporting functions in NWAdmin to see how much is being read from the cache. Don’t try to get a 100 percent cache hit rate for a given Web site. Either you won’t have enough disk space to hold the entire Web site in question, or your bandwidth will be consumed with caching the Web site.

The next question asks if you want to follow links to other hosts. I have found such links generally are tied to click-tracking or banner-providing companies. I have yet to find a reason that I would want to follow links to other sites, but you do have the option.

The next prompt is short and sweet. You are asked Verbose Y/N—this refers to how much information is displayed on the Novell Border Manager Proxy Cache Server screen while the site is being cached. While checking to see how much of a site would be downloaded during this part of the process, I chose Yes to see what would be downloaded when a permanent schedule was created. Taking into account how much a site changes or how quickly you need to get it on your system, you can specify how much processor power is given to the task. By changing the value of the Maximum Concurrent Download Requests setting, you can either speed up the retrieval process or slow it down if you don’t want Border Manager to slow down during the pre-fetching process. The default value for this option is 4, with a range of choices going from 1 to 20. You should stay with 4 until you get more comfortable with pre-fetching sites.

You can further tweak the pre-fetching process by limiting the number of objects (for example, Web graphics) that will be captured and how much disk space will be used to hold a particular site. Once you have entered all the requested information, you can see the progress of the site download for the Web location you are testing by switching to the Proxy Cache Server screen.

Setting a schedule for pre-fetching a site
At this point, you’ve decided which sites—and how much of each one—you want to pre-fetch for your users. The next step is to set a schedule for pre-fetching sites. Begin by going into NWAdmin and double-clicking the NDS server object that Border Manager is running on. Then, click the Border Manager Setup button and, on the next screen, click the Acceleration tab. Click the Caching button, then click the Scheduled Download tab.

On the next screen, click on the dotted square box to add a site to the download list. Make sure the Enable This Particular Download option is selected; otherwise, the pre-fetching process for the site you are configuring will never happen. Enter the complete URL for this site. For this option, you won’t have to include http:// as you did with Rconsole.exe; NWAdmin will add it automatically when you save the site download configuration. Enter the number of levels to download, just as you did when you tested the pre-fetching process for the site.

Unless you have a reason to adjust the remaining values, you can set the frequency of the site-fetching process. When you click the Frequency tab, you will be presented with three options for downloads: One Time Only, Once A Day, and Daily. The One Time Only option is the same as selecting Option 22 on the Proxy console screen. For sites that aren’t accessed heavily or don’t change substantially within a day’s time, you can use the Once A Day option. For sites such as www.usatoday.com that are updated several times a day, you will probably want to go with the Daily option. When you select that option, you can control at which times during the day and at what interval the site is fetched, and on what days of the week this schedule will be active. Once you complete this screen, click OK to submit the process to the proxy cache engine.

If you still have the RCONSOLE session to the server, check the server command console screen to see when the proxy NLM has reloaded with the updated configuration. The download process for the Web site you have just entered should begin in a moment. You can see which sites are currently set for download by going to the NetWare Proxy Cache Configuration Console screen, selecting 15, and pressing [Enter]. A list of currently configured sites will appear with the options you have entered. You can also view the same list in NWAdmin.

Conclusion
In this article, we walked you through the process of pre-fetching Web sites for your users. Use this function to get information to your users faster and to help decrease the load on your Internet connection.

Ronald Nutter is a senior systems engineer in Lexington, KY. He's an MCSE, Novell Master CNE, and Compaq ASE. Ron has worked with networks ranging in size from single servers to multiserver/multi-OS setups, including NetWare, Windows NT, AS/400, 3090, and UNIX. He's also the help desk editor for Network World. If you’d like to contact Ron, send him an e-mail . (Because of the large volume of e-mail that he receives, it's impossible for him to respond to every message. However, he does read them all.)

The authors and editors have taken care in preparation of the content contained herein, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

Editor's Picks

Free Newsletters, In your Inbox