Date Added: Sep 2012
The Web, the largest unstructured database of the world has greatly improved access to the documents. As the number of Internet users and the number of accessible web pages grow, it is becoming increasingly difficult for users to find documents that are relevant to their particular needs. Users must either browse through a hierarchy of concepts to find the information they need or submit a query to a Search Engine and wade through hundreds of results most of them irrelevant. Web Crawlers are one of the most crucial components used by the Search Engines to collect pages from the Web. It is an intelligent means of browsing used by the Search Engine.