International Journal of Advance Research in Science and Engineering (IJARSE)
Web crawlers are now-a-days, most important part of the internet. Can the authors imagine their life without GOOGLE, YAHOO and Bing? The Web-crawlers are heart of all those search engines. Besides just Web-crawler, many complex technologies such as page ranking system, effective key search techniques, disabling malicious sites and much more are involved in making a search engine. This essay is dedicated for complete understanding of a basic Web-crawler and some of its requirements. Keep in mind that for commercial Web crawlers, both hardware and software requirements are much more than what is described over here.