Assembling a Web-Crawler

Provided by: International Journal of Advance Research in Science and Engineering (IJARSE)
Topic: Developer
Format: PDF
Web crawlers are now-a-days, most important part of the internet. Can the authors imagine their life without GOOGLE, YAHOO and Bing? The Web-crawlers are heart of all those search engines. Besides just Web-crawler, many complex technologies such as page ranking system, effective key search techniques, disabling malicious sites and much more are involved in making a search engine. This essay is dedicated for complete understanding of a basic Web-crawler and some of its requirements. Keep in mind that for commercial Web crawlers, both hardware and software requirements are much more than what is described over here.

Find By Topic