Institute of Electrical & Electronic Engineers
Mercator is a scalable web crawler which crawl the user request for a corresponding documents and fetch the URL request as a response and send to the user in an efficient way. The authors enumerate the major components of scalable web crawler, comment on alternatives, extensibility, customizability and tradeoffs in their design. In this paper, they propose internal architecture model of R-R Proxy server of A New Mercator Web Crawler. With the help of this architecture, they will handle the similar and different types of request from sender. The design of this architecture will reduce the overload of similar and dissimilar types of request on the Rewind Input Stream (RIS).