A Users Search History Based Approach to Manage Revisit Frequency of an Incremental Crawler
With the tremendous growth of the Internet, World Wide Web has become a huge source of hyperlinked information contained in hypertext documents. Search engines use web crawlers to collect these documents from web for the purpose of storage and indexing. An incremental crawler visits the web for updating its collection. There is a need to regulate the frequency of the crawler to visit web sites and provide latest information to the user. In this paper a novel approach to manage the revisiting frequency of an incremental crawler based on the users search history is being proposed.