IJCTT-International Journal of Computer Trends and Technology
Internet usage has increased a lot in recent times. Users can find their resources by using different hypertext links. This usage of Internet has led to the invention of web crawlers. Web crawlers are full text search engines which assist users in navigating the web. These web crawlers can also be used in further research activities. For e.g. the crawled data can be used to find missing links, community detection in complex networks. In this paper, the authors have reviewed web crawlers: their architecture, types and various challenges being faced when search engines use the web crawlers.