The number of web pages is increasing into millions and trillions around the world. To make searching much easier for users, web search engines came into existence. Web search engines are used to find specific information on the World Wide Web. Without search engines, it would be almost impossible to locate anything on the Web unless or until a specific URL address is known. This information is provided to search by a web crawler which is a computer program or software. Web crawler is an essential component of search engines, data mining and other Internet applications.