Distributed Crawler Processes Applied to Internet Information Retrieval

Provided by: AICIT
Topic: Big Data
Format: PDF
Search engines and search catalogs make use of robots, which communicate with a web server and have the incumbency of collecting the content of web pages. This paper describes a preliminary research on Information Retrieval (IR) systems. The authors developed a prototype tool that runs on a master-slave model and uses distributed processing in order to decentralize the workload while retrieving information from the internet. Later, its viability is demonstrated with a set of executions and the discussion follows with an analysis of the master process overload versus the number of slaves connected.

Find By Topic