Not so Creepy Crawler: Easy Crawler Generation With Standard XML Queries

Download Now Date Added: Apr 2010
Format: PDF

Web crawlers are increasingly used for focused tasks such as the extraction of data from Wikipedia or the analysis of social networks like In these cases, pages are far more uniformly structured than in the general Web and thus crawlers can use the structure of Web pages for more precise data extraction and more expressive analysis. This paper presents a focused, structure-based crawler generator, the "Not so Creepy Crawler" (nc2). What sets nc2 apart, is that all analysis and decision tasks of the crawling process are delegated to an (Arbitrary) XML query engine such as XQuery or Xcerpt. Customizing crawlers just means writing (Declarative) XML queries that can access the currently crawled document as well as the metadata of the crawl process.