Outline Motivation and taxonomy of crawlers Basic crawlers and implementation issues Universal crawlers Preferential (focused and topical)crawlers Evaluation of preferential crawlers Crawler ethics and conflicts New developments: social, collaborative federated crawlers Slides 2007 Filippo Menczer, Indiana University School of Informatics Bing Liu: Web Data Mining. Springer, 200 Ch 8 Web Crawing by Filippo Menczer informaticsSlides © 2007 Filippo Menczer, Indiana University School of Informatics Bing Liu: Web Data Mining. Springer, 2007 Ch. 8 Web Crawling by Filippo Menczer Outline • Motivation and taxonomy of crawlers • Basic crawlers and implementation issues • Universal crawlers • Preferential (focused and topical) crawlers • Evaluation of preferential crawlers • Crawler ethics and conflicts • New developments: social, collaborative, federated crawlers