上一讲主要内容 crawle面临的难题 Scalable, fast, polite ync UDP robust, continuous (slack about DNS prefetch K expiry dates) Text indexing client (UDP) 画晶实现高效率的基本技术 cache Cache Hyperlink Prefetch HttpH normalizer receve a Concurrency Page fetching context'thread disPageKnown? 多进程/多线程 e Craw k+ 异步I/O Persistent work-threadK H isUrlVisited? K+ pool of URLs 〓■有趣的技术 Bloom filter Consistent Hashing上一讲主要内容 ◼ Crawler面临的难题 ◼ Scalable, fast, polite, robust, continuous ◼ 实现高效率的基本技术 ◼ Cache ◼ Prefetch ◼ Concurrency ◼ 多进程/多线程 ◼ 异步I/O ◼ 有趣的技术 ◼ Bloom filter ◼ Consistent Hashing