Web Crawling Agents

An essential component of information mining and pattern discovery on the Web is the Web Crawling Agent (WCA). General-purpose Web Crawling Agents, which were briefly described in Chapter 1, are intended to be used for building generic portals. The diverse and voluminous nature of Web documents presents formidable challenges to the design of high performance WCAs. They require both powerful processors and a tremendous amount of storage, and yet even then can only cover restricted portions of the Web. Nonetheless, despite their fundamental importance in providing Web services, the design of WCAs is not well-documented in the literature. This chapter describes the conceptual design and implementation of Web crawling agents.