Automatic generation of agents for collecting hidden Web pages for data extraction

Abstract

As the Web grows, more and more data has become available under dynamic forms of publication, such as legacy databases accessed by an HTML form (the so called hidden Web). In situations such as this, integration of this data relies more and more on the fast generation of agents that can automatically fetch pages for further processing. As a result, there is an increasing need for tools that can help users generate such agents. In this paper, we describe a method for automatically generating agents to collect hidden Web pages. This method uses a pre-existing data repository for identifying the contents of these pages and takes the advantage of some patterns that can be found among Web sites to identify the navigation paths to follow. To demonstrate the accuracy of our method, we discuss the results of a number of experiments carried out with sites from different domains.

Introduction

With the popularization of the World-Wide Web, a huge amount of data from a number of different domains has become available. However, managing and querying such data is not trivial since we cannot make use of traditional database techniques. One way to deal with this data is through the so called Web wrappers [1], programs that extract unstructured data from Web pages and store it in suitable formats such as XML and relational tables.

To accomplish their task, Web wrappers take as input a set of pages from a Web source. This set of pages is generally collected by Web agents such as spiders or crawlers. Traditionally, these agents cover only the so called Publicly Indexable Web (PIW) [2], which corresponds to the set of Web pages reachable only by following hyperlinks. However, recent studies point out that the great majority of the pages containing useful data on the Web is outside the PIW [2], [3]. This large portion of the Web is generally called the hidden [1], [2] or deep [3] Web. Pages in the hidden Web are dynamically generated by programs in response to forms submitted to searchable databases. Thus, to allow wrappers to deal with the data on such pages we need a special type of agent to collect them, a type we call the hidden Web agent.

To illustrate the functionality expected from a hidden Web agent, suppose we want to fetch a set of data-rich pages¹ from the Bookpool Web site.² We start from the site’s main page, depicted in Fig. 1(a). Although we have a simple search box there (see arrow 1), an advanced search form, where users can better specify their needs, is preferable. Selecting the “Search” hyperlink (arrow 2) takes us to the advanced search page depicted in Fig. 1(b). After filling in some fields and submitting the form, an answer page is presented, as shown in Fig. 2(a). Note that not all information available is presented in the first answer page. Thus we should select the “Next” hyperlink (arrow 3) to get other pages, like the one in Fig. 2(b). An agent to fetch these pages should act as described.

In this paper, we present a method to generate hidden Web agents for sites with common navigational characteristics. The method uses a set of heuristics and a sample data repository for automatically finding relevant forms, filling them in, and collecting pages containing useful data. We also describe the results of a number of experiments carried out with sites from different domains to evaluate the accuracy of our method. These results show that our method is successful on 80% of the sites we have used in our experiments.

The rest of the paper is organized as follows. Section 2 overviews related work. In Section 3 we introduce the navigation pattern concept. Section 4 outlines our method for generating hidden Web agents. Experimental results are described in Section 5. Finally, in Section 6 we conclude the paper.