Machine Discovery Based on the Co-occurrence of References in a Search Engine

This paper describes a new method of discovering clusters of related Web pages. By clustering Web pages and visualizing them in the form of graph, users can easily access to related pages. Since related Web pages are often referred from the same Web page, the number of co-occurrence of references in a search engine is used for discovering relation among pages. Two URLs are given to a search engine as keywords, and the value of the number of pages searched from both URLs divided by the number of pages searched from either URL, which is called Jaccard coefficient, is calculated as the criteria for evaluating the relation between the two URLs. The value is used for deciding the length of an edge in a graph so that vertices of related pages will be located close to each other. Our system based on the method succeeds in discovering clusters of various genres, although the system does not interpret the contents of the pages. The method of calculating Jaccard coefficient is easily processed by computer systems, and it is suitable for the discovery from the data acquired through the internet.