Web crawling and filtering for on-line auctions from a social network perspective

The on-line auction is one of the most successful types of electronic marketplace and has been the subject of many academic studies. In recent years, empirical research on on-line auctions has been flourishing because of the availability of large amounts of high-quality bid data from on-line auction sites. However, the increasingly large volumes of bid data have made data collection ever more complex and time consuming, and there are no effective resources that can adequately support this work. So this study focuses on the parallel crawling and filtering of on-line auctions from the social network perspective to help researchers collect and analyze auction data more effectively. The issues raised in this study include parallel crawling architecture, crawling strategies, content filtering strategies, prototype system implementation, and a pilot test of social network analysis. Finally we conduct an empirical experiment on eBay US and Ruten Taiwan to evaluate the performance of our crawling architecture and to understand auction customers’ bidding behavior characteristics. The results of this study show that our parallel crawling and filtering methods are able to work in the real world, and are significantly more effective than manual web crawling. The collected data are useful for drawing social network maps and analyzing bidding problems.