Unsupervised User-Generated Content Extraction by Dependency Relationships

User-generated contents are very valuable for event detection, opinion mining and so on, but the extraction of those data is difficult because users are given strong power to present their contents in Web 2.0 pages. Compared to machine-generated contents, user-generated contents are very personalized, which often take on complex styles, combine various information and embed much noise. Users’ deep participation makes data acquisition environment a great change and breaks the hidden assumption of traditional extraction methods, which is that Web pages should be relatively regular. The traditional extraction methods can not adapt complex user-generated contents well. In this paper, we consider user-generated contents as unstable contents and advise an unsupervised approach to extract high-quality user-generated contents without noise. Those stable information in machine-generated contents, which are often omitted by traditional extraction methods, are firstly picked up by a two-stage filtering operation, page-level filtering and template-level filtering. Path accompanying distance is then defined to compute the dependency relationships between unstable information and stable information, which guide us to locate user-generated contents. Our approach gives a full consideration on structures, contents and the dependency information between stable and unstable contents to assure the extraction accuracy of user data. The whole process does not need any artificial participation. The experimental results show its good performance and robustness.