Leveraging linguistic traits and semi-supervised learning to single out informational content across how-to community question-answering archives

Abstract

Community Question-Answering sites (e.g., Yahoo! Answers) have become large-scale knowledge bases of natural language questions formulated by their own members. In other to provide quick answers, these sites are compelled to make the best out of the content stored in their repository. Researchers have discovered that, on the one hand, many of these services are the confluence of an information-seeking and a social network that are constantly overlapping, and on the other hand, how-to questions are frequently published across these platforms. By and large, informational procedural questions are highly likely to expect informational answers, while non-informational manner questions target at socially interacting with other members of the community. In order to enhance user experience by reducing the delay in answering, these services are heartened to identify, retrieve and revitalize the content maintained in their knowledge bases. For this purpose, it is key to match the intent of new posted questions with the intention of archived answers that will be presented to the asker.

By manually annotating a reduced number of how-to questions and answers, we carried out an exploratory analysis that unveils a dichotomy between the interaction of these two networks. More precisely, we corroborate previous findings indicating that procedural questions are more likely to bear an informational goal, but our analysis is also extended to their answers, and it reveals that they exhibit a more conspicuous confluence. In substance, we find out that informational and non-informational answers are very likely to show up regardless of the end of the question. Then, we take advantage of this tagged set and of massive unlabelled material for exploiting two state-of-the-art single-view semi-supervised approaches aimed at discriminating informational from non-informational how-to content.

Moreover, our proposed models leverage assorted linguistically-motivated features, such as sentiment analysis and dependency parsing as well as named entity recognition. Our outcomes show that attributes, harvested from morphological and sentiment analysis, proven to be effective under a semi-supervised framework. At the expenses of low annotation costs, these linguistically-motivated semi-supervised models reached an accuracy of 84.25% and 74.41% for classifying questions and answers, respectively. In addition, we quantify the impact of automatically detecting informational/non-informational intents on the retrieval of best answers, i.e., an improvement of 4.12% in terms of precision at one.

Introduction

The most massively popular community Question Answering (cQA) services keep over 100 million resolved questions. As of December 2015, Yahoo! Answers¹ had about one hundred million members, and its knowledge base maintained more than four hundred million questions prompted since its inception (i.e., almost 250 million asked in English). As a logical consequence, cQA repositories are now perceived as vast sources of reusable information conveyed in natural language enriched with some metadata [33], [43]. Take for instance, search engines attempt at reusing best answers embodied in these knowledge bases for producing snippets from their best answers, whenever they detect that a question-like query was submitted by a web user. Needless to say, there are different types of cQA sites [9], for example some communities lack moderation, other are strongly regulated. In essence, many of these cQA platforms are conceived as the synergy of an information-seeking and a social network [32], because their fellows can ask any sort of simple, complex or detailed question, and they can additionally seek opinions and conduct polls. All these elements expect to receive several responses coming from multiple users of the community, reflecting different walks of life, word-of-mouth tips, solutions and opinions as well as answers.

Moreover, members yield social capital when they participate in the dynamic of these communities. For instance, they make comments, assess the quality or express their sympathy towards questions and answers (e.g., thumbs-up/thumbs-down). By means of these social interactions, users of these sites disseminate information (e.g., facts and word-of-mouth tips) resulting in a first-rate, quickly growing repository of questions and answers evaluated by humans. Nonetheless, there is an inherent lag between when questions are published and when first and good responses are posted. Hence, cQA platforms target at responding to the pressing need for automatically reusing past resolved questions, and as a by-product, they also lessen duplicate-asking.

By and large, Yahoo! Answers has been categorized as a social cQA, which facilitates the voluntary discussion among peers for questioning and answering [15]. Such sites encourage and thrive on communities built around information exchange, introducing a social aspect to information seeking [36]. This type of cQA site has demonstrated to be more attractive at responding complex, especially how-to, questions (e.g.,“How to remove pimples?” and “How to break up with my boyfriend?”) [17]. In effect, our exploratory analysis indicates that across manner questions the synergy between an information-seeking and a social network becomes more conspicuous. On the one hand, this sort of question is highly likely to be informational, but on the other hand, its answers show an almost even distribution across informational and non-informational questions. Therefore, when retrieving and revitalizing elements stored in the knowledge base, it is relevant to match the intent behind new posted questions with the goal of archived answers that will be presented to the asker. Put differently, informational manner questions, such as “How to make beer?” and “How to make easy home-made mayonnaise?”, require informational good answers, but in a similar way, non-informational procedural questions, including “How to recover from a broken heart?” and “How can I get beautiful hair?”, need good, but non-informational, answers. Textbook cases are questions aiming at attracting or finding answers bearing great social capital.

In this work, we seek to effectively separate the informational from the non-informational content across how-to questions and answers, and show how sorting this out cooperates on improving the retrieval, and thus the reuse, of their past best answers. We base this separation on discriminant models constructed on top of linguistically motivated properties. In other words, our reasoning is that good answers can be fetched from both groups, ergo their archival value depends on matching the intent of the question. This premise predicates upon the idea that there are linguistic traits characterizing both groups. To be more exact, this paper makes the following contributions with respect to our forerunners:

• Questions across cQA archives are composed of a title and a body. The former is utilized for: a) conveying the gist of the question; b) sometimes it is used as an attention catcher; or c) summarizing the whole question. Despite of the fact that the latter is likely to go into the details of the question, current studies have focused their attention on models to discriminate the informational/non-informational nature of questions on grounds of their titles only. This work furthers these studies by examining the contribution of their bodies, and it additionally aims at automatically discriminating informational/non-informational answers.

• In deed, manually annotating full questions and answers demands considerable efforts in time and human resources, especially if question bodies are taken into account. As a means of lowering this cost, our work tests two state-of-the-art semi-supervised approaches for the classification of both questions and answers into informational or non-informational. Both methods are compared to their supervised counterparts, in a scenario, where few training material is available.

• In order to generalize effective classifiers, we explored distinct models for questions and answers enriched with assorted linguistically-oriented properties. Contrary to previous works, we do not only take into account surface features such as frequent words across question titles, but also we leverage attributes distilled from natural language processing tools including sentiment and morphological analysis as well as dependency parsing.

• Lastly, we carry out several experiments that empirically prove and quantify how matching the intent of questions and answers, via these obtained semi-supervised models, aids in significantly improving the retrieval of best answers to how-to questions from cQA archives.

In a nutshell, this piece of research advances this task in the following aspects: a) a thorough investigation into which kind of linguistic processing is more effective when combining few labelled training instances with large volumes of unlabelled material; b) it extends earlier studies by examining the contribution of question bodies; and c) it also digs deeper into linguistically-motivated models to recognize informational/non-informational answers.

At the expense of few training samples, our outcomes show that a semi-supervised variation of Support Vector Machines (SVM) models enriched with attributes produced by linguistic processing, such as lemmatization and sentiment analysis, can considerably enhance the classification rate, and consequently, the retrieval of best answers. Our empirical results also indicate that question bodies aid in enhancing the performance, when they are available. Overall, we show that the obtained models assist in increasing the retrieval of best answers by 4.12%. The reminder of this paper is organized as follows. Section 2 outlines the related work, next Section 3 dissects our semi-supervised models, Section 4 describe our experiments and findings. Eventually, Section 5 draws some conclusions and sketches future work.