ABSTRACT
Question answering (QA) helps one go beyond traditional keywords-based querying and retrieve information in more precise form than given by a document or a list of documents. Several community-based QA (CQA) services have emerged allowing information seekers pose their information need as questions and receive answers from their fellow users. A question may receive multiple answers from multiple users and the asker or the community can choose the best answer. While the asker can thus indicate if he was satisfied with the information he received, there is no clear way of evaluating the quality of that information. We present a study to evaluate and predict the quality of an answer in a CQA setting. We chose Yahoo! Answers as such CQA service and selected a small set of questions, each with at least five answers. We asked Amazon Mechanical Turk workers to rate the quality of each answer for a given question based on 13 different criteria. Each answer was rated by five different workers. We then matched their assessments with the actual asker's rating of a given answer. We show that the quality criteria we used faithfully match with asker's perception of a quality answer. We furthered our investigation by extracting various features from questions, answers, and the users who posted them, and training a number of classifiers to select the best answer using those features. We demonstrate a high predictability of our trained models along with the relative merits of each of the features for such prediction. These models support our argument that in case of CQA, contextual information such as a user's profile, can be critical in evaluating and predicting content quality.
- Dervin, B. (1998). Sense-making theory and practice: An overview of user interests in knowledge seeking and use. In Journal of Knowledge Management, 2(2), 36--46.Google ScholarCross Ref
- Gazan, R. (2008). Social annotations in digital library collections. D-Lib Magazine, 11/12(14). Available from http://www.dlib.org/dlib/november08/gazan/11gazan.html.Google Scholar
- Harper, M. F., Raban, D. R., Rafaeli, S., & Konstan, J. K. (2008). Predictors of answer quality in online Q&A sites. In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems (pp. 865--874). New York: ACM. Google ScholarDigital Library
- Janes, J. (2003). The Global Census of Digital Reference. In 5th Annual VRD Conference. San Antonio, TX.Google Scholar
- Kim, S., Oh, J-S., & Oh, S. (2007). Best-Answer Selection Criteria in a Social Q&A site from the User Oriented Relevance Perspective. Proceeding of the 70th Annual Meeting of the American Society for Information Science and Technology (ASIST '07), 44.Google ScholarCross Ref
- Kresh, D. N. (2000). Offering High Quality Reference Service on the Web: The Collaborative Digital Reference Service (CDRS). D-Lib Magazine, 6.Google ScholarDigital Library
- Lee, J. H., Downie, J. S., & Cunningham, S. J. (2005). Challenges in cross-cultural/ multilingual music information seeking. In Proceedings of the 6th International Society for Music Information Retrieval (pp. 1--7). London, UK.Google Scholar
- Liu, Y., Bian, J., & Agichtein, E. (2008). Predicting Information Seeker Satisfaction in Community Question Answering. Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Pomerantz, J. (2008). Evaluation of Online Reference Services. Bulletin of the American Society for Information Science and Technology, 34(2), 15--19. Available from http://www.asis.org/Bulletin/Dec-07/pomerantz.html.Google ScholarCross Ref
- Pomerantz, J., Nicholson, S., Belanger, Y., & Lankes, R. D. (2004). The Current State of Digital Reference: Validation of a General Digital Reference Model through a Survey of Digital Reference Services. Information Processing & Management, 40(2), 347--363. Google ScholarDigital Library
- Saracevic, T. (1995). Evaluation of evaluation in information retrieval. Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval (pp. 138--146). Seattle, USA. Google ScholarDigital Library
- Shah, C., Oh, J. S., & Oh, S. (2008). Exploring characteristics and effects of user participation in online social Q&A sites. First Monday, 13(9). Available from http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/2182/2028.Google Scholar
- Shah, C., Oh, S., & Oh, J-S. (2009). Research Agenda for Social Q&A. Library and Information Science Research, 11(4), 205--209.Google ScholarCross Ref
- Su, Q., Pavlov, D., Chow, J., & Baker, W. (2007). Internet-scale collection of human-reviewed data. In C. L. Williamson, M. E. Zurko, P. E. Patel-Schneider, & P. J. Shenoy (Eds.), Proceedings of the 16th International Conference on World Wide Web (pp. 231--240). New York: ACM. Google ScholarDigital Library
- Voorhees, E. M (2003). Overview of the TREC 2003 question-answering track. In TREC 2003.Google Scholar
- Zhu, Z., Bernhard, D., & Gurevych, I. (2009). A Multi-dimensional Model for Assessing the Quality of Answers in Social Q&A Sites. Technical Report TUD-CS-2009-0158. Technische Universitat Darmstad.Google Scholar
Index Terms
- Evaluating and predicting answer quality in community QA
Recommendations
Analyzing and predicting question quality in community question answering services
WWW '12 Companion: Proceedings of the 21st International Conference on World Wide WebUsers tend to ask and answer questions in community question answering (CQA) services to seek information and share knowledge. A corollary is that myriad of questions and answers appear in CQA service. Accordingly, volumes of studies have been taken to ...
Using graded-relevance metrics for evaluating community QA answer selection
WSDM '11: Proceedings of the fourth ACM international conference on Web search and data miningCommunity Question Answering (CQA) sites such as Yahoo! Answers have emerged as rich knowledge resources for information seekers. However, answers posted to CQA sites can be irrelevant, incomplete, redundant, incorrect, biased, ill-formed or even ...
Predicting web searcher satisfaction with existing community-based answers
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information RetrievalCommunity-based Question Answering (CQA) sites, such as Yahoo! Answers, Baidu Knows, Naver, and Quora, have been rapidly growing in popularity. The resulting archives of posted answers to questions, in Yahoo! Answers alone, already exceed in size 1 ...
Comments