Aliaksei Severyn
ABSTRACT
Supervised learning applied to answer re-ranking can highly improve on the overall accuracy of question answering (QA) systems. The key aspect is that the relationships and properties of the question/answer pair composed of a question and the supporting passage of an answer candidate, can be efficiently compared with those captured by the learnt model.
In this paper, we define novel supervised approaches that exploit structural relationships between a question and their candidate answer passages to learn a re-ranking model. We model structural representations of both questions and answers and their mutual relationships by just using an off-the-shelf shallow syntactic parser. We encode structures in Support Vector Machines (SVMs) by means of sequence and tree kernels, which can implicitly represent question and answer pairs in huge feature spaces. Such models together with the latest approach to fast kernel-based learning enabled the training of our rerankers on hundreds of thousands of instances, which previously rendered intractable for kernelized SVMs. The results on two different QA datasets, e.g., Answerbag and Jeopardy! data, show that our models deliver large improvement on passage re-ranking tasks, reducing the error in Recall of BM25 baseline by about 18%. One of the key findings of this work is that, despite its simplicity, shallow syntactic trees allow for learning complex relational structures, which exhibits a steep learning curve with the increase in the training size.
- M. Bilotti, P. Ogilvie, J. Callan, and E. Nyberg. Structured retrieval for Question Answering. In Proceedings of ACM SIGIR, 2007. Google Scholar
Digital Library
- M. W. Bilotti, J. L. Elsas, J. Carbonell, and E. Nyberg. Rank learning for factoid Question Answering with linguistic and semantic constraints. In Proc. of CIKM, 2010. Google ScholarDigital Library
- M. W. Bilotti and E. Nyberg. Improving text retrieval precision and answer accuracy in question answering systems. In Proc. of IR4QA at COLING, 2008. Google ScholarDigital Library
- S. Blair-Goldensohn, K. R. McKeown, and A. H. Schlaikjer. Answering definitional questions: A hybrid approach. In Proc. of AAAI, 2004.Google Scholar
- N. Cancedda, E. Gaussier, C. Goutte, and J. M. Renders. Word sequence kernels. JMLR, 2003. Google ScholarDigital Library
- Y. Chen, M. Zhou, and S. Wang. Reranking answers from definitional QA using language models. In Proc. of ACL, 2006. Google ScholarDigital Library
- M. Ciaramita and Y. Altun. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proc. EMNLP, 2006. Google ScholarDigital Library
- M. Collins and N. Duffy. New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron. In Proc. of ACL, 2002. Google ScholarDigital Library
- H. Cui, M. Kan, and T. Chua. Generic soft pattern models for definitional QA. In Proc. of SIGIR, 2005. Google ScholarDigital Library
- A. Echihabi and D. Marcu. A noisy-channel approach to question answering. In Proc. of ACL, 2003. Google ScholarDigital Library
- D. Ferrucci. Build watson: an overview of deepqa for the Jeopardy! challenge. In Proc. of PACT, 2010. Google ScholarDigital Library
- A.-M. Giuglea and A. Moschitti. Knowledge Discovering using FrameNet, VerbNet and PropBank. In Proc. of Ontology and Knowledge Discovering at ECML 2004, Pisa, Italy, 2004.Google Scholar
- A.-M. Giuglea and A. Moschitti. Semantic Role Labeling via Framenet, Verbnet and Propbank. In Proc. of ACL, Sydney, Australia, 2006. Google ScholarDigital Library
- A. Hickl, J. Williams, J. Bensley, K. Roberts, Y. Shi, and B. Rink. Question answering with LCC chaucer at trec 2006. In Proc. of TREC, 2006.Google Scholar
- J. Jeon, W. B. Croft, and J. H. Lee. Finding similar questions in large question and answer archives. In Proc. of CIKM, NY, USA, 2005. Google ScholarDigital Library
- T. Joachims. Making large-scale SVM learning practical. In B. Sch$\ddoto$lkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods, 1999.Google Scholar
- T. Kudo and Y. Matsumoto. Fast methods for kernel-based text analysis. In Proc. of ACL, 2003. Google ScholarDigital Library
- Y. Mehdad, A. Moschitti, and F. M. Zanzotto. Syntactic/semantic structures for textual entailment recognition. In HLT-NAACL, 2010. Google ScholarDigital Library
- A. Moschitti. A study on convolution kernels for shallow semantic parsing. In Proc. of ACL, 2004. Google ScholarDigital Library
- A. Moschitti. Efficient convolution kernels for dependency and constituent syntactic trees. In Proceedings of ECML, 2006. Google ScholarDigital Library
- A. Moschitti. Kernel methods, syntax and semantics for relational text categorization. In Proc. of CIKM, NY, USA, 2008. Google ScholarDigital Library
- A. Moschitti and C. Bejan. A semantic kernel for predicate argument classification. In Proc. of CoNLL, Boston, MA, USA, 2004.Google Scholar
- A. Moschitti, S. Quarteroni, R. Basili, and S. Manandhar. Exploiting syntactic and shallow semantic kernels for question/answer classification. In Proc. of ACL, 2007.Google Scholar
- T.-V. T. Nguyen and A. Moschitti. Joint distant and direct supervision for relation extraction. In Proc. IJCNLP, Chiang Mai, Thailand, 2011.Google Scholar
- F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. CoRR, 2006.Google Scholar
- Y. Sasaki. Question answering as question-biased term extraction: A new approach toward multilingual QA. In Proc. of ACL, 2005. Google ScholarDigital Library
- A. Severyn and A. Moschitti. Large-scale support vector learning with structural kernels. In ECML/PKDD, 2010. Google ScholarDigital Library
- J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge Univ. Press, 2004. Google ScholarDigital Library
- D. Shen and M. Lapata. Using semantic roles to improve question answering. In Proc. of EMNLP-CoNLL, 2007.Google Scholar
- L. Shen, A. Sarkar, and A. Joshi. Using LTAG Based Features in Parse Reranking. In EMNLP, 2003. Google ScholarDigital Library
- S. Small, T. Strzalkowski, T. Liu, S. Ryan, R. Salkin, N. Shimizu, P. Kantor, D. Kelly, and N. Wacholder. Hitiqa: Towards analytical question answering. In Proc. of COLING, 2004. Google ScholarDigital Library
- A. F. Smeaton. Using NLP or NLP resources for Information Retrieval tasks. In T. Strzalkowski, editor, Natural Language Information Retrieval. Kluwer Ac. Pub., Dordrecht, NL, 1999.Google ScholarCross Ref
- T. Strzalkowski, J. P. Carballo, J. Karlgren, A. H. P. Tapanainen, and T. Jarvinen. Natural Language Information Retrieval: TREC-8 report. In Proc. of TREC, 1999.Google Scholar
- T. Strzalkowski, G. C. Stein, G. B. Wise, J. P. Carballo, P. Tapanainen, T. Jarvinen, A. Voutilainen, and J. Karlgren. Natural Language Information Retrieval: TREC-7 report. In Proc. of TREC, 1998.Google Scholar
- M. Surdeanu, M. Ciaramita, and H. Zaragoza. Learning to rank answers on large online QA collections. In Proc. of ACL-HLT, 2008.Google Scholar
- J. Suzuki, Y. Sasaki, and E. Maeda. SVM answer selection for open-domain Question Answering. In Proc. of Coling, 2002. Google ScholarDigital Library
- Y. Versley, A. Moschitti, M. Poesio, and X. Yang. Coreference systems based on kernels methods. In Coling, Manchester, England, 2008. Google ScholarDigital Library
- E. M. Voorhees. Using Wordnet to disambiguate word senses for text retrieval. In Proc. of SIGIR, 1993. Google ScholarDigital Library
- E. M. Voorhees. Query expansion using lexical-semantic relations. In Proc. of SIGIR, 1994. Google ScholarDigital Library
- E. M. Voorhees. Overview of the trec 2004 question answering track. In Proc. of TREC 2004, 2004.Google Scholar
- C.-N. J. Yu and T. Joachims. Training structural SVMs with kernels using sampled cuts. In KDD, 2008. Google ScholarDigital Library
- D. Zelenko, C. Aone, and A. Richardella. Kernel methods for relation extraction. JMLR, 2003. Google ScholarDigital Library
Index Terms
- Structural relationships for large-scale learning of answer re-ranking
Recommendations
Kernel-based learning to rank with syntactic and semantic structures
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrievalKernel Methods (KMs) are powerful machine learning techniques that can alleviate the data representation problem as they substitute scalar product between feature vectors with similarity functions (kernels) directly defined between data instances, e.g., ...
Assessing the Impact of Syntactic and Semantic Structures for Answer Passages Reranking
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge ManagementIn this paper, we extensively study the use of syntactic and semantic structures obtained with shallow and deeper syntactic parsers in the answer passage reranking task. We propose several dependency-based structures enriched with Linked Open Data (LD) ...
Linguistic kernels for answer re-ranking in question answering systems
Answer selection is the most complex phase of a question answering (QA) system. To solve this task, typical approaches use unsupervised methods such as computing the similarity between query and answer, optionally exploiting advanced syntactic, semantic ...
Comments