Jiaul H. Paik
Abstract
Stemming is a widely used technique in information retrieval systems to address the vocabulary mismatch problem arising out of morphological phenomena. The major shortcoming of the commonly used stemmers is that they accept the morphological variants of the query words without considering their thematic coherence with the given query, which leads to poor performance. Moreover, for many queries, such approaches also produce retrieval performance that is poorer than no stemming, thereby degrading the robustness. The main goal of this article is to present corpus-based fully automatic stemming algorithms which address these issues. A set of experiments on six TREC collections and three other non-English collections containing news and web documents shows that the proposed query-based stemming algorithms consistently and significantly outperform four state of the art strong stemmers of completely varying principles. Our experiments also confirm that the robustness of the proposed query-based stemming algorithms are remarkably better than the existing strong baselines.
- Bacchin, M., Ferro, N., and Melucci, M. 2005. A probabilistic model for stemmer generation. Inf. Process. Manage. 41, 1, 121--137. Google Scholar
Digital Library
- Baroni, M., Matiasek, J., and Trost, H. 2002. Unsupervised discovery of morphologically related words based on orthographic and semantic similarity. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning (MPL’02). Vol. 6, Association for Computational Linguistics, Stroudsburg, PA, 48--57. Google ScholarDigital Library
- Church, K. W. and Hanks, P. 1990. Word association norms, mutual information, and lexicography. Comput. Linguist. 16, 1, 22--29. Google ScholarDigital Library
- Collins-Thompson, K. and Callan, J. 2007. Estimation and use of uncertainty in pseudo-relevance feedback. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’07). ACM, New York, NY, 303--310. Google ScholarDigital Library
- Dolamic, L. and Savoy, J. 2009. Indexing and stemming approaches for the Czech language. Inf. Process. Manage. 45, 6, 714--720. Google ScholarDigital Library
- Dolamic, L. and Savoy, J. 2010. Comparative study of indexing and search strategies for the Hindi, Marathi, and Bengali languages. ACM Trans. Asian Lang. Inf. Process. 9, 3. Google ScholarDigital Library
- Goldsmith, J. 2001. Unsupervised learning of the morphology of a natural language. Comput. Linguist. 27, 2, 153--198. Google ScholarDigital Library
- Hammarstrom, H. 2009. Unsupervised learning of morphology and the languages of the world. Ph.D. thesis.Google Scholar
- Harman, D. 1991. How effective is suffixing. J. Amer. Soc. Inf. Sci. 42, 7--15.Google ScholarCross Ref
- Krovetz, R. 1993. Viewing morphology as an inference process. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’93). ACM, New York, 191--202. Google ScholarDigital Library
- Lovins, J. 1968. Development of a stemming algorithm. Mech. Tran. Comput. Linguistics, 22--31.Google Scholar
- Majumder, P., Mitra, M., Parui, S. K., Kole, G., Mitra, P., and Datta, K. 2007. YASS: Yet another suffix stripper. ACM Trans. Inf. Syst. 25, 4, 18. Google ScholarDigital Library
- Manning, C. D. and Schütze, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA. Google ScholarDigital Library
- Mayfield, J. and McNamee, P. 2003. Single n-gram stemming. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’03). ACM, New York, 415--416. Google ScholarDigital Library
- McNamee, P., Nicholas, C. K., and Mayfield, J. 2009. Addressing morphological variation in alphabetic languages. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 75--82. Google ScholarDigital Library
- Oard, D. W., Levow, G.-A., and Cabezas, C. I. 2000. CLEF experiments at Maryland: Statistical stemming and backoff translation. In Proceedings of the Workshop of the Cross-Language Evaluation Forum (CLEF’00). 176--187. Google ScholarDigital Library
- Paik, J. H. and Parui, S. K. 2011. A fast corpus-based stemmer. ACM Trans. Asian Lang. Inf. Process. 10, 2, 8:1--8:16. Google ScholarDigital Library
- Paik, J. H., Mitra, M., Parui, S. K., and Järvelin, K. 2011a. GRAS: An effective and efficient stemming algorithm for information retrieval. ACM Trans. Inf. Syst. 29, 19:1--19:24. Google ScholarDigital Library
- Paik, J. H., Pal, D., and Parui, S. K. 2011b. A novel corpus-based stemming algorithm using co-occurrence statistics. In Proceedings of the 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). ACM, New York, 863--872. Google ScholarDigital Library
- Peng, F., Ahmed, N., Li, X., and Lu, Y. 2007. Context sensitive stemming for web search. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’07). ACM, New York, 639--646. Google ScholarDigital Library
- Porter, M. F. 1997. An algorithm for suffix stripping. In Readings in Information Retrieval, 313--316. Google ScholarDigital Library
- Sakai, T., Manabe, T., and Koyama, M. 2005. Flexible pseudo-relevance feedback via selective sampling. ACM Trans. Asian Lang. Inf. Process. 4, 111--135. Google ScholarDigital Library
- Savoy, J. 2006. Light stemming approaches for the French, Portuguese, German and Hungarian languages. In Proceedings of the ACM Symposium on Applied Computing (SAC’06). ACM, New York, 1031--1035. Google ScholarDigital Library
- Smucker, M. D., Allan, J., and Carterette, B. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM’07). ACM, New York, 623--632. Google ScholarDigital Library
- Snyder, B. and Barzilay, R. 2008. Unsupervised multilingual learning for morphological segmentation. In Proceedings of the Annual Conference of the Association for Computational Linguistics.Google Scholar
- Voorhees, E. M. 2003. Overview of the trec 2003 robust retrieval track. In Proceedings of TREC. 69--77.Google Scholar
- Voorhees, E. M. 2005. The trec robust retrieval track. SIGIR Forum 39, 1, 11--20. Google ScholarDigital Library
- Xu, J. and Croft, W. B. 1998. Corpus-based stemming using cooccurrence of word variants. ACM Trans. Inf. Syst. 16, 1, 61--81. Google ScholarDigital Library
- Xu, J. and Croft, W. B. 2000. Improving the effectiveness of information retrieval with local context analysis. ACM Trans. Inf. Syst. 18, 79--112. Google ScholarDigital Library
- Zhao, L. and Callan, J. 2010. Term necessity prediction. In Proceedings of the 19th ACM Conference on Information and Knowledge Management (CIKM’10). ACM, New York, 259--268. Google ScholarDigital Library
Index Terms
- Effective and Robust Query-Based Stemming
Recommendations
GRAS: An effective and efficient stemming algorithm for information retrieval
A novel graph-based language-independent stemming algorithm suitable for information retrieval is proposed in this article. The main features of the algorithm are retrieval effectiveness, generality, and computational efficiency. We test our approach on ...
A Fast Corpus-Based Stemmer
Stemming is a mechanism of word form normalization that transforms the variant word forms to their common root. In an Information Retrieval system, it is used to increase the system’s performance, specifically the recall and desirably the precision. ...
A novel unsupervised corpus-based stemming technique using lexicon and corpus statistics
AbstractWord Stemming is a widely used mechanism in the fields of Natural Language Processing, Information Retrieval, and Language Modeling. Language-independent stemmers discover classes of morphologically related words from the ambient ...
Comments