Unsupervised Part-of-Speech Tagging

In this chapter, homogeneity with respect to syntactic word classes (partsof- speech, POS) is aimed at. The method presented in this section is called unsupervised POS-tagging, as its application results in corpus annotation in a comparable way to what POS-taggers provide. Nevertheless, its application results in slightly different categories as opposed to what is assumed by a linguistically motivated POS-tagger, which hampers evaluation methods that compare unsupervised POS tags to linguistic annotations. To measure the extent to which unsupervised POS tagging can contribute in application-based settings, the system is evaluated in supervised POS tagging, word sense disambiguation, named entity recognition and chunking, improving on the state-of-the-art for supervised POS tagging and word sense disambiguation. Unsupervised POS-tagging has been explored since the beginning of the 1990s. Unlike in previous approaches, the kind and number of different tags is here generated by the method itself. Another difference to other methods is that not all words above a certain frequency rank get assigned a tag, but the method is allowed to exclude words from the clustering, if their distribution does not match closely enough with other words. The lexicon size is considerably larger than in previous approaches, which results in a more robust tagging.