3. Lessons Learned from Evaluation

The performance evaluation of word sense disambiguation systems has only been more or less standardised in the last decade with the first three SENSEVAL and the more recent SEMEVAL exercises. These exercises have pointed to the superiority of supervised methods using multiple knowledge sources and ensembles of classifiers. Behind the apparently plateaued performance of state-of-the-art systems, some fundamental issues including sense granularity, sparseness of sense-tagged data, and contribution to real applications, still remain. But more importantly, evaluation results also suggest that there is something about the target words themselves which is responsible for the differential performance among systems trained on the same feature sets, and within systems on different target words. We suggest that such intrinsic nature of the target words has a direct impact on the accuracy of sense disambiguation, which cannot be addressed solely from the computational perspective.