Annotating Learner Errors

Abstract

The cornerstone of any NLP evaluation is the manual annotation of data to create a gold standard evaluation set. With this set, one can quickly compare a system’s predictions to the annotators’ judgments and report performance. Annotation tasks can range from simply labeling a picture or website, to assigning part-of-speech tags, or to more complicated tasks such as discourse structure labeling. Simple tasks may require just one annotator, but more complex tasks often require two or more annotators since judgments are more likely to differ. While there has been considerable emphasis in the field placed on system development, there has actually been little attention paid to developing best practices for annotation. In this chapter, we cover the issues surrounding error annotation, describe two different annotation schemes in detail, and describe work on ameliorating some of the problems that researchers have previously encountered when annotating learner productions for errors.