Fusing Text and Image Data with the Help of the OWLnotator

A central challenge for any approach to mining multimedia data concerns the availability of a unified semantics that allows for the fusion of multicodal information objects. To meet this challenge, a format is needed that enables the representation of multimedia data even across the border of different (e.g. iconic and symbolic) codes using the same ontology. In this paper, we introduce the OWLnotator as a first step to meeting this dual challenge by example of text-image relations. The OWLnotator is presented as part of the eHumanities Desktop, a browser-based, platform-independent environment for the support of collaborative research in the digital humanities. It focuses on modeling and analyzing multicodal, multimedia information objects as studied in the humanities. The eHumanities Desktop contains a wide range of tools for managing, analyzing and sharing resources based on a scalable concept of access permissions. Within this framework, we introduce the OWLnotator as a tool for annotating intra- and intermedia relations of artworks. The OWLnotator allows for modeling relations of symbolic and iconic signs of various levels of resolution: ranging from the level of elementary constituents to the one of complete texts and images. To this end, the OWLnotator integrates TEILex (a system for interrelating corpus and lexicon data as part of the eHumanities Desktop) with the expressiveness of OWL-based ontologies in order to meet the first part of our twofold challenge. As an evaluation, we illustrate the OWLnotator by means of “Illustrations of Goethes Faust”.