How do we measure and improve the quality of a hierarchical ontology?

Abstract

Hierarchical ontologies enable organising information in a human–machine understandable form, but constructing them for reuse and maintainability remains difficult. Often supporting tools available lack formal methodological underpinning and their developers are not supported by any concomitant metrics. The paper presents a formal underpinning to provide quality metrics of a taxonomy hierarchical ontology and proposes a methodology for semi-automatic building of maintainable taxonomies. Users provide terms to be used to describe different ontological elements as well as their attributes and their ranges of values. The methodology uses the formalised metrics to assess the quality of the users input and proposes changes according to given quality constraints. The paper illustrates the metrics and the methodology in constructing and repairing two medium size well-known taxonomies.

Introduction

Domain specific taxonomies represent information semantics in a shareable and reusable manner (Gruber, 1993). Suitably crafted, they facilitate exchange of organization-dependent information within and across organizational boundaries and can play a central role in alleviating political and technical obstacles involved. Taxonomies often come from different organisations and persons with varying agendas with varying quality criteria. Thus interest in their evaluation in the context of their design within semantically enabled technologies increased significantly in recent years. Some examples of recent such works can be found in (Middleton et al., 2004), where a similarity system is presented as a prelude to evaluation, or in (Staab et al., 2001), where the authors propose a complex framework consisting of 160 characteristics spread across five dimensions: content, language, development methodology, building tools, and usage costs.

The significance of a well formed taxonomic structure cannot be overstated. They are a key technology in supporting user preferences search systems over the Semantic Web (Chamiel and Pagnucco, 2008). For example, a hierarchical musical genre system is a user preferencing system: an album can be identified as Progressive Rock which may be a leaf concept in a taxonomy but at the same time an album can be identified with the concept Rock even though it serves as an abstract concept. Resolving such problems in generalizations/classifications is not easy and requires tuning and improving the quality of taxonomies to avoid them in the first place. Borst (1997) generalizes evaluation to one of three scenarios: where both users and machines need a taxonomy assessment guide to suit their needs; a second where designers need practical guides to build and evaluate ontologies before publishing them, and a third where automatic taxonomy machine-learning requires identifying a suitable option among varying different possibilities, to adjust the parameters of the learning algorithms appropriately. This paper is well suited to support the imposition of quality requirements in the second scenario. We present a well-founded evaluation framework for automatic structural assessment of taxonomic ontologies by means of a set of formalised metrics. The paper also formalizes a quality description which is also well suited for use in the other two scenarios.

We present and illustrate an algorithmic methodology for building taxonomies with formally specified content. Although there are some examples of such methodologies in literature which accommodate the psychological/mental/real processes that take place, very few of them provide a formal structure (as was also pointed out in Brewster and O’Hara, 2004, Ganter and Wille, 1999). This is particularly problematic in scenario three identified in (Borst, 1997) (see above). Our formally specified algorithm to fix malformed taxonomies is extremely valuable in settings where taxonomies are developed on the fly by non-computer literate users e.g., as in collecting user preferences (Balke and Wagner, 2004, Chamiel and Pagnucco, 2008). They can play a central role generally in knowledge structures (e.g., The Semantic Web) where every concept in the hierarchy (not only leaves) can be referred to, not only as an abstract concept (probably through the set of properties it contains), but also as a data concept – a concept which can be associated with asserted instances (i.e., real web objects). Various examples of using such structures can be found in recent literature e.g., in Fortuna et al. (2007), Kiefer et al. (2007) and Schickel-Zuber and Faltings, 2006, Schickel-Zuber and Faltings, 2007.

Our focus in this paper is on the generation of precise taxonomic levels with carefully chosen and evaluated inter-level links to connect the most appropriate concepts in the ontology. Various authors noted the importance to analyse taxonomies and their properties, to define ‘goodness’ of taxonomies (Mizoguchi, 2004, Welty and Guarino, 2001). We apply similarity measures between concepts with particular emphasis on their attributes. Although evaluation and generation of taxonomies are important parts of most conceptual models, as they provide substantial structural information, and are key elements in integration tasks, little research effort has been done as to how they affect the development of taxonomies. Simperl et al. (2009) present the results of an empirical survey on ontology development, in which, during 6 months 148 ontology engineering real-world projects were analysed. After calculating the correlation between different aspects of ontology engineering and the associated effort, they conclude that complexity of domain analysis and ontology evaluation are two of the most effort demanding tasks. Therefore any result which eases the difficulties of these two tasks would result in major efficiency gains in the use of taxonomic ontologies. Ontologies are envisioned to be developed by domain experts having limited to no skills in ontology engineering. They conclude it is paramount to provide appropriate techniques and tools which enable the effective and efficient development and evaluation of ontologies. There is a gap between this recognition and the availability of formal and technical tools available and the work here presented contributes to fill in this gap. The rest of this paper is structured as follows: Section 2 presents a formal framework to represent a taxonomy. Section 3 presents a formal framework to represent the quality of the taxonomy. Based on formalisms of Sections 2 Modelling maintainable taxonomies, 3 Modelling and measuring: the quality of a taxonomy, Section 4 presents automatic algorithms for evaluation and construction of a taxonomy. These are illustrated in Section 5 with an example. Finally Section 6 concludes with further discussion of related work and future plans for the research.