Participatory design for ontologies: a case study of an open science ontology for qualitative coding schemas

2.1 Designing ontologies

In accordance to Gruber (1995), Borst (1999) and Studer (1998) Guarino et al. (2009), there are important factors in ontology: The conceptualization, the formal, explicit specification and the “shared” part of it:

1. A “conceptualization” can be seen as an abstract, simplified view of the world that we wish to represent for some purpose (Genesereth and Nilsson, 1987).

2. The “explicit specification” is about choosing the right language and vocabulary for representing the ontology.

3. “Shared” means that this knowledge is agreed on within a domain.

The design of the ontology can be seen as trying to make these concepts explicit in order to understand the concepts people have in mind about a domain of knowledge. Still, these concepts will not all be the same, but it might be possible to find some common ground everyone can agree on. Since these conceptualizations already exist in people's minds, one could argue that the main purpose of creating ontologies might be the moderation of the process, where users explicate their concepts and creating a space for sharing their conceptualizations.

Several approaches exist for designing ontologies (e.g. the frameworks Enterprise (Uschold and King, 1998) or Methonotology (Fernández et al., ,1997)). Still, none of these methods involve users in the whole process of creation; rather they bring users in at one or more stages of the development process. Other approaches have introduced a user-centered evaluation (e.g. Zhang and Li, 2008 or Hu et al., 2018) or interviews with users at the beginning of the process in order to grasp the requirements (e.g. Reinhold, 2015; Lam et al., 2015). The goal of this paper is to go beyond these practices and involve researchers earlier in the process of ontology design using participatory design or codesign methods.

2.2 Participatory design for ontologies

The idea of participatory design or codesign is to involve people affected by change in the design process to secure a higher product quality, but also to foster the acceptance of a new technology. In the digital humanities, several authors have advocated for involving actual users more in research (see Warwick, 2012; Kemman and Kleppe, 2015). It has also been noted that users need to actively articulate their needs for new software, especially in the digital humanities (Borgman, 2009).

In digital humanities and open science projects in general, the future users of the software or system generally have a high level of expertise in their research field and a good understanding of what they need for their research. Therefore, it makes sense to involve them at an early state of research to use their knowledge for improving the product, i.e. software or system (see Warwick, 2012; Kemman and Kleppe, 2015; Womser-Hacker and Heuwing, 2015).

Muller (2007) defines participatory design as the “third space in HCI” and argues for using it to create “new settings and experiences that can assist computer professionals to work in partnership with diverse users in improving both computer technology and the understandings that make computer technologies successful in real use.” (Muller, 2007, p. 3). This can also be combined with the call for creating software so users also improve their handling of the technologies and also lower the barrier to use these new technologies (Borgman, 2009).

Gregory (2003) describes the following goals to be accomplished with participatory design heuristics:

1. Improving the knowledge upon which systems are built;

2. Enabling people to develop realistic expectations and reducing resistance to change;

3. Increasing workplace democracy by giving the members of an organization the right to participate in decisions that are likely to affect their work (Gregory, 2003, p. 2).

It becomes clear that these goals match the ideas proposed by authors in the digital humanities. For the development of an ontology, it can be added that it is not only about improving the knowledge of the system that is built but also it is a way to find out from the researchers about their research in order to create this shared conceptualization of a domain that is needed to describe and formalize it into an ontology.

Several authors have used participatory design practices for creating ontologies:

Palavatsinidis (2014) focuses on feedback rounds with domain experts. These feedback rounds take place at early stages in the design process and can roughly be described as participatory design. However, the approach only involves domain experts at an early stage of the process, whereas actual users are only considered at the end of the ontology design process.

For designing health services, Ongenae et al. (2011) propose a participatory design approach for an ontology based on workshops and observation. Kingsun et al. (2018) also suggest using workshops and observation for creating an ontology in the medical domain.

2.3 Ontology evaluation

The last phase of a conventional design process is the evaluation of the developed ontology. Sabou and Fernandez (2012) name several ways to evaluate ontologies. Many of these focus on technical metrics, but they also mention user-centered evaluation as an important approach to evaluate an ontology. Similarly to the user-centered way of designing the ontology, a user-centered way of evaluation is also fruitful.

The user-centered evaluation approaches use qualitative or quantitative methods to assess ontologies: Expert interviews with feedback options are an example of a qualitative approach (e.g. Zhang and Li, 2008 or Hu et al., 2018). Questionnaires are often used to let users provide quantitative ratings of ontologies. Reinhold (2015) used A/B-testing of a new version of an ontology comparing it to a baseline version. Tan et al. (2017) used usability/usefulness-tests.

A crucial aspect of user-centered evaluation concerns the criteria that are being evaluated. Palavitsinis (2014, p. 65) uses criteria for information quality developed by Lee et al. (2002), based on the information quality criteria by Wang and Strong (1996) to evaluate the developed metadata [1].

However, most of these criteria can only be measured within a system that is already live and contains some real data. This interferes with the approach of testing/evaluation early in order to find problems early thus being able to fix them when this is cheap. Hu et al. (2018) describe a different approach, doing feedback interviews as evaluation of a prototype and deriving criteria based on these interviews [2].

2.4 Summary and outcomes of related works

We can see that the formulation of an ontology and the call from the digital humanities to involve users early at the design process stage of an ontology can be met using methods of participatory design. Reinhold (2015) showed that it makes sense to use expert interviews for requirement analysis. We think this might also make sense for the participatory design since expert interviews allow bringing in a broader range of people than workshops and within the domain of research data it might not be easy for people to convene in one place at the same time for a workshop.

Regarding the evaluation, a qualitative evaluation makes sense, especially when evaluating a prototype that does not have any data in the system yet. The creation of two ontologies for A/B testing is also of limited use if the process before is done via participatory design since this process does not involve creating several versions of the ontology.

3.1 Theoretical background of case study

The case study is situated in the field of open science since it deals with the transparency of scientific results. Furthermore, the work is based on discussions within the community of qualitative social scientists.

Coding schemas are an important part of qualitative research in methods like grounded theory (e.g. Glaser and Strauss, 1967) and qualitative content analysis (Mayring, 2015; Saldaña, 2015; Schreier, 2012). Coding schemas refer to the collection of all codes in a qualitative research project. This includes the codes as well as the relationships between these codes and description of what these codes mean and how they are used. Codes can be based on data, theoretical models or reused from other research. Coding schemas can also be seen as a result of the research, especially in grounded theory or as a tool for data analysis (in qualitative content analysis).

Since open science, FAIR data principles (Mons et al., 2017) and the concept of research graphs (Auer, 2018) have become more important topics also in qualitative research; more effort has been invested in opening up qualitative research. FAIR data describe standards for research data. Accordingly, research data should be findable, accessible, interoperable and reuseable. Research graphs describe the connections of research papers, software and data, whereas the goal is to make more transparent how research data and research software was produced and reused.

There exist several research data centers for qualitative data (e.g. UK data service [3], Forschungsdatenzentum Bildung (Germany) [4] or DANS (Netherlands) [5]), as well as exchange formats to exchange coding schemas [6] between commercial software like NVivo [7] or Atlas.TI [8] (Evers, 2018). Furthermore, there is an ongoing discussion about the quality of qualitative research, which also addresses the traceability of results (Strübing et al., 2018), a topic which can also be supported by the structured description and publication of coding schemas.

3.2 Participatory ontology design for case study

Qualitative research is a field where some foundations have been laid, but as of now there is no shared understanding of the description of qualitative coding schemas. Therefore, a participatory approach is useful. Since this shift to open science is also a transformation process, a participatory design first democratizes the process and gives the members of the community the skills and possibilities to add their feedback (Gregory, 2003). The other goal was to create an ontology that works “in action”, in the practical research process.

As suggested in chapter 2, a participatory ontology design method was used in this case study. Given the current lack of standards in qualitative coding schemas this also makes sense. So far, approaches have been limited to textbooks or the publication of coding schemas by some researchers in the appendix of a thesis or a paper and there is no standardization yet.

The research from this point focused on two aspects: First, creating a description of coding schemas or qualitative research in general. There was no common concept about what can describe a qualitative project and makes it distinguishable from another qualitative research project.

The second aspect concerns formalizing this concept in order to create an ontology based on this knowledge. For this purpose, semantic web standards were taken on as well as principles for data sharing, mainly the FAIR data principles to use these as tools to have an explicit description (Gruber, 1995) of coding schemas in qualitative research.

The method consists of three parts:

1. Requirement analysis to grasp the processes in research and to understand the way research works in qualitative methods.

2. Design phase which allows users to become involved in the process and give feedback to prototypes.

3. Evaluation phase where the developed ontology is evaluated by users.

The following Figure 1 shows an overview of the process and the methods used in each step:

4.1 Preliminary results of applying a participatory design process

This chapter discusses the applicability of the participatory ontology design and the experiences made during the research process. We can see four main results in the participatory design process, i.e. commitment of users, mutual learning, quality of feedback during the process and quality of overall ontology.

During the design phase, it was clear that the participants felt a strong commitment to the project and were happy to be involved in creating standards for “their” research. Participatory ontology design allowed for the users' strong commitment during the interviews, observation and design phases. The further development of the ontology benefitted from the many perspectives and opinions. For instance, users mentioned that better descriptions of coding schemas could also serve teaching of qualitative methods. Given the participants' teaching experience, they can also function as multipliers of the ontology. This can be also seen when the interviewees referred to prior interviews saying things like “we already talked about this before”. During the research project, a shared frame of reference was thus created between the users and the developers of the ontology.

Throughout the whole design process, mutual learning could be seen. For the project manager, the learning of the methods of qualitative research was evident during the whole research process. This is also evident for the interviewees. At the beginning, they had little experience with standardization and metadata. During the design phase, they learned the perspective of the ontology engineer and were also able to add their thoughts to the proposed concepts. At the beginning, the ontology engineer also found it hard to access the field because of a lack of knowledge. Participatory research also helps build capacities regarding the usage of qualitative research methods, so this point of mutual learning was also achieved. The interviews during the design process also showed that the capacities increased on both parts, so the interviews were conducted on a higher level.

The quality of feedback also increased during the process: Interviewees in the design phase were more focused on several points as well as being more specific on what information is needed for the ontology. This makes this approach superior to only involving interviewees in the requirement analysis. It also became evident that when doing interviews in the design phase, the interviews were more and more focused, people had also made their minds up about the ideas as well as remembering things from the previous interviews. The other advantage was that during the design phase the researchers' understanding of the concepts and the ideas improved a lot compared to the initial interviews during the requirements analysis.

The design phase also showed that as soon as the interviewees are given a prototype, they understand the idea better and also give better feedback. This raises an issue. On the one hand, it is sensible to show early prototypes in order to grasp user requirements (see Warwick, 2012). Therefore it is essential to show early prototypes in the process instead of waiting until the end of the process to involve users. On the other hand, the more concrete examples are given, the more concrete the feedback is. We addressed this issue by giving the user as many opportunities as possible in the process phase to give feedback, which got more concrete in the course of development.

Two examples highlight the advantages of user involvement in the participatory design with respect to the quality of the ontology, introducing new ideas: The first example is the possibility to upload a visualization of a coding schema: In grounded theory, it is important to connect codes in a meaningful way. But this is not very well described in textbooks and only a few examples are given, which are – according to the interviewees – not sufficient. Therefore, a metadata visualization was added to get this information and to allow the researchers to upload and publish these figures.

Another example is information about the sampling of a study: in qualitative research sampling is often done via theoretical sampling. But this is not easy to formalize like in quantitative research, where the most important information is whether the sampling was representative. Therefore, a nonformalized field was added, where people can describe how they did their sampling and why.

Two problems persisted when using a participatory design, especially with interviews: contradicting comments of the interviewees and the higher effort involved in the participatory design.

For one, the interviewees submitted contradictory comments. In these cases a compromise was sought during the design process, by showing ideas and then talking to the interviewees again to find out how they felt with respect to their opinion from the last feedback round. Therefore it is important to balance in the design process between interested and positive participants. It is important to find participants who are open to the proposed ideas. Otherwise, the interviews will not get a great outcome. At the same time, this selection might also cross out critical voices that should still be considered. Using theoretical sampling is helpful in hearing different voices. It can also be useful to involve nonmainstream opinions in the research.

One limitation is the higher effort that a participatory and iterative approach needs, but this time is well invested because feedback quality is good in the development process. Qualitative interviews and observations are also time-consuming, yet especially in the case study with researchers not familiar with ontology engineering, this was useful in order to also build capacities. Qualitative methods also may yield contradictive results; therefore, moderating skills are needed to find a compromise.

4.2 Preliminary results on the case study

Another main result of the research is the ontology itself, which can be found on Github [12]. The requirements for the ontology were that it is crucial to add further metadata, e.g. the theoretical background, the type of data and the scientific discipline. The researchers mentioned it should be possible to get an overview of the coding schemas without needing to read the complete methods section of the publication.

The ontology is split into five categories: publications, studies, research data, coding schema and codes. This allows including all information that is important for describing coding schemas. The following Figure 2 shows the structure of the ontology. The relation between the coding schema, research data, project and publications is N:N, which denotes that as many instances of one can relate to the other, e.g. when for one project there were several publications. Only the codes depend on one coding schema, so one coding schema can have an unlimited number of codes, but not vice versa:

Users identified several areas where the ontology can help to make qualitative research more open: while searching for coding schemas, a standard would help find coding schemas and researchers would not have to read complete studies, reusage would also be easier. Publication could also be easier because many coding schemas have not been published yet and owing to a publication, researchers can get credit for their coding schemas, similar to the sharing of data or research software. Furthermore, this process might expand in two other ways, i.e. the validation of other researcher's work and usage for teaching because textbook examples are often rather short.

Two critical points were raised: first, it might be an issue that the amount of work might increase. All except for one interviewee were positive about sharing, but mentioned as a potential problem for many of the researchers. Therefore it is crucial to also involve developers of research software to provide basic tools for this in their software, so researchers can do more of the work during their research. One interviewee also mentioned that she would train her students to use the qualitative methods in a more traceable manner, e.g. she would teach the students to create code-memos when they created a code so they would not have to do this as a time-consuming task at the end. It was also mentioned that a structure of rewards within the scientific community (e.g. to treat the publication of a coding schema as meaningful as the publication of data) is needed.

Another concern raised by one person was the fear of mainstreaming research when publishing schemas. This was described as the problem of researchers just copying codes from other researchers rather than thinking for themselves and questioning if these codes fit the actual research.