An analysis and comparison of keyword recommendation methods for scientific data

To accurately classify vast amounts of scientific data and to allow the desired information to be quickly obtained, it is useful to annotate these data with metadata. Metadata are not data in their own right but rather describe information related to data. The annotation of each distinct datum or dataset with semantic information such as metadata can support the understanding of the data themselves and allow relevant data to be quickly extracted from a vast amount of available data. Examples of metadata for a scientific dataset include the title, creation date, author, data format, abstract text and keywords of an entry. Of these types of metadata, we particularly focus on keywords because of their importance. Viewing annotated keywords enables us to roughly understand the content of the corresponding dataset, to determine the associations between related datasets, and to support the searching, browsing and classification of various datasets. Therefore, keyword annotation of every dataset is very important for building a convenient and useful database of scientific datasets.

In the folksonomy approach to keyword annotation, which is adopted in many social networking service (SNS) systems, any user is allowed to freely annotate various data with keywords [1, 11, 13]. Because many general users are continuously adding keywords to a single dataset or other data object, the advantage arises that the keywords added to that data object will ultimately converge to a useful keyword set based on how the data are used. Unlike in such a folksonomy approach, in the keyword annotation of scientific data, data providers themselves add keywords to their own highly specialized data [4, 9, 19, 22, 24]. In this case, because the data provider is the only person to annotate his own data objects with descriptive keywords provided in their metadata, the utilization value of such keyword sets depends solely on the data providers. Therefore, once the metadata have been defined, their value will not be further improved by general users. Furthermore, in many cases, the keywords that can be added are restricted through the use of a specific controlled vocabulary for the relevant domain. This restriction effectively eliminates noise in data retrieval that may be caused by differences in word form and orthographic variations.

We focus on the latter case, in which data providers annotate their own scientific data, and consider methods for recommending suitable keywords from a controlled vocabulary for the annotation of scientific data. The selection of suitable keywords from a controlled vocabulary requires extensive knowledge of both the research domain and the controlled vocabulary, which typically includes thousands of keywords. Therefore, even an expert data provider will experience difficulty in selecting suitable keywords from the provided vocabulary. Controlled vocabularies exist in various research domains; examples for earth science, agriculture and biology, and the life sciences include GCMD Science Keywords [7], the Centre for Agricultural Bioscience (CAB) Thesaurus,¹ and Medical Subject Headings (MeSH),² respectively. Because it is hard to understand the whole context of a controlled vocabulary, keyword annotation is regarded as a very time-consuming task for data providers.

We investigated metadata for earth science data that are managed by the metadata portal called the Global Change Master Directory (GCMD)³ and the project called the Data Integration Analysis System (DIAS)⁴ [10] as examples. Both the GCMD and DIAS datasets are annotated with keywords from GCMD Science Keywords [7], and the GCMD Science Keywords vocabulary includes about 3000 keywords. Figures 1 and 2 show the distributions of the number of keywords from the GCMD Science Keywords vocabulary with which each dataset is annotated. An investigation of the metadata in the GCMD and DIAS databases revealed many poorly annotated datasets. In the GCMD metadata portal,⁵ the total number of datasets is 32,731, and approximately one-fourth of these datasets have fewer than 5 keywords. In DIAS, among 437 datasets⁶ annotated in English, 220 are annotated with no keywords from the GCMD Science Keywords vocabulary, and the average number of GCMD Science Keywords with which each dataset is annotated is only approximately 3. Therefore, there is a need to increase the number of keywords with which each dataset is annotated by recommending keywords for each dataset. Note that the focus of this paper is keyword recommendation methods, and we are assuming the framework that the final selection of right keywords from recommended keywords is done by data providers.

In this paper, we discuss methods for recommending keywords when data providers create their metadata. We consider the indirect method, which recommends keywords based on similar existing metadata, and the direct method which recommends keywords based on the keyword definitions. This study makes the following three contributions: Below, we describe each of the contributions listed above in greater detail.

Considering metadata quality in keyword recommendation As the first contribution of this paper, we discuss keyword recommendation methods from the viewpoint of metadata quality. The text information that is provided in metadata is often used for keyword recommendation. In general, this text information typically includes an abstract text, which is a free-text description of the data. For example, in the case of an earth science dataset, information regarding the items observed, the observation methods, the usage of the data and so on is provided in the abstract text, and this information is considered to be useful for recommending suitable keywords. Figure 3 shows an example of the abstract text for a dataset managed by DIAS. In this paper, we assume the abstract text in the metadata of a target dataset for which we wish to recommend keywords is available, and we utilize it for the recommendation. Previous studies on keyword recommendation [13, 24] have typically proposed methods for recommending keywords to be added to a target dataset based on those associated with similar existing data by calculating the similarity between the text information in the metadata of the target and in similar existing metadata. We call this method the indirect method. Figure 4 illustrates the underlying concept of the indirect method. However, because this method relies on existing metadata, the effectiveness of the indirect method depends on the quality of the existing metadata set, such as the number of existing datasets with available metadata, the keywords with which the datasets are annotated, and the words contained in the abstract texts. For example, if the datasets that are similar to the target dataset are poorly annotated with keywords, then the indirect method cannot provide useful recommendations even if similar datasets are available in a given portal. In fact, even beyond earth science metadata portals such as DIAS and the GCMD, it has been reported that the quality of the available metadata set is also a pressing problem in Europeana [5], which is the largest-scale digital library portal in Europe. The scope and the quantification of metadata quality as addressed in this paper will be discussed in detail later. The examples given above show that many cases exist in which the existing metadata quality is insufficient.

In this paper, we also consider a keyword recommendation method that does not depend on the quality of the existing metadata set, which we call the direct method. In the direct method, instead of existing metadata information, the definition sentences provided for each keyword are used in combination with the abstract text provided for the target data. Figure 5 illustrates the underlying concept of the direct method. In most cases, each keyword in a controlled vocabulary is associated with a keyword definition that explains its meaning; this is also the case for GCMD Science Keywords. Figure 6 shows an example of the definition of a keyword in GCMD Science Keywords. If a data provider provides an abstract text that includes sufficient information to describe his dataset, which is a target dataset of keyword recommendation, then the direct method can be applied to recommend suitable keywords for the dataset even if the quality of the existing metadata is insufficient. Therefore, if each data provider improves the quality of the abstract text associated with his data, then the direct method can be used to improve the quality of an entire metadata portal by increasing the number of annotated keywords. By contrast, if the existing metadata quality is insufficient, the indirect method will be unable to recommend suitable keywords for a dataset even if the data provider improves the quality of the associated abstract text. Therefore, the indirect method is less likely to increase the number of annotated keywords or to improve the quality of an entire portal. Note that if the existing metadata quality is sufficient, then the indirect method can be effectively applied for keyword recommendation. Hence, we can say the direct method, which is independent of the metadata quality, is preferable while the existing metadata quality is insufficient, and it can contribute to creating an environment in which the indirect method can function effectively.

In this paper, we consider the field of earth science as one possible target research domain. Earth science data include satellite observation data, vegetation index data, data on earthquakes and so on. In earth science, the desire is to integrate these diverse data and apply them in heterogeneous domains. Therefore, to promote the use of such data, we need to support their classification and the ability to extract relevant data through keyword annotation.

Keyword annotation for earth science data Metadata keywords for earth science data are added to a dataset by selecting keywords that are relevant to that dataset from a controlled vocabulary. For example, using the GCMD Science Keywords vocabulary, a dataset containing rainfall observations can be annotated with the keyword “PRECIPITATION AMOUNT.” In addition, as mentioned in Sect. 1.2, each keyword is managed hierarchically in GCMD Science Keywords. For example, consider the top-layer keyword “OCEANS”; below this keyword, “OCEAN TEMPERATURE” is one of the keywords in the second layer, and “SEA SURFACE TEMPERATURE” is a keyword in the next deeper layer. In this paper, we represent these keywords in their hierarchical structure as follows: “OCEANS > OCEAN TEMPERATURE > SEA SURFACE TEMPERATURE.” We provide examples of the hierarchical structures of several GCMD Science Keywords in Fig. 7.⁷

The remainder of this paper is organized as follows. Section 2 introduces previous studies on keyword recommendation for web pages, earth science data and academic research papers. This section also introduces research on metadata quality. Section 3 explains the two methods of keyword recommendation for scientific datasets. One is the indirect method, which relies on an existing metadata set, and the other is the direct method, which relies on the metadata of the target itself and the keyword definitions. In addition, we consider the role of metadata quality in relation to these two methods. Section 4 proposes evaluation metrics that consider a controlled vocabulary with a hierarchical structure. In Sect. 5, we present experiments conducted on real datasets managed by the GCMD and compare the results obtained using the direct and indirect methods by applying our evaluation metrics. Section 6 concludes the paper and discusses possibilities for future work.

Several studies have addressed the use of content-based methods to support social tagging [1, 11, 13]. For social bookmarking services such as Delicious, Lu et al. proposed a method of recommending suitable keywords for a web page lacking tag information [13]. In their method, an assignment probability is calculated for each potential tag for a web page based on the similarities between web pages and how often that tag appears in sets of tags added to similar web pages. However, this approach presupposes that multiple users will annotate a particular web page with the same tags as other users will. Hence, this method cannot be applied to the cases that only the data providers add keywords to their own data. The cited authors also calculated the trustworthiness of a web page based on the total number of tags added to that web page. However, in research domains, the numbers of keywords added to data have little relation to the reliability of those data. Belem et al. proposed a formula for calculating the relevance of each tag for a resource using learning-to-rank technologies, combining various indicators such as tag co-occurrence, descriptive power and term predictability [1]. However, a controlled vocabulary is not used in this approach; instead, recommended keywords are extracted from among all terms present in a document. Krestel et al. suggested tags for a web page by applying latent Dirichlet allocation (LDA) [2] to a set of preassigned tags [11]. This method assumes that less frequent or unique tags that have been added by only a few users are inappropriate. However, in the case of scientific data, such as earth science data, because controlled vocabularies are used, important keywords are often assigned that appear with low frequencies. Therefore, if LDA were to be used in the earth science domain, appropriate but less frequently used keywords would likely not be recommended.

In the context of keyword recommendation for earth science data, Tuarob et al. [24] proposed a method of recommending tags drawn from a controlled vocabulary for data lacking tag information. In this method, the feature vector for each dataset is created from the text information available in the metadata, and tags are recommended to be added to similar datasets by calculating the similarities between feature vectors. Each document is represented by either a term frequency-inverse document frequency (TF-IDF) vector [18] or an LDA probability distribution. However, when the quality of the existing metadata set in a metadata portal is insufficient, this method can be expected to be ineffective. Therefore, we also consider the direct method, which does not depend on an existing metadata set and can be applied to a new controlled vocabulary that has not seen extensive use. Shimizu et al. proposed a method of recommending keywords that represent different earth science categories using Labeled LDA [16, 22]. They defined 14 keywords as labels for learning the correspondence between the abstract text associated with a dataset and the keywords added to that dataset. Then, they recommended suitable keywords by applying the learned results to a target dataset. As demonstrated in that study, when the number of labels is small, labeled LDA is useful for recommendation. However, it is very difficult to prepare sufficient training data to define thousands of keywords as labels.

This section introduces several works that have addressed methods of supporting annotation for academic research papers. Chernyak proposed a method for recommending topics based on a controlled vocabulary called the ACM Computing Classification System⁸ [4]. Using techniques such as TF-IDF, BM25 and annotated suffix trees, this method calculates the similarities between the topics and an abstract text of a paper of interest. Santos and Rodrigues addressed the problem of multi-label classification for research papers using machine learning techniques such as support vector machines (SVMs), the k-nearest neighbor (k-NN) approach and naïve Bayes classification [19]. Certainly, these methods are also applicable to our topic of study. Because they utilize other existing research papers available in a portal, these methods depend on the quality of the existing metadata. Notably, although studies of annotation methods for research papers are different from those for sets of observational data, such as earth science data, in that annotation methods for research papers can utilize reference information, they have much in common with our study in that both types of approaches require a controlled vocabulary and the keywords are added by a specific person, such as the author, in both cases. Our discussion can thus be considered applicable to the annotation of research papers.

Dataset profiles consist of topics for a dataset and their relevance. The researches on dataset profiling proposed approaches based on graphical models [6], machine learning techniques [21] and calculation of cosine similarity [17]. Fetahu et al. [6] proposed an approach for creating dataset profiles through the combination of the topic extraction from reference datasets and the ranking based on graphical models. Schaible et al. addressed the problem of reusing appropriate vocabulary terms to represent linked data using a machine learning technique and a data mining approach. Those methods are based on how other data providers on the LOD cloud have used RDF classes and properties [20]. Ramnandan et al. proposed TF-IDF-based cosine similarity approach. They ranked semantic labels in decreasing order of the cosine of the angle between a target document vector and the other existing document vectors [17]. The target of these studies is general datasets such as the datasets from the LOD cloud, while the target of our study is scientific datasets and keywords from a controlled vocabulary with a hierarchical structure, which are associated with definition sentences. These studies can be considered as indirect methods and thus depend on metadata quality.

In this section, we introduce several studies on metadata quality. To evaluate metadata quality, Bruce and Hillmann introduced the seven metrics of “completeness,” “accuracy,” “conformance to expectations,” “logical consistency and coherence,” “accessibility,” “timeliness” and “provenance” [3]. These metrics indicate whether the metadata include sufficient information to understand the data contents, how accurate the information provided by the metadata is, whether the metadata satisfy user needs, how closely the metadata follow a standard definition, how accessible the data are, whether there are differences between the real data and the metadata because of the passage of time, and how trustworthy the manner in which the metadata were processed is, respectively. Ochoa and Duval [14] proposed formulas for quantifying these seven quality metrics for metadata in a digital library. For example, these authors quantified “conformance to expectations” by calculating the sum of the TF-IDF score of each word in the abstract text or the entropy of each keyword. However, there is no general consensus regarding how metadata quality should be quantified. Tani et al. [23] published a survey paper on the issue of metadata quality. This work summarizes various frameworks for evaluating metadata quality, including that of Bruce and Hillmann [3]. However, the authors state that there is no consensus with respect to the question “What is metadata quality?”. Because metadata quality refers to the complex concept of “fitness for use,” its definition changes depending on the real usage of the data in question. Moreover, these authors explain that it is difficult to define metadata quality in a way that is applicable in all contexts because the understanding and evaluation of metadata are different for each community unit. In this paper, we focus on the number of existing datasets with available metadata, the number of words in the abstract texts and the number of annotated keywords per dataset, which we regard as important aspects of the concept of metadata quality, and we define these quality measures as factors that can influence the performance of methods of keyword recommendation.

We consider that the quality of metadata reflects the degree to which the metadata include sufficient information to characterize the associated data. With regard to an abstract text that is included in the metadata for a dataset, the quality of the abstract text is considered to represent the degree to which it describes the information required to understand the contents of the dataset, such as the process by which the data were obtained and their utility value. In addition, with regard to annotated keywords, the quality of a set of keywords is considered to represent the degree of comprehensiveness with which the keywords express various aspects of the dataset and how useful each keyword is in finding the data.

To verify whether the performance of the indirect method depends on the quality of the available metadata, we consider several existing metadata sets that differ in quality and observe the changes in the values of various evaluation metrics when the indirect method is applied to these different existing metadata sets. To perform such a verification study, it is necessary to establish a standard on which we can judge whether the existing metadata quality is sufficient. However, as mentioned in Sect. 2.5, because the concept of metadata quality encompasses several abstract aspects that cannot be quantified, it is very difficult to calculate some measure of the quality in which all of these aspects are considered. Therefore, we approximate the metadata quality in terms of quantitative measures such as the number of words in the abstract text for each dataset and the number of annotated keywords per dataset, and we then consider several existing metadata sets created to exhibit different levels of quality based on these measures.

In addition, the number of existing datasets for which associated metadata are available can itself be regarded as a measure of the metadata quality. When only a small amount of existing metadata is available in a portal, a particular dataset that suits the specific needs of any given user is less likely to be found in that portal. In this case, the quality of the existing metadata set is considered to be insufficient. Thus, in this paper, we identify the following three measures, \(Q_\mathrm{t}\), \(Q_\mathrm{k}\) and \(Q_\mathrm{n}\), as factors related to the quality of an existing metadata set. This study addresses the case in which, based on the three measures listed above, the quality of the existing metadata set is insufficient; we assume that in such a case, the indirect method of keyword recommendation is ineffective.

In this paper, as the indirect method of keyword recommendation based on an existing metadata set, we adopt Tuarob et al.’s method, which was introduced in Sect. 2.2 [24]. This method is chosen because it was developed for keyword recommendation for earth science data based on textual metadata information. In this method, using the abstract texts contained in the metadata, feature vectors are created for the metadata of a target dataset and for the metadata of existing datasets in a given portal, and these feature vectors are used to calculate the similarity between the target metadata and the existing metadata for each dataset. Then, keywords are recommended from among those used to annotate the similar existing datasets. This process is described in detail below.where \(k (\in K)\) represents a keyword, K is a controlled vocabulary, q is the metadata for a target dataset, D represents the set of existing metadata in a portal, and M represents the method used for feature vector creation. Given K, D and M, \(P(k \mid q, K, D, M)\) represents the probability that q should be annotated with the keyword k. \({ TagScore}_M(k, q, D)\), as calculated using Eq. (2), measures the relevance between the metadata q of the target dataset and the keyword \(k (\in K)\).In Eq. (2), \(DocSim_M(q, d, D)\) calculates the cosine similarity between the metadata \(d \in D\) of an existing dataset and the metadata q of the target dataset. isTag(k, d) is a binary function that returns a value of 1 if the keyword k is present in the existing metadata \(d \in D\) and a value of 0 otherwise. Thus, \(TagScore_M(k, q, D)\) represents the sum of the similarities of the target dataset with all existing datasets that are annotated with a set of keywords that includes k. \(P(k \mid q,K,D,M)\) represents the normalization of \(TagScore_M(k, q, D)\) such that the output will lie on the interval \(\left[ 0,1\right] \), and keywords are recommended in descending order of \(P(k \mid q, K, D, M)\).

In Sect. 2.2, we noted two methods that can be used to create the feature vectors, namely the calculation of TF-IDF vectors and the determination of the topic distribution for the metadata of each dataset. Tuarob et al. reported that the latter method yields more suitable keyword recommendations than the former. Therefore, in the indirect method considered here, we adopt the latter approach, in which the topic distribution of the metadata of each dataset is used as the feature vector for that dataset. In the method of Tuarob et al., the metadata for each dataset are considered to consist of a mixture of topics, and LDA is used to calculate the metadata topic distribution for each dataset. LDA is a technique that is widely used for topic modeling and assumes that a data provider has a particular set of topics in mind when annotating his data, leading the data provider to define the metadata for a dataset by selecting words related to those topics. LDA is used to infer the topic distributions intended by the data provider by analyzing the text information present in a metadata set. In this paper, for consistency with the experimental conditions considered by Tuarob et al., we use the LDA algorithm with 300 topics and 1000 iterations. In addition, when using LDA, we must determine the parameters \(\alpha \) and \(\beta \) that describe the per-document topic distributions and the per-topic word distributions, respectively. No description of the detailed values is provided in Tuarob et al.’s paper, but \((\alpha , \beta ) =(50/L, 0.1)\) is often adopted, where L is the number of labels [8]. Therefore, we adopt \((\alpha , \beta ) =(50/300\simeq 0.167, 0.1)\). We use the Stanford Topic Modeling Toolbox⁹ to apply LDA.T represents the set of topics; thus, \(|T| = 300\). \(t_j \in T\) is the probability of assigning topic j to the metadata q of the target dataset. In this way, the indirect method treats the topic distribution of the metadata associated with each dataset as the feature vector for that dataset and recommends keywords by calculating the similarity between the target dataset and each existing dataset as indicated by their associated metadata.

In this paper, we use the abstract texts contained in the metadata associated with datasets of interest as a basis for the direct method of keyword recommendation. By viewing the abstract text, a user can gain a rough understanding of the content of a dataset. Initially, using a simple string matching method, we extracted keywords from the abstract text associated with a dataset in DIAS by matching the words present in the text with each keyword in the GCMD Science Keywords vocabulary. Note that we preprocessed the abstract texts and the keywords by removing stopwords and stemming each word. However, although the average number of annotated keywords per dataset in the GCMD is approximately 10, the string matching method was only able to recommend an average of approximately 2.7 keywords per dataset. Therefore, we decided to utilize the implicit information provided by the keyword definitions in addition to the explicit information of the keyword names. The GCMD Science Keywords vocabulary includes a definition for each keyword. We calculate the similarities between the abstract text provided in the metadata of a target dataset and the definition of each keyword, and we then recommend keywords with a high degree of similarity.

To calculate the similarities, we create feature vectors for the metadata of the target dataset and each keyword definition. Note that we preprocess the abstract texts and the keyword definitions by removing stopwords and stemming each word. We represent the abstract text associated with a target dataset by \(A_i\), and we represent the set of words contained in \(A_i\) by \(T(A_i)\). In addition, we represent the definition of a keyword by \(D_j\) and represent the set of words contained in \(D_j\) by \(T(D_j)\). Then, we represent a feature vector of \(A_i\) by \(\mathbf{DA} (A_i)\), and a feature vector of \(D_j\) by \(\mathbf{KD} (D_j)\). Each element of \(\mathbf{DA} (A_i)\) and \(\mathbf{KD} (D_j)\) is the weight for the word \(t_l \in T(A_i) \cup T(D_j)\), and thus, the sizes of both \(\mathbf{DA} (A_i)\) and \(\mathbf{KD} (D_j)\) are \(|T(A_i) \cup T(D_j)|\). For \(\mathbf{DA} (A_i)\), the weight of \(t_l\) represented by \(DA(t_l)\) is whether \(t_l\) appears in \(A_i\) or not.For \(\mathbf{KD} (D_j)\), the weight of \(t_l\) represented by \(KD(t_l)\) is the TF-IDF value of \(t_l\). For the term frequency (TF), we use the Length Regularized TF (LRTF) introduced in [15]. These quantities are described in detail below.Let C be the set of keyword definitions, where \(D_j \in C\), and let |C| be the number of keywords. In addition, \(\hbox {len}(D_j)\) is the length of keyword definition \(D_j\), ADL(C) is the average of all \(\hbox {len}(D_j)\), and \(\hbox {TF}({t_l}, D_j)\) is the appearance frequency of word \({t_l}\) in \(D_j\). The LRTF is calculated using a formula that normalizes the TF value by considering the ratio of \(\hbox {len}(D_j)\) to ADL(C). We consider the LRTF to be appropriate in this situation, in which an abstract text is treated as the object of a query, because it is stated in [15] that the LRTF is useful for long queries consisting of more than 5 words. The inverse document frequency (IDF) is calculated using the standard formula, in which |C| is divided by \(DF({t_l}, C)\). In this paper, we utilize the cosine similarity, which is generally used when comparing documents in vector-space models. The cosine similarity takes values in the interval \(\left[ 0,1\right] \), and it represents the angle formed by two vectors: a value close to 1 indicates that the vectors are similar to each other, whereas a value close to 0 indicates that they are independent of each other. We recommend keywords in descending order of cosine similarity. The calculation is presented in detail below.

In this section, we explain the need for our evaluation metrics by considering an example in which we use GCMD Science Keywords as the controlled vocabulary. As mentioned above, the keywords in a controlled vocabulary are often managed hierarchically. Figures 8 and 9 show the hierarchical structure of a controlled vocabulary, in which each node represents one keyword. In Fig. 8, we represent the correct keywords in the controlled vocabulary for a certain dataset as red bordered nodes. In Fig. 9, we present two patterns of recommended results for this dataset, in which each recommended keyword is indicated by a yellow node. In Result (1), three correct keywords are recommended, all in the top layer, whereas Result (2) also contains a total of three correct keywords, but they include keywords in the lower layers. In terms of precision or recall, both of these results earn the same evaluation score of 50%. However, we cannot consider both sets of recommendations to be of equivalent value. Although Result (1) provides correct keywords, which represent the earth science categories “ATMOSPHERE,” “BIOSPHERE” and “AGRICULTURE,” these keywords are easy for a data provider to select and also have many descendants. Therefore, the recommendation provided by Result (1) cannot significantly reduce the cost of keyword annotation. By contrast, Result (2) recommends correct keywords in lower layers, such as “HEAT FLUX.” The selection of such keywords requires extensive knowledge of both the domain and the controlled vocabulary. Therefore, because it recommends keywords that are more difficult to select, Result (2) is considered to reduce the cost incurred by the data provider to a greater extent. Considering that the cost differs depending on the positions of the keywords in the controlled vocabulary, these two sets of recommendations should not be regarded as equivalently valuable, even though their evaluation scores in terms of commonly used metrics such as precision and recall are identical. To differentiate between such recommendations, we must use evaluation metrics that consider a hierarchical vocabulary structure and place greater emphasis on keywords that are more difficult to select.

As an example of a recommendation evaluation metric that considers a hierarchical structure of the items to be recommended, we can consider \({ nxCG}\) [12]. \({ nxCG}\) (Normalized eXtended Cumulated Gain) is an evaluation metric for the retrieval of XML documents. In this context, the items recommended in search results can include the \({ Section}\), \({ Paragraph}\) or \({ Sentence}\) of interest or even divisions with greater granularity. The \({ nxCG}\) value represents the relative cumulated gain achieved by an actual system compared with that achieved by an ideal system. The features of \({ nxCG}\) include the ability to consider partially correct items and the overlaps between recommended items in a search result.

In the case of GCMD metadata used in our experiments, the average number of words in the abstract text is approximately 112 words; there are approximately 10,000 datasets whose abstract texts contain fewer than 50 words, including 6000 datasets whose abstract texts contain fewer than 25 words. Figure 15 shows the distribution of the number of words in the abstract text in the GCMD metadata set.¹⁰ Thus, if an abstract text contains fewer than 50 words or 25 words, or approximately one half or one quarter of the average number of words in the abstract text in the GCMD metadata set, respectively, then we consider the metadata quality for the corresponding dataset to be insufficient from the perspective of the abstract text. By contrast, if an abstract text contains more than 200 words, or approximately twice the average number of words in the abstract text in the GCMD metadata set, then we consider the corresponding metadata to include sufficient information to understand the contents of the associated dataset. We also consider the number of annotated keywords per dataset. For example, if fewer than 5 keywords are used to annotate a dataset (for which at least 10 keywords should be required), then the metadata quality for that dataset is considered to be insufficient from the keyword perspective. By contrast, considering the fact that most datasets in the GCMD are annotated with fewer than 10 keywords as we can see in Fig. 1, if a dataset is annotated with approximately 10 keywords, then those keywords can be considered to cover, to some extent, multiple aspects of the dataset.

Here, we explain the process used to create existing metadata sets that differ in quality for our experimental evaluation. To consider all cases in which the quality is either sufficient or not in terms of each of the three quality measures introduced in Sect. 3.1, namely \(Q_\mathrm{t}\) (average number of words in the abstract text), \(Q_\mathrm{k}\) (average number of annotated keywords) and \(Q_\mathrm{n}\) (number of existing datasets), we constructed one existing metadata set with each of the \(2^3\) corresponding patterns. Table 3 lists the created sets \(\hbox {MS}_1\) to \(\hbox {MS}_8\) and their characteristics as well as the number of datasets (metadata) assigned to each set. First, because we can observe most datasets in the GCMD are annotated with fewer than 10 keywords, we consider datasets that are annotated with more than 10 keywords to have sufficient quality in terms of keywords. Thus, to create a metadata set with an insufficient number of keywords, we randomly removed keywords from the metadata of these datasets until each had fewer than 5 keywords, which is less than half of the average number. Second, to obtain metadata sets with sufficient and insufficient quality in terms of the abstract text, we divided the datasets into those whose metadata contained abstract texts of fewer than 50 words and more than 200 words, respectively. Finally, by randomly selecting the metadata associated with 100 datasets from each metadata set created as described above, we created metadata sets with insufficient quality in terms of the number of existing datasets for which metadata are available. Thus, we created 8 metadata sets to serve as references for recommending keywords for a target dataset.

In this section, we discuss the variations in the evaluation values observed among the results for the different existing metadata sets and describes the influence of each of the three considered quality measures on the performance of the indirect method.

To consider the possibility that the values of the evaluation metrics may vary depending on the metadata quality of the target dataset, we prepared five target metadata sets, each consisting of metadata for 100 target datasets and one each with abstract texts consisting of fewer than 50 words, 51–100 words, 101–150 words, 151–200 words and more than 200 words. To ensure reliability, each target dataset was annotated with more than 10 keywords, and there was no overlap between the target metadata sets and the existing metadata sets.

We regarded the keywords that have been added to each target dataset in the GCMD as the set of correct keywords for that dataset. We evaluated the top 10 keywords recommended by the indirect method and by the direct method using the proposed evaluation metrics. Figures 16, 17, 18, 19, 20, 21, 22 and 23 show the results of applying both methods to the target metadata sets. Figures 16, 18, 20 and 22 present the results obtained using \(\hbox {MS}_1\) to \(\hbox {MS}_4\), in which \(Q_\mathrm{k}\) is large. Figures 17, 19, 21 and 23 present the results obtained using \(\hbox {MS}_5\) to \(\hbox {MS}_8\), in which \(Q_\mathrm{k}\) is small. The horizontal axis represents the target metadata set, and the vertical axis represents the average evaluation value.

The number of words in the abstract text For each of Figures 16, 17, 18, 19, 20, 21, 22 and 23, we discuss the influence of \(Q_\mathrm{t}\) on the performance of the indirect method. Because we should compare existing metadata sets for which the number of datasets with available metadata is comparable, we consider the changes in the evaluation values between the following pairs of existing metadata sets: (\(\hbox {MS}_1\), \(\hbox {MS}_3\)), (\(\hbox {MS}_2\), \(\hbox {MS}_4\)), (\(\hbox {MS}_5\), \(\hbox {MS}_7\)) and (\(\hbox {MS}_6\), \(\hbox {MS}_8\)). Except in the case of the target datasets with abstract texts of fewer than 50 words, the evaluation values obtained using \(\hbox {MS}_3\), \(\hbox {MS}_4\), \(\hbox {MS}_7\) and \(\hbox {MS}_8\) tend to be higher than those obtained using \(\hbox {MS}_1\), \(\hbox {MS}_2\), \(\hbox {MS}_5\) and \(\hbox {MS}_6\). This finding demonstrates that \(Q_\mathrm{t}\) influences the performance of the indirect method. Moreover, the results indicate that as the number of words in the abstract text of the target dataset increases, the differences in the evaluation values due to \(Q_\mathrm{t}\) become larger. This behavior can be explained as follows: if the metadata of the target dataset include sufficient information for the recommendation of suitable keywords, then the recommendation accuracy improves in proportion to an increasing amount of information available in the existing metadata set. From this result, it can be recommended that a data provider should provide an abstract text with a sufficient description of the corresponding dataset.

In the case that the abstract texts in both the existing and target metadata sets consist of fewer than 50 words, however, the evaluation values are significantly higher. To determine the cause of this phenomenon, we investigated the corresponding datasets. We found that for many of the target datasets, their abstract texts contained descriptions that were nearly identical to those in the existing metadata. For example, some data providers produce multiple abstract texts from the same template by simply replacing the names of the observation satellite, the research vessel and the observation location. The metadata of these datasets are likely to have been mass produced through copying and pasting, and the annotated keywords for these datasets are also nearly the same. Thus, the evaluation values of the indirect method become high as it can utilize the information of nearly identical metadata in the existing metadata sets. Such mass production of metadata provides an additional indication that the cost incurred by data providers for metadata annotation is very high.

The number of annotated keywords To investigate the influence of \(Q_\mathrm{k}\), we consider the changes in the evaluation values between the following pairs of existing metadata sets: (\(\hbox {MS}_1\), \(\hbox {MS}_5\)), (\(\hbox {MS}_2\), \(\hbox {MS}_6\)), (\(\hbox {MS}_3\), \(\hbox {MS}_7\)), and (\(\hbox {MS}_4\), \(\hbox {MS}_8\)). We can observe the changes by comparing lines of the same color between Figs. 16 and 17,  18 and 19,  20 and 21, and  22 and 23. The results show that when \(Q_\mathrm{k}\) is small, the evaluation values tend to be lower; however, \(Q_\mathrm{k}\) has less influence on the results of the indirect method in the cases of existing metadata sets with small \(Q_\mathrm{t}\) (\(\hbox {MS}_1\), \(\hbox {MS}_2\), \(\hbox {MS}_5\), and \(\hbox {MS}_6\)). This finding indicates that when \(Q_\mathrm{t}\) is small, the recommendation accuracy cannot be improved by increasing \(Q_\mathrm{k}\). From these results, we conclude that \(Q_\mathrm{k}\) affects the results, but \(Q_\mathrm{t}\) should receive greater emphasis.

The number of existing datasets with available metadata In each of Figures 16, 17, 18, 19, 20, 21, 22 and 23, we consider the influence of \(Q_\mathrm{n}\) on the performance of the indirect method. Because we should compare existing metadata sets of comparable \(Q_\mathrm{t}\), we consider the changes in the evaluation values between the following pairs of existing metadata sets: (\(\hbox {MS}_1\), \(\hbox {MS}_2\)), (\(\hbox {MS}_3\), \(\hbox {MS}_4\)), (\(\hbox {MS}_5\), \(\hbox {MS}_6\)) and (\(\hbox {MS}_7\), \(\hbox {MS}_8\)). The results show that the evaluation values obtained using \(\hbox {MS}_2\), \(\hbox {MS}_4\), \(\hbox {MS}_6\) and \(\hbox {MS}_8\) tend to be higher than those obtained using \(\hbox {MS}_1\), \(\hbox {MS}_3\), \(\hbox {MS}_5\) and \(\hbox {MS}_7\). This indicates that \(Q_\mathrm{n}\) affects the results of the indirect method. If \(Q_\mathrm{n}\) is small, then the number of comparisons considered when calculating the similarities is insufficient, resulting in a low recommendation accuracy. In addition, one of the reasons for this result is that \(Q_\mathrm{n}\) directly affects the accuracy of the topic distribution obtained via LDA, which is a kind of machine learning technique.

Discussion The results presented in this section show that all three quality measures considered in this study influence the results of the indirect method. Therefore, the performance of the indirect method tends to depend on the quality of the existing metadata set, and there is a high likelihood that the indirect method cannot offer reliable accuracy and comprehensiveness, especially when the existing metadata quality is insufficient. When \(\hbox {MS}_4\), whose quality is sufficient in terms of all three measures, is used as the reference for the indirect method, the evaluation values tend to be highest. By contrast, when \(\hbox {MS}_5\), whose quality is insufficient in terms of all three measures, is used as the reference, the evaluation values tend to be lowest. These observations provide an additional indication that the metadata quality influences the performance of the indirect method. As mentioned in Sect. 1, the quality of the metadata registered in DIAS can generally be regarded as insufficient. Therefore, it can be expected that if we were to use this metadata set as the existing metadata set for keyword recommendation, the indirect method would be ineffective.

In this section, we compare the direct and indirect methods. First, in the case that the abstract texts in both the existing and target metadata sets consist of fewer than 50 words, the evaluation values obtained using the direct method are lower than those obtained using the indirect method. The cause of this phenomenon is described in Sect. 5.2. With the exception stated above, the results in Figs. 16, 17, 18, 19, 20, 21, 22 and 23 show that the evaluation values achieved using the direct method are higher in most evaluation metrics than those achieved using the indirect method. To determine the reasons for this behavior, we investigated the results obtained by applying these two methods to several datasets. We found that regardless of the characteristics of the existing metadata set used, the indirect method tends to recommend keywords from the top layer of GCMD Science Keywords. This is because Eq. (2), which describes the keyword relevance calculation used in the indirect method, depends on the frequencies of the keywords with which the existing datasets are annotated, and keywords in the top layer appear more frequently. By contrast, our evaluation metrics assign a larger basic gain to keywords that are more difficult for data providers to select. Therefore, when the proportion of the correctly recommended keywords that are located in the top layer is higher, the evaluation values will inevitably be lower. For the recommendation of top-layer keywords in a controlled vocabulary with a hierarchical structure, the indirect method is likely to be more appropriate than the direct method, but it cannot support the selection of keywords in lower layers, which are more difficult to select. In other words, the indirect method cannot effectively reduce the cost incurred by data providers for keyword annotation. By contrast, the direct method is independent of the frequencies of the annotated keywords in the existing metadata because those metadata are not used. Therefore, the direct method tends to recommend more keywords in lower layers, which are more difficult to select, and can reduce the cost of keyword annotation to a greater extent. To illustrate this behavior, as shown in Figs. 24 and 25, we calculated the proportions of the correctly recommended keywords belonging to each hierarchical depth for each method. Because of space limitations, only the ratios among the top three hierarchical depths are shown in Fig. 24, which presents the ratios found when using \(\hbox {MS}_7\).

Notably, the evaluation values achieved using the direct method are often higher than even those achieved using the indirect method with \(\hbox {MS}_4\), whose quality is sufficient in all three considered measures. This finding indicates that the indirect method, which depends on the keyword frequencies, has a tendency to recommend keywords in the top layer even when the existing metadata quality is sufficient.

Finally, the results of the direct method in Figs. 16, 17, 18, 19, 20, 21, 22 and 23 show that when the abstract texts in the target metadata contain fewer than 50 words, the evaluation values are the lowest among all target metadata sets. This observation demonstrates that the keyword recommendation accuracy that is achieved by the direct method depends on the number of words in the target abstract text. Therefore, we recommend that data providers should be sure to provide abstract texts that contain sufficient information to adequately describe their datasets.

To support keyword annotation for data in various research domains, we have discussed methods of recommending keywords drawn from a controlled vocabulary. We believe that the indirect method of keyword recommendation is likely to be ineffective when the quality of the existing metadata set is insufficient. By contrast, the direct method utilizes the definition of each keyword in addition to the abstract text provided in the metadata associated with the recommendation target, and therefore, it does not depend on the quality of an existing metadata set. To verify the effectiveness of the direct and indirect methods, we created several metadata sets of differing quality in terms of the number of existing datasets with available metadata, the length of the abstract texts and the number of annotated keywords per dataset. We observed the changes in various evaluation values that occurred when these different datasets were used as the existing metadata set for keyword recommendation, and we showed that the performance of the indirect method depends on the quality of the existing metadata set. This observation shows that it is important to consider metadata quality when we adopt keyword recommendation methods. Additionally, we proposed evaluation metrics that consider a hierarchical vocabulary structure, which is a distinctive feature of most controlled vocabularies. Using our evaluation metrics, we evaluated the extent to which the cost incurred by data providers for keyword annotation is reduced through keyword recommendation. In our experimental evaluations, we compared the indirect method with the direct method using the proposed metrics. The results show that the indirect method tends to recommend keywords in the highest layer, which are easy to select, whereas the direct method tends to recommend keywords in lower layers, which are more difficult to select.

In this study, we considered keyword recommendation using only the keywords that are included in GCMD Science Keywords. In the future, we wish to investigate other controlled vocabularies in the earth science domain. We are also interested in applying our analysis to various other research domains, such as agriculture, chemistry, biology and computer science, in addition to earth science. Several such controlled vocabularies are available, including the CAB Thesaurus, the ACM Computing Classification System, and MeSH, which are relevant to agriculture, computer science and the life sciences, respectively. Each controlled vocabulary has a hierarchical structure, in which each domain is further subdivided into more specific domains. Notably, the direct method requires keyword definitions, but not all controlled vocabularies include keyword definitions. Therefore, by creating a definition for each keyword from words related to that keyword, we wish to improve the direct method to make it suitable for application to a controlled vocabulary that does not include keyword definitions.

Moreover, we would like to improve the direct and indirect methods with more consideration on a hierarchical vocabulary structure. In this paper, we used simple implementations for the direct and indirect methods in order to focus on the discussion of metadata quality. It is possible to handle each keyword considering the position of the keyword in the controlled vocabulary. Also, we are interested in integrating the direct and indirect methods. Because both of these methods are based on similarity calculations, the assignment probability of each keyword for a recommendation target can be acquired as a linear combination of the similarities calculated using the two methods. For this purpose, we need to define a parameter \(\alpha \) to indicate how much emphasis should be placed on each method. The value of this parameter \(\alpha \) should depend on the quality of the existing metadata set that is available for reference. In addition, we can consider utilizing the concept of keyword co-occurrence. In the earth science domain, sets of two or more keywords can be identified that tend to co-occur. For example, when researchers observe wind intensity, they also simultaneously observe wind velocity. However, an approach that relies on keyword co-occurrence will necessarily depend on the quality of the existing metadata set.

Finally, in our future work, we would like to incorporate our approach into a real metadata management system and obtain feedback on its performance from researchers and experts in earth science. Because we are assuming the framework that the final keyword selection from the recommended keywords is done by data providers, we would like to consider the significance of keyword recommendation with the keyword selection process.