ETM: Enrichment by topic modeling for automated clinical sentence classification to detect patients’ disease history

EHR data contain a large amount of text in which useful patterns need to be automatically identified. Machine learning and text mining algorithms with different data representation methods have been used to study the classification of clinical notes. Mujtaba et al. (2019) presented a comprehensive review of articles on clinical text classification published in 2013–2018. Based on their study, the most extensively employed clinical texts for classification are pathology reports, radiology reports, and Medline biomedical documents. In a majority of studies, bag of words (BOW) representations: binary, term frequency, and TFiDF feature representations were determined to be beneficial. A significant number of the studies have used either supervised machine learning or rule-based approaches.

Many approaches to clinical text classification rely on medical ontologies (dictionaries), such as the unified medical language system (UMLS) meta-thesaurus, and medical subject headings (MeSH), to glean knowledge from clinical notes. Yao et al. (2019) proposed an approach that combines rule-based features and a knowledge-guided convolutional neural network for effective disease classification. They used concepts from the UMLS meta-thesaurus. Similarly, Kocbek et al. (2016) combined three clinical reports—from pathology, radiology, and patients’ admission-related meta-data—and used a support vector machine (SVM) with a bag of phrases from the UMLS meta-thesaurus to predict the rate of admissions against disease.

On the contrary, some clinical text classification studies have used non-dictionary-based approaches instead of dictionary-based methods. For instance, Bui and Zeng-Treitler (2014) applied regular expressions to extract snippets of text from clinical notes containing specific words and built an SVM classifier to categorize them. Fodeh et al. (2018) used unstructured text narratives in the EHR to derive pain assessments from clinical notes on patients with chronic pain. They developed their system based on different machine learning classifiers, among which random forest achieved the best results. Blanco et al. (2019) used several deep learning classification models for assigning multiple ICD codes to clinical documents. They implemented binary logistic regression, a neural network with three fully connected hidden layers, and a bidirectional gated recurrent unit for text classification.

Nevertheless, the problem of clinical sentence classification was not covered in work by Mujtaba et al. (2019), because a few studies have sought to derive the knowledge hidden in clinical short notes (Friedman et al. 2004; Cao et al. 2017; Mujtaba et al. 2019; Hughes et al. 2017; Lv et al. 2016). Hughes et al. (2017) applied convolutional neural networks (CNNs) with a distributed word representation to medical text classification at the sentence level. They evaluated the learning of complex data representations using the algorithm instead of feature engineering for clinical knowledge representation. Lv et al. (2016) used sentence segmentation, word segmentation, part of speech and entity extraction for text preprocessing to extract features for short text classification in EHRs. In their approach, TFiDF and latent semantic analysis are used to select features that represent the vocabulary for short text classification from several entity dictionaries. In addition, a dependency parser is applied to texts where the dependency relations are used as features for text classification. Cao et al. (2017) proposed a knowledge-guided short text classification system for healthcare applications, and claimed that text in the healthcare domain contains domain-specific or infrequently appearing words that can lead to poor embedding owing to a lack of training data. They proposed a bidirectional long short-term memory deep neural network to perform short text classification tasks. Their approach is a domain knowledge-guided attention model that uses the domain dictionary at hand to refine classification performance.

The main difference between the above studies on clinical text classification and our approach is that the former studies used domain dictionaries and disregarded the unlabeled data. Our approach uses the unlabeled data for the unsupervised model-based smoothing, and deploys the labeled data for the sentence classification model.

Impressive progress has been made on the problem of text classification, but few studies have tackled sentence classification (Kozlowski and Rybinski 2019; Zelikovitz and Hirsh 2000; Cheng et al. 2014; Yin et al. 2017; Khoo et al. 2006; Kim 2014; Jurafsky and Martin 2019; Aggarwal 2018). Unlike the traditional text classification problem, sentence classification pose two main challenges. First, patterns of word co-occurrence are sparse in the feature space, where a sentence contains only several to a dozen words. Second, texts face the challenge of a large-scale and manual labeling task, where with sentences this task is more burdensome as they are very small samples causing to increase noise and reduce classification accuracy.

Several techniques have been proposed to tackle the challenges posed by sentence classification, including dimension reduction (Zelikovitz and Hirsh 2000; Sriram et al. 2010; Khoo et al. 2006; Bollegala et al. 2018), topic modeling (Cheng et al. 2014; Chen et al. 2011; Yang et al. 2015), clustering (Kozlowski and Rybinski 2019; Yin et al. 2017; Bollegala et al. 2018; Dai et al. 2013; Kozlowski and Rybinski 2017; Yang et al. 2019), and word embedding (Kozlowski and Rybinski 2019; Kim 2014; Lee and Dernoncourt 2016; Hill et al. 2016). Kim (Kim 2014) proposed a single layer of CNN applied for sentence classification. He concluded that despite little tuning of hyperparameters, unsupervised pre-training of word vectors is an important ingredient in deep learning for sentence classification. Zelikovitz and Hirsh (2000) developed a method to reduce error rates in short text classification by using a combination of labeled training data plus a large body of “uncoordinated background knowledge” that is a secondary corpus of unlabeled but related longer documents. They used the WHIRL method (Cohen 1998) for text classification, an information integration tool designed to query and integrate varied sources of text from the Web. Sriram et al. (2010) proposed an intuitive approach to classify the short texts in tweets by using author information and features of texts. Yin et al. (2017) proposed a short text classification technique based on a combination of the K-nearest neighbors (KNN) and hierarchical SVM classification. They used KNN to initially group labels of the samples to create subclasses and then they applied a SVM algorithm as a hierarchical multi-class classification to each group to classify labels. Cheng et al. (2014) proposed a biterm topic model to capture topics in short texts based on aggregated biterms in the entire corpus to tackle the sparsity of patterns of word co-occurrence in texts. They defined the biterm as an unordered word pair co-occurring in a short text. They considered the corpus as a mixture of topics, where each biterm is drawn independently from a specific topic. Yang et al. (2015) proposed a topic model to extract key phrases for short text classification using the idea that knowledge incorporation can solve the problem of sparsity. Their approach extracts topics from texts by focusing on phrases in the generative process of documents. Bollegala et al. (2018) developed ClassiNet, a network of binary classifiers trained to predict missing features from a given short text for text classification. ClassiNets solves the problem of feature sparseness by generalizing word co-occurrence graphs by considering implicit co-occurrences between features. Dai et al. (2013) proposed the Crest to generate topic clusters from training data by exploiting a clustering method. Crest uses topic information to extend the representation of short texts and define a new feature space. It subsequently measures the cosine similarity between a document and clusters as augmented features of the document for classification. Lee and Dernoncourt (2016) presented a model on the basis of recurrent and convolutional neural networks. Their model incorporates preceding short texts for sequential short text classification. This model comprises two parts. The first part generates a vector representation for each text and the second part classifies the vector representations of the current text as well as a few preceding short texts using a two-layer feed-forward neural network. Kozlowski and Rybinski (2019, 2017) used a neural network-based distributional model for enriching the semantic meaning of short texts for clustering. They proposed the SnSRC clustering algorithm that uses the SnS method (Kozlowski and Rybinski 2017), a knowledge-poor text mining algorithm to sense induction, a language-independent approach. They trained their model using continuous bag of words and negative sampling, and computed cosine similarity between the mean vector of the embeddings for the text and the vectors for each word in the distributional model. The retrieved words with the highest semantic similarity were added as additional term features to the initial BOW text representation. In their study, especially in cases involving a specific domain language, the semantic enrichment of texts by applying neural networks improved the quality of clustering. Hill et al. (2016) overcame feature sparseness in sentence representations by embedding them into a low-dimensional, dense space. They compared deep neural language models that compute sentence representations from unlabeled data with prevalent methods for word representation, and concluded that the unsupervised BOW models delivered the best performance in terms of sentence representation compared with supervised ones.

Current methods for sentence classification and short text classification either represent texts in a lower-dimensional space to reduce feature sparseness or add data to the text to enhance the quality of the feature space. The main outstanding challenge is the construction of external knowledge repositories, a labor-intensive task in applications of domain-specific clinical text mining. We propose an approach to tackle this challenge in clinical sentence classification that deploys an unsupervised scheme for enriching the original data set by internal knowledge acquisition, where the length of each document is considered by a dynamic weighting mechanism. The proposed approach uses the output of the unsupervised scheme as an internal source for enriching that does not employ any external dictionary.

DEDUCE (Menger et al. 2018), a pattern matching tool, is used for automatic de-identification of Dutch medical texts, to anonymize clinical notes for legal and privacy reasons. De-identification process removes patients-level private data that comprise names of patients, names and identification of nurses and doctors, addresses and dates. All texts are then tokenized using NLTK library (Bird et al. 2009) and the Python scikit-learn (Pedregosa et al. 2011) feature extractor. NLTK sentence tokenizer uses an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences; and then uses that model to find sentence boundaries. This approach has been shown to work well for many European languages (Bird et al. 2009). The punctuation marks that are used to separate sentences in our case study are period, question mark, exclamation point, and semicolon. To handle spelling errors in Dutch texts, the Python package language-check¹ is used, which is a wrapper for the LanguageTool² package. LanguageTool is an open source proofreading software that can detect and correct spelling errors in more than 20 different languages. The effect of the spell-checker on sentence classification is not evaluated in this study and will be discussed in detail in future work.

Each clinical sentence (document) from the data set is represented by a normalized V-dimensional vector weighted by the TFiDF measure. TFiDF is a BOW representation model that stands for term frequency—inverse document frequency, and is defined as follows: where V is the size of the vocabulary, n_d,i denotes the number of times the ith word appears in the dth document, |C| denotes the total number of documents in the data set, and |C_i| is the number of documents containing the ith word. TFiDF evaluates how important a word is to a document in a data set, where the importance increases proportionally to the number of times a word appears in the document but is offset by the document frequency of the word in the data set. Thus, with this representation, each document in the data set can be regarded as a multinomial distribution over V words, and each dimension reflects the semantic coherence between the ith word and dth document.

Topic modeling is a way of discovering topics in unlabeled text data (Cheng et al. 2014; Blei et al. 2003). The LDA is a generative topic model that represents documents as a mixture of topics and assigns certain probabilities to the words. In other words, the LDA model is an unsupervised learning method that seeks patterns by inferring hidden variables in texts by treating words as observations.

Given a document in the form of (w₁,w₂,...,w_N), and K asked-for topics, the LDA model estimates parameters 𝜃 and β. 𝜃 is the distribution of hidden topics in each document. β is the probability of each word given the topic. Figure 2 shows a graphical representation of the LDA model (Blei et al. 2003), where the nodes are random variables and the edges indicate the conditional dependencies between them. The shaded and unshaded variables indicate observed and latent (i.e., unobserved, hidden) variables, respectively, while the plates refer to repetitions of the steps of sampling with the variable in the lower-right corner referring to the number of samples. As Fig. 2 shows: the parameter α is a data set-level Dirichlet prior that can be interpreted as the prior number of observations of a topic being sampled in a document before having observed any words from the document. Similarly, parameter η is a data set-level Dirichlet prior that can be interpreted as the number of prior observations of words sampled from a topic before any word from the data set is observed. These two parameters are assumed to be sampled once in the LDA model when generating a data set of documents.

The variable 𝜃_d is a document-level variable that is sampled once for each document. 𝜃_d is the distribution of hidden topics in the dth document based on a multinomial distribution with the Dirichlet parameter α. The variable β_k is a topic-level variable that represents the probability distribution of words in topic k. Variables Z_d,n and W_d,n are word-level variables that are sampled once for each word in each document (n ∈{1,2,...,N}). The variable Z_d,n is a topic generated by a multinomial distribution with the parameter 𝜃, and variable W_d,n is a word sampled from the multinomial distribution with parameters β and Z.

The process of the LDA-clustering algorithm (Blei et al. 2003) implies a joint distribution over the latent and observed random variables (W,Z,β,𝜃) defined as follows: Standard statistical techniques can be used to invert the generative process of the LDA model, thus inferring the set of topics responsible for generating a collection of documents. To use the LDA, the key inferential problem to solve is that of computing the posterior distribution of the hidden random variables given the observed words in a document. This posterior distribution is defined in (3). This posterior distribution is intractable to compute, and thus approximate inference algorithms are needed for the posterior estimations of β,𝜃, and Z. The most common approaches used for making inferences in the LDA model are expectation maximization, Gibbs sampling, and variational inference.

Sentence classification is different from traditional text classification in the brevity of the text involved. A solution to improve classification is to enrich the data representation of sentences before training a machine learning model.

Two main approaches have been used to enrich the representation of sentences. The first is to obtain the contextual information of sentence and add more data, and the second approach involves uncovering latent topics from a data set and adding topic-related information to smoothen the representation of sentences. We combine the ideas underlying these two approaches by introducing the ETM algorithm as a topic-based smoothing method. To enrich the feature space of a sentence, the ETM algorithm matches the inferred probability distributions from the LDA model to words of sentences. sentences are represented as a TFiDF matrix, a N × V matrix where the rows denote the texts and the columns contain TFiDF values of the chosen words. Then, by applying a method of inference, the ETM extracts the topic distributions and topic assignments for the TFiDF matrix of texts.

The ETM exploits topic analysis to enhance features in sentences by assigning weights to words on the basis of the topics of inference, as the internal features of texts. This approach is applied to clinical notes to improve classification performance. When dealing with sentences, especially when manual labeling is labor intensive, the ETM can use unlabeled data to enrich the quality of the available labeled data. The overall procedure of the ETM is outlined in Algorithm 1.

In this algorithm, M is the TFiDF matrix and C is the enriched matrix. D is the number of documents in the data set, K is the number of topic clusters, and N is the number of words in the vocabulary. For each sentence, the ETM computes a dynamic enrichment weight γ as in (5):

m is the average length of the sentences and n_d is the number of words in the text document indexed by d. The weight γ is computed to consider the length of each sentence in the enrichment with the ETM. This means that if a text is longer than the average, the weight of the enrichment decreases. The ETM calculates ω as in (6), where ω is the enrichment value when information on the distributions 𝜃 and β is available. The ETM considers the original representations using β as the posterior probability of each word given the topic, and 𝜃 is the posterior distribution of hidden topics in each document. The ETM updates the representation by adding the enrichment value ω as in (6). The ETM algorithm enriches each document of the data set by incorporating the length of the document, the posterior distribution of hidden topics in the document, the probabilities of each word given the topic, and the value of the TFiDF of the word. This algorithm considers the length of each document as it incorporates an enrichment dynamic weight with greater values for shorter documents. The idea of considering the length of a text in the enrichment process is as in empirical data sets, where some sentences are long enough while others are short and need to be enriched. The ETM assumes that each document is a mixture of corpus-wide topics, and gains internal knowledge by taking advantage of the patterns of clustering. These patterns contain more contextual information on sentences that can improve clinical sentence classification performance. Because the ETM algorithm uses internal knowledge of the data set as the main source of enrichment, its effectiveness is data dependent.

The UMCU is one of the largest university hospitals in the Netherlands that provides specialized cardiac care. Given the structure of its EHRs, the data are available on a research data platform and can be extracted accordingly. The textual data set used in this study consisted of all clinical cardiovascular notes from doctors or physicians’ assistants between 2014 and 2018. A total of 1,002 clinical notes were manually annotated for medical history based on the International Classification of Diseases (ICD10)³ criteria, and were checked sample wise by doctors. The words in the clinical notes on which the annotation was based were also marked for text mining. These words determine the category of sentences in our data set describing medical history. The description of the data set is provided in Table 1. The train and unlabeled data contained 11,053 and 20,200 sentences, respectively. Sentences in the train data were labeled as two classes: with and without medical history. A total of 3,560 records had medical history and 7,493 records were labeled as without medical history.

We present an example (Fig. 4) to demonstrate the first three steps of the clinical sentence classification model, in this study. This example is used to describe the idea of how text representation could be enriched by incorporating probability distributions from the LDA clustering algorithm.

Data provided in this example contain five sentences and 14 unique words. M is the initial BOW representation, the TFiDF matrix and C is the output of the ETM algorithm, the enriched matrix. The LDA model was applied on the data set to learn two clusters of words (topics). β represents the probability distribution of words for the topics T¹ and T². 𝜃 represents the probability distribution of the topics T¹ and T² per sentence (document) S¹ to S⁵. As shown in Fig. 4 the ETM algorithm first calculates an enrichment weight (γ) for each sentence in terms of its length. Subsequently, the C matrix is calculated using the M matrix, γ and the clustering outputs: 𝜃 and β. The ETM algorithm creates a smoothed data set that balances initial observations with patterns extracted by the LDA algorithm.

We used an SVM and a multi-layer neural network (NN) as classification algorithms. In the definition of a learning classifier, the training data were the set of documents and the classes were medical history versus no medical history. The objective of the SVM was to find a hyperplane in a high-dimensional feature space that distinctly classified the input data set. By internally employing a kernel trick, it selected the discriminative hyperplane based on the computed support vectors. We used the SVM algorithm with the default parameter settings in scikit-learn⁴. The NN classifier in our experiments used a feed-forward architecture and learned to map the input data to the output labels through a series of nonlinear compositions. For sentence classification, the ReLU activation function along with the ADAM solver with two hidden layers of 100 units were used. Compared with other non-linearities, the ReLU activation function learns more quickly in deep architectures with many hidden layers. For the learning of the classification algorithms, we chose 80% of the data set as the training set and used the remaining for testing.

To compare the performance of the classifiers, accuracy, precision, recall, and the F1 score were used as the evaluation measures. Precision and recall are useful measures when classes are imbalanced. Precision is a measure of the relevance of the result while recall shows how many truly relevant results were returned. The F1 score is the harmonic mean of precision and recall. These evaluation measures were computed as follows: