AI-Powered Knowledge and Expertise Mining in Healthcare from a Field Experiment

The picture-word association exercise in the app prototype presents a picture of the target word on the right side of the screen and four written words (the target word and three distractors) on the left and centre. Participants must correctly identify and match the word with the corresponding picture. The exercise’s difficulty increased weekly based on factors such as the type of exercise, prototypicality and unambiguous representation of the target word in the picture, semantic field (category), and, more detailed below, target word/item and distractors.

Several factors determine the difficulty of the target word. These include individual frequency of use [29], degree of concreteness according to the Dual Coding Theory (DCT) [30], prototypicality (e.g., “Möhre” (carrot Eng.) is more prototypical than “Schwarzwurzel” (salsify Eng.) for the category vegetables), length and structure of the word (shorter words with simpler syllable structures (e.g., CV) are easier to learn than longer words with more syllables and complex syllable structures [31]), level of specification within a semantic hierarchy (e.g., “Hund” (dog Eng.) is easier to learn than “Dackel” (dachshund Eng.)), and properties such as dialectal peculiarities/synonyms, imageability, number of phonemes, compound words, and foreign words.

The properties of the three distractors (semantic, phonological, and non-relational) in relation to the target item also influence the difficulty level of this exercise. For semantic distractors, the level of difficulty increases with the closeness to the target item (equivalence in terms of prototypicality/type frequency, imageability/visual redundancy or distinctiveness, semantic/relational proximity). For example, “Keks” (cookie Eng.) is an easier semantic distractor for the target word “Möhre” in the semantic field food than “Rettich” (radish Eng.). For phonological distractors, it holds that the more similar the phonological-lexical properties (compared to the target word), the more difficult this distractor is [32]. For example, “Matratze” (mattress Eng.) has the same initial letter as “Möhre”, but a different number of syllables as well as a different number of graphemes and is therefore an easy phonological distractor. “Möwe” (seagull Eng.) is a more difficult distractor, as it has the same initial letter, same number of syllables, similar syllable structure, same vowel length, and it also rhymes with “Möhre”. Non-relational distractors must not be in a semantic or phonological-lexical relation to the target word or to any of the other distractors, like “Goldkette” (gold chain Eng.) for “Möhre” (carrot Eng.), as frequent errors with this distractor indicate guessing.

Data collection took place during and after the 12-week test phase and included questionnaires, recorded app usage data, a vocabulary database with items (e.g., target words, and distractors) created by the SLTs, and developed attributes. The criteria that determine the level of difficulty of an item were elaborated and described in terms of their characteristics.

This phase involved a thorough investigation of the provided database. The data on targets included around 25 features, with 11 having been identified as relevant for this investigation by the SLTs and data analysts. Hereby, certain features were neglected for this investigation (this mainly depended on the availability and validity of the data). Since the test phase was already completed, it wasn’t possible to acquire additional data from the test phase itself as well as patient-specific characteristics (e.g., displayed distractors which weren’t clicked; the patients’ familiarity with items from certain semantic fields).

The selected features included categorical values like difficulty and concreteness/abstractness, integer values like number of graphemes, Boolean values like compound words, and strings like structure syllables. The same attributes were provided for the distractors, with the additional feature “type” with its feature space {semantic, phonological, non-relational}. Additionally, data from the test phase was provided, including e.g., exercise ID, patient ID, time stamps of each click within the exercise-solving process by the patient, correct target value, falsely chosen distractors, used support (audio, joker). For a better understanding of the data, uni- and multivariate data analysis was performed: Histograms and boxplots allowed a deeper insight on distributions and possible outliers (which occurred, for example, regarding the duration of an exercise or the total number of errors per item). Additionally, the calculation of correlation coefficients and category-wise variance analysis provided a deeper understanding of the features’ connections (e.g., there is an obvious correlation between the length of a word and the number of its syllables, but the word type frequency is not correlated with the other item’s features and the type-specific number of errors). Appropriate visualisations, both individual and inter-individual, included e.g., the number of trainings per day and exercise (for each week) as stacked bar charts, the distribution of the types of errors (semantic, phonological, and non-relational) as pie charts as well as box plots, and the temporal distribution of the trainings as scatter plots.

In the context of univariate data analysis, it became apparent that the duration of some picture-word association exercises lasted up to 23 times as long as the average time and/or were not finished. Unfinished exercises were neglected; regarding the duration, a time limit was set in coordination with the SLTs. Furthermore, plausibility checks performed in Python 3.9 showed that for exercises exceeding one day, incorrect data was stored in the database regarding the type of exercise. These mistakes were corrected. There were no missing values.

Further, it was required to integrate the data from the different sources. The characteristics of the targets and distractors were provided in various Microsoft Excel files, as they were assembled manually by the SLTs. Data from the test phase were acquired automatically and made available in a MySQL database. The relevant data was extracted and structured with regard to the analysis to be performed in a new MySQL database. This also included the calculation of new features like the type-specific (semantic, phonological, and non-relational) number of errors per exercise.

After data cleaning and integration, the data was then prepared in accordance with the AI algorithms to be applied. For the decision tree algorithm, both numerical and categorical data are allowed, and the data do not need to be normalised. Regarding clustering methods, the preprocessing process included distribution and outlier handling, transformation and normalisation of numerical data, and removing highly correlated features.

Two different AI approaches were implemented. First, the decision tree as supervised method was applied to identify the features and their values leading to the difficulty of each target (with difficulty space {easy, medium, hard}). The main advantage of this approach were the explainable results, which means that the features and their respective values leading to the categorisation could be provided and discussed with the SLTs. As hyperparameters, the gini criterion was chosen, with a maximum depth of 10 being sufficient to provide unique classification results.

Second, clustering algorithms were applied as unsupervised methods to check which grouping of items was suggested. For centroid-based methods, the elbow method to identify the optimal number of clusters led to three (which matched the number of categories the SLTs chose). For comparison, two and four clusters were also set as hyperparameters. Additionally, connectivity-based methods were applied.

Besides the items’ characteristics (which formed the basis for the SLTs to derive the difficulty of the items), the results from the test phase could also be included, i.e., the number of mistakes for each of the three difficulty types. Hereby, the types semantic, phonological, and non-relational distractors were distinguished. Both approaches (leaving out and including data from the test phase) were applied and evaluated individually.

Additionally, statistical methods for multivariate analysis were applied to compare the feature values of targets with same difficulty. This allowed the identification of conspicuous targets or peculiarities with respect to the semantic fields. For example, Fig. 1 in Sect. 5.5 shows the comparison of selected feature values of target words with the same difficulty for the semantic fields clothing and food.

The results were discussed with the SLTs. Regarding the decision tree, an iterative investigation of the leaf nodes revealed items which weren’t categorised easily and required deep branching. Targets which were difficult to categorise had an 85% intersection for both approaches (leaving out and including test phase data), indicating a high consistency of the SLTs’ and patients’ perception. If considered, the relative number of errors heavily influenced the decision nodes, but not the results as much – especially if a sufficiently high depth was chosen as hyperparameter. Tab. 1 shows the impact of features selected for the algorithm with maximum depth value of 5. Here, the relative number of misclassified targets (also specifying the respective falsely stated difficulty) is depicted for three cases of selected features. It becomes obvious that including the data acquired during the test phase with the patients significantly improves the results. Simultaneously, the typical problem of words misclassified as “medium” is still apparent.

Five features were identified to have major impact on the categorisation. Certain semantic fields could be classified easily as suggested by the SLTs, whereas targets with “medium” difficulty were the most problematic ones. These items were further investigated using clustering results: Changing the number of clusters (and, hence, the number of difficulty categories) to two provided a deeper insight into the connection between the item characteristics and the results from the test phase. Most of the target values with difficulty “easy” and “hard” were assigned to the two different clusters, respectively, indicating that the SLTs’ classification of “easy” and “hard” was mostly in accordance with patients’ perception. Target values with “medium” difficulty were reassorted. It became apparent that for each semantic field with this difficulty, either a reassignment to “easy” or “hard” heavily predominated. The results brought deeper insight into which features might be relevant for the assignment to a cluster. Creating four clusters is however challenging from the application point of view: A further division of the difficulty “medium” would be necessary. As for three clusters, about half of the targets matched the SLTs’ classification, where the targets identified as critical with the decision tree method were assigned to different clusters with this method as well. These results in connection with the target analysis on targets with same difficulty provided the basis for the SLTs to find causes for targets with a high deviation from the original difficulty. Such reasons were, e.g., an unsuitable or ambiguous picture provided in the test phase.

Several working hypotheses were confirmed by cross-patient valid attribute rules. For example, the difficulty of exercises was found to depend on the type of exercise, and the percentage of phonological and semantic errors in the picture-word association exercise increased with higher difficulty, indicating appropriate distractors were developed for these levels of difficulty. Additionally, for the category food at an easy level, there was a positive correlation between word type frequency and error rate. This confirmed the hypothesis that words that occur more frequently in the German corpus (e.g., “Marmelade”, jam Eng.) would elicit fewer errors than words that occur less frequently (e.g., “Käse”, cheese Eng., or “Tee”, tea Eng.). However, word type frequency alone was not sufficient to explain the error patterns. Type frequency from the lexical database dlexDB [33] was used, which is based on the reference corpus of the German language of the 20th century from the Digital Dictionary of the German Language (DWDS) and ranges from 1900 to 1999. Therefore, the type frequency of certain words does not allow any conclusion on the current usage frequency of these words. For instance, we found more errors for words with shorter word length than for words with longer word length within the same category and level of difficulty. This contradicted the assumption regarding word type frequency and number of syllables. Possible factors that could account for this discrepancy are syllable structure, picture quality, distractor choice, or everyday relevance (depending on each patient’s perception). We also performed pair- and groupwise comparisons. For the two categories food and clothing at an easy level of difficulty, the similarity between features like word type frequency across both categories is depicted in Fig. 1. However, there were fewer phonological and semantic errors for food compared to clothing. A possible conclusion is that the visual representation of food items may show a greater degree of prototypicality and a closer alignment with everyday visual perception as compared to items of clothing. Furthermore, food might activate other sensory modalities such as taste.