Natural Language Processing and Information Systems

Using subword n-grams for training word embeddings makes it possible to subsequently compute vectors for rare and misspelled words. However, we argue that the subword vector qualities can be degraded for words which have a high orthographic neighbourhood; a property of words that has been extensively studied in the Psycholinguistic literature. Empirical findings about lexical neighbourhood effects constrain models of human word encoding, which must also be consistent with what we know about neurophysiological mechanisms in the visual word recognition system. We suggest that the constraints learned from humans provide novel insights to subword encoding schemes. This paper shows that vectors trained with subword properties informed by psycholinguistic evidence are superior to those trained with ad hoc n-grams. It is argued that physiological mechanisms for reading are key factors in the observed distribution of written word forms, and should therefore inform our choice of word encoding.

The paper presents the results of applying the BERT representation model in the named entity recognition task for the cybersecurity domain in Russian. Several variants of the model were investigated. The best results were obtained using the BERT model, trained on the target collection of information security texts. We also explored a new form of data augmentation for the task of named entity recognition.

The daily exponential increase of biomedical information in scientific literature and patents is a main obstacle to foster advances in biomedical research. A fundamental step hereby is to find key information (named entities) inside these publications applying Biomedical Named Entities Recognition (BNER). However, BNER is a complex task compared to traditional NER as biomedical named entities often have irregular expressions, employ complex entity structures, and don’t consider well-defined entity boundaries, etc. In this paper, we propose a deep neural network (NN) architecture, namely the bidirectional Long-Short Term Memory (Bi-LSTM) based model for BNER. We present a detailed neural network architecture showing the different NN layers, their interconnections and transformations. Based on existing gold standard datasets, we evaluated and compared several models for identifying biomedical named entities such as chemicals, diseases, drugs, species and genes/proteins. Our deep NN based Bi-LSTM model using word and character level embeddings outperforms CRF and Bi-LSTM using only word level embeddings significantly.

This paper shows the preliminary results of an initial effort to analyse whether explanations associated with a semantic parser help users to generalise the system’s mechanisms regardless of their technical background. With the support of a user-centred experiment with 66 participants, we evaluated the user’s mental model by associating the linguistic features from a set of explanations to the system’s behaviour.

The Bidirectional Encoder Representations from Transformers (BERT) model produces state-of-the-art results in many question answering (QA) datasets, including the Stanford Question Answering Dataset (SQuAD). This paper presents a query expansion (QE) method that identifies good terms from input questions, extracts synonyms for the good terms using a widely-used language resource, WordNet, and selects the most relevant synonyms from the list of extracted synonyms. The paper also introduces a novel QE method that produces many alternative sequences for a given input question using same-language machine translation (MT). Furthermore, we use a coreference resolution (CR) technique to identify anaphors or cataphors in paragraphs and substitute them with the original referents. We found that the QA system with this simple CR technique significantly outperforms the BERT baseline in a QA task. We also found that our best-performing QA system is the one that applies these three preprocessing methods (two QE and CR methods) together to BERT, which produces an excellent \(F_1\) score (89.8 \(F_1\) points) in a QA task. Further, we present a comparative analysis on the performances of the BERT QA models taking a variety of criteria into account, and demonstrate our findings in the answer span prediction task.

The popularization of Enterprise Knowledge Graphs (EKGs) brings an opportunity to use Question Answering Systems to consult these sources using natural language. We present CONQUEST, a framework that automates much of the process of building chatbots for the Template-Based Interactive Question Answering task on EKGs. The framework automatically handles the processes of construction of the Natural Language Processing engine, construction of the question classification mechanism, definition of the system interaction flow, construction of the EKG query mechanism, and finally, the construction of the user interaction interface. CONQUEST uses a machine learning-based mechanism to classify input questions to known templates extracted from EKGs, utilizing the clarification dialog to resolve inconclusive classifications and request mandatory missing parameters. CONQUEST also evolves with question clarification: these cases define question patterns used as new examples for training.

A mechanism to enable task oriented procedural question answering system for user assistance in English is presented in this paper. The primary aim is to create an answering “corpus” in a tree-form from unstructured document passages. This corpus is used to respond to the queries interactively to assist in completing a technical task. Reference manuals, documents or webpages are scraped to identify the sections depicting a “procedure” through machine learning techniques and then an integrated task tree with extracted procedural knowledge from text is generated. The automated mechanism breaks the procedural sections into steps, the appropriate “decision points” are identified, the interactive utterances are generated to gain user inputs and the alternative paths are created to complete the tree. Conventional tree traversal mechanism provides step by step guidance to complete a task. Efficacy of the proposed mechanism is tested on documents collected from three different domains and test results are presented.

Prior works on dialog generation focus on task-oriented setting and utilize multi-turn conversational utterance-response pairs. However, natural language generation (NLG) in the open-domain environment is more challenging. The conversations in an open-domain chit-chat model are mostly single-turn in nature. Current methods used for modeling single-turn conversations often fail to generate contextually relevant responses for a large dataset. In our work, we develop a transformer-based method for natural language generation (NLG) in an open-domain setting. Experiments on the utterance-response pairs show improvement over the baselines, both in terms of quantitative measures like BLEU and ROUGE and human evaluation metrics like fluency and adequacy.

In a long term exploitation environment, a Question Answering (QA) system should maintain or even improve its performance over time, trying to overcome the lacks made evident through the interactions with users. We claim that, in order to make progress in the QA over Knowledge Bases (KBs) research field, we must deal with two problems at the same time: the translation of Natural Language (NL) questions into formal queries, and the detection of missing knowledge that impact the way a question is answered. The research on these two challenges has not been addressed jointly until now, what motivates the main goals of this work: (i) the definition of the problem and (ii) the development of a methodology to create the evaluation resources needed to address this challenge.

A large number of applications in text data analysis use the Latent Dirichlet Allocation (LDA) as one of the most popular methods in topic modeling. Although the instability of the LDA is mentioned sometimes, it is usually not considered systematically. Instead, an LDA is often selected from a small set of LDAs using heuristic means or human codings. Then, conclusions are often drawn based on the to some extent arbitrarily selected model. We present the novel method LDAPrototype, which takes the instability of the LDA into account, and show that by systematically selecting an LDA it improves the reliability of the conclusions drawn from the result and thus provides better reproducibility. The improvement coming from this selection criterion is unveiled by applying the proposed methods to an example corpus consisting of texts published in a German quality newspaper over one month.

Online reviews are an important source of feedback for understanding customers. In this study, we follow novel approaches that target the absence of actionable insights by classifying reviews as defect reports and requests for improvement. Unlike traditional classification methods based on expert rules, we reduce the manual labour by employing a supervised system that is capable of learning lexico-semantic patterns through genetic programming. Additionally, we experiment with a distantly-supervised SVM that makes use of the noisy labels generated by patterns. Using a real-world dataset of app reviews, we show that the automatically learned patterns outperform the manually created ones. Also the distantly-supervised SVM models are not far behind the pattern-based solutions, showing the usefulness of this approach when the amount of annotated data is limited.

The Régie du Logement du Québec (RDL) is a tribunal with exclusive jurisdiction in matters regarding rental leases. Within the framework of the ACT (Autonomy Through Cyberjustice Technologies) project, we processed an original collection of court decisions in French and performed a thorough analysis to reveal biases that may influence prediction experiments. We studied a multilabel classification task that consists in predicting the types of verdict in order to illustrate the importance of prior data analysis. Our best model, based on the FlauBERT language model, achieves F1 score micro averages of 93.7% and 84.9% in Landlord v. Tenant and Tenant v. Landlord cases respectively. However, with the support of our in-depth analysis, we emphasize that these results should be kept in perspective and that some metrics may not be suitable for evaluating systems in sensitive domains such as housing law.

Aspect Based Sentiment Analysis (ABSA) is the task of identifying sentiment polarity of a text given another text segment or aspect. In ABSA, a text can have multiple sentiments depending upon each aspect. Aspect Term Sentiment Analysis (ATSA) is a subtask of ABSA, in which aspect terms are contained within the given sentence. Most of the existing approaches proposed for ATSA, incorporate aspect information through a different subnetwork thereby overlooking the advantage of aspect terms’ presence within the sentence. In this paper, we propose a model that leverages the positional information of the aspect. The proposed model introduces a decay mechanism based on position. This decay function mandates the contribution of input words for ABSA. The contribution of a word declines as farther it is positioned from the aspect terms in the sentence. The performance is measured on two standard datasets from SemEval 2014 Task 4. In comparison with recent architectures, the effectiveness of the proposed model is demonstrated.

In the sentiment attitude extraction task, the aim is to identify «attitudes» – sentiment relations between entities mentioned in text. In this paper, we provide a study on attention-based context encoders in the sentiment attitude extraction task. For this task, we adapt attentive context encoders of two types: (I) feature-based; (II) self-based. Our experiments (https://​github.​com/​nicolay-r/​attitude-extraction-with-attention) with a corpus of Russian analytical texts RuSentRel illustrate that the models trained with attentive encoders outperform ones that were trained without them and achieve 1.5–5.9% increase by \(F1\). We also provide the analysis of attention weight distributions in dependence on the term type.

Curriculum Learning (CL) is the idea that learning on a training set sequenced or ordered in a manner where samples range from easy to difficult, results in an increment in performance over otherwise random ordering. The idea parallels cognitive science’s theory of how human brains learn, and that learning a difficult task can be made easier by phrasing it as a sequence of easy to difficult tasks. This idea has gained a lot of traction in machine learning and image processing for a while and recently in Natural Language Processing (NLP). In this paper, we apply the ideas of curriculum learning, driven by SentiWordNet in a sentiment analysis setting. In this setting, given a text segment, our aim is to extract its sentiment or polarity. SentiWordNet is a lexical resource with sentiment polarity annotations. By comparing performance with other curriculum strategies and with no curriculum, the effectiveness of the proposed strategy is presented. Convolutional, Recurrence and Attention based architectures are employed to assess this improvement. The models are evaluated on standard sentiment dataset, Stanford Sentiment Treebank.

Users play a critical role in the creation and propagation of fake news online by consuming and sharing articles with inaccurate information either intentionally or unintentionally. Fake news are written in a way to confuse readers and therefore understanding which articles contain fabricated information is very challenging for non-experts. Given the difficulty of the task, several fact checking websites have been developed to raise awareness about which articles contain fabricated information. As a result of those platforms, several users are interested to share posts that cite evidence with the aim to refute fake news and warn other users. These users are known as fact checkers. However, there are users who tend to share false information, who can be characterised as potential fake news spreaders. In this paper, we propose the CheckerOrSpreader model that can classify a user as a potential fact checker or a potential fake news spreader. Our model is based on a Convolutional Neural Network (CNN) and combines word embeddings with features that represent users’ personality traits and linguistic patterns used in their tweets. Experimental results show that leveraging linguistic patterns and personality traits can improve the performance in differentiating between checkers and spreaders.

The area of Computational Creativity has received much attention in recent years. In this paper, within this framework, we propose a model for the generation of literary sentences in Spanish, which is based on statistical algorithms, shallow parsing and the automatic detection of personality features of characters of well known literary texts. We present encouraging results of the analysis of sentences generated by our methods obtained with human inspection.

In this paper, we use movies and series subtitles and applied text mining and Natural Language Processing methods to evaluate emotions in videos. Three different word lexicons were used and one of the outcomes of this research is the generation of a secondary dataset with more than 3658 records which can be used for other data analysis and data mining research. We used our secondary dataset to find and display correlations between different emotions on the videos and the correlation between emotions on the movies and users’ scores on IMDb using the Pearson correlation method and found some statistically significant correlations.

Word representation models have been successfully applied in many natural language processing tasks, including sentiment analysis. However, these models do not always work effectively in some social media contexts. When considering the use of Arabic in microblogs like Twitter, it is important to note that a variety of different linguistic domains are involved. This is mainly because social media users employ various dialects in their communications. While training word-level models with such informal text can lead to words being captured that have the same meanings, these models cannot capture all words that can be encountered in the real world due to out-of-vocabulary (OOV) words. The inability to identify words is one of the main limitations of this word-level model. In contrast, character-level embeddings can work effectively with this problem through their ability to learn the vectors of character n-grams or parts of words. We take advantage of both character- and word-level models to discover more effective methods to represent Arabic affect words in tweets. We evaluate our embeddings by incorporating them into a supervised learning framework for a range of affect tasks. Our models outperform the state-of-the-art Arabic pre-trained word embeddings in these tasks. Moreover, they offer improved state-of-the-art results for the task of Arabic emotion intensity, outperforming the top-performing systems that employ a combination of deep neural networks and several other features.

Explainability of deep learning models has become increasingly important as neural-based approaches are now prevalent in natural language processing. Explainability is particularly important when dealing with a sensitive domain application such as clinical psychology. This paper focuses on the quantitative assessment of user-level attention mechanism in the task of detecting signs of anorexia in social media users from their posts. The assessment is done through monitoring the performance measures of a neural classifier, with and without user-level attention, when only a limited number of highly-weighted posts are provided. Results show that the weights assigned by the user-level attention strongly correlate with the amount of information that posts provide in showing if their author is at risk of anorexia or not, and hence can be used to explain the decision of the neural classifier.

With the increasing popularity of personal assistant systems, it is crucial to build a chatbot that can communicate with humans and assist them to complete different tasks. A fundamental problem that any chatbots need to address is how to rank candidate responses based on previous utterances in a multi-turn conversation. A previous utterance could be either a past input from the user or a past response from the chatbot. Intuitively, a correct response needs to match well with both past responses and past inputs, but in a different way. Moreover, the matching process should depend on not only the content of the utterances but also domain knowledge. Although various models have been proposed for response matching, few of them studied how to adapt the matching mechanism to utterance types and domain knowledge. To address this limitation, this paper proposes an adaptive response matching network (ARM) to better model the matching relationship in multi-turn conversations. Specifically, the ARM model has separate response matching encoders to adapt to different matching patterns required by different utterance types. It also has a knowledge embedding component to inject domain-specific knowledge in the matching process. Experiments over two public data sets show that the proposed ARM model can significantly outperform the state of the art methods with much fewer parameters.

In this paper, we focus on the problem of question retrieval in community Question Answering (cQA) which aims to retrieve from the community archives the previous questions that are semantically equivalent to the new queries. The major challenges in this crucial task are the shortness of the questions as well as the word mismatch problem as users can formulate the same query using different wording. While numerous attempts have been made to address this problem, most existing methods relied on supervised models which significantly depend on large training data sets and manual feature engineering. Such methods are mostly constrained by their specificities that put aside the word order and ignore syntactic and semantic relationships. In this work, we rely on Neural Networks (NNs) which can learn rich dense representations of text data and enable the prediction of the textual similarity between the community questions. We propose a deep learning approach based on a Siamese architecture with LSTM networks, augmented with an attention mechanism. We test different similarity measures to predict the semantic similarity between the community questions. Experiments conducted on real cQA data sets in English and Arabic show that the performance of question retrieval is improved as compared to other competitive methods.

“A picture is worth a thousand words”, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more information than the sum of that contained in the single media. The combination of visual and textual information can be obtained through linking the entities mentioned in the text with those shown in the pictures. To further integrate this with agent background knowledge, an additional step is necessary. That is, either finding the entities in the agent knowledge base that correspond to those mentioned in the text or shown in the picture or, extending the knowledge base with the newly discovered entities. We call this complex task Visual-Textual-Knowledge Entity Linking (VTKEL). In this paper, after providing a precise definition of the VTKEL task, we present a dataset composed of about 30K commented pictures, annotated with visual and textual entities, and linked to the YAGO ontology. Successively, we develop a purely unsupervised algorithm for the solution of the VTKEL tasks. The evaluation on the VTKEL dataset shows promising results.

This paper describes a prototype system for partial automation of customer service operations of a mobile telecommunications operator with a human-in-the loop conversational agent. The agent consists of an intent detection system for identifying the types of customer requests that it can handle appropriately, a slot filling information extraction system that integrates with the customer service database for a rule-based treatment of the common scenarios, and a template-based language generation system that builds response candidates that can be approved or amended by customer service operators. The main focus of this paper is on the system architecture and machine learning system structure design, and the observations of a limited pilot study performed to evaluate the proposed system on customer messages in Latvian. We also discuss the business requirements and practical application limitations and their influence on the design of the natural language processing components.