Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top

2020 | Book

Hybrid Natural Language Processing: An Introduction Read chapter Read first chapter

A Practical Guide to Hybrid Natural Language Processing

Combining Neural Models and Knowledge Graphs for NLP

Authors: Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva

Publisher: Springer International Publishing

Part of: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Login to get access
insite
SEARCH

About this book

This book provides readers with a practical guide to the principles of hybrid approaches to natural language processing (NLP) involving a combination of neural methods and knowledge graphs. To this end, it first introduces the main building blocks and then describes how they can be integrated to support the effective implementation of real-world NLP applications. To illustrate the ideas described, the book also includes a comprehensive set of experiments and exercises involving different algorithms over a selection of domains and corpora in various NLP tasks.

Throughout, the authors show how to leverage complementary representations stemming from the analysis of unstructured text corpora as well as the entities and relations described explicitly in a knowledge graph, how to integrate such representations, and how to use the resulting features to effectively solve NLP tasks in a range of domains. In addition, the book offers access to executable code with examples, exercises and real-world applications in key domains, like disinformation analysis and machine reading comprehension of scientific literature. All the examples and exercises proposed in the book are available as executable Jupyter notebooks in a GitHub repository. They are all ready to be run on Google Colaboratory or, if preferred, in a local environment.

A valuable resource for anyone interested in the interplay between neural and knowledge-based approaches to NLP, this book is a useful guide for readers with a background in structured knowledge representations as well as those whose main approach to AI is fundamentally based on logic. Further, it will appeal to those whose main background is in the areas of machine and deep learning who are looking for ways to leverage structured knowledge bases to optimize results along the NLP downstream.

Table of Contents

Frontmatter

Preliminaries and Building Blocks

Frontmatter
Chapter 1. Hybrid Natural Language Processing: An Introduction
Abstract
The proliferation of knowledge graphs and recent advances in artificial intelligence have raised great expectations related to the combination of symbolic and data-driven approaches in cognitive tasks. This is particularly the case of knowledge-based approaches to natural language processing as near-human symbolic understanding relies on expressive, structured knowledge representations. Engineered by humans, knowledge graphs are frequently well curated and of high quality, but they can also be labor-intensive, rely on rigid formalisms and sometimes be biased towards the specific viewpoint of their authors. This book aims to provide the reader with means to address limitations like the above by bringing together bottom-up, data-driven models and top-down, structured knowledge graphs. To this purpose, the book explores how to reconcile both views and enrich the resulting representations beyond the possibilities of each individual approach. Throughout this book, we delve into this idea and show how such hybrid approach can be used with great effectiveness in a variety of natural language processing tasks.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 2. Word, Sense, and Graph Embeddings
Abstract
Distributed word representations in the form of dense vectors, known as word embeddings, are the basic building blocks for machine-learning based natural language processing. Such embeddings play an important role in tasks such as part-of-speech tagging, chunking, named entity recognition, and semantic role labeling, as well as downstream tasks including sentiment analysis and more in general text classification. However, early word embeddings were static context-independent representations that fail to capture multiple meanings for polysemous words. This chapter presents an overview of such traditional word embeddings, but also of alternative approaches that have been proposed to produce sense and concept embeddings using disambiguated corpora or directly from knowledge graphs. As a result, this chapter serves as a conceptual framework for the rest of book.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 3. Understanding Word Embeddings and Language Models
Abstract
Early word embeddings algorithms like word2vec and GloVe generate static distributional representations for words regardless of the context and the sense in which the word is used in a given sentence, offering poor modeling of ambiguous words and lacking coverage for out-of-vocabulary words. Hence a new wave of algorithms based on training language models such as Open AI GPT and BERT has been proposed to generate contextual word embeddings that use as input word constituents allowing them to generate representations for out-of-vocabulary words by combining the word pieces. Recently, fine-tuning pre-trained language models that have been trained on large corpora have constantly advanced the state of the art for many NLP tasks.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 4. Capturing Meaning from Text as Word Embeddings
Abstract
This chapter provides a hands-on guide to learn word embeddings from text corpora. To this purpose we choose Swivel, whose extension is the basis for the Vecsigrafo algorithm, which will be described in Chap. 6 . As introduced in Chap. 2 , word embedding algorithms like Swivel are not contextual, i.e. they do not provide different representations for the different meanings a polysemous word may have. As we will see in the subsequent chapters of the book, this can be addressed in a variety of ways. For the purpose of this chapter, we focus on a basic way to represent words using embeddings.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 5. Capturing Knowledge Graph Embeddings
Abstract
In this chapter we focus on knowledge graph embeddings, an approach to produce embeddings for concepts and names that are the main nodes in knowledge graphs, as well as the relations between them. The resulting embeddings aim to capture the knowledge encoded in the structure of the graph, in terms of how nodes are related to one another. This technique allows translating the symbolic representation of graphs in a format that simplifies manipulation without altering the inherent structure of the graph. Several algorithms to create knowledge graph embeddings have been proposed. In this chapter we will give a brief overview of the most important models and the libraries and tools that implement them. Lastly, we select one of such approaches (HolE) and provide practical guidance to learn embeddings based on WordNet.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva

Combining Neural Architectures and Knowledge Graphs

Frontmatter
Chapter 6. Building Hybrid Representations from Text Corpora, Knowledge Graphs, and Language Models
Abstract
In the previous chapter we saw how knowledge graph embedding algorithms can capture structured knowledge about concepts and relations in a graph as embeddings in a vector space, which then can be used in downstream tasks. However, this type of approaches can only capture the knowledge that is explicitly represented in the graph, hence lacking in recall and domain coverage. In this chapter, we focus on algorithms that address this limitation through the combination of information from both unstructured text corpora and structured knowledge graphs. The first approach is Vecsigrafo, which produces corpus-based word, lemma, and concept embeddings from large disambiguated corpora. Vecsigrafo jointly learns word, lemma, and concepts embeddings, bringing together textual and symbolic knowledge representations in a single, unified formalism for use in neural natural language processing architectures. The second and more recent approach is called Transigrafo, which adopts recent Transformer-based language models to derive concept-level contextual embeddings, providing state-of-the-art performance in word-sense disambiguation with reduced complexity.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 7. Quality Evaluation
Abstract
In the previous chapters we have discussed various methods for generating embeddings for both words and concepts. Once you have applied some embedding learning mechanism you may wonder how good are these embeddings? In this chapter we look at methods for assessing the quality of the learned embeddings: from visualizations to intrinsic evaluations like predicting alignment with human-rated word similarity and extrinsic evaluations based on downstream tasks. As in the previous chapters, we provide hands-on practical sections for gaining experience in applying evaluation methods. We also discuss the methodology and results used for a real-world evaluation of Vecsigrafo compared to various other methods, which provides a sense for how thorough real-world evaluations can be performed.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 8. Capturing Lexical, Grammatical, and Semantic Information with Vecsigrafo
Abstract
Embedding algorithms work by optimizing the distance between a word and its context(s), generating an embedding space that encodes their distributional representation. In addition to single words or word pieces, other features, which result from a deeper analysis of the text, can be used to enrich such representations with additional information. Such features are influenced by the tokenization strategy used to chunk the text and can include not only lexical and part-of-speech information but also annotations about the disambiguated sense of a word according to a structured knowledge graph. In this chapter we analyze the impact that explicitly adding lexical, grammatical and semantic information during the training of Vecsigrafo has in the resulting representations and whether or not this can enhance their downstream performance. To illustrate this analysis we focus on corpora from the scientific domain, where rich, multi-word expressions are frequent, hence requiring advanced tokenization strategies.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 9. Aligning Embedding Spaces and Applications for Knowledge Graphs
Abstract
In previous chapters we have seen a variety of ways to train models to derive embedding spaces for words and concepts and other nodes in knowledge graphs. As you often do not have control over the full training procedure, you may find yourself with several embedding spaces which have (conceptually) overlapping vocabularies. How can you best combine such embedding spaces?. In this chapter we look at various techniques for aligning disparate embedding spaces. This is particularly useful in hybrid settings like when using embedding spaces for knowledge graph curation and interlinking.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva

Applications

Frontmatter
Chapter 10. A Hybrid Approach to Disinformation Analysis
Abstract
Disinformation and fake news are complex and important problems where natural language processing can play an important role in helping people navigate online content. In this chapter, we provide various practical tutorials where we apply several of the hybrid NLP techniques involving neural models and knowledge graphs introduced in earlier chapters to build prototypes that solve some of the pressing issues posed by disinformation.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 11. Jointly Learning Text and Visual Information in the Scientific Domain
Abstract
In this chapter we address multi-modality in domains where not only text but also images or, as we will see next, scientific figures and diagrams are important sources of information for the task at hand. Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now remained untapped: the correspondence between a figure and its caption. In this chapter we show what can be learnt by looking at a large number of figures and reading their captions, and describe a figure-caption correspondence learning task that makes use of such observation. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also follow up on previous chapters and illustrate how transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, the positive impact of such hybrid and semantically enriched features is demonstrated in two transfer learning experiments involving scientific text and figures: multi-modal classification and machine comprehension for question answering.
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Chapter 12. Looking into the Future of Natural Language Processing
Abstract
It has been a long journey, from theory to methods to code. We hope the book took you painlessly through experiments and real NLP exercises, as much as this is possible, and that you enjoyed it as much as we did. Now, it is time to wrap up. Here, we provide guidelines for future directions in hybrid natural language processing and share our final remarks, additional thoughts, and vision. As a bonus, we also include the personal view of a selection of experts on topics related to hybrid natural language processing. We asked them to comment on their vision, foreseeable barriers to achieve such vision, and ways to navigate towards it, including opportunities and challenges in promising research fields and areas of industrial application. Now it is up to you. Hopefully this book gave you the necessary tools to build powerful NLP systems. Use them!
Jose Manuel Gomez-Perez, Ronald Denaux, Andres Garcia-Silva
Backmatter
Metadata
Title
A Practical Guide to Hybrid Natural Language Processing
Authors
Jose Manuel Gomez-Perez
Ronald Denaux
Andres Garcia-Silva
Copyright Year
2020
Electronic ISBN
978-3-030-44830-1
Print ISBN
978-3-030-44829-5
DOI
https://doi.org/10.1007/978-3-030-44830-1

Premium Partner

    Image Credits