nach oben

2021 | Buch

Kapitel lesen Erstes Kapitel lesen

Synthetic Data for Deep Learning

verfasst von: Sergey I. Nikolenko

Verlag: Springer International Publishing

Buchreihe : Springer Optimization and Its Applications

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Einloggen, um Zugang zu erhalten

Über dieses Buch

This is the first book on synthetic data for deep learning, and its breadth of coverage may render this book as the default reference on synthetic data for years to come. The book can also serve as an introduction to several other important subfields of machine learning that are seldom touched upon in other books. Machine learning as a discipline would not be possible without the inner workings of optimization at hand. The book includes the necessary sinews of optimization though the crux of the discussion centers on the increasingly popular tool for training deep learning models, namely synthetic data. It is expected that the field of synthetic data will undergo exponential growth in the near future. This book serves as a comprehensive survey of the field.

In the simplest case, synthetic data refers to computer-generated graphics used to train computer vision models. There are many more facets of synthetic data to consider. In the section on basic computer vision, the book discusses fundamental computer vision problems, both low-level (e.g., optical flow estimation) and high-level (e.g., object detection and semantic segmentation), synthetic environments and datasets for outdoor and urban scenes (autonomous driving), indoor scenes (indoor navigation), aerial navigation, and simulation environments for robotics. Additionally, it touches upon applications of synthetic data outside computer vision (in neural programming, bioinformatics, NLP, and more). It also surveys the work on improving synthetic data development and alternative ways to produce it such as GANs.

The book introduces and reviews several different approaches to synthetic data in various domains of machine learning, most notably the following fields: domain adaptation for making synthetic data more realistic and/or adapting the models to be trained on synthetic data and differential privacy for generating synthetic data with privacy guarantees. This discussion is accompanied by an introduction into generative adversarial networks (GAN) and an introduction to differential privacy.

Inhaltsverzeichnis

Frontmatter

Chapter 1. Introduction: The Data Problem

Abstract

Machine learning has been growing in scale, breadth of applications, and the amounts of required data. This presents an important problem, as the requirements of state-of-the-art machine learning models, especially data-hungry deep neural networks, are pushing the boundaries of what is economically feasible and physically possible. In this introductory chapter, we show and illustrate this phenomenon, discuss several approaches to solving the data problem, introduce the main topic of this book, synthetic data, and outline a plan for the rest of the book.

Sergey I. Nikolenko

Chapter 2. Deep Learning and Optimization

Abstract

Deep learning is currently one of the hottest fields not only in machine learning but in the whole of science. Since the mid-2000s, deep learning models have been revolutionizing artificial intelligence, significantly advancing state of the art across all fields of machine learning: computer vision, natural language processing, speech and sound processing, generative models, and much more. This book concentrates on synthetic data applications; we cannot hope to paint a comprehensive picture of the entire field and refer the reader to other books for a more detailed overview of deep learning [153, 289, 630, 631]. Nevertheless, in this chapter, we begin with an introduction to deep neural networks, describing the main ideas in the field. We especially concentrate on approaches to optimization in deep learning, starting from regular gradient descent and working our way towards adaptive gradient descent variations and state-of-the-art ideas.

Sergey I. Nikolenko

Chapter 3. Deep Neural Networks for Computer Vision

Abstract

Computer vision problems are related to the understanding of digital images, video, or similar inputs such as 3D point clouds, solving problems such as image classification, object detection, segmentation, 3D scene understanding, object tracking in videos, and many more. Neural approaches to computer vision were originally modeled after the visual cortex of mammals, but soon became a science of their own, with many architectures already developed and new ones appearing up to this day. In this chapter, we discuss the most popular architectures for computer vision, concentrating mainly on ideas rather than specific models. We also discuss the first step towards synthetic data for computer vision: data augmentation.

Sergey I. Nikolenko

Chapter 4. Generative Models in Deep Learning

Abstract

So far, we have mostly discussed discriminative machine learning models that aim to solve a supervised problem, i.e., learn a conditional distribution of the target variable conditioned on the input. In this chapter, we consider generative models whose purpose is to learn the entire distribution of inputs and be able to sample new inputs from this distribution. We will go through a general introduction to generative models and then proceed to generative models in deep learning. First, we will discuss explicit density models that model distribution factors with deep neural networks and their important special case, normalizing flows, and explicit density models that approximate the distribution in question, represented by variational autoencoders. Then we will proceed to the main content, generative adversarial networks, discuss various adversarial architectures and loss functions, and give a case study of style transfer with GANs that is directly relevant to synthetic-to-real transfer.

Sergey I. Nikolenko

Chapter 5. The Early Days of Synthetic Data

Abstract

It may appear that synthetic data has become instrumental only very recently, with the rise of modern computer graphics that allows for near-photorealistic imagery. But in fact, synthetic data has been used throughout the history of computer vision, starting from its very inception in the 1960s. In this chapter, we begin with the early days of synthetic data, show some of the earliest models and applications of computer vision, and discuss aspects of computer vision that have always been very hard or even impossible to do without synthetic data.

Sergey I. Nikolenko

Chapter 6. Synthetic Data for Basic Computer Vision Problems

Abstract

It is time to put the pedal to the metal: starting from this chapter, we will discuss the current state of the art in various aspects of synthetic data. This chapter is devoted to basic computer vision problems: we begin with low-level problems such as optical flow estimation and stereo image matching, proceed to datasets of basic objects that can be used to train computer vision models, discuss in detail the case study of synthetic data for object detection, and finish with several different use cases such as synthetic datasets of humans, OCR, and visual reasoning.

Sergey I. Nikolenko

Chapter 7. Synthetic Simulated Environments

Abstract

In this chapter, we proceed from datasets of static synthetic images, either prerendered or procedurally generated, to entire simulated environments that can be used either to generate synthetic datasets on the fly or provide learning environments for reinforcement learning agents. We discuss datasets and simulations for outdoor environments (mostly for autonomous driving), indoor environments, and physics-based simulations for robotics. We also make a special case study of datasets for unmanned aerial vehicles and the use of computer games as simulated environments.

Sergey I. Nikolenko

Chapter 8. Synthetic Data Outside Computer Vision

Abstract

While computer vision remains the main focus of synthetic data applications, other fields also begin to use synthetic datasets, with some directions entirely dependent on synthetic data. In this chapter, we survey some of these fields. Specifically, Section 8.1 discusses how structured synthetic data is used for fraud and intrusion detection and other applications in the form of network and/or system logs; in Section 8.2, we consider neural programming; Section 8.3 discusses synthetic data generation and use in bioinformatics, and Section 8.4 reviews the (admittedly limited) applications of synthetic data in natural language processing.

Sergey I. Nikolenko

Chapter 9. Directions in Synthetic Data Development

Abstract

In this chapter, we outline the main directions that we believe to represent promising ways to further improve synthetic data, making it more useful for a wide variety of applications in computer vision and other fields. In particular, we discuss the idea of domain randomization (Section 9.1) intended to improve the applications of synthetic datasets, methods to improve CGI-based synthetic data generation itself (Section 9.2), ways to create synthetic data from real images by cutting and pasting (Section 9.3), and finally possibilities to produce synthetic data by generative models (Section 9.4). The latter means generating useful synthetic data from scratch rather than domain adaptation and refinement, which we consider in a separate Chapter 10.

Sergey I. Nikolenko

Chapter 10. Synthetic-to-Real Domain Adaptation and Refinement

Abstract

Domain adaptation is a set of techniques aimed to make a model trained on one domain of data to work well on a different target domain. In this chapter, we give a survey of domain adaptation approaches that have been used for synthetic-to-real adaptation, that is, methods for making models trained on synthetic data work well on real data, which is almost always the end goal. We distinguish two main approaches. In synthetic-to-real refinement input synthetic data is modified, usually to be made more realistic, and we can actually see the modified data. In model-based domain adaptation, it is the training process or the model structure that changes to ensure domain adaptation, while the data remains as synthetic as it has been. We will discuss neural architectures for both approaches, including many models based on generative adversarial networks.

Sergey I. Nikolenko

Chapter 11. Privacy Guarantees in Synthetic Data

Abstract

In this chapter, we discuss another important field of applications for synthetic data: ensuring privacy. In many real-world problems, real data is sensitive enough that it is impossible to release. One possible solution could be to train generative models that would produce new synthetic datasets based on real data, while the real data itself would remain secret. But how can we be sure that real data will not be inadvertently leaked? Guarantees in this regard can be provided by the framework of differential privacy. We give a brief introduction to differential privacy, its relation to machine learning, and the guarantees that it can provide for synthetic data generation.

Sergey I. Nikolenko

Chapter 12. Promising Directions for Future Work

Abstract

In this concluding chapter, we discuss the next steps that we can expect from the field of synthetic data for deep learning. We consider four different ideas that are starting to gain traction in this field. First, procedural generation of synthetic data can allow for much larger synthetic datasets or datasets generated on the fly. Second, recent works try to make the shift from domain randomization to the generation feedback loop, adapting synthetic data generation to the model and problem at hand. Third, we discuss how to best incorporate additional knowledge into the domain adaptation architectures, and fourth, show examples of introducing extra modalities into synthetic datasets with the purpose to improve downstream tasks that formally might not even use these modalities.

Sergey I. Nikolenko

Backmatter

Titel: Synthetic Data for Deep Learning
verfasst von: Sergey I. Nikolenko
Verlag: Springer International Publishing
Electronic ISBN: 978-3-030-75178-4
Print ISBN: 978-3-030-75177-7
DOI: https://doi.org/10.1007/978-3-030-75178-4