Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top

2023 | Book

Introduction Read chapter Read first chapter

Introduction to Transfer Learning

Algorithms and Practice

Authors: Jindong Wang, Yiqiang Chen

Publisher: Springer Nature Singapore

Book Series : Machine Learning: Foundations, Methodologies, and Applications

Part of: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Login to get access
insite
SEARCH

About this book

Transfer learning is one of the most important technologies in the era of artificial intelligence and deep learning. It seeks to leverage existing knowledge by transferring it to another, new domain. Over the years, a number of relevant topics have attracted the interest of the research and application community: transfer learning, pre-training and fine-tuning, domain adaptation, domain generalization, and meta-learning.

This book offers a comprehensive tutorial on an overview of transfer learning, introducing new researchers in this area to both classic and more recent algorithms. Most importantly, it takes a “student’s” perspective to introduce all the concepts, theories, algorithms, and applications, allowing readers to quickly and easily enter this area. Accompanying the book, detailed code implementations are provided to better illustrate the core ideas of several important algorithms, presenting good examples for practice.

Table of Contents

Frontmatter

Foundations

Frontmatter
Chapter 1. Introduction
Abstract
In this chapter, we introduce the background of transfer learning to give an overview of this area. This chapter can be thought of as a broad introduction to readers who have never experienced transfer learning. Thus, this chapter is self-contained. Experienced readers can skip it with no harm.
Jindong Wang, Yiqiang Chen
Chapter 2. From Machine Learning to Transfer Learning
Abstract
Transfer learning is an important branch of machine learning. They have very tight connections. Therefore, we should first start familiarizing with the basics of machine learning. With these bases, we can then deeply understand their problems with more insights. We have briefly introduced the background and concepts of transfer learning in the last chapter. We will dive into this area starting from this chapter. We introduce the definition of machine learning and probability distribution. Then, we present the definition of transfer learning, along with its fundamental problems and negative transfer case. Finally, we present a complete transfer learning process.
Jindong Wang, Yiqiang Chen
Chapter 3. Overview of Transfer Learning Algorithms
Abstract
This chapter gives an overview of transfer learning algorithms so that readers can learn and understand detailed algorithms in other chapters with a thorough view. To facilitate such an understanding, we establish a unified representation framework through which most of existing methods can be derived. Then, other chapters will introduce more details on each kind of algorithms. We want to emphasize that you are encouraged to do such overview when learning new materials.
Jindong Wang, Yiqiang Chen
Chapter 4. Instance Weighting Methods
Abstract
Instance weighting methods are one of the most effective methods for transfer learning. Technically speaking, any weighting methods can be used for evaluating the importance of each instance. In this chapter, we mainly focus on two basic methods: instance selection and instance weight adaptation. These two kinds of methods are widely adopted in existing transfer learning research and also act as the basic module for more complicated systems.
Jindong Wang, Yiqiang Chen
Chapter 5. Statistical Feature Transformation Methods
Abstract
In this chapter, we introduce statistical feature transformation methods for transfer learning. This kind of approaches is extremely popular in existing literature with good results. Especially, they are often implemented in deep neural networks in recent research, demonstrating remarkable performance. Thus, it is important that we understand its very basic knowledge. Note that we will focus on its basics and will not introduce its deep learning extensions, which will be introduced in later sections.
Jindong Wang, Yiqiang Chen
Chapter 6. Geometrical Feature Transformation Methods
Abstract
In this chapter, we introduce the geometrical feature transformation methods for transfer learning, which is different from statistical feature transformation in the last section. The geometrical features can exploit the potentially geometrical structure to obtain clean and effective representations with remarkable performance. Similar to statistical features, there are also many geometrical features. We mainly introduce three types of geometrical feature transformation methods: subspace learning, manifold learning, and optimal transport methods. These methods are different in methodology, and they are all important in transfer learning.
Jindong Wang, Yiqiang Chen
Chapter 7. Theory, Evaluation, and Model Selection
Abstract
We have introduced several basic algorithms for transfer learning. However, we did not show how to select models and tune hyperparameters, which will be covered in this chapter. Moreover, we will also introduce theories behind existing approaches that act as the foundation for designing new algorithms.
Jindong Wang, Yiqiang Chen

Modern Transfer Learning

Frontmatter
Chapter 8. Pre-Training and Fine-Tuning
Abstract
In this chapter, we focus on modern parameter-based methods: the pre-training and fine-tuning approach. We will also step into deep transfer learning starting from this chapter. In next chapters, the deep transfer learning methods focus on how to design better network architectures and loss functions based on the pre-trained network. Thus, this chapter can be seen as the foundations of the next chapters. Pre-training and fine-tuning belongs to the category of parameter/model-based transfer learning methods that perform knowledge transfer by sharing some important information of the model structures. The basic assumption is that there exists some common information between source and target structures that can be shared.
Jindong Wang, Yiqiang Chen
Chapter 9. Deep Transfer Learning
Abstract
With the development of deep learning, more and more researchers adopt deep neural networks for transfer learning. Compared to traditional machine learning, deep transfer learning increases the performance on various tasks. In addition, deep learning can take the vanilla data as the inputs, thus it has two more benefits: automatic feature extraction and end-to-end training. This chapter will introduce the basic of deep transfer learning, including network structure of deep transfer learning, distribution adaptation, structure adaptation, knowledge distillation, and practice.
Jindong Wang, Yiqiang Chen
Chapter 10. Adversarial Transfer Learning
Abstract
Generative Adversarial Nets (GAN) is one of the most popular research topics in recent years. In this chapter, we introduce adversarial transfer learning methods, which belongs to the implicit feature transformation methods. Specifically, we will introduce GAN-based transfer learning with applications to distribution adaptation and maximum classifier discrepancy. Data generation is another important research topic in adversarial transfer learning. Finally, we present some practice.
Jindong Wang, Yiqiang Chen
Chapter 11. Generalization in Transfer Learning
Abstract
While fine-tuning and domain adaptation focus on the performance on the target domain, we discuss the generalization of transfer learning in this chapter. Specifically, we introduce the problem definition, main algorithms, and theory of domain generalization.
Jindong Wang, Yiqiang Chen
Chapter 12. Safe and Robust Transfer Learning
Abstract
In this chapter, we discuss the safety and robustness of transfer learning. By safety, we refer to its defense and solutions against attack and data privacy misuse. By robustness, we mean the transfer mechanism that prevents the model from learning from spurious relations. Therefore, we introduce the related topics from three levels: (1) framework level, which is the safe fine-tuning process against defect inheritance, (2) data level, which is the transfer learning system against data privacy leakage, and (3) mechanism level, which is causal learning.
Jindong Wang, Yiqiang Chen
Chapter 13. Transfer Learning in Complex Environments
Abstract
The application environment is dynamically changing, so does the algorithm. In order to cope with the changing environment, there are several new transfer learning algorithms being developed. We define the complex environment mainly based on the traits of the training data since data is the key to modern transfer learning. In this chapter, we briefly introduce some of these complex environments and show how transfer learning algorithms can be adapted to deal with such new challenges. Note that there are multiple new settings in recent literature and we cannot cover them all.
Jindong Wang, Yiqiang Chen
Chapter 14. Low-Resource Learning
Abstract
This chapter discusses another important topic that is closely related to transfer learning: low-resource learning. Low-resource learning refers to the situation where there are not sufficient hardware resources or labeled data to train a model or even perform fine-tuning. Low-resource learning has wide applications in real world. Particularly, we will first introduce how to compress transfer learning models. Then, we introduce three low-resource learning paradigms: semi-supervised learning, meta-learning, and self-supervised learning. While transfer learning has made consistent success in multiple fields, its combination with other learning paradigms can often generate bigger impacts beyond single transfer learning. We will introduce their basic problem definitions, algorithms, and possible applications.
Jindong Wang, Yiqiang Chen

Applications of Transfer Learning

Frontmatter
Chapter 15. Transfer Learning for Computer Vision
Abstract
Today, most of the deep learning algorithms, tutorials, and talks are using computer vision tasks as benchmarks. For instance, the common “Hello world” example of deep learning tutorial is MNIST digits classification and the ImageNet challenge has dramatically boosted the rapid of deep learning. To now, ImageNet is still the common benchmark in many areas.
Jindong Wang, Yiqiang Chen
Chapter 16. Transfer Learning for Natural Language Processing
Abstract
Recent years have witnessed the fast development of natural language processing (NLP). Particularly, the pre-training technique has been playing a key role in common NLP tasks. In this chapter, we show how to perform fine-tuning using the pre-trained language model on a sentence classification task. To save space, we will only introduce the important code snippets in this chapter. For complete code, please refer to the link: https://​github.​com/​jindongwang/​tlbook-code/​tree/​main/​chap15_​application/​nlp.
Jindong Wang, Yiqiang Chen
Chapter 17. Transfer Learning for Speech Recognition
Abstract
Speech recognition is also an important research area of transfer learning. Speech recognition has several scenarios: cross-domain ASR and cross-lingual ASR. In this chapter, we introduce how to implement these two applications using PyTorch and EspNet. Note that for easy installation of EspNet, we provide a docker environment for the users: jindongwang/espnet:all11. For easy usage of EspNet, we also provide a wrapper for it: https://​github.​com/​jindongwang/​EasyEspnet.
Jindong Wang, Yiqiang Chen
Chapter 18. Transfer Learning for Activity Recognition
Abstract
Sensor-based human activity recognition (HAR) plays an important role in people’s daily life. HAR makes it possible to recognize people’s daily activities, thus monitoring people’s health status in a pervasive manner. In this chapter, we show how to use transfer learning for cross-domain human activity recognition on a public dataset. The complete code of this chapter can be found at: https://​github.​com/​jindongwang/​tlbook-code/​tree/​main/​chap18_​app_​activity.
Jindong Wang, Yiqiang Chen
Chapter 19. Federated Learning for Personalized Healthcare
Abstract
Federated learning aims at building machine learning models without compromising data privacy from the clients. Since different clients naturally have different data distributions (i.e., the non-i.i.d. issue), it is intuitive to embed transfer learning technology into the federated learning system. In this chapter, we first introduce a healthcare task and show how to prepare data for federated learning.
Jindong Wang, Yiqiang Chen
Chapter 20. Concluding Remarks
Abstract
Transfer learning is extremely important to solve the label scarce situations and non-i.i.d issues in machine learning. In this book, we start from the basics of machine learning to the concepts of transfer learning. Then, based on the three fundamental problems of transfer learning: when to transfer, where to transfer, and how to transfer, we introduced the foundations and modern transfer learning algorithms. The last part of this book is transfer learning code practice in real applications and we hope readers can quickly get familiar with it.
Jindong Wang, Yiqiang Chen
Backmatter
Metadata
Title
Introduction to Transfer Learning
Authors
Jindong Wang
Yiqiang Chen
Copyright Year
2023
Publisher
Springer Nature Singapore
Electronic ISBN
978-981-19-7584-4
Print ISBN
978-981-19-7583-7
DOI
https://doi.org/10.1007/978-981-19-7584-4

Premium Partner

    Image Credits