Continual learning of neural networks for quality prediction in production using memory aware synapses and weight transfer

One major assumption of machine learning learning is that the training data for a model is drawn from the same domain and shares the same characteristics (e.g. input features, distributions) as the test data. This ensures that models generalize well to new unseen data. However, in many real world scenarios this is not the case and the training data for a learning task becomes available only during a certain time. In such cases, a new model would have to be trained from scratch on new data for each new task. Continual learning is a paradigm of machine learning that tackles this problem and deals with training machine learning models over time in such a way that they can both acquire knowledge for new tasks and retain knowledge from previously trained tasks (Parisi et al. 2019; Chen and Liu 2018). A related paradigm to continual learning is transfer learning, which has also been successfully investigated in industrial applications, such as by Zhao et al. (2020) or Zellinger et al. (2020). The main idea of transer learning is also to leverage the knowledge of a model pre-trained on a source task to a given new task (Pan and Yang 2009; Tan et al. 2018). However, when training neural networks with transfer learning, they may suffer from catastrophic forgetting. As soon as they are trained for sequentially occurring tasks, their performance for previously learned tasks drops due to changes of their parameters respectively network weights (Goodfellow et al. 2013). In contrast, continual learning methods address this issue and try to find a trade-off between the stability and plasticity of network parameters when training on new tasks. State-of-the-Art methods for continual learning can be roughly divided into three categories: memory-based rehearsal strategies, dynamic architectures, and regularization strategies (Parisi et al. 2019).

Rehearsal methods use a fixed sized memory to store data samples from previously trained tasks. Theses samples are then later revisited during the training of new tasks in order to mitigate catastrophic forgetting. For example, Rebuffi et al. (2017) keep an episodic memory with representative samples for each task. When training new tasks, they calculate an additional distillation loss to prevent the network’s predictions for these samples from changing significantly. In contrast to that, Lopez-Paz and Ranzato (2017) use the memory to compute the network’s gradients for previous tasks. They then formulate the learning of a new task as a dual optimization problem allowing the calculated gradients to minimize both the new loss and the previous losses. Shin et al. (2017) propose a pseudo-rehearsal approach that uses an autoencoder as a generative model to replay previous tasks and to generate new data for each of them when training a new task.

Approaches with dynamic architectures change the architecture of the network when training for new tasks. Often, they dynamically expand the capacity of a network in order to learn new patterns without conflicts. Parisi et al. (2017) use a growing when required (GWR) approach to train recurrent self-organizing neural networks that are hierarchically extended for new tasks. Rus et al. (2016) propose a progressively growing neural network, with the network being extended by an additional column when trained on a new task. By freezing the previous columns and using lateral connections to the new column, both catastrophic forgetting is prevented and previously learned knowledge is reused for the current task. Schwarz et al. (2018) consolidate previously learned knowledge in a base column (i.e. knowledge base) by means of knowledge distillation. For new tasks an additional active column is trained, which again is connected to the basis and can therefore leverage previous knowledge and at the same time acquire new knowledge.

Regularization strategies reduce catastrophic forgetting of neural networks by restricting network parameter updates while training on new tasks. In elastic weight consolidation (EWC) by Kirkpatrick et al. (2017), this is realized by penalizing changes of parameters that are important for previous tasks. The parameter importance is estimated via probability densities using the fisher information matrix. Because of that, EWC is suited for continual classification tasks. In contrast to that, the memory-aware synapses approach proposed by Aljundi et al. (2018, 2019) represents the importance of network parameters by the sensitivity of the network output to changes of the parameters. By incorporating changes of important parameters into the loss function, mainly those networks weights are adapted to a new task that are not yet important. A number of other regularization strategies exist that constrain the learning of a network in a different way. Zhizhong and Hoiem (2018) use a distillation loss to prevent the actual outputs (i.e. predictions) of the updated network from deviating too much from those of its older version. Pomponi et al. (2020) applies regularization to penalize significant changes of a network’s embeddings (i.e. activations) for previously learned tasks.

The mentioned methods show promising results and most of them could also be applied for predictive quality scenarios with varying process conditions. However, only few of the them fulfill the three requirements defined in Sect. 1. Although rehearsal methods can yield the best results (Hsu et al. 2018), they require access to old data and a sufficiently large memory for training. Dynamic architectures manage learning without forgetting but are mostly inefficient and do not scale well with the number of tasks. Among the established regularization methods, the memory-aware synapses approach by Aljundi et al. (2018) is most suitable for our use case because this method can be used for regression (unlike EWC by Kirkpatrick et al. (2017)) and it does not require access to old data (like by Lopez-Paz and Ranzato (2017)). The importance of the network weights for a task is computed only once and withheld for future training. Our proposed approach in this paper is therefore mostly based on memory-aware synapses (see detailed description in Sect. 3).

Methods of continual learning are not only researched in the core research areas of machine learning such as computer vision and natural language processing, but are also evaluated in applied research areas. In the field of medicine, for example, methods like elastic weight consolidation are used for semantic segmentation (Baweja et al. 2018; Karin van Garderen et al. 2019) and classification of medical images (Matthias Lenga et al. 2020) or for pattern recognition in physiological time series data (Kiyasseh et al. 2020). In the field of engineering, research effort is put into continual learning strategies for robotic systems based on supervised learning or deep reinforcement learning scenarios (Lesort et al. 2020; Wong 2016). The goals are mainly to enable intelligent and autonomous agents to perform lifelong learning of new tasks and to overcome the challenge of needing large data sets from real robotic controls by means of incremental learning strategies. Examples are found by Dehghan et al. (2019) and Ayub and Wagner (2020), where both work deal with visual object detection systems for robotic tasks that allow incrementally adding new objects.

With regard to manufacturing and production, few works exist yet on the use of continual learning methods. Tercan et al. (2019) continually train neural networks for predictive quality tasks, where the learning approach consists of a finetuning on the new task and a subsequent retraining on data of the old tasks, yielding better learning rates than training models from scratch. Maschler et al. (2020) use EWC for fault prediction of turbofan engines based on LSTM networks. Their results show that though EWC successfully prevents forgetting in similar tasks, its performance deteriorates as soon as the tasks are very different from each other. Tian et al. (2021) propose an online learning method for Long Short Term Memory (LSTM) networks in vibration signal prediction.

Injection molding is a manufacturing process for producing plastics in a single production steps (Kashyap and Datta 2015). During the process, plastics material is plasticized and subsequently injected into the cavity of the mold, where it is formed and cooled down to the final shape of the product. Injection molding involves complex behavior of interdependent process variables, with the mold design and the settings of the machine parameters in particular having a major impact on the quality of the product produced. The principle goal in this use case is to train a neural network for predicting the part quality, namely the maximum deformation under load, on the basis of six machine setting parameters (holding pressure level, holding pressure time, mold temperature, cooling time, melt temperature, and volume flow). Since the process behavior changes when producing a different product, we apply our proposed continual learning method to improve the performance of the model when applied to the data of a new part.

For evaluation purpose, we conducted molding simulations of different plastic brick specimens that provide the data basis. We designed 16 plastic bricks of different sizes by varying the number of studs on top of the brick (3, 4, or 6 studs per row, 1 or 2 rows) and the height of the bricks. Figure 3 illustrates two exemplary bricks with \(3\times 1\) studs and \(4\times 1\) studs. The simulations were performed with the software Cadmould 3D-F. For each part respectively prediction task, we varied the parameters in a central composite experimental design with 77 examples.

In our experiments, with the exception of those in Sect. 5.1, we train the neural network incrementally over sequences with 16 consecutive tasks (one base task and 15 following incremental tasks). Thereby, all increments are conducted in a sequential way. The learning of the first (base) task involves an untrained network and is regarded as the zeroth increment. In each increment, we train the model on training data of a new task and compute its loss a on separate test set. Thereby we use the last trained model for the new increment.

To obtain reliable results, we run the experiments on ten different task sequences, each with a randomly selected task order, and each sequence with five different shuffles for training and holdout testing data set (thus 60 experiments in total). In addition, we also run the experiments on a reduced data set (\(45\%\) of training data) to investigate how well the approaches perform on sparse data.

For the evaluation of the methods, we are mainly interested in both their ability to learn new tasks as well as the capability to retain already learned tasks. For the former, we compute the mean squared error (MSE) on the test data of a new task. For the latter, we compute the backward loss for an increment, which defines the (positive or negative) effect of the increment on the model’s performance on all previously trained tasks:where \(t>0\) is the number of the increment of interest and \(L_{i,j}\) refers to the test loss (MSE) of the ith task after the jth increment. A positive value—i.e. an increase in average loss—corresponds to a negative effect on previous tasks. Conversely, a negative value signifies that the tasks actually benefited from learning the new task. For a sequence of t increments, we can further compute the average backward loss over all its increments.

We compare both approaches, the regular MAS method and MAS-Cloning, with two baseline approaches:We implement our methods using PyTorch, an open source library for deep learning (Pytorch 2020). Prior to the main evaluation, we conducted initial grid search based tests on the data to identify the best performing topologies and hyperparameters for the neural network (see Table 1 for parameter ranges), resulting in a two-layer multi-layer perceptron (MLP) with 20 neurons per layer and the Rectified Linear Unit (ReLU) activation functions. Each training is performed using early stopping with a patience of 50 epochs and a validation set size of \(20\%\) of the available training set. The training is performed using the Adam optimizer (Kingma and Ba 2017) with a batch size of 16 and a learning rate of 0.001 throughout.

The result of the grid search, which was performed on the data sets using fivefold cross-validation, provided a MSE of 0.037 for the best performing neural network. In order to better assess this performance and to know how good the network is, we performed the same experiments with a polynomial regression of degrees two and three for comparison. The MSEs of these two methods are 0.046 and 0.044, respectively, thus significantly higher than the error of the neural network.

MAS has basically two hyperparameters, namely \(\lambda \) and \(\gamma \), which affect the abilities to learn new tasks and to avoid catastrophic forgetting. In order to find the best possible selection, we conduct an initial hyperparameter search by performing reduced experiments with 10 random sequences of seven tasks (one base task and six incremental tasks). Each experiment is further performed with five different data set shuffles. The box plots in Fig. 4 show the results for selected parameter combinations. Note that MAS with \(\lambda =0\) (green bars) does not use any regularization for the network training w.r.t. the \(\varOmega \)-loss.

The results clearly illustrate the trade-off between avoiding forgetting and learning of new tasks. On average, the model without regularization performs best on new tasks (mean loss of 0.04), but suffers greatly from forgetting. In contrast, MAS effectively prohibits forgetting for large \(\lambda \) values while only resulting in minor performance losses for new tasks. The best performing values are \(\lambda =1000\) and \(\gamma =1\) with a mean task loss of  0.05 and a mean backward loss of  0.04. We use these values in all subsequent experiments. It can also be seen that the loss values in the experiments vary significantly. We assume that this is mainly due to the fact that the plastic bricks and prediction tasks sometimes differ greatly from each other.

Table 2 (left columns) provides the computed average task losses and backward losses across all experiments when using \(100\%\) of the available training data. The task loss results show that though the MAS methods are constrained by regularization, they can learn new tasks similarly well as unconstrained models. In particular, MAS cloning performs best on new tasks—even slightly better than the finetuning approach. This also applies to the backward loss. While, as expected, simple finetuning causes the neural network to underperform on old tasks, both MAS approaches strongly prevent the network from forgetting when training on new tasks. Cloning has also a positive effect on forgetting. The reason might be that using pre-trained network weights leads to faster convergence of training and thus to fewer changes of the parameters. As a consequence, cloning leads to an average backward loss of almost 0.

The difference between regularization (MAS) and finetuning on forgetting becomes clear when looking at the network performances across the whole task sequences. Figure 5 shows how the task loss for the base task (increment 0) changes over all following increments. While the loss remains almost constant with MAS-Cloning and slightly increases for the regular MAS, finetuning significantly deteriorates the performance. Interestingly, its loss slowly decreases again with each task—a phenomenon we have observed often, but without the model ever reaching the original loss value again.

In order to evaluate the performances on sparse data, we conduct the same experiments with a reduced training data size of \(45\%\). Table 2 (right columns) provides their results. On the one hand, they show that still both MAS approaches prevent the network from forgetting, though the backward loss values slightly increase. On the other hand, finetuning and MAS-Cloning perform best for new tasks. In particular, cloning the output head significantly improves the data efficiency of our method. While the regular MAS method suffers from the data reduction, cloning ensures that even with sparse data very good performances are achieved, which are almost as good as a model trained from scratch on \(100\%\) of the training data. The differences between the methods are also illustrated in Fig. 6a, which presents the loss values for each increment. It can be seen that finetuning and MAS cloning perform better in each increment than regular MAS, which sometimes performs very poorly, and training a new network from scratch.

The finetuning approach performs almost as well with only \(45\%\) of the training data as with \(100\%\) of the data. This raises the question of how much data all other methods need for training a new task to achieve similarly good performances. For this purpose, we conduct further experiments by training the networks on different proportions of target training data. On this basis, we identify the proportion that is required to achieve the same test loss (with \(10\%\) margin) as with training on \(100\%\) of the data within the respective increment. Figure 6b depicts the proportions for all evaluated approaches. It shows that both finetuning and MAS-Cloning have a higher data efficiency than for example the regular MAS method. Starting with the first increment, MAS-Cloning requires only about \(50\%\) of the training data to achieve a similarly good performance as with \(100\%\) of the data. However, it can also be seen that its data efficiency does not improve over the course of the increments, so that a certain amount of training data is always necessary.

The results of our experiments show that cloning the output head of a similar previous task has a significant positive impact on the forward transfer of the neural network. To show that cloning must happen from a similar task, we compare our approach with a minor modification: instead of selecting the output head with the lowest loss on the new data for the initialization, we use the one with the highest error (called negative cloning). Figure 7a shows its test results, where the experiments are conducted the same as before. Compared to MAS-Cloning, negative cloning not only provides worse performance, it actually results in a larger loss than MAS with randomly initialized weights. This indicates a knowledge transfer that has a negative impact on new tasks, a problem that is called negative transfer in deep learning research.

A further closer look at the cloning approach reveals that the source task of which the output head is cloned for initialization is often similar to the new target task (similarity in terms of plastic bricks with similar dimensions). This leads us to the assumption that similar tasks also form similar network weights during training. To test this, we extract the trained weights (including the bias weight) of all output heads after a fully trained sequence of 16 tasks and visualize them in a two-dimensional space using the dimensionality reduction method UMAP (McInnes et al. 2018), see Fig. 7b. Each point in the plot accordingly represents one output head. For the representation of task similarity, we use the maximum flow distance of the plastic bricks. This parameter defines the distance the molten plastic must travel during molding and can be computed from the simulation for each brick. The figure shows that output heads for bricks that are more similar to each other are also closer to each other in the space and share similar network weights accordingly.