Deep learning study of induced stochastic pattern formation in the gravure printing fluid splitting process

The experimental setup from Schäfer mainly consisted of a modified laboratory sheet-fed rotogravure printing machine, a high-power white light emitting diode (LED) light source, a beam splitter for confocal light input, and a high-speed camera Photron Fastcam SA-4, see Fig. 3. The core of the modified machine was an optically accessible substrate carrier, which replaced the standard opaque substrate carrier. Schäfer used a printing cylinder with 120 electromechanically engraved quadratic test fields (13 mm x 13 mm) with different raster frequencies (40–140 lines/cm) and tonal values (5–100 %) at a constant raster angle of \(45^{o}\). This enabled him to study the impact of the underlying gravure raster pattern and transfer volume on transient pattern formation in the printing nip. Additionally, the printing velocity (0.5–1.5 m/s) was varied in the experiments. Ethanol was used as the printing fluid. Printing trials with 185 different parameter combinations of raster frequency, tonal value, and printing velocity were performed three times each, resulting in 555 high-speed video recordings. A simplified sketch of the setup as well as an exemplary video snapshot with annotations is presented in Fig. 4. For more details of the experimental setup see Schäfer et al.¹⁹

Schäfer’s high-speed video recordings revealed a remarkable variability of dynamic liquid bridge and wetting patterns, see Figs. 4 and 5. The video recordings display the rolling line where the printing cylinder is in direct contact with the substrate. In the case of lamella splitting, this line was enclosed in a fluid seam, a few millimeter in width, extending along the whole height of the engraved test field on both sides of that line. The quadratic test field is the region of interest (ROI) in the videos. The fluid menisci on the converging and on the diverging side had clear optical contrast by the light refraction at the liquid–air interface. While the meniscus situated on the converging side of the nip (‘converging meniscus’) was an almost straight border and oriented parallel to the cylinder axis, the meniscus on the diverging side (‘diverging meniscus’) showed characteristic corrugation into more or less periodic finger-like structures with periods that were typically in the range of a few 100 \(\mu\)m. The finger frequency was especially dependent on the raster frequency and independent from the printing velocity and the mean gravure cell volume.²⁰ Analogous to the finger phenomenon at the retracting meniscus in the Hele-Shaw cuvette experiments of Saffman and Taylor, the fingers were the effect of air intrusions penetrating into the nip. In certain cases, extended branched dendrite-like patterns evolved, and even complex, disconnected structures were observed. As the air intrusions were always topologically interconnected, the tips of the air fingers were apparently in a steady pressure equilibrium with the air outside. For this reason, we can exclude gas bubble cavitation as the origin of any air volume in the seam. The liquid phase, however, was only occasionally connected. Rather, isolated, mostly corrugated, and irregularly shaped liquid structures were detaching from the liquid seam, also drop-like structures. Each such structure was the footprint of liquid bridges between the substrate and the gravure cylinder surface.²⁴ In certain cases, the liquid structures (e.g., drops and fingers) coincided with the gravure raster of the printing cylinder, but there were also regimes with a larger drop and finger size than the gravure pattern, and periodic finger patterns of an integer multiple width of the gravure raster appeared. A frequent lock-in of the finger width at three or four times the raster width was observed even when the printing velocity was changed, and was apparent to the bare eye. In particular, extended finger patterns were either aligned with or, alternatively, inclined against the printing direction, and more correlated with the skew axes of the gravure raster. Thus, the transient patterns showed a considerable variety of regimes in meniscus dynamics, which we consider to deserve a phenomenological classification.

Accordingly, a complementary, extended-scale view on the classification should be useful which considers the spatial correlation of these patterns, revealing point symmetries in the autocorrelation function and in the spectral Fourier transform. We have already studied the pattern formation with 1D Fourier methods in reference (25). However, there are more options to detect hidden periodicities in a stochastic 2D pattern. Folding such functions with representations of 2D crystallographic, i.e., of the planar uniform discontinuous groups,²⁶ extracts the particular weights of these symmetries from the patterns. These weights and their variation with printing parameters could serve as order parameters of symmetry breaking transitions, and help to identify specific regimes of pattern formation. NNs could be efficient in recognizing such symmetries in the highly stochastic image data, and one might also train them by use of the respective representation functions.

Artificial intelligence (AI) is one of today’s worldwide major topics of interest. Self-driving cars, face recognition, and predictive maintenance are just some familiar examples for applications of AI in industry. A subdomain of AI is machine learning (ML). According to Zhou, ML ‘is the technique that improves system performance by learning from experience via computational methods [...] and the main task of ML is to develop learning algorithms that build models from data’.²⁷ ML problems can be divided into three classes: supervised learning, unsupervised learning, and reinforcement learning. In supervised learning which is for example used for regression or classification tasks, labeled data is needed on which the ML model is trained. For example, to classify certain classes of iris flowers according to the length and width of their sepals and petals as given in the famous iris data set,²⁸ the class or the so-called label needs to be known to train the ML model. In contrast, unsupervised learning does not need labeled data. For example, clustering tasks can be performed using unsupervised learning. Reinforcement learning cannot be considered as supervised or unsupervised, since it does not include labels but it involves taking feedback from the environment.

DL is a subdomain of ML. We speak of DL, when deep neural networks (DNNs) are used to, e.g., cluster or classify the data. DNNs are NNs with many layers of neurons and they try to mimic the behavior of biological NNs, e.g., the human brain. In this study, we use DL for the classification of videos showing different classes of hydrodynamic pattern formation. There are many well-known architectures for DNNs which are available from online repositories. For image classification, convolutional neural networks (CNNs) are commonly used. They incorporate so-called convolutional layers in their architecture which detect and enhance certain features of the input image, e.g., edges, horizontal lines, circles, or much more complex shapes. The extracted features are fed to subsequent NN-layers. In the case of pretrained CNNs, the convolutional layers have already been trained on well-known public data sets like ImageNet.²⁹ There is a difference between DNNs used for image classification and those used for video classification. Since videos are a chronological sequence of single images, DNNs designed for video classification take the temporal relationship between the images into account.

A DL model needs to be trained before it can be used for the intended purpose, e.g., for image classification. The training duration is specified by the number of training epochs. One training epoch describes one cycle in the process diagram in Fig. 6. First, the DL model receives input data. Then, the model makes predictions and the predictions are compared with the labels of the input data. According to the chosen loss function (e.g., crossentropy loss), a loss value is calculated and according to the chosen optimizer (e.g., Adam), the weights of the DL model are updated. The updated DL model again makes predictions and the next training epoch starts.

In order to effectively train a DL model, a large enough data set is needed, the so-called training data set. In addition, a test data set, and possible, also a validation data set for hyperparameter tuning are needed. Hyperparameters describe the parameters that can be changed during the training process, e.g., number of epochs and learning rate. The test data set is used to evaluate whether the DL model was able to generalize or if it rather learned the data by heart. The latter is called overfitting which leads to a poor performance of the DL model on unseen data even though it performed well on the training data set. Should overfitting occur, e.g., hyperparameters or even the architecture of the DL model have to be adjusted iteratively. This iteration process yields a final trained DL model. The performance of this final DL model is tested on the test data set which has not been involved in the iteration process. A validation data set is only necessary if the model’s hyperparameters have been tuned iteratively. To avoid overfitting, several techniques are generally used. One is called data augmentation which is used to increase the variety of the data set. Examples for data augmentation are rotation, translation, or mirroring of the images from the data set before feeding it to the DL model. Another common technique to avoid overfitting is data set balancing. An image data set is called unbalanced if there is a large discrepancy between the number of images for every class. To obtain a balanced data set, e.g., some images of over-represented classes can be ignored from the data set or a so-called cost-sensitive training can be performed. Figure 6 shows the general concept of training DL models. In the case of cost-sensitive training, the loss function is weighted according to the representation of the classes. The weights are higher for under-represented classes. The performance of DNNs can be assessed using different approaches, for example: A confusion matrix shows true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions in a matrix. Table 1 shows a confusion matrix for a binary classification problem.³⁰ Often, it is also interesting to know with which confidence a DL model has predicted a class. A prediction results from the class probability for each of the two classes in case of a binary classification problem. The class which is attributed to the higher probability is the predicted class. As an example assume that an image of a dog was classified with a class probability of 95 % as a dog and with a class probability of only 5 % as a cat. Thus, the model’s prediction is ‘dog’ with the DL model being very sure about its prediction. If the class probabilities were 55 % for dog and 45 % for cat, the prediction would also be ‘dog’ but the model would be rather unsure about its prediction. Based on the TP, TN, FP, and FN values, the so-called accuracy can be calculated [equation (1)]:³¹The accuracy of a DL model can be calculated for the training, the validation and the test data set. We refer to training accuracy, validation accuracy, and test accuracy. The higher the accuracy, the better the performance of a DL model at least in the case of a balanced data set, i.e., when there is a similar number of samples for each class. CAMs are heat maps that reveal the areas of an input image that were most important for the classification decision of the DL model.³² The warmer the color, the more important is the area for the classification decision. If applicable, the heat maps can be overlaid with the original image. For further reading on ML and DL, we suggest the textbooks from Zhou,²⁷ Joshi,³¹ and Rebala et al.,³⁰ from which much of the information in this subsection is taken.

We trained six different DL models over 20 training epochs on the training data set and compared the test accuracies of all trained DL models. All models achieved test accuracies of more than 94 %, see Table 3. Model #2 was found to be the best performing 3-class-model with a test accuracy of 99.5 %. We observed that for a resolution of 224 px x 224 px the 3D-CNN architecture achieved higher test accuracies than the RNN architecture. For the same architecture and resolution, the 2-class-models performed better than the 3-class-models and also had a shorter training duration. For the RNN architecture, an increase in resolution lead to higher test accuracies of 2- and 3-class-models, respectively. Concerning the training duration, the RNN architecture models trained on lower resolution frames of 224 px x 224 px (models #3 and #4) required the least amount of training time, about 1 h, whereas the 3D-CNN architecture models and the RNN architecture models with higher resolution (models #1, #2, #5 and #6) had about three times longer training durations.

The test accuracy plotted against the training epoch for all 2-class- and 3-class-models as well as for an optimized model #2 is depicted in Fig. 9. The 2-class-models curves (Fig. 9a) are more closely spaced in comparison to the 3-class-model curves (Fig. 9b). Additionally, we applied data set balancing and data augmentation for model #2. We found that the implementation of data set balancing did not exceed the previous best test accuracy (99.5 %) (Table 3). In fact, data augmentation slightly decreased the test accuracy to 97.8 %. However, the test accuracy curve for model #2 with data augmentation showed a rising trend, see Fig. 9c.

We also investigated the effect of increasing the number of frames per video for training and testing of model #2, see Table 4. Four frames were used as the default number throughout this study. Training of model #2 with default frame number took about 3.5 hours and yielded a test accuracy of 99.5 %. By reducing the number of frames per video down to one, the training duration of model #2 decreased to roughly a third compared to the default value. However, the test accuracy was also decreased by around 2 % to 97.3 %, leading to more confusions especially concerning the MR videos, compare Fig. 16 in Appendix. When doubling the number of frames per video to eight frames, the training duration also doubled but the test accuracy roughly stayed the same. This could be associated with the fact that the additional frames did not contribute relevant information about the printed patterns, since only a small fraction of the test field was visible in the additional frames, see Fig. 8. However, increasing the number of frames per video from one to four improved the classification accuracy specifically for the MR class. This indicates that time correlations in the transient patterns were important for the MR classification, whereas there was almost no effect for the LSR and PSR class recognition.

Confusion matrices for the test data set for all trained DL models can be found in Fig. 10. The confusion matrices reveal that the 3-class-models, model #2, #4, and #6, (Fig. 10a) never confused the LSR and PSR, apart from one exception: model #4 misclassified one PSR video as LSR, see Fig. 11a. Therefore, the 3-class-models had implemented the characteristics of PSR and LSR very well, with only one wrong event among more than 150 classifications. All other misclassifications that occurred for the 3-class-models, 17 in total, involved the MR class. Consequently, the MR evoked by far the greatest amount of confusions. The number of confusions for one model can be calculated by adding together the entries that are not on the main diagonal of its confusion matrix. The confusion matrices for the 2-class-models (Fig. 10b) illustrate that only one confusion happened in total, namely model #3 misclassified a LSR video as PSR, see Fig. 11b. A list of all confusions can be found in Table 6 in Appendix as well as confusion matrices for model #2 for different numbers of frames per video, see Fig. 16.

We investigated the class probabilities for models #1 and #2 and plotted them as boxplots in Fig. 12. Model #1 is a 2-class-model which was trained only on PSR and LSR videos and reached a test accuracy of 100.0 %. When observing the class probabilities, it became clear that model #1 was very decided of its classification. For example, LSR videos were assigned a class probability for PSR of around 0 % and a class probability for LSR of around 100 %. The same applied to videos labeled as PSR in an analogous manner. In contrast, for videos labeled as MR, model #1 was rather indecisive. This behavior was expected, since model #1 was not trained on MR videos. For videos labeled as MR, the class probabilities for PSR and LSR each ranged from 0 to 100 %, although the median class probability for PSR was around 10 % and for LSR 90 %. In other words, model #1 tended to classify a MR video as LSR. For model #2, a 3-class-model with 99.5 % test accuracy, class probabilities are also displayed in Fig. 12. LSR videos were assigned a class probability for PSR and MR of around 0 % and for LSR of around 100 % with only a few outliers. The same behavior applied to PSR videos in an analogous manner. Thus, model #2 was very confident about its classification decisions concerning videos labeled as LSR or PSR. However, for MR videos, the model was less confident, since there were more outliers and larger box sizes in the boxplot, but it was still more confident than model #1. Model #2 assigned a MR video the median class probabilities for PSR and LSR of around 0 % and for MR of around 100 %. Additional boxplots for the class probabilities of models #3, #4, #5, and #6 can be found in Appendix in Figs. 17 and 18.

CAMs of three exemplary videos applied to all four main layers of model #2 were evaluated, see Fig. 13 (LSR video), Fig. 14 (PSR video), and Fig. 15 (MR video). The CAMs visualize the areas that were most important for the model’s classification decision. The first layer yielded four CAMs that were overlaid with the four original video frames so that the hot parts of the CAM could be directly connected to certain features of the video frames. Whereas the first layer detected numerous small features, e.g., the gravure raster dots, small droplets, or single fluid fingers, the DL model focused on larger features in later layers, e.g., groups of features. The second layer of the DL model yielded two CAMs and the third and fourth layer resulted in one CAM each. Consequently, these CAMs could not be assigned to the four original video frames and were plotted without any overlay. This is because a 3D-CNN not only convolves the spatial dimensions but also the temporal dimension. In other words, the model looks at each input frame individually in the first layer and in later layers considers all input frames at once for the classification decision. Interestingly, especially in the first layer, model #2 also paid attention to features outside the ROI, e.g., as seen in Fig. 15a. The region outside the ROI was much hotter than the ROI itself. However, in later layers (15b-d), the model tended to focus on the ROI. We observed this behavior also for many other videos from the Schäfer data set, although not presented here.