Synthetic image data augmentation for fibre layup inspection processes: Techniques to enhance the data set

Lightweight structures are now commonly used in aero-space manufacturing. The Airbus A350 XWB and the wing and fuselage production of the Boeing 787 are examples of an increasing demand for these lightweight components (Marsh 2010; McIlhagger et al. 2020). Compared to metallic materials, Carbon Fiber Reinforced Plastic (CFRP) offers superior stiffness and strength properties. Thus, lightweight structures are often made from CFRP. The manufacturing of these mostly complex lightweight structures is typically quite expensive. In order to make production economical, fast and efficient production techniques are essential. To meet the high safety requirements of the aerospace industry a visual inspection follows the fibre layup process.

Typically today, this manual inspection takes between 32% (Rudberg et al. 2014) and 50% (Eitzinger 2019) of the total production time. Moreover, due to the manual inspection process, it is sometimes impossible to fulfil the required inspection accuracy. This aspect offers great potential for improvements in quality and speed.

A crucial stage of the automated inline inspection is the reliable classification of manufacturing defects within a sensor image. Machine learning methods are very well suited for this purpose (Schmidt et al. 2019). Unfortunately, especially the very common approaches based on Artificial Neural Network (ANN), often require very large training data sets, which are quite difficult to produce in a reliable production process or during the development of a classification system (Huang et al. 2019; Tan and Le 2019; Zambal et al. 2019b) For this reason we investigate methods for generating a large training data set based on a few previously acquired real defect images for the Automated Fiber Placement (AFP) inspection process, in this paper.

The AFP technology is relatively novel, but is increasingly applied in industry. Thus, we have chosen this technique for further investigations in this paper, aiming a proper transferability of our research results (Cemenska et al. 2015; Weimer et al. 2016; Black 2018) Since a Laser Line Scan Sensor (LLSS) is frequently used in research and development for the inline inspection of AFP processes, in this paper we will focus on greyscale depth images from such a sensor (Cemenska et al. 2015; Weimer et al. 2016; Ucan et al. 2019; Meister et al. 2020) A LLSS is based on the principle of triangulation to obtain topology data from a laser beam that is projected onto a surface and reflected to a camera sensor at an angle to the laser illumination. The research question of this publication is:

Which methods can be used to generate synthetic image data of fiber placement defects from the AFP process, using a small data set with mostly less than 100 images?

This section gives details on the experimental setup as well as the test procedure and evaluation. For the studies in this paper appropriate defect types were chosen. According to those introduced in “Manufacturing process” section, no defect regions, wrinkles, twists, foils representing foreign bodies, gaps and overlaps were selected for the following studies. Figure 2 schematically illustrates these defect types. Accordingly, Fig. 3 displays six randomly selected and smoothed real defect images per class, which were used as inputs for examinations in this paper. They have been acquired using the experimental setup described below. The individual defect images were manually labeled in the overall LLSS scan image using the tool LabelImg (Tzutalin 2015). Based on these labels, the individual defect images were extracted and used individually for the experiments. For the investigations considered, we used a different number of defect images per defect class. This is because defect types like gaps and overlaps can be considered as the combination of many individual partial areas and thus several real and independent defect images can be extracted from a single defect. In certain cases defects are located quite close to the edge of the overall defect image in our database. Thus, they become useless as training samples due to pre-processing steps and filter effects at the edge of the image. All origin input images were previously resized to a reasonable size of 128 \(\times \) 128 px. This image size was chosen because the essential characteristics of a defect are still represented here, but the amount of data has been significantly reduced. Larger images may require additional layers in the ANN which in turn increases the training effort. The actual amount of data considered per defect type and the corresponding rounded half amounts are presented in Table 4. These “rounded half amounts” are needed for data compilation at a later stage and hence they are mentioned here.

In order to perform reliable investigations, representative original data must be acquired. These fibre layup defect images must be generated in a reproducible and representative way with respect to the actual fiber placement process. For this reason, a feasible experimental setup was applied, as shown in Fig. 4. This assembly is independent of disturbing influences from the manufacturing process such as contamination, thermal radiation or tilting of the layup effector. This test setup consisted of a KUKA jointed-arm robot, the Automation Technology GmbH (AT) C5-4090 LLSS (Automation Technology GmbH 2019) and a CFRP prepreg material sample.

In the following, the results of the three preliminary tests and the final two validation experiments are presented and analysed.

Table 9 presents the results of the manual visual image assessment for the generated conditional DCGAN data with batch sizes 64 and 128. Both perform very similarly, as you can see from the Fig. 5.

Only for gap and overlap defects the DCGAN with batch size 64 generates slightly superior quality images. Since a manual and therefore uncertain evaluation method was used here, we can assume that there is no significant difference in the quality and variance of the images. However, it should be noted that for a comparable number of epochs the training time of an ANN increases with rising batch size, as already described in “Methodology” section. Thus, it makes sense to choose a batch size that is as small as possible but of sufficient quality. For this reason we have selected batch size 64 for this use case and for the following experiments.

In the second preliminary test the generated images from different previously defined DCGAN generator (G) and discriminator (D) layer structures (G/D) are examined. The results of the manual visual image assessment are evaluated qualitatively and compared in Table 10.

These results indicate again only a perceptible difference for gaps and overlaps. We can see clearly that the DCGAN architectures (6/5) and (6/6) generate the best quality synthetic images. The structure (5/5) produces only mediocre artificial gap and overlap defect images. The composition (5/6) synthesises especially very poor gap defect images. Since variant 3 with the architecture (6/5) presumably yield the best defect images for all classes, this configuration is applied for further investigations in this paper. It should be noted, that the difference to the architecture (6/6) is maginal and therefore this configuration provides a reasonable alternative. However, since the focus in this paper is on the investigation of the general feasibility of an artificial image data augmentation of this particular AFP defect images, only the presumably best configuration (6/5) was considered for the subsequent experiments.

The parameter configurations presented in Table 11 were compared in the third preliminary experiment. The resulting quality of the generated synthetic images was evaluated visually.

The performance of the individual settings is color coded. The parameter sets 1, 7 and 10 generate qualitatively good synthetic images. The other settings create rather poor or unsuitable images. Except for the configurations 4 and 11, a dropout factor \(> 0\) seems to have a major negative impact on the quality and variety of the synthetically generated images. For setting 11 this deterioration in quality is also evident but considerably less than in the other configurations with an equal dropout factor of 0.25. With setting 4 the image quality is poor and the variety between the images is smaller, despite a dropout factor of 0. This setting combines a learning rate of 0.0001 with a \(\beta _{2}\) of 0.9. It is possible that the combination of both parameters has a significant influence on the quality of the synthesised images for the considered depth map data set of the AFP fibre layup defects. However, a generally valid conclusion cannot be derived from this since the settings of the DCGAN and the resulting synthetic images are highly depend on the input data set. Based on the visual assessment, configuration 1 generates the highest quality synthetic defect images. Furthermore, this configuration contains a presumably beneficial learning rate and \(\beta _{2}\) parameter combination. These are the reasons for choosing this parameter set 1 for the subsequent validation experiments. Figure 6 illustrates a visual representation of six randomly selected images per class which were generated synthetically with the DCGAN using parameter set 1. Consequently, the following network architecture from Table 12 is applied for the conditional DCGAN for the following validation experiments. The corresponding specific layer structure for the Generator is presented in Table 13 and for the Discriminator in Table 14. Accordingly, for the Generator 99.98% and for the Discriminator 99.91% of all the parameters are trainable.

The training of the DCGAN takes about 77 min in this scenario. For this the settings and computing hardware specified above were used for passing through the 20000 epochs. The generation of 5000 training images for each of the six classes, making a total of 30000 images, takes \(< 3\) min in this experiment.

Within this first validation experiment the quality and diversity of the generated images are investigated. Furthermore, the manual visual assessment is compared with the results from the GAN-Train GAN-Test evaluation. To review the realism of the synthetically generated images, they are compared with the illustrated example images from Fig. 3. For this purpose, the conditional DCGAN with the previously defined parameter set was applied. Figure 7 gives the mean value and the standard deviation of the three similar performed runs using the data sets from Table 7. The GAN-Train results are presented on the left and the GAN-Test results on the right hand side of the figure.

When looking at the GAN-Train confusion matrix we notice values \(> 88\%\) along the diagonal for all class assignments, except the non defective assignment. This has a classification rate of only 78.07%. These results indicate that the diversity of the defect patterns are fairly high. Beyond that this also means that the non defective test patterns look very similar to each other. The large standard deviation of \(\sigma = 23.36\%\) for the three generated data sets further indicates that the CNN classifier probably has difficulties in deriving suitable features from the non defect images. Obviously, this finding is plausible as an accurate and reliable fiber placement process is designed to achieve a consistently good fibre placement quality. This results in a very smooth LLSS depth image. The diversity of the images without defects thus should be slightly less than for the images with defects. Actual defect images are therefore subject to very strong variations in the appearance due to the characteristic defect shape. Furthermore, we notice that non defect images are classified as overlap defects with a mean value of 17.77%. This is likely due to the fact that the overlap defects also have less distinctive geometric attributes present in a LLSS scan image. Additionally this could indicate that the classifier applied for this validation needs to be properly configured to correctly distinguish between these two types of defects. However, it is also conceivable that the DCGAN generates an insufficient representation of these overlap and non-defect images. The comparatively high standard deviation of \(\sigma = 21.72\%\) is another indicator for potential deficits in the synthetic data generation using DCGAN or in the GAN-Train GAN-Test evaluation with the CNN classifier. However, since the synthetic non-defect images and the overlap images from the Fig. 5 and the Fig. 6 are visually very well distinguishable, we assume here that the deviating classification results rather indicate an insufficient configuration of the CNN classifier regarding these particular defect types. For the geometrically more complex defect types wrinkle and twist even mean values of \(> 96\%\) are yielded. This is due to the very characteristic shape which can simply be varied from an image generator. Furthermore, these defect geometries can be mapped easily to the feature maps of a CNN classifier. The result for the GAN-Test investigations differs slightly with regard to the value range of all mean values and regarding the weakest classification result. In this observation all mean classification results are \(> 94\%\), except for overlap defects with 87.36% mean classification rate. Overlap fiber placement defects appear very similar to gap defects or non defect images. Therefore the classifier is more likely to identify these types as gap defects or non defect images. This is plausible for the mix-up of gaps as well as overlaps and basically matches the visual impression when viewing a LLSS scan image. Nevertheless, this result is different from the correct classification, which leads to a decrease of this value. The mixing up of overlaps and non-defect images probably has a similar origin as discussed above for the GAN-Train results.

In addition, Table 15 presents the results for a comparison of the false positive and false negative results. The false positives values refer to the amount of no defects which are categorised as defect. False negatives are the number of defects which are recognised as no defect. This false negative rate has a special meaning here, because it describes the amount of defects that are missed. This is particularly problematic in a manufacturing process.

The numbers in the table indicate the influence of gaps and overlaps on the individual false values. Without considering gaps and overlaps the false values are < 2 %. Taking all defect types into account these false values vary between 2.65 % and 21.93 %. In this case the GAN-Train values are significantly larger than for the GAN-Test scenario. This relationship changes substantially for the observations without gaps and overlaps. In particular it should be noted that the false negative rate for the GAN-Train evaluation with 0.24 % is significantly lower than for the GAN-Test procedure. This indicates that the CNN classifier has a significantly better performance with respect to critical defect misses if trained with the synthetic DCGAN data. However, this is only valid if the gap and overlap defects are excluded.

The following second validation experiment serves as a comparison of the Geometrical Transformation and the conditional DCGAN for differently sized initial input data sets. The used data sets are introduced in “Methodology” section in Table 8, with the aim of performing a slightly modified GAN-Train GAN-Test evaluation. The corresponding results are presented in Fig. 8. For the results from the Fig. 8a, c, e ten images, half of the total or all of the available real training data are enlarged with the Geometrical Transformation to 5000 images for training the CNN classifier. To generate the results in the Fig. 8b, d, f the previously introduced data set AUG_DCGAN_2 is applied for the CNN training. The test data sets each consist of the remaining available original data, which has not been previously utilised for the data augmentation.

The image data generated with the DCGAN provide classification results with a total mean classification rate of 90.17% for all the different data sets. Thus, this assessment appears to be relatively independent of the size of the test data set. We recognise once more a slightly increasing misclassification between no defect images, gaps and overlaps as previously discussed. Particularly noticeable is the increasing number of twists being classified as overlaps. This unexpected behaviour is especially noticeable when comparing the results for ten and all real training images per class, with miss classification rates of \(> 20\)%. However, this tendency is also clearly apparent when considering the half amount of available test images from the Fig. 8d, having a misclassification value of 7.69%.

For the training images generated via Geometrical Transformation a distinctly heterogeneous behaviour appears from the evaluation of the different data sets. When applying ten initial images for training and the remaining available images for tests obvious classification deficits are evident for the defect types twist, foreign body and overlap. This is displayed in the Fig. 8a. Furthermore, we notice that foreign bodies are often recognised as wrinkles or twists. However, the classification results for no defects, wrinkles and gaps are unexpectedly high compared to the classification rate of foreign bodies in this experiment. Compared to the findings from Fig. 8 of the DCGAN generated images using all-10 test samples it is clearly evident that ten input images are not enough to model a sufficient diversity of defects and to train a CNN, even after a geometric augmentation. In contrast, the classification of only a few test samples can lead to a similar diversity problem. This makes it difficult to assess their realism. Nevertheless a robustly trained classifier is capable of properly classifying such defects. The results in Fig. 8a appear unrepresentative in comparison to the remaining results of this study. They seemed affected from a beneficial or non-beneficial aggregation of the randomly composed training data set. Considering the results from Fig. 8, applying half of the available data set for the geometric augmentation leads to a significant increase in the classification rate compared to the usage of just ten initial training images per class. Noteworthy here is the increase in the classification rate for foreign bodies. Due to the small number of defect images available of this type only one more initial training image was additionally applied. This fact strengthens the previous assumption of the low representativeness of ten randomly selected initial training images. Except for foreign bodies and overlaps, the CNN classifier trained with the part data set listed in the Table 4 yields classification rates of \(> 95\%\). This indicates a sufficiently good CNN classification rate for the remaining defect types when trained with only 25 to 47 initial defect images, depending on the class. The relatively low classification rate for overlap defect images is quite surprising, since the applied part data set with 83 images contains the largest number of training images of all classes. Thus, the previous findings of this paper are strengthened that especially the characteristics of overlap defects are difficult to abstract appropriately using image features. When comparing the results from the Fig. 8e, f we realise that the classification rate for no defects, wrinkles, twists, and foreign bodies is 100%. For the difficult to characterise gaps and overlaps we observe classification rates of \(> 95\%\). This results in the very great mean classification rate of 98.89% with a standard deviation of \(\sigma = 1.66\%\). Compared to these results, the CNN classifier trained with the DCGAN enhanced data set only yields a mean classification rate of 90.3% with a standard deviation of \(\sigma = 51.81\%\). These results illustrate the limitations of the traditional data augmentation regarding the diversity of the generated images. In this case, the images initially applied for the traditional data augmentation are only geometrically transformed, while the actual image content remains identical. Hence, obviously very good classification results are achieved when in this case a classifier is trained with these images which are basically the same for their subsequent classification. In contrast, the data generated with DCGAN obviously must have a different appearance than the original data. For this reason, the classification rate is usually \(< 100\%\) with a significant standard deviation, despite good results have been achieved. In summary, we would like to mention again that the considered GAN-Train GAN-Test method serves to investigate the diversity and realism of different defect types for the stated image synthesis techniques. The GAN-Test results in the Fig. 8d, f partly show lower classification rates than the corresponding GAN-Train results in the Fig. 8c, e. This only indirectly characterises the performance of the DCGAN. As already mentioned above, a great advantage of data synthesis with DCGAN is the different but realistic appearance of the resulting artificial images compared to the real recorded defect images. As a result, the GAN-Test analysis often achieves classification rates of \(< 100\)%. As this indicates varying synthetic data, these results are desirable. The extremely high classification rates of often 100% in the GAN-Train analysis reveal an insufficient diversity for the image data augmentation with the Geometrical Transformation. Thus, this can quickly lead to an overfitting of a classifier and would not offer any added value in real applications. We also want to clarify that the original real input images provide the basis for the data augmentation for all the results presented above. However, they are not part of the actual CNN training data set. For the investigations of all-10 test images and half of the data set, the remaining original images are used for tests. This data has not previously been considered for data enhancement. For the tests on all real data these images were initially considered for data augmentation, but they are not part of the actual CNN training data set.

Additionally, Table 16 summarises the calculated false positive and false negative values corresponding to the results in Fig. 8. Similar to the investigations in Table 15 the false positives values describe the number of no defects predicted as defects and the false negatives consider the inverse case.

In addition to the results from Table 15 we see that the false negative rates for training with DCGAN synthesised training data are very similar for all runs. For training with Geometrical Transformation augmented data these false negative rates and the corresponding false positive values decreases continuously with increasing amount of applied input data. In particular for the runs x.1 and x.2 the false negatives rates using the DCGAN training data are much less than for training with the corresponding Geometrical Transformation training data. These results indicate that the data augmentation for fibre layup defects is very beneficial especially for very small data sets. Therefore it is crucial to apply a valid and high quality data augmentation model. Below the presented results are discussed and compared with related studies.

The research discussed above of Radford et al. (2016), Perarnau et al. (2016), Neff (2018), Salimans et al. (2016) and Brownlee (2019) illustrate fundamental approaches for designing a DCGAN architecture. Unfortunately, these investigations are based on everyday pictures and medical image data. This data is rather different from AFP inspection images considered in this study. Thus, the investigations in this paper complement the ideas of Sacco et al. (2018). As they have suggested, we applied a GAN in this paper for artificial data augmentation, which they had briefly mentioned in their publication. Similar to Zambal et al. (2019b; a), we demonstrated the possibility of generating synthetic AFP inspection topology data. Furthermore, we theoretically evaluated different data augmentation techniques and investigated selected approaches in detail. However, none of the previous studies cover the detailed investigation of different DCGAN configurations for the inspection in fiber composite manufacturing. These necessary investigations are initiated in this paper and different feasible methods are compared.

On the basis of a detailed literature analysis the traditional enhancement method Geometrical Transformation and the deep learning technique conditional DCGAN are determined as suitable methods for an artificial data enhancement. Their performances as well as their advantages and disadvantages are conclusively demonstrated in two validation experiments. We also demonstrated that for the classes no defect, wrinkles, twists, foreign bodies and gaps probably 25 to 47 representative origin defect images are already sufficient to achieve a good CNN classification result after a proper data augmentation to 5000 training images per class. This fits quite well with the results of Tabernik et al. (2019) which used 25 to 30 initial data sets for training their ANN classifier for the surface cracks detection.

Overlaps behave very unevenly in our investigations. Thus, for the use of overlaps for algorithm training or data augmentation probably considerably more data is required than was available for this work. However, it is also possible that the input data needs to be more representative.

In this regard, we need to mention that training the DCGAN with all possible variations of defect patterns is practically impossible. This is also not necessary, since the DCGAN abstracts the defect pattern and learns their appearance. Thus training the DCGAN with a various realistic defect images is more important than training with as many as possible images. In this study, fibre layup defects with different characteristics were recorded and then transformed geometrically. Nevertheless, it is important to note that this is a replicated process. For a reliable comparison of the synthetic defects with industrially arising process defects, an extensive empirical analysis in a real world production environment would be essential. However, this was not part of this study.

In order to provide a basic assessment of the diversity and realism of the synthetic defect images, the GAN-Train GAN-Test method was used successfully in this paper. With this GAN-Train GAN-Test method we have demonstrated that the DCGAN generates diverse but realistic training data, similar to the expectations from the literature review. According to its operating principle the Geometrical Transformation generates very realistic synthetic data, but the variety of these artificial images is very limited. This can easily lead to an overfitting of a classifier. Beside that, Cubuk et al. (2019) have already demonstrated for another applications, that a significant improvement in classification performance can be achieved with a reasonably configured Geometrical Transformation.

Considering the results mentioned above, the research question is answered. The Geometrical Transformation method and the DCGAN based technique can be used to generate synthetic image data of fiber placement defects from the AFP process, using less than 50 initial representative data samples. Only overlap defects shall be an exception in our studies. Since the data augmentation and training with overlap defects behave quite unexpectedly in our investigations, further tests should therefore be examined. For these investigations also the gap and no defect classes need to be taken into account. In further investigations, it seams to be reasonable to examine the robustness of the proposed methods for other sensor settings and different materials with deviating optical material properties. In this context, it might also be beneficial to investigate further promising methods from the literature summarised in the Table 2 for application in this adapted scenario.

In summary, the aim of this study was not to increase the classification rate of a CNN classifier but to compare and evaluate different data synthesis methods for the AFP inspection application case. In order to apply these techniques to a real world scenario, representative defect image data from the corresponding inspection process need to be recorded and utilised for data synthesis. Therefore an important aspect is to ensure the diversity of the initial training images. Thus these images should consist of many potential defect characteristics. The challenge is to extract these striking defect images from a real process. In most cases, typical defects occur, which each contain very similar characteristics. Obviously, this can lead to the fact that in real applications considerably more defect images have to be recorded than the actually required minimum quantity of representative training images. Moreover, the enhancement of the very simple CNN classifier used here is certainly advisable to achieve a very high and robust classification result.

The investigations in this paper demonstrate that the Geometrical Transformation and a reasonably configured conditional Deep Convolutional Generative Adversarial Network are well suited for the synthetic data generation from less than 50 representative origin images per class. The GAN-Train GAN-Test method proves to be a suitable tool for the independent evaluation of artificially generated image data. However, this method has the inherent property that it links the diversity and realism of the images always to another defined comparison data set, which has to be taken into account appropriately.

The data synthesis techniques investigated in this paper offer major advantages for the fibre composite industry. Firstly, this paper provides the necessary information for the utilisation of suitable data synthesis techniques for inspection image data from fibre composite production. Thus the results of our studies emphasise the application of such methods. Secondly, the presented approaches enable the abstraction and synthetic generation of potentially confidential manufacturing data. Thus a sufficient amount of realistic data can be provided easily to developers of inspection systems, without spreading potentially confidential manufacturing information. Furthermore, these artificially generated depth images can be applied for a simulative optimisation of an AFP inspection algorithms.