The extraction of meaningful features from the monitoring of laser processes is the foundation of new non-destructive quality inspection methods for the manufactured pieces, which has been and remains a growing interest in industry. We present ConvLBM, a novel approach to monitor Laser Based Manufacturing processes in real-time. ConvLBM uses a Convolutional Neural Network model to extract features and quality indicators from raw Medium Wavelength Infrared coaxial images. We demonstrate the ability of ConvLBM to represent process dynamics, and predict quality indicators in two scenarios: dilution estimation in Laser Metal Deposition, and location of defects in laser welding processes. Obtained results represent a breakthrough in the 3D printing of large metal parts, and in the quality control of welding processes. We are also releasing the first large dataset of annotated images of laser manufacturing.
Notes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
Laser technologies play a key role in high precision processes in sectors such as automotive industry and aeronautics. In these sectors a continuous improvement towards the goal of “zero defects manufacturing” is essential to reduce costs and avoid exploitation problems. For this reason, new techniques are continuously developed for on-line quality monitoring and fast defect detection. We focus in two laser processes, Laser Metal Deposition (LMD), and laser welding.
LMD is a manufacturing process based on the creation of a molten pool into which metal powder is injected as particles. This new additive technology is expected to enable 3D printing of net-shaped large metal components and is replacing previous techniques for coating and repairing critical parts in metallurgic industries. Currently, the biggest challenge for LMD is the lack of control in process dynamics at the high deposition rates required for large pieces (Song et al. 2012). Process monitoring in real time is critical for the efficient and fast manufacturing of larger and more complex parts (Hauser and De Weck 2007).
Advertisement
Laser welding industry is a mature technology implemented in most automotive industries. However, defects are a significant issue in production lines and any deviation in the position of the work piece produces defects and non-compliance with quality standards. Process monitoring in real-time and detecting any deviation from the desired parameters will lead to a dramatic reduction in repairing costs and significant increase in product quality.
Automatic quality control in laser processes is challenging and stable online monitoring systems are still in early development stages. Nowadays, destructive analysis and visual inspection are the main techniques for defect detection. The automation of this process will suppose a great improvement in the production lines. In addition, many parameters such as track geometry or dilution can only be obtained through destructive testing, as metallographic analysis and 3D scanning. These procedures cannot be used for on-line quality control since they are either destructive or time-consuming. A vision system inspection allows the online monitoring showing key indicators of the studied processes.
Monitoring and control of manufacturing processes have significantly evolved, and recent research efforts rely on the use of non-intrusive sensors that can work with accuracy in a harsh environment, such as Charged Couple Device (CCD) or Complementary Metal–Oxide–Semiconductor (CMOS) cameras, photodiodes (Liu et al. 2019) and pyrometers (Song et al. 2012). Among these sensors, CCD/CMOS cameras provide the most comprehensive information from the process. These cameras are commonly used to obtain cladding height or geometry (Meriaudeau and Truchetet 1996; Iravani-Tabrizipour and Toyserkani 2007; Bi et al. 2007), particle-in-flight velocity (Doubenskaia et al. 2004; Meriaudeau and Truchetet 1996), powder delivery distribution (Doubenskaia et al. 2004), and melt pool features such as surface temperature (Meriaudeau and Truchetet 1996) and thermal images (Hu and Kovacevic 2003; Bi et al. 2007).
Thermal imaging is an effective technique for defect detection (Jager and Hamprecht 2009; You et al. 2015) defect classification (Lapido et al. 2015) and laser power control (Hofman et al. 2012; Rodriguez-Araujo et al. 2012). In particular, high-speed Medium Wavelength Infrared (MWIR) imaging, has the benefits of being low cost and providing a high thermal dynamic range (Rodríguez-Araújo et al. 2017), and has been successfully applied to the control of laser processes (Panadeiro-Castro et al. 2018; Garcia et al. 2018).
Advertisement
The equipment used for laser welding and LMD is ideal for monitoring the molten pool with coaxial imaging. However, two major barriers limited the success of previous monitoring and control approaches. On one hand, images in the visible range from traditional sensors present problems of visual noise due to signal overflow and reflections exacerbated by dynamic range issues. These problems are solved with the appearance of low cost high-speed uncooled sensors in the MWIR range (Rodríguez-Araújo et al. 2017). On the other hand, obtaining measurements from images of the process is a challenging task due to the unintuitive nature of the images and the difficulty in obtaining annotated data with parameters of interest.
The image processing field is experiencing a huge transformation and many applications are meeting industry requirements with the use of Convolutional Neural Networks (CNN). A CNN is a type of feed-forward deep neural network based on a multilayer perceptron (Rosenblatt 1961), and designed to analyze visual information with minimal preprocessing. Schemes based on CNNs have a stunning ability to recognize images, objects, and image characteristics (Krizhevsky et al. 2012). These models use a labelled set of examples to train a model with convolutional elements that extract meaningful representations of the data without a handcrafted approach.
CNNs approaches are capable of analysing MWIR thermal images (see Fig. 1) to extract parameters of laser processes (Kwon et al. 2018) and quality indicators (Aminzadeh and Kurfess 2018; Zhang et al. 2019). We designed ConvLBM as a modular system that allows on-line quality control and defect detection in manufactured components. These capabilities will lead to the development of new control approaches for industrial laser manufacturing.
Fig. 1
Examples of MWIR images acquired during a Laser process
×
Materials and methods
We assumed that high-speed MWIR images capture well the heat flows that characterize the quasi-steady states of the melt pool, and that such steady states can be encoded in a CNN, providing an unequivocal representation of the process (Garcia et al. 2018). With this perspective, ConvLBM is trained using a two-step approach.
First, we built a large dataset labelled with laser power and process speed. The selected variables are easy to control and good indicators of the energy and mass input. This is supported by recent efforts on modelling (Pinkerton 2015) and control (Tapia and Elwany 2014), where parameterization studies assume that two intrinsic magnitudes, enthalpy and mass input, drive the process and determine key quality indicators such as dilution, clad bead geometry and porosity (Zhong et al. 2015).
To that end, 405 tests were recorded using different sets of parameters within acceptable ranges. Training ConvLBM on this dataset, the network captures the dynamic changes of the laser process from the raw images, without the need of manual feature extraction.
Then, we used a technique called transfer learning. This technique applies a second training to the model, using a smaller dataset specific for every process and laser configuration. With transfer learning, ConvLBM can predict new quality measurements while retaining the knowledge about the process gained from the initial training.
Transfer learning is used in two scenarios. In a first experiment we used images from the LMD process to estimate dilution, quality indicator that can only be measured trough destructive testing. In a second experiment laser welding images are used to detect defects in the manufactured parts.
ConvLBM network
ConvLBM is a CNN model that predicts laser power and speed from 28 × 28 raw MWIR coaxial images of a laser process. We implemented ConvLBM using Keras1 and TensorFlow2 frameworks.
Figure 2 details the architecture of ConvLBM. This design is a modification of ResNet (He et al. 2016), a network that shows good results on the CIFAR-10 dataset, a well-known benchmark for image applications. This architecture, uses blocks made of four operators:
Fig. 2
ConvLBM architecture
Convolutional layers, which performs the feature calculation on the image. They are a special type of perceptron with known performance in computer vision (Bengio 1997).
ReLu activations, that add non-linearities to the model (Ioffe and Szegedy 2015).
Batch normalization, a regularization technique to accelerate training and to reduce overfitting (Ioffe and Szegedy 2015).
Shortcut connection, which performs an identity mapping carrying the input to the next block and skipping the convolutional layers. This approach tries to solve the saturation problem of deep networks, that results in higher training errors when adding more layers (He et al. 2016).
×
ConvLBM is made of 4 residual blocks followed by one fully connected layer. Residual blocks add the output of a shortcut convolutional layer to a stack of three convolutional layers with filters of different sizes. We use a ReLU activation and a batch normalization step after every convolution.
We trained ConvLBM minimizing a mean squared error loss function on a first dataset and used the resulting net as a base model to apply transfer learning in different quality control applications. Thus, we replaced the last layer of ConvLBM and retrained it to measure dilution in LMD applications and to detect defects in laser welding scenarios.
To estimate the dilution, we added an additional neuron to the output layer in order to represent this variable, and trained the new model using the dilution dataset and minimizing mean squared error.
To detect defects in images from laser welding, we replaced the output layer with a softmax layer that represents the output as a category. The training minimizes the categorical cross entropy on the defects dataset, obtaining a classification tool. This tool detects defects “segment-by-segment” in a component that is being manufactured.
Datasets
We built three datasets of images for the different application scenarios and then, divided each one into a training and a validation set. The images were obtained with a Tachyon 1024 FPA camera at 1000 fps mounted coaxially to the laser head. The primary dataset is a large set of images annotated with the parameters of laser power and process speed. Then we created two smaller datasets; one with dilution information and one with labeled defects.
The primary dataset consists of over 1600,000 images from 405 tracks of laser cladding, with 316L steel as powder and base material. Every track was cladded with different values of laser head speed and of laser power chosen from a range of values that delivered good quality tracks. We divided the dataset into a training and a validation set. In the training dataset, the dataset was resampled to obtain a set of images for every tested combination of laser power and speed, resulting in a total of 269,235 images. The validation dataset contains 268,470 images from 90 tracks (2983 images per track), and represents evenly distributed parameters over the range of tested laser power and speed.
The dilution dataset consists of LMD images with M2 tool steel as powder and base material and using the same camera configuration as in the previous experiment. M2 is known to be more challenging than 316L, and has significant differences in process dynamics. The dataset contains over 8000 images from 20 clad tracks labeled with the dilution value obtained from a manual inspection by experts. We used a strategy of leave-one-out cross validation, training the system with 7600 images from 19 clad tracks and using the remaining one for validation. The process was repeated 20 times using a different track for validation in every iteration.
The dataset of defects was acquired from a welding process with an overlap joint laser configuration and DP600 as material. The camera configuration is the same as in the parametric dataset. We recorded 50 tracks with 24,444 images labeled with a binary classification (good joint or defected joint), obtained from a metallographic inspection by experts. The datasets are divided in two sets of 5/6 and 1/6 for the training and validation sets respectively.
Results and discussion
Process representation
In the first experiment, we trained ConvLBM to predict the known laser parameters power and speed, two parameters that drive heat and mass input in a laser process, and thus, are representative of its overall state. The first dataset was used for this experiment, with the objective of making a model capable of extracting meaningful features from raw images.
The results prove that ConvLBM is able to capture the process dynamics using a regression analysis. A comparison between the true and predicted laser power and laser head speed is shown in Fig. 3.
Fig. 3
Estimation of laser power (a) and laser head speed (b) using ConvLBM. The magenta line corresponds to the mean of predicted values, boxes represent the 50% confidence interval, and bars represent the 99.3% confidence interval
×
ConvLBM achieved an accuracy of about ± 0.05 kW for power and ± 1.75 mm/s for speed with over 99% confidence. This accuracy is better than one step in the training stage for power (0.1 kW), and slightly higher in the case of speed (1 mm/s). Figure 3 shows the complete validation results.
Dilution estimation
Dilution is a major quality indicator in LMD and cladding applications. During manufacturing, a low dilution may lead to insufficient bound and produce warping, while a value too high indicates large heat affected zones and a high probability of defects due to thermal expansion. Dilution can only be measured with accuracy through metallographic inspection, a destructive procedure that requires cutting the sample.
For the second experiment, we used the dilution dataset and applied transfer learning to the model of the previous experiment. The re-trained model was able of predicting dilution measurements (Fig. 4) with a RMSE of 2.8%.
Fig. 4
Estimation of dilution by ConvLBM. The magenta line corresponds to the mean of predicted values, the green represents 1 standard deviation (68.2% of values) and the blue area represents 3 standard deviations (99.7% of values)
×
It is worth noting that in metallographic inspection, dilution is measured at a specific transversal cut, and the assumption is made that dilution is constant over the track. In practise, variations of dilution may occur on the track and affect the apparent performance of the model, especially if a peak value occurs in the position of the cut.
A significant outcome of these results is that, despite the base model being built on LMD images and with a different material known to present a different thermal behaviour. ConvLBM achieved a good generalization from one material to the other.
Detection of defects
The detection of defects is a clear quality indicator in industrial laser welding scenarios. The thermal cycles in welding processes generates residual stresses due to thermal expansion, which results in cracks or deformations that weakens the manufactured part. In order to detect these defects, the analysis and visual inspection of the piece is needed. Usually a human operator checks the quality, weld by weld, and remakes the defective pieces so they are compliant with the quality standards. Most common defects are lack of fusion, undercuts, holes and pores.
For the third experiment, we used the defects dataset and applied transfer learning to the model of the first experiment, with the objective of creating a model able to detect defects during the laser welding process.
ConvLBM achieved in this experiment a F1-Score of 0.975 and 0.974 in the training and validation sets respectively. Table 1 display the confusion matrices for the training and validation set, and Table 2 summarizes the most common classification metrics for both validation and training sets.
Table 1
Confusion matrix for defects in laser welding in the validation set
True defective
True non defective
Predicted defective
2555
63
Predicted non defective
73
1383
Table 2
Classification metrics for the training set and the validation set
Training set
Validation set
Recall (TP rate)
0.973
0.972
Fallout (FP rate)
0.043
0.044
Specificity (TN rate)
0.957
0.956
Miss rate (FN rate)
0.027
0.028
Precision
0.976
0.976
Accuracy
0.968
0.967
F1 Score
0.975
0.974
The results indicate that the model is able to accurately identify defects in laser weld beads from raw IR images. Furthermore, this model was tested in an industrial scenario, detecting defects in real-time. Figure 5 shows an example of a successful defect detection in the laser welding case.
Fig. 5
Real-time defect detection on a laser weld bead. a and c show two side views of the weld bead where the blue rectangles mark a defective section in the first and final segments due to undercuts and the yellow ellipses mark a region where some points have excessive porosity. c and d show the corresponding output of ConvLBM where red colour show frames labelled as defective and green show non defective frames
×
It is important to note that, like in the dilution estimation, ConvLBM achieved a good generalization to the new material used in this experiment, which restates the generalization capabilities of the algorithm.
Conclusions
We present a novel approach to monitor laser processes in real time based on MWIR imaging. The ConvLBM model is based on a residual CNN architecture trained with a novel dataset of images from LMD to capture the basic dynamics of laser processes and can be extended to different scenarios by modifying the output layer, and re-training the network. This training procedure fits well to common calibration in industry practice.
ConvLBM is able to estimate the dilution for LMD applications and in the detection of defects for laser welding, obtaining very promising results.
In LMD processes, ConvLBM can monitor accurately parametric indicators that can be used in real time applications to define a control loop that keeps LMD processes in the desired window.
For laser welding, we developed an on-line defect detection tool capable of determining the position, size and length of defects. This tool can be easily integrated to increase the overall quality and reliability of the product.
These results show that ConvLBM can estimate laser parameters from high-speed thermal images and generalizes well to different processes and geometries. Furthermore, the good results of the model in three different materials (316L steel in the process dataset, M2 tool steel in the dilution dataset and DP600 dual phase steel in the defects dataset), proves its flexibility and adaptability to new materials.
In future work, we will validate ConvLBM with new indicators representative of cladding quality such as porosity or strength and more tests performed in a broader range of materials. Monitoring and controlling these indicators is crucial in 3D applications of LMD, and better reference magnitudes for new control systems should be achieved.
Acknowledgements
This publication and research was partially funded by European Union’s Horizon 2020 research and innovation programme funds under GA ref. H2020-FoF-2015-680481 and GA ref. H2020–FoF-2016-723945.
Compliance with ethical standards
Conflicts of interest
Carlos Gonzalez-Val, Adrián Pallas, Verónica Panadeiro and Alvaro Rodríguez declare that there is no conflict of interest regarding the publication of this paper.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
