Detecting Process Anomalies in the GMAW Process by Acoustic Sensing with a Convolutional Neural Network (CNN) for Classification

Abstract

Today, the quality of welded seams is often examined off-line with either destructive or non-destructive testing. These test procedures are time-consuming and therefore costly. This is especially true if the welds are not welded accurately due to process anomalies. In manual welding, experienced welders are able to detect process anomalies by listening to the sound of the welding process. In this paper, an approach to transfer the “hearing” of an experienced welder into an automated testing process is presented. An acoustic measuring device for recording audible sound is installed for this purpose on a fully automated welding fixture. The processing of the sound information by means of machine learning methods enables in-line process control. Existing research results until now show that the arc is the main sound source. However, both the outflow of the shielding gas and the wire feed emit sound information. Other investigations describe welding irregularities by evaluating and assessing existing sound recordings. Descriptive analysis was performed to find a connection between certain sound patterns and welding irregularities. Recent contributions have used machine learning to identify the degree of welding penetration. The basic assumption of the presented investigations is that process anomalies are the cause of welding irregularities. The focus was on detecting deviating shielding gas flow rates based on audio recordings, processed by a convolutional neural network (CNN). After adjusting the hyperparameters of the CNN it was capable of distinguishing between different flow rates of shielding gas.

1. Introduction

Gas metal arc welding (GMAW) is one of the most used welding processes in the world and provides high productivity besides low equipment costs. The process is highly automatable and is used in different industry sectors like the automotive, structural engineering or shipbuilding sectors [1]. Despite the high level of automation of the welding process, most test procedures are off-line. Non-destructive testing as well as destructive testing are time consuming and therefore costly. Destructive testing is particularly expensive due to the fact that the examined workpiece is gone after testing.

Due to the high costs of off-line testing, many researchers and industry sectors are moving towards on- and in-line testing. In particular, GMAW methods for on-line testing are being investigated. For this purpose, a wide range of sensing techniques have been used [2]. Two of the most used sensing techniques are the arc voltage and current signal [3,4]. These process parameters are easy to acquire and required sensors are not expensive [5]. Huang et al. showed that differing process parameters like shielding gas flow rates or voltage parameters change the stability of the welding arc in short arc mode, which can be seen in the multi-scale entropy (MSE). They used a genetic algorithm support vector machine to classify different formed weld beads with the MSE as the input feature and reached a classification result of 92.36%. Cao et al. presented a nonlinear model to estimate the weld penetration in pulsed GMAW with the acquired welding voltage.

Other investigations have used computer vision techniques to determine the weld quality [6,7,8]. Zhang et al. and Feng et al. used a CCD camera to acquire images of the weld pool during gas tungsten arc welding (GTAW), which is similar to GMAW. In GTWA the welding arc burns between a tungsten electrode and the workpiece and the filler material can be added externally. Due to the high intensity of the welding arc, an appropriate illumination or filter system is required. For this purpose, Feng et al. used a laser for illumination. Zhang et al. used a mirror and filter system to acquire images of the welding arc, the welding pool and the bottom of the welding pool with one camera so synchronous acquisition was not necessary. Both investigations used deep learning techniques to determine the weld penetration state. Ma et al. also used a laser but as a line scanner. With the filtered profile information of the welded bead and a neural network with an adaptive learning rate, weld surface defects could be detected. They achieved a recognition rate of 98.15%.

Another sensing technique is to collect the emitted radiation of the welding arc with a spectrometer [9,10,11]. Chen et al. stated that the resulting spectra provide a great opportunity to analyze the shielding gas composition or the welding wire composition during the weld process, which has a significant impact on the stability of the weld process and therefore the reachable weld bead quality. The combined visual images of the wire arc and the spectra are used as the input. Two sub-branches are implemented to handle the visual and spectral data and are finally fused to determine the shielding gas flow rate and differentiate between different welding wires.

In manual welding the welders control the process with their senses, for example through hearing. With their experience and knowledge, they can easily adjust process parameters and guarantee good weld quality during the whole process [12,13]. The adapting of this kind of in-line testing has also been a subject of different research in the past years. The audible sound emitted during the weld process has gained particular attention and is being examined by many researchers as an additional sensing technique to those mentioned before.

A correlation between arc current, arc voltage and the emitted arc sound was examined by Drouet et al. [14]. They described the relationship between the arc energy and the emitted sound in a formula that was verified by Cayo et al. [15,16]. The formula is shown in (1) with S_a representing the emitted sound pressure of the arc in dependence of time. Within this equation, k is used as a proportional factor, which considers a geometrical factor α, the adiabatic air expansion coefficient γ and the speed of sound in the air c to take the transmission in air into account. The arc energy is described as the temporal change in the arc voltage V and the arc current I. Drouet et al. performed their experiments with graphite electrodes, thus no metal transfer or shielding gas were included and the investigated relationship can be seen as a good approximation. To obtain a better approximation, Wang et al. proposed a new approach that uses the second and third derivatives of the energy. Due to this, the new formula enables getting closer to the real values, but as they are only fitted on measured data, the formula is not generally useable [17].

$$S_a\left(t\right)=k\frac{d\left(V\left(t\right)\ast I\left(t\right)\right)}{dt}withk=\alpha \ast \frac{\gamma -1}{c²}$$

(1)

Cayo et al. also stated that shielding gas fluctuations are not detectable by calculating the sound pressure with welding current and voltage. Horvat et al. [17] investigated the general behavior of the welding arc and the resulting sound. They welded bead on plates and recorded the emitted sound with a microphone, which had a constant distance to the arc of 350 mm. They determined that the main sound source is the arc ignition and extinction with high peaks in sound pressure. Between those peaks, the pulsation of shielding gas mainly influences the emitted sound. Wang et al. [18] used GTAW to characterize the sound arc. They examined that the arc emits sound in a dipole shape. Furthermore, experiments with different shielding gas flow rates were performed. It has been seen that the acquired sound pressure varies with the gas flow rate, but no prediction models have been developed. The mentioned investigations focused on analyzing the arc sound generation and the influencing parameters like welding current, welding voltage or shielding gas flow rate. They showed that the sound amplitude increases with increasing gas flow and is constant if the flow rate is over 15 L/min; nevertheless, they used the welding parameters as dependent variables and the sound as the resulting one, but for quality monitoring it should be vice versa.

However, many researchers have used the audible welding sound as the dependent variable to predict the welding penetration state [19,20,21,22] as one kind of weld irregularity. They used different methods for feature extraction and machine learning algorithms to develop prediction models. Zhang and Chen used a fusion of audible sound, voltage and spectrum signals to distinguish between the three penetration states full, normal and under penetration. The sound signal was decomposed with a wavelet transform to gather more information from the time domain as well as the frequency domain. They calculated the relative wavelet energy and used it as the input for a support vector machine (SVM). The resulting classification accuracy with all sensors fused was 96.58%. If only the sound was used as the input they reached an accuracy of 92.08%. Zhang et al. [23] performed the same experiments but with a new approach of selecting relevant frequency bands. They decomposed the acquired signal with a wavelet transformation, filtered it and then used Fisher distance and PCA for selecting the relevant frequency bands. The PCA features were then used to classify the three weld penetration states with the help of a support vector machine. They achieved an accuracy up to 98%.

Yusof et al. [24] used GMAW in their experiments and varied the welding parameters to force the appearance of weld irregularities as pores. Three different gas flow rates were set up for this. With wavelet decomposition they revealed that pores can be detected by different wavelet details.

Different sensing techniques and their main applications during the GMAW or GTAW process were discussed. GTAW is used more often for weld experiments owing to the fact that with a non-consumable electrode a much more stable process is the result. By contrast, the GMAW exhibits a greater dynamic during the process. The carried-out experiments are mainly performed in laboratory environment and similar background noise like in an industrial environment is not provided.

The described sensing techniques like spectra or welding current are abstract to human welders and need good visualization during the process to determine weld quality. Hearing as a second sense that can be used by the welder is more common and is already used by human welders. With their expertise a validation of the acquired sound can be performed and a better understanding of the algorithms used can be provided. The reliability of algorithms used can also be investigated [13].

The research about the generation of the audible arc sound has shown that it is mainly influenced by the electrical energy during welding. Other investigations have used the audible sound to detect weld penetration or the presence of pores in the welds. Those irregularities are caused by different process anomalies and are rare if the welds are performed normally with best fit parameters. To overcome this issue the researchers emulated such process anomalies by, for example, varying the shielding gas flow rate or adapting the welding current and voltage. This leads to the assumption that not the irregularity itself was detected, but the process anomaly.

In this study the aim was to detect a lack of shielding gas coverage as a process anomaly and not the resulting bead irregularity. For this purpose, GMAW weld experiments with different shielding gas flow rates on low alloyed steel were completed and the audible welding sound was acquired in a housed experimental setup to reduce environmental noise. Then, different data processing steps were applied to the acquired data to prepare them for classification in a CNN. With the developed and tuned model, relevant frequency bands were examined. The prediction results and the information value of different frequency bands will be discussed.

2. Experimental Setup

The experimental trials for the welds were carried out on a 3-axis linear motion system with an EWM Phoenix 355 Expert 2.0 puls MM TKM as the welding power source. The schematic setup is shown in Figure 1. The welding power source is a digitally controlled welding source, which uses synergic curves to control the welding process. The chosen curve for the weld experiments supplies a short arc. The substrate was fixed on the weld table to ensure reproducibility.

The audible sound emissions during the weld process were measured by a condenser microphone. It provides a nearly linear frequency response, so the whole bandwidth from 20 to 20,000 Hz can be used without adjustments. The polar pattern is cardioid, which reduces the acquisition of background noise or sound reflections from the back of the microphone. The required 48 V of phantom power was supplied by a Palmer PAN 48. The output of the phantom power supply was connected to the Dewetron DEWE-PCI 16 measuring system to acquire the sound pressure. To obtain the sound pressure, the sensitivity of the microphone with 25 mV/Pa was used to calculate it from the acquired voltage signal. A sample rate of 48 kHz was used to fulfill Nyquist Sampling Theorem for audible sound. Air temperature and humidity were not documented during the experiments. To ensure a synchronous acquisition, a trigger from the CNC controller was used. The CNC controller was also used to start the weld process and to set up the gas flow rate with a proportional valve.

In this research, low-alloyed welding wire (ER70S-6, G4Si1) with a diameter of 1.2 mm was used as a consumable electrode for the GMAW process. Furthermore, the welds were performed on base plates (S355J2+N) with the dimensions 200 × 150 × 5 mm to achieve comparable results. The materials used are of high interest in industrial sectors such as civil engineering, steel construction or architecture. The common chemical compositions of the welding wire and the base plate material are listed in

Table 1. Chemical composition of materials used.

Material	Cmax	Simax	Mnmax	Cumax	Femax
S355J2 + N (1.0577) [25]	0.20	0.55	1.60	0.55	balance
G4Si1 (1.5130) [26]	0.07	1.05	1.75	0.05	balance

. To reduce interference between the welded beads they were welded with 25 mm spacing between the middle axes of the beads, to fit eight beads on one base plate. The positions of the beads on the plate and the welding order are illustrated in Figure 2. The beads were welded from the left to the right side of the plates with a time difference of about two minutes between each weld, due to changing the position of the handling system and the gas hose flushing.

Due to the fact that the sound pressure is distance-dependent, it was necessary to maintain a constant distance between the microphone and welding torch, therefore the microphone was mounted to the robot flange in the same way as the welding torch. Additionally, not only the distance is a relevant factor. The directivity pattern of the sound source also has a strong impact on the acquired sound pressure. Wang et al. have shown that the directivity pattern of the arc can be described as a dipole [17], i.e., the sound amplitude is dependent on the angle between the arc and the microphone that is used for acquisition. If the microphone is placed orthogonal to the welding arc column the sound amplitude is low. Hence the amplitude is highest when the microphone is placed parallel to the welding arc. According to these results, the microphone was placed parallel to the welding torch with a distance of 380 mm to the arc as shown in Figure 2. The microphone was placed trailing the arc.

The wire feed rate was set to 3.5 m/min and welding voltage to 18.6 V to produce high-quality welds. The linear axes moved with a travel speed of 250 mm/min. The contact tube to workpiece distance (CTWD) was set to 12 mm. The short arc process used provides a lower heat input, which can reduce the component distortion. In Figure 3 the welding sound is shown in comparison to the welding current and voltage. The short circuits and the resulting unsteady droplet transfer can be seen. The achieved mean welding current is 150 A and the mean welding voltage is 19.4 V, which is 0.8 higher than the voltage that was set.

As the shielding gas, a mixture of 2% CO₂ and 98% argon [27] were used, and the initial flow rate was set to 15 L/min. During the experiments the gas flow rate was incrementally decreased in-line by steps of 1.5 L/min until no shielding gas was used.

The welding experiments were performed on the same day one after another. This reduced the effect of anomalies in data acquisition due to random events. The flushing of the welding hose after every gas flow rate adjustment should ensure a constant gas flow. This was necessary due to the fact that no measurement of the flow rate could be implemented right before the shielding gas flows through the gas nozzle in the welding torch.

For safety aspects and to reduce the influence of surrounding machines, which also emit audible sound, the whole experimental setup was housed in protective glass. This glass softened the environmental sound and reduced the ultraviolet rays emitted by the welding arc. Additionally, a welding fume extractor was placed on the top of the housing.

3. Data Preprocessing

Considering fast and unique identification of the acquired data and the physical plates, an ontological model was developed. It is displayed in Figure 4 and shows the core features of the model. Firstly, it contains the date and time of data acquisition and when the file was created. As it is only possible to create one sample at a time the combination of creation date and time can be used as unique indices. Secondly, the physical specimen is addressed. The plates used were marked with consecutive integers starting with one in the lower-right corner. The bead placed at the top of the plate was indexed as bead number one. All beads lower than the first one were also indexed by consecutive integers. The layer was kept constant to one in this research. Thirdly, the class of the weld is described. “gas” was used as a prefix for the classes and was followed by a two-digit integer that represents the gas flow rate as a percentage of the initial setup of 15 L/min. The acquired sensor signal was saved. In this research only, the emitted sound was acquired. With the help of the ontological model, a small database was created to store the information about the weld experiments and the records were named with it. Additional information can easily attach, and traceability is given at every time. The setup of machine learning algorithms also benefits from it.

Data augmentation was implemented by slicing the sound data into two second slices. This multiplies the number of samples that can be used for training purposes. Furthermore, this enables an examination of the gas flow rate during the welding process and an online monitoring could be set up. The obtained slices were then filtered by a bandpass filter with the lower cut-off frequency at 500 Hz and the upper frequency at 20 kHz. The upper frequency yields from the upper bandwidth of the used microphone. The lower frequency was set after analyzing the frequency spectrum of the welding fume extraction emitted sound without any welding. The main energy is in the frequencies below 500 Hz, which can be seen in Figure 5.

Since experienced welders can distinguish between different welding parameters by hearing, a second filter stage with a mel-band filter bank was implemented. The mel-band filter bank tries to emulate the human ear frequency response. With applying the mel-bands on the acquired sound data, the ability of the human ear to extract relevant frequencies will be adapted. The human ear has a non-linear behavior frequency response, so lower frequencies can be distinguished better than higher ones. In this research, 44 mel-bands were used.

After filtering the signals, the short-time Fourier transformation (STFT) was applied to transform the sound data into the time-frequency domain. For this purpose, the hamming window was used to reduce the spectral leakage effect. The window size was set to 256 samples with 50% overlap. A comparison of samples without shielding gas coverage during the process and samples with normal flow rate for comparison are shown in Figure 6. Differences can be seen in the higher frequency bands, which are marked as red rectangles and correspond to the 30th and 35th mel-bands. The magnitudes are much higher for full shielding gas supply as for no shielding gas supply. After applying the STFT, a patch of the STFT with a size of 128 and an overlap of 50% were selected. Those patches were used to train the CNN. The final step in the presented data pipeline is normalization. To achieve this, the bins in the patch were normalized by their own mean values. This avoids overflowing weights in the neural network during the training process. The implemented data pipeline is illustrated in Figure 7.

4. Convolutional Neural Network

A 2D convolutional neural network (CNN) was used to detect the various gas flows. As mentioned before, patches of the generated STFTs were shown to the network. The detection was here formulated as a multi-classification problem. Each gas flow rate was assigned to a certain class; 15 L/min, for example, was assigned to the class “i.O.”. The remaining flow rates were used as described in the developed ontological model.

Due to the fact that the CNN generates features on its own, no feature extracting is necessary. The chosen structure of the CNN is shown in Figure 8. At first a convolutional layer with a filter size of 5 is applied, which generates a 16 feature maps. After every convolutional layer a max-pooling layer with a filter size of 2 is connected to reduce the dimension of the data. A dropout layer is also implemented after every convolution with a rate of 20% to reduce overfitting. The first combination of convolutional and pooling layers is followed by two more of those combinations. The filter size of the pooling layers is kept constant in contrast to this; the filter sizes of the convolutional layers are reduced to 3. The count of feature maps is doubled after every combination. Consecutive to the feature extraction section of the CNN, a dense classification network is used. It starts with a 1024 dense layer followed by a dense layer with a size of 256. The ReLU function was used as the activation function. As a consequence of the multi-class classification and the categorical cross entropy used as the loss function, the last layer used soft max for activation.

The CNN was trained on 80% of the generated data. The other 20% were used for test and validation purposes during the training. In total there were 201 two-second slices for training, 52 for validation and 49 slices for testing. The samples were randomly shown to the network in batches of 512 slices. The dataset for testing was balanced before it was used to validate the model. The training set was left unbalanced. The Adam optimizer was used as an optimizer, which ensures a fast and stable training. For training a total of 500 epochs was set, but with the option of early stopping above 300 epochs. Early stopping is used to reduce the training time if the loss is not significantly decreasing after a few epochs. Significantly means in this study a loss reduction of 0.001.

A simple 10-fold cross-validation was used for better validation of the model. The generated slices are split into 10 data folds as seen in Figure 9. As a first step, a single fold is used for testing (Te), another one for validation (V) and the other folds for training (Tr). In the following steps, two folds that differ from the last chosen ones are selected for testing and validation and the ones left are used for training. The model is tested for every fold and with the resulting accuracies for the single folds an overall accuracy is calculated. Therefore, every data set is once used for validation and testing and the overall capability to predict the gas flow rate can be evaluated.

The relevant frequency bands for the shielding gas prediction were examined as additional analysis. This was performed by adding a third filter stage with a band stop filter. The band stop filter was applied right after the first filter stage. The first lower cut off frequency was set to zero and the upper frequency to 750 Hz. All other cut-off frequencies were multiples of these bounds until 24 kHz, which led to 32 frequency bands. The CNN was then trained by those datasets as described above and the resulting prediction accuracies were compared.

5. Results and Discussion

The training with all classes showed that the classification for gas flow rates below 7.5 L/min is a trivial problem for the CNN. Thus, in the following investigation, only no shielding gas as reference and flow rates above 7.5 L/min (50%) were used for training purposes. The welded beads for these flow rates are shown in Figure 10. During the visual inspection of the weld beads with flow rates between 70 and 100% of 15 L/min, no defects were detected, therefore these welds were defined as good samples. Shielding gas rates below 70% produce surface pores and are visually not acceptable. The welded bead with no shielding gas coverage was highly interspersed with pores and was an unacceptable weld.

To evaluate the developed model, a confusion matrix was created by means of the 10-fold cross-validations with the test folds. The ground truth is faced to the predicted labels. In Figure 11 the confusion matrix for the model is displayed. The values represent the prediction accuracy as percentages. As seen before, no shielding gas coverage can be detected easily by the CNN. Furthermore, 100% gas flow rate can be detected with an accuracy of about 94%. The left 6% were predicted as 90% flow rate, which is only 1.5 L/min less than 100%. Air turbulence or other influences could affect the flow rate as it is slightly less than the set up. This phenomenon can be seen in all class predictions. With an overall mean accuracy of 84%, the model is able to distinguish between the classes.

In classification problems, two types of error can occur. There are type I errors (False Positive) and type II errors (True Negative). If a model predicts a true negative class like in this investigation a gas flow rate below 100% as a positive class (100%) it is a type I error. If a true positive is predicted as a negative one it is called a type II error. In quality assurance type I errors must be avoided, because a bad quality is declared as good. The red marked column in the confusion matrix shows the area of type I errors for the 100% class. It can be clearly seen that the negative classes were not predicted as the positive class. Therefore, no type I error was made for the 100% gas flow rate.

Difficulties in the prediction were identified for the classes 50% and 90%. Considering the noisy environment during welding and the low damping of the housing these can affect the prediction model. Examinations by listening to the incorrectly predicted samples proved that during the welding processes noise was also acquired.

Additionally, developing an appropriate prediction model was the first goal for this research. The second goal was to examine the relevant frequency bands that contain the main information about the shielding gas flow rate. For this purpose, 750 Hz wide frequency bands were left out for classification and the resulting prediction accuracies were compared. The results are displayed in Figure 12. The red line represents the mean accuracy over all trained models.

At three frequency bands the accuracy drops significantly. This indicates that the model did not have enough features. This leads to the conclusion that the skipped bands are relevant features. Those features are located at higher frequencies. Furthermore, leaving out certain frequency bands does not lead to better prediction results, i.e., the left-out band did not contain misleading information.

Visual inspections are one method of testing weld beads, but internal irregularities cannot be detected. Therefore, radiographs were taken and analyzed to examine internal defects. A small extraction of the resulting images is displayed in Figure 10. The beads were evaluated based on DIN EN ISO 5817 [28], with the goal to detect weld irregularities according to DIN EN ISO 6520-1 [29]. Assessment group B was chosen, which represented the highest possible quality grade. The visual inspection suggested the beads with a shielding gas flow rate higher than 70% only had a few individual pores. At 70% there was a small pore nest with an outer diameter of 3 mm. Those irregularities were permitted by norm. For flow rates below 70% tube pores were found. The length of the pores exceeded the maximum allowed length of 25 mm. Since the visual inspections for 60% were similar to the radiographs, the inspections for 50% were inconspicuous. Nevertheless, in the radiographs a tube pore was found. As expected, the bead with no shielding gas coverage was permeated with pores. Therefore, the beads with lower shielding gas were not acceptable by norm. Acceptable beads can be detected with an average accuracy of 92.85% for the classes that are above 70% shielding gas flow rate.

6. Conclusions

Monitoring the audible sound emitted by the GMAW process is often based on the experience of the welders. Since the computational capabilities have increased in recent years, new research approaches using machine learning algorithms can achieve more knowledge from the emitted sounds. To obtain an appropriate dataset with bead irregularities, process anomalies must be emulated by varying the weld parameters. However, training the prediction models on these artificial datasets leads to the assumption that the process anomaly was detected and not the irregularity.

• Deviations during the process need to be detected and correlated to achieve high-quality results with a precise prediction capability.
• In this work, an optimized data acquisition setup was developed and implemented. The housing reduced the influence of environmental noise, but the prediction results have shown that the damping is not sufficient, and noise was acquired. Owing to the synchronization of the data acquisition system and the welding machine, other data streams can be recorded additionally, which enables multi-sensor fusion. This can improve the capability of the implemented prediction model.
• With an overall accuracy of 84%, the model is able to predict the shielding gas flow rate during the GMAW process. It is a highly dynamic process so small deviations can have a great impact. Neighbored flow rates were predicted as each other as they can behave similarly. Due to noise, lower accuracy was achieved in some data sets, especially in the 50 and 90% welds. Moreover, no type I errors for full shielding gas coverage have been made, which is an important feature for quality measurements.
• After developing the model, relevant frequency bands to predict the shielding as flow were examined. Three bands were identified to contain more information about the flow rate. They were investigated by leaving out certain bands with a bandwidth of 750 Hz and the model was then trained on these data. If the mentioned three bands were left out, the resulting accuracy dropped significantly.

Further investigations will analyze different gas mixtures or arc modes to validate a general use of audible sound for detecting process anomalies. Those changes will greatly affect the acquired sound samples due the fact that different gas mixtures have a great impact on the arc. The weld curve will change and so will the sound, as stated by Drouet. Similar effects will occur if the welding current or voltage are changed. To overcome this, more data must be acquired with different parameters. The developed model can then be used as a pretrained model and transfer learning can be applied. In addition, the spacing between the different flow rates will be reduced to set up a regression model. This can improve the prediction accuracy due to less blurred bounds. Adding more sensor data like welding current or voltage can also improve the prediction capability and will be investigated.

In manual welding, the algorithm can be used to support experienced welders as well as new welders. The necessary shielding gas flow rate is given before the welding starts as a threshold and a real-time online prediction of the actual shielding gas flow rate can be done for 0.5 s frames. If the flow rate is lower than the threshold, the welder can be informed. This can help both new and experienced welders to improve their torch handling, which greatly influences the shielding gas coverage. External effects like windy conditions can also be seen and will help to improve the weld quality.