Hyperspectral imaging for prediction of surface roughness in laser powder bed fusion

The area-scan hyperspectral camera MQO22HG-IM-SM5X5-NIR from Ximea (Table 1) was used in this work to capture process emissions. The special characteristic that makes this sensor unique is a coating of the CMOS sensor with 25 different Fabry-Pérot interference filters that each covers an individual pixel. These filters are arranged in 5 * 5 mosaics covering the 2048 * 1088 pixels from the entire sensor resulting in a reduced spatial resolution of 409 * 217 but with added spectral resolution represented by 25 channels. With a 875-nm short pass filter protecting the pixels from laser light reflections at 1070 nm, these 25 spectral bands lie between 600 and 875 nm. The filter also annihilates the second harmonic in the Fabry-Pérot filter’s spectral transmittance.

Figure 1a illustrates the pattern of the Fabry-Pérot interference filters on the CMOS sensor chip. The height of the filters correlates with the wavelength they are transmissive to. Figure 1b shows the plot of the quantum efficiencies of the 25 bands taking into account the effect of the short pass filter. The sensitivity in one pixel is not confined to one narrow wavelength band around the peak, but has several minor peaks at other wavelengths. This effect can be accounted for by a so-called correction matrix. Besides, the configuration is prone to noise from crosstalk induced by non-parallel light beams. Further noise is caused by dark current, vignetting effects as well as chromatic aberration of the optical setup.

The major benefit of this sensor technology, namely the combination of spectral as well as spatial resolution, comes with certain costs: comprehensive correction/calibration, reduced spatial and temporal resolution when compared to high-speed cameras [25], and reduced spectral resolution when compared to spectrometers [13]. Area-scan hyperspectral imaging thus fulfills a niche whose potential shall be evaluated and discussed in the remainder of this article.

Since powder properties strongly affect the reproducibility and quality of the LPBF process [19, 22], several powder characteristics were determined. Table 2 summarizes the crucial properties. The LPBF process is most reliable as long as powder particles are spherical and hence show good flowability. The SEM images in Fig. 2 give a qualitative reference for the particle’s shape whereas the so-called Dynamic Particle Image Analysis provides a quantitative measure for the particles sphericity. These results along with the Hausner-ratio imply an adequate flowability even though the SEM image reveals some fine particles that usually impede flowability [3]. It is also critical that the powder be sufficiently dry which was shown in the Karl Fischer Titration method.

The experiments were conducted with a laboratory LPBF machine providing high flexibility with regard to process control as well as hardware accessibility. The machine is schematically shown in Fig. 3a where the camera is integrated off-axially. The features of the machine are summarized in Table 3.

The area-scan hyperspectral camera was integrated off-axially at a tilt angle of 20^∘ such that the depth of focus of was sufficient to ensure a sharp image. The objective’s working distance was fixed at 190 mm, the magnification ×1.0, and its field of view (FOV) thus corresponding to the sensor size. Figure 3b shows the orientation of the manufactured part on the magnesium AZ31 substrate. One rectangular layer was built with the dimensions 14 mm * 10 mm with the camera’s FOV in the center with dimensions of 10 mm * 6 mm. This way, a maximum number of frames is collected from the center of the scan vectors. During the LPBF process, all parameters except laser power and scan speed were kept constant.

Data acquisition and preprocessing scripts were implemented using the camera manufacturer’s application programming interface for the programming language Python. This was beneficial for subsequent data evaluation through methods of machine learning since these also rely on Python. Figure 4 shows schematically how data acquisition, storage and subsequent evaluation were organized. At the start of the process, the frames were continually captured and the maximum grey value was determined. As soon as the laser spot reached the FOV of the camera, this value quickly shot up. The storage of the images therefore started when the maximum grey value had reached a threshold of 20 for at least 20 ms (at an exposure time of 4 ms this corresponds to five consecutive frames). The frames were then saved at full resolution as a numpy-array data type and were successively put into a list until the grey value dropped below 20 for 20 ms. This was the case when the laser spot moved out of the camera’s FOV and the coating process started. Then, the first data processing occurred: for every frame, the pixel coordinates of the maximum grey value were determined which then served as the center for converting the 2048 * 1088 frame to a 100 * 100 frame. The cropping was then done such that the spectral bands’ pixel locations were identical for each post-processed frame. Efficient data processing requires manageable amounts of data and so the rest of the image was discarded.

All frames that were captured in one layer were saved after cropping in a so-called pickle-file that served as a container of the Python list. The processing of the frames is usually interrupted as soon as the next layer is built. For the experiments presented here, however, only one layer was built. The following data processing happened only after the build job has finished. At first, the 100 * 100 * 1 frames were dissected into 20 * 20 * 25 frames to sort the different spectral channels. As soon as the samples were investigated and the quantities of interest were determined, each frame could be assigned a label—in this case the surface roughness R_z. The result was a Python list where each element was again a list containing the frame as a numpy-array and the label as a float. This list could be used as input for a convolutional neural network.

The target value of the investigations was the surface roughness of additively manufactured WE43 alloy. In order to keep ambient conditions as reproducible as possible, only one layer was manufactured while the process was monitored with the hyperspectral camera. Three parameter combinations of laser power and scan speed were selected from literature [8]. One was reported to lead to dense parts and good reproducibility (50 W; 75 mm/s), one to too much energy input (50 W; 40 mm/s), and the final one to too little energy input (20 W; 75 mm/s). The roughness R_z was measured according to DIN EN ISO 4288 using a Keyence VK X1050 confocal microscope. The cut-off wavelengths λ_C and λ_S were set to 2.5 mm and 8 μm, respectively.

The measuring section l_n was therefore 12.5 mm long. With a FOV of 10 mm * 6 mm, the roughness was measured only along one direction so that most of the measuring section lay within the FOV. Figure 5a shows the confocal image of the first sample and the location of the five measuring sections. The average of the five roughness values was taken as the final value that was used as a label for all the camera images taken at the respective sample. For each sample, the model is thus provided with a large number of images that are labeled with the identical roughness value. This means that the model is trained to find features in the hyperspectral data hinting at global process characteristics leading to a certain global roughness.

Table 4 shows the surface roughness measured of the different samples. Sorted by parameter set, for each sample, the average measured roughness R_z is given along with the number of frames acquired from the hyperspectral camera for this particular sample in brackets. The right column finally displays the mean roughness for the parameter set and the mean absolute error (or deviation) of that set. These values will help evaluate the prediction power of the convolutional neural network used subsequently.

The correlation between the surface roughness and the hyperspectral process recordings shall be determined. As laid out in previous sections, the images contain spatial as well as spectral information about the process emissions. This information could potentially contain crucial features for the prediction of the surface roughness. Since these features are unknown, a method from machine learning, a so-called convolutional neural network, is applied. The hyperspectral measurement is done in the high-temperature melt pool environment. Therefore, the temperature has an obvious influence on the hyperspectral measurement. Since the training data is labeled with the roughness, however, the model is trained to extract precisely the features related to roughness, not temperature.

Typically, the input data for a CNN are images with one or three color channels. The process recordings in this case, however, entail 25 channels, such that the network topology is adapted accordingly. Therefore, pre-trained networks that have achieved exceedingly high classification accuracies in other areas (e.g. ResNet-models) cannot be applied in this case.

On the one hand, the CNN shall provide a high prediction accuracy while on the other hand it may not require more computation power than was available, both for the training and for the later classification process. That is why an approach with four convolutional layers was chosen (Fig. 6) whose number of channels is doubled in each except for the last layer. The dimension of the convolutional matrix was fixed at 3 * 3. The ReLU-function served as activation function. The input data in the form of the process images entailed a large number of channels (25) but was otherwise very small (20 * 20) which is why no max-pooling layers were applied. To avoid overfitting, a dropout layer was added to the network followed by a fully connected layer before the final output is given. The mean squared error was chosen as a loss function to penalize high deviations between prediction and label during training. As evaluation metric, the mean absolute error was used, so that it could directly be compared to the values in Table 4. To compare the prediction power of different network topologies, a hyperparameter tuning was conducted during which the dropout rate (0.4 and 0.6), the number of perceptrons in the fully connected layer (8, 16, 32), and the optimizer (Adam and Nadam) were varied. Table 5 shows the results of the training for 10 epochs, respectively, with a batch size of 20 with the corresponding hyperparameters. Of the data available, 70% was used as training, 20% as validation, and the remaining 10% as test data. The test data was used to determine the prediction accuracies shown.

For better clarity, the results are visualized in Fig. 7, such that the influence of individual hyperparameters becomes apparent. It can be seen, for instance, that the Nadam-optimizer allows higher prediction accuracies than the Adam optimizer. The smallest mean absolute error and thus the highest prediction accuracy was achieved using the Nadam optimizer, a dropout rate of 0.6 and 16 perceptrons in the fully connected layer. A simplified scheme of this network topology is shown in Fig. 6.

With these hyperparameters, a longer training process of 40 epochs was carried out. To ensure that for each parameter set the same proportion of data went into training, validation and test set, the parameter sets were divided separately. This means that for each parameter set, 70% of the data was allocated to the training, 20% to the validation, and 10% to the test data set. Only after that, the training and validation data was shuffled. The test data was kept separate so that the prediction accuracy could be evaluated for each parameter set individually. This allows to determine whether the CNN really recognizes different surface roughnesses or only distinguishes between parameter sets.

Figure 8 shows the evolution of the loss function during the training process for training and validation data. Despite the high dropout rate of 0.6, overfitting can be observed starting at around epoch 22. The network learns the features of the training data “too well”.

Table 6 shows how accurately the CNN can predict the surface roughness based on the test data. For the complete data set, all three parameter sets were used. The prediction accuracy of the CNN can be expressed by an MAE of 4.1 μm which is much lower than the MAE of the roughness within the data set of 14.3 μm. It should be noted that the results of parameter set A contribute slightly more to this result than the other parameter sets due to sheer number of images in each set. As a next step, the separate test data containing only frames from one parameter set were used as input for the CNN. Table 6 shows that for parameter set A (50 W; 75 mm/s), the parameter itself is recognized, but the roughness cannot be predicted more accurately than is already implied by the statistical spread within that set. For parameter set B (50 W; 40 mm/s), on the other hand, the roughness can be predicted more accurately than implied by the inherent spread. For this set, there must be features in the hyperspectral data that contain information about the surface roughness. Parameter set C (20 W; 75 mm/s) shows a high spread and the roughness can be predicted even less accurately.

First of all, these results show that the different parameters can be recognized fairly well. Within the different parameter sets, however, the surface roughness R_z can be predicted with diverging accuracy when compared to the statistical spread in that set. The spread of the roughness in parameter set A is very small, making a more accurate prediction difficult. Parameter set C consists of a very low laser power, so that too little light was available for a more accurate prediction. Parameter set B, on the other hand, contains sufficient statistical spread as well as enough light so that the prediction is more accurate than the statistical spread. Besides, the largest amount of training data was available for this parameter set.