A new approach for automated measuring of the melt pool geometry in laser-powder bed fusion
verfasst von:
Simon Schmid, Johannes Krabusch, Thomas Schromm, Shi Jieqing, Stefan Ziegelmeier, Christian Ulrich Grosse, Johannes Henrich Schleifenbaum
Additive manufacturing (AM) offers unique possibilities in comparison to conventional manufacturing processes. For example, complex parts can be manufactured without tools. For metals, the most commonly used AM process is laser-powder bed fusion (L-PBF). The L-PBF process is prone to process disturbances, hence maintaining a consistent part quality remains an important subject within current research. An established indicator for quantifying process changes is the dimension of melt pools, which depends on the energy input and the cooling conditions. The melt pool geometry is normally measured manually in cross sections of solidified welding seams. This paper introduces a new approach for the automated visual measuring of melt pools in cross-sections of parts manufactured by L-PBF. The melt pools are first segmented in the images and are then measured. Since the melt pools have a heterogeneous appearance, segmentation with common digital image processing is difficult, deep learning was applied in this project. With the presented approach, the melt pools can be measured over the whole cross section of the specimen. Furthermore, remelted melt pools, which are only partly visible, are evaluated. With this automated approach, a high number of melt pools in each cross-section can be measured, which allows the examination of trends over the build direction in a specimen and results in better statistics. Furthermore, deviations in the energy input can be estimated via the measured melt pool dimensions.
Hinweise
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1 Introduction
Laser-powder bed fusion (L-PBF) is an AM process, which can create three-dimensional geometries out of metal especially for individualized products in small series [1]. The process can be approximated by stacking thousands of welds together in a predefined order in a layer-wise manner [2]. The parts are manufactured in a powder bed, which is melted selectively at the locations where parts are built. Due to the wide variety of influencing factors and environmental conditions, maintaining a consistent part quality within L-PBF is still a challenge even after two decades of research. The same machine with the same powder and identical parameters, might produce parts of varying quality. In addition, the part geometry and local thermal history affect, among other things, the properties of the printed part.
A schematic illustration of a part manufactured with L-PBF and the exposure strategy used in this work is given in Fig. 1a. The scan direction is varied by a rotation angle of 90° from each layer to the next. To reach a more isotropic material behavior a rotation angle with a prime number, e.g. 67°, is usually applied. In this case, the cutting plane of the micro-sections is inclined and the melt pools appear distorted, resulting in patterns which are more difficult to analyze. For this reason, a rotation angle of 90° is used in this work for measuring the melt pools. Furthermore, the tensile strengths of specimens manufactured with a rotation angle of 67° and 90° are comparable according to [3]. From this can be concluded that the process for these rotation angles is similar. In Fig. 1b an excerpt of a typical micro-section is shown, which corresponds to the marked region in a.
Fig. 1
Schematic illustration of a part manufactured with L-PBF in a. Excerpt of a micro-section in b corresponding to the illustration in a
×
Anzeige
The borders of the melt pools are visible in the micro-sections because their microstructure is coarser compared to the inner microstructure of the melt pool. This microstructure is a result of different solidification conditions, in particular a higher temperature gradient, a higher cooling rate and a lower solidification growth velocity [4].
In literature, melt pool dimensions are a commonly used criterion for quantifying process deviations. Lee et al. [5] presented a data analytics approach, which correlates influence factors such as process or material parameters to the melt pool dimensions. A machine learning model was trained on the data. The melt pool dimensions could accurately be predicted for melt pools measured on the substrate but showed low accuracy for melt pools, which were melted with powder [5]. The melt pool dimensions not only depend on the process condition, but also on the material used. For this reason, Johnson et al. [6] introduced printability maps for different metal alloys based on the melt pool dimensions. Hereby, a finite element model was used to distinguish regions in the parameter space where keyhole formation, balling or lack of fusion would occur. The distinction of these process regimes was made based on the melt pool geometry [6].
To the best of the authors’ knowledge, in literature melt, pools are measured manually in cross sections using software such as ImageJ. Furthermore, the melt pools are typically measured in single tracks, because then remelting by neighboring welding tracks does not have to be taken into account. The melt pools are not only remelted by the neighboring welding tracks, but also by the layer above the respective layer. Therefore, only the melt pools in the top layer are measured in the literature. In our study, an algorithm is developed for measuring remelted melt pools as well. The algorithm was developed with deep learning and digital image processing.
For computer vision tasks, deep learning algorithms like convolution neural networks (CNNs) have shown strong performance by learning features in an automated way. In this work, we use the U-net model, proposed by Ronneberger et al. [7], for semantic segmentation of the micro-sections. Based on the segmented borders of the melt pool, the watershed algorithm is applied for instance segmentation of the melt pools. With the segmented melt pool, the welding depth \({d}_{\mathrm{mp}}\) and melt pool width \({w}_{\mathrm{mp}}\) are measured.
Anzeige
2 Materials and methods
In this chapter, the methodology for measuring the melt pools is described. The implemented algorithms consist of deep learning and digital image processing. In the segmented image, the layers in which the melt pools were manufactured must be identified to subsequently measure the melt pool dimensions towards the respective layers.
2.1 Procedure for measuring the melt pools
First, the melt pools in the micro-section are segmented using the U-net model. Based on the semantic segmented image, instance segmentation of the melt pool is performed with the watershed algorithm. In the next step, the segmented melt pools are assigned to the layer in which they were manufactured. Then the welding depth \({d}_{\mathrm{mp}}\) and the melt pool width \({w}_{\mathrm{mp}}\) of the melt pools are measured with respect to the layer in which the melt pools were generated. The welding depth \({d}_{\mathrm{mp}}\) is estimated as the distance between the deepest point of a melt pool, which is the location of the pixel with the highest y value in the image, to the assigned layer. The procedure for measuring the melt pools is schematically displayed in Fig. 2. It shall be noted that the y-direction is the opposite direction of the build direction.
Fig. 2
Schematic illustration of the procedure for measuring the melt pools
×
Hereby, the position of the layers is determined by the layer thickness \({d}_{\mathrm{L}}\) and the position of the top layer. To measure the melt pool width, an ellipse equation is fit through the downward facing boundary of the melt pool. This is done because the melt pools below the top layer are only partly visible in the micro-sections, due to remelting. An ellipse equation is also used in Mitchell et al. [8] for approximating the melt pool shape in a thermography camera image. The ellipse equation is fit as a constrained non-linear least square optimization problem. Constraints for the ellipse middle points (\({x}_{0}\),\({y}_{0}\)) are needed because otherwise the data points of the boundary would not be enough to fit an ellipse equation in the right location. The optimization problem was solved with the Levenberg–Marquardt algorithm [9]. The objective function for fitting an ellipse equation, which was minimized, is the following:
The parameters \({x}_{0}\),\({y}_{0}\), \({a}_{\mathrm{e}}\) and \({b}_{\mathrm{e}}\) in mm have to be adapted. \({x}_{0}\) was constrained by \(\pm 20\) pixel around the x value of the deepest point in the melt pool. \({y}_{0}\) was constrained by the upper boundary being the location of the assigned layer and the lower boundary was set to that location − 20. To measure the melt pool width for specimens measured with a rotation other than 90°, which is not done in this article, the distortion of the melt pools resulting from the inclined cutting plane has to be taken into account. This can be done by dividing the measured width by the cosines of the angle between the scanning direction and the cutting plane.
2.2 Generation of the training data
The algorithm was trained using specimens, which were manufactured under different process conditions. This was done in order to capture different melt pool shapes. The process parameters laser power, scan velocity and hatch distance were varied. The layer thickness was kept constant at 50 µm. Furthermore, rotation angles of 67° and 90° were used. The rotation angle of 67° was only used for the generation of training data and not for measuring the melt pools. In total, 12 cubic specimens of different size and geometry out of AlSi10Mg were manufactured. Nine of them were produced on a L-PBF machine AconityONE and three of them on a SLM \({500}^{\mathrm{HL}}\). The specimens were mounted in a mould via hot-mounting (see Fig. 3b).
Fig. 3
Location of the cross section in a specimen in a. Four specimens in a mould in b
×
Subsequently, the mounted specimens were grinded and polished. Normally the specimens are cut before mounting them. However, this can introduce an inclination of the cutting plane, which can lead to distortion of the melt pool dimensions. For this reason, the specimens were not cut and the observed level cross section was reached by grinding. Grinding was carried out with great care in order to ensure that the preparation was as good as possible without scratches.
To make the melt pool borders more visible etching was applied. Therefore, the specimens were immersed into a NaOH base (2 \(\frac{{{\text{mol}}}}{{\text{l}}}\)) for a prescribed period of time, which was varied between 30 and 45 s. This was done in order to train the algorithms with different etching conditions, thereby augmenting the dataset through etching.
The prepared specimens were examined with the digital microscope Keyence VHX-6000. A magnification of 100 was used resulting in a pixel size of 2 μm with the given microscope. The microscope records images of smaller sizes and stitches them together to the final illustration. The resulting images had different sizes from about 3000 to 8000 pixels for each dimension.
To train the U-net algorithm, the images were first annotated. Four different classes were distinguished: defect, melt pool border, material and mounting mass. The micro-section of a specimen and its annotations are shown in Fig. 4.
Fig. 4
On the left micro-section of specimen and on the right annotated image
×
At first, the image of the specimen was rotated based on a principal component analysis (PCA). With the rotated image, the annotations for the classes defect, material and mounting material were extracted using a combination of thresholding via Otsu’s method [10], morphological and logical operations. The extracted annotations were then improved manually. The class melt pool border was annotated manually with the MATLAB Image Labeler and a brush with a size of 8 × 8 pixels. Five different people performed the annotation in order to reduce bias. During the manual labeling, the melt pool borders were painted over the classes defect and mounting material. Afterwards, the labels were combined in such a way that pixels were only annotated with the class melt pool border if the class material was present in the image yielded by thresholding and morphological and logical operations.
Because of the large image size, it could occur that the data do not fit into the GPU memory when using the whole images for training the algorithm. For this reason, random cropping with a patch size of 256 × 256 was applied. Out of each image, \(\frac{2 \cdot n\cdot m}{256\cdot 256}\) patches were cropped, where n and m are the dimensions of the image in pixel. Random cropping was used for the training and validation dataset with 11 of the 12 micro-sections. One micro-section was reserved for testing. The testing was performed on the whole uncropped micro-section 70% of the data were used for training and 30% for validation.
2.3 Semantic segmentation of the melt pools with deep learning
The semantic segmentation was performed applying an U-net. The implementation of Yakubovskiy [11] in Keras Tensorflow was used. Hereby, the downsampling path was replaced by VGG-19, which was pretrained on ImageNet. Only the upsampling path of U-net was trained with the new data.
The training was performed with AWS on the GPU instance p3.2xlarge. To increase the dataset and improve the generalization of the algorithm, augmentation was applied in each training epoch. For comparison, the model was also trained without augmentation. The following methods were used: Gaussian noise, CLAHE, histogram equalization, random contrast, and a sharpening filter. Each method was applied with a probability between 0.2 and 0.9 in each epoch.
The models were trained with stochastic gradient descent and the adaptive moment estimation (Adam) optimizer. For each model, hyperparameter tuning was performed for the hyperparameters learning rate \({l}_{r}\) ϵ [0.0001, 0.001] and batch size \({b}_{s}\) ϵ [32, 64] using a grid search. The model with the tuned hyperparameters was trained until a minimal validation loss was reached with the early stopping criterion. As a loss function, the sum of the dice loss and the focal loss was used. The combination of dice loss and focal loss was chosen to combine the advantages of the dice loss, which is better boundary segmentation and focal loss considering class imbalance. The loss is calculated for each batch in the epoch and is averaged over all batches.
The performance of the model was evaluated on the test dataset using the following metrics. Hereby, \({k}_{ij}\) is the number of pixels in class i predicted as class j, \({m}_{i}\) is the total number of pixels in class i and K is the number of different classes [12]. The class, which should be the most accurate for evaluating the melt pools, is the melt pools border. Therefore, this class is evaluated with a separate metric: the intersection over union for the class melt pool border:
2.4 Instance segmentation of the melt pools with the watershed algorithm
The image resulting from the semantic segmentation is not perfectly segmented (see Fig. 9), which results in holes in the predicted melt pool border. For this reason, the melt pools are not always separated from each other by simple methods like region growing. Therefore, the watershed algorithm was applied additionally. Hereby, an image is interpreted as a topographic surface. The topographic surface was determined based on the segmented image and the Euclidean distance transform. The Euclidean distance transform calculates the distance from each pixel to pixel with the class melt pool border. Hereby, pixels with the class melt pool border have the value 0 and all other pixel values are increasingly negative with a higher distance from the border. Due to this, "valleys" emerge. From the bottom of the valley the watershed algorithm starts "flooding" the regions until the flooded areas merge.
Between the different flooded areas, a so-called dam or watershed ridge line is constructed, which has a width of two pixels. In order to prevent over segmentation marker-controlled watershed was used. The markers determine the seed points from which the flooding starts, and therefore gives the number of segmented objects. To determine the seed points the h-minima transform was used. The h-minima transform suppresses all minima for which the distance value is over a certain threshold.
In Fig. 5, the procedure for applying the watershed algorithms on the semantic segmented image is displayed. The watershed algorithm applied in this way does introduce artefacts. Long melt pools, which have been cut in the welding direction, can be over segmented because the distance map often includes more than one minimum.
Fig. 5
Procedure for applying the watershed algorithm
×
2.5 Identifying the layers
The layers in which the melt pools were manufactured were determined by manually annotating them in the image, which were segmented by the watershed algorithm. The melt pools, which were cut perpendicularly to the welding directions, were marked with a brush. Hereby, incorrectly segmented melt pools were excluded. Three different classes were annotated, which were labeled in a certain sequence (class 1, class 2, class3, class1…) over the layers starting from the top layer. Afterwards, the melt pools intersected by each class were extracted. Based on the centroids and the class number, the extracted melt pools were ordered according to the order of the layer in which they were manufactured. In Fig. 6, the assigned layers to the melt pools are displayed. The layers were randomly color coded.
Fig. 6
Layers assigned to the melt pools. The layers are color coded randomly. The white lines indicate the positions of the layers
×
The y-position of the layers was determined with the y-position of the melt pools in each layer. Therefore, the position of the layers in the micro-section has to be parallel to the x-axis in the image. To achieve that, the upper boundary of the top layer was extracted. Through this boundary, a line equation was fit and the image was rotated in a way that the slope of the line equation became zero. The layer thickness \({d}_{\mathrm{L}}\) was determined with the distances of the melt pool between the determined layers. The upper plot in Fig. 7 shows the y-position of the identified layers from the top to the bottom of the specimen. The y-position of the layer was determined by averaging the y-value of the lowest point for all melt pools in the respective layer. The bottom plot in Fig. 7 shows the distances between one layer and the subsequent layer.
Fig. 7
Top: the y value of the deepest points in the melt pools in each layer is plotted against the identified layers. Bottom: the distances between the layers are displayed. The red line is the average of the distances
×
It should be noted that only every second layer cut perpendicular to the welding direction was taken into account. To measure the distance between each layer to the subsequent layer, the determined distances were divided by two. It can be seen that the distances between the layers vary between 45 and 57 µm. The plant layer thickness dL was 50 µm. The varying distances can be a result of varying welding depths and measurement inaccuracies. To estimate the layer thickness of the specimen and considering inclination of the cutting plane the layer thickness was determined by averaging all the calculated distances between the layers. For the given specimen this results in a layer thickness of 51.6 µm. It should be noted that the mean and not the median was taken, because the mean also averages over process changes (outliers), which lead to a change in the welding depth. The white lines on the left side of Fig. 6 indicate the determined y positions of the layers.
To prevent the time-consuming manual labeling of the layers an automated approach was developed. For this, the segmented melt pools were filtered based on their shape. Elongated melt pools, which have been cut perpendicular to the welding direction, were discarded. This can be seen in Fig. 8 on the left, where the yellow melt pools have been filtered. Afterwards, the y position of each melt pool was determined. The k-means clustering algorithm was applied on these values and the number of clusters was found with the elbow criterion. The resulting assignment of the melt pools is displayed in Fig. 8 on the right. This procedure is not robust enough, because in some cases layers are missed which leads to a wrong estimation of the welding depth. For that reason, the algorithm needs to be improved further. Therefore, the manual assignment of the layers was chosen for further investigations.
Fig. 8
Procedure for automated identifying the layers of the melt pools
×
To identify the layers automatically for specimens manufactured with a rotation angle other than 90°, the distortion of the melt pools has to be taken into account. On the one hand, this makes it easier to assign the right layers to the melt pools, because their size differs from layer to layer. On the other hand, is it not possible to measure the width of melt pool, which have been cut in scan direction. Because of this not all layers could be measured.
3 Results and discussion
In this chapter, the results of the procedure described are shown and discussed. At first, the performance of the semantic segmentation algorithm is investigated. Afterwards, the whole measuring procedure is conducted on one specimen and the results are discussed.
3.1 Evaluation of the semantic segmentation performance
In Table 1, the evaluated metrics of the test dataset of the trained model with and without augmentation are summarized. The values are rounded to two decimal places. As mentioned above, the models were tested on one micro-section, which was not used for training. All metrics show higher values for the model trained with augmentation. The model trained with augmentation was chosen for the further procedure because of its better performance. The model trained with augmentation reaches on the training dataset a pixel accuracy of 95.14% in comparison to the pixel accuracy of 91.11% on the test dataset.
Table 1
Performance of the models trained with and without augmentation on the test dataset
Pixel accuracy (%)
Mean accuracy (%)
Mean IoU (%)
\({\mathrm{IoU}}_{\mathrm{MP}} \left(\%\right)\)
Without augmentation
91.10
74.54
62.90
38.83
With augmentation
91.11
75.81
63.58
39.75
In Table 2, a confusion matrix of the model trained with augmentation on the test dataset is shown. In the confusion matrix, the number of pixels predicted in each class together with a ground truth label is displayed. The number of pixels with the class defect is much smaller in comparison to the other classes. This is also true for the training dataset. The most important class for measuring the melt pools is the melt pool border. Hereby, 53.30% of the pixels have been predicted in the same class as the ground truth melt pool border. This smaller accuracy in comparison to the classes mounting material and material can be explained by annotation errors. For example, the size of the brush was always kept the same in the manual annotation procedure of the melt pool borders (see Fig. 9 in the middle). In comparison to the classes material and mounting material, the small accuracy of the class defect can result from an imbalance in the dataset. In Fig. 9, a section of the micro-section used for testing is displayed on the left and the corresponding ground truth image in the middle. The prediction is shown on the right. It can be seen that the prediction matches the ground truth accurately, despite the small accuracy of the predicted melt pool borders of 53.30%. However, some gaps in the melt pool borders are present. The predicted melt pool borders vary in their thickness and are smoother than the manually annotated ones.
Table 2
Confusion matrix of the model trained with augmentation
Ground truth class labels
Mounting material
Material
Defect
Melt pool border
Predicted class labels
Mounting material
9898641
99.12%
49965
0.36%
1695
1.45%
275
0.01%
Material
80787
0.81%
13101670
93.24%
60056
51.25%
1292703
47.61%
Defect
708
0.01%
46914
0.33%
52261
44.60%
2067
0.08%
Melt pool border
536
0.01%
853694
6.08%
3174
2.71%
1420124
53.30%
The matrix shows the number of pixels for each class. The percentage numbers give the the fraction of pixels from all the pixels in the respective ground truth class
Fig. 9
On the left: section of the test image. In the middle: corresponding ground truth image. On the right: prediction of the image
×
3.2 Measurement of one specimen
In Fig. 10, the welding depth \({d}_{\mathrm{mp}}\) is displayed for all measured melt pools. Hereby, the determined layer thickness (see Sect. 2.5) of 51.6 µm was used to calculate the positions of the layers. It can be seen that towards the left of the specimen and the bottom of the specimen the welding depths increases. On the top right welding depths of approx. 200 µm and the bottom right of approx. 300 µm are measured. This can be a result of the exposure direction, which is from right to left in the micro-section. Through the exposure in each layer, the material is heated and, therefore the welding depth increases with exposure time. The larger welding depth at the bottom of the specimen can be explained by the preheating of the building platform, which has a stronger influence at the bottom of the specimen.
Fig. 10
Welding depth \({\mathrm{d}}_{\mathrm{mp}}\) plotted on top of the measured welding lenses
×
Additionally, the melt pool width \({w}_{\mathrm{mp}}\) was measured using an ellipse fit as described in Sect. 2.1 and is plotted against the specimen in Fig. 11. A similar trend as for the welding depth can be seen to the left and bottom part of the specimen. However, some melt pools show high values for the melt pool widths. Measured melt pool width with unrealistically high values of over 900 µm, which were approximately 5% of the measured melt pools, were removed from the data. These high values can be a result of errors in the segmentation and therefore wrong melt pool shapes.
Fig. 11
Melt pool width \({w}_{\mathrm{mp}}\) plotted on top of the measured melt pools
×
4 Conclusions and outlook
With the proposed approach, melt pools could be measured automatically in cross sections of parts manufactured with L-PBF. The approach has to be validated further to use it in production. For this, the influence of geometry changes on the melt pool geometry should be investigated for example. Comparisons to in situ measurements of the melt pool dimensions can be drawn. For in situ melt pool measurements thermography cameras are commonly used [13]. With cameras however only the top of the melt pool can be investigated and the welding depths cannot be measured. An alternate method for investigating also the welding depth in an in-situ manner is using X-ray imaging [14].
Furthermore, improvements have to be made to use it. First, the size of the dataset should be increased by building more specimens with different parameter combinations and annotating them. In addition, different materials can be taken into account. Furthermore, the result of the instance segmentation by the watershed algorithm should be improved. An alternate approach to the h-minima transform is given by Wang et al. [15]. Hereby, an object detection algorithm is used to determine the location of the objects, and therefore the locations of the seed points for the watershed algorithm. Thus, the over segmentation resulting from the h-minima transform would not occur. Instead of using the watershed algorithm on the semantic segmented image, the image could also be directly segmented with an instance segmentation algorithm. For this, Mask R-CNN, introduced by He et al. [16], could be applied. Furthermore, the automated approach for identifying the layers of the melt pools should be improved.
Another measurement value, which can be extracted from the semantic segmented micro-sections, is the thickness of the melt pool borders. The melt pool border thickness is dependent on the cooling conditions. Therefore, the cooling condition can be quantified with the welding line thickness. The melt pool borders differ in their thickness in comparison to the ground truth image (see Fig. 5). This is the case, although the welding lines in the ground truth images used for training always had the same thickness.
Using the measured melt pool dimensions, conclusions to the process state and changes in the process parameter can be drawn. On the one hand this can be done using surrogate models as proposed by Tapia et al. [17]. On the other hand, a thermal FE simulation can be applied as an inverse problem to describe the influence of the process parameters on the melt pool dimensions. Belitzki et al. [18] adapt the input parameters for FE simulations of melt pools using an optimization algorithm such that the shape of the simulated melt pool matches the measured one as closely as possible. This was done for laser beam welding.
Compliance with ethical standards
Conflict of interest
The authors have no conflicts of interest.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Die im Laufe eines Jahres in der „adhäsion“ veröffentlichten Marktübersichten helfen Anwendern verschiedenster Branchen, sich einen gezielten Überblick über Lieferantenangebote zu verschaffen.
