Quality prediction for milling processes: automated parametrization of an end-to-end machine learning pipeline

In signal processing, the techniques for extracting relevant features to perform process monitoring tasks can be divided into time domain, frequency domain, and time-frequency domain methods [1, 26, 27, 36]. Typically, it is necessary to manually identify and select the appropriate features with respect to the underlying process. To reduce the manual effort and maximize the automation of the feature extraction process, the Python library TSFEL [3] is used to compute the process describing features. The extracted features are defined using a features.json file, in which the available features are categorized by domain into statistical, temporal, and spectral features.

To additionally consider information from the time-frequency domain an extension of TSFEL by the domain temporal-spectral is implemented. The extension applies the discrete wavelet transform (DWT) and Hilbert-Huang transform (HHT) to the input time series data. The DWT decomposes the input signal into individual frequency bands by repeating high-pass and low-pass filtering. The decomposition level determines the number of transformation steps. In each decomposition level, the high-pass filtered signal components are coded as wavelet coefficients. The low-pass filtered signal components finally serve as a basis for the subsequent decomposition step. The calculation of the wavelet coefficients \(c \left( \tau , s \right)\) within the DWT is done for discrete values of the scaling parameter s and shifting parameter \(\tau\) according to Eq. (1)whereby \(x\left( t \right)\) corresponds to the time series under investigation and \(\psi\) represents the selected wavelet basis function [10, 23, 29]. For the determination of characteristic features, the calculation of the mean, root mean square, standard deviation, kurtosis, crest factor, and the peak to peak distance for the coefficient profiles of each decomposition level is performed according to [36]. Additionally, the residual was taken into account during feature extraction.

The empirical mode decomposition as the initial step of the HHT decomposes the signal into a finite number of intrinsic mode functions (IMF) based on the sifting algorithm. To identify an IMF, the local maxima are connected via a cubic spline to create the upper envelope. Similarly, the lower envelope is obtained by using the local minima. By averaging the two envelopes, the resulting time series \(m_{1}\) is obtained. The subtraction of \(m_{1}\) from the original signal \(x\left( t \right)\) results in the first IMF component \(h_{1}\), which serves as input signal for the following iteration. These steps are repeated until a previously defined stopping criterion is reached. The second step of the HHT, is the application of the Hilbert transform to the IMFs, which uses the resulting instantaneous frequencies and instantaneous amplitudes of the signal to form the energy-frequency-time spectrum. This is known as Hilbert-Huang Transform [15‐17, 33]. Due to the high output dimension of the HHT matrix, the characteristic features mean, root mean square, standard deviation, crest factor, peak to peak distance and absolute energy are calculated in the same manner as described above for the DWT [7, 31].

A majority of the features obtained from the automated feature extraction based on Sect. 3.1 typically yield non-relevant information regarding the prediction task or exhibit indifferent behavior under changing process conditions. Additionally, an enlarged number of input dimensions leads to an increased demand for training data and the risk of overfitting rises. Therefore, in order to achieve highest prediction performances with correspondingly high generalization capability of the predictive models, subsequent to feature extraction, the designed feature selection algorithm (illustrated in Fig. 1) is applied. It is based on the presented feature selection method from Fertig et al. [12], which has been extended within this work to ensure a more general application. Initially, all features exhibiting a variance of 0 across the analyzed dataset are removed by means of a variance threshold filter. The next step applies a set S of feature selection algorithms on the reduced feature set. S represents a subset of the implemented features selection algorithms (eg. S = \(\left[ \text {uniStat, LogisRe} \right]\), cf. Sect. 3.4). A final proprietary feature set (propFeatSet) is obtained for each geometric element (geomElem), located on a workpiece (cf. the workpiece shown in Fig. 3 consists of seven quality-relevant geomElem) by the set of features which are selected by the previously defined amount of \(SW_{\text {fs,prop}}\) feature selection algorithms. Thereby \(SW_{\text {fs,prop}}\) (eg. \(SW_{\text {fs,prop}} = 2\), cf. Sect. 3.4) can be varied from 1 to the quantity of implemented and thus available feature selection algorithms.

The implementation provides four selection algorithms, which can be combined arbitrarily. The first one consists of a univariate feature selection (uniStat) based on the determination of the mutual information between the feature vector and target variable. In addition, logistic regression (LogisRe) with elastic net regularization is utilized for feature selection. This regularization technique is particularly suited for a large number of features and a small number of training samples [39]. Lasso regression (Lasso), a shrinkade method, uses L1 and L2 regression penalty terms to shrink the coefficients of irrelevant features to 0. This model-based selection method allows a straightforward selection of the influential features by analyzing the model coefficients [13, 19]. For Lasso feature selection, the Least Angle Regression (LARS) algorithm developed by Efron et al. [9] is used to compute the coefficients, which calculates all Lasso estimates at high computational efficiency. In particular, LARS shows its advantages in high-dimensional datasets. Efron et al. [9] The fourth method for feature selection consists of the efficient wrapper approach boruta (Boruta). It aims to the identification of all relevant features for the prediction task. For this purpose, shadow features exhibiting random values are taken into account in addition to the real features. Finally, the feature selection is performed by comparing the feature importance, given by the used random forest, between the real and the shadow features. Kursa and Rudnicki [25]

Adapted from Fertig et al. [12] the 6 ML algorithms, Support Vector Machine (SVM), k-Nearest Neighbors (KNN), Ridge Regression classifier (RidgeRe), Gaussian Naive Bayes classifier (GNB) Decision Tree (DT), Multilayer Perceptron (MLP) and the 3 ensemble algorithms Random Forest (RF), Extra Trees classifier (XT), AdaBoost classifier were implemented within the MLPL. The optimization of the hyperparameters for each algorithm is done using a grid search combined with stratified 3-fold cross-validation using precision as scoring function. For implementing the algorithms as well as the feature selection the python library scikit-learn was used [30].

By using the underlying pre-processed internal data from the machine tool and quality assurance, the developed MLPL enables automated individualized modeling for each quality-relevant geomElem on a workpiece. The following parameterization of the MLPL was applied by Fertig et al. [12]:The promising prediction results show the potential of the obtained models for quality prediction. Nevertheless, the numerous parameterization options of the MLPL lead to the assumption that the performance of the models can be further increased by suitable parameter combinations. After each run, 9 models are available for each geomElem. Thus the appropriate model suitable for the quality prediction must be selected to be used in the application. The presented algorithm in the following sections provides the automated parameterization of the MLPL via the PipelineConfig.json to obtain and select the best suitable models for the quality prediction task.

The developed algorithm consists of multiple consecutive optimization steps (optSteps), which are executed sequentially to identify the optimal parameters gradually. The decision to perform a stepwise parameter optimization was motivated by the consideration that a full factorial implementation of the selected parameter combinations requires more than 600,000 MLPL runs, which leads to an unacceptable computational effort. After performing an optimization step, the corresponding identified parameters are set and the optimized configuration is applied to perform the next optimization step. Fig. 2 shows the operation procedure of the algorithm, which consists of running the MLPL, described in Sect. 3, with different configurations to identify the best-performing values for each parameter. The following six optSteps are considered within the algorithm:

These are defined in a configTable_optSteps. For each optStep a separate configTable_optStep_runs provides the specified parameter values run_param to be set for the MLPL runs. Accordingly, the MLPL is iterated within an optStep according to the number of parameter combinations of the configTable_optStep_runs with individually adjusted parameter values. The results for each run are stored as a metrics report, which contains the resulting prediction metrics. Additionally, the MLPL provides the trained models for the corresponding run. Basically, the main concept consists of a modular and extensible design, which allows the algorithm to run through additional optimization steps by extending the configTable_optSteps with the corresponding configTable_optStep_runs.

After performing an optimization iteration, the metricsReports are read for each run to rank and sort the results per geomElem following the developed scoring approach (cf. Sect. 4.2). These results are subsequently used to determine which parameter values lead to the highest performance metrics. Finally, the PipelineConfig gets reparameterized for the next optStep based on the identified parameters.

Both the DWT and the HHT exhibit hyperparameters, which need to be adapted to the characteristics of the available data for adequate feature extraction. For this purpose, the optStep\(_{\text {DWT}}\) thus envisages runs under altered basis-wavelets as well as decomposition levels. Included are the wavelet families daubechies, coiflet, symlet, biorthogonal with 9 shapes and the decomposition levels 3–7. Owing to the different processes resulting from the individual geometry of each geomElem, individual DWT hyperparameters are selected for each geomElem, which leads to the parameterization of the created individual features.json configuration files. The same applies to the HHT, in which the number of IMFs (in this case 1–7) is selected individually for each geomElem. Following the determination of the individual hyperparameters, a subsequent decision is required on whether features extracted using HHT should be considered in addition to DWT based features. Preliminary tests showed that the DWT is considerably more powerful compared to the HHT in terms of prediction performance. The combination of both time-frequency methods yielded no improvement in the results. Since it cannot be excluded that in particular cases the additional use of HHT features may achieve better performance, the described optStep 2 is included as well. The final step of optimizing the feature extraction is to select which domains should be included in the model building process. For the spectral domain, the Hanning, Hamming, and Blackman window functions are additionally examined to improve the quality of the spectral analysis [32]. The selection of domains to be considered is based on different subsets of the available domains. The subsets consist of each domain individually (4 subsets), the combinations of temporal-spectral with the other domains (3 subsets) as well as all domains together (1 subset).

The feature selection is optimized based on the feasible combinations of the four implemented algorithms. \(SW_{\text {fs,prop}}\) is adjusted according to the number of algorithms used per run. When using one feature selection method, a threshold \(SW_{\text {fs,prop}} = 1\) follows. Above two to four methods, \(SW_{\text {fs,prop}}\) iterates between 2 and 4 respectively. Each of the resulting 21 parameter combinations is executed once with the standardization and normalization of the scaling methods, requiring 42 runs for optimizing the feature selection. The final run finally performs the MLPL using the optimal configuration identified by the algorithm to build and select the final prediction models for each geomElem.

The modular architecture and the various configuration files as well as tables enable a flexible extension of the algorithm by simple adaptation. This ensures broad applicability by allowing additional values to be easily added to the verification procedure if desired.

At the completion of each optStep, the results need to be analyzed and the parameter values that lead to the best prediction performance need to be determined based on the metricsReports created for each geomElem, which contain the classification performance measures for accuracy (ACC), precision (PREC), recall (REC) and specificity (SPEC) [12]. Additionally, the ROC AUC and the number of false positive predictions (FP). The metrics within the metricsReports are determined on the validation set. To achieve the identification of the parameters, a rank is assigned to each model. The ranks are determined using the number of FP predictions, the values for specificity and accuracy, as well as ROC AUC. A lower value for the number of FP as well as higher values of the remaining metrics lead to a better rank and thus to a preferred selection as a final model of a geomElem. Due to the desired application in quality prediction, FP predictions are considered to be particularly critical in a production environment. These lead to further processing and assembly of a workpiece that has been manufactured in violation of its tolerances. As a result, it may not fulfill its function and may not be able to withstand the operating loads acting on it. For this reason, the ascending sorted ranks belonging to the number of FP forms the first basis to select the best performing models. If the number of FP is equal for several models, the subsequent sorting base considers the sum of ranks across all metrics. If this is still insufficient for unambiguous identification, the sorting procedure takes into account the ranks of ROC AUC, specificity, and accuracy for decision-making. This individual analysis for specific geomElem allows the hyperparameters of the time-frequency feature extraction methods to be determined and adjusted. The subsequent optSteps are evaluated globally across all geometric elements. Preliminary tests have shown that this individualized consideration yields significant improvements in model performance. However, no improvements and thus no advantages were obtained by individualized evaluation of the other optSteps. To reduce complexity and preserve comprehensibility, the subsequent optSteps are evaluated globally across all geometric elements. For each parameter combination, the resulting sum of FPs on the validation set predicted from the previously determined best-performing models per run is calculated across the entire workpiece thus all geometric elements. The parameter combination which leads to the lowest number of FPs is finally selected.

This domain-adapted scoring approach allows the identification of the best-performing models per geomElem within each optimization step. The underlying configuration parameters are finally set in the corresponding config files to be considered during the next optStep.

The datasets were generated in the TEC-Lab of PTW using the 3-axis DMC 850 V machining center from the manufacturer DMG MORI (DMG) and the 5-axis GROB G350 2. Gen (GROB) machining center, which both are equipped with a Sinumerik 840 D control system. For data acquisition per machine, an installed edge computing solution was used, that provides the internal drive signals from the controller at a sampling frequency of 500 Hz. The respectively recorded signals and the experimental design as well as the data acquisition and matching from quality measurement can be obtained from Fertig et al. [12]. In addition to the dataset \(\text {DS}_\text {DMG1}\) presented in Fertig et al. [12], two further datasets were generated using the given experimental design and the considered reference geometry. \(\text {DS}_\text {DMG2}\) was produced approximately 1.5 years apart from \(\text {DS}_\text {DMG1}\) on the same machine tool. \(\text {DS}_\text {GROB}\) accordingly using the GROB machining center. Table 1 summarizes additional information about the datasets.

The fourth dataset is applied for validation of the presented algorithm using a different workpiece geometry. For this purpose the pocket geometry made from the material 42CrMo4V, which is shown in Fig. 3 is considered. It consists of 7 quality-relevant geometric elements and was manufactured using the DMG machine tool. A total of 392 pockets were produced based on the previously mentioned experimental setup. For finishing the pocket, a carbide end milling tool (article number: 203089 10) made by Hoffmann Group was used. The tool features N\(_{\text {z}}\) = 5 teeth at a nominal diameter of D\(_{\text {tool}}\) = 10 mm. The technology parameters were summarized in Fig. 3. The pockets were arranged on 8 cuboidal plates made of 42CrMo4V with the dimensions l × w × h = 305 × 305 × 30 mm\(^{3}\) in a 7 × 7 grid for optimized experimental procedure. This results in 392 pockets for the subsequent analysis.

Figure 4 summarizes the evaluated metrics on the test dataset for each trained model obtained from the final run for each dataset. Each dot represents the corresponding score of one trained model. The black bars represent the average value across all models. The gray bar represents the results of the MLPL in default configuration (cf. Sect. 3.4). The improvements in specificity and ROC AUC are clearly visible for each dataset. The average values of the other metrics also exhibit minor improvements. To analyze the improvements in more detail, Table 2 summarizes the percentage improvement of the average values relative to the default configuration. Except for the recall at \(\text {DS}_\text {DMG1}\) and \(\text {DS}_\text {DMG2}\), the metrics in the total view across all models and geomElems show a partially significant improvement by the optimization algorithm. The improvements can be particularly observed when considering the scores of specificity and ROC AUC. This reflects the optimization target of minimizing the FP predictions since the number of FP, in this case, impacts the specificity values most. After optimizing the parameters, it is shown that for each of the 4 datasets for feature extraction, the temporal-spectral domain without HHT is considered solely for the prediction model development. The base-wavelets and decomposition levels selected individually for each geomElem thus provide the highest information density for quality prediction regarding the analyzed datasets. These best-performing hyperparameter values used for DWT furthermore differ among the geomElems and datasets. This supports the design of the algorithm toward individual parameter identification. In addition, the optimization algorithm selects different model algorithms for each geometric element, which leads to the conclusion that multiple model algorithms are required for the most accurate prediction.

Nevertheless, these results only provide an overall impression of the algorithm’s performance. When considering Fig. 4 several cases can be identified where models show poor scores, especially for the specificity. For this reason, it is necessary to select the best performing model for each geomElem based on the presented scoring approach (cf. Sect. 4.2). Table 3 summarizes the results of the final model selection. Shown are the mean values of the achieved metrics on the test dataset across all geomElems. Additionally, the achieved metrics of the non-optimized pipeline are displayed. This clearly shows that for each data set the optimization according to the presented algorithm leads to significant improvements regarding the final selected models. Overall, all average scores achieved high values above 90\,%. The average number of FP per geomElem is reduced to less than 0.5 for each dataset. It is worth mentioning that the FP predictions are not equally distributed among the geomElems. The total of 7 FP on the test dataset \(\text {DS}_\text {DMG1}\) are allocated to 5 geomElem. On \(\text {DS}_\text {DMG2}\), 2 FP occurred in two geomElem and on \(\text {DS}_\text {GROB}\) 6 FP predictions were made on the test dataset which consists of 50 Workpieces thus 750 geomElems. The size of the test dataset was kept the same for each of these datasets. The final selected models perform on the test dataset, consisting of 98 pockets and accordingly 686 geomElems, of \(\text {DS}_\text {DMG3}\) only 3 FP predictions which are associated to one element.