Multi-model anomaly detection for industrial inspection with dynamic loss weighting and soft-hard features loss

Deep learning-based methods [4, 16, 17] address these issues by capturing complex relationships across multiple layers of abstraction and automatically learning the feature representations. Based on label availability during training, anomaly detection methods can be further categorized into supervised [18, 19], semi-supervised [16, 20], and unsupervised [4, 17] methods. In the context of anomaly detection, supervised and semi-supervised methods require labeled instances for both normal and abnormal cases during training, while unsupervised methods solely rely on normal data. Due to the challenges in collecting abnormal data in manufacturing, unsupervised learning has been extensively utilized across various benchmark datasets [2, 21] in the manufacturing domain. In this paper, we specifically focus on improving unsupervised deep learning methods for anomaly detection.

In unsupervised learning, deep learning-based methods for anomaly detection can be broadly categorized into reconstruction-based and pre-trained-based algorithms. VAE-based [22, 23] and GAN-based [17, 24] algorithms aim to reconstruct images and detect anomalies from scratch by analyzing differences between the reconstructed and input images in pixel spaces. This approach may not be optimal due to straightforward per-pixel comparisons or suboptimal reconstructions [4]. VAEs often produce overly smoothed outputs, making it difficult to preserve fine-grained details necessary for detecting subtle anomalies. GANs are prone to mode collapse, leading to unreliable reconstructions and difficulty in capturing diverse anomaly patterns. On the other hand, pre-trained-based methods [4‐6, 25, 26], extract features from models trained from large datasets like ImageNet and perform anomaly detection by analyzing the differences in extracted features between normal and abnormal instances. Utilizing features from pre-trained networks tends to yield better results than methods relying on autoencoders or GANs built from scratch [27]. However, pre-trained models are typically trained on large-scale natural image datasets like ImageNet, which may not fully align with the characteristics of industrial anomaly detection datasets. Recently, AAND [28] proposed a two-stage framework using knowledge distillation methods for industrial anomaly detection that enhances feature discrepancy through Anomaly Amplification and Normality Distillation (AADD). The Residual Anomaly Amplification (RAA) module increases sensitivity to anomalies while preserving the integrity of the pre-trained model. Additionally, the hard feature distillation loss facilitates the reconstruction of subtle or rare patterns in normal samples.

In this work, we adopt a recently popular student–teacher-based method [4‐6] which falls under the broader umbrella of pre-trained-based methods, where the teacher model can be seen as a pre-trained model that guides the learning of the student model for detecting anomalies. Specifically, our approach utilizes a student–teacher-based model in which multiple models, i.e., one student and two teacher models, are incorporated to detect the anomalies.

Loss weighting techniques have been widely utilized in deep learning-based approaches that incorporate multiple loss functions. These techniques aim to balance the contributions of different loss functions, each corresponding to distinct objectives or models, to optimize overall performance. In semi-supervised learning methods [29, 30], where supervised and unsupervised learning occur concurrently, adjusting the weight between the supervised and unsupervised losses plays a critical role in influencing overall performance. Basically, total loss, \(L_\text{total}\) is formulated as a weighted sum of the supervised loss \(L_\text{supervised}\) and unsupervised loss, \(L_\text{unsupervised}\), where \(\mathrm {\alpha }\) represents the weighting coefficient that controls the contribution of the unsupervised loss term. This can be expressed as follows:The weighting factor \(\mathrm {\alpha }\) plays a crucial role in balancing the impact of supervised and unsupervised learning components, ensuring that the model effectively leverages both labeled and unlabeled data during training.

Recent semi-supervised learning approaches have employed a single fixed hyperparameter to regulate the influence of the unsupervised loss on the total loss function [29, 30]. A similar strategy has been adopted in anomaly detection, where a single fixed hyperparameter is used to adjust the importance of the distillation loss in a multitask learning framework [31]. Likewise, a fixed hyperparameter has been utilized to control the contribution of a cosine similarity-based loss within an overall loss function that also incorporates a Euclidean distance-based loss [32]. A fixed weighting strategy may not be optimal across different datasets or anomaly types, as some tasks require stronger global feature representations, while others necessitate higher sensitivity to localized anomalies.

While manually adjusting the hyperparameters through extensive experiments is possible, finding optimal values requires testing and comparing a comprehensive range of values. In many anomaly detection methods where multiple models or objectives are employed [4‐6, 33], a balancing mechanism is often absent, resulting in equal weighting for all losses and neglecting the importance of balancing between different models or objectives. In contrast, compared to these previous methods, we dynamically adjust the importance between multiple losses, corresponding to different teacher models. Instead of fixing the importance of the two teacher networks with different functionalities as constants, dynamic weighting is applied to enable efficient fusion, effectively transferring specialized knowledge from each teacher to the student network. This dynamically controlling the weights of the losses during the training process helps our framework to achieve a better overall performance without requiring extensive tuning of hyperparameters.

Recently, using a student–teacher framework [4, 5, 32, 34] for spotting anomalies in computer vision has emerged as a promising approach. The student–teacher framework initially emerged from knowledge distillation [35], a model compression technique in which a complex teacher model transfers its learned representations to a more compact and computationally efficient student model while preserving performance. While traditionally designed to enhance model efficiency through size reduction, recent applications of this framework in anomaly detection have been adapted to address the challenges of unsupervised anomaly detection. In unsupervised anomaly detection, the student network is not necessarily smaller than the teacher network. Instead, it is trained using only normal images to mimic the output of the pre-trained teacher network. When presented with data not part of the training set, including anomalous images, the output of the student and the teacher could diverge. A higher discrepancy, especially on anomalous images, arises because the student model is only trained on anomaly-free data.

While some student–teacher-based methods focus on minimizing discrepancies across multiple intermediate layers between the student and teacher networks, many approaches [4‐6, 34] primarily minimize the discrepancy between the output features of the student and teacher networks. In the EfficientAD [6] method, however, to avoid outliers on normal images, only a small percentage of the largest pixel-level errors, i.e., the discrepancy between the output features of the student and the teacher network, are considered. However, given that a significant amount of information in the images is disregarded, there is a chance that the student network may fail to capture essential details. In contrast, instead of entirely neglecting the majority of relatively lower pixel-level errors, our approach reduces these error values while retaining the largest ones. This approach is not only able to avoid the outliers on normal images, but also enables the capture of more relevant details, resulting in improved pixel-level detection performance.

Given the dataset \(\mathcal {D}_\text{ad}\), a student network \(\mathcal {S}\) learns from the static teacher and the trainable teacher on normal images by mimicking the output of \(\mathcal {T}_\text{static}\) and \(\mathcal {T}_\text{trainable}\), respectively. The architecture of \(\mathcal {S}\) has a similar architecture as \(\mathcal {T}_\text{static}\) to have a low inference time. This is because the PDN model [6] which is used for the \(\mathcal {T}_\text{static}\), has a low computational complexity consisting of only six convolutional layers. Note that the output layer of \(\mathcal {S}\) has twice the number of channel dimensions compared to \(\mathcal {T}_\text{static}\) and \(\mathcal {T}_\text{trainable}\), i.e., \(\mathcal {S}(.) \in \mathbb {R}^{ 2C \times W \times H}\). This expansion allows \(\mathcal {S}\) to simultaneously learn from both \(\mathcal {T}_\text{static}\) and \(\mathcal {T}_\text{trainable}\). Specifically, half of the total channels correspond to the static teacher, while the remaining half corresponds to the trainable teacher. The illustration of the student architecture, which is similar to the static teacher architecture, is shown in Fig. 4.

The hard feature loss, a traditionally used approach in anomaly detection, focuses on the highest values during backpropagation, thereby prioritizing the most critical regions for anomaly detection. While effective in cases where the difference between normal and abnormal features is significant, this approach becomes less reliable when anomalies exhibit subtle deviations, as it may fail to capture minor but important variations. In contrast, soft feature loss, which normalizes and propagates all feature values without filtering, may retain excessive information, potentially leading to an increased false-positive rate by misclassifying normal samples as anomalies. To overcome this issue as illustrated in Fig. 5, we introduce a soft-hard feature loss to avoid completely ignoring most of the data points while maintaining the importance of the most relevant parts of the image. Soft-hard feature loss is a hybrid loss function designed to balance the trade-off between hard feature loss and soft feature loss in anomaly detection tasks. This loss function aims to enhance feature-level anomaly representation by selectively focusing on high-error regions while still incorporating global feature information.

Our soft-hard feature loss is defined as follows:where the loss \(L_i^\mathrm {\Omega \_{\tiny {softhard}}}\) consists of hard features loss and soft features loss with \(\alpha \) parameter controlling the contribution of the two losses. The soft features loss \(L_i^\mathrm {\Omega \_{\tiny {soft}}}\) is the loss that is weighted by the min–max normalized errors. Intuitively, this gives the higher weights to the higher error values and gives the lower weights to the lower error values. By extending hard features loss techniques to the soft-hard features loss, we ensure that a broader range of loss values contributes to the optimization process, while concurrently mitigating the influence of less relevant parts or outliers. This approach allows the model to focus on minimizing errors where they are most significant while still considering the entirety of the loss distribution, thereby enhancing the model’s pixel-level performance.

The soft-hard feature loss technique is expected to be particularly effective in scenarios where there are many areas in the images considered hard or difficult to train, such as background clutter, object variability, and other complexities inherent in real-world image data. Only considering the top 0.1% error values as in the hard features loss may not adequately address the complexities present in the whole area. We provide an example highlighting the influence of the soft-hard feature loss in Fig. 6.

In addition to \(L_i^\mathrm {\Omega \_{\tiny {softhard}}}\) loss, we also incorporate penalty loss \(L_i^\mathrm {\Omega \_{\tiny {penalty}}}\) [6] which is defined as follows:where \(\hat{I}_i\) is a randomly selected image from the pretraining dataset, i.e., ImageNet, at the i-th training step. This penalty loss discourages the student from imitating the teacher on images beyond the normal images on the training set. Hence, the final loss between the student and the static teacher is formulated as:

Dynamic loss weighting is a mechanism that automatically adjusts the contribution of different teacher models during training. Instead of using fixed weighting, this approach dynamically balances the importance of the static and trainable teachers based on their reliability at each iteration. The model ensures that when the trainable teacher is unreliable in early training stages, the static teacher takes precedence. Conversely, as the trainable teacher improves, it gradually contributes more to the final loss calculation. This adaptive strategy eliminates the need for extensive hyperparameter tuning while improving learning stability and performance. During training on the \(\mathcal {D}_\text{ad}\) dataset, we train the \(\mathcal {S}\) and \(\mathcal {T}_\text{trainable}\) models simultaneously, and aggregate each of the loss terms with a weighted sum:where \(w_i^\mathrm {\Omega }\) and \(w_i^\mathrm {\Psi }\) control the weighting at the i-th training step, for the student model to dynamically balance the importance between the static teacher and the trainable teacher, respectively. The previous works [5, 6] did not consider balancing the importance between multiple models based on the reliability of the model. The individual loss terms w.r.t. the different models are considered equal. Although the weight parameters can be manually tuned using a single fixed value over the training iterations, the number of experiments on exhaustive combinations of the hyperparameters could easily increase largely with the number of models or the number of parameters increasing. In our approach, we introduce a novel dynamic loss weighting technique, in which the weight hyperparameters are dynamically adjusted during the training process based on the reliability of the related model. Since we are dealing with an unsupervised problem, where there is no ground truth to validate the actual detection performance of the models on the training or validation set, in our framework, we consider the \(L_i^\mathrm {\Theta }\) loss, i.e., the trainable teacher loss between trainable teacher and static teacher, as the reference to adjust the weight parameters with the formula is defined as follows:During the early stages of training, the \(L_i^\mathrm {\Theta }\) loss tends to be relatively high, as the trainable teacher has not yet been sufficiently trained. Consequently, it is necessary to dynamically decrease \( w_i^\mathrm {\Psi }\) and increase \(w_i^\mathrm {\Omega }\) to mitigate the influence of the trainable teacher when its reliability is low, allowing the model to rely more on the static teacher for stable guidance. As training progresses and the trainable teacher becomes more reliable, the weight distribution gradually shifts in the opposite direction, increasing the contribution of the trainable teacher. Notably, the sum of \(w_i^\mathrm {\Omega }\) and \(w_i^\mathrm {\Psi }\) is always equal to 1, ensuring a balanced distribution of contributions throughout training. The proposed dynamic loss weighting mechanism effectively adjusts these weights in an adaptive manner, optimizing the balance between the static and trainable teachers without requiring extensive hyperparameter tuning.

The dynamic loss weighting approach is grounded in the principle of adaptive optimization [41], emphasizing the model’s capability to adjust its learning strategy dynamically in response to the available information at each training iteration. Through dynamic weight adjustment of the losses, the dynamic loss weighting method ensures that the model prioritizes guidance from the more reliable teacher, leading to better optimization. In the scenario where time and resources for hyperparameter optimization are limited, the dynamic loss weighting technique eliminates the need for extensive manual tuning of hyperparameters related to the balance between the teachers. In addition, we also conducted experiments on using a single fixed parameter for the weight parameters and compared the performance with our proposed dynamic loss weighting.

Algorithm 1 presents the training procedure of the student–teacher network under the proposed framework. Instead of employing a frozen pre-trained model as the static teacher, our approach utilizes the patch description network (PDN), which is derived through knowledge distillation from a larger pre-trained model. This process follows the methodology of EfficientAD [6], ensuring consistency with prior works while leveraging the benefits of distilled knowledge for enhanced anomaly detection performance.

The MVTec dataset [2] is widely recognized as a prominent benchmark in the field of anomaly detection in computer vision. It consists of 15 categories, comprising 10 distinct objects and 5 unique textures. The training data exclusively consists of normal (anomaly-free) data, with an average of 240 image samples per category. The test dataset encompasses a variety of abnormal cases and a single normal case, with each case organized into different folders. Additionally, each abnormal case is paired with a corresponding masking image, serving as the ground truth.

The VisA dataset [21] is more recent and larger than the MVTec dataset. This dataset comprises 12 categories with a total of 10,821 high-resolution images. Among these images, 9621 represent normal samples, while 1200 represent abnormal samples, with image-level annotations provided in CSV files. Similarly to the MVTec dataset, the VisA dataset also includes annotations of ground truth masks for abnormal images.

The availability of both image-level and pixel-level labeling in both the MVTec and VisA datasets facilitates the evaluation of models for two distinct tasks: the detection task and the localization or segmentation task. The detection task involves identifying anomalies as we would in an image classification problem, distinguishing between normal and abnormal instances at the image level. In contrast, the localization or segmentation task focuses on precisely locating anomalies at the pixel level, providing detailed information about the specific pixels that make up anomalous regions.

We evaluate our method using the two widely used metrics in anomaly detection, i.e., AU-ROC and AU-PRO. Specifically, we evaluate our method’s detection and localization performance using AU-ROC and AU-PRO, respectively. As suggested by [4, 6, 42], the AU-PRO performance is computed until false-positive rate of 30%. In addition, we evaluate the computational performance of our method using some computational efficiency metrics, including latency, throughput, CUDA memory usage, number of model parameters, and floating-point operations (FLOPs). Latency measures the time taken to infer detection results for a single input image, while throughput quantifies the number of input images processed per second through batch processing. To compute CUDA memory usage and FLOPs, we utilize PyTorch’s official profiling framework [43].

For \(\mathcal {T}_\text{static}\) and \(\mathcal {S}\) networks, we employ the medium variant of the PDN architecture [6]. To speed up the forward pass, we also disable the padding in the convolutional layers of the PDN architecture. Meanwhile, for \(\mathcal {T}_\text{trainable}\), we utilize an autoencoder-based model following the same reference. Given the structural similarity between our approach and the EfficientAD method [6], we establish the EfficientAD method as the baseline for our work.

Before training, we initialized the convolutional layer parameters using PyTorch’s default initialization method [43]. The image input is resized to a resolution of \(256 \times 256\) using bilinear interpolation, and then normalized using the mean and standard deviation values of the torchvision [43] pre-trained models for the RGB channels, that is [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively. For all models, we set the output dimensions to \(C=384\), \(W=56\), and \(H=56\). For the output of the \(\mathcal {T}_\text{static}\), we follow EfficientAD to normalize the output channel-wise using the training set of the anomaly detection dataset.

For each experiment, the training process consists of 70,000 iterations, where a random training image is selected for each of the iterations. The Adam [44] optimizer is utilized with an initial learning rate of 1e-4 and a weight decay factor of 1e-5. We then reduce the learning rate after reaching 95% of the total iterations to 1e-5. For the hyperparameters \(\beta _1\) and \(\beta _2\) of the Adam optimizer, we use the default values of 0.9 and 0.999, respectively. For the augmentation techniques as shown in Fig. 7, which correspond to the trainable teacher, we randomly change either brightness, contrast, or saturation, for up to 20% in either direction.

During inference, we utilize quantile-based normalization as described in [6]. This process generates the final anomaly map, which is a combination of two anomaly maps: one based on the mean squared difference between the output of \(\mathcal {S}\) and \(\mathcal {T}_\text{static}\) and another based on the mean squared difference between the output of \(\mathcal {S}\) and \(\mathcal {T}_\text{trainable}\). The combined anomaly map is then resized to match the original image input size utilizing bilinear interpolation.

We used PyTorch to implement our method and used a computer with NVIDIA RTX 3090 GPU and AMD Ryzen 9 5900X CPU, and 64 GB RAM. With these settings, our training process takes about 2 h for the 70,000 iterations.

In this section, we quantitatively compare our method with the baseline and other existing methods on the MVTec AD and VisA datasets. Besides, we conduct an ablation study of the proposed methods on the MVTec AD dataset. Finally, we provide computational performance results for our method compared to the baseline.

We qualitatively analyze and compare our proposed method to the baseline model with six samples from both MVTec and VisA datasets as illustrated in Fig. 8 and Fig. 9, respectively. As shown in the transistor and zipper images in Fig. 8 and cashew, pipe_fryum, and pcb1 images in Fig. 9, compared with the baseline method, we can see that our proposed method avoids the false detection of an anomaly on the part of images where the anomaly has not occurred. At the same time, our proposed model is still able to detect the anomalous parts equally well. We believe this is because our proposed model does not completely neglect most of the parts in the image while still giving higher importance to some by utilizing the proposed soft-hard feature loss.

From the image of the capsule and chewing gum, we can see that the prediction of the proposed method is more localized than the baseline method, by having a smaller detected anomaly area. From the carpet image, the proposed method covered more anomalous areas than the baseline method. From the sample image of the tile, wood, macaroni1, and fryum, we can notice that our proposed method has better confidence in predicting the anomaly, indicated by more red color shown in the images. We expect these behaviors because the dynamic loss helps to balance the importance of the model, optimizing the final detection performance.