Spectro Temporal Fusion with CLSTM-Autoencoder based approach for Anomalous Sound Detection

The task of abnormal sound detection (ASD) is challenging and demanding in monitoring applications such as industrial machines, animal care, and public surveillance [1]. When compared to visual surveillance, audio sensors offer few advantages: (i) audio data acquisition and basic processing is much less expensive with respect to memory space and computational load (ii) microphones offer omnidirectional coverage (iii) insensitive to luminosity and critical weather conditions and (iv) audio cues can supplement video cues in cases where anomalous events/objects are occluded. Neural network-based techniques have been widely employed to address the ASD problem as deep learning models have grown in popularity [2, 3]. Using data from solely normal sound occurrences, the self-supervised anomaly detection approach trains the auto encoders (AE) to learn the patterns of normal sound events and to spot abnormalities.

Acoustic characteristics in sound signal can also be interpreted as time-frequency representation in the form of an auditory image. AEs with convolutional layers (CAE) are commonly used for image reconstruction. To capture the sequential patterns from normal sound signals, we propose to use convolutional long short-term memory auto encoder (CLSTM-AE) based approach for anomalous sound detection. With an encoder-decoder paradigm, the CLSTM Autoencoder is an autoencoder suitable for sequential data. The trained encoder can be used to obtain compressed representation of speech, text, and other sequential data. CLSTM-AE has the ability to recall or use long term dependencies over lengthy input sequences with the help of internal memory. The decoder model that reconstructs the input sequence uses these learnt representation as input. The reconstruction error serves as the basic measure for this model’s efficacy. In order to lower the reconstruction error or anomaly score of the sound data, the CLSTM-AE learns the network parameters during the training phase. During testing, anomalous sounds won’t be compressed properly and so provide high anomaly scores or reconstruction error. Optimal thresholding identifies anomalous sounds.

In addition to spectrogram image input, we explore CNN based Temporal gram Network to learn temporal (Tgram) features from normal sound data. The log-Mel spectrogram (Sgram) and the Tgram are fused to form a Mel-Spectro-tempogram which is then used to train CLSTM-AE. The rest of this paper is structured as follows: state of the art deep learning based approaches for anomalous sound detection are reviewed in “Related work”. The proposed CLSTM-AE based approach is presented in Sect. 3. Experimental study in Sect. 4 covers ablation studies of proposed approach on DCASE 2020 Task 2 dataset and performance analysis with state of the art deep model based approaches.

A wide range of deep anomaly detection algorithms have been introduced, showing noticeably superior performance than traditional machine learning based anomaly detection in tackling difficult detection problems [4‐6]. In Koizumi et al. [7], Variational Auto Encoder (VAE) is employed which used normal distribution in the latent layer of VAE. In addition to preventing overfitting, this can help in data reconstruction. Acoustic input was transformed into auditory images by convolutional layers in the AE. The authors used the MIMII dataset, which included four machine sounds that were both normal and abnormal (fans, pumps, slide rails and valves).

According to Purohit et al. [8], the Neyman-Pearson lemma is used to describe the loss function of AE, and ASD is seen as a statistical hypothesis test to mimic anomalous sound. The authors conclude that their method improves ASD performance measures with a low false positive rate. The drawback of this approach is that it simulates aberrant sound via costly rejection sampling. In Purohit et al. [8], WaveNet is used to predict sound data rather than generating it, and prediction error is used to calculate the anomaly score. Studies were carried out using real-world dataset recorded in a subway station. According to experimental findings, WaveNet marginally boosts an ASD system’s effectiveness in comparison to AE-based methods. However, the WaveNet incurs higher computational cost.

Recent study using evaluation data of DCASE 2020 Task 2 covered (1) Ensemble approaches using "outlier exposure" (OE)- and "inlier modelling" (IM)-based detectors, and (2) approaches using IM-based detection for features discovered during a machine-identification task [9]. Group Masked Autoencoder for Density Estimation (GMADE) is an unique density estimation-based anomaly detector that is built on an ensemble with a self-supervised classification-based anomaly detector [10]. A unique approach [11] uses audio representations based on Gammatones and convolutional autoencoders (both unsupervised and semi-supervised).These architectures significantly outperform the baseline results.

According to a technical report [12] that summarized performance of Task 2 of the DCASE 2020 challenge dataset, few attempts followed self supervised learning based techniques to detect abnormal sounds. Training used only samples of normal machine operations. These techniques employ deep autoencoders that leverage melspectogram-processed sound features and dense convolutional architectures. The dense and convolutional AE trained and tested using six types of machine operations produced results that were competitive with and even better than the baseline technique given in DCASE challenge. The baseline consists of a dense Autoencoder (AE) with three layers, in both the encoder and decoder components, with 128 units, and a latent space with 8 units, all with the ReLU activation function.

Long short term memory (LSTM) Autoencoder and Support Vector Machines used in another approach [8] takes into account the fact that, due to anomalies occurring in real-time industrial environments on an infrequent basis, there would be fewer aberrant samples than normal samples. One Class SVM receives the compressed patterns from the input data that the autoencoder model has presented. The normal sound data is used to train the model, and both the normal and anomalous data are used to test the learned model.

Another framework proposed a dense, convolutional autoencoder framework [8]. The mel-spectrograms of the audio data were used as input. When compared to classification based on raw audio data, the mel-spectrogram image aids in emphasising the spatial features and hence helps in improved classification outcomes. The MIMII Dataset and a portion of the ToyAdmos Dataset were used to train this model.

A deep learning system called MAABL (MobileNetV2 with enhanced attention block), which combines Deep Convolutional Neural Networks with Attention Blocks, is presented in this work [13]. An augmented attention block and an inverted residual block from MobileNetV2 are combined to create the neural network. In order to train neural networks, attention blocks assist in highlighting the aspects that are most important. MIMII and the ToyAdmos datasets were utilised for the experimentation. In comparison to the MobileNetV2 basic architecture and a few other frameworks that combined convolutional LSTM, and attention modules, the proposed framework displayed promising results.

One of the main challenges when working with the anomaly detection is high dimensional data. Combination of an anomaly detection mechanism and a feature extractor model is explored in [14]. They proposed a hybrid model that combines Deep Belief Networks (DBN) and One-Class SVM. The OneClass SVM is trained using the features that the DBN framework has learned from the input data. Results are reported on par with a Deep Autoencoder in terms of performance.

A similarity function was put forth by the author [7] for use in one-shot anomaly detection of sounds. The proposed method is called the SPecific anomaly IDentifiER network (SPIDERnet). As the similarity function used was based on the naive mean squared error between input data and the memorised spectrogram, previous research that used memory-based one-shot learning were only able to detect short anomalous sounds. SPIDERnet is viewed as a fusion of attention mechanisms for capturing time-frequency stretching and feature extractors based on neural networks for similarity measurement. ToyADMOS and MIMII datasets were utilised for experimenting. In comparison to other approaches [15‐17] utilised for industrial anomaly identification, better performance is reported in this work.

Self supervised (one class) deep model based anomaly detection approaches train the autoencoders (AE) using data of normal sounds only to learn the patterns of normal sound events. Convolutional AEs (CAE) are commonly used for image reconstruction. To capture the sequential patterns from normal sound signals, we propose to use Convolutional Long Short-Term Memory Auto Encoder (CLSTM-AE) based approach for anomalous sound detection. Spectrograms are commonly used with CAE for sound anomaly detection. In addition to spectrogram image input, we explore CNN based Temporal gram Network to learn temporal features from normal sound data.

The proposed approach is evaluated on the DCASE 2020 Challenge Task 2 (Unsupervised Anomalous Sound Identification for Machine Condition Monitoring). The dataset includes normal and abnormal sounds of six types of machines namely TGoyCar, ToyConveyor, Fan, Pump, Slide rail, and Valve. Each sound clip is of 10 s duration and includes environmental noise. In unsupervised anomalous sound detection tasks, it is assumed that anomaly detectors are trained entirely with typical sound clips excluding anomalies, but after training, are capable of accurately recognising anomalies.

The training data (normal sounds) from the development and supplementary datasets of Task 2 are utilized as the training set in the experiments, while the test data (normal and anomaly sounds) from the development dataset is used for evaluation. Initial experiments were conducted using Mean Square Error (MSE). However, on further analysis and experimentation on other datasets, we concluded that MSE is not a promising error metric as it lacks local sensitivity since it ignores the spatial organization in images. Then, we used Proximal Sensitive error [25], a locally sensitive error function that considers the relative positions in the spectrograms also. For the purpose of illustration, we used spectrograms of few samples of normal and abnormal sounds from the Toy Car dataset and the error values are plotted in Figs. 4 and 5. Based on our analysis on the average error values for all the spectrograms of normal and abnormal sounds, we fixed a threshold of 0.8 for classification. All the samples with reconstruction error below 0.8 would be classified as Normal and those with reconstruction error above would be classified as Anomalous. We studied the performance of proposed approach and SCR-LSTM (stacked convolutional Residual LSTM) approach for three types of inputs namely, spectrogram, Tgram and SpectroTempogram (STgram) for ToyCar dataset and the STgram based CLSTM-AE approach significantly improves the performance as shown in Table 1. The epoch vs training loss curve in case of training using ToyCar dataset is shown in Fig. 6.

In addition to DCASE 2020 Task 2 data, the MIMII (Malfunctioning Industrial Machine Investigation and Inspection) dataset is also used for verifying the robustness of the proposed approach. It consisted of data from four kinds of industrial machines which include Pump, Valve, Slide Rail and Fan. The data was taped down with a microphone placed at 10 cm distances. Each category of dataset consisted of audio signals with three kinds of noise—6dB, 0dB and 6dB and sampled at 16kHz and the AUC results of proposed approach are shown in Table 2. Comparison of AUC results with two other baseline methods such as AE and IDNN [24] are presented in Table 3. Interpolating deep neural network (IDNN) model is trained to forecast a given time frame of a representation from surrounding frames which leads to slightly better performance compared to the basic autoencoder.

Studies were carried out on DCASE Task 2 dataset using few state of the art methods and the comparison of performance for seven types of machines is shown in Table 4. ANP [23] is an approach namely Attentive Neural Process. It is an encoder/decoder architecture, which works based on encoding each of the elements in the context set along with the observed values to predict the output. It is trained to estimate the conditionally independent Gaussian parameters for each of the elements of the target set by attending to the context points of nearby coordinates. It seems to give reasonable results for few scenarios like Toy Car, Slide Rail. However, for other sounds, it seems to give poor results.

SCR-LSTM [12] consists of stacked two- and one-dimensional CNNs and residual long short-term memory (LSTM) and is not so promising in detecting anomalies. Group-Masked Autoencoder based Density Estimation (Group-MADE) approach for audio anomaly detection yields good result for sound anomaly detection [18]. This approach gave slightly better performance than Dense AE based approach. Glow aff. [21] is basically based on exact likelihood estimation with the help of Normalizing Flows. The model is trained to output higher likelihood to the target machine sounds than that compared to sounds from other machines of same machine type. This is to overcome the issues with out-of-distribution detection where the likelihood will be affected by the smoothness of the data.

MobileNetV2 architecture [19] used a sequence of 2 or more identical layers. Every layer in the same sequence will have same number of output channels. All spatial convolutions use 3 kernels. MobileNet V2 detected anomalous sounds of Slide rail and valve machine types better compared to other methods. ResNet-50 model consists of 5 stages each of which has a convolution and Identity blocks. A convolution block has 3 convolution layers and an identity block has 3 convolution layers [20]. ResNet provided comparable performance with that of MobileNet. IDCAE [22] is an adaptive class conditioned autoencoder which best suits for open-set recognition problems. It is able to provide reasonable good results on comparison with other models. The proposed CLSTM-AE with STgram input significantly improved the anomaly detection performance consistently for all 7 types of machines.

We have proposed spectro-tempograms with CLSTM-AE based approach for anomalous sound detection. Fusion of temporal features learnt using TgramNet and the spectrogram features enhances the quality of input to the CLSTM-AE model. The proposed model was validated using DCASE Challenge Task 2 dataset that comprised of sound data from seven types of machines namely Toy Car, Toy Train, pump, slide rails, fans, and valve. The model significantly improved the performance over few baseline and state of the art deep models .