Bi-STAN: bilinear spatial-temporal attention network for wearable human activity recognition

With the progressive development of ubiquitous computing, wearable human activity recognition is playing an increasingly important role in many fields, such as health monitoring, disease-assisted diagnostic rehabilitation, and exercise assessment. Internal measurement unit in wearable devices provides a rich representation of motion. Human activity recognition based on sensor sequence has proven to be crucial in machine learning research. The key challenge is to extract powerful representational features from multi-sensor data to capture subtle differences in human activities. Beyond this challenge, due to the lack of attention to the temporal and spatial dependence of the data, critical information is often lost in the feature extraction process. Few previous papers can jointly address these two challenges. In this paper, we propose an efficient Bilinear Spatial-Temporal Attention Network (Bi-STAN). Firstly, a multi-scale ResNet backbone network is used to extract multimodal signal features and jointly optimize the feature extraction process. Then, to adaptively focus on what and where is important in the original data and to mine the discriminative part of the features, we design a spatial-temporal attention network. Finally, a bilinear pooling with low redundancy is introduced to efficiently obtain second-order information. Experiments on three public datasets and our real-world dataset demonstrate that the proposed Bi-STAN is superior to existing methods in terms of both accuracy and efficiency. The code and models are publicly available at https://​github.​com/​ilovesea/​Bi-STAN.​