Toward Data Augmentation and Interpretation in Sensor-Based Fine-Grained Hand Activity Recognition

Recognizing fine-grained hand activities has widely attracted the research community’s attention in recent years. However, rather than enriched sen-sor-based datasets of whole-body activities, there are limited data available for acceler-ator-based fine-grained hand activities. In this paper, we propose a purely convolution-based Generative Adversarial Networks (GAN) approach for data augmentation on accelerator-based temporal data of fine-grained hand activities. The approach consists of 2D-Convolution discriminator and 2D-Transposed-Convolution generator that are shown capable of learning the distribution of re-shaped sensor-based data and generating synthetic instances that well reserve the cross-axis co-relation. We evaluate the usability of synthetic data by expanding existing datasets and improving the state-of-the-art classifier’s test accuracy. The in-nature unreadable sensor-based data is interpreted by introducing visualization methods including axis-wise heatmap and model-oriented decision explanation. The experiments show that our approach can effectively improve the classifier’s test accuracy by GAN-based data augmentation while well preserving the authenticity of synthetic data.