Human Activity Sensing

From the increase in the number of rehabilitation patients, precise and detailed records of rehabilitation have become difficult due to the busyness of physical therapists. This creates difficulty for quantitative analysis of rehabilitation. Therefore, we are constructing a system that automatically records rehabilitation using activity recognition techniques with wearable sensors. The system can offer the dual prospect of decreasing a practitioners’ load and enabling quantitative analysis of rehabilitation. In general, a large number of wearable sensors are demanded to be placed on each part of a patient’s body for accurate activity recognition. However, managing a larger number of wearable sensors incurs significant time and effort of therapists and patients to apply them and adds to a patients’ discomfort, consequently it significantly decreases practicality. Therefore, we investigate the suitable number and positions of wearable sensors for activity recognition for rehabilitation. Experiments were carried out with 16 healthy subjects wearing seven wearable inertial measurement units (IMUs). The subjects performed 10 different rehabilitation activities chosen by a qualified physical therapist as typical ones used in real rehabilitation therapy. The activities were recognized while reducing the number of sensors with all combinations of sensor placements using six classification algorithms. Then, the accuracy on each setting was examined. As a result, 0.833 of F-measure value was obtained when using three sensors on the waist, right thigh, and right lower leg.

Every benchmark dataset that contains inertial data (acceleration, rate-of-turn, magnetic flux) requires a thorough description of the datasets itself. This description tends often to be unstructured, and supplied to researchers as a conventional description, and in many cases crucial details are not available anymore. In this chapter, we argue that each sensor modality exhibits particular statistical properties that allow to reconstruct the modality solely from the sensor data itself. In order to investigate this, tri-axial inertial sensor data from five publicly available datasets are analysed. We found that in particular three statistical properties, the mode, the kurtosis, and the number of modes tend to be sufficient for classification of sensor modality—requiring as the only assumption that the sampling rate and sample format are known, and the fact that that acceleration and magnetometer data is present in the dataset. With those assumption in place, we found that \(98\%\) of all 1003 data points were successfully classified.

There is an inherent problem of error accumulation in Pedestrian Dead Reckoning (PDR). In this chapter, we introduce a PDR error compensation scheme based on the assumption that can obtain sparse locations. Sparse locations are discontinuous locations obtained by using an absolute localization method or passage detection devices (ex. RFID tag, BLE beacon, Spinning Magnet Marker). Our proposal scheme focuses on being able to install anywhere in the indoor environment. In our scheme, we define error models that represent errors in PDR, including moving distance error and orientation change error. We apply the error models to counteract the error that occurs in PDR estimation. Moreover, the error models are tuned each time when a sparse location is measured. As a result, the proposed scheme improves the position error rate by approximately 10% and the route distance error rate by approximately 7%. In addition, we discuss the effectiveness of our scheme by each test route for future consideration.

As sensor-based inference models move out of laboratories into the real-world, it is of crucial importance that these models retain their performance under changing hardware and environment conditions that are expected to occur in-the-wild. This chapter motivates this challenging research problem in the context of audio sensing models, by presenting three empirical studies which evaluate the impact of hardware and environment variabilities on cloud-scale as well as embedded-scale audio models. Our results show that even the state-of-the-art deep learning models show significant performance degradation in the presence of ambient acoustic noise, and more surprisingly under scenarios of microphone variability, with accuracy losses as high as 15% in some scenarios. Further, we provide intuition on how this problem of model robustness relates to the broader topic of dataset-shift in the machine learning literature, and highlight future research directions for the mobile sensing community which include the investigation of domain adaptation and domain generalization solutions in the context of sensing systems.

The importance of location information is increasing more and more. Therefore, research on indoor positioning technology is actively conducted. Among them, Wi-Fi positioning is attracting attention because of its low introduction cost. However, since the Wi-Fi is blocked by the human body, the RSSI decreases. Therefore, we investigated radio wave attenuation by human body in preliminary experiments. We describe related research and its problems.

In this chapter we present a case study on drinking gesture recognition from a dataset annotated by Experience Sampling (ES). The dataset contains 8825 “sensor events” and users reported 1808 “drink events” through experience sampling. We first show that the annotations obtained through ES do not reflect accurately true drinking events. We present then how we maximise the value of this dataset through two approaches aiming at improving the quality of the annotations post-hoc. Based on the work presented in Ciliberto et al. (2018), we extend the application of template-matching (Warping Longest Common Subsequence) to multiple sensor channels in order to spot a subset of events which are highly likely to be drinking gestures. We then propose an unsupervised approach which can perform drinking gesture recognition by combining K-Means clustering with WLCSS. Experimental results verify the effectiveness of the proposed method, with an improvement of the F1 score by 16% compared to standard K-Means using Euclidean distance.

Inexpensive, low-power sensors and microcontrollers are widely available along with tutorials about how to use them in systems that sense the world around them. Despite this progress, it remains difficult for non-experts to design and implement event recognizers that find events in raw sensor data streams. Such a recognizer might identify specific events, such as gestures, from accelerometer or gyroscope data and be used to build an interactive system. While it is possible to use machine learning to learn event recognizers from labeled examples in sensor data streams, non-experts find it difficult to label events using sensor data alone. We combine sensor data and video recordings of example events to create a better interface for labeling examples. Non-expert users were able to collect video and sensor data and then quickly and accurately label example events using the video and sensor data together. We include 3 example systems based on event recognizers that were trained from examples labeled using this process.

Exchanging data as character-separated values (CSV) is slow, cumbersome and error-prone. Especially for time-series data, which is common in Activity Recognition, synchronizing several independently recorded sensors is challenging. Adding second level evidence, like video recordings from multiple angles and time-coded annotations, further complicates the matter of curating such data. A possible alternative is to make use of standardized multi-media formats. Sensor data can be encoded in audio format, and time-coded information, like annotations, as subtitles. Video data can be added easily. All this media can be merged into a single container file, which makes the issue of synchronization explicit. The incurred performance overhead by this encoding is shown to be negligible and compression can be applied to optimize storage and transmission overhead.

OpenHAR is a toolbox for Matlab to combine and unify 3D accelerometer data of ten publicly open data sets. This chapter introduces OpenHAR and provides initial experimental results based on it. Moreover, OpenHAR provides an easy access to these data sets by providing them in the same format, and in addition, units, measurement range, sampling rates, labels, and body position IDs are unified. Moreover, data sets have been visually inspected to fix visible errors, such as sensor in wrong orientation. For Matlab users OpenHAR provides code which user can use to easily select only desired parts of this data. This chapter also introduces OpenHAR to users without Matlab. For them, the whole OpenHAR data is provided as a one .txt-file. Altogether, OpenHAR contains over 280 h of accelerometer data from 211 study subjects performing 17 daily human activities and wearing sensors in 14 different body positions. This chapter shown the first experimental results based on OpenHAR data. The experiment was done using three classifiers: linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and classification and regression tree (CART). The experiment showed that using LDA and QDA classifiers and OpenHAR data, as high recognition rates can be achieved in a previously unseen test data than by using a data set specially collected for this purpose. With CART the results obtained using OpenHAR data were slightly lower.

Human Activity Recognition from accelerometer sensors is key to enable applications such as fitness tracking or health status monitoring at home. However, evaluating the performance of activity recognition systems in real-life deployments is challenging to the multiple differences in sensor number, placement and orientation that may arise in real settings. Considering such differences requires a large amount of labeled data. To overcome the challenges and costs associated to the collection of a wide range of heterogeneous data, we propose a simulator, called MEASURed, which uses motion capture to simulate accelerometer data on different settings. Then, using the simulated data to estimate the performance of activity recognition models under different scenarios. In this chapter, we describe MEASURed and evaluate its performance in estimating the accuracy of activity recognition models. Our results show that MEASURed can estimate the average accuracy of an activity recognition model using real accelerometer magnitude data. By using motion capture to simulate accelerometer data, the sensor research community can profit from visual datasets that have been collected by other communities to evaluate performance of activity recognition in a wide range of activities. MEASURed can be used to evaluate activity recognition classifiers in settings with different number, placement, and sampling rate of accelerometer sensors. The evaluation on a broad spectrum of scenarios gives a more general view of models and their limitations.

The Sussex-Huawei Transportation-Locomotion (SHL) Recognition Challenge 2018 aims to recognize eight transportation activities (Still, Walk, Run, Bike, Bus, Car, Train, Subway) from the inertial and pressure sensor data of a smartphone. In this chapter, we, as part of competition organizing team, present reference recognition performance obtained by applying various classical and deep-learning classifiers to the testing dataset. The classical classifiers include naive Bayes, decision tree, random forest, K-nearest neighbours and support vector machine, while the deep-learning classifiers include fully-connected and convolutional deep neural networks. We feed different types of input to the classifier, including hand-crafted features, raw sensor data in the time domain, and in the frequency domain. We additionally employ a post-processing scheme, which smoothens the predictions in order and improves the recognition performance. Results show that convolutional neural network operating on frequency-domain raw data achieves the best performance among all the classifiers. Finally, we achieve a benchmark result with F1 score 92.9%, which is comparable to the best result from the team that won the competition (achieving F1 score 93.9%). The competition dataset and the benchmark implementation is made available online (http://​www.​shl-dataset.​org/​).

Design of neural architectures is a critical aspect in deep-learning based methods. In this chapter, we explore the suitability of different neural architectures for the recognition of mobility-related human activities. Neural architecture search (NAS) is getting a lot of attention in the machine learning community and improves deep learning models’ performances in many tasks like language modeling and image recognition. Deep learning techniques were successfully applied to human activity recognition (HAR). However, the design of competitive architectures remains cumbersome, time-consuming, and rely strongly on domain expertise. To address this, we propose a large-scale systematic experimental setup in order to design and evaluate neural architectures for HAR applications. Specifically, we use a Bayesian optimization (BO) procedure based on a Gaussian process surrogate model in order to tune architectures’ hyper-parameters. We train and evaluate more than 600 different architectures which are then analyzed via the functional ANalysis Of VAriance (fANOVA) framework to assess hyper-parameters relevance. We experiment our approach on the Sussex-Huawei Locomotion and Transportation (SHL) dataset, a highly versatile, sensor-rich and precisely annotated dataset of human locomotion modes.

Most submissions to the 2018 Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge strongly overestimated the performance of their algorithms in relation to their performance achieved on the challenge evaluation data. Similarly, recent studies on smartphone based trip data collection promise accurate and detailed recognition of various modes of transportation, but it appears that in field tests the available techniques cannot live up to the expectations. In this chapter we experimentally demonstrate potential sources of upward scoring bias in the evaluation of travel activity classifiers. Our results show that (1) performance measures such as accuracy and the average class-wise F1 score are sensitive to class prevalence which can vary strongly across sub-populations, (2) cross-validation with random train/test splits or large number of folds can easily introduce dependencies between training and test data and are therefore not suitable to reveal overfitting, and (3) splitting the data into disjoint subsets for training and test does not always allow to discover model overfitting caused by lack of variation in the data.

The Sussex-Huawei Locomotion-Transportation Recognition Challenge presented a unique opportunity to the activity-recognition community to test their approaches on a large, real-life benchmark dataset with activities different from those typically being recognized. The goal of the challenge was to recognize eight locomotion activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway). This chapter describes the submissions winning the first and second place. They both start with data preprocessing, including a normalization of the phone orientation. Then, a wide set of hand-crafted domain features in both frequency and time domain are computed and their quality evaluated. The second-place submission feeds the best features into an XGBoost machine-learning model with optimized hyper-parameters, achieving the accuracy of 90.2%. The first-place submission builds an ensemble of models, including deep learning models, and finally refines the ensemble’s predictions by smoothing with a Hidden Markov model. Its accuracy on an internal test set was 96.0%.