Introduction
Contributions
Outline
Review of related works
Authors | Activities | Sensors/position | Algorithms/evaluation | Fusion type | Strength | Weakness |
---|---|---|---|---|---|---|
Static: sitting, standing, lying, sleeping, idle Mobility based: walking, climbing stairs, leaving home Household: prepare breakfast, prepare dinner, drink Daily hygiene: shower, use toilet | Sensors: Camera, accelerometer, gyroscope, magnetometer, binary sensor Position: ankle, toilet flush, cupboard Sampling rate: ~ 100 Hz | Algorithm: dynamic Bayesian network, Kalman filtering, Dempster–Shafer Evaluation metrics: error rates, accuracy, computation time, precision, recall, F-measure | Data fusion | Provide a simple and real-time, computationally efficient and independent implementation of human activity recognition | Inability to handle a long sequence of activities. Moreover, the approach is sensitive to sensor position, noise and sometimes impractical to implement | |
Static: sitting, lying down, standing, reading, making calls, kneeling Mobility based: walking, jogging, ascending descending stairs, biking, stretching, object-lifting, bending, falling forward, falling left, falling backward, talking, cycling, nordic walking, jumping, car, running Household: eating, watching TV, open door, close door, open fridge, close fridge, open dishwasher, close dishwasher, open drawer, close drawer, close table, drink form cup, vacuuming, ironing, shopping, cooking Daily hygiene: brush teeth, bathing, washing clothes, drying clothes, house cleaning Transition activities: stand up from sitting, stand up from laying, going up, going down, laying down from standing Office: Typing, Writing, walking at computer Harmful habit: smoking | Sensor: accelerometer, gyroscope, magnetometer, linear acceleration, gravity, heart rate, location sensor, air pressure, video Position: front pocket, chest, ankle, thigh, forearm, wrist, waist, back, feet, Right shoulder Sampling rate: 20–200 Hz | Algorithms: k-NN, ANN, DT, SVM, NB, LR, RF, Hoeffding tree, HMM, RNN, CNN, LSTM, LDC, QDC, POLYC, PARZENC, Gaussian Mixture Model, MLP, FNN, DNN Evaluation metric: AUC, accuracy, precision, recall, F-measure, specificity, computation time, error rate, Kappa, FP, FN, Confusion Matrix | Feature fusion | Use to fuse sensor of diverse modalities and less sensitive to noise | Feature incompatibility, instability to sensor failure and signal variation reduce performance | |
Static: lying, sitting, standing, kneeling, sleeping, recreational activities Mobility based activities: running, walking, cycling, ascending stairs, descending stairs, jogging, travel, sports Household: washing dishes, mop floor, sweep the floor, eat chips, watch TV, shopping, brush teeth Hygiene based: cleaning, wash hand Office activities: working on the computer, meeting | Sensor: accelerometer, calorimeter, biosensor, body-media, accelerometer, sound, accelerometer, location data, vital signs Positions: chest, wrist, mouth, pocket, table (sound), shirt pocket, belt, bag sampling rate: 20–200 Hz | Base classifiers: MLR, SVR, GPR, M5P, MLP, SVM, RF, BDT, DNN, Adaboost, LR, J48, KNN Fusion method: posterior probability, majority voting, single classifier based multi-view stacking Evaluation metrics: F-measure, recall, precision, accuracy, Confusion Matrix | Multiple classifier systems | Can handle complex activity details, high dimensional sensor data and uncertainty by systematic classifier fusion. Combine heterogeneous and homogeneous classifier to reduce variance and ambiguity that are likely to occur in the single classifier | External knowledge dependencies and may be computationally complex based on the base classifier |
Data-level fusion
Feature-level fusion
Multiple classifier systems
Problem formulation
Proposed methodology
Signal processing
Feature extraction and normalization
Feature | Formula | Feature | Formula |
---|---|---|---|
Mean (µ) |
\(\overline{s} = \frac{1}{N}\sum\nolimits_{i = \,1}^{N} {s_{i} }\)
| Root mean square (\(R_{ms}\)) |
\(rms = \sqrt {\frac{1}{n}} \sum\nolimits_{i = 1}^{N} {\left( {s_{i} } \right)}^{2}\)
|
Median (\(M_{e}\)) |
\(median_{i} \left( {s_{i} } \right)\)
| Peak amplitude (\(\,P_{a}\)) |
\({ \hbox{max} }(s_{i} ) - { \hbox{min} }(s_{i} )\)
|
Maximum (\(\,M_{a}\)) |
\({ \hbox{max} }_{i} \left( {s_{i} } \right)\)
| Pitch angle (\(\,P_{k}\)) |
\(\arctan \left( {\frac{{x_{i} }}{{\sqrt {y^{2} + x_{i}^{2} } }}} \right)\)
|
Minimum (\(\,M_{i}\)) |
\({ \hbox{min} }_{i} \left( {s_{i} } \right)\)
| Signal power (\(\,S_{p}\)) |
\(\sum\nolimits_{i = 1}^{N} {s_{i}^{2} }\)
|
Harmonic mean (\(H_{m}\)) |
\(\frac{1}{N}\sum\nolimits_{i = 1}^{n} {\frac{1}{{s_{i} }}}\)
| Kurtosis (\(\,K_{r}\)) |
\(E\left[ {\left( {s_{i} - \overline{s} } \right)^{4} } \right]/E\left[ {\left( {s_{i} - \overline{s} } \right)^{2} } \right]^{2}\)
|
Standard deviation (\(\,\sigma\)) |
\(\sigma \, = \,\sqrt {\frac{1}{N}} \sum\nolimits_{i = 1}^{N} {\left( {s_{i} - \mathop s\limits^{\_} } \right)}^{2}\)
| Skewness (\(\,S_{k}\)) |
\(E\left[ {\left( {\frac{{s_{i} - \overline{s} }}{\sigma }} \right)^{3} } \right]\)
|
Variance (\(\,\sigma^{2}\)) |
\(\sigma^{2} \, = \,\frac{{\sum\nolimits_{{}}^{{}} {\left( {s_{i} - \overline{s} } \right)^{2} } }}{N}\)
| Energy (\(\,E\)) |
\(\frac{{\sum\nolimits_{i = 1}^{N} {\left[ {s_{i} } \right]^{2} } }}{{length(s_{i} )}}\)
|
Coefficient of variation (\(\,C_{v}\)) |
\(\,\frac{{\sigma_{si} }}{{\mu_{si} }}\)
| Entropy (\(\,H\)) |
\(\frac{{ - \sum\nolimits_{i = 1}^{N} {\left[ {S_{i} } \right]} \log \left[ {S_{i} } \right]}}{{length(S_{i} )}}\)
|
Interquartile range (\(\,I_{r}\)) |
\(Q_{3} (s_{i} ) - Q_{1} (s_{i} )\)
| Mean frequency (µF) |
\({{\sum\nolimits_{i = 1}^{N} {\left( {is_{i} (F)} \right)} } \mathord{\left/ {\vphantom {{\sum\nolimits_{i = 1}^{N} {\left( {is_{i} (F)} \right)} } {\sum\nolimits_{j = 1}^{N} {s_{j} } }}} \right. \kern-0pt} {\sum\nolimits_{j = 1}^{N} {s_{j} } }}(F)\)
|
Feature selection
Activity class imbalanced distribution
Classification algorithms
Decision tree (J48)
Support vector machine (SVM)
K-Nearest Neighbors (k-NN)
Logistic regression (LR)
Decision fusion using multi-view stacking ensemble method
-
Multi-view stacking with k-NN, J48, and k-NN (MST-k-NN–J48–k-NN): k-NN and decision tree were used as based base classifiers and k-NN as the Meta classifier;
-
Multi-view stacking with LR, k-NN, and J48 (MST-LR–k-NN–J48–LR): LR, k-NN, J48 were used as base classifiers and LR as the Meta classifier.
-
Multi-view stacking with LR, k-NN, J48 and (k-NN-LR): (MST-LR–k-NN–J48 (k-NN–LR)): LR, k-NN, and J48 as base classifiers and LR and k-NN as the Meta classifier. The final result is the average of the prediction of the two classification algorithms.
Experiments
Datasets description
Dataset 1 | Dataset 2 | |
---|---|---|
Sensors | Shimmer sensor devices containing 3D accelerometer, 3D gyroscope | Shimmer2 sensor device IMU containing 3D accelerometer, 3D gyroscope, 3D magnetometer and 2-lead electrocardiography (ECG) |
Placement | Right ankle, chest and right wrist | Ankle and wrist |
Physical activities performed | sitting, lying, standing, washing dishes, vacuuming, sweeping, walking, ascending stairs, descending stairs, running, bicycling on the ergometer (50 w), bicycling on the ergometer (100 w), rope jumping. | Standing still, sitting and relaxing, lying down, walking, climbing stairs, waist bend forward, the frontal elevation of arms, knee bending(crouching), cycling, jogging, running, jumping |
Number of activities | 13 | 12 |
Number of participants | 19 | 10 |
Sampling rate | 204.8 Hz | 50 Hz |
Filtering method | Linear interpolation | Linear interpolation |
Window type and size | 5 s with 50% overlap | 2 s |
Feature selection methods | Evolutionary search method, CorrelationAttributeEval, Ranker | Evolutionary search method, CorrelationAttributeEval, Ranker |
Evaluation method | Tenfold cross validation | Tenfold cross validation |
Experimental setup
Classification algorithm | Parameters |
---|---|
Support vector machine | batchSize=100;buildCalibrationModels=False;c=1.0;calibrator=Logistic-R 1.0E-8-M-1-num-decimal-places 4;checksTurnedOff=False; debug=False;doNotCheckCapabilities=False;epsilon=1.0E-12;filterType=Normalize;kernel=PolyKernel-E 1.0-C 250007; numDecimalPlaces=2;numFolds=-1;randomSeed=1;toleranceParameter=0.001 |
k-Nearest Neighbors | KNN=10;batchSize=100;crossValidate=False;debug=False;distanceWeighing=No distance weighing;doNotCheckCapabilities=False;meanSquared=false;nearestNeighbourSearchAlgorithm=LinearNNSearch;numDecimalPlaces=2;windowSize=0 Standard |
Decision tree (J48) | batchsize=100;binarysplits=false;collapseTree=True;confidenceFactor=0.25; debug=false;doNotCheckCapabilities=false;doNotMakeSplitPointActualValue=false; minNumObj=2;numDecimalPlaces=2;numFolds=3;reduceErrorPrunning=false;saveInstanceData=false; seed=1;subtreeRaising=True; unpruned=false;useLaplace=false;useMDLcorrection=true |
Logistic regression | batchSize=100;debug=false;doNotCheckCapabilities=false;maxit=-1;numDecimalPlaces=4; ridge=1.0E-8;useConjugateGradientDescent=false |
Model validation
Performance evaluation
Evaluation measures | Equation | Description | |
---|---|---|---|
Higher values indicate better performances | Accuracy |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {\frac{{(TP + TN)_{i} }}{{\left( {TP + FP + TN + FN} \right)_{i} }}}\)
| Calculate the rate of correctly classified activities classes out of the total number of activity instances |
Recall |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {\frac{{(TP)_{i} }}{{\left( {TP + FN} \right)_{i} }}}\)
| Measure the number of correctly predicted instances as positive instances | |
Precision |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {\frac{{(TP)_{i} }}{{\left( {TP + FP} \right)_{i} }}}\)
| Measure the ability of the proposed algorithm to accurate classify actual activity details | |
F-measure |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {2.\frac{{(precision*recall)_{i} }}{{\left( {precision + recall} \right)_{i} }}}\)
| Calculate the weighted harmonic mean of precision and recall | |
Area under the curve (AUC) |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {0.5*\left[ {\frac{{TP_{i} }}{{\left( {TP + FN} \right)_{i} }} + \frac{{TN_{i} }}{{\left( {TN + FP} \right)_{i} }}} \right]}\)
| Measure the rate of performance of the algorithms across all activity details. AUC is the plot between recall and specificity drawn from the different threshold | |
Lower values indicate higher performances | Error |
\(\frac{1}{N}\sum\limits_{i = 1}^{N} {\frac{{(FP + FN)_{i} }}{{\left( {TP + FP + TN + FN} \right)_{i} }}}\)
| Calculate the rate of activities incorrectly classified out of all the activity details |
Experimental results and discussions
Baseline method: analysis of individual motion sensor
Sensor fusion for improved human activity detection system
Positions | Methods | Accuracy (%) | Recall | Precision | F-measure | Errors | AUC |
---|---|---|---|---|---|---|---|
Ankle | SVM | 93.08 | 0.9029 | 0.9312 | 0.8987 | 0.0692 | 0.9465 |
KNN | 94.67 | 0.9243 | 0.9303 | 0.9272 | 0.0533 | 0.9272 | |
J48 | 95.13 | 0.9283 | 0.9380 | 0.9330 | 0.0487 | 0.9603 | |
LR | 95.60 | 0.9430 | 0.9382 | 0.9404 | 0.0440 | 0.9683 | |
Stacking–KNN–J48–KNN | 96.79 | 0.9592 | 0.9583 | 0.9587 | 0.0321 | 0.9772 | |
Stacking-LR–KNN–J48–LR | 97.49 | 0.9648 | 0.9673 | 0.9660 | 0.0251 | 0.9804 | |
Stacking–LR–KNN–J48–MV–LR–KNN |
97.57
|
0.9672
|
0.9669
|
0.9670
|
0.0243
|
0.9818
| |
Chest | SVM | 94.09 | 0.9107 | 0.9355 | 0.9210 | 0.0591 | 0.9505 |
KNN | 94.45 | 0.9148 | 0.9428 | 0.9275 | 0.0545 | 0.9528 | |
J48 | 93.35 | 0.9107 | 0.9185 | 0.9145 | 0.0665 | 0.9500 | |
LR | 95.32 | 0.9379 | 0.9373 | 0.9375 | 0.0468 | 0.9654 | |
Stacking-KNN–J48–KNN | 95.05 | 0.9323 | 0.9405 | 0.9362 | 0.0495 | 0.9621 | |
Stacking-LR–KNN–J48–LR | 95.67 | 0.9398 | 0.9477 | 0.9435 | 0.0433 | 0.9665 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
96.02
|
0.9425
|
0.9528
|
0.9474
|
0.0398
|
0.9681
| |
Wrist | SVM | 90.96 | 0.8432 | 0.9293 | 0.8758 | 0.0904 | 0.9134 |
KNN | 93.32 | 0.8828 | 0.9514 | 0.9111 | 0.0668 | 0.9311 | |
J48 | 91.89 | 0.8885 | 0.9020 | 0.8947 | 0.0811 | 0.9376 | |
LR | 91.38 | 0.8780 | 0.8884 | 0.8828 | 0.0862 | 0.9323 | |
Stacking-KNN–J48–KNN | 94.74 | 0.9209 | 0.9448 | 0.9319 | 0.0526 | 0.9560 | |
Stacking-LR–KNN–J48–LR | 95.32 | 0.9324 | 0.9472 | 0.9395 | 0.0468 | 0.9623 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
95.48
|
0.9330
|
0.9501
|
0.9412
|
0.0452
|
0.9628
|
Positions | Methods | Accuracy (%) | Recall | Precision | F-measure | Errors | AUC |
---|---|---|---|---|---|---|---|
Ankle |
SVM
| 90.24 | 0.9021 | 0.9029 | 0.9019 | 0.0976 | 0.9441 |
KNN | 98.17 | 0.9617 | 0.9818 | 0.9817 | 0.0183 | 0.9896 | |
J48 | 96.42 | 0.9642 | 0.9643 | 0.9642 | 0.0358 | 0.9795 | |
LR | 96.18 | 0.9617 | 0.9619 | 0.9616 | 0.0382 | 0.9781 | |
Stacking-KNN–J48–KNN |
98.50
|
0.9849
|
0.9850
|
0.9849
|
0.0150
|
0.9914
| |
Stacking-LR–KNN–J48–LR | 98.46 | 0.9845 | 0.9845 | 0.9845 | 0.0154 | 0.9912 | |
Stacking-LR–KNN–J48–MV–LR–KNN | 98.37 | 0.9837 | 0.9837 | 0.9837 | 0.0163 | 0.9907 | |
Wrist | SVM | 94.31 | 0.9430 | 0.9454 | 0.9431 | 0.0569 | 0.9674 |
KNN | 97.44 | 0.9743 | 0.9748 | 0.9744 | 0.0256 | 0.9653 | |
J48 | 95.33 | 0.9534 | 0.9535 | 0.9534 | 0.0467 | 0.9734 | |
LR | 94.15 | 0.9415 | 0.9414 | 0.9414 | 0.0585 | 0.9666 | |
Stacking-KNN–J48–KNN | 97.93 | 0.9793 | 0.9794 | 0.9793 | 0.0207 | 0.9882 | |
Stacking-LR–KNN–J48–LR | 97.76 | 0.9777 | 0.9777 | 0.9777 | 0.0224 | 0.9872 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
98.05
|
0.9805
|
0.9806
|
0.9805
|
0.0195
|
0.9889
|
Impact of over-sampling on the performance results
Positions | Methods | Accuracy (%) | Recall | Precision | F-measure | Errors | AUC |
---|---|---|---|---|---|---|---|
Ankle | SVM | 93.98 | 0.9060 | 0.9299 | 0.8991 | 0.0602 | 0.9488 |
KNN | 95.33 | 0.9295 | 0.9314 | 0.9303 | 0.0467 | 0.9614 | |
J48 | 96.45 | 0.9537 | 0.9560 | 0.9547 | 0.0355 | 0.9742 | |
LR | 96.48 | 0.9504 | 0.9505 | 0.9502 | 0.0352 | 0.9727 | |
Stacking-KNN–J48–KNN | 97.25 | 0.9629 | 0.9621 | 0.9624 | 0.0275 | 0.9794 | |
Stacking-LR–KNN–J48–LR | 97.92 | 0.9709 | 0.9713 | 0.9711 | 0.0208 | 0.9838 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
97.89
|
0.9718
|
0.9717
|
0.9717
|
0.0211
|
0.9844
| |
Chest | SVM | 94.43 | 0.9170 | 0.9386 | 0.9254 | 0.0557 | 0.9541 |
KNN | 95.20 | 0.9307 | 0.9434 | 0.9364 | 0.0480 | 0.9616 | |
J48 | 93.79 | 0.9197 | 0.9240 | 0.9218 | 0.0621 | 0.9551 | |
LR | 95.42 | 0.9419 | 0.9443 | 0.9430 | 0.0458 | 0.9675 | |
Stacking-KNN–J48–KNN | 95.87 | 0.9466 | 0.9465 | 0.9464 | 0.0413 | 0.9702 | |
Stacking-LR–KNN–J48–LR | 96.54 | 0.9565 | 0.9586 | 0.9575 | 0.0346 | 0.9756 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
96.73
|
0.9586
|
0.9603
|
0.9594
|
0.0327
|
0.9768
| |
Wrist | SVM | 91.26 | 0.8956 | 0.9254 | 0.9085 | 0.0874 | 0.9407 |
KNN | 95.30 | 0.9381 | 0.9534 | 0.9451 | 0.0470 | 0.9654 | |
J48 | 92.39 | 0.9222 | 0.9243 | 0.9232 | 0.0761 | 0.9552 | |
LR | 91.72 | 0.9040 | 0.9067 | 0.9052 | 0.0828 | 0.9457 | |
Stacking-KNN–J48–KNN | 95.91 | 0.9504 | 0.9522 | 0.9513 | 0.0409 | 0.9721 | |
Stacking-LR–KNN–J48-LR | 96.17 | 0.9524 | 0.9545 | 0.9477 | 0.0383 | 0.9733 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
96.63
|
0.9572
|
0.9616
|
0.9593
|
0.0337
|
0.9760
|
Positions | Methods | Accuracy (%) | Recall | Precision | F-Measure | Errors | AUC |
---|---|---|---|---|---|---|---|
Ankle | SVM | 93.67 | 0.9079 | 0.9573 | 0.9241 | 0.0633 | 0.9489 |
KNN | 98.45 | 0.9780 | 0.9868 | 0.9820 | 0.0155 | 0.9878 | |
J48 | 96.08 | 0.9537 | 0.9561 | 0.9549 | 0.0392 | 0.9740 | |
LR | 96.87 | 0.9642 | 0.9603 | 0.9622 | 0.0313 | 0.9799 | |
Stacking-KNN–J48–KNN |
99.18
|
0.9892
|
0.9924
|
0.9908
|
0.0082
|
0.9940
| |
Stacking-LR–KNN–J48–LR | 99.05 | 0.9880 | 0.9898 | 0.9889 | 0.0095 | 0.9933 | |
Stacking-LR–KNN–J48–MV–LR–KNN | 99.13 | 0.9897 | 0.9905 | 0.9901 | 0.0087 | 0.9942 | |
Wrist | SVM | 94.92 | 0.9392 | 0.9427 | 0.9407 | 0.0508 | 0.9658 |
KNN | 98.18 | 0.9740 | 0.9840 | 0.9787 | 0.0182 | 0.9856 | |
J48 | 96.74 | 0.9577 | 0.9605 | 0.9590 | 0.0326 | 0.9765 | |
LR | 94.59 | 0.9396 | 0.9383 | 0.9389 | 0.0541 | 0.9658 | |
Stacking-KNN–J48–KNN | 98.99 | 0.9864 | 0.9869 | 0.9866 | 0.0101 | 0.9925 | |
Stacking-LR–KNN–J48–LR | 98.98 | 0.9846 | 0.9869 | 0.9842 | 0.0120 | 0.9925 | |
Stacking-LR–KNN–J48–MV–LR–KNN |
99.02
|
0.9868
|
0.9871
|
0.9869
|
0.0098
|
0.9927
|
Significance of the proposed methods for human activity detection
Methods | Accuracy (%) | Recall | Precision | F-measure | AUC | |
---|---|---|---|---|---|---|
Dataset 1 | Saha et al. [16] | 91.85 | 0.8806 | 0.8963 | 0.8872 | 0.9340 |
Chowdhury et al. [26] | 95.32 | 0.9315 | 0.9414 | 0.9342 | 0.9622 | |
Ghojeski et al. [27] Random subspace | 95.05 | 0.9272 | 0.9365 | 0.9303 | 0.9599 | |
Ghojeski et al. [27] Bagging | 95.71 | 0.9381 | 0.9383 | 0.9378 | 0.9660 | |
Proposed method | 97.89 | 0.9718 | 0.9717 | 0.9717 | 0.9844 | |
Dataset 2 | Saha et al. [16] | 77.52 | 0.7749 | 0.7769 | 0.7718 | 0.8714 |
Chowdhury et al. [26] | 92.28 | 0.9227 | 0.9239 | 0.9224 | 0.9559 | |
Ghojeski et al. [27] Random subspace | 87.52 | 0.8750 | 0.8770 | 0.8746 | 0.9286 | |
Ghojeski et al. [27] Bagging | 86.26 | 0.8623 | 0.8643 | 0.8620 | 0.9213 | |
Proposed method | 99.18 | 0.9892 | 0.9924 | 0.9908 | 0.9940 |