ABSTRACT
Mobile sensing and computing applications usually require time-series inputs from sensors, such as accelerometers, gyroscopes, and magnetometers. Some applications, such as tracking, can use sensed acceleration and rate of rotation to calculate displacement based on physical system models. Other applications, such as activity recognition, extract manually designed features from sensor inputs for classification. Such applications face two challenges. On one hand, on-device sensor measurements are noisy. For many mobile applications, it is hard to find a distribution that exactly describes the noise in practice. Unfortunately, calculating target quantities based on physical system and noise models is only as accurate as the noise assumptions. Similarly, in classification applications, although manually designed features have proven to be effective, it is not always straightforward to find the most robust features to accommodate diverse sensor noise patterns and heterogeneous user behaviors. To this end, we propose DeepSense, a deep learning framework that directly addresses the aforementioned noise and feature customization challenges in a unified manner. DeepSense integrates convolutional and recurrent neural networks to exploit local interactions among similar mobile sensors, merge local interactions of different sensory modalities into global interactions, and extract temporal relationships to model signal dynamics. DeepSense thus provides a general signal estimation and classification framework that accommodates a wide range of applications. We demonstrate the effectiveness of DeepSense using three representative and challenging tasks: car tracking with motion sensors, heterogeneous human activity recognition, and user identification with biometric motion analysis. DeepSense significantly outperforms the state-of-the-art methods for all three tasks. In addition, we show that DeepSense is feasible to implement on smartphones and embedded devices thanks to its moderate energy consumption and low latency.
- Intel edison compute module. http://www.intel. com/content/dam/support/us/en/documents/ edison/sb/edison-module_HG_331189.pdf .Google Scholar
- Qualcomm snapdragon 800 processor. https://www.qualcomm.com/products/ snapdragon/processors/800.Google Scholar
- W. T. Ang, P. K. Khosla, and C. N. Riviere. Nonlinear regression model of a low-g mems accelerometer. IEEE Sensors Journal, 2007.Google Scholar
- D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.Google Scholar
- I. G. Y. Bengio and A. Courville. Deep learning. Book in preparation for MIT Press, 2016.Google Scholar
- S. Bhattacharya and N. D. Lane. From smart to deep: Robust activity recognition on smartwatches using deep learning. In PerCom Workshops, 2016.Google ScholarCross Ref
- G. Chandrasekaran, T. Vu, A. Varshavsky, M. Gruteser, R. P. Martin, J. Yang, and Y. Chen. Tracking vehicular speed variations by warping mobile phone signal strengths. In PerCom, 2011. Google ScholarDigital Library
- J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014.Google Scholar
- T. Cooijmans, N. Ballas, C. Laurent, and A. Courville. Recurrent batch normalization. arXiv:1603.09025, 2016.Google Scholar
- G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE TASLP, 2012. Google ScholarDigital Library
- J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.Google ScholarCross Ref
- D. Figo, P. C. Diniz, D. R. Ferreira, and J. M. Cardoso. Preprocessing techniques for context recognition from accelerometer data. Pers. Ubiquit. Comput., 2010. Google ScholarDigital Library
- M. Gadaleta and M. Rossi. Idnet: Smartphone-based gait recognition with convolutional neural networks. arXiv:1606.03238, 2016.Google Scholar
- K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. Lstm: A search space odyssey. arXiv:1503.04069, 2015.Google Scholar
- N. Y. Hammerla, R. Kirkham, P. Andras, and T. Ploetz. On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution. In ISWC, 2013. Google ScholarDigital Library
- K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv:1512.03385, 2015.Google Scholar
- S. Hu, L. Su, S. Li, S. Wang, C. Pan, S. Gu, M. T. Al Amin, H. Liu, S. Nath, et al. Experiences with enav: a low-power vehicular navigation system. In UbiComp, 2015. Google ScholarDigital Library
- S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.Google Scholar
- L. Kang, B. Qi, D. Janecek, and S. Banerjee. Ecodrive: A mobile sensing and control system for fuel efficient driving. In MobiCom, 2015. Google ScholarDigital Library
- J. Ko, C. Lu, M. B. Srivastava, J. A. Stankovic, A. Terzis, and M. Welsh. Wireless sensor networks for healthcare. Proc. IEEE, 2010.Google Scholar
- N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao, L. Qendro, and F. Kawsar. Deepx: A software accelerator for low-power deep learning inference on mobile devices. In IPSN, 2016. Google ScholarDigital Library
- N. D. Lane, P. Georgiev, and L. Qendro. Deepear: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In UbiComp, 2015. Google ScholarDigital Library
- N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury, and A. T. Campbell. A survey of mobile phone sensing. IEEE Commun. Mag., 2010. Google ScholarDigital Library
- C.-Y. Li, C.-H. Yen, K.-C. Wang, C.-W. You, S.-Y. Lau, C. C.-H. Chen, P. Huang, and H.-H. Chu. Bioscope: an extensible bandage system for facilitating data collection in nursing assessments. In UbiComp, 2014. Google ScholarDigital Library
- T. Li, C. An, Z. Tian, A. T. Campbell, and X. Zhou. Human sensing using visible light communication. In MobiCom, 2015. Google ScholarDigital Library
- R. LiKamWa, Y. Hou, J. Gao, M. Polansky, and L. Zhong. Redeye: analog convnet image sensor architecture for continuous mobile vision. In ISCA, pages 255--266, 2016. Google ScholarDigital Library
- K. Lin, A. Kansal, D. Lymberopoulos, and F. Zhao. Energy-accuracy aware localization for mobile devices. In MobiSys, 2010.Google Scholar
- G. Milette and A. Stroud. Professional Android sensor programming. John Wiley & Sons, 2012. Google ScholarDigital Library
- E. Miluzzo, A. Varshavsky, S. Balakrishnan, and R. R. Choudhury. Tapprints: your finger taps have fingerprints. In MobiSys, 2012. Google ScholarDigital Library
- F. J. O. Morales and D. Roggen. Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In ISWC, 2016. Google ScholarDigital Library
- S. Nath. Ace: exploiting correlation for energy-efficient and continuous context sensing. In MobiSys, 2012. Google ScholarDigital Library
- M. Park. Error analysis and stochastic modeling of MEMS-based inertial sensors for land vehicle navigation applications. Library and Archives Canada= Bibliothèque et Archives Canada, 2005.Google Scholar
- M. Rabbi, M. H. Aung, M. Zhang, and T. Choudhury. Personal sensing: Understanding mental health using ubiquitous sensors and machine learning. In UbiComp, 2015.Google Scholar
- V. Radu, N. D. Lane, S. Bhattacharya, C. Mascolo, M. K. Marina, and F. Kawsar. Towards multimodal deep learning for activity recognition on mobile devices. In UbiComp: Adjunct, 2016. Google ScholarDigital Library
- Y. Ren, Y. Chen, M. C. Chuah, and J. Yang. Smartphone based user verification leveraging gait recognition for mobile healthcare systems. In SECON, 2013.Google ScholarCross Ref
- O. Rippel, J. Snoek, and R. P. Adams. Spectral representations for convolutional neural networks. In NIPS, 2015. Google ScholarDigital Library
- M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Trans Sig. Process., 1997. Google ScholarDigital Library
- N. Srivastava and R. R. Salakhutdinov. Multimodal learning with deep boltzmann machines. In NIPS, 2012. Google ScholarDigital Library
- A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, and M. M. Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Sensys, 2015. Google ScholarDigital Library
- H. M. Thang, V. Q. Viet, N. D. Thuc, and D. Choi. Gait identification using accelerometer on mobile phone. In ICCAIS, 2012.Google ScholarCross Ref
- C. Wang, X. Guo, Y. Wang, Y. Chen, and B. Liu. Friend or foe?: Your wearable devices reveal your personal pin. In AsiaCCS, 2016. Google ScholarDigital Library
- C. Xu, S. Li, G. Liu, Y. Zhang, E. Miluzzo, Y.-F. Chen, J. Li, and B. Firner. Crowd++: unsupervised speaker count with smartphones. In UbiComp, 2013. Google ScholarDigital Library
- W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neural network regularization. arXiv:1409.2329, 2014.Google Scholar
- L. Zhang, B. Tiwana, Z. Qian, Z. Wang, R. P. Dick, Z. M. Mao, and L. Yang. Accurate online power estimation and automatic battery behavior based power model generation for smartphones. In CODES+ISSS, 2010. Google ScholarDigital Library
- Y. Zhao, S. Li, S. Hu, L. Su, S. Yao, H. Shao, and T. Abdelzaher. Greendrive: A smartphone-based intelligent speed adaptation system with real-time traffic signal prediction. In ICCPS, 2017. Google ScholarDigital Library
- Y. Zhu, Y. Zhu, B. Y. Zhao, and H. Zheng. Reusing 60ghz radios for mobile radar imaging. In MobiCom, 2015. Google ScholarDigital Library
Index Terms
- DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Recommendations
Multimodal Deep Learning for Activity and Context Recognition
Wearables and mobile devices see the world through the lens of half a dozen low-power sensors, such as, barometers, accelerometers, microphones and proximity detectors. But differences between sensors ranging from sampling rates, discrete and continuous ...
Can Deep Learning Revolutionize Mobile Sensing?
HotMobile '15: Proceedings of the 16th International Workshop on Mobile Computing Systems and ApplicationsSensor-equipped smartphones and wearables are transforming a variety of mobile apps ranging from health monitoring to digital assistants. However, reliably inferring user behavior and context from noisy and complex sensor data collected under mobile ...
DeepSense: A GPU-based Deep Convolutional Neural Network Framework on Commodity Mobile Devices
WearSys '16: Proceedings of the 2016 Workshop on Wearable Systems and ApplicationsRecently, a branch of machine learning algorithms called deep learning gained huge attention to boost up accuracy of a variety of sensing applications. However, execution of deep learning algorithm such as convolutional neural network on mobile ...
Comments