Maximilian Schrapel
Michael Rohs
ABSTRACT
Digital pens emit ink on paper and digitize handwriting. The range of the pen is typically limited to a special writing surface on which the pen's tip is tracked. We present Pentelligence, a pen for handwritten digit recognition that operates on regular paper and does not require a separate tracking device. It senses the pen tip's motions and sound emissions when stroking. Pen motions and writing sounds exhibit complementary properties. Combining both types of sensor data substantially improves the recognition rate. Hilbert envelopes of the writing sounds and mean-filtered motion data are fed to neural networks for majority voting. The results on a dataset of 9408 handwritten digits taken from 26 individuals show that motion+sound outperforms single-sensor approaches at an accuracy of 78.4% for 10 test users. Retraining the networks for a single writer on a dataset of 2120 samples increased the precision to 100% for single handwritten digits at an overall accuracy of 98.3%.
Supplemental Material
- Anoto AB. 2016. Anoto. Digital pen and paper technology. (2016). http://www.anoto.com.Google Scholar
- E. Alpaydin. 1992. Multiple neural networks and weighted voting. In Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems. 29--32.Google ScholarCross Ref
- Toshifumi Arai, Dietmar Aust, and Scott E. Hudson. 1997. PaperLink: A Technique for Hyperlinking from Real Paper to Electronic Content. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI '97). ACM, New York, NY, USA, 327--334. Google ScholarDigital Library
- Hiroki Asai and Hayato Yamana. 2013. Detecting Student Frustration Based on Handwriting Behavior. In Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13 Adjunct). ACM, New York, NY, USA, 77--78. Google ScholarDigital Library
- Won-Chul Bang, Wook Chang, Kyeong-Ho Kang, Eun-Seok Choi, Alexey Potanin, and Dong-Yoon Kim. 2003. Self-contained Spatial Input Device for Wearable Computers. In Proceedings of the 7th IEEE International Symposium on Wearable Computers (ISWC '03). IEEE Computer Society, Washington, DC, USA, 26--. http://dl.acm.org/citation.cfm?id=946249.946869 Google ScholarDigital Library
- Muzaffar Bashir and Jürgen Kempf. 2008. Reduced Dynamic Time Warping for Handwriting Recognition Based on Multi dimensional Time Series of a Novel Pen Device. 3 (01 2008).Google Scholar
- Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and Measuring the Benefits of Bimanual Pen and Direct-touch Interaction on Horizontal Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 154--161. Google ScholarDigital Library
- H. Bunke, T. Von Siebenthal, T. Yamasaki, and M. Schenkel. 1999. Online handwriting data acquisition using a video camera. In Document Analysis and Recognition, 1999. ICDAR '99. Proceedings of the Fifth International Conference on. 573--576. Google ScholarDigital Library
- François Chollet and others. 2015. Keras. https://github.com/fchollet/keras. (2015).Google Scholar
- Wacom Company. 2011. Wacom Inkling. (2011). http://inkling.wacom.eu.Google Scholar
- S. d. Choi, A. S. Lee, and S. y. Lee. 2006. On-Line Handwritten Character Recognition with 3D Accelerometer. In 2006 IEEE International Conference on Information Acquisition. 845--850.Google ScholarCross Ref
- Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 205--208. Google ScholarDigital Library
- Sungjae Hwang, Andrea Bianchi, and Kwangyun Wohn. 2012. MicPen: Pressure-sensitive Pen Interaction Using Microphone with Standard Touchscreen. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 1847--1852. Google ScholarDigital Library
- Koichi Kise, Megumi Chikano, Kazumasa Iwata, Masakazu Iwamura, Seiichi Uchida, and Shinichiro Omachi. 2010. Expansion of Queries and Databases for Improving the Retrieval Accuracy of Document Portions: An Application to a Camera-pen System. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems (DAS '10). ACM, New York, NY, USA, 309--316. Google ScholarDigital Library
- Narayanan C. Krishnan, Colin Juillard, Dirk Colbry, and Sethuraman Panchanathan. 2009. Recognition of Hand Movements Using Wearable Accelerometers. J. Ambient Intell. Smart Environ. 1, 2 (April 2009), 143--155. http://dl.acm.org/citation.cfm?id=1735835.1735841 Google ScholarDigital Library
- F. F. Li. 2004. Handwriting authentication by envelopes of sound signature. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Vol. 1. 401--404 Vol.1. Google ScholarDigital Library
- Wenzhe Li and Tracy Anne Hammond. 2011. Recognizing Text Through Sound Alone. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7--11, 2011. http://www.aaai.org/ocs/index.php/ AAAI/AAAI11/paper/view/3791 Google ScholarDigital Library
- Cheng-Lin Liu, Kazuki Nakashima, Hiroshi Sako, and Hiromichi Fujisawa. 2003. Handwritten digit recognition: Benchmarking of state-of-the-art techniques. 36 (10 2003), 2271--2285.Google ScholarCross Ref
- M. Miura, S. Kunifuji, and Y. Sakamoto. 2007. AirTransNote: An Instant Note Sharing and Reproducing System to Support Students Learning. In Seventh IEEE International Conference on Advanced Learning Technologies (ICALT 2007). 175--179.Google ScholarCross Ref
- Tohru MIYAGAWA, Yoshimichi YONEZAWA, Kazunori Itoh, and Masami Hashimoto. 2002. Handwritten Pattern Reproduction Using 3D Inertial Measurement of Handwriting Movement. 38 (01 2002), 1--8.Google Scholar
- Mario E. Munich and Pietro Perona. 2002. Visual Input for Pen-Based Computers. IEEE Trans. Pattern Anal. Mach. Intell. 24, 3 (March 2002), 313--328. Google ScholarDigital Library
- Shinji Nabeshima, Shinichirou Yamamoto, Kiyoshi Agusa, and Toshio Taguchi. 1995. MEMO-PEN: A New Input Device. In Conference Companion on Human Factors in Computing Systems (CHI '95). ACM, New York, NY, USA, 256--257. Google ScholarDigital Library
- A. Osborne. 1980. An Introduction to Microcomputers: Basic concepts. Osborne/McGraw-Hill. https://books.google.de/books?id=ScAjAAAAMAAJ Google ScholarDigital Library
- Stan Salvador and Philip Chan. 2007. Toward Accurate Dynamic Time Warping in Linear Time and Space. Intell. Data Anal. 11, 5 (Oct. 2007), 561--580. http://dl.acm.org/citation.cfm?id=1367985.1367993 Google ScholarDigital Library
- Andrew Seniuk and Dorothea Blostein. 2009. Pen Acoustic Emissions for Text and Gesture Recognition. In Proceedings of the 2009 10th International Conference on Document Analysis and Recognition (ICDAR '09). IEEE Computer Society, Washington, DC, USA, 872--876. Google ScholarDigital Library
- Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15 (2014), 1929--1958. http://jmlr.org/papers/v15/srivastava14a.html Google ScholarDigital Library
- Jürgen Steimle, Iryna Gurevych, and Max Mühlhäuser. 2007. Notetaking in University Courses and its Implications for eLearning Systems. In DeLFI 2007: 5. e-Learning Fachtagung Informatik, Christian Eibl, Johannes Magenheim, Sigrid Schubert, and Martin Wessner (Eds.). Gesellschaft für Informatik, Bonn, 45--56.Google Scholar
- Jelena Ljucović Adis Balota Zoran ?evarac Tijana Vujičić, Tripo Matijević. 2016. Comparative Analysis of Methods for Determining Number of Hidden Neurons in Artificial Neural Network. Proceeding of the Central European Conference on Information and Intelligent Systems 27 (2016), 219--223. http://archive.ceciis.foi.hr/app/index.php/ceciis/ index/pages/view/ProceedingsArchive2016Google Scholar
- Jeen-Shing Wang and Fang-Chen Chuang. 2012. An Accelerometer-Based Digital Pen With a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition. IEEE Trans. Industrial Electronics 59, 7 (2012), 2998--3007.Google ScholarCross Ref
- Daniel Wigdor, Clifton Forlines, Patrick Baudisch, John Barnwell, and Chia Shen. 2007. Lucid Touch: A See-through Mobile Device. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 269--278. Google ScholarDigital Library
- Myrna S. Zamarripa, Victor M. Gonzalez, and Jesus Favela. 2007. The Augmented Patient Chart: Seamless Integration of Physical and Digital Artifacts for Hospital Work. In Proceedings of the 4th International Conference on Universal Access in Human-computer Interaction: Applications and Services (UAHCI'07). Springer-Verlag, Berlin, Heidelberg, 1006--1015. http://dl.acm.org/citation.cfm?id=1757148.1757261 Google ScholarDigital Library
- Bin Zhang, Sargur N. Srihari, and Sangjik Lee. 2003. Individuality of Handwritten Characters. In Proceedings of the Seventh International Conference on Document Analysis and Recognition - Volume 2 (ICDAR '03). IEEE Computer Society, Washington, DC, USA, 1086--. http://dl.acm.org/citation.cfm?id=938980.939484 Google ScholarDigital Library
Index Terms
- Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition
Recommendations
Indic script family and its offline handwriting recognition for characters/digits and words: a comprehensive survey
AbstractHandwriting recognition has become an active area of research in pattern recognition and machine learning in recent years. Handwriting recognition systems have a variety of applications ranging from digital character conversion to signboard ...
A novel comprehensive database for offline Persian handwriting recognition
Developing a standard database for offline handwriting recognition is an essential task. This paper offers a novel comprehensive database for conducting research on offline Persian handwriting recognition. Seven pages of forms were designed and ...
Numeral characters and capital letters segmentation recognition in mixed handwriting context
ICDAR '95: Proceedings of the Third International Conference on Document Analysis and Recognition (Volume 2) - Volume 2For the analytic on-line recognition of handwriting, the range of pattern recognition problems can be described by the severity of letter segmentation required. More difficult problems require an interaction of letter segmentation and recognition. These ...
Comments