A New Deep Learning-Based Handwritten Character Recognition System on Mobile Computing Devices

Optical character recognition technology refers to the process of using electronic devices to scan printed characters, determine their shape by detecting edge information, and then translate the shapes into computer characters by character recognition [1]. OCR technology combines digital image processing, computer graphics and artificial intelligence and is one of the most active research topics in the field of pattern recognition. In China, there is an urgent need to use OCR technology for digital preservation of Shui characters. Shui characters are hieroglyphic, except for the Dong Ba character, which shape resembles that of an oracle bone and a gold inscription. Cultural inheritance of this language currently depends on oral communication and the handwriting of certain people; thus, most of the existing Shui characters are illegible and the books are unreadable. Therefore, by using advanced information processing methods, such as machine learning as well as big data collection and analysis, we can change the traditional document preservation methods and urgently establish meaningful digital preservation.

With the rapid development of mobile Internet services and the popularity of various intelligent devices, more and more users receive and transmit various information through mobile devices, which brings great convenience in terms of data collection, storage and use. Advances in mobile service computing and embedded devices have led to the development of the Internet of Things (IoT), which has increasingly linked physical content in everyday environments, dramatically changing the interaction between people and things. Especially in mobile phones, artificial intelligence technology represented by deep learning makes mobile phones capable of machine learning. There are many potential applications, such as object detection and recognition; speech-to-text translation; media information retrieval; and multimodal data analysis. Deep learning brings tremendous opportunities for mobile computing and can greatly improve the performance of various applications. In addition, due to the rapid spread of smart portable devices and the development of mobile service technology [2, 3], the possibility of introducing smart applications in mobile environments is receiving increased attention. Therefore, people are increasingly concerned about the possibility of applying deep neural networks (DNNs) in mobile environments [4]. Deep learning not only improves the performance of existing mobile multimedia applications, but also paves the way for more complex applications for mobile devices. Many of these devices (including smart watches, smartphones, and smart cameras) can perform some sensing and processing, making them smart objects that can learn. Despite the large potential for mobile DNNs, the design of neural network architectures for mobile environments is not well developed. From basic deep learning techniques to network architecture, training and reasoning algorithms, there are still many important issues that need to be addressed.

The most widely used neural network for deep learning work is the convolutional neural network, which converts unstructured image data into structured object tag data [5]. Generally, the working principle of a CNN is as follows: first, the convolution layer scans the input image to generate a feature vector; second, the activation layer determines which feature should activate the image under inference; third, the pooling layer reduces the size of the feature vector; and finally, a fully connected layer connects each potential tag to all outputs of the pooling layer. Although current deep learning technology has made a major breakthrough in the field of OCR, the computing and storage resources of mobile intelligent devices are limited, and the convolutional neural network model usually has hundreds of megabytes of parameters, which makes it difficult to implement in mobile devices. It is a challenge to realize the unification of the interconnection and interoperability between an object and the cloud, and to guide the IoT application system that supports horizontal interconnection, heterogeneous integration, resource sharing and dynamic maintenance [6, 7]. How to abstract the capabilities provided by the “object” and “cloud” resources into a unified system of software components according to the application requirements, and define the interaction topology between these components to establish the software architecture based on IoT is also a test [8, 9].

In this article, we focus on two main research efforts. The first research work is to design a neural network structure suitable for mobile devices to enable the recognition of Shui characters. The second study aimed to design a real-time character recognition system. The proposed system uses an edge computing service paradigm and distributes data analysis throughout the network. In particular, the proposed system will split the character recognition task between the edge device (physically close to the user) and the server (typically located in a remote cloud).

In this section, before introducing mobile computing, let us talk about the research status of the IoT and then briefly introduce the principle of convolutional neural networks and their application in OCR; finally, let us talk about the development status of deep learning on mobile devices.

The Internet of Things is an emerging concept that is likely to change our lives as much as the Internet has. In the Internet of Things, IoT devices with smart features or objects can communicate with each other. Each device will be given a unique identifier that allows the device to access the Internet and make decisions through calculations, thus creating a new world of interconnected devices. These IoT applications are sure to make a large difference in our lives. For example, in the medical industry, HIT (Health Internet of Things) sensors can be used to monitor important patient parameters (blood pressure, heart rate, etc.), to record data to better respond to emergencies, and to provide a better quality of life for people with disabilities [10]. Another area of research based on the Internet of Things is the IoUT (Underwater Internet of Things), whose goal is to monitor a wide range of inaccessible waters to explore and protect these areas [11]. The application of interest in this article is OCR.

Optical character recognition is one of the litmus tests of pattern recognition algorithms. In addition to pattern recognition, optical character recognition involves some other fields of knowledge, such as image processing, artificial intelligence and linguistics [12‐14]. Optical character recognition can make computers handle real world texts directly. The research results can be applied to a large number of practical problems, such as mail splitting, check recognition, image retrieval, intelligent robots and handwriting input. Commonly used methods in character recognition can be divided into three categories: template matching; feature extraction and classification; and deep learning methods. In the template matching method, a standard template set is first established for the character to be recognized, and then the preprocessed images of the character to be recognized are sequentially projected onto the templates for matching. The best matching template corresponds to the character being recognized. Feature extraction and recognition are the most commonly used methods in optical character recognition. The process consists of two parts. The first step uses an algorithm to extract the features of the character image. The commonly used feature extraction algorithms are: SIFT, SURF, and HOG. The second step is to use a classifier to classify the acquired features. Common classifiers include the Bayesian classifier, support vector machine, K nearest neighbor algorithm, artificial neural network algorithm, etc. [15, 16] Deep learning has become the most commonly used method since its appearance. The most commonly used deep learning model in the field of character recognition is the convolutional neural networks.

A CNN is a feedforward neural network inspired by biological processes. Unlike traditional network architectures, a CNN contains very special convolution and down sampling layers [17]. LeNet5 was born in 1994 and is one of the earliest convolutional neural networks, which has promoted the development of deep learning [18]. This network is mainly used for handwritten character recognition. The features of the image are distributed over the entire image. Convolution with learnable parameters is an effective way to extract similar features at multiple locations with a small number of parameters. Therefore, the ability to save the parameters and the calculation process is a key development. The two main features of a CNN are local connections and weight sharing. The so-called local connection means that the nodes of the convolution layer are only connected with some nodes of the previous layer and are only used to learn local features. This kind of local connection greatly reduces the number of parameters, speeds up the learning rate, and reduces the possibility of overfitting to some extent. In addition, the convolutional neurons are grouped into feature maps that share the same weight; thus, the entire process is equivalent to convolution, and the shared weight is the filter for each map. Weight sharing greatly reduces the number of network parameters, thereby increasing efficiency and preventing overfitting. The first few layers of the convolutional network alternate between a convolutional layer and a down sampling layer, followed by a number of fully connected layers, each followed by an activation layer, resulting in a classification result. By using gradient-descent based methods and backpropagation to minimize the loss function, a CNN is trained in a similar manner to other artificial neural networks.

Since the 1990s, CNNs have gained continuous research interest. During this time, Hinton and his colleagues conducted extensive and meaningful fundamental research on deep neural networks to improve algorithmic performance and optimize architecture [19‐22]. In 2012, Alex Krizhevsky proposed AlexNet, a deep convolutional neural network model, which can be regarded as a deeper and wider version of LeNet [23]. AlexNet includes several newer technical points. In the CNN, functions such as ReLU, Dropout, and LRN have been added. At the same time, GPUs have also been used to accelerate the calculations, gaining the first place in the ILSVRC 2012 competition with significant advantages. This network structure has also become a representative network structure of current convolutional neural networks. Zeiler and Fergus used random pools to tune the CNN. Simonyan and Zisserman proposed a very deep CNN to layers 16–19, achieving the greatest accuracy (ILSVRC) for ImageNet’s massive visual recognition challenge [22]. The width of the network also affects the ultimate effect of deep learning. Google has proposed a deep convolutional neural network structure called “Inception” that stacks 1 × 1, 3 × 3, 5 × 5 convolution layers and 3 × 3 pooling layers. On the one hand, this increases the network’s width; on the other hand, it also increases the adaptability of the network to the scale of a particular problem [24]. This network raises the level of technology for classifying and identifying the ILSVRC14 data set. He Kaiming et al. proposed a residual learning framework to reduce network training, so that each layer can learn an incremental transformation, which changes the internal network parameters and characteristics of the transmission method [25, 26]. The network is deeper than previously used networks, and training speeds have increased significantly. In March 2016, AlphaGo defeated Lee Sedol in five games. This is the first time that the Go program has defeated professional chess players [27]. It is an important milestone in artificial intelligence research.

In pattern recognition tasks, traditional machine learning tools often provide limited precision, and deep learning has been shown to be one of the most promising methods to overcome these limitations and achieve more powerful and reliable reasoning [28]. However, although deep learning technology has existed and has been successfully applied in many fields, only a few DL-based mobile applications have been produced, mainly due to the low-power computing resources provided by mobile devices [29, 30]. There are two main models for deploying DL applications on mobile devices: the client-server model and the client-only model. The former uses a mobile device as a sensor (without data preprocessing), or a smart sensor (with some data preprocessing), whose data are sent to a server or a simple cloud that runs the DL engine and sends the results back to the client. In this framework, maintaining good connectivity is the most important requirement for efficient operation of the entire system. The latter model runs DL applications directly on the mobile device. This solution can produce faster results without an operational network connection but must cope with the lack of device resources and therefore requires the right software and hardware solutions. In terms of mobile services, Honghao Gao, Wanqiu Huang and others have researched and innovated service patterns and workflows [31, 32]. In the development of deep learning, scientists have optimized model compression and neural network structure, especially for the CNN. The SqueezeNet [33] structure proposed in 2016 is not designed to achieve higher precision, but to simplify network complexity, reduce the size and calculation time of the model, and achieve performance similar to that of some typical large networks (such as Alexnet and VGG). In 2017, Google Inc. proposed a small network structure, MobileNet for embedded devices such as mobile phones, which can be applied to various tasks such as target detection, face attribute analysis and scene recognition [34]. Similarly, there is an efficient network ShuffleNet designed for mobile devices by Vision Technology [35].

Different input data and computing environments have different structural requirements for neural networks. In this paper, our work is to design a mobile computing framework and a convolutional neural network structure based on data characteristics and mobile computing requirements to achieve Shui character recognition.

In this section, we designed a mobile service computing framework, which is divided into three modules. And we detail the main content of Shui character recognition, which is divided into two parts: first, we describe the construction process of the data set used in the study; then, we build the CNN model and explain its details.

In this paper, we apply a convolutional neural network to handwritten character recognition. First, we construct a Shui character data set. Then during the training process of the convolutional neural network, we compared the results of different parameters so that we proposed the parameter tuning recommendations. The application of Shui character recognition shows that the CNN model we proposed can classify characters effectively and is more suitable for deployment on mobile devices.

Deep convolutional neural networks have made a great breakthrough in machine learning, but there are still some research challenges. The first is the determination of the number of CNN network layers and the number of neurons, which can only rely on many experiments. Second, efficient deep learning algorithms still rely on large-scale data sets. To improve the accuracy of Shui character recognition, we also need to provide many training samples. In addition, there are many parameters in CNN, and determining the optimal parameters is also a research problem.

Furthermore, the rapid development of deep learning technology has also been widely used in the fields of virtual reality, augmented reality and mobile devices. Today, when the mobile device fully enters the “AI era”, considering that CNN models are getting deeper, larger in size and more computationally intensive, it is especially important to design more efficient CNN models and compression techniques to reduce the resources (memory, storage, power, computation, band-width, etc.) required. At present, the deep learning model compression technology for mobile devices is mostly for a network structure. Therefore, from a software and hardware perspective, designing a complete deep learning model for mobile device deployment is still worth exploring.