Designing architectures of convolutional neural networks to solve practical problems

Highlights

• Our approach aims to support the estimation of Convolutional Neural Network (CNN) parameters.

• It intends to produce simpler CNN, reducing the complexity.

• This estimation was based on False Nearest Neighbors method.

• Caffe deep learning framework was used to conduct the training of CNN.

• Our results are comparable even to very complex and empirical CNN architectures.

Abstract

The Convolutional Neural Network (CNN) figures among the state-of-the-art Deep Learning (DL) algorithms due to its robustness to support data shift, scale variations, and its capability of extracting relevant information from large-scale input data. However, setting appropriate parameters to define CNN architectures is still a challenging issue, mainly to tackle real-world problems. A typical approach consists in empirically assessing different CNN settings in order to select the most appropriate one. This procedure has clear limitations, including the choice of suitable predefined configurations as well as the high computational cost involved in evaluating each of them. This work presents a novel methodology to tackle the previously mentioned issues, providing mechanisms to estimate effective CNN configurations, including the size of convolutional masks (convolutional kernels) and the number of convolutional units (CNN neurons) per layer. Based on the False Nearest Neighbors (FNN), a well-known tool from the area of Dynamical Systems, the proposed method helps estimating CNN architectures that are less complex and produce good results. Our experiments confirm that architectures estimated through the proposed approach are as effective as the complex ones defined by empirical and computationally intensive strategies.

Introduction

The vast amount of data currently available has fostered the development of methodologies capable of processing and extracting meaningful features to assist the interpretation, understanding, and solution of complex problems. In this context, the area of Deep Learning (DL) has emerged as a main alternative to analyze massive data, presenting breakthrough results in tasks such as speech recognition (Graves, Mohamed, & Hinton, 2013), machine translation (Luong, Sutskever, Le, Vinyals, & Zaremba, 2014), and data classification (Lauer, Suen, Bloch, 2007, LeCun, Bottou, Bengio, Haffner, 1998, Sharif Razavian, Azizpour, Sullivan, Carlsson, 2014, Zhou, Lapedriza, Xiao, Torralba, Oliva, 2014).

Deep Learning algorithms operate in multiple levels, each of which composed of a set of regression models that involve linear and nonlinear components. The combination of multiple models makes possible the representation of complex functions (LeCun, Bengio, & Hinton, 2015). Most DL algorithms resemble Artificial Neural Networks, such as the Multilayer Perceptron (Haykin & Network, 2004), where input vectors are processed throughout consecutive layers containing operation units to emphasize or inhibit features (LeCun, Bottou, Bengio, Haffner, 1998, Sharif Razavian, Azizpour, Sullivan, Carlsson, 2014).

A particularly important DL algorithm is the so-called Convolutional Neural Network (CNN), which has gained prestige mainly due to its good performance in computer vision tasks (Oquab, Bottou, Laptev, Sivic, 2015, Scherer, Müller, Behnke, 2010a, Scherer, Schulz, Behnke, 2010b, Sharif Razavian, Azizpour, Sullivan, Carlsson, 2014, Zhou, Lapedriza, Xiao, Torralba, Oliva, 2014), and its feature extraction ability from time-dependent data such as audio and video (Karpathy, Toderici, Shetty, Leung, Sukthankar, Fei-Fei, 2014, Osadchy, Cun, Miller, 2007, Schluter, Bock, 2014). However, the performance of CNN strongly depends on its architecture, including the number of layers, units per layer, and convolutional mask sizes.

Moreover, an architecture that works well for a given problem may not be appropriate when dealing with different data types or tasks. An usual alternative to remedy this issue is to evaluate different CNN architectures, choosing the one with best performance. Such procedure clearly bears a number of drawbacks: (i) the training and evaluation of a single architecture is already computationally intensive, thus the overall assessment of several of those may be unfeasible to many scenarios; and (ii) the empirically defined architectures may not be appropriate for the problem under consideration (Lappas, Chen, 2009, Menotti, Chiachia, Falcao, Oliveira Neto, 2014), thus acceptable results may never be reached.

A strategy commonly employed to avoid the assessment of multiple CNN settings considers an additional training stage to tune weights associated with convolutional units. This also increases the computational burden and requires thousands/millions of examples to produce reasonable results (Lauer, Suen, Bloch, 2007, LeCun, Bottou, Bengio, Haffner, 1998, Simard, Steinkraus, Platt, 2003). Another strategy employed is the ensemble of CNNs, which usually allows a reduction in the maximum number of iterations at the cost of training more architectures to obtain relevant results. It is also time consuming though, threatening its application into production environments (Ciregan, Meier, & Schmidhuber, 2012).

In this work, we present a novel methodology to assist the definition of CNN architectures that differs substantially from the described alternatives. Specifically, our approach analyzes the input and output images produced by convolutional operations at each CNN layer in order to estimate the adequate dimensions for the convolutional masks, and to suitable set the number of convolutional units per layer. In addition, motivated by the Occam’s razor problem-solving principle (Blumer, Ehrenfeucht, Haussler, & Warmuth, 1987), we also aim to design as simple as possible, but yet efficient, architectures to address target problems.

The faster convergence of simpler CNN architectures is here demonstrated using the Statistical Learning Theory, most specifically as a result of the Chernoff bound and the Hoeffiding inequality (Devroye, Györfi, Lugosi, 2013, Vapnik, 2013, Von Luxburg, & Schölkopf, 2011). Eq. (1) presents the condition that ensures learning bounds, in which $\mathcal{N}(\mathcal{F},2n)$ corresponds to the Shattering coefficient, $\mathcal{F}$ is the set of all possible functions provided by an algorithm (a.k.a. algorithm bias), and n is the sample size (here two samples with n elements are provided). The Shattering coefficient is indeed a function of n which defines the maximum number of admissible functions contained in $\mathcal{F}$ that produce distinct classifications considering the worst possible sample organization with size n (Smola & Schölkopf, 1998).

$${\displaystyle \frac{\log \mathcal{N}(\mathcal{F},2n)}{n}}\to 0$$

As already discussed in Von Luxburg and Schölkopf (20011), the Shattering coefficient can be approximated to an² when considering just one neuron of some deep network (see Eq. (2)), for some constant a > 0. Thus, Eq. (3) provides the Shattering coefficient for a given deep architecture composed of k units in total.

$${\displaystyle \frac{\log \mathcal{N}(\mathcal{F},2n)}{n}}={\displaystyle \frac{\log a{n}^2}{n}}$$

$${\displaystyle \frac{\log \mathcal{N}(\mathcal{F},2n)}{n}}={\displaystyle \frac{\log {\left(a{n}^2\right)}^k}{n}}={\displaystyle \frac{k\log a{n}^2}{n}}$$

The sample size required to ensure convergence is defined in Eq. (4), in which R(f) represents the real risk (the expected value for the loss function, in range [0, 1]), $R_{\text{emp}}$ corresponds to the empirical risk (the average loss computed on a given sample, in range [0, 1]), 0 < ϵ < 1 is a threshold that indicates an acceptable divergence limit for risks, $\mathcal{F}$ is the set of all functions provided by some supervised learning algorithm, n is the sample size. Thus, the convergence of some algorithm for an given network architecture is here analyzed in terms of its number of units and sample size. Eq. (5) defines the generalization probability, in which $P\left({\sup }_{f\in \mathcal{F}}\right|R\left(f\right)-R_{\text{emp}}|>ϵ)$ represents the probability of an algorithm does not generalize. So, the main goal of the Statistical Learning Theory, as proved by Vapnik (2013), is to make sure term $2{\left(a{n}^2\right)}^k{e}^{-nϵ/4}$ converges to zero, as a greater sample size is provided to guarantee generalization.

$$P(\underset{f\in \mathcal{F}}{\sup }|R\left(f\right)-R_{\text{emp}}|>ϵ)\le 2\mathcal{N}(\mathcal{F},2n)e^{-nϵ/4}$$

$$P(\underset{f\in \mathcal{F}}{\sup }|R\left(f\right)-R_{\text{emp}}|>ϵ)\le 2{\left(a{n}^2\right)}^k{e}^{-nϵ/4}$$

From Eq. (5), it is possible to conclude that additional CNN units directly require greater sample sizes in order to ensure the right-side term of such inequality approaches zero. Fig. 1 illustrates the generalization probability produced according to Eq. 5, in which the sample size n varies from 1 to 1 million, the number of network units k is set as {10, 50, 100, 250}, $a={10}^{-10}$ and $ϵ=0.05$ (5% percent of divergence is accepted between the expected and the empirical risks). Convergences initiate when curves start decaying, but they only occur when an enough sample size is provided, so that it tends to zero.

The Shattering coefficient is the main responsible for holding back the convergence and make it require more data examples, as shown in Fig. 1. The greater the complexity of such Shattering coefficient, the longer it behaves as an exponential function, only after a great enough sample size it is dominated by the other Chernoff terms and, finally, decreases and converges. In addition, the theoretical convergence could not be illustrated for any network with 250 or more neurons: (i) first of all, term a should be smaller to make the Chernoff bound approximate zero; (ii) secondly, it tended to infinity as more data examples were provided; and finally, (iii) not even using 1 million examples it could converge in theory. We believe this theoretical demonstration is necessary and enough to confirm the need for designing simpler (less neurons and layers) architectures.

Based on this theoretical formulation after Vapnik (2013), it is obvious that large CNN networks require greater samples to guarantee the learning convergence and, finally, prove a good classifier was obtained. Thus, the goal of this paper is to estimate adequate parameters in order to design simpler and yet efficient CNN architectures, producing networks that converge faster with a reasonable number of training examples. In summary, we propose a method to estimate the adequate dimensions for the convolutional masks (convolutional kernels), and the number of convolutional units (CNN neurons) at each layer of a CNN architecture for general purpose classification tasks.

Motivated by the False Nearest Neighbors (FNN) (Kennel, Brown, & Abarbanel, 1992), a well-known tool from the area of Dynamical Systems, our method analyzes the input and output images produced by the convolutional operation of each CNN layer in order to estimate the adequate dimensions for the convolutional masks, and the suitable number of convolutional units per layer. In more details, this analysis takes each image and builds up vectors to embed data into high-dimensional spaces in attempt to increase the recurrence levels. Recurrences are here associated with the prevalent and most similar patterns happening in a given input image.

The CNN architectures produced by our approach were compared against more complex ones, using the Caffe deep learning framework (Jia et al., 2014). Four datasets were used: i) CMU Face Images (Roweis & Saul, 2000), that contains images of human faces; ii) MNIST, that is a dataset of handwritten digits; iii) Columbia University Image Library, which is referred as COIL-100 (Nene, Nayar, & Murase, 1996), that contains object images divided in 100 classes; and iv) German Traffic Sign Recognition Benchmark, which is referred as GTSRB (Stallkamp, Schlipsing, Salmen, & Igel, 2012), that is a dataset of traffic sign images.

Experimental results confirm our CNN architectures have error rates similar with the ones listed in the literature, but presenting a significant lower complexity (less layers and units per layer). Therefore, our methodology turns out to be a viable alternative to design simpler CNN architectures that are faster to be trained and less prone to produce overfitting (Cogswell, Ahmed, Girshick, Zitnick, & Batra, Lappas, Chen, 2009).

This paper is organized as follows: Section 2 discusses about Deep Learning approaches and their results, with particular attention to the datasets used in our experiments; Section 3 uses Linear Algebra to formulate CNN operations; Section 4 addresses the original False Nearest Neighbors (FNN) method, proposed by Kennel et al. (1992); in Section 5, we describe all necessary FNN modifications in order to make possible the estimation of CNN architectures; Section 6 shows experimental results performed on the four selected datasets in order to evaluate the mask sizes and the number of convolutional units for a CNN with one, two, and three convolutional layers; Section 7 presents the concluding remarks and perspectives for future work.