A new hybrid approach for grapevine leaves recognition based on ESRGAN data augmentation and GASVM feature selection

Plants are one of the most important basic food sources for the survival of other living things. It is estimated that there are approximately 400,000 plant species in nature [1]. In addition to being one of the most important food sources, it is also used in medicine and industry. Recognition of plant species is important not only for consumption, but also for the conservation of species in nature. Therefore, experts are needed in the determination of plant species. Botanists often make the distinction by observing the leaf characteristics of plants. Traditional methods like this are a very time-consuming, extensive, and complex process.

The grapevine leaves discussed in this article are a very useful type of leaf that is widely used in traditional Turkish and Mediterranean food cultures. Grapevine leaves are rich in low-calorie, high amounts of dietary fiber, calcium, phylloquinone (K1), phenolic compounds, phosphorus, and vitamin C. In addition, grapevine leaves can be stored frozen, canned, and in brine and therefore have a long shelf life. Also, some types of grapevine leaves provide a better economic return than grapes. It is reported that approximately 13.5 million dollars of export revenue is obtained from Turkey’s grapevine leaf exports, and 135 million dollars of export revenue is obtained from wrapping and stuffing produced from leaves [2]. The leaves have different properties in terms of thickness, thinness, hairiness, and shape. Not all types of grapevine leaves are preferred as food. Those that will be used for food, i.e., consumption, should be as fine-grained, hairless, and thin as possible. In addition, varieties without slices and with a sour taste on the palate are preferred. For this reason, it is an important need to recognize the grapevine leaf species to be used for consumption. In particular, to export a large number of grapevine leaves, it is necessary to determine the grapevine leaf types with fast, error-free, and cost-effective solutions. The most suitable of these solutions is to detect grapevine leaves autonomously with industrial machines. In this study, an approach with a very high accuracy rate is presented that will perform the recognition of grapevine leaves autonomously.

There are many studies on the classification of plant species with artificial intelligence applications developed in the field of agriculture [3‐7]. First, classification studies were carried out with datasets such as Flavia [8], Swedish [9], ICL [10], and Foliage [11], which include plant species in the literature. In the past, many researchers have evaluated the performance of these manually extracted features based on shape, texture, and color features using machine learning methods. In feature-based studies, Histogram of Oriented Gradient (HOG) [12, 13], Centroid-Center Distance [14‐16], Fourier Descriptor [9, 17, 18], geometric or morphological features [19‐21], Gray-Level Co-occurrence Matrix (GLCM) [11, 22, 23], and local binary pattern [24] were used. In recent years, with the development of deep learning instead of machine learning techniques, low-performance problems of complex images in the field of computer vision have been solved. With the rapid progress of deep learning techniques, many studies have been carried out in the literature to automatically extract leaf features for plant species recognition [25]. Compared to traditional feature extraction models, deep learning techniques are completely data-driven and generally have higher feature possibilities for capturing distinctive features. The deeper and wider the mesh of the architecture, the more image information is obtained. In deep learning, architectures like GoogleNET [26], VGG [27], ResNet [28], DenseNet [29], MobileNetV2 [30] and Inception V4 [31] and VGG19 [27] have gotten a lot of attention. In addition, there are studies on diseases of plant leaves with deep learning methods. Ma et al. [32] carried out a study on the detection of diseases in cucumber leaves. In the deep architecture they proposed based on VGG, 4 diseases were classified in plant leaves. Dhivyaa et al. [33] proposed a method for the detection of plant diseases that includes dense blocks, an expanded complexity network, and a bidirectional long short-term memory. Bhujel et al. [34] detected tomato leaf diseases using a light convolutional neural network model they developed. Prabu and Chelliah [35] proposed a MobileNetv2 and SVM-based method to detect mango leaf diseases. Singh et al. [36] used the AlexNet network model to detect diseases in corn leaves. Li et al. [37] developed a lightweight convolutional neural network RegNet model for apple leaf diseases. Five different apple leaf diseases were identified in this study. Anari [38] proposed a CNN-based ResNet 18 model to detect six different plant diseases, namely Apple, Corn, Cotton, Grape, Pepper, and Rice. For deep learning-based plant classification, architectures such as DeepPlant [39], Multi-Scale Convolution Neural Network (MSF-CNN) [25], ResNet-50 [40], and BLeafNet [1] have been proposed. Pawara et al. [41] proposed a new plant leaf classification method based on Inception-v3 and ResNet-50 network models. On the other hand, there are very few studies in the literature on recognizing grapevine leaf species. Carneiro et al. proposed an approach based on the transfer learning Xception Convolution Neural Network (CNN) model for the automatic classification of 12 grape varieties found in the Douro Demarked region of Portugal. They achieved an accuracy of 93% with this approach [42]. Another study, Koklu et al. by using a dataset created from grape leaf varieties collected from the Central Anatolian Region of Turkey. They used the pre-trained MobileNetV2 CNN model for feature extraction. In addition, they achieved 97.6% accuracy by using the Chi-square algorithm to select the most important features and the SVM algorithm to classify the selected features [43]. However, the performance successes obtained from the methods suggested in these studies are insufficient for recognizing the grape leaf. Therefore, in this study, we propose a new hybrid method with very high performance.

In this study, a proposed new hybrid approach for grapevine leaf recognition consists of 5 basic steps: data augmentation, training of CNN, feature extraction, feature selection, and classification. In the data augmentation step, augmentation was performed using the Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) model alongside traditional data augmentation techniques to preserve the detailed texture feature of the data. Then, CNN models such as MobileNetV2 and VGG19, which are known for their success in object recognition and pre-trained with ImageNet, were used to extract features from the data. In addition to these network models, the Top block-1 structure has been developed to provide the best learning, and these layers have been integrated into the top layer of the CNN models and training has been carried out. In the feature extraction step, the features were extracted from the dense (160) layer of the Top block-1 structure and combined (fusion). Then, feature selection was made from these combined features with the support vector classifier (SVC) method. However, when these selected features were classified with SVM, the classification success remained at 98.86%. In this study, the Genetic Algorithm-Based Support Vector Machine (GASVM) algorithm was developed and applied to select the best feature subset. As a result, 314 important features were selected from 640 features with GASVM. In the last stage, classification was made using the SVM algorithm, whose success is known. As a result of the experimental tests, 99.57% accuracy was obtained with this proposed approach.

In summary, the main contributions of this study are:

The remaining parts of this article are organized as follows; in Chapter 2, there is information about the dataset, data augmentation methods, block structures used by CNN models, and feature subset selection method. The approach proposed in Chapter 3 and the experimental results in Chapter 4 are evaluated and analyzed. Finally, Chapter 5 contains general conclusions.

The proposed approach consists of five basic steps. In the first step of the model shown in Fig. 5, the dimension of the original dataset consisting of 500 images was set to 224 × 224. Image augmentation techniques are then applied to the original dataset to ensure that the CNN models are best learned. The new dataset created from 3500 images is called augmentation data. Then, 80% of the augmentation data is separated as the train (2800 images) and 20% as the test (700 images) data. Then, new synthetic data are derived from this train and test data separately with the ESRGAN method. In this way, the reason why train and test data are derived separately, and the train and test data are prevented from mixing with each other. While 2800 train and 700 test data were generated in the new data derivation, this dataset was called super-resolution (SR) data. The reason for using the ESRGAN method in the derivation of new data is to take advantage of its ability to produce more natural and detailed tissue information than other super-resolution methods. In the second step, transfer learning was performed with MobileNetV2 and VGG19 CNN models to learn classes from augmentation and SR data. The reason for choosing these CNN models is because of their successful performance on small datasets. Also, the minimum requirement top block structure used with transfer learning CNN models is top block-2 in Fig. 4. However, in the experimental tests made with this structure, it was determined that learning was insufficient. For this reason, the top block-1 structure, which can be used with these CNN models, has been developed to make learning more effective with the datasets used. In this step, top block-1 structured CNN models are trained with augmentation and SR data. In the third step, 160 features are extracted from the dense (160) layer in the top block-1 of the trained CNN models. Features extracted from each model are combined (fusion), resulting in 640 features. In the fourth step, important features are selected by using the genetic algorithm and support vector classifier together. In the fifth step, classification is done using the SVM method. The schematic view of the proposed approach is given in Fig. 5.

The hyper-parameters used in these CNN models are given in Table 1. In addition, the categorical cross-entropy loss function is used in these CNN models. As the optimization method, Adam, SGD, and RMSprop were tried and the SGD algorithm, which gave the best results, was selected. Momentum and learning rate values were tested iteratively and the values that gave the best results were selected.

In this study, performance metrics such as accuracy, precision, recall, F1-score, specificity, sensitivity, and Matthews correlation coefficient obtained from the complexity matrix were used to measure the performance of the proposed approach. For the calculation of these metrics, 4 basic parameters are needed, namely true positive (TP), true negative (TN), false positive (FP), and false negative (FN) [43, 56, 57]. These metrics are formulated in Eqs. (2), (3), (4), (5), (6), (7), and (8).

Experimental studies were compiled using software technologies such as Python 3.7.13, TensorFlow 2.8.0, Keras 2.8.0, and hardware technologies such as Intel(R) Xeon(R) CPU @ 2.30 GHz, 27.6 GB RAM, Tesla T4—16 GB.

In this study, experimental research is basically divided into two stages according to feature extraction or not. In this way, the reason why the experimental study is in two stages is to provide an understanding of how feature extraction or non-extraction will contribute to the proposed approach. In the first stage, experimental results were obtained by using MobileNetV2 (MNV2) and VGG19 CNN models without extracting attributes with augmentation and super-resolution (SR) data and by classifying grapevine leaf according to the use of top block-1 or top block-2 structures in these CNN models. In this way, it is aimed that each component used in the proposed approach, such as augmentation and SR data, MobileNetV2 and VGG19 CNN models, and top block-1 and top block-2 structures, can be evaluated separately. In the second stage, MobileNetV2 (MNV2) and VGG19 CNN models trained with augmentation and super-resolution (SR) data were used as feature extractors. The top block-1 structure provides the output of these CNN models used in the feature extraction task. The top block-1 structure aims to keep the transfer and data learning levels of these CNN models at an optimum level. In addition, feature extraction was performed with 160 features from the dense layer of the top block-1 structure. Different feature numbers such as 320, 160, and 80 were tried iteratively and the most successful results were obtained with 160 features. At this stage, the extracted features were firstly combined and 640 features were obtained. Then, ineffective and unimportant features were discarded from 640 features, and the most important features were selected by the support vector classifier (SVC) method. At this point, after the feature selection was made with the SVC method, grapevine leaf classification was made with the SVM method and the classification measurements were recorded. Then, the GASVM algorithm was developed and used to further increase the success achieved by the SVC method. In the feature selection made with the GASVM method, 314 features were selected. Finally, to measure the final classification success of the proposed approach, 314 features were classified and the results were recorded.

On the other hand, by applying data augmentation methods to the original dataset, a total of 3500 images, 500 images from each class, were derived and this derived data was called augmentation data. Then, 80% of the augmentation dataset is reserved for train and 20% for test data to be used in the training and testing of CNN methods.

In addition, this train and test dataset was passed through the ESRGAN method separately to obtain a new synthetic dataset, which is called super-resolution data (SR data). In particular, since the train and test datasets are given separately to ESRGAN, the newly derived dataset is 80% train and 20% test data. The reason for using ESRGAN in this study is to benefit from its ability to produce more natural and detailed texture information than other super-resolution GANs. The accuracy and loss graphs obtained when training the MobileNetV2 (MNV2) and VGG19 CNN models with top block-1 with augmentation and SR data are given in Fig. 6. Figure 6a and b indicates the graph of training with augmentation data, while Fig. 6c and d indicates the graph of training with SR data.

As shown in Fig. 6, the accuracy of the MobileNetV2 and VGG19 CNN models is 100% in the training made with augmentation and SR datasets, while their losses are close to zero. However, as shown in Fig. 6a, the accuracy graph of MobileNetV2 reaches up to 97.5% with the augmentation dataset, while the accuracy graph of VGG19 reaches up to 96.5%. As shown in Fig. 6b, the loss graph of MobileNetV2 reaches up to 0.075, while the loss graph of VGG19 reaches up to 0.099. In addition, when these CNN models are trained with the SR dataset, the accuracy graph of MobileNetV2 reaches 94%, while the accuracy graph of VGG19 reaches 96%, as shown in Fig. 6c. As shown in Fig. 6d, the loss graph of MobileNetV2 reaches up to 0.184, while the loss graph of VGG19 reaches up to 0.196.

The top block structures were used in training and classification tests with MobileNetV2 and VGG19 CNN models of augmentation and SR data. The standard block structure used for training and testing CNN models in studies with transfer learning is top block-2. Since the number of learnable parameters in CNN models with this block structure was insufficient, it was necessary to develop the top block-1 structure. The comparison of these two top block structures according to the dataset and CNN models is given in Table 2. Macro-average measurement results are given in precision, recall, and F1-score metrics in this table. As seen in this table, the highest accuracy rate of 97.42% was achieved with the triple combination of Top block-1, augmentation data, and MobileNetV2. When the top block-1 and top block-2 structures are compared according to the same dataset and CNN model, the accuracy has increased by about 10%, especially with the binary combination of SR data and VGG19. When the binary combinations of the other dataset and CNN models are examined, it is observed that there is a remarkable increase in the accuracy metric of the top block-1 structure compared to the top block-2 structure.

In each CNN model, F1, F2, F3, and F4 features obtained from the dense layer of the top block-1 structure were combined to obtain 640 features. These fused features were firstly subjected to the feature selection process with the SVC algorithm and then classified by the SVM method, and the measurement results were obtained. Then, the final results were obtained by using the GASVM algorithm developed from GA and SVC algorithms to select the combined features and the SVM method for classification. The contribution of GA to the proposed approach can be seen in the tests performed in this way. In Table 3, the classification results of the combined features are given according to the feature selection method. Accordingly, the contribution made to the accuracy metric by using GA in the proposed approach is approximately 0.7%.

One of the tables that are often used to evaluate the performance of a classification model on a set of test data with known actual values is the confusion matrix. The confusion matrix table in Fig. 7a is given to evaluate the classification performance of the proposed approach. Accordingly, two and one images were predicted incorrectly for the Dimnit and Nazli classes, respectively. This means that 3 images out of 700 images are incorrect and approximately 0.43% of incorrect predictions are made. Another performance evaluation method is the receiver operating characteristics (ROC) curve in Fig. 7b. According to this curve, the distinction can be made very successfully in all classes. In addition, the class-based classification performance measurement results of the proposed approach are presented in Table 4. Accordingly, a very high classification success was achieved with an accuracy rate of 99.57%.

As shown in Table 5, only Koklu et al. [43] have a study with the same dataset. The comparison of the current method using the same dataset, which includes a total of 5 classes, including Ak, Ala İdris, Büzgülü, Dimnit, and Nazlı, and 500 images, and the proposed approach is presented in this table. As shown in Table 5, while Koklu et al. obtained a 97.6% accuracy rate with 250 features, a 99.57% success rate was obtained with 314 features in the proposed study. This success rate is approximately 2% higher than the success of Koklu et al. In addition, the proposed approach was more successful in performance metrics such as specificity, precision, sensitivity, F1-score, and Matthews correlation coefficient compared to Koklu et al.’s method.

The results obtained when the test images of the grapevine leaf dataset were classified with the proposed approach are given in Fig. 8 visually and as a class. Accordingly, the class name extending along the y-axis to each image represents the actual (true) class, while the class name extending along the x-axis represents the classes predicted by the proposed approach. As can be seen, the leaf image belonging to the 2 Dimnite class was incorrectly predicted as Ak, while the leaf image belonging to the Nazli class was incorrectly predicted as Ala İdris. These incorrect predictions are considered to have occurred because the proposed approach failed to effectively extract the distinctive features of these three leaf images.

As a result, in this study, a new hybrid approach is proposed to increase the classification accuracy of grapevine leaves. In this approach, feature extraction is done with convolutional neural networks MobileNetV2 and VGG19 to provide automatic recognition. When grapevine leaves were classified using feature extraction methods and the top block-2 structure, the highest accuracy score obtained from the combination of augmentation data and MobileNetV2 was 96.71%. However, since this accuracy score was also insufficient, a new top block structure was designed. This top block structure was named top block-1, and the accuracy score of the combination of augmentation data and MobileNetv2 with this block structure was increased by 0.72% and reached 97.43%. Since this accuracy score is also insufficient for perfect leaf recognition, insignificant features should be eliminated. Therefore, with the GASVM method, which we adapted to this problem, we selected 314 of the most important features. Then, the selected features were classified using the SVM method and the accuracy score was increased by 2.14%, achieving a perfect accuracy of 99.57%. With this proposed approach, 697 images out of 700 test images were classified correctly, while only 3 images were misclassified. It is thought that the reason why the proposed approach misclassifies the 3 images is due to the insufficient number of leaf species in the original dataset. Even if data augmentation techniques are applied, feature extraction cannot be performed perfectly if the leaf types in the original dataset do not have a wide variety of sizes, textures, and maturity.

In this study, we propose a new hybrid approach for the automatic recognition of grapevine leaf species. In the proposed approach, data augmentation techniques are applied to provide more effective learning from the dataset. In addition, it was reconstructed with the ESRGAN model to obtain more detailed textures and high-resolution synthetic images from the newly created dataset. In our hypothesis, we designed a deep feature extractor with new detailed information by adding the top block-1 structure to the pre-trained MobileNetV2 and VGG19 CNN architectures. According to the results, the proposed block structure contributed to the increase in performance. The features taken from the datasets trained with CNN models are then combined in the following stage. The GASVM feature selection method has been developed and applied for the selection of the most efficient deep features. To classify the features combined in the previous stage, SVM was employed. 99.57% accuracy, 99.89% specificity, 99.56% precision, 99.57% sensitivity, 99.56% F1-score, and 99.46% MCC were achieved using the designed hybrid approach model. In addition, the experimental results were compared with other studies using the relevant dataset. It has been shown that the proposed approach is approximately 2% more successful in terms of accuracy, precision, sensitivity, F1-score, and MCC metrics compared to Koklu et al., which is the only study conducted with the grapevine leaf dataset. The observed findings in the results showed both the significance of the deep features obtained by the model and its success in grapevine leaf classification.

In the future, it is aimed to further improve the proposed approach by training and testing large datasets consisting of different plant species and producing cost-effective solutions that can work on end devices.