Custom CornerNet: a drone-based improved deep learning technique for large-scale multiclass pest localization and classification

The agriculture sector is the most significant part of the economy, and the majority population of the entire world depends on it. Meanwhile, the recent growth in the world’s population rate necessitates an increase in crop quantity to meet the food requirements of people all over the world [1]. However, several challenging factors like the climate conditions and crop pests hinder the farmers in taking care of crops and improving the yield. Conventional crop pest identification is performed by manually arranging pest traps to analyze and assess the category of pests in crops. However, such methods are unreliable and often suffer from high error rates.

Furthermore, we often experience delays in the pest detection process due to the limited availability of agronomists. Moreover, because of the lack of technical information about the various pest types and high visual similarity between different insect species, it is difficult to accurately specify the related insecticide types, resulting in an extensive and blind application of pesticides [2]. The delay in pest recognition can prevent the farmers from taking timely measurements, which causes massive damage to both the quality and quantity of crops. However, identifying the type and amount of crop pests is a tedious and time-consuming activity. At the same time, early recognition and usage of timely spraying pesticides on the plants can improve the yield and enhance the economy. Recent progress in machine learning (ML) and computer vision (CV) has urged the researcher present computer-aided approaches to simplify this time-consuming task and build effective automated insect identification systems.

Initially, conventional ML-based approaches used local descriptors like local binary pattern (LBP) [3], regional ternary pattern (LTP) [4], SIFT [5], SURF [6], etc., with classifiers such as SVM [7], K-means [8], etc. Such approaches are heavily explored for pest detection and classification [9‐14]. Although hand-crafted keypoint computation techniques are easier to use and require less training data, they are slow and require the skill of experienced human specialists. Moreover, the recent progression in image acquisition has introduced such challenging datasets. These conventional approaches employ ML as ready-to-use solutions are not promising for real-world pest detection, and they face a severe reduction in classification performance. This might be because of several reasons, such as ineffective hand-crafted keypoints computation.

Moreover, the exact pest can appear with different motions and positions in various images, which causes to generate the varying keypoints vectors for the same insects. Primarily, the research community is working to improve the detection performance for specific categories of insects by proposing new keypoints solutions. Such work lacks focus on introducing novel frameworks for multi-category pest recognition missions that need to acquire insect localization and classification information to assist in pest monitoring [15, 16].

Recently, we observed the robustness of Deep learning (DL)-based frameworks, i.e., convolution neural networks (CNNs) [17], recurrent neural networks (RNNs) [18], deep belief networks [19] in a variety of areas covering the agriculture sector as well. DL is a robust approach for image analysis and object recognition with superior effectiveness in classifying various categories of pests [20]. Transfer learning is an essential task in DL, in which pre-trained frameworks are modified to perform a new job. Deep transfer learning (DTL) proposes a new network for processing digital samples and performing predictive analytics, with better generalization power and the ability for pest recognition. DL-based approaches employ CNNs that can automatically detect the discriminative keypoints from input data without the assistance of human specialists. Because of the considerable evolution of hardware, DL frameworks are extensively being used to handle complicated problems in a reasonable amount of time. In agriculture, DL-based algorithms have shown to be entirely accurate and have been effectively adapted to perform various tasks [21].

As progress in DL methods [22, 23] has exhibited promising results in the area of object identification, so, the extensive research work concerns proposing more complicated object localization frameworks for better detection accuracy, i.e., Super-FAN [22] and unsupervised multi-stage keypoints learning [23], etc. Moreover, several CNN-based approaches namely GoogLeNet [24], AlexNet (AN) [25], VGG [26], ResNet (RN) [27], R-CNN [28], Fast R-CNN [29], Faster R-CNN [30], and YOLO [31] are also evaluated for pest detection and classification. Even though the aforementioned DL-based object identification frameworks have demonstrated robust performance in general object identification systems, their applications for pest detection are still limited. Pest recognition has its characteristics and differs from existing object identification and classification tasks [21]. The insect pests are small target and they are usually surrounded by complex environment in the real-field images; thus, the identification network can be easily deceived by the background while computing keypoints. In addition, because of the varying angles and distance at which they are captured in the field, there is a considerable difference in pest size and postures which makes accurate recognition more challenging. Moreover, distinct insect pest species often have a high degree of resemblance in appearance, and the same species may exist in many states such as egg, larva, pupa, and adult, indicating a large intra and inter-class variances. Furthermore, the existence of poor lighting and harsh environments complicates the automated identification process. Therefore, a low-complexity automated framework for precise pest recognition in the field and performance improvement both in terms of classification robustness and computational complexity is still required. In this work, we presented a cost-effective DL-based model for pest recognition and categorization using drone. The presented framework is based on custom CornerNet model with the DenseNet-100 serving as a backbone for deep feature extraction from the input samples. Our results show that the proposed technique is capable of effectively localizing and classifying the multiple pest species in the presence of high variation in shape, size, color, position, and variability across and within classes. The main contributions of our work are as follows:

The rest of the paper is structured as follows: “Related work” reviews related work for insect pest recognition, while “Proposed method” provides a detailed description of the proposed framework. In “Experimental details and results”, we provide the details of experiments performed and a discussion on the results. Lastly, “Conclusion” concludes our research.

Recently, pest localization and classification have attained the consideration of the research community due to immense development in the area of computer vision. Numerous standard datasets are available for this purpose. However, these datasets consist of the minimum number of samples compared to the normal data compulsory for the latest DL-based frameworks. This section conducted a thorough examination of previous work for the automated identification and categorization of pests from crops.

Nanni et al. [32] presented a method to merge CNNs with saliency approaches to recognize and classify the pest from crops automatically. Initially, the saliency method was applied for data augmentation, and then five different CNN models, namely AN, DenseNet201, ShuffleNet (SN), GoogLeNet (GL), and MobileNetv2 (MN), were trained to classify the insects. This approach [32] improves the pest classification accuracy. However, the performance degrades for identifying pest species with significant intra-class differences. In [33], the authors presented a novel CNN framework and compared it with existing DL models, i.e., AN, RN, GL, and VGG. Transfer learning and data augmentation were employed to prevent the network from overfitting and improve classification accuracy.

Similarly, Li et al. [34] proposed an approach for automated recognition and categorization of crop insects. Initially, the adaptive threshold (AT) algorithm was applied over the input sample to convert it to a binary image, on which morphological operations together with the watershed algorithm were used to acquire the region of interest (ROI). Then, the GrabCut technique was utilized to remove the background, and several DL models, namely VGG, GL, and RN, were applied to classify the pest from the input samples. However, these methods [33, 34] exhibit better insect classification accuracy at the expense of a longer computing time. Wang et al. [35] introduced a DL framework for mobile devices, namely DeepPest, to automatically detect and categorize insects. The method [35] employed contextual information as prior information for its training and worked well for the localization of small size insects. However, the approach [35] is inadequate for different mobile devices due to processing limitations. Jiao et al. [36] introduced an anchor-free region CNN (AF-RCNN) for automated localization and categorization of various classes of crop insects. Initially, a keypoints fusion unit was proposed to compute the representative set of features, particularly for small-sized pests. Next, an anchor-free region proposal network (AFRPN) was introduced to calculate practical object proposals based on pest positions by employing fusion feature maps. Lastly, AF-RCNN was trained to identify 24 classes of insects by integrating the AFRPN with Fast R-CNN into a single framework. This method [36] works well for the localization of small insects. However, the performance relies on extensive hyper-parameter choices selected during the training process.

Rodríguez et al. [37] proposed a framework for pest identification. Initially, the RGB sample was transformed into a quaternion matrix, on which Quaternion Gaussian Low-Pass Filter was applied to remove the noise. The processed sample was then subjected to Sangwine’s method to obtain the two colored keypoints maps and horizontal and vertical directions. Both maps were converted to the HSV domain to differentiate the monotone horizontal and vertical edge maps. In the following steps, the obtained maps were combined, on which binarization together with the morphological approach was applied to achieve the ROIs. This method [37] is robust to pest detection under chrominance and size variations. However, the generalization performance of this work can be further improved. Nam et al. [38] suggested a DL-based approach to locate and categorize crop insects. The Single-shot detector (SSD) framework was used to calculate the in-depth features from input samples and classify the pests into respective classes. The approach [38] resulted in higher accuracy than previously developed methods. However, it was unable to detect small insects. CNN-based techniques need diverse training samples to show better accuracy. However, datasets lack this aspect. Li et al. [39] proposed a data augmentation-based approach to deal with such challenges. Data augmentation was applied during the training step by rotating input samples to different angles together with a cropping operation. This step assisted in obtaining diverse multi-scale samples that could be utilized to train a multi-scale insect identification framework. Various CNN models were trained to demonstrate the effectiveness of the proposed strategy. This technique [39] detects insects despite significant variations in position. However, it is computationally costly. A two-stage CNN framework was proposed in [40] to locate and categorize crop pests. Initially, the Global activated Feature Pyramid Network (GaFPN) was applied to compute the representative set of features from the input images. The calculated feature vector was passed to the Local activated Region Proposal Network (LaRPN) to identify and classify the pests in the next phase. The method [40] shows better pest classification performance. However, it is prone to overfitting issues, thus resulting in poor performance on unseen data. Another framework for pest detection was proposed in [41]. Initially, the input image was converted to greyscale. In the next step, the processed sample was compared to a reference image to identify the changes saved as the feature vector. In the next step, the Density-Based Spatial Clustering (DBSCAN) was applied over the calculated key points to cluster the pests from samples. This approach [41] can effectively identify the insects from the noisy samples; however, this approach is computationally complex.

Nieuwenhuizen et al. [42] presented an approach to locate and classify the insects from the input sample. In the first step, annotations were developed from the input images, which were employed for transfer learning. In the next step, the annotated images were passed to a DL framework, namely Faster R-CNN, to localize and classify the insects. In the last step, insects were counted manually. The approach [42] improves the insect classification accuracy. However, few results were reported. In [43], the authors conducted a comparative analysis of various CNN-based frameworks, namely VGG, ResNet-50, ResNet-101, AN, and InceptionNet, together with SVM, KNN, and ELM classifiers. The mentioned CNN models were employed to compute the in-depth features, which were later used to train the classifiers to classify the pest from crops. It is concluded in [43] that in-depth features with SVM and ELM classifiers exhibit better classification accuracy. Liu et al. [44] proposed a DL-based model, namely PestNet, for classifying crop pests. In the first step, a module, namely Channel-Spatial Attention (CSA), was introduced to be replaced with CNN for keypoints computation. Then, Region Proposal Network (RPN) was employed for calculating region proposals to locate the positions of insects based on extracted keypoints maps from the samples. Finally, a Position-Sensitive Score Map (PSSM) was applied to show the located pest together with their computed class. This approach [44] works well for multiclass classification of crop pests, however, at the expense of increased computational complexity. Another automated pest detection framework was introduced in [45]. After performing image preprocessing, EM and KMM were applied to obtain the ROIs. In the following steps, the GLCM matrix was employed to compute the image features from computed ROIs. The obtained feature vector was used to train the SVM to classify the insects. The method [45] is robust to pest detection. However, it requires a substantial amount of time for data preparation and training. Rustia et al. [46] proposed an approach to localize and classify crop insects. After performing preprocessing, YOLO-V3 has been employed to calculate the deep features and classify the pests from the input samples. This technique [46] shows better insect detection accuracy. However, it is unable to locate pests under intense chrominance and light variations. Another CNN model, namely AN, was utilized in [47] to identify and categorize the insect from images. The technique [47] exhibits better recognition accuracy. However, performance decreases when multiple pest species are present. Xia et al. [48] presented a technique to identify and classify insects. Initially, a data augmentation step was applied to improve the diversity of data. Then the samples were used to train the VGG-19 framework to localize and categorize the pests. This approach [48] works well for insect classification, however, performance is evaluated on limited insect species data. In a very recent work [73], the Custom CenterNet framework with DenseNet-77 method was presented to automate the plant diseases detection and categorization efficiently. The model outperformed the latest plant disease approaches and were able to efficiently locate and classify 38 types of crop diseases from the PlantVillage dataset.

Table 1 presents the analysis of existing techniques employed for pest detection and classification along with their limitations. From Table 1, it can be seen that although the research community has presented extensive work in the field of automated pest categorization, however, there is still a need for performance improvement.

In this section, we have discussed the framework presented for the automated identification and categorization of several crop pests in the field. The aim of this work is to propose a technique that is computationally efficient and capable of automatically extracting reliable image features without the need for any manual inspection. The proposed approach follows two phases: in the first step, a set of samples comprising images from a standard dataset, namely IPI02, is used to prepare the annotations, which are later used for the model training. In the test phase, suspected samples are passed to the trained framework to evaluate the model’s performance. More specifically, we have proposed an improved CornerNet model [49] by employing the DenseNet-100 as its base network. Initially, the deep features from the input images are calculated by using the DesneNet-100 framework, which is later used by CornerNet to locate the presence of pests on plant leaves and determine the corresponding category. In the last step, performance is evaluated by employing several standard metrics used in the field of object detection. Figure 1 shows the structural description of the introduced pest detection and classification methodology.

This section describes the implementation details and the experiments carried out to evaluate the performance of the suggested model. To comprehensively demonstrate the efficacy of the custom CornerNet model, we have evaluated pest recognition and classification and compared it with other models.

In our work, we have presented a low-cost DL-based framework for the automated recognition and categorization of crop pests in the field using drones. The presented method is based on a custom CornerNet model that employs DenseNet architecture as a backbone network for feature extraction. More precisely, we employed the DenseNet-100 network to extract a discriminative set of keypoints from the input samples. The custom CornerNet model is then trained to recognize various types of pests. We evaluated our approach on the IP102 dataset, a large-scale challenging pest recognition benchmark database comprising in-field captured images. Through extensive experimentation, we have shown the efficacy of our approach for real-world pest monitoring applications. The reported results showed that our method could accurately localize and classify pests of various categories in the presence of complex background and variations in pest shape, color, size, orientation, and luminosity. In the future, we intend to develop a more effective feature fusion approach to improve the performance of our method for fine-grained pest categorization.