Abstract

Weeds are unwanted plants that grow among crops. These weeds can significantly reduce the yield and quality of the farm output. Unfortunately, site-specific weed management is not followed in most of the cases. That is, instead of treating a field with a specific type of herbicide, the field is treated with a broadcast herbicide application. This broadcast application of the herbicide has resulted in herbicide-resistant weeds and has many ill effects on the natural environment. This has prompted many research studies to seek the most effective weed management techniques. One such technique is computer vision-based automatic weed detection and identification. Using this technique, weeds can be detected and identified and a suitable herbicide can be recommended to the farmers. Therefore, it is important for the computer vision technique to successfully identify and classify the crops and weeds from the digital images. This paper investigates the multiple classifier systems built using support vector machines and random forest classifiers for plant classification in classifying paddy crops and weeds from digital images. Digital images of paddy crops and weeds from the paddy fields were acquired using three different cameras fixed at different heights from the ground. Texture, color, and shape features were extracted from the digital images after background subtraction and used for classification. A simple and new method was used as a decision function in the multiple classifier systems. An accuracy of 91.36% was obtained by the multiple classifier systems and was found to outperform single classifier systems.

1. Introduction

Computer vision often abbreviated as CV is defined as the process of analyzing images and videos automatically to obtain meaningful inference or measurements without human intervention. It is a multidisciplinary field that involves artificial intelligence (AI) and machine learning concepts. The goal of computer vision is to understand the content of digital images and videos [1]. This involves developing methods and techniques which attempt to reproduce the human capability of cognition and recognition. Computer vision applications are being extensively used in the agricultural domain for the discrimination of crops and weeds, plant classification, crop disease identification, and so forth. The automatic weeding machine was proposed in [2], which was based on computer vision techniques [3]. Automation of various agricultural tasks such as disease detection in crops, precise spraying of pesticides, prediction of crop yield, estimation of soil texture, automatic grading of fruits, estimation of crop biomass, management of water balance in the irrigation system, and monitoring plant growth has been done using computer vision techniques [4, 5]. In [6], diseased plant leaves were identified using image processing and soft computing techniques. In [7], discrimination of corn crop and weed species was done using the C5 algorithm based on textural and spectral features. Wireless sensor networks and wireless visual sensor networks are contributing to sending the sensed data of the field either in image or in text form to the remote machine where it will be processed and analyzed for some kind of decision-making [8]. In [9], a wireless sensor network was used to predict the occurrence of bud necrosis virus in groundnut crops. In [10], a wireless sensor network was implemented to predict the water requirement in semi-arid regions. Soft computing techniques along with wireless sensor network technology are helping farmers and agronomists to make the right decision at the right time. A precision agriculture model resulted due to the advances in wireless sensor networks and computer vision techniques [11]. This model can help to reorganize the entire farming system with low input, high efficiency, and sustainable farming.

Weeds can be defined as undesirable plants growing with crops. Generally, they are referred to as plants out-of-space. They compete with the crops for soil nutrients and water. Here, the term competition can be referred to as weed plants nourishing at the cost of rice [12]. This competition usually does not result in the death of the rice crop but definitely leads to reduced yield. The problem caused by weeds and the importance of weed management is not recognized by the general population and sometimes even by agriculturists. One of the main reasons for agriculturists to ignore weeds is high cost involving manual weeding and expensive herbicides. Other than competing with rice plants for soil nutrients, water, and space, they also act as alternate hosts to various pests, which in turn attack crops and destroy them. Weeds are the main reason for heavy yield losses and sometimes are responsible for complete crop failure [13]. It is reported that annually India is incurring a loss of INR 1050 million because of weeds in paddy fields [14]. Classification of weeds in rice fields is usually based on their life cycle, their habitat, and their gross morphological features [15]. They are broadly categorized as sedges, grass, and broadleaved weed [16], as shown in Figures 1(a)1(c), respectively [17, 18].

The multiple classifier system (MCS) is a way of using many classifiers to make a final decision in a classification process. The ensemble of different classifiers has been used lately in pattern recognition to improve the performance and aims at increasing the accuracy of the single classifier system [19]. The idea is that two or more diverse classifiers when grouped help in negating the errors made by the individual classifiers [20]. There are two types of decision functions that are commonly used in the design of MCSs. They are classifier fusion (combination-based) and classifier selection (selection-based) [21, 22]. In classifier fusion, the outputs of several classifiers in the MCS or ensemble are fused to obtain the final decision. Some of the classifier fusion techniques are majority voting, sum rule, product rule, and so forth [23]. In classifier selection, one classifier’s output is selected from the pool of classifier members. An example of the classification selection method is cross-validation accuracy (CVM). The classifier selection method can be static or dynamic. CVM is a static classifier selection method. In [24], Kuncheva has proposed clustering-based dynamic classifier selection methods.

In [25], an MCS was proposed for the classification of leaves from the digital images based on texture and shape features. Heterogeneous classifiers are made of SVM and neural network classifiers with different texture features such as local binary patterns, histogram features (HoG), and Zernike moments. The dataset was obtained from ImageCLEF. Results show that improvement relative to the classification score was reported in the literature for the ImageCLEF-2011 and ImageCLEF-2012 datasets.

In [26], an SVM-based MCS was developed to recognize diseased wheat leaves. The MCS was designed as a two-level structure. Three SVM classifiers were trained with color features, texture features, and shape features separately. At level-0, the output of the individual three classifiers was fed as training data to the level-1 classifiers. This way the errors made by level-0 classifiers are compensated and the direct mapping limitation at level-0 was solved.

In [23], a review was made about the different topology of MCSs. Also, different types of decision functions are explained that can be used when implementing the MCS. The paper gives insight for the developers working on MCSs.

In [27], a review is made about MCSs as hybrid systems for information fusion. The paper explains the different types of hybrid systems built using the MCS. In addition, this paper explains different combinatorial functions to fuse the results.

MCSs have been used widely in the field of biometrics. Research favors the solutions where information about the iris, voice, fingerprints as well as features are provided through a combination of information rather than information generated from a single source. MCSs are also used in document analysis, remote sensing data analysis systems, medical decision support systems, and so forth [28]. However, MCSs are less explored in the plant classification domain. In this study, MCSs were designed to classify paddy crops and weeds from digital images. The main contributions of this study are as follows:(i)Creation of a digital image dataset of paddy crops and weeds. The dataset contains around 1500 images taken under natural varying lighting conditions. The images were acquired using Sony Cybershot (DSC-W220), Canon PowerShot SD3500 IS, and Raspberry Pi (RP-v5647) cameras. The dataset also contains annotated images.(ii)Two selection-based MCSs were designed: one with calibrated RF classifier and calibrated SVM classifier called MCS-1; another with uncalibrated RF classifier and uncalibrated SVM called MCS-2. The performance of two MCSs is compared and analyzed.(iii)A new dynamic classifier selection method is proposed.

2. Materials and Methods

2.1. Dataset Creation

Two digital cameras (Canon PowerShot SD3500 IS) and Sony Cybershot (DSC-W220) were used to acquire images from paddy fields around Manipal region of Karnataka state in India. Images were acquired under natural variable lighting conditions with the camera fixed at different heights such as 0.61 m, 0.91 m, and 1.22 m from the ground. The cameras were fixed on a tripod and were facing down towards the ground. Images were also acquired from the Raspberry Pi (RP ov5647) camera installed in the paddy field. The weeds and crops of varying canopy sizes were selected for acquisition to increase the difficulty of identification. The dataset contains paddy plants and weeds from the early growth stage (1-leaf seedling stage) to the flowering stage. These growth stages are very critical because the effect of weed will be maximum [29]. Both transplanted and direct-seeded rice fields were selected for acquisition. The images were stored in RGB color space in JPG format. The MATLAB R2018a was used to process the images. While working in MATLAB, images were resized to 1296 X 966. The dataset consists of 1517 images. Figure 2(a) shows an image acquired from the Canon camera fixed at a distance of 0.61 m from the ground. Figure 2(b) shows Figure 2(a) after background subtraction. Figure 3(a) shows the image acquired from the Canon camera fixed at a distance of 1.22 m from the ground and Figure 3(b) is its background subtracted image. Similarly, Figures 4(a) and 5(a) show images taken from the Sony camera fixed at a distance of 0.61 m and 1.22 m from the ground, respectively, and Figures 4(b) and 5(b) are its background subtracted images. Figure 6(a) shows the image acquired by the Raspberry Pi camera and Figure 6(b) is its corresponding background subtracted image.

2.1.1. Feature Extraction and Feature Selection

The soil and water background was removed to retain only green vegetation. Connected component algorithm and successive erosion and dilation [30] were used to remove possible overlapping and to extract individual plants for feature extraction as shown in Figures 7 and 8. The images with too heavy overlapping were not considered and left out from classification. Color features were extracted using the methods explained in [31]. Two different types of shape features were extracted, namely, size independent features and Hu’s moments using the method as explained in [8]. Texture features were extracted using the Laws’ texture masks [32] using the method explained in [33]. Ninety-three features were extracted. From these, seventy-one best-performing features were selected using the feature selection method based on analysis of variance (ANOVA) [34, 35]. Table 1 gives information about the number of samples classwise used in the classification task.

2.2. Design of Multiple Classifier Systems

Multiple classifier systems (MCSs) can be built using two or more same types of classifiers or different types of classifiers whose outputs are then fused to determine the final class. The individual classifiers can be trained on the same data, or with different data partitions or data with different feature subsets. If the participating classifiers are trained with different partitions, then techniques such as bagging and boosting can be used [36]. Since the decision support system is built on either homogenous or heterogeneous models, MCSs are the subcategory of hybrid intelligent systems.

2.2.1. Diversity Metric

MCSs can show improved performance than single classifier systems provided very diverse classifiers be used in the design of MCSs [37]. Therefore, before designing the MCS, the diversity metric was calculated for different pairs of classifiers. There are many diversity metrics such as entropy, double fault, correlation, and Yule’s statistic [38]. In this study, Yule’s statistic was used to calculate the diversity of two classifiers based on the pairwise diversity metric [39]. If the value of Yule’s statistic is nearer to zero, then the classifiers are more diversified. The Yule’s statistic Q is given bywhere a, b, c, and d are given in Table 2.

Steps in finding Yule’s Statistic:(1)Take sample data (300) and split into training data and test data(2)Fit the two models on training data.(3)Predict on test data.(4)Calculate the diversity metric using Yule’s statistic

From Table 3, the combination of RF and SVM has the lowest Qi,j. The value of a for this combination was 0.8334. Therefore, these two classifiers were selected to design a selection-based MCS. In selection-based MCSs, for each input sample (test), the output of one classifier among many classifiers will be selected dynamically based on a selection mechanism. Since heterogeneous classifiers were used in this study, it is important to evaluate the quality of individual classifiers and use them for a proper classifier selection. Therefore, the weights of each class for each classifier was obtained. To do this, the whole dataset was divided into two sets called Train set-1 and Test set in the ratio 80 : 20. Trainset-1 was again divided into two sets in the ratio 80 : 20 as Train set-2 and Weight set. After fitting the classifiers with Train set-2 data, it was tested by Weight set. The weight for each class of each classifier was obtained by dividing the number of Weight set samples belonged to each class that was correctly classified by the classifier by the total number of Weight set samples. This procedure is summarized in Algorithm 1. For the final class selection, the probability of the sample belonging to each class is taken and summed with the corresponding weight. The class that has the highest value is selected as the final class. The final class selection by MCS-2 is summarized in Algorithm 2.

(1)Input
(2) Train set-1 Train data set with class labels
(3) Test-set Test data
(4)k Number of classes
(5)n Number of classifiers
(6)Output
(7)W[n][k] Weight of each class of each classifier
(8)X_train, X_ ⟵ weight divide the Train set-1 into two sets in the ratio of 80 : 20
(9)len ⟵ length of X_weight
(10)for i 1 ⟵ to n do
(11) Train Ci on X_train
(12) Predict using Ci on X_weight
(13)for j ⟵ 0 to k − 1 do
(14)  W[i][j] ⟵ (number of samples that belong to j class that were correctly classified by classifier Ci)/len
(15)end for
(16)end for
(1)Input
(2)Train set-1 Train data set with class labels
(3)Test-set Test data
(4)k Number of classes
(5)n Number of classifiers
(6)W [n] [k] Weight of each class of each classifier
(7)Output
(8)Y [ ] Class labels
(9)len ⟵ length of Test set
(10)Y [ ] ⟵ NULL
(11)for i ⟵ 1 to n do
(12) Train Ci on Train set-1
(13)end for
(14)for i ⟵ 1 to len do
(15)p Predict probability of classes for sample Si using C1, where Si Є Test-set
(16)q Predict probability of classes for sample Si using C2, where Si Є Test-set
(17)end for
(18)for i ⟵ 1 to len do
(19)for j ⟵ 0 to k − 1 do
(20)  p[i] [j] ⟵ W [1][j] + p[i][j]
(21)  q[i][j] ⟵ W[2][j] + q[i][j]
(22)end for
(23)end for
(24)for i ⟵ 1 to len do
(25) max1 ⟵ 0
(26) max2⟵0
(27) index1 ⟵ 0
(28) index2 ⟵ 0
(29)for j 0 to k − 1 do
(30)  if p[i][j] > max1 then
(31)   max1 ⟵ p[i][j]
(32)   index1 ⟵ j
(33)  end if
(34)  if q[i][j] > max2 then
(35)    max2 q[i][j]
(36)   index2 j
(37)  end if
(38)end for
(39)if max1 > max2 then
(41)  Y [i] ⟵ index1
(42)else
(43)  Y [i] ⟵ index2
(44)end if
(45)end for
2.2.2. Designing of MCS with Calibrated Classifiers

(1) Calibrating SVM and RF Classifiers. There are many ways of calibrating the classifiers. The most commonly used methods are platt scaling and isotonic regression [40, 41].

(2) Platt Scaling. This type of scaling is used when the distortion in the predicted probabilities is in the sigmoid shape. The outputs of the classification system (scores) are passed through the sigmoid function, where parameters A and B are fitted using maximum likelihood estimation from a fitting training set and is given by

This fitting is done by gradient descent [42] to reducewhere .

(3) Isotonic Regression. Let fi be the prediction from the classifier and the corresponding actual target be yi, then the isotonic regression can be defined as where m is a nondecreasing function.

The isotonic regression will show superior performance when there are large data (beyond 1000). In this study, isotonic regression was used for calibrating the classifiers. Cross-validation is helpful to solve the problem of overfitting. In this research work, 5-fold cross-validation was used. In this, the base estimator (RF and SVM) was fit on the train set of the cross-validation generator, and the test set is used for calibration. The probabilities for each of the folds are then averaged for prediction. Algorithm 3 summarizes the design of the MCS using calibrated classifiers.

(1)Input
(2)Train set-1 Train data set with class labels
(3)Test-set Test data
(4)k Number of classes
(5)n Number of classifiers
(6)W [n] [k] Weight of each class of each classifier
(7)Output
(8)Y [ ] Class labels
(9)len ⟵ length of Test-set
(10)Y [ ] ⟵ NULL
(11)for i ⟵ 1 to n do
(12) Fit calibrated Ci using isotonic regression on Train set-1
(13)end for
(14)for i ⟵ 1 to len do
(15)p Predict probability of classes for sample Si using C1, where Si Є Test-set
(16)q Predict probability of classes for sample Si using C2, where Si Є Test-set
(17)end for
(18)for i ⟵ 1 to len do
(19)for j ⟵ 0 to k − 1 do
(20)  p[i] [j] ⟵ W [1][j] + p[i][j]
(21)  q[i][j] ⟵ W [2][j] + q[i][j]
(22)end for
(23)end for
(24)for i ⟵ 1 to len do
(25) max1⟵0
(26) max2 ⟵ 0
(27) index1 ⟵ 0
(28) index2 ⟵ 0
(29)for j 0 to k − 1 do
(30)  if p[i][j] > max1 then
(31)   max1 ⟵ p[i][j]
(32)   index1 ⟵ j
(33)  end if
(34)  if q[i][j] > max2 then
(35)    max2 ⟵ q[i][j]
(36)   index2 ⟵ j
(37)  end if
(38)end for
(39)if max1 > max2 then
(40):
(41)  Y [i] ⟵ index1
(42)else
(43)  Y [i] ⟵ index2
(44)end if
(45)end for

3. Results and Discussion

3.1. Evaluation Using Confusion Matrix

Table 4 shows the multiclass confusion matrix, and the diagonal element gives you the true positives (TP) of respective classes. Here, we have a four-class confusion matrix. According to [43], to evaluate the performance of the classifier in a multiclass classification case, for each separate class Ci, TPi, true negative (TNi), false positives (FPi), false negative (FNi), Accuracyi, Recalli, and Specificityi can be calculated from the counts, counti from each class Ci. The performance of the classifier is calculated in two ways, one using macroaveraging and another microaveraging. In the case of macroaveraging, an evaluation parameter is the average of the same parameter, but in the case of microaveraging, cumulative sum of counts to get the cumulative values of TP, TN, FP, and FN is obtained, and then evaluation parameters are calculated. In this study, macroaveraging was used. The calculation of evaluation parameters for multiclass classification is as shown in Table 5 where l is the number of classes.

3.2. Evaluation Using Area Under Curve

The receiver operating characteristics (ROC) curve is used in machine learning to depict the performance of a classifier visually. It is a graph of the false-positive rate (FPR) against the true-positive rate (TPR). The area under the ROC curve commonly referred to as AUC is another measure used to assess the performance of the given classifier [44, 45]. It gives us the discriminative ability of the given classifier. That is, it gives us the probability with which the classifier will rank a randomly chosen positive instance. For example, if we get an AUC value as 0.8, then it means that a randomly chosen positive instance has a higher score than for a negative instance 80% of the time. If most of the time classifier cannot clearly distinguish between the groups, we have AUC as 0.5, for a random classifier. For the best classifier, we have AUC as 1. Thus, this area gives us the predictive accuracy of a classifier model. Higher the value of AUC, the better the model is. Figures 9 and 10 show AUC for MCS-1 and MCS-2. The value of AUC for all the class types in both the MCSs is above 0.95 indicating the excellent performance of the MCSs.

Cohen’s kappa [46, 47], calculated for different partitions of the train and test data, is shown in the box plot in Figure 11.

Tables 68 show the classification results obtained by MCS-1, MCS-2, and single classifiers (RF and SVM), respectively. The following are the observations made from this study.

(1)Accuracy of 91.36% is obtained in classifying paddy crops and three types of weeds. Even though it is difficult to compare the result of this study with other related works due to a different field and other conditions, three studies [4850] have been selected for comparison. Even though the method described in [50] achieved better results when compared to this study, the work was carried out on a single image, and we believe that, it cannot be considered as a model for rice and weed discrimination. In [49], the work has been performed to detect the rice plant based on rice ears, while the proposed method works for different crop and weed sizes. The work carried out in [48] achieved a slightly better result than this study, but it is computationally intensive and resource-intensive.

4. Case Study

The methodology of feature extraction and classification of this research work was applied to the rice and weed dataset created by [48]. Table 9 shows the classification result obtained on this dataset.

The result in Table 9 shows that the feature set extracted has the potential for effective feature vector representation of paddy crop and weed images for the classification process.

(1)The two MCSs designed outperformed the single classifier systems. This result indicates that the MCS can help in improving crop and weed classification task and has to be explored with different designs.(2)The MCS designed with calibrated classifiers performed slightly better than the MCS designed with uncalibrated classifiers. Therefore, this study supported the claim made by the [51] that the calibrated classifier does not degrade the performance of the classification task.(3)The misclassification of paddies is found to be more in MCS-1 and it is misclassified as grass. This could be attributed to the fact that features of paddies and grass are more or less the same [52, 53]. Hence, the chances of the true probability values of paddy class (class 2) matching with grass class (class 1) are largely resulting in misclassification. This could be reduced by increasing the number of samples of grass weeds with different growth sizes.(4)Kappa between two MCSs around 0.929 indicates that two MCSs agree with each other greatly.(5)The extracted features are good enough to discriminate paddy crops and weeds. The method is device-independent. This is the first dataset of paddy crops and three types of weeds to the best knowledge of authors.

5. Conclusion

In this study, two MCSs were designed and assessed for the classification of paddy crops and weeds from the digital images. The approach was first to create digital images of paddy crops and weeds from paddy fields using digital cameras, and the Raspberry Pi camera was fixed at different heights from the ground to make the method device-independent. The soil and water background was removed. Texture, color, and shape features were extracted. Two selection-based MCSs were designed, one with calibrated random forest and calibrated SVM classifiers called as MCS-1 and another MCS with uncalibrated random forest classifier and uncalibrated SVM classifier called as MCS-2. MCS-1 and MCS-2 outperformed the single classifier systems. In addition, it was found that the MCS designed with calibrated classifiers performed slightly better than the MCS designed with uncalibrated classifiers. The features extraction and classification process proposed in this research work were applied on a publicly available paddy crop and weed dataset and results obtained are very promising. The study showed that the extracted features are good enough to classify paddy crops and weeds. This work could be used to recommend suitable herbicide for a particular type of weed based on the classification results to avoid broadcast application of the herbicides. This could lead to the reduction of herbicide-resistant weeds, contamination of the groundwater, and other ill effects of overuse of the herbicides.

In the next phase of this research, we intend to expand the paddy crop and weed digital image dataset. We intend to design different types of MCSs for the classification of paddy crops and weeds and analyze the performance. In addition, we intend to carry out extended research to develop a general crop and weed discrimination model using a selection based-MCS.

Data Availability

The datasets are not made public yet. However, interested readers can get the sample images upon request from the corresponding author through e-mail: [email protected].

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding this article.