Automated apple stem end and calyx detection using evolution-constructed features

Highlights

• The importance of distinguishing stem end and calyx from defect is discussed.

• A new feature construction method is introduced to food engineering applications.

• ECO features automatically construct useful features without human input.

• This stem end and calyx detection method can be adapted for other stone fruit.

Abstract

Distinguishing stem end and calyx from true defects is the main challenge for an automatic apple blemish detection systems in real time. This paper presents the result of using newly developed Evolutionary COnstructed (ECO) features for distinguishing bruises and blemishes from the stem end and calyx of apple images acquired in near-infrared spectrum. Rather than relying on human experts to build features sets to tune their parameters, our method uses simulated evolution to construct series of transforms that convert the input apple images into high quality features. The use of this method demonstrates the feasibility of using machine vision technology with the off-the-shelf optical and electronics components to detect true bruises and blemishes on apples with higher than 94% accuracy. Although Gala apple, a very challenging case, is used in this paper as an example to prove its feasibility, this method could be easily adapted for other stone fruits.

Introduction

Statistics show the United States alone produced 9.4 million pounds of apples in 2011 (USDA, 2012). It represents the total revenue of close to US$2 billion (USDA, 2011) accounting for both fresh fruits and other apple products such as applesauce and apple juice. Before this huge amount of apples is sent to their designated markets, a reliable inspection process to grade and sort them must be performed. Manual inspection of apples is a labor-intensive task and subject to many drawbacks such as subjectivity, cost, inefficiency, and inconsistency (ElMasry et al., 2008). As a consequence, an automatic system is required to increase the speed of inspection and eliminate the errors and variations introduced by human laborers.

According to a consumer survey (Ricks et al., 2002), 81–91% of consumers reported that the most important characteristics of apples are their flavor, that they are unbruised and unblemished, and their crispness. Far fewer consumers reported other factors such as color, variety, and price as “very important”. Considering the consumer demand for crisp unblemished apples, a reliable inspection system is required to identify bruises and blemishes before the apples are distributed to markets. Machine vision systems grading apples into different quality grades based on color and size have been around since the early 1990s. However, vision systems sorting surface defects such as rotten spots and insect bites and subsurface defects such as bruises are not yet commercialized. The biggest challenge preventing this from happening is the similarity between these defects and apple stem end and calyx.

An ideal inspection procedure would be a high-speed nondestructive measurement on the apple packing line to achieve 100% inspection and allow only high quality apples to reach the fresh fruit market. Since the last decade of the 20th century, machine vision has been successfully employed in inspecting fruits with respect to size, color and other appearance indices, for tasks such as shape classification, defect detection, quality grading, and variety classification (Xing et al., 2007, Nicolai et al., 2007, Brosnan and Sun, 2004). Machine vision systems for real-time color classification (Lee and Anbalagan, 1995, Lee, 2000, Zhang et al., 1998) have been commercialized to grade food products based on color. Xing et al. (2005) investigated a multi-spectral imaging system for detecting bruises on Golden Delicious apples. Lee et al., 2008a, Lee et al., 2008b developed a machine vision system for automatic date quality evaluation for commercial production. Ahmad et al. (2010) proposed an automatic grading prototype for citrus using machine vision. Machine vision based automatic systems are also reported for inspecting other kinds of fruits, including peaches (Esehaghbeygi et al., 2010), bananas (Wang et al., 2009), grapes (Kim et al., 2009), kiwifruits (Rashidi and Gholami, 2008), etc. Many of these systems have shown very promising results. However, detecting the true defects and distinguishing them from the stem end and calyx of apple is still a critical and open issue for machine vision-based apple inspection systems (Unay and Gosselin, 2007).

Yang (1996) identified the stem end and calyx of mono-colored apples with an image analysis based method and achieved 95% average recognition rate. Lu (2003) proposed to use near-infrared (NIR) hyperspectral imaging for detecting bruises on apples in the spectral region between 1000 nm and 1340 nm and achieved a correct detection rate from 62% to 88% for Red Delicious and from 59% to 94% for Golden Delicious. In order to distinguish the stem end and calyx from the true defects of apples, Tao et al. (2003) used both a NIR camera and an expensive mid-infrared (MIR) camera for combined sensing. The image taken by MIR camera was sensitive to stem end and calyx while the image taken by NIR camera was sensitive to both stem end/calyx and true defects. The MIR image was used to segment the stem end and calyx from the NIR image. Tao’s system achieved 92% classification accuracy. However, the expensive MIR camera may limit the usage of this system. Li et al. (2002) proposed an artificial neural networks-based system that uses two CCD cameras as an alternative to the more expensive MIR camera and achieved 93% classification accuracy.

Unay and Gosselin (2007) proposed to recognize stem and calyx regions in ‘Jonagold’ variety apples by three stages, i.e. threshold-based segmentation, features extraction, and classification algorithm-based recognition. Statistical, textural, and shape features were extracted from the segmented area and provided to Support Vector Machine for classification. The shortcoming of this system is the region of stem end and calyx is prone to be misclassified if it is close to the edge and likely to be incorrectly recognized if it is partially segmented. ElMasry et al. (2008) proposed a hyperspectral imaging system for early detection of apple bruises. But the experiment was performed with the assumption that bruises appear in the middle areas between stem end and calyx. Literature review shows stem end and calyx detection is still an unsolved problem for real-time high volume production line applications.

Previous research methods for automated stem end and calyx classification depend on a human expert to design the features for the identification algorithm. Although those proposed features are able to describe the object of interest well and produce sound accuracy for recognition, specific features created by human experts that are good for one class of products may do poorly for others. New features must be developed or grading parameters must be reset for different varieties. This manual process often requires change of algorithm and software or fine-tuning sorting parameters, which need unique skills and extensive training. In this paper, we propose, based on our previous work on a novel feature construction method called Evolution-COnstructed (ECO) features (Lillywhite et al., 2012), to automatically construct features that are then used by AdaBoost to correctly distinguish bruises and blemishes from the stem end and calyx of apples. This method automatically discovers good and useful features of apple images without the use of human expert designed features. It is capable of constructing non-intuitive features that are often overlooked by human experts. Gala apples are used in the paper as an example to demonstrate the robustness of our algorithm. Gala variety was chosen because of its mixed shades of colors that are more challenging than other single-color varieties such as granny smith and golden delicious. ECO features can be learned and determined off-line for any apple variety, and can be easily adapted for other stone fruit such as peaches, plums, and pears.