1.

Introduction

The advent of modern hyperspectral imaging modalities opened the possibility of implementing a large spectrum of applications that cannot be robustly solved using the standard trichromatic [red, green, blue (RGB)] data. Among many possible applications that were well served by the inclusion of hyperspectral imaging systems (such as remote sensing, biomedical imaging, industrial inspection, etc.),^1–4 the accurate recognition of nonferrous materials represents a prominent example.^5–12 The motivation behind the decision to use hyperspectral information in the nonferrous material sorting process is multifold, where the chief reason being the fact that the color information is not sufficiently descriptive to robustly sample the characteristics associated with each nonferrous fraction. This issue proved to be particularly apparent in our study as the different fractions of nonferrous materials show considerable intraclass dispersion and a relative high interclass similarity in the RGB domain.

As opposed to the standard RGB image-formation process, hyperspectral imaging involves the acquisition and interpretation of multidimensional digital images that are able to sample the properties of the imaged materials in a substantially wider spectral domain than that covered by the visible spectrum.^12,13 In this regard, it is worth mentioning that the current range of hyperspectral sensors are able to simultaneously capture hundreds of spectral bands from ultraviolet to far infrared with a spectral resolution of less than 10 nm.^14,15 Thus, the main characteristic of the hyperspectral images is that each pixel is defined by a large vector whose elements are the spectral components captured from the light arriving at the spectral sensor.^4,13,16–19 While there is no doubt that the availability of hyperspectral data is opportune in the classification process, it is important to note that the process of capturing a large number of spectral bands results in a high dimensional data that may raise substantial challenges from practical and computational perspectives. In addition to the extensive dimensionality of the hyperspectral data, the substantial correlation between the closely spaced spectral bands may represent another issue that may affect the accuracy of the material classification process. Thus it is clear that the inclusion of the unprocessed hyperspectral data in the classification process is not only suboptimal, but also this approach will compromise the real-time operation of the developed machine vision system.

To answer the computational constrains and to avoid the Hughes phenomenon,^9,15 a large number of studies have been devoted to address the optimal decorrelation of the hyperspectral data, where the ultimate objective has been the reduction of the feature vectors that describe the materials that define the scene objects. To this end, principal component analysis (PCA),¹ linear discriminant analysis (LDA),²⁰ wavelet decomposition,² and uniform band design (UBD),²¹ were previously used for hyperspectral decorrelation. While these methods proved efficient when applied to diverse practical scenarios, they have the disadvantage of having to require a substantial level of user intervention during the training stage, and in addition these decorrelation approaches need retraining if new materials are included in the classification process. To circumvent these issues, unsupervised decorrelation techniques based on fuzzy sets were proposed,⁹ which proved particularly efficient when applied to material classification tasks. The experimental results reported in Ref. 9 indicate that the application of decorrelation not only improves the computational efficiency, but also increases the accuracy of the classification process. Our prior investigations¹⁰ revealed that the pixel-wise features calculated after the application of data decorrelation proved insufficient to accurately model the within-class variability associated with the nonferrous fractions.⁹ This inadequacy of the pixel-wise features was particularly exacerbated by the specular properties and the various levels of oxidization that are characteristic for the nonferrous materials contained in the waste of electric and electronic equipment (WEEE) scrap. A robust solution resides in the combination of the spectral and spatial features with the aim of generating a descriptor that is able to factor in not only the spectral properties of the imaged materials, but also their textural characteristics as well.^{4,9,15,22–24} One approach toward spectral-spatial integration was proposed by Mercier and Lennon,²⁵ where the authors attempted to model the texture in the hyperspectral domain by computing the marginal distribution of the wavelet coefficients using generalized Gaussian density (GDD). Other approaches implemented texture decomposition using Gabor filters,²⁶ directional filter banks,²⁷ Markov random fields,²⁸ or analyzed the texture using statistical schemes based on the calculation of co-occurrence matrices.²⁹ While these methods have shown promising results when applied to various practical applications, their major limitation is the onerous computational overhead associated with the calculation of the spectral-spatial descriptors (SSDs), a fact that rendered them as unfeasible when deployed in the development of real-time vision systems.

In a recent paper⁹ we detailed a novel framework where the textural and spectral features are integrated using local distributions, a concept that is well suited for sampling the spectral descriptors that are characteristic for nonferrous materials. The use of local distributions for nonferrous material sorting proved opportune from a classification performance standpoint (accuracy $>98\%$), but although the computational complexity of the proposed algorithm was substantially lower than that associated with standard hyperspectral texture analysis methods,^26–29 it was still too high to be feasible for real-time operation. Thus, the major goal of this paper is to redesign the process behind the calculation of the local distributions in the decorrelated hyperspectral domain that was presented in Ref. 9 with the objective of reducing the computational time to a level that offers real-time operation when the nonferrous material-sorting algorithm is included in the development of a conveyor-based WEEE recycling system. Another important objective in our research was to attain the real-time operation while maintaining the classification accuracy at a similar level when compared to the original implementation.

This paper is organized as follows. In Sec. 2, an overview of the WEEE recycling process is provided. Section 3 details the proposed material classification algorithm and its real-time implementation. Section 4 describes and discusses the experimental results, while Sec. 5 concludes the paper.

2.

WEEE Recycling: Background Information

The most recent statistics indicate that the WEEE constitutes 4% of the total municipal waste in Europe, and it is increasing by 16 to 28% every five years.^5,30 Although certain sectors of the electrical and electronic equipment (EEE) market show signs of stagnation (e.g., TV sets and large kitchen appliances), others, including information and telecommunication equipment, car electronics, and electronic toys, still experience a robust growth. In this context, it is useful to note that the European economic area (EEA) countries generate 6.5 million tons of WEEE per annum, and currently approximately 90% of this potentially hazardous waste is disposed as unsorted in generic municipal landfills. According to the current EU statistics, the WEEE is expected to increase to 12 million tons by 2015, and, as a consequence, legislation that sets specific requirements in regard to WEEE collection and recycling has been recently introduced.³¹ While the introduction of strict targets toward WEEE recycling had a direct impact on EEE manufacturers, as they had to adjust their environmental policies, it also opened an opportunity for companies that are active in the WEEE recycling sector.

Prior to the application of various technological recycling processes, the WEEE is shredded to allow the separation of its constituent parts. After mechanical, electrostatic, and densiometric sorting,^18,32 a distinct component of the WEEE scrap is formed by nonferrous materials (stainless steel, aluminum, copper, zinc, brass, and lead) that cannot be sorted by standard mechanical recycling procedures. This fact has negative economic implications as the unsorted nonferrous materials are sold at a much lower price than the individual nonferrous fractions. Thus the availability of new technologies that would allow the robust sorting of nonferrous materials will increase considerably the profitability of the overall WEEE recycling process. To achieve this goal, the classification of the nonferrous materials has been initially carried out using the RGB data. These approaches proved inefficient when applied to the separation of several fractions of nonferrous materials (such as stainless steel, aluminum, and zinc),⁸ and as a result new methodologies that perform the classification in the hyperspectral domain were actively explored.^9,24 While hyperspectral classification techniques proved accurate when applied to nonferrous sorting, they proved challenging when included in the development of industrial systems due to their high computational demand. To provide an insight into this issue, the specifications for WEEE recycling outlined in the SORMEN European project documentation³⁰ indicate that a sorting speed of one ton/h is required to justify the automatic recycling process from an economical perspective. This translates into a processing speed of 50 m/min (approximately 170 camera lines/s). The algorithms based on pixel-wise classification schemes can meet this computational constraint, but they are not able to properly model the interclass variations associated with different nonferrous materials. As indicated in the introductory section of this paper, the solution to this problem resides in the integration of the spectral and spatial information in the classification process, but there are substantial computational challenges that need to be overcome. In the following sections of the paper we will detail a real-time nonferrous material classification algorithm that is able to attain the processing speed required for an industrial recycling system, while maintaining the classification accuracy at a level above 96%.

2.1.

System Overview

In order to devise a flexible machine vision solution, we have adopted a modular approach. In this regard, the proposed system consists of several computational modules that include the hyperspectral image acquisition and nonferrous material classification, and mechanical subcomponents that implement the WEEE vibratory delivery system, particle transport module (conveyor), and the pneumatic particle sorting mechanism. The logical arrangement of the constituent modules of the developed machine vision solution is illustrated in Fig. 1, and in the remainder of the paper we will be primarily focused on the description and analysis of the nonferrous material classification algorithm, which is the core component of the system. Details about all mechanical components of the developed machine vision solution can be found in Ref. 24.

Fig. 1

Schematic of the nonferrous material sorting system.

3.

Real-Time Nonferrous Material Classification

The computational stages associated with the proposed real-time material classification algorithm can be summarized as follows:

For clarity, Fig. 2 presents a block diagram that describes the complete overview of the proposed real-time material classification algorithm. In this diagram, arrows illustrate the logical links between the distinct computational components of the proposed nonferrous classification algorithm. It is useful to mention that the main computational bottleneck is associated with the calculation of the SSD for each nonferrous particle (see the shaded component in Fig. 2). Thus the main objective of our work was to devise an optimized approach for the robust integration of spectral-spatial information to allow the implementation of a real-time nonferrous material-sorting system. In this sense, the new approach constructs a single descriptor for each nonferrous particle, as opposed to the approach detailed in Ref. 9, which requires the calculation of fuzzy spectral-spatial distributions for each pixel in the hyperspectral image and the application of computationally complex procedures relating to region merging and reclassification. Full details about each computational component shown in Fig. 2 will be provided in the remainder of this paper.

Fig. 2

Outline of the real-time nonferrous material classification algorithm.

3.1.

Hyperspectral Data Decorrelation

In the first step the unprocessed hyperspectral image is subjected to data decorrelation using an unsupervised approach based on spectral fuzzy sets.⁹ This approach is based on the knowledge that the spectral information varies smoothly over successive spectral bands, and as a result the characteristics associated with nonferrous materials should be sampled by groups of spectral bands rather than selective spectral bands. The data flow associated with the developed fuzzy-based decorrelation scheme is detailed in Fig. 3.

Fig. 3

Fuzzy sets-based hyperspectral decorrelation.

The developed hyperspectral decorrelation (see Fig. 4) involves the partitioning of the spectral domain in a predefined number of fuzzy sets, $m$, and an energy value $E_q$ for each fuzzy set $q$ ($q\in [1,m]$) is calculated as follows:

Eq. (1)

$$E_q=\sum _{i\in [1,k]}{M}_{fq}({\lambda }_i)I({\lambda }_i),q\in [1,m],\begin{array}{c}m<<k\end{array}\mathrm{.}$$

It can be observed that the energy measure defined in Eq. (1) samples the strength of the intensity signal $I(x,y)=[I_1(x,y),I_2(x,y),\dots ,I_k(x,y)]$ in each fuzzy set. Using this approach, the dimensionality of each hyperspectral pixel $I(x,y)$ is reduced from $k$ to $m$ ($k>>m$). To provide some insight into the computational efficiency attained by the proposed data decorrelation scheme, the dimensionality of the unprocessed hyperspectral data, $k$, is 80 (i.e., consists of 80 spectral bands), and our earlier studies revealed that optimal classification results are obtained when the hyperspectral data is decorrelated using eight fuzzy sets (i.e., $m=8$) (for more details please refer to Ref. 9 where a comprehensive set of experimental results are provided).

Fig. 4

Calculation of the membership grade for the spectral band ${\lambda }_i$. In this diagram ${\lambda }_D$ defines the half-width of each fuzzy set.

3.2.

Background Segmentation and Particle Labeling

The next step of the classification procedure involves the segmentation of the decorrelated spectral data into background and nonferrous particles. As indicated in our earlier paper, where the mechanical design of the WEEE recycling system is outlined,²⁴ the WEEE delivery component (vibratory feeder) was carefully built to ensure a uniform placement of the nonferrous particles onto the belt of the conveyor without any particle overlaps.

Based on the hyperspectral characteristics of the nonferrous materials, it was decided to choose a belt that is coated with matte black material. The main reason behind this decision was to maximize the contrast in the hyperspectral domain between the nonferrous materials and the background information (conveyor belt) and to reduce as much as possible the occurrence of specular reflections due to the adherence of the small WEEE particles to the conveyor belt. By taking advantage of these characteristics, a sole decorrelated energy vector component (i.e., the energy component whose central wavelength corresponds to 510 nm) proved sufficient for robust background segmentation using an experimentally determined threshold. However, when the recycling system has been operated in an industrial environment, we have discovered that vertical scratches occurred on the conveyor belt due to the friction with nonferrous particles (see Fig. 5). Nonetheless, the occurrence of these vertical scratches on the conveyor belt has negative effects on the background segmentation process, as illustrated in Fig. 5(b), and to robustly eliminate these undesired features we devised a filtering approach in the Fourier domain by masking the contribution of the frequencies that are generated by the vertical scratches as shown in Fig. 5(d) and Eq. (2).

Eq. (2)

$$\begin{gathered} F(u,v)=I(u,v)H(u,v) \\ H(u,v)=\{\begin{array}{ll}0& if(u,v)\in \mathrm{\Omega }\\ 1& otherwise\end{array}, \end{gathered}$$

where $u$,$v$ are the spatial frequencies, $I(u,v)$ is the input image after the application of the two-dimensional (2-D) fast Fourier transform (FFT), $H(u,v)$ denotes the Fourier filter, $\mathrm{\Omega }$ defines the section of the Fourier spectrum associated with vertical scratches, and $F(u,v)$ is the filtered image. The filtered image $F(u,v)$ is converted to the spatial domain using the inverse FFT, and the image resulting from the background segmentation process [see Fig. 5(f)] is subjected to a computationally optimized labeling procedure. Figure 6 illustrates the image resulting from the labeling process.

Fig. 5

Background segmentation process. (a) Input image where vertical scratches due to the friction between nonferrous particles and conveyor’s belt are visible. (b) Background segmentation. (c) Fourier spectrum of image (a). (d) Fourier filtering (c). (e) Inverse Fourier transform (d). (f) Background segmentation.

Fig. 6

Nonferrous particle labeling.

3.3.

Calculation of the Particle Spectral-Spatial Descriptor

After the application of the particle segmentation and labeling process, the next step involves the calculation of the spectral descriptor for each nonferrous particle. In our initial studies we have analyzed the feasibility of using pixel-wise spectral information for material classification, but the experimental results indicated a classification success rate of only 71.52% when dealing with six nonferrous fractions (aluminium, copper, brass, lead, stainless steel, and white copper). This classification accuracy clearly indicates that the spectral information provided by a single pixel is not sufficient to robustly model the significant intraclass dispersion caused by the various levels of oxidization that are characteristic for each nonferrous material. Another important factor that increased the intraclass dispersion was generated by the specular properties of the nonferrous materials and by the shadows caused by the three-dimensional (3-D) geometrical profile of each particle. While the issues inserted by the various oxidization levels can be (at least theoretically) modeled in a statistical sense, the incidence of the problems caused by the specular highlights and shadows cannot be predicted as they are caused by external factors. To alleviate the problems introduced by shadows and highlights we normalized the intensity of the decorrelated spectral information as illustrated in Eq. (3) (for more details regarding this hyperspectral intensity normalization scheme the reader can refer to Ref. 33).

Eq. (3)

$$E_{\mathrm{norm}}({\lambda }_j)=E_n({\lambda }_j)-\underset{i\in [1,m]}{\min }[E_n({\lambda }_i)],$$

where $E_n({\lambda }_j)=\frac{E({\lambda }_j)}{\sum _{i=1}^{m}E({\lambda }_i)}$and $m$ denotes the number of fuzzy sets that is a parameter of the spectral decorrelation scheme detailed in Sec. 3.1.

However, as indicated earlier, the most important problem we confronted is the inability of the pixel-wise spectral features to properly sample the characteristics of the nonferrous materials, and a solution to this problem resides in the integration of the spatial and spectral features in a composite descriptor. Using this concept, the spectral characteristics of the nonferrous materials are sampled by the distribution of the spectral features that are calculated over a region in the image. In the context of the application detailed in this paper, we propose to calculate the distribution of the spectral-spatial information using a fuzzy-based method similar to the approach used for data decorrelation (see Sec. 3.1). This approach is motivated by the fact that several factors such as image noise, uneven illumination, and different levels of oxidization of the nonferrous materials induce undesired and unpredictable changes in the calculation of the SSDs. To alleviate these issues we mapped each component of the normalized spectral energy vectors $E_i$ into a new fuzzy-based representation. The process required to calculate the fuzzy distributions (histograms) entails two steps that are depicted in Figs. 7 and 8. In the first step, the intensity of each energy vector $E_i$, $i\in [1,m]$ is subjected to a quantization procedure where each spectral component is projected into a higher dimensional vector space as illustrated in Fig. 7. In this process, $p$ fuzzy sets are employed from which only two will return a value different than zero. In the second step, the SSD is calculated as shown in Fig. 8. As indicated in Fig. 8, for each pixel $(x,y)$ the quantized vectors $Q_i$ for each energy $E_i$ are concatenated into a pixel-wise feature vector $Q(x,y)$, while the SSD for a given region $R$ is calculated as follows,

Eq. (4)

$$\mathrm{SSD}(R)=\sum _{\forall (x,y)\in R}Q(x,y)\mathrm{.}$$

The calculation of the SSD, as illustrated in Figs. 7 and 8, has several advantages that are well suited for the application targeted in this paper. First, the vector quantization procedure applied to measure the response of each energy $E_i(x,y)$ allows the construction of soft histograms that avoid the disadvantages associated with crisp binning (i.e., as in the case when the spectral-spatial histograms would be calculated using the $E_i$ values). This is particularly obvious in the case of material classification, as the properties of different nonferrous fractions vary smoothly within the hyperspectral domain. Second, another important advantage resides in the fact that the compound SSD involves the simple summation of the quantized spectral response that is calculated for each pixel in the region of interest $R$, which opens the opportunity of real-time operation. While the nonferrous particles have various sizes, prior to the classification process, each component (bin) of the SSD is normalized with respect to the number of pixels contained in the region $R$.

Fig. 7

The fuzzy-based vector quantization procedure ($p=6$ fuzzy sets are applied to sample the decorrelated hyperspectral domain).

Fig. 8

Calculation of the spectral-spatial descriptor (SSD).

4.

Experimental Results

The developed machine vision system consists of four distinct components: the particle feeding device, conveyor, material classification, and particle separation mechanism. The shredded WEEE mixture is automatically loaded onto a nonspecular black conveyor belt (600 mm wide) via a vibratory feeder that has been specifically designed to ensure the nonferrous materials are arranged into a thin layer prior to their arrival at the inspection line. The conveyor speed was set at 20 m/min, and the particle separation mechanism was implemented using a pneumatic part-extractor.

The WEEE mixture is defined by six nonferrous materials: white copper, aluminum, stainless steel, brass, copper, and lead (see Fig. 9). The material samples have been provided by Indumetal Recycling S.A. and IGE Hennemann Recycling GmbH, both members of the SORMEN project consortium.³⁰ The nonferrous materials have been manually sorted by expert operators, and full information was provided about each nonferrous waste fraction. In this study the captured datasets were divided into training and testing sets, where half of the data was used for training, and the remaining half was used for testing. From each of these datasets more than 500,000 SSDs were extracted for validation purposes.

Fig. 9

The nonferrous materials investigated in this study.³⁰

Table 1 shows the experimental results obtained by the nonferrous material sorting system detailed in this paper. The processing times reported in Table 1 were obtained when the proposed algorithm, and the algorithms described in Ref. 9, were implemented using the Microsoft Visual C++ environment (Windows XP) and executed on a standard PC that is fitted with an Intel Core 2 Duo 2.4-GHz processor and 2-GB RAM. To emphasize the computational advantages associated with the real-time classification technique presented in Sec. 3, we also provide the computational time attained by the classification method for which the SSD is calculated in predefined neighborhoods for each pixel in the image.⁹ As opposed to the approach presented in Ref. 9 where the classification process involves $N$ class assignments, $N$ being the number of pixels present in the image, in the implementation detailed in this paper the classification of each nonferrous particle resulting after segmentation and labeling (see Sec. 3.2) involves a single class membership assignment using Eq. (6). This is illustrated in Eq. (7) where we show the computational cost entailed by the method discussed in Ref. 9 ($C_{\text{neighborhood}}$) and the new approach detailed in Sec. 3 ($C_R$). In Eq. (7) $f_1$ defines the computational cost associated with the extraction of the vector $Q(x,y)$ (see Fig. 8), $M$ is the size of the neighborhood (see Table 1), $f_2$ is the marginal computational cost associated with the class assignment for the unknown SSD, as indicated in Eq. (6), and $c$ denotes the number of classes.

Table 1

Classification results.

5.

Conclusions

The major objective of this paper was to detail the development of an industrial compliant nonferrous material sorting system. In this paper we have reviewed the practical issues associated with the WEEE recycling process, and we have analyzed the challenges that have to be addressed during the development of an integrated nonferrous material sorting system when operated in an industrial environment. Among many practical and theoretical challenges, the computational efficiency and the accuracy of the nonferrous classification technique are critical requirements that justify the automatic recycling process from an economical standpoint. To answer these requirements, in this paper we have introduced a redesigned classification scheme in the hyperspectral domain, where the main novelty resides in the optimized computational approach that allows the real-time calculation of the SSDs in the context of nonferrous material sorting. The experimental results reveal that the proposed machine vision system is able to process the nonferrous shredded WEEE scrap at a rate of 2.28 m/s with a classification accuracy of 96.87%. The performance attained by our system exceeds the economic thresholds required for automatic WEEE nonferrous material sorting, and currently the developed machine vision system is fully evaluated in an industrial recycling environment.