Vision-based techniques for automatic marine plankton classification

Linear discriminant analysis (LDA) is a popular supervised classification technique that assumes a normal or Gaussian distribution for data points and identical covariance matrices for each class. It is also known as Fisher’s LDA, as Ronald A. Fisher developed it in the 1930s. LDA aims to find the projection hyperplane that maximizes the distance between the projected means of the given classes and minimizes the variance within the classes; this can be stated as a generalized eigenvalue problem in a multiclass scenario. If c is the number of training classes, we can find \(c-1\) discriminatory directions that separate the c classes as much as possible (the largest eigenvectors, Fig. 3).

Uhlmann et al. (1978) designed a pattern recognition system for phytoplankton. For image acquisition purposes, they built a custom camera device (vidicon) to scan either phytoplankton photographs or species preserved in formalin. After performing contrast enhancement, the control unit transferred the scanned images to a computer (Robotron KRS 4200), which processed the patterns into a 2-D power frequency spectrum, defining a position in the polar space that was independent of translations and rotations. Two years later, Schlimpert et al. (1980) published the details of this work. To construct a linear classifier, they prepared a set of training patterns comprising five classes, which were iteratively presented to the classifier while tuning its parameters. As this technique produced unsatisfactory results, they combined two interlacing classes, trained the classifier again with four classes, and separated these combined classes in a posterior second stage. They obtained a 93.2% correct classification rate when categorizing the five classes. Even with this remarkable result, they expected that a better classification accuracy could be obtained if the scan resolution was improved. The 2-D power frequency spectrum processing step took approximately 10 s per pattern, and the classifier converged between 3 and 170 s, depending on the class. Nevertheless, the authors were aware that higher speeds could be attained by using parallel processing. Given the resolution of the scanned images shown in their paper, this work was a remarkable effort at the time. However, LDA is not the most appropriate method when some input variables are correlated, as shown in this work.

Jeffries et al. (1980) also employed LDA but instead categorized zooplankton. The idea was to classify species into ecologically and taxonomically meaningful groups. To this end, they identified which species were viable for classification according to morphometric relations. They employed two datasets: photocopied plates of species obtained from published literature and contours of preserved specimens traced by hand on acetate sheets; the first was captured with a vidicon, and the second was obtained with the Bausch and Lomb QMS system. The first dataset was classified into six shape categories using LDA by the jack-knife procedure, yielding a 93.1% correct classification rate. The second dataset was classified into 19 major species using the LDA-based nonpooled covariance procedure, reaching a 97.9% correct classification rate due to the combination of four morphometric relations. This work marked the beginning of automatic zooplankton image classification, achieving an 85% time reduction compared to manual processing in routine laboratory work.

Four years later, Jeffries et al. (1984) built a custom vidicon system to capture better plankton images. As a novelty, six satellite microprocessors were employed for feature extraction. These microprocessors worked in parallel under a central computer, obtaining an 89% correct classification rate when categorizing eight groups with the same five features as those in their previous work (Jeffries et al. 1980). Katsinis et al. (1984) improved this classification process by employing eight satellite microprocessors to extract nine morphometric features, obtaining a 92% correct classification rate in less than half the time required by Jeffries et al. (1984). These studies employed custom edge detection techniques to erase all zooplankton antennae and swimming legs, which led to problems when performing feature extraction. Although the plankton group classification results were promising, species identification was still impossible due to the poor quality of the input images. Additionally, it was necessary to manually check that no organisms overlapped before scanning. These works are noteworthy due to the star architectures of their central computers and satellites (the microprocessors), demonstrating how easy it is to implement an LDA classifier. However, as stated before, an LDA classifier may not be the most appropriate classifier when similar plankton species exist.

Hierarchical clustering (HC) is an unsupervised classification algorithm that organizes a group of unlabelled data points into a tree-shaped structure (called a dendrogram). The dendrogram can be built in a bottom-up manner, in which all data points are initially considered as single clusters that are iteratively merged until only one cluster remains, or in a top-down manner, in which all data points initially belong to a single cluster that is recursively split until each cluster includes only a single data point. These approaches are named agglomerative and divisive HC, respectively (Fig. 4). HC gained notoriety after the publication of the "Numerical Taxonomy" book (Sneath and Sokal 1973), where the authors presented the advantages of HC in the field of biology and its application to organism classification.

Chehdi et al. (1986) from ENST Bretagne collaborated with the IFREMER in Brest using an agglomerative HC approach to classify 16 groups of zooplankton (Fig. 4A). A camera installed on a microscope captured images, and Freeman’s chain code was used to encode the contours. The appendages were removed to focus on bodies, separating those that seemed to overlap. They tested this system on 320 specimens by employing ten morphometric features; the algorithm could only find specimens falling into eight biological groups. Although this result was not the best, biologists liked how the approach gathered specimens, but improvements in image quality were mandatory for better classification. Six years later, Chehdi and Coquin (1992) optimized this method to classify 14 groups of zooplankton. As a novelty, they preprocessed the input images to resolve their problems concerning shadows, lights, and high-frequency noises. Seven features, five morphometric and two inertial features, were extracted to orient the specimens. They successfully classified 280 specimens into 14 groups, obtaining a 78.8% correct classification rate. As a drawback, this method required manual supervision to control the grouping at each aggregation level.

Thonnat and Gandelin (1988) applied a divisive HC method to classify Mediterranean zooplankton and their developmental stages for the first time (Fig. 4B). After performing photogram digitization and organism detection, they extracted morphological, densitometric, external and internal parameters. Then, marine biologists defined 59 prototypes containing descriptions for each class and subclass of zooplankton. Finally, they proceeded classify the plankton with 66 conditional rules, where each image took 122 s to process. They tested this classifier with only 40 zooplankton images but provided no information on accuracy, stating that they were working on constructing a more extensive dataset. This plankton application resulted from extending their previous work (classifying galaxies with HC) to other areas. While applying HC to plankton classification seems interesting, it may not be the most appropriate method because it does not use prior knowledge about the relationships between the different species. However, the employed tree-like structure provides a valuable visual representation, helping scientists understand how different species are related to one another and enabling them to identify patterns.

To mimic the architecture of the human brain, artificial neural networks (ANNs) were introduced in the 1940s. ANNs are composed of artificial neurons. Each artificial neuron has inputs and generates a single output, which can be provided as an input to other neurons. These connections, such as the synapses in a biological brain, can transmit a signal from one neuron to another. A weight value increases or decreases the strength of the signal traversing a connection. A typical ANN organizes its neurons into several layers: one input layer and one output layer with hidden layers between them. The weight values are usually adjusted by minimizing the difference between the processed output of the ANN and the target output. This training procedure is the most relevant aspect of forming a reliable model.

Simpson et al. (1991, 1992) published the first work that applied a ANN for plankton identification. Three years later, Culverhouse et al. (1994) employed the same network but increased the number of categories from two to five congeneric species, which were difficult to classify even for taxonomists. The topology of the ANN was a 15-3-5 fully connected network (every neuron in one layer was connected to every neuron in the next layer). Backpropagation was used to tune the weight values of the ANN based on the error rate obtained in the previous iteration. The employed dataset was the same as that used by Williams et al. (1994), which was comprised of digitized photomicrographs that were validated by six experts into two subdatasets: one for training and another for testing. After generating Fourier spatial frequencies as features, they ran several trials with a randomized dataset size. The minimum obtained mean test error was 23% on unseen images. Species that were difficult to classify by experts were also difficult to classify with the ANN due to how bad the Fourier components of these species were, as clearly seen in Fig. 5. This motivated the authors to look for other preprocessing techniques in conjunction with adjustments to the Fourier technique and their ANN. The authors knew that there was room for improvement because Williams et al. (1994) proved that this dataset was discriminable with 95% confidence by conducting a multivariate statistical analysis on the Fourier-transformed data. This work is exemplary for understanding how ANNs can be sensitive to data quality, performing poorly when the input data are noisy or contain outliers.

Ellis et al. (1994) also employed the same technique used by Simpson et al. (1991, 1992) to classify four plankton species. As a novelty, they combined the Fourier spatial frequencies with the internal textures of the plankton bodies extracted using the Gabor function. They achieved a 73% accuracy rate using these combined features, but the accuracy was only 69.8% or 43.7% when using only the Fourier-transformed spatial frequencies or the texture features alone, respectively. This ground-breaking work was the first to look at the advantage of combining both types of features in the field of plankton classification. However, as shown in their paper, image preprocessing is crucial for attaining high accuracy, making this technique very time-consuming for plankton classification and other applications.

Culverhouse et al. (1996) developed an automatic system for classifying 23 species of dinoflagellates identified as having negative impacts on the aquaculture industry. They first digitized the specimens archived at the Plymouth Marine Laboratory and fresh samples collected by the Spanish Institute of Oceanography. Afterwards, these samples were labelled by experts to create a database. Then, they determined the edges using the Sobel operator, where 60 features were extracted, including shape and texture features. Later, they compared the classification performance of two ANNs with that of two classic multivariate statistical techniques. The radial basis function (RBF) ANN performed the best with an 83% classification accuracy compared to that of taxonomists, who achieved an 85% rate. The authors highlighted these results because the classifiers were trained and tested from field-collected specimens instead of cultured specimens. While this system could not handle overlapping specimens, detritus-contacting samples did not appear to affect the accuracy of classifier because they were much smaller than plankton. This system evolved slightly, and in 2000, the software was designated as dinoflagellate categorization by an ANN (DiCANN) (Culverhouse et al. 2000). Although taxonomists slightly outperformed the ANN, this study evaluated how ANNs were superior to other discriminant methods, even to taxonomists, given that their better accuracy could fade away with psychological factors such as short-term memory, fatigue, boredom, recency effects, and positive bias. The potential sample analysis time reduction from human taxonomists to an ANN was from 120 to 5 min. Additionally, this work confirmed that data preprocessing is important for ANNs; therefore, the authors had to develop a specific preprocessing technique to manage debris.

Tang and his colleagues (Tang and Stewart 1996; Tang et al. 1998) produced a pattern recognition system to classify the massive quantity of images collected with the Video Plankton Recorder (VPR) (Davis et al. 1992). First, a video processing system rejected objects that were out of focus. Next, the system extracted features with three methods: moment invariants, Fourier boundary descriptors, and granulometric features; the three techniques grabbed shape information, while the latter method also acquired texture information. Later, the authors removed redundant information by reducing the feature vectors using the Karhunen-Loeve transform (KLT). Finally, they applied the supervised learning vector quantization (LVQ) ANN classifier to six classes of plankton. Due to the time-consuming nature of the classifier, they used a parallel training strategy. Regarding the results, they focused on which feature or feature combination gave the best accuracy instead of studying which were the best network parameters, obtaining a 92.2% accuracy on the test set using the combined feature vector. This work demonstrated the benefit of using granulometric features alone or in combination with other features, as shown in Table 1 of their paper, which helped counteract the noise, occlusion, and variability of the dataset. It is also worth noting the innovative parallel strategy used to reduce the training time of the LVQ algorithm.

Ellis et al. (1997) developed an alternative method for classifying six phytoplankton species. It combined the results of different individual classifiers into one result with an arbitration algorithm; this approach was established based on the idea that a group of experts produces a more accurate classification than an individual expert. They named their technique "classification by committee". First, they described the image textures with a Gabor filter, which enabled the input image to be segmented into a background and foreground. Then, five different shape and texture feature vectors were extracted from the foreground and individually employed to train five ANNs, obtaining five different classification performances. These five different results were combined with three algorithms based on several criteria: the majority of votes, the sum of votes, and the most confident network, where each network had a single vote. Among the three criteria, the most confident network combination offered a maximum accuracy of 84%. Finally, the authors compared these “classification by committee" techniques with another method called “classification by collective machines", which combined the five different obtained results with an ANN instead of criteria. This latter method slightly outperformed “classification by committee” (70.3% versus 68.7% accuracy, respectively). In all cases, this work showed that combining networks’ results with a criterion or an ANN resulted in higher accuracy than picking the best single ANN, but this was only true when the individual ANNs had comparable accuracies. This work tried to increase classification performance based on perspectives other than those used in previously reviewed papers. Although these techniques were innovative for plankton classification, the authors were inspired by prior works (Jacobs et al. 1991; Wolpert 1992; Battiti and Colla 1994).

Two years later, Dollfus and Beaufort (1999) applied position-normalized images directly on an ANN instead of using feature vectors. The authors employed free weights because neurons specialized better in specific places (in normalized images), increasing the probability of correct classification. Thus, they obtained an average correct classification rate of 86% when categorizing 13 species of calcareous nanoplankton. This classification system was named SYRACO2, proving helpful for natural forms with high intravariability. In addition, they found that algorithms performing position normalization needed to be more robust. Therefore, in 2004, they employed parallel ANNs (Beaufort and Dollfus 2004) in a system capable of categorizing nonposition-normalized images. Thus, the images were presented directly to the ANN (vision module), and in the case of weak identification, a second ANN (motor modules) performed up to five actions on each image (translation, rotation, dilatation, contrast, and symmetry) that helped to confirm or deny the initial weak identification result. This updated system achieved 91% accuracy in classifying 11 categories, with a speed of 7 s per image. It is notable that in this work, the authors attained a better accuracy by confronting the weak points of ANNs: the complicated preprocessing step and their effectiveness with correlated inputs. The direct image input process used by the network was similar to the procedure employed in a convolutional neural network (CNN).

Hu and his colleagues (Hu and Davis 2006; Hu 2006) presented a new system employing two classifiers: the first was an ANN using shape-based features (the same as in Tang and Stewart (1996); Tang et al. (1998)), and the second was a support vector machine (SVM) using texture-based features (the same as that in Hu and Davis (2005)). Then, both classifiers’ outputs were combined, resulting in a final correct identification if both agreed to the same class or an unknown result otherwise, as shown in Fig. 6. This system achieved 64.7% accuracy compared to the 61% accuracy attained by Davis et al. (2004) on the same dataset using an LVQ ANN classifier. The dataset was collected with the VPR and comprised seven major categories. This dual classifier performed better for taxa with low relative abundance than a single classifier fed with a unique feature vector containing stacked shape and texture information. Although this work employed a different approach for combining classifiers’ results than the “classification by committee” technique of Ellis et al. (1997), they both documented that when using classifiers in parallel and merging their results with any strategy, a broader spectrum of image details can be covered, thus resulting in higher accuracies.

Support Vector Machines (SVMs) were introduced in the 1990s. The aim of an SVM is to find the hyperplane that classifies the given data points and has the maximum margin (the maximum distance between the data points of both classes). Contrary to LDA, an SVM makes no assumptions about the data at all. Moreover, if LDA produces an analytical solution, SVM-based classification is an optimization problem. An SVM does not make use of the entire input dataset. The subsets of data points that are closest to the hyperplane are called support vectors, and the SVM is optimized over these vectors. Then, the support vectors determine how the SVM discriminates between classes.

Luo et al. (2003) published the first work applying an SVM to classify the massive quantity of images collected with the Shadow Image Particle Profiling Evaluation Recorder (SIPPER) (Samson et al. (2001)). After performing noise suppression, they calculated general and specific descriptors to produce 29 features. Due to the binary nature of the images, they lacked texture information. They classified two datasets comprising six plankton species with a soft-margin SVM. The first was classified with 90% accuracy, and the second was classified with 75.1% accuracy. The low accuracy achieved on the second dataset was due to insufficient image quality and unknown objects that could not even be manually identified. To speed up the feature calculation process without compromising accuracy, they published an expansion of this work (Luo et al. 2004a) where they described a selection process to reduce the number of features from 29 to 15. Hence, they obtained similar accuracies for both datasets attaining a processing time of 160 s for 6000 images with a one-versus-one approach. They acknowledged the need to develop new features for the higher-resolution version of SIPPER (25mm p/pixel). It is remarkable how the authors wisely picked a soft-margin SVM to cope with the noisy nature of the dataset. A soft-margin SVM tries to maximize accuracy while allowing some misclassification.

Hu and Davis (2005) employed co-occurrence matrix (COM) features with an SVM to classify images captured with the VPR. Co-occurrence matrices are textural features that describe the frequency at which one grey tone appears in a specified location in relation to another grey tone on an image (Haralick et al. 1973). Thus, their system computed eight COMs per image to perform posterior classification with an SVM. They obtained a 72% accuracy compared with the 61% accuracy of Davis et al. (2004). Both approaches classified the same dataset comprising seven categories, but the latter with an ANN was obtained from Tang and Stewart (1996). In another attempt, the authors combined texture features with shape-based features as classifier inputs, obtaining a mere 1% accuracy improvement over that achieved with COM features alone. After several tests, they concluded that the main accuracy improvement was achieved due to the COM features instead of the nature of the classifier, confirming that texture-based features are ideal for classifying field-collected images. This work highlighted that features should be selected depending on the given image’s nature (captured in the laboratory or in situ); this step is more critical than selecting or tuning a specific classifier. Moreover, a COM has the advantage of low computational requirements. The COM-SVM technique reduced the manual identification process from weeks to several hours.

Lisin et al. (2005) tried using a new method to deal with plankton images in which occlusion and clutter were present. First, they computed global features based on shape and texture information and local features based on the scale invariant feature transform (SIFT). Then, the features were independently classified; the global features were classified by an SVM, and the local features were classified by a nonparametric density (NPD) classifier, attaining 55% and 52% accuracy, respectively. Later, the authors tested two methods for combining the previous classifiers’ outputs: stacking and 2-tiered hierarchical classification. They obtained the highest accuracy (65.5%) with the stacking combination method on a 14-category dataset captured with the VPR (Fig. 7). Stacking consisted of concatenating the outputs of different classifiers (in this case, the SVM and NPD outputs) and using the result as an input vector (meta-features) for a meta-classifier (an SVM), which produced the final output. As in previous works (Ellis et al. 1997; Hu 2006), the authors demonstrated that combining different classifier outputs resulted in higher accuracy. Nevertheless, they stated that stacking was rarely exploited in artificial vision applications. Although this ensemble technique delivered a low accuracy, it was something new in plankton classification.

Wang et al. (2006) also benefited from COM features, but this time they operated at the geometric and pixel levels instead of the texture level. First, they searched for the points of interest for use as the local extraction centres with a Harris detector. After performing normalization, they computed a 128-dimensional SIFT descriptor. Later, they generated patterns with an associative rule mining algorithm, keeping the most discriminant features by a Pearson correlation analysis. The authors called this local feature extraction process local co-occurring patterns (LCPs). Finally, they classified five phytoplankton species with an SVM-based one-versus-one approach, obtaining an accuracy of 83%. They expressed how this result was better than the 72% accuracy obtained by Culverhouse et al. (2003a, 2003b). Nevertheless, this comparison was inappropriate because the authors here categorized five species instead of the six species used in the cited work. It is interesting to note how during the heyday of SVMs, the authors reviewed thus far tried to achieve improved accuracy by researching new features instead of dealing with SVM parameters, which led to novel visual representations such as LCPs.

Unlike previous SVM frameworks, Al-Barazanchi et al. (2015) decided to substitute the time-consuming step of feature engineering with a CNN as a feature extractor. Hence, they fed a 10-layer CNN with resized images possessing 32x32 pixels, which were comprised of seven plankton categories captured with the SIPPER. Then, they trained this CNN until the maximum accuracy was achieved. From the obtained network, they employed each hidden layer as a feature for training another classifier. Thus, they separately evaluated a random forest and an SVM classifier with two different training sets, the original dataset and the same dataset extended via data augmentation (random image rotation), and with every layer as a feature vector. Finally, they obtained the best accuracy of 96.7% with the SVM classifier and the augmented training set using the features of hidden layer 1. With this technique, they outperformed other works’ accuracies in terms of classifying the same dataset (Tang et al. 2006; Zhao et al. 2005, 2009, 2010; Li et al. 2014). This was the second reviewed paper, after the work of Dollfus and Beaufort (1999), that replaced the cumbersome feature extraction step with another original technique, allowing for the quick addition of new plankton species.

In a dataset, a plankton category contains dozens of images of the same species in multiple postures and positions. Nevertheless, a model requires even more images to attain a high classification accuracy. Some works, such as the previous study described above, employed data augmentation, which consists of artificially increasing the dataset's size by applying various transformations to the already labelled data, such as rotation, translation and flipping. Cheng et al. (2020) presented a new method to make a classifier invariant to plankton rotation, thereby eliminating the need for data augmentation. They first described each image with a polar representation, which consisted of depicting each pixel in terms of its distance and angle to the centre of the image, converting the rotational problem into a translational task. Thus, there exist classifiers such as CNNs that are robust to small translations due to their internal architectures (pooling layers). The combination of polar representations and their use in a CNN is a powerful technique for preventing the misclassification of two similar plankton images when one appears rotated relative to the other. With this idea in mind, the authors inputted each original image into a CNN and inputted its polar representation into another CNN. They trained each CNN independently and merged the outputs of their fully connected layers as the features for a subsequent classifier: an SVM (Fig. 8). Finally, the SVM obtained a 98% accuracy in terms of categorizing seven plankton species captured with PlanktonScope. This technique even outperformed a DenseNet CNN employing augmented data with 14 times the original size, which achieved 96.5% accuracy. This work solved one of the major issues of underwater artificial vision applications: the misclassification of similar objects captured at different angles. While polar representations were employed for plankton in the past (see the first reviewed paper, Schlimpert et al. (1980)), the authors presented an advancement based on the properties of CNNs. This framework can be valuable if combined with other techniques that address the problem of unbalanced categories in a dataset.

Semiautomated classification aims to improve the accuracy of a classification model by combining the strengths of machines and human intelligence. Thus, a machine performs some tasks while a human performs other tasks, such as the initial grouping of data and the final classification decision. However, the requirement for human interaction is kept to a minimum. This approach is advantageous when labelled data are scarce, or new images are complex for a machine to label. This method’s application dates to the early 1990s when researchers needed to perform simple classification tasks, such as sorting images into categories, with some input and guidance from human operators.

Luo et al. (2004b) updated their previous SVM classification approach (Luo et al. 2004a) by adding active learning. First, they built a multiclass SVM based on a training set and an unclassified set. Then, they ran the model, and an expert labelled the images from the unclassified set that were classified with the lowest confidence. Later, these manually labelled images were added to the training set and the model was retrained. This procedure was iterated several times until the desired accuracy of 86.7% was achieved. The main advantage of this method was that it required four times fewer images to produce the same accuracy as that of the classifier built on a random training set, as shown in Fig. 1 of their paper. Additionally, the iterative retraining process saved valuable time for the operators because it only requested labels for the most challenging images. This method is a reliable way to build a plankton dataset.

Grosjean et al. (2004) tested different automatic classifiers to categorize 29 groups of zooplankton. These images were acquired with ZooScan (Gorsky et al. 2009), a system built to scan biological samples. Among the 15 tested classifiers (such as LDA, HC, ANN), they obtained the highest accuracy of 74.6% with a double-bagging LDA classifier employing 27 textural and shape features. Nevertheless, the authors thought they should try another approach to attain higher accuracy due to species intravariability and the likelihood of training set mislabelling. Thus, they implemented a system to manually retag the items classified with low accuracy, improving the performance to 85% with a speed of 10,000 items in less than 5 s. The authors confirmed that human intervention is essential for managing the complexity of categorizing many species.

Bell and Hopcroft (2008) assessed the open-source ZooImage software on Alaska zooplankton to determine if it would be reliable for future research on salmon survival rates. ZooImage was based on the R project and the ImageJ software developed by the National Institutes of Health of the U.S. government to process biomedical images (Rasband 1997; Abràmoff et al. 2004). First, they scanned the samples using a conventional flatbed desktop scanner (Epson Perfection 4990). Then, the authors used the default ZooImage configuration, which created vignettes of every particle (segmentation) found on the image for posterior feature extraction. In this study, no information about the type of features employed by ZooImage was included. However, they probably consisted of textural and shape features, as employed in other works using ZooImage (Table I, Fernandes et al. (2009)). Next, they created a training set and evaluated six classification algorithms: LDA, a recursive partitioning tree, K nearest neighbours, LVQ, an ANN, and a random forest (RF). They obtained the best accuracy of 81.7% with the RF algorithm by categorizing 53 species on field-collected samples, reducing the time required for quantitative analysis from hours to less than one hour. When they removed the discarded category, which accounted for more than 75% of all particles, the accuracy fell to 63.3%. Although they considered that the results were valuable, they anticipated that a semiautomated procedure was required to attain higher accuracies, expecting its inclusion in future software updates. As mentioned in this section’s reviewed works, the semiautomated method is regarded as a technique for improving the accuracy of any model.

Gorsky et al. (2010) developed a new software for plankton classification: the Plankton Identifier (PkID). They performed a classification sequence composed of ZooScan, ZooProcess and Plankton ID. First, the images were acquired with ZooScan, as mentioned in a previous work. Next, they employed ZooProcess software (based on ImageJ software) to preprocess, segment, and extract features. Then, they manually created a training set with 30 categories, which was reduced to 20 due to the low accuracy shown on some categories in PkID cross-validation tests. Nevertheless, they kept some categories with low accuracy because of their ecological values. This reduction was made iteratively by merging categories with similar ecological values, resulting in higher accuracy (Fernandes et al. 2009). This training set achieved an overall accuracy of 78% using an RF algorithm. According to the authors, this performance was inaccurate for ecological studies because it is driven primarily by less abundant taxa. Thus, they decided to increase the number of categories from 14 to 42, assisting the machine with labelling when needed and resulting in 100% accuracy. The authors clearly showed how this semiautomated classification method is the best way to obtain the highest accuracy in taxonomic research at the expense of taking more time than the fully automated approach.

A year later, Ye et al. (2011) presented a semiautomated method based on a Bayesian probabilistic model. A Bayesian probabilistic model represents uncertainty about events using probabilities; it starts with a prior premise about an event (prior probability) and then updates that premise with new information (likelihood) to obtain a new probability (posterior probability). After performing ZooScan image acquisition, the authors manually categorized every object, building unbalanced and balanced training sets comprised of 28 planktonic and nonliving categories. The balanced training set had the same number of images in each class. In contrast, the unbalanced class had different numbers of images in various classes. ZooProcess extracted features, and a naive Bayesian classifier was trained, obtaining a relationship between the posterior probability and cumulative recognition accuracy for each category. New images were automatically classified based on the posterior probabilities, obtaining accuracies of 69% and 68% for the unbalanced and balanced training sets, respectively. Second, over 31% of the images with the lowest confidence levels were manually reclassified by experts, obtaining 92% accuracy for both the unbalanced and balanced sets. This semiautomatic approach outperformed the method of Grosjean et al. (2004), and it offered the possibility of providing the results for any aggregation level with a single calculation step due to the Bayesian probability model. As stated in Bayesian probability model theory, this technique seems ideal for plankton classification because it can handle category intravariability, missing data in unbalanced categories, and noisy datasets. However, as embodied in the previously reviewed works, a semiautomated approach can raise any model to its maximum accuracy.

Bochinski et al. (2019) applied a variation of active learning to zooplankton for the first time: cost-effective active learning (CEAL, Fig. 9 (Wang et al. 2017)). Similarly, while active learning tries to maximize a model’s accuracy regardless of the required human effort, CEAL tries to avoid unnecessary human intervention, hence the “cost-effective" aspect of the name. First, images were segmented using Otsu thresholding, but only when rotation was needed because the authors found that segmentation techniques removed plankton details. Then, they employed a CNN based on AlexNet and tested this method with two databases comprised of four zooplankton species acquired with the In Situ Ichthyoplankton Imaging System (ISIIS) (Cowen and Guigand 2008). As a result, CEAL achieved the highest accuracy compared to the bare CNN and other active-learning methods; therefore, CEAL should be considered for future developments focused on semiautomated approaches.

Convolutional Neural Networks (CNNs) are extensions of ANNs that can take an image as input. The name is derived from convolution, which is an operator that can merge two matrices by multiplying them. When correctly employed, convolutions can be used to perform edge detection, image blurring or sharpening. The topology of a CNN considers multiple convolution layers to extract key image features. Interspersed with these layers, a CNN includes pooling layers, which are in charge of summarizing the features within a group of cells in the previous layer. Thus, the CNN also compresses the input data further. The final layer of the CNN serves as an input for an ANN to conduct pattern classification. CNNs were first introduced in the late 1980s by Yann LeCun and his colleagues at the Pierre and Marie Curie University in Paris and AT &T Bell Labs. They have become state-of-the-art approaches in computer vision applications, including image and video classification, object detection, and image generation.

Members of the reservoir laboratory (Ghent University - Belgium) won the US National Data Science Bowl (NDSB) competition by classifying 121 classes of plankton acquired with the ISIIS (Dieleman et al. 2015). First, they performed zero-mean unit-variance (ZMUV) normalization and data augmentation. ZMUV normalization is a common preprocessing step used in machine learning to ensure that the given data possess a consistent scale and have similar properties. The authors combined two different rescaling strategies due to the species’ size diversity; one was based on image size, and the other was based on a fixed factor. For classification purposes, they employed a 13-layer CNN structure inspired by OxfordNet and an ANN employing the most discriminative conventional features, such as image sizes, image moments, and Haralick textures. Inspired by their previous participation in the Galaxy Challenge, they added cyclic pooling to the CNN to make the network more robust to the rotational plankton variations. Finally, they fused the results of these methods, obtaining a top-5 accuracy of over 98%, demonstrating their approach as a helpful tool for aiding experts in labelling tasks. They trained the models on Nvidia GPUs with convergence times from 24 to 48 h, depending on the model parameters. As a drawback, CNNs tend to overfit due to their complexity and the number of parameters relative to the training data, resulting in poor performance on new images. The authors overcame this issue for classes with as few as 20 samples per category by applying dropout, weight decay, data augmentation, pretraining, pseudolabelling, and parameter sharing techniques. In addition, cyclic pooling has become so powerful that neural networks centred on it have emerged, such as CyCNN and R2CNN. This work proved that high accuracy is possible with few samples, contrary to what is conveyed in deep learning theory.

Dai et al. (2016) presented ZooplanktoNet, a framework based on a CNN to classify a 13-class dataset acquired with ZooScan and provided by the Laboratoire d’Océanologie de Villefranche-sur-Mer, France. First, they normalized and augmented the dataset via rotation, translation, rescaling, shearing, and flipping. Then, they tested four popular CNN architectures on an augmented and nonaugmented dataset as reference points: AlexNet, CaffeNet, VGGNet and GoogleNet. The models’ accuracies achieved on the augmented dataset were approximately 10% superior to those of the nonaugmented versions. Then, the authors chose AlexNet to continue their experiments but tested different optimizations for this CNN: different network depths, sizes and numbers of convolutional filters and the advantages of using local response normalization (LRN) and rectified linear units (ReLUs). LRN is a preprocessing step that is commonly used in CNNs to prevent overfitting. A ReLU is an activation function that enables the learning of complex shapes and textures, improving overall model accuracy. The researchers obtained the best results with an 11-layer network, using LRN and parametric ReLU (PReLU) activation functions instead of the widely known ReLU function. The PReLU function resolves the dying ReLU problem and performs better, as shown in this work. Thus, this architecture was denoted as ZooplanktoNet, achieving an accuracy of 93.7% versus the 91.3% of the standard AlexNet at the cost of requiring ten more minutes of training time (a total of 38 min). The authors documented how a standard CNN such as AlexNet could be adjusted to obtain higher accuracies depending on the nature of the given dataset. Nonetheless, this model was designed specifically for images acquired with ZooScan, with unknown performance for images acquired by other means.

Xiu and Zuoying (2016) applied the residual CNN (ResNet, He et al. (2016)) that was recently developed by Microsoft Research to the 121 classes of plankton dataset employed in the 2015 National Data Science Bowl (NDSB) competition. The idea of this work was to overcome the accuracy decrease induced when increasing the depth of a CNN to 20 layers or more. After rescaling and augmenting the dataset, they trained three residual CNNs with depths of 19, 32 and 50 layers plus a VGG-19 network as a reference, keeping the remaining parameters the same. The best network was the 32-layer network, with a top-5 accuracy of 95.8%. Although this accuracy was lower than the 98% achieved by the winning team in the NDSB competition, its computational cost was stunning, running 6.5x faster at a rate of 0.1 s per test set image. In general, ResNets perform better than traditional CNNs because they can train much deeper networks without incurring the vanishing gradient problem. The vanishing problem has been overcome by allowing the gradients to bypass one or more layers (as shown in Fig. 4 of their paper); this new internal structure is called “residual connection”. Moreover, during inference, ResNets are expected to run slower than traditional CNNs because they must perform calculations with both the original and residual inputs. They also use more layers, increasing the required computation time. Contrary to the above, the authors achieved a lower accuracy and a faster inference time compared to the VGG-19 network used as a reference. Perhaps this behaviour was due to the small dataset size, which the authors blamed for the result obtained in the 50-layer network test.

One year later, Dai et al. (2017) presented a classification system composed of three CNNs in parallel. The idea was to take advantage of textural features that describe differences when shapes do not and vice versa. To extract shape information, they first removed noise, smoothed shapes, applied the Scharr operator and optimized the contrast. To obtain texture information, they employed a Canny edge detector. Then, three CNNs were arranged in parallel; the first was fed with shape features, the second was directly fed with the images, and the third was given textures, converging into a softmax function (final layer) by a pyramid structure, as shown in Fig. 10. They classified a dataset containing 30 plankton classes acquired with the Imaging FlowCytobot (IFCB, Olson and Sosik (2007)). After completing several tests, they obtained maximum accuracies of 95.8% when employing networks based on AlexNet and 96.3% when using GoogleNet. AlexNet is an 8-layer CNN introduced in 2012 by researchers at the University of Toronto. It won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 with a top-5 error rate of 15.3%. GoogleNet is a 22-layer CNN introduced in 2014 by researchers at Google. It won the ILSVRC in 2014 with a top-5 error rate of 6.6%. All tests were run on four 6-GB Nvidia GTX 980Ti GPUs. Although the authors tried to employ as much image information as possible, they could barely achieve a 1% accuracy improvement over a single CNN (either AlexNet or GoogleNet) fed with only the original images as input, as shown in Tables 2 and 3 of the original paper. Thus, this method runs contrary to the trend of avoiding complicated feature calculations, showing no benefits.

Cui et al. (2018) presented an alternative approach to that used in the previous work. The authors also combined original images with shape and textural features as classifier information. However, they obtained shape features by applying Gaussian low-pass filtering and textural features by applying Gaussian high-pass filtering followed by logarithmic image enhancement. Then, they merged the shape features, the texture features, and the original image into a more complex feature representation using a concatenation layer. Finally, the authors input the merged vector into a single CNN based on AlexNet (Fig. 11). This approach obtained 96.6% accuracy, outperforming the framework based on AlexNet from Dai et al. (2017) by 0.8%. Although this procedure did not achieve a noticeable accuracy improvement, its simplicity compared to the previous method makes it preferable for future framework developments.

Al-Barazanchi et al. (2018) applied a CNN classifier to the same dataset employed in past works comprising seven plankton species (Tang et al. 2006; Zhao et al. 2005, 2009, 2010; Li et al. 2014; Al-Barazanchi et al. 2015). After resizing all the dataset images, they fed the images into an 18-layer CNN (based on VGGNet), achieving 98.2% accuracy on the testing dataset. Their CNN outperformed the best result from previous works, which achieved 96.7% accuracy using a 10-layer CNN for feature extraction and an SVM as the classifier (Al-Barazanchi et al. 2015). This paper demonstrated how the unique use of a deep CNN outperforms any other method without data augmentation. However, it would have been interesting to compare this architecture with their previous work but instead using a CNN with the same depth as the feature extractor.

Luo et al. (2018) improved the parameters of the spatially sparse CNN formerly employed in the 2015 NDSB competition, which finished third in the final rankings. In a traditional CNN, each neuron in a layer receives input from all the neurons of the previous layer. In a spatially sparse CNN, each neuron in a layer receives input from a subset of the previous layer’s neurons through a process called pruning. With these connection reductions, the network first learns the most important features. Then, it continues to learn more complex features with less important connections. Researchers created this type of CNN to recognize Chinese characters, where strokes were considered sparse pixels and the background was omitted for processing purposes. In this work, the authors considered a plankton image to be similar to a handwritten character, where most of it was background. In this way, a spatially sparse CNN could learn faster, with less overfitting, and with a lower computational cost than that of a dense CNN. They classified the 2015 NDSB competition dataset by regrouping the training set into 37 classes according to ecological significance. After conducting contrast normalization and discarding noisy and low-confidence images, data augmentation was performed on the rare classes via rotation, skewing, and scaling. Their 13-layer spatially sparse CNN obtained an F1-score of 0.881 and a precision of 90.7% on nonrare biological groups (23 classes), taking approximately 24 h for training and 165 h for prediction on new images using 1536 Nvidia CUDA cores. The result was quite impressive, considering the dataset size of 23.4 million images. However, this model may not be suitable for general plankton images, as they have high variability in their texture and illumination conditions, requiring more connections among neurons to process complex images. Moreover, this technique may soon become obsolete with the emergence of colour plankton databases, which present high degrees of variability.

Soh et al. (2018) presented a CNN architecture capable of identifying individual plankton in images with homogeneous clumping, heterogeneous interspersion or both. Homogeneous clumping occurs when plankton of the same species cluster together homogeneously. Heterogeneous interspersion occurs when different plankton species are randomly distributed in a sample. These phenomena can affect the accuracy of plankton classification. Thus, the authors developed a 19-layer CNN based on the “You Only Look Once" (YOLO) architecture, which was capable of identifying and categorizing multiple objects in an image using just one computation. After conducting manual labelling, they performed data augmentation: six geometric distortions and two-pixel value modifications. Next, they applied this classifier to 18 species of plankton, detecting 100% of the validation set with an average intersection of union (IoU) of 86%. The IoU is a metric for object detection algorithms that measures the overlap between the predicted box and the ground-truth bounding box for an object in an image. This model took 20 h for training, running at 50 frames per second on the test set using one Nvidia GTX980. Due to the nature of the classifier, this method is a perfect fit for coping with scattered plankton, but distinguishing between clumped plankton and individuals is a different challenge, perhaps requiring extra preprocessing steps not covered in this work.

Lumini and Nanni (2019) tested a classification approach consisting of the fusion of different CNNs. They tested three plankton datasets on each of the following pretrained networks: AlexNet, GoogleNet, InceptionV3, VGG16, VGG19, ResNet50, ResNet101, and DenseNet. A pretrained network is a model that has already trained on a large dataset, the ImageNet database in this case. Then, the parameters of these models were adapted to the new dataset. For this task, the authors employed the sequential floating forward selection (SFFS) feature picking method, which selects the most informative features from a pretrained models that are useful for a new dataset. The retraining step was performed under three different strategies: one round of tuning (1R), two rounds of tuning (2R) and preprocessing tuning (PR). 1R consisted of fine-tuning a pretrained model with a new dataset in a single round, 2R uses two rounds, and PR involves fine-tuning a preprocessed new dataset in a single round. For PR, the authors implemented preprocessing by applying data augmentation with four techniques: gradient, orient, local binary pattern, and local ternary pattern transformations. The three tested datasets were already used in past studies; they were acquired with the IFCB, ZooScan, and ISIIS. The best single pretrained model was DenseNet, but the highest accuracy was obtained with an ensemble method that consisted of employing the sum rule among the eight previous networks using one round and two rounds of fine-tuning, as shown in Table 6 of their paper (Fus_2R + Fus_1R). In a comparison, this approach obtained an F-measure of 0.953 versus the 0.9 achieved by Zheng et al. (2017) on the same IFCB dataset. They published another study that outperformed this ensemble’s accuracy. It employed the SFFS feature selection approach but instead selected which classifier to add to the ensemble in each iteration based on its performance (Lumini et al. 2019). Thus, the SFFS ensemble consisted of 11 classifiers, obtaining an F-measure of 0.958 on the same IFCB dataset. As in the previously reviewed works, the authors revealed how an ensemble classifier could outperform individual CNNs. The use of pretrained models could be a powerful time-saving technique when the new target is a high-dimensional and complex dataset. Nevertheless, the complexity of running several networks to fuse their results later could be challenging. Further research may be needed to determine if the time required to implement this ensemble technique is shorter than that required to optimize a single CNN with layers adapted to the nature of the input images.

Li et al. (2021c) presented a new model that addressed two problems encountered in plankton classification: imbalanced categories and the loss of subtle features by a CNN during training. Imbalanced categories frequently occur in datasets for plankton classification tasks, where rare or poorly known taxa exist, negatively affecting a model's accuracy. Retaining subtle features, such as the shape of a head or the presence of appendages, is crucial for accurately classifying different plankton species. The authors solved the imbalance issue by creating fake images in the rare taxa categories using CycleGAN, a generative adversarial neural network that generates new images similar to those employed as inputs and described by Yann LeCun, Chief Artificial Intelligence Scientist at Facebook, as “the most interesting idea in the last ten years in machine learning" (LeCunn 2017). Then, the authors proposed a densely connected structure based on the YOLOv3 model to better capture the subtle features of plankton. In a densely connected structure, each layer is connected to every other layer, rather than just the previous and successive layers (as in traditional CNNs), allowing for a more robust representation of the input data. The authors applied this method to classify the IFCB dataset, obtaining an average precision of 97.1% for eight classes. This method outperformed others, with a detection time of 51 ms per testing image, which is suitable for real-time applications. Although the experiments were run in a desktop environment (Nvidia Titan RTX GPUs), they successfully integrated this method into a low-power Jetson Nano board, looking for its implementation in autonomous underwater vehicles (AUVs). This latest work marked a turning point in plankton image classification, presenting a model that not only offers high accuracy, but is also likely to be seen soon in an in situ device performing real-time plankton classification.