The automatic classification of marine species based on images is a challenging task for which multiple solutions have been increasingly provided in the past two decades. Oceans are complex ecosystems, difficult to access, and often the images obtained are of low quality. In such cases, animal classification becomes tedious. Therefore, it is often necessary to apply enhancement or pre-processing techniques to the images, before applying classification algorithms. In this work, we propose an image enhancement and classification pipeline that allows automated processing of images from benthic moving platforms. Deep-sea (870 m depth) fauna was targeted in footage taken by the crawler “Wally” (an Internet Operated Vehicle), within the Ocean Network Canada (ONC) area of Barkley Canyon (Vancouver, BC; Canada). The image enhancement process consists mainly of a convolutional residual network, capable of generating enhanced images from a set of raw images. The images generated by the trained convolutional residual network obtained high values in metrics for underwater imagery assessment such as UIQM (~ 2.585) and UCIQE (2.406). The highest SSIM and PSNR values were also obtained when compared to the original dataset. The entire process has shown good classification results on an independent test data set, with an accuracy value of 66.44% and an Area Under the ROC Curve (AUROC) value of 82.91%, which were subsequently improved to 79.44% and 88.64% for accuracy and AUROC respectively. These results obtained with the enhanced images are quite promising and superior to those obtained with the non-enhanced datasets, paving the strategy for the on-board real-time processing of crawler imaging, and outperforming those published in previous papers.
Notes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
The use of Machine Learning (ML) techniques by Artificial Intelligence (AI) is growing at a constant pace in several scientific fields, with applications in medicine [15], [55], agriculture [23], industry [53], and marine ecology [6, 7, 59, 81]. As a matter of fact, the implementation of AI-based algorithms for tracking and classification of animals in seafloor (i.e., benthic) realms has grown spectacularly in the past decade both in data from cabled observatories [60, 61, 71, 105], Remotely Operated Vehicles (ROVs) [65], [100] and Autonomous Underwater Vehicles (AUVs) [16], [101]. Thus, AI processing innovation is rather conditioned by the different operational scenarios, as for example the monitoring of deep-sea biodiversity and stock assessment [1, 2, 24] or ecosystem recovery from different disturbances [13, 75, 82].
Marine imaging acquisition
Although most of the surface of this planet is covered by seawater, a large part of its volume and projected seafloor remains unexplored [25, 40, 92]. This is principally a consequence of the particular characteristics of the ocean environment such as high pressure, low temperature and absence of light, which make it hostile to humans and pose practical challenges to its exploration [79]. However, advances in robotic platforms and sensor technologies have made possible to dive in virtually any realm of the deep marine biosphere [12, 33, 67], obtaining relevant information about many marine environments and their inhabiting species [3, 22, 41, 68]. Oceans provide us with goods and services that, if exploited without control, can be depleted [29, 99]. Therefore, much more effort is required for the acquisition of baseline knowledge on marine ecosystems in terms of species and their habitats characteristics, in order to promote sound and scientifically sustained management policies [27, 28]. This data acquisition is up to date with the framework of the UNESCO Ocean Decade Initiative (https://en.unesco.org/ocean-decade) (C. N. [64]. The study of imaging data provided by new robotic platforms is progressively playing a central role for fisheries [1, 2, 14], with evident industrial applications in terms of impact quantifications at off-shore decommissioning and projected mining activities [8, 43, 46], [65, 87].
Advertisement
During the last two decades, underwater imaging assets have increased with the improvement of High-Definition (HD) optics and the introduction of low-light equipment that, in association with acoustic multi-beam cameras, are presently allowing vision in darkness [3]. Complex sensor packages are being installed on cabled observatories and their docked mobile platforms such as crawlers (i.e., Internet Operated Vehicles, IOVs), combining oceanographic and geo-chemical assets with cameras [20, 20, 21, 21], [32, 50, 93]. This combined image and multiparametric environmental data acquisition is allowing to link visual counts of animals for the different species within a marine community with concomitant habitat quality changes, as an experimental field measure of their ecological niche [1, 2]. The technological developments are also paralleled by vessel-assisted robotic technologies such as AUVs and ROVs [17, 26, 81], [101].
Underwater image quality challenges
The automation of image acquisition and processing for animal tracking prior the classification is of relevance for the prolonged and continuous monitoring of marine biodiversity in any different operational scenario [6, 7, 10, 49, 81]. Although the use of better imaging technologies has made it possible to obtain HD outputs of increased quality, in almost all cases a pre-processing is necessary, due to strong environmental variability (i.e., “real world” scenarios, [61, 86]). While in controlled laboratory environments the bottom is static and with hardly any details [36, 37, 56, 88, 89, 103], in uncontrolled field settings such as the seas, images are acquired under variable environmental light or artificial lighting (for costal to deep-sea applications), floating particles and variable substrates as background [18, 31, 57, 62]. Such a variability represents a challenge in the identification and tracking of animals within the Field of Views (FOVs) and the extraction of their morphological features for classification [5, 57, 60, 61, 71, 85].
In this framework, image pre-processing is essential to improve animals’ detection and their posterior classification. Image enhancement encompasses both Computer Vision (CV) methods (i.e. noise removal, contrast change or colour adjustment: e.g., [30] and Deep Learning (DL) methods such as neural networks, capable of generating a new enhanced image from the original one [54], [96, 105]. However, given the amount of images collected, manual processing becomes infeasible. Therefore, an automatic process is needed to improve these huge datasets, either to analyse them manually or to use them later in a detection and classification process to obtain better results. In order to manage the processing difficulties of image treatment in real world case scenarios some authors [57, 88, 89, 105] established a pipeline based on different automated treatment steps toward quality enhancement at cabled observatories. Such an effort has not yet been done with their docked mobile platforms such as IOVs, although some work has already been developed for ROVs (e.g., [100] and AUVs (e.g., [16]. The images and videos obtained by crawlers in particular are composed of different scenarios, as IOVs are often in constant motion. Typically, multiple sediment particles can be seen in the images, as they are lifted by the movement of the crawler and can impede visibility [20, 21, 32].
Objectives
The objective of this paper is to design an automatic image enhancement process pipeline using Deep Learning techniques, in order to enhance the images and subsequently obtain better animals’ classification results. Based on previous experience at cabled observatories [57], this process can be applied to ROV and AUV data. This enhancement was carried out on images acquired by the deep-sea crawler “Wally” with innovative deep-learning techniques, as baseline condition for improved animal classification which is required by monitoring and conservation strategies [28]. This processing provides a solution to the problem of enhancing very dark underwater images, since existing enhancement and classification solutions are still too dependent upon high illumination levels.
Advertisement
The article is organised as follows: Sect. “Materials and methods” describes the image set used in this work, some existing methods for image enhancement, and the chosen evaluation metrics (both for image quality assessment and species classification). Sect. “Proposed methods” presents the proposed methods, including the image enhancement pipeline and the classification process. Sect. “Experimental design” details the experimental design used to carry out the different experiments. Sect. “Results” shows all the obtained results, first regarding image generation and then animal classification, while Sect. “Discussion” discusses the results. Finally, Sect. “Conclusions and Future Work” gives our conclusions and future work.
Materials and methods
The IOV and the study area
The crawler operates since 2009 at the Hydrates site (~ 870 m depth; 48° 18′ 46″ N, 126° 03′ 57″ W) of the Barkley Canyon Node of the North-East Pacific Undersea Networked Experiments (NEPTUNE) seafloor cabled observatory, operated and maintained by the Ocean Networks Canada (ONC; www.oceannetworks.ca). The site is a soft sediment plateau, characterised by the presence of outcrops of methane hydrates forming small mounds, with chemosynthetic bacteria forming thin mats suspect to erosion by currents [94]. A prevalent down-canyon current flow of intermediate velocity, i.e., seldom higher than 0.3 m/s [20, 21] and mainly SW direction can generate turbidity and phytodetritus fluxes [20, 21] with varying quantities of particles, potentially interfering with the FOV. The tidal regime is mixed semi-diurnal, with two unequal pairs of highs and lows [47], determining the image capturing protocol (see Sect. “Data collection”).
Data collection
The image capturing process is described in [20, 21]. In brief, a total of 18 imaging transects (i.e., 9 forth and 9 back) of ~ 30 m length were carried out between 2 November and 2 December 2016. The currents in the area are generally stronger than the crawler’s established speed (i.e., ~ 0.04 m/s), meaning that moving towards the same direction as the current would constantly place the entire sediment cloud in front of the camera. Moving against the current was an efficient strategy to avoid that, but did not impede the generation of sediment cloud, parts of which would occasionally interfere with the camera’s field of view.
Imaging was performed at 1 Hz (i.e., 1 image/s) with a Basler dart USB 3.0 camera (daA1600-60 μm/μc; 1600 × 1200 pixel), which was mounted on a structured light system (i.e., pan/tilt unit; PTU) developed by DFKI Bremen, with illumination provided by two 33 W, 2000 m rated LED lamps. For standardization purposes, angles were set to − 76° left for pan and − 2° up for tilt in all but one transect (for further details on camera calibration see Supplementary Table S2 in [20, 21]. The camera was facing towards the right of the crawler at all times, so that the background was different between the back and forth transects.
The targeted group of species
The main megafauna (i.e., animals with body size above 2–3 cm) from a total of 14 morphospecies (i.e., identified down to species level or to higher taxonomic ranking, based on general morphology) present in the dataset, were identified by visual inspection with the help of the NEPTUNE Canada Marine Life Field Guide in [38]. However, species for which few records were available were combined into one class which was later added. The remaining considered species were:
Demersal fish of the family Sebastidae. This group included rockfishes of the genus Sebastes and thornyheads of the genus Sebastolobus, which are mainly observed inactive on the seabed and are characterised by their orange-red colour pattern.
The blackfin poacher (Bathyagonus nigripinnis) a small, thin, dark coloured fish also often observed as inactive on the seabed.
The Pacific hagfish (Eptatretus stoutii), with a characteristic grey-violet colour and long, slender body, observed either laying on the seabed or swimming in sudden bursts.
The grooved tanner crab (Chionoecetes tanneri) of varying size between adults and juveniles, orange body and 4 pairs of walking legs, observed both as immobile or walking.
Sea stars of the class Asteroidea, appearing as white and stationary (compared to the temporal scale of each transect).
The ctenophore Bolinopsis infundibulum, transparent and drifting with the current flow at varying velocities or actively swimming across the FOV.
The jellyfish Poralia rufescens, orange-red and round-bell shaped, also moving with a combination of active swimming and drifting with the current.
Moreover, three classes were added a posteriori due to their recurrent occurrence: firstly, the class encompassing floating and resuspended sediment particles in water, because the movement of the IOV leads to their occurrence on countless occasions and the currents drag visible phytodetritus from shallower waters; on the other hand, the class containing transect plastic metric reference marks (for remote navigation) has been added, as they can be observed in many of the images; finally, the class which includes unclassified species, as too dark or too far in the FOV to be recognised, or because they belong to the group of species for which few records were available in the current dataset. All species and classes used can be seen in Figs. 1 and 2.
Fig. 1
An example of the species in the dataset: A Asteroidea, BChionoecetes tanneri, CBathyagonus nigripinnis, DEptatretus stoutii, E Sebastidae, FBolinopsis infundibulum, GPoralia rufescens
Fig. 2
An example of other classes added to the dataset: A floating particles and B transect plastic metric reference marks and C other species (unclassified or few records available)
×
×
Table 1 shows the number of samples per class (before data augmentation) considered for the generation of the datasets.
Table 1
Number of samples of species used for building the dataset for reference at automated classification
Species
Samples in dataset
Asteroidea
731
Chionoecetes tanneri
793
Bathyagonus nigripinnis
604
Eptatretus stoutii
651
Sebastidae
1102
Bolinopsis infundibulum
849
Poralia rufescens
748
Sand particles
520
Objects
656
Other species
318
Image enhancement methods
To enhance the images, a pipeline has been generated using different techniques to deliver an input for a neural network that would then generate properly enhanced images.
Although many techniques were finally discarded, it was necessary to carry out a preliminary survey on those techniques to seek for best enhancement results. The following list contains a brief description of each of the techniques that were used:
The Contrast Limited Adaptive Histogram Equalization (CLAHE) [106] reduces the problem of its predecessor, the traditional Adaptive Histogram Equalization [76], which tends to amplify the noise in constant and homogeneous regions.
Gamma Correction (or Gamma Encoding) is the non-linear operation to encode and decode the luminance values in images or videos, used to compensate human vision in order to maximize bit bandwidth in relation light/colour perception and details hidden in dark images can be appreciated [77].
Colour Balance (i.e., as white balance) is the global adjustment of colour intensities to correct the representation of neutral colours [9, 84].
Convolutional Neural Networks (CNNs) are a type of neural network models commonly used for image recognition and classification [51, 52].
Autoencoders are used as unsupervised learning neural networks and have three main components: the encoder, the code (also known as latent space representation) and the decoder [11, 39].
Metrics
To evaluate the improvements in image quality after applying the processes described above and also to evaluate the further classification, we selected a set of evaluation metrics.
Image quality metrics
For the evaluation of the images, the following image quality assessment metrics have been used:
In this subsection we present the image enhancement process in terms of composing steps and the description of the residual network that mainly constitutes this process. The complete pipeline can be seen in Fig. 3. The original images were in raw format and appeared in grayscale, so a chromatic interpolation algorithm by demosaicking or debayering, a digital image process used to reconstruct an image in colour, was applied.
Fig. 3
Underwater image enhancement Pipeline
×
Since the images still retained the greenish hue characteristic, we model and train a residual CNN network to generate the enhanced images and thus eliminate the greenish hue characteristic. Those neural networks are known for skip connections, or shortcuts to jump over some layers. The omitted connections aim to avoid the problem of vanishing gradients or mitigate the problem of Degradation (accuracy saturation), where adding more layers to a deep model leads to a larger training and test error [42]. This network has the structure of an autoencoder, which usually presents a structure made by three parts: the encoder, which extracts features from the input image, a central part that performs feature processing, and the decoder, the final part, which decodes the processed features into an output image. In the elaborated residual CNN network, the optimizer, batch size and layers of were modified until the results were improved. Techniques such as White Balance, Gamma Correction and the CLAHE algorithm were applied to generate the images with which the network would be trained.
Each convolutional layer was followed by a ReLU activation layer [70], a linear function whose output, if positive, will be the same as the input value, while if negative, the output will be zero, as indicated in Eq. (1):
$$f\left(x\right)=\mathrm{max}(0,\mathrm{x})$$
(1)
After the input layer, there were two pairs of convolutional and ReLU layers followed by a max pooling layer. Next, there was a larger block consisting of three convolutional and ReLU layers and a max pooling layer. This was followed by a group of four convolutional layers. The last group was composed of three convolutional layers. The optimiser chosen for this network was Adam [48], while the loss function chosen was MSE loss. The layered structure can be seen in Fig. 4. This residual network has two residual blocks which skip connections. In this way, these shortcuts perform identity mapping, where their outputs are added to the outputs of the stacked layers.
Fig. 4
Structure of the residual convolutional neural network
×
Classification pipeline
For the detection and classification of animals within the different species as categories, a modified version of the pipeline previously proposed in [57] was used, omitting the application of CLAHE at the image processing step. However, here, background subtraction procedure did not work properly since FOV characteristics slightly changed over consecutive frames/image due to the crawler’s motion. Therefore, the frame difference technique was applied, where each frame was subtracted from the previous one [66].
The used classification algorithms are detailed in subSect. “Metrics” of [57]. On one hand, 8 classical algorithms: two versions of Support Vector Machine (LSVM and SVM_SGD), two K-Nearest Neighbours (K-NN1 and K-NN2), two Decision Trees (DT1 and DT2) and two Random Forests (RF1 and RF2). On the other hand, 8 neural networks: four Convolutional Networks (CNN1, CNN2, CNN3 and CNN4) and four Deep Neural Networks (DNN1, DNN2, DNN3 and DNN4) with different configuration and structure parameters. The parameters chosen for training were the same as in [57].
Experimental design
This section explains the experimental setup of the experiments carried out using both proposed pipelines, i.e., the image enhancement pipeline and the classification pipeline.
As the original images were too large and slowed down the training too much, they were resized to 400 × 300 pixels.
The implemented residual CNN had a 14-layer structure, separated in different blocks. The network for image enhancement was configured to train a maximum of 50 epochs of 100 iterations each. An epoch is one complete presentation of the data set to be learned, while an iteration is the number of batches needed to complete one epoch. The batch size is the total number of training examples present in a single batch. The higher is this parameter, the more memory space is needed. In addition, it was also designed to save the model every time the loss value decreased, and also to stop training if the loss value did not improve within 5 epochs (to avoid the over-fitting problem).
To train the residual CNN that was part of the image enhancement process, a dataset of 13,548 images was used to create the training and test data sets for the new datasets. 80% of the images (10,838 images) constituted the training dataset, while the remainder was used to test the network. Different datasets were incrementally generated, to test the techniques that best affected the final classification results. For the purpose of simplify the names of the datasets used and generated and used in this paper, the following table has been created in which all the names and their description are listed in Table 2.
Table 2
Simplified names of datasets used and their description
Techniques
Short name
Description
RAW
Dataset0
Dataset with the original images
DBY
Dataset1
Dataset with the original images, but in colour (after the application of the debayering technique)
DBY + WB
Dataset2
Dataset containing images of the DBY dataset to which the White Balance (WB) method has been applied
DBY + WB + GC
Dataset3
Dataset containing images of the DBY + WB dataset to which the Gamma Correction (GC) method has been applied
DBY + WB + GC + CLAHE
Dataset4
Dataset containing images of the DBY + WB + GC dataset to which the CLAHE algorithm has been applied
DBY + WB + ResidualCNN
Dataset5
Dataset generated by the trained Residual CNN, using as input the colour images of the DBY dataset to which the WB method has been applied
DBY + WB + GC + ResidualCNN
Dataset6
Dataset generated by the trained Residual CNN, using as input the images of the DBY + WB + GC dataset
DBY + WB + GC + CLAHE + ResidualCNN
Dataset7
Dataset generated by the trained Residual CNN, using as input the images of the DBY + WB + GC + CLAHE dataset
To evaluate the CNN-generated underwater images, we chose SSIM, PSNR, UIQM and UCIQE values.
As for the classification, the size of the collected set was only of 6972 elements. As there were not too many images, we decided to apply data augmentation techniques to 80% of the images (a total of 5573 images), which are the ones that made up the training set. After applying data augmentation techniques, the training set increased from 5573 to 35,020 images, obtaining 3502 samples per class.
All the selected classifiers were tested by tenfold cross-validation by considering that the elements of each class were distributed evenly in each fold [45, 58, 90]. The performance of the models was evaluated by the accuracy, the loss (both train and testing), the AUROC average scores [34], as well as by the confusion matrix. The accuracy and AUROC values were calculated by the multiclass implementation from Scikit-learn, which estimates the metrics for each label, without considering the label imbalance.
All experiments were conducted in Python. The implementation of all the classical algorithms used is within the Scikit‐learn library [74] (https://scikit-learn.org), while the neural networks were implemented with the Keras and Tensorflow libraries. The environment used for training the selected algorithms and the defined models was Google Colaboratory (also known as Colab). It operates currently under Ubuntu 18.04 (64 bits) and it is provided by an Intel Xeon processor and 12 GB RAM. It is also provided with Nvidia K80, T4, P4 and P100 GPUs.
On the one hand, a classification of 4 datasets (the original-coloured dataset, Dataset1, and the three generated by the network, Dataset5, Dataset6 and Dataset7), whose number of classes amounts to 7, was carried out. In addition, the dataset for which the lowest loss value and the highest accuracy and AUROC test values were obtained was selected for another classification, in which three more classes will be added to make a total of 10.
Results
In this section, the results obtained following the experimental design are presented: the results of the image enhancement pipeline are presented in SubSect. “Image enhancement pipeline results”, while the results of the classification pipeline in SubSect. “Classification results”.
Image enhancement pipeline results
The network was finally trained for 15 epochs (due to the configured stop condition) and approximately each epoch took 1100 s to run (approx. 20 min). The visual comparison between the different image datasets is reported in Fig. 5, where the original, colourised and CNN-generated images are shown.
Fig. 5
Comparison among the different image datasets: original, processed with CV techniques and generated by the CNN. Dataset0: dataset of original images in RAW format. Dataset1: dataset of colourised images. Dataset2: dataset of images to which the WB technique has been applied. Dataset3: dataset of images with WB and GC applied. Dataset4: dataset of images to which the WB, GC and CLAHE techniques have been applied. Dataset5: dataset of images generated by the CNN having as input the images of the Dataset2 dataset. Dataset6: dataset of images generated by the CNN having as input the images of the Dataset3 dataset. Dataset7: dataset of images generated by the CNN having as input the images of the Dataset4 dataset
×
Regarding the visual aspect, the greenish colour that characterised the images to which techniques such as WB, GC and CLAHE were applied (Dataset2, Dataset3 and Dataset4), was almost eliminated. The images generated by the residual network did anyway maintain a bluish tone, but the colours of the visible animals were more pronounced. In any case, at a simple visual inspection, the images generated by CNN were somewhat blurrier compared to those processed by CV methods.
Figures 6 and 7 show in detail the comparison between two images as an example. Figures 6A and 7A are part of the input dataset to which CV methods were applied, while Figs. 6B and 7B are images of the dataset generated by the enhancement network. Figure 6 shows an example of image processing where the generated result is quite similar to the input image. As a result of that processing, the two animals belonging to two species (i.e., a floating ctenophore and a rockfish lying on the seabed) can be better visualised by the naked eye.
Fig. 6
Comparison of CNN input and output images. A Shows an input image as after WB, GC and CLAHE processes (from Dataset4). B Shows an image generated by the CNN (from Dataset7)
Fig. 7
Comparison of CNN input and output images. A Shows an input image after WB, GC and CLAHE processes (from Dataset4). B Shows an image generated by the CNN (from Dataset7)
×
×
Similarly, a different comparison between two images can be seen in Fig. 7 also for two species (i.e., a hagfish and another rockfish). This time, the residual CNN generated a superior colour and illumination quality in the output image, except for warm colours, which maintained some of the bluish tone, as common characteristic of untreated underwater images.
Figure 8 shows other two images from input Dataset4 and the output Dataset7. As can be seen in both images, a floating jellyfish can be observed. In this case, the network has transformed the orange tones while maintaining some bluish tones.
Fig. 8
Comparison of CNN input and output images. A Shows an input image after WB, GC and CLAHE processes (from Dataset4). B Shows an image generated by the CNN (from Dataset7)
×
The values of the UIQM and UCIQE metrics for the evaluation of images of the two scenes are summarised in Table 3. The UIQM and UCIQE mean values were slightly higher for the input images of the network. The difference is even greater for coloured images (by debayering).
Table 3
UIQM and UCIQE mean values for the different datasets
UIQM
UCIQE
Dataset0
1.845
0.2719
Dataset1
1.936
3.515
Dataset2
2.421
2.778
Dataset3
2.299
2.475
Dataset4
2.023
3.096
Dataset5
2.585
2.164
Dataset6
2.454
1.892
Dataset7
1.908
2.406
The bold values refer to the highest (and best) value
The values of the SSIM and PSNR metrics for the evaluation of the quality and similarity of the images corresponding to the different datasets are shown in Tables 4 and 5 respectively. As for the SSIM values (Table 4), we can observe that the highest values have been obtained by the datasets generated by the network, while the datasets to which CV techniques were applied (used to train the residual network) obtained lower values. In the same form, the PSNR values have been slightly higher for the datasets generated by the residual network, as can be seen in Table 5
Table 4
SSIM mean values between the different datasets
SSIM
Dataset1
Dataset2
Dataset3
Dataset4
Dataset5
Dataset6
Dataset7
Dataset0
0.5464 ± 0.0663
0.5298 ± 0.0668
0.6465 ± 0.0496
0.5199 ± 0.0723
0.5355 ± 0.0585
0.6496 ± 0.0382
0.5348 ± 0.0620
The bold values are the highest (and therefore best) value
Table 5
PSNR mean values between the different datasets
PSNR
Dataset1
Dataset2
Dataset3
Dataset4
Dataset5
Dataset6
Dataset7
Dataset0
15.3835 ± 0.6807
15.4722 ± 0.6855
19.5634 ± 0.9484
16.3320 ± 0.7462
15.7494 ± 0.6848
20.1172 ± 0.9509
17.0195 ± 0.7969
The bold values are the highest (and therefore best) value
Classification results
The classification results have been divided in two parts: first the results are shown for the seven-class dataset and then for the modified dataset containing ten classes.
The 7-classes classification results
The results obtained in the classification of the first 7 selected classes (Sebastidae, Bathyagonus nigripinnis, Eptatretus stoutii, Chionoecetes tanneri, Asteroidea, Bolinopsis infundibulum, Poralia rufescens) are shown in Tables 6 and 7.
Table 6
Test accuracy and AUROC values obtained by the classical algorithms with the 7 classes datasets
Classifier
Dataset1
Dataset5
Dataset6
Dataset7
Test AUROC
Test accuracy
Test AUROC
Test accuracy
Test AUROC
Test accuracy
Test AUROC
Test accuracy
LSVM
0.6443
0.3390
0.7365
0.4976
0.7484
0.5141
0.7506
0.5171
SVM_SGD
0.6098
0.2868
0.6457
0.3911
0.6773
0.4534
0.7220
0.4850
K-NN1
0.6557
0.3579
0.7077
0.4083
0.6999
0.4050
0.7179
0.4282
K-NN2
0.6136
0.2999
0.6715
0.3462
0.6590
0.3446
0.6763
0.3558
DT1
0.6780
0.4135
0.7047
0.4634
0.6972
0.4570
0.7065
0.4744
DT2
0.6743
0.4112
0.6998
0.4559
0.6966
0.4571
0.7020
0.4648
RF1
0.6744
0.4067
0.7033
0.4571
0.7050
0.4665
0.7062
0.4721
RF2
0.7524
0.5214
0.8009
0.6147
0.7965
0.6034
0.8041
0.6110
The bold values refer to the highest (and best) value
Table 7
Training accuracy, training loss, test AUROC and test accuracy and loss values obtained by the deep learning approaches with the 7 classes datasets
Classifier
Train accuracy
Train loss
Test AUROC
Test accuracy
Test loss
Dataset1
CNN-1
0.6097
1.2148
0.5839
0.2508
2.4334
CNN-2
0.6568
1.0570
0.5983
0.2779
2.3739
CNN-3
0.9704
0.1289
0.7960
0.6106
3.4402
CNN-4
0.9875
0.0490
0.7918
0.6043
4.3183
DNN-1
0.6330
0.7024
0.8264
0.6046
2.3056
DNN-2
0.6311
0.7548
0.8181
0.6042
3.6272
DNN-3
0.5799
0.9233
0.8097
0.6373
1.1579
DNN-4
0.5693
0.9631
0.8090
0.6358
1.1581
Dataset5
CNN-1
0.6345
1.1534
0.6602
0.4595
1.4518
CNN-2
0.6603
1.0562
0.6509
0.4374
1.4733
CNN-3
0.9807
0.1263
0.7910
0.6075
3.8420
CNN-4
0.9883
0.0723
0.7923
0.6083
4.2763
DNN-1
0.6395
0.6841
0.8208
0.6563
2.4390
DNN-2
0.6297
0.7835
0.8169
0.6573
4.1019
DNN-3
0.5952
0.8647
0.8227
0.6531
1.1528
DNN-4
0.5978
0.8466
0.8215
0.6525
1.2127
Dataset6
CNN-1
0.6236
1.1459
0.6679
0.4296
1.5100
CNN-2
0.6709
1.0069
0.6721
0.4348
1.5241
CNN-3
0.9632
0.1421
0.7846
0.5925
3.5784
CNN-4
0.9809
0.0775
0.7837
0.5851
4.3230
DNN-1
0.6377
0.6915
0.7987
0.6193
2.7051
DNN-2
0.6300
0.7649
0.8026
0.6218
4.2045
DNN-3
0.5653
0.9716
0.7990
0.6106
1.2135
DNN-4
0.5715
0.9392
0.8076
0.6264
1.2195
Dataset7
CNN-1
0.6375
1.1002
0.6668
0.4738
1.4635
CNN-2
0.7033
0.9094
0.6584
0.4335
1.4909
CNN-3
0.9741
0.1291
0.7889
0.5950
3.8693
CNN-4
0.9866
0.0735
0.7970
0.6144
3.9761
DNN-1
0.6371
0.6880
0.8210
0.6520
2.6627
DNN-2
0.6312
0.7615
0.8246
0.6616
3.8673
DNN-3
0.5908
0.8806
0.8291
0.6616
1.1330
DNN-4
0.6018
0.8376
0.8277
0.6644
1.2163
The bold values refer to the best value
Table 6 shows the results obtained by the classic classifiers on the different datasets. The dataset that obtained the lowest values overall during this classification was the Dataset1. The remaining datasets obtained fairly equal results. The algorithm that obtained the highest performance on all the datasets (both accuracy and AUROC values) was the RF2. The highest accuracy value of 0.6147 was achieved with the Dataset5 dataset, while the highest AUROC value of 0.8041 was achieved with the Dataset7 dataset.
The results obtained by the deep learning techniques can be seen in Table 7. The dataset that obtained the lowest values overall during classification was the so-called Dataset1. Several of the networks obtained high values of accuracy and AUROC, and low values of loss (also during test).
The 10-classes classification results
Three more classes were added to represent floating particles and sand suspended in the water, objects placed on the ground and unclassified species and species that contained very few records, in order to improve the results obtained at the time of detection. In fact, elements within these three categories were detected but were not classified in initially available classes. This dataset was only considered in its Dataset7 processing trials, where it delivered the best result. As can be seen in Table 8, RF2 obtained an accuracy value of more than 0.75 and an AUROC value of almost 0.87.
Table 8
Test accuracy and AUROC values obtained by the classical algorithms with the 10 classes for the images from Dataset7
Classifier
Test AUROC
Test accuracy
LSVM
0.7549
0.5458
SVM_SGD
0.7089
0.4719
K-NN1
0.7787
0.5802
K-NN2
0.7207
0.4727
DT1
0.7587
0.5610
DT2
0.7582
0.5592
RF1
0.7608
0.5637
RF2
0.8691
0.7568
The bold values refer to the highest (and best) value
The confusion matrix (Fig. 9) corresponds to the results obtained by RF2, as the best performing algorithm. It classified quite correctly several classes, like Asteroidea, Bathyagonus nigripinnis, Human-made objects and Floating particles, but it frequently misclassified two classes (Eptatretus stoutii and Sebastidae), for which it achieved a 60% of success rate.
Fig. 9
Confusion matrix for the classification results (accuracy) obtained by RF2
×
As for neural networks, training accuracy and loss did not show major differences compared to those obtained for 7 classes datasets. However, other metrics such as the AUROC value, the test accuracy and test loss values improved (Table 9). The highest AUROC value was 0.8864, achieved by DNN-2. The best test accuracy value was 0.7944, much higher than in any of the previous datasets of 7 classes (see Table 7). Finally, the test loss also decreased to 0.8389 in the case of DNN-4.
Table 9
Training accuracy, training loss and test AUROC, accuracy and loss values obtained by the deep learning approaches for the 10 classes Dataset7
Classifier
Train accuracy
Train loss
Test AUROC
Test accuracy
Test loss
CNN-1
0.5860
1.4096
0.6170
0.3124
2.4164
CNN-2
0.5314
1.6716
0.6370
0.3410
3.5452
CNN-3
0.9224
0.2387
0.8242
0.6749
1.9160
CNN-4
0.9552
0.1326
0.8136
0.6359
2.8140
DNN-1
0.5929
0.8642
0.8859
0.7944
1.2662
DNN-2
0.5813
0.9618
0.8864
0.7888
1.4685
DNN-3
0.5594
1.0097
0.8479
0.7299
0.8745
DNN-4
0.5626
0.9914
0.8760
0.7578
0.8389
The bold values refer to the best value
Figure 10 shows the confusion matrix for the classification results obtained by DNN-4, which achieved good results for almost every class. In this case, five classes (Asteroidea, Bathyagonus nigripinnis, human-made objects, Poralia rufescens and Floating Particles) were correctly classified with a rate above 80%, and the worst ranked class (Sebastidae) had 56% correctly labeled.
Fig. 10
Confusion matrix for the classification results (accuracy) obtained by the DNN-4
×
The performance during the training of the DNN-4 can be seen in Fig. 11. This network was trained during 889 epochs. Figure 11A shows the progress of the accuracy value, while Fig. 11B shows the decreasing of the loss value.
Fig. 11
Training accuracy and loss plots of the DNN-4. The X axis of the plots shows the number of epochs, while the Y axis shows the accuracy and the loss value respectively that was reached during the training. A Accuracy values obtained in every epoch at training time and B Loss values obtained in every epoch at training time
×
Some examples of detection and classification by DNN-4 are shown in Figs. 12 and 13. Figure 12 shows cases of correct detection and classification, while Fig. 13 shows cases where the algorithm confused the classes, leading to a wrong labelling.
Fig. 12
Some examples of detections and correct classification. A Shows a correctly labelled ctenophore, B shows a correctly labelled jellyfish, C shows a correctly labelled rockfish and D shows a correctly labelled hagfish
Fig. 13
Some examples of detections and incorrect classification. A Shows floating particles incorrectly labelled as ctenophore, B shows ground object incorrectly labelled as hagfish, C shows floating particles incorrectly labelled as jellyfish and D shows a jellyfish incorrectly labelled as objects
×
×
It is possible that the misclassification of floating particles as ctenophore shown in Fig. 13 is due to their similarity in colour, as this species body is transparent and has some white spots. The objects classified as species may have been confused because of their location, as these are common areas where such species are found, and probably also because of their shape.
Discussion
In this study, we presented a novel pipeline for the enhancement of dark deep-sea images and the automated classification of visible fauna, in footage taken by a crawler as a moving benthic platform on a changing background. We elaborated an enhancement procedure that allowed to improve the animals classification capability, hence the functionalities previously achieved with static cameras at cable observatories [57]. For this purpose, different image enhancement techniques were first investigated and then applied to generate different datasets. Then, a residual network was modelled and trained with these datasets in order to generate a new set of enhanced images. Although the evaluation metrics of the image sets generated by the residual network could be improved, the best values of test accuracy, loss and AUROC in classification were achieved with one of the datasets generated by the neural network, which is the principal objective.
The residual convolutional network shows some problems with some hues when generating new images. For example, orange colours have been generated with bluish hue, transforming them into pinks. This is probably because these colours do not appear very often in the whole set of images. The UIQM and UCIQE values were slightly higher for the input images of the network. This may be because the images generated by the network are more blurred than the input images, as are the images transformed by applying techniques such as white balance, gamma correction and CLAHE. Similar studies, e.g. [88, 89], applied similar methods to pre-process the images, such as CLAHE, in order to obtain a mask on the Norway lobster (Nephrops norvegicus) detection, and then apply CV techniques and a Mask-RCNN for detection and segmentation comparison. Other studies in which the dataset was also obtained by an ROV, as in [69], we can observe that although the images are not as dark as those in the present paper, they do have that characteristic blue-green colour of the water. The method proposed in [16] provides colour enhancement and restoration to marine images, and although they tested it on not very dark images to which turbidity has been added, it is intended for AUVs and ROVs. They obtained a PSNR value of 21.840 dB, while ours was 20.117 dB. In [78] authors present an image enhancement method in which they also apply CLAHE, as well as other techniques, such as gray-level co-occurrence matrix (GLCM) feature extraction. However, the images they used were obtained from a dataset whose characteristics are totally different from the one used here, since it is collected by static cameras and at a shallower depth, since the images have natural light.
For the classification of marine species, two different types of methods were used in this study, i.e., classical algorithms and DL techniques. Data augmentation techniques were applied to the species with the most elements, and on the other hand, classes with insufficient number of elements were discarded. Similar studies also detected the advantages of DL over ML methods in marine environments (X. [19, 80, 83, 95]. However, the datasets used by these studies were obtained in coral reef areas, where there is still some sunlight, while the dataset we used was obtained at depths of more than 800 m, where visibility is low. For deeper water applications mimicking environmental conditions similar to those where the crawler is deployed, [88, 88, 89, 89] evidenced that advanced DL techniques, such as segmentation networks, can be an efficient tool for monitoring catches in pelagic fishery. In addition, the crawler generated clouds of sand and prevented the observation of species and objects on several occasions, which would not happen with a fixed camera.
For the test values, there are notable differences both among datasets and among neural networks. Regarding the classical algorithms, the RF2 was clearly the model that obtained the best results on all the datasets. With regard to the neural networks, it can be seen that, in the case of the CNNs, good results were not obtained, since CNN-1 and CNN-2 networks obtained quite low validation accuracy and AUC values, while CNN-3 and CNN-4, due to the loss values, were probably over trained. The deep neural networks (DNN-1 to DNN-4) achieved better AUROC, accuracy and loss test values than other algorithms. However, the sequential networks DNN-1 and DNN-2 have performed rather poorly for datasets of 7 and 10 classes, reaching loss values as high as 1.2662 and 1.4685 respectively. On the other hand, the other two deep networks (DNN-3 and DNN-4), have had a good performance and result, obtaining the best value for the test accuracy (0.6644) and the best value for the test loss (1.1330) for 7 classes datasets, while for the 10 classes dataset DNN-4 obtained a test accuracy value of 0.7578 and a test loss value of 0.8389.
In [88, 89] authors also used CV techniques for the classification of marine species, which they compared with the results obtained by a Mask R-CNN. In their case, they obtained higher results with CV techniques, although in a later work they improved their classification with the segmentation network in a dataset of four classes [88, 89]. In [78] authors performed classification on the enhanced images using SVM, DT and k-NN, among others. The SVM achieved an accuracy value of 79.66%, while the k-NN obtained a value of 72.96% and the DT of 64.03%. They also used a backpropagation neural network (BPNN), which achieved an accuracy of 93.73%. The values achieved in the present study were quite close to those, despite the fact that the dataset is totally different, more complex and darker.
If we compare the results obtained for the 7 classes dataset and the 10 classes dataset, we can see that the best results were obtained for the dataset with more classes. This may be due to the fact that the classification pipeline in [57] detects the elements that move along the different images, and that these elements were not correctly classified because they did not correspond to any class. In the 10 classes dataset, these extra classes have been added and those elements can be assigned to a class and then be correctly classified.
Compared to the results of [57], the results here obtained achieved better metrics. As for the ML methods, RF2 was the algorithm that obtained the best test values for accuracy and AUROC in both investigations. In this paper the test values of 0.7568 for accuracy and 0.8691 for AUROC were reached, while in [57] the accuracy value for this algorithm was 0.6527 and that for AUROC was 0.8210. Regarding DL techniques, in [57] the network that obtained the highest values was the DNN-4, with test accuracy value of 0.7140 and an AUROC value of 0.8503, whereas here higher values have been achieved with several networks. DNN-4 obtained a test accuracy value of 0.7578 and an AUROC value of 0.8389. DNN-3 also obtained, anyway, higher values than the previous paper. DNN-1 and DNN-2 also outperformed the previous results but obtained high error values. We can state definitely that the results obtained in this paper outperform those of [57].
The next technological application scenario
Marine robotics is creating platforms that can be transformed into intelligent tools for autonomous ecosystem monitoring needs [4], as is nowadays required for monitoring an increasing number of marine activities, e.g., oil extraction and mining [46, 87, 91]. Implementing routines for automated individual tracking and classification and later integrate all those component routines into an operational hardware and software product, is a key aspect to improve the ecological monitoring functionality of all mobile platforms, including the crawler [8, 35, 63, 98, 104]. The proposed approaches and results represent a first step toward the establishment of an autonomous software focused on image processing to be installed on-board of the crawler. This represents a critical bottleneck for full autonomous monitoring of deep-sea ecosystem functions and services, by this class of IOVs.
Extracting the essential information on species presence, counts (as an indicator for abundance) and derived spatiotemporal changes, picturing community dynamism, is a time-consuming manual process. The tasks proposed in this research with the state-of-the-art of CNN algorithms indicate the possibility to allow embedded pre-processing of acquired images for object tracking via image quality enhancing/rendering with CNN approaches. At the same time, a refinement of species classification procedure is available with the post-processing of imaging products on a server with the help of newly added morphological descriptors [1, 2]. Moreover, the integration of all the detection and classification connected with the video capture processes would allow to transmit and store only the frames where the algorithms detect some kind of labelled species.
Conclusions and future work
The designed neural network, in combination with the detection and classification pipeline, generated enhanced underwater images leading to a more accurate classification process. The improvement and enhancement of underwater images also play an important role in feature detection, since a clear improvement of the images could reduce the subsequent work of feature detection and obtains better classification rates. We demonstrated that a neural network is a good option for generating enhanced images automatically, without the need to apply multiple techniques to an image. Due to their particular characteristics, enhancement of underwater images prior to detection and classification is indispensable for the improvement of classification results, regardless of the use of traditional classifiers or DL approaches.
As future work in this line of research, the current developed CNN for image enhancement could be modified by adding or removing layers, modifying the number of units in each layer, or applying different parameter settings, i.e., modifying the number of epochs, the batch size or using different activation functions. Another step which would be of interest for practical applications would be the optimization of image quality vs. computational cost when applying these procedures to the original-sized (1600 × 1200 pixel) images, in order to minimize the processing time without compromising the extracted amount of valuable information. As for classification, to improve the results, other strategies like transfer learning, or even object detection networks and segmentation networks could be used. However, the amount of floating particles in some images, and the small size of some species, could hinder the performance and results of this type of networks.
Acknowledgements
This work was developed at Deusto Seidor S.A. (01015, Vitoria-Gasteiz, Spain) within the framework of the Tecnoterra (ICM-CSIC/UPC) and the following project activities: ARIM (Autonomous Robotic sea-floor Infrastructure for benthopelagic Monitoring); MarTERA ERA-Net Cofund; Centro para el Desarrollo Tecnológico Industrial, CDTI; and RESBIO (TEC2017-87861-R; Ministerio de Ciencia, Innovación y Universidades).
Declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
