Pollen recognition through an open-source web-based system: automated particle counting for aerobiological analysis
Authors:
Antonio Jesús Chaves, Cristian Martín, Luis Llopis Torres, Manuel Díaz, Rocío Ruiz-Mata, Enrique de Gálvez-Montañez, Marta Recio, M. Mar Trigo, Antonio Picornell
Airborne pollen is produced by plants for their sexual reproduction and can have negative impacts on public health. The current monitoring systems are based on manual sampling processes which are tedious and time-consuming. Due to that, pollen concentrations are often reported with a delay of up to one week. In this study, we present an open-source user-friendly web application powered by deep learning for automatic pollen count and classification. The application aims to simplify the process for non-IT users to count and classify different types of pollen, reducing the effort required compared to manual methods. To overcome the challenges of acquiring large labelled datasets, we propose a semi-automatic labelling approach, which combines human expertise and machine learning techniques. The results demonstrate that our approach significantly reduces the effort required for users to count and classify pollen taxa accurately. The model achieved high precision and recall rates (\(\varvec{>96\%}\) mAP@0.5), enabling reliable pollen identification and prediction.
Notes
Communicated by: H. Babaie.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
Many plant species have anemophilous strategies to disperse their pollen and ensure their sexual reproduction. As a result, pollen is frequently detected in the atmosphere of almost every part of the world (D’amato et al. 2007; Buters et al. 2018; Charalampopoulos et al. 2021). Airborne pollen can be considered an organic pollutant, given that it can produce adverse effects on public health. Pollen can trigger allergic reactions in the sensitized population, and it can also promote the symptoms of other respiratory diseases such as asthma.
Climate change is exacerbating the negative impacts of pollen on public health. During the last decades, there was an upward trend in allergy symptoms, and further increases are projected in the upcoming years (Clot et al. 2020; Lake et al. 2017). In this context, the implementation of airborne monitoring systems is crucial to minimize the impacts of pollen on public health and the future economy. However, such systems are generally based on a manual sampling process: trap maintenance, sample preparation, and analysis with a light microscope. In addition, the dissemination of information to the public requires specialised staff and a lot of workforce. Due to that, pollen concentrations are usually reported with a week of delay, constituting a handicap for the pollen forecast systems (Clot et al. 2020; Galán et al. 2007).
Advertisement
In recent decades, several prototypes of automatic samplers of airborne pollen have been developed. Some are based on air-flow cytometry, such as the WIBS sensor (Perring et al. 2015), Plair PA-300 (Crouzy et al. 2016), or the Yamatronics KH3000 (Kawashima et al. 2017), while others are on detecting chemicals (Buters et al. 2012), DNA (Kraaijeveld et al. 2015), or image recognition (Clot et al. 2020; Oteros et al. 2015; Crouzy et al. 2016; Buters et al. 2022). These new methods and devices have reported accuracy rates of over 80% in most cases, suggesting that they are useful in identifying the most common pollen types in certain regions of Europe, America, and Asia (Oteros et al. 2020). However, these automatic devices are currently expensive and most of the already existing aerobiological networks are based on manual sampling devices (Buters et al. 2018). In this scenario, the development of algorithms that can identify and count pollen from manually collected samples would constitute a needed intermediate step for the transition from manual to automatic sampling. It would reduce the time required to process the collected samples and the labour costs of the current aerobiological networks.
At present, there are a few algorithms that can identify pollen through image recognition, such as those implemented in the BAA500 sampling device (Oteros et al. 2015), in the PS-400 Pollen Sensor (Jiang et al. 2022), or in the Swisens Poleno based on images obtained by digital holography (Sauvageat et al. 2020). However, these algorithms are developed by private companies and thus, they are solely available in a paid subscription model (Khanzhina et al. 2018). In addition, these algorithms are linked to their specific sampling devices. They cannot be directly applied to manually collected samples, such as the ones collected by Hirst-type pollen traps, which are the current standard for pollen monitoring (Buters et al. 2018). Frequently, pollen samples are collected but not counted due to a lack of time or trained staff. Therefore, developing an online and open-source tool for identifying airborne pollen from these already collected samples would reduce costs in aerobiological sampling worldwide. Open-source algorithms trained in pollen recognition could also constitute a relevant advance for assessing pollen production of trees (Rojo et al. 2015) or estimating the fruit production of a certain crop through pollen detection (Ribeiro et al. 2008). This tool would allow researchers to evade the economic barrier that prevents them from continuing with the data collection. With deep learning capabilities (LeCun et al. 2015) these pollen samples can be analysed and classified automatically without the need for human intervention.
Previous work has already achieved results with deep learning algorithms for particle detection and classification, such as in Andriopoulou et al. (2023), where an encoder-decoder neural network is proposed for the classification and segmentation of four phytolith classes. Some of these techniques have already been tested on pollen recognition, such as in Kubera et al. (2022a) where several state-of-the-art model architectures were trained and evaluated on the identification of three pollen types. Other works such as Daunys et al. (2022) compared the performance of different model architectures in pollen identification through image recognition, and Minowa et al. (2022) evaluated the ability of deep learning models to recognise 20 different pollen types. In Monteiro et al. (2021), different deep-learning architectures were evaluated in the recognition of a large pollen dataset (73 types). Earlier works have also attempted the same task with more classical computer vision algorithms, as in García et al. (2012), where contours and colour techniques were used to correctly account for a good part of the test datasets. Similar techniques have helped us to automate certain tasks such as image labelling. However, most algorithms were trained using pollen types that are abundant in central Europe or Japan, but few studies have included pollen types predominant in the Mediterranean area. In addition, we have not found any reference to the availability of the trained algorithms and datasets in most cases.
Facing the growing demand for open-source code algorithms and datasets, this study had a threefold aim: i) to train and select a deep learning algorithm to identify the most abundant and representative pollen types detected in the Mediterranean region by reviewing some state-of-the-art models; ii) to release a public dataset of different Mediterranean pollen types; and iii) to create an open-access web application for the automatic image processing of pollen samples.
Advertisement
Materials and methods
An overview of the workflow used as materials and methods in this work is shown in Fig. 1. First, pollen samples were collected directly from the plants at different locations in Málaga by the staff of the Aerobiology and Palynology lab belonging to the Botany and Plant Physiology department of the University of Málaga, Spain. These samples were stored and frozen in Eppendorf tubes until the samples of individual pollen types were produced. Afterwards, the samples were digitized using an optical microscope and pre-processed to obtain a standard image format. Subsequently, with the help of different techniques, a semi-automatic labelling approach of the images was performed, which resulted in the dataset that was used to train the deep learning algorithm You Only Look Once version 7 (YOLOv7). Finally, once the algorithm had been trained and learned from the pollen images, it was used to classify and count pollen through new and unseen images. This workflow has been integrated into the web architecture presented in Section 3. Next, each of these workflow steps is described in detail.
Fig. 1
Pollen workflow adopted in this work: from pollen acquisition to pollen classification
×
Sample preparation
Pollen samples were manually collected from open flowers of a mixture of individuals belonging to various plant species that have the same pollen types. The pollen types considered for this study were Amaranthaceae, Casuarina, Cupressaceae, Olea, Palmaceae, Pinus, Platanus, Plantago, Poaceae, Rumex and Urticaceae, as they are usually detected in relevant concentrations in the atmosphere of Mediterranean cities and they can trigger allergic symptoms in the sensitized population (Trigo et al. 2008). Samples were collected in Málaga (Southern Spain) and were stored in Eppendorf tubes and frozen under \(-17^{\underline{\textrm{o}}}\)C to avoid their deterioration. For each pollen type, two microscope slides were prepared with the collected pollen using glycerin jelly as the mounting media. The glycerin jelly was stained with basic fuchsin following the same procedure usually applied to aerobiological samples (Galán et al. 2007). Each sample was cooled until the glycerin jelly was solid again and was sealed with Eukitt®. The samples had a total surface of 1440 mm2. This ensures that the pollen grains have the same appearance as they usually have when collected by the regular aerobiological procedure, and that the samples contain enough pollen grains to train the algorithms.
Microscopy and image acquisition
For imaging, an Olympus VS120 optic motorised microscope was used with a UPLSAPO40x/0.9 objective lens. The microscope was equipped with a VC50 Olympus camera that captured images of a longitudinal sweep of the pollen samples. The pictures were initially processed and compiled using the VS-NIS-SWL-V2.3 software, provided by Olympus. In the first step, samples were scanned at different focus plains and, then, the pictures were stacked to provide clear sights of the pollen grains.
Fig. 2
Step-by-step computer-guided analysis of pollen sample fragment image
×
Image analysis and dataset preparation
Initially, the images captured by the microscope were in a proprietary format (Olympus CellSens VSI). To analyze them, we needed to extract the different slides and convert them to a standard format (such as TIFF, PNG, etc.). To accomplish this, we utilized the BioFormats library created by The Open Microscopy Environment (OME),1 which allowed us to convert the microscope scans to a standard format to proceed with their analysis.
After converting the images to a standard TIFF format, we proceeded to analyze them. In this study, our initial approach involved analyzing the images based on their colours and sizes. To achieve this, we utilized the widely-used Computer Vision library, OpenCV.2 This preliminary analysis allowed automatic labelling using specific colour and shape filters, thus minimizing the additional workload involved in manual labelling.
In particular, a colour filter on the HSV colour pattern was carried out, which allows us to select only the regions of a selected range of colours, which in our case, is the colour spectrum of the dyed pollen particles. Afterwards, a noise removal technique (Opening (Jamil et al. 2008)) was applied to eliminate possible false positives obtained in the previous step. Finally, we compute the remaining contours of our filtered image and calculate the bounding box of these contours. Moreover, as the pollen samples are monospecific, we also get to which pollen taxa they correspond, so our images are already labelled. A similar workflow was followed in previous studies (García et al. 2012).
Figure 2 depicts step-by-step the process made using OpenCV in order to make the first step in automatic image labelling: 1) example of sample to be processed (Fig. 2a); 2) image after applying the colour filter (Fig. 2b); and 3) result obtained after applying noise reduction operation (Fig. 2c).
After the images were pre-labelled, they were provided in a monospecific dataset to multiple instances of object detection models ("YOLOv7" was used as recommended in (Kubera et al. 2022a)). Then, they were trained to identify possible new particles in the monospecific images. This approach further automated the labelling process, where the human review was only necessary to confirm that all images were correctly labelled.
Table 1
Model evaluation results
Model
mAP@0.5 (Val)
mAP@0.5 (Test)
Inf time (s)
YOLOv7
96.9%
97.1%
337.566s
YOLOX
96.7%
96.8%
318.690s
Faster R-CNN
96.0%
95.8%
453.193s
RetinaNet
87.6%
87.9%
598.186s
Object detection algorithms studied
In order to find the best model that fits our use case, we reviewed the state-of-the-art object detection models and selected the following models (and different versions of them), which we trained and evaluated to select the most accurate one:
You Only Look Once (YOLO) is an object detection algorithm that can detect multiple objects in an image or video in real time. It divides the input image into a grid and predicts bounding boxes and class probabilities for each grid cell, using non-maximum suppression (Hosang et al. 2017) to eliminate redundant detections. YOLO is known for its speed and efficiency, processing the entire image at once. However, it may have problems with detecting small objects or those with low contrast. Multiple versions of YOLO have been released, such as YOLO (Redmon et al. 2016), YOLOv4 (Bochkovskiy et al. 2020), YOLOv7 (Wang et al. 2022), and YOLOv8 (Jocher et al. 2023). Each time a new version is released, it consistently incorporates enhancements, optimizations, or novel methodologies for information processing, with the aim of enhancing latencies and/or accuracies. A comparison between the different versions and their performance is presented in Terven and Cordova-Esparza (2023), which presents a comprehensive analysis of the evolution of YOLO releases as well as its innovations and contributions.
Fig. 3
Example of the YOLOv7 algorithm prediction for four different pollen taxa particles in a microscopy image
Fig. 4
Web Application Architecture
×
×
Faster R-CNN (Ren et al. 2016) is an object detection model that combines deep learning and region proposal methods. It revolutionized computer vision by introducing a two-stage architecture, enabling efficient and accurate object localization and classification. It uses a region proposal network to generate potential object regions and a region of interest pooling layer for refinement and classification.
RetinaNet (Lin et al. 2018) is an accurate and computationally efficient object detection model designed to overcome the difficulties associated with detecting objects at different scales and handling imbalanced class distributions. The model introduces a unique component known as focal loss, which dynamically adapts the loss function to prioritize challenging examples during the training process. Furthermore, RetinaNet incorporates a feature pyramid network to extract features at multiple resolutions.
As demonstrated in the literature (Kubera et al. 2022a, b), the YOLO model usually provides better results in terms of accuracy for the detection and classification of pollen. In this work, we evaluate the above models to validate which one is the best suited for the general Mediterranean pollen dataset acquired.
YOLOv7 algorithm for pollen detection
In a second iteration, we incorporated the YOLOv7 deep learning algorithm to detect pollen particles in samples (YOLOv7 has been used as it is the one with the best results as seen in Table 1.). Following semi-automatic image processing and labelling, we incorporated the images into a YOLOv7 instance for training and validation. The dataset contained images of various pollen taxa mentioned previously, and the annotations comprised bounding boxes around each pollen particle in the image and its corresponding type. The YOLOv7 algorithm proficiently identified the visual characteristics of pollen particles and precisely predicted their locations in the image.
Fig. 5
Pollen sample submission form
Fig. 6
General overview of the application
Fig. 7
Confusion matrix result of applying the algorithm into the validation dataset
Fig. 8
Well-predicted images of different pollen taxas
Fig. 9
Potentially problematic images of different pollen taxas
×
×
×
×
×
The YOLOv7-trained model for this research used our released open-source pollen dataset comprising 19,579 samples.3 For training purposes, 70% of the data was used, and the remaining 30% was considered for both validation and testing (15% each). The training involved rescaling images to a size of 1024x1024 and utilizing the default hyperparameters of YOLO-P6. Moreover, various data augmentation techniques, such as random scaling, rotation, and horizontal flipping, were employed to augment the model’s robustness.
To demonstrate image prediction using the YOLOv7 algorithm, we present a representative image from our test dataset in Fig. 3. The image shows a pollen sample containing four different pollen taxa particles: Platanus (green), Palmaceae (light green), Causarina (pink), and Poaceae (blue). The ground truth annotations for the image indicate that there are three Platanus particles, eight Palmaceae particles, seven Causarina particles, and four Poaceae particles in the image. The YOLOv7 algorithm was able to detect all the pollen particles in the image and correctly classify them into their respective classes.
Web application for automatic image processing and object detection
To facilitate the adoption of the system for the identification and categorization of pollen particles by non-technical users, we created a web application that enables users to upload their microscopy images and perform automatic processing as discussed in Section 2. The web application was designed to operate seamlessly with microscopy images acquired using the Olympus microscope. However, it can also be customized to be compatible with other microscopy image types.
Figure 4 shows a general overview of the application architecture and its components. The backend provides a web server and a RESTful API that is consulted by the frontend, the visual part that is accessed by the users of the platform for the execution of all the actions provided in the tool, such as consulting the history of the pollen samples analyzed, or the analysis of a new sample. The backend also invokes the YOLO executor, which is responsible for serving the trained YOLO model for pollen classification through a new API, i.e., this executor receives pollen images from the backend and returns the list of pollen found.
The web application was developed with the Angular, Django, and FastAPI web frameworks and is hosted on-premises cloud-based server. It can be easily deployed in a distributed way thanks to the containerization of the components through Docker and its integration with Kubernetes. Kubernetes enables the orchestration of containers in multi-node deployments.
Users can conveniently access the application through a web browser and upload their microscopy images saved in the .vsi file format. The web application utilizes the BioFormats and OpenCV libraries to extract the separate images from the .vsi file and preprocess them slicing them into smaller images in order to give a better input to the model. Figure 5 shows the web application form where the user uploads the samples they want to analyse.
Subsequently, the preprocessed images are forwarded to a FastAPI instance with the model ready to run inferences. The YOLOv7 is developed using the PyTorch ML framework and is hosted on a dedicated server to ensure optimal performance and scalability. The YOLOv7 algorithm can identify and categorize the pollen particles in the images rapidly and with high accuracy as shown before. The YOLOv7 outcomes are conveyed back to the web application in a JSON format.
Once analyzed, the user can check the results obtained by the algorithm in the Web User Interface and download the JSON report and the inferred images and their labels if needed. Figure 6 shows a general overview of the user dashboard once the analysis has been completed. In this form, users can visualize the pollen concentrations obtained from the microscope images submitted.
Overall, the web application provides a user-friendly and efficient platform for the automatic processing of microscopy images and the detection and classification of pollen particles using the YOLOv7 algorithm. The web application can be easily integrated into research workflows and can lead to new applications in plant biology, environmental sciences, and related fields. The application is open source, so anyone can adapt the application to their specific requirements easily and free of charge. The source code is available at GitHub.4
Evaluation
As discussed in Section 2.5, the model instances have been trained with more than 19,000 images, 70% of which are considered for training, 15% for validation, and the remaining 15% for testing. State-of-the-art deep learning models have been considered and evaluated to compare the performance with the proposed model.
The models have been evaluated with the same or similar hyperparameters, small differences in batch size, and the number of iterations because each model requires more or less memory. The training was performed on a single GPU (NVIDIA™ GeForce™ RTX® 3090) for a total of 512 epochs and it was stopped when the first signs of overfitting were observed on the evaluation metrics (such as a significant increase in the validation loss). Table 1 shows the mean average precision (mAP) and the inference time (Inf time) of the evaluated models. Compared with the other models, YOLOv7 obtained the highest accuracy rates in the validation and test, and it is close to the one with the lowest latency (YOLOX).
As observed in the confusion matrix (Fig. 7), some pollen types are more difficult to detect than others. Pollen types such as Urticaceae have high variability in their sight at the optic microscope. In addition, these pollen grains usually have irregular surfaces due to being wrinkled, broken, or partially overlapped. However, the accuracy obtained is higher than 90% in all the pollen types.
Fig. 10
Confusion matrix result of applying the algorithm into the multiple pollen taxa set
×
Figure 8 shows some examples of well-predicted images, and Fig. 9 shows some examples of different pollen that could not be correctly classified by the YOLOv7 model. These issues are the main problems contributing to the appearance of false positives in pollen taxa detection and classification. In Fig. 9a b, some prediction errors are visible due to the overlapping of pollen particles, and in Fig. 9c d, it can be seen how there are missing pollen particles to be identified due to lack of dye or due to blurring of the sample.
This evaluation only involved monospecific images. Building upon findings from Kubera et al. (2022a), the YOLO algorithm would be able to recognise multiple pollen taxa in a single image even when trained on monospecific samples.
In order to assess this statement, we carried out an evaluation with an image set with different pollen taxa, resulting in similar accuracies to those previously shown (Fig. 10). This may be due to new overlaps between different pollen taxa or to other factors already discussed.
Regarding the response time, we have measured the time it takes for the different models to infer all the images in the test set (Table 1). It can be seen that unlike in Wang et al. (2022), YOLOX was slightly faster than YOLOv7. However, this may be due to the fact that YOLOX has not been deployed on PyTorch like the rest of the models. It was exported to ONNX (an open standard for machine learning interoperability5) and executed following the indications of the official repository. Response time depends on both the size of the images and the server infrastructure where the application is deployed. In our infrastructure,6 average response time is 291.4 seconds. Given the small difference in time between the YOLOX and YOLOv7 models, and the effort involved in working with YOLOX (problems with libraries, exporting to ONNX, supporting models in ONNX, etc.), we decided to use YOLOv7 on PyTorch. Lower inference latencies mean that images will be analysed faster, so the user experience will be smoother.
Conclusion and future work
In this paper, an open-source web application powered by deep learning for automatic pollen count and classification is presented. The application offers a user-friendly interface, making it accessible for non-IT users to efficiently count and classify different types of pollen. The challenges of acquiring large labelled datasets were overcome by employing a semi-automatic labelling approach that combines human expertise and machine learning techniques. The results, which demonstrate a rate of precision over 97% for the YOLOv7, support the use of this model to reduce the effort and time required for accurate pollen identification. The model provided reliable results for pollen monitoring, and its implementation in the pollen alert systems would contribute to the management of allergies and understanding of pollen’s impact on public health. This open-access tool represents a significant step towards simplifying and automating the process of pollen monitoring, ultimately benefiting both researchers and the wider community. The web application would be also useful for quantifying pollen from monospecific samples, such as those obtained for assessing the pollen production of trees, or the pollen production of crops. The system, the machine learning model, and the dataset used (and acquired in southeastern Spain) have been released openly so that the community can adopt this pollen counting and classification system.
As a future work, some of the following improvements and/or integrations are proposed:
Complete automation of the sample upload process. An interesting feature to include would be the ability to connect the microscopes with the platform in order to automatically send the images of the pollen samples to the application. This would considerably speed up the process by avoiding the transfer of data between intermediate devices.
Multiple model integration. Although at the moment there is only one trained and integrated model and the platform automatically selects that model for inference, it would be interesting to enable the possibility of integrating other YOLO instances, allowing models trained for other pollen taxa, or even allowing new tools that include multiple object detection models such as Detectron2 (Wu et al. 2019). This would give more flexibility and a wide range of models from which people could start to count different particles.
Providing support for new image types. Similarly, as before, we have currently supported the VSI format, which belongs to the Olympus microscope we use. However, the BioFormats library supports a multitude of formats from a variety of systems. It would be interesting to adapt our use case to other types in a way that is simple and transparent to the user.
Declarations
Competing interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Pollen recognition through an open-source web-based system: automated particle counting for aerobiological analysis
Authors
Antonio Jesús Chaves Cristian Martín Luis Llopis Torres Manuel Díaz Rocío Ruiz-Mata Enrique de Gálvez-Montañez Marta Recio M. Mar Trigo Antonio Picornell
