Skip to main content
Top
Published in: International Journal of Multimedia Information Retrieval 4/2022

18-10-2022 | Regular Paper

Video deblurring and flow-guided feature aggregation for obstacle detection in agricultural videos

Authors: Keyang Cheng, Xuesen Zhu, Yongzhao Zhan, Yunshen Pei

Published in: International Journal of Multimedia Information Retrieval | Issue 4/2022

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Autonomous agricultural vehicles are increasingly common on farms, where they can replace humans in tasks such as irrigation, harvesting, and weeding, reducing labor costs. Real-time obstacle avoidance is a prerequisite for their work. At present, vehicles equipped with vision sensors cannot perform end-to-end video object detection, and their accuracy is also affected by motion blur. We propose a novel agricultural obstacle detection method based on RNN and flow-guided feature aggregation, combining video deblurring and object detection tasks for joint optimization. In addition, to make full use of the region proposals, a region shared strategy is proposed to improve the efficiency of video deblurring. The proposed method can solve the common motion blur problem in agricultural video and is expected to be suitable for all kinds of obstacle detection tasks in agricultural scenes. We experimented with this method on the FieldSAFE and GOPRO datasets. Our method provides better detection performance and is computationally less costly than other methods according to experimental results.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Ross P, English A, Ball D, et al (2014) Novelty-based visual obstacle detection in agriculture. In: 2014 IEEE international conference on robotics and automation (ICRA), IEEE, pp 1699–1705 Ross P, English A, Ball D, et al (2014) Novelty-based visual obstacle detection in agriculture. In: 2014 IEEE international conference on robotics and automation (ICRA), IEEE, pp 1699–1705
2.
go back to reference Campos Y, Sossa H, Pajares G (2016) Spatio-temporal analysis for obstacle detection in agricultural videos. Appl Soft Comput 45:86–97CrossRef Campos Y, Sossa H, Pajares G (2016) Spatio-temporal analysis for obstacle detection in agricultural videos. Appl Soft Comput 45:86–97CrossRef
3.
go back to reference Murthy CB, Hashmi MF, Keskar AG (2021) Optimized mobilenet+ ssd: a real-time pedestrian detection on a low-end edge device. Int J Multimed Inf Retr 10(3):171–184CrossRef Murthy CB, Hashmi MF, Keskar AG (2021) Optimized mobilenet+ ssd: a real-time pedestrian detection on a low-end edge device. Int J Multimed Inf Retr 10(3):171–184CrossRef
4.
go back to reference Suresha M, Kuppa S, Raghukumar D (2020) A study on deep learning spatiotemporal models and feature extraction techniques for video understanding. Int J Multimed Inf Retr 9(2):81–101CrossRef Suresha M, Kuppa S, Raghukumar D (2020) A study on deep learning spatiotemporal models and feature extraction techniques for video understanding. Int J Multimed Inf Retr 9(2):81–101CrossRef
5.
go back to reference Pan J, Bai H, Tang J (2020) Cascaded deep video deblurring using temporal sharpness prior. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3043–3051 Pan J, Bai H, Tang J (2020) Cascaded deep video deblurring using temporal sharpness prior. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3043–3051
6.
go back to reference Ruan L, Chen B, Li J, et al (2022) Learning to deblur using light field generated and real defocus images. arXiv preprint arXiv:2204.00367 Ruan L, Chen B, Li J, et al (2022) Learning to deblur using light field generated and real defocus images. arXiv preprint arXiv:​2204.​00367
7.
go back to reference Guo C, Fan B, Zhang Q, et al (2020) Augfpn: improving multi-scale feature learning for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12,595–12,604 Guo C, Fan B, Zhang Q, et al (2020) Augfpn: improving multi-scale feature learning for object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12,595–12,604
8.
9.
go back to reference Bastian BT, CV J (2019) Pedestrian detection using first-and second-order aggregate channel features. Int J Multimed Inf Retr 8(2):127–133CrossRef Bastian BT, CV J (2019) Pedestrian detection using first-and second-order aggregate channel features. Int J Multimed Inf Retr 8(2):127–133CrossRef
10.
go back to reference Kang K, Li H, Yan J et al (2017) T-cnn: tubelets with convolutional neural networks for object detection from videos. IEEE Trans Circuits Syst Video Technol 28(10):2896–2907CrossRef Kang K, Li H, Yan J et al (2017) T-cnn: tubelets with convolutional neural networks for object detection from videos. IEEE Trans Circuits Syst Video Technol 28(10):2896–2907CrossRef
12.
go back to reference Lee B, Erdenee E, Jin S, et al (2016) Multi-class multi-object tracking using changing point detection. In: European conference on computer vision, Springer, pp 68–83 Lee B, Erdenee E, Jin S, et al (2016) Multi-class multi-object tracking using changing point detection. In: European conference on computer vision, Springer, pp 68–83
13.
go back to reference Isobe T, Jia X, Tao X, et al (2022) Look back and forth: video super-resolution with explicit temporal difference modeling. arXiv preprint arXiv:2204.07114 Isobe T, Jia X, Tao X, et al (2022) Look back and forth: video super-resolution with explicit temporal difference modeling. arXiv preprint arXiv:​2204.​07114
14.
go back to reference Sayed M, Brostow G (2021) Improved handling of motion blur in online object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1706–1716 Sayed M, Brostow G (2021) Improved handling of motion blur in online object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1706–1716
15.
go back to reference Wang Z, Wu Z, Lu J, et al (2020) Bidet: an efficient binarized object detector. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2049–2058 Wang Z, Wu Z, Lu J, et al (2020) Bidet: an efficient binarized object detector. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2049–2058
16.
go back to reference Pathak D, Krahenbuhl P, Donahue J, et al (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544 Pathak D, Krahenbuhl P, Donahue J, et al (2016) Context encoders: feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544
17.
go back to reference Zamir SW, Arora A, Khan S, et al (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831 Zamir SW, Arora A, Khan S, et al (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831
18.
go back to reference Zhang K, Luo W, Zhong Y et al (2018) Adversarial spatio-temporal learning for video deblurring. IEEE Trans Image Process 28(1):291–301MathSciNetCrossRef Zhang K, Luo W, Zhong Y et al (2018) Adversarial spatio-temporal learning for video deblurring. IEEE Trans Image Process 28(1):291–301MathSciNetCrossRef
19.
go back to reference Hyun Kim T, Mu Lee K, Scholkopf B, et al (2017) Online video deblurring via dynamic temporal blending network. In: Proceedings of the IEEE international conference on computer vision, pp 4038–4047 Hyun Kim T, Mu Lee K, Scholkopf B, et al (2017) Online video deblurring via dynamic temporal blending network. In: Proceedings of the IEEE international conference on computer vision, pp 4038–4047
20.
go back to reference Wieschollek P, Hirsch M, Scholkopf B, et al (2017) Learning blind motion deblurring. In: Proceedings of the IEEE international conference on computer vision, pp 231–240 Wieschollek P, Hirsch M, Scholkopf B, et al (2017) Learning blind motion deblurring. In: Proceedings of the IEEE international conference on computer vision, pp 231–240
21.
go back to reference Zhou J, Cheng J et al (2011) Moving obstacle detection based on machine vision for agricultural mobile robot. Nongye Jixie Xuebao Trans Chinese Soc Agric Mach 42(8):154–158 Zhou J, Cheng J et al (2011) Moving obstacle detection based on machine vision for agricultural mobile robot. Nongye Jixie Xuebao Trans Chinese Soc Agric Mach 42(8):154–158
22.
go back to reference Christiansen P, Nielsen LN, Steen KA, et al (2016) Deepanomaly: combining background subtraction and deep learning for detecting obstacles and anomalies in an agricultural field. Sensors, 16(11), 1904 Christiansen P, Nielsen LN, Steen KA, et al (2016) Deepanomaly: combining background subtraction and deep learning for detecting obstacles and anomalies in an agricultural field. Sensors, 16(11), 1904
23.
go back to reference Zhang Y, Tian Y, Kong Y et al (2020) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495CrossRef Zhang Y, Tian Y, Kong Y et al (2020) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495CrossRef
24.
go back to reference Zhu X, Wang Y, Dai J, et al (2017) Flow-guided feature aggregation for video object detection. In: Proceedings of the IEEE international conference on computer vision, pp 408–417 Zhu X, Wang Y, Dai J, et al (2017) Flow-guided feature aggregation for video object detection. In: Proceedings of the IEEE international conference on computer vision, pp 408–417
25.
go back to reference Dosovitskiy A, Fischer P, Ilg E, et al (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766 Dosovitskiy A, Fischer P, Ilg E, et al (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766
26.
go back to reference Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448 Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448
27.
go back to reference He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778 He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
28.
go back to reference Szegedy C, Ioffe S, Vanhoucke V, et al (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence Szegedy C, Ioffe S, Vanhoucke V, et al (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence
29.
go back to reference Dai J, Li Y, He K, et al (2016) R-fcn: object detection via region-based fully convolutional networks. Adv Neural Inf Process Syst, 29 Dai J, Li Y, He K, et al (2016) R-fcn: object detection via region-based fully convolutional networks. Adv Neural Inf Process Syst, 29
30.
go back to reference Kragh MF, Christiansen P, Laursen MS et al (2017) Fieldsafe: dataset for obstacle detection in agriculture. Sensors 17(11):2579CrossRef Kragh MF, Christiansen P, Laursen MS et al (2017) Fieldsafe: dataset for obstacle detection in agriculture. Sensors 17(11):2579CrossRef
31.
go back to reference Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3883–3891 Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3883–3891
32.
go back to reference Zhu X, Xiong Y, Dai J, et al (2017) Deep feature flow for video recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2349–2358 Zhu X, Xiong Y, Dai J, et al (2017) Deep feature flow for video recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2349–2358
33.
go back to reference Wang S, Zhou Y, Yan J, et al (2018) Fully motion-aware network for video object detection. In: Proceedings of the European conference on computer vision (ECCV), pp 542–557 Wang S, Zhou Y, Yan J, et al (2018) Fully motion-aware network for video object detection. In: Proceedings of the European conference on computer vision (ECCV), pp 542–557
34.
go back to reference Bertasius G, Torresani L, Shi J (2018) Object detection in video with spatiotemporal sampling networks. In: Proceedings of the European conference on computer vision (ECCV), pp 331–346 Bertasius G, Torresani L, Shi J (2018) Object detection in video with spatiotemporal sampling networks. In: Proceedings of the European conference on computer vision (ECCV), pp 331–346
35.
go back to reference Deng J, Pan Y, Yao T, et al (2019) Relation distillation networks for video object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7023–7032 Deng J, Pan Y, Yao T, et al (2019) Relation distillation networks for video object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7023–7032
36.
go back to reference Chen Y, Cao Y, Hu H, et al (2020) Memory enhanced global-local aggregation for video object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10,337–10,346 Chen Y, Cao Y, Hu H, et al (2020) Memory enhanced global-local aggregation for video object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10,337–10,346
37.
go back to reference Jiang Z, Liu Y, Yang C, et al (2020) Learning where to focus for efficient video object detection. In: European conference on computer vision, Springer, pp 18–34 Jiang Z, Liu Y, Yang C, et al (2020) Learning where to focus for efficient video object detection. In: European conference on computer vision, Springer, pp 18–34
38.
go back to reference Xu Z, Hrustic E, Vivet D (2020) Centernet heatmap propagation for real-time video object detection. In: European conference on computer vision, Springer, pp 220–234 Xu Z, Hrustic E, Vivet D (2020) Centernet heatmap propagation for real-time video object detection. In: European conference on computer vision, Springer, pp 220–234
39.
go back to reference Zhou Q, Li X, He L, et al (2022) Transvod: end-to-end video object detection with spatial-temporal transformers. arXiv preprint arXiv:2201.05047 Zhou Q, Li X, He L, et al (2022) Transvod: end-to-end video object detection with spatial-temporal transformers. arXiv preprint arXiv:​2201.​05047
40.
go back to reference Zhu X, Su W, Lu L, et al (2020) Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 Zhu X, Su W, Lu L, et al (2020) Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:​2010.​04159
Metadata
Title
Video deblurring and flow-guided feature aggregation for obstacle detection in agricultural videos
Authors
Keyang Cheng
Xuesen Zhu
Yongzhao Zhan
Yunshen Pei
Publication date
18-10-2022
Publisher
Springer London
Published in
International Journal of Multimedia Information Retrieval / Issue 4/2022
Print ISSN: 2192-6611
Electronic ISSN: 2192-662X
DOI
https://doi.org/10.1007/s13735-022-00263-4

Other articles of this Issue 4/2022

International Journal of Multimedia Information Retrieval 4/2022 Go to the issue

Premium Partner