Top

Autonomous Robots

Published in:

21-07-2020

High precision control and deep learning-based corn stand counting algorithms for agricultural robot

Authors: Zhongzhong Zhang, Erkan Kayacan, Benjamin Thompson, Girish Chowdhary

Published in: Autonomous Robots | Issue 7/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This paper presents high precision control and deep learning-based corn stand counting algorithms for a low-cost, ultra-compact 3D printed and autonomous field robot for agricultural operations. Currently, plant traits, such as emergence rate, biomass, vigor, and stand counting, are measured manually. This is highly labor-intensive and prone to errors. The robot, termed TerraSentia, is designed to automate the measurement of plant traits for efficient phenotyping as an alternative to manual measurements. In this paper, we formulate a Nonlinear Moving Horizon Estimator that identifies key terrain parameters using onboard robot sensors and a learning-based Nonlinear Model Predictive Control that ensures high precision path tracking in the presence of unknown wheel-terrain interaction. Moreover, we develop a machine vision algorithm designed to enable an ultra-compact ground robot to count corn stands by driving through the fields autonomously. The algorithm leverages a deep network to detect corn plants in images, and a visual tracking model to re-identify detected objects at different time steps. We collected data from 53 corn plots in various fields for corn plants around 14 days after emergence (stage V3 - V4). The robot predictions have agreed well with the ground truth with \(C_{robot}=1.02 \times C_{human}-0.86\) and a correlation coefficient \(R=0.96\). The mean relative error given by the algorithm is \(-3.78\%\), and the standard deviation is \(6.76\%\). These results indicate a first and significant step towards autonomous robot-based real-time phenotyping using low-cost, ultra-compact ground robots for corn and potentially other crops.

previous article Guest Editorial: Robotics: Science and Systems 2018 (RSS 2018)

next article Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Abendroth, L.J., Elmore, R.W., Boyer, M.J., & Marlay, S.K. (2011). Corn growth and development. PMR 1009. Iowa State University

Araus, J. L., & Cairns, J. E. (2014). Field high-throughput phenotyping: The new crop breeding frontier. Trends in Plant Science, 19(1), 52–61.CrossRef

Biber, P., Weiss, U., Dorna, M., & Albert, A. (2012). Navigation system of the autonomous agricultural robot bonirob. In Workshop on Agricultural Robotics: Enabling Safe, Efficient, and Affordable Robots for Food Production (Collocated with IROS 2012), Vilamoura, Portugal

Chen, S. W., Shivakumar, S. S., Dcunha, S., Das, J., Okon, E., Qu, C., et al. (2017). Counting apples and oranges with deep learning: A data-driven approach. IEEE Robotics and Automation Letters, 2(2), 781–788.CrossRef

Das, J., Cross, G., Qu, C., Makineni, A., Tokekar, P., Mulgaonkar, Y., & Kumar, V. (2015). Devices, systems, and methods for automated monitoring enabling precision agriculture. In 2015 IEEE International Conference on Automation Science and Engineering (CASE), IEEE, (pp. 462–469).

Diehl, M., Bock, H., Schölder, J. P., Findeisen, R., Nagy, Z., & Allgöwer, F. (2002). Real-time optimization and nonlinear model predictive control of processes governed by differential-algebraic equations. Journal of Process Control, 12(4), 577–585.CrossRef

Ferreau, H., Kraus, T., Vukov, M., Saeys, W., Diehl, M. (2012) High-speed moving horizon estimation based on automatic code generation. In Decision and Control (CDC), 2012 IEEE 51st Annual Conference on (pp. 687–692).

Furbank, R. T., & Tester, M. (2011). Phenomics-technologies to relieve the phenotyping bottleneck. Trends in plant science, 16(12), 635–644.CrossRef

Halstead, M., McCool, C., Denman, S., Perez, T., & Fookes, C. (2018). Fruit quantity and quality estimation using a robotic vision system. arXiv preprint arXiv:1801.05560

Haseltine, E. L., & Rawlings, J. B. (2005). Critical evaluation of extended kalman filtering and moving-horizon estimation. Industrial and Engineering Chemistry Research, 44(8), 2451–2460. https://doi.org/10.1021/ie034308l.CrossRef

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 770–778).

He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, IEEE (pp. 2980–2988).

Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015). High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 583–596.CrossRef

van Henten, E. J., Goense, D., & Lokhorst, C. (2009). Precision agriculture. The Netherlands: Wageningen Academic Publishers.CrossRef

Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., & Guadarrama, S, et al. (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In IEEE CVPR, vol 4.

Iagnemma, K., Kang, S., Shibly, H., & Dubowsky, S. (2004). Online terrain parameter estimation for wheeled mobile robots with application to planetary rovers. IEEE Transactions on Robotics, 20(5), 921–927. https://doi.org/10.1109/TRO.2004.829462.CrossRef

Kayacan, E., & Chowdhary, G. (2019). Tracking error learning control for precise mobile robot path tracking in outdoor environment. Journal of Intelligent and Robotic Systems, 95(3–4), 975–986.CrossRef

Kayacan, E., Kayacan, E., Ramon, H., & Saeys, W. (2013) Modeling and identification of the yaw dynamics of an autonomous tractor. In 2013 9th Asian Control Conference (ASCC) (pp. 1–6).

Kayacan, E., Peschel, J.M., & Kayacan, E. (2016). Centralized, decentralized and distributed nonlinear model predictive control of a tractor-trailer system: A comparative study. In 2016 American Control Conference (ACC) (pp. 4403–4408), https://doi.org/10.1109/ACC.2016.7525615

Kayacan, E., Kayacan, E., Chen, I. M., Ramon, H., & Saeys, W. (2018a). On the Comparison of Model-Based and Model-Free Controllers in Guidance (pp. 49–73). Navigation and Control of Agricultural Vehicles: Springer International Publishing, Cham.

Kayacan, E., Saeys, W., Ramon, H., Belta, C., & Peschel, J. M. (2018b). Experimental validation of linear and nonlinear mpc on an articulated unmanned ground vehicle. IEEE/ASME Transactions on Mechatronics, 23(5), 2023–2030.CrossRef

Kayacan, E., Young, S. N., Peschel, J. M., & Chowdhary, G. (2018c). High-precision control of tracked field robots in the presence of unknown traction coefficients. Journal of Field Robotics, 35(7), 1050–1062.CrossRef

Kayacan, E., Zhang, Z., & Chowdhary, G. (2018d). Embedded high precision control and corn stand counting algorithms for an ultra-compact 3d printed field robot. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, https://doi.org/10.15607/RSS.2018.XIV.036

Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).

Lee, S.U., Iagnemma, K. (2016). Robust motion planning methodology for autonomous tracked vehicles in rough environment using online slip estimation. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (pp. 3589–3594), https://doi.org/10.1109/IROS.2016.7759528

Lee, S.U., Gonzalez, R., Iagnemma, K. (2016). Robust sampling-based motion planning for autonomous tracked vehicles in deformable high slip terrain. In 2016 IEEE International Conference on Robotics and Automation (ICRA), (pp. 2569–2574), https://doi.org/10.1109/ICRA.2016.7487413

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C.L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740–755). Springer

Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., & Berg, A.C. (2015). Ssd: Single shot multibox detector. arXiv preprint arXiv:1512.02325

Liu, X., Chen, S.W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., Taylor, C.J., Das, J., Kumar, V. (2018). Robust fruit counting: Combining deep learning, tracking, and structure from motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (pp. 1045–1052)

Liu, X., Chen, S. W., Liu, C., Shivakumar, S. S., Das, J., Taylor, C. J., et al. (2019). Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letters, 4(3), 2296–2303.CrossRef

Long, J., Shelhamer, E., & Darrell, T. (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp 3431–3440)

Lu, H., Cao, Z., Xiao, Y., Zhuang, B., & Shen, C. (2017). Tasselnet: Counting maize tassels in the wild via local counts regression network. Plant methods, 13(1), 79.CrossRef

Mayne, D., Rawlings, J., Rao, C., & Scokaert, P. (2000). Constrained model predictive control: Stability and optimality. Automatica, 36(6), 789–814.MathSciNetCrossRef

Mehndiratta, M., Kayacan, E., Patel, S., Kayacan, E., & Chowdhary, G. (2019). Learning-Based fast nonlinear model predictive control for custom-made 3D printed ground and aerial robots (pp. 581–605). Cham: Springer.

Mueller-Sim, T., Jenkins, M., Abel, J., & Kantor, G. (2017). The robotanist: A ground-based agricultural robot for high-throughput crop phenotyping. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp 3634–3639), https://doi.org/10.1109/ICRA.2017.7989418

Rahnemoonfar, M., & Sheppard, C. (2017). Deep count: Fruit counting based on deep simulated learning. Sensors, 17(4), 905.CrossRef

Ray, L. E. (2009). Estimation of terrain forces and parameters for rigid-wheeled vehicles. IEEE Transactions on Robotics, 25(3), 717–726. https://doi.org/10.1109/TRO.2009.2018971.CrossRef

Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems(pp. 91–99)

Shafiekhani, A., Kadam, S., Fritschi, F. B., & DeSouza, G. N. (2017). Vinobot and vinoculer: Two robotic platforms for high-throughput field phenotyping. Sensors, 17(1). https://doi.org/10.3390/s17010214

Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

Stein, M., & Bargoti, S. (1915). Underwood J (2016) Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors, 16(11). https://doi.org/10.3390/s16111915

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9)

Young, S. N., Kayacan, E., & Peschel, J. M. (2019). Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum. Precision Agriculture, 20(4), 697–722.CrossRef

Title: High precision control and deep learning-based corn stand counting algorithms for agricultural robot
Authors: Zhongzhong Zhang
Erkan Kayacan
Benjamin Thompson
Girish Chowdhary
Publication date: 21-07-2020
Publisher: Springer US
Published in: Autonomous Robots / Issue 7/2020
Print ISSN: 0929-5593
Electronic ISSN: 1573-7527
DOI: https://doi.org/10.1007/s10514-020-09915-y

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 7/2020

A unified kinematics modeling, optimization and control of universal robots: from serial and parallel manipulators to walking, rolling and hybrid robots

Nonlinear observability of unicycle multi-robot teams subject to nonuniform environmental disturbances

Guest Editorial: Robotics: Science and Systems 2018 (RSS 2018)

Planar max flow maps and determination of lanes with clearance

Wind disturbance rejection for unmanned aerial vehicles using acceleration feedback enhanced method

Multi-objective drone path planning for search and rescue with quality-of-service requirements