AI in Current and Future Agriculture: An Introductory Overview

As pointed out in Sect. 2, many agricultural applications require a reliable model of the environment and suitable methods to interpret environment data. In autonomous agricultural machines or assistive systems, collecting information about the environment is of critical importance.

Localisation and navigation are among the core tasks that help guide a machine on a field (c.f. Sect. 3.4). Perception systems are used to support such tasks by detecting, for example, crops and rows [4, 11, 45, 78]. This could occur before harvesting or weeding processes and can rely on top-view images from drones [11, 142]. Real-time detection systems have been developed for decades, based on e.g., the Hough transformation [9], the random sample consensus (RANSAC) algorithm [195] or masks based on vegetation indices [4]. Furthermore, crop poses on semantic maps enable localisation with visual features [27]. Obstacle detection is essential for safety reasons, and agriculture is no exception. In contrast to autonomous driving on roads, classification of the obstacle is mandatory to distinguish between plants, which are to be harvested and therefore touched, and objects that might cause a harmful collision [29].

Pose estimation is another important task, necessary when objects need to be manipulated (e.g., by grippers, pickers or other special purpose end effectors) as is the case in applications such as weeding [112, 113, 118, 158, 198] as well as harvesting and fruit picking[7, 10, 117, 182].

While classical object detection and pose estimation algorithms often provide only axis-aligned bounding boxes, weeding requires tight boundaries between crops and weeds. Instance segmentation [113, 122, 158] addresses this problem by providing individual polygons on single plants or plant groups. RGB and NIR data are combined in the normalised difference vegetation index (NDVI)[17, 65, 158], and crops and weeds can be distinguished by local binary patterns (LBP) [17, 117] or covariance features [17].

As pointed out in Sect. 2.5, handling single fruits of high-value crops, like sweet pepper or strawberries, is important since their price directly correlates with their quality [7, 10, 182]. Pose estimation is fundamental for automated fruit harvesting [56, 139], so object detectors like SSD [137, 145] or instance segmentation algorithms like Mask R-CNN [56, 139] are often used together with RGB-D data. Harvesting selection is based on different quality parameters [35, 115], such as firmness and ripeness. This can be implemented through vision systems, albeit clutter and occlusions pose major challenges [139]. In [115], quality measures get derived from Near-Infrared bands (NIR). Firmness, for example, can be determined from hyperspectral data with regression models [35].

Other applications of sensor data processing include plant monitoring, for instance through a perception system that can track the growth state of a plant [99] or detect pests and diseases. This allows an early intervention and is often approached using convolutional neural networks (CNN)[169, 180, 199]. Region Proposal Networks like Faster R-CNN-based architectures [180] and augmentation learning [128] can be used to detect citrus diseases like black spot.

Although sensor data may be collected and interpreted in various ways and for different reasons, we can identify a common core of four main techniques:

In recent years, many such models and algorithms were released and have continued to increase their levels of performance [205]. Perception algorithms are mostly trained for specific tasks and domains, and the underlying datasets constitute a determining factor on the resulting applicability and performance of algorithms. Publicly available datasets exist, which contain plenty of common objects in everyday scenes with some focus on urban or driving scenarios [49, 57, 108, 157]. In general, the development and evaluation of AI-based perception systems requires large amounts of data, together with a suitable ground truth. Preparing these datasets is a core challenge when bringing detection systems to the market. Especially in the agricultural sector, the prevailing variety of scenarios and environmental conditions must be taken into account.

In addition to common object datasets, off-road and agricultural scenarios have also been made available. Marulan [143], RELLIS-3D [79], NREC [144], Robot Unstructured Ground Driving (RUGD) [193] and FieldSAFE [94] in particular can be used to train and evaluate environment perception algorithms that deal with obstacle detection. For 3D Mapping and localisation, the Bacchus Long Term Dataset [146] contains RTK-GPS and IMU, as well as RGB-D cameras and 3D LIDAR data. The data was recorded in vineyards over an entire growing season. In agriculture, environmental conditions are often harsh and changing, due to weather and vegetation. These conditions are often just partly covered, instead of comprehensively for the whole operational design domain. For this reason, recording additional data is essential to develop and validate application systems suitable for commercial use. This can be done via sensors mounted on vehicles like tractors and trucks [93, 196, 153].

While data recording with vehicles offers flexibility, rail-guided test stands like AgroSafety [120] and AI-TEST-FIELD [95] are designed to achieve higher comparability between test runs under many different environmental conditions. For the comparability of different test runs, dummies are usually employed as test specimens to represent real obstacles. In order to standardise the evaluation, ISO 18497 [1] describes a barrel-like obstacle, which roughly compares to a seated person. A more realistic replicate of a human is proposed in ISO 19206-2 [2]. The reflectivity of test targets should represent the worst-case scenario, so [121] proposed a modified pedestrian test target based on ISO 19206-2 [2] with new cotton material.

In addition to the general perception of the surroundings, process-related perception algorithms are essential in agricultural domains and need to be trained accordingly. To that end, published datasets are available including the Swedish Leaf Dataset [173], which contains images of single leaves before a white background and can be used to classify different Swedish tree classes. Other datasets like Flavia [197], Leafsnap [96] and PlantVillage [126] contain images of plants under laboratory conditions. The Crop/Weed Field Image Dataset (CWFID) [64] contains 60 RGB and hyperspectral images, recorded by the autonomous field robot Bonirob in an organic carrot farm, that can be used for single plant phenotyping. The Vegetable Crops Dataset for Proximal Sensing (VCD) [100] contains RGB images of vegetable crops at an early stage of growth. The Plant-Doc dataset [171] contains around 2500 images of plant species with up to 17 classes of diseases. A crowd-sourced dataset with more than 50,000 images also exists, created on the PlantVillage platform [73]. Additionally, 375 pixel-wise labeled sugar beet and weed images (RGB and NIR) are published in the weedNet dataset [158].

Since there might not be free or public datasets available for every application, data recording becomes necessary in many cases but data acquisition comes with challenges of its own. Existing work includes that of Sankaran et al. [162], who provide a survey of sensor systems for phenotyping. Recording can be conducted on different devices such as UAVs [168], RC-Copters [26], airships [107], robots [184], self-propelling chairs [6, 33, 80], handcarts [12, 77, 192], spidercams [13, 89] or specific test stands [85].

In general, collecting data is a challenge and calls for special care. Data acquisition and labelling are costly and time-consuming tasks, especially when pixel-wise annotation is required as is the case in semantic instance segmentation. This can become very challenging very quickly if the labelling processes is manual, especially when dealing with occluded and young plants. Hence, synthetic training data has become more prominent, since images can be generated from models and the ground truth is known in advance for every pixel [16, 38, 77].

One last issue in dataset creation and usage that affects all topics in computer vision, are biases in the data [50], and agriculture is of course no exception [135]. Seasonal vegetation and irregular occurrences of diseases, pests, and weeds exacerbate the challenge.

Ontologies and knowledge graphs are commonly used representation methods with connections to semantic web technologies which allow these models to be stored and queried in a distributed manner. Examples spanning various sub-domains of agriculture include [21, 24, 66, 83, 104, 156]. A comprehensive overview can be found in a 2019 survey [43], where the authors argue that the increasing amount of sensor data in the agricultural domain has little benefit to the farmer in its raw form, and that its value lies in the insight gained by processing such data with suitable technologies.

Among the more representative semantic technology tools used in agriculture are AGROVOC [23], a type of thesaurus under the supervision of the Food and Agriculture Organization (FAO), the Crop Ontology [116], the Agro Portal [82] and, on the topic of plant science, AgroLD [103]. It should be noted, however, that most of these examples either refer to very specific subfields or take a very general approach to agriculture, and not much work exists that addresses the practical aspects in between. Some recent work also discusses the connection between ontology modelling and data mining, and provides an example application in crop farming [133].

Whether the input data is obtained automatically via mining and whether the result is an ontology or not, these models ultimately fulfil the need for a repository or “knowledge base”, often used in combination with an inference engine or some other information processing tool to generate useful or meaningful results.

A decision-support system (DSS) is a tool that relies on various AI techniques to process multiple types and sources of information, in order to provide useful and non-trivial insight that may inform the decisions made by an external agent. In agriculture, information sources might include a description of crops, fields, the soil and the weather forecast and the system may generate suggestions to facilitate e.g., farm-level management. DSS technologies and expert systems in general have a wide array of applications but, as some studies show, agriculture was not a primary focus at least until the turn of the current century: in a survey spanning the period between 1995 and 2001, the authors found only a couple of agricultural applications [46]. Furthermore, these few examples constituted more traditional computer programs than knowledge- or inference-based AI systems. Since then, and perhaps as a result of considerable improvements in the acquisition and storage of large amounts of data, there appears to be some renewed interest, particularly when the inference component of a DSS is combined with automated data analysis and machine learning tools.

DSSAT is a current example that focuses on Ag Tech transfer and is capable of simulating crop growth, development and yield [70, 71]. Other existing systems provide support in a variety of agricultural tasks ranging from farm operation through scheduling and optimisation [150], irrigation activities based on fuzzy inference [60] or a fuzzy neural network [130], as well as impact assessment with respect to sustainability and climate change [163, 191]. A more in-depth discussion of these and other support systems can be found in a recent survey [203]. In this paper, the authors compare the performance and functionality of several agricultural DSSs in the era of remote sensing, large amounts of data, cloud computing and artificial intelligence, which is collectively referred to as “Agriculture 4.0”.

From a functional perspective, planners and decision-support systems are both AI technologies that can inform human activities, but a DSS is limited to generating suggestions or contextual information while a planning problem is solved by identifying an optimal sequence of actions, meant to be executed. Similar to planning, optimisation approaches generate optimal value assignments that may be pursued or implemented directly.

An exemplary approach is the unmanned monitoring of crop health utilising a UAV capable of choosing its own paths and targets, in order to perform a close inspection, apply herbicide or collect higher-resolution images [5]. Work also exists on route planning and optimisation that can guide auto-steering and navigation-aiding systems [20]. Another approach uses a combination of aerial and ground vehicles, where information about crops and weeds is gathered from the air and used to plan possible interventions, reducing the application of herbicide [30].

AI-adjacent topics such as control optimisation may also have an impact in agricultural applications, as exemplified by a drip irrigation system that approaches moisture transport in unsaturated soil as a linear optimisation problem, providing correct amounts of water [91]. Another optimising approach focuses on agricultural resource management and suggests optimal land allocation for diversified crop planning, relying on combinatorial methods that consider socioeconomic and environmental objectives as well as farm- and district-level goals [160].

Finally, a recent paper shows that the application of optimisation methods through multiple AI technologies can lead to more efficient crop protection by applying more suitable products at the right time, in the right amounts and only where necessary, which also may contribute to reaching the United Nations Sustainable Development Goals of responsible consumption and production [166].

Planning and optimisation techniques are commonly used as components of autonomous robotics and agricultural vehicle platforms, discussed in more detail in Sect. 3.4.

Field robots are autonomous systems oftentimes described as the technological means to meet ambitious goals, regarding sustainability and development of the agricultural sector [181]. Robots are also a particularly good choice for the application of AI as they can, in principle, address many agricultural domains and integrate – to some degree – many if not all of the technologies provided by AI [97]. In the context of the very active field of object detection with machine learning techniques, robotics can be seen as the actuating counterpart to the advances in perception and sensing [161]. The large amount of papers on robotics applications, however, makes it difficult to provide an exhaustive overview of a field as active as agricultural robotics.

Topics of special importance to mobile robotics include localisation and path planning, sensing, and actuating, which can be found at different levels of technological maturity [200, 202].

In the last years, a rich account of surveys has been published hinting at a plenitude of prototypes, in general, [81, 149, 189] and with a focus on monitoring and phenotyping [15, 52, 76, 147], seeding[52], weeding [47, 52, 63], and harvesting [8, 52]. Fundamentally, it can be observed, that recent developments in image processing have led to an enormous increase in work related to monitoring and detection use cases.

Actuation on the other hand, and therefore fully integrated farming robots, accounts for much less published articles as the required mechanical components are still under development. Especially in the fields of weeding [17, 119, 148, 185] and harvesting [7, 167, 170] (see also 3.4.2 below), there are some recent examples of working prototypes in both science and industry.

It is worth mentioning that the ideal of fully automating food production on farms is not devoid of social and ethical considerations. Farming continues to play a fundamental role in the course of societies worldwide, and continues to involve small stakeholders [40, 176]. The farmers’ own perspectives are of great importance, as robots are expected to become commercially available. Work such as [177] becomes particularly interesting also from the technological perspective, as issues regarding human-robot collaboration, but also the choice of size and a fitting role for robots on farms will depend on commercial demand.

Additionally, integrating multiple robots in swarms as well as enabling and orchestrating cooperation between robots, and between humans and robots, seems to be a promising trend as suggested by several studies [3, 40, 114].

An additional perspective, albeit not yet as visible in the community, is that of the expected long-term behaviour of autonomous systems in agriculture. Robots in this context have to deal with a dynamic, partially controllable, and partially observable environment. When tasks take long enough to complete that such dynamics become relevant, for instance when tending to a large field, or when tasks must be repeated multiple times, as is the case of weed management, dealing with these aspects becomes a complementary but no less important challenge that poses many interesting research questions. In 2017, some aspects of long-term autonomy were examined in an indoor setting [67], while the authors of [97] describe different technologies necessary to enable long-term capabilities of robots in different contexts, including agriculture. In these examples, long-term autonomy is described as a central challenge based on the integration of various building blocks.

Physical interactions between robots and the environment demand special attention and the development of well-suited tools. If conventional implements are to be used, this is often a matter of interfaces and control; when a robot is supposed to fulfill a task typically performed manually, like the plucking of larger fruit, new types of tools or approaches are necessary.

As mentioned in section 2.5, there is a wealth of ongoing research on special end-effectors and their integration into fruit harvesting robots operating in both fields a greenhouse contexts. Examples include tomatoes [84], pumpkins [155], sweet pepper [8], strawberries [201] and others, although many questions remain unanswered so far what harvesting strategy and which tools will prove best. Soft end-effectors which are experimented on but not yet widely used might prove as a promising avenue for further investigation [131].

Judging by the abundance of different individual solutions for different fruits, it seems that it is not easy to transfer robotic systems from one use case to another. This might have to do with the detection systems or individual requirements of the end-effectors and future research seems worthwhile.

In agriculture, drones are typically used as low-cost but efficient sensing platforms, especially for RGB aerial imagery but other sensor data such as lidar or multi-spectral may also be collected, although less commonly. Overlapping with the field of remote sensing, drone images are used for plant monitoring, especially for pest and disease control [32, 37, 47, 109, 134, 167]. Although less frequently due to challenges in engineering and administration, drones can also be used to precisely spray pesticides onto small patches to drastically reduce overall agrochemical usage [61].

Although originally dominated by academic institutions, in the last decades the industrial sector has also joined the field of robotics research and prototyping. Small and large corporations alike have contributed their own innovations and initiated collaboration efforts with the scientific community. Websites like ducksize.com,³ industry-oriented conferences like World Fira, or the robotics components in Ag-Tech fairs such as Agritechnica can offer a good starting point to showcase agricultural robots, in the form of products and prototypes both currently available and under development. As previously mentioned, robotic systems come in many shapes and configurations, but a coarse distinction can be observed between hybrid or upgraded conventional machines, large robots without driver cabins, and small or very small robots that might be used in newer and less conventional ways.

Despite the progress illustrated by such products and prototypes, many open questions still remain regarding architecture, data sovereignty, and interfacing, in addition to technological challenges. Especially in larger machines, which are meant to be coupled with different farming implements for multiple farming tasks, there are several open problems related to interfacing, compatibility, and automated decision-making in addition to concerns about functional safety.