Data analytics for crop management: a big data view

DA, (also called digital farming or smart farming)¹ [78, 105, 130], is a modern approach that uses digital and smart devices [sensors, cameras, satellite, drones, the Global Positioning System (GPS)] in conjunction with Data Mining (or data analytics) to improve productivity and to optimise the use of resources. Digital Agriculture (DA) comes as a response to the increasing demand for improving productivity while reducing farming operational costs. Moreover, the improvement of productivity should not be done at any cost, e.g., overuse of natural resources and chemical products. DA can, for example, manage crop growth by finding appropriate fertilisation program for each farming field and can help farmers to reduce their operational costs and respect the environment by refining their farming operations based on the needs of each part of the farming field.

Since agriculture has a direct and significant impact on the population and therefore its economic environment, DA in its turn should be viewed as the next natural step to respond to the world population’s needs while protecting the environment, by taking advantage of the recent technological advances in digital devices, communications systems, and artificial intelligence. These allow us to construct multidimensional domains, where the farms and farmers are their central subjects. Figure 1 shows the agriculture ecosystem and its direct impact on other sectors of the economy.

Besides, since DA involves the development, adoption and iteration with digital technologies [39], and Artificial Intelligence (data analytics, ...), these developments and interactions should be well-defined (laws, regulations and policies) to guarantee rights and benefits of all the involved actors (farmers, farm holders’, data owners’, developers and analysts, technology vendors’,...) [70, 77, 78, 92, 113, 146].

DA can be regarded as a data driven form of farming, in which decision-making processes are based on explicit information derived from data collected through various sources [148]. DA and Precision Agriculture (PA) seem to refer to the same thing, however, as stated in [148], DA involves the development and adoption of modern technologies in both collecting the data and its analysis in various farming contexts, while PA takes into account only the in-field variability [147]. DA aims to exploit advanced digital devices, ranging from a simple sensor to complex robots, to offer the required farmland treatment with high accuracy. DA can be applied in almost all agricultural fields. For instance, in crop production: DA allows accurate management of crops, which includes fields, wasteland, crop, pest, and irrigation management, soil classification, etc. In Animal production: DA allows monitoring the animal over its whole life cycle, its food quantity, health control and protection from diseases, and so on. Fishery, animal Husbandry, livestock and dairy farming are some examples [14]. In Forestry: We can efficiently manage forests by supporting the environmental and sustainable decision [36]. DA can help in detecting unhealthy trees, air pollution, discriminate different tree species, protect the wildlife, etc. From the economy point of view, the application of DA for forest management enhances the wood quality and its production, which can augment profits; reduce waste and maintain the environment [138].

Addressing DA from all the above mentioned views is a challenging task and cannot be achieved without the participation of specialists from all these sectors. In this study, we focus on the use of Big Data in crop management, it is, not only one of the pillars in agriculture but also it can profoundly affect biodiversity. Moreover, crop growth is a very complex process involving various endogenous and exogenous factors. Recent advances in digital technologies allow us to collect data about all these factors. DA has the ability to elucidate the correlations and interactions of these factor to help farmers and agronomists optimise the productivity while reducing the side effects on the environment. DA exhibits several benefits to agriculture as shown in Figure  2. These benefits were discussed in [10, 13, 70, 98, 104, 112, 113, 130, 135, 148] and summarised in the following:

The contributions of this study are in the investigation of big data analytics applications to crop production. Crop farming is a complex task, and it depends on many factors that should be taken into account. To optimise the operational cost and reduce the impact on the environment, the big data analytics emerges as one of the most cost effective approaches nowadays. The contributions, therefore, include the following:

To study the impact of data analytics and big data on DA based on previous works, we conducted a systematic review approach that consists of three steps: (1) collection of related work, (2) selection of relevant work, and (3) examination and analysis of the filtered related work.

In the first step, we performed keyword-based research and We gathered a large number of studies from well-known and popular online sources (Web of Science, Scopus, IEEEXplore, ACM, etc.). We used a combination of keywords from the two sets (Big data, data mining, data analytics, machine learning, Internet Of Things, sensors) and (Digital agriculture, smart farming, precision agriculture, agriculture, farming). We gathered more than 327 articles. In the next steps, We selected a small number of articles, which are considered relevant for further analysis, based on their ideas, methods, data types and sources, addressed problems, proposed solutions, tools used and quality of the results.

Through the literature analysis, the study aims to find responses to the following research questions and discuss findings in the following sections.Figure 3 summarises the overall approach, adopted from the PRISMA² flow diagram.

Despite that DA and Big Data being relatively recent research fields, their scientific literature is rich and covers several concepts. As DA is at the cross boundaries between agriculture and ICT, three major dimensions have emerged as of a very high importance; technology, social economics and ethics, and decision-making based on Machine Learning. The first dimension focuses on the use of advanced technologies to improve practices and productivity [56, 124]. In Ref. [124], the authors studied the impact of sensor networks in agriculture, including remote sensing technologies, wireless devices, and other IoT devices. Ref. [56] reviewed some developments in remote sensing within Big Data processing and management in agriculture. The second dimension concerns legal, ethics, social and economic factors of DA, to provide insights into the impact of digitised information and its analysis on the farm management; farmer identity, skills, privacy, production, and value chains in food systems [39, 70, 77, 78, 92, 113, 146, 148]. The third dimension focuses on the application of big data analysis and machine learning (ML), to optimise and forecast the production and the use of resources. In this paper, we only consider this dimension.

Various studies have been conducted on the application of data analytics to crop yield management. For instance, [71] presented a systematic review on crop yield prediction using ML techniques, and extracted major ML algorithms, features and evaluation metrics used in those studies. Ref.[35] discussed the yield estimation by integrating agrarian factors in ML techniques. This allowed them to show a strong relationship between crop yield and climatic factors. Ref. [103] Provided a systematic review on the use of computer vision and AI to enhance the grain quality of five crops (maize, rice, wheat, soybean and barley), disease detection and phenotyping. Ref. [64] reviewed the application of big data analysis in some fields of agriculture. It highlighted solutions to some key well-known problems, used tools and algorithms, along with input datasets. The authors concluded that big data analytics in agriculture is still at its early stage, and many barriers need to be overcome, despite the availability of the data and tools to analyse it. To measure the level of usage of big data in DA, the authors defined big data metrics (low, medium, high) for each of its dimensions (volume, velocity, and variety). However, while it is a very simple model, it is not easy to specify thresholds, as some dimensions, such as volume and velocity depend on technological advances. Ref. [12] presented a review on the use of ML methods to detect biotic stress in crop protection. The authors analysed the potential of these techniques and their suitability to deal with crop protection from weeds, diseases and insects. In addition, they provided very good instructive examples from different fields of DA. An earlier similar study was presented in [89], where the authors studied four very popular learning approaches; Artificial Neural Network (ANN), Support Vector Machine (SVM), K-means, and K-Nearest Neighbour (KNN). Ref. [25] presented a survey on data mining clustering methods applied to food and agricultural domains. It first described major techniques of unsupervised classification, then it examined some existing techniques applied to agriculture products; like fruit classification, wine classification, analysis of remote sensing in forest images and machine vision.

This study is not just an update of previous surveys. The main objective is to examine the effectiveness of big data analytics in crop yield monitoring and discuss the challenges of such paradigm shift in the agriculture domain. Moreover, It is important to understand the sources of datasets, their types, and which ML techniques are more suitable to analyse them.

Digital Agriculture (DA) relies heavily on the data sources and techniques used to collect it. This data is then organised in agricultural data warehouses and analysed [93]. The results of this data analysis provide significant insights to farmers and agronomists about how to improve the production, minimise the farming operational costs, manage risks, and protect the environment. The process of deploying DA is derived from data science.

Big Data analytics system architecture is depicted in Fig.  5. While this system is targeted specifically to crop yield management, it can be adapted to any data-driven application. This architecture implements faithfully what we have highlighted in the previous sections. In this section, we will focus on the data analysis layer of the architecture, moreover, we will pay attention to the data types and their sources, techniques of data acquisition, the learning algorithms. The main objective of the crop management data analysis is to get some insights about the crop monitoring problems and show the potential of DA through big data analytics, also called data mining. Data mining and its techniques are involved in several roles in crop production. Farmers may want to know the future yield of their crop, specific areas of their farms suffer from the spread of weeds or under-nutrition. Researchers can look for information such as plant growth patterns, optimum growing conditions, best pest and disease control environment and so on. Data mining offers panoply of sophisticated techniques required to meet all of these needs.

There are two major categories of data analysis: Classification and Clustering. In the work of [24], authors studied applications of data mining techniques in crop management and proposed a classification of these applications. They found that the classification and clustering are the main used categories, where the classification includes prediction, detection, protection, and categorisation). The choice between classification or clustering analysis is very simple. If the models or classes we are looking for were known in advance and we have an annotated data to support the training of the learning algorithms, then classification is the right choice. However, the annotated data is not always available and easy to generate, and in many cases we do not know even which models or patterns we are looking for. In these situations, clustering analysis is the right alternative.

In this section, we focus on the studies that use classification methods for their data analysis. Clustering analysis will be covered in the next section. We structure these classification studies based on the application objectives or targets which arecategorisation, prediction, detection, and protection.

Clustering techniques are not widely employed in DA, few efforts have been deployed to investigate the potential of these techniques for zones’ delineation within a field. There are several reasons for splitting an agricultural field into zones. Some traditional reasons include crop diversification within a field, crop-rotation, facilitating the management tasks, and more recently we defined the zones based on yield maps. This usually helps to improve the overall crop yield of the field, by managing the zones more effectively. Therefore, delineation of Management zones (DMZ) is a very important task for farming operations since determining zones of low-or-high yields, and understanding the reasons behind low yields, can help come up with specific solution for each zone with the view to increase the yields. In addition, it has other economic benefits, because we can target each zone with the right amount of fertilisers, water, and other nutrients.

According to [69], delineation of management zones is an effective way to manage the variability of soil within a field, such that each zone will receive specific management. In [145], a management zone is defined as a subregion of a field that has a relatively homogeneous combination of yield-limiting factors, for which a single rate of a specific crop input is appropriate to reach maximum efficiency of farm inputs. In [53], it is defined as a subregion of a field that is relatively homogeneous with regard to soil attributes.

DMZ is a complex spatial problem, which is addressed in the literature from several perspectives. This has attracted interest from many researchers [61, 85, 87, 110, 140]. A literature review has been presented in [90], where the authors discussed the delineation of soil management zones from the variable-rate fertilisation point of view. many other studies presented the delineation based on various criteria. Some techniques that have been used include topographic maps, direct soil sampling, non-invasive soil sampling by electrical conductivity equipment, soil organic matter or organic estimated by remote sensing, and yield maps built using data collected over several seasons/years [99].

Figure 6 depicts the general process of delineation of management zones designed according to methodologies followed by the majority of the literature.

The majority of problems that are related to crop management imply the management of fields and zones. Therefore, the collected data is usually characterised by geographic coordinates and time associated with each sample, which leads to the use of data mining techniques that are more suitable for spatial and temporal datasets. It is well recognised that agricultural datasets are typically spatio-temporal, as the data is always associated with location and time. However, these datasets contain a significant amount of noise, outliers, and even missing values. For instance, GPS capture devices introduce some noise, imprecisions, and even outliers in the data. Satellite imagery also faces huge imprecision and noise (such as clouds, ...).

Because of the type of the datasets, which is spatio-temporal, it is not surprising to notice that the majority of the clustering algorithms used are of type partitional. K-means and Fuzzy C-Mean (FCM) are considered among the most popular clustering techniques and heavily used to cluster agricultural data [17, 18, 84, 134, 137, 142, 151, 154]. The FCM approach has an advantage over K-means, as it deals better with imprecision and noisy data. Moreover, other types of clustering algorithms have also been proven to be efficient in DA, such as density-based and hierarchical-based clustering techniques applied to DMZ [48, 116].

As mentioned above, besides its huge importance in crop management, delineation of management zone (DMZ) has received much attention, as the data is now available not only from traditional sources but also from refined sources, including advanced data pre-processing techniques. In addition, the recently collected data integrates knowledge of experts and farmers experiences on their fields, which improves significantly the quality of the data [84, 141]. Advanced imaging enhancement techniques improve further the data quality, and they offer the ability to track the development of crops and provide a Geo-referenced data that can describe the spatial and the temporal variability of soil and crops variables at high resolution, covering large areas [17, 84, 101, 132, 133, 141, 151].

In the following we will explore the application of data analytics in DA and its extension to big data, and illustrate the practical challenges that hinder the full adoption of DA by farmers.

Digital agriculture (DA) is a data-driven approach that exploits the hidden information within the collected data to gain new insights; transforming the farming practices from intuitive-based decision-making to informed-based decision-making. DA relies on efficient data collection practices, efficient data preparation and storage techniques, efficient data analytics, and efficient deployment and exploitation of the gained insights to make optimal farming decisions.

In this study, we presented a systematic review of the potential use of the data mining process in crop production and management and highlighted serious gaps which can be considered in future studies. The majority of the current practices were dominated by statistical analyses and small machine learning systems. However, these can only give some ideas within a very limited view of the overall system. Agricultural data-driven applications collect a significant amount of data from various sources. This constitutes an excellent opportunity to the field to answer numerous research and practical questions that were not possible before. Nevertheless, despite all the advantages that can be gained from DA, there are several other challenges and obstacles that need to be addressed, among them lack of data, lack of skills, and lack of maturity and standards so that it can be adopted and deployed quickly and easily.

In this study, we cover approaches that deal the entire process of data mining; from data collection to knowledge deployment. We cover this process from big data view, with more focus on crop monitoring and management in an attempt to understand the challenges that DA is currently facing. We defined the research questions addressed by the study and provided a classification of data mining techniques used in the field. For each class, a set of representative existing works have been reviewed, and an analytical study has been provided to highlight the category of machine learning method applied and for which purpose. We discussed the big data concepts and its current impact on DA, and showed that from the data analyst’s view, the transition towards DA is ready to embrace big data analytics concepts. This provides new opportunities of investment into these challenges and allows for a efficient ways of managing crops. Besides, it will provide farmers with new insights into how they can grow crops more efficiently, while minimising the impact on the environment. It also promises new levels of scientific discovery and innovative solutions to more complex problems.