Crop yield prediction is an essential tool for food security research, agricultural resource management, climate risk assessment, climate adaptation, and policy decisions in a changing climate. However, projecting crop yield is fraught with uncertainty, depending on the quality of input data and models at every turn¹. Crop yield is a complex trait and accurate crop yield prediction requires the use of data on a multitude of variables that have been demonstrated to impact crop plant growth such as climate, soil, and agricultural inputs². The impact of all these variables differs between crop cultivars and genotypes, and changes throughout development. The two major modelling approaches for crop yield prediction are process-based (dynamic) crop models and data-driven approaches³. The difference between these two approaches is that process-based models are driven by the physiological principles of plant growth, drawing from the domain expertise and theory of fields such as crop physiology, crop ecology, meteorology, and soil science, while data-driven model parameters are derived directly from the data, using domain knowledge to inform selection of model inputs^4,5. The importance of collecting data for dynamic crop models has been extensively discussed in a previous paper by Corcoran et al.⁶. In this paper we focus on the importance of data for data-driven approaches, with an emphasis on those that apply machine learning.

The term “data-driven modelling” (DDM) refers to the overarching paradigm of using historical data derived from real-world systems in conjunction with advanced computational techniques, including machine learning and artificial intelligence, to create models that can reveal underlying trends and patterns, and potentially predict future outcomes^3,5. Unlike process-based models, data-driven models can optionally be developed and trained to make predictions without detailed knowledge of the underlying processes governing the system behaviour, which makes these models particularly valuable when knowledge of these processes is incomplete^7,8. In this regard, machine learning-based approaches have been used with increasing frequency in agriculture and crop yield prediction over the past several years due to their ability to model non-linear processes and complex relationships between multiple inputs⁹. Another advantage of machine learning approaches is that regularisation techniques can be applied to these models, which have been shown to improve accuracy when dealing with noisy data⁸. However, a major trade-off for the flexibility of these approaches is that machine learning algorithms require larger amounts of data compared to process-based models to distinguish between genuine patterns and noise, accompanied by a risk of overfitting¹⁰.

To reduce the risk of overfitting and improve the generalisability of machine learning approaches, large amounts of high-quality training data are required to train models¹¹. Ground truth data, direct observations or measurements that are known to be real and true, are of particular importance when developing machine learning models as they can be used to quantify how accurately models perform, a process commonly referred to as model validation¹². Ground truth data that can be used to develop and validate machine learning models for crop yield prediction are also potentially beneficial for testing and verifying the accuracy of predictions made with conventional process-based models, as the actual yield observed can be compared to the yield predictions, providing evidence for the robustness and generalisability of process-based models¹³. However, for many scientific domains, scarcity of data on which to train models, and in particular scarcity of ground truth data for use in validating models, is a major constraint on development of machine learning models¹⁴. A rise in the availability and accessibility of spatiotemporal data with high coverage collected by remote sensors on the climate, soil, and land use variables that impact crop yield has provided a potential way to gather large and varied training datasets for crop yield prediction, but efforts to develop these models thus far have been limited by a lack of ground truth data⁵. Direct observations of yield have often only been available for a small number for fields for limited crops, resulting in a lack of generalisability of machine learning models to areas and crops outside those included in the restricted validation dataset⁸.

The widespread adoption of precision agriculture in recent years has meant that huge volumes of data on crop yield at high spatial resolutions are routinely collected by farm machinery^15–17. These data are of great potential benefit for training and validating yield models, including machine learning approaches, which can then be used to extrapolate predicted yield beyond the locations and time frames over which precision yield data were collected. Many studies have successfully demonstrated these approaches, using a wide variety of satellite data sources and indices as predictors of yield^18,19. However, fundamental constraints on building and using such models are the restrictions around access and use of precision yield data. Although large data volumes are widely collected, these data remain the property of individual farmers, so such data are commercially sensitive and not widely accessible. Even when precision yield data are made available to researchers, they are not generally approved for wider publication¹⁹. This is an issue in common with much data on agricultural practice, from pesticide use²⁰ to agri-environmental uptake²¹. Therefore, to use precision yield data effectively in the production of yield models for widespread deployment requires procedures to ensure that the source and location of precision yield data can be effectively anonymised. In this study, we aim to address the scarcity of ground truth data for use in the development of machine learning models and validation of process-based models for yield prediction for crops in the United Kingdom (UK). To address this issue, we have assembled a dataset of anonymised yield data derived from precision agriculture, as well as data on crop type, soil conditions, daily time-series of climate data and 5-day time-series of Sentinel-1 Radar Band data for 2015–2017. This Crop Yields, Climate, Soils, and Satellites (CYCleSS) dataset will provide invaluable resources for crop yield prediction across the UK and its relative issues. It will also help to promote the FAIR Guiding Principles (in which data should be findable, accessible, interoperable, and reusable)²² and in facilitating data-driven modelling in environmental and agricultural research.

A potential use of the CYCLeSS dataset is in the development of machine-learning based models to predict field-scale crop yield for the UK. The CYCLeSS dataset may be particularly beneficial for the development of deep learning approaches, such as convolutional neural networks (CNN), deep neural networks (DNN), and long-short term memory models (LSTM). These methods have been found to produce accurate and efficient results when previously used to predict crop yield in other geographic areas and for different scales^23–25. There are two main aspects of the CYCLeSS that address current data gaps that have hindered development of a large-scale crop yield model for the UK. Firstly, a lack of ground truth yield observations on which models can be trained and validated^6,7. The inclusion of anonymised precision yield data within the CYCLeSS dataset addresses this gap, providing field-level observations of yield that can be used to assess the accuracy of models through comparison to the yield predicted by machine learning models. Secondly, development of a UK wide crop model has thus far been limited by the differing scale and coverage of available data of factors potentially impacting crop yield such as climate and soil conditions^5,26. The CYCLeSS dataset addresses this data gap by aggregating and aligning climate, soil, and satellite data to the same 1 km² grid. Since this climate, soil and satellite data is available for the entirety of the UK, the process of aggregation and alignment outlined in this study can be used by future researchers to efficiently retrieve data on grids beyond those included in the dataset in order to test model predictions on novel data. This process can also be used to guide retrieval of climate, soil and satellite data for areas where further yield data becomes available, allowing expansion of the model training and validation dataset, or fine-tuning of trained models. The dataset also provides information on temperature, rainfall, and soil type, which have been found to be among the most commonly used inputs into previous predictive machine-learning and deep-learning models for crop yield⁵.

This dataset can also be used to assess the validity of models built and parameterised using spatially more limited, but otherwise richer sources of data. The scope of the models that could be assessed in this way is broad, from (semi-)mechanistic, process-based models (such as APSIM²⁷ or Sirius²⁸) to more empirical, statistical regression models. Assessing how well such models can predict yield responses across a broader landscape is an important component of the development and out-scaling of these models to provide Digital Twins for crop productivity beyond the field or farm scales at which the models are usually developed²⁹. For all models, the main limitations in the use of these datasets are the matching of the input variables for the model to the variables that are available for each of the locations in the dataset, and the availability of suitable values for any parameters that need to be set for each location³⁰. The CYCLeSS dataset addresses this limitation by providing a link between precision crop yields, and the soil type and daily weather data associated with each location. Comparing model predictions to the observed yield data provided in the CYCLeSS dataset will then both allow the assessment of how widely a model can be implemented, and potentially identify the location-specific information that is required for the models to provide more reliable predictions. As an illustrative example, Addy et al.³¹ constructed a model for the impact of inter-annual weather variation on the yield response of winter wheat to applied nitrogen inputs using multiple years of yield data from the Broadbalk long-term experiment at Rothamsted Research. Modelling the impact of weather, aggregated within months for each growing season, allows a space-for-time substitution, and hence the prediction of winter wheat crop yields at different locations based on the observed weather in the growing season for each location³¹. As applied nitrogen inputs are not available for the precision crop yields, model predictions are based on optimal applied nitrogen levels³¹. Prediction errors can then be used to identify the need for model adjustments (re-parameterisations) for differences in crop characteristics at different locations, for example soil type, variety, or sowing date. Further models are being developed using data from the Broadbalk experiment to assess the impact of climate and nitrogen inputs on first (following a non-wheat crop) or second (following a first wheat crop) winter wheats rather than the continuous wheat crops considered in Addy et al.³¹.

There is also scope to improve existing crop models by assimilating satellite data into existing process-based models. This is because satellite data can provide measurements of crop parameters that many process-based models simulate via mathematical and statistical means. This concept is by no means new, and dates back to the availability of earth observation (EO)-derived measures of crop growth such as canopy leaf area index (LAI), some 40 years ago^32,33. However, in recent years, increases in the spatial and temporal frequency of EO data capture have made these approaches increasingly viable, to the point where hybrid process-based and EO models can now be conceived for real time crop growth monitoring at the farm scale³⁴. The assimilation of EO-derived metrics into process-based models confers several advantages³⁵. Firstly, and most obviously, it corrects modelled data to observed data, by adjusting the time series of a given crop growth parameter (or multiple parameters) simulated by the model to match that measured by EO. This can be done at various points within a model, so that there are multiple ties back to observed data, increasing model accuracy. This is particularly valuable for accurate forecasting of yields within a given year, such as in systems designed to support real-time farm decision making. Secondly, EO data incorporates the impacts on crop parameters of factors such as local management history, fertiliser input, pests and diseases, unexpected weather events etc, all of which are challenging for process-based models to simulate and parametrise beyond a single site. Assimilation thus adjusts models from providing ‘potential’ yields excluding these factors closer to those likely to be observed in the field. The relationship between pre- and post-assimilation modelled yields can thus be used to adjust, and improve the accuracy, of model outputs even for spatial or temporal extents lacking EO data (e.g. forecasting future crop yields), as detailed by Hayman et al.³⁵. Finally, assimilating EO data can allow models constructed and parametrised at the site level to be run effectively over larger spatial extents, by supplying parameters that would otherwise require assumptions. There are many approaches to EO data assimilation in crop yield models^35–38 and the field is a rapidly developing one. CYCLeSS offers a potential route to explore and validate such approaches.

The finalised dataset contains data from 934 areas of arable land in England, each measuring 1 km² in size, where crop plants were grown between 2015–2017, hereafter referred to as ‘grid squares’. Each grid square is given an individual identification number that links the data on the grid square location, the crop type grown within it each year, the yearly mean crop yield per field, time series of Sentinel-1 radar band data collected at 5-day intervals averaged over the 1 km² area, as well as data on climate and soil conditions for each grid square. This dataset with ground-truth precision yield data is therefore suitable for training and validating a machine learning model of crop yield prediction. We would like to emphasise that the satellite data, climate data, and soil data are all publicly available for any area in the UK. Data is available for 201 grid squares in 2015, 292 grid squares in 2016, and 443 grid squares in 2017 (Table 3). The dataset primarily contains grid squares where winter wheat was grown, followed by grid squares where oilseed rape and spring barley was grown, and a smaller number of grid squares where winter barley and beans were grown (Table 3).Data on yield for crops present within less than 10 grid squares has been omitted from the final dataset to ensure the location of farms on which these rarer crops are grown remains anonymous.Table 3

Number of 1 km grid squares in dataset with available climate, soil, crop yield and Sentinel-1 radar band data by year and crop type.

Data comprising the final CYCleSS dataset is available through Figshare repository (10.6084/m9.figshare.27225807)⁴⁸. The main folder provided (‘CYCLeSS_dataset_141024’) is divided into three subfolders: the ‘crop_yield_type_and_satellite_data’ subfolder contains data on crop yield per field, crop type, and Sentinel-1 radar band data for grid squares, the ‘climate_data’ subfolder contains all climate data extracted and aggregated to 1 km² scale from the CHESS meteorological dataset for each grid square, and the ‘soil_data’ subfolder contains data on soil conditions extracted from the ‘Maps of indicators of soil hydraulic properties for Europe’ dataset, and ‘Mapping topsoil physical properties at European scale using the LUCAS database’ dataset for each grid square. For each grid square data is linked across all subfolders and files by a unique identification number found in the ‘ID’ column of each file.

The subfolder ‘crop_yield_type_and_satellite_data’ contains data on crop type, mean yield per field calculated in tonnes per hectare for each field, and 5-daily time series of Sentinel-1 radar band data for each grid square. The.csv files within this folder contain this data split by radar band (‘VV’, ‘VH’ and the ratio between these values, referred to simply as ‘Ratio’) and year, designated by the following file name format; ‘Band_Year_MeanYieldperField.csv’. For example, the file ‘VV_2015_MeanYieldperField.csv’ contains data on crop type, mean yield per field, and 5-daily ‘VV’ radar band data for the year 2015. Within each.csv file the ‘east’ and ‘north’ columns provide the six-digit easting northing coordinates for the 10 km² British National Grid square containing each field, with the ‘grid_ID’ column providing a unique identification number for each of these grid squares. The ‘ID’ column denotes the individual field identification number, the ‘Year’ column denotes the year data was collected, the ‘Crop’ column contains the crop type grown within each grid square, and the ‘Yield’ column contains the mean yield per field of each crop type for each grid square. Time-series of the relevant Sentinel-1 radar band value denoted by the file name is then included in a series of columns with names in the following format; ‘XYear.Day.Month’ where for instance a column with the name ‘X2015.01.03’ would contain the band value on 1st March 2015.

Data on climate conditions for each grid square is provided in the ‘climate_data’ subfolder, with data further divided into sub-folders by year from 2015 to 2017. Data is then divided into individual.csv files containing time-series of a single variable extracted from the CHESS meteorological dataset for each year with the name format ‘variable_year.csv’. For instance, the file ‘dtr_2015.csv’ contains daily time series data on the daily temperature range for each 1 km² grid square. See Table 4 for the complete list of sub-folder names and the corresponding variable the.csv files within contain data on. For each.csv the unique identification number of each field is given in the ‘ID’ column, and a unique identifier for the 10 km² British National grid square each field was contained within is provided in the ‘grid_ID’ column, along with columns containing the six-digit easting (‘east’) and northing (‘north’) of that grid square. The variable measurement for each day is given in columns with the format ‘XYear.Day.Month’ where for instance a column with the name ‘X2015.01.03’ would contain the observed value for the variable on 1st March 2015.Table 4

Abbreviation for data on each variable extracted from the ‘Climate, Hydrology and Ecology research Support System (CHESS)’ dataset used as file names.

Data on soil conditions for each grid square is provided in the ‘soil_data’ subfolder. This data is divided into two.csvs by year, with ‘LandUseandSoil_2015_2016’ containing data extracted for each grid square for both 2015 and 2016, and ‘LandUseandSoil_2017’ providing data extracted for 2017. Within each.csv file the ‘ID’ column provides a unique identification number for each field, ‘grid_ID’ provides a unique identification number for the 10 km² square area of the British National Grid containing the field, the ‘east’ and ‘north’ columns provide the easting northing coordinates of each grid square. The ‘p_arable’ column provides the percentage of arable land within each grid square derived from the EUCROPMAP 2018 land cover class dataset. The rest of the columns provide data on variables extracted from the ‘Maps of indicators of soil hydraulic properties for Europe’ dataset, and ‘Mapping topsoil physical properties at European scale using the LUCAS database’ dataset. See Table 5 for a key to abbreviations used as column names for each variable⁵⁹.Table 5

Abbreviation for data on each variable extracted from the ‘Maps of indicators of soil hydraulic properties for Europe’ dataset, and ‘Mapping topsoil physical properties at European scale using the LUCAS database’ dataset used as column names.