A myriad of components contribute to yield, as plant performance is regulated by a combination of genetic factors (G), environmental factors (E), and the interaction between them (G × E; Juliana et al., 2018; Montesinos-López et al., 2018). Because of the high complexity that underlies plant performance, breeders have to submit potential varieties to extensive field testing to determine their potential yield (Hunt et al., 2020). Field HTP can substantially accelerate the breeding process by allowing breeders to predict end-of-season traits, such as yield and biomass at early growth stages. Early yield prediction allows researchers to bypass plant growth time, a key limiting factor in crop breeding. In a soybean (Glycine max) study, 2,551 genotypes were grown in different locations, and it was observed that yield, plant maturity, and seed size can be predicted at an early stage using Cubist regression because it presented the best result in comparison to Partial Least Squares Regression, Random Forests, Artificial Neural Networks, and Support Vector Regression (Yuan et al., 2019). Similar results were observed for wheat (Triticum aestivum), barley (Hordeum vulgare), and other soybean genotypes (Bai et al., 2016; Nevavuori et al., 2019). Although promising, the results are constrained to the conditions evaluated, since interannual weather variation, changes in agroecological zones, differences in farm management practices, sensor use and specifications, and other factors can cause instability in model accuracy.

The broad diversity of remote sensors enables capturing different aspects of the plant phenotype. Different combinations of RGB, multispectral, and thermal image data associated with weather and soil have been employed to train deep learning models for crop yield forecasting (Vega et al., 2015; Gracia-Romero et al., 2019; Zhang et al., 2019a; Maimaitijiang et al., 2020; da Silva et al., 2020). The models can support differentiating crop performance in relation to irrigation regimes (Gracia-Romero et al., 2019), quantify growth rate under nitrogen treatment (Holman et al., 2016; Arroyo et al., 2017; Aranguren et al., 2020), estimate variation of wheat grain protein content (Rodrigues et al., 2018; Sharabiani et al., 2019), and monitor crop height variation during the season (Ziliani et al., 2018). A systematic review on machine learning models for crop yield prediction was published by van Klompenburg et al. (2020), showing that deep learning models are increasing in popularity. The most used architectures were Convolutional Neural Networks (CNNs) and Long-Short Term Memory (LSTM). These architectures were created for different purposes: LSTM is designed specifically for sequence prediction tasks, while CNN’s structure is suited to extract features from complex image data. These architectures can benefit from transferred learning, in which pretrained weights obtained with different data can be implemented in the new model (He et al., 2016) that allows rapid and high performance. Multimodal machine learning can be employed to analyze datasets with multiple data sources (rainfall, temperature, multispectral image, soil data), each data type is a modality that will be analyzed and combined to increase model performance (Baltrušaitis et al., 2017). van Klompenburg et al. (2020) observe in the review that temperature, rainfall, and soil type were the most used data types in machine learning models, but different feature combinations and the volume of data can directly impact the model performance and should be tested during development.

A few HTP datasets were recently released with the goal to improve yield prediction and more specifically support genotype to phenotype prediction. The Genomes to Field (G2F) datasets comprise genotype (single nucleotide polymorphism information), manual phenotype measurements, climatic data, soil information, inbred ear images, and UAV collected multispectral and hyperspectral images of several maize (Zea mays) varieties grown over multiple years (Supplemental Data Set 1; McFarland et al., 2020). Detailed metadata are essential for understanding genotype to phenotype relationships in each season/environment. However, the G2F field trials were carried out in a single location, which limit the robustness of the traits identified. For sorghum (Sorghum bicolor) and wheat, the Transportation Energy Resources from Renewable Agriculture Phenotyping Reference Platform (TERRA-REF) database offers a comprehensive resource of sensor data (five thermal, spectral, and shape imaging sensors), phenotypic measurements, environmental and genomic data, including genome sequencing of 384 varieties and genotyping by sequencing of 768 varieties (Supplemental Data Set 1; LeBauer et al., 2017; Burnette et al., 2018). TERRA-REF has four sensing platforms to collect image-based phenotyping and agronomic traits from both controlled environment and field grown plants. TERRA-REF maintains a manuscript management section in their website where researchers willing to use the data can register their proposed manuscript to prevent overlap and encourage collaboration. Oftentimes, researchers will delay publishing datasets until their planned publications are completed. Nonetheless, the TERRA-REF approach to register publications enables early publishing of the data and allows other groups to explore different aspects of the dataset or collaborate. Federated learning is another strategy that can be used when the data must be protected due to privacy or security concerns, showing increasing use in medical research (Lee et al., 2018; Huang et al., 2019; Rieke et al., 2020). Federated learning allows for training machine learning models collaboratively without exchanging the data, in this framework, each dataset owner institution downloads the model and trains it locally. The trained parameters from each institution are exported and aggregated, creating a model that benefits from previously inaccessible datasets while the data governance and accessibility remain in the control of the data owner (Konečný et al., 2016).

Both G2F and TERRA-REF datasets present limitations regarding the types of environments represented, species grown, and the treatments that they were subjected to. Other similar phenotyping initiatives covering different locations and plant species (including noncrop plants) are needed to depict phenotypic variation. Nonetheless, the above datasets offer an extensive resource that can assist the identification of quantitative trait loci (QTLs) related to crop performance, develop tools for genotype to phenotype prediction based on the multidimensional dataset, and ultimately these could be used as benchmark datasets to assess tool performance. Moreover, smaller datasets for field trial experiments can be found at the Global Agricultural Research Data Innovation Acceleration Network and the International Maize and Wheat Improvement Center described in Supplemental Data Set 1.

Grain yield in wheat is directly related to spike head population density, size, and maturity stage. The Annotated Crop Image Dataset (ACID) provides images with coordinates to identify wheat spikes under greenhouse conditions (Pound et al., 2017b). ACID was designed for training novel tools for identifying the spike heads, and measuring individual head traits, but the tool could be further applied to new datasets and to link measured traits with genotypic variability. Limited metadata annotation in ACID prevents further exploration of the dataset itself for identification of yield-related traits because the genotypes and experimental conditions are not described. The global wheat head database compiles multiple RGB wheat images collected in the field, from several countries using different cameras (David et al., 2020). The dataset was used in a challenge hosted on Kaggle (https://www.kaggle.com) with the goal to benchmark wheat head detection approaches. Top solutions used object detection deep learning architectures (EfficientDet, Faster-RCNN, and Yolo-v3), with data augmentation techniques playing a major role for their success. Data augmentation is a computer vision technique to increase dataset size through a series of transformations, such as flipping or rotating the image (Buslaev et al., 2020). It is important to note that with field images, a greater variability of conditions can occur such as genotype differences, head orientation, and mixed developmental stages, which can cause the object detection model to present performance instability such as mislabeling plant organs at a higher rate when the conditions differ from what was seen in the training data. The global wheat head dataset provides a valuable resource for developing and benchmarking tools due to the high variability of wheat genotypes and conditions represented. Similar datasets for different species can be developed collaboratively by annotating previously released HTP data (such as the G2F and TERRA-REF datasets). This can decrease the cost of producing a dataset and benefit from the described metadata.

The development of climate resilient crops must consider the effect of combined abiotic stresses occurring in the region (Cammarano et al., 2019). As a result, datasets featuring combined abiotic stresses provide a resource to understand how their interaction impacts plant health and development. The Eschikon dataset (Supplemental Data Set 2) includes temporal images of beet (Beta vulgaris) under multiple independent and combined drought, nitrogen deficiency, and weed stresses (Khanna et al., 2019). This dataset was employed to develop 3D representations of the plants from which the authors were able to extract canopy cover, height, and vegetation indices. These traits were used to classify stress in plants with 83%–93% accuracy depending on the stress measured. The dataset can be further explored to measure agronomic traits related to each stress and understand plant response, it can also be employed in further developing computer vision tools for stress classification (Khanna et al., 2019). Plant researchers willing to predict the effects of climatic change in crop species will require the creation (or release) of more datasets in which the combined stresses are observed. These datasets must offer a detailed description of the environmental conditions and if possible, of the genetic data to enable accurate interpretation of the results. Ideally, the aggregated datasets must depict the diversity of agroecological zones including low latitude locations, which are currently underrepresented.

Crop water management is essential in regions currently facing or predicted to face water scarcity. Infrared thermography has been successfully implemented to assess water use by crops (Nhamo et al., 2020), and for measuring genotype performance under salinity or water deficit stress (Raza et al., 2014; Kumar et al., 2017; Thapa et al., 2018; Hou et al., 2019; Zhang et al., 2019b; Kumar et al., 2020; Masina et al., 2020). In cotton (Gossypium arboreum) monitored by infrared thermography, it was observed that yield, fiber length, and micronaire suffered reduction after canopy temperature exceeded a given threshold (Conaty et al., 2015). Canopy temperature and evapotranspiration (ET) maps are used as a proxy for measuring the phenotypic response to both stresses as they influence stomatal conductance (Fischer et al., 1998; Sirault et al., 2009), and are observed from close range, at the aerial and spatial level. Remotely sensed thermal data collected by satellite platforms allow mapping water resource use through the prediction of ET maps (Anderson et al., 2012). In 2018, a space station mission (ECOSTRESS) was launched to measure ET and identify plant stress (Fisher et al., 2020). It provides a higher spatial and temporal resolution ratio (60 m with 1–5 d interval) in comparison to Landsat (>60 m, 16-d interval) or MODIS (>375 m, daily; Anderson et al., 2012). The ECOSTRESS library provides satellite imagery associated with laboratory measurements of vegetation to help correlate the spectral patterns (Meerdink et al., 2019; Fisher et al., 2020). This dataset has been employed to assess plant species diversity in restoration areas, showing that sites with higher species diversity present lower temperatures (Hamberg et al., 2020). For this review, we chose to focus on HTP images collected by aerial or ground devices since satellite images currently do not yet provide enough resolution to be used for assessing plants at the field level. However, satellite HTP imagery offers the potential to help understand abiotic stress at a large-scale (Anderson et al., 2012; Miralles et al., 2014), thus we have included links to satellite libraries (ECOSTRESS, Landsat and MODIS) in the resources in Supplemental Data Set 2.

Besides infrared thermography, multispectral and hyperspectral sensors are also used in HTP. These sensors are capable of detecting physiological changes in the plant leaf composition (Bruning et al., 2020). For example, decomposition of foliar hyperspectral signatures showed that C3 and C4 plants have divergent and well-defined patterns of reflectance (Baranoski et al., 2016). Hyperspectral images were employed to quantitatively rank salt tolerance between four wheat varieties (dataset available in Supplemental Data Set 2) using machine learning and dimensionality reduction. The authors observed that multiple trait measurements would be required to correctly assess the plants, whereas with hyperspectral images they were able to score them in a fast noninvasive way, dataset is described in Supplemental Data Set 2 (Moghimi et al., 2018). Multispectral and hyperspectral images have also been employed to identify salt stress in sugarcane (Saccharum officinarum L.) and wheat irrigated with saline water (Hamzeh et al., 2013; El-Hendawy et al., 2019), acidic and heavy metal stress (Liu et al., 2011; Jin et al., 2013; Li et al., 2015; Zhang et al., 2017; Wang et al., 2018a), nutrient deficiency (Pacumbaba and Beyl, 2011), heat stress (Gautam et al., 2015; Trachsel et al., 2019), and frost (Fitzgerald et al., 2019; Nuttall et al., 2019; Murphy et al., 2020).

Frost damage in wheat can have a major impact, as a single frost event can severely reduce quality and yield (Boer et al., 1993; Frederiks et al., 2012; Martino and Abbate, 2019). Rapid detection of frost damage would enable growers to take management decisions to avoid losses. A study using hyperspectral images indicated that under controlled conditions, frosted and nonfrosted individuals, present significant spectral differences (Murphy et al., 2020). Mixed results were observed when detecting frost under field conditions, indicating that more research is needed (Fitzgerald et al., 2019; Nuttall et al., 2019). The Frost nursery dataset provides multispectral images of several commercial wheat varieties grown in the field and were affected by frost at different developmental stages (AgReFed, 2019). This dataset includes final yield, leaf protein, and abundance of metabolites, which can be used to characterize the effect of frost (Supplemental Data Set 2). The association of hyperspectral data with physiological measurements may assist frost damage quantification, which can support crop breeders screening for more tolerant varieties. Hyperspectral imagery has the potential to capture traits related to the biochemical composition of target tissues. However, due to various technical factors, the recorded data are usually noisy with nonnegligible redundancy (Mishra et al., 2019). Datasets including hyperspectral data may benefit from detailed description of the experimental conditions and sensors used, helping guide researchers how to better extract the information.

Platforms such as Quantitative Plant (Lobet et al., 2013), Phenopsis (Granier et al., 2006), and BrAPI (Selby et al., 2019) are dedicated to assemble a wide range of phenotyping datasets that can be used to compare phenotypic response to stress within and between species. These platforms are focused on making phenotypic datasets more findable. Information and website links for other abiotic stress datasets are described in Supplemental Data Set 2.

Changes in environmental conditions are likely to shift pathogen and pest regional distributions (Hovmøller et al., 2008; Shaw and Osborne, 2011; Bebber et al., 2013; Garrett, 2013; Mariette et al., 2016; Skelsey et al., 2016). To provide suitable crop varieties and agricultural management recommendations for these new conditions, it is necessary to gain greater understanding of the ecological, phenotypic, and molecular basis of the interaction between plant and pathogens (Skelsey et al., 2016). Pathogen identification and disease severity estimation are an important part of characterizing their distribution in the field (Ali and Hodson, 2017). The detection and quantification of disease are usually performed by visually assessing crop symptoms, which may be subjected to bias and human error, besides being labor and time intensive. Various datasets have been released to assist the development of automated systems for disease identification and assessment (Supplemental Data Set 3). The Plant Village, BRACOL, RoCoLe, citrus (Citrus sp.), cassava (Manihot esculenta), and apple (Malus sp.) datasets offer close range annotated images of infected plant organs against a clean background, offering a resource for disease diagnosis and severity scoring in collected leaves (Mohanty, 2016; Arsenovic et al., 2019; Chouhan et al., 2019; Krohling, 2019; Parraga-Alava et al., 2019; Rauf et al., 2019; Tian et al., 2019; Nakatumba-Nabende et al., 2020; Singh et al., 2020). Machine learning models using support vector machines, CNNs, and self-attention CNNs trained on similar datasets were published recently (Abdu et al., 2020; El Abidine et al., 2020; Zeng and Li, 2020), some of which report increased efficiency when using segmented regions for pathogen identification (Esgario et al., 2020; Karlekar and Seal, 2020). A comprehensive review on machine learning for disease assessment in crops was published by Hasan et al. (2020).

Although disease detection models trained with the above datasets can be used in the field, the input samples have to be manually collected and imaged which can be time consuming. Hence, many researchers have focused on developing models that use UAV-collected images to accelerate disease detection (Vergara-Diaz et al., 2015; Sugiura et al., 2016; Moriya et al., 2019; Qiu et al., 2019; Tetila et al., 2020; Zhao et al., 2020). (Marzougui et al., 2019) combined HTP images from greenhouse and field experiments to quantify Aphanomyces root rot resistance in lentils (Lens culinaris). The authors developed 12 normalized spectral indices that correlate with disease symptoms and severity, allowing breeders to objectively quantify genotype resistance. Another study used hyperspectral data and machine learning for early identification of charcoal rot disease in soybean, obtaining classification accuracy of 90% for plants 3 d after infection (Nagasubramanian et al., 2018). These studies demonstrate the potential of image-based HTP to enable growers and breeders to automatically screen plants. To the best of our knowledge, there are no available datasets for disease-related tasks in the field which prevents the development and benchmarking of computer vision-based tools. Benchmark datasets created for this task requirements are shown in Box 1, with specific image annotations depending on the target task (disease detection, identification, severity scoring, and lesion segmentation).

Field HTP is widely applied to the detection and quantification of pests. Rapid pest identification is important so growers can take action to control pest spread and limit damage to crops. A large benchmark dataset for insect pest detection was released containing 75,000 close range images of annotated pests belonging to 102 categories (see Supplemental Data Set 3 for a detailed description; Wu et al., 2019). This benchmark dataset is a valuable resource for the development of crop monitoring and management approaches, allowing researchers to test model performance over a wide range of pests. This dataset can also be complemented with the mango (Mangifera indica) pest classification dataset, which has images of mango plants infected with 15 different categories of pests, with a large volume of augmented images to increase model robustness (Kusrini et al., 2020a). Precise algorithms for the detection of pests can support assessing crop resistance by counting the pests, helping identify pest species in the field, and monitoring pest spread. Employing HTP datasets to measure plant–insect interactions can allow the use of RGB sensors to quantify leaf damage and defoliation (O’Neal et al., 2002). Thermal infrared and hyperspectral images can also be used to capture physiological changes, such as stomata regulation (Backoulou et al., 2011; Nabity et al., 2013). Novel datasets targeting the plant–insect interactions should follow the guidelines proposed in Box 1 with special attention to providing detailed metadata (view MIAPPE project) and ground-truth measurements and labeling.

The development of navigation maps is particularly important for weed management systems, in which the map can be used for targeted herbicide application or by weed killing robots (Somerville et al., 2019; Gašparović et al., 2020; Hunter et al., 2020; Raja et al., 2020). Weed detection systems can reduce herbicide application by up to 60% in comparison to broadcast applications (Somerville et al., 2019; Hunter et al., 2020) and increase efficiency in organic production systems. A key challenge for implementing weed detection in the field using image-based HTP data is the difficulty in establishing robust computer vision-based models that can distinguish between crop and weed species under varying field conditions. To help overcome this challenge, many datasets have been released consisting of RGB and multispectral images of a wide variety of weed and crop species, some of which contain pixel level annotations to separate the weed from background (Supplemental Data Set 3; Haug and Ostermann, 2015; Dos Santos Ferreira, 2017; Giselsson et al., 2017; Sa et al., 2018; Teimouri et al., 2018; Skovsen et al., 2019; Sudars et al., 2020). A few datasets feature images of weed seedlings, enabling the development of models that can detect weed infestation at an early stage. Studies using similar datasets employed computer vision and machine learning algorithms for weed detection, though these presented a high variability in the precision rate (69%–98%) depending on the crop field analyzed (Wang et al., 2007; dos Santos Ferreira et al., 2017; Pallottino et al., 2018; Umamaheswari et al., 2018; Bah et al., 2019; Partel et al., 2019). These results emphasize the need to produce more datasets with an increased variety of crop and weed species at different growth stages. Furthermore, the datasets need to reflect the management practices (e.g. sowing density) that the weed detection model would encounter in the field. Increasing model robustness to varied field conditions is essential to enable its adoption in agricultural management systems and allow plant researchers to quantify herbicide or other weed control practices efficiency.

Root system architecture (RSA) greatly influences nutrient access, efficient water uptake, and plant tolerance to stress (Mary et al., 2018; York et al., 2018; Mattupalli et al., 2019; Busener et al., 2020; Griffiths et al., 2020; McKay Fletcher et al., 2020; Seo et al., 2020). Increased efforts in breeding for desirable RSA traits can drive a breakthrough in crop productivity and resource efficiency (Lynch, 2007). To leverage RSA potential in crop breeding, it is important that we improve current root phenotyping strategies.

Noninvasive RSA imaging is extremely challenging due to soil opacity. At the same time, soil replacements such as transparent gels or hydroponic mediums often lead to phenotypes that diverge substantially from the ones observed in regular soil (Hargreaves et al., 2009; Wojciechowski et al., 2009; Clark et al., 2011; Ma et al., 2019). A wide variety of sensors can be employed to acquire 2D or 3D images of plant root grown in the glasshouse, such as X-ray computed tomography, magnetic resonance imaging, positron emission tomography, and hyperspectral imaging (Jahnke et al., 2009; Garbout et al., 2012; Mooney et al., 2012; van Dusschoten et al., 2016; Bodner et al., 2018). In Supplemental Data Set 4, we list available RSA datasets with metadata at varied levels of detail including from plants grown in multiple types of media, such as gellan gum, soil, and hydroponics. In addition, a synthetic root system dataset is available. This large dataset was produced for tool calibration and modeling since it provides ground-truth of fibrous and tap-root images which help identify artifacts generated by the model when dealing with complex, overlapping root structures. The data were produced using ArchiSimple with three levels of noise, and the roots present varying degrees of complexity (Lobet et al., 2017).

Field root phenotyping frequently requires the manual excavation of individual plants followed by imaging of the washed root crown system for quantitative trait analysis (Trachsel et al., 2011; Bucksch et al., 2014; Colombi et al., 2015). Root crown datasets of multiple crop species are described in Supplemental Data Set 4, some of which were produced with the aim to automatically quantify RSA traits using different tools. Noninvasive alternative approaches are not as commonly employed, but offer the potential to undertake a time-series analysis of crop development. These include electrical resistance tomography, electromagnetic inductance, and ground penetrating radar (Diaz and Herrero, 1992; Zenone et al., 2008; Srayeddin and Doussan, 2009), which are used to characterize root water uptake of wheat and vine plants in the field (Shanahan et al., 2015; Whalley et al., 2017; Mary et al., 2018).

Overall, image-based RSA phenotyping has many applications, such as linking RSA traits to micronutrient concentration and heritability (Busener et al., 2020; McKay Fletcher et al., 2020), the effect of dwarf genes in seedling roots (Wojciechowski et al., 2009), changes in the root crown in response to disease (Corona-Lopez et al., 2019; Mattupalli et al., 2019), to investigate root plasticity (Rosas et al., 2013), genetically driven root architecture differences (Jiang et al., 2019), and QTL mapping of regions controlling RSA (Topp et al., 2013). Most of the studies cited above use a combination of tools for RSA trait extraction (DIRT; Das et al., 2015), RhizoVision (Seethepalli and York, 2019), RSA-GiA (Galkovskyi et al., 2012; Topp et al., 2013), or Rootscape (Ristova et al., 2013)) followed by statistical analysis (variations of ANOVA, three-parameter logistic function, PCA) or linear regression to test if the observed traits relate to environmental or genetic data. The wide range of approaches used reflects the diversity of input data formats. The sensors employed to collect RSA traits are very diverse and capture different aspects of the root. Hence, the decision for which feature extraction tool and analysis method to implement must be decided case by case. Even more important in this case is tool and data interoperability because it will allow researchers to explore the resources efficiently. Root image datasets from several major crop species can be downloaded from the Quantitative Plant platform (quantitative-plant.org/dataset) and Zenodo database(zenodo.org/).

The reconstruction of the data as 2D or 3D representations of the root system, and root segmentation from the medium usually assumes a high contrast between root and background, which is not always the case (Atkinson et al., 2019). Machine and deep learning-based tools have been developed for root segmentation in 2D or 3D (Iyer-Pascuzzi et al., 2010; Bucksch et al., 2014; Falk et al., 2020; Yasrab et al., 2020a), including very thin (1–3 pixels) roots grown in visible medium (RootNet; Yasrab et al., 2020b) and in soil (Soltaninejad et al., 2020), while other tools aimed for RSA trait quantification (Atkinson et al., 2017a; Falk et al., 2020). Although there are many potential approaches to perform root segmentation, most are not suited for newer image data types. In addition, few tools are capable of linking observed RSA to genotypic information. Recently, deep learning models have been employed to attempt to bridge phenotype to genotype predictions (Pound et al., 2017a; Yasrab et al., 2020a) and can achieve similar results for QTL identification as user supervised methods (Pound et al., 2017a). However, to effectively integrate high-throughput phenotype to genotype tools into the breeding process requires refined tools. These tools must be capable of dealing with phenotype and sensor variability and of aggregating experimental metadata into the analysis. The success in the development of such tools relies on the quality and size of the available datasets because these are the sole source of information for the deep learning model to adjust its internal parameters.

The description of plant morphological traits, for example, number of leaves, canopy cover, number of flowers, and seeds, provides a foundation to characterize plant phenotypic response, which is directly related to plant developmental stage, yield potential, and overall health (Kouressy et al., 2008). The quantification of agronomic traits often relies on manual measurements, which are costly, labor-intensive, and prone to errors. Several approaches including neural networks and other machine learning models have been published to perform leaf counting, area estimation, folding and plant growth stage classification, stem–leaf segmentation, and seed counting (Parmar et al., 2016; Pereira et al., 2016; Sodhi et al., 2017; Teimouri et al., 2018; Uzal et al., 2018; Jin et al., 2019; Rascio et al., 2020). Deep learning models are widely applied to image analysis due to the high complexity of the data and their potential for quantitative morphology lies partially in their capacity to segment the target object from the nontarget objects in the image. Hence, it is possible to measure the traits of the segmented object (number of seeds, color, fruit shape, fruit, or seed size). This measurement ability was shown in a study for fish morphology quantification that used Mask R-CNN for pixel-wise segmentation of the fish body followed by measurement of its morphological features (Yu et al., 2020). A variety of trait phenotyping datasets have been released to develop pipelines for trait measurement, such as the hypocotyl dataset with images of A. thaliana seedlings for length determination (Dobos et al., 2019), image time-series of A. thaliana growth that can be used to predict performance (Taghavi Namin et al., 2018), and species identification datasets (Kumar et al., 2012; Lee et al., 2015, 2017; Fricker et al., 2019; Zheng et al., 2019) as shown in Supplemental Data Set 5.

PlantCV and Deep Plant Phenomics are the two platforms that offer packaged pretrained deep learning models to run as applications for phenotyping (Fahlgren et al., 2015; Ubbens and Stavness, 2017). However, tools for quantitative morphology analysis can only guarantee performance if under restricted image conditions and may require further image processing steps. Producing and sharing annotated datasets from a diverse set of species are the most efficient way to ensure new tools can be developed to incorporate them. The Plant Phenotyping Datasets (Supplemental Data Set 5) are the collection of annotated top-view images of A. thaliana and tobacco (Nicotiana tabacum) undergoing different treatments (Minervini et al., 2016). It is a benchmark dataset (Box 1), that was employed in the leaf segmentation and leaf counting challenges at the Computer Vision Problems in Plant Phenotyping conference, and propelled the development of tools for leaf segmentation and counting (Aich and Stavness, 2017; Dobrescu et al., 2017; Giuffrida et al., 2018; Praveen Kumar and Domnic, 2020), which can be later used for assessing plant growth and biomass. Other datasets focused on seed and fruit organs are available. Some datasets are useful to compare variance in seed morphological traits (Ducournau et al., 2020), while others can be used for the development of computer vision tools for fruit counting and automatic quality assessment. In this category, there is a soybean image dataset to assess seed damage from mechanical and biological sources (Pereira et al., 2019), a dataset for the identification of Indian basmati rice (Oryza sativa) seed varieties (Sharma et al., 2020), sugar beet (Beta vulgaris) seed traits (Ducournau et al., 2020), a cocoa bean (Theobroma cacao) dataset for quality assessment (Santos et al., 2019), a banana (Musa sp.) tier abnormality classification (Piedad, 2019), and hyperspectral images of different loose tea (Camellia sinensis; Mishra, 2018; Supplemental Data Set 5).

Determining leaf inclination and distribution on the plant is an important morphological trait, it impacts the plant spectral reflectance and is a mechanism to increase tolerance to abiotic stress, with impacts on leaf temperature, water loss, and drought tolerance (Ehleringer and Comstock, 1987; Fuchs, 1990; He et al., 1996; Werner et al., 1999). In common bean (Phaseolus vulgaris L.) the extent of leaf movement increases as the water availability drops, allowing the plants to maintain leaf temperature despite stomata closure (Pastenes et al., 2005). A dataset for leaf angle estimation with ground-truth angles for 71 Eucalyptus species (Pisek and Adamson, 2020) is described in Supplemental Data Set 5, it contains images of Eucalyptus canopies that can be used to estimate leaf angle distribution in trees. Automated pipelines for leaf angle extraction have been developed and tested for A. thaliana, beet, apple (Malus domestica), maize, and sorghum (Müller-Linow et al., 2015; Kenchanmane Raju et al., 2020), allowing researchers to track leaf angle variability and distribution over time. Identifying varieties with desired leaf angle distribution can assist breeders to select the varieties best adapted to specific environmental conditions, such as high planting densities where a narrow angle prevents the leaf from being shadowed by others (Pepper et al., 1977; Lambert and Johnson, 1978).

A multitask pipeline capable of phenotyping a comprehensive array of traits in different tissues would produce a snapshot that can be used to identify new QTLs. It was observed that genetic traits may contribute to different tissues causing multiple trait variance (Li et al., 2018). This would provide a resource to detect QTLs and improve our understanding of the genetic basis of complex phenotypes (Topp et al., 2013). Trait phenotyping can also be used for the construction of 3D representations of the plant structure (Topp et al., 2013; Vadez et al., 2015; van Dusschoten et al., 2016; McCormick et al., 2016; Bengochea-Guevara et al., 2017; Sodhi et al., 2017; Vázquez-Arellano et al., 2018; Wang et al., 2018b). This avoids loss of information caused by 2D compression and prevents the generation of artifacts that can occur due to lighting, occlusion, and overlaps.