Plant phenotyping is an emerging science of characterizing and quantifying the physical, physiological, and biochemical traits of crops [1]. It can provide quantitative information for assessing crop performance under specific environmental and management conditions. The most commonly used phenotyping traits include geometric parameters (e.g., plant height, crop canopy area, and leaf area index (LAI)), abiotic/biotic resistance (e.g., canopy temperature, stomatal conductance, and leaf water potential), biophysical/chemical parameters (e.g., biomass, chlorophyll, and photosynthesis), and yield [2]. The breeding process, which involves identifying the genotypes with the most desirable traits (e.g., diseases resistant and high yield), relies on the extraction and description of these traits [3, 4]. Traditional phenotypic analysis methods are labor-intensive as it requires professionals of this field to score plant samples, record plant characteristics (e.g., plant height), and manually harvest selected plants to the laboratory for further testing [5]. Over the last ten years, computer vision and robotics technology augmented with high-throughput plant phenotyping systems (HTTPs) have been developed to relieve the phenotyping bottleneck [6].

The use of multiple remote sensing platforms (satellites, drones, and ground devices) and sensors (RGB, multi- and hyper-spectral, thermal, and LiDAR) has facilitated the collection of large amounts of image data. The emergence of big data coupled with the use of machine/deep learning technique enables a rapid, non-invasive, and detailed acquisition of plant phenotyping traits throughout the whole life cycle of the crop [7]. The identification and localization of crops, fruits, and weeds from images are critical step in the subsequent extraction of plant phenotype information. The typical pipeline of machine learning algorithms is extract features from multiple color spaces then implement threshold segmentation, edge detection, and morphological processing to separate objects from background [8]. This method, although effective, often lacks robustness since it heavily depends on the quality of handcrafted features and hyperparameter tuning, especially when faced with complex environmental conditions such as lighting, weather, multi-target adhesion, and uncontrolled external interventions [9].

In recent years, deep learning, especially Convolution Neural Networks (CNN), has proven to be a powerful tool for detecting and delineating plants from remote sensing images. Mask R-CNN is one of the most popular instance segmentation algorithms [10] and has also been shown effective in many previous agricultural studies. For example, Machefer et al. [11] used Mask R-CNN model to detect and count two low-density crops (potato and lettuce) from the UAV imagery with a spatial resolution of ~2 cm. The mean average precision (mAP) was 0.418 for potato plants and 0.660 for lettuces in the individual plant segmentation tasks. Wang et al. [12] built a Fruits 360 Dataset using the Labelme tool and then proposed an improved Mask R-CNN framework for the detection of various fruits such as apricot, corn, and pineapple. The overall test accuracy reached 99.66%. [13, 14] applied Mask R-CNN to generate mask images of ripe strawberry fruits and presented a visual localization method to get the strawberry picking points from these masks. The average detection precision rate was 95.78% and the picking point error was within 1.2 mm, which is a valuable advancement for fruit picking robot.

Biomass is an important plant attribute that offers insights into the ability of the plant to utilize sunlight, water, and mineral nutrients to grow various tissues and organs [15, 16]. At present, the estimation of biomass through various sensing technologies has become a research hotspot in precision agriculture [17–19]. According to our literature query, there are still few studies using deep learning regression models for crop biomass prediction. Ma et al. [20] predicted the above ground biomass of winter wheat at early growth stages using pre-designed deep convolutional neural network (DCNN) with an accuracy (R²) of 0.80. Castro et al. [21] evaluated two CNN-based architectures (AlexNet and ResNet-18) in the estimation of forage biomass from high-resolution UAV RGB images and found that the prediction accuracy (R²) using the AlexNet framework can reach 0.88. Most previous studies focused on establishing models to determine the underlying relationships between various image-derived features and biomass. Commonly used image-based parameters include plant geometric indicators (canopy area, volume, plant height, shape, etc.) and spectral characteristics (individual band values, spectral derivatives, and vegetation indices), which have been proven quite effective in biomass estimation.

For example, Chen et al. [22] extracted 36 high-quality features from RGB, NIR, and fluorescence images, including geometric descriptors and physiological indicators (spectral band values and fluorescence, and near-infrared (NIR) related). Based on these parameters, the authors developed biomass prediction models for barley (Hordeum vulgare) using four machine learning methods: multivariate linear regression (MLR), multivariate adaptive regression splines (MARS), random forest (RF), and support vector regression (SVR). The results showed that RF performs better than other methods with prediction accuracy (R²) higher than 0.9. Quirós et al. [23] selected 6 image features derived from high-resolution multispectral imagery taken by drones as input variables for field pea (Pisum sativum L.) biomass prediction, including the Green Red Vegetation Index (GRVI), Normalized Difference Red Edge Index (NDRE), NDVI, plot volume, canopy height, and coverage. The Lasso method was implemented for prediction and the average accuracy (R) was approximately 0.84 [24]. Johansen et al. [25] presented a novel approach for tomato (Solanum pimpinellifolium) fresh shoot mass prediction. The author designed an object-based segmentation rule in eCognition software to delineate individual tomato plants and extracted several image-related variables, such as shape descriptors, vegetation indices, and entropy texture. The random forest method was adopted for biomass prediction resulting in 87.95% explained variance.

Strawberry (Fragaria ×ananassa) is widely appreciated all over the world in virtue of its delicious taste, high nutrition, and pleasant flavor. Several studies have been implemented to predict strawberry biomass, showing encouraging results. Guan et al. [26] applied the Structure from Motion (SfM) and Object-Based Image Analysis (OBIA) method to obtain strawberry canopy geometric indicators from high-resolution images (~5 mm), such as planimetric canopy area, canopy average height, and canopy smoothness metric. Multiple linear regression (MLR) was used for strawberry leaf area and dry biomass prediction. It showed that the prediction accuracy (R²) was about 0.79 and 0.84 for biomass and leaf area, respectively. Abd-Elrahman et al. [27] further developed geospatial analysis workflows for automated extraction of strawberry canopy size metric parameters (area, volume, average height, and height standard deviation) from high spatial resolution images and Digital Surface Model (DSM). Although this approach improved the throughput to a certain extent, it still involved sophisticated and relatively time-consuming geospatial analysis procedures and relied on models built using proprietary GIS software (ArcMap) [28]. Moreover, current attempts to estimate plant biomass using traditional machine learning and statistical methods require feature extraction and selection, involving significant manual intervention.

This study aims to establish an automatic workflow for strawberry canopy delineation and biomass prediction using deep convolutional neural networks, applied on ultra-high-resolution visible, near-infrared (VNIR) and Digital Surface Model (DSM) images. Mask R-CNN was used to identify, localize, and delineate each strawberry plant in the imagery. Canopy boundaries produced by the Mask R-CNN analysis and image data (VNIR and DSM) were input to a vanilla deep regression model (referred as to convolutional neural network with a linear regression layer) to predict biomass and leaf area. Two networks (VGG-16 and ResNet-50) were used as the base architecture for deep regression. This work automated the canopy delineation and biomass prediction processes using the powerful capabilities of deep learning, eliminating the need for feature selection and extraction.

All our experiments were implemented under the deep learning framework of Tensorflow 2.1.0 and Keras 2.3.1, with an Nvidia TITAN X (Pascal generation) for GPU acceleration. Strawberry canopy delineation and biomass prediction experiments were carried out separately, and the results are as follows.

Biomass and total leaf area are important but complex biophysical traits that can be used to monitor the health and nutritional status of crops in precision agriculture. These two parameters are also key factors in breeding and genetic research to assess plant growth, yield potential, and plant regeneration ability. The primary aim of this study is to apply deep learning technologies to rapidly and automatically predict dry biomass weight and total leaf area of individual strawberry plants from images, which helps increase the throughput of phenotyping. In fact, several previous studies conducted by Guan et al. [26] and Abd-Elrahman et al. [27] have demonstrated that the geometric traits of plants extracted from high-resolution imagery are effective in the estimation of strawberry biomass and leaf area. However, their approaches contain many complicated procedures which require substantial human effort. We used Mask R-CNN and deep regression models in the task of strawberry canopy delineation and dry biomass/leaf area prediction. This is a good attempt to use deep learning models to directly and automatically predict canopy biophysical parameters of bush crops without extracting any features from the image. The results of this experiment are contributing to the rapid acquisition of strawberry biomass using deep learning methods in the breeding program, which can also be easily transferred to other plants.

In terms of canopy delineation of individual plants, the experimental scenario in this study is relatively simple as the background interference only includes bed, soil, and sometimes grass. The most difficult problems are the multi-canopy adhesion and overlapping/interference as well as the shadow caused by the illumination difference. The Mask R-CNN appears capable of overcoming the above difficulties in multiple growth stages of strawberry plants. Compared with the marker-controlled watershed algorithm introduced by Abd-Elrahman et al. [27], Mask R-CNN does not rely on human interventions, provides more accurate strawberry canopy boundaries, and exhibits strong versatility and robustness even in complex environments.

For deep regression modeling, we choose to import two types of data with different band combinations: DSM-R-NIR and DSM-RGB-NIR-Mask to compare the prediction performance. Each experiment was trained three times and an average result was taken. Meanwhile, the multilinear regression model was also performed as benchmark using ten-fold cross-validation. The result is summarized as Table 6. With respect to the network variants, we conclude that the ResNet-50 was generally more stable and worked better than VGG-16 as the behavior of VGG-16 was more variable in the training process. Of note, the ResNet-50 has a faster convergence speed than VGG-16 in a short training time. This result may be due to the design of identity shortcuts in ResNet, which augmented the network depth but reduced the number of parameters. Regarding data import, six-band images seemed to achieve a little higher prediction accuracy than three-band images. This result also demonstrates that the DSM-NIR-R images were sufficient to provide the three-dimensional spatial information of individual strawberry plants to the network. This result indicates that the limited geometric information in the image itself may be the ultimate limit to prediction accuracy. Unexplained error could be caused by differences in the canopy leaf density among genotypes that are not captured in the images. Overall, the deep regression models showed similar predictive power to the MLR method. Although the deep learning model does not improve the prediction accuracy and is complex, this method avoids the process of feature extraction and also has great potential for fully automated strawberry biophysical parameter estimation.

Remote sensing data collection and deep learning-based big data analysis methods have broad application prospects in precision agriculture and plant phenotyping. Recent advances in deep learning analysis in the field of image recognition provide more advanced frameworks for crop instance segmentation, such as the YOLO series [39] and the transformer-based methods [40]. In addition to the RGB images, high spectral dimension data have the potential to improve predictive model performance in the future. More CNN backbone networks such as Xception (Chollet et al., 2017), DenseNet (Iandola et al., 2014), and EfficientNet (Tan et el., 2019) can be evaluated and applied to the estimation of plant biophysical parameters from remote sensing images, especially for UAV images with coarse spatial resolution but more spectral information.

Phenotyping of field crops based on high-resolution imagery has become a hot research topic in recent years. Most of the current research has focused on extracting representative image-derived features and then importing these parameters to various regression or machine learning models to predict parameters such as plant leaf area and biomass. This manuscript presents a workflow that includes state-of-the-art deep learning methods for strawberry canopy delineation and leaf area/biomass prediction based on ground-based high-resolution imagery. Mask R-CNN was used for individual strawberry plant extraction, and a deep regression approach with two architectures (VGG-16 and ResNet-50) was adopted to directly estimate leaf area and biomass. For dry biomass prediction, R² values were approximately 0.76 and 0.79, and the RMSE values were 8.73 and 8.31 g using VGG-16 and ResNet-50 networks, respectively. For leaf area, the R² values were 0.82 and 0.84, and the RMSE values were both 0.05 m² using VGG-16 and ResNet-50 networks, respectively. These results demonstrate the feasibility of deep regression models for predicting plant biophysical parameters without any intermediate feature (e.g., canopy height and area) extraction. Future research could apply deep learning and deep regression to the phenotyping of other traits and plant species as well as to lower-resolution images from drones.