Data was collected over two years, across two locations. The first location was the Peanut Belt Research Station (PBRS) near Lewiston, North Carolina, located at 36.1333, − 77.1705. The second location was the Upper Coastal Plain Research Station (UCPRS) near Rocky Mount, North Carolina, situated at 35.8943, − 77.6812. A total of 2120 plots were grown for the purpose of studying leaf spot over the course of the 2021 and 2022 growing seasons. Field trials consisted of 530 plots, with each trial containing 265 different genotypes of peanut to be monitored. Each plot consisted of two rows of a single line. Rows were approximately 0.9 meters in width and 7.6 meters in length. Further information regarding planting, visual scoring, and image capture dates can be found in Table 1 and a complete description of the trials can be found in [20].

To facilitate image acquisition and development of necessary data sets, a protocol was developed to image in the field. To ensure that the canopy was at a consistent distance (3 ft) from the camera, handheld mounts were constructed to serve as references. A schematic of this simple setup is shown in Fig. 2a in which a 36-inch wooden dowel was combined with a small phone mount. A spatial calibration block was fixed to the end of the dowel; This calibration block was captured in all images and ensured that a spatial reference was included in all images, allowing the determination of the real world size of objects in the images [33–35]. Per Fig. 2b, this configuration allowed the user to hold the mount so that the reference block rested just above the canopy while the phone was positioned at the opposite end of the dowel. The pixel sampling distances in each image were converted to physical distance by d=a/b, where d is the calibrated sampling distance, a is the width of the block in pixels, and b is the width of the block in pixels. For this setup, the block had a width of 1 inch, and generally the camera was oriented such that b=282 pixels, so these images had an approximate sampling distance of 282 pixels / inch. These images were 3042 x 4032 pixels in size, the maximum resolution of the Samsung Galaxy S8 phones used. The effective focal length of this camera was 24 mm. Employing a wide angle lens ensures that off-axis regions are included in the captured images, ensuring that more than just the highest leaves are imaged. All images were captured on days with minimal cloud cover in the early afternoon, ensuring that consistent lighting was present in all images.

A minimum of three images per plot were captured. The first and last images were captured at the extreme ends of the length of the plot, with the image arranged to capture the edge of the plot while minimizing non-plant area in frame. Often, more than the minimum number was taken as the imaging personnel walked from one end of the plot to the other. This ensured good coverage was maintained while also allowing each plot to be rapidly imaged. An example of these images can be seen in Fig. 2b. Table 1 contains the dates at which the imaging protocol was performed. On dates image capture took place, a random number generator was used to select a subset of plots for imaging.

A goal of having included many different peanut lines at different times in the growing season was to ensure that the data set included peanuts with different canopy structures, specifically as it relates to leaf density. An example of images that showcase different structures can be found in Fig. 3. These images were used in the development process of this pipeline as part of the Image Scoring Data Set described in Transfer learning and selecting the neural network architecture section.

Three primary data sets were created over the course of this study, each with a different purpose. The first will be referred to as the Neural Network Training Data Set. This data set was made of images captured at both UCPRS and PBRS locations during the 2021 growing season. All of the 2021 imaging dates found in Table 1 were included in this data set. The final result of this data set was 514 image / semantic segmentation labels that were used for developing the neural network used in the pipeline. The second data set is the Image Scoring Data Set. This data consists of 476 images that were collected at the PBRS location on 5 Oct. 2022 and includes data from plots that were manually scored from an infection level 1 to an infection level 9. This data set was used to develop the scoring function and model its success against a manual ground truth scoring by a trained expert. The last major data set is the Temporal Data Set. This data set includes randomly sampled field trials toward the end of the 2022 growing season. Two, subsets of this data set were created by randomly imaging plots from the UCPRS location in accordance with the imaging protocol. The first subset was collected at the UCRPS location on 29 September 2022 and contained 1,434 images. The second data set was collected at the same UCPRS location on 11 October 2022 and contained 714 images. Both subsets involved a different set of randomly sampled plots with 118 plots in total imaged for the Temporal Data Set.

A neural network was transfer learned to perform semantic segmentation, such that each pixel would be classified as either healthy, infected, dead, or non-plant material. Four networks were chosen and trained prior to down-selecting the best-performing model. The U-Net architecture was first chosen given its origin in biomedical image segmentation and powerful encoder-decoder structure [39]. Three additional deep learning networks were also employed: MobileNet, ResNet18, and ResNet50 [36, 37]. These were implemented using DeepLabV3+ in MatLab, allowing the networks to all perform pixel-wise classification on the images [40].

The training data subset consisted of 420 image and semantic segmentation label pairs. The validation and holdout data subsets were 48 images each, representing a split of 81.4% / 9.3% / 9.3% for the training / validation / holdout split from the original Neural Network Development data set. Table 2 describes the raw pixel counts of each of the individual classes in the training data set as well as the normalized frequency for the classes.

All four networks were trained using the following settings:

The main metric for success with respect to the imaging protocol was the number of images captured and plots imaged compared to the speed of a human scorer. To that end, imaging per plot takes an individual approximately 1 minute. For a field of 530 plots, this would take almost 9 hours for an individual to properly image for scoring protocols. By comparison, an individual scoring in compliance with the traditional 9-point can score the same number of field trials in approximately 1–1.5 hours. To this end, the imaging protocol was much less time efficient than traditional field surveys, with imaging taking almost five times longer compared to a traditional field survey.

We also see a reduction in total plant canopy area used for scoring through the imaging protocol as described. Given the typical ground sampling distance of 94 pixels / inch, we are left with an approximate 0.90 m2 area per image taken. Compared to a human scorer who can see the entirety of the 6.8 m2 plot at a glance, this is a massive reduction in sampled area.

As previously described, a holdout data set of 48 image / segmentation label pairs was used to evaluate how well the models generalized to new information. The performance on this data set would demonstrate how the model performs on future new data sets. The hold out data set was used to generate two primary metrics of evaluation. The first metric was the Jaccard Index, which evaluates the similarities in the semantic segmentation labels versus the generated labels by comparing the intersection areas of each class to the overlapping areas of each class. The ResNet50 model scored the highest, with a Jaccard index of 0.770. A full list of Jaccard indices for each network, alongside global accuracy, can be seen in Table 6. The time for each model to perform inference was determined by running each model on one thousand 224×224 pixel images, with the mean time to perform interference recorded. The GPU utilization was logged during this using the program GPU-Z version 0.8.9, with the mean GPU Utilization during this time period recorded for each model. Each of these tasks were performed using the aforementioned system with Ryzen 5600X CPU and RTX 3060 graphics card.

The normalized Confusion Matrices for all four of the preliminary networks can be seen in Fig. 6, which shows how well the architectures could identify each pixel belonging to each class. While each model demonstrated a mean accuracy of between 0.795 and 0.853, the minimum performance of each model across the four classes was the metric used to decide which model would be used in the finished pipeline. Despite a larger model and longer inference time compared to the three other models that were tested, ResNet50 was the only network to achieve an accuracy of over 50% on the dead leaf material during testing (See Figure 5d). ResNet50 was selected for the application due to its highest minimum accuracy across classes. While mean accuracy and even Jaccard index indicates similar performance of each network, we determined that ResNet50 provides a good balance of all classes, spanning early leaf spot (visual scores 1–5) and defoliation (visual scores 6–9) based on the confusion matrices.

In addition to simple numeric testing of the network’s accuracy on test sets, it was important to perform visual inspections of how well the model generalized to new data. An example of this qualitative testing can be seen in Fig. 5, which shows an image that was passed into the preliminary ResNet50 architecture and its resulting label. As depicted, semantic labels were applied correctly, demonstrating proper semantic segmentation of the image—this is supported by the confusion matrix. Within the image, green represents Healthy plant area, red sections are infected lesions, yellow representing Dead plants, and black represents Non-Plant or non-distinguishable area.

All of the associated networks exhibited the lowest correct classification on the “Dead” class. ResNet50 saw the best result here, with more than half of the Dead pixels being properly classified. The other networks all saw under 50% proper classification of the Dead classed pixels, reinforcing our choice to keep the ResNet50 network in the pipeline. This low classification rate was noticeably exacerbated in images where there was no green plant area - this behavior was seen routinely.

To examine the relationship between our two primary classes and visual severity level, a simple data set was created. This data set, referred to as the Image Scoring Data Set, was comprised of 475 images captured at the end of the growing season on 5 Oct. 2022 at PBRS per Table 1. Each of the full-size 12MP images in the Image Scoring Data Set was segmented by our network and the labels were converted into their relative class frequency. Each visual severity level represents a small range of possible severities—for example, a score of 4 is described as having “many spots; mostly on lower and middle leaves; disease evident” [11]. This description represents a range of severities before increasing to the next score, effectively introducing quantization or round-off error to the measurements [48]. To reduce error caused by this range, all samples of a specific visual severity were averaged to a single value, representing the typical infection for each visual severity score. Figure 7 shows a plot of the average normalized frequency of each of these class counts as compared to the visual severity level. The r2 for this fitting can be found in Table 7 alongside the root-mean-square-error (RMSE) when the fit function is applied to all images in the Image Scoring Data Set. A fit was performed on the classification frequency to the linear function f=p00+p01sv where f is the normalized frequency, sv is the ground truth visual severity level, p00 is an initial offset, p01 is the multiplication coefficient for the visual severity level. The results of fitting to a linear trend as described here can be seen in Table 7. A continuous decrease in the Healthy plant frequency is observed as the visual severity level increases. We also see an increase in Infected plant material and Dead plant material as the severity increases. A correlation coefficient matrix can be seen in Figure 8, showing the correlation coefficients of each pixel class and the visual level.

Using the Image Scoring Data Set, the ratios of Infected and Dead pixels within the image can be used as inputs and converted into a predicted visual score by

where δ is the normalized frequency of Infected pixels, ω is the normalized frequency of dead pixels, and sp is the predicted visual level. This equation was determined by performing a linear regression across all samples in the Image Scoring Data Set. The Healthy classification ratio was not needed as it is linearly dependent upon the Infected and Dead values.

Using this linear regression equation, we can determine a new level that is directly comparable to the 9-point scale. By converting from the normalized frequency of plant classes to the 9-point scale, it can be determined how well this image scoring system replicates the results of our visual severity levels. Using the Image Scoring Data Set, the visual severity level can be estimated from the ratio of Infected and Dead pixels 1. As mentioned previously, occasionally more images were taken per plot, which can lead to an imbalance in the scored data set as not all plots were imaged equally. To eliminate this possible bias, a random subsampling of images was performed. Only three images per plot were selected, and each image was passed through the scoring pipeline to determine an estimated leaf spot severity rating. The root mean square error (RMSE) value was calculated to be 0.996. Some of this error can be attributed to the 9-point scale’s integer basis compared to the continuous levels of the predicted level. While the visual severity levels force a small range of infection severities to be equal to a single integer level, the predicted level is continuous and will not always match up exactly with the original level.

To identify how the number of images and sampling area impacts the expected error, a second comparison was performed where the predicted level of three images per plot, the same randomly sampled images described previously, were averaged before being passed into the scoring function. The resulting predicted score was then compared to the ground truth measurement for the plot as a whole. This increases the area of the plot that is being measured by a factor of three. When performing this average across three images the RMSE for each plot decrease to approximately 0.800 from the previous 0.996.

Additionally, non-linear fitting functions were investigated to determine the suitability of the linear function. Three support vector machines (SVM) were fit to the entire Image Scoring Data Set using the Regression Learner from MatLab 2023b. The kernel scale, box constraint, and epsilon were all determined dynamically by the Regression Learner application. The Linear SVM was found to be the highest performing with an RMSE of 1.064, followed by the Cubic SVM with an RMSE of 1.071, and lastly was the Quadratic SVM with an RMSE of 1.081. Due to the performance of the SVM models all being slightly worse than the linear model, the linear model was selected as the final model for converting segmentation values into an estimated leaf spot score.

One limitation of our study is that visual severity levels were only conducted at the end of each growing season; however, image data were collected at earlier time-points with our scoring protocol, our previously described Temporal Data Set. A simple hypothesis to test is whether the scoring algorithm produces a lower average level early on in the season, when leaf spot severity should be lower due to the reduced duration of the infection period [5]. To quantify trends in the data over time, our temporal dataset was used to predict the distribution of levels from the UCPS location at two different points in time toward the end of the 2022 growing season (exact dates can be found in Table 1). To avoid biasing the data, a random subsampling was performed in the cases where more than three images were captured of a particular plot, resulting in each of our plots in the data set contributing three images to the analysis. A graph of normalized frequency versus predicted levels for both sampling dates can be seen in Fig. 9. The mean predicted level for the early season subset was 3.87 where the mean predicted level for the late season subset was 4.29, an increase of 0.42 over the course of 12 days. A wider spread of levels is observed on the late season subset, with the standard deviation of 1.33 on the early subset and a standard deviation of 2.18 on the late subset. Finally, we see a higher maximum predicted level in the late subset; the maximum predicted level being 9.32 in the late subset and 7.71 in the early subset. The minimum predicted level in the early subset is 1.05 compared to 1.11 in the late subset.

The first step in this automated leaf spot scoring pipeline involved the handheld imaging component. As described previously, our imaging personnel used a cell phone camera, combined with a simple handheld tool, to perform rapid imaging of a random selection of plots. For our purpose, we needed to obtain hundreds of images across numerous plots in our field trials. Compared to a traditional visual scoring, our imaging protocol was more time-intensive than traditional expert visual scoring by a factor of five. While this limitation may mean that field surveys require five times as many personnel to survey at the same rate as an individual using traditional field survey protocols, the training time is reduced with the imaging protocol and scalability is increased. The element of human subjectivity present in traditional field surveys is greatly reduced using our protocol, which reduces the error introduced by having more personnel present in the scoring process. An additional benefit is that the raw data is stored in the form of captured images, where a traditional field survey does not store data that can easily be revisited if errors are identified.

When the plots were imaged, a minimum of three images were taken, two at either end and one in the center. This does not fully image the plot and means that the entire 6.8 square meter plot must be scored on a combined area of 2.7 square meters, a reduction in area scored by 60%. As described in the results, an RMSE of 0.996 visual scores was observed when evaluating the individual images in the Image Scoring Data Set, and when randomly averaging the results of three images, the RMSE was reduced to 0.800. This reduction in error indicates that a higher number of images reduces the error seen when evaluating a plot for infection severity. This is reinforced by the r2 values when fitting the normalized frequency of each class per the severity score, as seen in Fig. 7 and Table 7. Here, we see a high r2≈0.88-0.95 for all classes when averaging all images associated with a particular severity score. As previously described, the discrete nature of the 9-point scale and the continuous output of the scoring function introduces round-off errors into the measurements. Representing a range of infection severities as an integer value causes unavoidable error due to compressing each severity range to a single integer. Round-off error in the ground truth measurements leads to an unavoidable error when evaluating the performance of the pipeline.

As a limitation of this study, only three images were captured per plot, which limits the multi-angle coverage of each area and could result in leaf occlusion. This imaging protocol was performed to minimize time spent capturing images and minimize the barrier of entry. More advanced image collection protocols are under investigation by peanut breeding teams around the country, such as automated drone image capturing pipelines; future work should could investigate the possibility of combining the pipeline developed here with these systems that are capable of more fully imaging plots and minimizing leaf occlusion [20]. Future work could examine integrating the pipeline developed here with such systems to achieve more comprehensive plot imaging and reduce the impact of leaf occlusion.

A key neural network metric is how predictably it performs on images associated with specific levels of infection severity. The network must both classify pixels correctly and fit into the overall scoring pipeline. One key identifiers of the semantic segmentation network’s generalizability is the normalized confusion matrix that was generated from the testing (holdout) subset of the Neural Network Development Data Set as seen in Fig. 6. These results demonstrate the varying degrees of accuracy for the four classes. The least important class—Non-Plant—shows the highest degree of accuracy. The Healthy and Infected classes show similar levels of accuracy, with accurate classification >80.0% of the time. The Dead class was frequently mistaken for the Non-Plant class, with 40.0% of the Dead pixels being mistaken for Non-Plant pixels and only 53.8% of the Dead class being identified correctly. Despite having the second-highest mean accuracy across all classes, the ResNet50-based model had the worst identification of Non-Plant material of any of the four models. The next closest performer in mean accuracy was MobileNetV2. However, this model suffered from a low accuracy (66.5%) in identifying infected material. ResNet18 performed poorly across almost all classes, with Dead pixel classification accuracy of only 18.5% and Infected pixel classification accuracy of 70.4%. The U-Net model performed similarly to ResNet50 on both the Healthy and Infected classes, and even outscored ResNet50 in regards to the Non-Plant class, but suffered from a low accuracy of 34.9% on the Dead class, indicating that it may struggle in identifying one of the key classes to our scoring pipeline. Finally, the Jaccard index, shown in Table 6, was another key metric for success when evaluating the networks tested for developing our pipeline. The ResNet50 model scored the highest Jaccard Index of the four networks created. While the mean accuracy was slightly lower than the MobileNet-based model, the higher Jaccard Index indicated that the labels produced by the ResNet50 model were more similar to the semantic labels created for testing compared to the labels produced by any other model. The Jaccard Index is based on the overlapping area of the labels rather than a raw accuracy of correctly applied labels; this behavior means the metric is less impacted by imbalanced datasets than the overall accuracy metric, making it key to evaluating the performance of the trained neural network [49].

Because many other damage detection pipelines treat damage as a singular class rather than differentiating between different classifications of damage, the networks’ ability to differentiate between damage to the canopy in the form of infected lesions and dead plant material was of key interest [50]. In the ResNet50 model that was chosen, Infected pixels were misclassified as Dead 5.7% of the time, and Dead pixels were misclassified as Infected 1.6% of the time. Other models also showed a low misclassification rate between the two different damage classifications. The ResNet18 model for example showed no misclassification where Infected was mislabelled as Dead, and only 3.6% of Dead pixels were misclassified as Infected; the U-Net and MobileNetV2 also showed low misclassification rates between Infected and Dead, as seen in Fig. 6. These results are indicative that the four models tested are capable of differentiating between the types of damage seen in our field trials, implying suitability for these networks in multi-class classification tasks across different timespans. Future work may focus on developing pipelines capable of detecting and scoring additional types of damage in peanut, to allow for a more complete tool.

We saw a small level of defoliation that was present even at low levels of severity infection. Referring to Fig. 6, we observed that Non-Plant was mistakenly for Dead plant material 2.5% of the time. The average frequency of Dead plant material in an image associated with a level 1 severity infection was 0.03, or approximately 3% of pixels classified as plant material. The standard deviation of our Dead pixel frequency from images associated with a level 1 infection was only 0.01, or 1%. It is believed that much of this remaining error in Dead classified pixels can be attributed to misclassification by the neural network; the confusion matrix seen in Fig. 6 shows that 40% of the Dead pixels in the holdout set were misclassified as Non-Plant. Much of the Non-Plant was soil, which indicated that many of our Dead pixels were mistaken for soil or other background material. Despite the misclassification, the dead class showed a strong linear relationship with the visual ground truth score, as shown in Fig. 7. The averaged dead classified pixel fit demonstrated an r2 of 0.883 and an RMSE of 0.11 as detailed in Table 7. This indicated that, while misclassification on the full Score Set may be higher than that of the test subset from the Neural Network Development Data Set, it was low enough and consistent enough for use in the scoring pipeline. This predictable behavior was reinforced by explicitly quantifying the Healthy class’s linearity, where the average fitting function had an r2 of 0.960 when predicting the visual score as seen in Table 7. The RMSE values were on the order of magnitude of 0.1, indicating a variance of almost 10% of the pixels within an image versus the linear function’s prediction. Given that each severity score represents a range of infection severity, this range of possible outputs within a predicted score is to be expected. A high correlation coefficient between the visual score and the healthy number of pixels, with a correlation coefficient of − 0.81. It should be noted that the three normalized classification values are linearly dependent on one another; the healthy pixel value is equivalent to 1-δ-ω, so while the Dead and Infected classes suffered from a lower correlation, the combined sum of the two provided a strong correlation to the visual severity score that was equivalent to the healthy class score.

The overall pipeline had an RMSE of 0.996 visual scores when using individual images and 0.800 visual scores when using three images per plot. The average error for an image will be just under one level of severity on the 9-point scale, while imaging more of the plot and averaging the scores of several images can reduce the error to below one severity level, indicating the suitability of the pipeline.

The pipeline includes an intermediary step to extract ratios of Healthy, Infected, and Dead plant regions. While the 9-point scale is used by NC State’s breeding program, other scales, such as that of Chiteka et al. have been historically applied [6, 11, 20]. If the breeding community adopts a new scale, the neural network can still be used, requiring only adjustments to the scoring function to accommodate the new severity standard.

The randomly sampled scores over time served as a demonstration of the pipeline’s scalability. Given this pipeline’s purpose as a tool for monitoring severity in peanuts, it is important to show that the pipeline can be used not just to score at the end of field trials (when scoring would normally be done), but also as a tool to monitor the severity’s progression over time, which impacts yield [5]. Using the Temporal Data Set, the early season subset demonstrated a significantly lower maximum predicted score on individual images than the late season subset—a maximum score of 7.71 versus 9.32, respectively. A higher mean single image score was also seen, with the mean increasing from 3.87 to 4.29—an increase of 0.42 in the predicted visual score between the two subsets. The late-season subset also exhibited a higher standard deviation compared to the early-season data set of 2.18 versus 1.33, respectively, which indicated that the spread of predicted severities increased as the growing season progressed.

While these observations aligned with our expectations, the limited number of sampling dates makes it difficult to draw definitive conclusions about the sensitivity of our pipeline to small changes in the severity of the infection over time. However, such a capability would allow for the detection of increasing severities over time and would enable recurring monitoring of leaf spot progression during field trials—offering a new dimension of analysis for peanut breeding programs. While future studies—which capture visual score data at different times of the year—will help to further justify this in the future, these results indicated that capturing a wider range of scores across a larger span of time should be possible using this pipeline.

Not applicable.

Not applicable.

The author(s) declare(s) that there is no conflict of interest regarding the publication of this article.

The datasets generated and/or analysed during the current study are available in the Automated Leaf Spot Scoring GitHub repository: https://github.ncsu.edu/jclarse2/AutomatedLeafSpotScoring.

Not applicable.