Recently, there have been a few works focusing on algorithms both for branch and node detection to support agricultural task decisions, such as for pruning. Cuevas-Velasquez et al. [9] developed a cutting location algorithm focusing on completing pruning tasks on rose bushes. The authors’ approach was based on stem detection, making use of a Convolutional Neural Network (CNN) to segment the branches from the background of the image. The cutting point location was then selected according to the segmented stem’s height. Marset et al. [10] proposed the use of computer vision algorithms to detect buds in grapevine branches. For this purpose, the authors implemented a Fully Convolutional Network (FCN) with a MobileNet, which segmented pixels in the image corresponding to buds. Gentilhomme et al. [11] proposed a novel method for grapevine structure estimation to assist in pruning activities. The authors analysed the grapevines’ structure considering the similarities with the problem of human pose estimation, with the plant’s nodes as the landmarks and the branches as the spatial dependencies. Gentilhomme et al.’s method [11] consists essentially of a Stacked Hourglass Network (SHG), which outputs both the nodes and vector fields of the branches out of two-dimensional (2D) images of the grapevines. Afterwards, these features are associated according to the points’ coordinates, resulting in a complete vine structure.

This paper aims to present efficient light-computing deep learning models for node detection on 2D images of grapevines. Despite the fact that recent works have been able to use 2D images to compute cutting locations for pruning tasks, they present constraints. In the work of Cuevas-Velasquez et al. [9], the cutting decision was based on the stems’ heights, not providing data on the detection of small objects such as nodes. The authors acknowledged that, in future work, the pruning criteria should be updated to consider more phenotypic traits. Meanwhile, in the research of Marset et al. [10], although a detection algorithm was implemented for bud detection in grapevines, the images used were close-ups of the branch, not allowing for visibility over the whole grapevine. The work of Gentilhomme et al. [11] successfully detected nodes in grapevines. However, the proposed network was trained with images containing an artificial background, and the authors considered that the algorithm was not robust enough to successfully perform on environments where the artificial background was removed from behind the grapevines.

This work aims to surpass the constraints present in the previous approaches for node detection by providing a solution that guarantees the following:

In order to achieve these requirements, this paper studies the adaptability of four distinct You Only Look Once (YOLO) [12] models (YOLOv7 [13], YOLOv8 [14], YOLOv9 [15] and YOLOv10 [16]) on performing node detection on grapevines. These models were trained with a public grapevine dataset containing images with artificial backgrounds and were further tested on detecting nodes of grapevines with distinct configurations on natural environments without artificial backgrounds, as detailed in Section 2.1.

Furthermore, this allowed us to understand the requirements to correctly detect the grapevine’s nodes on two distinct and highly relevant viticulture regions in Portugal, in which the grapevines’ configurations differ significantly, as well as the backgrounds created by the region’s landscapes and climate [17]. The work developed also resulted on the creation of a publicly available dataset of Portuguese grapevines for node detection [18], contributing as a first approach towards the development of a perception module for an autonomous robotic system capable of performing pruning tasks with equivalent or higher performance than the same type of task performed by human labour.

In order to train the models, we used the publicly available 3D2cut Single Guyot dataset [11], whose respective authors created to develop a method for pruning task assistance. This choice was made mainly due to three reasons: the images captured contained the entire grapevine on each frame, making it possible to visually distinguish each constituent and to simultaneously perform the detection of the same plant on all the canes; a single grapevine was captured per frame in most of the images, avoiding the presence of additional unwanted information; the dataset was collected with an artificial background behind the grapevines, which resulted in a more intuitive annotation procedure, validation of the bounding boxes’ location and distinction of the different constituents of the plant.

New annotations to the existing dataset have been performed to consider only the nodes on the primary canes of the grapevine in the centre of each image, which are the main detection goals since pruning will be conducted on these primary canes by considering the number of nodes present as a criterion. This step was performed to guarantee that it would achieve lightweight models for node detection by removing non-relevant information on the detection framework, such as nodes in areas of the grapevine not corresponding to primary canes. These annotations were performed using the Computer Vision Annotation Tool (CVAT) (CVAT. https://www.cvat.ai/ (accessed on 17 October 2024)) platform, with bounding boxes surrounding the portions of the canes containing the nodes, as seen in Figure 1.

To comprehensively assess the resilience of the deep learning models proposed in this study to varying grapevine conditions, particularly those with cluttered backgrounds, it became imperative to curate new datasets distinct from the 3D2cut Single Guyot dataset [11], mainly containing images without artificial backgrounds. Thus, images from two distinct Portuguese viticulture regions were used to improve this evaluation.

The first set of images was collected at Centro de Estudos Vitivinícolas do Dão, located in the Dão region (40°31′31.5″ N 7°51′22.7″ W), while the second dataset was obtained from Quinta do Seixo in the Douro region (41°10′05.4″ N 7°33′17.5″ W). The performance of the trained YOLO models on images from these two locations was considered relevant to this study because the grapevine configuration is different from the 3D2cut Single Guyot dataset [11]. Additionally, these images were captured without using artificial backgrounds on regions with distinct background environment information due to natural light sources and landscape, which might have influenced the outcome of the detection. Both datasets were acquired during the winter months, between December and January. The Dão dataset was captured on a clear afternoon with abundant light, whereas the Douro dataset was captured on a cloudy afternoon, where, despite sufficient lighting, the sunlight was less intense when compared to the Dão location.

For both datasets, two camera perspective variations in relation to the grapevine were employed for further performance comparison. The first was a horizontal perspective, which allowed the background of the image to have more objects in sight, such as adjacent vineyard rows and hills. The second was a bottom-to-top perspective using the sky as a more uniform background of the image, as illustrated in Figure 2. The images were acquired using a digital OM System camera (OM System. https://explore.omsystem.com/us/en/ (accessed on 17 October 2024)) at a distance of approximately 80 cm of the captured grapevine with a resolution of 4608 × 3456 pixels (px).

YOLO deep learning models have been widely used in real-time object detection and applied in various fields, including robotics [19]. These models are single-stage objective detection algorithms, granting them high speed combined with precision [20]. Due to the detection performance reported by their respective authors on the Microsoft Common Objects in Context (MS COCO) dataset [21], the following state-of-the-art YOLO versions were selected for this work.

YOLOv7 was published in 2022 and at the time surpassed the previously developed object detectors in terms of speed and accuracy. This accuracy was improved without incrementing the inference speed at the cost of increased training time [22]. The authors also designed a version of this model to be run in edge devices—the YOLOv7-tiny [13]—and since the goal of this work was to study detection algorithms capable of real-time inference on an autonomous robotic system, this version was chosen as one of the models for this purpose.

The YOLOv8 is one of the most recent object detection models, released in 2023. It has a backbone similar to the one used on YOLOv5 [23] with modifications that allow for improvements on detection accuracy [22]. Moreover, the YOLOv8 uses Distribution Focal Loss (DFL) [24] and Complete Intersection over Union (CIoU) [25] loss functions for the bounding-box loss, which is expected to contribute to improving the object detection performance in smaller objects [22], as it is relevant for node detection scenarios due to the size of the nodes when compared to the full image of a grapevine. In order to guarantee that the model used could compute light and was capable of being implemented on a real-time object detection system, the model selected was the YOLOv8 Small (YOLOv8s), which is the second-smallest scaled version of the model provided by Ultralytics (Ultralytics. https://www.ultralytics.com/ (accessed on 17 October 2024)). Furthermore, previous works have already developed on bud detection based on this version of the YOLOv8 model. Xie et al. [20] implemented the YOLOv8s to detect buds on tea trees, achieving a Mean Average Precision (mAP) of 88.27% and an inference time of 37.1 ms.

In 2024, the YOLOv9 was published by the same authors of the YOLOv7. The authors designed a novel lightweight network architecture called the Generalized Efficient Layer Aggregation Network (GELAN), which, when combined with the concept of Programmable Gradient Information (PGI), also developed by the same authors, allowed it to surpass the performance of the existing object detectors. The PGI framework on the GELAN architecture helped to reduce information bottleneck during the feedforward process, which resulted in more retained complete information when compared to architectures such as PlainNet, ResNet and CSPNet [15]. In order to compare the performance of this novel YOLO model in grapevine node detection with the previously presented ones, we used the light-computing YOLOv9 Small (YOLOv9-S).

Also in 2024, YOLOv10 was announced with a novel approach for real-time object detection, eliminating the use of non-maximum suppression (NMS). The model is based on the YOLOv8 from Ultralytics, which the authors have chosen due to the efficient balance achieved between latency and accuracy. Despite being based on the YOLOv8, according to the authors, the YOLOv10 achieves higher average precision (AP) while requiring less parameters and fewer calculations and also achieving lower latencies. The YOLOv10 was developed containing dual-label assignments, which, during the training phases, recurs to two heads to optimize the model. However, during inference, only one head is maintained (one-to-one) to make predictions, avoiding using NMS in post-processing. Similarly to the previous ones, the model used was the small variant: the YOLOv10 Small (YOLOv10-S).

The four distinct YOLO models presented were trained on a total of 13,539 images of grapevines, whereby 759 of the images were from the 3D2cut Single Guyot dataset [11] and the remaining 12,780 images were from augmentations of these original images.

The 3D2cut Single Guyot dataset [11] contained varied images of grapevines with significant differences between vines not only on their main structure, but also on cane orientation and size. However, this dataset is not a complete representation of possible environments and scenarios in a real-life application, lacking variations in the image capture perspective as well as external elements that might be present in the field. To address the lack of variability in this dataset, the following augmentations presented in Table 1 were implemented. These augmentations used to improve the training image set presented geometric modifications to the existing dataset (image flip, scaling, and angle change), and also inserted some external elements that can be present in real-life applications on the field, such as rain, mud, fog, blur, variations in hue and saturation, ISO noise, and optical distortion, as can be seen in Figure 3. These altered images allowed us to transform the dataset to a more reliable approximation of the agricultural work environment, which is often unpredictable and subjected to harsh and variable climatic conditions [26,27].

The deep learning models used were initially trained with similar hyperparameters for 50 epochs. However, there was a lack of convergence on the training loss for all the models. Therefore, the number of training epochs was gradually incremented to 150 for all the models to guarantee that the training loss value stabilized. Despite using similar hyperparameters in the first set of training conducted, eventually, a few suffered different tuning strategies to guarantee that the best results from each model were achieved as well as to ensure correct adaptation to the hardware used. Parameters such as input and batch sizes had to be modified on all models due to constraints on the hardware of the device used for training (NVIDIA GeForce RTX 3090, NVIDIA Corporation, Santa Clara, CA, USA) (NVIDIA GeForce RTX 3090. https://www.nvidia.com/en-eu/geforce/graphics-cards/30-series/rtx-3090-3090ti/ (accessed on 17 October 2024)). However, YOLOv8s, YOLOv9-S and YOLOv10-S presented more difficulties to reach convergence during training. For this purpose, the Cosine learning Rate Scheduler Function [28] was activated during training, and mosaic data augmentation was deactivated on the last 50 epochs to improve loss stabilization at the last training stages. Furthermore, different optimizers were tested on different trainings, and it was discovered that on all the models tested, the AdamW contributed to better convergence. The basic configurations of the models and the hyperparameters that suffered further modifications in each deep learning model used, which resulted in the most favourable results, can be seen in Table 2.

Three different sets of images were selected to evaluate the performance of the models in detecting the nodes on grapevines. The first set was composed of images from the 3D2cut Single Guyot dataset [11], containing an artificial background behind the captured grapevines. The remaining two sets consisted of images selected from the datasets gathered on the Portuguese Dão and Douro regions, containing 30 original images and 150 augmented images, which added geometric modifications to the original images in order to increase the variability of the dataset by containing grapevines captured in different layouts.

The images from the 3D2cut Single Guyot dataset [11] were used as a benchmark to compare the robustness of the YOLO models trained. Since these images were similar to the ones used for training, it was expected that the models would achieve the best results. Dão and Douro datasets constituted of images taken in open fields without covering the objects behind the grapevines. This feature allowed these datasets to be used as indicators to evaluate the robustness of the models developed in detecting nodes in images with natural backgrounds (cluttered).

Furthermore, despite the fact that the models have been trained with an input size of 640 × 640 px, tests were also conducted with changes in the input parameter to consider a size of 1280 × 1280 px. Since nodes on grapevines are relatively small objects on the overall size of the image, this test was crucial to reduce the loss of information on the images used. However, it was expected that this increase in the input size would result in delays in the inference times of the models [30].

The results obtained on the validation phase of the detection models trained on the four different datasets are presented in Table 3. The metrics obtained were analysed at confidence levels of at least 10% and at the best F1-Score obtained in each model. Confidence levels below 10% were excluded from the analysis, as lower confidence scores are likely to lead to an increase in false positive detections [31]. An increase in false positives would negatively affect the performance of the node detection task, resulting in inaccurate conclusions and potentially compromising the reliability of the system to be integrated into an autonomous pruning robot. Furthermore, Table 4 displays the average inference time per image on each of the models with different input sizes.

As expected, the models’ inference performance was considerably higher on the 3D2cut Single Guyot dataset [11], where all the models presented a similar detection capability, as seen in Figure 6. This behaviour occurs not only because this set constituted of images similar to the ones used for training, but also due to the amount of additional information on the background of the Dão and Douro datasets, which induced the occurrence of some false positive detections on the background of the images.

In both of these two datasets (Dão and Douro), we noticed that the YOLOv7-tiny achieved higher F1-Score values than the other three models, with this difference being more noticeable on these datasets in comparison to the results obtained on the 3D2cut Single Guyot dataset [11]. Furthermore, although YOLOv8s was the second best performing model on the Dão dataset, the performance changed significantly on the Douro dataset, in which this model presented varying results according to the input size used, resulting in lower F1-Score values for an input size of 640 × 640 px. Although the Precision of YOLOv10-S was usually higher than the other models on the Portuguese datasets, the Recall was lower, especially when considering an input size of 640 × 640 px. This allowed us to notice that when using the YOLOv10-S, despite not encountering many false positives, the model presented more difficulties in detecting small objects on each frame, ignoring a few nodes in the image. This behaviour makes it less valuable for the intended task than the other models analysed, since this behaviour results in a considerable amount of nodes not being considered as such.

Furthermore, when comparing the results obtained between the two datasets without an artificial background, the Douro dataset resulted in less-accurate detections by the models than the Dão dataset. Analysing the images present in each one, it is possible to observe what features might be causing this discrepancy. The grapevines on the Dão dataset, despite having a structure different from the grapevines on the images of the training dataset, make distinguishing each cane on the plant easily feasible. Meanwhile, on the Douro dataset, the grapevines have more canes overlapped, and thus sometimes make it more difficult to distinguish the primary canes; this also results in nodes that can be occluded by nearby branches, as can be observed in Figure 7.

Regarding the annotations performed on the 3D2cut Single Guyot dataset [11], only the nodes on the primary canes of the centred grapevine on the image were selected. The same logic was implemented on the construction of the ground truth for the Dão and Douro image sets. However, the models also detected a few nodes on canes in adjacent grapevines that slightly appeared on the images captured. This consequently resulted in these detections being considered as false positives, which is not entirely wrong since the nodes do not belong to the focused grapevine, but they are correctly assumed as being nodes. Although these situations slightly affect some of the images, they were not considered relevant for the overall performance of the models in detecting nodes on these datasets, since it only happened in a few cases. Furthermore, in these sets of images taken in open field, the pictures captured in a bottom-to-top perspective are preferable to achieve better results. This approach helped reduce the background information, resulting in less false positives detected. In Figure 8, it is possible to notice that a horizontal camera perspective captures additional information, not only from other grapevines behind the one in foreground, but also from the ground of the figure. Meanwhile, a bottom-to-top perspective reduces the amount of content in the background of the image, allowing for a more precise detection.

Regardless of the performance differences between datasets and between images containing artificial backgrounds or not, grapevine node detection recurring to YOLO detection models presents an efficient and more versatile approach when compared to the previously implemented methods. The previously published work by Gentilhomme et al. [11] managed to achieve an overall Precision of 95% and Recall of 90% in the task of node detection, having an execution time between 0.55 s and 8 s. Despite obtaining good detection performance on the selected testing environment, the authors acknowledged that the algorithm requires optimization to be implemented on a real-time system and also considered that the algorithm proposed for pruning assistance would be severely impacted in work conditions without uniform artificial backgrounds. However, this work assures that grapevine node detection is feasible in heterogeneous environments recurring to state-of-the-art YOLO detection models, despite presenting a slight performance decrease.

Furthermore, it was possible to assess the capability of performing node detection on the entirety of the grapevines even when configurations different from those used in training are provided to the models. In Figure 9 and Figure 10, it is possible to analyse the inference behaviour of the models used on each of the datasets acquired in Portugal, Dão and Douro, respectively, considering an input size of 1280 × 1280 px.

According to the work by Botterill et al. [1], a human takes approximately 2 min to prune a grapevine. In a robotic system, there might be several necessary additional steps from the detection of the cutting location until the cut is completed. These steps are mainly related to navigation towards the cutting location and actuation of the cutting tool [7,32]. Considering this, minimizing the inference time of the detection algorithm is crucial to ensure the maximum efficiency of the system by reducing the necessary time to execute the pruning task on each vine.

The average inference time of the analysed YOLO models varied significantly. These inference time differences were not only between the distinct versions of the models, but also between different input sizes. YOLOv7-tiny was the fastest model at grapevine node detection, presenting inference times below 100 ms. YOLOv8s, YOLOv9-S and YOLOv10-S presented significantly larger times, with YOLOv8s reaching an average of around 502.5 ms when detecting nodes with a 1280 × 1280 px input size.

Despite the most recent models (YOLOv9-S and YOLOv10-S) being faster than the 502 ms of YOLOv8s, their inference times on 1280 × 1280 px sized images are still near 300 ms, which is significantly higher than the inference time obtained using the YOLOv7-tiny. Furthermore, the YOLOv7-tiny model presents less computational complexity than the other models tested, having 6.2 million (M) parameters and 13.8 Giga (G) Floating Point Operations Per Second (FLOPS). This number of operations corresponds to around 36% fewer operations than YOLOv10-S, which is the second model tested with fewer FLOPS, as can be seen in Table 5. Considering this, as a matter of the capability of implementation in a real-time system, the YOLOv7-tiny would be the preferred model of the four in order to ensure the system’s efficiency.

The data presented in this study are openly available in the digital repository Zenodo: Douro & Dão Grapevines Dataset for Node Detection—https://doi.org/10.5281/zenodo.10991688.