Stand counting is an important task in breeding and crop management, providing farmers and breeders with information about the germination rate and population density of crops. In the breeding stage, the corn stand count can provide the seed germination rate for breeders and geneticists. As for the planting stage, stand counting can provide the plant population density to farmers, which is an important trait for crop yield and quality [1,2,3,4,5,6,7]. This information can also help farmers with some essential management practices, such as determining the necessity of replanting [8].

The traditional method for stand counting usually samples the corn population manually with quadrats [9], which is time consuming, laborious, and cost intensive. In addition, manual stand counts are not feasible for large fields and are prone to human error. Recent advances in machine vision provide automatic methods for farmers and breeders to count plant seedlings and determine yield-related traits such as germination rate and plant population density. One popular method has been to use a fixed camera on an unmanned aerial vehicle (UAV) to capture digital images of plants, then process the images to obtain the plant stand count offline [10,11,12,13,14].

UAV-based stand counting usually takes images with camera heights varying from 10 m to 50 m [11,12,13], with some even over 100 m [15,16]. Thus, it is challenging to acquire a high-resolution image of every single plant, especially for the small plants that are in their early stages. As a result, these small objects cannot be detected accurately. Apart from UAVs, there are many ways to collect seedling images, such as moving-cart-based systems [15], tractor-based systems [17,18], and automatic field robots [19], which can collect images and information accurately and efficiently [20]. Besides RGB-camera-based methods, there are also some other sensors that can be used for object counting in the field, such as Lidar sensors [21,22].

Detecting plants accurately from images is the foundation of image-based stand counting. Different methods and their counting accuracies are summarized in Table 1. UAV-based stand counting usually requires a stitching and alignment operation. Some commercial software such as Agisoft Photoscan (Agisoft LLC, St. Petersburg, Russia) and Pixel4D (Pix4D S.A., Lausanne, Switzerland), as well as some algorithms, have been developed [11] to align UAV-collected images automatically. Then objects need to be identified from the images. In traditional image processing, different plant features have been extracted or designed to identify objects, like edge [23], color [24] and shape [12]. Some are also equipped with a hyperspectral sensor, and identify objects through NDVI index [11].

Since the significant evolution of computation in 1999, graphics processing units (GPUs) have been developed for image processing and a revolution has taken place in the application of deep learning. State-of-the-art networks include the faster region-based convolutional network (Faster R-CNN) [25], the mask region-based convolutional network (Mask-RCNN) [26], and You Only Look Once (Yolo) [27,28,29]. It is a promising method to develop algorithms for agricultural object detection and counting based on deep convolutional neural networks (CNNs) [13,17,30]. In particular, Faster R-CNN [25] and Yolo [28] are representative two-stage and one-stage detection networks, respectively. Each of them has made significant breakthroughs in the detection of different objects. Faster R-CNN features high accuracy, while Yolo features a high processing speed. The two networks have been widely applied in agricultural object detection. For example, DeepSeedling integrated Faster R-CNN as the target detection network, then developed a framework to track and count cotton seedlings in the videos [17]. In addition, Faster R-CNN was used as the detection network for detecting other agricultural objects such as plant leaves [31], maize seedlings [32], kiwi fruit [33], grapes [34], and apples [35]. However, most Faster-R-CNN-based approaches are offline. Real-time detection for similar objects usually implements the Yolo network [36,37,38,39,40], which can reach an F1 score of 0.817 for apple detection (with YoloV3) [39], and 0.944 for mango detection (with YoloV3-tiny) [36]. For this study, we chose the Yolo model to satisfy the online counting requirement.

The above image-based counting studies have achieved satisfactory counting accuracies (Table 1). For example, Rawla et al. (2018) achieved corn stand counting with an accuracy of over 99%, using a MicaSense RedEdge Multispectral UAV Camera with a 6 mm lens, mounted on a Hexacopter UAV. The images were processed manually through an ImageJ-based plugin [41]. Cameras equipped with expensive hyperspectral sensors can also achieve 98% count accuracy [11]. These methods are costly, however, both in money and time.

In summary Table 1, the most commonly used counting system is UAV-based, but no real-time counting system has really been developed. The UAV-based counting systems have two major disadvantages. The first is a need for reliable hardware. A stable UAV is the basic piece of equipment, and a high-resolution camera is also needed to collect detailed information on plants from a long distance. Expensive sensors are added to improve the count accuracy. The second disadvantage is a reliance on trained experts. A trained pilot is needed to operate the UAV expertly, which is essential to the success of image acquisition and plant detection. In some methods, the image preprocessing stage also requires human intervention [8,11,12,23,42]. Apart from this, battery life and payload are also limitations if considering long-duration work.

Our objective for the study was to develop a machine learning-based image processing pipeline for real-time early-season corn stand counting, which could be integrated into an unmanned robotic system for plant growth monitoring and phenotyping. This proposed corn stand counting pipeline was expected to be an accurate, efficient, and reliable solution for automatically counting early-season corn plants. No extra sensor was needed, only a camera. No professional operator was needed. The system could be easily mounted on manned cart, tractor, or field robotic systems for counting similar plants, like sorghum.

Counting for corn plants at stages V2 and V3 achieved high accuracies of 98.66% and 98.06%, respectively. However, performance deteriorated when counting plants at stages V1 and V4. Detection precision for both stages V1 and V4 is lower than for the other two stages, but the main reason for this low counting accuracy is different. At stage V1, the YoloV3-tiny model can usually detect objects correctly but the Kalman filter fails to track them (Figure 9b). When detected bounding boxes are small, the values of s and r are small and change slightly along the time axis. In our discrete linear tracking system, the noise from motion is high as a result of the cart vibration and uneven road in the field, and this makes it difficult to track the small objects. Thus, the performance of counting is much lower than the detection. As for counting at stage V4, the counting accuracy decreases because of the overlapping and occlusions of the plant canopies. The similarity between V4, V2, and V3 will confuse the neural network. The YoloV3 model tends to detect a single corn plant as several objects (Figure 9c), and new tracking IDs are assigned to them. Thus, when they cross the finish line, the counting number is larger than the ground truth. That is the reason why counting accuracies are higher than detecting precisions when using YoloV3 and YoloV3-tiny original weight at stage V4.

Aside from the two aforementioned reasons, other random noise introduced by the detection network can also affect performance, such as missed detections (Figure 9d) and false detections (Figure 9c,e). However, these falsely detected bounding boxes generally have a very short life (≤2 frames). It is rare that a new tracking ID will be assigned to them, and even if one is assigned, it usually expires before they cross the finish line successfully.

To improve the performance of stand counting with corn plants at the V1 stage, the camera height may need to be adjusted to obtain detailed information for tracking. If one chooses a high-resolution camera to capture images, zooming into a suitable size on the region of interest will also work (but the network must also retrain based on the old weight). In addition, there are some advanced filters which have been developed to track small targets [52,53,54]. As for counting plants at stage V4 and later stages, the main issue is caused by the overlap of plant canopies in top-view images. In this case, detecting plant stems in side-view images may be a better option for stand counting.

In this paper, we present a pipeline for counting corn seedlings in the field at growth stages from V1 to V4. The pipeline combines a powerful high-speed one-stage detection network, YoloV3-tiny, with a Kalman filter. A customized training method has also been introduced to improve YoloV3-tiny’s detection accuracy. Experiments showed that this method can count corn plants at stages V2–V3 with accuracies over 98% under different outdoor lighting conditions, outperforming other CNN-based approaches for fruit counting [10,17]. Furthermore, the YoloV3-tiny-based model can achieve real-time counting. Thus, the pipeline is a cost-effective, high-speed, and real-time method. In addition, only one camera is needed for the system. We believe this pipeline can most likely be applied to count similar agricultural objects such as sorghum and soybean seedlings. There are still some improvements that can be made in this method, which is also our future study plan. For example, designing a height-changeable mechanical structure to suit different seedling heights could provide a big enough object (enough features) for the network to learn. A side-view of corn will be adopted to count the V4 stage and later stages, to overcome the overlapping problem.