O2RNet: Occluder-occludee relational network for robust apple detection in clustered orchard environments

Automated apple harvesting has attracted significant research interest in recent years because of its great potential to address the issues of labor shortage and rising labor costs. One key challenge to automated harvesting is accurate and robust apple detection, due to complex orchard environments that involve varying lighting conditions, fruit clustering and foliage/branch occlusions. Apples are often grown in clusters on trees, which may be mis-identified as a single apple and thus causes issues in fruit localization for subsequent robotic harvesting operations. In this paper, we present the development of a novel deep learning-based apple detection framework, called the Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in clustered situations. A comprehensive dataset of RGB images were collected for two varieties of apples under different lighting conditions (overcast, direct lighting, and back lighting) with varying degrees of apple occlusions, and the images were annotated and made available to the public. A novel occlusion-aware network was developed for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations of the developed O2RNet were performed using the collected images, which outperformed 12 other state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection. O2RNet provides an enhanced method for robust detection of clustered apples, which is critical to accurate fruit localization for robotic harvesting.