Computer Vision Algorithms for Yield Mapping in Specialty Farms

Abstract

Precision farming and phenotyping technologies have the potential to drastically transform the agricultural landscape. For commodity crops such as maize, wheat and soy recurring farming tasks such as seeding, weeding, irrigation, fertilization, application of pesticides, harvesting, and storage are in the process of being completely automated. Specialty crops (tree fruit, flowers, vegetables, and nuts) are excellent candidates for similar automation as they have high monetary value, high management cost and high variability in growth. An important capability for both precision agriculture and phenotyping is yield mapping. Yield mapping for tree fruit is challenging because it involves solving multiple computer vision (fruit detection, counting, recovering underlying 3D geometry for tracking fruit across different frames in continuously changing illumination) as well as planning problems (path planning for covering all fruit, picking fruit). The main goal of this dissertation is to develop computer vision and deep learning algorithms for yield mapping in specialty farms. The dissertation is divided into three parts. The first part is dedicated to developing practical solutions for yield mapping in specialty farms. We present solutions for fruit detection, counting, recovering the underlying scene geometry and fruit tracking. We integrate these individual solutions in a modular manner and create a flexible framework for complete yield estimation. Additionally, we perform an extensive experimental evaluation of the developed system and sub-components. Our algorithms successfully predict 97% of the ground truth yield and outperform all existing state-of-the-art methods. Some of these efforts are now in the process of being commercialized. In the second part of the dissertation, we study a problem where a manipulator equipped with a camera, mounted on a ground robot is charged with accurately counting fruit by taking a minimum number of views. We present a method for efficiently enumerating combinatorially distinct world models most likely to generate the captured views. These are incorporated into single and multi-step planners for accurate fruit counting. We evaluate these planners in simulation as well as with experiments on a real robot. In the third part, we study the problem of realistic synthetic data generation for training deep neural networks. We present a method that jointly translates the synthetic images and their underlying semantics to the domain of the real data so that an adversarial discriminator (a deep neural network) cannot distinguish between the real and synthetic data. This method enables us to stylize the synthetic data to any fruit, lighting condition and environment. It can be applied to a wide variety of domain transfer tasks beyond fruit detection and counting (e.g from Grand Theft Auto (GTA) to Cityscapes for autonomous driving). Additionally, it enables us to perform image to image translation with significant changes in underlying geometry (e.g circles to triangles, sheep to giraffe, etc). These results in this dissertation together present a complete yield monitoring system for specialty crops, view planning strategies for accurate fruit counting and a framework for generating realistic synthetic data. These methods together push the state-of-the-art and take us one step closer toward building a sustainable infrastructure for intelligent integrated farm management.