Computer Vision Algorithms for Yield Mapping in Specialty Farms

Thumbnail Image

Persistent link to this item

View Statistics

Journal Title

Journal ISSN

Volume Title


Computer Vision Algorithms for Yield Mapping in Specialty Farms

Published Date




Thesis or Dissertation


Precision farming and phenotyping technologies have the potential to drastically transform the agricultural landscape. For commodity crops such as maize, wheat and soy recurring farming tasks such as seeding, weeding, irrigation, fertilization, application of pesticides, harvesting, and storage are in the process of being completely automated. Specialty crops (tree fruit, flowers, vegetables, and nuts) are excellent candidates for similar automation as they have high monetary value, high management cost and high variability in growth. An important capability for both precision agriculture and phenotyping is yield mapping. Yield mapping for tree fruit is challenging because it involves solving multiple computer vision (fruit detection, counting, recovering underlying 3D geometry for tracking fruit across different frames in continuously changing illumination) as well as planning problems (path planning for covering all fruit, picking fruit). The main goal of this dissertation is to develop computer vision and deep learning algorithms for yield mapping in specialty farms. The dissertation is divided into three parts. The first part is dedicated to developing practical solutions for yield mapping in specialty farms. We present solutions for fruit detection, counting, recovering the underlying scene geometry and fruit tracking. We integrate these individual solutions in a modular manner and create a flexible framework for complete yield estimation. Additionally, we perform an extensive experimental evaluation of the developed system and sub-components. Our algorithms successfully predict 97% of the ground truth yield and outperform all existing state-of-the-art methods. Some of these efforts are now in the process of being commercialized. In the second part of the dissertation, we study a problem where a manipulator equipped with a camera, mounted on a ground robot is charged with accurately counting fruit by taking a minimum number of views. We present a method for efficiently enumerating combinatorially distinct world models most likely to generate the captured views. These are incorporated into single and multi-step planners for accurate fruit counting. We evaluate these planners in simulation as well as with experiments on a real robot. In the third part, we study the problem of realistic synthetic data generation for training deep neural networks. We present a method that jointly translates the synthetic images and their underlying semantics to the domain of the real data so that an adversarial discriminator (a deep neural network) cannot distinguish between the real and synthetic data. This method enables us to stylize the synthetic data to any fruit, lighting condition and environment. It can be applied to a wide variety of domain transfer tasks beyond fruit detection and counting (e.g from Grand Theft Auto (GTA) to Cityscapes for autonomous driving). Additionally, it enables us to perform image to image translation with significant changes in underlying geometry (e.g circles to triangles, sheep to giraffe, etc). These results in this dissertation together present a complete yield monitoring system for specialty crops, view planning strategies for accurate fruit counting and a framework for generating realistic synthetic data. These methods together push the state-of-the-art and take us one step closer toward building a sustainable infrastructure for intelligent integrated farm management.


University of Minnesota Ph.D. dissertation. November 2019. Major: Computer Science. Advisor: Volkan Isler. 1 computer file (PDF); xii, 175 pages.

Related to




Series/Report Number

Funding information

Isbn identifier

Doi identifier

Previously Published Citation

Suggested citation

Roy, Pravakar. (2019). Computer Vision Algorithms for Yield Mapping in Specialty Farms. Retrieved from the University Digital Conservancy,

Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use. Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories.