Active vision for efficient 3D reconstruction and rendering

Abstract

Active vision (AV) has been in the spotlight of robotics research due to its emergence in numerous applications, including urban surveillance, agriculture, and biomedicine just to list a few. Two major AV problems that gained popularity are the 3D reconstruction and scene rendering of targeted environments using different views of 2D images. While collecting and processing a large number of arbitrarily taken 2D images may become an arduous process in several practical settings, an efficient solution is to seek the optimal placement of available cameras in the 3D space so as to obtain the necessary visual information from fewer yet more informative images to effectively reconstruct or render environments of interest. In the first chapter, we introduce the problems of view planning (VP) and scene rendering, and the open challenges that this thesis addresses. The second chapter aims to alleviate the reliance on a pre-defined mesh model of the environment, which is a common assumption in existing model-based VP approaches. To that end, this chapter presents an efficient and realistic VP pipeline, which aims to optimize the viewpoints of cameras and hence the quality of the 3D reconstruction of a field of row crops without need for a given mesh model. The goal of the third chapter is to cope with the challenge of existing environmental noise that is not explicitly accounted for in existing VP approaches. To that end, a novel geometric-based reconstruction quality function is introduced for VP, that accounts for the existing noise of the environment, without requiring its closed-form expression. With no analytic expression of the objective function, this chapter presents an adaptive Bayesian optimization framework for accurate 3D reconstruction in the presence of environmental noise. The fourth chapter aims (i) to overcome the errors induced by certain geometric proxies in geometric-based VP approaches arising from e.g. non-informative and insufficient geometric cues, and (ii) to identify optimal VP solutions with adaptivity to similar yet unknown environments without need for re-training/re-optimization. This chapter presents an alternative VP framework that considers a reconstruction quality-based optimization formulation. The fifth chapter focuses on the 3D scene rendering task, relying on the so-termed Gaussian splatting (GS) models that provide outstanding visual quality along with computational efficiency. It proposes an active image selection framework to assist GS models aiming to address the limitations of existing passive GS approaches. Specifically, these approaches rely on either (i) densely collected views that introduce redundancy and increase processing and computational costs, or (ii) sparse-view settings that may result in reduced scene coverage and limited expressiveness of the 3D scene representation.