Towards Reality: Egocentric Environment Understanding for Mobile Devices

Abstract

Mixed reality (MR) allows the users to overlay the virtual contents onto the physical world. Given an image taken by a mounted camera on the MR headset, two fundamental questions must be solved: (1) Where am I? and (2) Where is what?. The first question concerns estimating the 6 degree-of-freedom of the camera and the second question concerns predicting the scene layout, contents and their geometries (depth, surface normal, shape) from the captured egocentric image. Even though these problems have been studied in robotics and computer vision, deploying the existing methods to the mobile MR headsets faces two challenges: (1) high computational cost and power consumption for tracking and localization algorithms, and (2) degeneration of existing scene understanding models due to the domain gap caused by the users' large head that motions capture egocentric views. This dissertation seeks to address these two major challenges to enable MR into our daily usage. Specifically, it explores two methods to obtain compact map representations that are suitable for camera localization (collection of scene landmarks) and navigation (topological map). Furthermore, it proposes a multimodal spatial rectifiers that allows the scene geometry (depth, surface normal) predictor module to focus on only a few modes of the head orientation, significantly reduces the requirement for high capacity model and large scale 3D-annotated egocentric data. The proposed methods are validated on exisiting datasets, as well as newly introduced challenging egocentric datasets. Finally, a large scale egocentric dataset with 3D objects is introduced to study the object's 3D shape and pose estimation benchmark tasks.