Efficient Robotic Manipulation with Scene Knowledge

Abstract

In recent years, robots have transformed manufacturing, logistics, and transportation. However, extending the success to unstructured real-world environments (e.g., domestic kitchens, warehouses, grocery stores, etc.) remains difficult due to three key challenges: (1) assumption of structured environments (such as organized bottles in the factories); (2) hand-engineered solutions that are difficult to generalize to novel scenarios; (3) limited flexibility of action primitives, which prevents the robot from reaching target objects. In this thesis, we address these challenges by learning scene knowledge that improves the efficiency of robotic manipulation systems. Grasping is a fundamental manipulation skill that is constrained by the scene arrangement (i.e., the locations of the robot, the objects, and the environmental structures). Understanding scene knowledge, such as the robot's reachability to objects, is crucial to improve the robot's capability. We developed a reachability-aware grasp pose generator that predicts feasible 6-degree-of-freedom (6-DoF) grasp poses (i.e., approaching with an arbitrary direction and wrist orientation). Then, we extended to target-driven grasping in constrained environments and added collision awareness to our scene knowledge. When objects are densely cluttered, we improved the robot's efficiency by employing graph neural networks (GNN) to exploit the underlying relationships in the scene. To accomplish complex manipulation tasks in constrained environments, such as rearranging adversarial objects, we hierarchically integrated a heterogeneous graph neural network (HetGNN)-based coordinator and the 3D CNN-based actors. The system reasons about the relational knowledge between scene components and coordinates multiple robotic skills (e.g., grasping, pushing) to minimize the planning cost. As we anticipate an increase in the number of domestic robots, the robotics community necessitates a framework that not only commands the robot accurately, but also reasons about the unstructured scene to improve robots' efficiency. This thesis contributes to the goal by equipping robotics manipulation with learned scene knowledge. We present 6-DoF robotic systems that can grasp novel objects in dense clutter with reachability awareness, retrieve target objects within arbitrary structures, and rearrange multiple objects into goal configurations in constrained environments.