Learning from Pixels: Image-Centric State Representation Reinforcement Learning for Goal Conditioned Surgical Task Automation

Abstract

Over the past few years, significant exploration has occurred in the field of automating surgical tasks through off-policy Reinforcement Learning (RL) methods. These methods have witnessed notable advancements in enhancing sample efficiency (such as with the use of Hindsight Experience Replay - HER) and addressing the challenge of exploration (as seen in Imitation Learning approaches). While these advancements have boosted RL model performance, they all share a common reliance on accurate ground truth state observations. This reliance poses a substantial hurdle, particularly in real-world scenarios where capturing an accurate state representation becomes notably challenging.This study addresses the aforementioned challenge by exploiting an Asymmetric Actor-Critic framework while addressing the issues of sample efficiency and exploration burden by using HER and behavior cloning. Within this framework, the Critic component is trained on the complete state information, whereas the Actor component is trained on partial state observations, thus diminishing the necessity for pre-trained state representation models. The proposed methodology is evaluated within the context of SurRoL, a surgical task simulation platform. The experimental results showcased that the RL model, operating with this configuration, achieves task performance akin to models trained with complete ground truth state representations. Additionally, we delve into the necessity for Sim-to-Real transfer methods and elucidate some of the formidable challenges inherent in this process and present a comprehensive pipeline that addresses the intricacies of domain adaptation. This research thus presents a promising avenue to mitigate the reliance on pre-trained models for state representation in the pursuit of effective surgical task automation.