Learning with Structured Priors for Robust Robot Manipulation

Robust and generalizable robots that can autonomously manipulate objects in semi-structured environments can bring material benefits to society. Datadriven learning approaches are crucial for enabling such systems by identifying and exploiting patterns in semi-structured environments, allowing robots to adapt to novel scenarios with minimal human supervision. However, despite significant prior work in learning for robot manipulation, large gaps remain before robots can be widely deployed in the real world. This thesis addresses three particular challenges to advance toward this goal: sensing in semi-structured environments, adapting manipulation to novel scenarios, and flexible planning for diverse skills and tasks. A common theme among the discussed approaches is enabling efficient and generalizable learning by incorporating “structures," or priors specific to robot manipulation, into the design and implementation of learning algorithms. The thesis works follow the three challenges above. 

We first leverage contact-based sensing in scenarios that are difficult for visionbased perception. In one work, we use contact feedback to track in-hand object poses during dexterous manipulation. In another, we learn to localize contacts on the surface of robot arms to enable whole-arm sensing. 

Next, we explore adapting manipulation to novel objects and environments for both model-based and model-free skills. We show how learning task-oriented interactive perception can improve the performance of downstream modelbased skills by identifying relevant dynamics parameters. We also show how using object-centric action spaces can make deep reinforcement learning of model-free skills more efficient and generalizable. 

Lastly, we explore flexible planning methods to leverage low-level skills for more complex manipulation tasks. We develop a search-based task planner that relaxes assumptions on skill and task representations from prior works by learning skill-level dynamics models. This planner is then applied in a followup work that uses learned preconditions of Hybrid Force-Velocity Controllers to perform multi-step contact-rich manipulation tasks. We also explore planning for more flexible tasks described by natural language by using code as the structured action space. This is done by prompting Large Language Models to directly map natural language task instructions to robot policy code, which orchestrates existing robot perception and skill libraries to complete the task.