Robot Learning for 3D Deformable Object Manipulation

<p dir="ltr">Much of the world we interact with as humans are deformable, yet current robotic systems often struggle to successfully manipulate deformable objects. If we hope to develop robust and generalizable robotics systems for manufacturing, food preparation, assistive robotics and surgery, among others, it is critical that we develop systems that are able to predict the behavior of deformable objects and generate desirable action plans that are informed by the deformation behavior. In this dissertation, we aim to explore the challenges of deformable object manipulation with the task of creating 3D simple sculptures in an unstructured 3D deformable object, what we refer to as the robotic shaping task. This task requires consideration of key challenges within deformable object manipulation - state representation, partial observability and occlusions, complex deformations due to robot interaction, and long-horizon action sequences to reach the desired goal shape. </p><p dir="ltr">This dissertation explores a variety of learning-based methods for the robotic clay shaping task to explore different facets of the challenge of deformable object manipulation. We introduce SculptBot, a learned dynamics model that leverages a pre-trained point cloud embedding for more efficient dynamics predictions. To generate sculpting action sequences, we employed sample-based model predictive control with the learned dynamics model. To follow this work, we present SculptDiff, a point cloud-based v imitation learning method that directly generates sculpting actions from observation. We further present LLM-Craft to explore how to incorporate the relevant world knowledge of LLMs to directly generate robotic sculpting trajectories. Next, we present a text-to-3D sculpting system [3] that leverages LLMs and a low-level action model to create simple shapes with a hierarchical framework. Finally, we develop PinchBot, an imitation learning model with task-progress and sub-goal guidance for the highly long horizon task of robotic pinch pottery. The aim of this dissertation is to develop a robotic sculpting framework and demonstrate the importance and effectiveness of learning-based methods for the challenging, long horizon and often multi-modal task of 3D clay sculpting.</p>