Learning and Inference in Factor Graphs with Applications to Tactile Perception

Factor graphs offer a flexible and powerful framework for solving largescale, nonlinear inference problems encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, highdimensional sensor observations. For instance, consider a robot manipulating an object in hand and receiving high-dimensional tactile observations from which it must infer latent object poses. How do we couple machine learning to extract salient information from observations and graph optimization to efficiently fuse such information? This thesis addresses three principal challenges: (1) How do we learn observation models from data with optimizers in the loop? We show that learning observation models can be viewed as shaping energy functions that graph optimizers, even non-differentiable ones, optimize. (2) How do we impose hard constraints in graph optimizers derived from real-world physics or geometry? We expand incremental Gauss-Newton solvers into a broader primal-dual framework to efficiently solve for constraints in an online manner. (3) Finally, we look at different learned feature representations that extract salient information from tactile image observations.

We evaluate these approaches for a real-world application of tactile perception for robot manipulation, where we demonstrate reliable object tracking in hundreds of trials across planar pushing and in-hand manipulation tasks. This thesis establishes novel connections between factor graph inference, energybased learning, and constrained optimization, opening avenues for new research problems at the intersection of these topics.