Tactile sensing for Robot Learning: Development to Deployment

 The role of tactile sensing is widely acknowledged for robots interacting with the  physical environment. However, few contemporary sensors have gained widespread  use among roboticists. This thesis proposes a framework for incorporating tactile  sensing into a robot learning paradigm, from development to deployment, through the  lens of ReSkin– a versatile and scalable magnetic tactile sensor. By examining design,  integration, policy learning and representation learning in the context of ReSkin, this  thesis aims to provide guidance on the implementation of effective sensing systems  for robot learning.  

We begin by proposing ReSkin– a low-cost, compact, and diverse platform  for tactile sensing. We develop a self-supervised learning technique that enables  sensor replaceability by adapting learned models to generalize to new instances of  the sensor. Next, we investigate the scalability of ReSkin in the context of dexterous  manipulation: we introduce the D’Manus, an inexpensive, modular, and robust  platform with integrated large-area ReSkin sensing, aimed at satisfying the large scale data collection demands of robot learning.  

Based on the learnings from the development of ReSkin and the D’Manus, we  propose AnySkin– an upgraded sensor tailored for robot learning that further reduces  variability in response across sensor instances. AnySkin is as easy to integrate as  putting on a phone case, eliminates the need for adhesion and demonstrates enhanced  signal consistency. We deploy AnySkin in a policy learning setting for precise  manipulation, demonstrate improved task performance when augmenting camera  information, and exhibit zero-shot transfer of learned policies across sensor instances.  

Going beyond sensor design and deployment, we explore representation learning  for sensors including but not limited to ReSkin. Sensory data is typically sequential  and continuous; however, most research on existing sequential architectures like  LSTMs and Transformers focuses primarily on discrete modalities such as text and  DNA. To address this gap, we propose Hierarchical State Space (HiSS) models,  a conceptually simple and novel technique for continuous sequence-to-sequence  prediction (CSP). HiSS creates a temporal hierarchy by stacking structured state space models on top of each other, and outperforms state-of-the-art sequence models  such as causal Transformers, LSTMs, S4, and Mamba. Further, we introduce CSP Bench, a new benchmark for CSP tasks from real-world sensory data. CSP-Bench  aims to address the lack of real-world datasets available for CSP tasks, providing a  valuable resource for researchers working in this area.  

Finally, we conclude by summarizing our takeaways throughout the journey  of ReSkin from development to deployment, and outline promising directions for  bringing tactile sensing into the fold of mainstream robotics research.