Carnegie Mellon University
Browse

Learning and Translating Temporal Abstractions of Behaviour across Humans and Robots

Download (26.9 MB)
thesis
posted on 2024-10-25, 19:22 authored by Tanmay ShankarTanmay Shankar

Humans are remarkably adept at learning to perform tasks by imitating other people demonstrating these tasks. Key to this is our ability to reason abstractly about the high-level strategy of the task at hand (such as the recipe of cooking a dish) and the behaviours needed to solve this task (such as the behaviour of pouring liquid into a pan), while ignoring irrelevant details (such as the precise angle at which to pour).

In this thesis, we describe steps towards imbibing robots with these abilities;i.e. to learn and translate temporal abstractions of behaviour across humans and robots. In Part II, we first explore the question “How can we learn and represent temporal abstractions of agent behaviours and their effects on their environment?”, We present two methods to do so in Chapter 2 and Chapter 3, adopting an unsupervised representation learning perspective to skill learning.

In Part III, we then address the question “How can we understand demonstrator task strategies in terms of these abstractions, and translate these to corresponding abstractions for a robot to execute?”. Specifically, In Chapter 4, we explore how agents can learn correspondences between their own skills, and those of a demonstrator, or how we can translate skills from demonstrators to agents, inspired by unsupervised machine translation in natural language.

In Part IV, we begin to consider the effects of agent behaviors on their environments. In Chapter 5, we consider how agents can jointly reason about the skills they execute and the effects these skills have on the objects and environments that these agents interact with. We do this by extending our prior skill learning work to learn temporally abstract representations of agent-environment interactions.

In Part V, we revisit these questions; addressing translating temporal abstractions of behaviour from humans to robots from a perspective of imitating desired environmental change. In Chapter 6, we explore how we can translate environment-aware task strategies across humans and robots. We introduce TransAct, a framework to address this. TransAct empowers robots to consume in-domain human task demonstrations, retrieve and compose corresponding robot-environment interactions with similar environmental effects to perform similar tasks themselves in a zero shot manner, without access to paired demonstrations or dense annotations.

In Part VI, we explore other application domains that our work enables. In particular, in Chapter 7, we explore the domain of encouraging older adults to exercise more with an interactive robot artist. We employ the idea of mapping abstract representations from human exercise routines to robot paint strokes.

In Part VII, we offer a broader perspective on the paradigm of unsupervised translation of temporal abstractions of behaviour across humans and robots. We finally discuss future research directions enabled by our work.

History

Date

2024-08-29

Degree Type

  • Dissertation

Department

  • Robotics Institute

Degree Name

  • Doctor of Philosophy (PhD)

Advisor(s)

Jean Oh