Learning Task-Oriented Dialog with Neural Network Methods

Dialog system is class of intelligent system that interacts with human via natural language
interfaces with a coherent structure. Based on the nature of the conversation, dialog systems
are generally divided into two sub-classes, task-oriented dialog systems that are created
to solve specific problems, and chit-chat systems that are designed for casual chat and
entertainment. This thesis focuses on task-oriented dialog systems.
Conventional systems for task-oriented dialog are highly handcrafted, usually built with
complex logic and rules. These systems typically consist of a pipeline of separately developed
components for spoken language understanding, dialog state tracking, dialog policy,
and response generation. Despite the recent progress in spoken language processing and dialog
learning, there are still a variety of major challenges with current systems. Firstly, the
handcrafted modules designed with domain specific rules inherently make it hard to extend
an existing system to new domains. Moreover, modules in current system are interdependent
in the processing pipeline. Updating an upper-stream module may change its output
distribution which can make other down-stream modules sub-optimal. Last but not least,
current systems are mostly configured and trained offline. They lack the flexibility to learn
continuously via interaction with users.
In this thesis, we address the limitations of the conventional systems and propose a datadriven
dialog learning framework. We design a neural network based dialog system that
can robustly track dialog state, interface with knowledge bases, and incorporate structured query results into system responses to successfully complete task-oriented dialogs. The system
can be optimized end-to-end with error signals backpropagating from system output to
raw natural language system input. In learning such system, we propose imitation and reinforcement
learning based methods for hybrid offline training and online interactive learning
with human-in-the-loop. The system is enabled to continuously improve itself through the
interaction with users. In addition, we address several practical concerns with interactive
dialog learning. In addressing the impact of inconsistent user ratings (i.e. the rewards) for
dialog policy optimization, we propose an adversarial learning method which can be used
to effectively estimate the reward for a dialog. In addressing the sample efficiency issue
in online interactive learning with users, we propose a method by integrating the learning
experience from real and imagined interactions to improve the dialog learning efficiency. We
perform the system evaluation in both simulated environments and real user evaluation settings.
Empirical results show that our proposed system can robustly track dialog state over
multiple dialog turns and produce reasonable system responses. The proposed interactive
learning methods also lead to promising improvement on task success rate and human user
ratings.