Integrating Multiple Knowledge Sources for Utterance-Level Confidence Annotation in the CMU Communicator Spoken Dialog System

In the recent years, automated speech recognition has been the main drive behind the advent of spoken language interfaces, but at the same time a severe limiting factor in the development of these systems. We believe that increased robustness in the face of recognition errors can be achieved by making the systems aware of their own misunderstandings, and employing appropriate recovery techniques when breakdowns in interaction occur. In
this paper we address the rst problem: the development of an utterance-level condence annotator for a spoken dialog system. After a brief introduction to the CMU Communicator spoken dialog system (which provided the target platform for the developed annotator), we cast the condence annotation problem as a machine learning classication task, and focus on selecting relevant features and on empirically identifying the best classication techniques for this task. The results indicate that signicant reductions in classication error rate can be obtained using several dierent classiers. Furthermore, we propose a data driven approach to assessing the impact of the errors committed by the condence annotator on dialog performance, with a view to optimally ne-tuning the annotator. Several models were constructed, and the resulting error costs were in accordance with our intuition. We found, surprisingly, that, at least for a mixed-initiative spoken dialog system as
the CMU Communicator, these errors trade-o equally over a wide operating characteristic range.