file.pdf (49.77 kB)
Download fileModeling the Cost of Misunderstanding Errors in the CMU Communicator Dialog System
journal contribution
posted on 2007-08-30, 00:00 authored by Dan Bohus, Alexander RudnickyAlexander RudnickyWe describe a data-driven approach that allows us to quantify the costs of various types of errors made by the utterance-level confidence annotator in the Carnegie Mellon Communicator system. Knowing these costs we can determine the optimal tradeoff point between these errors, and tune the confidence annotator accordingly. We describe several models, based on concept transmission efficiency. The models fit our data quite well and the relative costs of errors are in accordance with our intuition. We also find,
surprisingly, that for a mixed-initiative system such as the CMU Communicator, false positive and false negative errors trade-off equally over a wide operating range.