Online Active Learning with Corrective Feedback: A New Paradigm Applied to Streaming Audio

 Machine Learning (ML) is a powerful tool that can assist humans in doing a wide variety of tasks. However, typical ML paradigms tend to remove the human element from the equation as frequently as possible in the name of automation. This human-independent approach is referred to generally as Passive Machine Learning (PML). PML approaches may be useful for creating powerful solutions to a selection of pre-defined tasks, but it has led to a dearth of methods for quickly training a machine to assist a human domain expert in a task previously undefined for ML. This problem is compounded in cases where data for the intended task are scarce or proprietary, since typical ML paradigms rely on large amounts of training data. To address this and other related issues with typical PML paradigms, an Online Active Learning with Corrective Feedback (OAL-CF) paradigm, which is a special case of Online Active Learning, is introduced in this thesis. In OAL-CF, a machine classifier is trained to perform a new task alongside a human domain expert as part of his or her operational workflow.

In this thesis, the OAL-CF paradigm is defined—including its framework and methods for evaluation—and distinguished from other existing paradigms. The efficacy of OAL-CF is demonstrated experimentally by applying it to a variety of audio-based detection and verification tasks. To measure the improvement of OAL-CF over PML and other existing online methods, a new evaluation metric based on both annotation cost and prediction error is introduced. Additionally, components of the framework that could be improved are highlighted, and methods to implement the improvements are presented. These methods include strategies for obtaining more informative feedback from the expert, loss functions based on detection cost to handle extreme class imbalance, and dynamic allocation of the machine's queries to the domain expert.

Considering the operational suitability and the experimental results of OAL-CF presented in this thesis, it is clear that this paradigm has significant advantages over PML training in certain contexts. Models trained with OAL-CF can achieve dramatically lower error rates using only a small fraction of the labeled data required by PML training.