Adaptive Proactive Learning with Cost-Reliability Tradeoff

Proactive Learning is a generalized form of active learning where the learner must reach out to multiple oracles exhibiting different costs and reliabilities (label noise). One of the its major goals is to capture the cost-noise tradeoff in oracle selection. Sequential active learning exhibits coarse accuracy at the beginning and progressively refine prediction at later stages. The ability to learn oracle accuracies over time and select better oracles or oracle ensembles lead to potentially faster error reduction rate as a function of total cost, and thus improve its cost complexity. To realize this potential, we propose a statistical model that adapts to a range of accuracies at different stages of active learning. In a more general scenario, we formulate the problem as maximum submodular coverage subject to a budget envelope.