Learning In The Wild With Limited Supervision

Over the past decade, machine visual perception has experienced remarkable progress due to advancements in the field of deep learning. However, the performance of deep learning systems remain far from ideal in real-world tasks that lack large training datasets. In this thesis, we study learning under limited supervision with a focus on unsupervised domain adaptation (no labelled examples) and few-shot learning (few labelled examples). We propose adaptation schemes that can leverage prior knowledge from a large-labelled base domain and transfer it to the domain of limited supervision (target domain). We con-sider both image classification and semantic segmentation tasks in the limited supervision regime. The key findings in this thesis are as follows.

In unsupervised domain adaptation, object-level adaptation is more effective than pixel-level adaptation for semantic segmentation. We propose a multi-modal objectness constraint that improves self-training based approaches for this problem. In the few-shot learning setup, full model finetuning is crucial for effective transfer when the base domain lacks sufficient diversity. We propose contrastive finetuning approach that leverages negative exemplars (distractors) to alleviate the issue of data scarcity in few-shot learning. As the size and diversity of base domain scales up, parameter efficient techniques can out-perform full-finetuning on variety of tasks including image classification and semantic segmentation. To that end, we propose expres, that augments a frozen base model with a few learnable parameters in the form of input and residual prompts and optimizes them for the few-shot task. While base domain scaling improves few-shot performance, using the right pretraining objective is equally important. We show that, compared to supervised representations, self-supervised representations are more suitable for few-shot semantic segmentation and a combination of the two achieves the best of both worlds.

Through our research work, we expand the palette of adaptation techniques suitable for different scales of base domain and degrees of target domain super-vision. We show that a careful design of the adaptation method can strike a much better trade-off between performance and forgetting. On popular bench-marks for few-shot image classification and semantic segmentation, our pro-posed approaches lead to significant performance gains reducing the gap with fully supervised methods.