Towards Data-Efficient Machine Learning

Deep learning works well when the problem is regular enough and there is abundant training data to adequately and in a representative way reflect all the regularity. As the ambition of researchers grows, problems with less regularity are being addressed, where more data is needed to achieve great performance. In addition, as researchers push the boundary of deep learning, state-of-the-art models become more and more data-hungry due to the growing capacity. Hence, data annotation is necessary to train deep learning models to perform well. However, data annotation is a costly process that requires a significant amount of work for each new task of interest. 

To tackle this difficulty, we present algorithms that can leverage other kinds of information to achieve a better performance given a certain amount of data. In this thesis, we show how to leverage several kinds of information including: (1) unlabeled data; (2) data from another domain; (3) prior knowledge. First, when unlabeled data of the domain of interest is available, semi-supervised learning can effectively improve the performance of deep learning models by regularizing the models to make consistent predictions for similar examples; Second, when data from another domain is available, transfer learning or domain adaptation can be applied to transfer general knowledge or task-specific knowledge learned from another domain to the domain of interest; Last, with prior knowledge, we can inject targeted inductive biases into the models and make use of external knowledge bases.

With three possible directions, one might wonder what direction should be taken given a new task. To offer practical suggestions to researchers and practitioners, we analyze the effectiveness, the applicability and the engineering difficulty of each algorithm. Specifically, we present the performance of different algorithms on different problems and study whether different algorithms can be combined together for improved performance, analyze whether an algorithm can be applied to a broad range of tasks or is restricted to certain tasks and discuss the required engineering efforts for each algorithm.