Towards Efficient and Reproducible Natural Language Processing

<p dir="ltr">Machine learning has reported remarkable progress on a broad range of tasks, including machine translation, object recognition, and game playing (Shoham et al., 2018). Much of this progress has been achieved by increasingly large and computationally-intensive deep learning models. Figure 1.1, reproduced from Amodei and Hernandez (2018), plots training cost increase over time for state-of-the-art deep learning models starting with AlexNet in 2012 (Krizhevsky et al., 2012) to AlphaZero in 2017 (Silver et al., 2017a). The chart shows an overall increase of 300,000x, with training cost doubling every few months. An even sharper trend can be observed in NLP word embedding approaches by looking at ELMo (Peters et al., 2018) followed by BERT (Devlin et al., 2019), openGPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), Megatron-LM (Shoeybi et al., 2019), and T5 (Raffel et al., 2019). </p><p dir="ltr">This trend is driven by the strong focus of the AI community on obtaining state-of-the art results, as exemplified by the popularity of leaderboards (Wang et al., 2019b,a) which typically report performance1 but omit any mention of cost or efficiency. Despite the clear benefits of improving model performance, the focus on one single metric ignores the economic, environmental, and social cost of reaching the reported results. </p><p dir="ltr">This increase in computational expense has led to some types of research being prohibitively expensive, raising barriers to participation. In addition, recent research estimates a significant carbon footprint for NLP experiments (Strubell et al., 2019). This thesis advocates for increasing research activity in Green AI--AI research that is more efficient, inclusive, and environmentally friendly. We emphasize that Red AI has been yielding valuable scientific contributions, but it has been overly dominant in driving the direction of research in the eld; we want to shift the balance towards the Green AI option. Specifically, the work in this thesis makes efficiency a evaluation criterion alongside accuracy and related measures.</p>