External Knowledge Augmented Language Models for Code Generation and Agents

We interact with computers everyday. Although the user experience of operating a computer has never been easier nowadays, there are still hurdles that prevent users from making use of their full potential. One such example is the learning curve of programming languages. Programmers have always dreamt of easier and more intelligent tools to assist them with their work, and thus make developing more effortless. End users that do not know how to program would also benefit from a more natural way of instructing computers to accomplish certain customized tasks, i.e., transforming computers into their personal agents that complete various tasks with high level of autonomy and instruction-following ability. This sharply contrasts to how effortlessly we communicate our goals and desires in natural language such as English.

In this thesis, we propose to bridge natural language with programming language and executable actions in daily life online environments. The anticipated outcomes of this work aim to construct models, environments and evaluations for both code generation tasks for programming jobs and large language model (LLM) agents for online daily life and workplace, with a special focus on utilizing large data sources and external knowledge for model pre-training and retrieval-augmented models. We examine the problem from four perspectives that constitute the four parts of the dissertation. In the first part we explore pre-training for code generation models. Next, in the second part we perform human study of code generation. Then, in the third part we improve retrieval-augmented models. Finally, In the fourth part we explore interactive use of LLMs as agents.