Hybrid Knowledge Architectures for Question Answering

Question answering (QA) is a knowledge-intensive task in natural language processing (NLP) that requires the system to provide answers to user queries expressed in natural language. The types of knowledge a QA system is equipped with broadly fall into two categories, namely explicit knowledge and implicit knowledge. The explicit knowledge takes formats that are human readable, e.g. raw text, knowledge graphs, structured tables etc, while the implicit knowledge resides in the model parameters that are learned by training on the explicit knowledge. Due to the complementary nature of these two types of knowledge, most recent QA research has tried to leverage both of them for modeling. However, existing work on this front mostly focuses on building customized models for specific datasets, which do not generalize well to other use cases. Moreover, while using end-to-end models to directly learn to fuse different knowledge is a simple solution and often works well, it’s hard to interpret the model’s reasoning process, leading to untrustworthy predictions. Finally, most systems are designed without considering memory and computation efficiency, which hinders’ their application to real-world use cases. 

With the aforementioned issues in mind, in this thesis, we present solutions for building generalizable, interpretable, and efficient QA systems. Specifically, we present three solution elements, namely 1) hybrid knowledge fusion, 2) modularized knowledge framework, and 3) modularized knowledge sharing. In the first part, we study various ways of injecting commonsense knowledge into QA systems powered by pretrained language models. Our results show that instance-level late fusion of knowledge subgraphs is promising in a supervised setting and pretraining on transformed knowledge graphs (KGs) provides substantial gains across a diverse set of tasks in a zero-shot setup. These findings show that combining explicit and implicit knowledge is a step towards generalization across different domains of questions. In the second part, we explored two different modularized frameworks for open-domain question answering that bridge the gap across knowledge types and question types. We show that text can serve as a universal knowledge interface for different types of structured knowledge, and decomposing the reasoning process into discrete steps enables a single unified system to solve both single-hop and multi-hop questions. Modularized frameworks not only offer generalization across modalities of knowledge and question types but also bring improved interpretability of the reasoning process. In the third part, we extend the modularized framework from the previous part by allowing implicit knowledge sharing among different modules. Multiple reasoning modules are merged together and learned simultaneously through multi-task learning, and we further add skill-specific specialization for each module to reduce task interference. Such an architecture not only greatly reduced the overall model size but also improved the inference efficiency, therefore achieving all three target properties generalizability, interpretability, and efficiency. Finally, we discuss open challenges and ways forward beyond this thesis.