Knowledge-augmented deep learning and its applications

Abstract

Deep learning models have achieved remarkable success in many different fields over the past years thanks to advanced algorithmic techniques; great computational power provided by processors; and, most importantly, tremendous amounts of data. Though designed to mimic the behavior of human brains, existing deep models are still far from matching human learning abilities. Particularly, existing deep models are usually data hungry, fail to perform well on unseen samples, and lack of interpretability. In contrast, human beings can learn from limited observations, generalize well to novel settings, and explain well their predictions, due to their ability to extract, understand, and make use of domain knowledge. To mimic this ability, this thesis aims to identify domain knowledge and encode and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning, which we refer to as \textit{knowledge-augmented deep learning}. Existing knowledge-augmented deep learning techniques face two main challenges: diverse knowledge representation formats and imperfect knowledge. Knowledge from different domains can be represented in different formats, including probabilistic relationships, symbolic rules, or mathematical equations. Domain knowledge is usually imperfect because it can be incomplete, fragmented, and ambiguous. Knowledge imperfection leads to uncertainty during inference. To address the first challenge, we propose different knowledge encoding and integration schemes to ensure that domain knowledge is efficiently and accurately encoded, and effectively integrated with data. To address imperfect knowledge, we propose to employ probabilistic models for compact and systematic encoding of the uncertainties. To evaluate the proposed knowledge encoding and integration methods, we consider four use cases. In use case 1, we show how to use a Bayesian network to encode the facial anatomy knowledge on probabilistic relationships among facial muscles and integrate it with data for facial action unit detection. In use case 2, we demonstrate how facial mechanics knowledge represented as ordinary differential equations is integrated into an encoder-decoder framework for facial action unit detection. In use case 3, we demonstrate that a prior probability as a prior model is used to encode ontological knowledge represented by symbolic rules and combined with a deep learning method for a knowledge graph completion task. Finally, in use case 4, we demonstrate how algorithmic knowledge about variational belief propagation is encoded into a message passing neural network through a custom loss function for probabilistic inference tasks on probabilistic graphical models.