Associative learning for text and graph data

Abstract

This dissertation focuses on the application of associative learning into text data and graphstructured data. Associative learning is the process of learning to associate two stimuli. If the connection between two events are repeatedly strengthened, or some similar events happen over and over again, our brain memorizes those patterns which are frequently shown together. When we encounter a similar pattern, or one of the event pairs we have in our memory, then we can retrieve the relevant associated pattern. This kind of phenomenon is extensively studied in biology, and we want to apply this idea to develop biologically plausible neural networks. We identified three main challenges of associative learning. The first challenge is that most of the early associative learning methods use local learning rules to update the weights. Though the local leaning rule is efficient and biologically plausible, it may not be able to extract features that are good enough for representation learning. On the other hand, backpropagation can guide the network’s weights towards the loss we select, which can be more helpful for the given task. How to train the weights effectively in the associative memory network is still an open question. The second challenge is that some recent work has shown the connection between Hopfield networks and attention module that is widely used in modern deep neural network architectures. How to make the improvement of existing attention module from the perspective of associative memory is worth exploring. And also how to integrate the associative memory into modern deep neural network architectures is an interesting task. The third challenge is to apply associative learning to different types of data (e.g., text, graph, tables and so on). The network has to learn to extract the feature, and learn to distinguish and associate between different features. The network also has to let the memories remember diverse patterns to increase the model capacity given the limited resources. In this dissertation, we propose solutions to address the challenges mentioned above. Specifically, to address the first challenge, we explore different ways to train the associative memory networks. In our text application, we propose a way to learn the memory of the model using a local update rule, and in our graph application, we focus on training the whole network using backpropagation. To address the second challenge, we propose two different ways to apply Modern Hopfield Networks in graph applications (for structure graph and featured graph, respectively). To address the third challenge, we propose novel approaches for associative learning in both text data and graph data. Our associative learning based model is competitive with existing deep learning architectures, and can also help us interpret the network from the angle of information retrieval and completion.