Associative learning for text and graph data
Loading...
Authors
Liang, Yuchen
Issue Date
2023-05
Type
Electronic thesis
Thesis
Thesis
Language
en_US
Keywords
Computer science
Alternative Title
Abstract
This dissertation focuses on the application of associative learning into text data and graphstructured data. Associative learning is the process of learning to associate two stimuli.
If the connection between two events are repeatedly strengthened, or some similar events
happen over and over again, our brain memorizes those patterns which are frequently shown
together. When we encounter a similar pattern, or one of the event pairs we have in our
memory, then we can retrieve the relevant associated pattern. This kind of phenomenon
is extensively studied in biology, and we want to apply this idea to develop biologically
plausible neural networks.
We identified three main challenges of associative learning. The first challenge is that
most of the early associative learning methods use local learning rules to update the weights.
Though the local leaning rule is efficient and biologically plausible, it may not be able
to extract features that are good enough for representation learning. On the other hand,
backpropagation can guide the network’s weights towards the loss we select, which can be
more helpful for the given task. How to train the weights effectively in the associative
memory network is still an open question. The second challenge is that some recent work
has shown the connection between Hopfield networks and attention module that is widely
used in modern deep neural network architectures. How to make the improvement of existing
attention module from the perspective of associative memory is worth exploring. And also
how to integrate the associative memory into modern deep neural network architectures is
an interesting task. The third challenge is to apply associative learning to different types of
data (e.g., text, graph, tables and so on). The network has to learn to extract the feature,
and learn to distinguish and associate between different features. The network also has to
let the memories remember diverse patterns to increase the model capacity given the limited
resources.
In this dissertation, we propose solutions to address the challenges mentioned above.
Specifically, to address the first challenge, we explore different ways to train the associative
memory networks. In our text application, we propose a way to learn the memory of the
model using a local update rule, and in our graph application, we focus on training the whole
network using backpropagation. To address the second challenge, we propose two different
ways to apply Modern Hopfield Networks in graph applications (for structure graph and
featured graph, respectively). To address the third challenge, we propose novel approaches
for associative learning in both text data and graph data. Our associative learning based
model is competitive with existing deep learning architectures, and can also help us interpret
the network from the angle of information retrieval and completion.
Description
May2023
School of Science
School of Science
Full Citation
Publisher
Rensselaer Polytechnic Institute, Troy, NY