Symbolic information in recurrent neural networks: Issues of representation and training

Abstract

In recent years, there has been a renewed interest in artificial neural networks. The discovery of new learning algorithms has made them promising tools in diverse applications such as pattern recognition, signal processing, knowledge acquisition for expert systems, prediction of protein structures, and dynamical system modeling. Recurrent neural networks which are able to store state information over indefinite time spans are particularly well-suited for modeling dynamical systems such a stock markets, speech, and physical systems.

In this thesis, we examine how symbolic knowledge is represented in recurrent networks trained to recognize regular languages. We demonstrate methods for inserting and extracting the learned languages in the form of deterministic finite-state automata (DFAs). We focus our attention on the quality of the extracted grammatical rules and on how partial prior knowledge can be effectively utilized to improve training performance. We prove that DFAs can be naturally embedded in a special recurrent network architecture with higher order weights such that the finite-state encoding remains stable indefinitely. We investigate means for constructing neural DFAs which are tolerant to faults in their internal structure.