Novel greedy deep learning algorithms

Abstract

The new algorithms are designed to simulate human’s learning process, where features for different objects are identified, understood and memorized iteratively. The main difference between the two new algorithms is the partition function used to split the input data into subsets. A certain feature will be learned from one subset, instead of from the whole data set. The Greedy-By-Node (GN) algorithm is based on an additive-feature assumption which to some extent resembles the boosting algorithms, where the input data is sorted and partitioned based on their distance to the most common feature learned by the first inner node. The subsets closer to the common feature will be learned earlier, while harder problems are intrinsically covered by more inner nodes and learned at later stage. The Greedy-By-Class-By-Node (GCN) algorithm directly utilizes the data labels and assumes that data in each class share common features. A special cache mechanism and a parameter called ”amnesia factor” are also introduced in order to keep the speed while provide control over the ”orthogonality” between learned features. Our algorithms are orders of magnitude faster in training, create more interpretable internal representations at the node level, while not sacrificing on the ultimate out-of-sample performance.

Multilayer neural networks have seen a resurgence under the umbrella of deep learning. Current deep learning algorithms train the layers of the network sequentially, improving the algorithmic performance as well as providing some regularization. While the current algorithms have shown great prediction power in many problems, disadvantages in model interpretability and training time still exist and may not be easily overcome under the current framework. In order to solve the problems, we developed two new training algorithms for deep networks which train each node in the network sequentially.