Design of experiments for reinforcement learning

Abstract

Reinforcement learning is one of the fundamental learning techniques that humans use to learn how to interact with and make decisions within challenging or uncertain environments. In essence, it is based on an iterative process of trial-and-error, and learning from both positive and negative feedback. This learning paradigm also serves as the basis for a machine learning technique that can be used to solve sequential decision making problems by learning a sequence of actions to achieve a goal or maximize performance. As a machine learning method, this method was developed decades ago and there have been numerous variations and developments. However, extending these methods to real-world challenging problems is often not successful, and there are relatively few noteworthy applications. Furthermore, the reasons behind successful applications are not completely understood.

The purpose of this work is to gain a better understanding of reinforcement learning in a variety of problems. We are interested in fundamental knowledge, basic science if you will, of what affects reinforcement learning and what contributes to a successful implementation. This work takes an empirical approach to understanding the behavior and interactions between the two main components of reinforcement learning: the learning algorithm and the functional representation of the learned knowledge. We study these entities using an approach that is not commonly employed to study machine learning methods, that being design of experiments. We use contemporary design of experiments methods, including a novel sequential experimentation procedure that finds convergent learning algorithm parameter subregions, and stochastic kriging for response surface metamodeling. We consider three problems in this work, one benchmark reinforcement learning problem and two more real world and challenging problems: the mountain car problem, the truck backer-upper problem, and the tandem truck backer-upper problem. The knowledge gained from this work provides insight as to what enables and what has an effect on successful reinforcement learning implementations so that this learning method can be applied to more challenging problems.