Emergence of multi-step discrete state transition through reinforcement learning with a recurrent neural network

Abstract

For developing a robot that learns long and complicated action sequences in the real-world, autonomous learning of multi-step discrete state transition is significant. To realize the multi-step discrete state transition in a neural network is generally thought to be difficult because of basically the needs to hold the state while performing the transition between the states when needed. In this paper, only through the reinforcement learning using rewards and punishments in a simple learning system consisting of a recurrent neural network (RNN), it is shown that a multi-step discrete state transition emerged through learning in a continuous state-action space. It is shown that in a two-switch task, two states transition represented by two types of hidden nodes emerged through the learning. In addition, it is shown that the contribution of the dynamics by the interaction between the RNN and the environment based on the discrete state transitions leads to repetition of the interesting behavior when no reward is given at the goal.