A novel neural reinforcement learning algorithm based on critic-only architecture for continuous state control problems

Many real-world control problems inherently have large and continuous domains, leading to the curse of dimensionality in the learning process. This paper proposes a novel critic-only based Neural Reinforcement Learning (NRL) algorithm designed for continuous state spaces. Our approach, called Neural Least Square Policy Iteration (NLSPI), utilizes LSPI method with Radial Basis Function (RBF) network as a function approximator. RBF network offers a continuous and compact representation for continues sensory spaces, enabling the learning system to generalize the learned policy to unseen states. LSPI is employed to adjust the weight parameters of RBF network. Additionally, we present positive theoretical results regarding an error bound between the optimal and the approximated Action Value Function (AVF) for NLSPI. Our proposed method boasts favorable features such as positive mathematical analysis, independence from learning rate, and comparatively good convergence properties. Simulation studies demonstrate the applicability and performance of our learning framework. The overall results indicate that the proposed idea can outperform previously known reinforcement learning algorithms.