Invited Talk: Increasing Representational Power and Scaling Inference in Reinforcement Learning

As robots are starting to perform everyday manipulation tasks, such as cleaning up, setting a table or preparing simple meals, they must become much more knowledgeable than they are today. Natural environments are composed of objects, and the possibilities to manipulate them are highly structured due to the general laws governing our relational world. All these need to be acknowledged when we want to realize thinking robots that efficiently learn how to accomplish tasks in our relational world.

Triggered by this grand vision, this talk discusses the very promising perspective on the application of Statistical Relational AI techniques to reinforcement learning. Specifically, it reviews existing symbolic dynamic programming and relational RL approaches that exploit the symbolic structure in the solution of relational and first-order logical Markov decision processes. They illustrate that Statistical Relational AI may give new tools for solving the “scaling challenge”. It is sometimes mentioned that scaling RL to real-world scenarios is a core challenge for robotics and AI in general. While this is true in a trivial sense, it might be beside the point. Reasoning and learning on appropriate (e.g. relational) representations leads to another view on the “scaling problem”: often we are facing problems with symmetries not reflected in the structure used by our standard solvers. As additional evidence for this, the talk concludes by presenting our ongoing work on the first lifted linear programming solvers for MDPs. Given an MDP, our approach first constructs a lifted program where each variable presents a set of original variables that are indistinguishable given the objective function and constraints. It then runs any standard LP solver on this program to solve the original program optimally.

Acknowledgements.

This talk is based on joint works with Babak Ahmadi, Kurt Driessens, Saket Joshi, Roni Khardon, Tobias Lang, Martin Mladenov, Sriraam Natarajan, Scott Sanner, Jude Shavlik, Prasad Tadepalli, and Marc Toussaint.