Scheduling Virtual Machine Migration During Datacenter Upgrades with Reinforcement Learning

Physical machines in modern datacenters are routinely upgraded due to their maintenance requirements, which involves migrating all the virtual machines they currently host to alternative physical machines. For this kind of datacenter upgrades, it is critical to minimize the time it takes to upgrade all the physical machines in the datacenter, so as to reduce disruptions to cloud services. To minimize the upgrade time, it is essential to carefully schedule the migration of virtual machines on each physical machine during its upgrade, without violating any constraints imposed by virtual machines that are currently running. Rather than resorting to heuristic algorithms, we propose a new scheduler, Raven, that uses an experience-driven approach with deep reinforcement learning to schedule the virtual machine migration process. With our design of the state space, action space and reward function, Raven trains a fully-connected neural network using the cross-entropy method to approximate the policy of a choosing destination physical machine for each migrating virtual machine. We compare Raven with state-of-the-art heuristic algorithms in the literature, and our results show that Raven effectively leads to shorter time to complete the datacenter upgrade process.