A learning system based on lazy metareasoning

Metareasoning has been widely studied in the literature, with a wide variety of algorithms and partially overlapping methodological approaches. However, these methods are typically either not targeted toward practical machine learning systems or alternatively are focused on achieving the best possible performance for a particular domain, with extensive human tuning and research, and vast computing resources. In this paper, our goal is to create systems that perform sustained autonomous learning, with automatically determined domain-specific optimizations for any given domain, and without requiring human assistance. We present Alma, a metareasoning architecture that creates and selects reasoning methods based on empirically observed performance. This is achieved by using lazy learning at the metalevel, and automatically training and combining reasoning methods at run-time. In experiments across diverse data sets, we demonstrate the ability of Alma to successfully reason about learner performance in different domains and achieve a better overall result than any of the individual reasoning methods, even with limited computing time available.