Empowerment as a Task-Agnostic Measure of Domain Competence

Abstract

A ubiquitous challenge for humans and learning agents is the ability to measure and forecast how competent they can become in a specific domain. The problems created by the inability to forecast and measure achievable competence are manifold. We don't know who will be good at which jobs - we don't know which problems a machine learning architecture can solve, we don't know whether other agents might be better, etc. Performance measures exist in all sorts of domains: e.g. video games, athletics, academics, etc. that traditionally capture task-specific performance heuristics such as points accrued, time remaining, accuracy, etc. Assessment is problematic because we don't have the ideal battery of tests, and the time-cost of extensive testing on all plausible tests is prohibitive. In reinforcement learning we desire that agents are able to learn reasonable behavior on novel tasks in new environments, but it is unclear on how to best design tasks and scheduling to provide the agent with a general understanding of its capabilities. These quantities based on task heuristics may not even be appropriate for measuring an agent’s general ability if they do not accurately reflect the agent’s true goals, especially in environments with multiple available tasks. What does it really mean to be competent in an environment? Rather than domain-specific heuristics, a more fundamental notion of skill is the agent’s ability to understand, predict and control their environment. While we normally impute a player’s capabilities indirectly from their score or rank, in this dissertation we show that it is possible to create a direct measure of a player’s capabilities via the empowerment measure. We then use this measure to show the value of using a better universal objective in the context of reinforcement learning. Navigation is a task where it is advantageous to understand, predict, and control the environment. Methods for localization using information-theoretic quantities focus on where to sample signals to reduce uncertainty, rather than use the agent’s understanding of its own capabilities to understand where it might fail. We demonstrate a proof of concept using empowerment to predict navigation failure, and also how it can be used to produce a safer route. This measure is poised to have broad impact across multiple domains, including RL, education, and entertainment.