Skill-Driven Neurosymbolic State Abstractions

David Abel (Google DeepMind) · George Konidaris (Brown University) · Alper Ahmetoglu (Brown University) · Steven James (University of the Witwatersrand) · Cameron Allen (UC Berkeley) · Sam Lobel (Brown University)

abstract actionsabstract markov decision processapproximate solutionsbellman equationconstruction frameworkdecision-making processesdistributions over statesfactored abstract statesfactored actionsmodel-preservingmontezuma's revengeneurosymbolicplanning algorithmsstate abstractionsvisual maze task

We consider how to construct state abstractions compatible with a given set of abstract actions, to obtain a well-formed abstract Markov decision process (MDP). We show that the Bellman equation suggests that abstract states should represent distributions over states in the ground MDP; we characterize the conditions under which the resulting process is Markov and approximately model-preserving, derive algorithms for constructing and planning with the abstract MDP, and apply them to a visual maze task. We generalize these results to the factored actions case, characterizing the conditions that result in factored abstract states and apply the resulting algorithm to Montezuma's Revenge. These results provide a powerful and principled framework for constructing neurosymbolic abstract Markov decision processes.