WHAT MAKES MATH PROBLEMS HARD FOR REINFORCEMENT LEARNING: A CASE STUDY

Ali Shehper (California Institute of Technology) · Anibal Medina-Mardones (University of Western Ontario) · Lucas Fagan (University of California, Santa Barbara) · Bartłomiej Lewandowski (University of Warsaw) · Angus Gruen (Independent Researcher) · Yang Qiu (School of Computer Science and Technology, Huazhong University of Science and Technology) · Piotr Kucharski (University of Warsaw) · Zhenghan Wang (University of California, Santa Barbara) · Sergei Gukov (California Institute of Technology)

agent behaviorakbulut--kirby seriesandrews--curtis conjecturecombinatorial group theoryconjectural frameworkscounterexampleshigh rewardsinfinite subfamilieslength reducibilitymathematical questionsmiller--schupp seriesoptimization challengesproblem hardnessrare instancesreinforcement learning theoretical analysis

Using a long-standing conjecture from combinatorial group theory, we explore, from multiple perspectives, the challenges of finding rare instances carrying disproportionately high rewards. Based on lessons learned in the context defined by the Andrews--Curtis conjecture, we analyze how reinforcement learning agents handle problems of varying hardness. We also address many mathematical questions as a part of our study. Notably, we demonstrate the length reducibility of all but two presentations in the Akbulut--Kirby series (1981), and resolve various potential counterexamples in the Miller--Schupp series (1991), including three infinite subfamilies.