A Differential and Pointwise Control Approach to Reinforcement Learning

Minh Nguyen (University of Texas at Austin) · Chandrajit Bajaj (The University of Texas at Austin)

consistent trajectoriescontinuous state-action spacesdifferential dual formulationdifferential policy optimizationdifferential reinforcement learninggrid controlhamiltonian structurelocal movement operatorsmolecular dynamicsphysics priorspointwise convergence guaranteesreinforcement learning sample efficiencysurface modelingtheoretical regret bound

Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement Learning (Differential RL), a novel framework that reformulates RL from a continuous-time control perspective via a differential dual formulation. This induces a Hamiltonian structure that embeds physics priors and ensures consistent trajectories without requiring explicit constraints. To implement Differential RL, we develop Differential Policy Optimization (dfPO), a pointwise, stage-wise algorithm that refines local movement operators along the trajectory for improved sample efficiency and dynamic alignment. We establish pointwise convergence guarantees, a property not available in standard RL, and derive a competitive theoretical regret bound of $\mathcal{O}(K^{5/6})$. Empirically, dfPO outperforms standard RL baselines on representative scientific computing tasks, including surface modeling, grid control, and molecular dynamics, under low-data and physics-constrained conditions.