NS-Gym: A Comprehensive and Open-Source Simulation Framework for Non-Stationary Markov Decision Processes

Nathaniel S. Keplinger (Vanderbilt University) · Baiting Luo (Vanderbilt University) · Yunuo Zhang (Vanderbilt University) · Kyle H Wray (Stanford University) · Aron Laszka (Pennsylvania State University) · Abhishek Dubey (Vanderbilt University) · Ayan Mukhopadhyay (Vanderbilt University (Incoming Asst. Professor at William & Mary))

agent’s decision-making modulealgorithmic approaches benchmark problemsdecision-making modelsdynamic environmentsenvironmental parametersflexible adaptationsgymnasium frameworkmodular adaptationsnon-stationary environmentsnon-stationary markov decision processesns-gymns-mdpsreproducible evaluationsimulation frameworksstandardized interfaces

Many real-world applications require decision-making where the environmental dynamics evolve over time. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, there are no standardized simulation frameworks for NS-MDPs, as opposed to widely popular frameworks for stationary problems. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent’s decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark several algorithmic approaches from prior work on NS-MDPs using NS-Gym. We envision that NS-Gym will enable researchers to study decision-making under non-stationarity by providing standardized interfaces, simulation frameworks, and benchmark problems.