Non-Markovian Discrete Diffusion with Causal Language Models

Yangtian Zhang (Yale University) · Sizhuang He (Yale University) · Daniel Levine (Yale University) · Lawrence Zhao (Yale University) · David Zhang (Yale University) · Syed Rizvi (Yale University) · Shiyang Zhang (Yale University) · Emanuele Zappala (Idaho State University) · Rex Ying (Yale University) · David van Dijk (Yale University)

architectural changesautoregressive transformerscausal language models discrete diffusion models error accumulation generative trajectorymarkovian assumptionmodel unificationnatural-language benchmarksnon-markovian transformerpretrained llm weightssequential reasoning state-of-the-art baselinesstructured sequence generationtemporal reasoning

Discrete diffusion models offer a flexible, controllable approach to structured sequence generation, yet they still lag behind causal language models in expressive power. A key limitation lies in their reliance on the Markovian assumption, which restricts each step to condition only on the current state, leading to potential uncorrectable error accumulation. In this paper, We introduce CaDDi, a discrete diffusion model that conditions on the entire generative trajectory, thereby lifting the Markov constraint and allowing the model to revisit and improve past states. By unifying sequential (causal) and temporal (diffusion) reasoning in a single non‑Markovian transformer, CaDDi also treats standard causal language models as a special case and permits the direct reuse of pretrained LLM weights with no architectural changes. Empirically, CaDDi outperforms state‑of‑the‑art discrete diffusion baselines on natural‑language benchmarks, substantially narrowing the remaining gap to large autoregressive transformers.