Informed Correctors for Discrete Diffusion Models

Feng Chen (Microsoft AI / Stanford University) · Shaul Druckmann (Stanford University) · Scott Linderman (Stanford University) · Lester Mackey (Microsoft Research) · Jiaxin Shi (Google DeepMind) · Yixiu Zhao (Basis Research Institute)

approximation errors architectural modifications discrete diffusion failure modesfast generationfid scores generative modeling high-fidelity generationhollow transformersinformed correctorspredictor-corrector schemesample quality sampling strategiestailored training objectivetokenized imagenettraining signal

Discrete diffusion has emerged as a powerful framework for generative modeling in discrete domains, yet efficiently sampling from these models remains challenging. Existing sampling strategies often struggle to balance computation and sample quality when the number of sampling steps is reduced, even when the model has learned the data distribution well. To address these limitations, we propose a predictor-corrector sampling scheme where the corrector is informed by the diffusion model to more reliably counter the accumulating approximation errors. To further enhance the effectiveness of our informed corrector, we introduce complementary architectural modifications based on hollow transformers and a simple tailored training objective that leverages more training signal. We use a synthetic example to illustrate the failure modes of existing samplers and show how informed correctors alleviate these problems. On the Text8 dataset, the informed corrector improves sample quality by generating text with significantly fewer errors than the baselines. On tokenized ImageNet 256x256, this approach consistently produces superior samples with fewer steps, achieving improved FID scores for discrete diffusion models. These results underscore the potential of informed correctors for fast and high-fidelity generation using discrete diffusion.