Large Language Diffusion Models

Ji-Rong Wen (Renmin University of China) · Jun Zhou (Ant Financial) · Shen Nie (Renmin University of China) · Chongxuan LI (Renmin University of China) · Fengqi Zhu (Renmin University of China) · Zebin You (Gaoling School of AI, Renmin University of China) · Xiaolu Zhang (Ant Financial Services Group) · Jingyang Ou (Renmin University of China) · Jun Hu (Ant Group) · Yankai Lin (Renmin University of China)

autoregressive models benchmarks diffusion modelforward data maskingin-context learning instruction-followinglikelihood lower boundlladapre-training probabilistic inferencereversal cursereverse generation processscalability supervised fine-tuning

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing *LLaDA*, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong *scalability* and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in *in-context learning* and, after SFT, exhibits impressive *instruction-following* abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: \url{https://ml-gsai.github.io/LLaDA-demo/}.