L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models

Xiaohao Liu (National University of Singapore) · Xiaobo Xia (The University of Sydney) · Weixiang Zhao (Harbin Institute of Technology) · Manyi Zhang (Huawei Technologies Ltd.) · Xianzhi Yu (University of the Chinese Academy of Sciences) · Xiu Su (Central South University) · Shuo Yang (University of California, Berkeley) · See-Kiong Ng (National University of Singapore) · Tat-Seng Chua (National Univ. of Singapore)

benchmarkscontextual coveragedecoding strategy experimental validation inference efficiency inference speedleap-based mechanismlong-range dependencies multi-token prediction next-token predictionnon-sequential token generationperformance boostingstructured leaptheoretical demonstration

Large language models (LLMs) have achieved notable progress. Despite their success, next-token prediction (NTP), the dominant method for LLM training and inference, is constrained in both contextual coverage and inference efficiency due to its inherently sequential process. To overcome these challenges, we propose leap multi-token prediction~(L-MTP), an innovative token prediction method that extends the capabilities of multi-token prediction (MTP) by introducing a leap-based mechanism. Unlike conventional MTP, which generates multiple tokens at adjacent positions, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass. This structured leap not only enhances the model's ability to capture long-range dependencies but also enables a decoding strategy specially optimized for non-sequential leap token generation, effectively accelerating inference. We theoretically demonstrate the benefit of L-MTP in improving inference efficiency. Experiments across diverse benchmarks validate its merit in boosting both LLM performance and inference speed. The source code is available at https://github.com/Xiaohao-Liu/L-MTP.