The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

Haitao Mi (Tencent AI Lab) · Dong Yu (Tencent AI Lab) · Ke Ji (The Chinese University of Hong Kong, Shenzhen) · Benyou Wang (The Chinese University of Hong Kong, Shenzhen) · Zhaopeng Tu (Tencent AI Lab) · Xingyu Chen (Shanghai Jiaotong University) · Zhiwei He (Shanghai Jiao Tong University) · Jiahao Xu (Tencent AI Lab) · Tian Liang (Tencent AI Lab) · Qiuzhi Liu (Tencent AI Lab) · Xiaoyuan Liu (The Chinese University of Hong Kong, Shenzhen) · Junying Chen (Chinese University of Hong Kong, Shenzhen)

computationally expensive samplinginitial prefix substringslabeled dataminimal unsupervised fine-tuningprefix self-consistencyreasoning benchmarks reasoning efficiencyrejection sampling fine-tuningresource-efficient methodssampling cost reductionscalable alternativestructural knowledge preservationsupervised fine-tuning training time reductionunsupervised prefix fine-tuning

Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency -- the shared initial reasoning steps across diverse solution trajectories -- to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75\% and sampling cost by 99\%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model’s structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.