SkyLadder: Better and Faster Pretraining via Context Window Scheduling

Tianyu Pang (Sea AI Lab) · Qian Liu (TikTok (Singapore)) · Haonan Wang (The Hong Kong University of Science and Technology) · Min-Yen Kan (National University of Singapore) · Shiqi Chen (City University of Hong Kong) · Tongyao Zhu (National University of Singapore) · Xiangming Gu (National University of Singapore)

benchmark performancecommon benchmarkscontext window schedulingcontext windowscontrolled studyextensive experiments llm pretraining long-context tasks model efficiency parameter modelspretraining efficiencysequence processingshort-to-long transitionskyladdertoken budgettraining speeds

Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our controlled study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates us to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose SkyLadder, a simple yet effective approach that implements a short-to-long context window transition. SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long-context tasks. Through extensive experiments, we pretrain 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of up to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines.