llm pretraining

3 papers

• MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
• SkyLadder: Better and Faster Pretraining via Context Window Scheduling
• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text