language model pretraining
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
- Group-Level Data Selection for Efficient Pretraining
- Multi-Token Prediction Needs Registers
- PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts