scaling laws
Empirical observations in machine learning that describe how the performance of models improves as a function of increased data, model size, or computational resources. Understanding scaling laws helps predict how enhancements will affect model training.
- Diffusion Beats Autoregressive in Data-Constrained Settings
- Dimension-adapted Momentum Outscales SGD
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules
- Gemstones: A Model Suite for Multi-Faceted Scaling Laws
- HollowFlow: Efficient Sample Likelihood Evaluation using Hollow Message Passing
- LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought
- Language Modeling by Language Models
- Learning in Compact Spaces with Approximately Normalized Transformer
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
- Provable Scaling Laws for the Test-Time Compute of Large Language Models
- Scaling Laws For Scalable Oversight
- Scaling Laws for Optimal Data Mixtures
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
- Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
- TabDPT: Scaling Tabular Foundation Models on Real Data
- TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
- Unified Scaling Laws for Compressed Representations
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization