next-token prediction
Next-token prediction is a common task in natural language processing where a model predicts the next word or token in a sequence based on the preceding context. It is the foundation for many language models, including transformers.
- AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
- Causal Discovery and Inference through Next-Token Prediction
- Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- Interpretable Next-token Prediction via the Generalized Induction Head
- L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models
- OmniGen-AR: AutoRegressive Any-to-Image Generation
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
- Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation