self-distillation
A learning method that involves a model generating its own training data or labels, leveraging its predictions to refine itself iteratively, enhancing the model's performance and efficiency without requiring additional labeled data.
- How to build a consistency model: Learning flow maps via self-distillation
- SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling
- Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
- Vision Transformers with Self-Distilled Registers