Distilling step-by-step: Outperforming larger language models with less training

knowledge-distillationllmchain-of-thoughtmodel-efficiency

Abstraction: Rationale-based distillation trains small models to outperform 540B LLMs with far less data

Key points:

Extracts chain-of-thought rationales from large LLMs via few-shot CoT prompting, then trains small models on both labels and rationales as a multi-task problem
A 770M T5 model outperforms few-shot 540B PaLM on ANLI using only 80% of the dataset — over 700x model size reduction
A 220M T5 surpasses 540B PaLM on e-SNLI; standard fine-tuning with 100% of data cannot match PaLM's performance on ANLI
Dataset reduction of 12.5–75% over standard fine-tuning while still achieving superior performance across NLI, QA, and math word problems
Rationale generation task provides richer supervision, teaching the small model intermediate reasoning steps rather than just input-output mappings
Method is available for private preview on Google Vertex AI (as of Sep 2023)

Connections: Google · Palm · T5 · Knowledge Distillation · Chain Of Thought · Large Language Models