The False Promise of Imitating Proprietary LLMs
llm-imitationfine-tuningopen-source-llmevaluationalpaca
Abstraction: Finetuning open models on ChatGPT outputs mimics style but not factuality or capability
Key points:
- Imitation models (Alpaca, Self-Instruct-style) appear competitive with ChatGPT to human raters but fail targeted automatic evaluations
- Models tested: base LMs 1.5B–13B parameters, 0.3M–150M imitation tokens from ChatGPT outputs
- Key finding: imitation models close little to none of the gap between base LM and ChatGPT on tasks not heavily supported in imitation data
- Human raters deceived because imitation models mimic ChatGPT's style but not its factuality
- Authors conclude imitation is a "false promise": bridging the open/closed LM gap requires either an unwieldy amount of imitation data or better base LMs
- Recommended path: develop better open base LMs rather than taking the imitation shortcut
Connections: Chatgpt · Openai · Large Language Models · Fine Tuning · Model Evaluation
Source: https://arxiv.org/abs/2305.15717