GitHub - tatsu-lab/stanford_alpaca: Code and documentation to train Stanford's Alpaca models, and generate the data.
instruction-fine-tuningllamaalpacaopen-source-ailarge-language-models
Abstraction: Instruction-following LLaMA model fine-tuned via self-instruct data
Key points:
- Stanford Alpaca fine-tunes LLaMA-7B on 52K instruction-following examples generated using OpenAI's text-davinci-003 via a Self-Instruct pipeline, for under $500
- Dataset (CC BY NC 4.0) uses 20-instruction batch decoding and a single output per instruction, more diverse than original Self-Instruct seed data
- Preliminary human evaluation: Alpaca-7B performs similarly to text-davinci-003 on the Self-Instruct evaluation suite
- Fine-tuning 7B requires ~112 GB VRAM naively; FSDP full_shard mode on 4× A100 80G is the reference setup; LoRA can reduce this to ~28 GB
- Released as research-only (CC BY NC 4.0); model has not been fine-tuned for safety/harmlessness; live demo suspended
- Weight diff between Alpaca-7B and LLaMA-7B released on Hugging Face; original LLaMA weights must be obtained separately from Meta
Connections: Stanford · Meta · Openai · Instruction Fine Tuning · Large Language Models · Open Source AI