Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality

Abstraction: LLaMA fine-tune on ShareGPT data achieving near-ChatGPT quality for $300

Key points:

• Vicuna-13B is fine-tuned from Meta LLaMA on ~70K user-shared ChatGPT conversations from ShareGPT.com; training cost ~$300 using spot instances via SkyPilot
• GPT-4 judge evaluation (80 diverse questions, 8 categories) scores Vicuna at 92% of ChatGPT total score; preferred over LLaMA and Alpaca in >90% of comparisons
• Key training improvements over Alpaca: multi-turn conversation loss, context window expanded from 512 to 2048 tokens (gradient checkpointing + flash attention), spot-instance cost reduction
• Trained with PyTorch FSDP on 8 A100 GPUs in one day; inference served via lightweight distributed system with fault-tolerant controller
• Limitations: weak at reasoning/math, not optimized for safety or toxicity; OpenAI moderation API used for demo filtering
• Introduced GPT-4-as-judge evaluation framework later extended into the "Judging LLM-as-a-Judge" paper and Chatbot Arena

Connections: Vicuna · Chatgpt · GPT-4 · Lmsys · Fine Tuning · LLM Evaluation

Source: https://lmsys.org/blog/2023-03-30-vicuna/