Hugging Face Introduces StackLLaMA: A 7B Parameter Language Model Based on LLaMA and Trained on Data from Stack Exchange Using RLHF

Abstraction: Hugging Face RLHF fine-tuning of LLaMA 7B on Stack Exchange Q&A data

Key points:

• StackLLaMA is a 7B parameter model fine-tuned from Meta's LLaMA using three stages: supervised fine-tuning (SFT), reward/preference modeling (RM), and RLHF via the TRL library
• Training data: StackExchange dataset with 10M+ question-answer pairs including Stack Overflow code; upvote counts and accepted-answer labels served as reward signals
• Reward model achieved 67% accuracy — considered strong given the difficulty of the task even for human annotators
• To prevent reward hacking (model generating gibberish that tricks the reward model), a KL divergence penalty was added to the reward signal
• Released publicly for educational/research purposes; user prompts are being collected for further fine-tuning (sensitive data warning issued)
• Demonstrates a complete open-source RLHF pipeline comparable to what powers ChatGPT, GPT-4, and Claude

Connections: Hugging Face · Llama · Stack Exchange · Reinforcement Learning · Large Language Models · RLHF

Source: https://www.marktechpost.com/2023/04/12/hugging-face-introduces-stackllama-a-7b-parameter-language-model-based-on-llama-and-trained-on-data-from-stack-exchange-using-rlhf/