Fine-tuning · Hugging Face

Abstraction: Fine-tuning pretrained LLMs with Hugging Face Trainer API

Key points:

• Fine-tuning continues training on a smaller task-specific dataset; uses pretrained weights (not random init); requires far less compute, data, and time than pretraining
• Pipeline: load dataset → tokenize (AutoTokenizer with truncation/max_length) → DataCollatorForLanguageModeling (dynamic padding, mlm=False for causal LM) → load model with dtype="auto" to avoid float32 memory doubling
• TrainingArguments key params: num_train_epochs, per_device_train_batch_size, learning_rate, gradient_accumulation_steps, gradient_checkpointing, bf16, eval_strategy, save_strategy
• Example uses Qwen3-0.6B on horoscope dataset; trained with lr=2e-5, batch_size=2, gradient_accumulation=8, 3 epochs
• trainer.push_to_hub() uploads weights, tokenizer, generation config, and model config to the Hugging Face Hub

Connections: Hugging Face · Qwen · Fine Tuning · Large Language Models · Transformers

Source: https://huggingface.co/docs/transformers/training