How to train a new language model from scratch using Transformers and Tokenizers

Abstraction: End-to-end tutorial training RoBERTa-like LM on Esperanto from scratch

Key points:

• Trains "EsperBERTo," a 84M-parameter RoBERTa-like model (6 layers, 768 hidden, 12 heads) on 3 GB of Esperanto text from OSCAR and Leipzig corpora
• Uses a byte-level BPE tokenizer (vocab size 52,000) trained with Hugging Face tokenizers; byte-level BPE eliminates unknown tokens entirely
• Model trained on masked language modeling (MLM) objective with learning rate 1e-4, 5 epochs, batch size 16
• Evaluates with FillMaskPipeline showing coherent Esperanto sentence completions
• Fine-tunes on POS tagging (token classification) using run_ner.py; achieves near-perfect performance on the highly-regular Esperanto grammar
• Recommends sharing trained models via transformers-cli upload with a model card on Hugging Face Hub

Connections: Hugging Face · Language Model Pretraining · Masked Language Modeling · Byte Pair Encoding · Transformers

Source: https://huggingface.co/blog/how-to-train