Language Model Pretraining

concepts · 1 notes linked

Related: Hugging Face · Masked Language Modeling · Byte Pair Encoding · Transformers

Notes

• How to train a new language model from scratch using Transformers and Tokenizers — End-to-end tutorial training RoBERTa-like LM on Esperanto from scratch