Subword Tokenization

concepts · 1 notes linked

Related: Google · Large Language Models · Transformers

Notes

• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Language-independent subword tokenizer trained from raw sentences