SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
tokenizationnlpsubwordneural-machine-translation
Abstraction: Language-independent subword tokenizer trained from raw sentences
Key points:
- SentencePiece trains subword models directly from raw sentences, eliminating pre-tokenization requirement
- Existing subword tools (BPE, wordpiece) assume input is already word-tokenized; SentencePiece does not
- Enables purely end-to-end, language-independent NLP pipelines
- Validated on English-Japanese NMT achieving accuracy comparable to direct subword training
- Provides open-source C++ and Python implementations under Apache 2 license
- Available at github.com/google/sentencepiece
Connections: Google · Subword Tokenization · Transformers
Source: https://arxiv.org/abs/1808.06226