tokenizers

3 papers

• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
• UniTok: a Unified Tokenizer for Visual Generation and Understanding