vocabulary size

4 papers

• Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees
• Explaining and Mitigating Crosslingual Tokenizer Inequities
• Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law
• UniTok: a Unified Tokenizer for Visual Generation and Understanding