Test-Time Training (TTT): A New Approach to Sequence Modeling

Abstraction: Hidden state as a learnable model updated at inference

Key points:

• TTT replaces the fixed hidden state of RNNs with a small ML model (linear or MLP) that is updated via gradient descent on a self-supervised loss at each token during inference
• Two variants: TTT-Linear (fast, matches Mamba wall-clock time) and TTT-MLP (more expressive but higher memory I/O cost)
• At 8k context TTT-Linear is faster than Transformer; both TTT variants significantly outperform Mamba at 8k+ context on perplexity
• Models tested up to 1.3B parameters on The Pile and Books3 datasets
• Key insight: context becomes an "unlabeled dataset" and the hidden state model is trained on it via a self-supervised reconstruction loss ℓ(W; xₜ) = ‖f(x̃ₜ; W) − xₜ‖²
• Mini-batch TTT parallelizes gradient computation across b timesteps to improve hardware efficiency

Connections: Large Language Models · Transformers · Sequence Modeling · Self Supervised Learning

Source: https://medium.com/thedeephub/test-time-training-ttt-a-new-approach-to-sequence-modeling-8baf1ea79ed7