Scaling Recommender Transformers to a Billion Parameters | Towards Data Science

Abstraction: ARGUS autoregressive recommender transformer scaled to one billion parameters

Key points:

• ARGUS (AutoRegressive Generative User Sequential Modeling) trains on two simultaneous tasks: next-item prediction (including negative interactions) and feedback prediction, unlike SASRec's positive-only next-item approach
• Scaling study over four configs (3.2M to 1.007B parameters) shows linear quality improvement on log-parameter scale; HSTU architecture from Meta's "Actions Speak Louder than Words" matched but did not beat standard transformers
• Training dataset for music streaming exceeded 300 billion listens; autoregressive pretraining processes the entire year of user history in a single transformer pass — orders of magnitude faster than per-impression runs
• History length of 8,000+ events (vs. previous 1,500–2,000) enabled by the new approach produces noticeable quality gains
• First ARGUS deployment achieved about the same metric gains as all previous transformer model generations combined; Unfamiliar setting saw +12% total listening time

Connections: Yandex · Recommender Systems · Transformers · Reinforcement Learning · Scaling Laws

Source: https://towardsdatascience.com/scaling-recommender-transformers-to-a-billion-parameters/