Innovating Faster on Personalization Algorithms at Netflix Using Interleaving

netflixinterleavingab-testingrecommendationonline-experimentssensitivity

Abstraction: Netflix two-stage experimentation using interleaving for sensitive ranking algorithm comparison

Key points:

Netflix uses a two-stage process: fast interleaving pruning to identify best candidates, then traditional A/B test on finalists
Interleaving blends rankings from two algorithms (A and B) into a single list for each user; preference is determined by share of hours viewed attributed per ranker
Uses "team draft" interleaving: algorithms alternate picking their highest-ranked available video (coin flip decides who goes first) to eliminate position bias
Interleaving requires >100x fewer users than the most sensitive A/B metric to achieve 95% statistical power
Strong correlation found between interleaving preference and A/B test outcomes, validating it as a predictor
Limitation: interleaving measures relative preference only, not absolute metrics like retention; A/B test is still needed for those

Connections: Netflix · Interleaving · Ab Testing · Recommendation Systems