A Survey of Causal Inference Applications at Netflix

causal-inferenceexperimentationab-testingrecommendation-systemsnetflix

Abstraction: Netflix internal summit survey of causal inference methods and applications

Key points:

Double machine learning was used to estimate incremental value of content localization (subtitles/dubs) across languages, controlling for confounders; synthetic control measured impact of pandemic-era dub delays
Holdback AB tests (showing users an experience without a specific feature) measure long-term effects of features and are used more broadly after the team codified best practices
The "Causal Ranker Framework" adds a causal adaptive layer atop existing recommendation models, framing recommendations as incrementally useful actions rather than pure correlation-based prediction
"Bellmania" methodology estimates incremental account lifetime value (LTV) using causal interpretation and Markov chain models to infer off-Netflix LTV from minimal non-subscriber transition data
Applications span content, product, and member-experience teams; topics at the internal summit included difference-in-difference, Bayesian AB testing, and causal inference in recommender systems

Connections: Netflix · Causal Inference · Ab Testing · Recommendation Systems · Quasi Experimentation