Experimentation & Causal Inference

Netflix consistently employs a simple but powerful approach to product innovation: we ask our members, through online experiments, which of several possible experiences resonate with them.

This culture of experimentation and a commitment to data-driven decision making extend through all levels of the company, from our data scientists to our founder and former CEO Reed Hastings. Hypotheses and results are subject to rigorous analysis plus robust and open debate from a wide variety of viewpoints. Our executives make time to understand experimental methods and test results, and their close interaction with data scientists helps ensure that our data-driven decisions are sound from both statistical and business perspectives.

We use controlled A/B experiments to test nearly all proposed changes to our product, including new recommendation algorithms, user interface (UI) features, content promotion tactics, title launch and scheduling strategies, streaming algorithms, new member signup process, and payment methods. We also use A/B testing as part of our software deployment process, to ensure that we can quickly catch any bugs that impact the member experience.

Sometimes, the questions we have as a business cannot be answered using fully randomized A/B tests. In these cases, we rely on quasi-experiments and causal inference methods to inform decisions.

We maintain an active applied research agenda aimed at improving all aspects of experimentation and causal inference at Netflix, and present our work at external forums like MIT’s Conference on Digital Experimentation and the American Causal Inference Conference. To learn more, dive into our blogs and publications.