We describe how to support content decision makers by leveraging machine learning and statistical modeling.

Supporting content decision makers with machine learning

Senior Machine Learning Scientist

ML Researcher, Content & Media ML Foundations

Senior Research Scientist

Director, Content & Media ML Foundations

In a [previous blog post](https://netflixtechblog.com/artwork-personalization-c589f074ad76) we explained how our artwork personalization algorithm can pick the best image for each member, but how do we create a good set of images to choose from? What data would you like to have if you were designing an asset suite?

In this blog post, we talk about two approaches to create effective artwork. Broadly, they are:

The top-down approach, where we preemptively identify image properties to investigate, informed by our initial beliefs.
The bottom-up approach, where we let the data naturally surface important trends.

Discovering Creative Insights in Promotional Artwork

CV Senior Research Scientist

Head of Machine learning

We introduce Magenta Green Screen, a novel machine–learning enabled matting technique for recording the color image of a foreground actor with a simultaneous high-quality alpha channel without requiring a special camera or manual keying techniques. We record the actor on a green background, but light them with only red and blue foreground lighting. In this configuration, the green channel shows the actor silhouetted against a bright, even background, which can be used directly as a holdout matte, the inverse of the actor's alpha channel. We then restore the green channel of the foreground using a machine learning colorization technique. We train the colorization model with an example sequence of the actor lit by white lighting, yielding convincing and temporally stable colorization results. We further show that time-multiplexing the lighting between magenta green screen and green magenta screen allows the technique to be practiced under what appears to be mostly normal lighting. We demonstrate that our technique yields high-quality compositing results when implemented on a modern LED virtual production stage. The high-quality alpha channel data obtainable with our technique can provide significantly higher quality training data for natural image matting algorithms to support future ML matting research.

Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization

Research Scientist

Senior Research Scientist • Production Innovation

 Hardware Engineer

Director, Research, Creative Algorithms & Technologies • Production Innovation

In this blog post, we introduce RecSysOps a set of best practices and lessons that we learned while operating large-scale recommendation systems at Netflix. These practices helped us to keep our system healthy while 1) reducing our firefighting time, 2) focusing on innovations and 3) building trust with our stakeholders.

RecSysOps: Best Practices for Operating a Large-Scale Recommender System

Senior Research Engineer

Director, Recommendation Systems Research and Engineering

This writeup is about using reinforcement learning to construct an optimal list of recommendations when the user has a finite time budget to make a decision from the list of recommendations.

Reinforcement Learning for Budget Constrained Recommendations

Research Advisor at Netflix and Assistant Professor, Cornell University

Data, Machine Learning and AI

At Netflix, we aim to provide recommendations that match our members’ interests. To achieve this, we rely on Machine Learning (ML) algorithms. ML algorithms can be only as good as the data that we provide to it. This post will focus on the large volume of high-quality data stored in Axion — our fact store that is leveraged to compute ML features offline. We built Axion primarily to remove any training-serving skew and make offline experimentation faster. We will share how its design has evolved over the years and the lessons learned while building it.

Evolution of ML Fact Store

Senior Software Engineer, Machine Learning Platform

This is the fifth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products.

Building confidence in a decision

Data Science

Director, Experimentation Platform

Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation.

Experimentation is a major focus of Data Science across Netflix

Data Scientist 6

Manager, Payments Data Science and Engineering

Using experimentation as a mechanism to safely and quickly update Netflix client applications.

Safe Updates of Client Applications at Netflix

Engineering Leader

Last January, Netflix launched globally, reaching many new members in 130 countries around the world. In many of these countries, people access the internet primarily using cellular networks or still-developing broadband infrastructure. 

More Efficient Mobile Encodes for Netflix Downloads

Software Engineer - Video & Image Encoding

Senior Director, Encoding Technologies

A recap of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019 on the topic of Infrastructure for Contextual Bandits and Reinforcement Learning. Features slides and videos from LinkedIn, Netflix, Facebook, and Dropbox


ML Platform Meetup: Infra for Contextual Bandits and Reinforcement Learning

Director of Engineering, Personalization Infrastructure

Sean shares his perspective on how nonlinear career trajectories make us better, and can bring unexpected benefits during unforeseen circumstances.

My (Seemingly) Random Walk to Netflix

Senior Data Scientist

To continue the story of Dataflow for workflow asset management in the Netflix data ecosystem, we’d like to introduce readers to the data pipeline project management and to ready-to-go sample workflows.

Ready-to-go sample data pipelines with Dataflow

In the third installment of the Netflix analytics series, Alex helps mythbust some common misconceptions about succeeding in analytics at a big tech company.

Mythbusting the Analytics Journey

Senior Analytics Engineer

Netflix was a proud sponsor of the 2020 Conference on Digital Experimentation (CODE), hosted by the MIT Initiative on the Digital Economy. As well as providing sponsorship, Netflix data scientists were active participants, with three contributions.

Netflix at MIT CODE 2020

In the second in the analytics branding series, Rocio shares her perspective of the projects, technical skills, and day-to-day of working in the Content analytics and metrics research space.

A Day in the Life of a Content Analytics Engineer

This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering at Netflix through the eyes of two team members: Chris Pham and Julie Beckley.

How Our Paths Brought Us to Data and Netflix

Analytics Engineer - Live Streaming

An Introduction to Analytics and Visualization Engineering at Netflix

Analytics at Netflix: Who We Are and What We Do

Manager, Product Content Data Science & Engineering

Manager, Content Delivery Science & Analytics

We discuss some key challenges and approaches Netflix has been using to handle small sample size and limited pre-intervention data in quasi experiments.

Key Challenges with Quasi Experiments at Netflix

Director, Experimentation Platform 

The Netflix experience is powered by a family of ranking algorithms, each optimized for a different purpose. For instance, the Top Picks row on the homepage makes recommendations based on a personalized ranking of videos, and the Trending Now row also incorporates recent popularity trends. These algorithms, along with many others, are used together to construct personalized homepages for over 100 million members.

Innovating Faster on Personalization Algorithms at Netflix Using Interleaving

Director, Experimentation

Manager, Experimentation

Manager, Algorithms Science & Analytics

Computational causal inference brings a software implementation focus to causal inference, especially in regards to high performance numerical computing.

Computational Causal Inference

Senior Modeling Architect, Computational Causal Inference at Netflix

In 2012, Netflix launched its first original title to members in Canada and the US. Lilyhammer, starring Steve Van Zandt, was a fish out of water tale about a New York mobster hiding out in rural Norway. As a data scientist starting out my career at Netflix, it felt like an earth-shattering event. Netflix … launching its own original content! It was my role to design and analyze the foundational A/B Test in order to optimize the way we promoted a Netflix show to our members within our service. I felt very Hollywood and very Silicon Valley at the same time. Fast forward to May 8th, 2017 when (within a single week) Netflix debuted about a dozen new original titles to 100 million members in almost 190 countries simultaneously. An amazing transformation - and we’ve only just begun!

Data Science @ Netflix & Promoting Netflix Originals

Vice President, Content and Marketing Science and Analytics

With a global launch in January 2016 that brought Netflix to over 130 new countries, Netflix is now a truly global Internet TV network, available in almost every part of the world. Our member base is over 100 million strong, and approximately half of our members live outside the US. 

How do we ensure that the Netflix streaming experience is as enjoyable in São Paulo, Mumbai, and Bangkok as it is in San Francisco, London, or Paris? The engineers and scientists at Netflix continuously innovate to ensure that we can provide the best QoE possible. To enable this, Netflix has a culture of experimentation and data-driven decision making that allows new ideas to be tested in production so we get data and feedback from our members.

A/B Testing and Beyond: Improving the Netflix Streaming Experience

Director, Studio Production & Streaming Data Science

This post introduces the Dataflow CLI and showcases one of its features: workflow asset management. This would be the first in a series of articles about Dataflow, its features and the problems it solves.

Data pipeline asset management with Dataflow

This blog post describes our approach to predictive CPU isolation for containers at Netflix -- enabling tasks to run faster, and in a more predictable way, by limiting  co-location with noisy neighbors. This approach blends Machine Learning (quantile regression) and optimization techniques (Mixed-Integer Programming).

Predictive CPU isolation of containers at Netflix

Senior Machine Learning Researcher

Senior Software Engineer

We discuss salient challenges of building a search experience for a streaming media service such as Netflix. We provide an overview of the role of recommendations within the search context to aid content discovery and support searches for unavailable (out-of-catalog) entities. We also stress the importance of keystroke-level Instant Search experience, and the technical challenges associated with implementing it across different devices and languages for a global audience.

Challenges in Search on Streaming Services: Netflix Case Study

Perfecting video adaptive bitrate algorithm (ABR) in a commercial setting is challenging, given the endless array of network environments, device capabilities, and content properties. To identify shortcomings effectively, a scalable methodology is needed to evaluate ABR algorithms under various scenarios. The state-of-the-art method is to evaluate a production ABR retrospectively with an optimal ABR algorithm. However, optimal ABR is an NP-hard problem and therefore is costly to be deployed. In this paper, we present Hindsight, an efficient approximation for the optimal ABR problem. We will also share how Hindsight is used as a tool to identify sessions with suboptimal ABR performance in our production environment. This task was previously infeasible at scale due to its high computational complexity. We hope the discussion in
this work can draw attention from the community and help further advance the understanding of this area.

Hindsight: Evaluate Video Bitrate Adaptation at Scale

Senior Engineering Manager, Adaptive Streaming

Manager, Video Codecs and Quality

Over 69 million Netflix members stream billions of hours of movies and shows every month in North and South America, parts of Europe and Asia, Australia and New Zealand. Soon, Netflix will be available in every corner of the world with an even more global member base.

As we expand globally, our goal is to ensure that every member has a high-quality experience every time they stream content on Netflix. This challenging problem is impacted by factors that include quality of the member’s Internet connection, device characteristics, content delivery network, algorithms on the device, and quality of content.

Optimizing Content Quality Control at Netflix with Predictive Modeling

’Tis the season for holiday binging. How do seasonal viewing patterns affect how Netflix stores and streams content?

Our solution for delivering streaming content is Open Connect, a custom-built system that distributes and stores the audio and video content our members download when they stream a Netflix title (a movie or episode). Netflix has a unique library of large files with relatively predictable popularity, and Open Connect’s global, distributed network of caching servers was designed with these attributes in mind. 

Caching Content for Holiday Streaming

In the internet-connected world, it's almost expected that any form of entertainment (movies, TV shows, sports, music, etc.) is available anytime, anywhere, and on any device. We're in the midst of a significant change in the way we consume content. For example, as a result of new models of content distribution, you no longer have a 13-week wait to find out what happens at the end of a House of Cards season. Instead, you can binge-watch the entire season in one sitting! At the core of this cultural shift is the ability to smoothly stream video over the internet to millions of members around the globe.

Streaming Science at Netflix: Turning Data into a Better Viewing Experience

At Netflix, the Content Platform Engineering and Global Product Creative teams know that imagery plays an incredibly important role in how viewers find new shows and movies to watch. We take pride in surfacing the unique elements of a story that connect our audiences to diverse characters and story lines. As our Original content slate continues to expand, our technical experts are tasked with finding new ways to scale our resources and alleviate our creatives from the tedious and ever-increasing demands of digital merchandising. 

AVA: The Art and Science of Image Discovery at Netflix

Senior UI Architect

Senior Product Designer

Senior Software Engineer - Computer Vision

Senior Software Engineer in Test

Design Program Manager

Program Manager, ML Automation & Systems Infrastructure

In this two-part blog post, we will open the doors of one of the most valued Netflix assets: our recommendation system. In Part 1, we will relate the Netflix Prize to the broader recommendation challenge, outline the external components of our personalized service, and highlight how our task has evolved with the business.

Netflix Recommendations: Beyond the 5 stars (Part 1)

In this second part of the blog post, we will give more insight into our broader personalization technology. We will discuss some of our current models, data, and the approaches we follow to lead innovation and research in this space.

Netflix Recommendations: Beyond the 5 stars (Part 2)

Having launched the Netflix service globally in January, we now support search in 190 countries. We currently support 20 languages, and this will continue to grow over time. Some of the most challenging language support was added while launching in Japan and Korea as well as in the Chinese and Arabic speaking countries. We have been working on tuning the language specific search prior to each launch by creating and tuning the localized datasets of the documents and their corresponding queries. While targeting a high recall for the launch of a new language, our ranking systems focus on increasing the precision by ranking the most relevant results high on the list.

Global Languages Support at Netflix

Netflix uses machine learning to understand the underlying structure of movies and TV shows, including scenes, sequences, and acts. Recognizing these elements is crucial for video summarization, content retrieval, quality assessment, and video editing.
The blog post presents two approaches to scene boundary detection in audiovisual content. The first method leverages auxiliary data from a screenplay, aligning screenplay text with timed text (like closed captions) to assign timestamps to scene headers. This approach, however, requires high-quality screenplays and has to account for changes made during filming or post-production.
The second approach uses a supervised sequential model, such as bidirectional LSTM or GRU, that utilizes pretrained shot-level embeddings. These embeddings, which are rich in content information, can outperform current state-of-the-art baselines on Netflix's internal benchmarks.

Detecting Scene Changes in Audiovisual Content

Today, in partnership with Google, we have open sourced Kayenta, a platform for Automated Canary Analysis (ACA). Kayenta leverages lessons learned over the years of delivering rapid and reliable changes into production at Netflix. It is a crucial component of delivery at Netflix as it reduces the risk from making changes in our production environment. In addition, Kayenta has increased developer productivity by providing engineers with a high degree of trust in their deployments.

Automated Canary Analysis at Netflix with Kayenta

Data Scientist

Netflix has over 139M members streaming on more than half a billion devices spanning over 1,700 different types of devices from hundreds of brands. This diverse device ecosystem results in a high dimensionality feature space, often with sparse data, and can make identifying device performance issues challenging. 

Detecting Performance Anomalies in External Firmware Deployments

Senior Data Scientist at Netflix

Learn more about how Netflix uses data science and analytics to improve the reliability, security and efficiency of its infrastructure.

How Data Inspires Building a Scalable, Resilient and Secure Cloud Infrastructure At Netflix

Manager - Science & Analytics at Netflix

Cloud Infra and Security Data Engineering Leader at Netflix

Today we’re exploring another important piece of the puzzle: how to create a software architecture that can deliver this experience and support rapid innovation. Coming up with a software architecture that handles large volumes of existing data, is responsive to user interactions, and makes it easy to experiment with new recommendation approaches is not a trivial task. In this post we will describe how we address some of these challenges at Netflix.

System Architectures for Personalization and Recommendation

Providing high-fidelity, near real-time data is important for multi-armed bandit and RL models to converge quickly. In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state.


Real-time Processing with Flink for Machine Learning at Netflix

What's deeper than deep learning? A look at how recommender systems have grown from matrix factorization, to deep learning, to hyperbolic neural networks.

Deeper Things

Director, Machine Learning

Both router buffer sizing and Active Queue Management (AQM) are concepts deeply embedded inside the computer networks, such that people don't usually draw connections between them and video quality of experience (QoE). In this talk, we will describe our observations at Netflix on how buffer sizing and AQM affects QoE. We hope to draw attention from the community to them through this talk. 

https://ripe78.ripe.net/archives/video/128/



Buffer Sizing and AQM Observations at Netflix

Network Architect

Content Delivery Operations Engineer

Engineering Manager

As we’ve described in our previous blog posts, at Netflix we use personalization extensively and treat every situation as an opportunity to present the right content to each of our over 57 million members. The main way a member interacts with our recommendations is via the homepage, which they see when they log into Netflix on any supported device. The primary function of the homepage is to help each member easily find something to watch that they will enjoy. A problem we face is that our catalog contains many more videos than can be displayed on a single page and each member comes with their own unique set of interests. Thus, a general algorithmic challenge becomes how to best tailor each member’s homepage to make it relevant, cover their interests and intents, and still allow for exploration of our catalog.

Learning a Personalized Homepage

Senior Machine Learning Engineer

Have you ever wondered where your video comes from when you watch Netflix? We serve video streams out of our own content delivery network (CDN), called Open Connect, which is tailored to one specific application: delivering internet TV to our members around the world. This system is the cornerstone of every Netflix video experience — serving 100% of our video, over 125 million hours every day, to 100 million members across the globe! In this post, we introduce some of the challenges in the content-delivery space where our data science and engineering teams collaborate to optimize the Netflix service.

How Data Science Helps Power Worldwide Delivery of Netflix Content

Another day, another custom script to analyze an A/B test. Maybe you’ve done this before and have an old script lying around. If it’s new, it’s probably going to take some time to set up, right? Not at Netflix.

Reimagining Experimentation Analysis at Netflix

Scientific Product Manager, Experimentation Platform

At Netflix, we have created millions of artwork to represent our titles. Each artwork tells a story about the title it represents. From our testing on promotional assets, we know which of these assets have performed well and which ones haven’t. Through this, our teams have developed an intuition of what visual and thematic artwork characteristics work well for what genres of titles. A piece of promotional artwork may resonate more in certain regions, for certain genres, or for fans of particular talent. The complexity of these factors makes it difficult to determine the best creative strategy for upcoming titles.
Our assets are often created by selecting static image frames directly from our source videos. To improve it, we decided to invest in creating a Media Understanding Platform, which enables us to extract meaningful insights from media that we can then surface in our creative tools. In this post, we will take a deeper look into one of these tools, AVA Discovery View.

AVA Discovery View: Surfacing Authentic Moments

As we have described previously on this blog, at Netflix we are constantly innovating by looking for better ways to find the best movies and TV shows for our members. When a new algorithmic technique such as Deep Learning shows promising results in other domains (e.g. Image Recognition, Neuro-imaging, Language Models, and Speech Recognition), it should not come as a surprise that we would try to figure out how to apply such techniques to improve our product. In this post, we will focus on what we have learned while building infrastructure for experimenting with these approaches at Netflix.

Distributed Neural Networks with GPUs in the AWS Cloud

We are pleased to announce the open-source launch of Polynote: a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more.

Open-sourcing Polynote: an IDE-inspired polyglot notebook

Senior Software Engineer, Machine Learning

Invited talk at the Adaptive and Multi-Task Learning (AMTL) workshop at ICML 2019 on 2019-06-15 in Long Beach, CA. Covers some recent trends in research in the personalization and recommender systems space along with some potential opportunities for using multi-task and meta learning in that area.


Recent Trends in Personalization: A Netflix Perspective

In order to produce software that improves a scientist's productivity we have come up with the following design principles.

Design Principles for Mathematical Engineering in Experimentation Platform at Netflix

On January 16, 2007, Netflix started rolling out a new feature: members could now stream movies directly on their browser without having to wait for the red envelope in the mail. This event marked a substantial shift for Netflix and the entertainment industry. A lot has changed since then. Today, Netflix delivers over 1 billion hours of streaming per month to 48 million members in more than 40 countries. And Netflix accounts for more than a third of peak Internet traffic in the US. This level of engagement results in a humungous amount of data.

Optimizing the Netflix Streaming Experience with Data Science

When Netflix launched in 130 additional countries in January, we wanted our service to work well everywhere in the world, and that meant solving many complex technical problems with a global approach.

For example, the personalized recommendations we offer members are central to the Netflix experience, helping people find something great to watch quickly and easily. In general, our algorithms analyze vasts amounts of data to produce these recommendations, and making this process work globally was challenging on several fronts.

A Global Approach to Recommendations

With 83+ million members watching billions of hours of TV shows and movies, Netflix sends a huge amount of video bits through the Internet. As we grow globally, more of these video bits will be streamed through bandwidth-constrained cellular networks. Our team works on improving our video compression efficiency to ensure that we are good stewards of the Internet while at the same time delivering the best video quality to our members. Part of the effort is to evaluate the state-of-the-art video codecs, and adopt them if they provide substantial compression gains.



A Large-Scale Comparison of x264, x265, and libvpx — a Sneak Peek

Manager, Video and Image Encoding

We’ve spent years developing an approach, called per-title encoding, where we run analysis on an individual title to determine the optimal encoding recipe based on its complexity. Imagine having very involved action scenes that need more bits to encapsulate the information versus unchanging landscape scenes or animation that need less. This allows us to deliver the same or better experience while using less bandwidth, which will be particularly important in lower bandwidth countries and as we expand to places where video viewing often happens on mobile networks.

Per-Title Encode Optimization

 Senior Software Engineer, Video Algorithms

Director of Encoding Technologies

At Netflix we care about video quality, and we care about measuring video quality accurately at scale. Our method, Video Multimethod Assessment Fusion (VMAF), seeks to reflect the viewer’s perception of our streaming quality. We are open-sourcing this tool and invite the research community to collaborate with us on this important project.

Toward A Practical Perceptual Video Quality Metric

Video encoding has fueled academic research over the past 25 years and enabled compelling products and services. Many companies are built around video encoding and transmission — Netflix and Google’s YouTube are two prime examples of video-centric companies. The fundamentals of video encoding haven’t changed for all these years, as modern video streams are produced with the same type of encoding parameters that have been used since MPEG-1: a certain frame resolution is chosen, together with a group-of-pictures (GOP) structure that imposes periodic Intra pictures, and a target bitrate that is (approximately) met by either a single-pass or two-passes over the input video frames.

Dynamic optimizer — a perceptual video encoding optimization framework

"Netflix uses machine learning (ML) to personalize promotional content and optimize its creation process. Their media-focused ML infrastructure enables efficient access and processing of media data, large-scale model training, and model productization.
To address challenges like standardization of media assets and efficient computation and storage of media features, Netflix has developed tools like Jasper for media access and Amber Feature Store for media feature storage.
A case study on Match Cutting, a video editing technique, illustrates the effectiveness of this infrastructure. It helped overcome hurdles like lack of standardization and repeated computations, leading to a more efficient process.
As the intersection of media and ML opens up numerous innovation opportunities, Netflix is extending its ML infrastructure to facilitate more use cases, including ML-based VFX tooling and enhancing recommendations using content understanding models."

Scaling Media Machine Learning at Netflix

Computer Vision Systems Architect

Senior Creative Technologist

ML Engineer

A key aspect of Netflix is providing our members with a personalized experience so they can easily find great stories to enjoy. A collection of recommender systems drive the main aspects of this personalized experience and we continuously work on researching and testing new ways to make them better. As such, we were delighted to sponsor and participate in this year’s ACM Conference on Recommender Systems in Boston, which marked the 10th anniversary of the conference. For those who couldn’t attend or want more information, here is a recap of our talks and papers at the conference.

Netflix at RecSys 2016 Recap

Director, Algorithm Engineering

Because treatments in streaming experimentation may have much larger impacts on high (or low) values of a given metric, changes in the mean, the median, or other summary statistics are not generally sufficient to understand if and how the test treatment has changed the behaviour of that metric. In general, we are interested in how the distributions of relevant metrics differ between test experiences (called “cells” here at Netflix). Our goal is to arrive at an inference and visualization solution that simultaneously indicates which parts of the distribution have changed, and both the practical and statistical significance of those changes.

Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance

We want to make it easy for Netflix members to find great content to fulfill their unique tastes. To do this, we follow a data-driven algorithmic approach based on machine learning, which we have described in past posts and other publications. We aspire to a day when anyone can sit down, turn on Netflix, and the absolute best content for them will automatically start playing.

Distributed Time Travel for Feature Generation

With the deluge of deep learning libraries and software innovation in the field over the last few years, it is an exciting time to be working on machine learning problems. Most of the libraries available evolved from fairly specialized computational code for large dense problems such as image classification into general frameworks for neural-network-based models offering marginal support for sparse models.



Introducing Vectorflow: a lightweight neural network library for sparse data

Machine Learning is making fast inroads into many areas of business and is being employed in an increasingly widening array of commercial applications. At Netflix, ML has been used for several years on our Recommendations and Personalization problems. Over the last couple of years, ML has expanded into a wide range of newer applications here, such as Content Promotion, Content Price Modeling, Programmatic Marketing, Efficient Content Delivery, Adaptive Streaming QoE, to name only a handful.

Machine Learning Platform Meetup Recap

We’re developing a new product within the wider Experimentation Platform, to automate some aspects of the scientists workflow so we can free up our scientists and run more quasi-experiments in parallel. 

Quasi Experimentation at Netflix

Senior Product Manager, Experimentation Platform

In January 2016 we launched our service globally to begin bringing joy to a much larger audience.  Marketing drives awareness, original content drives organic growth, and we actively engage in partnerships to create inroads to new member bases.  Our partnerships include consumer electronics companies, cable TV operators, mobile operators, and of course Apple and Google. No matter where you are in the world, a partner is helping bring the same amazing Netflix experience to you.

Scaling Netflix through Partnerships

Manager, GPP S&A Data Products

It’s 2 a.m. and half of our reliability team is online searching for the root cause of why Netflix streaming isn’t working. None of our systems are obviously broken, but something is amiss and we’re not seeing it. After an hour of searching we realize there is one rogue server in our farm causing the problem. We missed it amongst the thousands of other servers because we were looking for a clearly visible problem, not an insidious deviant.

Tracking down the Villains: Outlier Detection at Netflix

Director of Engineering, Discovery & Playback Services

Senior Reliability Engineer

Netflix has released hundreds of Originals and plans to spend $8 billion over the next year on content. Creators of these stories pour their hearts and souls into turning ideas into joy for our viewers. The sublime art of doing this well is hard to describe, but it necessitates a careful orchestration of creative, business and technical decisions. Here we will focus on the latter two — business & technical decisions like planning budgets, finding locations, building sets, and scheduling guest actors that enable the creative act of connecting with viewers.

Publications

Netflix Research regularly publishes a variety of updates, talks, and other presentations.

Recent Articles

Netflix Artwork Personalization via LLM Post-training

Integrating Netflix’s Foundation Model into Personal...

Lost in Transmission: When and Why LLMs Fail to Reas...

A Course Correction in Steerability Evaluation: Reve...

Provably Learning from Language Feedback

Trace is the Next AutoDiff: Generative Optimization ...

Browse Content