In a [previous blog post](https://netflixtechblog.com/artwork-personalization-c589f074ad76) we explained how our artwork personalization algorithm can pick the best image for each member, but how do we create a good set of images to choose from? What data would you like to have if you were designing an asset suite?

In this blog post, we talk about two approaches to create effective artwork. Broadly, they are:

The top-down approach, where we preemptively identify image properties to investigate, informed by our initial beliefs.
The bottom-up approach, where we let the data naturally surface important trends.

Discovering Creative Insights in Promotional Artwork

We introduce Magenta Green Screen, a novel machine–learning enabled matting technique for recording the color image of a foreground actor with a simultaneous high-quality alpha channel without requiring a special camera or manual keying techniques. We record the actor on a green background, but light them with only red and blue foreground lighting. In this configuration, the green channel shows the actor silhouetted against a bright, even background, which can be used directly as a holdout matte, the inverse of the actor's alpha channel. We then restore the green channel of the foreground using a machine learning colorization technique. We train the colorization model with an example sequence of the actor lit by white lighting, yielding convincing and temporally stable colorization results. We further show that time-multiplexing the lighting between magenta green screen and green magenta screen allows the technique to be practiced under what appears to be mostly normal lighting. We demonstrate that our technique yields high-quality compositing results when implemented on a modern LED virtual production stage. The high-quality alpha channel data obtainable with our technique can provide significantly higher quality training data for natural image matting algorithms to support future ML matting research.

Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization

Audiovisual representation learning typically relies on the correspondence between sight and sound. However, there are often multiple audio tracks that can correspond with a visual scene. Consider, for example, different conversations on the same crowded street. The effect of such counterfactual pairs on audiovisual representation learning has not been previously explored. To investigate this, we use dubbed versions of movies to augment cross-modal contrastive learning. Our approach learns to represent alternate audio tracks, differing only in speech content, similarly to the same video. Our results show that dub-augmented training improves performance on a range of auditory and audiovisual tasks, without significantly affecting linguistic task performance overall. We additionally compare this approach to a strong baseline where we remove speech before pretraining, and find that dub-augmented training is more effective, including for paralinguistic and audiovisual tasks where speech removal leads to worse performance. These findings highlight the importance of considering speech variation when learning scene-level audiovisual correspondences and suggest that dubbed audio can be a useful augmentation technique for training audiovisual models toward more robust performance.

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

To continue the story of Dataflow for workflow asset management in the Netflix data ecosystem, we’d like to introduce readers to the data pipeline project management and to ready-to-go sample workflows.

Ready-to-go sample data pipelines with Dataflow

In this blog post, we introduce RecSysOps a set of best practices and lessons that we learned while operating large-scale recommendation systems at Netflix. These practices helped us to keep our system healthy while 1) reducing our firefighting time, 2) focusing on innovations and 3) building trust with our stakeholders.

RecSysOps: Best Practices for Operating a Large-Scale Recommender System

This writeup is about using reinforcement learning to construct an optimal list of recommendations when the user has a finite time budget to make a decision from the list of recommendations.

Reinforcement Learning for Budget Constrained Recommendations

Netflix: A Culture of Learning 

Image for Culture of Learning on Carousel

Beyond Customer Lifetime Valuation: Measuring the Value of Acquisition and Retention for Subscription Services

Bellmania screen shot for carousel for Allen & Reza paper

Image for carousel for Discovering Creative Insights in Promotional Artwork

Building In-Video Search

Image from Building In-Video Search blog post

Research at Netflix aims to improve various aspects of our business. Research applications span many areas including our recommendations, content valuation, streaming optimization, and user insights. To maximize the impact of our research, we do not centralize it into a separate organization. Instead, we have many teams that pursue research in close collaboration with business teams, engineering teams, and other researchers. Research that applies to the same methodological area or business area is shared and highlighted in discussion and debate forums to strengthen the work and its impact. These forums also serve to identify and motivate future research directions.

Researchers at Netflix love working in an unique environment enabled by the Netflix Culture that values curiosity, courage with smart risks, innovation, science, rigor, and high impact. Across the company, we strive to run experiments to back up our hypotheses with evidence, which often uncover surprises that redirect or refine our research. We relish the freedom to try new ideas and the opportunity to debate their implications with colleagues spanning all parts and levels of the company.


What is Netflix Research?

For more information on the different Research areas at Netflix, including papers and talks, click on any of the areas below.

Our Areas of Expertise

Analytics

Seeing Beyond Pixels: Revolutionizing Perception with Computer Vision and Graphics

Recent Articles

FM-Intent: Predicting User Session Intent with Hierarchical Multi-Task Learning

Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems

Equilibria under Dynamic Benchmark Consistency in Non-Stationary Multi-Agent Systems

Part 3: A Survey of Analytics Engineering Work at Netflix

Part 2: A Survey of Analytics Engineering Work at Netflix

Part 1: A Survey of Analytics Engineering Work at Netflix

Job Postings

What is Netflix Research?

Our Areas of Expertise

For more information on the different Research areas at Netflix, including papers and talks, click on any of the areas below.

Analytics

Computer Vision and Graphics

Consumer Insights

Encoding & Quality

Experimentation & Causal Inference

Machine Learning

Machine Learning Platform

NLP & Conversations

Recommendations