AI Case Study

Netflix increases quality control efficiency by using machine learning to predict which video assets are likely to fail

Netflix's customer base and product offering is growing globally. In order to ensure customer satisfaction, quality control (QC) is undertaken to identify product assets (e.g. video) which are defective. Using AI predictive modelling reduces the need for manual checks quality control checks, thereby increasing efficiency.

Industry

Consumer Goods And Services

Entertainment And Sports

Project Overview

From the Netflix Tech Blog: "We looked at the data on manual QC failures and observed that certain factors affected the likelihood of an asset failing QC. For example, some combinations of content and fulfillment partners had a higher rate of defects for certain types of assets. Metadata related to the content also showed patterns of failure... These types of factors were used to build a machine learning model that predicts the probability that a delivered asset would not meet the Netflix quality standards."

Reported Results

From the Netflix Technical Blog: "A predictive model to identify defective assets helps in two significant ways:

* Scale the content QC process by reducing QC effort on assets that are not defective.
* Improve member experience by re-allocating resources to the discovery of hard-to-find quality issues that may otherwise be missed due to spot checks."

Technology

Netflix Technical Blog: "Using results from past manual QC checks, a supervised machine learning (ML) approach was used to train a predictive quality control model that predicts a “fail” (likely has content quality issue) or “pass.” If an asset is predicted to fail QC, it is sent to manual QC... Given that only a small fraction of the delivered assets are defective, one of the main challenges is class imbalance in the training data, i.e. we have a lot more data on “pass” assets than “fail” assets. We tackled this by using cost-sensitive training that heavily penalizes misclassification of the minority class (i.e. defective assets)."

Function

Information Technology

Quality Assurance

Background

Netflix's video offerings come from a variety of suppliers and the quality of video, audio and text must meet rigorous standards. Developing a predictive model to identify which assets are likely to fail a quality control inspection improves the efficiency of the manual quality control process.

Benefits

Data