AI Case Study
Facebook's Marketplace achieves a nearly 100% increase in consumer engagement using computer vision and natural language processing
Facebook has leveraged computer vision and natural language processing to enhance the customer's experience on Marketplace. The company has built a product index that draws information both from the text accompanying a product as well as the photos, through their image and text understanding systems. As a result, when searching for products, the recommended results are ranked based on a content retrieval system. This has enabled Facebook's Marketplace to achieve a nearly 100% increase in consumer engagement.
Internet Services Consumer
"Building a product index:
Visitors to Marketplace seamlessly interact with these AI-powered features from the moment they arrive to buy or sell something. From the very first search, results are recommended by a content retrieval system coupled with a detailed index that is compiled for every product. Since the only text in many of these listings is a price and a description that can be as brief as “dresser” or “baby stuff,” it was important to build in context from the photos as well as the text for these listings. In practice, we’ve seen a nearly 100 percent increase in consumer engagement with the listings since we rolled out this product indexing system.
To accomplish this, we built a multimodal ranking system that incorporates Lumos, our image understanding platform, and DeepText, our text understanding engine. As a result, the index for each product includes layers for both the text and the images. To model the word sequence, we map each word from the title and description to an embedding and then feed them into the convolutional neural networks. The image portion comes from a 50-layer ResNet encoder pretrained on the image classification task. The output of the image encoder and the text embedding are concatenated and sent into a multilayer perceptron, where they can be trained and stored together as the complete product model.
Using more similarity searches and Lumos, we recently launched another feature to recommend products based on listing images. So if a buyer shows interest in a specific lamp that is no longer available, the system can search the database for similar lamps and promptly recommend several that may be of interest to the buyer."
"In practice, we’ve seen a nearly 100 percent increase in consumer engagement with the listings since we rolled out this product indexing system."
"To model the word sequence, we map each word from the title and description to an embedding and then feed them into the convolutional neural networks."
"Facebook Marketplace was introduced in 2016 as a place for people to buy and sell items within their local communities. Today in the U.S., more than one in three people on Facebook use Marketplace, buying and selling products in categories ranging from cars to shoes to dining tables. Managing the posting and selling of that volume of products with speed and relevance is a daunting task, and the fastest, most scalable way to handle that is to incorporate custom AI solutions."
Text and photos