AI Case Study

Rosetta Stone identifies objects for real-time translation using object recognition technology

Rosetta Stone has enhanced its translation app with machine learning and object identification technology to allow its users to name objects in different languages in real time. App users can point their phones to objects and see the vocabulary associated with the objects the phone camera detects.

Industry

Project Overview

"Language teaching developer Rosetta Stone is taking the next step forward by adding augmented reality and machine learning to its iPhone app, enabling users to identify and translate the names of real-world objects — the first time object recognition technology has been used in a language app in this manner.

The machine learning technology is a new component of Rosetta Stone’s Dynamic Immersion feature, which lets the user simulate being in a foreign language environment, surrounded by new words, phrases, and native speakers’ voices. A scavenger hunt-like game called Seek & Speak has the user point the iPhone camera at an object and see the vocabulary word in a chosen language, then practice a conversation using the word. Seek & Speak is launching in beta for English, French, German, Italian, and Spanish, with plans to add additional languages and challenges during 2019.

These five languages are also getting two more improvements: Embedded Translations, enabling Dynamic Immersion users to long-press any word during the experience to reveal its translation, and an updated Phrasebook — an offline collection of over 164 spoken phrases that can be used in common situations. Rosetta Stone also says it’s using machine learning to continuously improve the performance of its TruAccent speech recognition feature, leveraging thousands of hours of recorded speech data to help users match their pronunciation of words to that of native speakers."

Reported Results

Users are now able to get translation of objects in different languages, in real time.

Technology

Function

R And D

Product Development

Background

"Historically, language translation apps were confined to whatever developers could store in their on-board databases, limitations that were surpassed with internet connections and AI-assisted text recognition tools."

Benefits

Data

live images