AI Case Study
Yahoo! researchers investigate optimising click-through rates by predicting optimal news story to feature online
Yahoo! researchers create and trial a method of trialling main page news stories to optimise clickthrough rate without having to implement them online first, a disruptive and more costly option.
Consumer Goods And Services
Media And Publishing
"To draw visitors’ attention, Yahoo! would like to rank available articles according to individual interests, and highlight the most attractive article for each visitor at the story position.
This paper studies an offline evaluation method of bandit algorithms that relies on log data directly rather than on a simulator. Common practice is to create a simulator which simulates the online environment for the problem at hand and then run an algorithm against this simulator. However, creating simulator itself is often difficult and modeling bias is usually unavoidably introduced." Recommendation algorithms should be tested before implemented, but directly experimenting on users can cause disruption, and could fail. In order to then test a new recommendation algorithm, the Yahoo! researchers investigate using offline data from previous recommendation algorithms to evaluate how well the new one will work.
"The Today Module is the most prominent panel on the Yahoo! Front Page, which is also one of the most visited pages on the Internet. The default “Featured” tab in the Today Module highlights four high-quality news articles, selected from an hourly-refreshed article pool maintained by human editors. A user can click on the highlighted article at the story position to read more details if interested in the article. The event is recorded as a story click."
The "evaluation method provides a solution that is accurate (like bucket tests) without the cost and risk of running the policy in the real system. These encouraging results suggest the usefulness of our evaluation method, which can be easily applied to other related applications such as online refinement of ranking results and ads display".
Contextual bandits (in this case, the context is user features
including age, gender, location).
"For offline evaluation, millions of events were collected from a 'random bucket”'from Nov. 1, 2009 to Nov. 10, 2009. In the random bucket, articles are randomly selected from the article pool to serve users. There are about 40 million events in the offline evaluation data set, and about 20 articles available in the pool at every moment. We focused on user interactions with the story article at the story position only. The user interactions are recorded as two types of events, user visit event and story click event."