AI Case Study
Tumblr has an automatic adult content detection method but has been flagging the wrong images for removal
In adhering to its new platform policy, Tumblr has implemented a system for automatically detecting and removing images it deems as inappropriate (e.g. containing adult content). However, Wired reports that the system has had numerous failings in incorrectly identifying user posts as violating the new policy when they in fact do not.
Industry
Technology
Internet Services Consumer
Project Overview
As reported by Wired, Tumblr has automated the detection of images which may be inappropriate (e.g. adult content). "Image-recognition classifiers—like the one Tumblr ostensibly deployed—are trained to spot explicit content using datasets typically containing millions of examples of porn and not-porn. The classifier is only as good as the data it learned from".
Reported Results
Wired reports on anecdotal evidence that the image classification system is currently flagging images as inappropriate when they are not, much to the annoyance of its users. "Tumblr did acknowledge in a blog post announcing its new rules that “there will be mistakes” as it begins enforcing them. “Filtering this type of content versus say, a political protest with nudity or the statue of David, is not simple at scale,” Tumblr’s new CEO Jeff D’Onofrio wrote."
Technology
"While it’s true that machine learning capabilities have improved dramatically in recent years, computers still don’t “see” images the way humans do. They detect whether groups of pixels appear similar to things they’ve seen in the past. Tumblr’s automated content moderation system might be detecting patterns the company isn’t aware of or doesn’t understand.
When asked, Tumblr didn’t specify what tech it would be using to enforce its new rules for adult content. A source familiar with the company said it’s using modified proprietary technology."
Function
Legal And Compliance
Compliance
Background
Tumblr has recently "announced that it would no longer allow 'adult content'" on the platform. |The announcement was greeted with dismay in the Tumblr community, which has long been a bastion for DIY and non-mainstream porn. But the policy change appears to be having an even wider effect than anticipated. Posts are being flagged that seem to fall well outside Tumblr’s definition of adult content, which “primarily includes photos, videos, or GIFs that show real-life human genitals or female-presenting nipples, and any content—including photos, videos, GIFs and illustrations—that depicts sex acts.” (Users can appeal to a human moderator if they believe their posts were incorrectly labeled as adult content, and nothing will be censored until the new policy goes into effect later this month.)|
Benefits
Data
Microblog posts on Tumblr