AI Case Study

Researchers at the University of Cambridge decode passwords by listening to finger taps on mobile phone screens

Researchers at the University of Cambridge have developed a machine learning system that records the sound signals of mobile phone screen taps to decode passwords. The team of researchers fed the recorded taps in different frequencies into the machine learning classifier and when tested it showed (worrying) results. The model accurately predicted single digits, PINs after 10 attempts and passwords including letters and words, outperforming random guesses.


Project Overview

"Researchers describe a novel attack that recovers characters typed on a virtual keyboard from sounds generated by finger taps.

The team’s approach employs an app that recovers the sounds of taps and correlates them with keystrokes, using a machine learning algorithm that’s trained offline and tuned to a particular smartphone or tablet model. Architecting the algorithm required overcoming a significant engineering challenge: It needed to be able to account for the interfering vibrations produced by tapping fingertips. In the end, the researchers cross-correlated the feedback sound to disambiguate it from the vibration feedback, and subtracted out the vibration data.

With a model in hand, they set about calculating the time difference between the reception of the sound signals on the dual-mic devices they tested: LG’s Nexus 5 and Samsung’s Nexus 9. Roughly 70 percent of the recorded taps — which were in the frequency ranges 1,300-1,700Hz, 8000-8500Hz, 4000-4400 Hz, and 60-70 Hz — were fed into a machine learning classifier, while the remaining 30 percent were reserved for testing.

To validate their approach, the researchers developed an Android app that had users enter letters, words, and digits into fields while it collected audio through the on-device microphones. About 45 test subjects used it in environments with a fair amount of ambient noise, including a common room, a reading room, and a library.

Ten participants were asked to press each of nine digits (1 to 9) ten times in a random ordered, and 10 others were told to type 200 unique four-digit PINs. A third group was instructed to type letters (also randomly ordered), and a fourth was told to type five-character words from an open source data set."

Reported Results

"The researchers report that, with two microphones, the model correctly predicted single digits three times better than a random guess in the worst case and 100 percent of digits in the best case. Moreover, it recovered 54 percent of PINs after 10 attempts and 91 out of 150 four-digit PINs in 20 attempts. Where letters and words were concerned, it outperformed a random guess by a factor of three with a single microphone. More alarmingly, it managed to recover seven words on the Nexus 5 and 19 on the Nexus 9 in 27 passwords within 10 attempts."



Information Technology



Acoustic attacks targeting keyboards aren’t new, the researchers note — previous studies have investigated the use of mics to identify physical keys by their unique physical characteristics or defects. But soft keyboards naturally make for more difficult targets, because each tap happens on the same surface.



Recorded taps in different frequencies