Engineers from the US and China have created a prototype of wireless headphones that can be controlled by gestures, touching or swiping your fingers on the skin around the ear. For gesture recognition used data from the microphone, so this can be added to already manufactured headphones using the software updates, the authors of the articlepresented at the conference CHI 2020.
In the field of human-computer interaction is the direction in which the researchers propose to use the human body as a touch surface. For example, we talked about smart watchesthat can recognize the touch to the skin of the hands, and smart glassesthat can recognize the touch to the nose. But although by themselves, these developments can be quite convenient, they do not go beyond the laboratories, because they require to incorporate into the device the individual sensors.
Blue And (Xin Yi) of Tsinghua University together with his colleagues decided to create a method for recognizing gestures on the skin, based on the already used in practice hardware — microphones wireless headphones. They used serial wireless headphones Samsung Gear IconX, connected via Bluetooth to a laptop for data processing.
When running algorithm headphones continuously transmit microphone audio to
a computer for processing. In the first phase with three-layer sound works
a fully connected neural network which analyzes a 180-millisecond segments of audio, and determines whether there is at them a gesture. If the gesture is not detected, the segment is clipped, and if detected, it is passed to another algorithm. Before the second stage, the sound is converted to the Mel-spectrogram, and is then passed to convolutional neural networks DenseNet, dataset pre-trained on ImageNet, and then doubochinski on dataset authors. This neural network works as a classifier that gives the final result — the perfect type of user gesture.