The American-Chinese group of engineers unveiled a prototype wristband that reads the position and gestures of the brush. There are four infrared cameras that are directed in the direction of the fingers. The algorithms combine the video from the cameras and counting the current position of all fingers and the palm, say the authors of the article in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.
Tracking of the brush used in VR systems, the user a natural way to touch virtual objects in a contactless control interfaces, and in some other areas. Since this area is already highly developed, there are several standard approaches. Most commonly used tracking with the help of mechanical sensors in the gloves and tracking by video sensors on the helmet, as implemented in the Oculus Quest, or near hand, for example, on a table or on the hand using the Leap Motion controller or similar sensors. The second option is not very convenient and reliable, because the location is not on the body of the user becomes attached to a certain area and should keep a hand in it. And if you mount the sensor on the hand, it will disturb, because it is quite cumbersome and will not be seen bent fingers.
Engineers under the direction of Cheng Zhang (Zhang Cheng) from Cornell University have created worn on the wrist bracelet that is small in size and tracks the position of the brush four sides at once. The bracelet consists of two sections: closest to the brush installed four cameras, uses far infrared range with a resolution of 32 by 24 pixels. Data from the cameras is coming to the Raspberry Pi single Board computer, and he in turn forwards them to a more powerful computer, which employs neural network algorithms.
First, four frames are processed by a convolutional neural network ResNet-44, and then supplied to the last polnotsvetnyj layer, which essentially combines the frames and acts as a decoder. At the output of the algorithm provides a model set of 21 points, related to each other and having a certain position.
The developers revealed that the bracelet allows to determine the position of the brush even if a person is holding an object. Test on 11 volunteers showed that the average error of determining the position of the segments among themselves is 8.06 degree.
An unusual approach to the determination of the position and movement of the brush use CTRL-Labs, which in 2019 engulfed Facebook. Its engineers have developed a prototype bracelet that reads muscle activity with multiple electrodes. It allows you to capture the user’s intentions, even when he is very weak straining muscles and hardly moves fingers.