American developers have created a method that allows you to type text, drawing his finger in the air. They developed an algorithm that uses data from a stereo camera or a glove with sensors to track your finger tip and handwriting recognition. The method can be applied to input data when using the VR helmet, say the authors of the articlepresented at the conference CV4ARVR 2020.
Typically, in VR-helmets used to manage a controller which you hold in your hand or both hands. With it, he can point to the objects of interest and interact with them using the buttons. For most interactions this interface is quite suitable and does not cause strong discomfort. But if the user needs to enter text, for example, to search for his desired movie or video, this method works much worse because he has to hover over the individual letters and click them.
Ejo Yang (Yezhou Yang) and his colleagues from Arizona state University have developed a method of continuous handwriting input for VR helmets, which makes typing more natural. They chose two devices that can be used for recognition. One of them is the glove, which at the end of the index finger mounted accelerometer and gyroscope to track the trajectory. The second device is the Leap Motion controller, consisting of two infrared cameras and infrared light. It independently detects the current position of all the parts of the hand and gives the computer data is ready. In both cases, the user when you enter a word or combination of characters you just have to draw them with your finger in the air.
After the device collected data about the trajectory classification algorithm based on convolutional neural networks or support vector machines depending on the task compares the user entered data, data from dataset. The developers have tested the algorithms on several tasks, including handwriting recognition on the set of 210 words. For English recognition accuracy amounted to 79.7 per cent when using an infrared controller and 78.5 when using gloves. However, results for Chinese was higher than that: 87.4 and 83.4 percent, respectively.
The developers have published on GitHub the datasets and code libraries for recognition in Python. The article also notes that along with the code must be published and the technical details of the algorithms.
Recognition of handwritten words in a VR helmet could potentially be used with existing devices. For example, Oculus Quest last year received an update that adds the ability to track in real time the position of the hands and all fingers.