This Weeks Topic: Hand Tracking

Technically, it does involve a few challenges that are more tricky than tracking controllers. The easiest way to illustrate this is by comparing finger vs hand tracking Oculus Quest. After all, it's one of the few devices available that support both.

The Touch controllers are tracked by utilizing something called Infrared LED matching. It is done by having cameras on the headset itself, that can see infrared LED lights built into the controllers.

Imagine a box with 4 infrared lights on them, here illustrated by two red, and two green dots. The camera sees the object and can distinguish the bright dots, and "count" the distance between them in pixels. The pixels in the below illustration are represented by millimetres.

As you see in the picture above, when the object (1) is close to the camera, the distance between green and red dots is 400mm. The distance is measured on the image itself and does not account for perspective/depth.
Thus, when the object (2) is further away, the distance between the same dots is 85mm. The lights on the object are always fixed, as is the camera's field of view.

Based on this fact, we can accurately estimate the distance of the object to the camera based on how far apart the dots appear. In the right-most object (3) you can see that, from the perspective of the camera, the green dots appear closer together than even object (2).
If we only had the two green lights, it would therefore appear to be further away than object (1).However, if you observe the red dots in object (3) being further apart than object (1), we can correctly deduce that it is in fact closer to camera but rotated on its own axis.

In the case of the Oculus Touch controllers, the large donut on top has many infrared LEDs for the Quest headset to track in relation to itself. You cannot see them with the naked eye, but below you can see an image that shows both normal and infrared spectrum colors.

The dots are not differentiated by colors, but rather by blinking in a specific order that lets the Quest cameras identify each light and comparing the lights in relation to each other.

This way you get a very stable and accurate tracking of the controller's position.

Hand tracking is a bit harder. Rather than having dedicated lights that can be individually identified by blinking them, the hands must be tracked simply by analyzing a video feed of the hands from multiple angles. The computer vision has some sense of depth based on comparing these different angles.

But it mostly relies on estimating each finger position by utilizing a trained model of what gesture a large number of different hands had to make to look like that to the camera. Unlike the constant, known shape of manufactured Touch controllers, human hands come in all shapes and sizes.
People wearing rings, different skin types, and lots of other things can make hands a tricky thing to track.

The technology is luckily maturing fast. The Leap Motion has been fairly impressive for many years, and Oculus is getting closer and closer to that performance. I would dare to say the technology itself is already good enough, but we are still waiting for the tracking to be available for desktop users.

Above is a great example of how good the tracking of the Oculus Quest has become

The anticipation of hand tracking is a big part of the reason why our menus are operated with the avatars finger. That finger will, instead of poking out of the controller, be perfectly matched to your own finger. Not only will it be more intuitive, but it will also feel a lot more natural.

User experience

From a user experience perspective, there is a lot of things that need to be figured out. Since you are touching something that doesn't exist, it is not possible to provide the correct tactile feedback.

If you grab an object, you can have your fingers too far inside it without noticing it. When you decide to let it go and open your fingers wider, your fingers could still technically be inside the model still. It is then our job to figure out if you intent was to let go of the model.

In the video below you can see someone trying to paint on a surface using the finger.Since the surface is not really there, it is hard for the user to feel how much pressure to apply, making it harder to do something that is child's play in real life.

Many people think gloves and suits to provide feedback is the future solution. I am not so sure this will ever happen. Not only is it very cumbersome to put on, but it is also technically more challenging to do right than many realize.
There will probably be a future article about the subject of tactile feedback.

There are also some actions that you do not normally perform with finger tracking that is mapped to buttons in Dimension10 today.
A few examples are teleportation, cross-sectioning, and menu activation. Since, in the case of eye tracking, all you have is your fingers - these actions need to be moved to virtual menu buttons or finger gestures.

The quick menu on your wrist is a step towards providing menu options without physical buttons, and as hand tracking is introduced in Dimension10 you will see more of that. Gestures is a fantastic input method for many, but for some, it is very counter-intuitive. Therefore, we need to find the right balance for the average user vs the super-user.

Rest assured we are taking our time to figure out the best path forward while also keeping compatibility for users who still want to use controllers. The coming generation of VR is only going to be an improvement, and we are excited to take this journey with you.

Until next time, I hope you all have a productive and inspiring time in Dimension10.