This Weeks Topic: Face Tracking

1589901045631268550342024883074

How face-tracking will improve your experience in Dimension10

Published 19/05/2020 by Christer-André

Face tracking

Last week I wrote about eye-tracking, one of the stepping stones for next-generation VR/AR. In that article, I briefly talked about face-tracking as one of the critical steps towards next-generation VR. Unlike eye-tracking, which brings a multitude of improvements and new features, face-tracking has one single purpose;
More natural communication.

However, having a single purpose doesn't mean it is any less impactful in how you will interact with and experience your projects in Dimension10.

Today I will be talking about what face-tracking will mean to you as an end-user.

Being there


We are proud of our work at Dimension10, bringing people together and allowing for smarter collaboration. Every day we work towards making you forget you are using devices and software, allowing you to just focus on "being there", interacting and focusing on the task at hand.

One exciting example of the excellent sense of presence we have achieved, is the fact that many of our users instinctively put their hand out before and after meetings. Shaking each other's hands is a standard greeting and parting tradition, and even tho the person at the
other end is represented by a neon-colored transparent avatar - it feels like human interaction.

That being said, it does occur in VR that a person holds out their hand while the
other look at it in confusion.
The present situation excluded, it would rarely happen in real life that a handshake gesture is
unreciprocated.

There are numerous reasons for this, one is the fact that some are too aware of the fact that
they are already holding an object in their hands (controllers).Hand-tracking will remedy this, and we will talk about hand tracking in a separate article soon. Then there is the lack of eye and face-tracking, in addition to a need for photorealistic avatars.
The latter is not strictly required to achieve this goal - your facial expressions can be translated to another avatar- but will most likely be preferred by most.

All of which will combine, and convey the subtleties of personal expression and
body language in a significantly more convincing way.

Research from Facebook

In the above video, you can see how convincing Facebook's VR avatars have become already.
Left is the actual people talking, right side is their virtual faces. This is despite the paradox hat the thing you are trying to track (face), is covered by the thing trying to track it (headset).
By putting multiple cameras at outer edges pointing at areas around eyes,
plus cameras underneath headset pointing at mouth/cheeks and some clever computation/extrapolation,
it is possible to manipulate an existing 3D-scan of the persons face.

Kamera

Based on this, one of my personal milestones is to get to a level where absolutely everyone reacts
properly at things like a hand being held out and similar - without hesitation. We will achieve this by implementing improvements to all of the above aspects,
as soon as available hardware allows for it in the not so distant future.


Dimension10 3D-scanning


In my own eagerness to achieve photorealistic avatars in VR, I actually built a human size 3D-scanner with more than 100 DSLR cameras. It was capable of scanning a standing adult in milliseconds.That whole endeavor is a story of its own.

Image

Below you can see an early prototype scan of my face.
3D-scanned faces that can mimic your likeness will be possible using simple devices such as a
normal handheld camera or even your phone.

D10 3D Scan example

In conclusion, face-tracking is fantastic and you can expect future sessions in Dimension10 to further close the gap between VR and Reality. It will feel more natural to both new and experienced users, and even the most subtle changes in peoples facial expression will be relayed.