Eyetracking and the potential it brings in next-generation VR

Eyetrack
Published 13/05/2020 by Christer-André

As more and more people become aware of the possibilities of current-generation VR, the demand also grows for improvements in technology. Some of the most obvious and frequently requested improvements include;

  • Smaller form factor/Better comfort
  • Higher Resolution/better picture quality
  • Higher Field Of View
  • Wireless support

The list goes on- and luckily, we will see vast improvements in all the major areas over the coming years. Today I will be writing about eye-tracking, and its potential to improve your experience as a user.

Like what our name Dimension10 alludes to, we are seeking to add extra dimensions to people's communication and interactions. When you meet a colleague in Dimension10, it is supposed to feel like you are there and not only equal but surpass real-life meetings.

Eye-tracking is one of the tools that will allow us to further improve on the experience, below is a breakdown of how that will happen.

Both in a direct way (by adding functionality) and indirectly (by improving performance etc). We will also touch upon what eye-tracking technology can offer in the future, with some functionally the current generation most likely will not offer.

Benefits

Better communication

In the coming months, D10 will continue working towards next-generation avatars for VR. The first step in this improvement will be to add pupils to the avatar, in order for users to get a better connection with their conversational partners through realistic eye contact.

This adds a whole new dimension to the conversation, both in very obvious practical ways (one example being seeing more accurately who someone is addressing while speaking) and more subtle (like making it feel slightly more like you are talking to an actual person)

The first iterations will look more like our current avatars with some addons, however, our end goal is that each user will look exactly like themselves with realtime photo-realistic facial expressions. You will be reading more about this in future posts.

Performance Improvements

Over the coming years, VR headsets will get increased resolution/sharpness, higher framerates, wider field of view, and ultimately things like depth of focus. Since we also want higher graphical fidelity rendered by the computer, simply put more realistic graphics, it is safe to say that it will be exponentially more demanding on the computer hardware. Foveated Rendering builds on the fact that our peripheral vision is dramatically less sharp than our central vision.

Our central vision, where we see the sharpest, makes up a mere 7degrees out of the roughly 120 that the human eye perceives (albeit very blurry) in our peripheral vision. Once we have eye-tracking, we can dynamically detect where on the screen you are looking at any given time, enabling us to detail/graphical fidelity in areas you cannot perceive detail in the first place. Nvidia has a good video detailing some of their solutions that we will be looking to implement this year.

Not only is this beneficial for the graphical rendering/processing power required, but dramatically reduced the need for bandwidth in image transfer.

This means significantly higher quality wireless streams, potentially even allowing for an ultra-high-resolution experience over 5G - eliminating the need to even be in the same building as the computer while getting the full VR experience.

The user could achieve what is perceived to be an 8K resolution per eye minimum, using only two 720p streams per eye.

Heatmaps

From the level of accuracy possible with existing tracking technology, we can produce heatmaps of anything from door handles to street signs, entire buildings or even larger.

By collecting analytics of where users are looking more than others, we can help pinpoint uninspected areas of a model, give data on what is more engaging to look at for customers, or anything you can imagine using such data for.

It can both be visualized as colored areas in vr, or exported as different types of analytics data.

Security

Unlike "eye-scanners" used for unlocking devices or physical doors, eye-tracking in VR will become a much more reliable way of secure interaction for a specific user.

The reason why its harder to cheat with a "copy" of the eye (such as a picture or a physical replica) Is that in VR the user can be prompted to look at very specific points in the scene, requiring the user to move the eyes dynamically while it's being scanned. This is hard if not impossible to pull off with an artificial eye.

This also means effortless login for personal accounts for those who want this in the future.

A challenging task

Human eyes have been surprisingly hard to track accurately and has traditionally been a very expensive technology to incorporate into a computer interface.

The tracking is done by computer vision, which is provided by infrared camerasand infrared illumination fed to a computer which then analyses the footage. After a calibration process, which prompts you to look at dots appearing on the screen, the software will track the position of the pupils relative to each other. The most common eye-tracking interfaces have been either built into monitors/laptops or attached as a peripheral.

One example is the Tobii 4C as seen below

Pasted image 0

Tobii is one of the leading companies developing eye-tracking devices for multiple use cases. Eye-tracking has been around for many years, but for VR-usage it is still fairly new. Luckily tho, building eye-tracking into VR-headsets is actually in many ways a lot simpler than almost any other case such as peripheral devices attached to monitors or similar.

In VR-headsets, the cameras and IR-LEDs are located around the rim of the lens, just a few millimeters away from your eyes. They therefore move with the headset and thus the users head/eyes, conveniently keeping the cameras directly in front of each of the users eyes at all times.

This also means each eye completely fills the camera’s view, significantly improving how much detail is captured. This is hugely beneficial compared to cases where the camera is sitting on a monitor, a few feet away from the users head. The users can move their heads independently - back and forth, looking away and so on - Making for a lot more complex tracking tasks.

The tracking works by simply observing the position of the pupils in relation to each other. If the pupils are closer together, - or converging, that means you are looking at something close, if the pupils are up you are looking up and so on.

Front view of adults eyes focusing on object normally 403209

That is not to say that eye-tracking in VR does not come without its challenges.

Some eyelids partially cover the pupil at all times, some have one eye-shaped differently than the other, and things such as glasses or contact lenses can cause reflections that confuse the cameras.

Furthermore, the iris in your eye is soft and squishy and not rigid. You can see this phenomenon in the slow-motion footage below of someone moving their eye. The iris is floating somewhat independently and “wobbling"

That being said, the tracking technology is mature enough that we can safely say it will vastly improve the experience for more than 90% of our users.

In the future the eye-tracking technology will make adjustments in dioptre/IPD, correcting vision automatically, and adapting to individual users. This will eliminate the need for glasses or contact lenses.

In addition, we will see great leaps forward in realism through varifocal displays, a concept that will deliver depth of focus and deserves its own article.

If we sum up all these benefits, it's actually impressive how many areas it will improve. Increased comfort, big leaps in performance, better mobility, enhances human to human communication, and even higher security.

D10 will continue to explore new improvements and concepts in more articles in the future.