Facebook Patent | Time-of-flight depth sensing for eye tracking

Patent: Time-of-flight depth sensing for eye tracking

Publication Number: 10154254

Publication Date: 2018-12-11

Applicants: Facebook, Inc.

Abstract

A head-mounted display (HMD) includes an eye tracking system that determines user’s eye tracking information based on depth information derived from time-of-flight methods. The eye tracking system includes an illumination source, an imaging device and a controller. The illumination source illuminates the user’s eye with a temporally varying irradiance pattern. The imaging device includes a detector that captures temporal phase shifts (temporal distortions) caused by a local geometry and the illumination pattern being reflected from a portion of the eye. The detector comprises multiple pixels, each pixel having multiple units for capturing, over multiple time instants, light signals related to the temporally distorted illumination pattern. The controller determines phase differences between the temporally distorted illumination pattern and the temporally varying irradiance pattern, based on the captured light signals. The controller determines depth information related to eye surfaces and updates a model of the eye, based on the phase differences.

BACKGROUND 

The present disclosure generally relates to eye tracking, and specifically relates to using time-of-flight based depth information for eye tracking in virtual reality and/or augmented reality applications.

Eye tracking refers to the process of detecting the direction of a user's gaze, which may comprise detecting the angular orientation of the eye in three-dimensional space. Eye tracking may further comprise detecting the location of the eye (e.g., the center of the eye), the torsion (i.e., the roll of the eye about the pupillary axis) of the eye, the shape of the eye, the current focal distance of the eye, the dilation of the pupil, other features of the eye's state, or some combination thereof. One known technique for eye tracking is capturing video images of a user and identifying the orientation of the user's pupils using a machine vision algorithm. However, this technique requires substantial computing resources, and is susceptible to occlusion of the eye by eyelashes and eyelids. Furthermore, this method is dependent on the contrast between the iris and the pupil, which is not invariant across users. Thus, video based pupil tracking may not be able to accurately track the eyes of certain users. Similarly, this technique may place constraints on the proximity of the camera to the user's eye. Furthermore, this technique may perform poorly when the camera is located off the axis of the user's gaze. However, when eye tracking is used in an HMD, it may be preferred that the detection element of the eye tracking system be small, be close to the eye, and be off the axis of the user's gaze.

SUMMARY 

Embodiments of the present disclosure support a head-mounted display (HMD) that comprises an electronic display, an optical assembly, and an eye tracking system. The HMD may be, e.g., a virtual reality (VR) system, an augmented reality (AR) system, a mixed reality (MR) system, or some combination thereof. The electronic display is configured to display content to a user wearing the HMD, and the optical assembly is configured to direct light from the electronic display to an exit pupil of an eye of the user. The eye tracking system includes an illumination source, an imaging device (camera) and a controller coupled to the imaging device or being part of the imaging device. The illumination source illuminates the eye with a temporally varying irradiance pattern producing a controlled illumination pattern on a portion of the eye. The imaging device captures perceived temporal distortions in the illumination pattern (i.e., a temporally distorted illumination pattern) associated with the portion of the eye and a local geometry of the eye. A temporal distortion is defined by a change in a measured phase of an output temporal frequency or signal, both in absolute and relative terms across an imaged area or region. The imaging device includes a detector comprising a plurality of pixels, wherein each pixel is associated with multiple storage units for capturing light signals related to the temporally distorted illumination pattern captured in multiple time instants. The controller determines one or more phase differences between the temporally distorted illumination pattern and the temporally varying irradiance pattern, based on the light signals captured in the storage units during the multiple time instants. Based on the one or more phase differences, the controller calculates one or more distances from the detector to one or more surfaces of the eye. Based on the one or more distances, the controller updates a model of the eye and estimates a position and orientation of the eye based on the updated model.