Meta Patent | Binary-encoded illumination for corneal glint detection
Patent: Binary-encoded illumination for corneal glint detection
Patent PDF: 加入映维网会员获取
Publication Number: 20230055268
Publication Date: 2023-02-23
Assignee: Meta Platforms Technologies
Abstract
An apparatus, system, and method for detecting glints includes a pair of light sources positioned on a head-mounted frame and driven to illuminate a corneal surface of an eye with binary-encoded differential light signals. An image sensor is positioned on the head-mounted frame to receive reflections of the differential light signals from the corneal surface. An orientation of a cornea (e.g., corneal sphere) of the eye may be determined at least partially based on the reflections of the binary-encoded differential light signals.
Claims
What is claimed is:
1.A glint detection system comprising: a plurality of pairs of light sources positioned on a head-mounted frame to illuminate a corneal surface of an eye with differential light signals; an image sensor positioned on the head-mounted frame to receive reflections of the differential light signals from the eye; processing logic coupled to the plurality of pairs of light sources and to the image sensor; and one or more memories coupled to the processing logic, the one or more memories storing instructions that, when executed by the processing logic, cause the glint detection system to perform operations comprising: encode the differential light signals with binary identifiers that are unique to each pair of the plurality of pairs of lights; receive, from the image sensor, image data that is representative of reflections of the differential light signals; and identify each of the plurality of pairs of light sources within the image data at least partially based on the binary identifiers.
2.The glint detection system of claim 1, wherein the one or more memories store further instructions that when executed by the processing logic, cause the glint detection system to perform further operations comprising: identify locations, within the image data, of the reflections of the differential light signals; and determine a location of a corneal sphere of the eye at least partially based on the locations.
3.The glint detection system of claim 1, wherein the binary identifiers are binary numbers having at least 2 bits.
4.The glint detection system of claim 1, wherein the image sensor is responsive to changes in brightness in a field of view of the image sensor, wherein the image data is representative of the changes in brightness in the field of view of the image sensor.
5.The glint detection system of claim 1, wherein the image sensor is an event camera that is responsive to changes in brightness in a field of view of the image sensor.
6.The glint detection system of claim 1, wherein the differential light signals from each pair of the plurality of pairs of light sources represent binary-encoded square-wave patterns of opposite polarity.
7.The glint detection system of claim 1, wherein the plurality of pairs of light sources emit infrared light.
8.The glint detection system of claim 1, wherein the plurality of pairs of light sources emit the binary identifiers at an illumination frequency of at least 1 kHz.
9.The glint detection system of claim 1, wherein an intra-light source angle between each light source in each pair of the pairs of light sources is at least 2.5 degrees.
10.The glint detection system of claim 1, wherein the plurality of pairs of light sources are light emitting diodes (LEDs), vertical external-cavity surface-emitting lasers (VCSELs), fiber optics, or out-coupling gratings.
11.The glint detection system of claim 1, wherein the one or more memories store further instructions that when executed by the processing logic, cause the glint detection system to perform further operations comprising: identify first bits of the binary identifiers of glints detected at first locations; confirm second bits of binary identifiers of the detected glints around the first locations; and associate detected glints with corresponding ones of plurality of pairs of light sources based on the binary identifiers.
12.The glint detection system of claim 11, wherein the one or more memories store further instructions that when executed by the processing logic, cause the glint detection system to perform further operations comprising: update a location of a cornea sphere after first bits and second bits of binary identifiers have been confirmed and associated with corresponding ones of the plurality of pairs of light sources.
13.A head-mounted display comprising: a frame; a display positioned on the frame to provide display light; a lens assembly positioned on the frame to transmit the display light from the display to an eyebox; and a glint detection system comprising: a plurality of pairs of light sources positioned on the frame to illuminate a corneal surface of an eye with differential light signals; an image sensor positioned on the frame to receive reflections of the differential light signals from the eye; processing logic coupled to the plurality of pairs of light sources and to the image sensor; and one or more memories coupled to the processing logic, the one or more memories storing instructions that, when executed by the processing logic, cause the glint detection system to perform operations comprising: encode the differential light signals with binary identifiers that are unique to each pair of the plurality of pairs of lights; receive, from the image sensor, image data that is representative of reflections of the differential light signals; and identify each of the plurality of pairs of light sources within the image data at least partially based on the binary identifiers.
14.The head-mounted display of claim 13, wherein the image sensor is an event camera that is responsive to changes in brightness in a field of view of the image sensor.
15.The head-mounted display of claim 13, wherein the differential light signals from each pair of the plurality of pairs of light sources represent binary-encoded square-wave patterns of opposite polarity.
16.The head mounted display of claim 13, wherein the differential light signals include square wave patterns that repeat at a frequency of at least 1 kHz.
17.A method of corneal glint detection using binary-encoded illumination comprising: driving a plurality of pairs of light sources to emit differential light signals towards an eye of a user; encoding the differential light signals with binary identifiers, wherein each of the binary identifiers is unique to a corresponding pair of the plurality of pairs of light sources; receiving image data from an image sensor, wherein the image data is representative of reflections of the differential light signals off of the eye of the user; and identifying each of the plurality of pairs of light sources within the image data at least partially based on the binary identifiers.
18.The method of claim 17 further comprising: identifying locations, within the image data, of the reflections of the differential light signals; and determining an orientation of a cornea of the eye at least partially based on the locations.
19.The method of claim 18, wherein the differential signals for each pair of the plurality of pairs of lights sources include square-wave patterns having opposite polarities.
20.The method of claim 17, wherein the binary identifiers are binary numbers having at least 2 bits.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority to U.S. provisional Application No. 63/234,621 filed Aug. 18, 2021, which is hereby incorporated by reference.
TECHNICAL FIELD
This disclosure relates generally to glint detection, and in particular to illumination for glint detection systems.
BACKGROUND INFORMATION
Eye tracking inaccuracies can undermine a user's trust in a system that relies on an eye tracking system. For example, if an eye tracking system is used for a head-mounted display, inaccurate eye tracking could make the system less enjoyable to use or less functional.
BRIEF DESCRIPTION OF THE DRAWINGS
Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.
FIG. 1 illustrates a glint detection system, in accordance with aspects of the disclosure.
FIG. 2 illustrates an ocular environment that includes example implementations of a glint detection system, in accordance with aspects of the disclosure.
FIG. 3 illustrates an example graph of potential metrics that are based on light source spacing in the disclosed glint detection system, in accordance with aspects of the disclosure.
FIG. 4 illustrates a flow diagram of a process for acquiring and using glint locations, in accordance with aspects of the disclosure.
FIG. 5 illustrates a timing diagram of binary encoding for complementary light sources, in accordance with aspects of the disclosure.
FIG. 6 illustrates a flow diagram of a process for tracking illumination patterns in a glint detection system, in accordance with aspects of the disclosure.
FIG. 7 illustrates a flow diagram of a process for detecting corneal glints using binary-encoded illumination, in accordance with aspects of the disclosure.
FIG. 8 illustrates a head mounted display, in accordance with aspects of the disclosure.
DETAILED DESCRIPTION
Embodiments of a glint detection system using differential illumination are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In aspects of this disclosure, visible light may be defined as having a wavelength range of approximately 380 nm to 700 nm. Non-visible light may be defined as light having wavelengths that are outside the visible light range, such as ultraviolet light and infrared light. In aspects of this disclosure, red light may be defined as having a wavelength range of approximately 620 to 750 nm, green light may be defined as having a wavelength range of approximately 495 to 570 nm, blue light may be defined as having a wavelength range of approximately 450 to 495 nm, and infrared light may be defined as having a wavelength range of approximately 700 nm to 1 mm.
In aspects of this disclosure, a glint may be defined as a corneal reflection that is a virtual image of a light source (e.g., infrared) that illuminates the eye and that is created by the outward facing surface (i.e., corneal surface) of the cornea. The corneal reflection is a specular reflection off of the corneal surface, which acts as a convex mirror.
Embodiments of the present disclosure include a glint detection system that uses binary-encoded differential illumination to improve glint identification while reducing saturation of images caused by the illumination of diffuse surfaces, such as skin, an iris, or a sclera. When a system is configured to use corneal reflections to determine the orientation of the eye, a better signal-to-noise ratio can result in more accurate detection of glints and therefore a more accurate identification of gaze orientation. Depending upon the application, accurate identification of gaze orientation can provide a more reliable and enjoyable user experience. For example, a glint detection system that is integrated into an eye tracking system that is part of a head-mounted display may be used to adjust a focal point of a display, may be used to customize user interface elements, and/or may be used to interact with or control applications run on the head-mounted display.
A glint detection system may include one or more pairs of light sources, an image sensor, and glint detection logic, according to an embodiment. The glint detection logic may be communicatively coupled to the pairs of light sources and to the image sensor. The glint detection logic may drive the pairs of light sources to emit binary-encoded differential light signals. The binary-encoded differential light signals transmit binary identifiers that identify the light source from which the binary-encoded differential light signals are emitted.
The binary-encoded differential light signals may be switched in opposition polarities and cause reflections on the cornea. Advantageously, by switching in opposite directions, the binary-encoded differential light signals cause diffuse surfaces to maintain a net illumination that is approximately constant.
The image sensor (e.g., an event camera) can be configured to identify changes in an image scene while ignoring portions of the image scene that remain the same. Without using differential illumination, driving a light source with a square wave can cause the image sensor to saturate, so that glints are barely detectable or undetectable. By generating glints with the binary-encoded differential light signals, the image sensor detects changes in the image scene as events that represent the corneal glints with strong signal-to-noise ratios. The image sensor transmits the captured events to the glint detection logic as image data, to identify locations of the glints in the image scene (on the corneal surface). The captured events enable the glint detection logic to identify the binary identifier that is encoded in the glint.
The glint detection system may be used in an eye tracking system that is incorporated into a head-mounted display, according to an embodiment. The eye tracking system may use the glint locations and binary identifiers to determine (e.g., calculate) a gaze vector or a gaze orientation. Corneal sphere location and/or gaze orientation may be determined, for example, using corneal sphere regression. In corneal sphere regression, corneal glints may be modeled as ideal specular reflections on a perfect sphere. Under this model, there is exactly one ray which passes from each light source to the corresponding glint and from each glint to the corresponding point on the image plane (e.g., on a pixel array), according to an embodiment. By fixing the locations of light sources in the image plane with calibration, corneal sphere dimensions and location may be determined. The gaze orientation may be provided to the head-mounted display to enable the head-mounted display to personalize a user's experience based on the user's gaze orientation.
The apparatus, system, and method for a glint detection system that uses binary-encoded differential illumination is described in this disclosure and enables improved determination of corneal sphere orientation and eye tracking in, for example, a head-mounted display. These and other embodiments are described in more detail in connection with FIGS. 1-8.
FIG. 1 illustrates an example of a glint detection system 100, according to an embodiment of the disclosure. Glint detection system 100 is configured to drive a binary-encoded differential illumination pattern onto a pair of light sources to generate specular reflections of the corneal surface of an eye (i.e., glints) while concurrently generating an approximately constant net illumination from diffuse portions of a user's eyes and face. Binary-encoding the specular reflections enables identification of the origin/source of the of the reflection, the constant diffuse illumination improves glint detection, and associating binary-encoding with the source of a reflection enables determination of corneal sphere orientation. Glint detection system 100 is configured to reduce the amount of data captured and processed from image scenes by monitoring the changes to an image scene while ignoring the portions of the image scene that remain relatively unchanged. Advantageously, the disclosed differential illumination systems and processes enhance the detectability of corneal surface specular reflections, which enables the determination of corneal and gaze orientation, according to embodiments of the disclosure.
Glint detection system 100 includes glint detection logic 102, light sources 104 (individually, light source 104A and 104B), and image sensor 108 carried by a head-mounted frame 110, according to an embodiment. Glint detection logic 102 includes circuitry, processor logic, memory, and/or instructions that support operation of glint detection system 100. Glint detection logic 102 is communicatively coupled to light sources 104 and image sensor 108 with communication channels 112. Glint detection logic 102 may include or use an illumination pattern 114 to generate image data 106 of specular reflections to enable a determination of glint locations 116, according to an embodiment.
Glint detection logic 102 is configured to drive illumination pattern 114 onto light sources 104 to cause light sources 104 to emit encoded differential light signals 118 (individually, encoded differential light signal 118A and 118B) toward a user 120. User 120 includes diffuse surfaces and reflective surfaces. Some of the diffuse surfaces/objects include the skin, the iris, and the sclera of user 120, which are approximately Lambertian surfaces that scatter light diffusely. An eye 122 of user 120 includes a corneal surface 124. Corneal surface 124 is a specular (reflective) surface that produces mirror-like reflections of encoded differential light signals 118. Encoded differential light signals 118 are diffusely scattered off of the diffuse surfaces and appear as glints in an image by reflecting off of corneal surface 124.
Encoded differential light signals 118 include a high state H and a low state L. High state H represents a level of illumination that is greater than the level of illumination of low state L. Low state L may be the level of illumination of a light source that is turned off, and high state H may be the level of illumination of a light source that is turned on, according to an embodiment. Low state L may be the level of illumination of a light source that is turned on and emits, for example, a light signal that is relatively dim (e.g., less than 20-30 lumens), and high state H may be the level of illumination of a light source that is turned on and emits, for example, a light signal that is relatively bright (e.g., anything over 50 lumens), according to an embodiment. In other words, high state H and low state L may represent two different levels of illumination that image sensor 108 may be configured to distinguish between. Although example values of illumination are described in terms of lumens, digital pixel values or another metric of illumination for image sensor 108 may be used.
Encoded differential light signals 118 include a binary identifier that identifies light sources 104, according to an embodiment. The binary identifier is a number encoded in the encoded differential light signals 118. The binary identifier can be a 2-bit, 3-bit, or larger binary number. A 2-bit number can be used to identify four different pairs of light sources (e.g., shown in FIG. 5), and a 3-bit number can be used to identify eight different pairs of light sources. An advantage of binary-encoding encoded differential light signals 118 is that a number of light sources (e.g., 8, 12, 16, etc.) can operate at the same frequency (e.g., 1 kHz) and be distinguished from each other based on individual binary identifiers. An example binary-encoding technique is described with reference to FIG. 5 below, and an example binary identifier tracking process is described below with reference to FIG. 6.
Light sources 104 emit encoded differential light signals 118 according to illumination pattern 114, according to an embodiment. Illumination pattern 114 may define both a frequency, waveform shape, binary identifier, and duty cycle of encoded differential light signals 118. The binary identifier is unique to each pair of light sources used in glint detection system 100, according to an embodiment. For example, encoded differential light signals 118 may be emitted with a square wave pattern, with a binary identifier of [01], and at a frequency of 1.2 kHz. With this pattern, light source 104A may be operated to initially emit encoded differential light signal 118A with high state H, and in a complementary configuration, light source 104B may be operated to initially emit encoded differential light signal 118B with a low state L, prior to switching back and forth between high and low states. In some embodiments, encoded differential light signals 118 may have a frequency in the range of 1 kHz to 2 kHz, or a frequency that is greater than or equal to 1 kHz to reduce blurring of glint detection. In an embodiment, encoded differential light signal 118A may have a duty cycle that is less than 50% to represent a binary ‘0’ and a duty cycle that is greater than 50% to represent a binary ‘1’. Complementary encoded differential light signal 118B has a duty cycle that is greater than 50% to represent a binary ‘0’ and a duty cycle that is less than 50% to represent a binary ‘1’, for example. In an embodiment, the combined duty cycles of encoded differential light signals 118 is 100%, so that the net illumination remains approximately constant for diffuse surfaces to enable improved detection of glints from corneal surface 124.
Light sources 104 are positioned on head-mounted frame 110 and have an intra-light source angle θ, according to an embodiment. Intra-light source angle θ may be used to define how far apart light source 104A and light source 104B are positioned from each other on head-mounted frame 110. For example, as disclosed below in connection with FIG. 3, intra-light source angle θ may determine a signal-to-noise ratio (SNR) for glint detection and may determine the proximity of clusters of glints to each other. In an embodiment, light sources 104 are positioned apart from each other to define intra-light source angle θ to be at least 2.5 degrees, so clusters of glints captured by image sensor 108 are adjacent to each other and not overlapping. One implementation of hardware may achieve adjacent (while not overlapping) pixel clusters of glints with at a 2.5 degree intra-light source angle θ. However, with different hardware implementations another intra-light source angle θ may result in adjacent (while not overlapping) pixel clusters of glints that result in an increased SNR of captured glints.
Light sources 104 may be implemented with a variety of types of light sources and in a number of configurations on head-mounted frame 110, according to an embodiment. Light sources 104 may be implemented as light-emitted diodes (LEDs), vertical external-cavity surface-emitting lasers (VCSELs), fiber optics, out-coupling gratings, or the like. Light sources 104 may be positioned to be above eye 122, below eye 122, near the nose, away from the nose, within a field of view of eye 122 (e.g., mounted or integrated into lenses of AR/VR glasses), and/or outside of a field of view of eye 122 (periphery) on head-mounted frame 110. Although a single pair of light sources 104 is illustrated and discussed in FIG. 1, multiple pairs of light sources may be incorporated into glint detection system 100, as shown in FIG. 2 and described below.
Glint detection logic 102 is configured to determine glint locations 116 at least partially based on image data 106 received from image sensor 108, according to an embodiment. Image sensor 108 is positioned on head-mounted frame 110 a distance D from eye 122. Distance D may partially be defined by hardware that mounts head-mounted frame 110 to the head of user 120. Image sensor 108 is oriented to receive reflections 126 (individually, reflection 126A and 126B) of encoded differential light signals 118 that reflect off of corneal surface 124. Image sensor 108 includes a number of pixels 128 that are photosensitive elements that convert light into an electrical signal. Front view 130 of image sensor 108 illustrates pixels 128 arranged in an addressable pixel array 132 of rows R and columns C. Image sensor 108 may include millions of pixels (e.g., 8 megapixels) to provide resolution in glint detection.
Image sensor 108 may be implemented as an event camera, a complementary metal oxide semiconductor (“CMOS”) image sensor, a charge-coupled device (“CCD”) image sensor, a time-of-flight sensor, or another photosensitive sensor. When implemented as an event camera, image sensor 108 detects changes in an image scene without capturing portions of the image scene that are unchanged. More specifically, as an event camera, image sensor 108 captures data from specific pixels that are illuminated above a particular threshold of illumination. Examples of thresholds of illumination can be in terms of brightness in lumens or brightness in digital pixel values (dp). Examples of these thresholds may depend on the bit depth of the image sensor (e.g., 8 bit, 10 bit, 20 bit) and may be 50 lumens, 100 lumens, 150 lumens, 20 dp, 50 dp, 100 dp, etc. The threshold value that defines whether a glint is detected may be adjusted based on environmental noise (e.g., external sunlight), according to an embodiment. When implemented as a CMOS or CCD image sensor, image sensor 108 may include on-chip circuitry to support event-based operations that ignore portions of an image scene that remain relatively constant and that captures or transmits events associated with pixels that are illuminated to a level that exceeds a pre-determined threshold.
Image data 106 is data that represents glints or clusters of glints captured by image sensor 108. Image data 106 includes events that may represent one or more of pixels 128 that exceed a threshold. An event may include a pixel address, a time stamp, a digital pixel value, and/or a polarity (e.g., high-to-low or low-to-high illumination changes). Image data 106 may be transmitted as each event is detected, may be transmitted periodically (e.g., every 10 ms), or may be detected each time a certain number of events are detected (e.g., every 10 events).
Front view 130 of image sensor 108 depicts an example mapping of pixel clusters that may represent glints detected by image sensor 108. Pixel cluster 134 represents a number of events or glints detected from reflection 126A of encoded differential light signal 118A, and pixel cluster 136 represents a number of events or glints detected from reflection 126B of encoded differential light signal 118B. Pixel cluster 134 and pixel cluster 136 represent events or glints detected around a first time t1, which may include a small period of time (e.g., 0.1 ms). Depending upon the size and/or pitch of pixels 128, detection of a single event (e.g., a change from low-to-high of one of encoded differential light signals 118) may cause several pixels 128 to register a digital pixel value that exceeds a threshold. Glint detection logic 102 may be configured to track individual events, or glint detection logic 102 may be configured to integrate several events (e.g., 20-50 events) together prior to updating glint locations 116, according to various embodiments. Hence, pixel cluster 134 and pixel cluster 136 may be representative of a single detected event or of several integrated events, according to various embodiments. As illustrated, at a second time t2 (which may be a small period of time), pixel cluster 134 and pixel cluster 136 have, for example, changed locations within pixel array 132, which may be indicative of a change of orientation of a corneal sphere 138 of which corneal surface 124 is a part of.
Glint detection logic 102 may use one or more synchronization pulses to associate encoded differential light signals 118 with reflections 126, according to an embodiment. For example, glint detection logic 102 may transmit a short synchronization pulse to light sources 104 and to image sensor 108 at the beginning of each period of illumination pattern 114 to correlate patterns emitted with patterns received. Glint detection logic 102 may then decode binary identifiers that are encoded in reflections 126 to identify light sources 104 as the source of encoded differential light signals 118 that produced reflections 126. When there are multiple pairs of light sources, the binary encoding can be used to distinguish one pair of light sources from another. In the absence of multiple pairs of light sources, the binary encoding can be used to distinguish an intentionally generated glint from noise or from light sources in the external environment. By knowing which light source caused a particular glint, glint detection system may use a number of techniques to determine the orientation of the corneal sphere 138 and/or a gaze orientation of user 120.
Glint detection system 100 may be incorporated into one or more systems to support operations of those systems. As described above, glint detection system 100 may be configured to use encoded differential light signals 118 and image sensor 108 (e.g., an event camera) to dynamically identify glint locations 116, which change as eye 122 changes orientation. Accordingly, glint detection system 100 may be used in an eye tracking system to determine a gaze orientation of a user and may be used in a head-mounted display (HMD) to adjust a display's focal point, brightness, user interface, etc., at least partially based on where a user's eyes are oriented within the HMD.
FIG. 2 illustrates an ocular environment 200, in accordance with embodiments of the disclosure. Ocular environment 200 illustrates features of glint detection system 100 at least partially integrated into an eye tracking system 202 that is further integrated into an HMD 204, according to an embodiment. Ocular environment 200 includes multiple pairs of light sources and multiple image sensors included in HMD 204 to support glint detection and eye tracking.
Ocular environment 200 may include one or more pairs of light sources 206 that are configured to emit encoded differential light signals 208, according to an embodiment. Pair of light sources 206A may be an implementation of light sources 104 (shown in FIG. 1). Pairs of light sources 206 are mounted to head-mounted frame 110 and may be mounted in a variety of locations (e.g., periphery, in-field, above the eye, below the eye, integrated into lenses, etc.). Pairs of light sources 206 emit light in the infrared (e.g., near infrared) or other non-visible wavelength range, according to an embodiment. Glint detection logic 102 may be configured to drive pairs of light sources 206 with illumination pattern 114, according to an embodiment. Illumination pattern 114 may set each of encoded differential light signals 208A, 208B, 208C, 208D to be different for each pair of light sources 206A, 206B, 206C, 206D. For example, illumination pattern 114 may define binary identifiers 209 that are unique to each pair of pairs of light sources 206. For example, illumination pattern 114 may define that: pair of light sources 206A is associated with binary identifier 209A, which could be [00]— binary value of ‘0’; pair of light sources 206B is associated with binary identifier 209B, which could be [01]— binary value of ‘1’; pair of light sources 206C is associated with binary identifier 209C, which could be [10]— binary value of ‘2’; and pair of light sources 206D is associated with binary identifier 209D, which could be [11]— binary value of ‘3’. Illumination pattern 114 can be configured to associate more binary identifiers 209 with additional light sources using, for example, 3-bit, 4-bit, or larger binary numbers as binary identifiers 209. Illumination pattern 114 may define a different periodic frequency for each of encoded differential light signals 208A, 208B, 208C and 208D, according to an embodiment. For example, illumination pattern 114 may define that encoded differential light signals 208A are emitted at 1 kHz, encoded differential light signals 208B are emitted at 1.2 kHz, encoded differential light signals 208C are emitted at 1.4 kHz, and encoded differential light signals 208D are emitted at 1.8 kHz. Glint detection logic 102 and/or eye tracking system 202 can be configured to correlate pairs of light sources 206 with their corresponding return signals (e.g., glints or events) by, for example, identifying binary identifiers 209 in reflections and/or by applying a Fourier transform to the detected return signals to identify a frequency of the return signals, according to an embodiment.
Ocular environment 200 may also include an image sensor 210 (in addition to image sensor 108) that is configured similarly to image sensor 108 to support glint detection. In one embodiment, image sensor 210 is configured to capture images of a pupil 212. Image sensor 210 may use a dedicated light source or light source pair to capture images of pupil 212 and may be configured to capture pupil images concurrently with detected events, concurrently with the detection of a number of events (e.g., 50), and/or periodically. Image sensor 210 and image sensor 108 may include bandpass filters that pass infrared light and filter out other wavelengths.
Eye tracking system 202 may use glint locations 116, corneal map data 214, and pupil characteristics 216 to determine a gaze orientation 218, according to an embodiment. Corneal map data 214 may include correlation mappings between glint locations 116 and an orientation of corneal sphere 138, according to an embodiment. Pupil characteristics 216 may include a shape, size, or center of pupil 212 that are determined from image data from image sensor 210, for example. Eye tracking system 202 may apply various techniques to determine a vector of gaze orientation 218 that is at least partially based on, for example, pupil characteristics 216 (e.g., a pupil center) and glint locations 116, according to an embodiment.
Eye tracking system 202 may provide gaze orientation 218 to HMD 204 to support various operations, according to an embodiment. For example, HMD 204 may use gaze orientation 218 to customize user interface elements incorporated into a user interface 220 and/or to define operations of one or more applications 222, according to an embodiment. HMD 204 may at least partially use gaze orientation 218 to drive a display 224, according to an embodiment. For example, based on gaze orientation 218, HMD 204 may drive or operate display 224 to adjust brightness, a focal point, or features included in display light 226 (e.g., user interface elements).
HMD 204 includes a lens assembly 228 that transmits display light 226 to eye 122, according to an embodiment. Lens assembly 228 may be carried by head-mounted frame 110 within HMD 204 and may include one or more lenses, grates, and/or other optical elements.
HMD 204 includes processing logic 230 and memory 232, according to an embodiment. Processing logic 230 and/or memory 232 may include instructions 234 that are machine-readable and executable by processing logic 230 and/or glint detection logic 102, according to various embodiments. Processing logic 230 may be communicatively coupled to image sensors 108 and 210, pairs of light sources 206, display 224, and glint detection logic 102 to support operation of HMD 204, according to an embodiment. Processing logic 230 may fully or partially include glint detection logic 102. Processing logic 230 may include circuitry, logic, instructions stored in a machine-readable storage medium, ASIC circuitry, FPGA circuitry, and/or one or more processors.
FIG. 3 illustrates an example graph 300 of potential operational characteristics that are based on intra-light source angle θ (shown in FIG. 1) between a light source 104A and a light source 104B, according to an embodiment. Intra-light source angle θ may be applied to any pairs of light sources 206 (shown in FIG. 2). Graph 300 includes a signal-to-noise ratio (SNR) graph line 302 and an events rate graph line 304 plotted against changes in intra-light source angle θ. A type of light source that may be used in embodiments of the disclosure includes an LED, so intra-light source angle θ may be referred to as an intra-LED angle θ. As illustrated, when intra-light source angle θ is approximately 2.5 degrees, pixel clusters of glints 306 become adjacent to each other while not overlapping. As illustrated, pixel clusters of glints 308 are adjacent to each other but overlap somewhat, which may result in lower SNR and lower events rate, as compared to that of pixel clusters of glints 306. One implementation of hardware may achieve adjacent (while not overlapping) pixel clusters of glints with at a 2.5 degree intra-light source angle θ. However, with different hardware implementations another intra-light source angle θ (greater or less than 2.5 degrees) may result in adjacent (while not overlapping) pixel clusters of glints that provide an increased SNR of captured glints.
FIG. 4 illustrates a process 400 for acquiring and using glint locations, for example, in a head-mounted display or other ocular environment, according to embodiments of the disclosure. Process 400 may be incorporated into glint detection system 100, ocular environment 200, eye tracking system 202, and/or HMD 204, according to embodiments of the disclosure. The order in which some or all of the process blocks appear in process 400 should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.
In process block 402, process 400 drives light sources 404 to emit differential light signals, according to an embodiment. The differential light signals are binary-encoded to enable association of glints with light sources, according to an embodiment. The differential light signals can be defined by an illumination pattern that specifies a particular binary encoding, waveform shape, frequency of signaling, duty cycle, and wavelength of light emitted. Process block 402 proceeds to process block 406, according to an embodiment.
At process block 406, process 400 receives, from image sensor 410, image data 408 that is representative of reflections of the differential light signals, according to an embodiment. The differential light signals reflect off of the corneal surface of an eye and are referred to as glints. When the differential light signals change state (e.g., high-to-low or low-to-high), the change in the image scene monitored by image sensor 410 may register each changed portion of the image scene as an individual event. Image sensor 410 may be configured to register or capture an event if an illumination change (positive or negative) in the image scene changes by more than a pre-determined threshold value (e.g., 30-50 digital pixel values), according to an embodiment. The events and image sensor 410 may be synchronized in time with a synchronization pulse to associate a particular light emission with one or more events. Process block 406 proceeds to process block 412, according to an embodiment.
At process block 412, process 400 identifies glint locations based on image data 408, according to an embodiment. Glint locations are a reference of where, in an image map, glints are detected. A glint location may be determined by selecting a highest digital pixel value within a cluster of pixels. A glint location may be defined by finding a weighted mean pixel coordinate, since glints may appear as Gaussian blobs. A glint map may be created from a combination or integration of multiple events, and the glint map may be searched for local peaks to detect glints. Process block 412 proceeds to process block 414, according to an embodiment.
At process block 414, process 400 identifies a binary identifier from a glint, according to an embodiment. A binary glint may be identified based on the duty cycle of the detected glint. For example, the duration of illumination of a glint may be used to identify a glint as a binary ‘0’ (e.g., a short pulse) or as a binary ‘1’ (e.g., a long pulse). In one embodiment, a glint having an illumination duration that is less than a 50% duty cycle may correspond to a short pulse or a binary ‘0’, and a glint having an illumination duration that is greater than a 50% duty cycle may correspond to a long pulse or binary ‘1’. The illumination duration of glints may be determined based on when an image sensor (e.g., event camera) detects an event for the presence of a glint as compared to an event for the disappearance of a glint. Process block 414 proceeds to process block 420, according to an embodiment.
At process block 420, process 400 tracks glints using binary identifiers to correlate glints with light sources. Glint tracking may be performed in accordance with one or more portions of process 600 (shown in FIG. 6), according to an embodiment. Process 400 may track glints by identifying a first binary digit for a particular glint, searching the vicinity of the location of the first particular glint, identifying a second (or subsequent) binary digit, and associating the glint with the light sources assigned to the binary identifier made with the first and second binary digits, according to an embodiment. Subsequent glints can be used to confirm the association of a glint with a light source or can be used to identify a new/different light source glint at a particular location. Process block 420 proceeds to process block 422, according to an embodiment.
At process block 422, process 400 provides glint location data and corresponding light source identification data 424 to eye tracking system 426, according to an embodiment. Process block 422 proceeds to process block 428, according to an embodiment.
At process block 428, process 400 determines corneal sphere orientation and/or gaze orientation based on glint location data and corresponding light source identification data 424, according to an embodiment. Corneal sphere location and/or gaze orientation may be determined, for example, using corneal sphere regression. In corneal sphere regression, corneal glints may be modeled as ideal specular reflections on a perfect sphere. Under this model, there is exactly one ray which passes from each light source to the corresponding glint and from each glint to the corresponding point on the image plane (e.g., on a pixel array), according to an embodiment. By fixing the locations of light sources in the image plane with calibration, corneal sphere dimensions and location may be determined by optimizing (e.g., iteratively adjusting) the reprojection error of the lights onto the image plane. Process block 428 proceeds to process block 430, according to an embodiment.
At process block 430, process 400 provides corneal sphere orientation data and/or gaze orientation data 432 to a display controller 434, according to an embodiment. Display controller 434 may be configured to control images and/or user experience elements, focal points, etc. that for the display. The display may be a component of, for example, a head-mounted display, according to an embodiment. Process block 430 proceeds to process block 436, according to an embodiment.
At process block 436, process 400 adjusts display properties, application renderings, and/or user interface elements, based on corneal sphere orientation data and/or gaze orientation data 432, according to an embodiment. Process block 436 proceeds to process block 402, to continue to iterate through process 400, according to an embodiment.
FIG. 5 illustrates an example timing diagram 500 of binary encoding for complementary light sources, according to an embodiment. Timing diagram 500 illustrates complementary pairs of encoded differential signals 502, 504, 506, and 508, as an example of 2-bit encoding for four pairs of light sources that can be used to cause the light sources to emit encoded differential light signals. The encoded differential signals 502, 504, 506, and 508 provide a first bit in a first period 510 and a second bit in a second period 512. The encoded differential signals 502, 504, 506, and 508 may be synchronized with image sensors using a CLK signal 514 having a synchronization pulse 516 at the beginning of each period, according to an embodiment.
The encoded differential signals 502, 504, 506, and 508 are each associated with a primary light source LSP and a secondary light source LSS. The primary light source LSP and the secondary light source LSS together make a complementary pair. Encoded differential signals 502 may correspond to a primary light source LS0P that emits light with a binary pattern of [00] according to a waveform 518 and may correspond to a secondary light source LSos that emits light with a complementary binary pattern of [00] according to a waveform 520, according to an embodiment. Encoded differential signals 504 may correspond to a primary light source LS1P that emits light with a binary pattern of [01] according to a waveform 522 and may correspond to a secondary light source LSis that emits light with a complementary binary pattern of [01] according to a waveform 524, according to an embodiment. Encoded differential signals 506 may correspond to a primary light source LS2P that emits light with a binary pattern of [10] according to a waveform 526 and may correspond to a secondary light source LS2S that emits light with a complementary binary pattern of [10] according to a waveform 528, according to an embodiment. Encoded differential signals 508 may correspond to a primary light source LS3p that emits light with a binary pattern of [11] according to a waveform 530 and may correspond to a secondary light source LS3S that emits light with a complementary binary pattern of [11] according to a waveform 532, according to an embodiment. As shown, for the primary light sources, a binary ‘0’ is represented as having a duty cycle that is less than 50% and that may be approximately 33%, and a binary ‘1’ is represented as having a duty cycle that is greater than 50% and that may be approximately 66%, according to an embodiment. For the secondary (complementary) light sources, the waveform is a complementary (or inverted) representation of the primary light source signal. Other example duty cycles or illumination durations may be used, and this binary encoding scheme may be incorporated into glint detection system 100 (shown in FIG. 1) and into ocular environment 200 (shown in FIG. 2).
FIG. 6 illustrates a process 600 for tracking illumination patterns in a glint detection system, according to embodiments of the disclosure. Process 600 may be incorporated into glint detection system 100, ocular environment 200, eye tracking system 202, and/or head-mounted display 204, according to embodiments of the disclosure. The order in which some or all of the process blocks appear in process 600 should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.
Process 600 may categorize glints into three categories, according to an embodiment. The three categories may include unidentified glints 602, partially-identified glints 604, and fully-identified glints 606. In operation, process 600 may dynamically move a particular glint between these identification categories based on binary bits received, binary bits expected, and the locations in an image map at which glints are located.
At process block 608, process 600 detects a glint, according to an embodiment. The detected glint may be an unidentified glint for which at least one bit has not yet been received. At this point, process 600 may categorize a glint as one of unidentified glints 602. Process block 608 proceeds to process block 610, according to an embodiment.
At process block 610, process 600 receives a first bit of a glint at a first (new or previously not known) location, according to an embodiment. Process block 610 proceeds to process block 612, according to an embodiment.
At process block 612, process 600 stores a first bit and a first location for a detected glint, according to an embodiment. At this point, process 600 may categorize a glint as one of partially-identified glints 604. Process block 612 proceeds to process block 614, according to an embodiment.
At process block 614, process 600 waits for a second bit for a glint in a first area of pixels around a first location of the glint, according to an embodiment. The first area may be 2, 5, 10, or some other number of pixels around the location at which a glint was previously detected to account for the glint moving in response to corneal sphere orientation changes. Process block 614 proceeds to process block 616, according to an embodiment.
At process block 616, process 600 receives a second bit and second location for a glint, according to an embodiment. Process block 616 proceeds to process block 618, according to an embodiment.
At process block 618, process 600 associates a second location of a glint in a pixel array with a light source that was assigned the binary identifier represented by the first and second bits, according to an embodiment. At this point, a glint may be categorized as one of fully-identified glints 606. Process block 618 proceeds to process block 620, according to an embodiment.
At process block 620, process 600 determines a corneal sphere location based on the second location, according to an embodiment. Process block 620 proceeds to process block 622, according to an embodiment.
At process block 622, process 600 waits for a next expected bit from the glint within a second area (window) of pixels, according to an embodiment. The second area (window) of pixels may be the same, smaller, or larger than the size of the first area. If the next glint is in the same location (within the second area) and provides a next bit that is expected from a particular light source, process block 622 loops back to repeat process block 622, according to an embodiment. If the next glint is in a new location but within the second area of pixels and provides a next bit that is expected, process block 622 loops back to process block 620, according to an embodiment. If the next glint provides an unexpected next bit, process block 622 loops to process block 608 and categorized the glint as one of unidentified glints 602, according to an embodiment.
FIG. 7 illustrates a process 700 for detecting corneal glints using binary-encoded illumination, for example, in an eye tracking system, according to embodiments of the disclosure. Process 700 may be incorporated into glint detection system 100, ocular environment 200, eye tracking system 202, and/or head-mounted display 204, according to embodiments of the disclosure. The order in which some or all of the process blocks appear in process 700 should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.
In process block 702, process 700 drives a plurality of pairs of light sources to emit differential light signals towards an eye of a user, according to an embodiment. Process block 702 proceeds to process block 704, according to an embodiment.
In process block 704, process 700 encodes the differential light signals with binary identifiers, wherein each of the binary identifiers is unique to a corresponding pair of the plurality of pairs of light sources, according to an embodiment. Process block 704 proceeds to process block 706, according to an embodiment.
In process block 706, process 700 receives image data from an image sensor, wherein the image data is representative of reflections of the differential light signals off of the eye of the user, according to an embodiment. Process block 706 proceeds to process block 708, according to an embodiment.
In process block 708, process 700 identifies each of the plurality of pairs of light sources within the image data at least partially based on the binary identifiers, according to an embodiment.
FIG. 8 illustrates a head-mounted device (HMD) 800, in accordance with aspects of the present disclosure. As described further below, in embodiments, HMD 800 may include a glint detection system that uses binary-encoded illumination and that includes light sources 104, image sensor 108, glint detection logic 102, and any related processes, as described above in connection with FIGS. 1-7. An HMD, such as HMD 800, is one type of head mounted device, typically worn on the head of a user to provide artificial reality content to a user. Artificial reality is a form of reality that has been adjusted in some manner before presentation to the user, which may include, e.g., virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and/or derivative thereof. The illustrated example of HMD 800 is shown as including a viewing structure 840, a top securing structure 841, a side securing structure 842, a rear securing structure 843, and a front rigid body 844. In some examples, the HMD 800 is configured to be worn on a head of a user of the HMD 800, where the top securing structure 841, side securing structure 842, and/or rear securing structure 843 may include a fabric strap including elastic as well as one or more rigid structures (e.g., plastic) for securing the HMD 800 to the head of the user. HMD 800 may also optionally include one or more earpieces 820 for delivering audio to the ear(s) of the user of the HMD 800.
The illustrated example of HMD 800 also includes an interface membrane 818 for contacting a face of the user of the HMD 800, where the interface membrane 818 functions to block out at least some ambient light from reaching the eyes of the user of the HMD 800.
Example HMD 800 may also include a chassis for supporting hardware of the viewing structure 840 of HMD 800 (chassis and hardware not explicitly illustrated in FIG. 8). The hardware of viewing structure 840 may include any of processing logic, wired and/or wireless data interface for sending and receiving data, graphic processors, and one or more memories for storing data and computer-executable instructions. In one example, viewing structure 840 may be configured to receive wired power and/or may be configured to be powered by one or more batteries. In addition, viewing structure 840 may be configured to receive wired and/or wireless data including video data.
Viewing structure 840 may include a display system having one or more electronic displays for directing light to the eye(s) of a user of HMD 800. The display system may include one or more of an LCD, an organic light emitting diode (OLED) display, or micro-LED display for emitting light (e.g., content, images, video, etc.) to a user of HMD 800.
Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
The term “processing logic” (e.g., processing logic 230, glint detection logic 102) in this disclosure may include one or more processors, microprocessors, multi-core processors, Application-specific integrated circuits (ASIC), and/or Field Programmable Gate Arrays (FPGAs) to execute operations disclosed herein. In some embodiments, memories (not illustrated) are integrated into the processing logic to store instructions to execute operations and/or store data. Processing logic may also include analog or digital circuitry to perform the operations in accordance with embodiments of the disclosure.
A “memory” or “memories” (e.g., memory 232) described in this disclosure may include one or more volatile or non-volatile memory architectures. The “memory” or “memories” may be removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVD), high-definition multimedia/data storage disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.
A computing device may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.
The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.
A tangible non-transitory machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-readable storage medium includes recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).
The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.
These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.
