Sony Patent | Use Of Eye Tracking To Adjust Region-Of-Interest (Roi) For Compressing Images For Transmission
Patent: Use Of Eye Tracking To Adjust Region-Of-Interest (Roi) For Compressing Images For Transmission
Publication Number: 10684685
Publication Date: 20200616
Applicants: Sony
Abstract
Gaze tracking data may representing a user’s gaze with respect to one or more images transmitted to a user are analyzed to determine one or more regions of interest. The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.
FIELD OF THE DISCLOSURE
Aspects of the present disclosure are related to digital graphics. In particular, the present disclosure is related to varying the quantization within compressed image to maximize compression and minimize perceptible artifacts presented to a user.
BACKGROUND
Graphical display devices having a wide field of view (FOV) have been developed. Such devices include head mounted display (HMD) devices. In an HMD device, a small display device is worn on a user’s head. The display device has a display optic in front of one eye (monocular HMD) or each eye (binocular HMD). An HMD device typically includes sensors that can sense the orientation of the device and change the scene shown by the display optics as the user’s head moves. Conventionally, most stages of rendering scenes for wide FOV displays are performed by planar rendering where all parts of the screen have the same number of pixels per unit area.
However, rendering for virtual reality (VR) programs, which is often performed in conjunction with HMD devices, requires a higher frame rate than conventional flat screen rendering to prevent a user from experiencing motion sickness. HMD for VR has optical systems to show rendered scenes in wide FOV for immersive experiences. While the screen area around a primary gaze point (sometimes called the foveal region) requires high resolution, the areas outside the primary gaze point are observed only by the peripheral vision and can therefore be rendered at a lower resolution, or may contain less detail. Such rendering is sometimes referred to as foveated rendering.
Research has been performed that seeks to apply foveated rendering at the pixel level by selectively adjusting the pixel resolution for different parts of the screen. See co-pending U.S. patent application Ser. No. 14/246,066, to Mark Evan Cerny, filed Apr. 5, 2014, which is incorporated herein by reference. Furthermore, the foveated rendering concept may be applied at earlier stages of a graphics processing pipeline, such as the geometry level, e.g., by adjusting the tessellation of computer generated objects for different parts of the screen on which they are displayed. See co-pending U.S. patent application Ser. No. 14/927,157 to Jun Murakawa et al. filed Oct. 29, 2015, which is incorporated herein by reference. These approaches, and others, can reduce the computational load on graphics processing hardware by concentrating computational resources on rendering more important parts of an image on a display.
It is within this context that the present disclosure arises.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1A-1B are schematic diagrams illustrating gaze tracking within the context of aspects of the present disclosure.
FIG. 2 is a flow diagram depicting a method according to aspects of the present disclosure.
FIG. 3 is a block diagram depicting a system according to aspects of the present disclosure.
FIG. 4A is a simplified diagram illustrating an example of normal tessellation performed in accordance with the prior art.
FIG. 4B is a simplified diagram illustrating an example of foveated tessellation in accordance with aspects of the present disclosure.
FIGS. 5A-5B are flow diagrams depicting a graphics processing methods according to an aspect of the present disclosure.
FIG. 6A is a schematic diagram of a screen space illustrating an example of a region of interest in accordance with aspects of the present disclosure.
FIGS. 6B-6D are graphs depicting examples of transitions in vertex density over a screen space in accordance with aspects of the present disclosure.
FIG. 7 is a block diagram of a graphics processing system in accordance with aspects of the present disclosure.
FIG. 8 is a block diagram of a graphics processing pipeline that may be implemented, e.g., by the system of FIG. 7 in accordance with aspects of the present disclosure.
FIGS. 9A-9H are schematic diagrams illustrating examples of the use of eye gaze and face tracking in conjunction with embodiments of the present invention.
FIG. 10A-10D are schematic diagrams illustrating facial orientation characteristic tracking setups according to aspects of the present disclosure.
FIG. 10E is a schematic diagram illustrating a portable device that can utilize facial orientation tracking according to an aspect of the present disclosure.
DETAILED DESCRIPTION
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, the illustrative implementations of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.
* Introduction*
Eye gaze tracking has use in a wide range of applications, including medical research, automobile technology, computer entertainment and video game programs, control input devices, augmented reality glasses, and more. There are a number of techniques for eye tracking, also known as gaze tracking. Some of these techniques determine a user’s gaze direction from the orientation of the pupils of the user’s eyes. Some known eye gaze tracking techniques involve illuminating the eyes by emitting light from one or more light sources and detecting reflections of the emitted light off of the corneas with a sensor. Typically, this is accomplished using invisible light sources in the infrared range and capturing image data (e.g., images or video) of the illuminated eyes with an infrared sensitive camera. Image processing algorithms are then used to analyze the image data to determine eye gaze direction.
Generally, eye tracking image analysis takes advantage of characteristics distinctive to how light is reflected off of the eyes to determine eye gaze direction from the image. For example, the image may be analyzed to identify eye location based on corneal reflections in the image data, and the image may be further analyzed to determine gaze direction based on a relative location of the pupils in the image.
Two common gaze tracking techniques for determining eye gaze direction based on pupil location are known as Bright Pupil tracking and Dark Pupil tracking. Bright Pupil tracking involves illumination of the eyes with a light source that is substantially in line with the optical axis of the camera, causing the emitted light to be reflected off of the retina and back to the camera through the pupil. The pupil presents in the image as an identifiable bright spot at the location of the pupil, similar to the red eye effect which occurs in images during conventional flash photography. In this method of gaze tracking, the bright reflection from pupil itself helps the system locate the pupil if contrast between pupil and iris is not enough.
Dark Pupil tracking involves illumination with a light source that is substantially off line from the optical axis of the camera, causing light directed through the pupil to be reflected away from the optical axis of the camera, resulting in an identifiable dark spot in the image at the location of the pupil. In alternative Dark Pupil tracking systems, an infrared light source and cameras directed at eyes can look at corneal reflections. Such camera based systems track the location of the pupil and corneal reflections which provides parallax due to different depths of reflections gives additional accuracy.
FIG. 1A depicts an example of a dark pupil gaze tracking system 100 that may be used in the context of the present disclosure. The gaze tracking system tracks the orientation of a user’s eye E relative to a display screen 101 on which visible images are presented. While a display screen is used in the example system of FIG. 1A, certain alternative embodiments may utilize an image projection system capable of projecting images directly into the eyes of a user. In these embodiments, the user’s eye E would be tracked relative to the images projected into the user’s eyes. In the example of FIG. 1A, the eye E gathers light from the screen 101 through a variable iris I and a lens L projects an image on the retina R. The opening in the iris is known as the pupil. Muscles control rotation of the eye E in response to nerve impulses from the brain. Upper and lower eyelid muscles ULM, LLM respectively control upper and lower eyelids UL,LL in response to other nerve impulses.
Light sensitive cells on the retina R generate electrical impulses that are sent to the user’s brain (not shown) via the optic nerve ON. The visual cortex of the brain interprets the impulses. Not all portions of the retina R are equally sensitive to light. Specifically, light-sensitive cells are concentrated in an area known as the fovea.
The illustrated image tracking system includes one or more infrared light sources 102, e.g., light emitting diodes (LEDs) that direct non-visible light (e.g., infrared light) toward the eye E. Part of the non-visible light reflects from the cornea C of the eye and part reflects from the iris. The reflected non-visible light is directed toward a suitable sensor 104 (e.g., an infrared camera) by a wavelength-selective mirror 106. The mirror transmits visible light from the screen 101 but reflects the non-visible light reflected from the eye.
The sensor 104 is preferably an image sensor, e.g., a digital camera that can produce an image of the eye E which may be analyzed to determine a gaze direction GD from the relative position of the pupil. This image may be produced with a local processor 120 or via the transmission of the obtained gaze tracking data to a remote computing device 160. The local processor 120 may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The image tracking data may be transmitted between the sensor 104 and the remote computing device 160 via a wired connection (not shown), or wirelessly between a wireless transceiver 125 included in the eye tracking device 110 and a second wireless transceiver 126 included in the remote computing device 160. The wireless transceivers may be configured to implement a local area network (LAN) or personal area network (PAN), via a suitable network protocol, e.g., Bluetooth, for a PAN.
The gaze tracking system 100 may also include an upper sensor 108 and lower sensor 109 that are configured to be placed, for example, respectively above and below the eye E. Sensors 108 and 109 may be independent components, or may alternatively be part of a component 110 worn on the user’s head that may include, but is not limited to, any combination of the sensor 104, local processor 120, or inertial sensor 115 described below. In the example system shown in FIG. 1A, sensors 108 and 109 are capable of collecting data regarding the electrical impulses of the nervous system and/or the movement and/or vibration of the muscular system from those areas surrounding the eye E. This data may include for example, electrophysiological and/or vibrational information of the muscles and/or nerves surrounding the eye E as monitored by the upper sensor 108 and lower sensor 109. The electrophysiological information collected by sensors 108 and 109 may include, for example, electroencephalography (EEG), electromyography (EMG), or evoked potential information collected as a result of nerve function in the area(s) surrounding the eye E. Sensors 108 and 109 may also be capable of collecting, for example, mechanomyogram or surface electromyogram information as a result of detecting the muscular vibrations or twitches of the muscles surrounding the eye E. The data collected by sensors 108 and 109 may be delivered with the image tracking data to the local processor 120 and/or the remote computing device 160 as described above.
The gaze tracking system 100 may also be capable of tracking a user’s head. Head tracking may be performed by an inertial sensor 115 capable producing signals in response to the position, motion, orientation or change in orientation of the user’s head. This data may be sent to the local processor 120 and/or transmitted to the remote computing device 160. The inertial sensor 115 may be an independent component, or may alternatively be part of a component 110 worn on the user’s head that may include, but is not limited to, any combination of the sensor 104, local processor 120, or sensors 108 and 109 described above. In alternative embodiments, head tracking may be performed via the tracking of light sources on the component 110.
The remote computing device 160 may be configured to operate in coordination with the eye tracking device 110 and the display screen 101, in order to perform eye gaze tracking and determine lighting conditions in accordance with aspects of the present disclosure. The computing device 160 may include one or more processor units 170, which may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The computing device 160 may also include one or more memory units 172 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), and the like).
The processor unit 170 may execute one or more programs, portions of which may be stored in the memory 172, and the processor 170 may be operatively coupled to the memory 172, e.g., by accessing the memory via a data bus 178. The programs may be configured to perform eye gaze tracking and determine lighting conditions for the system 100. By way of example, and not by way of limitation, the programs may include gaze tracking programs 173, the execution of which may cause the system 100 to track a user’s gaze, e.g., as discussed above, error and state parameter determination programs 174, and foveation rendering programs 175, the execution of which render foveated images to be presented on the display. The foveation rendering programs 175 may use error and/or state parameters to determine potential adjustments that can be made to images presented and to adjust the foveation of images to be presented on the display, respectively, e.g., as discussed below with respect to FIG. 2.
By way of example, and not by way of limitation, the gaze tracking programs 173 may include processor executable instructions which cause the system 100 to determine one or more gaze tracking parameters of the system 100 from eye tracking data gathered with the image sensor 104 and eye movement data gathered from the upper and lower sensors 108 and 109, respectively, while light is emitted from the lighting source 102. The gaze tracking programs 173 may also include instructions which analyze images gathered with the image sensor 104 in order to detect a presence of a change in lighting conditions, e.g., as described below with respect to FIG. 2.
As seen in FIG. 1B, the image 181 showing a user’s head H may be analyzed to determine a gaze direction GD from the relative position of the pupil. For example, image analysis may determine a 2-dimensional offset of the pupil P from a center of the eye E in the image. The location of the pupil relative to the center may be converted to a gaze direction relative to the screen 101, by a straightforward geometric computation of a three-dimensional vector based on the known size and shape of the eyeball. The determined gaze direction GD is capable of showing the rotation and acceleration of the eye E as it moves relative to the screen 101.
As also seen in FIG. 1B, the image may also include reflections 187 and 188 of the non-visible light from the cornea C and the lens L, respectively. Since the cornea and lens are at different depths, the parallax and refractive index between the reflections may be used to provide additional accuracy in determining the gaze direction GD. An example of this type of eye tracking system is a dual Purkinje tracker, wherein the corneal reflection is the first Purkinje Image and the lens reflection is the 4th Purkinje Image. There may also be reflections 190 from a user’s eyeglasses 193, if these are worn a user.
Current HMD panels refresh at a constant rate of 90 or 120 Hertz (Hz) depending on the manufacturer. The high refresh rate increases power consumption of the panel and bandwidth requirements of the transmission medium to send frame updates. However, the image displayed on the panel does not always need to be refreshed during events that interrupt the user’s visual perception. For example, when a user blinks, visual information is shut off by the eyelids. When a user’s eyes undergo a saccade, the brain effectively shuts off interpretation of visual information. Human eyes also exhibit rapid eye movements known as saccades. A phenomenon known as saccadic masking occurs during a saccade. Saccadic masking causes the brain to suppress visual information during eye movements. Power and computational resources devoted to rendering frames during a vision interrupting event, such as a saccade or blink, are therefore wasted.
Many types of displays, such as HMD systems, are strobed systems that utilize persistence of vision to keep the image stable. There is a relatively large variation in the duration of a saccade or blink. For example, a saccade typically lasts from 20 to 200 ms. This corresponds to between 2 and 25 frames at a frame rate of 120 frames per second (fps). Even if it takes 10 ms to detect the start of saccade and the saccade only lasts 20 ms, the graphics system can save one frame, e.g., not render to reduce computation or turn off the display to save power or both. A blink typically lasts from about 100 ms to about 150 ms, which sufficient time for 12 to 18 frames at 120 fps.
As discussed with respect to FIG. 1A, camera-based eye tracking can be augmented with other methods to update eye tracking during a blink phase. Examples of augmentation include providing EEG information in addition to the image information in order to detect nerve impulses that trigger eye muscle activity. This information can also be used help detect the start and end of blinks and saccades, or to predict the duration of blinks and saccades. Eye tracking systems can determine whether the vision system is in a saccade or not by high-pass filtering based on rate of eye movement.
With knowledge that a blink or a saccade has begun and when it will end, a graphics rendering system could selectively disable transmission of frames until the time a user’s saccade or blink will finish. Then the system could schedule work to finish the transmission in time to update the display. The end result is a savings in computation time, transmission bandwidth, and power consumption. Computational resources that would otherwise be devoted to graphics may be used for other things, e.g., physics simulations and AI processing for rendering subsequent frames can run during this time.
For further computational savings, gaze tracking may also be analyzed to predict the user’s gaze point on the display at the end of the saccade or blink and render the frame using foveated rendering. Furthermore, as the user uses the system over a period of time, software running on the computing device 160 can analyze gaze tracking data to improve detection and estimates of duration estimation and final gaze point for saccades or blinks.
FIG. 2 shows an example method 200 wherein a system could adjust the compression or transmission of graphics transmitted to a user in ways that take into account saccades and/or blinks by a viewer. In this example, gaze tracking data 202 is obtained as discussed with respect to FIGS. 1A-1B. The eye tracking data may then be analyzed to detect a saccade and/or blink, as indicated at 204. If, at 206 no saccade or blink is detected normal transmission of image data to the display may take place, as indicated at 210A followed by presentation of images, as indicated at 212. The normal transmission takes place with normal transmission parameters and/or data compression parameters. If instead, a saccade and/or blink is detected at 206 the transmission of image data may be disabled at 210B for a period that accounts for the nature of the saccade/blink, which may potentially include determining the duration of the saccade blink at 208 through analysis of the gaze tracking data. Determining the saccade/blink duration at 208 may also include predicting when the saccade/blink will end by utilizing historical gaze tracking data of the user. When the system determines that the saccade/blink is ending, normal compression/transmission of the image data resumes, and the resulting images may then be presented at 212.
In alternative implementations the normal transmission at 210A may involve compression of the transmitted images. For example, the images may be selectively compressed based on additional parameters determined from the gaze tracking data. For example, and not by way of limitation, the quantization parameters may be determined for each foveal region of the image presented to the user, and this parameter may be used to selectively compress the foveal regions of the image before transmission and subsequent presentation to the user.
Furthermore, the disabling transmission may involve reducing transmission power, ignoring packet loss or both. Normal compression may resume, or transmission power may be increased and/or packet loss recovery may be reinstituted so that images are transmitted normally following the saccade or blink.
FIG. 3 depicts an example system for eye tracking 300 to further illustrate various aspects of the present disclosure. The example system 300 may include a computing device 360 which is coupled to an eye tracking device 302 and a display device 304 in order to perform eye gaze tracking and/or calibration for eye tracking in accordance with aspects of the present disclosure. The display device 304 may be in the form of a cathode ray tube (CRT), flat panel screen, touch screen, or other device that displays text, numerals, graphical symbols, or other visual objects. According to aspects of the present disclosure, the computing device 360 may be an embedded system, mobile phone, personal computer, tablet computer, portable game device, workstation, game console, and the like. Moreover, the computing device 360, the eye tracking device 302, the display device 304, or any combination thereof may form an integral unit or be implemented as separate components which may be in communication with each other.
The eye tracking device 302 may be coupled to the computing device 360, and may include a dynamic lighting source 310 similar to light sources 110 of FIGS. 1A-1B. By way of example, and not by way of limitation, the lighting source 310 may be an invisible lighting source in the form of one or more infrared LEDs, which may be configured to illuminate a user’s eyes in order to gather eye tracking data with the sensor 312. The sensor 312 of the eye tracking device may be a detector which is sensitive to light emitted from the light source 310. For example, the sensor 312 may be a camera sensitive to the light source such as an infrared camera, and the camera 312 may be positioned relative to the eye tracking device and the lighting source so that it may capture images of an area illuminated by the lighting source 310.
The computing device 360 may be configured to operate in coordination with the eye tracking device 302 and the display device 304, in order to perform eye gaze tracking and determine lighting conditions in accordance with aspects of the present disclosure. The computing device 360 may include one or more processor units 370, which may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The computing device 360 may also include one or more memory units 372 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), and the like).
The processor unit 370 may execute one or more programs, portions of which may be stored in the memory 372, and the processor 370 may be operatively coupled to the memory 372, e.g., by accessing the memory via a data bus 376. The programs may be configured to perform eye gaze tracking and determine lighting conditions for the system 300. By way of example, and not by way of limitation, the programs may include gaze tracking programs 373, execution of which may cause the system 300 to track a user’s gaze, e.g., as discussed above with respect to FIG. 1, saccade/blink detection programs 374, execution of which analyze gaze tracking information to determine onset and/or duration of a blink or saccade, e.g., as discussed above with respect to FIG. 2, and rendering adjustment programs 375, execution of which varies graphics processing and rendering of images to be presented on the display device 304 to account for saccades and blinks, e.g., as implemented by a method having one or more features in common with the method of FIG. 2. By way of example, and not by way of limitation, the gaze tracking programs 373 may include processor executable instructions which cause the system 300 to determine one or more gaze tracking parameters of the system 300 from eye tracking data gathered with the camera 312 while light is emitted from the dynamic lighting source 310. The gaze tracking programs 373 may also include instructions which analyze images gathered with the camera 312, e.g., as described above with respect to FIG. 1B.
The computing device 360 may also include well-known support circuits 378, such as input/output (I/O) circuits 379, power supplies (P/S) 380, a clock (CLK) 381, and cache 382, which may communicate with other components of the system, e.g., via the bus 376. The I/O circuits may include a wireless transceiver to facilitate communication with similarly configured transceivers on the eye tracking device 302 and display device 379. The processor unit 370 and wireless transceiver may be configured to implement a local area network (LAN) or personal area network (PAN), via a suitable network protocol, e.g., Bluetooth, for a PAN. The computing device 360 may optionally include a mass storage device 384 such as a disk drive, CD-ROM drive, tape drive, flash memory, or the like, and the mass storage device 384 may store programs and/or data. The computing device 360 may also include a user interface 388 to facilitate interaction between the system 300 and a user. The user interface 388 may include a keyboard, mouse, light pen, game control pad, touch interface, or other device.
The system 300 may also include a controller (not pictured) which interfaces with the eye tracking device 302 in order to interact with programs executed by the processor unit 370. The system 300 may also execute one or more general computer applications (not pictured), such as a video game, which may incorporate aspects of eye gaze tracking as sensed by the tracking device 302 and processed by the tracking programs 373, foveation adjustment programs 374, and foveation rendering programs 375.
The computing device 360 may include a network interface 390, configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods. The network interface 390 may incorporate suitable hardware, software, firmware or some combination thereof to facilitate communication via a telecommunications network. The network interface 390 may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The network interface 390 may also include the aforementioned wireless transceiver that facilitates wireless communication with the eye tracking device 302 and display device 379. The computing device 360 may send and receive data and/or requests for files via one or more data packets 399 over a network.
* Foveated Rendering*
In some implementations, foveated rendering may augment computational resource savings from leveraging knowledge of blinks or saccadic masking. Foveated rendering can reduce computational complexity in rendering graphics, while still preserving essential details in regions of interest in the image presented by the display. Foveated rendering reduces computation by performing high resolution rendering on regions of interest (ROI) of the displayed image where the fovea is focused and low resolution rendering outside this region. In addition to lowering the computational cost, one can also use these ROIs to compress the frame for transmission. To utilize foveated rendering, an image display device, such as a head-mounted display (HMD) would use eye gaze tracking technology to determine where the user is focusing on the screen.
Foveated rendering may be implemented by adjusting certain parameters of the rendering process based on screen location. Such adjustment may, e.g., vary the pixel resolution of the rendered image based on screen location. Alternatively, the density of vertices used to render three-dimensional objects may vary by screen location.
Using Foveated rendering images as input to compression, one can use varying levels of compression for each rendered region. The output is one or more compression streams with varying levels of compression or quality. For the foveal ROI, the highest quality settings are used giving minimal or no compression. However, for regions outside the fovea, the eye is less sensitive and therefore higher compression is acceptable. The result is a reduction in the bandwidth required for frame transmission while preserving quality of important regions.
FIGS. 4A-4B illustrate an example of adjustment of vertex density to implement foveated rendering in the context of a Virtual Reality (VR) environment. In conventional FOV displays, three-dimensional geometry is rendered using a planar projection to the view plane. However, rendering geometry onto displays, especially high FOV view planes, can be very inefficient and result in significant latency and performance issues. These issues can cause the displayed frame rate to drop below desired levels, creating a jarring, non-immersive experience for a user, and potentially inducing motion sickness in a user immersed in a VR environment.
Additionally, regions of the display near the edge of the screen, or regions which the user is not viewing, or not likely to view, hold much less meaningful information than regions near the center or to which a user’s attention is currently directed. When rendering a scene conventionally, these regions have the same number of vertices and the time spent rendering equal sized regions on the screen is the same. Other parts of the screen, such as the rear-view mirrors in the driving scene depicted in FIG. 4A, may be more important. These parts of the image are referred to as regions of interest 480.
FIG. 4B illustrates an example of a VR environment in which the scene information is rendered using foveated tessellation in accordance with aspects of the present disclosure. By utilizing foveated tessellation of real-time graphics rendering, detail may be added and subtracted from a 3D mesh for regions of interest 480 and corresponding silhouette edges based on a variety of parameters, e.g., camera distance, user attention, user eye movement, or depth of field. Detail in the areas surrounding regions of interest 480 can be defined as transition regions 482, and detail in these areas may be rendered such that the areas contain less detail than the areas of interest 480 but more detail than the peripheral regions 483. This may be accomplished by rendering the transition regions 482 to establish, for example, a mathematical relationship between the pixel density distributions of the area of interest 480 and the peripheral region 483 (See FIGS. 6C-6D, below). Such foveated tessellation can reduce computational load and or rendering time for an image. Reductions in computational load and/or rendering time may alternatively by achieved in other parts of the graphics processing pipeline by selectively reducing the pixel resolution outside of the regions of interest 480.
Experiments have shown, e.g., that by utilizing foveated tessellation, the rendering time of a 3D mesh or wireframe can be reduced by a factor of roughly 4.times. or more, as fewer vertex computations are required in rendering the image in the tessellation and certain parts of the graphics pipeline subsequent to tessellation.
In some implementations, subsequent graphics processing may utilize a rasterization stage that approximates a projection of the vertices onto a curved viewport. In such implementations, the density of the projected vertices may be determined for selected portions of the screen space corresponding to the region(s) of interest 480, such that a higher density of vertices is present in the region(s) of interest, while the density of the projected vertices is lower in remaining regions of the screen space. This can be accomplished by reducing the density of vertices for portions of the screen that are determined to be outside the region(s) of interest 480. In alternative embodiments, the density of vertices may be increased in selected portions of the screen space such that a higher density of vertices is present in a portion or portions of interest, and the density of vertices in the remaining portion or portions of the screen space is not increased. Accordingly, aspects of the present disclosure utilize a screen space transformation of the type described above to reduce a GPU’s computational load by effectively reducing the number of vertex computations for the area of the screen space that is to be rendered.
Foveated rendering may be limited by the capabilities of the gaze tracking system. Performance of gaze tracking systems depend on a multitude of factors, including the placement of light sources (IR, visible, etc.) and cameras, whether user is wearing glasses or contacts, HMD optics, frame rate, exposure time, camera optics, tracking system latency, rate of eye movement, shape of eye (which changes during the course of the day or can change as a result of movement), eye conditions, e.g., lazy eye, macular degeneration, gaze stability, fixation on moving objects, scene being displayed to user, and user head motion.
In systems and devices that utilize eye tracking, errors in eye tracking and associated latencies in tracking, as well as the inability to track eye state information, cause these systems to need a much greater radius of high resolution on the display than is theoretically needed to preserve the high fidelity for the user. This issue is particularly prevalent in virtual reality systems, wherein performance is dependent on screen resolution. In such systems, high levels of rendering are required in order to maintain an ideal resolution, however, much of rendering performed is unnecessary since a user’s eyes only focus on a small part of the screen. Foveated rendering techniques allow for a system to provide high resolution to the foveal region and lower resolution to transitional and/or peripheral regions outside the foveal region. However, even for systems utilizing foveated rendering, the rendered foveal region is often larger than necessary as compared to the theoretical foveal region, as the region is rendered to account for the variability in human vision. An example of this variability involves the speed and accuracy of a user’s saccade to fixation.
Aspects of the present disclosure address these problems with an adaptive foveated rendering technique. Using error bounds and information regarding the state of the eye collected with the eye tracking data the system could adjust the fovea rendering radius to compensate for state changes or errors in the tracking results. The fovea rendering radius may be adjusted with respect to state changes occurring in real time, or alternatively, may be adjusted in anticipation of a state change. Additionally, using knowledge of latencies in the system one could scale the fovea region. The end result would allow for more savings in rendering complexity while maintaining the highest possible resolution for the user.
In order to provide the most accurate scaling of the fovea region and maximize the savings in rendering complexity while maintaining the highest possible resolution for the user, aspects of the present disclosure may be configured to determine the size and shape of the foveal region in advance based on a “worst case” scenario that accounts for the variability in human vision, determine estimates of error, state information, and latencies during gaze tracking, and dynamically resize the foveal region to provide the best balance of resolution quality and rendering performance.
According to aspects of the present disclosure, real-time adjustment of foveated rendering of an image containing one or more regions of interest may be implemented by a graphics processing method 500 illustrated in FIG. 5A. To understand the context of the graphics processing method, certain conventional elements of computer graphics processing are shown. Specifically, a computer graphics program may generate three-dimensional object vertex data 501 for one or more objects in three-dimensional virtual space. The object vertex data 501 may represent the coordinates of points in the virtual space that correspond to points on the surfaces of geometric shapes that make up one or more objects. An object may be made up of one or more geometric shapes. The object vertex data 501 may define a set of object vertices that correspond to points on the surfaces of one or more shapes that make up an object. By way of example, and not by way of limitation, each geometric shape may be represented by a shape identifier (e.g., cone, sphere, cube, pyramid, etc.), coordinates in virtual space for a relevant location of the object, e.g., coordinates of a centroid of the shape, and relevant geometric parameters for determining coordinates of a point on the surface of the shape. By way of example, for the case of a sphere, the relevant location could be the location of the center of the sphere and the relevant geometric parameter could be the radius of the sphere.
As indicated at 502, the object vertex data 501 may be subject to a process that projects the object vertices onto a screen space in a conventional manner for 3D graphics processing. In some implementations, the projection may approximate a projection of the vertices onto a curved viewport. Polygons may then be generated from the projected vertices, as indicated at 504. The generation of polygons from the projected vertices may be done in a conventional manner. Specifically, edges may be defined between selected pairs of polygons and selected edges may be associated together as polygons. The resulting polygon data 503 includes information identifying the vertices and edges that make up the polygons. The polygon data 503 is used by the method 500, which tessellates the polygons represented by the polygon data in accordance with aspects of the present disclosure.
The method 500 includes determining foveation data 505 for one or more regions of interest of the screen space, as indicated at 506 and determining vertex density information 507V and/or pixel resolution date 507P, as indicated at 508. The polygon data 503, foveation data 505 and vertex density information 507V are used to tessellate the polygons in accordance with aspects of the present disclosure, as indicated at 510 to produce tessellated vertex data 509. The resulting tessellated vertex data is then used in subsequent graphics processing, as indicated at 512.
Determining the foveation data 505 may involve obtaining the gaze tracking data as indicated at 506A, determining gaze tracking error and/or state parameters at 506B, and adjusting regions of interest at 506C. Gaze tracking data may be obtained, e.g., as discussed above with respect to FIG. 1A-1B, and FIG. 2. The size and/or shape of ROI may be adjusted to compensate for errors in the gaze tracking by determining the error bounds of gaze tracking data. The ROI may also be adjusted to compensate for the state of the user’s eye in the gaze tracking data by determining the state information parameters of the gaze tracking data. Such adjustment of foveated rendering is described in detail in U.S. patent application Ser. No. 15/086,645 filed the same date as the present application, the entire contents of which are incorporated by reference herein.
Gaze tracking error parameters determined at 506B may include a confidence interval regarding the current gaze position, which may be determined by examining the rotational velocity and acceleration of a user’s eye for change from last position. Alternatively, the gaze tracking error and/or state parameters may include a prediction of future gaze position determined by examining the rotational velocity and acceleration of eye and extrapolating the possible future positions of the user’s eye. In general terms, the fixed sampling rate or exposure time of the gaze tracking system may lead to a greater error between the determined future position and the actual future position for a user with larger values of rotational velocity and acceleration. To accommodate for the larger error the size of the foveal region may increase accordingly.
The gaze tracking error parameters may also include a measurement of the eye speed, e.g., the rotation rate. For a slow moving eye, the region of interest may be adjusted at 506C to be smaller, and peripheral and/or transition regions may be adjusted so that they are larger. For a fast moving eye, the size of the foveal region may increase, and the peripheral and/or transition regions may be made smaller.
The regions of interest may also be adjusted at 506C based on state parameters established from the metrics of a user’s blink. During a blink, a user’s vision may not be focused on the presented images for up to 20-30 frames. However, upon exiting the blink, the user’s gaze direction may not correspond to the last measured gaze direction as determined from the gaze tracking data. The metrics of a user’s blink or blinks may be established from the gaze tracking data and regions of interest for subsequent images may be adjusted based on those metrics. For example, the metrics may include, but are not limited to, the measured start and end times of the blink of a user, as well as the predicted end times. The adjustment may involve, for example, decreasing the size of the foveal region and increasing the size of the peripheral and/or transition regions during the blink, and increasing the size of the foveal region and decreasing the size of the peripheral and/or transition regions as the blink is determined or predicted to be ending as a result of the blink cycle data.
Gaze tracking state parameters may also be related to saccades. A user’s gaze direction will have shifted to a different region of interest when the saccade is exited. The metrics of a user’s saccade(s) may be established from the gaze tracking data 506A. These metrics may include, but are not limited to, the measured start and end times of the saccades of a user as well as the predicted end times. The regions of interest for subsequent images may accordingly be adjusted at 506C, e.g., based on the predicted time that will elapse during the saccade. This may involve, for example, decreasing the size of the foveal region while increasing the size of the peripheral and/or transition regions during the saccade, and increasing the size of the foveal region and decreasing the size of the peripheral and/or transition regions as the saccade is determined to be ending as a result of the saccade cycle data. Alternatively, the foveal region may be eliminated completely when it determined that a saccade is either occurring or about to occur, and a new foveal region and peripheral/transition region boundaries may be established based on gaze tracking data 506A obtained during the saccade. Gaze tracking state parameters may also account for a transition in gaze direction between areas of interest as a result of a change in depth of field between presented images that triggers a saccade.
Gaze tracking state parameters may be used to adapt for color blindness. For example, regions of interest may be present in an image presented to a user such that the regions would not be noticeable by a user who has a particular color blindness. Gaze tracking data may be analyzed to determine whether or not the user’s gaze identified or responded the area of interest, for example, as a result of the user’s changed gaze direction. The region or regions of interest in subsequent images presented to a color blind user may be adjusted order to account for the user’s condition, by, for example, utilizing a different color scheme in subsequently presented areas of interest.
Gaze tracking data may also be analyzed to provide a measurement of the gaze stability of a user. Gaze stability may be determined, e.g., by measuring the microsaccadic radius of the user’s eye; smaller fixation overshoot and undershoot equates to a more stable gaze in a user. Accordingly, the regions of interest for subsequent images may be adjusted at to be smaller for a user with greater gaze stability, or larger for a user with less gaze stability.
Gaze tracking error or state parameters may also measure a user’s ability to fixate on moving objects. These parameters may include the measurement of the capability of a user’s eye to undergo smooth pursuit and the maximum object pursuit speed of the eyeball. Typically, a user with excellent smooth pursuit capabilities experiences less jitter in the movement of the eyeball. The region of interest in subsequent images may be adjusted correspondingly at to decrease the size of the region where a user experiences less jitter, or increased where a user experiences increased jitter. The region may also be adjusted at 506C in accordance with a maximum pursuit speed of a user’s eye, as a faster measured pursuit speed would require a larger region of interest as compared to the region of interest necessary for a person with a slower pursuit speed. Gaze tracking error parameters may also include determination of eye movement as a precursor to head movement. Offset between head and eye orientation can affect certain error parameters as discussed above, e.g., in smooth pursuit or fixation. As a result, a larger offset between head an eye orientation may require the adjustment of a region of interest for a subsequent image so to make the region larger, whereas a smaller offset would result in a smaller region of interest.
Once the adjustments at 506C have taken place, foveated images may be generated and presented to the user. By way of example, and not by way of limitation, in tessellating the polygons at 510, the foveation data 505 and vertex density information 207V may define tessellation parameters that vary with respect to location in screen space and are used by a hardware or software tessellator to generate a triangle-based tessellation of the polygons. Examples of such tessellation parameters include the so-called TessFactor, which controls the degree of fineness of the mesh generated by the Direct3D 11 programmable graphics pipeline, which is part of Windows 7 from Microsoft Corporation.
In general terms the foveation data 505 and vertex density information 507V are used to modify a conventional tessellation process to account for the fact that not all regions of the screen space are equally important to the one who views images of the screen space on a display. The foveal regions represent portions of the screen space that are determined by an application to be important to the viewer and are therefore allocated a greater share of available graphics computation resources. The foveal region data 205 may include information identifying a location of a centroid of the foveal region in the screen space, a size of the foveal region relative to the screen space, and shape of the foveal region. A foveal region may be determined at 506 by an application to be of interest to a viewer because (a) it is a region the viewer is likely look at, (b) it is a region the viewer is actually looking at, or (c) it is a region it is desired to attract the user to look at.
With respect to (a), the foveal region may be determined to be likely to be looked at in a context sensitive manner. In some implementations, the application may determine that certain portions of the screen space or certain objects in a corresponding three-dimensional virtual space are “of interest” and such objects may be consistently drawn using a greater number of vertices than other objects in the virtual space. Foveal regions may be contextually defined to be of interest in a static or dynamic fashion. As a non-limiting example of static definition, a foveal region may be a fixed part of the screen space, e.g., a region near the center of the screen, if it is determined that this region is the part of the screen space that a viewer is most likely to look at. For example, if the application is a driving simulator that displays an image of a vehicle dashboard and a windshield, the viewer is likely to be looking at these portions of the image. In this example, the foveal region may be statically defined in the sense that the region of interest is a fixed portion of the screen space. As a non-limiting example of dynamic definition, in a video game a user’s avatar, fellow gamer’s avatars, enemy artificial intelligence (AI) characters, certain objects of interest (e.g., the ball in a sports game) may be of interest to a the user. Such objects of interest may move relative to the screen space and therefore the foveal region may be defined to move with the object of interest.
With respect to (b) it is possible to track the viewers gaze to determine which portion of a display the viewer is looking at. Tracking the viewer’s gaze may be implemented by tracking some combination of the user’s head pose and the orientation of the pupils of the user’s eyes. Some examples of such gaze tracking are described e.g., in U.S. Patent Application Publications Numbers 2015/0085250, 2015/0085251, and 2015/0085097, the entire contents of all of which are incorporated herein by reference. Further details of estimation of head pose can be found, e.g., in “Head Pose Estimation in Computer Vision: A Survey” by Erik Murphy, in IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, Vol. 31, No. 4, April 2009, pp 607-626, the contents of which are incorporated herein by reference. Other examples of head pose estimation that can be used in conjunction with embodiments of the present invention are described in “Facial feature extraction and pose determination”, by Athanasios Nikolaidis Pattern Recognition, Vol. 33 (Jul. 7, 2000) pp. 1783-1791, the entire contents of which are incorporated herein by reference. Additional examples of head pose estimation that can be used in conjunction with embodiments of the present invention are described in “An Algorithm for Real-time Stereo Vision Implementation of Head Pose and Gaze Direction Measurement”, by Yoshio Matsumoto and Alexander Zelinsky in FG ‘00 Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000, pp 499-505, the entire contents of which are incorporated herein by reference. Further examples of head pose estimation that can be used in conjunction with embodiments of the present invention are described in “3D Face Pose Estimation from a Monocular Camera” by Qiang Ji and Ruong Hu in Image and Vision Computing, Vol. 20, Issue 7, 20 Feb. 2002, pp 499-511, the entire contents of which are incorporated herein by reference.
With respect to (c), it is a common cinematic device to change the depth of focus of a scene to focus on a portion of interest, e.g., a particular actor who is speaking. This is done to draw the viewer’s attention to the portion of the image that is in focus. According to aspects of the present disclosure, a similar effect may be implemented with computer graphics by moving the foveal region to a desired portion of the screen so that that portion has a greater density of vertices and is rendered in greater detail as a result.
……
…..
……
