Facebook Patent | Dynamic Tiling For Foveated Rendering

Patent: Dynamic Tiling For Foveated Rendering

Publication Number: 20200394830

Publication Date: 20201217

Applicants: Facebook

Abstract

A method for providing imagery to a user on a display includes receiving eye tracking data and determining a gaze location on the display or a gaze vector using the eye tracking data. The method can also include defining a first tile using the gaze location on the display or the gaze vector. The first tile may have a height and a width, the height and width being determined using the eye tracking data. The method can further include defining multiple additional tiles to fill an entire area of the display. The method can also include providing a portion of an image using the first tile at a first image quality and providing another portion of the image at a second image quality using at least one of the multiple additional tiles.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims the benefit of and priority to U.S. Provisional Patent No. 62/861,106, filed Jun. 13, 2019, the entire disclosure of which is incorporated by reference herein.

FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to display systems. More particularly, the present disclosure relates to systems and methods for using eye tracking with foveated rendering.

BACKGROUND

[0003] The present disclosure relates generally to augmented reality (AR), mixed reality (MR) and/or virtual reality (VR) systems. AR, MR, and VR systems can be used to present various images, including three-dimensional (3D) images, to a user. For example, AR, MR or VR headsets can be used to present images to the user in a manner that is overlaid on a view of a real world environment or that simulates a virtual environment. To render convincing, life-like AR/MR/VR images, the AR/MR/VR systems can use eye tracking to track the user’s eye and accordingly present images.

SUMMARY

[0004] One implementation of the present disclosure is a method for providing imagery to a user on a display. The method may include receiving eye tracking data. The method may further include determining a gaze location on the display or a gaze vector using the eye tracking data. The method can further include defining a first tile using the gaze location on the display or the gaze vector. The first tile may have a height and a width, the height and width being determined using the eye tracking data. The method can further include defining multiple additional tiles to fill an entire area of the display. The method can also include providing a portion of an image using the first tile at a first image quality and providing another portion of the image at a second image quality using at least one of the multiple additional tiles.

[0005] Another implementation of the present disclosure is a head mounted display for providing imagery to a user. The head mounted display can include a display, and eye tracker, and a processor. The eye tracker can be configured to provide eye tracking data. The processor can be configured to determine a gaze location on the display or a gaze vector using the eye tracking data. The processor can further be configured to define a first tile using the gaze location or the gaze vector. The processor can further be configured to define multiple additional tiles to fill an area of the display. A portion of an image may be provided on the display using the first tile at a first image quality and other portions of the image may be provided on the display at a second image quality using at least one of the multiple additional tiles.

[0006] Another implementation of the present disclosure is a display for providing foveated imagery to a user. The display may include processing circuitry configured to track a gaze direction or a gaze vector of the user’s eye. The processing circuitry may be further configured to define a first tile based on the gaze direction or gaze vector of the user’s eye. A location of the first tile is defined using the gaze direction or the gaze vector and is for a first image quality. The processing circuitry may be configured to define one or more additional tiles for a second image quality different than the first image quality. The processing circuitry can be configured to provide imagery using the first tile and each of the one or more additional tiles. The processing circuitry may also be configured to redefine the location of the first tile in response to a change in the gaze direction or gaze vector.

[0007] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings:

[0009] FIG. 1 is a block diagram of a display system, according to some embodiments.

[0010] FIG. 2 is a schematic diagram of a head-mounted display (HMD) system, according to some embodiments.

[0011] FIG. 3 is a spherical coordinate system showing a gaze vector of a user’s eye, according to some embodiments.

[0012] FIG. 4 is a top view of the gaze vector of FIG. 3 directed towards a display screen, according to some embodiments.

[0013] FIG. 5 is a side view of the gaze vector of FIG. 3 directed towards a display screen, according to some embodiments.

[0014] FIG. 6 is a tiled display screen with foveated rendering, according to some embodiments.

[0015] FIG. 7 is the tiled display screen of FIG. 6 with foveated rendering after the user’s gaze has shifted to a new location, according to some embodiments.

[0016] FIG. 8 is the tiled display screen of FIG. 6 with foveated rendering for a first gaze position error, according to some embodiments.

[0017] FIG. 9 is the tiled display screen of FIG. 6 with foveated rendering for a second gaze position error, according to some embodiments.

[0018] FIG. 10 is a tiled display screen with smaller tiles than the display screen of FIGS. 6-9 and foveated rendering, according to some embodiments.

[0019] FIG. 11 is the tiled display screen of FIG. 10 with foveated rendering after a user’s gaze has shifted to a new location, according to some embodiments.

[0020] FIG. 12 is a portion of a tiled display screen with a user’s gaze centered at a first tile, according to some embodiments.

[0021] FIG. 13 is the portion of the tiled display screen of FIG. 12 after the user’s gaze has shifted to a new location near a tile border and a new tile has been defined, according to some embodiments.

[0022] FIG. 14 is the portion of the tiled display screen of FIG. 12 with smaller gaze position error and smaller tiles, according to some embodiments.

[0023] FIG. 15 is the portion of the tiled display screen of FIG. 14 after the user’s gaze has shifted to a new location near a tile border and new tiles have been defined to account for the shifted user’s gaze, according to some embodiments.

[0024] FIG. 16 is a flow diagram of a process for dynamically tiling a display screen and providing foveated rendering based on gaze direction, according to some embodiments.

[0025] FIG. 17 is a flow diagram of a process for dynamically defining one or more new tiles based on a user’s gaze position, according to some embodiments.

[0026] FIG. 18 is a flow diagram of a process for determining if new tiles should be defined based on a user’s gaze, according to some embodiments.

[0027] FIG. 19 is a flow diagram of a process for determining if new tiles should be defined based on gaze error, according to some embodiments.

[0028] FIG. 20 is a block diagram of a computing environment that the systems of FIGS. 1 and 2 can be implemented in, according to some embodiments.

[0029] FIG. 21 is a front view of a lens of a display with various image quality regions, according to some embodiments.

DETAILED DESCRIPTION

Overview

[0030] Referring generally to the FIGURES, systems and methods for providing foveated images to a user are shown, according to some embodiments. Tiles are used to display the foveated images, according to some embodiments. A user’s eye is tracked to determine gaze direction and/or focal point, according to some embodiments. The gaze direction and/or focal point is used to determine a gaze location on a display, according to some embodiments. The gaze location can be a gaze location (X, Y) on a two dimensional display or a gaze location (X, Y, Z) on a three dimensional display. In some embodiments, an error associated with the gaze direction and/or the gaze location on the display is also determined.

[0031] Various eye tracking sensors, devices, hardware, software, etc., are used to track the user’s eye and to determine the gaze location on the display, according to some embodiments. A tile is defined that is centered at the gaze location on the display, and additional tiles are also defined to fill out remaining area of the display, according to some embodiments. The tile that is centered at the gaze location on the display is updated in real-time to track the user’s gaze direction as it changes, according to some embodiments. In some embodiments, the tile that is centered at the gaze location is for imagery at a high image quality, and tiles that are adjacent or near or otherwise on the display are for imagery at a same or lower quality.

[0032] The use of the tile centered at the gaze location and the additional tiles facilitates a foveated display, according to some embodiments. In some embodiments, imagery of the tile centered at the gaze location and the additional tiles are rasterized to achieve a foveated display of the imagery. In some embodiments, sizes and/or shapes of the various tiles are adjusted in real-time to account for error associated with the gaze direction of the user’s eye. The system can rasterize image data for tiles with lower resolution, lower detail, or lower image quality and upscale (e.g., using nearest neighbor) the rasterized imagery to provide a smooth transition between tiles, according to some embodiments. In some embodiments, tiles that are further away from the gaze location are associated with lower image quality. Advantageously, the systems and methods described herein facilitate reduced power consumption of processing circuitry, but still provide detailed imagery within the fovea region, according to some embodiments.

Virtual Reality or Augmented Reality System

[0033] Referring now to FIG. 1, a system 100 can include a plurality of sensors 104a … n, processing circuitry 116, and one or more displays 164. System 100 can be implemented using HMD system 200 described in greater detail below with reference to FIG. 2. Ssytem 100 may be configured as an HMD system or a head wearable display (HWD) system. System 100 can be implemented using the computing environment described with reference to FIG. 4. System 100 can incorporate features of and be used to implement features of virtual reality (VR) systems. At least some of processing circuitry 116 can be implemented using a graphics processing unit (GPU). The functions of processing circuitry 116 can be executed in a distributed manner using a plurality of processing units.

[0034] Processing circuitry 116 may include one or more circuits, processors, and/or hardware components. Processing circuitry 116 may implement any logic, functions or instructions to perform any of the operations described herein. Processing circuitry 116 can include any type and form of executable instructions executable by any of the circuits, processors or hardware components. The executable instructions may be of any type including applications, programs, services, tasks, scripts, libraries processes and/or firmware. Any of eye tracker 118, error manager 120, tile generator 122, an image renderer 124 may be any combination or arrangement of circuitry and executable instructions to perform their respective functions and operations. At least some portions of processing circuitry 116 can be used to implement image processing executed by sensors 104.

[0035] Sensors 104a … n can be image capture devices or cameras, including video cameras. Sensors 104a … n may be cameras that generate images of relatively low quality (e.g., relatively low sharpness, resolution, or dynamic range), which can help reduce the size, weight, and power requirements of system 100. For example, sensors 104a … n can generate images having resolutions on the order of hundreds of pixels by hundreds of pixels. At the same time, the processes executed by system 100 as described herein can be used to generate display images for presentation to a user that have desired quality characteristics, including depth characteristics.

[0036] Sensors 104a … n (generally referred herein as sensors 104) can include any type of one or more cameras. The cameras can be visible light cameras (e.g., color or black and white), infrared cameras, or combinations thereof. Sensors 104a … n can each include one or more lenses 108 a … j generally referred herein as lens 108). In some embodiments, sensor 104 can include a camera for each lens 108. In some embodiments, sensor 104 include a single camera with multiple lenses 108 a … j. In some embodiments, sensor 104 can include multiple cameras, each with multiple lenses 108. The one or more cameras of sensor 104 can be selected or designed to be a predetermined resolution and/or have a predetermined field of view. In some embodiments, the one or more cameras are selected and/or designed to have a resolution and field of view for detecting and tracking objects, such as in the field of view of a HMD for augmented reality. The one or more cameras may be used for multiple purposes, such as tracking objects in a scene or an environment captured by the image capture devices and performing calibration techniques described herein.

[0037] The one or more cameras of sensor 104 and lens 108 may be mounted, integrated, incorporated or arranged on an HMD to correspond to a left-eye view of a user or wearer of the HMD and a right-eye view of the user or wearer. For example, an HMD may include a first camera with a first lens mounted forward-facing on the left side of the HMD corresponding to or near the left eye of the wearer and a second camera with a second lens mounted forward-facing on the right-side of the HMD corresponding to or near the right eye of the wearer. The left camera and right camera may form a front-facing pair of cameras providing for stereographic image capturing. In some embodiments, the HMD may have one or more additional cameras, such as a third camera between the first and second cameras an offers towards the top of the HMD and forming a triangular shape between the first, second and third cameras. This third camera may be used for triangulation techniques in performing the depth buffer generations techniques of the present solution, as well as for object tracking.

[0038] System 100 can include a first sensor (e.g., image capture device) 104a that includes a first lens 108a, first sensor 104a arranged to capture a first image 112a of a first view, and a second sensor 104b that includes a second lens 108b, second sensor 104b arranged to capture a second image 112b of a second view. The first view and the second view may correspond to different perspectives, enabling depth information to be extracted from first image 112a and second image 112b. For example, the first view may correspond to a left eye view, and the second view may correspond to a right eye view. System 100 can include a third sensor 104c that includes a third lens 108c, third sensor 104c arranged to capture a third image 112c of a third view. As described with reference to FIG. 2, the third view may correspond to a top view that is spaced from an axis between first lens 108a and second lens 108b, which can enable system 100 to more effectively handle depth information that may be difficult to address with first sensor 104a and second sensor 104b, such as edges (e.g., an edge of a table) that are substantially parallel to the axis between first lens 108a and second lens 108b.

[0039] Light of an image to be captured by sensors 104a … n can be received through the one or more lenses 108 a … j. Sensors 104a … n can include sensor circuitry, including but not limited to charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) circuitry, which can detect the light received via the one or more lenses 108a … j and generate images 112a … k based on the received light. For example, sensors 104a … n can use the sensor circuitry to generate first image 112a corresponding to the first view and second image 112b corresponding to the second view. The one or more sensors 104a … n can provide images 112a … k to processing circuitry 116. The one or more sensors 104a … n can provide images 112a … k with a corresponding timestamp, which can facilitate synchronization of images 112a … k when image processing is executed on images 112a … k, such as to identify particular first and second images 112a, 112b representing first and second views and having the same timestamp that should be compared to one another to calculate gaze information.

[0040] Sensors 104 can include eye tracking sensors 104 or head tracking sensors 104 that can provide information such as positions, orientations, or gaze directions of the eyes or head of the user (e.g., wearer) of an HMD. In some embodiments, sensors 104 are inside out tracking cameras configured to provide images for head tracking operations. Sensors 104 can be eye tracking sensors 104 that provide eye tracking data 148, such as data corresponding to at least one of a position or an orientation of one or both eyes of the user. Sensors 104 can be oriented in a direction towards the eyes of the user (e.g., as compared to sensors 104 that capture images of an environment outside of the HMD). For example, sensors 104 can include at least one fourth sensor 104d (e.g., as illustrated in FIG. 2) which can be oriented towards the eyes of the user to detect sensor data regarding the eyes of the user.

[0041] In some embodiments, sensors 104 output images of the eyes of the user, which can be processed to detect an eye position or gaze direction (e.g., first gaze direction) of the eyes. In some embodiments, sensors 104 process image data regarding the eyes of the user, and output the eye position or gaze direction based on the image data. In some embodiments, sensors 104 optically measure eye motion, such as by emitting light (e.g., infrared light) towards the eyes and detecting reflections of the emitted light.

[0042] As discussed further herein, an eye tracking operation can include any function, operation, routine, logic, or instructions executed by system 100 or components thereof to track data regarding eyes of the user, such as positions or orientations (e.g., gaze directions) of the eyes of the user as the eyes of the user move during use of the HMD. For example, the eye tracking operation can be performed using at least one of one or more sensors 104 or eye tracker 118. For example, the eye tracking operation can process eye tracking data 148 from sensor 104 to determine an eye position, gaze direction, gaze vector, focal point, point of view, etc., shown as gaze vector 136 of eye(s) of the user. In some embodiments, the eye tracking operation can be performed using eye tracker 118 that is implemented using a portion of processing circuitry 116 that is coupled with, mounted to, integral with, implemented using a same circuit board as, or otherwise provided with one or more sensors 104 that detect sensor data regarding the eyes of the user. In some embodiments, the eye tracking operation can be performed using an eye tracker 118 that receives sensor data by a wired or wireless connection from the one or more sensors 104 that are configured to detect sensor data regarding the eyes of the user (e.g., images of the eyes of the user); for example, eye tracker 118 can be implemented using the same processing hardware as at least one of error manager 120, tile generator 122, and/or image renderer 124. Various such combinations of sensor hardware of sensors 104 and/or processing hardware of processing circuitry 116 may be used to implement the eye tracking operation.

[0043] Eye tracker 118 can generate gaze vector 136 in various manners. For example, eye tracker 118 can process eye tracking data 148 to identify one or more pixels representing at least one of a position or an orientation of one or more eyes of the user. Eye tracker 118 can identify, using eye tracking data 148, gaze vector 136 based on pixels corresponding to light (e.g., light from light sources/light emitting diodes/actuators of sensors 104, such as infrared or near-infrared light from actuators of sensors 104, such as 850 nm light eye tracking) reflected by the one or more eyes of the user. Eye tracker 118 can use light from various illumination sources or reflections in the HMD or AR system, such as from waveguides, combiners, or lens cameras. Eye tracker 118 can determine gaze vector 136 or eye position by determining a vector between a pupil center of one or more eyes of the user and a corresponding reflection (e.g., corneal reflection). Gaze vector 136 can include position data such as at least one of a position or an orientation of each of one or more eyes of the user. The position data can be in three-dimensional space, such as three-dimensional coordinates in a Cartesian, spherical, or other coordinate system. Gaze vector 136 can include position data including a gaze direction of one or more eyes of the user. In some embodiments, eye tracker 118 includes a machine learning model. The machine learning model can be used to generate eye position or gaze vector 136 based on eye tracking data 148.

[0044] Processing circuitry 116 can include an error manager 120. Error manager 120 is configured to receive eye tracking data 148 from sensor(s) 104 and determine gaze error 126 associated with gaze vector 136. Gaze error 126 can include error for eye position, gaze direction, eye direction, etc., of gaze vector 136 (e.g., gaze location, gaze vector 302, etc.). Error manager 120 can receive eye tracking data 148 from sensor(s) 104 and perform an error analysis to determine gaze error 126. Error manager 120 monitors eye tracking data 148 over time and/or gaze vector 136 over time and determines gaze error 126 based on eye tracking data 148 and/or gaze vector 136, according to some embodiments. In some embodiments, error manager 120 provides gaze error 126 to tile generator 122. Eye tracker 118 also provides gaze vector 136 to tile generator 122, according to some embodiments. Error manager 120 can be configured to identify, determine, calculate, etc., any of rotational velocity, prediction error, fixation error, a confidence interval of gaze vector 136, random error, measurement error of gaze vector 136, etc.

[0045] Processing circuitry 116 includes tile generator 122, according to some embodiments. Tile generator 122 is configured to receive gaze vector 136 from eye tracker 118 and gaze error 126 from error manager 120, according to some embodiments. Tile generator 122 is configured to define one or more tiles 128 (e.g., tiles 602 shown in FIGS. 6-15 and 21), superpixels, collection of pixels, render areas, resolution areas, etc., for image renderer 124, according to some embodiments. Tile generator 122 generates tiles 128 based on gaze vector 136, a focal gaze location of the user’s eyes, a reference gaze location, a direction of gaze, eye position, a point of interest, etc., according to some embodiments. Tile generator 122 generates various subsets of tiles 128 for display on display(s) 164 and corresponding resolutions, according to some embodiments. In some embodiments, tile generator 122 defines a first set of tiles 128 that should have a high resolution (e.g., a high level of detail, high image quality, etc.), a second set of tiles 128 that should have a medium resolution, and a third set of tiles that should have a low resolution. Tiles 128 include a corresponding size (e.g., height and width, number of pixels, gaze angles, etc.) for each tile 128, according to some embodiments.

[0046] In some embodiments, tiles 128 include data regarding a corresponding position on display(s) 164. For example, tile generator 122 generates multiple tiles 128 that collectively cover an entirety of display(s) 164 and associated positions within display(s) 164, according to some embodiments. Tile generator 122 provides tiles 128 to image renderer 124 for use in generating a rendered image 130, according to some embodiments. Tile generator 122 also generates or defines tiles 128 based on gaze error 126, according to some embodiments. In some embodiments, tile generator 122 divides a total area of display(s) 164 into various subsections, collection of pixels, etc., referred to as tiles 128. Tile generator 122 assigns a corresponding resolution to each of tiles 128, according to some embodiments. In some embodiments, tile generator 122 redefines tiles 128 periodically or dynamically based on updated or new gaze error 126 and/or gaze vector 136. In some embodiments, tile generator 122 defines a size, shape, position, and corresponding resolution of imagery for each of tiles 128. In some embodiments, any of the size, position, and corresponding resolution of imagery for each of tiles 128 is determined by tile generator 122 based on gaze vector 136 and/or gaze error 126.

[0047] Processing circuitry 116 includes image renderer 124, according to some embodiments. In some embodiments, image renderer 124 is configured to receive tiles 128 from tile generator 122 and use tiles 128 to generate an image for display(s) 164. In some embodiments, image renderer 124 receives image data 132 and uses tiles 128 to display the image data on display(s) 164. In some embodiments, image renderer 124 receives tiles 128 and image data 132 and generates a rendered image 130 based on tiles 128 and image data 132. Image renderer 124 uses the size, shape, position, and corresponding resolution of each of tiles 128 to rasterize image data 132 to generate rendered image 130, according to some embodiments.

[0048] Image renderer 124 is a 3D image renderer or 2D image renderer, according to some embodiments. Image renderer 124 uses image related input data to process, generate and render display or presentation images to display or present on one or more display devices, such as via an HMD, according to some embodiments. Image renderer 124 generates or creates 2D images of a scene or view for display on display 164 and representing the scene or view in a 3D manner, according to some embodiments. The display or presentation data (e.g., image data 132) to be rendered includes geometric models of 3D objects in the scene or view, according to some embodiments. Image renderer 124 determines, computes, or calculates the pixel values of the display or image data to be rendered to provide the desired or predetermined 3D image(s), such as 3D display data for images 112 captured by the sensor 104, according to some embodiments. Image renderer 124 receives images 112, tiles 128, and head tracking data 150 and generates display images using images 112.

[0049] Image renderer 124 can render frames of display data to one or more displays 164 based on temporal and/or spatial parameters. Image renderer 124 can render frames of image data sequentially in time, such as corresponding to times at which images are captured by the sensors 104. Image renderer 124 can render frames of display data based on changes in position and/or orientation to sensors 104, such as the position and orientation of the HMD. Image renderer 124 can render frames of display data based on left-eye view(s) and right-eye view(s) such as displaying a left-eye view followed by a right-eye view or vice-versa.

[0050] Image renderer 124 can generate the display images using motion data regarding movement of the sensors 104a … n that captured images 112a … k. For example, the sensors 104a … n may change in at least one of position or orientation due to movement of a head of the user wearing an HMD that includes the sensors 104a … n (e.g., as described with reference to HMD system 200 of FIG. 2). Processing circuitry 116 can receive the motion data from a position sensor (e.g., position sensor 220 described with reference to FIG. 2). Image renderer 124 can use the motion data to calculate a change in at least one of position or orientation between a first point in time at which images 112a … k were captured and a second point in time at which the display images will be displayed, and generate the display images using the calculated change. Image renderer 124 can use the motion data to interpolate and/or extrapolate the display images relative to images 112a … k. Although image renderer 124 is shown as part of processing circuitry 116, the image renderer may be formed as part of other processing circuitry of a separate device or component, such as the display device, for example within the HMD.

[0051] System 100 can include one or more displays 164. The one or more displays 164 can be any type and form of electronic visual display. The displays may have or be selected with a predetermined resolution and refresh rate and size. The one or more displays can be of any type of technology such as LCD, LED, ELED or OLED based displays. The form factor of the one or more displays may be such to fit within the HMD as glasses or goggles in which the display(s) are the lens within the frame of the glasses or goggles. Displays 164 may have a refresh rate the same or different than a rate of refresh or frame rate of processing circuitry 116 or image renderer 124 or the sensors 104.

[0052] Referring now to FIG. 2, in some implementations, an HMD system 200 can be used to implement system 100. HMD system 200 can include an HMD body 202, a left sensor 104a (e.g., left image capture device), a right sensor 104b (e.g., right image capture device), and display 164. HMD body 202 can have various form factors, such as glasses or a headset. The sensors 104a, 104b can be mounted to or integrated in HMD body 202. The left sensor 104a can capture first images corresponding to a first view (e.g., left eye view), and the right sensor 104b can capture images corresponding to a second view (e.g., right eye view). HMD system 200 may also be a HWD system.

[0053] HMD system 200 can include a top sensor 104c (e.g., top image capture device). Top sensor 104c can capture images corresponding to a third view different than the first view or the second view. For example, top sensor 104c can be positioned between the left sensor 104a and right sensor 104b and above a baseline between the left sensor 104a and right sensor 104b. This can enable top sensor 104c to capture images with depth information that may not be readily available to be extracted from the images captured by left and right sensors 104a, 104b. For example, it may be difficult for depth information to be effectively extracted from images captured by left and right sensors 104a, 104b in which edges (e.g., an edge of a table) are parallel to a baseline between left and right sensors 104a, 104b. Top sensor 104c, being spaced from the baseline, can capture the third image to have a different perspective, and thus enable different depth information to be extracted from the third image, than left and right sensors 104a, 104b.

[0054] HMD system 200 can include processing circuitry 116, which can perform at least some of the functions described with reference to FIG. 1, including receiving sensor data from sensors 104a, 104b, and 104c as well as eye tracking sensors 104, and processing the received images to calibrate an eye tracking operation.

[0055] HMD system 200 can include communications circuitry 204. Communications circuitry 204 can be used to transmit electronic communication signals to and receive electronic communication signals from at least one of a client device 208 or a server 212. Communications circuitry 204 can include wired or wireless interfaces (e.g., jacks, antennas, transmitters, receivers, transceivers, wire terminals) for conducting data communications with various systems, devices, or networks. For example, communications circuitry 204 can include an Ethernet card and port for sending and receiving data via an Ethernet-based communications network. Communications circuitry 204 can communicate via local area networks (e.g., a building LAN), wide area networks (e.g., the Internet, a cellular network), and/or conduct direct communications (e.g., NFC, Bluetooth). Communications circuitry 204 can conduct wired and/or wireless communications. For example, communications circuitry 204 can include one or more wireless transceivers (e.g., a Wi-Fi transceiver, a Bluetooth transceiver, a NFC transceiver, a cellular transceiver). For example, communications circuitry 204 can establish wired or wireless connections with the at least one of the client device 208 or server 212. Communications circuitry 204 can establish a USB connection with the client device 208.

[0056] HMD system 200 can be deployed using different architectures. In some embodiments, the HMD (e.g., HMD body 202 and components attached to HMD body 202) comprises processing circuitry 116 and is self-contained portable unit. In some embodiments, the HMD has portions of processing circuitry 116 that work in cooperation with or in conjunction with any type of portable or mobile computing device or companion device that has the processing circuitry or portions thereof, such as in the form of a staging device, a mobile phone or wearable computing device. In some embodiments, the HMD has portions of processing circuitry 116 that work in cooperation with or in conjunction with processing circuitry, or portions thereof, of a desktop computing device. In some embodiments, the HMD has portions of processing circuitry 116 that works in cooperation with or in conjunction with processing circuitry, or portions thereof, of a server computing device, which may be deployed remotely in a data center or cloud computing environment. In any of the above embodiments, the HMD or any computing device working in conjunction with the HMD may communicate with one or more servers in performing any of the functionality and operations described herein.

[0057] The client device 208 can be any type and form of general purpose or special purpose computing device in any form factor, such as a mobile or portable device (phone, tablet, laptop, etc.), or a desktop or personal computing (PC) device. In some embodiments, the client device can be a special purpose device, such as in the form of a staging device, which may have the processing circuitry or portions thereof. The special purpose device may be designed to be carried by the user while wearing the HMD, such as by attaching the client device 208 to clothing or the body via any type and form of accessory attachment. The client device 208 may be used to perform any portion of the image and rendering processing pipeline described in connection with FIGS. 1 and 3. The HMD may perform some or other portions of the image and rendering processing pipeline such as image capture and rendering to display 164. The HMD can transmit and receive data with the client device 208 to leverage the client device 208’s computing power and resources which may have higher specifications than those of the HMD.

[0058] Server 212 can be any type of form of computing device that provides applications, functionality or services to one or more client devices 208 or other devices acting as clients. In some embodiments, server 212 can be a client device 208. Server 212 can be deployed in a data center or cloud computing environment accessible via one or more networks. The HMD and/or client device 208 can use and leverage the computing power and resources of server 212. The HMD and/or client device 208 can implement any portion of the image and rendering processing pipeline described in connection with FIGS. 1 and 3. Server 212 can implement any portion of the image and rendering processing pipeline described in connection with FIGS. 1 and 3, and in some cases, any portions of the image and rendering processing pipeline not performed by client device 208 or HMD. Server 212 may be used to update the HMD and/or client device 208 with any updated to the applications, software, executable instructions and/or data on the HMD and/or client device 208.

[0059] System 200 can include a position sensor 220. The position sensor 220 can output at least one of a position or an orientation of the body 202. As the image capture devices 104a, 104b, 104c can be fixed to the body 202 (e.g., at predetermined locations relative to the position sensor 220), the position sensor 220 can output at least one of a position or an orientation of each sensor 104a, 104b, 104c. The position sensor 220 can include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, or a magnetometer (e.g., magnetic compass).

[0060] System 200 can include a varifocal system 224. Varifocal system 224 can have a variable focal length, such that varifocal system 224 can change a focus (e.g., a point or plane of focus) as focal length or magnification changes. Varifocal system 224 can include at least one of a mechanical lens, liquid lens, or polarization beam plate. In some embodiments, varifocal system 224 can be calibrated by processing circuitry 116 (e.g., by a calibrator), such as by receiving an indication of a vergence plane from a calibrator which can be used to change the focus of varifocal system 224. In some embodiments, varifocal system 224 can enable a depth blur of one or more objects in the scene by adjusting the focus based on information received from the calibrator so that the focus is at a different depth than the one or more objects.

[0061] In some embodiments, display 164 includes one or more waveguides. The waveguides can receive (e.g., in-couple) light corresponding to display images to be displayed by display 164 from one or more projectors, and output (e.g., out-couple) the display images, such as for viewing by a user of the HMD. The waveguides can perform horizontal or vertical expansion of the received light to output the display images at an appropriate scale. The waveguides can include one or more lenses, diffraction gratings, polarized surfaces, reflective surfaces, or combinations thereof to provide the display images based on the received light. The projectors can include any of a variety of projection devices, such as LCD, LED, OLED, DMD, or LCOS devices, among others, to generate the light to be provided to the one or more waveguides. The projectors can receive the display images from processing circuitry 116 (e.g., from image renderer 124). The one or more waveguides can be provided through a display surface (e.g., glass), which can be at least partially transparent to operate as a combiner (e.g., combining light from a real world environment around the HMD with the light of the outputted display images).

[0062] Display 164 can perform foveated rendering based on the calibrated eye tracking operation, which can indicate a gaze point corresponding to the gaze direction generated by the eye tracking operation. For example, processing circuitry 116 can identify at least one of a central region of the FOV of display 164 (e.g., a plurality of pixels within a threshold distance from the gaze point) peripheral region of the FOV of display 164 based on the gaze point (e.g., a peripheral region represented by a plurality of pixels of the display images that are within a threshold distance of an edge of the display images or more than a threshold distance from the gaze point). Processing circuitry 116 can generate the display images to have a less quality (e.g., resolution, pixel density, frame rate) in the peripheral region than in the central region, which can reduce processing demand associated with operation of HMD system 200.

Gaze Vector and Point of Interest

[0063] Referring now to FIGS. 3-5, the gaze vector is shown in greater detail, according to some embodiments. Gaze vector 136 as used by processing circuitry 116 is represented graphically in FIGS. 3-5 as gaze vector 302, according to some embodiments. It should be understood that while gaze vector 136 is represented in a spherical coordinate system, gaze vector 136 can also be represented in a Cartesian coordinate system, a polar coordinate system, a cylindrical coordinate system, etc., or any other coordinate system. Gaze vector 302 is used by processing circuitry 116 to determine a focal point or gaze location 402 of the user’s eyes, according to some embodiments.

[0064] Referring particularly to FIG. 3, a spherical coordinate system includes gaze vector 302, and a user’s eye (or eyes) 140. Eye 140 is shown as a centerpoint of the spherical coordinate system, and gaze vector 302 extends radially outwards from eye 140, according to some embodiments. In some embodiments, a direction of gaze vector 302 is defined by one or more angles, shown as angle .theta..sub.1 and angle .theta..sub.2. In some embodiments, angle .theta..sub.1 represents an angular amount between gaze vector 302 and a vertical axis 304. In some embodiments, angle .theta..sub.2 represents an angular amount between gaze vector 302 and a horizontal axis 306. In some embodiments, vertical axis 304 and horizontal axis 306 are substantially perpendicular to each other and both extend through eye 140.

[0065] In some embodiments, eye tracker 118 of processing circuitry 116 is configured to determine values of both angle .theta..sub.1 and angle .theta..sub.2 based on eye tracking data 148. Eye tracker 118 can determine the values of angles .theta..sub.1 and .theta..sub.2 for both eyes 140, according to some embodiments. In some embodiments, eye tracker 118 determines the values of angles .theta..sub.1 and .theta..sub.2 and provides the angles to error manager 120 and/or tile generator 122 as gaze vector 136.

[0066] Referring particularly to FIGS. 4 and 5 gaze vector 302 can be used to determine a location of a point of interest, a focal point, a gaze point, a gaze location, a point, etc., shown as gaze location 402. Gaze location 402 has a location on display 164, according to some embodiments. In some embodiments, gaze location 402 has an x location and a y location (e.g., a horizontal and a vertical location) on display 164. In some embodiments, gaze location 402 has a location in virtual space, real space, etc. In some embodiments, gaze location 402 has a two dimensional location. In some embodiments, gaze location 402 has a three-dimensional location. Gaze location 402 can have a location on display 164 relative to an origin or a reference point on display 164 (e.g., a center of display 164, a corner of display 164, etc.). Gaze location 402 and gaze vector 302 can be represented using any coordinate system, or combination of coordinate systems thereof. For example, gaze location 402 and/or gaze vector 302 can be defined using a Cartesian coordinate system, a polar coordinate system, a cylindrical coordinate system, a spherical coordinate system, a homogeneous coordinate system, a curvilinear coordinate system, an orthogonal coordinate system, a skew coordinate system, etc.

[0067] In some embodiments, tile generator 122 and/or eye tracker 118 are configured to use a distance d between the user’s eye 140 and display 164. The distance d can be a known or sensed distance between the user’s eye 140 and display 164, according to some embodiments. For example, sensors 104 can measure, detect, sense, identify, etc., the distance d between the user’s eye 140 and display 164. In some embodiments, the distance d is a known distance based on a type or configuration of the HMD.

[0068] The distance d and the angles .theta..sub.1 and .theta..sub.2 can be used by eye tracker 118 to determine gaze vector 302/136. In some embodiments, eye tracker 118 uses the distance d and the angles .theta..sub.1 and .theta..sub.2 to determine the location of gaze location 402. In some embodiments, eye tracker 118 provides the distance d and the angles .theta..sub.1 and .kappa..sub.2 to tile generator 122. Tile generator 122 uses the distance d and the angles .theta..sub.1 and .theta..sub.2 to determine the location of gaze location 402 relative to a reference point on display 164.

[0069] FIG. 4 is a top view of display 164 and the user’s eye 140, according to some embodiments. FIG. 4 shows the angle .theta..sub.1, according to some embodiments. Likewise, FIG. 5 is a side view of display 164 and the user’s eye 140 and shows the angle .theta..sub.2, according to some embodiments. Tile generator 122 and/or eye tracker 118 use the distance d and the angles .theta..sub.1 and .theta..sub.2 to determine the position/location of gaze location 402, according to some embodiments. In some embodiments, tile generator 122 uses the position/location of gaze location 402 to define tiles 128. It should be understood that while display 164 is shown as a generally flat display screen, in some embodiments, display 164 is a curved, arcuate, etc., display screen. A rectangular display screen is shown for ease of illustration and description only. Accordingly, all references to “local positions,” “local coordinates,” “Cartesian coordinates,” etc., of display 164 may refer to associated/corresponding angular values of angle .theta..sub.1 and/or angle .theta..sub.2.

Tile Definition

[0070] Referring to FIG. 6, display 164 is shown to include tiles 602, according to some embodiments. In some embodiments, tiles 602 are defined by tile generator 122 based on the location/position of gaze location 402. Gaze location 402 represents an approximate location that the user is viewing, according to some embodiments. In some embodiments, gaze location 402 represents the point or location that the user’s gaze is directed towards.

[0071] Display 164 includes tiles 602 having a width w and a height h, according to some embodiments. In some embodiments, the width w is referred to as a length along a central horizontal axis of display 164 (e.g., a straight horizontal axis if display 164 is straight, a curved horizontal axis if display 164 is curved). Likewise, the height h is referred to as a height along a vertical axis of display 164 (e.g., a straight vertical axis if display 164 is straight, a curved vertical axis if display 164 is curved about horizontal axis 306). In some embodiments, the width w and the height h are angular values of angle .theta..sub.1 and .theta..sub.2 for a given distance from the user’s eye 140. For example, the width w of tiles 602 may be an 11 degrees (e.g., an amount of 11 degrees for angle .theta..sub.1 from opposite sides of tile 602), and the height h of tiles 602 may be 17 degrees (e.g., an amount of 17 degrees for angle .theta..sub.2 from top and bottom sides of tile 602).

[0072] In some embodiments, each of tiles 602 have an area A=wh. In some embodiments, each of tiles 602 includes a collection of pixels that display a portion of an image that is displayed on display 164 to the user. Tiles 602 collectively display the image to the user on display 164, according to some embodiments. The image can be a rendered image of three dimensional objects, particles, characters, terrain, maps, text, menus, etc. In some embodiments, the image is a virtual reality image. In some embodiments, the image is an augmented reality image (e.g., imagery is overlaid or projected over a real-world image). For example, if display 164 is a display of a HMD virtual reality system, the image can be a representation of a virtual reality, a virtual space, a virtual environment, etc. Likewise, if display 164 is a display of a HMD augmented reality system, the image can be a representation of projected objects, characters, particles, text, etc., having a location in virtual space that matches or corresponds or tracks a location in real space.

[0073] Referring still to FIG. 6, display 164 includes a first set of tiles 602a, a second set of tiles 602b, and a third set of tiles 602c, according to some embodiments. In some embodiments, the resolution of tiles 602c is greater than the resolution of tiles 602b, and the resolution of tiles 602b is greater than the resolution of tiles 602a. In some embodiments, processing power of processing circuitry 116 can be reduced by decreasing the resolution of tiles 602 that are in the user’s peripheral view. For example, tiles 602 that are currently being viewed out of the corner of the user’s eye may be rendered at a lower resolution without the user noticing the reduced or lower resolution.

[0074] Tile generator 122 determines which tiles 602 should be rendered by image renderer 124 at a higher resolution, according to some embodiments. In some embodiments, tile generator 122 includes a predetermined number of tile groups with ascending or descending resolutions. Tile generator 122 can monitor the location/position of gaze location 402 in real-time (e.g., before every frame is rendered and provided to the user on display 164) to determine which of tiles 602 should be rendered at the highest resolution and to determine which of tiles 602 should be rendered in a lower resolution to save processing power of processing circuitry 116. In some embodiments, tile generator 122 determines that tiles 602 which correspond to the position/location of gaze location 402 should have the highest resolution. For example, if gaze location 402 is located within a specific tile 602, tile generator 122 can determine that the specific tile 602 should have the highest resolution or should be associated with the highest resolution set of tiles 602. In some embodiments, tile generator 122 also determines that tiles 602 that are adjacent, neighboring, or nearby (e.g., directly adjacent) the specific tile 602 should also be rendered with the same high resolution (e.g., tiles 602a). In some embodiments, tile generator 122 determines that tiles 602 that are adjacent, next to, neighbor, nearby, etc., the high resolution tiles 602 should be rendered at a medium resolution (e.g., tiles 602b). Tile generator 122 can determine that all other tiles 602 of display 164 which are in the user’s peripheral view should be rendered at the low resolution (e.g., tiles 602c).

[0075] In some embodiments, tile generator 122 adjusts which of tiles 602 should be rendered at the high resolution, the medium resolution, and the lower resolution. For example, tile generator 122 can re-assign tiles 602 to redefine the various tile groups in response to the location/position of gaze location 402 changing. For example, as shown in FIGS. 6 and 7, the location/position of gaze location 402 on display 164 changes as the user re-directs their gaze, according to some embodiments. Tile generator 122 redefines the first set of tiles 602a so that tiles 602 that are near or at the location/position of gaze location 402 are rendered at the highest quality/resolution, according to some embodiments. Likewise, tile generator 122 redefines the second set of tiles 602b and the third set of tiles 602c to account for the shift in the location/position of gaze location 402. In this way, tile generator 122 can redefine the resolution of each of tiles 602 in response to the user changing or redirecting their gaze. In some embodiments, tile generator 122 uses gaze vector 302 in real-time to re-calculate the location/position of gaze location 402. Tile generator 122 uses the re-calculated or updated location/position of gaze location 402 to redefine or reassign tiles 602 (e.g., to re-determine positions, sizes, shapes, definitions, locations, resolutions, etc., of tiles 602).

[0076] Referring now to FIGS. 8 and 9, tile generator 122 uses gaze error 126 to adjust or redefine tiles 602, according to some embodiments. Gaze error 126 is represented by error 604, according to some embodiments. Error 604 indicates an area or range of locations that gaze location 402 may be located, according to some embodiments. In some embodiments, error 604 is determined based on eye tracking data 148 by error manager 120. In some embodiments, error 604 is used by tile generator 122 to determine if the user’s gaze may be directed to an area or location other than the location/position of gaze location 402. Error 604 represents an area that gaze location 402 may be located, corresponding to an error of angle .theta..sub.1 and angle .theta..sub.2, according to some embodiments. For example, if angle .theta..sub.1 has a high amount of error, error 604 (represented by a two-dimensional area on display 164) may have a larger height, according to some embodiments. Likewise, if angle .theta..sub.2 has a high amount of error, error 604 may have a larger width, according to some embodiments. Error 604 indicates angular jitter, wobble, measurement uncertainty, etc., of gaze vector 302 (and/or gaze location 402) about vertical axis 304 and/or horizontal axis 306, according to some embodiments.

[0077] If error 604 increases, a corresponding number of tiles 602 that the user’s gaze may be directed towards also increases, according to some embodiments. In response to error 604 increasing, tile generator 122 can redefine the first set of tiles 602a, the second set of tiles 602b, and the third set of tiles 602c to account for the increased error 604. For example, as shown in FIG. 8, five tiles 602 are included in the first set of tiles 602a and are rendered at the high resolution, according to some embodiments. However, as error 604 increases (shown in FIGS. 8 and 9), tile generator 122 includes additional tiles 602 to ensure that the user’s gaze is not directed towards a lower resolution tile 602, according to some embodiments. Error manager 120 can receive eye tracking data 148 in real-time, determine gaze error 126, translate gaze error 126 (e.g., the error of angle .theta..sub.1 and/or angle .theta..sub.2) to a range of positions/locations on display 164 (shown as error 604), and provide error 604 and/or gaze error 126 to tile generator 122. Tile generator 122 defines the size, area, number of pixels, location, resolution, etc., of each tile 602 based on gaze error 126 (e.g., based on error 604), according to some embodiments. In some embodiments, tile generator 122 provides any of the defined information regarding tiles 602 to image renderer 124. In some embodiments, tile generator 122 uses both the location/position of gaze location 402 as well as error 604 to define tiles 602. In some embodiments, the positions, locations, sizes, etc., of tiles 602 are predefined or predetermined, and tile generator 122 defines a corresponding resolution for each tile 602 based on the location/position of gaze location 402 as well as error 604.

[0078] For example, if angle .theta..sub.1 has an error .DELTA..theta..sub.1 and angle .theta..sub.2 has an error .DELTA..theta..sub.2, error manager 120 determines an error .DELTA.x in the x position of gaze location 402, and an error .DELTA.y in the y position of gaze location 402, according to some embodiments. In some embodiments, the error .DELTA.x and the error .DELTA.y define error 604. In some embodiments, error manager 120 translates the uncertainty, error, range of values, etc., of the angles .theta..sub.1 and .theta..sub.2 to errors .DELTA.x and .DELTA.y in the x direction and they direction of display 164 to define error 604.

[0079] In some embodiments, error 604 has the shape of a circle (as shown in FIGS. 6-15). In some embodiments, error 604 has the shape of an ellipse, a square, a rectangle, etc. Tile generator 122 can identify, based on the location/position of gaze location 402 and error 604, which tiles 602 lie within error 604 (e.g., if tiles 602 have predetermined/predefined locations and sizes) and assigns these tiles 602 with high resolution rendering. If error 604 increases, additional tiles 602 that now are within error 604 can be re-assigned by tile generator 122 to render with high resolution. Likewise, if error 604 decreases, tiles 602 that are not within error 604 can be re-assigned by tile generator 122 to render with lower resolution. In this way, foveated rendering can be achieved that accounts for error in the gaze vector, gaze direction, point of interest, focal point, etc., of the user.

[0080] In some embodiments, display 164 includes smaller tiles 602 as shown in FIGS. 10-11. The height h and width w of tiles 602 are defined by tile generator 122 based on display capabilities of display 164, according to some embodiments. In some embodiments, the height h and width w of tiles 602 are predefined for display 164. For example, displays that are capable of displaying higher resolution images (e.g., displays with more pixels) can have additional tiles 602 compared to displays with lower resolution (e.g., less pixels). Tile generator 122 can dynamically re-define positions, sizes, shapes, etc., of each of tiles 602 and corresponding render/resolution qualities in response to gaze location 402 shifting (e.g., shifting to the right as shown in FIGS. 10-11).

[0081] Referring particularly to FIGS. 12-13, gaze location 402 may sometimes remain within a corresponding tile 602, according to some embodiments. However, gaze location 402 can also move near an intersection of multiple tiles 602. For example, if gaze location 402 moves to the intersection between the four tiles 602 shown in FIG. 12, a situation arises in which tile generator 122 must determine which of the four tiles 602 should be rendered in the high resolution, according to some embodiments. Tile generator 122 can render all four tiles 602 in the high resolution, however, this may not necessarily be the most processing efficient solution.

[0082] In cases when gaze location 402 moves near an intersection of one or more tiles 602, tile generator 122 can redefine a layout, size, position, arrangement, etc., of tiles 602. In some embodiments, tile generator 122 redefines the layout, size, position, arrangement, etc., of tiles 602 in response to gaze location 402 moving a predetermined distance from a centerpoint of a tile 602. For example, if gaze location 402 is within a predetermined distance from the centerpoint of the bottom left tile 602 as shown in FIG. 12, tile generator 122 maintains the current layout, size, position, arrangement, etc., of tiles 602, according to some embodiments. However, if gaze location 402 moves the predetermined distance or more from the centerpoint of the bottom left tile 602 shown in FIG. 12, tile generator 122 redefines the layout, size, position, location, arrangement, etc., of tiles 602, according to some embodiments. In some embodiments, the predetermined distance is zero (e.g., if gaze location 402 moves from the centerpoint of tile 602 any amount, tile generator 122 redefines tiles 602). In some embodiments, the predetermined distance is a number of pixels, an angular amount for a given distance from the user’s eye 140 (e.g., an amount of angular rotation about either vertical axis 304 and/or horizontal axis 306), a distance (e.g., in x and y directions) on display 164, etc. In some embodiments, the predetermined distance is a portion of, a percentage of, or is proportional to a height h or a width w of a tiles 602 that gaze location 402 is currently within. The predetermined distance includes both x and y components or corresponding angular amounts, according to some embodiments.

[0083] In some embodiments, tile generator 122 redefines tiles 602 in response to an intersection (e.g., a border, a boundary, etc.) of one or more tiles 602 lying within error 604. For example, gaze location 402 as shown in FIG. 12 includes error 604, but none of the intersections of adjacent tiles 602 lie within error 604, according to some embodiments. However, when gaze location 402 moves to the position shown in FIG. 13, the intersections or borders of adjacent tiles 602 are now within error 604, according to some embodiments. In response to gaze location 402 moving off center of a corresponding tile 602 that gaze location 402 is currently within, or in response to an intersection, border, boundary, etc., of the corresponding tile 602 lying within error 604, tile generator 122 redefines the layout, size, position, arrangement, etc., of tiles 602, according to some embodiments. Tile generator 122 redefines the layout, size, positions, arrangements, etc., of tiles 602 such that gaze location 402 is at a centerpoint of a corresponding tile 602, according to some embodiments. Advantageously, this allows processing circuitry 116 to provide high detail or high resolution imagery on new tile 602n instead of displaying high detail/high resolution imagery on all four of tiles 602, thereby decreasing processing power requirements.

[0084] In the example shown in FIGS. 12-13, tile generator 122 defines a new tile 602n that is centered about the location/position of gaze location 402, according to some embodiments. Tile generator 122 assigns new tile 602n the highest resolution for rendering, according to some embodiments.

[0085] In some embodiments, tile generator 122 generates, defines, determines definitions of, etc., tiles 602 for the rest of display 164 based on new tile 602n. Tile generator 122 uses any of the techniques, functionality, etc., described in greater detail above with reference to FIGS. 6-11 to assign or map resolution qualities to the rest of tiles 602 that surround new tile 602n and fill out the entirety of display 164, according to some embodiments. New tile 602n is defined as being positioned centrally at gaze location/position 402, according to some embodiments. In some embodiments, a size of new tile 602n is very small compared to gaze angles .theta..sub.1 and/or .theta..sub.2 or gaze angles .theta..sub.1 and/or .theta..sub.2 are very large compared to the size of new tile 602n. If the size of new tile 602n is very small compared to gaze angles .theta..sub.1 and/or .theta..sub.2, new tile 602n can include multiple tiles (e.g., 2 tiles, 4 tiles, etc.). In some embodiments, new tile 602n does not have a fixed size or ratio between height and width (e.g., new tile 602n can be a square tile or a rectangular tile). In some embodiments, surrounding tiles 602 can be different shapes and/or different sizes (e.g., differently sized squares and/or rectangles).

……
……
……