Google Patent | Foveated Compression of Display Streams

Patent: Foveated Compression of Display Streams

Publication Number: 20190019315

Publication Date: 2019-01-17

Applicants: Google

Abstract

A processing unit is configured to render first pixels representative of a high-acuity region in the image and second pixels representative of a low-acuity region in the image. A shaper is configured to reorganize the first pixels based on at least one dimension of the low-acuity region. A multiplexer is configured to multiplex the reorganized first pixels and the second pixels to form a display stream. An encoder is configured to compress the display stream for transmission to a display. A decoder configured to decompress the display stream. A demultiplexer is configured to demultiplex the first pixels and the second pixels. Another processing unit is configured to blend the first pixels and the second pixels to form blended pixel values representative of the image for presentation on a screen.

Background

Foveated rendering exploits the falloff in acuity of the human eye at the visual periphery to conserve power and computing resources that are consumed while generating contents for head mounted displays (HMDs) in augmented reality (AR) and virtual reality (VR) applications. In foveated rendering, the user’s central gaze direction is determined, either by the center of system field-of-view or by eye tracking devices. The user’s field-of-view is then subdivided into a high-acuity region that surrounds the central gaze direction and one or more lower-acuity regions in the visual periphery. The high-acuity region includes a portion of the field-of-view that is within some angular distance of the central gaze direction. The angular distance from the central gaze direction is also referred to as the eccentricity. The lower-acuity regions include portions of the field-of-view that are at larger eccentricities. For example, the high-acuity region can include a portion of the field-of-view that is within an eccentricity of 5-10.degree., which corresponds to a portion of the field-of-view that projects to a retinal region in the human eye called the fovea. Content is rendered at high resolution within the high-acuity region, e.g., by rendering the pixels at a resolution corresponding to the native resolution supported by the display. Content in the low-acuity regions at eccentricities larger than 5-10.degree. are rendered at lower resolutions, thereby reducing the power and computing resources needed to render the pixels. The rendered pixels in the low-acuity region can subsequently be upsampled and blended with the pixels in the high-acuity region to generate display pixels at the native resolution of the display, e.g., using well-known interpolation techniques such as bilinear interpolation.

The limited bandwidth of current standard transmission protocols (e.g. DisplayPort) can become a bottleneck for uncompressed image data produced by high-resolution applications. For example, a Ultra High Definition (UHD) display at 60 frames per second with a 30-bit color depth requires a data rate of about 17.3 gigabits per second, which is the current limit of the DisplayPort specification. Higher interface data rates demand more power, can increase the interface wire count, and require more shielding to prevent interference with the device’s wireless services. These attributes increase system hardware complexity and weight, which is particularly undesirable in an HMD that is worn by a user. Graphics processing systems can therefore compress the display stream using techniques such as display stream compression (DSC), which is a standardized, visually lossless method of performing inline video compression for standard displays. A DSC encoder includes a frame buffer to store pixel values for an incoming frame, a line buffer to store values of a line of reconstructed pixel values, and a rate buffer to store the output bitstream. Dimensions of the buffers correspond to dimensions of images in the display stream. For example, each line in a buffer can store values for 1280 pixels to correspond to the number of pixels in a line of a 1280.times.1280 image. A DSC decoder implements a complementary set of buffers.