Google Patent | Video Frame Codec Architectures

Patent: Video Frame Codec Architectures

Publication Number: 10659797

Publication Date: 20200519

Applicants: Google

Abstract

Techniques and apparatuses are described for video frame codec architectures. A frame decompressor decompresses compressed frames to produce decompressed frames. A frame decompressor controller arbitrates shared access to the frame decompressor. Multiple cores of an SoC request to receive a decompressed frame from the frame decompressor via the frame decompressor controller. The frame decompressor controller can implement a request queue and can order the servicing of requests based on priority of the requests or requesting cores. The frame decompressor controller can also establish a time-sharing protocol for access by the multiple cores. In some implementations, a video decoder is logically integrated with the frame decompressor and stores portions of a decompressed frame in a video buffer, and a display controller retrieves the portions for display using a synchronization mechanism. In analogous manners, a frame compressor controller can arbitrate shared access to a frame compressor for the multiple cores.

BACKGROUND

Noon Electronic devices play integral roles in manufacturing, communication, healthcare, commerce, social interaction, and entertainment. For example, electronic devices power the server farms that provide cloud-based, distributed computing functionality for commerce and communication. Devices with computing power are also embedded in many different types of modern equipment, from medical devices to appliances and from vehicles to industrial tools. Further, one electronic device–the smartphone–has become a necessity to literally always have at hand.

Many electronic devices, such as those with a camera or a display screen, manipulate video data. For example, a video may be obtained using a security camera and then enhanced to improve some visual aspect, such as the clarity or contrast. Existing video data can also be manipulated to improve the appearance of individual video frames for presentation on a display screen of a smartphone or television monitor. For example, the video data of a movie may be processed to improve the realism of artificial graphics or to upscale the display resolution. Video image data is also manipulated in industrial and medical environments. For instance, the image data from a three-dimensional body scan can be stitched together into a video presentation for review and analysis by a doctor.

In any of these situations, the manipulation of video data is a processing-intensive task. This is due in part to the size or amount of information typically present in video data. Consequently, the area of an integrated circuit (IC) chip that is devoted to handling video data can be greater than that for other types of data. The difficulty of handling video data has been exacerbated by the ever-increasing display resolution of videos that electronic devices are expected to handle. For example, high-definition (HD) video has approximately four times more video data than standard-definition (SD) video, and ultra-HD (UHD) or 4K video has approximately four times more video data than HD video.

The amount of video data that electronic devices are expected to process has therefore increased dramatically over the last decade or so. Video data processing demands are expected to further increase in the coming years as virtual reality (VR) and artificial reality (AR) usage becomes more common. Accordingly, electronic device manufacturers continue to strive to improve the ability of electronic devices to handle ever-increasing amounts of video data.

This background description is provided to generally present the context of the disclosure. Unless otherwise indicated herein, material described in this section is neither expressly nor impliedly admitted to be prior art to the present disclosure or the appended claims.

SUMMARY

Techniques and apparatuses are described for video frame codec architectures. These techniques and apparatuses enable integrated circuit (IC) chips to process high bandwidth video data using a lower amount of circuitry resources while also facilitating a streamlined workflow to upgrade to newer frame compression/decompression technologies, including lossless ones. To do so, the inclusion of multiple separate frame decompressing units at multiple different cores of an IC chip is obviated. Instead, a frame decompressor can provide a frame decompression service to multiple different cores that function as frame decompressor client circuits. A frame decompressor controller facilitates the sharing of the decompression service using a queueing or priority mechanism that orders decompression requests received from one or more cores of the multiple cores. The frame decompressor controller can also arbitrate access to the frame decompressor in accordance with a time-sharing protocol. In an example implementation, the frame decompressor is co-located with a video decoder client circuit, and the frame decompressor is shared at least with a display controller. Similarly, a frame compressor can provide a frame compression service to multiple different cores that function as frame compressor client circuits. A frame compressor controller facilitates the sharing of the compression service with the multiple cores. Further, a frame compression service and a frame decompression service can both be provided in a single IC chip to be shared across multiple cores.

Aspects described below include an electronic device comprising a frame decompressor and a frame decompressor controller. The frame decompressor is configured to decompress multiple compressed frames to produce multiple decompressed frames. The frame decompressor controller is coupled to the frame decompressor and configured to arbitrate access to the frame decompressor for multiple cores. The multiple cores include a first core and a second core. The first core is coupled to the frame decompressor controller and is configured to obtain via the frame decompressor controller a decompressed frame of the multiple decompressed frames produced by the frame decompressor. The second core is coupled to the frame decompressor controller and is configured to obtain via the frame decompressor controller another decompressed frame of the multiple decompressed frames produced by the frame decompressor.

Aspects described below also include a method for sharing frame decompression circuitry between multiple cores. The method comprises accepting, from a first core, a first request for a first decompressed frame. The method also comprises decompressing a first compressed frame to produce the first decompressed frame. The first decompressed frame is provided to the first core responsive to the first request. The method additionally comprises accepting, from a second core, a second request for a second decompressed frame. The method further comprises decompressing a second compressed frame to produce the second decompressed frame. The second decompressed frame is provided to the second core responsive to the second request.

Aspects described below include another electronic device comprising a video decoder and a display controller. The video decoder is configured to decode a video stream to produce multiple decoded frames. The video decoder includes a frame compressor, a frame decompressor, and a frame decompressor controller. The frame compressor is configured to compress the multiple decoded frames to produce multiple compressed frames. The frame decompressor is configured to decompress the multiple compressed frames to produce multiple decompressed frames. The frame decompressor controller is coupled to the frame decompressor and is configured to arbitrate access to the frame decompressor. The display controller is coupled to the frame decompressor controller. The display controller is configured to obtain, via the frame decompressor controller, a decompressed frame of the multiple decompressed frames produced by the frame decompressor.

Aspects described below also include a system comprising a frame decompressor and multiple cores. The frame decompressor is configured to decompress multiple compressed frames to produce multiple decompressed frames. The multiple cores include a first core and a second core. The first core is coupled to the frame decompressor and is configured to obtain a decompressed frame of the multiple decompressed frames. The second core is coupled to the frame decompressor and is configured to obtain another decompressed frame of the multiple decompressed frames. The system also comprises control means for controlling the frame decompressor to arbitrate access to the frame decompressor for the multiple cores, including for the first core and the second core. Additionally or alternatively, the system can comprise a frame compressor configured to compress multiple uncompressed frames to produce multiple compressed frames. The first and second cores can each obtain a respective compressed frame of the multiple compressed frames. Thus, the system can also comprise control means for controlling the frame compressor to arbitrate access to the frame compressor for the multiple cores, including for the first core and the second core.

BRIEF DESCRIPTION OF THE DRAWINGS

Apparatuses of and techniques for implementing video frame codec architectures are described with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:

FIG. 1 illustrates an example environment including a printed circuit board in which video frame codec architectures can be implemented.

FIG. 2 illustrates other aspects of an example environment in which video frame codec architectures can be implemented.

FIG. 3 illustrates a system-on-chip (SoC) having an example implementation of a video frame codec architecture that includes a frame compressor-decompressor, a frame compressor-decompressor controller, and multiple cores.

FIG. 3-1 illustrates an SoC having an example implementation of a video frame codec architecture that includes a frame compressor, a frame compressor controller, and multiple cores.

FIG. 3-2 illustrates an SoC having an example implementation of a video frame codec architecture that includes a frame decompressor, a frame decompressor controller, and multiple cores.

FIG. 4 illustrates an example frame compressor-decompressor controller in conjunction with a frame decompressor and a core.

FIG. 5 illustrates an example approach to implement video frame codec architectures in which the multiple cores include a video decoder and a display controller.

FIG. 6 illustrates an example technique for routing a decompressed display frame from a video decoder to a display controller.

FIG. 7 illustrates an example scheme by a frame decompressor controller to manage requests for decompressed frames issued by multiple cores.

FIG. 8 illustrates an example scheme by a frame compressor-decompressor controller to establish a time-sharing protocol for sharing a frame compression resource or a frame decompression resource.

FIG. 9 illustrates example methods for operating video frame codec architectures as described herein.

FIG. 10 illustrates various components of an example electronic device that can implement video frame codec architectures in accordance with one or more implementations.

DETAILED DESCRIPTION

* Overview*

Data for a movie or other video consumes significant bandwidth in terms of both storage while at rest and transmission while propagating between electronic devices or the internal components thereof. As the display resolution for video has increased, the bandwidth demands have likewise increased. With ultra-HD (UHD) or 4K video, for example, there is approximately 15-20 times more video data to handle than with SD video, which was commonly used just a decade ago. This increased amount of data makes managing video difficult, even within a single integrated circuit (IC) chip. A typical system-on-chip (SoC), for instance, has a system bus that can become overloaded if raw decoded video data is transported between different SoC components using the system bus.

As is known, the bandwidth of a video can be reduced by encoding the video using some lossy codec, e.g. H.264. The encoded video can then be streamed, such as from a cloud server to a tablet computer, or saved, such as on a Blu-Ray disc or in flash memory. An end-user electronic device, for example, is then responsible for decoding the video for presentation on a display screen. Decoding entails transforming a stream of binary 1s and 0s, or bits, into individual decoded video frames, which can be displayed sequentially to represent the video. As part of this video encoding and decoding procedure, the video data is compressed and decompressed. With this video-level procedure (e.g., with an H.264 codec), the amount of compression is significant, but the compression is also lossy. Thus, because of the video-level compression/decompression procedure, some video information is lost to the point that image quality may be visibly changed.

This loss of video information is accepted to enable the large quantity of video data to be transmitted between devices or stored using a reasonable amount of memory. After decoding a video stream at the electronic device to produced decoded video frames, this large quantity of video data is recreated on a frame-by-frame basis as a group of frames or many individual frames. Each of these individual frames is still an appreciable amount of data. To handle this amount of video data at the electronic device, if a decoded video frame is not currently being processed or displayed, the decoded video frame is compressed to produce a compressed video frame. Because a finite number of frames are decoded from a stream at any given time, a lossless compression algorithm can be applied to the decoded frames in some implementations. A lossless compression and decompression frame-level procedure can prevent any further image degradation. A compressed video frame occupies less space in memory and consumes less bandwidth on an interconnect, such as a system bus of an SoC. Compressed video frames can therefore be transferred between different chips on a printed circuit board (PCB), between an SoC and a main memory, or even between different components of a single SoC.

To enable the use of per-video-frame, or frame-level, compression in existing systems, each respective component, or core, of an SoC that works with video data includes a respective decompression unit. With this straight-forward conventional approach, each core can independently produce a decompressed video frame from a compressed video frame and then process the compressed video frame in accordance with a given core’s video-related purpose. However, this straightforward approach has several attendant costs. First, an appreciable area on the IC die is devoted to duplicative video-frame decompression circuitry or compression circuitry. Second, the display path for presenting videos on a display screen includes a separate decompression unit. Third, the workflow to upgrade video frame compression and decompression technology for an SoC is significantly complicated. This complication results from multiple different cores each including an individual decompression unit or an individual compression unit (including, in some cases, both units). In other words, to upgrade to a newer, more-efficient decompression algorithm, each core that includes a compression unit or a decompression unit has to be simultaneously modified and then reintegrated with the rest of the SoC. This workflow upgrade complication therefore slows the adoption of improved compression/decompression algorithms in multi-core chips.

In contrast, certain implementations that are described herein use a frame compressor-decompressor that is shared. The frame compressor-decompressor can include a frame compressor or a frame decompressor (or both). Multiple cores of an IC chip, such as an SoC, can obtain a decompressed version of a compressed frame from, e.g., the shared frame decompressor. A given core can make a request for a decompressed frame, and the frame decompressor can provide a response that includes the requested decompressed frame. Thus, the frame decompressor can function to provide a frame decompression service, and each core can function as a client to the frame decompression service. Similarly, the frame compressor can function to provide a frame compression service, and each core can function as a client to the frame compression service. These frame-level compression and decompression services can use a lossless algorithm.

In some implementations, an SoC includes multiple different components, such as a video decoder and a display controller, that operate as different cores. The video decoder includes a stream decoder to decode a video stream to produce decoded frames. The video decoder also includes a frame compressor that produces compressed frames from the decoded frames. Multiple different components of the SoC are configured to process decompressed frames. For example, the video decoder may use a decompressed reference frame for further decoding of the video stream. Additionally, the display controller may use a decompressed display frame to present a video on a display screen.

To obtain any of these example types of decompressed frames, a frame decompressor decompresses the corresponding compressed frames. The frame decompressor can route a requested decompressed frame to a requesting core using one or more busses, at least one buffer, or some other routing mechanism. A frame decompressor controller arbitrates shared access to the frame decompressor between at least two cores, such as the video decoder and the display controller. The frame decompressor controller can manage a request queue of compressed frame requests. The management can entail ordering the servicing of the frame requests in accordance with a priority scheme. The frame decompressor controller can also establish a time-shared protocol for access to the frame decompressor. The time-shared protocol can include time slots that are assigned to different cores, the acceptance of interrupts that are issued by cores to seize control of the frame decompressor, and so forth.

In these manners, a frame compressor resource or a frame decompressor resource can be shared between two or more client cores that process video data. By sharing, e.g., a frame decompressor, the area of a SoC devoted to frame decompression is reduced. Additionally, a separate frame decompressor does not need to be inserted along a video display path. Further, the compression/decompression algorithm is decoupled from the overall SoC architecture. Consequently, the compression/decompression algorithm can be updated more easily and thus more frequently because fewer, or even one, frame compressor or frame decompressor is included on the SoC and thus subject to the updating workflow.

Example implementations in various levels of detail are discussed below with reference to the associated figures. The discussion below first describes an example operating environment, then example schemes and hardware, followed by example methods, and ends with an example electronic device and related example aspects.

* Example Environment*

FIG. 1 illustrates an example environment 100 including a printed circuit board 104 (PCB) in which video frame codec architectures can be implemented. As shown, the environment 100 includes an electronic device 102. The electronic device 102 includes at least one PCB 104. The PCB 104 includes one or more integrated circuits, such as an integrated circuit 106 (IC). As described below with reference to FIGS. 2 and 3, the PCB 104 can include other integrated circuits, such as at least one memory that is discrete from the IC 106. The IC 106 includes at least one frame decompressor 108-2; multiple cores 110-1, 110-2 … 110-n, with n representing an integer greater than one; and multiple video frames, such as at least one compressed frame 112 and at least one decompressed frame 114. FIG. 1 depicts just the frame decompressor 108-2 for simplicity while describing the example environment 100. However, a frame compressor-decompressor 108 and a frame compressor 108-1 are described below with reference to FIG. 3 and FIG. 3-1, respectively.

In example implementations, the frame decompressor 108-2 is communicatively coupled to at least a portion of the multiple cores 110-1 to 110-n. The frame decompressor 108-2 includes circuitry to decompress a compressed video frame. Thus, the frame decompressor 108-2 can decompress the compressed frame 112 to produce the decompressed frame 114. In operation, a core 110 uses the frame decompressor 108-2 to obtain a decompressed version of a compressed frame. For example, the nth core 110-n sends a request 116 to the frame decompressor 108-2 to indicate that a decompressed version of an identified compressed frame 112 is being requested. The frame decompressor 108-2 provides a response 118 that includes the decompressed frame 114. Although video frames are referenced as an example context, the frames described herein can include any frames with visual data, including computer-generated graphic frames, video frames, combination frames, and so forth.

In this manner, the frame decompressor 108-2 provides a decompression service to individual cores of the multiple cores 110-1 to 110-n. Similarly, the frame compressor 108-1 (e.g., of FIG. 3-1) can provide a compression service to individual cores of the multiple cores 110-1 to 110-n. Thus, the frame decompressor 108-2 or the frame compressor 108-1 (or both) realizes at least part of frame compression-decompression (FCD) server circuitry 122 for the IC 106. Similarly, the multiple cores 110-1 to 110-n realize multiple frame compression-decompression (FCD) client circuits 120. Using this client-server architecture, a frame compressor unit or a frame decompressor unit can be shared between two or more cores 110 to save space on the IC 106 and simplify a workflow for upgrading the compression/decompression technology used to compress/decompress video frames for the IC 106. A more detailed example architecture for an IC 106 and a PCB 104 is described below with reference to FIGS. 3, 3-1, and 3-2. However, additional aspects of example implementations are described next with reference to FIG. 2.

FIG. 2 illustrates other aspects of an example environment 200 in which video frame codec architectures as described herein can be implemented. The electronic device 102 is illustrated with various non-limiting example devices: a smartphone 102-1, a notebook computer 102-2, a television 102-3, a desktop computer 102-4, a tablet 102-5, and a wearable device 102-6. As shown on the right, the electronic device 102 includes one or more processors 202, one or more computer-readable media 204, and at least one interconnect 216. The computer-readable media 204 can store, hold, or otherwise include code, data, instructions, other information, and so forth. The electronic device 102 can also include an operating system 212. Although depicted separately, the operating system 212 can be stored on one or more computer-readable media 204.

Applications (not shown) or the operating system 212 that is embodied as computer-readable instructions on the computer-readable media 204 can be executed by the processor 202. The operating system 212 or a basic input/output system (BIOS) can include a frame codec parameter module 214. The frame codec parameter module 214 can set one or more parameters to enable, authorize, tune, or otherwise facilitate performance of the shared frame compression and decompression functionalities described herein.

As illustrated, the computer-readable media 204 can include at least one video buffer 206, at least one shared local cache 208, and at least one main memory 210. In some implementations, the video buffer 206 and the shared local cache 208 are separate memory blocks on an IC. In other implementations, the video buffer 206 and the shared local cache 208 are part of a same memory block, such as if part of the shared local cache 208 is used as the video buffer 206 in a dynamically changing or a fixed allocation scheme. The interconnect 216 can include at least one system bus 218, at least one video bus 220, and at least one external bus 222. In some implementations, the video bus 220 and the system bus 218 are different buses. In other implementations, there is no separate video bus, and video data is therefore propagated around an IC using the system bus 218. Example implementations of these computer-readable media 204 and these interconnects 216 are described below with reference to FIGS. 4-6.

* Example Components and Techniques*

FIG. 3 illustrates a portion of a PCB 104. The PCB 104 includes a system-on-chip 302 (SoC) and a main memory 210. The SoC 302 depicts an example implementation of a video frame codec architecture that includes a frame compressor-decompressor 108 (FCD), a frame compressor-decompressor controller 304, and multiple cores 110-1 to 110-n. The SoC 302 also includes a shared local cache 208 and a system bus 218. The main memory 210 is coupled to the SoC 302 via an external bus 222 that is included as part of (e.g., disposed on) the PCB 104.

The PCB 104 can be implemented with a rigid or a flexible material for mounting or securing multiple IC chips, interconnects, interfaces, and so forth. The main memory 210 can be realized using, for example, dynamic random-access memory (DRAM) that is refreshed periodically to maintain memory contents or flash memory that can maintain memory contents without power. Generally, more energy is consumed accessing data stored on the main memory 210 than is consumed accessing data stored on the SoC 302, such as at the shared local cache 208. The shared local cache 208 can be realized using, for example, static random-access memory (SRAM), DRAM, flash memory, or some combination thereof.

The system bus 218 interconnects the multiple cores 110-1 to 110-n, the shared local cache 208, the frame compressor-decompressor controller 304, and other components and interfaces (e.g., the frame compressor-decompressor 108 may be directly coupled to the system bus 218). Each core 110 of the multiple cores 110-1 to 110-n can store data at or retrieve data from the shared local cache 208 using the system bus 218. Similarly, each core 110 of the multiple cores 110-1 to 110-n can store data at or retrieve data from the main memory 210 using the external bus 222, such as by also using the system bus 218 or the shared local cache 208. For example, a first core 110-1 can store data in the shared local cache 208, and a second core 110-2 can then retrieve the stored data from the shared local cache 208.

In an example operation, the frame compressor-decompressor 108 (FCD) processes (e.g., compresses or decompresses) multiple unprocessed frames 306 to produce multiple processed frames 308. The frame compressor-decompressor controller 304 (FCD controller) is coupled to the frame compressor-decompressor 108 and arbitrates access to the frame compressor-decompressor 108 for multiple cores 110-1 to 110-n. Although shown separately, the frame compressor-decompressor 108 and the frame compressor-decompressor controller 304 may be logically integrated together. At least two of the cores 110 of the multiple cores 110-1 to 110-n are coupled to the frame compressor-decompressor controller 304. Thus, each core 110 can obtain, via the frame compressor-decompressor controller 304, a processed frame 308 that is produced by the frame compressor-decompressor 108 from an unprocessed frame version thereof. The processed frame 308 can be obtained using, for instance, a request 116 and a corresponding response 118.

The frame compressor-decompressor 108 can include a frame compressor 308-1 (e.g., as depicted in FIG. 3-1), a frame decompressor 108-2 (e.g., as depicted in FIG. 3-2), or both. Similarly, the frame compressor-decompressor controller 304 can include a frame compressor controller 304-1 (e.g., as depicted in FIG. 3-1), a frame decompressor controller 304-2 (e.g., as depicted in FIG. 3-2), or both. One of the unprocessed frame 306 or the processed frame 308 corresponds to a compressed frame, and the other corresponds to a decompressed frame, depending on whether the processing operation by the frame compressor-decompressor 108 is a compression or a decompression operation. If a given implementation includes both a frame compressor 108-1 and a frame decompressor 108-2 (or both a frame compressor controller 304-1 and a frame decompressor controller 304-2), such components may be co-located, located proximate to one another, or located at different places on an IC. For instance, each may be located nearer a core that is likely to be the most common client of the corresponding compression or decompression service. Further, if a chip includes a frame compressor 108-1 and a frame decompressor 108-2 that are co-located, each may include fully separate circuitry or they may share circuitry. In some implementations, the frame compressor-decompressor 108 can be realized as a lossless frame data manipulator that adjusts a memory size of the data for the frame as part of a lossless compression operation or a lossless decompression operation, depending on the request 116. Example implementations for compression operations and decompression operations are described below with reference to FIGS. 3-1 and 3-2, respectively.

FIG. 3-1 illustrates an SoC having an example implementation of a video frame codec architecture that includes a frame compressor 108-1 (FC), a frame compressor controller 304-1 (FC controller), and multiple cores. The frame compressor 108-1 compresses uncompressed frames (e.g., decompressed frames 114 or not-yet-compressed frames) to produce compressed frames 112. The frame compressor controller 304-1 is coupled to the frame compressor 108-1 and arbitrates access to the frame compressor 108-1 for the multiple cores 110-1 to 110-n. Although shown separately, the frame compressor 108-1 and the frame compressor controller 304-1 may be logically integrated together.

At least two of the cores 110 of the multiple cores 110-1 to 110-n are coupled to the frame compressor controller 304-1. Here, each core 110 can include a component or block that produces visual data. Thus, each core 110 can obtain, via the frame compressor controller 304-1, a compressed frame 112 that is produced by the frame compressor 108-1 from a decompressed frame 114 version thereof. Thus, the compressed frame 112 can be obtained using, for instance, a request 116 and a corresponding response 118. In operation, the frame compressor controller 304-1 can grant access to the compression engine of the frame compressor 108-1 to some requesting core 110 on the system bus 218 in a pipelined manner to avoid adding traffic to the external bus 222, which provides access to the main memory 210. In some implementations, the frame compressor controller 304-1 can temporarily grant exclusive access to the requesting core 110 to access the compression resource.