Facebook Patent | Parallel Texture Sampling

Patent: Parallel Texture Sampling

Publication Number: 20200143580

Publication Date: 20200507

Applicants: Facebook

Abstract

In one embodiment, a computing system may receive a number of texels organized into a texel array including a number of sub-arrays. The system may determine a number of texel subsets with the texels in each subset having a same position within their respective sub-arrays. The system may store the texel subsets into a number of buffer blocks, respectively, with each buffer block storing one texel subset. The system may retrieve a sampling texel array from the buffer blocks for parallelly determining pixel values of a number of sampling points. Each texel of the sampling texel array may be retrieved from a different buffer block.

PRIORITY

[0001] This application claims the benefit, under 35 U.S.C. .sctn. 119(e), of U.S. Provisional Patent Application No. 62/755,281, filed 02 Nov. 2018, which is incorporated herein by reference.

TECHNICAL FIELD

[0002] This disclosure generally relates to artificial reality, such as virtual reality and augmented reality.

BACKGROUND

[0003] Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

SUMMARY OF PARTICULAR EMBODIMENTS

[0004] Particular embodiments described herein relate to a method of rendering artificial reality objects using pre-warped surfaces as the rendering primitives, and parallelly retrieving all the texels that are needed to determine the properties (e.g., colors or distance fields) of a set of pixel samples (e.g., a 2.times.2 set of pixels) within a tile (e.g., a collection of pixels, such as 16.times.16 pixels, defined by its four corners) in one operation. The system may generate pre-warped surfaces on one or more CPUs/GPUs associated with a computing system (e.g., a body wearable computing system, such as a mobile phone, or a laptop, desktop, etc.). The pre-warped surfaces may be generated based on the rendering results of the CPU(s)/GPU(s), such as a 2D image that visually take into account the 3D contour of the underlying object. The pre-warped surfaces may be generated with particular shape, size, orientation based on a viewpoint (e.g., view distance, view angle) of a viewer. The 2D image may be stored as the texture data of the surface. The surface may be considered as a flat planar canvas for the 2D texture and is positioned in 3D view space facing the viewer (e.g., with a normal vector substantially pointing towards the viewer). As a result, the pre-warped texture data of the surface may be represented by a single mipmap level instead of multiple texture mipmap levels.

[0005] A headset system may receive the pre-warped surfaces from the body wearable computing system and render the surfaces on a head mounted display by transforming the pre-warped surfaces. Visibility of pre-warped surfaces may be tested by projecting rays from a viewer’s screen. In particular embodiments, the rays may be cast from a number of tiles (e.g., an aligned block of 16.times.16 pixels), each of which may be defined by its four corner positions in screen space. Once it is determined that a particular surface is visible from a tile, colors for pixels within the tile may be sampled from the texture of the surface. In particular embodiments, pixel sampling may be performed concurrently for four aligned 2.times.2 pixels. The system may restrict minification by zooming out operations to be within a two-time range. With this constraint, the 2.times.2 sampling points may always fall within a 4.times.4 texels region. In particular embodiments, the system may use a multi-level memory architecture including 16 independent texel buffer blocks for texel buffer. The system may use a pre-determined texel storage pattern to store 4.times.4 texels regions in the 16 independent quad buffer blocks that can be addressed separately and can be readout parallelly (e.g., in one operation). Therefore, the system may access the 4.times.4 texels region in one read operation and parallelly sample all the texels that are needed to determine the four pixels values (rather than sequentially access four quads). Since the pre-warped surfaces are generated based on one texture mipmap level, the headset system may only need to access single texture mipmap and use bilinear interpolation instead of trilinear interpolation, which would require reading another set of texel data from a different mipmap level, to determine the pixels values.

[0006] In an embodiment, a method may comprise, by a computing system: [0007] receiving a plurality of texels organized into a texel array comprising a plurality of sub-arrays; [0008] determining a plurality of texel subsets, wherein the texels in each subset have a same position within their respective sub-arrays; [0009] storing the plurality of texel subsets into a plurality of buffer blocks, respectively, wherein each buffer block stores one texel subset of the plurality of texel subsets; and [0010] retrieving a sampling texel array from the plurality of buffer blocks for parallelly determining pixel values of a plurality of sampling points, wherein each texel of the sampling texel array is retrieved from a different buffer block of the plurality of buffer blocks.

[0011] The plurality of sub-arrays may form a repeated pattern within the texel array.

[0012] Each of the plurality of buffer blocks may be addressed separately and accessed parallelly.

[0013] The plurality of buffer blocks may be grouped into a plurality of groups, and each texel used for determining a pixel value of a sampling point may be retrieved from a different group of the plurality of groups.

[0014] In one embodiment, the plurality of texel subsets may be determined by: determining a reference texel having a first coordinate (u, v) indicating a position of the reference texel within the texel array; [0015] determining a plurality of intermediate texel groups comprising texels having least significant bits of binary representations of first coordinates (u, v) equal to (0, 0), (1, 0), (0, 1), and (1, 1), respectively; [0016] determining a second coordinate (x, y) for each texel of each intermediate texel group; and [0017] determining the plurality of texel subsets each comprising texels having least significant bits of binary representations of second coordinates (x, y) equal to (0, 0), (1, 0), (0, 1), and (1, 1) with respect to a corresponding intermediate texel group.

[0018] The plurality of texel subsets may comprise 16 texel subsets, and the plurality of buffer blocks may comprise 16 buffer blocks.

[0019] The 16 texel subsets may be organized into four texel groups. The plurality of sampling points may comprise four sampling points, and the four texel groups may be used for parallelly determining the pixels values of the four sampling points, respectively.

[0020] Each pixel value may be determined based on a 2.times.2 texel sub-array of the sampling texel array, and each texel of the 2.times.2 texel sub-array may be selected from one of the four texel groups.

[0021] The plurality of sampling points may be associated with an intersection area of a display region to a two-dimensional representation of a portion of a scene.

[0022] The two-dimensional representation may comprise three-dimensional information of the portion of the scene, and the two-dimensional representation of the portion of the scene may be visible in the display region.

[0023] The two-dimensional representation of the portion of the scene may be represented with a single texture resolution, and the texel array may comprise texels having the single texture resolution.

[0024] The sampling texel array may be associated with an aligned texel region or an unaligned texel region.

[0025] In one embodiment, one or more computer-readable non-transitory storage media may embody software that is operable when executed to: [0026] receive a plurality of texels organized into a texel array comprising a plurality of sub-arrays; [0027] determine a plurality of texel subsets, wherein the texels in each subset have a same position within their respective sub-arrays; [0028] store the plurality of texel subsets into a plurality of buffer blocks, respectively, wherein each buffer block stores one texel subset of the plurality of texel subsets; and [0029] retrieve a sampling texel array from the plurality of buffer blocks for parallelly determining pixel values of a plurality of sampling points, wherein each texel of the sampling texel array is retrieved from a different buffer block of the plurality of buffer blocks.

[0030] The plurality of sub-arrays may form a repeated pattern within the texel array.

[0031] Each of the plurality of buffer blocks is addressed separately and accessed parallelly.

[0032] The plurality of buffer blocks may be grouped into a plurality of groups, and each texel used for determining a pixel value of a sampling point may be retrieved from a different group of the plurality of groups.

[0033] In one embodiment, a system may comprise: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to: [0034] receive a plurality of texels organized into a texel array comprising a plurality of sub-arrays; [0035] determine a plurality of texel subsets, wherein the texels in each subset have a same position within their respective sub-arrays; [0036] store the plurality of texel subsets into a plurality of buffer blocks, respectively, wherein each buffer block stores one texel subset of the plurality of texel subsets; and [0037] retrieve a sampling texel array from the plurality of buffer blocks for parallelly determining pixel values of a plurality of sampling points, wherein each texel of the sampling texel array is retrieved from a different buffer block of the plurality of buffer blocks.

[0038] The plurality of sub-arrays may form a repeated pattern within the texel array.

[0039] Each of the plurality of buffer blocks may be addressed separately and accessed parallelly.

[0040] The plurality of buffer blocks may be grouped into a plurality of groups, and each texel used for determining a pixel value of a sampling point may be retrieved from a different group of the plurality of groups.

[0041] In an embodiment, one or more computer-readable non-transitory storage media may embody software that is operable when executed to perform a method according to or within any of the above mentioned embodiments.

[0042] In an embodiment, a system may comprise: one or more processors; and at least one memory coupled to the processors and comprising instructions executable by the processors, the processors operable when executing the instructions to perform a method according to or within any of the above mentioned embodiments.

[0043] In an embodiment, a computer program product, preferably comprising a computer-readable non-transitory storage media,* may be operable when executed on a data processing system to perform a method according to or within any of the above mentioned embodiments*

[0044] The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed above. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG. 1A illustrates an example artificial reality system.

[0046] FIG. 1B illustrates an example eye display system of the headset system.

[0047] FIG. 2 illustrates an example 3D object.

[0048] FIG. 3 illustrates an example pre-warped surface generated on a body wearable computing system.

[0049] FIG. 4 illustrates an example pre-warped surface which appears transformed by the headset system for rendering on eye display systems.

[0050] FIG. 5 illustrates an example pre-warped surface that is visible through an example tile.

[0051] FIGS. 6A-B illustrate an example process for determining the texels that are needed for determining the color or distance field of a sampling point.

[0052] FIG. 7A illustrates an example 4.times.4 texel region and an example sampling region that represents a set of 2.times.2 orthogonal sampling points located at the four corners of the sampling region.

[0053] FIG. 7B illustrates example 2.times.2 orthogonal sampling points within an aligned 4.times.4 texel region.

[0054] FIG. 7C illustrates example 2.times.2 orthogonal sampling points within an unaligned 4.times.4 texel region.

[0055] FIG. 7D illustrates example 2.times.2 non-orthogonal sampling points within a 4.times.4 texel region.

[0056] FIG. 8A illustrates an example 4.times.4 texel array stored in a 32 bytes texel memory of control block with an interleaved pattern.

[0057] FIGS. 8B-C illustrate an example 8.times.8 texel array stored in 16 independent texel buffer blocks to allow any 4.times.4 texel array to be read in one read operation.

[0058] FIGS. 8D-E illustrate an example 4.times.4 texel array selected from an 8.times.8 texel array stored in 16 independent texel buffer blocks.

[0059] FIGS. 8F-G illustrate an example 2.times.2 texel array which is selected from an 8.times.8 texel array stored in 16 independent texel buffer blocks and can be read from the texel buffer with reduced multiplexing operations.

[0060] FIG. 9 illustrates a system diagram for a display engine.

[0061] FIG. 10 illustrates an example diagram for pixel block.

[0062] FIG. 11A illustrates an example diagram for filter block including four quad buffer blocks.

[0063] FIG. 11B illustrates an example diagram for quad buffer block including four texel buffer blocks.

[0064] FIG. 12A illustrates an example diagram for texel buffer block.

[0065] FIG. 12B illustrates example data formats for texel data stored in texel buffer blocks.

[0066] FIG. 12C illustrates an example diagram for sample filter block.

[0067] FIG. 13 illustrates an example method of parallelly sampling multiple groups of texels to determine multiple pixel values using bilinear interpolation.

[0068] FIG. 14 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

[0069] Traditional graphics rendering systems may need to perform separate read operations to obtain the necessary texel data from a texture to determine the color (for images) or distance field (for labels, such as fonts, characters, glyphs, etc.) for a single pixel. During a sampling process, traditional rendering pipelines implemented on traditional GPUs access texel buffer memory in quads when performing bilinear interpolation to determine the pixel value (e.g., color/distance field). For example, traditional GPUs may need to perform four separate read operations to retrieve the four closest texels, relative to the sample location, that are needed to perform filtering (e.g., via bilinear interpolation). Such memory-access operations are slow and consume more power. In addition, if multiple pixel samples are being filtered concurrently, different sampling locations may require texels from different texture mipmap levels, further adding to memory-access time. For example, if a virtual box is drastically slanted relative to the viewer, the portion of the box that is closer to the viewer may use a high-resolution mipmap texture than the portion of the box that is farer away from the viewer. Similarly, if the viewer zooms out from the box, more texels may need to be retrieved or the system may need to switch to a coarser mipmap level. Such operations, especially when performed in large volume, significantly adds to the overall rendering time, power consumption, and complexity of the system.