Nvidia Patent | System and method for near-eye light field rendering for wide field of view interactive three-dimensional computer graphics

Patent: System and method for near-eye light field rendering for wide field of view interactive three-dimensional computer graphics

Drawings: Click to check drawins

Publication Number: 20210181674

Publication Date: 20210617

Applicant: Nvidia

Abstract

A method for rendering a light field comprises projecting rays from a viewpoint positioned at a first side of a spatial light modulator (SLM) to a clipping plane positioned at an opposing side of the SLM to form an elemental view frustum within a three-dimensional scene and rendering objects within the elemental view frustum to generate components of a first elemental image for the first elemental region. The SLM may include a tiled array of non-overlapping elemental regions and a top edge and a bottom edge of a first elemental region of the non-overlapping elemental regions are intersected by the rays to form the elemental view frustum. Furthermore, the light field may include the first elemental image and additional elemental images corresponding to the array of elemental regions and each one of the additional elemental images is rendered using an additional elemental view frustum.

Claims

1. A method for rendering a light field that comprises a plurality of elemental images arranged in a grid, the method comprising: generating, by a processor, for each elemental image of the plurality of elemental images, components of the light field corresponding to that elemental image by: projecting rays from a viewpoint positioned at a lateral offset from a first side of a spatial light modulator (SLM) to form an elemental view frustum within a three-dimensional scene; and rendering objects within the elemental view frustum to generate components of the elemental image, wherein each elemental image of the plurality of elemental images corresponds to a holographic element (hogel) associated with one or more picture elements (pixels) of the SLM, and wherein the SLM is divided into a plurality of hogels having size w.sub.h and the lateral offset is calculated based on the size w.sub.h.

2. The method of claim 1, wherein the rendering comprises, for each pixel of the hogel corresponding to the elemental image: projecting second rays from the pixel to a clipping plane to define a pixel diffraction cone having a base of a first width; and removing a portion of the components of the elemental image outside of the pixel diffraction cone.

3. The method of claim 1, wherein the components include a color and a position in three-dimensional (3D) space.

4. The method of claim 1, wherein the components include a phase and an amplitude.

5. The method of claim 4, further comprising computing the phase and the amplitude as a product of an object wave and a conjugate reference wave corresponding to a plane wave illumination source.

6. The method of claim 4, further comprising computing the phase and the amplitude as a product of an object wave and a conjugate reference wave corresponding to a spherical wave illumination source.

7. A method for rendering a light field that comprises a plurality of elemental images arranged in a grid, the method comprising: computing, by a processor, a lateral offset based on a size w.sub.h of a holographic element (hogel) associated with a spatial light modulator (SLM), a pixel pitch of the SLM, and a wavelength associated with a monochromatic light source, wherein the SLM is divided into a plurality of hogels, each hogel associated with one or more picture elements (pixels) of the SLM; and for each elemental image, rendering, by the processor, a three-dimensional scene from a viewpoint located at the lateral offset from a first side of the SLM to produce components of the elemental image, wherein each elemental image is associated with a different viewpoint.

8. The method of claim 7, wherein the rendering comprises computing a color array and a depth array for at least one elemental image of the plurality of elemental images.

9. The method of claim 8, further comprising: calculating a phase value for a pixel of the SLM based on at least one depth value of the depth array.

10. The method of claim 8, further comprising: calculating an amplitude value for a pixel of the SLM based on at least a corresponding color value of the color array.

11. The method of claim 7, further comprising computing a phase and an amplitude for a pixel of the SLM as a product of an object wave and a conjugate reference wave corresponding to a spherical illumination source.

12. The method of claim 7, wherein the rendering comprises, for each pixel of the hogel corresponding to the elemental image: projecting rays from the pixel to a clipping plane to define a pixel diffraction cone having a base of a first width.

13. The method of claim 12, wherein the rendering further comprises removing a portion of the components of the elemental image outside of the pixel diffraction cone.

14. A system for rendering a light field that comprises a plurality of elemental images arranged in a grid, the system comprising: a spatial light modulator (SLM) divided into a plurality of holographic elements (hogels) having size w.sub.h, each hogel associated with one or more picture elements (pixels) of the SLM, and each elemental image of the plurality of elemental images corresponding to a particular hogel of the plurality of hogels; and a processor coupled to the SLM and configured to: generate, for each elemental image of the plurality of elemental images, components of the light field corresponding to that elemental image by: projecting rays from a viewpoint positioned at a lateral offset from a first side of a spatial light modulator (SLM) to form an elemental view frustum within a three-dimensional scene, wherein the lateral offset is calculated based on the size w.sub.h; and rendering objects within the elemental view frustum to generate components of the elemental image.

15. The system of claim 14, wherein the rendering comprises, for each pixel of the hogel corresponding to the elemental image: projecting second rays from the pixel to a clipping plane to define a pixel diffraction cone having a base of a first width; and removing a portion of the components of the elemental image outside of the pixel diffraction cone.

16. The system of claim 14, wherein the components include a color and a position in three-dimensional (3D) space.

17. The system of claim 14, wherein the components include a phase and an amplitude.

18. The system of claim 17, further comprising computing the phase and the amplitude as a product of an object wave and a conjugate reference wave corresponding to a plane wave illumination source.

19. The system of claim 17, further comprising computing the phase and the amplitude as a product of an object wave and a conjugate reference wave corresponding to a spherical wave illumination source.

20. The system of claim 14, wherein the processor comprises a graphics processing unit.

Description

CLAIM OF PRIORITY

[0001] This application is a continuation of U.S. Non-Provisional application Ser. No. 15/946,576, filed on Apr. 5, 2018, which claims the benefit of U.S. Provisional Application No. 62/525,644 titled “NEAR-EYE LiGHT FIELD HOLOGRAPHIC RENDERING,” filed Jun. 27, 2017, the entire contents of which are incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present invention relates to computer generated holography, and more particularly to a system and method for near-eye light field rendering for wide field of view interactive three-dimensional computer graphics.

BACKGROUND

[0003] Creating a comfortable visual experience is important to the success of modern virtual reality (VR) and augmented reality (AR) systems. A wide field of view, high resolution, interactivity, view-dependent occlusion, and continuous focus cues are significant features for providing a comfortable visual experience. However, conventional VR systems typically fail to provide many of these features, resulting in user discomfort. Thus, there is a need for addressing these issues and/or other issues associated with the prior art.

SUMMARY

[0004] A method, computer readable medium, and system are configured to render a light field. The method comprises projecting rays from a viewpoint positioned at a first side of a spatial light modulator (SLM) to a clipping plane positioned at an opposing side of the SLM to form an elemental view frustum within a three-dimensional scene. Objects within the elemental view frustum are rendered to generate components of a first elemental image for the first elemental region. In one embodiment, the SLM is tiled with an array of elemental regions and a top edge and a bottom edge of a first elemental region of the non-overlapping elemental regions are intersected by the rays to form the elemental view frustum. In certain embodiments, the light field includes the first elemental image and additional elemental images corresponding to the array of elemental regions and each one of the additional elemental images is rendered using an additional elemental view frustum.

[0005] The computer readable medium includes instructions that, when executed by a processing unit, perform the method. Furthermore, the system includes circuitry configured to perform the method.

[0006] A second method, second computer readable medium, and second system are configured to render a light field. The second method comprises computing a lateral offset between a view position and a spatial light modulator (SLM) based on a size of the SLM and a width of a holographic element. A three-dimensional scene is rendered from the view position to produce an array of elemental images. In one embodiment, an array of holographic elements covers a surface of the SLM.

[0007] The second computer readable medium includes instructions that, when executed by a processing unit, perform the second method. Furthermore, the second system includes circuitry configured to perform the second method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1A illustrates a flowchart of a method for rendering a light field, in accordance with one embodiment.

[0009] FIG. 1B illustrates a flowchart of a method for rendering objects within an elemental view frustum, in accordance with one embodiment.

[0010] FIG. 1C illustrates computer generated holography, in accordance with one embodiment.

[0011] FIG. 1D illustrates a holographic element, in accordance with one embodiment.

[0012] FIG. 2A illustrates conventional hogel rendering, in accordance with the prior art.

[0013] FIG. 2B illustrates hogel rendering with plane wave illumination, in accordance with one embodiment.

[0014] FIG. 2C illustrates a region of an ambiguity segment, in accordance with one embodiment.

[0015] FIG. 2D illustrates hogel rendering with spherical wave illumination, in accordance with one embodiment.

[0016] FIG. 2E illustrates a region of an ambiguity segment, in accordance with one embodiment.

[0017] FIG. 2F illustrates algorithmic operations of a method for rendering a light field using spherical illumination, in accordance with one embodiment.

[0018] FIG. 2G illustrates a comparison of elemental image resolution results, in accordance with one embodiment.

[0019] FIG. 2H illustrates a flowchart of a method for rendering a light field, in accordance with one embodiment.

[0020] FIG. 3 illustrates a parallel processing unit, in accordance with one embodiment.

[0021] FIG. 4A illustrates a general processing cluster within the parallel processing unit of FIG. 3, in accordance with one embodiment.

[0022] FIG. 4B illustrates a memory partition unit of the parallel processing unit of FIG. 3, in accordance with one embodiment.

[0023] FIG. 5A illustrates the streaming multi-processor of FIG. 4A, in accordance with one embodiment.

[0024] FIG. 5B is a conceptual diagram of a processing system implemented using the PPU of FIG. 3, in accordance with one embodiment.

[0025] FIG. 5C illustrates an exemplary system in which the various architecture and/or functionality of the various previous embodiments may be implemented.

[0026] FIG. 6 is a conceptual diagram of a graphics processing pipeline implemented by the PPU of FIG. 3, in accordance with one embodiment.

DETAILED DESCRIPTION

[0027] Embodiments of the present invention improve field of view, interactivity at high resolution, and view dependent occlusion in computer generated holography (CGH). Furthermore, various embodiments advantageously provide continuous focus cues, thereby substantially avoiding vergence-accommodation-conflict in near eye displays. In one embodiment, a near eye display comprises liquid crystal (LC) and/or spatial light modulator (SLM) structures configured to display a CGH light field to a user. The CGH light field may be computed according to plane wave illumination, spherical wave illumination, or any other technically feasible wave propagation illumination model.

[0028] A CGH light field provides an object wave for a given observable point in a three-dimensional (3D) scene, based on a reference wave. The form of the reference wave (e.g., plane wave) may be specified, and CGH processing computes a diffraction pattern which will perform a conversion from the reference wave to an object wave at a given location within a hologram. In one embodiment, computing the object wave includes projecting rays from a viewpoint (e.g., rendering camera position) positioned in front of an SLM towards a clipping plane positioned in back of the SLM. In general, the viewpoint and the clipping plane may be positioned on opposing sides of the SLM. A given ray may be computed to have an amplitude and phase relative to other rays. Regions of the SLM may be organized into elemental images, each with an elemental view frustum within the 3D scene so that each elemental image may comprise a single, different representative view of the 3D scene. Furthermore, multiple elemental images may be rendered to form a complete 3D scene presented to a user.

[0029] FIG. 1A illustrates a flowchart of a method 110 for rendering a light field, in accordance with one embodiment. Although method 110 is described in the context of a processing unit, the method 110 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 110 may be executed by a GPU (graphics processing unit), a CPU (central processing unit), or any other technically feasible processor. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 110 is within the scope and spirit of embodiments of the present invention.

[0030] At step 112, the processing unit projects rays from a viewpoint positioned in front of an SLM to a clipping plane positioned in back of the SLM to form an elemental view frustum within a 3D scene. More generally, the viewpoint may be positioned at a first side of the SLM, and the clipping plane may be positioned at an opposing side of the SLM. In one embodiment, viewpoint is positioned on the observer’s side of the SLM and the near clipping plane is positioned on the opposing side (opposite side relative to the observer) of the SLM. In one embodiment, the near clipping plane is located coincident with the surface of the SLM. In one embodiment, the SLM is tiled with an array of non-overlapping elemental regions and a top edge and a bottom edge of a first elemental region of the non-overlapping elemental regions are intersected by the rays to form the elemental view frustum.

[0031] At step 114, the processing unit renders objects within the elemental view frustum to generate components of a first elemental image for the first elemental region. In one embodiment, the light field includes the first elemental image and additional elemental images corresponding to the array of elemental regions and each one of the additional elemental images is rendered using an additional elemental view frustum.

[0032] At step 116, the processing unit computes phase and amplitude components for driving the SLM as a product of an object wave and a conjugate reference wave. Furthermore, the components may include color and position within the 3D scene. In one embodiment, the conjugate reference wave comprises a plane wave illumination source. In another embodiment, the conjugate reference wave comprises a spherical wave illumination source. In other embodiments, the conjugate reference wave comprises an arbitrary illumination source.

[0033] In one embodiment, for each pixel of the SLM within the first elemental region, rendering comprises projecting second rays from the pixel of the SLM to the clipping plane to define a pixel diffraction cone having a base of a first width and removing a portion of the components of the first elemental image that are outside of the pixel diffraction cone to perform ambiguity segment culling.

[0034] FIG. 1B illustrates a flowchart of a method 120 for rendering objects within the elemental view frustum, in accordance with one embodiment. Although method 120 is described in the context of a processing unit, the method 120 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 120 may be executed by a GPU, CPU, or any other technically feasible processor. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 120 is within the scope and spirit of embodiments of the present invention. As shown in FIG. 1B, in one embodiment, step 114 of method 110 comprises steps 122 and 124.

[0035] At step 122, the processing unit projects second rays from the pixel of the SLM to the clipping plane to define a pixel diffraction cone having a base of a first width. At step 124, the processing unit removes a portion of the components of the first elemental image that are outside of the pixel diffraction cone. In one embodiment, ambiguity segment culling is performed by removing the portion of components outside of the pixel diffraction cone.

[0036] Methods 110 and 120 may be performed in the context of computer generated holography (CGH) for generating light field data used to drive an SLM device. A description of CGH will now be set forth, along with implementation details relevant to various embodiments.

[0037] FIG. 1C illustrates computer generated holography (CGH), in accordance with one embodiment. As shown, a rendering viewpoint is indicated by a virtual camera 142, which is positioned to view a scene object 140 through an SLM 144. A point j is shown on the scene object 140, and a distance r separates point j from a pixel location x on the SLM 144.

[0038] In general, a hologram converts an input reference light wave E.sub.R (x) to an appropriate output object light wave E.sub.O(x). In CGH, generating the output object light wave requires knowledge of both the reference light wave and the object light wave. The form of the reference light wave may be given and various CGH techniques may be applied to compute a diffraction pattern that will yield the object light wave at each location on SLM 144. A diffraction pattern may be computed for each location based on a desired output waveform for the location on the SLM 144. To compute a given output waveform resulting from scene object 140, light is propagated backwards towards the SLM 144 using a Fresnel diffraction integral. For a scene object 140 comprising discrete points j, a summation of spherical waves originating from the points j may operate in place of a diffraction integral. Such a summation is calculated by Equation 1.

E O ( x ) = j A j r j ( x ) e i ( 2 .pi. .lamda. r j ( x ) + .phi. j ) ( 1 ) ##EQU00001##

[0039] In Equation 1, .lamda. is the wavelength of a monochromatic light source, A.sub.j is the amplitude of the point j on the scene object 140, and r.sub.j(x) is the Euclidean distance from the point j to a pixel location x on the SLM 144 to a given point j on the scene object 140. Furthermore, .PHI..sub.j is a random initial phase associated with each point j.

[0040] The resulting electric field E.sub.O(x) is complex-valued. In CGH, a corresponding illumination wavefront is generated by multiplying the resulting electric field with an appropriate illumination field. For example, in plane wave (collimated beam) illumination the resulting electric field is multiplied by a constant (e.g., 1). For spherical wave illumination, the electric field may be multiplied by a complex exponential with a quadratic phase to cancel out the quadratic phase of a spherical reference wave. Displaying a correct diffraction pattern on the SLM 144 is provided by spatially varying both amplitude and phase delays, according to a resulting product.

[0041] In one embodiment, a CGH rendering and display pipeline starts with a polygon-based holographic light field rendering and includes a point-based approach (i.e., summation of propagating fields from points on scene object 140) with local partitioning for view-dependent effect. Occlusion is handled through the pipeline using a z-buffer. Sampled fragments allow for parallel full-parallax CGH computation on a GPU at interactive speed with high-resolution (e.g., 1080p) image quality. In certain embodiments, a CGH rendering pipeline includes a polygon surface approach (i.e., summation of propagating fields from visible polygon surfaces comprising scene object 140) that may operate independently or in conjunction with the point-based approach. Any technically feasible technique may be performed to compute fields from the polygon surfaces at different pixels of SLM 144. Furthermore, while various techniques taught herein are described with reference to points on a scene object, persons of ordinary skill in the art will understand that the techniques may be applied to polygons and/or arbitrary shapes or surfaces without departing the scope and spirit of various embodiments.

[0042] Rendering a full light field generates highly-overlapped views for adjacent hologram pixels and conventionally results in significant computational redundancy. For example, in a point-based approach, conventional rendering requires sequential scanning of the scene to accumulate wavefronts emitted from depth-sorted scene points. Such an operation is equivalent to adding densely sampled angular views in conventional light field rendering, an approach known in the art to be computationally impractical for real-time graphics.

[0043] However, assuming Lambertian surfaces for scene object 140, a single recording of each point is sufficient to determine the wavefront. Leveraging this observation, a hologram can be spatially partitioned into abutting grids, with an individual grid referred to herein as a holographic element (hogel), illustrated in FIG. 1D.

[0044] FIG. 1D illustrates a holographic element, in accordance with one embodiment. As shown, a color intensity map includes an abutting grid of elemental images. Each elemental image comprises a single representative view of a 3D scene. A location and depth map includes a corresponding grid of depth information for the elemental images. A given elemental image is rendered and used to calculate each hogel, assuming all captured points are visible to all pixels in the hogel. In one embodiment, each hogel has an associated phase map and an associated amplitude map. The phase map and the amplitude map may be computed based on the color intensity map and the location and depth map.

[0045] Monocular occlusion parallax is bounded by hogel size (w.sub.h) within an eye box. In one embodiment, an eye box is a region at a user’s eye position that is sufficiently large as to allow a user’s eye to move freely while allowing the user (viewer) to see the entire 3D scene depicted by SLM 144 (e.g., all points on scene object 140). Approximating a complete holographic light field display as a grid of hogels substantially reduces rendering passes and computational effort, allowing conventional GPU systems to support real-time rendering applications. However, conventional hogel rendering projects to a given hogel center, thereby failing to render an accurate per-pixel diffraction cone gathering, and conventional hogel rendering may scale poorly in spherical illumination scenarios.

[0046] FIG. 2A illustrates conventional hogel rendering, in accordance with the prior art. As shown, a rendering configuration 200 includes a virtual camera 210 positioned at the center of a hogel 218 included within an SLM 203. The virtual camera 210 is aimed at a scene to be rendered. The position of virtual camera 210 results in a conventional view frustum 212, which only provides accurate rendering for pixels centered within the hogel 218. In prior art hogel rendering systems, the conventional view frustum 212 is used for rendering all pixels in hogel 218 because virtual camera 210 is statically positioned at the center of the hogel 218. Consequently, for a bottom pixel in the hogel 218, region 215 is mistakenly incorporated into the pixel during rendering, while region 217 will be incorrectly excluded from the pixel during rendering. To accurately render the bottom pixel in hogel 218, view frustum 216 should be used. Similarly, to accurately render the top pixel in hogel 218, view frustum 214 should be used.

[0047] As shown, hogels 218, 219 on SLM 203 have a hogel size w.sub.h. Furthermore, a near clipping plane 204 is positioned a distance d.sub.1 from SLM 203, and a far clipping plane 206 is positioned a distance d.sub.2 from SLM 203. Hogel size w.sub.h sets a depth limit of z.ltoreq.d.sub.min to scene objects and the near clipping plane 204 to prevent geometric clipping. In certain scenarios, this depth limit, along with potential geometric clipping, inaccurate per-pixel diffraction cone gathering, and/or additional limitations of conventional hogel rendering reduce the comfort and quality of a user experience.

[0048] FIG. 2B illustrates hogel rendering with plane wave illumination, in accordance with one embodiment. As shown, a rendering configuration 220 includes a virtual camera 230 positioned at a lateral offset d.sub.CZ along the Z (depth) axis with respect to an SLM 223. In contrast, with conventional techniques, as shown in FIG. 2A, where the virtual camera 210 is positioned at a lateral offset of zero, the lateral offset d.sub.CZ is greater than zero. The virtual camera 230 is aimed at a scene to be rendered, including a near clipping plane 224 and a far clipping plane 226. The position of virtual camera 230 results in a view frustum 232 that intersects at least hogel 238 on SLM 223. Hogels 238, 239 on SLM 223 have a hogel size w.sub.h. The near clipping plane 224 is positioned a distance d.sub.1 from SLM 223, and the far clipping plane 226 is positioned a distance d.sub.2 from SLM 203. The lateral offset d.sub.CZ may be equal to a depth limit of z.ltoreq.d.sub.min relative to scene objects and the near clipping plane 224. In one embodiment, the lateral offset is calculated according to Equation 2:

d CZ = d min = w h 2 tan ( sin - 1 ( .lamda. 2 .DELTA. p ) ) ( 2 ) ##EQU00002##

[0049] In Equation 2, .DELTA.p is a pixel pitch size for SLM 223, and .lamda. is the wavelength of a monochromatic light source, such as a light source used for rendering. As shown, the offset position of virtual camera 230 allows the entire visible area for view frustum 232 to be rendered. This visible area is indicated by w.sub.1, and calculated by Equation 3:

w.sub.1=2 sin.sup.-1(.lamda./2.DELTA.p)(d.sub.1+d.sub.CZ) (3)

[0050] In one embodiment, view frustum 232 intersects the edge of hogel 238 and near clipping plane 224, with an extent of w.sub.1. Furthermore, each pixel in SLM 223 may be generated using only a valid diffraction cone, bounded by projection 234. A per-pixel perspective may be obtained by aligning the diffraction cone in the far clipping plane 226 with a sliding window, defined by w.sub.3. The sliding window (w.sub.3) may be calculated according to Equation 4:

w 3 = ( 1 - d CZ d 1 + d C Z ) w 1 ( 4 ) ##EQU00003##

[0051] Diffraction culling may be used on an ambiguity segment, illustrated in FIG. 2C showing region 240 in detail, to provide more accurate rendering. Diffraction culling may include, without limitation, removing certain covered scene object geometry associated with an ambiguity segment from contributing to a given pixel on SLM 223. The ambiguity region may be obtained by extending the sliding window to w.sub.2, as calculated in Equation 5. Furthermore, w.sub.3 and w.sub.2 bound projection 234.

w 2 = ( 1 - d CZ d 2 + d C Z ) w 1 ( 5 ) ##EQU00004##

[0052] Arranging an array of virtual cameras 230 (e.g., one virtual camera per elemental view or elemental region) according to the disclosed configuration allows for unrestricted disposition of scene objects. Lateral offset d.sub.CZ ensures adjacent camera views overlap immediately in front of the SLM 223, and a resulting tiled frustum array fully covers the field of view of the entire hologram (the entire 3D scene). This allows the near clipping plane 224 to be advantageously set at an arbitrary depth in front of the SLM 223.

[0053] FIG. 2C illustrates a region 240 of an ambiguity segment 242, in accordance with one embodiment. Projection 234 intersects near clipping plane 224 and far clipping plane 226. Projection 234 may define a pixel diffraction cone with a base of a certain width. Furthermore, projection 234 may intersect an included region 245 (within the width) that should be included in rendering an associated pixel on SLM 223, and an excluded region 244 (outside the width) that should be excluded from rendering the pixel. Components outside the pixel diffraction cone may be removed as part of rendering one or more pixels within the pixel diffraction cone.

[0054] FIG. 2D illustrates hogel rendering with spherical wave illumination, in accordance with one embodiment. As shown, a rendering configuration 250 includes a virtual camera 260 positioned at a lateral offset d.sub.CZ along the Z (depth) axis with respect to an SLM 263. The virtual camera 260 is aimed at a scene to be rendered, including a near clipping plane 254 and a far clipping plane 256. The position of virtual camera 260 results in a view frustum 262 that intersects at least hogel 268 on SLM 263. Hogels 268, 269 on SLM 263 have a hogel size w.sub.h. The near clipping plane 254 is positioned a distance d.sub.3 along the Z axis from virtual camera 260, and the far clipping plane 256 is positioned a distance d.sub.4 along the Z axis from virtual camera 260. A user eye 261 is positioned a distance d.sub.F along the Z axis from SLM 263. As shown, an eye box is shown to be we in size. In one embodiment, a projection of view frustum 262 through virtual camera 260 is at least as large as the eye box.

[0055] In various embodiments that implement spherical illumination, view frustum 262 (and other view frustums associated with an array of virtual cameras or camera positions) may undergo a spatially varying transform because spherical illumination wavefronts introduce curvature and an off-axis rotation to a local incident ray direction of a diffraction cone for a given position of virtual camera 260. Such diffraction cones collectively widen the field of view of a given hogel.

[0056] Extending the rendering configuration 220 of FIG. 2C, to rendering configuration 250 for spherical illumination sets virtual camera 260 at the intersection of marginal rays restricted by the eye box and skews available field of view. In one embodiment, the lateral offset d.sub.CZ of virtual camera 260 relative to the position of SLM 263 is given by Equation 6:

d CZ = d F w h w e ( d F ) + w h ( 6 ) ##EQU00005##

[0057] An offset between a center view of virtual camera 260 and a hogel center along the X-axis and Y-axis depends on the position of the hogel relative to the eye box. Assuming 2m+1 by 2n+1 partitioning of the SLM 263 along the X-axis and Y-axis, respectively, the displacement from an (m, n)-th hogel center to a corresponding virtual camera is given by Equations 7 and 8:

d C X = m w h d C Z d F ( 7 ) d C Y = nw h d C Z d F ( 8 ) ##EQU00006##

[0058] As shown, displacement d.sub.CY is a displacement along the Y-axis from the center of hogel 268 to the center of view for virtual camera 260. In camera space, an appropriate off-axis projection matrix is defined by Equation 9:

P { m , n } = [ 2 d CZ w h 0 2 d CX w h 0 0 2 d CZ w h 2 d CY w h 0 0 0 - d 4 + d 3 d 4 - d 3 - 2 d 4 d 3 d 4 - d 3 0 0 - 1 0 ] ( 9 ) ##EQU00007##

[0059] A sliding window w.sub.2 inside each elemental image may be used to disambiguate a projected pixel, wherein w.sub.2 is calculated according to Equation 10:

w 2 = ( 1 - d C Z ( d 4 + d F - d C Z ) d 4 d F ) w 1 ( 10 ) ##EQU00008##

[0060] Diffraction culling may be used on an ambiguity segment, illustrated in FIG. 2E showing detail of region 270, to provide more accurate rendering. Diffraction culling may include, without limitation, removing certain covered scene object geometry associated with an ambiguity segment from contributing to a given pixel on SLM 263.

[0061] A fraction of an error-free segment within the sliding window may be used to derive a hogel size required to obtain an acceptable sampling error. Equation 11:

w 3 w 2 = 1 - ( d 4 - d 3 ) d CZ d 3 ( d 4 - d CZ ) ( 11 ) ##EQU00009##

[0062] Hogel size can have a significant impact on visual quality as well as computational effort. In one extreme case of hogel size, a hogel is one pixel within the SLM 263. In this first case, the holographic light field rendering expands to full light field rendering, which may be impractical. In another extreme case of hogel size, a hogel extends to the entire size of the SLM 263. In this second case, the rendered light field recedes to a single map of points rendered from the nearest distance where the SLM 263 is fully observable to a viewer (e.g., a user). In a practical scenario, hogel size is selected between these two extremes, as discussed further in conjunction with FIG. 2G.

[0063] In a holographic light field, a one-to-one mapping between a hogel on an SLM and a corresponding visible elemental image, as shown in FIG. 1D, facilitates parallel computation using a point-based method for Fresnel integration (i.e., summation). For example, light field calculation may proceed as a parallel operation on pixels comprising a hogel, a parallel operation on different virtual camera views, or a combination thereof. Furthermore, a parallel operation on pixels may include parallel computation of summation terms comprising Fresnel integration/summation. In one embodiment, a parallel processing unit, such as the PPU 300 shown in FIG. 3 may be used to perform the parallel computations.

[0064] FIG. 2E illustrates a region 270 of an ambiguity segment 272, in accordance with one embodiment. Projection 264 intersects near clipping plane 254 and far clipping plane 256. Projection 264 may define a pixel diffraction cone with a base of a certain width. Furthermore, projection 264 may intersect an included region 275 (within the width) that should be included in rendering an associated pixel on SLM 263, and an excluded region 274 (outside the width) that should be excluded from rendering the pixel. Components outside the pixel diffraction cone may be removed as part of rendering one or more pixels within the pixel diffraction cone.

[0065] FIG. 2F illustrates algorithmic operations of a method for rendering a light field using spherical illumination, in accordance with one embodiment. In the algorithmic operations, p denotes an SLM pixel in the (m, n)-th hogel, at a displacement (.DELTA.x, .DELTA.y) to the hogel center. A CGH fringe calculation for E(p) of each SLM pixel under spherical illumination multiplies the object wave E.sub.O(p) by a conjugate reference wave E.sub.R*(p). A position q is located on a scene object to be rendered, the position being identified by an index j. In an associated virtual camera space under spherical illumination, p’s spatial coordinate is given by (.DELTA.x+d.sub.CX, .DELTA.y+d.sub.CY, -d.sub.CZ). In one embodiment, p’s estimated view is a sliding window of k.times.k pixels. Furthermore, q.sub.j is the elemental pixel with a rendered point located at (x.sub.q.sub.j, y.sub.q.sub.j, z.sub.q.sub.j), an amplitude A.sub.q.sub.j, and an initial phase .PHI..sub.q.sub.j. This computation is based on Equation 1, and is shown in detail in Equations 12-16.

E(p)=E.sub.O(p)E.sub.R*(p) (12)

[0066] In Equation 12, E.sub.O(p) may be calculated according to Equation 13:

E o ( p ) = ( j = 1 k 2 A q j r ( p , q j ) e i 2 .pi. r ( p , q j ) .lamda. + .phi. q j ) ( 13 ) ##EQU00010##

[0067] Furthermore, E.sub.R*(p) may be calculated according to Equation 14:

E R * ( p ) = ( A F r ( p , F ) e i 2 .pi. r ( p , F ) .lamda. ) ( 14 ) ##EQU00011##

[0068] Euclidean distances r(p, q.sub.j) and r(p, F) may be calculated according to Equations 15 and 16, respectively:

r ( p , q j ) = ( x q j - .DELTA. x - d C X ) 2 + ( y q j - .DELTA. y - d C Y ) 2 + ( z q j + d C Z ) 2 ( 15 ) r ( p , F ) = ( d F ) 2 + ( m w h + .DELTA. x ) 2 + ( n w h + .DELTA. y ) 2 ( 16 ) ##EQU00012##

[0069] In steps 1-4 of FIG. 2F, i is defined as a pixel index for a pixel within an SLM (e.g., SLM 263), and a subset of pixels are identified as being within a hogel. Pixels within a sliding window for the hogel are given an index j. A for loop in step 5 is configured to iterate over pixels within the sliding window to compute a field value for each pixel. A wavefront phase (.PHI.) from a point q to a pixel within the SLM is computed in step 8, while an amplitude (A) for the wave front is computed in step 9. Field summation of Equation 12 is completed in step 11.

[0070] FIG. 2G illustrates a comparison of elemental image resolution results, in accordance with one embodiment. Angular sampling rate (pixels per observed degree) is varied, with image (a) having an angular sampling rate of 6, image (b) having an angular sampling rate of 18, image (c) having an angular sampling rate of 30, and image (d) having an angular sampling rate of 45. An inset (bottom left) of each image depicts the rendered elemental image of resolution varying resolution having a corresponding angular sampling rate, while a detail (top right) illustrates reconstructions at the corresponding angular sampling rate. Lower resolution reconstructions (top row of images) exhibits obvious aliasing; however, higher resolution reconstructions (bottom row of images) is smooth in appearance, without obvious signs of aliasing. In general, an angular sampling rate above 30 pixels per degree provides a good approximation with little noticeable aliasing. Consequently, in one embodiment, an angular sampling rate above 30 pixels per degree is implemented.

[0071] Although a small hogel size (w.sub.h) and dense partitioning increases the number of rendered views needed, a smaller hogel size also reduces ambiguity regions and produces more accurate perspectives for intra-ocular occlusion. A balance may be achieved between competing parameters by evaluating hogel size based on a ratio between error-free segment and approximated sliding window. In one embodiment the SLM includes a resolution of 3840.times.2160 and, w.sub.h.apprxeq.1 mm. This configuration may produce an ambiguity region of less than 0.16% for a two-dimensional view with 16.times.9 hogel partitioning. Note that larger pixel pitch may require denser hogel partitioning.

[0072] FIG. 2H illustrates a flowchart of a method 280 for rendering a light field, in accordance with one embodiment. Although method 280 is described in the context of a processing unit, the method 280 may also be performed by a program, custom circuitry, or by a combination of custom circuitry and a program. For example, the method 280 may be executed by a GPU, a CPU, or any other technically feasible processor. Furthermore, persons of ordinary skill in the art will understand that any system that performs method 280 is within the scope and spirit of embodiments of the present invention.

[0073] At step 282, the processing unit computes a lateral offset (e.g., d.sub.CZ) between a view position and an SLM (e.g., SLM 223, SLM 263) based on a size of the SLM and a width of a holographic element (hogel). In one embodiment, the view position specifies a view position for a virtual camera (e.g., virtual camera 230, virtual camera 260). Furthermore, in one embodiment, an array of hogels covers a surface of the SLM. At step 284, the processing unit renders a three-dimensional scene from the view position to produce an elemental image included within an array of elemental images. The processing unit may render each elemental image within the array of elemental images. In one embodiment, the array of elemental images includes a corresponding array of depth maps (e.g., rendered along with the elemental images). A phase map and an amplitude map are then computed from the elemental images and depth maps, as depicted in FIG. 1D. The phase map and amplitude map may be partitioned to form a one-to-one mapping to the array of hogels. Any technically feasible technique may be implemented to compute the phase map and the amplitude map.

[0074] More illustrative information will now be set forth regarding various optional architectures and features with which the foregoing framework may or may not be implemented, per the desires of the user. It should be strongly noted that the following information is set forth for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.

Parallel Processing Architecture

[0075] FIG. 3 illustrates a parallel processing unit (PPU) 300, in accordance with one embodiment. In one embodiment, the PPU 300 is a multi-threaded processor that is implemented on one or more integrated circuit devices. The PPU 300 is a latency hiding architecture designed to process many threads in parallel. A thread (i.e., a thread of execution) is an instantiation of a set of instructions configured to be executed by the PPU 300. In one embodiment, the PPU 300 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data in order to generate two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device. In other embodiments, the PPU 300 may be utilized for performing general-purpose computations. While one exemplary parallel processor is provided herein for illustrative purposes, it should be strongly noted that such processor is set forth for illustrative purposes only, and that any processor may be employed to supplement and/or substitute for the same

[0076] One or more PPUs 300 may be configured to accelerate thousands of High Performance Computing (HPC), data center, and machine learning applications. The PPU 300 may be configured to accelerate numerous deep learning systems and applications including autonomous vehicle platforms, deep learning, high-accuracy speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimizations, and personalized user recommendations, and the like.

……
……
……