Facebook Patent | High Dynamic Range Camera Assembly With Augmented Pixels

Patent: High Dynamic Range Camera Assembly With Augmented Pixels

Publication Number: 20200389587

Publication Date: 20201210

Applicants: Facebook

Abstract

A camera assembly for generating high dynamic range images. The camera assembly includes a sensor that images a portion of a local area, and a controller. The sensor includes a plurality of augmented pixels, each augmented pixel having a plurality of gates and at least some of the gates have a respective local storage location. An exposure interval of each augmented pixel is divided into intervals associated with the gates, and each local storage location stores image data during a respective interval. The controller reads out, after the exposure interval of each augmented pixel, the image data stored in the respective local storage locations of each augmented pixel to form intermediate images that each have a dynamic range. The controller then generates an image for the portion of the local area using the intermediate images, the image having a higher dynamic range than each of the intermediate images.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of co-pending U.S. application Ser. No. 16/219,502, filed Dec. 13, 2018, which is incorporated by reference in its entirety.

BACKGROUND

[0002] The present disclosure generally relates to a camera assembly, and specifically relates to a high dynamic range camera assembly with augmented pixels.

[0003] To achieve compelling user experience in artificial reality systems, it is essential to rely on an accurate and efficient camera for sensing a two-dimensional (2D) and three-dimensional (3D) surrounding environment.

[0004] Imaging systems with a large high dynamic range (HDR) facilitate 2D imaging of scenes that have a broad spectrum of brightness values. Typically, a number of image frames with different exposures are captured to generate one HDR image. However, this conventional approach features a long latency as the camera needs to read out multiple frames with different exposures. This approach may also feature a motion blur when capturing fast moving objects.

[0005] For achieving compelling 3D user experience of the surrounding environment in artificial reality systems, it is challenging to design a depth camera having a high performance and low computational power, which is also robust to the environment, flexible to operate, and have a compact form factor. Moreover, conventional methods for depth sensing typically involve either a triangulation or time of flight based depth determination, and have several drawbacks. For example, the triangulation based methods generally have a high computational cost to generate a depth map that involves rectification and searching for corresponding points using a pair of stereo images. The depth resolution achieved with the triangulation-based methods also relies on a baseline (e.g., distance between source and camera), and a size of the baseline increases with increasing depth. The time-of-flight methods for depth sensing experience a limited lateral resolution due to a limited number of pixels in conventional sensors. Also, the time-of-flight methods for depth sensing feature motion blur when capturing moving objects due to a relatively high number of image frames required to estimate the depth.

SUMMARY

[0006] A camera assembly presented herein generates high dynamic range images. The camera assembly includes a sensor and a controller. The sensor is configured to image at least a portion of a local area. The sensor includes a plurality of augmented pixels. Each augmented pixel has a plurality of gates and at least some of the gates have a respective local storage location. An exposure interval of each augmented pixel is divided into intervals that are associated with the gates, and each local storage location stores image data during a respective interval. The controller is configured to read out, after the exposure interval of each augmented pixel, the image data stored in the respective local storage locations of each augmented pixel to form a plurality of intermediate images that each have a dynamic range. The controller then generates an image for the portion of the local area using the plurality of intermediate images, wherein the image has a higher dynamic range than each of the intermediate images.

[0007] In some embodiments, the camera assembly is integrated into a headset. The headset may be, e.g., a near-eye display (NED), a head-mounted display (HMD), or some other type of headset. The headset further includes a display and an optical assembly. The headset may be part of an artificial reality system. The display of the headset is configured to emit image light. The optical assembly of the headset is configured to direct the image light to an eye box of the headset corresponding to a location of a user’s eye. The image light may comprise the image for the portion of the local area determined by the camera assembly.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a diagram of a headset, in accordance with one or more embodiments.

[0009] FIG. 2 is a cross-section of the headset in FIG. 1, in accordance with one or more embodiments.

[0010] FIG. 3A is an example sensor having a plurality of augmented pixels, in accordance with one or more embodiments.

[0011] FIG. 3B is an example augmented pixel of the sensor in FIG. 3A, in accordance with one or more embodiments.

[0012] FIG. 4 is an example pixel timing diagram for high-dynamic range imaging, in accordance with one or more embodiments.

[0013] FIG. 5 is a flow chart illustrating a process of generating a high dynamic range image of a local area, in accordance with one or more embodiments.

[0014] FIG. 6 is an example of timing diagrams for time-of-flight operation, in accordance with one or more embodiments.

[0015] FIG. 7A is an example of timing diagrams for time-of-flight operation that includes drain intervals, in accordance with one or more embodiments.

[0016] FIG. 7B is an example of timing diagrams for time-of-flight operation and reduced noise from ambient light, in accordance with one or more embodiments.

[0017] FIG. 8 is an example of timing diagrams for continuous wave time-of-flight operation, in accordance with one or more embodiments.

[0018] FIG. 9A is an example of timing diagrams for continuous wave time-of-flight operations that include a repeating capture interval in each group of capture intervals, in accordance with one or more embodiments.

[0019] FIG. 9B is an example of timing diagrams for continuous wave time-of-flight operations that include multiple repeating capture intervals in each group of capture intervals, in accordance with one or more embodiments.

[0020] FIG. 10 illustrates a sensor of a camera assembly performing a mix of time-of-flight and HDR operations, in accordance with one or more embodiments.

[0021] FIG. 11A is an example of timing diagrams for mixed operation for one augmented pixel of a sensor, in accordance with one or more embodiments.

[0022] FIG. 11B is an example of timing diagrams for mixed operation for another augmented pixel of the sensor, in accordance with one or more embodiments.

[0023] FIG. 12 is a flow chart illustrating a process of determining depth information for a local area, in accordance with one or more embodiments.

[0024] FIG. 13A is an example of timing diagrams for differential operation, in accordance with one or more embodiments.

[0025] FIG. 13B is another example of timing diagrams for differential operation, in accordance with one or more embodiments.

[0026] FIG. 14 is a flow chart illustrating a process of generating differential images for a local area, in accordance with one or more embodiments.

[0027] FIG. 15 is a block diagram of a system environment that includes a headset, in accordance with one or more embodiments.

[0028] The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a near-eye display (NED), a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0030] Some embodiments of the present disclosure relate to a camera assembly configured to generate high dynamic range images of a local area surrounding the camera assembly. The camera assembly includes a sensor having augmented pixels. Each augmented pixel has a plurality of gates and a plurality of associated local storage locations. The camera assembly controls the gates such that gates can have different exposure times. The camera assembly images the local area over a time period. During the time period the camera assembly instructs the augmented pixels to cycle through different gates such that over the time period a plurality of exposures are stored in the local storage locations associated with their respective gates. The camera assembly then reads out the data from the local storage locations of the augmented pixels to form a plurality of intermediate images. The camera generates a high dynamic range image using at least some of the intermediate images.

[0031] Some other embodiments of the present disclosure relate to a camera assembly configured as a depth camera assembly (DCA) to determine depth information using, e.g., time-of-flight based depth sensing techniques. The DCA includes a source assembly and a camera assembly. The source assembly includes a light source (e.g., a pulsed laser) that emits pulse light. The camera assembly includes a sensor that includes augmented pixels that have a plurality of gates (e.g., 3 or 4 gates) and a plurality of associated local storage locations. The camera assembly is configured to image a local area over a time period in sync with pulsed light from the source assembly. During the time period the DCA instructs the augmented pixels to cycle through different gates such that over the time period a plurality of exposures are stored in local storage areas associated with their respective gates. The DCA then reads out the data from the local storage areas of the augmented pixels to form a plurality of intermediate images. The DCA uses the intermediate images to determine depth information for the local area. The DCA may operate in multiple modes, e.g., continuous wave phasor, pulsed phasor, etc. Additionally, in some embodiments, the DCA is configured to adjust the number of repeating cycles on a per augmented pixel basis to control exposure of each of the augmented pixels.

[0032] Some other embodiments of the present disclosure relate to a camera assembly configured to generate differentiated images of a local area. The camera assembly includes a sensor that includes augmented pixels that have two gates, and for each gate an associated local storage location, e.g., a first storage location and a second storage location. The camera assembly images the local area over a time period. During the time period the augmented pixels cycle through different gates such that a plurality of exposures are stored in the first storage location and the second storage location. The camera assembly then determines, for each augmented pixel, a difference between the data in the data stored in the first local storage location and the second local storage location. The camera reads out the difference as a differential image of the local area.

[0033] The camera assembly presented herein may be integrated into a headset. The headset may be, e.g., a NED, a HMD, or some other type of headset. The headset may be part of an artificial reality system. The headset further includes a display and an optical assembly. The display of the headset is configured to emit image light. The optical assembly of the headset is configured to direct the image light to an eye box of the headset corresponding to a location of a user’s eye. In some embodiments, the image light may include depth information for a local area surrounding the headset determined by the camera assembly.

[0034] FIG. 1 is a diagram of a headset 100, in accordance with one or more embodiments. The headset 100 presents media to a user. Examples of media presented by the headset 100 include one or more images, video, audio, or some combination thereof. In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from the headset 100, a console (not shown), or both, and presents audio data based on the audio information. As shown in FIG. 1, the headset 100 is a NED, but in embodiments not shown, the headset may be a HMD, or some other headset. The headset 100 may be part of an artificial reality system (not shown). The headset 100 is generally configured to operate as an artificial reality headset. In some embodiments, the headset 100 may augment views of a physical, real-world environment with computer-generated elements (e.g., images, video, sound, etc.).

[0035] The headset 100 shown in FIG. 1 includes a frame 105 and a display 110. The frame 105 includes one or more optical elements which together display media to users. The display 110 is configured for users to see the content presented by the headset 100. The display 110 generates an image light to present media to an eye of the user.

[0036] The headset 100 further includes a camera assembly 115 configured to image a portion of a local area surrounding some or all of the headset 100. In some embodiments, the camera assembly 115 is implemented as a DCA configured to determine depth information for the local area. The headset 100 also includes an imaging aperture 120, and an imaging device of the camera assembly 115 captures ambient light coming from the local area through the imaging aperture 120. In some embodiments, e.g., when the camera assembly 115 is implemented as a DCA, the camera assembly 115 further includes an illumination aperture 125, and a light source assembly of the camera assembly 115 emits light (e.g., polarized light, structured light, etc.) through the illumination aperture 125. The imaging device of the camera assembly 115 captures light from the light source assembly that is reflected from the local area through the imaging aperture 120. Light emitted from the light source assembly of the camera assembly 115 through the illumination aperture 125 may comprise pulses of light.

[0037] Position of the camera assembly 115 within the headset 100 shown in FIG. 1 is only illustrative, and the camera assembly 115 can be located elsewhere on the frame 105, e.g., on any temple of the frame 105. Also, components of the camera assembly 115 can be located on different locations of the frame 105, e.g., the light source assembly and the illumination aperture 125 can be located on one temple of the frame 105 whereas the imaging device and the imaging aperture 120 can be located on another temple of the frame 105. The headset 100 shown in FIG. 1 is only an example of an artificial reality system. In alternate embodiments (not shown in FIG. 1), the camera assembly 115 could be part of an artificial reality system that includes a HMD, e.g., the camera assembly 115 can be integrated into the HMD.

[0038] In one embodiment, the camera assembly 115 presented in this disclosure provides generated images and/or the determined depth information to a console coupled to the headset 100. The console is then configured to generate content for presentation on an electronic display of the headset 100, based on the images and/or the depth information. In another embodiment, the camera assembly 115 presented herein provides the generated images and/or the determined depth information to a module of the headset 100 that generates content for presentation on the electronic display of the headset 100, based on the images and/or the depth information. In an alternate embodiment, the camera assembly 115 is integrated into the headset 100 as part of an AR system. In this case, the headset 100 may be configured to sense and display objects behind a head of a user wearing the headset 100 or display objects recorded previously. In yet other embodiment, the camera assembly 115 is integrated into a base station or a sensor bar external to the headset 100. In this case, the camera assembly 115 may be configured to sense various body parts of a user wearing the headset 100, e.g., the user’s lower body. In yet other embodiment, the camera assembly 115 is configured as part of a controller or as part of a capture system to capture VR content, AR content, and/or MR content.

[0039] In some embodiments, the camera assembly 115 presented herein and mounted on the headset 100 (or integrated into a frame of the headset 100) can be configured for hand tracking of a user wearing the headset 100. In other embodiments, the camera assembly 115 presented herein is configured to generate a hand gesture recognition input. In yet other embodiments, the camera assembly 115 presented herein is configured to determine object occlusion information, which may be displayed on the display 110. In yet other embodiments, the camera assembly 115 presented herein is configured for depth reconstruction. Alternatively, the camera assembly 115 presented herein can be part of a device external to the headset 100, and can be configured for recording the geometry of a user or a local area, hand-tracking, eye-tracking, etc.

[0040] FIG. 2 is a cross section 200 of the headset 100 illustrated in FIG. 1, in accordance with one or more embodiments. The cross section 200 includes at least one display assembly 210 integrated into the display 110, and the camera assembly 115. An eye box 220 is a location where an eye 225 is positioned when a user wears the headset 100. In some embodiments, the frame 105 may represent a frame of eye-wear glasses. For purposes of illustration, FIG. 2 shows the cross section 200 associated with a single eye 225 and a single display assembly 210, but in alternative embodiments not shown, another display assembly which is separate from the display assembly 210 shown in FIG. 2, provides image light to another eye 225 of the user.

[0041] The display assembly 210 is configured to direct the image light to the eye 225 through the eye box 220. In some embodiments, when the headset 100 is configured as an AR headset, the display assembly 210 also directs light from a local area surrounding the headset 100 to the eye 225 through the eye box 220. The display assembly 210 may be configured to emit image light at a particular focal distance in accordance with varifocal instructions, e.g., provided from a varifocal module (not shown in FIG. 2).

[0042] The display assembly 210 may be composed of one or more materials (e.g., plastic, glass, etc.) with one or more refractive indices that effectively minimize the weight and present to the user a field of view of the headset 100. In alternate configurations, the headset 100 includes one or more optical elements between the display assembly 210 and the eye 225. The optical elements may act to, e.g., correct aberrations in image light emitted from the display assembly 210, magnify image light, perform some other optical adjustment of image light emitted from the display assembly 210, or some combination thereof. The example for optical elements may include an aperture, a Fresnel lens, a convex lens, a concave lens, a liquid crystal lens, a diffractive element, a waveguide, a filter, a polarizer, a diffuser, a fiber taper, one or more reflective surfaces, a polarizing reflective surface, a birefringent element, or any other suitable optical element that affects image light emitted from the display assembly 210.

[0043] The camera assembly 115 mounted on the frame 105 includes an imaging device 240 and a controller 245 coupled to the imaging device 240. In some embodiments, e.g., the camera assembly 115 is implemented as a DCA, and the camera assembly 115 further includes a light source assembly 230 that is coupled to the controller 245. In alternative embodiments (not shown in FIG. 2), the light source assembly 230 and the imaging device 240 each may include its own internal controller. In some embodiments (not shown in FIG. 2), the light source assembly 230 and the imaging device 240 can be separated, e.g., the light source assembly 230 and the imaging device 240 can be located in different assemblies.

[0044] The light source assembly 230 may illuminate the local area with light in accordance with emission instructions generated by the controller 245. The light source assembly 230 may be configured to project pulses of light into the local area, e.g., through the illumination aperture 120. The light source assembly 230 may include a plurality of light sources, e.g., a two-dimensional array of light sources. Examples of light sources include but are not limited to: light-emitting diodes (LEDs), micro light-emitting diodes (.mu.LEDs), micro super luminescent diodes (.mu.SLDs), vertical-cavity surface-emitting lasers (VC SELs), micro resonant cavity light-emitting diodes (.mu.RCLEDs), some other light source that emits light, or some combination thereof. Alternatively, the light source assembly 230 includes a single light source emitting the pulses of light. In some embodiments, the light source assembly 230 includes projection optics that in-couples light from one or more sources of the light source assembly 230 and projects the light over at least a portion of the local area. The light source assembly 230 may be able to emit light in different bands. Furthermore, the light source assembly 230 may be configured to emit different patterns of light, e.g., structured light, diffuse flash, etc. The timing of light source assembly 230 being activated to emit light can be controlled based on, e.g., emission instructions from the controller 245.

[0045] The imaging device 240 is configured to image at least a portion of the local area. The imaging device 240 may be configured to operate with a frame rate in the range of approximately 30 Hz to approximately 1 KHz for fast detection of objects in the local area. The imaging device 240 includes a sensor (not shown in FIG. 2) having a plurality of augmented pixels for capturing light coming from the local area, e.g., through the imaging aperture 120. In some embodiments, the light captured by the augmented pixels of the imaging device 240 is ambient light. In some embodiments, the light captured by the augmented pixels of the imaging device 240 is light emitted by the light source assembly 230 and reflected from the local area. Each augmented pixel includes a plurality of gates, and at least some of the gates have a respective local storage location. An exposure interval of each augmented pixel may be divided into intervals and some of the intervals are synchronized to the pulses of light projected by the light source assembly 230 such that each respective local storage location of each augmented pixel stores image data during a respective interval. The sensor comprising the augmented pixels is described in more detail in conjunction with FIGS. 3A-3B. In some embodiments, e.g., when the camera assembly 115 is implemented as a DCA, the imaging device 240 includes one or more filters (e.g., an infrared filter) for blocking at least portion of ambient light from reaching the sensor of the imaging device 240. The imaging device 240 may be sensitive to different bands of light, e.g., infrared light including one or more bands between 700nm and 1mm.

[0046] The controller 245 may generate the emission instructions and provide the emission instructions to the light source assembly 230 for controlling operation of each individual light source in the light source assembly 230. The controller 245 may also generate capture instructions and provide the capture instructions to the imaging device 240 to control timing of the intervals during which respective local storage locations of each augmented pixel in the sensor element of the imaging device 240 stores image data. The controller 245 may generate images for the local area and/or depth information for the local area based in part on the image data stored in the respective local storage locations of each augmented pixel in the sensor of the imaging device 240, as described in detail in conjunction with FIG. 5, FIG. 12, and FIG. 14. The controller 245 may provide the images and/or the depth information to a console (not shown in FIG. 2) and/or an appropriate module of the headset 100 (e.g., a varifocal module, not shown in FIG. 2). The console and/or the headset 100 may utilize the images and/or the depth information to, e.g., generate content for presentation on the display 110.

[0047] In some embodiments, the headset 100 further includes an eye tracker (not shown in FIG. 2) for determining and tracking a position of the eye 225, i.e., an angle and orientation of eye-gaze. Note that information about the position of the eye 225 also includes information about an orientation of the eye 225, i.e., information about user’s eye-gaze. Based on the determined and tracked position and orientation of the eye 225, the headset 100 adjusts image light emitted from the display assembly 210. In some embodiments, the headset 100 adjusts focus of the image light and ensures that the image light is in focus at the determined angle of eye-gaze in order to mitigate the vergence-accommodation conflict. Additionally or alternatively, the headset 100 adjusts resolution of the image light by performing foveated rendering of the image light, based on the position of the eye 225. Additionally or alternatively, the headset 100 uses the information on a gaze position and orientation to provide contextual awareness for the user’s attention, whether on real or virtual content. The eye tracker generally includes an illumination source and an imaging device (camera). In some embodiments, components of the eye tracker are integrated into the display assembly 210. In alternate embodiments, components of the eye tracker are integrated into the frame 105.

[0048] FIG. 3A is an example sensor 300 having a plurality of augmented pixels 310, in accordance with one or more embodiments. The sensor 300 may be part of the camera assembly 115, e.g., the sensor 300 may be integrated into the imaging device 240 of the camera assembly 115 shown in FIG. 2. Alternatively, the sensor 300 may be implemented as part of some other camera assembly. The sensor 300 may capture, via the augmented pixels 310, light from a local area surrounding some or all of the headset 100. The light captured by the augmented pixels 310 of the sensor 300 may originate from light emitted by the light source assembly 230 of the camera assembly 115 and reflected from one or more objects in the local area, may originate from light emitted by some other light source (e.g., of the headset 100) and reflected from the local area, may originate from ambient light of the local area, or some combination thereof. The augmented pixels 310 may be organized within the sensor 300 as a two-dimensional array of augmented pixels 310. While the sensor 300 illustrated in FIG. 3A is 16.times.16 sensor, in other embodiments the sensor 300 may have some other number of columns and/or rows. For example, the sensor 300 may be 3000.times.4000. The sensor 300 can be generally described as an N.times.M sensor having N rows of augmented pixels 310 and M columns of augmented pixels 310.

[0049] FIG. 3B is an example augmented pixel 310 of the sensor 300 in FIG. 3A, in accordance with one or more embodiments. The augmented pixel 310 captures, over multiple time intervals, intensities of light incident on a surface of the augmented pixel 310. The augmented pixel 310 includes a detector 315 and a plurality of gates, e.g., gates 320A, 320B, 320C, and 320D as shown in FIG. 3B. Although the augmented pixel 310 shown in FIG. 3B includes three gates, each augmented pixel 310 of the sensor 300 may include less than three gates (e.g., two gates) or more than three gates (e.g., four or five gates). In some embodiments, different augmented pixels of the sensor 300 have different numbers of gates and/or storage areas associated with the gates.

[0050] The detector 315 represents a photo-sensitive area of the augmented pixel 310. The detector 315 in-couples, e.g., through an aperture, photons of incident light and transforms the captured photons into electrons (charge). An appropriate electric field generated at a specific time interval based on, e.g., capture instructions from the controller 245, guides the charge from the detector 315 to a local storage location associated with a specific gate, e.g., the gate 320A, 320B, or 320C or to a ground that may be coupled to the gate 320D.

[0051] Some or all of the gates 320A, 320B, 320C, 320D have a respective local storage location for storing a charge related to light captured by the detector 315 at a specific time interval. For example, the gates 320A, 320B, 320C may all have different respective local storage locations. Each gate 320A, 320B, 320C, 320D functions as a switch that is biased to either pass or not pass the charge collected at the detector 315 to, e.g., a local storage area associated with the gate or a drain (e.g., ground). In some embodiments, at least one of the gates 320A, 320B, 320C, 320D operates as a drain gate and does not have a local storage location, and instead drops a charge (e.g., via a ground) received from the detector 315 while active. For example, the gate 320D may be implemented as a drain.

[0052] The respective local storage location of each gate 320A, 320B, 320C may be implemented as an analog electron storage area (e.g., a capacitor) for storing an analog charge generated from the light captured by the detector 315. In some embodiments, the respective local storage location of each gate 320A, 320B, 320C may be coupled to an analog-to-digital converter for converting the analog charge into digital information (i.e., a digital value quantifying an amount of analog charge). Note that in a single augmented pixel 310 there is a plurality of local storage locations that are each associated with a respective gate, and the local storage locations are all local to the single augmented pixel 310 and do not belong to any other augmented pixels 310. Thus, the local storage locations are associated with the particular augmented pixel 310, and they do not represent e.g., a memory bank separate from the sensor 300.

High-Dynamic Range Imaging

[0053] FIG. 4 is an example pixel timing diagram 400 for high-dynamic range imaging, in accordance with one or more embodiments. The pixel timing diagram 400 relates to an embodiment for operating an augmented pixel of a camera assembly (e.g., the augmented pixel 310 of the camera assembly 115) for generating a high dynamic range image of a local area. The augmented pixel 310 may operate in accordance with the pixel timing diagram 400 based on, e.g., capture instructions from the controller 245. The augmented pixel 310 may include n gates each having a respective local storage location, where n>2. The augmented pixel 310 may be configured to image, over an exposure interval 405, at least a portion of the local area. The exposure interval 405 may be divided into capture intervals, e.g., capture intervals 410.sub.1, 410.sub.2, … , 410.sub.n, 415.sub.1, 415.sub.2, … , 415.sub.n, 420.sub.1, 420.sub.2, … , 420.sub.n, as shown in FIG. 4. During each capture interval, a corresponding gate is activated to enable an associated local storage location to store image data (i.e., a charge) associated with light captured by the detector 315 during that capture interval. Note that FIG. 4 is only the illustrative example, and either more or less capture intervals may be associated with each gate of the augmented pixel 310.

[0054] In some embodiments, the augmented pixel 310 includes three gates 320A, 320B, 320C each having a respective local storage location (i.e., n=3). The capture intervals 410.sub.1, 415.sub.1, 420.sub.1 may be associated with a first of the plurality of gates of the augmented pixel 310 (e.g., the gate 320A); the capture intervals 410.sub.2, 415.sub.2, 420.sub.2 may be associated with a second of the plurality of gates of the augmented pixel 310 (e.g., the gate 320B); and as n=3 the capture intervals 410.sub.n, 415.sub.n, 420.sub.n may be associated with a third of the plurality of gates of the augmented pixel 310 (e.g., the gate 320C). Thus, during the capture intervals 410.sub.1, 415.sub.1, 420.sub.1, one of the gates 320A, 320B, 320C (e.g., the gate 320A) may be activated to enable an associated local storage location to store image data (i.e., intensities of light) captured by the detector 315, and the other gates (e.g., all but 320A) are inactive during the capture intervals 410.sub.1, 415.sub.1, 420.sub.1. Similarly, during the capture intervals 410.sub.2, 415.sub.2, 420.sub.2, one of the gates 320A, 320B, 320C (e.g., the gate 320B) may be activated to enable an associated local storage location to store image data (i.e., intensities of light) captured by the detector 315, and the other gates (e.g., all but 320B) are inactive during the capture intervals 410.sub.2, 415.sub.2, 420.sub.2. Similarly, during the capture intervals 410.sub.3, 415.sub.3, 420.sub.3, one of the gates 320A, 320B, 320C (e.g., the gate 320C) may be activated to enable an associated local storage location to store image data (i.e., intensities of light) captured by the detector 315, and the other gates (e.g., all but 320C) are inactive during the capture intervals 410.sub.3, 415.sub.3, 420.sub.3.

[0055] As shown in FIG. 4, a duration of each capture interval 410.sub.1, 415.sub.1, 420.sub.1 associated with the first gate is different than a duration of each capture interval 410.sub.2, 415.sub.2, 420.sub.2 associated with the second gate, which is also different than a duration of each capture interval 410.sub.3, 415.sub.3, 420.sub.3 associated with the third gate, and so on. Thus, a duration of each capture interval 410.sub.i, 415.sub.i, 420.sub.i associated with the i-th gate is different than a duration of each capture interval 410.sub.i+1, 415.sub.i+1, 420.sub.i+1 associated with the (i+1)-th gate, where i=1, 2, … , n-1. Thus, image data stored in each local storage location of a respective gate is associated with a different exposure value of the imaging device 240. The exposure value of the imaging device 240 is based on, e.g., ISO of the imaging device 240, a gain of the imaging device 240, a size of the imaging aperture 120, a length of a shutter of the imaging device 240, a speed of the shutter, some other parameter, or combination thereof. In alternative embodiments (not shown in FIG. 4), each capture interval is a same time interval for each gate that has a respective local storage location, but has a different number of repetitions within the exposure interval 405. In this way, image data stored in each local storage location of a respective gate is again associated with a different exposure value of the imaging device 240.

[0056] In some embodiments, the pixel timing diagram 400 is identical for all augmented pixels 310. A number of capture intervals for all augmented pixels 310 may be identical and depend on, e.g., a signal-to-noise ratio (SNR) associated with image data stored in the local storage locations of each augmented pixel 310.

[0057] In some other embodiments, the pixel timing diagram 400 can be different for two different augmented pixels 310. For example, an exposure interval 405 of one augmented pixel 310 is different than an exposure interval 405 of another augmented pixel 310. The controller 245 may be configured to assign a duration of the exposure interval 405 for an augmented pixel 310, based on a distance between the augmented pixel 310 and at least one object in the local area imaged at least partially by the augmented pixel 310. The controller 245 may obtain information about the distance from previously captured image data related to the at least one object or from some other camera assembly of an artificial reality system that includes the headset 100. For example, an exposure interval 405 for an augmented pixel 310 that images at least a portion of a far object would be longer than an exposure interval 405 for another augmented pixel 310 that images at least a portion of a close object.

[0058] In some embodiments, the controller 245 may be configured to assign a duration of the exposure interval 405 for an augmented pixel 310, based on a brightness of at least one object in the local area imaged at least partially by the augmented pixel 310. The controller 245 may determine the brightness of the at least one object based on image data stored in local storage locations of the augmented pixel 310 during a previous exposure interval. For example, an exposure interval 405 for an augmented pixel 310 imaging at least a portion of an object having a low level of brightness would be longer than an exposure interval 405 for another augmented pixel 310 imaging at least a portion of an object having a high level of brightness.

[0059] As a duration of the exposure interval 405 may depend on a number of repeated capture intervals within the exposure interval 405, the controller 245 may assign the number of intervals for repetition an augmented pixel 310, based on a distance between the augmented pixel 310 and at least one object in the local area imaged at least partially by the augmented pixel. In some embodiments, the controller 245 may assign the number of intervals for repetition for an augmented pixel 310, based on a brightness of at least one object in the local area imaged at least partially by the augmented pixel 310.

[0060] In some embodiments (not shown in FIG. 4), drain intervals occur between two capture intervals. During each of the drain intervals, a drain gate (e.g., 320D) of the augmented pixel 310 may be activated so that charges related to any light incident to a surface of the augmented pixel 310 during the drain interval are not recorded. The drain interval may be placed between two capture intervals, e.g., the capture intervals 410.sub.n and 415.sub.1, to prevent that image data related to the same portion of light is saved in two different local storage locations of two different gates (e.g., the gates 320C, 320A) of the augmented pixel 310. For example, without the drain interval inserted between the two capture intervals 410.sub.n and 415.sub.1, a certain charge related to a portion of light reflected from, e.g., a distant object in the local area might be saved during a portion of the capture interval 415.sub.1 in a local storage location of the first gate. The drain interval inserted between a pair of capture intervals (e.g., 410.sub.n and 415.sub.1) may allow for enough time to pass such that all portions of light of that originate from, e.g., a distant object have already reached the detector 315 of the augmented pixel 310 and would not be saved in a second captured interval in the par (e.g., the capture interval 415.sub.1).

[0061] After the exposure interval 405, e.g., during a read out interval 425, the controller 245 initiates read-out of the image data stored in the local storage locations of each augmented pixel 310 of the imaging device 240. Note that, for the conventional cameras, instead of the single read out interval 425, there are multiple read out intervals, e.g., n read out intervals after each capture interval associated with a corresponding gate. Thus, the approach presented herein based on the single read out interval 425 allows much faster operation of the imaging device 240 in the camera assembly 115 in comparison with the conventional cameras.

[0062] The controller 245 obtains the stored image data and generates a plurality of intermediate images that each have a dynamic range. The controller 245 may then generate an image for the portion of the local area using the plurality of intermediate images, wherein the image has a higher dynamic range than each of the intermediate images. Each pixel value of an intermediate image may be generated using image data stored in a corresponding local storage area of a corresponding gate of the augmented pixel 310 captured during corresponding capture intervals. For example, the controller 245 generates each pixel value of a first of the intermediate images using image data stored in a local storage location of a gate 320A of an augmented pixel 310 captured during the capture intervals 410.sub.1, 415.sub.1, 420.sub.1. Similarly, the controller 245 generates each pixel value of a second of the intermediate images using image data stored in a local storage location of a gate 320B of an augmented pixel 310 captured during the capture intervals 410.sub.2, 415.sub.2, 420.sub.2; and the controller 245 generates each pixel value of a third of the intermediate images using image data stored in a local storage location of a gate 320C of an augmented pixel 310 captured during the capture intervals 410.sub.3, 415.sub.3, 420.sub.3. Then, the controller 245 generates each pixel value of the high dynamic range image by combining corresponding pixel values of the first, second and third intermediate images.

[0063] In some embodiments, the controller 245 generates the high dynamic range image for the portion of the local area by applying a reconstruction algorithm on the plurality of intermediate images. In one or more embodiments, the controller 245 applies the reconstruction algorithm to compute each pixel value of the high dynamic range image based on means of corresponding pixel values of the intermediate images. For example, in the case of three intermediate images, the controller 245 computes each pixel value of the high dynamic range image based on a first mean of corresponding pixel values of the first and second intermediate images and on a second mean of corresponding pixel values of the second and third intermediate images. In some other embodiments, the controller 245 generates the high dynamic range image for the portion of the local area by applying a machine learning algorithm on the plurality of intermediate images. In one or more embodiments, the machine learning algorithm is applied to learn weights of different exposures in the reconstruction at different regions, e.g., bright and dark regions. The machine learning algorithm may be applied to determine weights applied to pixel values of different intermediate images when combined to generate the high dynamic range image. For example, in one embodiment, each intermediate image can be associated with a corresponding single weight applied to all pixel values of that intermediate image. In another embodiment, different regions (e.g., dark and bright regions) of an intermediate image can be associated with different weights.

[0064] FIG. 5 is a flow chart illustrating a process 500 of generating a high dynamic range image for a local area, which may be implemented at the headset 100 shown in FIG. 1, in accordance with one or more embodiments. The process 500 of FIG. 5 may be performed by the components of a camera assembly, e.g., the camera assembly 115 of FIGS. 1-2. Other entities (e.g., a headset and/or console) may perform some or all of the steps of the process in other embodiments. Likewise, embodiments may include different and/or additional steps, or perform the steps in different orders.

[0065] The camera assembly images 510 at least a portion of a local area. The camera assembly images the portion of the local area using a sensor of a camera (e.g., the imaging device 240). The sensor includes a plurality of augmented pixels, each augmented pixel having a plurality of gates and at least some of the gates have a respective local storage location. An exposure interval of each augmented pixel may be divided into intervals (e.g., capture intervals) that are associated with the gates, wherein each local storage location stores image data during a respective interval. In some embodiments, each interval is different for each gate that has a respective local storage location. In some other embodiments, each interval is a same time interval for each gate that has a respective local storage location, but has a different number of repetitions within the exposure interval. In yet some other embodiments, a first exposure interval of a first augmented pixel of the plurality of augmented pixels is different than a second exposure interval of a second augmented pixel of the plurality of augmented pixels.

[0066] The camera assembly reads out 520 (e.g., via a controller), after the exposure interval of each augmented pixel, the image data stored in the respective local storage locations of each augmented pixel to form a plurality of intermediate images that each have a dynamic range. Each intermediate image may associated with a different exposure value of the camera assembly.

[0067] The camera assembly generates 530 (e.g., via the controller) an image for the portion of the local area using the plurality of intermediate images, wherein the image has a higher dynamic range than each of the intermediate images. In some embodiments, the camera assembly generates (e.g., via the controller) the image for the portion of the local area by applying a reconstruction algorithm on the plurality of intermediate images. In some other embodiments, the camera assembly generates (e.g., via the controller) the image for the portion of the local area by applying a machine learning algorithm on the plurality of intermediate images.

Time-of Flight Depth Sensing

[0068] FIG. 6 is an example 600 of timing diagrams 605 and 610 for time-of-flight operation, according to one or more embodiments. The timing diagrams 605, 610 relate to an embodiment for operating a DCA (e.g., the camera assembly 115) to determine depth information for a local area, e.g., based on time-of-flight.

[0069] The timing diagram 605 relates to operation of the light source assembly 230 operating as a continuous wave light source assembly, e.g., based on emission instructions from the controller 245. The controller 245 may instruct the light source assembly 230 to project pulses of light into a local area, e.g., pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R, where R is an integer greater than one, as shown in FIG. 6. For example, the pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R may have a specific pulse rate. As shown in FIG. 6, the projected pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R may form a pulse train that repeats in a periodic manner, e.g., R times, where R is greater than one. The number of repetitions, R, may depend on, e.g., a SNR associated with intensities of light stored in the local storage locations of the augmented pixel 310. In some embodiments, the light source assembly 230 includes a plurality of light sources emitting the pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R. In some other embodiments, the light source assembly 230 includes a single light source configured to emit the pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R. In addition to the pulses of light 615.sub.1, 615.sub.2, … , 615.sub.R illuminating the local area surrounding the DCA, a certain level of ambient light 617 may be also present in the local area. Although the ambient light 617 is illustrated in the timing diagram 605 to have a constant level of intensity over time, the intensity of ambient light 617 may vary over time.

……
……
……