Facebook Patent | Correction for lateral chromatic aberration in images

Patent: Correction for lateral chromatic aberration in images

Drawings: Click to check drawins

Publication Number: 20210152791

Publication Date: 20210520

Applicant: Facebook

Abstract

A camera calibration system includes at least two monochrome cameras and a controller. Each monochrome camera includes a sensor array and a sparse array of filter elements. The sensor array captures one or more images of a local area and includes a plurality of broadband pixels. The sparse array of filter elements is coupled to a portion of the broadband pixels. At least some of the portion of the broadband pixels are along a periphery of the sensor array. Each of the filter elements has a respective passband that is narrower than a passband of the broadband pixels. The controller corrects for lateral chromatic aberration in an image generated from the images captured by the monochrome cameras by using calibration information and portions of the image corresponding to the portion of the broadband pixels that are coupled to the sparse array of filter elements.

Claims

1. A camera calibration system comprising: a camera assembly configured to capture one or more images of a local area, the camera assembly comprising at least two monochrome cameras, each monochrome camera comprising: a sensor array including a plurality of broadband pixels, and a sparse array of filter elements coupled to a portion of the broadband pixels and at least some of the portion of the broadband pixels are along a periphery of the sensor array, the filter elements each having a respective passband that is narrower than a passband of the broadband pixels; and a controller configured to correct for lateral chromatic aberration in the one or more captured images using calibration information and portions of the one or more images of the local area corresponding to the portion of the broadband pixels that are coupled to the sparse array of filter elements.

2. The camera calibration system of claim 1, wherein the calibration information comprises information describing offset for light in a target band as a function of positions of at least some of the plurality of broadband pixels.

3. The camera calibration system of claim 2, wherein the one or more capture images are one or more images of a target image in the target band, and the controller is further configured to: determine offsets for light in the target band captured at the portion of the broadband pixels that is coupled to the sparse array of filter elements using the portions of the one or more captured images; and extrapolate offsets for other broadband pixels of the sensor array of the monochrome camera.

4. The camera calibration system of claim 2, wherein the target band comprises a color channel of a visible band.

5. The camera calibration system of claim 1, wherein a subset of the portion of the broadband pixels are at a center of the sensor array,

6. The camera calibration system of claim 1, wherein the broadband pixels have a passband that includes a visible band and an infrared band.

7. The camera calibration system of claim 6, wherein the passband of the filter elements is at least one of a color channel of a visible band and an infrared band.

8. The camera calibration system of claim 7, wherein the color channel is green.

9. The camera calibration system of claim 1, wherein the passband of a first portion of the filter elements is in an infrared band and has a bandwidth of at least 5 nanometers, and the controller is further configured to: detect a wavelength shift in light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels, wherein at least some of the captured light is from a peripheral device; and instruct the peripheral device to adjust a wavelength of the emitted light, and an amount of adjustment is based in part on the detected wavelength shift.

10. The camera calibration system of claim 1, wherein the controller is further configured to: detect a brightness in ambient light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels; and instruct an illuminator to adjust brightness of light emitted by the illuminator, and an amount of adjustment is based in part on the detected brightness of the ambient light.

11. The camera calibration system of claim 1, wherein the controller is further configured to use the one or more corrected images to generate depth information for the local area.

12. The camera calibration system of claim 1, further comprising a color camera comprising a red-green-blue (RGB) sensor array including a plurality of RGB pixels, the RGB sensor array configured to capture one or more RGB images of a local area, wherein the controller is further configured to correct for lateral chromatic aberration in the one or more captured images based in part on the one or more RGB images.

13. A method comprising: capturing one or more images of a local area by a camera assembly comprising at least two monochrome cameras, each monochrome camera comprising: a sensor array including a plurality of broadband pixels, and a sparse array of filter elements coupled to a portion of the broadband pixels and at least some of the portion of the broadband pixels are along a periphery of the sensor array, the filter elements each having a respective passband that is narrower than a passband of the broadband pixels; and correcting for lateral chromatic aberration in the one or more captured images using calibration information and portions of the one or more captured images corresponding to the portion of the broadband pixels that are coupled to the sparse array of filter elements.

14. The method of claim 13, wherein the calibration information comprises information describing offset for light in a target band as a function of positions of at least some of the plurality of broadband pixels.

15. The method of claim 14, further comprising: capturing one or more images of a target image in the target band; determining, by using the portions of the one or more captured images, offsets for light in the target band captured at the portion of the broadband pixels that is coupled to the sparse array of filter elements; and extrapolating offsets for other broadband pixels of the sensor array of the monochrome camera.

16. The method of claim 14, wherein the target band comprises a color channel of a visible band.

17. The method of claim 13, wherein the passband of a first portion of the filter elements is in an infrared band and has a bandwidth of at least 5 nanometers, and the method further comprising: detecting a wavelength shift in light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels, wherein at least some of the captured light is from a peripheral device; and instructing the peripheral device to adjust a wavelength of the emitted light, and an amount of adjustment is based in part on the detected wavelength shift.

18. The method of claim 13, further comprising: detecting a brightness in ambient light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels; and instructing an illuminator to adjust brightness of light emitted by the illuminator, and an amount of adjustment is based in part on the detected brightness of the ambient light.

19. A method comprising: capturing, by a camera assembly, one or more images of a target image in a target band, the camera assembly comprising at least two monochrome cameras, each monochrome camera comprising: a sensor array including a plurality of broadband pixels, the sensor array configured to capture one or more images of a local area, and a sparse array of filter elements coupled to a portion of the broadband pixels and at least some of the portion of the broadband pixels are along a periphery of the sensor array, the filter elements each having a respective passband that is narrower than a passband of the broadband pixels; determining offsets for light in the target band captured at the portion of the broadband pixels that is coupled to the sparse array of filter elements; extrapolating offsets for other broadband pixels of the sensor array of the monochrome camera; and generating calibration information for correcting lateral chromatic aberration in the camera assembly based on the determined offsets and extrapolated offsets, the calibration information comprising information describing offset for light in a target band as a function of positions of at least some of the plurality of broadband pixels.

20. The method of claim 19, wherein the target band comprises a color channel of a visible band.

Description

FIELD OF THE INVENTION

[0001] This disclosure relates generally to correcting for chromatic aberration, and more specifically to correction for lateral chromatic aberration in a camera assembly of a stereo based depth system.

BACKGROUND

[0002] Head-mounted display (HMD) system typically uses one or more optical elements to provide a magnified view of its display. However, HMD systems usually use positive optical elements that are refractive in nature. These optical elements cause chromatic aberration, which may degrade image quality and introduce error in stereo matching algorithm. Conventional solutions correct for chromatic aberration is to use an achromatic optical element (generally includes 2 or more optical elements). However, this solution adds optical components, increases form factor, and can be expensive.

SUMMARY

[0003] Embodiments of the present disclosure support a camera calibration system and method for generating calibrating information for a camera assembly and correcting for lateral chromatic aberration in images captured by the camera assembly.

[0004] In some embodiments, the camera assembly captures one or more images of a target image. The camera assembly includes at least two monochrome cameras. Each monochrome camera of the camera calibration system captures image data of the target image. An image captured by the camera assembly can be a stereo image generated based on the image data captured by the monochrome cameras. The target image is displayed in a target band. Each monochrome camera includes a sensor array and a sparse array of filter elements. The sensor array includes a plurality of broadband pixels. The sparse array of filter elements is coupled to a portion of the broadband pixels. At least some of the portion of the broadband pixels are along a periphery of the sensor array. Each of the filter elements has a respective passband that is narrower than a passband of the broadband pixels. Offsets for light in the target band captured at the portion of the broadband pixels that is coupled to the sparse array of filter elements are determined. Offsets for other broadband pixels of the sensor arrays of the monochrome cameras are extrapolated. Calibration information for correcting lateral chromatic aberration in the camera assembly is generated based on the determined offsets and extrapolated offsets. The calibration information comprising information describing offset for light in a target band as a function of positions of at least some of the plurality of broadband pixels.

[0005] A controller of the camera calibration system corrects for lateral chromatic aberration in the one or more images captured by the camera assembly. The controller uses the calibration information and portions of the one or more captured images of the local area corresponding to the portion of the broadband pixels that are coupled to the sparse array of filter elements.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 illustrates lateral chromatic aberration of an optical element, in accordance with one or more embodiments.

[0007] FIG. 2 is a block diagram of a camera calibration system coupled to a display device, in accordance with one or more embodiments.

[0008] FIG. 3 is a perspective view of a headset implemented as a head-mounted display, in accordance with one or more embodiments.

[0009] FIG. 4A illustrates a sensor array of a monochrome camera, in accordance with one or more embodiments.

[0010] FIG. 4B illustrates another sensor array of a monochrome camera, in accordance with one or more embodiments.

[0011] FIG. 5 illustrates sensor arrays of a camera assembly comprising a color camera and two monochrome cameras, in accordance with one or more embodiments.

[0012] FIG. 6 is a flowchart illustrating a method for generating calibration information used for correcting lateral chromatic aberration, in accordance with one or more embodiments.

[0013] FIG. 7 is a flowchart illustrating a method for correcting lateral chromatic aberration, in accordance with one or more embodiments.

[0014] FIG. 8 is a system that includes a headset, in accordance with one or more embodiments.

[0015] The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

DETAILED DESCRIPTION

[0016] Embodiments relate to a camera calibration system that corrects for lateral chromatic aberration in images captured by a camera assembly comprising at least two monochrome cameras. The two monochrome cameras are separated by a base line and can capture images of one or more objects in a local area from different angles. The camera assembly can be a component of a stereo based depth system. Depth information of the local area can be extracted from the images by examining positions of an object in the images.

[0017] Each monochrome camera includes a sensor array and a sparse array of filter elements. The sensor array includes broadband pixels that can capture images of a local area. The filter elements are coupled to broadband pixels at a periphery of the sensor array. In some embodiments, some of the filter elements are coupled to broadband filters at a center of the sensor array. Each of the filter elements have a passband that is significantly narrower than a passband of the corresponding broadband pixels. For example, the broadband pixels have a passband that includes a visible band (e.g., .about.380 nm to 750 nm) and at least a portion of an infrared band (e.g., .about.750 nm to 1 mm). The passband of the filter elements can be a subset of the visible band and/or a subset of the infrared band. A subset of the visible band can be a band corresponding to a color, i.e., a color channel, such as 500-550 nanometers for green. The passband of the filter elements can also include an infrared band in addition or alternative to the color channel.

[0018] Each monochrome camera captures one or more images of a target image that is displayed in a target band. The target band can be a color channel of a visible band or an infrared band. The camera calibration system generates a stereo image by combining images captured by the at least two monochrome cameras and compares portions of the stereo image that correspond to the broadband pixels coupled to the filter elements with corresponding portions of the target image. Based on the comparison, the camera calibration system determines offsets for light in the target band captured at the broadband pixels coupled to the filter elements and extrapolates offsets for the rest of the broadband pixels of the sensor array of the monochrome cameras. Based on the determined offsets and extrapolated offsets, the camera calibration system generates calibration information that describes offset for light in the target band as a function of positions of broadband pixels. In some embodiments, an offset is a shift in a position of a point of the local area in the captured images. The camera calibration system applies the calibration information to correct for lateral chromatic aberration in an image of a local area captured by the monochrome camera, e.g., for determining depth information of the local area.

[0019] The filter elements can have a passband that is a subset of the visible band, the correction of lateral chromatic aberration using the filter elements can improve stereo view of the location area that captured by the camera assembly. Also, determination of depth information of the local area based on images captured by the camera assembly can be improved by using filter elements that have a passband in the infrared band. Furthermore, filter elements having passband in an infrared band can also be used to detect wavelength shift in emitted light, e.g., light emitted by a peripheral device associated with the camera assembly and/or light projected by an illuminator associated with the camera assembly.

[0020] The camera calibration system is more advantageous than convention techniques for correction lateral chromatic aberration. It does not require additional optical components other than overlaying the existing sensor array of the camera with filter elements. It also does not increase form factor or volume of the headset. Rather, it reduces complexity of module design and is less expensive.

[0021] Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to create content in an artificial reality and/or are otherwise used in an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a wearable device (e.g., headset) connected to a host computer system, a standalone wearable device (e.g., headset), a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0022] FIG. 1 illustrates lateral chromatic aberration of an optical element 110, in accordance with one or more embodiments. The optical element 110 is a positive optical element that refracts light projected onto it. The optical element 110 may be made of various types of materials, such as plastic, glass, crystalline materials, or some combination thereof.

[0023] As shown in FIG. 1, the optical element 110 has an optical axis 120. Light 130 is project toward the optical element 110. The light 130 comprises various wavelengths of light. In some embodiments, a band of the light 130 includes various color channels of a visible band.

[0024] The optical element 110 is configured to focus the light 130 at a focal point 160 on a focal plane 170. However, lateral chromatic aberration in the optical element 110 causes different wavelengths to be focused at different positions on the focal plane 170. As shown in FIG. 1, the optical element 110 focuses a first light 180 at a first focal point 185 and focuses a second light 190 at a second focal point 195. The first light 180 (e.g., red light) has a larger wavelength than a wavelength of the second light 190 (e.g., blue light), and the first focal point 185 is below the second focal point 195. Lateral chromatic aberration in an optical element of a camera can cause aberration in images captured by the camera, such as color shifting, purple fringing, blur, etc. Lateral chromatic aberration also affects black-and-white images. Although there are no colors in black-and-white images, chromatic aberration can blur the images.

[0025] FIG. 2 is a block diagram of a camera calibration system 200 coupled to a display device 210, in accordance with one or more embodiments. The camera calibration system 200 generates calibration information for correcting lateral chromatic aberration in images captured by a camera assembly 230. The camera calibration system 200 includes the camera assembly 230 and a controller 240. In some embodiments, some or all of the camera calibration system 200 is integrated into a headset. In other embodiments, functionality of the camera calibration system 200 can be performed by third party work stations coupled to the headset. The camera calibration system 200 may be a part of an imaging system of a headset that generates depth information and color images of a local area surrounding the headset.

[0026] A network 220 couples the display device 210 to the camera calibration system 200. The network 220 may include any combination of target area and/or wide area networks using both wireless and/or wired communication systems. For example, the network 220 may include the Internet, as well as mobile telephone networks. In one embodiment, the network 220 uses standard communications technologies and/or protocols. Hence, the network 220 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G/3G/4G mobile communications protocols, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the networking protocols used on the network 220 can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network 220 can be represented using technologies and/or formats including image data in binary form (e.g. Portable Network Graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. The network 220 may also connect multiple display devices to the headset 220 or connect the display device 210 to multiple headsets.

[0027] The display device 210 displays a target image in a target band. The target image is an image of a specific pattern that can be used to detect aberrations in the image caused by lateral chromatic aberration. In some embodiments, the target image includes a sine wave pattern, such as square grid or circular grid of different sizes, etc. In some embodiments, the target band is a color channel (e.g., green, red, blue, some other color, etc.) of a visible band or an infrared band. The target band can also be an infrared band. The display device 210 can display the target image according to an instruction received from the camera calibration system 200, such as the controller 240 of the camera calibration system 200, through the network 220. The instruction can include specification of a pattern in the target image, the target band, a period of time of displaying the target image, or other information related to display of the target image by the display device 210. The display device 210 comprises one or more electronic display panel. Examples of an electronic display panel include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, an active-matrix organic light-emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, some other display, or some combination thereof.

[0028] The camera assembly 230 is configured to capture images of a local area, e.g., an area surrounding the headset. The local area may be a room that a user wearing the headset is inside, or the user wearing the headset may be outside and the local area is an outside area. An image captured by the camera assembly 230 can have poor quality due to optical errors in the camera assembly 230, such as lateral chromatic aberration of an optical assembly of the camera assembly 230, shifts in positions of the camera assembly 230, wear to tear of the camera assembly 230, etc.

[0029] The camera assembly 230 captures one or more images, e.g., stereo images, of the target image in accordance with an instruction from the controller 240. The instruction can include specification of the number of images to be captured by the camera assembly 230, times when images of the target image should be captured, or other parameters regarding capturing images of the target image. The camera assembly 230 can include at least two monochrome cameras. Each captured image can be generated by combining image data generated by the monochrome cameras. The captured images of the target image can be used for generating the calibration information.

[0030] Each monochrome camera includes a sensor array and a sparse array of filter elements. The sensor array captures images of the local area. The sensor array includes a plurality of broadband pixels. The filter elements are coupled to a subset of the broadband pixels. Some broadband pixels in the subset are arranged along a periphery of the sensor array. In some embodiments, the camera assembly 230 has one filter element for every 32 broadband pixels. Each of the filter elements has a passband that is narrower than a passband of the broadband pixels. For example, the passband of the broadband pixels includes a visible band and an infrared band, and the passband of a filter element is a color channel of a visible band and/or an infrared band (such as a bandwidth of at least 5 nanometers). In some embodiments, the camera assembly 230 can include a RGB (red-green-blue) sensor array. More details regarding sensor arrays of the camera assembly 230 are described below in conjunction with FIGS. 4A, 4B and 5.

[0031] The controller 240 generates the instruction to the display device 210 for displaying the target image and the instruction to the camera assembly 230 to capture the one or more images of the target image. For instance, the controller 240 selects a pattern and instructs the display device 210 to display an image of the pattern, i.e., the target image, in the target band within a period of time. The controller 240 also instructs the camera assembly 230 to capture the one or more images of the target image within the period of time. An image captured by the camera assembly 230 can be a stereo image or reconstructed image generated from images captured by monochrome cameras of the camera assembly 230.

[0032] The controller 240 receives the one or more images captured by the camera assembly 230 and generates calibration information for the camera assembly 230 based on the one or more captured images. The captured images are at wavelengths corresponding to the passband of the filter elements. In some embodiments, the controller 240 extracts portions of each captured image that corresponds to the broadband pixels that are coupled to the filter elements. The controller 240 compares the extracted portions of each captured images with corresponding portions of the target image. From the differences between the extracted portions of the captured image and the corresponding portions of the target image, the controller 240 determines aberrations in the portions of the captured image based on the comparison. The aberrations are at least partially caused by lateral chromatic aberration. Based on the determined aberrations, the controller 240 determines offsets for counterbalancing the aberrations. Each offset can correspond to a broadband pixel of the sensor array of the camera assembly 230 that is coupled to a filter element.

[0033] The controller 240 extrapolates offsets for the other broadband pixels of the sensor array that are not coupled to the filter elements based on the determined offsets. For instance, the controller 240 determines that the offset for one or more broadband pixels in a center of the sensor array is zero. The controller 240 then extrapolates the offsets for these other for other broadband pixels based on the determined offsets for the broadband pixels that is coupled to the sparse array of filter elements and the zero offset for the one or more broadband pixels in the central region of the sensor array. In some embodiments, the extrapolation is based on a camera projection model.

[0034] In some embodiments, the controller 240 determines an offset function that describes as a function of distance from the center of the sensor array how offset changes. The controller 240 determines the offset function using the determined offset values and information describing the optical assembly of the camera assembly 230, such as indices of refraction, curvature of surfaces, lens thickness, etc. The controller 240 generates the calibration information based on the offset function.

[0035] The controller 240 uses the calibration information to correct for lateral chromatic aberration for the passbands of the filter elements in images captured by the camera assembly 230. For example, the controller 240 uses the calibration information to eliminate aberrations in some (e.g., correspond to the portion of the broadband pixels that are coupled to the sparse array of filter elements) or all of an image captured by the camera assembly 230.

[0036] The controller 240 can also use corrected images to generate depth information of the local area. The controller 240 can also use the depth information to generate content to be presented to a user by the headset 210. The content can be virtual images, or a mixture of virtual images and real images of the local area. In some embodiments, the controller 240 generates a model of the local area based on the depth information. The controller 240 can also generate position information of an object in the local area that indicates a position of the object in the model. For example, the depth information indicates that the object has moved further from the headset, the controller 240 generates content for the headset that mirrors the object’s movement in an augmented reality environment.

[0037] In some embodiments, the controller 240 uses images captured by other cameras of the camera assembly 230 to correct for the lateral chromatic aberration in images captured by the camera. For instance, the controller 240 can use images captured by a color camera to correct for lateral chromatic aberration in images captured by the monochrome cameras. For instance, the controller 240 can use color images generate by the color camera to generate calibration information at different color bands from the passband of the filter elements. In some embodiments, the controller 240 also generates instructions for a peripheral device. The peripheral device projects light that can be captured by the camera assembly 230. In some embodiments, the peripheral device is a hand-held controller coupled to the headset. The projected light can be used for tracking positions/orientations of the peripheral device. The projected light can be in an infrared band, and the passband of some of the filter elements of a monochrome camera can be near IR band, e.g., having a bandwidth of at least 5 nanometers. The controller 240 detects a wavelength shift in light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels. For instance, the controller 240 obtains an intended wavelength of light, e.g., from a component associated with the projected light that controls projection of the light. The intended wavelength is a wavelength of light that the peripheral device should project. The controller 240 determines a wavelength of the projected light based on one or more image of the peripherical device captured by the camera assembly 230 and compares the intended wavelength with the wavelength of the projected light to detect the wavelength shift. In response to the detection, the controller 240 generates instructions for the peripheral device and instructs the peripheral device to adjust a wavelength of the emitted light. The generated instruction can include an amount of adjustment determined by the controller 240 based in part on the detected wavelength shift.

[0038] The controller 240 can also instruct the peripheral device (or a different system, e.g., an eye tracking system) to adjust brightness of emitted light so that it is detectable over ambient light. The controller 240 detects a brightness in ambient light filtered by the portion of the filter elements and captured by a corresponding portion of the broadband pixels. In response to the detection, the controller 240 generates instructions for the peripheral device or the different system and instructs it to adjust a brightness of the emitted light. For instance, the controller 240 instructs the peripheral device to increase the brightness of its emitted light to a level that is higher than the detected brightness of the ambient light. The generated instruction can include an amount of adjustment determined by the controller 240 based in part on the detected brightness of the ambient light.

[0039] FIG. 3 is a perspective view of a headset 300 implemented as a HMD, in accordance with one or more embodiments. The headset 300 is an embodiment of the headset 220 in FIG. 2. In embodiments where the headset 300 is used in an AR system and/or a MR system, portions of a front side of the headset 300 are at least partially transparent in the visible band (.about.380 nm to 750 nm), and portions of the headset 300 that are between the front side of the headset 300 and an eye of the user are at least partially transparent (e.g., a partially transparent electronic display).

[0040] In general, the headset 300 may be worn on the face of a user such that content (e.g., media content) is presented using a display assembly and/or an audio system. However, the headset 300 may also be used such that media content is presented to a user in a different manner. Examples of media content presented by the headset 300 include one or more images, video, audio, or some combination thereof.

[0041] The headset 300 includes a front rigid body 310, a depth camera assembly (DCA) that includes two monochrome cameras 330 and 335, a color camera 320, an illuminator 340, an audio controller 350, speakers 360, a band 370, acoustic sensors 380, and a position sensor 390. The headset 300 also includes components partially shown or not shown in FIG. 3, such as a display assembly including one or more display elements and an audio system. While FIG. 3 illustrates the components of the headset 300 in example locations on the headset 300, the components may be located elsewhere on the headset 300, on a peripheral device paired with the headset 300, or some combination thereof. For example, the speakers 360 may be located in various locations, such as coupled to the band 370 (as shown in FIG. 3), coupled to the front rigid body 310, or may be configured to be inserted within the ear canal of a user. There may be more or fewer components on the headset 300 than what is shown in FIG. 3.

[0042] The front rigid body 310 holds the other components of the headset 300. The front rigid body 310 includes a front part that holds the one or more display elements. The one or more display elements provide light to a user wearing the headset 300. The headset 300 can include a display element for each eye of a user.

[0043] In some embodiments, a display element generates image light that is provided to an eyebox of the headset 300. The eyebox is a location in space that an eye of user occupies while wearing the headset 300. For example, a display element is an electronic display panel. Examples of an electronic display panel include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an inorganic light emitting diode (ILED) display, an active-matrix organic light-emitting diode (AMOLED) display, a transparent organic light emitting diode (TOLED) display, some other display, or some combination thereof.

……
……
……