Facebook Patent | Dynamically programmable image sensor

Patent: Dynamically programmable image sensor

Drawings: Click to check drawins

Publication Number: 20210044742

Publication Date: 20210211

Applicant: Facebook

Abstract

In one example, an apparatus comprises a pixel cell array and a controller formed within a semiconductor package. The pixel cell array is configured to: generate, at a first time and based on first programming signals received from the controller, a first image frame; transmit the first image frame to a host processor; and transmit the first image frame or a second image frame to the controller, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame. The controller is configured to receive second programming signals from a host processor, the second programming signals being determined by the host processor based on the first image frame; update the first programming signals based on the second programming signals; and control the pixel cell array to generate a subsequent image frame based on the updated first programming signals.

Claims

1. An apparatus comprising: a pixel cell array, each pixel cell in the pixel cell array including a photodiode and processing circuits, the photodiodes being formed in a first semiconductor substrate and the processing circuits formed in one or more second semiconductor substrates; and a controller formed in at least one of the one or more second semiconductor substrates, the first semiconductor substrate and the one or more second semiconductor substrates forming a stack and housed within a semiconductor package; wherein the pixel cell array is configured to: generate, at a first time and based on first programming signals received from the controller, a first image frame; transmit the first image frame to a host processor; and transmit the first image frame or a second image frame to the controller, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame; and wherein the controller is configured to: receive the first image frame or a second image frame from the pixel cell array; receive second programming signals from a host processor, the second programming signals being determined by the host processor based on the first image frame; update the first programming signals based on the second programming signals; and control the pixel cell array to generate a subsequent image frame at a second time based on the updated first programming signals.

2. The apparatus of claim 1, further comprising the host processor; wherein the controller is further configured to update the first programming signals based on the first image frame or the second image frame and at a frame rate at which the pixel cell array generates image frames; and wherein the host processor is configured to update the second programming signals at a rate lower than the frame rate.

3. The apparatus of claim 1, wherein the first programming signals comprise a first subset of the first programming signals identifying a first subset of pixel cells of the pixel cell array and a second subset of first programming signals identifying at a second subset of pixel cells of the pixel cell array; and wherein the controller is further configured to update the first subset of pixel cells identified by the first subset of the first programming signals, and update the second subset of pixel cells identified by the second subset of the first programming signals.

4. The apparatus of claim 3, wherein the first subset of the first programming signals control the first subset of pixel cells to output pixel data at a first resolution to the host processor; and wherein the second subset of the first programming signals control the second subset of pixel cells not to output pixel data, or to output pixel data at a second resolution lower than the first resolution, to the host processor.

5. The apparatus of claim 1, wherein the controller is further configured to: perform, based on the second programming signals, an image processing operation on the first image frame or the second image frame to generate a processing result; and update the first programming signals based on the processing result.

6. The apparatus of claim 5, wherein the image processing operation comprises: determining, for each pixel included in the first image frame or the second image frame, a pixel value difference with respect to a corresponding pixel in a third image frame generated by the pixel cell array prior to the first time; and identifying a subset of pixels in the first image frame or the second image frame for which the pixel value differences exceed a threshold; wherein the controller is further configured to update the first programming signals based on identifying a subset of pixel cells of the pixel cell array, the subset of pixel cells being identified based on the subset of pixels; and wherein the second programming signals define the threshold.

7. The apparatus of claim 6, further comprising the host processor, wherein the host processor is configured to set the threshold based on determining an ambient light intensity from the first image frame.

8. The apparatus of claim 5, wherein the image processing operation comprises identifying a subset of pixels in the first image frame or in the second image frame that include target features of an object of interest; wherein the controller is further configured to update the first programming signals are updated based on identifying a subset of pixel cells of the pixel cell array, the subset of pixel cells being identified based on the subset of pixels; and wherein the second programming signals include information about the target features.

9. The apparatus of claim 8, wherein the second programming signals indicate an initial subset of pixels in the first image frame that include the target features; and wherein the image processing operation comprises identifying the subset of pixels based on determining whether the initial subset of pixels in the first image frame or in the second image frame include the target features of the object of interest.

10. The apparatus of claim 8, wherein the controller is further configured to: based on information about a movement of the object of interest and a time difference between the first time and the second time, determine the subset of pixels in the subsequent image frame that include the target features; and update the first programming signals based on determining a subset of pixel cells of the pixel cell array based on the subset of pixels in the subsequent image frame; and wherein the second programming signals include the information about the movement of the object of interest.

11. The apparatus of claim 8, wherein the second programming signals indicate that an initial subset of the pixels in the first image frame, defined based on the first programming signals, do not include all the target features; and wherein the image processing operation comprises, based on the second programming signals, identifying additional pixels that include the target features.

12. The apparatus of claim 8, wherein the controller includes a neural network model to perform the image processing operation; and wherein the second programming signals include at least one of: weights of the neural network model, backward propagation gradients to update the weights, a predicted accuracy of image processing operation, or intermediate outputs from the image processing operation.

13. The apparatus of claim 1, wherein each pixel cell of the pixel cell array or each block of pixel cells of the pixel cell array is individually addressable; and wherein the first programming signals comprise pixel-level signals individually targeted at each pixel cell or block-level signals targeted at each block of pixel cells.

14. The apparatus of claim 1, further comprising: pixel interconnects implemented by chip-to-chip copper bonding between the first semiconductor substrate and the one or more second semiconductor substrates to transmit signals generated by the photodiodes in the first semiconductor substrate to the processing circuits of pixel cell array in the one or more second semiconductor substrates; and through silicon vias (TSV) between the first semiconductor substrate and the one or more second semiconductor substrates to transmit the image frames from the pixel cell array to the controller, and to transmit the first programming signals from the controller to the pixel cell array.

15. The apparatus of claim 1, wherein the pixel cell array is coupled with the host processor via a point-to-point serial interface and is configured to transmit the image frames to the host processor via the point-to-point serial interface.

16. The apparatus of claim 1, wherein the controller is coupled with the host processor via a shared bus interface, the shared bus interface further coupled with other controllers; and wherein the controller is configured to receive the second programming signals from the host processor via the shared bus interface.

17. A method, comprising: controlling a pixel cell array to generate a first image frame at a first time based on first programming signals, and to transmit the first image frame to a host processor; receiving, from the pixel cell array, the first image frame or a second image frame from the pixel cell array, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame; receiving second programming signals from the host processor, the second programming signals being determined by the host processor based on the first image frame; updating the first programming signals based on the second programming signals and the first image frame or the second image frame received from the pixel cell array; and controlling the pixel cell array to generate a subsequent image frame at a second time based on the updated first programming signals.

18. The method of claim 17, further comprising: performing, based on the second programming signals, an image processing operation on the first image frame or the second image frame to generate a processing result; and updating the first programming signals based on the processing result.

19. The method of claim 18, wherein the image processing operation comprises identifying a subset of pixels in the first image frame or in the second image frame that include target features of an object of interest; wherein the first programming signals are updated based on identifying the subset of pixel cells; and wherein the second programming signals include information about the target features.

20. The method of claim 17, wherein the pixel cell array transmits the first image frame to the host processor via a point-to-point serial interface; and wherein the second programming signals are received via a shared bus interface.

Description

RELATED APPLICATION

[0001] This patent application claims priority to U.S. Provisional Patent Application Ser. No. 62/883,014, filed Aug. 5, 2019, entitled “ON-SENSOR PREDICTIVE SPARSE CAPTURE,” which is assigned to the assignee hereof and is incorporated herein by reference in its entirety for all purposes.

BACKGROUND

[0002] The disclosure relates generally to image sensors, and more specifically to dynamically programmable image sensors.

[0003] A typical image sensor includes a pixel cell array. Each pixel cell may include a photodiode to sense light by converting photons into charge (e.g., electrons or holes). The charge converted at each pixel cell can be quantized to become a digital pixel value, and an image can be generated from an array of digital pixel values. The light sensing and image generation operations of an image sensor can be configured based on pre-determined programming signals.

SUMMARY

[0004] The present disclosure relates to image sensors. More specifically, and without limitation, this disclosure relates to an image sensor that is dynamically programmable by an integrated controller and by a host device based on image data captured by the image sensor.

[0005] In one example, an apparatus is provided. The apparatus comprises: a pixel cell array, each pixel cell in the pixel cell array including a photodiode and processing circuits, the photodiodes being formed in a first semiconductor substrate and the processing circuits formed in one or more second semiconductor substrates; and a controller formed in at least one of the one or more second semiconductor substrates, the first semiconductor substrate and the one or more second semiconductor substrates forming a stack and housed within a semiconductor package. The pixel cell array is configured to: generate, at a first time and based on first programming signals received from the controller, a first image frame; transmit the first image frame to a host processor; and transmit the first image frame or a second image frame to the controller, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame. The controller is configured to: receive the first image frame or a second image frame from the pixel cell array; receive second programming signals from a host processor, the second programming signals being determined by the host processor based on the first image frame; update the first programming signals based on the second programming signals; and control the pixel cell array to generate a subsequent image frame at a second time based on the updated first programming signals.

[0006] In some aspects, the apparatus further comprises the host processor. The controller is further configured to update the first programming signals at a frame rate at which the pixel cell array generates image frames. The host processor is configured to update the second programming signals at a rate lower than the frame rate.

[0007] In some aspects, the first programming signals comprise a first subset of the first programming signals identifying a first subset of pixel cells of the pixel cell array and a second subset of first programming signals identifying at a second subset of pixel cells of the pixel cell array. The controller is further configured to update the first subset of pixel cells identified by the first subset of the first programming signals, and update the second subset of pixel cells identified by the second subset of the first programming signals.

[0008] In some aspects, the first subset of the first programming signals control the first subset of pixel cells to output pixel data at a first resolution to the host processor. The second subset of the first programming signals control the second subset of pixel cells not to output pixel data, or to output pixel data at a second resolution lower than the first resolution, to the host processor.

[0009] In some aspects, the controller is further configured to: perform, based on the second programming signals, an image processing operation on the first image frame or the second image frame to generate a processing result; and update the first programming signals based on the processing result.

[0010] In some aspects, the image processing operation comprises: determining, for each pixel included in the first image frame or the second image frame, a pixel value difference with respect to a corresponding pixel in a third image frame generated by the pixel cell array prior to the first time; and identifying a subset of pixels in the first image frame or the second image frame for which the pixel value differences exceed a threshold. The controller is further configured to update the first programming signals based on identifying a subset of pixel cells of the pixel cell array, the subset of pixel cells being identified based on the subset of pixels. The second programming signals define the threshold.

[0011] In some aspects, the apparatus further comprises the host processor. The host processor is configured to set the threshold based on determining an ambient light intensity from the first image frame.

[0012] In some aspects, the image processing operation comprises identifying a subset of pixels in the first image frame or in the second image frame that include target features of an object of interest. The controller is further configured to update the first programming signals are updated based on identifying a subset of pixel cells of the pixel cell array, the subset of pixel cells being identified based on the subset of pixels. The second programming signals include information about the target features.

[0013] In some aspects, the second programming signals indicate an initial subset of pixels in the first image frame that include the target features. The image processing operation comprises identifying the subset of pixels based on determining whether the initial subset of pixels in the first image frame or in the second image frame include the target features of the object of interest.

[0014] In some aspects, the controller is further configured to: based on information about a movement of the object of interest and a time difference between the first time and the second time, determine the subset of pixels in the subsequent image frame that include the target features; and update the first programming signals based on determining a subset of pixel cells of the pixel cell array based on the subset of pixels in the subsequent image frame. The second programming signals include the information about the movement of the object of interest.

[0015] In some aspects, the second programming signals indicate that an initial subset of the pixels in the first image frame, defined based on the first programming signals, do not include all the target features. the image processing operation comprises, based on the second programming signals, identifying additional pixels that include the target features.

[0016] In some aspects, the controller includes a neural network model to perform the image processing operation. The second programming signals include at least one of: weights of the neural network model, backward propagation gradients to update the weights, a predicted accuracy of image processing operation, or intermediate outputs from the image processing operation.

[0017] In some aspects, each pixel cell of the pixel cell array or each block of pixel cells of the pixel cell array is individually addressable. The first programming signals comprise pixel-level signals individually targeted at each pixel cell or block-level signals targeted at each block of pixel cells.

[0018] In some aspects, the apparatus further comprises: pixel interconnects implemented by chip-to-chip copper bonding between the first semiconductor substrate and the one or more second semiconductor substrates to transmit signals generated by the photodiodes in the first semiconductor substrate to the processing circuits of pixel cell array in the one or more second semiconductor substrates; and through silicon vias (TSV) between the first semiconductor substrate and the one or more second semiconductor substrates to transmit the image frames from the pixel cell array to the controller, and to transmit the first programming signals from the controller to the pixel cell array.

[0019] In some aspects, the pixel cell array is coupled with the host processor via a point-to-point serial interface and is configured to transmit the image frames to the host processor via the point-to-point serial interface.

[0020] In some aspects, the controller is coupled with the host processor via a shared bus interface, the shared bus interface further coupled with other controllers. The controller is configured to receive the second programming signals from the host processor via the shared bus interface.

[0021] In some examples, a method is provided. The method comprises: controlling a pixel cell array to generate a first image frame at a first time based on first programming signals, and to transmit the first image frame to a host processor; receiving, from the pixel cell array, the first image frame or a second image frame from the pixel cell array, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame;

[0022] receiving second programming signals from the host processor, the second programming signals being determined by the host processor based on the first image frame; updating the first programming signals based on the second programming signals; and controlling the pixel cell array to generate a subsequent image frame at a second time based on the updated first programming signals.

[0023] In some aspects, the method further comprises: performing, based on the second programming signals, an image processing operation on the first image frame or the second image frame to generate a processing result; and updating the first programming signals based on the processing result.

[0024] In some aspects, the image processing operation comprises identifying a subset of pixels in the first image frame or in the second image frame that include target features of an object of interest. The first programming signals are updated based on identifying the subset of pixel cells. The second programming signals include information about the target features.

[0025] In some aspects, the pixel cell array transmits the first image frame to the host processor via a point-to-point serial interface. The second programming signals are received via a shared bus interface.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Illustrative embodiments are described with reference to the following figures.

[0027] FIG. 1A and FIG. 1B are diagrams of an embodiment of a near-eye display.

[0028] FIG. 2 is an embodiment of a cross section of the near-eye display.

[0029] FIG. 3 illustrates an isometric view of an embodiment of a waveguide display with a single source assembly.

[0030] FIG. 4 illustrates a cross section of an embodiment of the waveguide display.

[0031] FIG. 5 is a block diagram of an embodiment of a system including the near-eye display.

[0032] FIG. 6A and FIG. 6B illustrate examples of an image sensor and its operations.

[0033] FIG. 7A, FIG. 7B, FIG. 7C, and FIG. 7D illustrate examples of applications to be supported by imaging systems according to examples of the present disclosure.

[0034] FIG. 8A and FIG. 8B illustrate examples of an imaging system and its operations.

[0035] FIG. 9A, FIG. 9B, FIG. 9C, and FIG. 9D illustrate examples of components of the imaging system of FIG. 8A and FIG. 8B.

[0036] FIG. 10A, FIG. 10B, and FIG. 10C illustrate example physical arrangements of the imaging system of FIG. 8A-FIG. 9D.

[0037] FIG. 11 illustrates a flowchart of an example process for generating image data.

[0038] The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated may be employed without departing from the principles of, or benefits touted in, this disclosure.

[0039] In the appended figures, similar components and/or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.

DETAILED DESCRIPTION

[0040] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain inventive embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0041] An image sensor includes a pixel cell array. Each pixel cell in the pixel cell array includes circuit components to perform a light sensing operation. For example, each pixel cell may include a photodiode to sense incident light by converting photons into charge (e.g., electrons or holes) and a charge sensing unit (e.g., a floating drain and a buffer) to convert the charge into a voltage. The image sensor may also include one or more analog-to-digital converters (ADCs) to quantize the voltage’s output by the charge sensing units of the pixel cells into digital values. The ADC can quantize the charge by, for example, using a comparator to compare a voltage representing the charge with one or more quantization levels, and a digital value can be generated based on the comparison result. The digital values can then be stored in a memory to generate the image. An image sensor typically includes a controller to send out one or more chip-level programming signals to configure the operations of the pixel cells of the image sensor. For example, the controller can turn on or off all the pixel cells of the image sensor, set a global exposure time in which the pixel cells perform light sensing operations, etc.

[0042] The pixel data from an image sensor can support various applications, such as fusion of 2D and 3D sensing, recognition and tracking of objects of interest, location tracking, etc. These applications can extract features of one or more objects from the image, and perform computations based on the extracted features. For example, to perform 3D sensing, an application can identify pixels of reflected structured light (e.g., dots), compare a pattern extracted from the pixels with the transmitted structured light, and perform depth computation based on the comparison. The application can also identify 2D pixel data from the same pixel cells that provide the extracted pattern of structured light to perform fusion of 2D and 3D sensing. To perform object recognition and tracking, an application can also identify pixels of image features of the object, extract the image features from the pixels, and perform the recognition and tracking based on the extraction results. The object recognition and tracking results can support higher level applications, such as a simultaneous localization and mapping (SLAM) application, an eye tracking application, etc. These applications are typically executed on a host processor, which can be electrically connected with the image sensor and receive the pixel data via off-chip interconnects (e.g., Mobile Industry Processor Interface Camera Serial Interface (MIPI CSI) and I3C bus). The host processor, the image sensor, and the interconnects can be part of a system of a mobile device to support a particular application such as, for example, an object tracking application, a location tracking application (e.g., simultaneous localization and mapping (SLAM)), and/or a virtual/mixed/augmented reality application. In some examples, the host processor can also be in a cloud system, in which case the host processor can receive the image data from the image sensor via a wired communication network, a wireless communication network, etc.

[0043] While these host applications can benefit from the image data generated by the pixel cell array, the performance of the overall imaging system, such as power consumption, speed, accuracy, etc., can be limited by various factors. First, typically those applications have no control over the generation of the image data as well as the light sensing operations of these pixel cells. The lack of input from the host applications on the configuration of the pixel cells can impose limits on the achievable performance of the image sensor and these applications. For example, the host applications can benefit from high-resolution images and/or high frame rates. Higher-resolution images allow the application to extract more detailed features/patterns (e.g., more refined patterns of reflected structured light, more detailed image features), whereas providing images generated at a higher frame rate enables an application to track the location of an object, the location of the mobile device, etc., at a higher sampling rate, both processes of which can improve the performances of the applications. However, high-resolution images and high frame rates can lead to generation, transmission, and processing of a large volume of pixel data, which can present numerous challenges. For example, transmitting and processing a large volume of pixel data at a high data rate can lead to high power consumption at the image sensor, the interconnect, and the host processor. Moreover, the image sensor and the host processor may impose bandwidth limitations on and add latency to the generation and processing of large volumes of pixel data. The high power and high bandwidth requirement can be especially problematic for a mobile device which tends to operate with relatively low power and at a relatively low speed due to form factor and safety considerations.

[0044] In addition, typically the image sensor and the host processor are designed and optimized individually according to different specifications. Such arrangements can lead to inefficiency and waste of resources (e.g., power, bandwidth) at the image sensor and at the host processor when they are combined to form the imaging system, which in turn degrades the overall system performance. For example, the image sensor may be configured to generate pixel data from each pixel cell and transmit the pixel data to the host processor in every image frame, but the host processor may not need pixel data from each pixel cell in every image frame to track an object, especially in a case where the object of interest is only a small part of a scene captured in an image frame. On the other hand, the bandwidth of the interconnects, as well as the processing capabilities of the host processor, can impose a limit on the frame rate and the resolutions of the image frames. The limited frame rate can reduce the rate at which the host processor tracks the object, while the limited resolution can reduce the accuracy in distinguishing pixels of the object from the rest of the image frame. Both the limited frame rate and limited resolution can degrade the overall system performance in tracking the object.

[0045] This disclosure relates to an imaging system that can address at least some of the issues above. The imaging system includes an image sensor comprising an pixel cell array and a controller, and a host processor. The pixel cell array and the controller can be formed on the same semiconductor substrate, or formed on two semiconductor substrates forming a stack and housed within a semiconductor package. The pixel cell array and the controller can be communicatively coupled to each other using on-chip interconnects, such as through-silicon vias (TSV), in a case where the image sensor and the controller form a stack. The host processor can be an off-chip device communicatively coupled to the image sensor and the controller based on chip-to-chip interconnects such as MIPI CSI and I3C buses. In some examples, the host processor can be part of a cloud system and can be communicatively coupled with the image sensor and the controller via a wired communication network, a wireless communication network, etc.

[0046] Each pixel cell, or block of pixel cells, of the pixel cell array can be individually programmable to, for example, enable/disable outputting of a pixel value, set a resolution of the pixel value output by the pixel cell, etc. The pixel cell array can receive a first programming signal, which can be in the form of a programming map, from the controller, and generate a first image frame, at a first time, based on sensing light from a scene and based on the first programming signal. Specifically, the pixel cell array can be controlled by the first programming signals to operate in multiple sparsity modes, such as in a full-frame mode in which the first image frame includes a full image frame of pixels, and/or in a sparse mode in which the first image frame only includes a subset of the pixels specified by the programming map. The pixel cell array can output the first image frame to both the host processor and to the controller. In some examples, the pixel cell array can also generate a second image frame at the same first time as the first image frame, and transmit the first image frame to the host processor and the second image frame to the controller. The first image frame can have sparse pixels, whereas the second image frame can have a full frame of pixels.

[0047] The controller and the host processor, together with the image sensor, can form a two-tier feedback system based on the first image frame to control the image sensor to generate a subsequent image frame. In a two-tier feedback operation, the controller can perform an image processing operation on the first image frame or the second image frame to obtain a processing result, and then adjust the first programming signals based on the processing result. The image processing operation at the controller can be guided/configured based on a second programming signals received from the host processor, which can generate the second programming signals based on the first image frame. The pixel cell array then generates a subsequent image frame, at a second time, based on the updated first programming signal.

[0048] In some examples, the image processing operation can include extracting/detecting pixels having certain target spatial and/or temporal features. As an example, the objective of the image processing operation may include identifying pixels that include spatial features, keypoints, etc., of a pre-determined object (e.g., a human face, a body part, or certain physical objects in a scene). As another example, the objective of the image processing operation may include identifying pixels that experience a pre-determined degree of changes in the pixel values between frames (e.g., having a pre-determined temporal contrast), to enable detection/measurement of a motion of an object. The programming map can be generated based on determining which of the pixels in the first frame contain the target feature, to control the image sensors such that only pixel cells corresponding to those pixels will output the pixel data to the host processor and/or to the controller. Meanwhile, the host processor can define, as part of the second programming signal, the target image feature to be detected, an approximate location of the target image feature in the first image frame, etc. The controller can either perform the image feature extraction operation directly based on the second programming signal, or further refine the second programming signals (e.g., to estimate a more precise location of the target image feature) and perform the image feature extraction operation based on the refined second programming signal.

[0049] The controller can employ various techniques to extract/detect target features from an image. For example, the controller may implement a neural network, such as convolution neural network (CNN), to perform arithmetic operations on the pixel data with weights to perform the extraction. The controller may include memory devices (e.g., spin tunneling random access memory (STRAM), non-volatile random access memory (NVRAM)) to store the weights.

[0050] In the aforementioned two-tier feedback system, the host processor can execute an application to perform an analysis of the first image frame to generate the second programming signal. In some examples, the second programming signals can be in the form of a teaching/guidance signal, the result of a neural network training operation (e.g., backward propagation results), a predicted accuracy of image processing operation, or intermediate outputs from the image processing operation, etc., to influence the image processing operation and/or programming map generation at the controller. For example, the host processor and the controller can take part in a training operation, with the host processor performing the training on the lower-level layers based on more generic images, whereas the controller continues the training on the higher-level layers based on images captured by the pixel cell array. As another example, the host processor can provide a predicted accuracy of the image processing operation performed by the neural network as feedback, which allows the neural network to update the weights to improve the predicted accuracy of the image processing operation.

[0051] The host processor can perform the analysis based on not just the first image frame but also other sensor data (e.g., other image frames captured by other image sensors, audio information, motion sensor outputs, inputs from the user) to determine a context of the light sensing operation, and then determine the teaching/guidance signal. The context may include, for example, an environmental condition the image sensor operates in, a location of the image sensor, or any other requirements of the application that consumes the pixel data from the image sensor. The teaching/guidance signals can be updated at a relatively low rate (e.g., lower than the frame rate) based on the context, given that the context typically changes at a much lower rate than the frame rate, while the image processing operation and the updating of the programming map can occur at a relatively high rate (e.g., at the frame rate) to adapt to the images captured by the pixel cell array.

[0052] With the disclosed techniques, a closed loop feedback system can be implemented to dynamically configure the operations at the image sensor, which can further improve the overall performance of the system. Specifically, the controller can program the pixel cell array to perform a sparse capture operation, in which only output a subset of pixels based on performing an image feature operation on the first image frame, which allows the image sensor to output only the pixel data of interest to the host processor. The pixel data of interest include pixel data needed to support an application at the host processor. For example, in a case where the host processor tracks an object, the pixel data of interest include the pixel data of the object being tracked, while pixel data not part of the object are not transmitted to the host processor. Due to the sparsity of the pixel data of interest, such pixel data can be transmitted at a higher speed or generated at a higher precision compared with a case where full frames of pixels are transmitted to the host processor. By providing only pixel data of interest to the host processor at a higher speed and/or at a higher accuracy, the performance and efficiency of the system can be improved. Moreover, as the pixel cell array and the controller are tightly integrated within a single semiconductor package, the feedback system can be operated at a high speed. For example, the controller can update the programming map at the same frame rate as the generation of the image frames by the pixel cell array. As a result, the updating of the first programming signals can be more responsive to the image data captured by the pixel cell array and the change in the operation condition of the image sensor, which can further improve the performance of the system.

[0053] In addition, by having the host processor to send the second programming signals to configure the image processing operations at the controller, an additional closed loop feedback system can be provided to further improve the image processing operations at the controller (and the generation of the first programming signals) based on better knowledge of the operation condition of the image sensor. Specifically, the host processor may execute an application that uses the image data from the image sensor to perform various operations, such as an object tracking operation, a location determination operation, a virtual/mixed/augmented reality operation, etc. The host processor can combine data from different sensors, or otherwise track a context/environment of the image sensor, to determine the most up-to-date context/environment.

[0054] The host processor can then generate, as part of the second programming signal, guidance information based on the most up-to-date context/environment, and provide the guidance information to the controller to influence its image feature extraction operation. For example, the host processor can determine the target object to be tracked, an initial/approximate location of the target object, the target feature (spatial feature, temporal feature, etc.) to be tracked, etc., and provide those information to adapt the image processing operation at the controller to the changes in the context. Such arrangements can improve the likelihood of the controller extracting/detecting the target features from the image data. For example, due to a change in the scene and/or the location of the image sensor, the host processor may request the controller to track a different object, to set a different temporal contrast threshold for selecting pixels that change between frames, etc. This allows the image processing operation to be adaptive to the change in the operation condition of the image sensor, which in turn can improve the likelihood that the controller can identify pixels having the target feature for a given application.

[0055] There are also various advantages of having the host processor to provide guidance information to the controller. Specifically, having the host processor to provide the guidance information can alleviate the controller from the burden of tracking the context of the image sensing operation, as the controller may not have the computation resource to execute the aforementioned applications to track/determine the context, while the controller can receive the context information from the host processor. Moreover, given that the change in the context typically occurs at a much slower pace than the generation of images, the host processor can generate and transmit the guidance information at a lower rate and using a low-bandwidth chip-to-chip interconnect, such as I3C, to further reduce power consumption of the system. All these can further improve the efficiency and overall performance of the system, especially in capturing images of fast-changing scenes to support object tracking applications, SLAM applications, etc.

[0056] The disclosed techniques may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0057] FIG. 1A is a diagram of an embodiment of a near-eye display 100. Near-eye display 100 presents media to a user. Examples of media presented by near-eye display 100 include one or more images, video, and/or audio. In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from the near-eye display 100, a console, or both, and presents audio data based on the audio information. Near-eye display 100 is generally configured to operate as a virtual reality (VR) display. In some embodiments, near-eye display 100 is modified to operate as an augmented reality (AR) display and/or a mixed reality (MR) display.

[0058] Near-eye display 100 includes a frame 105 and a display 110. Frame 105 is coupled to one or more optical elements. Display 110 is configured for the user to see content presented by near-eye display 100. In some embodiments, display 110 comprises a waveguide display assembly for directing light from one or more images to an eye of the user.

[0059] Near-eye display 100 further includes image sensors 120a, 120b, 120c, and 120d. Each of image sensors 120a, 120b, 120c, and 120d may include a pixel array configured to generate image data representing different fields of views along different directions. For example, sensors 120a and 120b may be configured to provide image data representing two fields of view towards a direction A along the Z axis, whereas sensor 120c may be configured to provide image data representing a field of view towards a direction B along the X axis, and sensor 120d may be configured to provide image data representing a field of view towards a direction C along the X axis.

[0060] In some embodiments, sensors 120a-120d can be configured as input devices to control or influence the display content of the near-eye display 100 to provide an interactive VR/AR/MR experience to a user who wears near-eye display 100. For example, sensors 120a-120d can generate physical image data of a physical environment in which the user is located. The physical image data can be provided to a location tracking system to track a location and/or a path of movement of the user in the physical environment. A system can then update the image data provided to display 110 based on, for example, the location and orientation of the user, to provide the interactive experience. In some embodiments, the location tracking system may operate a SLAM algorithm to track a set of objects in the physical environment and within a view of field of the user as the user moves within the physical environment. The location tracking system can construct and update a map of the physical environment based on the set of objects, and track the location of the user within the map. By providing image data corresponding to multiple fields of views, sensors 120a-120d can provide the location tracking system a more holistic view of the physical environment, which can lead to more objects to be included in the construction and updating of the map. With such an arrangement, the accuracy and robustness of tracking a location of the user within the physical environment can be improved.

[0061] In some embodiments, near-eye display 100 may further include one or more active illuminators 130 to project light into the physical environment. The light projected can be associated with different frequency spectrums (e.g., visible light, infra-red light, ultra-violet light), and can serve various purposes. For example, illuminator 130 may project light in a dark environment (or in an environment with low intensity of infra-red light, ultra-violet light, etc.) to assist sensors 120a-120d in capturing images of different objects within the dark environment to, for example, enable location tracking of the user. Illuminator 130 may project certain markers onto the objects within the environment, to assist the location tracking system in identifying the objects for map construction/updating.

[0062] In some embodiments, illuminator 130 may also enable stereoscopic imaging. For example, one or more of sensors 120a or 120b can include both a first pixel array for visible light sensing and a second pixel array for infra-red (IR) light sensing. The first pixel array can be overlaid with a color filter (e.g., a Bayer filter), with each pixel of the first pixel array being configured to measure intensity of light associated with a particular color (e.g., one of red, green or blue colors). The second pixel array (for IR light sensing) can also be overlaid with a filter that allows only IR light through, with each pixel of the second pixel array being configured to measure intensity of IR lights. The pixel arrays can generate an RGB image and an IR image of an object, with each pixel of the IR image being mapped to each pixel of the RGB image. Illuminator 130 may project a set of IR markers on the object, the images of which can be captured by the IR pixel array. Based on a distribution of the IR markers of the object as shown in the image, the system can estimate a distance of different parts of the object from the IR pixel array, and generate a stereoscopic image of the object based on the distances. Based on the stereoscopic image of the object, the system can determine, for example, a relative position of the object with respect to the user, and can update the image data provided to display 100 based on the relative position information to provide the interactive experience.

[0063] As discussed above, near-eye display 100 may be operated in environments associated with a very wide range of light intensities. For example, near-eye display 100 may be operated in an indoor environment or in an outdoor environment, and/or at different times of the day. Near-eye display 100 may also operate with or without active illuminator 130 being turned on. As a result, image sensors 120a-120d may need to have a wide dynamic range to be able to operate properly (e.g., to generate an output that correlates with the intensity of incident light) across a very wide range of light intensities associated with different operating environments for near-eye display 100.

[0064] FIG. 1B is a diagram of another embodiment of near-eye display 100. FIG. 1B illustrates a side of near-eye display 100 that faces the eyeball(s) 135 of the user who wears near-eye display 100. As shown in FIG. 1B, near-eye display 100 may further include a plurality of illuminators 140a, 140b, 140c, 140d, 140e, and 140f. Near-eye display 100 further includes a plurality of image sensors 150a and 150b. Illuminators 140a, 140b, and 140c may emit lights of certain frequency range (e.g., NIR) towards direction D (which is opposite to direction A of FIG. 1A). The emitted light may be associated with a certain pattern, and can be reflected by the left eyeball of the user. Sensor 150a may include a pixel array to receive the reflected light and generate an image of the reflected pattern. Similarly, illuminators 140d, 140e, and 140f may emit NIR lights carrying the pattern. The NIR lights can be reflected by the right eyeball of the user, and may be received by sensor 150b. Sensor 150b may also include a pixel array to generate an image of the reflected pattern. Based on the images of the reflected pattern from sensors 150a and 150b, the system can determine a gaze point of the user, and update the image data provided to display 100 based on the determined gaze point to provide an interactive experience to the user.

[0065] As discussed above, to avoid damaging the eyeballs of the user, illuminators 140a, 140b, 140c, 140d, 140e, and 140f are typically configured to output lights of very low intensities. In a case where image sensors 150a and 150b comprise the same sensor devices as image sensors 120a-120d of FIG. 1A, the image sensors 120a-120d may need to be able to generate an output that correlates with the intensity of incident light when the intensity of the incident light is very low, which may further increase the dynamic range requirement of the image sensors.

[0066] Moreover, the image sensors 120a-120d may need to be able to generate an output at a high speed to track the movements of the eyeballs. For example, a user’s eyeball can perform a very rapid movement (e.g., a saccade movement) in which there can be a quick jump from one eyeball position to another. To track the rapid movement of the user’s eyeball, image sensors 120a-120d need to generate images of the eyeball at high speed. For example, the rate at which the image sensors generate an image frame (the frame rate) needs to at least match the speed of movement of the eyeball. The high frame rate requires short total exposure time for all of the pixel cells involved in generating the image frame, as well as high speed for converting the sensor outputs into digital values for image generation. Moreover, as discussed above, the image sensors also need to be able to operate at an environment with low light intensity.

[0067] FIG. 2 is an embodiment of a cross section 200 of near-eye display 100 illustrated in FIG. 1. Display 110 includes at least one waveguide display assembly 210. An exit pupil 230 is a location where a single eyeball 220 of the user is positioned in an eyebox region when the user wears the near-eye display 100. For purposes of illustration, FIG. 2 shows the cross section 200 associated eyeball 220 and a single waveguide display assembly 210, but a second waveguide display is used for a second eye of a user.

[0068] Waveguide display assembly 210 is configured to direct image light to an eyebox located at exit pupil 230 and to eyeball 220. Waveguide display assembly 210 may be composed of one or more materials (e.g., plastic, glass) with one or more refractive indices. In some embodiments, near-eye display 100 includes one or more optical elements between waveguide display assembly 210 and eyeball 220.

[0069] In some embodiments, waveguide display assembly 210 includes a stack of one or more waveguide displays including, but not restricted to, a stacked waveguide display, a varifocal waveguide display, etc. The stacked waveguide display is a polychromatic display (e.g., a red-green-blue (RGB) display) created by stacking waveguide displays whose respective monochromatic sources are of different colors. The stacked waveguide display is also a polychromatic display that can be projected on multiple planes (e.g., multi-planar colored display). In some configurations, the stacked waveguide display is a monochromatic display that can be projected on multiple planes (e.g., multi-planar monochromatic display). The varifocal waveguide display is a display that can adjust a focal position of image light emitted from the waveguide display. In alternate embodiments, waveguide display assembly 210 may include the stacked waveguide display and the varifocal waveguide display.

[0070] FIG. 3 illustrates an isometric view of an embodiment of a waveguide display 300. In some embodiments, waveguide display 300 is a component (e.g., waveguide display assembly 210) of near-eye display 100. In some embodiments, waveguide display 300 is part of some other near-eye display or other system that directs image light to a particular location.

[0071] Waveguide display 300 includes a source assembly 310, an output waveguide 320, and a controller 330. For purposes of illustration, FIG. 3 shows the waveguide display 300 associated with a single eyeball 220, but in some embodiments, another waveguide display separate, or partially separate, from the waveguide display 300 provides image light to another eye of the user.

[0072] Source assembly 310 generates and outputs image light 355 to a coupling element 350 located on a first side 370-1 of output waveguide 320. Output waveguide 320 is an optical waveguide that outputs expanded image light 340 to an eyeball 220 of a user. Output waveguide 320 receives image light 355 at one or more coupling elements 350 located on the first side 370-1 and guides received input image light 355 to a directing element 360. In some embodiments, coupling element 350 couples the image light 355 from source assembly 310 into output waveguide 320. Coupling element 350 may be, e.g., a diffraction grating, a holographic grating, one or more cascaded reflectors, one or more prismatic surface elements, and/or an array of holographic reflectors.

[0073] Directing element 360 redirects the received input image light 355 to decoupling element 365 such that the received input image light 355 is decoupled out of output waveguide 320 via decoupling element 365. Directing element 360 is part of, or affixed to, first side 370-1 of output waveguide 320. Decoupling element 365 is part of, or affixed to, second side 370-2 of output waveguide 320, such that directing element 360 is opposed to the decoupling element 365. Directing element 360 and/or decoupling element 365 may be, e.g., a diffraction grating, a holographic grating, one or more cascaded reflectors, one or more prismatic surface elements, and/or an array of holographic reflectors.

[0074] Second side 370-2 represents a plane along an x-dimension and a y-dimension. Output waveguide 320 may be composed of one or more materials that facilitate total internal reflection of image light 355. Output waveguide 320 may be composed of e.g., silicon, plastic, glass, and/or polymers. Output waveguide 320 has a relatively small form factor. For example, output waveguide 320 may be approximately 50 mm wide along x-dimension, 30 mm long along y-dimension and 0.5-1 mm thick along a z-dimension.

[0075] Controller 330 controls scanning operations of source assembly 310. The controller 330 determines scanning instructions for the source assembly 310. In some embodiments, the output waveguide 320 outputs expanded image light 340 to the user’s eyeball 220 with a large field of view (FOV). For example, the expanded image light 340 is provided to the user’s eyeball 220 with a diagonal FOV (in x and y) of 60 degrees and/or greater and/or 150 degrees and/or less. The output waveguide 320 is configured to provide an eyebox with a length of 20 mm or greater and/or equal to or less than 50 mm; and/or a width of 10 mm or greater and/or equal to or less than 50 mm.

[0076] Moreover, controller 330 also controls image light 355 generated by source assembly 310, based on image data provided by image sensor 370. Image sensor 370 may be located on first side 370-1 and may include, for example, image sensors 120a-120d of FIG. 1A. Image sensors 120a-120d can be operated to perform 2D sensing and 3D sensing of, for example, an object 372 in front of the user (e.g., facing first side 370-1). For 2D sensing, each pixel cell of image sensors 120a-120d can be operated to generate pixel data representing an intensity of light 374 generated by a light source 376 and reflected off object 372. For 3D sensing, each pixel cell of image sensors 120a-120d can be operated to generate pixel data representing a time-of-flight measurement for light 378 generated by illuminator 325. For example, each pixel cell of image sensors 120a-120d can determine a first time when illuminator 325 is enabled to project light 378 and a second time when the pixel cell detects light 378 reflected off object 372. The difference between the first time and the second time can indicate the time-of-flight of light 378 between image sensors 120a-120d and object 372, and the time-of-flight information can be used to determine a distance between image sensors 120a-120d and object 372. Image sensors 120a-120d can be operated to perform 2D and 3D sensing at different times, and provide the 2D and 3D image data to a remote console 390 that may be (or may not be) located within waveguide display 300. The remote console may combine the 2D and 3D images to, for example, generate a 3D model of the environment in which the user is located, to track a location and/or orientation of the user, etc. The remote console may determine the content of the images to be displayed to the user based on the information derived from the 2D and 3D images. The remote console can transmit instructions to controller 330 related to the determined content. Based on the instructions, controller 330 can control the generation and outputting of image light 355 by source assembly 310, to provide an interactive experience to the user.

……
……
……