Apple Patent | Global-shutter image sensor with time-of-flight sensing capability

Patent: Global-shutter image sensor with time-of-flight sensing capability

Drawings: Click to check drawins

Publication Number: 20210392283

Publication Date: 20211216

Applicant: Apple

Abstract

Apparatus for optical sensing includes first matrix of optical sensing elements, arranged on a semiconductor substrate in rows and columns. A second matrix of storage nodes is arranged on the substrate such that respective first and second storage nodes in the second matrix are disposed in proximity to each of the sensing elements within the first matrix. Switching circuitry couples each of the sensing elements to transfer photocharge to the respective first and second storage nodes. Control circuitry controls the switching circuitry in a depth sensing mode such that over a series of detection cycles, each of the sensing elements and a first neighboring sensing element are connected together to the respective first storage node during the first detection interval, and each of the sensing elements and the second neighboring sensing element are connected together to the respective second storage node during the second detection interval.

Claims

1. Apparatus for optical sensing, comprising: a semiconductor substrate; a first matrix of optical sensing elements, arranged on the substrate in rows and columns such that any given sensing element within the first matrix has first and second neighboring sensing elements on different sides of the given sensing element in the first matrix; a second matrix of storage nodes, arranged on the substrate such that respective first and second storage nodes in the second matrix are disposed in proximity to each of the sensing elements within the first matrix; switching circuitry configured to couple each of the sensing elements to transfer photocharge to the respective first and second storage nodes; control circuitry, which is configured to control the switching circuitry in a depth sensing mode such that over a series of detection cycles comprising at least first and second detection intervals in each cycle, each of the sensing elements and the first neighboring sensing element are connected together to the respective first storage node during the first detection interval, and each of the sensing elements and the second neighboring sensing element are connected together to the respective second storage node during the second detection interval; and readout circuitry, which is configured to output the photocharge from the storage nodes.

2. The apparatus according to claim 1, and comprising: an illumination assembly, which is configured to direct optical radiation toward a target scene while modulating the optical radiation with a carrier wave having a predetermined carrier frequency; and objective optics, which are configured to image the target scene onto the first matrix of optical sensing elements, wherein the control circuitry is configured to synchronize the detection cycles with the carrier frequency.

3. The apparatus according to claim 2, wherein the control circuitry is configured to compare the photocharge output by the readout circuitry from the first and second storage nodes in order to compute depth coordinates of points in the target scene.

4. The apparatus according to claim 3, wherein the control circuitry is configured to vary respective phases of the first and second detection intervals relative to the carrier wave over the series of the detection cycles, and to compute the depth coordinates using the photocharge output over the varying phases.

5. The apparatus according to claim 4, wherein the first and second detection intervals have phases that are 180.degree. apart relative to the carrier wave.

6. The apparatus according to claim 5, wherein the control circuitry is configured to control the switching circuitry so that the first and second detection intervals have phases of 0.degree. and 180.degree. relative to the carrier wave, and so that the storage nodes in at least some of the rows in the matrix collect the photocharge in third and fourth detection intervals having phases of 90.degree. and 270.degree. relative to the carrier wave.

7. The apparatus according to claim 5, wherein the control circuitry is configured to control the switching circuitry so that the first and second detection intervals have phases of 0.degree. and 180.degree. relative to the carrier wave, and so that the storage nodes in at least some of the rows in the matrix collect the photocharge in third, fourth, fifth and sixth detection intervals having phases of 60.degree., 120.degree., 240.degree. and 300.degree. relative to the carrier wave.

8. The apparatus according to claim 5, wherein the phases of the first and second detection intervals are reversed from row to row in successive rows of the second matrix.

9. The apparatus according to claim 1, wherein the control circuitry is further configured to control the switching circuitry in a two-dimensional (2D) sensing mode such that each of the sensing elements is connected individually to a single, respective one of the storage nodes that is in proximity thereto.

10. The apparatus according to claim 9, wherein the switching circuitry comprises shutter gates coupled to apply a global shutter to the sensing elements in the 2D sensing mode.

11. The apparatus according to claim 1, wherein the switching circuitry comprises first and second transfer switches coupled respectively between each of the sensing elements and the respective first and second storage nodes.

12. The apparatus according to claim 1, wherein the optical sensing elements are arranged such that the first and second neighboring sensing elements of any given sensing element within any given row of the first matrix are disposed on opposing sides of the given sensing element along the given row.

13. The apparatus according to claim 12, wherein the storage nodes are interleaved with the sensing elements along the rows of the first matrix.

14. A method for optical sensing, comprising: arranging a first matrix of optical sensing elements on a semiconductor substrate in rows and columns such that any given sensing element within the first matrix has first and second neighboring sensing elements on different sides of the given sensing element in the first matrix; arranging a second matrix of storage nodes on the substrate such that respective first and second storage nodes in the second matrix are disposed in proximity to each of the sensing elements within the first matrix; coupling each of the sensing elements to transfer photocharge via switching circuitry to the respective first and second storage nodes; controlling the switching circuitry in a depth sensing mode such that over a series of detection cycles comprising at least first and second detection intervals in each cycle, each of the sensing elements and the first neighboring sensing element are connected together to the respective first storage node during the first detection interval, and each of the sensing elements and the second neighboring sensing element are connected together to the respective second storage node during the second detection interval; and outputting the photocharge from the storage nodes.

15. The method according to claim 14, and comprising: directing optical radiation toward a target scene while modulating the optical radiation with a carrier wave having a predetermined carrier frequency; imaging the target scene onto the first matrix of optical sensing elements; and synchronizing the detection cycles with the carrier frequency.

16. The method according to claim 15, and comprising comparing the photocharge output by the readout circuitry from the first and second storage nodes in order to compute depth coordinates of points in the target scene.

17. The method according to claim 16, wherein synchronizing the detection cycles comprises varying respective phases of the first and second detection intervals relative to the carrier wave over the series of the detection cycles, and comparing the photocharge comprises computing the depth coordinates using the photocharge output over the varying phases.

18. The method according to claim 14, and comprising controlling the switching circuitry in a two-dimensional (2D) sensing mode such that each of the sensing elements is connected individually to a single, respective one of the storage nodes that is in proximity thereto.

19. The method according to claim 18, wherein controlling switching circuitry comprises applying a global shutter to the sensing elements in the 2D sensing mode.

20. The method according to claim 14, wherein the switching circuitry comprises first and second transfer switches coupled respectively between each of the sensing elements and the respective first and second storage nodes.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application 63/037,626, filed Jun. 11, 2020, which is incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present invention relates generally to imaging, and particularly to optical sensors for both two-dimensional imaging and depth mapping.

BACKGROUND

[0003] Some wearable and portable consumer devices, such as smartphones, augmented reality (AR) devices, virtual reality (VR) devices, and smart glasses, comprise both two-dimensional (2D) image sensors and optical depth mapping systems. The image sensor typically requires high spatial resolution in order to capture high-quality images. At the same time, the physical size of the sensor is limited due to the space requirements of these devices, leading to small pixel size. The optical depth mapping system may be used, for example, in conjunction with 2D imaging in order to perform reliable face recognition.

[0004] Various methods are known in the art for optical depth mapping, i.e., generating a three-dimensional (3D) profile of the surface of an object by processing optical radiation reflected from the object. This sort of 3D profile is also referred to as a 3D map, depth map or depth image, and depth mapping is also referred to as 3D mapping. (In the context of the present description and in the claims, the terms “optical radiation” and “light” are used interchangeably to refer to electromagnetic radiation in any of the visible, infrared, and ultraviolet ranges of the spectrum.)

[0005] Some depth mapping systems operate by measuring the time of flight (TOF) of radiation to and from points in a target scene. In direct TOF (dTOF) systems, a light transmitter, such as a laser or array of lasers, directs short pulses of light toward the scene. A receiver, such as a sensitive, high-speed photodiode (for example, an avalanche photodiode) or an array of such photodiodes, receives the light returned from the scene. Control circuitry measures the time delay between the transmitted and received light pulses at each point in the scene, which is indicative of the distance traveled by the light beam, and hence of the depth of the object at the point, and uses the depth data thus extracted in producing a 3D map of the scene.

[0006] Indirect TOF (iTOF) systems, on the other hand, operate by modulating the amplitude of an outgoing beam of radiation at a certain carrier frequency, and then measuring the phase shift of that carrier wave (at the modulation carrier frequency) in the radiation that is reflected back from the target scene. The phase shift can be measured by imaging the scene onto an optical sensor array, and gating or modulating the integration times of the sensors in the array in synchronization with the modulation of the outgoing beam. The phase shift of the reflected radiation received from each point in the scene is indicative of the distance traveled by the radiation to and from that point, although the measurement may be ambiguous due to range-folding of the phase of the carrier wave over distance.

SUMMARY

[0007] Embodiments of the present invention that are described hereinbelow provide apparatus and methods for optical depth sensing.

[0008] There is therefore provided, in accordance with an embodiment of the invention, apparatus for optical sensing, including a semiconductor substrate and a first matrix of optical sensing elements, arranged on the substrate in rows and columns such that any given sensing element within the first matrix has first and second neighboring sensing elements on different sides of the given sensing element in the first matrix. A second matrix of storage nodes is arranged on the substrate such that respective first and second storage nodes in the second matrix are disposed in proximity to each of the sensing elements within the first matrix. Switching circuitry is configured to couple each of the sensing elements to transfer photocharge to the respective first and second storage nodes. Control circuitry is configured to control the switching circuitry in a depth sensing mode such that over a series of detection cycles including at least first and second detection intervals in each cycle, each of the sensing elements and the first neighboring sensing element are connected together to the respective first storage node during the first detection interval, and each of the sensing elements and the second neighboring sensing element are connected together to the respective second storage node during the second detection interval. Readout circuitry is configured to output the photocharge from the storage nodes.

[0009] In some embodiments, the apparatus includes an illumination assembly, which is configured to direct optical radiation toward a target scene while modulating the optical radiation with a carrier wave having a predetermined carrier frequency, and objective optics, which are configured to image the target scene onto the first matrix of optical sensing elements. The control circuitry is configured to synchronize the detection cycles with the carrier frequency. In some of these embodiments, the control circuitry is configured to compare the photocharge output by the readout circuitry from the first and second storage nodes in order to compute depth coordinates of points in the target scene.

[0010] In some embodiments, the control circuitry is configured to vary respective phases of the first and second detection intervals relative to the carrier wave over the series of the detection cycles, and to compute the depth coordinates using the photocharge output over the varying phases. In one embodiment, the first and second detection intervals have phases that are 180.degree. apart relative to the carrier wave. In a disclosed implementation, the control circuitry is configured to control the switching circuitry so that the first and second detection intervals have phases of 0.degree. and 180.degree. relative to the carrier wave, and so that the storage nodes in at least some of the rows in the matrix collect the photocharge in third and fourth detection intervals having phases of 90.degree. and 270.degree. relative to the carrier wave.

[0011] Alternatively, the control circuitry is configured to control the switching circuitry so that the first and second detection intervals have phases of 0.degree. and 180.degree. relative to the carrier wave, and so that the storage nodes in at least some of the rows in the matrix collect the photocharge in third, fourth, fifth and sixth detection intervals having phases of 60.degree., 120.degree., 240.degree. and 300.degree. relative to the carrier wave.

[0012] Additionally or alternatively, the phases of the first and second detection intervals are reversed from row to row in successive rows of the second matrix.

[0013] In some embodiments, the control circuitry is further configured to control the switching circuitry in a two-dimensional (2D) sensing mode such that each of the sensing elements is connected individually to a single, respective one of the storage nodes that is in proximity thereto. In one embodiment, the switching circuitry includes shutter gates coupled to apply a global shutter to the sensing elements in the 2D sensing mode.

[0014] Additionally or alternatively, the switching circuitry includes first and second transfer switches coupled respectively between each of the sensing elements and the respective first and second storage nodes.

[0015] In some embodiments, the optical sensing elements are arranged such that the first and second neighboring sensing elements of any given sensing element within any given row of the first matrix are disposed on opposing sides of the given sensing element along the given row. In a disclosed embodiment, the storage nodes are interleaved with the sensing elements along the rows of the first matrix.

[0016] There is also provided, in accordance with an embodiment of the invention, a method for optical sensing, which includes arranging a first matrix of optical sensing elements on a semiconductor substrate in rows and columns such that any given sensing element within the first matrix has first and second neighboring sensing elements on different sides of the given sensing element in the first matrix. A second matrix of storage nodes is arranged on the substrate such that respective first and second storage nodes in the second matrix are disposed in proximity to each of the sensing elements within the first matrix. Each of the sensing elements is coupled to transfer photocharge via switching circuitry to the respective first and second storage nodes. The switching circuitry is controlled in a depth sensing mode such that over a series of detection cycles including at least first and second detection intervals in each cycle, each of the sensing elements and the first neighboring sensing element are connected together to the respective first storage node during the first detection interval, and each of the sensing elements and the second neighboring sensing element are connected together to the respective second storage node during the second detection interval. The photocharge is outputted from the storage nodes.

[0017] The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG. 1 is a block diagram that schematically illustrates a combined depth mapping and 2D imaging apparatus, in accordance with an embodiment of the invention;

[0019] FIG. 2 is a schematic circuit diagram of an array of optical sensing elements and associated circuitry, in accordance with an embodiment of the invention;

[0020] FIGS. 3a and 3b are schematic circuit diagrams of an array of optical sensing elements, showing the operation of switching circuitry in the array in an iTOF depth sensing mode, in accordance with an embodiment of the invention;

[0021] FIGS. 4a and 4b are schematic frontal views of an image sensor showing phases of detection of storage nodes in the image sensor in an iTOF depth sensing mode, in accordance with an embodiment of the invention;

[0022] FIG. 5 is a schematic frontal view of an image sensor showing phases of detection of storage nodes in the image sensor in an iTOF depth sensing mode, in accordance with another embodiment of the invention;

[0023] FIG. 6 is a schematic frontal view of an image sensor showing phases of detection of storage nodes in the image sensor in an iTOF depth sensing mode, in accordance with yet another embodiment of the invention;

[0024] FIG. 7 is a schematic frontal view of an image sensor showing phases of detection of storage nodes in the image sensor in an iTOF depth sensing mode, in accordance with still another embodiment of the invention;

[0025] FIG. 8 is a schematic circuit diagram of an array of optical sensing elements, showing the operation of switching circuitry in the array in a 2D imaging mode, in accordance with an embodiment of the invention; and

[0026] FIG. 9 is a schematic circuit diagram showing details of sensing elements and associated circuitry in an image sensor, in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

[0027] Optical indirect TOF (iTOF) systems that are known in the art use multiple different acquisition phases in the receiver in order to measure the phase shift of the carrier wave in the light that is reflected from each point in the target scene. For this purpose, many iTOF systems use special-purpose image sensing arrays, in which each sensing element is gated individually to receive and integrate light during a respective phase of the cycle of the carrier wave. At least three different gating phases are needed in order to measure the phase shift of the carrier wave in the received light relative to the transmitted beam. For practical reasons, most systems acquire light during four distinct gating phases.

[0028] In a typical image sensing array of this sort, the optical sensing elements are arranged in groups of four sensing elements. Each optical sensing element in a given group integrates, in one or more respective storage nodes, photocharge that is generated due to light received over one or more different, respective detection intervals, which are synchronized at different phase angles relative to the carrier frequency, for example at 0.degree., 90.degree., 180.degree. and 270.degree.. A processing circuit combines the respective signals from the group of sensing elements in the four detection intervals (referred to as I.sub.0, I.sub.90, I.sub.180 and I.sub.270, respectively) to extract a depth value, which is proportional to the function tan.sup.-1[(I.sub.270-I.sub.90)/(I.sub.0-I.sub.180)]. The constant of proportionality and maximal depth range depend on the choice of carrier wave frequency. Alternatively, other combinations of phase angles may be used for this purpose, for example six phases that are sixty degrees apart (0.degree., 60.degree., 120.degree., 180.degree., 240.degree. and 300.degree.), with corresponding adjustment of the TOF computation.

[0029] Other iTOF systems use smaller groups of sensing elements, for example pairs of sensing elements that integrate photocharge in respective storage nodes due to received light in phases 180.degree. apart, or even arrays of sensing elements that all share the same detection interval or intervals. In such cases, the synchronization of the detection intervals of the entire array of sensing elements is shifted relative to the carrier wave of the transmitted beam over successive image frames in order to acquire sufficient information to measure the phase shift of the carrier wave in the received light relative to the transmitted beam. The processing circuit then combines the pixel values over two or more successive image frames in order to compute the depth coordinate for each point in the scene.

[0030] In order to maximize the signal/noise ratio (SNR) of the iTOF image, it is desirable that each optical sensing element in the image sensing array have two storage nodes. Switching circuitry couples the photocharge from the sensing element to each of the storage nodes in turn during respective detection intervals in each carrier cycle. In this manner, for example, each sensing element will generate and store, in each carrier cycle, the I.sub.0 intensity component in its first storage node and the I.sub.180 intensity component in its second storage node (or the I.sub.90 and I.sub.270 components). This same image sensing array can also be used to capture 2D images, typically under flood illumination, in a 2D sensing mode, in which photocharge is stored in only one of the storage nodes (or in both storage nodes together, rather than in opposite phases). Alternatively, a 2D image can be obtained by summing signals from both storage nodes obtained during 3D mapping. The need to accommodate dual storage nodes and switching circuitry in each cell of the image sensor, however, increases the pitch of the sensing elements in the array, and therefore limits the spatial resolution of the 2D images, relative to images that can be captured by a conventional image sensor with only one storage node per cell.

[0031] Embodiments of the present invention that are described herein address this problem by providing an image sensing array with storage nodes that are shared between neighboring sensing elements. In iTOF mode, switching circuitry in each cell of the array transfers the photocharge to the two neighboring storage nodes during different, respective detection intervals in respective phases of each carrier cycle. The switching circuitry is operated so that each storage node receives and stores photocharge generated by the two neighboring sensing elements during the same detection interval. On the other hand, in 2D imaging mode, each sensing element is connected individually to a single, respective storage node. Because the array contains the same number of storage nodes as sensing elements, the pitch of the sensing elements–and hence the resolution of the 2D images–is comparable to that of conventional, high-resolution image sensors. At the same time, the novel mode of sharing of neighboring storage nodes enables the array to capture 3D iTOF data with SNR and resolution comparable to iTOF sensors having two storage nodes per cell.

[0032] The disclosed embodiments provide apparatus for optical sensing in which a matrix of optical sensing elements and a matrix of charge storage nodes are arranged on a semiconductor substrate such that each storage node is in proximity to two neighboring sensing elements. Thus, each sensing element within the matrix is in proximity to two neighboring storage nodes on different sides of the sensing element. In the embodiments that are described below and shown in the figures, the storage nodes are interleaved along the rows of the matrix of sensing elements. The term “interleaving” is used in this context to mean that any given sensing element within any given row of the matrix shares a first storage node with a first neighboring sensing element on one side of the given sensing element along the row, while sharing a second storage node with a second neighboring sensing element on the opposite side along the row (with the possible exception of the sensing elements at the ends of the row).

[0033] In alternative embodiments, however, the sensing elements and storage nodes may be arranged in different relations. For example, a given sensing element may share at least one of its two neighboring storage nodes with a neighboring sensing element in another row of the matrix (such as an adjacent sensing element along the same column of the matrix). Additionally or alternatively, the matrix of storage nodes may be positioned below the matrix of sensing elements, for example using chip-stacking techniques. All such alternative arrangements of sensing elements and neighboring storage nodes are considered to be within the scope of the present invention, regardless of the specific topology that is used.

[0034] Switching circuitry connects each sensing element to transfer photocharge to the two neighboring storage nodes, such that the sensing element shares these storage nodes with its two neighboring sensing elements. When operating the array in iTOF depth sensing mode, control circuitry controls the switching circuitry to switch between the neighboring storage nodes during respective detection intervals over a series of detection cycles (which are typically synchronized with the carrier frequency of illumination of the target scene). In a first detection interval in each cycle, each sensing element is connected together with one of its two neighboring sensing elements to one of the neighboring storage nodes. Then, in the second detection interval in the cycle, each sensing element is connected together with its other neighboring sensing element to the other neighboring storage node. Readout circuitry outputs the photocharge from the storage nodes in the array.

[0035] As noted earlier and will be further described hereinbelow, the present apparatus can be used for both depth mapping and for high-resolution 2D imaging, by appropriate control of the switching circuitry. The switching circuitry may also comprise shutter gates coupled to apply a global shutter to the sensing elements in the 2D sensing mode. In some embodiments, the apparatus further comprises an illumination assembly directing optical radiation toward a target scene, and objective optics, which form an image of the target scene onto the matrix of the optical sensing elements. Alternatively, however, the novel switching and control circuitry in the sensing array may be used in other modes of operation, with or without a dedicated illumination assembly.

[0036] In the disclosed embodiments, the apparatus performs depth mapping using the method of iTOF. The illumination assembly modulates the optical radiation directed toward the target scene with a carrier wave having a predetermined carrier frequency. Each sensing element samples, under the control of the control circuitry, the image of the target scene in two discrete detection intervals during each cycle of the carrier wave. Numbering the storage nodes in each row (for the sake of convenience) as “even” and “odd”, the control circuitry controls the switches associated with each sensing element in synchronization with the carrier wave in such a manner that, during the first detection interval, the photocharge generated by their adjacent sensing elements is transferred to the even storage nodes, and, during the second detection interval, the photocharge generated by their adjacent sensing elements is transferred to the odd storage nodes.

[0037] For example, using the above-described scheme of four detection intervals, during a first cycle (or sequence of cycles) of the carrier wave, the even storage nodes in a row accumulate photocharge from the detection interval at 0.degree. phase relative to the carrier wave, and the odd storage nodes accumulate photocharge from the detection interval at 180.degree.. During the next cycle (or sequence of cycles), the storage nodes accumulate photocharge from the respective detection intervals at 90.degree. and 270.degree.. Alternatively, the alternating storage nodes in even-numbered rows of the matrix may accumulate photocharge in detection intervals at 0.degree. and 180.degree., while those in the odd-numbered rows accumulate photocharge in detection intervals at 90.degree. and 270.degree..

[0038] Following each series of detection cycles, the photocharges are output from the storage nodes by the readout circuitry. Since the photocharges from two neighboring sensing elements are accumulated in the same storage node, each pixel will comprise these two sensing elements. (The term “pixels” is used in the present description and in the claims to refer to the image data that are read out from the storage nodes, in response to the accumulated charge.) Thus, for a matrix of sensing elements with N columns and M rows, the spatial resolution of the depth map in pixels will be (N/2).times.M, or (N/2).times.(M/2) if alternating rows are used to acquire alternating detection phases, as noted above.

[0039] In the 2D sensing mode, according to the disclosed embodiments, the switches connect each sensing element to a single, respective storage node. For example, each sensing element may be connected to a storage node shared to the left of the sensing element. Thus, for the above N.times.M matrix, the spatial resolution in pixels will be the full N.times.M resolution of the matrix. Due to the sharing of storage nodes between sensing elements, the pitch of the sensing elements can be as small as 2 .mu.m using current fabrication technology. This fine pitch enables high-resolution 2D imaging, while simultaneously providing a sensor for iTOF.

System Description

[0040] FIG. 1 is a block diagram that schematically illustrates a combined depth mapping and 2D imaging apparatus 20, in accordance with an embodiment of the invention. Apparatus 20 comprises an illumination assembly 24 and a detection assembly 26, under control of control circuitry 22. In the pictured embodiment, the illumination and detection assemblies are boresighted, and thus share the same optical axis outside apparatus 20, without parallax; but alternatively, other optical configurations may be used.

[0041] Illumination assembly 24 comprises a beam source 30, for example a suitable semiconductor emitter, such as a semiconductor laser or high-intensity light-emitting diode (LED), or an array of such emitters, which emits optical radiation toward a target scene 28 (in this case containing a human subject). Typically, beam source 30 emits infrared radiation, but alternatively, radiation in other parts of the optical spectrum may be used. The radiation may be collimated by projection optics 34.

[0042] For depth mapping, a synchronization circuit 44 in control circuitry 22 modulates the amplitude of the radiation that is output by source 30 with a carrier wave having a specified carrier frequency. For example, the carrier frequency may be 100 MHz, meaning that the carrier wavelength (when applied to the radiation output by beam source 30) is about 3 m, which also determines the effective range of apparatus 20. (Beyond this effective range, i.e., 1.5 m in the present example, depth measurements may be ambiguous due to range folding.) Alternatively, higher or lower carrier frequencies may be used, depending, inter alia, on considerations of the required range, precision, and signal/noise ratio.

[0043] Detection assembly 26 receives the optical radiation that is reflected from target scene 28 via objective optics 35. The objective optics form an image of the target scene on a matrix 36 of optical sensing elements 40, comprising photodiodes or phototransistors, for example, in an iTOF image sensor 37. Sensing elements 40, formed on a semiconductor substrate 41, such as a silicon chip, are connected to a corresponding matrix 38 of charge storage and readout circuits 42 on the semiconductor substrate. Circuits 42 comprise storage nodes and switching circuitry, as shown in the figures that follow, which gates the detection intervals during which the sensing elements integrate the optical radiation that is focused onto matrix 36. Circuits 42 also comprise switches for reading the charge out of the storage nodes onto the column lines of image sensor 37 (as shown in FIG. 9).

[0044] Typically, although not necessarily, image sensor 37 comprises a single integrated circuit device, in which sensing elements 40 and circuits 42 are integrated. Alternatively, sensing elements 40 may be connected to circuits 42 using other technologies, such as chip stacking, for example, and may comprise either silicon or other materials, such as III-V semiconductor materials.

[0045] Synchronization circuit 44 controls charge storage and readout circuits 42 so that the photocharge generated by sensing elements 40 in response to the optical radiation that is incident on image sensor 37 is stored in different storage nodes during different detection intervals, which are synchronized with the carrier frequency that is applied to beam source 30. For example, circuits 42 comprise charge storage nodes and switches that are controlled to select different storage nodes during different detection intervals, which are synchronized with the carrier frequency at different, respective temporal phase angles, as illustrated further in the figures that follow.

[0046] Objective optics 35 form an image of target scene 28 on matrix 36 such that each point in the target scene is imaged onto a corresponding sensing element 40. To compute the depth coordinates of each point, control circuitry 22 combines the signals output by the sensing elements, as gated by circuits 42, as described further hereinbelow. Control circuitry 22 may then output a depth map 46 made up of these depth coordinates, and possibly a two-dimensional image of the scene, as well.

[0047] For 2D imaging, the amplitude of the optical radiation emitted by beam source 30 toward a target scene 28 may be either modulated by synchronization circuit 44 or emitted as unmodulated radiation. As will be detailed further hereinbelow, for 2D imaging sensing, circuits 42 are not required to be synchronized with the emitted radiation, but are rather configured to collect a sufficient amount of radiation for generating a 2D image. In this case, synchronization circuit 44 may drive charge storage and readout circuits 42 to apply a global shutter so as to control the exposure level of the entire image sensor.

[0048] Control circuitry 22 typically comprises a general-or special-purpose microprocessor or digital signal processor, which is programmed in software or firmware to carry out the functions that are described herein. The control circuitry also includes suitable digital and analog peripheral circuits and interfaces, including synchronization circuit 44, for outputting control signals to and receiving inputs from the other elements of apparatus 20. The detailed design of such circuits will be apparent to those skilled in the art of depth mapping devices after reading the present description.

Sensor Design and Operation

[0049] FIG. 2 is a schematic circuit diagram showing an array of optical sensing elements and associated circuitry, in accordance with an embodiment of the invention. The sensing elements in this embodiment are assumed to be photodiodes, and are accordingly labeled PD. In this and subsequent figures, for the sake of clarity of illustration the sensing elements are shown as being interleaved with storage nodes along the rows of the array; but other topological arrangements may alternatively be used, as noted above.

[0050] FIG. 2 shows three adjacent sensing elements PD.sub.i-1, PD.sub.i, and PD.sub.i+1 in a row of matrix 36. Each sensing element is coupled to a pair of neighboring storage nodes SN by respective transfer switches TX. Each sensing element is also connected to a common drain 66 by a switch referred to as a shutter gate SG. For the sake of simplicity, readout circuitry for reading charges out of storage nodes SN is omitted here, but is shown in FIG. 9, below. All of switches TX and SG together form switching circuitry 45, which controlled by control circuitry 22.

[0051] The matrix of storage nodes SN is interleaved with the sensing elements PD along the rows of matrix 36, so that each sensing element PD within any given row is in proximity to two neighboring storage nodes SN on its opposing sides along the row. (The term “within any given row” may exclude the sensing elements at the ends of the row, depending on the detailed structure of the image sensor.) Each neighboring pair of sensing elements PD thus shares a storage node SN between them. For example, sensing element PD.sub.i shares with sensing element PD.sub.i-1 a charge storage node SN.sub.i-1, and with sensing element PD.sub.i+1 a charge storage node SN.sub.i. (The terms “charge storage node” and “storage node” are used interchangeably.)

[0052] Each sensing element PD.sub.i-1, PD.sub.i, and PD.sub.i+1 is connected to two switches TX.sub.X.sup.Y for connecting each sensing element independently to two each of its neighboring storage nodes SN. In the notation TX.sub.X.sup.Y, index X refers to the storage node SN to which the switch connects, and Y is either L or R, corresponding to whether the switch is to the left or the right of the storage node, with “left” and “right” chosen for convenience within the view of FIG. 2. For example, switch TX.sub.i-1.sup.R connects sensing element PD.sub.i to storage node SN.sub.i-1, whereas switch TX.sub.i.sup.L connects the same sensing element to storage node SN.sub.i.

[0053] As a further example, the configuration of switches TX.sub.X.sup.Y for coupling sensing elements PD.sub.i-1, PD.sub.i, and PD.sub.i+1 to storage nodes SN.sub.1-1 and SN.sub.i is shown in Table 1:

[0054] Additional switches couple sensing elements PD.sub.i-1 and PD.sub.i+1 to storage nodes shared with sensing elements outside FIG. 2, but for the sake of simplicity, these switches are not labelled in this figure, nor shown in Table 1.

[0055] As noted above, each sensing element PD is also connected by a respective shutter gate SG to common drain 66, for draining any remaining photocharge from the sensing element prior to acquiring a new image. For the sake of simplicity, only shutter gate SG.sub.i (for sensing element PD.sub.i) and common drain 66 are labelled in FIG. 2.

[0056] FIGS. 3a and 3b are schematic circuit diagrams showing the operation of switching circuitry 45 in an iTOF depth sensing mode, in accordance with an embodiment of the invention. The figures show the settings of switches TX in two different detection intervals, synchronized to respective phase angles of 0.degree. and 180.degree. of the carrier frequency applied to illumination assembly 24.

[0057] Each of FIGS. 3a and 3b shows four sensing elements PD of matrix 36, labelled PD.sub.i-1, PD.sub.i, PD.sub.i+1, and PD.sub.i+2. All sensing elements PD, switches TX, and drain switches SG are labelled using the same numbering scheme as in FIG. 2. Optical radiation impinging on sensing elements PD.sub.i-1, PD.sub.i, PD.sub.i+1, and PD.sub.i+2 is indicated by double wavy arrows 70.sub.i-1, 70.sub.i, 70.sub.i+1, and 70.sub.i+2, respectively.

[0058] Before each exposure, all shutter gates SG are closed so as to drain from sensing elements PD into common drain any remaining charge from previous exposures. The shutter gates are then opened before the exposure, as shown in the figures.

[0059] At the detection interval synchronized to phase angle of 0.degree. (FIG. 3a), control circuitry 22 closes switches TX.sub.i-1.sup.L, TX.sub.i+1.sup.R, TX.sub.i+1.sup.L, and TX.sub.i+1.sup.R (with the other switches left open). Closing these switches transfers the photocharges generated by sensing elements PD.sub.i-1 and PD.sub.i to storage node SN.sub.i-1, as shown by arrows 72.sub.i-1.sup.L and 72.sub.i-1.sup.R, and the photocharges generated by sensing elements PD.sub.i+1 and PD.sub.i+2 to storage node SN.sub.i+1, as shown by arrows 72.sub.i+1.sup.L and 72.sub.i+1.sup.R. Consequently, sensing elements PD.sub.i-1 and PD.sub.i together define one pixel during this detection interval, and sensing elements PD.sub.i+1 and PD.sub.i+2 define a neighboring pixel.

[0060] At the detection interval synchronized to phase angle of 180.degree. (FIG. 3b), control circuitry 22 closes switches TX.sub.i-2.sup.R, TX.sub.i.sup.L, TX.sub.i.sup.R, and TX.sub.i+2.sup.L (with the other switches left open). Closing these switches transfers the photocharges generated by sensing elements PD.sub.i and PD.sub.i+1 to storage node SN.sub.i, as shown by arrows 72.sub.i.sup.L and 72.sub.i.sup.R. The photocharges from sensing elements PD.sub.i-1 and PD.sub.i+2 are transferred to respective storage nodes SN.sub.i-2 and SN.sub.i+2, as shown by arrows 72.sub.i-2.sup.R and 72.sub.i+2.sup.L, along with the photocharges from sensing elements PD.sub.i-2 and PD.sub.i+3 (not shown in FIG. 3b). The only full pixel (pixel receiving photocharge from two sensing elements) shown in FIG. 3b is defined by sensing elements PD.sub.i and PD.sub.i+1, whereas sensing elements PD.sub.i-1 and PD.sub.i+2 each define pixels with, respectively, sensing elements PD.sub.i-2 and PD.sub.i+3.

[0061] Control circuitry 22 controls switching circuitry 45 in this manner to alternate between the configurations of FIGS. 3a and 3b over a series of detection cycles. Thus, each of the sensing elements PD and one of its neighboring sensing elements (to its left or right) are connected together to one of the neighboring storage nodes SN during the detection interval illustrated in FIG. 3a; and each of the sensing elements and its other neighboring sensing element are connected together to the other neighboring storage node during the detection interval illustrated in FIG. 3b. After having stored the photocharges for the detection intervals at 0.degree. and 180.degree. to the respective storage nodes, typically (although not necessarily) over multiple cycles, the readout circuitry reads the photocharges out as signals I.sub.0 and I.sub.180.

[0062] Comparing FIGS. 3a and 3b, it is seen that the pixels acquired in the detection interval at 180.degree. are shifted by one sensing element PD from the pixels acquired at 0.degree. in each row.

[0063] In some embodiments, the photocharges in the detection intervals at 90.degree. and 270.degree. are collected in each row in similar fashion over a subsequent series of detection cycles, with appropriate adjustment of the phases of switching circuitry 45. Alternatively, the alternating rows of matrix 36 may be driven 90.degree. out of phase, so that one row collects the photocharges in the detection intervals at 0.degree. and 180.degree., while the next row collects the photocharges in the detection intervals at 90.degree. and 270.degree.. In either case, referring to FIGS. 3a-3b, the photocharges collected at 90.degree. are stored, similarly to those collected at 0.degree., in storage nodes SN.sub.i-1 and SN.sub.i+1, and the photocharges collected at 270.degree. are stored, similarly to those collected at 180.degree., in storage nodes SN.sub.i-2, SN.sub.i, and SN.sub.i+2.

[0064] After having stored the respective photocharges for the detection intervals at 90.degree. and 270.degree., the readout circuitry reads the photocharges out as signals I.sub.90 and I.sub.270. These signals, together with the signals I.sub.0 and I.sub.180 are used by control circuitry 22 to calculate a depth value for each pixel, which is proportional to the function tan.sup.-1[(I.sub.270-I.sub.90)/(I.sub.0-I.sub.180)] for that pixel.

[0065] FIGS. 4a and 4b are schematic frontal views of image sensor 37 showing phases of detection of the storage nodes in iTOF depth sensing mode, in accordance with an embodiment of the invention. Image sensor 37 is represented schematically as a matrix of cells 80, each of which comprises a respective sensing element PD, with connections to the neighboring storage nodes as illustrated in the preceding figures. Although the pictured matrix comprises only several hundred sensing elements, in practice the matrix is typically much larger, for example 1000.times.1000 sensing elements. The insets in FIGS. 4a and 4b show an enlarged view of an 8.times.8 group 82 of cells 80.

[0066] FIG. 4a illustrates the spatial distribution of pixels and 86 associated respectively with the two signal values I.sub.0 and I.sub.180. As shown in FIG. 3a, above, each of these signals is produced in a pixel comprising two adjacent sensing elements PD in response to the generated photocharges, with a one-element overlap between adjacent pixels at 0.degree. and 180.degree.. Thus, pixel 84 produces signal I.sub.0, and pixel 86 produces signal I.sub.180. In order to show the overlap conveniently, pixels 84 and 86 are drawn in successive rows.

[0067] Similarly, FIG. 4b illustrates the spatial distribution of pixels 88 and 90 producing the two signal values I.sub.90 and I.sub.270.

[0068] The sampling scheme demonstrated in FIGS. 4a and 4b shows that the pixels have a width of two cells 80 and a height of one cell. Thus, for an image sensor 37 with N columns and M rows of sensing elements 40, the spatial resolution in pixels will be (N/2).times.M. After collecting the photocharges at 0.degree. and 180.degree. (FIG. 4a), storage nodes SN need to be read out before collecting the photocharges at 90.degree. and 270.degree. (FIG. 4b). Thus, the photocharges at all four phases (0.degree., 90.degree., 180.degree., and 270.degree.) are collected over (at least) two image frames. Alternatively, if the alternating rows of matrix 36 are driven 90.degree. out of phase, a complete iTOF image, including all four phases, can be collected in a single image frame, with spatial resolution of (N/2).times.(M/2) pixels.

[0069] FIG. 5 is a schematic frontal view of image sensor 37 in an iTOF depth sensing mode, in accordance with another embodiment of the invention. Insets 90a and 90b show two enlarged views of a 4.times.4 group 92 of cells 80. Similarly to FIGS. 4a and 4b, the pixels producing the signal values I.sub.0 and I.sub.180 are shown in inset 92a, and the pixels producing the signal values I.sub.90 and I.sub.270 are shown in inset 92b. The layout of the pixels differs from those in FIGS. 4a and 4b in that the polarity, i.e., the horizontal order, of the phases of the detection intervals of the pixels is reversed from row to row. Control circuitry 22 can implement the polarization reversal by varying the control signals applied to switching circuitry 45 in successive rows. Such polarization reversal may be utilized beneficially to reduce artifacts in the resulting depth map.

[0070] FIG. 6 is a schematic frontal view of image sensor 37 in an iTOF depth sensing mode, in accordance with yet another embodiment of the invention. As in FIGS. 4a-4b and FIG. 5, image sensor 37 is represented schematically as a matrix of cells 80. Insets 96a, 96b, and 96c show three enlarged views of a 4.times.4 group 96 of cells 80. In this embodiment, the carrier wave is sampled at six detection intervals at 0.degree., 60.degree., 120.degree., 180.degree., and 240.degree. in three pairs, 0.degree./180.degree., 60.degree./240.degree., and 120.degree./300.degree., with the two members of each pair always 180.degree. degrees apart. The carrier wave is first sampled at 0.degree. and 180, as shown in inset 96a. After the stored photocharges are read out from respective storage nodes SN, the carrier wave is sampled at 60.degree. and 240.degree., as shown in inset 96b, followed again by readout of charges. Finally, the carrier wave is sampled at 120.degree. and 300.degree., followed by readout of charges.

[0071] Thus, for reading out all the photocharges for all six detection intervals, three image frames are required. Although a full cycle of data acquisition is thus longer than, for example, a four-interval scheme, the additional data may improve the robustness of the depth mapping, while providing the same (N/2).times.M spatial resolution.

[0072] FIG. 7 is a schematic frontal view of image sensor 37 in an iTOF depth sensing mode, in accordance with still another embodiment of the invention. The inset in FIG. 7 shows an enlarged view of a group 100 of 6.times.8 cells 80. In this embodiment, the carrier wave is sampled at six detection intervals at 0.degree., 60.degree., 120.degree., 180.degree., and 240.degree., as in FIG. 6. In this case, all detection intervals are sampled during a single image frame by a suitable timing of switching circuitry 45 in each set 102 of three rows, thus enabling a three-fold increase in the sampling speed as compared to FIG. 6. However, due to the use of three rows for different detection intervals, the spatial resolution in pixels will be (N/2).times.(M/3).

[0073] In all of FIGS. 4a-4b, 5, 6, and 7, image sensor 37 may be physically substantially the same sensor, with both the temporal and spatial sampling schemes controlled by control circuitry 22. In alternative embodiments, a polarization reversal of alternating rows similar to that shown in FIG. 5 may be implemented in the schemes shown in FIGS. 6 and 7, as well. Alternatively, switching circuitry 45 may be configured and controlled to enable spatial and temporal sampling schemes, as long as at least three detection intervals are sampled relative to the cycles of the carrier wave.

[0074] FIG. 8 is a schematic circuit diagram showing the operation of switching circuitry 45 in a 2D imaging mode, in accordance with an embodiment of the invention. As in the preceding figures, FIG. 8 shows three sensing elements PD.sub.i-1, PD.sub.i, and PD.sub.i+1 in a row of matrix 36. For 2D imaging, one switch TX of each sensing element PD is closed such that each sensing elements is connected individually to a single, respective neighboring storage node SN. Thus, the photocharge generated by each sensing element in response to incident radiation, shown by respective double wavy arrows 110, is collected in a storage node SN to the left of the sensing element. For example, for sensing element PD.sub.i, switch TX.sub.i-1.sup.R is closed, and the photocharge generated by sensing element PD.sub.i in response to incident radiation 110.sub.i is collected in storage node SN.sub.i-1, as shown by an arrow 112.sub.i-1. Similarly, the photocharge generated by sensing elements PD.sub.i-1 and PD.sub.i+1 is stored in storage nodes SN.sub.i-2 and SN.sub.i, respectively.

[0075] Thus, each storage node SN receives photocharge from only one sensing element PD, and each sensing element PD and its neighboring storage node define a corresponding pixel of the 2D image, with an image resolution of N.times.M. The choice of storing the generated photocharge to a storage node to the left of the sensing element is arbitrary, and the storage node to the right of each sensing element could as well have been chosen.

[0076] FIG. 9 is a schematic circuit diagram showing an example implementation of sensing elements PD and associated circuitry 120 in image sensor 37, in accordance with an embodiment of the invention. In this example, sensing elements PD are implemented as fully depleted pinned photodiodes, providing full charge transfer to selected storage nodes SN.sub.i when the appropriate transfer switches TX.sub.i.sup.L or TX.sub.i.sup.R are activated. (Storage nodes SN.sub.i and transfer switches TX.sub.i.sup.L or TX.sub.i.sup.R are selected as previously described in the context of FIGS. 3a-3b and 8.) Similarly, each storage node SN.sub.i is implemented in a way that provides full charge transfer to the respective floating diffusion FD.sub.i when transfer gate TRF.sub.i to the floating diffusion is activated. Circuitry 120 comprises sensing elements PD.sub.i and PD.sub.i+1, along with switching circuitry 45 and readout circuitry 122 (labeled as readout circuitry 122.sub.i and 122.sub.i+1 for storage nodes SN.sub.i and SN.sub.i+1, respectively). The same notation and labelling are used as in FIGS. 2, 3a-3b, and 8.

[0077] Readout circuitry 122.sub.i (and similarly readout circuitry 122.sub.i+1) comprises a floating diffusion node FD.sub.i, which is connected to storage node SN.sub.i through a transfer gate TRF.sub.i. Floating diffusion node FD.sub.i is also connected via a reset gate RST.sub.i to a reset voltage V.sub.pix and to the gate of a source follower SF.sub.i. The source and drain of source follower SF.sub.i are connected between a baseline voltage V.sub.DD and a column line 124.sub.i via a row select gate RS.sub.i. Column line 124.sub.i is connected to output stored charges (or digital values of stored charge) to control circuitry 22.

[0078] Photocharges generated by sensing elements PD.sub.i and PD.sub.i+1 are collected to one or both of storage nodes SN.sub.i and SN.sub.i+1, as shown in FIGS. 3a-3b and FIG. 8. From storage node SN.sub.i (and similarly from storage node SN.sub.i+1), the photocharge is transferred to floating diffusion node FD.sub.i by actuating transfer gate TRF.sub.i, which generates a voltage at the gate of source follower SF.sub.i. Prior to transferring the collected photocharge from storage node SN.sub.i to the corresponding floating diffusion node FD.sub.i, control circuitry 22 actuates reset gate RST.sub.i in order to reset the floating diffusion node to reset voltage V.sub.pix. Following the transfer of the photocharge from storage node SN.sub.i, reset voltage V.sub.pix of floating diffusion FD.sub.i is subtracted from the voltage generated by the transferred photocharge at the floating diffusion node, thus providing compensation for KTC noise injected into the floating diffusion node during the reset operation.

[0079] Reset of floating diffusion nodes FD and charge transfer from storage nodes SN is performed simultaneously for an entire row of matrix 80 of sensing elements (FIG. 4a) or group of sensing elements in the row. When control circuitry 22 actuates row select gate RS.sub.i, a current source 126.sub.i drives a current through source follower SF.sub.i, and a signal from the gate of source follower SF.sub.i is transferred to column line 124.sub.i for further processing. Readout of the entire matrix 80 is performed sequentially, row by row.

[0080] After the photocharge has been transferred from sensing element PD.sub.i to storage node SN.sub.i, any residual charge from the sensing element is drained to common drain 66 by actuating shutter gate SG.sub.i. Sensing element PD.sub.i+1 is drained in a similar way.

[0081] The structure of image sensor 37 in the disclosed embodiments enables the image sensor to be operated, in both 2D imaging and 3D mapping, in a global shutter mode, wherein the acquisition of charges in storage nodes SN is performed simultaneously for all (or one or more groups of) sensing elements in matrix 80. It also enables operating image sensor 37 in a rolling shutter mode, as well as in a mode combining a rolling shutter and a global shutter.

[0082] As compared to 2D global-shutter image sensors that are known in the art, circuitry 120 includes one additional transistor per sensing element PD. However, as compared to commonly used iTOF circuits, each sensing element PD in circuitry 120 requires one less transistor and one less storage node. This feature enables the fabrication of smaller iTOF sensor elements, which can also provide high-resolution 2D imaging using these sensors.

[0083] It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.