Facebook Patent | Three Dimensional Depth Mapping Using Dynamic Structured Light

Patent: Three Dimensional Depth Mapping Using Dynamic Structured Light

Publication Number: 20180324412

Publication Date: 2018-11-08

Applicants: Facebook, Inc.

Abstract 

Apparatus for generating a dynamic structured light pattern for optical tracking in three-dimensional space, comprises an array of lasers, such as a VCSEL laser array, to project light in a pattern into a three-dimensional space; and an optical element or elements arranged in cells. The cells are aligned with subsets of the laser array, and each cell individually applies a modulation, in particular an intensity modulation, to light from the laser or lasers of the subset, to provide a distinguishable and separately controllable part of the dynamic structured light pattern. A method of generating a structured light pattern is disclosed, in which light is provided from an array of lasers, and light is individually projected from subsets of the array of lasers to provide differentiated parts of the structured light pattern.

BACKGROUND 

The present invention relates to three-dimensional depth mapping using structured light, and more particularly, but not exclusively, to a system for tracking in order to provide input to a computer or like device.

Tracking in order to provide input to a computer, or other computational device, involves such ubiquitous technologies as the computer mouse. Tracking of styluses and fingers in a three-dimensional field in front of the computer is also available and uses various tracking technologies such as visual and IR imaging and ultrasonic. The term `tracking` may refer to following the positioning and motion of an object, and includes processing of inputs received at the tracking computer in order to determine the position or motion. Thus in the case of a computer mouse, tracking may include processing the mouse outputs to determine motion. In the case of an object being followed visually, the term tracking may include image processing of successive frames capturing the object.

One method of imaging simply uses cameras to view and process a scene. The cameras may follow specific marks that are placed in the scene or the imaging system can look for specifically recognizable features such as fingers.

Drawbacks of such visual imaging include a requirement that the three-dimensional area is sufficiently lit up. Furthermore the only features that can be tracked are features that are recognized in advance, and motion tracking combined with feature recognition may not give accurate results.

In order to overcome the above problems, tracking using structured light was introduced. A known pattern of pixels is projected onto the space where tracking is to occur. The way that the pattern deforms on striking surfaces allows the vision system to calculate the depth and surface information of objects in the scene. Typical patterns used are grids or horizontal bars. Various devices use structured light patterns to enable the use of gesture recognition and 3D depth mapping. The structured light patter transmitter includes a laser emitter and a diffractive optical element (DOE).

Projecting a narrow band of light onto a three-dimensionally shaped surface produces a line of illumination that appears distorted from other perspectives than that of the projector, and can be used for an exact geometric reconstruction of the surface shape. A faster and more versatile method is the projection of patterns consisting of many stripes at once, or of arbitrary fringes, as this allows for the acquisition of a multitude of samples simultaneously. Seen from different viewpoints, the pattern appears geometrically distorted due to the surface shape of the object.

Although many other variants of structured light projection are possible, patterns of parallel stripes are widely used. The displacement of the stripes allows for an exact retrieval of the 3D coordinates of any details on the object's surface.

One known method of stripe pattern generation is the laser interference method, which utilizes two wide planar laser beam fronts. Interference between the beam fronts results in regular, equidistant line patterns. Different pattern sizes can be obtained by changing the angle between these beams. The method allows for the exact and easy generation of very fine patterns with unlimited depth of field. Disadvantages are high cost of implementation, difficulties providing the ideal beam geometry, and laser typical effects such as speckle noise and the possible self-interference with beam parts reflected from objects. Furthermore, there is no means of modulating individual stripes, such as with Gray codes.

Specifically, a disadvantage of using a single source emitter such as an edge emitter laser diode is the fact that the light pattern that it produces can be controlled only as a single unit. This means that the light pattern can be entirely turned on, off or dimmed but cannot be changed dynamically.

Structured light patterns may be constructed using invisible light such as IR light.

Alternatively high frame rates may render the structured light imperceptible to users or avoid interfering with other visual tasks of the computer.

The vertical-cavity surface-emitting laser, or VCSEL is a type of semiconductor laser diode in which laser beam emission is perpendicular from the top surface, as opposed to conventional edge-emitting semiconductor lasers which emit from surfaces formed by cleaving the individual chip out of a wafer.

There are several advantages to producing VCSELs, as opposed to edge-emitting lasers. Edge-emitters cannot be tested until the end of the production process. If the edge-emitter does not function properly, whether due to bad contacts or poor material growth quality, the production time and the processing materials have been wasted. VCSELs can be tested at several stages throughout the process to check for material quality and processing issues. For instance, if the vias have not been completely cleared of dielectric material during the etch, an interim testing process may be used to determine that the top metal layer is not making contact with the initial metal layer. Additionally, because VCSELs emit the beam perpendicularly to the active region of the laser, tens of thousands of VCSELs can be processed simultaneously on a three inch Gallium Arsenide wafer. Furthermore, even though the VCSEL production process is more labor and material intensive, the yield can be controlled to a more predictable outcome.

There is a significant advantage in that the use of VCSEL laser array for a structured light system, in that use of the array allows for a reduction in the size of the structured light transmitter device. The reduction is especially important for embedding the transmitter in devices with size restrictions such as a mobile phone or wearable devices.

However, despite the above advantages, the VCSEL array is not currently used for structured light scanning systems for a number of reasons. Many diffraction patterns require a coherent Gaussian shaped beam in order to create the high density patterns needed for high resolution tracking. The VCSEL array merely provides multiple individual Gaussian beams positioned next to each other and usually with overlap between them. The multiple points and overlap between them reduce the detection performance in high density areas in the light pattern and restrict the use of various diffractive design techniques that requires a pre-defined Gaussian beam. Such designs include a Top-Hat design, Homogeneous line generators, and other complex high performance structures.

Indeed the problem with a standard diffractive design is that the entire VCSEL laser array is used as a single light source. Thus, when using a multiple spot design the array image is obtained for each spot instead of having a focused Gaussian beam. A diffractive design that requires a Gaussian beam as an input will not get the required output at all. The problem becomes more severe in dense light patterns, because in these light patterns there is a need to focus the source beam onto a tiny spot in order to separate the features and this is not possible if the light source is an array of lasers.

SUMMARY 

The present embodiments provide an array of VCSEL lasers, where the lasers of the array are modulated individually or in groups. The individual lasers or groups may be modulated statically or dynamically to provide and alter a structured light pattern as needed.

Thus each laser in the array, or group of lasers being moderated together, is provided with its own optical element, typically a diffraction element. The diffraction element can be individually controlled so that the overall structured light pattern can be selected for given circumstances and/or can dynamically follow regions of interest.

According to a first aspect of the present invention there is provided apparatus for generating a structured light pattern, comprising:

an array of lasers arranged to project light in a pattern into a three-dimensional space; and a plurality of optical elements, respective optical elements defining cells, the cells being aligned with respective subsets of the array of lasers, the optical element of each cell individually applying a modulation to light passing through the optical element from the respective subset to provide a distinguishable part of the structured light pattern.

In an embodiment, the optical modulation is any of a diffractive modulation, a refractive modulation, and a combination of a diffractive and a refractive modulation.

In an embodiment, the optical elements and the subset of the array of lasers comprising a respective cell are constructed from a single molded element.

In an embodiment, a width of the cell is 1 mm or less.

In an embodiment, a width of the optical element is 1 mm or less.

In an embodiment, the cells are individually controllable to change the diffractive modulation.

In an embodiment, the cells are controllable dynamically to provide changes to the structured light pattern based on receiving and analyzing at least one captured frame, the frame comprising a plurality of pixels in a two-dimensional layout.

In an embodiment, the cells are further controllable in respect of positioning within the structured light pattern and in respect of a shape applied to the light.

In an embodiment, the dynamic control is configurable to apply increased resolution of the structured light pattern to parts of the scene to apply reduced resolution of the structured light pattern to other parts of the scene.

In an embodiment, the dynamic changes to the structured light pattern comprise changes to orientation of the structured light pattern.

In an embodiment, the dynamic changes to the structured light pattern comprise cell wise change.

In an embodiment, the change is any of a change in intensity, a change in polarization, a change in filtering parameters, and a change in focus.

In an embodiment, the subsets are any of individual lasers, pairs of lasers, triplets of lasers, combinations of different sizes of lasers, and dynamically changing combinations of lasers.

In an embodiment, light projected from respective subsets is tiled or overlapped.

In an embodiment, the laser array comprises a VCSEL laser array.

In an embodiment, the laser array comprises a laser bar.

The apparatus may be incorporated into any of: a computer, a laptop computer, a mobile communication device, a tablet device, a game console, and a movement capture device.

The apparatus may be used for tracking of a three-dimensional scene.

According to a second aspect of the present invention there is provided a method of generating a structured light pattern for three-dimensional tracking, the method comprising: Providing light from an array of lasers; and individually projecting light from subsets of the array of lasers to provide differentiated parts of the structured light pattern.

According to a third aspect of the presenting invention there is provided apparatus for generating a structured light pattern, comprising: an array of lasers arranged to project light in a pattern into a three-dimensional space; and a plurality of optical element cells, the cells being aligned with respective subsets of the array of lasers, each cell individually applying an intensity modulation to light passing through the element from the respective subset to provide a distinguishable part of the structured light pattern.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods, and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

Implementation of the method and/or system of embodiments of the invention can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of embodiments of the method and/or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof using an operating system.

For example, hardware for performing selected tasks according to embodiments of the invention could be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to exemplary embodiments of method and/or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions.

Optionally, the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk and/or removable media, for storing instructions and/or data. Optionally, a network connection is provided as well. A display and/or a user input device such as a keyboard or mouse are optionally provided as well.