Tobii Patent | Method of estimating a three-dimensional position of an object

In one embodiment of the first aspect, the method further comprises transforming the estimated three-dimensional position of at least one target object to a position in a coordinate system of the first image.

In one embodiment of the first aspect, the method further comprises obtaining a second image depicting at least a part of the scene captured by a second physical camera, wherein the at least part of the scene comprises the at least one target object, detecting the at least one target object, wherein the at least one target object is detected using the object detector and the second image, wherein detecting comprises identifying, by the object detector, the at least one target object and generating, by the object detector, second coordinates indicating a position of the at least one detected target object, generating a second set of images depicting subsets of the captured at least part of the scene, the subsets of the scene being depicted by at least one second virtual camera, estimating parameters of a second probability distribution indicative of a likelihood that the at least one target object is in a particular position in the second image by providing the generated set of images to the trained model, generating a combined probability distribution by combining the probability distribution and the second probability distribution, estimating the three-dimensional position of the at least one target object using the combined probability distribution.

According to a second aspect of the invention the object of the invention is achieved by a system, the system comprising a computer comprising processing circuitry; and a memory, said memory containing instructions executable by said processing circuitry, whereby said computer is operative to perform the method according to the first aspect.

According to a third aspect of the invention the object of the invention is achieved by a computer program comprising computer-executable instructions for causing a computer, when the computer-executable instructions are executed on processing circuitry comprised in the computer, to perform the method according to the first aspect.

According to a fourth aspect of the invention the object of the invention is achieved by a computer program product comprising a computer-readable storage medium, the computer-readable storage medium having the computer program according to the third aspect comprised therein.

The scope of the invention is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a method of estimation of three-dimensional positions of target objects according to one or more embodiments of the present disclosure.

FIG. 2 illustrates further details of the method for estimation of three-dimensional positions of target objects according to one or more embodiments of the present disclosure.

FIG. 3 shows a flowchart of a method according to one or more embodiments of the present disclosure.

FIG. 4 shows an embodiment of the method using two or more physical cameras according to one or more embodiments of the present disclosure.

FIG. 5A illustrates an example of obtaining a first image comprising two target objects according to one or more embodiments of the present disclosure.

FIG. 5B illustrates an example of detection of two target objects according to one or more embodiments of the present disclosure.

FIG. 6 illustrates an example of generating the set of images according to one or more embodiments of the present disclosure.

FIG. 7A illustrates further details of obtaining a first image comprising two target objects according to one or more embodiments of the present disclosure.

FIG. 7B illustrates an image captured by a physical camera.

FIG. 7C illustrates a set of images depicting subsets of the captured scene.

FIG. 8 illustrates an example of estimating two three-dimensional positions of respective target objects according to one or more embodiments of the present disclosure.

FIG. 9 illustrates an example of estimating a three-dimensional position of a target object using a combined probability distribution according to one or more embodiments of the present disclosure.

FIG. 10 shows a system according to one or more embodiments of the present disclosure.

A more complete understanding of embodiments of the invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by consideration of the following detailed description of one or more embodiments. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

DETAILED DESCRIPTION

An “or” in this description and the corresponding claims is to be understood as a mathematical OR which covers “and” and “or” and is not to be understand as an XOR (exclusive OR). The indefinite article “a” in this disclosure and claims is not limited to “one” and can also be understood as “one or more”, i.e., plural.

This application refers to documents Linden 2019, Mohlin 2021, Liu 2018, Robbins 1951 and Kingma 2014 which are hereby included in full and by reference.LINDÉN, Erik; SJOSTRAND, Jonas; PROUTIERE, Alexandre. Learning to personalize in appearance-based gaze tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019. p. 0-0.

MOHLIN, David; BIANCHI, Gerald; SULLIVAN, Josephine. Probabilistic Regression with Huber Distributions. arXiv preprint arXiv:2111.10296, 2021.

LIU, Katherine, et al. Deep inference for covariance estimation: Learning gaussian noise models for state estimation. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018. p. 1436-1443.

ROBBINS, Herbert; MONRO, Sutton. A stochastic approximation method. The annals of mathematical statistics, 1951, 400-407.

KINGMA, Diederik P.; BA, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

This invention relates to estimation of three-dimensional positions, e.g., positions of target objects in an image. The main concept involves capturing an image of a real-world scene by a physical camera having a particular field of view.

The captured image is then provided to a suitable object detector that returns coordinates of a sought-after object/target object and/or a patch of pixels representing the object in the captured image and/or a bounding box enclosing the target object.

A virtual camera is created, e.g., based on the detected object coordinates and predetermined calibration parameters to create a virtual camera. Creating a virtual camera is further described in relation to FIG. 2. The predetermined calibration parameters may comprise e.g., camera intrinsic and camera extrinsic. The calibration parameters are selected such that the entire detected object (in the captured image) is represented in a subset of pixels in the captured image. In a sense, the calibration parameters are selected such that the virtual camera “zooms in” on the detected object in the captured image. In other words, a virtual camera that “zooms in” on the detected object and related coordinates is created.

In one example, the camera is configured with a fixed image resolution, a principal axis which intersects the estimated center of the detected object, e.g., based on the coordinates of a sought-after object/target object and/or the patch of pixels representing the object in the captured image and/or the bounding box enclosing the object. It is understood that the principal axis of a virtual camera intersects with the position of the physical camera and the detected object. A focal length is selected for the virtual camera to be as small as possible, with the condition that the whole object is captured using the selected focal length.

The resulting virtual camera is configured with a principal axis that intersects the position of the physical camera and a target object. It is then possible to find a correspondence or projection between the coordinates in the original camera's image space and the virtual camera's image space. The pixel values for the virtual camera can be sampled from the captured image by standard image interpolation methods, such as nearest neighbour, bilinear or bi-cubic, to create an image which is indistinguishable from an image captured by a camera with the calibration parameters of the virtual camera, except for some interpolation artifacts.

The image from the virtual camera is then used as input to a trained model, e.g., a convolutional neural network, which predicts parameters of a probability distribution of the position of the target object relative to the location and orientation of the virtual camera. The output of the trained model is then optionally sent through a function N which maps/transforms the parameters to a standard form and enforces the constraints of the parameters of the distribution.

The mapped parameters and probability distribution can then be mapped back to a coordinate system of the original physical camera/s, using the previously determined correspondence or projection between the coordinates in the original camera's image space and the “virtual camera's” image space.

The trained model is initially trained using training data comprising images captured by the physical camera and corresponding known coordinates in the captured image for a target object. The trained model, or the parameters of the trained model, is fitted to output estimates of the probability distribution and resulting estimated three-dimensional position of at least one target object, which are consistent with the training data, by iteratively optimizing a fitness function, also known as loss or loss function, to the training data. A standard way to design such a loss or loss function is to use a negative log likelihood function for the probability distribution in question.

At least one purpose of the present disclosure is to improve accuracy of the generated probability distribution and thereby the accuracy of the estimated three-dimensional position/coordinate of at least one target object.

Definitions

In the present disclosure the term “calibration parameters” denotes parameters of a mapping from the captured image space to a real-world scene space. Calibration parameters typically comprise camera intrinsic and/or camera extrinsic. The extrinsic of a camera describes the position and orientation of the camera in 3D space. The intrinsic of the camera describes other properties of the camera, such as a principal axis, a field of view and a focal length.

In the present disclosure, the term “position” or “position of at least one target object” is used interchangeably.

The term “scene” used herein denotes an observed part or section of the real-world or a simulated world.

The term “physical camera” used herein denotes an imaging device, such as a video camera or a still camera, and comprises at least an optical system, lenses etc., and/or an image sensor with an array of image pixel sensors. The imaging device is configured to capture digital images of a real-world scene.

The term “video” or “image” used herein denotes an electronic medium for the recording, copying, playback, broadcasting, and display of moving or static visual media. The video comprises one or a plurality of subsequent video frames. Each video frame comprises a representation of a field of view of a scene. The representation comprises a plurality of pixels arranged to a particular aspect ratio, and a particular resolution.

FIG. 1 illustrates a method of estimation of three-dimensional position/s of at least one target object according to one or more embodiments of the present disclosure.

In FIG. 1 a real-world scene 110 is shown. The scene 110 comprises at least one (target) object 111, e.g., a human face in front of a screen. The scene 110 is captured as an image 130 by the physical camera 120. The physical camera 120 may be a video camera or a still camera and includes at least an optical system, lenses etc., or an image sensor with an array of image pixel sensors.

A parameter estimator 140 estimates parameters (A, a, μ) of a probability distribution P indicative of a likelihood that the at least one target object 111 is in or is located in a particular position 150 (x, y, z) in the scene captured in the image 130. In FIG. 1, the probability distribution P is illustrated with increasing density of dots corresponding to increased likelihood that the at least one target object 111 is in a particular position 150 (x, y, z) in the scene captured by the camera 120 in the image 130. The probability distribution P is in FIG. 1 illustrated in the two dimensions of the captured image 130 (horizontal axis x, vertical axis y); however, it is understood that the probability distribution P indicates a likelihood of at least one target object. In one example, three degrees of freedom probability distribution P may relate to the estimated three-dimensional position and four degrees of freedom relate to uncertainty of the estimation.

A three-dimensional position 150 of at least one target object 111 is then estimated using the probability distribution P. The estimation may e.g., be performed by a processor/processing circuitry as illustrated in FIG. 10.

FIG. 2 illustrates further details of the method for estimation of three-dimensional positions of at least one target object according to one or more embodiments of the present disclosure.

In a first step, a first image 130 is obtained that depicts the scene 110, shown in FIG. 1.

The first image 130 is typically obtained by capturing the image 130 by the physical camera 120, shown in FIG. 1. As shown in FIG. 2, the scene 110 and the captured image depicting the scene 130 comprise at least one target object 111.

In a subsequent step, at least one object is detected wherein at least one target object 111 is detected using an object detector 210 and the first image 130.

Detecting comprises, in embodiments, identifying, by the object detector 210, the at least one target object 111 and generating, by the object detector 210, coordinates 150 (x″, y″, z″) indicating a position in the first image of the at least one detected target object 111. The coordinates may, e.g., be defined as coordinates and size of the target object, as a patch of pixels representing the target object in the captured image and/or as a bounding box enclosing the object.

The object detector 210 may be any suitable object detector, such as any of Viola-Jones object detection framework based on Haar features, Scale-invariant feature transform (SIFT), Histogram of oriented gradients (HOG) features, Region Proposals (R-CNN), Fast R-CNN, Faster R-CNN, cascade R-CNN. Single Shot MultiBox Detector (SSD), You Only Look Once (YOLO), Single-Shot Refinement Neural Network for Object Detection (RefineDet), Retina-Net, Deformable convolutional networks.

Optionally, the method further comprises generating 220 a virtual camera configured with a principal axis that intersects a position of the physical camera and the detected object. Additionally, or alternatively, the method further comprises generating one or more target areas 305, 306, as shown in FIG. 6, using predetermined camera calibration parameters for the physical camera, wherein each one target areas 305, 306 defines a respective subset of the scene being depicted by a respective virtual camera, wherein the respective subset comprises a respective detected target object 111. It is to be understood that the target area may be defined by a set of generated coordinates. In one example, the generated coordinates would be corners of a bounding box around the target objects. In another example, the generated coordinates may have other forms to ensure enclosing the target object with a geometrical shape. Such coordinates may for example be generated based on the object detector. The object detector may be a face detector. The target area with a predetermined aspect ratio is illustrated as the frame outlining a set of images 230.

In a subsequent step, the set of images 230 is generated that depict subsets of the captured scene 110. The subsets of the scene 110 being depicted by at least one virtual camera. In other words, the virtual camera extracts pixels from the image captured by the physical camera, e.g., by selection of pixels/pixel values or by interpolation of pixels/pixel values.

The detected coordinates are optionally used to create a virtual camera with predetermined calibration parameters. The virtual camera is generated such that the entire detected target object 111 in the captured image 130 is represented in a subset of pixels derived from the captured image 130. In a sense, the virtual camera is generated, and calibration parameters are selected such that the virtual camera “zooms in” on the detected object in the captured image. In other words, the calibration parameters are selected to ensure that the field of view of the virtual camera includes the entire detected object 111.

In one example, this includes generating 220 a virtual camera configured with a principal axis that intersects a position of the physical camera and the detected object. The focal length of the virtual camera is then selected such that the field of view of the virtual camera is fully enclosing the detected object. In FIG. 2 this is illustrated by “zooming” in on the face in the scene 110, shown in FIG. 1, and generating a subset of the first image 230.

The pixel values for the “virtual camera” can be sampled from the image captured by the physical camera by standard image interpolation methods, such as nearest neighbor, bilinear or bi-cubic, to create an image which is indistinguishable from an image captured by a camera with the calibration parameters of the virtual camera, except possibly some interpolation artifacts.

In a subsequent step, parameters of a probability distribution P indicative of a likelihood that the at least one target object 111 is in a particular position 150 (x, y, z) in the scene are estimated by providing the generated set of images 230 to a trained model, e.g., a Convolutional Neural Network, depicted as 240 in FIG. 2.

In an intermediate step, an intermediate output of the trained model, illustrated by a vector v (A′, a′, μ′) in FIG. 2, is transformed, by applying a transformation function N to the estimated parameters/vector v, to transform the estimated parameters (A′, a′, μ′) within predetermined constraints of the probability distribution P to transformed and constrained parameters (A, a, μ). In other words, the transformation function N forces the intermediate output parameter of the trained model (A′, a′, μ′) to output parameters fulfilling the constraint (A, a, μ). Such constraints are boundary conditions, such as positive definiteness for A, μ_z>0 and the constraint that parameter a is large enough to guarantee convexity of the negative log likelihood.

The probability distribution P with a density function f can then be derived using the parameters fulfilling the constraint (A, a, μ).

In a subsequent step, the three-dimensional position (x, y, z) 150 of the at least one target object 111 is estimated using the probability distribution P based on the transformed and constrained parameters (A, a, μ).

At least one advantage of this embodiment is that accuracy when estimating a distance to the target object and estimating the direction/ray to the target object from images is increased and computational complexity is simultaneously reduced.

The disclosed method may further be illustrated in a flowchart.

FIG. 3 shows a flowchart of a method 300 according to one or more embodiments of the present disclosure. The method 300 estimates a three-dimensional position 150 of at least one target object 111 and is performed by a computer. The method comprises:

Step 310: obtaining a first image 130 depicting a scene 110, wherein the scene 110 comprises at least one target object 111. The first image may be captured by a physical camera 120 or generated by other suitable image sources with known calibration parameters.

The calibration parameters of the physical camera are known, and therefore the relation between a point on the image space/plane and a ray in the real-world space from the image sensor of the camera to objects in the real-world scene.

Step 320: detecting at least one target object 111, wherein at least one target object 111 is detected using an object detector 210 on the first image 130. Additionally, or alternatively, detecting comprises identifying 321, by the object detector 210, the at least one target object 111 and generating 323, by the object detector 210, coordinates indicating a position of the at least one detected target object. The object detector further described in relation to FIG. 2.

The object detector typically returns coordinates of a sought-after object/target object and/or a patch of pixels representing the target object in the captured image and/or a bounding box enclosing the object. Coordinates indicating the position of at least one detected target object may be derived as the returned coordinates, as any of the pixel coordinates of the patch (in the respective image) or as center or corner coordinates of the bounding box. In other words, the returned coordinates from the detector may e.g., be defined as coordinates and size of the target object, as a patch of pixels representing the target object in the captured image and/or as the bounding box enclosing the object.

Step 330: generating a set of images 230 depicting subsets of the captured scene 110. The subsets of the scene are depicted by at least one virtual camera.

In one example, a single target object is detected, e.g., an eye or nose tip in a human face. A virtual camera is generated that is configured with a principal axis that intersects with the physical camera and the center of the eye or nose tip. The focal length of the virtual camera is then selected such that the field of view of the virtual camera fully encloses the detected object, such as the human eye. This may, for example, involve adapting the field of view of the virtual camera to a bounding box generated by the object detector 210. Generating the set of images 230 by the virtual camera is further described in relation to FIG. 2.

In one further example, two target objects are detected, e.g., two eyes in a human face. Two virtual cameras are then generated. A first virtual camera is generated that is configured with a principal axis that intersects with the physical camera and the center of the pupil of the left eye. A second virtual camera is generated with a principal axis that intersects with the physical camera and the center of the pupil of the right eye. The focal length of the first virtual camera is then selected such that the field of view of the virtual camera fully encloses the left eye. The focal length of the second virtual camera is then selected such that the field of view of the second virtual camera fully encloses the right eye. This may, for example, involve adapting the field of view of the virtual cameras to bounding boxes generated by the object detector 210. Generating the set of images 230 then comprises generating the images depicted by the first virtual camera of the first eye and depicted by the second virtual camera of the second eye respectively.

FIG. 5B and related text illustrates one further example where two target objects are detected.

Step 340: estimating parameters of the probability distribution P indicative of a likelihood that the at least one target object 111 is in a particular position (x, y, z) 150 in the scene 110 by providing the generated set of images 230 to a trained model 240, e.g., a Convolutional Neural Network CNN.

As further described in relation to FIG. 2, the intermediate output of the trained model CNN, illustrated by a vector v (A′, a′, μ′) in FIG. 2, is transformed, by applying a transformation function N to the estimated parameters/vector v, to transform the estimated parameters (A′, a′, μ′) within predetermined constraints of the probability distribution P to transformed and constrained parameters (A, a, μ) or transformed and constrained output of the trained model. In other words, the transformation function N forces the intermediate output parameter of the trained model (A′, a′, μ′) to output parameters fulfilling the constraint (A, a, μ). The probability distribution P with the density function f can then be derived using the parameters whilst fulfilling the constraint.

A probability distribution for each set of images 230 is estimated or generated and is initially in a coordinate system of the respective virtual camera. See FIG. 7A-C for further details regarding coordinates systems. FIG. 7A-C illustrates that all cameras, physical and virtual cameras respectively, have a coordinate system associated with them, typically related to the principal axis and orientation of the camera/s' field of view. The coordinate systems in these examples are shown with a shared origin.

However, as both the calibration parameters for the one or more virtual cameras and the calibration parameters for the physical camera/s 120 are known, the probability distribution for each of the set of images 230 can be transferred/projected to the coordinate system of the physical camera/s 120. Any suitable method in the art may be used to transform/project probability distribution for each of the set of images 230 from the coordinate system of the respective virtual camera to the coordinate system of the physical camera/s, such as rotation and/or translation

The probability distribution P can for example be defined as:

$v^{*}=\arg \underset{v}{\max }{\prod }_{i=1}^{❘"\backslash [LeftBracketingBar]"C❘"\backslash [RightBracketingBar]"}{p}_i\left(R_{i}v+t_i\right)$

R_iv is a matrix multiplication of R_i and v.

This distribution has, in one example, seven degrees of freedom. The intermediate output of the trained model, e.g., a neural network, is referred to as vector v, as described in relation to FIG. 2. The mapping between intermediate outputs from the trained model v and the probability distribution parameters of probability distribution P may, for example, be constructed as follows.

Examples of output vectors are presented in Liu 2018. In one further example in Mohlin 2021, a symmetric matrix B is formed from the output vector v using a bijection. An eigenvalue composition is then performed. Further a positive definite matrix is constructed by applying a function on each eigenvalue independently.

The parameter A can be derived from the first 3 values of vector v, by for example using the mapping described in in Liu 2018 or in Mohlin 2021. For example, in Liu 2018 they use these 3 values to parameterize an LDL decomposition of the matrix A. L is a lower unit triangular (unitriangular) matrix, and D is a diagonal matrix. The LDL decomposition is often called the square-root-free Cholesky decomposition.

The parameters μ_x and μ_y have no constraints, and can therefore be picked directly from the 4^th and 5^th value in the output vector v. The parameter z has a constraint in that it needs to be positive. This constraint can be achieved by, for example, applying an exponential function on the 6^th value of the output vector v. The parameter “a” needs to be larger than:

$a>\frac{\sqrt{5}}{4}$

This constraint can be enforced by, for example, the constraint:

$a=\min \left(❘"\backslash [LeftBracketingBar]"\mathrm{v\_}7❘"\backslash [RightBracketingBar]"+1,e^{v\_7}+\sqrt{\frac{5}{4}}\right)$

This probability distribution can then be mapped back to the coordinate system of the original camera since it is known via the calibration parameters how the virtual camera and the physical camera are related.

For example, if the virtual camera is rotated relative to the real camera by the rotation R, then the probability in the original camera system is

p_reai(v)=p_virtual(RV)