Qualcomm Patent | Point-based object localization from images

Patent: Point-based object localization from images

Drawings: Click to check drawins

Publication Number: 20210209797

Publication Date: 20210708

Applicant: Qualcomm

Abstract

Techniques and systems are provided for determining features of one or more objects in one or more images. For example, an image of an object and a three-dimensional model associated with the object can be obtained. From the image, a sample point on the object can be determined. A depth and an angle of the sample point of the object can be determined. A pose and a shape of the three-dimensional model associated with the object can be determined based on the depth and the angle.

Claims

1. A method of determining features of one or more objects in one or more images, the method comprising: obtaining an image of an object; obtaining a three-dimensional model associated with the object; determining, from the image, a sample point on the object; determining a depth and an angle of the sample point of the object; and determining, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

2. The method of claim 1, wherein the depth and the angle of the sample point are determined using the sample point and no other sample points of the object.

3. The method of claim 1, wherein the angle indicates an orientation of the object relative to a directional vector from a center of a camera used to capture the image to the sample point.

4. The method of claim 3, wherein the angle includes a yaw angle.

5. The method of claim 1, wherein the depth includes a distance from a center of a camera used to capture the image to the sample point.

6. The method of claim 1, further comprising: obtaining a two-dimensional bounding region for the object in the image; and determining the sample point from within the two-dimensional bounding region in the image.

7. The method of claim 1, further comprising: obtaining a two-dimensional bounding region for the object in the image; and aligning a projection of a three-dimensional bounding region to back-projected rays of at least two sides of the two-dimensional bounding region of the object.

8. The method of claim 7, further comprising: determining a first three-dimensional vector defining a forward direction of a camera used to capture the image; determining a second three-dimensional vector from the camera to a first corner point of the two-dimensional bounding region; determining a third three-dimensional vector from the camera to a second corner point of the two-dimensional bounding region; and determining a fourth three-dimensional vector from the camera to the sample point.

9. The method of claim 8, wherein the first corner point includes a top-left corner point of the two-dimensional bounding region, and wherein the second corner point includes a bottom-right corner point of the two-dimensional bounding region.

10. The method of claim 8, wherein the first corner point includes a bottom-left corner point of the two-dimensional bounding region, and wherein the second corner point includes a top-right corner point of the two-dimensional bounding region.

11. The method of claim 8, further comprising: obtaining a pitch angle relative to the camera, the pitch angle indicating an angle between a ground plane and a forward direction of a camera used to capture the image; determining, using the pitch angle, a first two-dimensional directional vector for the first three-dimensional vector; determining, using the pitch angle, a second two-dimensional directional vector for the second three-dimensional vector; determining, using the pitch angle, a third two-dimensional directional vector for the third three-dimensional vector; and determining, using the pitch angle, a fourth two-dimensional directional vector for the fourth three-dimensional vector.

12. The method of claim 11, further comprising: determining a first angle between the first two-dimensional directional vector and the fourth two-dimensional directional vector; determining a second angle between the second two-dimensional directional vector and the fourth two-dimensional directional vector; determining a third angle between the third two-dimensional directional vector and the fourth two-dimensional directional vector; determining a fourth angle between the first corner point of the two-dimensional bounding region and a forward direction of the object; determining a fifth angle between the second corner point of the two-dimensional bounding region and a forward direction of the object; determining a first distance from the first corner point of the two-dimensional bounding region to the sample point; and determining a second distance from the second corner point of the two-dimensional bounding region to the sample point.

13. The method of claim 12, further comprising: determining the depth and the angle of the sample point using the first angle, the second angle, the third angle, the fourth angle, the fifth angle, the first distance, and the second distance.

14. The method of claim 11, further comprising: determining, using the pitch angle, a first rotation matrix from the ground plane to a coordinate system of the camera; wherein the first two-dimensional directional vector, the second two-dimensional directional vector, the third two-dimensional directional vector, and the fourth two-dimensional directional vector are determined using the first rotation matrix.

15. The method of claim 14, wherein determining, based on the depth and the angle, the pose of the three-dimensional model associated with the object includes: determining a second rotation matrix using the angle of the sample point; determining a three-dimensional location of the sample point; and computing a pose parameter for the three-dimensional model using the depth of the sample point, the first rotation matrix, the second rotation matrix, and the three-dimensional location of the sample point.

16. The method of claim 1, further comprising: determining, based on the pose and the shape, a three-dimensional bounding region for the object; and outputting the three-dimensional bounding region for display with the image.

17. An apparatus for determining features of one or more objects in one or more images, comprising: a memory configured to store at least one image; and a processor implemented in circuitry and configured to: obtain an image of an object; obtain a three-dimensional model associated with the object; determine, from the image, a sample point on the object; determine a depth and an angle of the sample point of the object; and determine, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

18. The apparatus of claim 17, wherein the depth and the angle of the sample point are determined using the sample point and no other sample points of the object.

19. The apparatus of claim 17, wherein the angle indicates an orientation of the object relative to a directional vector from a center of a camera used to capture the image to the sample point.

20. The apparatus of claim 19, wherein the angle includes a yaw angle.

21. The apparatus of claim 17, wherein the depth includes distance from a center of a camera used to capture the image to the sample point.

22. The apparatus of claim 17, wherein the processor is configured to: obtain a two-dimensional bounding region for the object in the image; and determine the sample point from within the two-dimensional bounding region in the image.

23. The apparatus of claim 17, wherein the processor is configured to: obtain a two-dimensional bounding region for the object in the image; and align a projection of a three-dimensional bounding region to back-projected rays of at least two sides of the two-dimensional bounding region of the object.

24. The apparatus of claim 23, wherein the processor is configured to: determine a first three-dimensional vector defining a forward direction of a camera used to capture the image; determine a second three-dimensional vector from the camera to a first corner point of the two-dimensional bounding region; determine a third three-dimensional vector from the camera to a second corner point of the two-dimensional bounding region; and determine a fourth three-dimensional vector from the camera to the sample point.

25. The apparatus of claim 24, wherein the first corner point includes a top-left corner point of the two-dimensional bounding region, and wherein the second corner point includes a bottom-right corner point of the two-dimensional bounding region.

26. The apparatus of claim 24, wherein the first corner point includes a bottom-left corner point of the two-dimensional bounding region, and wherein the second corner point includes a top-right corner point of the two-dimensional bounding region.

27. The apparatus of claim 24, wherein the processor is configured to: obtain a pitch angle relative to the camera, the pitch angle indicating an angle between a ground plane and a forward direction of a camera used to capture the image; determine, using the pitch angle, a first two-dimensional directional vector for the first three-dimensional vector; determine, using the pitch angle, a second two-dimensional directional vector for the second three-dimensional vector; determine, using the pitch angle, a third two-dimensional directional vector for the third three-dimensional vector; and determine, using the pitch angle, a fourth two-dimensional directional vector for the fourth three-dimensional vector.

28. The apparatus of claim 27, wherein the processor is configured to: determine, using the pitch angle, a rotation matrix from the ground plane to a coordinate system of the camera; wherein the first two-dimensional directional vector, the second two-dimensional directional vector, the third two-dimensional directional vector, and the fourth two-dimensional directional vector are determined using the rotation matrix.

29. The apparatus of claim 17, wherein the processor is configured to: determine, based on the pose and the shape, a three-dimensional bounding region for the object; and output the three-dimensional bounding region for display with the image.

30. The apparatus of claim 17, wherein the apparatus is one of a vehicle or a robot.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of, and priority to, U.S. Provisional Patent Application No. 62/957,627, filed on Jan. 6, 2020, entitled “POINT-BASED OBJECT LOCALIZATION FROM IMAGES,” the contents of which are hereby expressly incorporated by reference in its entirety.

FIELD

[0002] The present disclosure generally relates to performing object localization, and more specifically to techniques and systems for performing point-based object localization to determine the pose of an object in an image.

BACKGROUND

[0003] Systems are available for determining objects that are present in real images and attributes of those objects. In some cases, object localization can be performed to localize an object in a scene. For example, a digital image or a video frame of a video sequence can be processed using object localization to determine a pose of the object in three-dimensional space.

[0004] Object localization for pose estimation is useful for many applications and systems, including augmented reality (AR), virtual reality (VR), mixed reality (MR), robotic systems, manufacturing systems, quality assurance, automotive and aviation (e.g., manufacturing, autonomous driving or navigation, etc.), three-dimensional scene understanding, object grasping, object tracking, video analytics, security systems, among many others. For instance, a three-dimensional model can be determined for representing an object in an image, and can be used to facilitate effective operation of various systems. In AR environments, as an illustrative example, a user may view images that include an integration of virtual content with the user’s natural surroundings. AR applications allow real images to be processed to add virtual objects to the images and to align the virtual objects to the image in multiple dimensions. For instance, a real-world object that exists in reality can be represented using a model that resembles or is an exact match of the real-world object. The user may be able to manipulate the model while viewing the real-world scene. In another illustrative example, an autonomous vehicle can determine shapes and poses of other vehicles driving on a road in order to navigate through traffic.

[0005] Inferring three-dimensional information (e.g., localization (pose) and shape) of an object from a single two-dimensional image can be challenging, due in part to the lack of geometric cues for objects in single images.

SUMMARY

[0006] Systems and techniques are described herein for determining poses of objects in images using point-based object localization. For example, the systems and techniques described herein provide an efficient way to estimate a six degrees of freedom (6-DoF) object pose and the shape of the object from an image (e.g., a single image) including the object. In some cases, the systems and techniques can determine the pose (and in some cases the shape) of the object using one sample point from the data, greatly reducing the complexity of performing the point-based objection localization.

[0007] According to one illustrative example, a method of determining features of one or more objects in one or more images is provided. The method includes: obtaining an image of an object; obtaining a three-dimensional model associated with the object; determining, from the image, a sample point on the object; determining a depth and an angle of the sample point of the object; and determining, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

[0008] In another example, an apparatus for determining features of one or more objects in one or more images is provided that includes a memory configured to store at least one image and one or more processors implemented in circuitry and coupled to the memory. The one or more processors are configured to and can: obtain an image of an object; obtain a three-dimensional model associated with the object; determine, from the image, a sample point on the object; determine a depth and an angle of the sample point of the object; and determine, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

[0009] In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processor to: obtain an image of an object; obtain a three-dimensional model associated with the object; determine, from the image, a sample point on the object; determining a depth and an angle of the sample point of the object; and determine, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

[0010] In another example, an apparatus for determining features of one or more objects in one or more images is provided. The apparatus includes: means for obtaining an image of an object; means for obtaining a three-dimensional model associated with the object; means for determining, from the image, a sample point on the object; means for determining a depth and an angle of the sample point of the object; and means for determining, based on the depth and the angle, a pose and a shape of the three-dimensional model associated with the object.

[0011] In some aspects, the depth and the angle of the object in the image are determined using the sample point and no other sample points of the object.

[0012] In some aspects, the angle indicates an orientation of the object relative to a directional vector from a center of a camera used to capture the image to the sample point. In some cases, the angle includes a yaw angle.

[0013] In some aspects, the depth includes distance from a center of a camera used to capture the image to the sample point.

[0014] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: obtaining a two-dimensional bounding region for the object in the image; and determining the sample point from within the two-dimensional bounding region in the image.

[0015] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: obtaining a two-dimensional bounding region for the object in the image; and aligning a projection of a three-dimensional bounding region to back-projected rays of at least two sides of the two-dimensional bounding region of the object.

[0016] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: determining a first three-dimensional vector defining a forward direction of a camera used to capture the image; determining a second three-dimensional vector from the camera to a first corner point of the two-dimensional bounding region; determining a third three-dimensional vector from the camera to a second corner point of the two-dimensional bounding region; and determining a fourth three-dimensional vector from the camera to the sample point.

[0017] In some examples, the first corner point includes a top-left corner point of the two-dimensional bounding region, and the second corner point includes a bottom-right corner point of the two-dimensional bounding region. In some examples, the first corner point includes a bottom-left corner point of the two-dimensional bounding region, and the second corner point includes a top-right corner point of the two-dimensional bounding region.

[0018] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: obtaining a pitch angle relative to the camera, the pitch angle indicating an angle between a ground plane and a forward direction of a camera used to capture the image; determining, using the pitch angle, a first two-dimensional directional vector for the first three-dimensional vector; determining, using the pitch angle, a second two-dimensional directional vector for the second three-dimensional vector; determining, using the pitch angle, a third two-dimensional directional vector for the third three-dimensional vector; and determining, using the pitch angle, a fourth two-dimensional directional vector for the fourth three-dimensional vector.

[0019] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: determining, using the pitch angle, a first rotation matrix from the ground plane to a coordinate system of the camera; wherein the first two-dimensional directional vector, the second two-dimensional directional vector, the third two-dimensional directional vector, and the fourth two-dimensional directional vector are determined using the first rotation matrix.

[0020] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: determining a first angle between the first two-dimensional directional vector and the fourth two-dimensional directional vector; determining a second angle between the second two-dimensional directional vector and the fourth two-dimensional directional vector; determining a third angle between the third two-dimensional directional vector and the fourth two-dimensional directional vector; determining a fourth angle between the first corner point of the two-dimensional bounding region and a forward direction of the object; determining a fifth angle between the second corner point of the two-dimensional bounding region and a forward direction of the object; determining a first distance from the first corner point of the two-dimensional bounding region to the sample point; and determining a second distance from the second corner point of the two-dimensional bounding region to the sample point.

[0021] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: determining the depth and an angle of the object in the image using the first angle, the second angle, the third angle, the fourth angle, the fifth angle, the first distance, and the second distance.

[0022] In some aspects, determining, based on the depth and the angle, the pose of the three-dimensional model associated with the object includes: determining a second rotation matrix using the angle of the sample point; determining a three-dimensional location of the sample point; computing a pose parameter for the three-dimensional model using the depth of the sample point, the first rotation matrix, the second rotation matrix, and the three-dimensional location of the sample point.

[0023] In some aspects, the method, apparatuses, and computer-readable medium described above further comprise: determining, based on the pose and the shape, a three-dimensional bounding region for the object; and outputting the three-dimensional bounding region for display with the image.

[0024] In some aspects, the apparatus is, is part of, and/or includes a vehicle or a computing device or component of a vehicle (e.g., an autonomous vehicle), a camera, a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data. In some aspects, the apparatuses described above can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyrometers, one or more accelerometers, any combination thereof, and/or other sensor).

[0025] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0026] The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Illustrative embodiments of the present application are described in detail below with reference to the following figures:

[0028] FIG. 1A and FIG. 1B are images illustrating an example of object detection and object localization results on the images, in accordance with some examples;

[0029] FIG. 1C and FIG. 1D are images illustrating another example of object detection and object localization results on the images, in accordance with some examples;

[0030] FIG. 2 is a diagram illustrating an example of a point-based object localization technique using multiple sample points, in accordance with some examples;

[0031] FIG. 3 is a diagram illustrating a geometric relationship between two-dimensional coordinates of an object in an image and a three-dimensional point of the object in a the three-dimensional real-world space, in accordance with some examples;

[0032] FIG. 4A-FIG. 4C are diagrams illustrating an example of coordinate transformations from an object coordinate system to a camera coordinate system, in accordance with some examples;

[0033] FIG. 4D is a diagram illustrating an example of a projection of a three-dimensional point on an object coordinate system to a two-dimensional point on an image, in accordance with some examples;

[0034] FIG. 4E is a diagram illustrating an example of back-projection of a two-dimensional point on an image to a camera coordinate system, in accordance with some examples;

[0035] FIG. 4F is a conceptual diagram illustrating an example of a pinhole camera model, in accordance with some examples;

[0036] FIG. 5-FIG. 8D are diagrams illustrating a perspective-n-point (PnP) technique, in accordance with some examples;

[0037] FIG. 9 is a diagram illustrating an example of a system for determining poses (and in cases shapes) of objects in images, in accordance with some examples;

[0038] FIG. 10 is a diagram illustrating a high level overview of a process that can be performed by the system shown in FIG. 9, in accordance with some examples;

[0039] FIG. 11A and FIG. 11B are diagrams illustrating an example of a relationship between an object and a camera, in accordance with some examples;

[0040] FIG. 12A is a diagram illustrating an example of pitch angle of a camera relative to a ground plane, in accordance with some examples;

[0041] FIG. 12B-FIG. 12J are diagrams illustrating an example of determining various parameters from a three-dimensional model, in accordance with some examples;

[0042] FIG. 13 is a diagram illustrating the Sine rule, in accordance with some examples;

[0043] FIG. 14A-FIG. 14E are diagrams illustrating examples of four solutions that can be determined and selected using the techniques described herein, in accordance with some examples;

[0044] FIG. 15A-FIG. 15D are images illustrating examples of keypoint-based vehicle pose estimation, in accordance with some examples;

[0045] FIG. 16A-FIG. 16D are images and FIG. 16E is a diagram illustrating an example of results of keypoint-based vehicle pose estimation using an image from a front-facing camera of a vehicle, in accordance with some examples;

[0046] FIG. 17A-FIG. 17D are images and FIG. 17E is a diagram illustrating an example of results of keypoint-based vehicle pose estimation using an image from a rear-facing camera of a vehicle, in accordance with some examples;

[0047] FIG. 18 is a block diagram illustrating an example of a deep learning network, in accordance with some examples;

[0048] FIG. 19 is a block diagram illustrating an example of a convolutional neural network, in accordance with some examples;

[0049] FIG. 20 is a diagram illustrating an example of a neural network used for keypoint detection, in accordance with some examples;

[0050] FIG. 21 is a diagram illustrating an example of the Cifar-10 neural network, in accordance with some examples;

[0051] FIG. 22A-FIG. 22C are diagrams illustrating an example of a single-shot object detector, in accordance with some examples;

[0052] FIG. 23A-FIG. 23C are diagrams illustrating an example of a You Only Look Once (YOLO) detector, in accordance with some examples;

[0053] FIG. 24A-FIG. 24E are diagrams illustrating examples of appearance properties of various objects used for defining keypoints, in accordance with some examples;

[0054] FIG. 25A-FIG. 25D are diagrams illustrating examples of keypoints defined for various objects, in accordance with some examples;

[0055] FIG. 26A is a diagram illustrating examples of three-dimensional models used for various objects, in accordance with some examples;

[0056] FIG. 26B and FIG. 26C are diagrams illustrating examples of three-dimensional keypoints defined for objects from FIG. 26A, in accordance with some examples;

[0057] FIG. 27A-FIG. 27C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0058] FIG. 28A-FIG. 28C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0059] FIG. 29A-FIG. 29C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0060] FIG. 30A-FIG. 30C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0061] FIG. 31A-FIG. 31C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0062] FIG. 32A-FIG. 32C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0063] FIG. 33A-FIG. 33C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0064] FIG. 34A-FIG. 34C are diagrams illustrating examples of results achieved using the techniques described herein compared with an alternative method, in accordance with some examples;

[0065] FIG. 35 is a flowchart illustrating an example of a process of determining features of one or more objects in one or more images using the techniques described herein, in accordance with some examples; and

[0066] FIG. 36 is a block diagram of an exemplary computing device that may be used to implement some aspects of the technology described herein, in accordance with some examples.

DETAILED DESCRIPTION

[0067] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0068] The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0069] Object detection can be used to detect objects in images, and in some cases various attributes of the detected objects. For instance, object localization is a technique that can be performed to localize an object in a digital image or a video frame of a video sequence capturing a scene or environment. Using object localization, a pose of the object can be determined in three-dimensional (3D) space. For instance, object localization can be performed on a single, monocular image to estimate a six-dimensional (6D) pose parameter (including a 3D translation vector and a 3D rotation vector, providing the six dimensions) in the camera coordinate system of an object in the image.

[0070] Object localization for pose estimation is useful for many applications and systems, such as extended reality (XR) systems (e.g., augmented reality (AR) systems, virtual reality (VR) systems, mixed reality (MR) systems, and/or other XR systems), robotic systems, manufacturing systems, quality assurance systems, automotive and aviation systems (e.g., for manufacturing, for autonomous driving or navigation for autonomous vehicles and/or unmanned aerial vehicles, etc.), three-dimensional scene understanding systems, object grasping systems, object tracking systems, video analytics systems, security systems, among others. In some examples, a 3D model can be determined for representing an object in an image, and can be used to facilitate effective operation of one or more of the systems noted above. In one illustrative example, AR systems and/or applications can process images and can add virtual objects to the images, in some cases aligning the virtual objects to the image in multiple dimensions. For instance, a real-world object that exists in reality can be represented using a model that resembles or is an exact match of the real-world object. The user may be able to manipulate the model while viewing the real-world scene. In another example, in object tracking systems (e.g., used in an autonomous vehicle, robotics system, or other system), an object (referred to as a tracking object) tracking other objects (referred to as target objects) in an environment can determine poses and sizes of the other objects. Determining the poses and sizes of target objects in the environment allows the tracking object to accurately navigate through the environment by making intelligent motion planning and trajectory planning decisions.

[0071] Systems, apparatuses, methods (also referred to as processes), and computer-readable media are described herein for determining poses of objects in images using point-based object localization techniques. The systems and techniques described herein provide an efficient way to estimate a 6-degrees-of-freedom (6-DoF) object pose and the shape of the object from an image including the object (e.g., from a single image of the object). For instance, the systems and techniques can solve a Perspective-n-Point (PnP) problem in a more efficient manner as compared to existing point-based localization techniques (described below).

[0072] In some cases, the systems and techniques can implement a keypoint-based solution to determine an initial pose (and in some cases the shape) of an object in an image using one sample point (e.g., a 1-point Random sample consensus (RANSAC)-based method) from the input data. In one illustrative example (e.g., using the 1-point RANSAC-based method), only one sample point from the input data is needed for the systems and techniques to compute an object pose hypothesis. In addition to the one point needed to compute a pose candidate, other information can also be used, such as a two-dimensional (2D) bounding box of an object (e.g., determined using object detection) and a pitch angle of the camera. For instance, if it is assumed that an object is on the ground and the relationship (e.g., the pitch angle) between the ground and a camera used to capture an image is pre-calibrated, the pose parameterization for the PnP problem is reduced to 1-degrees-of-freedom (1-DoF) parameterization (e.g., including a yaw angle and/or a depth of an object). As described in more detail below, a pose hypothesis can be computed per a point correspondence sample. A best sample consensus can be determined using a RANSAC process (or other suitable process) with a computational complexity of O(m) (with m being the number of points sampled in the RANSAC process). Because only one sample point is needed in such an example, the number of required iterations of the RANSAC process is at most m times or less than m times, which reduces the computational complexity of performing the point-based objection localization from O(m.sup.3) or greater (up to O(m.sup.n)) to O(m).

[0073] Various aspects of the application will be described with respect to the figures. One example of a field where a tracking object needs to be able to determine the pose of target objects is autonomous driving by autonomous driving systems (e.g., of autonomous vehicles). For instance, an autonomous driving system of an autonomous vehicle (a tracking vehicle) can determine the shapes and poses of other vehicles (target vehicles) on a road in order to successfully navigate itself through traffic. FIG. 1A and FIG. 1B are images 100 and 101 (from a side-facing camera of a tracking vehicle) illustrating an example of object detection and object localization results of objects in images. As shown in the image 100 of FIG. 1A, two cars are detected with respective 3D bounding boxes and classification (class) or category labels of “car.” A 3D mesh (of a respective 3D model) is shown for each of the cars in the image 101 of FIG. 1B. To generate the 3D bounding boxes and the 3D meshes with an accurate pose and shape, object localization can be performed on the underlying image shown in FIG. 1A and FIG. 1B. The object localization can be used to estimate a 6D pose parameter (3D translation vector and 3D rotation vector) of the two cars in the camera coordinate system. FIG. 1C and FIG. 1D are images 102 and 103 (from a rear-facing camera of a tracking vehicle) illustrating another example of object detection and object localization results, with 3D bounding boxes shown in the image 102 and the 3D meshes shown in image 103.

[0074] Various techniques can be used to perform 3D object localization. In some examples, direct localization methods can use neural networks (e.g., convolutional neural networks (CNNs) or other neural network based system or algorithm) to regress the road object state. Examples of such techniques include ROI-10D (described in Fabian Manhardt et al., “ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape,” Computer Vision and Pattern Recognition (CVPR), 2019) and 3D-RCNN (described in Abhijit Kundu et al., “3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare,” CPVR, 2018).

[0075] Some techniques solve a Perspective-n-Point (PnP) problem to estimate object poses. In some cases, when a pose is estimated from data that contains many outliers, a Random sample consensus (RANSAC) technique can be performed for robust model fitting. For instance, keypoint-based systems can use a PnP solver and RANSAC to perform 3D object localization. Keypoint-based systems can estimate 6D poses using extracted keypoints (also referred to herein as sample points) and 3D prior models. FIG. 2 is a diagram illustrating an example of a keypoint-based object localization system using multiple keypoints. As shown in FIG. 2, an input image 201 captured by a front-facing camera of a tracking vehicle is input to a convolutional neural network (CNN) 202. The CNN 202 is designed to perform 2D object detection on the input image 201 to generate a 2D bounding box representing each detected object. The input image is shown as image 203 with the 2D bounding boxes generated by the CNN 202. The input image 201 can then be cropped using the 2D bounding boxes. For example, the region of the input image 201 inside each bounding box can be cropped to produce individually cropped images 204.

[0076] Each of the cropped images 204 is input to a separate CNN, shown in FIG. 2 as CNNs 1-c 206, where c has a value greater than or equal to 0 (where if c=0, a single CNN is present for processing a single detected object). For instance, a first cropped image can be input to a first CNN, a second cropped image can be input to a second CNN, and so on. The CNNs 1-c 206 are designed to process the cropped images 204 to detect 2D object keypoints. The input image is shown as image 207 with the 2D object keypoint detection results generated by the CNNs 1-c 206. Each of the keypoints is represented in the image 207 by a dot, with dots having different colors being associated with different vehicles.

[0077] The 2D object keypoint detection results and prior information 208 (including a 3D object model and associated 3D keypoints) are provided as input to an N-Point RANSAC-based pose estimation system 210. The N-Point RANSAC-based pose estimation system 210 can use a PnP solver along with RANSAC to remove outliers. The object localization results from the N-Point RANSAC-based pose estimation system 210 are shown in the input image shown as image 211.

[0078] Knowledge of the general camera geometry can be utilized for performing PnP and other techniques. In the pinhole camera model, a scene view is formed by projecting 3D points into the image plane using a perspective transformation. FIG. 3 is a diagram illustrating an example of the pinhole camera model. The geometric relationship between 2D coordinates on an image (including 2D image coordinate 301) and the corresponding 3D points (including 3D point coordinate 302) in the 3D real-world space (here, the camera coordinate system) are shown. A set of formulas 304 are also shown in FIG. 3, which can be used for projection of the 3D point coordinate 302 (and/or other 3D point coordinates on the 3D object 308) onto the image plane 306 of a 2D image. As shown, a 2D homogenous image coordinate x (or point) is equal to the product of a camera projection matrix P and a 3D homogenous world coordinate X (or point).

[0079] Coordinate transformations can be used to transform coordinates from one coordinate system to another coordinate system. In one illustrative example, a coordinate system and points within the coordinate system can be transformed onto another coordinate system using the following 4.times.4 transformation matrix comprised of a 3.times.3 rotational matrix R and a 3D translational vector t:

T = [ R , t 0 1 .times. 3 , 1 ] Equation ( 1 ) ##EQU00001##

[0080] FIG. 4A-FIG. 4C are diagrams illustrating an example of coordinate transformations from an object coordinate system to a camera coordinate system. FIG. 4A illustrates the origin 402 of the camera coordinate system (also referred to as the camera center), a 3D point X.sub.O from a plurality of 3D points in an object coordinate system, and the origin 404 of the object coordinate system. A transformation matrix T.sub.co is also shown. As illustrated in FIG. 4A, FIG. 4B, and FIG. 4C, the points (including the point X.sub.O) on the object coordinate system are transformed into points (including point X.sub.C) on the camera coordinate system. In some examples, the following equation can be used to compute the transformation:

X

c = [ X c 1 ] = T co X

o = T co [ X o 1 ] = [ R co X o + t co 1 ] Equation ( 2 ) ##EQU00002##

[0081] FIG. 4D is a diagram illustrating an example of a projection of the 3D point X.sub.O on the object coordinate system (from FIG. 4A-FIG. 4C) to a 2D point on the image. The 3D point X.sub.O on the object coordinate system can include a vertex on a 3D model of the object illustrated in the image. In some examples, the 3D point X.sub.O can projected to a 2D point on the image using the following equation:

PT co X

o = PT co [ X o 1 ] = P [ R co X o + t co 1 ] = P X

c = P [ X c 1 ] = KX c = [ x

y

z

] = x

x = [ u v ] = [ x

z

y

z

] Equation ( 3 ) ##EQU00003##

[0082] The relationship between a 6D pose vector and a 4.times.4 transformation matrix can be defined and used to determine the 6D pose of an object in an image. For instance, a 4.times.4 transformation matrix belongs to the Special Euclidean group SE(3) and has six degrees of freedom (6-DoF). The real space SE(3) is a six-dimensional manifold, with the six dimensions corresponding to the number of degrees of freedom of a free-floating rigid body in space. The 4.times.4 transformation matrix belongs to SE(3). The 4.times.4 transformation matrix can be converted into a 6D vector (belonging to se(3) and comprised of a 3D rotational vector and a 3D translational vector) using a logarithmic map, in which case the 4.times.4 transformation matrix is identical to the 6D vector under SE(3). For instance, se(3) denotes a group of the 6D vectors and SE(3) denotes a group of the 4.times.4 transformation matrices. An example of the 6-DoF pose parameter definition is as follows (with w.sub.x, w.sub.y, and w.sub.z being the 3D rotational vector and t.sub.x, t.sub.y, and t.sub.z being the 3D translational vector that make up the 6D pose parameter):

T = [ R t 0 1 .times. 3 1 ] .di-elect cons. SE ( 3 ) .xi. = [ .omega. t ] .di-elect cons. se ( 3 ) , where w = [ w x w y w z ] .di-elect cons. so ( 3 ) , t = [ t x t y t z ] .di-elect cons. 3 , Equations ( 4 ) ##EQU00004##

[0083] and the pose parameter conversion can be formulated using the transformation matrix T as follows:

ln(T).xi.

e.sup..xi.T Equations (5)

[0084] Continuing with the examples from FIG. 4A-FIG. 4D, FIG. 4E is a diagram illustrating an example of back-projection of a 2D point on an image to a camera coordinate system. Given a 2D image, the ray direction from each pixel can be determined. However, the depth (a distance from the origin 402 (or camera center) to a 3D point) is not known. Back-projection of a 2D pixel point (a directional vector) can be computed and used when performing PnP, as discussed below. For instance, back-projection of a 2D pixel point 406 can be computed as a directional vector as follows:

{right arrow over (r)}=K.sup.-1{circumflex over (x)} Equation (6)

[0085] FIG. 4F is another diagram providing an illustration of the pinhole camera model 400. The pinhole camera is conceptualized as a simple camera without a lens and with a single small aperture. Light rays pass through the aperture and project an inverted image on the opposite side of the camera. The image can be captured onto photographic film. The surface where film would be placed is commonly called the image plane 405 or the retinal plane. The aperture is called the pinhole or center of the camera (shown as O 410). The distance between the image plane 405 and O 410 is the focal length (f) 412 (shown as f 412).

[0086] The pinhole camera model 400 can be used to map three-dimensional, real-world coordinates to the two-dimensional coordinate system 407 of the image plane. For example, a point P 422 on an object 420 in the real world can have 3D coordinates [X, Y, Z]. The point P 422 can be projected or mapped to a point p 424, whose 2D coordinates within the image plane are [x, y]. Note that, for convenience and clarity, capital letter variables (e.g., [X, Y, Z]) will be used herein to express three-dimensional real-world coordinates, and lower-case variables (e.g., [x, y]) will be used to express two-dimensional coordinates within the image plane 405.

[0087] As illustrated in this example, the pinhole camera model includes three coordinate references systems: the three-dimensional real-world coordinate system 403 centered at O 410, the three-dimensional (3D) camera reference system 408 [i, j, k] centered at O 410, and the two-dimensional image plane reference system 407, centered at one corner of the image plane 405. Transforming the 3D location of P 422 to P’=[X’, Y Z’] in the 3D camera reference system 408 can be accomplished using the following equation;

P ’ = [ X ’ Y ’ Z ’ 1 ] = [ R t 0 1 ] [ X Y Z 1 ] Equation ( 7 ) ##EQU00005##

[0088] In this equation, R includes the rotational parameters of the camera (e.g., pitch, yaw, and/or roll), and t is a translation vector (e.g., the physical location of the camera). Rotation and translation are intrinsic parameters of the camera. The rotational parameters R can be expressed using the following equation:

R = [ 1 0 0 0 cos .gamma. - sin .gamma. 0 sin .gamma. cos .gamma. ] [ cos .beta. 0 sin .beta. 0 1 0 - sin .beta. 0 cos .beta. ] [ cos .alpha. - sin .alpha. 0 sin .alpha. cos .alpha. 0 0 0 1 ] Equation ( 8 ) ##EQU00006##

[0089] In the above equation, .alpha. is the yaw (horizontal rotation), .beta. is the pitch (up-and-down rotation), and .gamma. is the roll (side-to-side rotation). The pitch, roll, and yaw relative to a camera can be conceptualized as the yaw being the camera’s horizontal rotation relative to the ground (e.g., left-to-right relative to the horizontal axis), the pitch being the camera’s vertical rotation relative to the ground (e.g., up and down relative to the horizontal axis), and the roll being the camera’s side-to-side rotation relative to the horizon (e.g., side-to-side relative to the horizontal axis). The translation vector t can be expressed as:

t = [ X T Y T Z T ] Equation ( 9 ) ##EQU00007##

[0090] The camera’s intrinsic parameters, K, can next be used to map P’ from the 3D camera reference system 408 to the image plane 405. This mapping is also referred to as a projective transformation. The camera’s intrinsic parameters can be expressed as follows:

K = [ f x S x c 0 f y y c 0 0 1 ] Equation ( 10 ) ##EQU00008##

[0091] In the above matrix, f.sub.x and f.sub.y are the focal length of the camera along the x and y axis, respectively; (x.sub.c, y.sub.c) is the center of the image plane 405; and S is a skew factor. Skew occurs when the 3D camera reference system 408 is not precisely perpendicular to the image plane 405.

[0092] Using the camera matrix, the 2-D location of p can now be determined from the 3D coordinates of P’ in the 3D camera reference system 408, using the following equation:

p [ x y 1 ] = [ f x s x 0 0 f y y 0 0 0 1 ] [ X ’ Y ’ Z ’ ] Equation ( 11 ) ##EQU00009##

[0093] The above group of equations (7)-(11) provide a mapping for a point P 422 in the real world to a point p 424 in the image plane. A mapping from the 2D image plane to the 3D real world coordinate system (for example, to identify where an object in a video frame is positioned) can also be accomplished using these equations, when the extrinsic parameters are known.

[0094] FIG. 5-FIG. 8D are diagrams illustrating a perspective-n-point (PnP) technique. PnP can be used to estimate the 6-DoF pose parameter (defined by a transformation matrix T), given a set of n 3D points in the world space (or the object coordinate system) and the 2D projection points in the image that correspond to the set of n 3D points. The transformation matrix T defining the 6-DoF pose parameter includes a 3D rotational vector (including angles for pitch along the transverse axis, roll along the longitudinal axis, and yaw along the normal axis) and a 3D translational vector (including translation in the horizontal (x) direction, vertical (y) direction, and depth (z) direction)). As noted above, the yaw is the camera’s horizontal rotation relative to the ground (e.g., left-to-right relative to the horizontal axis), the pitch is the camera’s vertical rotation relative to the ground (e.g., up and down relative to the horizontal axis), and the roll is the camera’s side-to-side rotation relative to the horizon (e.g., side-to-side relative to the horizontal axis).

[0095] When performing PnP, it can be assumed that the camera is calibrated (the intrinsic parameters K are known). PnP requires three or more point correspondences (between three 3D points and three corresponding 2D projection points). For example, as shown in FIG. 5, a first 3D point X.sub.O,1 and corresponding 2D projection point x.sub.1, a second 3D point X.sub.O,2 and corresponding 2D projection point x.sub.2, and a third 3D point X.sub.O,3 and corresponding 2D projection point x.sub.3 can be identified for use in the PnP algorithm. A transformation matrix T.sub.co is also shown in FIG. 5.

[0096] In some cases, there can be outliers in the 2D-to-3D point correspondences. For example, as shown in FIG. 6, the point correspondence between the second 3D point X.sub.O,2 and the corresponding 2D projection point x.sub.2 can be considered an outlier. When outliers are present, they can produce an incorrect pose estimate. It can thus be beneficial to filter out any outliers that are present. One illustrative technique that can be used to filter out outliers is Random sample consensus (RANSAC). RANSAC is an algorithm that can be used to estimate (or fit) a model in the existence of outlier point correspondences.

[0097] FIG. 7 illustrates an example of a pose estimation using RANSAC and PnP algorithms. For example, given a set of 2D-3D sample point (also referred to as keypoints) correspondences, C, n sample point (or keypoint) correspondences are sampled. Such sampling can be denoted as a set of sample point correspondences {(x.sub.1, X.sub.1), … , (x.sub.n, X.sub.n)|(x.sub.i, X.sub.i).di-elect cons.C}. A pose hypothesis T.sub.cand can be computed using a PnP technique. Reprojection errors can be computed for all the sample point correspondences and the sample points with reprojection errors that are within an outlier threshold value are regarded as inliers. This process is repeated N times and the pose with the maximum number of inliers is selected as the best pose estimate. The maximum number of iteration can be reduced by considering a inlier probability in some cases. The pose estimate can be optimized by minimizing the reprojection errors of the inlier sample points.

[0098] FIG. 8A illustrates a first step of RANSAC. Given m 2D-3D point correspondences, the RANSAC algorithm randomly samples three 2D-3D point correspondences among the m correspondences. For example, as shown in FIG. 8A, three points (n=3, labeled X.sub.O,1, X.sub.O,2, and X.sub.O,3) are sampled out of five total 2D-3D point correspondences (m=5). FIG. 8B illustrates a second step of RANSAC, where a pose parameter T.sub.co is computed using the PnP solver (with n=3 in the example of FIG. 8B), as described above. FIG. 8C and FIG. 8D illustrates a third step of RANSAC. For example, the algorithm can check whether each point correspondence is an inlier or an outlier. A point correspondence can be determined to be an inlier if a reprojection error is within an outlier threshold value. The outlier threshold value can be set to any suitable value, and in some cases is a user-defined value. The threshold value can vary depending on the camera configuration (e.g. image resolution, a focal length, the size of the 2D bounding box, etc.). In one illustrative example, the threshold value can be defined as 0.03.about.0.1 times the width and/or height of the 2D object bounding box (e.g., outlier threshold=0.03*BB.sub.Width or outlier threshold=0.03*BB.sub.Height, with BB referring to the bounding box). The reprojection error can include any suitable distance or error metric, such as Euclidean distance, Manhattan distance, or other suitable metric. For instance, the reprojection error can be defined as a Euclidean distance between the 2D image point and the projection point of the corresponding 3D point onto the image using the computed pose parameter T (e.g., T.sub.co), as follows:

e reproj ( x , X | T ) = x - x _ = ( u - u _ ) 2 + ( v - v _ ) 2 Equation ( 12 ) where x _ = [ u _ v _ ] = [ x z y z ] and [ x y z ] = PT X ^ Equation ( 13 ) ##EQU00010##

[0099] The projection error shown in FIG. 8A is a small projection error (below the outlier threshold) between the actual 2D image point x.sub.1 and the projection point 802 of the 3D point X.sub.O,1, in which case the projection point 802 of the 3D point X.sub.O,1 is considered an inlier. The projection error shown in FIG. 8B is a large projection error (above the outlier threshold) between the actual 2D image point x.sub.1 and the projection point 804 of the 3D point X.sub.O,1, in which case the projection point 802 of the 3D point X.sub.O,1 is considered an outlier.

[0100] In a fourth step of RANSAC, the first three steps can be repeated N times, and the pose parameter T with the maximum number of inliers among the N pose parameter candidates can be selected. In some cases, post-processing can be performed in the PnP technique. For example, after filtering out the outliers, the best pose parameter can be optimized using all the inliers. The optimal pose parameter minimizing the reprojection errors of all the inliers can be found as:

T co = arg min T co i e reprol 2 ( x i , X i | T co ) Equation ( 14 ) ##EQU00011##

[0101] Various optimization methods can be used, such as the Gauss-Newton method, the Levenberg-Marquardt method, among others.

[0102] Inferring 3D information (e.g., pose and shape) of an object from a single two-dimensional image can be challenging, due at least to the problem being inherently ill-posed due to the lack of geometric cues in a single image. Keypoint-based techniques have been shown to achieve superior performance in terms of localization accuracy as opposed to other methods. As described above, keypoint-based approaches assume an object shape is defined by a 3D model (which can be a parametric deformable model) that is statistically modeled from prior shape models (e.g., active shape models (ASM)). Such approaches simultaneously optimize the 6-DoF pose and deformable model parameters by minimizing re-projection errors for 2D-3D point correspondences of the object. Most keypoint-based methods initialize an object pose by solving a perspective-n-point (PnP) problem. In cases for which the data contains many outliers, RANSAC can be used to filter out outlier correspondences and to initialize an object pose.

[0103] In general, the computational complexity of the RANSAC process for the PnP problem has a complexity of O(m.sup.n) (where m is the number of 2D-3D point correspondences), and at least O(m.sup.3) because at least three points (n=3) need to be sampled for algebraic pose computation (e.g., as at least three points should be sampled to initialize a pose candidate in the RANSAC process). Accordingly, limitations of existing keypoint-based pose determination systems that use PnP solvers include that they require at least three points to compute a pose candidate, in which case the pose computation is repeated at most m.sup.3 times (e.g., usually at least hundreds of times) in the RANSAC process. The high computational complexity of keypoint-based techniques can make real-time performance difficult in real-world scenarios. In many cases, the high complexity of such techniques reduces the maximum number of objects that can be localized (with pose and/or shape determinations) in a real-time system, imposing a severe strain on devices and systems that estimate multiple object poses in real-time scenarios (e.g., in manufacturing systems, autonomous driving systems of autonomous vehicles and/or aviation systems, robotics systems, augmented reality (AR) systems, virtual reality (VR) systems, among others).

[0104] As noted above, systems and techniques are described herein for determining poses of objects in images using point-based object localization techniques. The systems and techniques described herein provide an efficient way to estimate a 6-degrees-of-freedom (6-DoF) object pose and the shape of the object from an image including the object (e.g., from a single image of the object). For instance, the systems and techniques can solve the PnP problem noted above in a more efficient manner as compared to existing point-based localization techniques. In some examples, the systems and techniques can implement a keypoint-based solution to determine an initial pose (and in some cases the shape) of an object in an image using one sample point (e.g., a 1-point RANSAC-based method) from the input data.

[0105] For example (e.g., using the 1-point RANSAC-based method), one sample point from the input data is needed for the systems and techniques to compute an object pose hypothesis. In addition to the one point needed to compute a pose candidate, additional information can also be used. For instance, the additional information can include a 2D bounding box of an object (from object detection) and a pitch angle of the camera. If it is assumed that an object is on the ground and the relationship (e.g., the pitch angle) between the ground and a camera used to capture an image is pre-calibrated, the pose parameterization for the PnP problem is reduced to 1-DoF parameterization (a yaw angle and/or a depth of an object).

[0106] In some examples, a pose hypothesis can be computed for each point correspondence sample. In such examples, the best sample consensus can be determined using a RANSAC process (or other suitable process) with a computational complexity of O(m) (with m being the number of points sampled in the RANSAC process). Because only one sample point is needed, the number of required iterations of the RANSAC process the systems and techniques described herein is at most m times or less than m times, reducing the computational complexity of performing the point-based objection localization from O(m.sup.3) or greater (up to O(m.sup.n)), as noted above, to O(m).

[0107] FIG. 9 is a diagram illustrating an example of a system 900 for determining poses (and in cases shapes and/or sizes) of objects in images. Given a 3D model (e.g., including a set of vertices) out of one or more 3D models 907, the system 900 can be used to estimate a 6D pose hypothesis (and in some cases the shapes and/or sizes) of an object (or multiple objects) in an environment using a sample point (also referred to as a keypoint sample) and a two-dimensional bounding box (2D bounding box). The 2D bounding box can be generated by the object detection engine 902 from an input image (out of the one or more images 901) of the environment.

[0108] The system 900 includes various components, including an object detection engine 902, a sample point selection engine 904, a vector determination engine 906, an object orientation and depth determination engine 908, and a pose determination engine 910. In some cases, the system 900 can include one or more cameras (not shown) that can be used to capture the one or more images 901. In some cases, the system 900 can receive, retrieve, or otherwise obtain the one or more images 901 from other sources (e.g., from another device, from storage, from a network-based location such as a server over the Internet, or other source). The components of the system 900 can include software, hardware, or both. For example, in some implementations, the components of the system 900 can include and/or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or other suitable electronic circuits), and/or can include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The software and/or firmware can include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the computing device implementing the system 900.

[0109] While the system 900 is shown to include certain components, one of ordinary skill will appreciate that the system 900 can include more or fewer components than those shown in FIG. 9. For example, the system 900 can include, or can be part of a computing device that includes, one or more input devices and one or more output devices (not shown). In some implementations, the system 900 may also include, or can be part of a computing device that includes, one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and/or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and/or other processing devices) in communication with and/or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor for each wireless interface) for performing wireless communications, one or more wired interfaces (e.g., a serial interface such as a universal serial bus (USB) input, a lightening connector, and/or other wired interface) for performing communications over one or more hardwired connections, and/or other components that are not shown in FIG. 9.

[0110] As noted above, the system 900 can be implemented by and/or included in a computing device or other object. In some cases, multiple computing devices can be used to implement the system 900. For example, a computing device used to implement the system 900 can include a computer or multiple computers that are part of a device or object, such as a vehicle, a robotic device, a surveillance system, and/or any other computing device or object with the resource capabilities to perform the techniques described herein. In other examples, a computing device used to implement the system 900 can include a personal computer, a tablet computer, a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a smart watch, a virtual reality headset, an augmented reality headset, and/or other wearable device), a server or multiple servers (e.g., in a software as a service (SaaS) system or other server-based system), and/or any other computing device with the resource capabilities to perform the techniques described herein.

[0111] In some implementations, the system 900 can be integrated with (e.g., integrated into the software, added as one or more plug-ins, included as one or more library functions, or otherwise integrated with) one or more software applications, such as a modeling software application, an autonomous driving or navigation software application, or suite of software applications. The one or more software applications can be installed on the computing device or object implementing the system 900. The software application can be a mobile application installed on a mobile device (e.g., a mobile phone, such as a smartphone, a tablet computer, a wearable device, or other mobile device), a desktop application installed on a desktop computer, a web-based application that can be accessed using a web browser or other application, or other software application. In some implementations, the system 900 can be implemented in a suite of software applications.

[0112] FIG. 10 is a diagram illustrating a high level overview of a process that can be performed by the system 900 shown in FIG. 9. The process of FIG. 10 illustrates a 1-Point RANSAC-based pose estimation technique. For example, given a 3D model 1007 of a vehicle 1011 (from the one or more 3D models 907) including a set of vertices, the system 900 can compute a 6-dimensional (e.g., having 6-degrees-of-freedom or 6-DoF) pose hypothesis of the object in an image 1001 using a sample point 1014 (selected by sample point selection engine 904) and a 2D bounding box 1012 shown in the image 1001. The 2D bounding box 1012 can be generated by the object detection engine 902. As shown in FIG. 10, and as described in greater detail below, using information determined by the vector determination engine 906, the object orientation and depth determination engine 908 determines a depth 1017 and a yaw angle 1016 relative to the camera center 1018 of the camera used to capture the image. The yaw angle 1016 is the angle between a direction (represented by directional vector 1013) of the sample point 1014 from the camera center 1018 and the forward direction (represented by directional vector 1015) of the vehicle 1011. The forward direction of the camera is perpendicular to the image plane of the camera. As can be seen from FIG. 10, the yaw angle 1016 represents the orientation of the object relative to the direction (shown by directional vector 1013) of the sample point 1014. The pose determination engine 910 can determine the best 6D pose hypothesis from among a plurality of 6D pose hypotheses using RANSAC or other suitable technique. The final object localization result for a detected object includes best 6D pose hypothesis selected using RANSAC or other technique. The 6D pose hypothesis is shown in image 1019, which is the input image with object localization result illustrated thereon.

[0113] In some aspects, prior-known information can be used to reduce the number of points required for computing an object pose hypothesis. For example, it can be assumed that the 3D model 1007 (and the 2D-3D sample point or keypoint correspondences of the 3D model) and a pitch angle (the angle from the camera center to the ground plane or horizontal plane) are already known as prior information. With respect to the pitch angle, it can be assumed that the tilt of the camera to the ground where an object of interest is placed is given as the pitch angle .theta..sub.p. Given the prior information, a 2D bounding box is obtained by the object detection engine 902 and one sample point (also referred to as a keypoint sample) is obtained by the sample point selection engine 904. For instance, from the pitch angle .theta..sub.p, a rotation transformation R.sub.cg.di-elect cons.SO(3) from the ground to the camera is defined as R.sub.cg=e.sup..omega..sup.p, where .omega..sub.p=[-.theta..sub.p, 0,0].sup.T .di-elect cons.so(3) and e:so(3)SO(3) is an exponential map. In one example, if an object is fronto-parallel to the image plane, R.sub.cg=I.sub.3 where I.sub.3 is a 3.times.3 identity matrix. Further, in addition to the 3D model 1007 of the object and its 2D-3D sample point (or keypoint) correspondences, which are provided as input by default in the PnP problem, the 2D bounding box of the detected object is provided as input to the vector determination engine 906. The vector determination engine 906 can project the rays of both sides of the 2D bounding box on a horizontal plane (e.g., a road surface). The sample point can be redefined as an origin in the object coordinate system of the 3D model 1007.

[0114] The object orientation and depth determination engine 908 can compute the yaw angle 1016 and the depth 1017 of the sample point relative to the direction of the sample point (shown by directional vector 1013). In one illustrating example, the object orientation and depth determination engine 908 can compute the yaw angle 1016 and the depth 1017 of the sample point relative to the direction of the sample point by solving an equation which is derived from the Sine rule or other rule. For instance, using the prior information, the problem can be redefined as aligning the projection of a 3D bounding box of the object into the back-projected rays of both sides of the 2D bounding box in a bird-eye view (BEV) of the object (i.e., from directly above the object), after which the yaw angle 1016 of the vehicle 1011 and the depth 1017 of the sample point 1014 are computed, as shown in FIG. 10. As described in more detail below, an equation can be formulated (e.g., based on the Sine rule) that computes the pose of the object of interest by 1-DOF parameterization. As noted above and described further below, a unique pose hypotheses per sample point can be obtained by solving the equation. The RANSAC process (e.g., the algorithm shown in FIG. 7) or other suitable technique can be used to determine the optimal pose parameter with the maximum number of inliers among those hypotheses. In some cases, the determined pose can then be optimized.

[0115] Returning to FIG. 9, the one or more images 901 can include still images or video frames. The one or more images 901 each contain images of a scene or environment. An example of an image 903 is shown in FIG. 9. The image 903 illustrates an example of an image captured by a camera of a tracking vehicle (as an example of a tracking object). The image 903 can include one or multiple target vehicles (as an example of target objects). When video frames are captured, the video frames can be part of one or more video sequences. As noted above, in some cases, the one or more images 901 can be captured by one or more cameras included in the system 900 or included outside of the system 900 in the computing device comprising the system 900. In some cases, the one or more images 901 can be stored in a storage device (not shown), and the one or more images 901 can be retrieved or otherwise obtained by the system 900 from the storage device. The one or more images 901 can be raster images composed of pixels (or voxels) optionally with a depth map, vector images composed of vectors or polygons, or a combination thereof. The one or more images 901 may include one or more two-dimensional representations of a scene along one or more planes (e.g., a plane in a horizontal or x-direction and a plane in a vertical or y-direction), or one or more three dimensional representations of the scene.

[0116] The object detection engine 902 can obtain and process the one or more images 901 to detect and/or track one or more objects in the one or more images 901. The object detection engine 902 can generate and output bounding regions (e.g., a bounding region 905) as detected and tracked objects. In some cases, the object detection engine 902 can determine a classification (referred to as a class) or category of each object detected in an image. As noted above, a bounding region can be generated for identifying each object detected in an image. For instance, an object can be detected in an image, and a bounding region and in some cases a class label (also referred to as a category label) can be output by the object detection engine 902 for the detected object. The bounding region can be used by other components of the system 900 to identify a region of the image that includes the detected object (e.g., bounding region identifying a target vehicle). In some cases, the dimensions of a bounding region (e.g., the width and/or height, the length of a diagonal, such as from a bottom-left corner to a top-right corner of from a top-left corner to a top-right corner, or other dimensions) can also be output by the object detection engine 902. A bounding region assigned to a detected object can include a bounding box, a bounding circle, a bounding ellipse, or any other suitably-shaped region representing a detected object. While examples are described herein using bounding boxes for illustrative purposes, the techniques and systems described herein can also apply using other suitably shaped bounding regions. A bounding box associated with a detected object can have a rectangular shape, a square shape, or other suitable shape. In some cases, the object detection engine 902 can output multiple classes for a detected object, along with a confidence score indicating a confidence that the object belongs to each of the classes (e.g., a confidence score of 0.85 that the object is a car, a confidence score of 0.14 that the object is a truck, and a confidence score of 0.01 that the object is a motorcycle).

[0117] Any suitable object detection and/or classification technique can be performed by the object detection engine 902. In some cases, the object detection engine 902 can use a machine learning based object detector, such as using one or more neural networks. For instance, a deep learning-based object detector can be used to detect and classify objects in the one or more images 901. One illustrative example of a neural network based detector is described below with respect to FIG. 20. In another illustrative example, a Cifar-10 neural network based detector can be used to perform object classification to classify objects. In some cases, the Cifar-10 detector can be trained to classify only certain objects, such as vehicles. Further details of the Cifar-10 detector are described below with respect to FIG. 21.

[0118] Another illustrative example of a deep learning based detector is a fast single-shot object detector (SSD) including a neural network and that can be applied for multiple object categories. A feature of the SSD model is the use of multi-scale convolutional bounding box outputs attached to multiple feature maps at the top of the neural network. Such a representation allows the SSD to efficiently model diverse bounding box shapes. It has been demonstrated that, given the same VGG-16 base architecture, SSD compares favorably to its state-of-the-art object detector counterparts in terms of both accuracy and speed. An SSD deep learning detector is described in more detail in K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, abs/1409.1556, 2014, which is hereby incorporated by reference in its entirety for all purposes. Further details of the SSD detector are described below with respect to FIG. 22A-FIG. 22C.

[0119] Another illustrative example of a deep learning-based detector that can be used to detect and classify objects in the one or more images 901 includes the You only look once (YOLO) detector. The YOLO detector, when run on a Titan X, processes images at 40-90 fps with a mAP of 78.6% (based on VOC 2007). A YOLO deep learning detector is described in more detail in J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” arXiv preprint arXiv:1506.02640, 2015, which is hereby incorporated by reference in its entirety for all purposes. Further details of the YOLO detector are described below with respect to FIG. 23A-FIG. 23C. While certain object detectors are provided as illustrative examples, one of ordinary skill will appreciate that any other suitable object detection and classification can be performed by the object detection engine 902.

[0120] The object detection engine 902 can provide a 2D bounding box for an object detected in an image to the sample point selection engine 904. In one illustrative example, the 2D bounding box can be represented by at least two points of the bounding box, such as the top-left-most point and the bottom-right-most point of the bounding box, the top-right-most point and the bottom-left-most point of the bounding box, a center point of the bounding box, or other points. The following can be used to represent a bounding box using the top-left-most point and the bottom-right-most point of the bounding box:

(x.sub.lt,x.sub.rb)=([u.sub.lt,v.sub.lt].sup.T ,[u.sub.rb,v.sub.rb].sup.T)

[0121] In some cases, the number of sample points (keypoints) and their locations can be defined by a user in advance, as shown in FIG. 25. 3D sample points can be manually annotated by a user in advance and used as prior information. 2D sample points can be obtained by CNNs (e.g., the CNNs 1-c 206 from FIG. 2) implemented by the sample point selection engine 904. The CNNs are trained using image datasets with 2D keypoint annotations, which can be annotated manually by users. Once the CNNs are trained using the image datasets, given an input image, the CNNs provide a 2D coordinate and a confidence value of each sample point (or keypoint). If the confidence value is more than a threshold value (e.g., a threshold value of 0.6, 0.7, 0.8, or other threshold value), the sample point is regarded as a detected sample point. For instance, referring FIG. 25 as an illustrative example, if 53 sample points are defined, the CNNs can provide 53 2D coordinates and 53 confidence values from a cropped image of an object.

[0122] For the detected object detected by the object detection engine 902 in an image, the sample point selection engine 904 can select a sample point (or keypoint) from the image. In some examples, the sample point selection engine 904 can randomly sample a sample point from within the region of the image that is contained in the 2D bounding box (which includes the pixels of the image representing the object). In one illustrative example of random sampling, assuming that 20 sample points (or keypoints) are detected among 53 sample points, one keypoint can be randomly sampled among the 20 sample points.

[0123] A sample point x.sub.K selected by the sample point selection engine 904 can be represented as follows:

X.sub.K=[u.sub.K,v.sub.K].sup.T

[0124] FIG. 11A and FIG. 11B are diagrams illustrating an example of a relationship between a 3D bounding box 1105 of a model obtained for an object detected in an image, a selected sample point 1114, a 2D bounding box generated for the object, the forward direction (represented by 3D directional vector denoted as V.sub.C) of the camera used to capture the image, and the forward direction (represented by 3D directional vector denoted as V.sub.O) of the object. The 2D bounding box of the object is represented in FIG. 11A by the 3D directional vectors V.sub.L and V.sub.R, which are back-projected rays of the left (L) and right (R) sides of the 2D bounding box in a bird-eye view (BEV) of the object. The camera is represented in FIG. 11A by camera center 1118.

[0125] The vector determination engine 906 can compute the various 3D directional vectors shown in FIG. 11A. For example, the forward direction (3D directional vector V.sub.C) of the camera center 1118 can be determined as follows:

V.sub.C=K.sup.-1[p.sub.u,p.sub.v,1].sup.T Equation (15)

[0126] The back-projection rays (3D directional vectors V.sub.L and V.sub.R) of the top-left point and the bottom-right points of the 2DBB shown in FIG. 11A can be determined as follows:

V.sub.L=K.sup.-1[p.sub.u,p.sub.v,1].sup.T Equation (16)

V.sub.R=K.sup.-1{circumflex over (x)}.sub.rb Equation (17)

[0127] The back-projection ray of the sample point 1114 (shown in FIG. 11A as 3D directional vector V.sub.K) can be determined as follows:

V.sub.K=K.sup.-1{circumflex over (x)}.sub.K Equation (18)

……
……
……