Apple Patent | Automatic measurements based on object classification

Patent: Automatic measurements based on object classification

Drawings: Click to check drawins

Publication Number: 20210224516

Publication Date: 20210722

Applicant: Apple

Abstract

Various implementations disclosed herein include devices, systems, and methods that obtain a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, generate a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, classify the object based on the 3D bounding box and the 3D semantic data, and display a measurement of the object, where the measurement of the object is determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.

Claims

1. A method comprising: at an electronic device having a processor: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data; generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation; classifying the object based on the 3D bounding box and the 3D semantic data; and displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.

2. The method of claim 1, wherein classifying the object based on the 3D bounding box and the 3D semantic data comprises: determining, based on the 3D semantic data, a class of the 3D bounding box using an object classification neural network; and classifying the object corresponding to the 3D bounding box based on the classification of the 3D bounding box.

3. The method of claim 1, wherein a first class-specific neural network is trained to determine specific points on a first classification of objects that are used for measurements of objects in the first classification.

4. The method of claim 3, wherein a second class-specific neural network is trained to determine specific points on a second classification of objects that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects.

5. The method of claim 4, wherein the measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects.

6. The method of claim 1, wherein the 3D bounding box is a refined 3D bounding box, wherein generating a refined 3D bounding box comprises: generating a proposed 3D bounding box for an object using a first neural network; and generating the refined 3D bounding box by inflating the proposed 3D bounding box based on a bounding box inflation scale, identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and refining the proposed 3D bounding box based on the identified features.

7. The method of claim 6, wherein the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object.

8. The method of claim 6, wherein the second neural network identifies the features of the object based on the 3D semantic data associated with the object.

9. The method of claim 6, wherein a third neural network is trained to: refine the accuracy of the identified features from the second neural network based on the 3D semantic data associated with the object and the light intensity image data; and output a further refined 3D bounding box based on the refined accuracy of the identified features from the second neural network.

10. The method of claim 1, wherein the 3D bounding box provides a location, an orientation, and a shape of an identified object.

11. The method of claim 1, wherein the 3D representation comprises a 3D point cloud and the associated 3D semantic data includes semantic labels associated with at least a portion of 3D points within the 3D point cloud.

12. The method of claim 11, wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment.

13. A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the device to perform operations comprising: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data; generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation; classifying the object based on the 3D bounding box and on the 3D semantic data; and displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.

14. The device of claim 13, wherein classifying the object based on the 3D bounding box and the 3D semantic data comprises: determining, based on the 3D semantic data, a class of the 3D bounding box using an object classification neural network; and classifying the object corresponding to the 3D bounding box based on the classification of the 3D bounding box.

15. The device of claim 13, wherein a first class-specific neural network is trained to determine specific points on a first classification of objects that are used for measurements of objects in the first classification.

16. The device of claim 15, wherein a second class-specific neural network is trained to determine specific points on a second classification of objects that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects.

17. The device of claim 16, wherein the measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects.

18. The device of claim 13, wherein the 3D bounding box is a refined 3D bounding box, wherein generating a refined 3D bounding box comprises: generating a proposed 3D bounding box for an object using a first neural network; and generating the refined 3D bounding box by inflating the proposed 3D bounding box based on a bounding box inflation scale, identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and refining the proposed 3D bounding box based on the identified features.

19. The device of claim 18, wherein the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data; generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation; classifying the object based on the 3D bounding box and on the 3D semantic data; and displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application Ser. No. 62/962,494 filed Jan. 17, 2020, which is incorporated herein in its entirety.

TECHNICAL FIELD

[0002] The present disclosure generally relates to generating geometric representations of objects in physical environments, and in particular, to systems, methods, and that generate geometric representations based on information detected in physical environments.

BACKGROUND

[0003] Object detection and accurate measurements of objects play an important role in designing, understanding, and remodeling indoor spaces and generating accurate reconstructions. There are numerous hurdles to providing computer-based systems to automatically generate object measurements based on sensor data. The sensor data obtained regarding a physical environment (e.g., images and depth data) may be incomplete or insufficient to provide accurate measurements. As another example, images and depth data typically lacks semantic information and measurements generated without such data may lack accuracy. Existing techniques do not allow for automatic, accurate, and efficient generation object measurements using a mobile device, for example, based on a user capturing photos or video or other sensor data while walking about in a room. Moreover, existing techniques may fail to provide sufficiently accurate and efficient measurements in real time (e.g., immediate measurement during scanning) environments.

SUMMARY

[0004] Various implementations disclosed herein include devices, systems, and methods that generate measurements using three-dimensional (3D) representations of a physical environment. The 3D representations of the physical environment may be generated based on sensor data, such as image and depth sensor data. The generation of object detection and object measurements is facilitated in some implementations using semantically-labelled 3D representations of a physical environment. Some implementations perform semantic segmentation and labeling of 3D point clouds of a physical environment.

[0005] According to some implementations, measurements of objects (e.g., furniture, appliances, etc.) detected in a physical environment may be generated using multiple, different techniques. In some implementations, an object is measured by first generating a 3D bounding box for the object based on the depth data, refining the bounding box using various neural networks and refining algorithms described herein, and acquiring measurements based on the refined bounding box and the associated 3D data points for the respective bounding box. In some implementations, an object is measured using different types of measurements for different object types using machine learning techniques (e.g., neural networks). For example, different types of measurements may include a seat height for chairs, a display diameter for TVs, a table diameter for round tables, a table length for rectangular tables, and the like.

[0006] Some implementations disclosed herein may achieve various advantages by measuring an object using multiple, class-specific machine learning models (e.g., class specific neural networks). In some implementations, multiple machine learning models are trained to determine different measurements for different object classes. For example, one model may be trained and used to determine measurements for chair type objects (e.g., determining a seat height, arm length, etc.) and another model may be trained and used to determine measurements for TV type objects (e.g., determining a diagonal screen size, greatest TV depth, etc.). Such class-specific measurements may provide more information than the simple length, width, and height of the bounding boxes that may be identified for each object.

[0007] In some implementations, an auto-measure technique uses slices or horizontal planes to identify surfaces (e.g., a seat top) and uses those detected surfaces to provide measurements (e.g., distance of the seat to the ground).

[0008] Some implementations involve an exemplary method of providing a measurement for an object within a physical environment. The exemplary method involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with light intensity images. In some implementations, the 3D representation may be associated with 3D semantic data. For example, an algorithm may perform semantic segmentation and labeling of 3D point clouds points.

[0009] The exemplary method further involves generating a 3D bounding box corresponding to objects in the physical environment based on the 3D representation. For example, the 3D bounding box may provide location, pose (e.g., orientation and location), and shape of each piece furniture and appliance in a room or portion of a room. A 3D bounding box may be refined using an inflate and cut technique. In some implementations, generating a refined 3D bounding box includes generating a proposed 3D bounding box for an object using a first neural network and generating a refined 3D bounding box by inflating the proposed 3D bounding box based on a bounding box inflation scale (e.g., inflate the bounding box by 10%), identifying features of the object of the inflated proposed bounding box using a second neural network, and refining the proposed 3D bounding box based on the identified features. In some implementations, the first neural network generates the proposed bounding box based on the 3D semantic data associated with the object. In some implementations, the second neural network identifies the features of the object based on the 3D semantic data associated with the object. In some implementations, a third neural network is trained to refine the accuracy of the identified features from the second neural network based on the 3D semantic data associated with the object and light intensity image data (e.g., RGB data) obtained during the scanning process, and output a further refined 3D bounding box based on the refined accuracy of the identified features.

[0010] The exemplary method further involves classifying the objects based on the 3D bounding box and the 3D semantic data. For example, providing a class designation or label for each generated 3D bounding box. In some implementations, classifying an object based on the 3D bounding box and the 3D semantic data includes determining, based on the 3D semantic data, a class of the 3D bounding box using an object classification neural network, and classifying the object corresponding to the 3D bounding box based on the classification of the 3D bounding box. In some implementations, a first class-specific neural network is trained to determine specific points on a first classification of objects (e.g., chairs) that are used for measurements of objects in the first classification. For example, an arm length and a seat height of a chair could be determined. In some implementations, a second class-specific neural network is trained to determine specific points on a second classification of objects (e.g., tables) that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects. For example, table height, table top dimensions specific to round or rectangular table tops. The measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects. For example, a chair may include more or at least different measurements than a table or a TV.

[0011] The exemplary method further involves displaying a measurement (e.g., arm length, seat height, TV diameter, etc.) of the object, where the measurement of the object is determined using one of a plurality of class-specific neural networks selected based on the classifying of the object. For example, a first network is trained to determine specific points on a chair that are used for chair measurements, and a second network trained to determine different points on tables that are used for table measurements. In use, a user may scan a room with a device (e.g., a smartphone) and the processes described herein would identify an object (e.g., a chair), and provide specific measurements for the object (e.g., chair height, seat height, based width, etc.). In some implementations, the measurements may be automatically displayed on the user device overlaid or next to the object. In some implementations, the measurements may be provided after some type of user interaction with the identified object. For example, the user may be shown a transparent bounding box surrounding an object, and the user may select or click on the bounding box and the measurements would then be displayed.

[0012] In some implementations, the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud. Additionally, in some implementations, the semantic labels identify walls, wall attributes (e.g., doors and windows), objects, and classifications of the objects of the physical environment.

[0013] Some implementations of this disclosure involve an exemplary method of providing measurement data for an object within a physical environment. The exemplary method first involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images. In some implementations, the 3D representation is associated with 3D semantic data. For example, algorithms may be used for semantic segmentation and labeling of 3D point clouds of indoor scenes.

[0014] The exemplary method further involves generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation. For example, a 3D bounding box may provide location, pose (e.g., orientation and location), and shape of each piece furniture and appliance in the room. Bounding boxes may be refined using an inflate and cut technique. In some implementations, generating a refined 3D bounding box includes generating a proposed 3D bounding box for an object using a first neural network and generating the refined 3D bounding box by inflating the proposed 3D bounding box based on a bounding box inflation scale (e.g., inflate the bounding box by 10%), identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and refining the proposed 3D bounding box based on the identified features. In some implementations, the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object. In some implementations, the second neural network identifies the features of the object based on the 3D semantic data associated with the object. In some implementations, a third neural network is trained to refine the accuracy of the identified features from the second neural network based on the 3D semantic data associated with the object and light intensity image data (e.g., RGB data) obtained during the scanning process, and output a further refined 3D bounding box based on the refined accuracy of the identified features.

[0015] The exemplary method further involves determining a class of the object based on the 3D semantic data. For example, providing a class designation or label for the generated 3D bounding box. In some implementations, classifying an object based on the 3D semantic data includes determining, based on the 3D semantic data, a class of the object using an object classification neural network, and classifying the 3D bounding box corresponding to the object based on the determined class of the object. In some implementations, a first class-specific neural network is trained to determine specific points on a first classification of objects (e.g., chairs) that are used for measurements of objects in the first classification. For example, arm length and seat height of a chair. In some implementations, a second class-specific neural network is trained to determine specific points on a second classification of objects (e.g., tables) that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects. For example, for tables the table height and table top dimensions specific to round or rectangular table tops. The measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects. For example, a chair may include more or at least different measurements than a table or a TV.

[0016] The exemplary method further involves determining a location of a surface of the object based on the class of the object. The location is determined by identifying a plane within the 3D bounding box having semantics in the 3D semantic data satisfying surface criteria for the object. For example, identifying that a large number of chair voxels are within a horizontal plane indicates that the plane is the seat surface of a chair type object.

[0017] The exemplary method further involves providing a measurement (e.g., seat height, etc.) of the object. The measurement of the object is determined based on the location of the surface of the object. For example, measurements may be acquired from the seat surface to the floor to provide a seat height measurement. For example, a user may scan a room with a device (e.g., a smartphone) and the processes described herein would identify an object (e.g., a chair), and provide measurements for the identified object (e.g., seat height, etc.). In some implementations, the measurements may be automatically displayed on the user device overlaid or next to the object. In some implementations, the measurements may be provided after some type of user interaction with the identified object. For example, the user may be shown a transparent bounding box surrounding an object, and the user may select or click on the bounding box and the measurements would then be displayed.

[0018] In some implementations, identifying a plane within a bounding box includes identifying that a number of 3D data points (e.g., voxels) are within a particular plane (e.g., horizontal) indicates that the plane is a surface of a particular feature (e.g., chair seat) of a type of the object.

[0019] In some implementations, the number of 3D data points that are within the particular plane that indicates that the plane is the surface of the particular feature of the type of the object is determined based on a comparison to a data point plane threshold. For example, a plane threshold is a particular number of data points. In some implementations, a plane threshold is a percentage of data points compared to the other data points that are semantically labeled. For example, if 30% or more points are on the same horizontal plane (i.e., same height level), then it can be determined that a detected horizontal plane is the seat of a chair. In some implementations, different threshold percentages could be used for other object classifications. For example, tables would have a higher percentage of data points on the same horizontal plane. In some implementations, different detected planes may be used and compared to determine different features of identified objects.

[0020] In some implementations, the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud. Additionally, in some implementations, the semantic labels identify walls, wall attributes (e.g., doors and windows), objects, and classifications of the objects of the physical environment.

[0021] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be had by reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0023] FIG. 1 is a block diagram of an example operating environment in accordance with some implementations.

[0024] FIG. 2 is a block diagram of an example server in accordance with some implementations.

[0025] FIG. 3 is a block diagram of an example device in accordance with some implementations.

[0026] FIG. 4 is a system flow diagram of an example generation of a semantic three-dimensional (3D) representation using 3D data and semantic segmentation based on depth and light intensity image information according to some implementations.

[0027] FIG. 5 is a flowchart representation of an exemplary method that generates and displays a measurement of an object determined using a class-specific neural network based on a 3D representation of a physical environment in accordance with some implementations.

[0028] FIGS. 6A-6B are system flow diagrams of an example generation of a measurement of an object determined using one of a plurality of class-specific neural networks of a physical environment based on a 3D representation of the physical environment according to some implementations.

[0029] FIG. 7 is a flowchart representation of an exemplary method that generates and provides measurements of objects determined based on the location of the surface in accordance with some implementations.

[0030] FIG. 8 is a system flow diagram illustrating an example generation of measurements of objects determined based on the location of the surface according to some implementations.

[0031] In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.

DESCRIPTION

[0032] Numerous details are described in order to provide a thorough understanding of the example implementations shown in the drawings. However, the drawings merely show some example aspects of the present disclosure and are therefore not to be considered limiting. Those of ordinary skill in the art will appreciate that other effective aspects and/or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein.

[0033] FIG. 1 is a block diagram of an example operating environment 100 in accordance with some implementations. In this example, the example operating environment 100 illustrates an example physical environment 105 that includes walls 130, 132, 134, chair 140, table 142, door 150, and TV 152. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment 100 includes a server 110 and a device 120. In an exemplary implementation, the operating environment 100 does not include a server 110, and the methods described herein are performed on the device 120.

[0034] In some implementations, the server 110 is configured to manage and coordinate an experience for the user. In some implementations, the server 110 includes a suitable combination of software, firmware, and/or hardware. The server 110 is described in greater detail below with respect to FIG. 2. In some implementations, the server 110 is a computing device that is local or remote relative to the physical environment 105. In one example, the server 110 is a local server located within the physical environment 105. In another example, the server 110 is a remote server located outside of the physical environment 105 (e.g., a cloud server, central server, etc.). In some implementations, the server 110 is communicatively coupled with the device 120 via one or more wired or wireless communication channels (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0035] In some implementations, the device 120 is configured to present an environment to the user. In some implementations, the device 120 includes a suitable combination of software, firmware, and/or hardware. The device 120 is described in greater detail below with respect to FIG. 3. In some implementations, the functionalities of the server 110 are provided by and/or combined with the device 120.

[0036] In some implementations, the device 120 is a handheld electronic device (e.g., a smartphone or a tablet) configured to present content to the user. In some implementations, the user wears the device 120 on his/her head. As such, the device 120 may include one or more displays provided to display content. For example, the device 120 may enclose the field-of-view of the user. In some implementations, the device 120 is replaced with a chamber, enclosure, or room configured to present content in which the user does not wear or hold the device 120.

[0037] FIG. 2 is a block diagram of an example of the server 110 in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the server 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated-circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, and/or the like), one or more input/output (I/O) devices 206, one or more communication interfaces 208 (e.g., universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global system for mobile communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0038] In some implementations, the one or more communication buses 204 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and/or the like.

[0039] The memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices remotely located from the one or more processing units 202. The memory 220 comprises a non-transitory computer readable storage medium. In some implementations, the memory 220 or the non-transitory computer readable storage medium of the memory 220 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 230 and one or more applications 240.

[0040] The operating system 230 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the applications 240 are configured to manage and coordinate one or more experiences for one or more users (e.g., a single experience for one or more users, or multiple experiences for respective groups of one or more users).

[0041] The applications 240 include a 3D representation unit 242, a object detection unit 244, and a measurement unit 246. The 3D representation unit 242, the object detection unit 244, and the measurement unit 246 can be combined into a single application or unit or separated into one or more additional applications or units.

[0042] The 3D representation unit 242 is configured with instructions executable by a processor to obtain image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more of the techniques disclosed herein. For example, the 3D representation unit 242 fuses RGB images from a light intensity camera with a sparse depth map from a depth camera (e.g., time-of-flight sensor) and other sources of physical environment information to output a dense depth point cloud of information. Additionally, the 3D representation unit 242 is configured with instructions executable by a processor to obtain light intensity image data (e.g., RGB) and perform a semantic segmentation algorithm to assign semantic labels to recognized features in the image data and generate semantic image data (e.g., RGB-S) using one or more of the techniques disclosed herein. The 3D representation unit 242 is further configured with instructions executable by a processor to obtain light intensity image data (e.g., RGB) and depth image data and generate a semantic 3D representation (e.g., a 3D point cloud with associated semantic labels) using one or more of the techniques disclosed herein. In some implementations, the 3D representation unit 242 includes separate units, such as an integration unit to generate the 3D point cloud data, a semantic unit for semantic segmentation based on light intensity data (e.g., RGB-S), and a semantic 3D unit to generate the semantic 3D representation, as further discussed herein with reference to FIG. 4.

[0043] The object detection unit 244 is configured with instructions executable by a processor to generate and display measurements of objects determined using one of a plurality of class-specific neural networks of a physical environment based on a 3D representation (e.g., a 3D point cloud, a 3D mesh reconstruction, a semantic 3D point cloud, etc.) of the physical environment using one or more of the techniques disclosed herein. For example, the object detection unit 244 obtains a sequence of light intensity images from a light intensity camera (e.g., a live camera feed), a semantic 3D representation (e.g., semantic 3D point cloud) generated from the 3D representation unit 242, and other sources of physical environment information (e.g., camera positioning information from a camera’s SLAM system). The object detection unit 244 can identify objects (e.g., furniture, appliances, etc.) in the sequence of light intensity images based on the semantic 3D representation, generate bounding boxes for each identified object, and perform post processing using a fine-tuning neural network technique further disclosed herein.

[0044] In some implementations, the object detection unit 244 includes separate units, such as an object detection neural network unit to identify objects and generate proposed bounding boxes, an associated post processing unit to fine-tune the bounding boxes for each object identified, and an object classification neural network to classify each type of object, as further discussed herein with reference to FIGS. 6 and 8.

[0045] The measurement unit 246 is configured with instructions executable by a processor to generate measurement data based on the 3D representation (e.g., a 3D point cloud, a 3D mesh reconstruction, a semantic 3D point cloud, etc.) for the identified objects using one or more techniques disclosed herein. For example, the measurement unit 246 obtains data associated with bounding boxes (e.g., classified and refined bounding boxes) for identified objects from the object detection unit 244. The measurement unit 246 is configured with instructions executable by a processor to generate and provide measurement data based on the 3D representation for the identified objects using one or more processes further disclosed herein with reference to FIGS. 6 and 8.

[0046] In some implementations, the measurement unit 246 includes a plurality of machine learning units for each specific type of object. For example, a class–1 neural network for chairs, a class–2 neural network for tables, a class–3 neural network for TVs, etc. The plurality of machine learning units can be trained for a different subset of objects such that the measurement unit 246 can provided different types of measurements for each subset of object (e.g., diameter for a round table versus length and width of a rectangular table). The measurement unit 246 is configured with instructions executable by a processor to generate and provide measurement data based on the 3D representation for the identified objects for each subset of neural networks for each class of object using one or more processes further disclosed herein with reference to FIG. 6.

[0047] In some implementations, the measurement unit 246 includes a plane detection unit for identifying a plane within the bounding box having semantics in the 3D semantic data satisfying surface criteria for the object. For example, identifying that a large number of chair voxels are within a horizontal plane indicates that the plane is the seat surface of a chair type object. The measurement unit 246 is configured with instructions executable by a processor to generate and provide measurement data based on the 3D representation for the identified objects and particular plane detection using one or more processes further disclosed herein with reference to FIG. 8.

[0048] Although these elements are shown as residing on a single device (e.g., the server 110), it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover, FIG. 2 is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules shown separately in FIG. 2 could be implemented in a single module and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

[0049] FIG. 3 is a block diagram of an example of the device 120 in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, 120, and/or the like type interface), one or more programming (e.g., I/O) interfaces 310, one or more AR/VR displays 312, one or more interior and/or exterior facing image sensor systems 314, a memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0050] In some implementations, the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like.

[0051] In some implementations, the one or more displays 312 are configured to present the experience to the user. In some implementations, the one or more displays 312 correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays 312 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the device 120 includes a single display. In another example, the device 120 includes an display for each eye of the user.

[0052] In some implementations, the one or more image sensor systems 314 are configured to obtain image data that corresponds to at least a portion of the physical environment 105. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, and/or the like. In various implementations, the one or more image sensor systems 314 further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems 314 further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data including at least a portion of the processes and techniques described herein.

[0053] The memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices remotely located from the one or more processing units 302. The memory 320 comprises a non-transitory computer readable storage medium. In some implementations, the memory 320 or the non-transitory computer readable storage medium of the memory 320 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 330 and one or more applications 340.

[0054] The operating system 330 includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the applications 340 are configured to manage and coordinate one or more experiences for one or more users (e.g., a single experience for one or more users, or multiple experiences for respective groups of one or more users). The applications 340 include include a 3D representation unit 342, an object detection unit 344, and a measurement unit 346. The 3D representation unit 342, the object detection unit 344, and the measurement unit 346 can be combined into a single application or unit or separated into one or more additional applications or units.

……
……
……