Facebook Patent | Feed-forward collision avoidance for artificial reality environments

Patent: Feed-forward collision avoidance for artificial reality environments

Drawings: Click to check drawins

Publication Number: 20210183135

Publication Date: 20210617

Applicant: Facebook

Abstract

An artificial reality system includes a head mounted display (HMD) and a physical overlay engine that generates overlay image data, referred to herein as a physical overlay image, corresponding to the physical objects in a three-dimensional (3D) environment. In response to an activation condition, a rendering engine of the artificial reality system renders the overlay image data to overlay artificial reality content for display on the HMD, thereby apprising a user of the HMD of their position with respect to the physical objects in the 3D environment.

Claims

1. An artificial reality system comprising: one or more image capture devices configured to capture image data representative of a three-dimensional (3D) physical environment having one or more physical objects; a head mounted display (HMD) configured to output artificial reality content associated with a 3D virtual environment; a physical overlay engine; and a rendering engine configured to render, based on a pose of the HMD, output image data comprising the artificial reality content without including image data representative of the physical objects in the 3D physical environment; wherein in response to an activation condition, the physical overlay engine determines, from the image data, data representing the physical objects, and generates, from the data representing the physical objects, overlay image data representing the physical objects and wherein the rendering engine is configured to temporarily render the overlay image data representing the physical objects with the output image data comprising the artificial reality content in response to the activation condition.

2. The artificial reality system of claim 1, wherein the physical overlay engine is configured to generate the overlay image data to include contour data for the physical objects.

3. The artificial reality system of claim 1, wherein the artificial reality content comprises fully immersive artificial reality content.

4. (canceled)

5. The artificial reality system of claim 1, further comprising: a hand-held device; and a hand-held device tracker configured to determine a current orientation of the hand-held device, wherein the activation condition comprises one of: a detection of a button press of at least one button of the hand-held device, or a determination that the current orientation of the hand-held device comprises an overlay activation orientation.

6. The artificial reality system of claim 1, further comprising a pose tracker, wherein the pose tracker is configured to detect that the HMD has crossed a virtual boundary, and wherein the activation condition comprises detection of the HMD crossing the virtual boundary.

7. The artificial reality system of claim 1, further comprising a pose tracker, wherein the pose tracker is configured to: detect a distance of the HMD from one of the physical objects in the 3D physical environment; and wherein the activation condition comprises a determination that the distance is less than a configurable or predetermined threshold.

8. The artificial reality system of claim 1, further comprising: a hand-held device; and a hand-held device tracker configured to determine a distance of the hand-held device from one of the physical objects in the 3D physical environment; wherein the activation condition comprises a determination that the distance is less than a configurable or predetermined threshold.

9. The artificial reality system of claim 1, wherein the physical overlay engine includes an edge detector configured to determine one or more edges of the one or more physical objects in the 3D physical environment, and wherein the data representing the physical objects is determined according to the one or more edges of the one or more physical objects.

10. A method comprising: obtaining image data representative of a three-dimensional (3D) physical environment having one or more physical objects; rendering, by a rendering engine of an artificial reality system and based on a pose of a head mounted display (HMD) representing an estimated position and orientation of the HMD within the 3D physical environment, output image data comprising artificial reality content without including image data representative of the one or more physical objects; and in response to an activation condition, determining, from the image data, data representing the one or more physical objects, generating, from the data representing the physical objects, overlay image data representing the physical objects, and temporarily rendering, by the rendering engine, the overlay image data representing the physical objects with the output image data comprising the artificial reality content.

11. The method of claim 10, wherein generating the overlay image data comprises generating contour data for the one or more physical objects.

12. The method of claim 10, wherein generating the overlay image data comprises generating texture data for the one or more physical objects.

13. (canceled)

14. The method of claim 10, wherein the activation condition comprises one of: determining that a current orientation of a hand-held device of the artificial reality system comprises an overlay activation orientation; or detecting a button press on a hand-held device of the artificial reality system.

15. The method of claim 10, wherein the activation condition comprises detection of the HMD crossing a virtual boundary.

16. The method of claim 10, wherein the activation condition comprises detecting that the HMD is within a configurable or predetermined distance from one of the physical objects in the 3D physical environment.

17. The method of claim 10, wherein the activation condition comprises detecting that a hand-held device is within a configurable or predetermined distance from one of the physical objects in the 3D physical environment.

18. The method of claim 10, further comprising: determining one or more edges of the one or more physical objects in the 3D physical environment, wherein the data representing the one or more physical objects is determined according to the one or more edges of the one or more physical objects.

19. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system including a head mounted display (HMD) to: obtain image data representative of a three-dimensional (3D) physical environment having one or more physical objects; determine an HMD pose representing an estimated position and orientation of the HMD within the 3D physical environment; render, based on the HMD pose, output image data comprising artificial reality content without including image data representative of the one or more physical objects; and in response to an activation condition, determine, from the image data, data representing the one or more physical objects, and generate, from the data representing the one or more physical objects, overlay image data representing the physical objects, and temporarily render the overlay image data representing the physical objects with the output image data comprising the artificial reality content.

20. (canceled)

21. The artificial reality system of claim 1, wherein to temporarily render the overlay image data representing the physical objects with the output image data comprising the artificial reality content comprises one of: to render the overlay image data while the activation condition is satisfied; or to render the overlay image data for a predetermined or configurable time after the activation condition is satisfied.

22. The method of claim 10, wherein temporarily rendering the overlay image data representing the physical objects with the output image data comprising the artificial reality content comprises one of: rendering the overlay image data while the activation condition is satisfied; or rendering the overlay image data for a predetermined or configurable time after the activation condition is satisfied.

23. The non-transitory, computer-readable medium of claim 19, wherein the instructions to cause the one or more processors to temporarily render the overlay image data representing the physical objects with the output image data comprising the artificial reality content comprise instruction to cause the one or more processors to perform one of: render the overlay image data while the activation condition is satisfied; or render the overlay image data for a predetermined or configurable time after the activation condition is satisfied.

Description

TECHNICAL FIELD

[0001] This disclosure generally relates to artificial reality systems, such as virtual reality, mixed reality, and/or augmented reality systems, and more particularly, to avoiding collisions in artificial reality environments.

BACKGROUND

[0002] Artificial reality systems are becoming increasingly ubiquitous with applications in many fields such as computer gaming, health and safety, industrial, training and education. As a few examples, artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks. In general, artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof.

[0003] Typical artificial reality systems include one or more devices for rendering and displaying content to users. As one example, an artificial reality system may incorporate a head mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include completely computer generated content or generated content combined with captured content (e.g., real-world video and/or images). During operation, the user may utilize a hand-held device, keyboard or other devices to interact with applications and the artificial reality system.

SUMMARY

[0004] In general, this disclosure describes artificial reality systems and, more specifically, an overlay engine for artificial reality systems that constructs overlay image data providing visual indications of physical objects in a physical environment in which a user of the artificial reality system is located. The overlay image data can be overlaid on artificial reality content, such as fully immersive virtual reality content, being presented to the user by the artificial reality system. Thus, a user of the artificial reality system can use the overlay image data to avoid collisions with any nearby physical objects in the physical environment, thereby keeping the user safe and helping to ensure an uninterrupted artificial reality experience.

[0005] As further described herein, artificial reality content presented by an artificial reality system does not necessarily coincide with the physical dimensions and physical objects of a physical environment, such as a room in which a user of an artificial reality system is located. When a user interacts with an artificial reality system, the user initially may have a mental model of the physical environment, including the location of physical objects in the physical environment. However, as the user continues using the artificial reality system and potentially moves about the physical environment, the user’s mental model may become less accurate, thereby increasing the risk of collisions with objects in the physical environment. For example, a user may be using an artificial reality system within a room of their home. There may be furniture and other physical objects in the room. The user may start an artificial reality application while in one location of the room, but as the artificial reality application progresses, may move to other locations with the room. A technical problem with conventional artificial reality systems is that the systems do not provide any way for the user to determine the user’s location with respect to physical objects in the room. For example, the artificial reality content presented by the artificial reality application may obscure or completely cover some or all of the physical objects in the room. Thus, the user may be forced to temporarily displace or remove their HMD in order to determine where they are in the physical environment to avoid collisions with physical objects that may be nearby. This can be a distraction to the user, leading to user dissatisfaction and frustration with the operation of the artificial reality system.

[0006] As a technical solution to the aforementioned technical problem, the artificial reality system of this disclosure includes a physical overlay engine that generates overlay image data representing physical objects in the room. The overlay image data can include, as examples, partial images, a single image or a set of images. In general, the overlay image data can be overlaid with the artificial reality content, thereby allowing the users to determine their location with respect to physical objects in the room without interrupting their artificial reality experience. In some aspects, overlaying the image data with the artificial reality content includes rendering the overlay image data with the artificial reality content such that the user can see both the artificial reality content and the overlay image data at the same time.

[0007] The overlay image data can be generated and overlaid in response to a request by the user, for example via a button press on a hand-held device of the artificial reality system. The overlay image data may be temporary and may be shown only when activated by the user. Thus, the physical overlay engine can provide an image of the physical objects in a physical environment. The user can utilize the overlaid image to refresh their mental map of the room and avoid collisions with objects in the room. Further, the user can utilize the overlaid image to plan future movements within the room.

[0008] An overlay image can be generated and displayed in response to other activation conditions besides a button press. For example, the overlay image may be generated and displayed in response to user input besides a button press such as touching or hovering over a touch sensitive surface or presence sensitive device, a voice command, hand gesture or other form of user input. Further, the overlay image may be automatically generated and displayed in response to crossing or coming within a threshold distance of a virtual boundary defined by the user or artificial reality system. Also, the overlay image may be automatically generated and displayed in response to coming within a threshold distance of a physical object.

[0009] The aspects described above, and further aspects described below can provide a technical improvement over conventional artificial reality system implementations, and can provide one or more practical applications, such as enabling an artificial reality system to provide overlay image data representative of physical objects in the user’s physical environment. The overlay image data may be overlaid with artificial reality content. Thus, the user can plan and make movements in the physical environment to avoid collisions with physical objects without interrupting their artificial reality experience.

[0010] In one or more example aspects, an artificial reality system includes one or more image capture devices configured to capture image data representative of a three-dimensional (3D) physical environment having one or more physical objects; a head mounted display (HMD) configured to output artificial reality content associated with a 3D virtual environment; a physical overlay engine configured to: determine, from the image data, data representing the physical objects, and generate, from the data representing the physical objects, overlay image data representing the physical objects; and a rendering engine configured to render, based on a pose of the HMD, output image data comprising the artificial reality content without including image data representative of the physical objects in the 3D physical environment; wherein in response to an activation condition, the physical overlay engine generates the overlay image data representing the physical objects and the rendering engine renders the overlay image data representing the physical objects with the output image data comprising the artificial reality content.

[0011] In one or more further example aspects, a method includes obtaining image data representative of a three-dimensional (3D) physical environment having one or more physical objects; rendering, by a rendering engine of an artificial reality system and based on a pose of a head mounted display (HMD) representing an estimated position and orientation of the HMD within the 3D physical environment, output image data comprising artificial reality content without including image data representative of the one or more physical objects; and in response to an activation condition, determining, from the image data, data representing the one or more physical objects, and generating, from the data representing the physical objects, overlay image data representing the physical objects, and rendering, by the rendering engine, the overlay image data representing the physical objects with the output image data comprising the artificial reality content.

[0012] In one or more additional example aspects, a non-transitory, computer-readable medium comprises instructions that, when executed, cause one or more processors of an artificial reality system to obtain image data representative of a three-dimensional (3D) physical environment having one or more physical objects; determine an HMD pose representing an estimated position and orientation of the HMD within the 3D physical environment; render, based on the HMD pose, output image data comprising artificial reality content without including image data representative of the one or more physical objects; and in response to an activation condition, determine, from the image data, data representing the one or more physical objects, and generate, from the data representing the one or more physical objects, overlay image data representing the physical objects, and render the overlay image data representing the physical objects with the output image data comprising the artificial reality content.

[0013] The details of one or more examples of the techniques of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

[0014] FIG. 1A is an illustration depicting an example artificial reality system that generates and displays an overlay image as an aid in collision avoidance in accordance with the techniques of the disclosure.

[0015] FIG. 1B is an illustration depicting another example artificial reality system that generates and displays an overlay image as an aid in collision avoidance in accordance with the techniques of the disclosure.

[0016] FIG. 2A is an illustration depicting an example HMD that operates in accordance with the techniques of the disclosure.

[0017] FIG. 2B is an illustration depicting another example HMD that operates in accordance with the techniques of the disclosure.

[0018] FIG. 3 is a block diagram depicting an example in which generating and displaying the overlay image is performed by an example instance of the HMD of the artificial reality systems of FIGS. 1A, 1B in accordance with the techniques of the disclosure.

[0019] FIG. 4 is a block diagram showing example implementations in which generating and displaying the overlay image is performed by example instances of the console and the HMD of the artificial reality systems of FIGS. 1A, 1B.

[0020] FIG. 5 illustrates an example physical environment.

[0021] FIG. 6 illustrates an example overlay image corresponding to the example physical environment of FIG. 5.

[0022] FIG. 7A illustrates example artificial reality content.

[0023] FIG. 7B illustrates an example integrated image in which the example overlay image of FIG. 6 is overlaid on the example artificial reality content of FIG. 7A.

[0024] FIG. 8 is a flowchart illustrating example operations of a method for generating and displaying overlay image data in accordance with aspects of the disclosure.

[0025] FIG. 9 illustrates an example virtual boundary for the example physical environment of FIG. 5.

[0026] Like reference characters refer to like elements throughout the figures and description.

DETAILED DESCRIPTION

[0027] FIG. 1A is an illustration depicting an example artificial reality system 10 that generates and displays overlay image data as an aid in collision avoidance in accordance with the techniques of the disclosure. In some example implementations, artificial reality system 10 generates and renders artificial reality content 122 to a user 110. The artificial reality content 122 can include one or more graphical elements 120, 128 that represent avatars and virtual objects that may be part of an AR or VR experience. In addition, overlay image data can be presented with the artificial reality content 122 that may show physical objects that are in, or in proximity to, a physical environment occupied by user 110. In the example illustrated in FIG. 1A, the overlay image data includes a representation 144 of a couch that is in the physical environment occupied by user 110.

[0028] In the example of FIG. 1A, artificial reality system 10 includes head mounted device (HMD) 112. As shown, HMD 112 is typically worn by user 110 and includes an electronic display and optical assembly for presenting artificial reality content 122 to user 110. In addition, HMD 112 includes one or more motion sensors (e.g., accelerometers) for tracking motion of the HMD 112 and may include one or more image capture devices 138, e.g., cameras, infrared (IR) detectors, Doppler radar, line scanners and the like, for capturing image data of the surrounding physical environment.

[0029] In some example implementations HMD 112 operates as a stand-alone, mobile artificial reality system. In other implementations, an artificial reality system 10 can optionally include a console 106 and/or one or more external sensors 90 in addition to, or instead of HMD 112. In the example illustrated in FIG. 1A, console 106 is shown as a single computing device, such as a gaming console, workstation, a desktop computer, or a laptop. In other examples, console 106 may be distributed across a plurality of computing devices, such as a distributed computing network, a data center, or a cloud computing system. Console 106, HMD 112, and sensors 90 may, as shown in this example, be communicatively coupled via network 104, which may be a wired or wireless network, such as WiFi, a mesh network or a short-range wireless communication medium.

[0030] In general, artificial reality system renders artificial reality content 122 for display to user 110. In the example of FIG. 1A, user 110 views the artificial reality content 122 constructed and rendered by an artificial reality application executing on HMD 112 and/or console 106. In some aspects, the artificial reality content 122 may be completely computer constructed, as in the case of a fully immersive VR experience. For example, the artificial reality content 122 may not include representations of some or all the physical objects in the physical environment occupied by user 110. As one example, artificial reality content 122 may be a consumer gaming application in which user 110 is rendered as avatar 120 with one or more virtual objects 126, 128A, 128B. In some aspects, the artificial reality content 122 can include representations of physical objects in a different physical environment from that of user 110. As an example, the artificial reality content may be associated with a video conferencing application, a virtual navigation application, or a virtual travel application. In the case of a video conferencing application, the artificial reality content may include the physical objects in a room occupied by a participant of the video conferencing application other than user 110. In the case of a virtual travel application, the physical object may include the physical objects in a travel destination. In such cases, the artificial reality content 122 may include physical objects that are in a separate physical environment from that occupied by the user 110 but does not include physical objects in the same physical environment as that occupied by the user 110.

[0031] During operation, the artificial reality application constructs artificial reality content 122 for display to user 110 by tracking and computing pose information for a frame of reference, typically a viewing perspective of HMD 112. Using HMD 112 as a frame of reference, the artificial reality application renders 3D artificial reality content 122. During this process, the artificial reality application uses sensed data received from HMD 112 and the one or more hand-held devices 114, such as movement information and user commands, and, in some examples, data from any external sensors 90, such as external cameras, to capture 3D information within the real world, physical environment, such as motion by user 110 and/or motion of the one or more hand-held devices 114. Based on the sensed data, the artificial reality application determines a current pose for the frame of reference of HMD 112, a current pose for the one or more hand-held devices 114 and, in accordance with the current poses of the HMD 112 and hand-held devices 114, renders the artificial reality content 122. In the example illustrated in FIG. 1A, the artificial reality application can render virtual objects such as virtual sword 126 based on a pose of the HMD 112 and one or more hand-held devices 114. Further, in response to an activation condition, the artificial reality application can render overlay image data with the artificial reality content. In the example illustrated in FIG. 1A, the overlay image data includes an image 144 of a couch that is in the physical environment occupied by user 110.

[0032] More specifically, as further described herein, image capture devices 138 of HMD 112 capture image data representative of objects in the real world, physical environment that are within a field of view 130 of image capture devices 138. Field of view 130 typically corresponds with the viewing perspective of HMD 112. Physical objects in the physical environment can include objects such as walls, furniture, televisions, monitors, plants, etc. In some aspects, the artificial reality content 122 may not include any representations of the physical objects. For example, in the case of a totally immersive VR experience, the artificial reality content may be generated by one or more processors according to the artificial reality application. Thus, the user 110 may not be fully aware of the location of physical objects in the physical environment because they are not presented as part of the artificial reality content 122. As a result, the user 110 may rely on their “mental map” of the physical environment formed prior to the user wearing HMD 112, or by temporarily removing or displacing the HMD 112 in order to view the physical environment.

[0033] In some aspects, overlay image data can be presented with the artificial reality content to apprise the user 110 of the location of physical object in the physical environment occupied by the user 110. Thus, the user can refresh their “mental map” of the location of the objects in order to avoid colliding with the object during their AR/VR experience. The display of the overlay image data may be temporary and may be presented in response to an activation condition. The overlay image data may be presented during the presence of the activation condition, and not be presented if the activation condition is not satisfied. Using the example illustrated in FIG. 1A as an example, the activation condition can be detection that a button 142 on a hand-held device 114 has been pressed. The image 144 of the couch may not be present prior to pressing button 142. In response to detection that button 142 has been pressed, the image 144 of the couch may be presented along with artificial reality content 122. The image may be presented for a predetermined or configurable amount of time after the button 142 is pressed, or it may be presented for as long as the button 142 is pressed. When the activation condition is no longer satisfied, for example, after the time period for display has passed or if the button is no longer pressed, the image 144 of the couch may no longer be presented.

[0034] Other activation conditions are possible. Examples of such other activation conditions include detection of touching or hovering over a touch sensitive surface or presence sensitive device, detection of a voice command associated with the activation condition, detection of hand gesture or position associated with the activation condition, etc. Further, the overlay image may be automatically generated and displayed in response to detection that the HMD 112 or hand-held device 114 has crossed or come within a threshold distance of a virtual boundary defined by the user or artificial reality system. Also, the overlay image may be automatically generated and displayed in response to detection that the HMD 112 or hand-held device 114 is within a threshold distance of a physical object.

[0035] The graphical representation of physical objects in the overlay image data may be created in various ways. In some aspects, the graphical representation of physical objects in the overlay image data may be “shadow objects”, e.g. semi-transparent images of the physical objects, that allow at least a portion of the artificial reality content 122 to be viewed through the graphical representation of the physical object. In some aspects, the graphical representation of the physical objects in the overlay image data may be rendered as line representations (e.g., as outlines) of the corresponding physical object as depicted by image 144 in FIG. 1A. The line representation may be rendered such that edges corresponding to the associated physical object are visible while the interior of the line representation is partially or fully transparent.

[0036] Accordingly, the techniques of the disclosure provide specific technical improvements to the computer-related field of rendering and displaying content by an artificial reality system. For example, artificial reality systems as described herein may provide a high-quality fully immersive artificial reality experience to a user, such as user 110, of the artificial reality application by generating and rendering graphical elements that are not necessarily based on the object in the physical environment occupied by the user. Further, artificial reality systems as described herein may provide a mechanism by which the user may participate in an immersive artificial reality experience and avoid collisions with physical objects in the physical environment occupied by the user. Overlay image data that includes graphical representations of the physical objects in the physical environment occupied by the user may be rendered, at least temporarily, with the generated artificial reality content in response to an activation condition.

[0037] A further technical improvement is that the collision avoidance capability provided by the techniques described herein are feed-forward as opposed to conventional feedback implementations that merely warn the user when the system determines that the user is too close to a physical object. In other words, the feed-forward technique described herein provides the ability for a user to quickly refresh their mental map of the physical environment they occupy prior to any collisions with physical objects. The user can refresh their mental map without the interruption of the artificial reality experience that may be caused by removing or displacing their HMD 112. Thus, the user may utilize their refreshed mental map to plan future strategies for the artificial reality experience that may involve movements within the physical environment without fear of collisions with physical objects. This is in contrast to the feedback approach of conventional systems that typically provide warnings to a user when the user is too close to a physical object. Thus, in the feedback collision avoidance system of conventional artificial reality systems, the user reacts to avoid a collision, which may interrupt their artificial reality experience.

[0038] FIG. 1B is an illustration depicting another example artificial reality system 20 that generates and displays overlay image data as an aid in collision avoidance in accordance with the techniques of the disclosure. Similar to artificial reality system 10 of FIG. 1A, in some examples, artificial reality system 20 of FIG. 1B may, in response to an activation condition, render overlay image data with artificial reality content.

[0039] In the example of FIG. 1B, artificial reality system 20 includes external cameras 102A and 102B (collectively, “external cameras 102”), HMDs 112A-112C (collectively, “HMDs 112”), hand-held devices 114A, 114B and 114C (collectively, “hand-held devices 114”), console 106, and sensors 90. As shown in FIG. 1B, artificial reality system 20 represents a multi-user environment in which an artificial reality application executing on HMDs 112 and/or console 106 presents artificial reality content to each of users 110A-110C (collectively, “users 110”) based on a current viewing perspective of a corresponding frame of reference for the respective user. That is, in this example, the artificial reality application constructs artificial reality content by tracking and computing pose information for a frame of reference for each of HMDs 112 and respective hand-held devices 114. Further, artificial reality system 20 uses data received from cameras 102, HMDs 112, and hand-held devices 114 to capture 3D information regarding physical objects within the real world physical environment. As one example, the artificial reality application may render, based on a current viewing perspective determined for HMD 112C, immersive artificial reality content 122 having virtual objects 128A-128C (collectively, “virtual objects 128”). Additionally, artificial reality system can capture image data with respect to real world objects 108A-108C (collectively, “real world objects 108”). Further, from the perspective of HMD 112C, artificial reality system 20 renders avatars 120A, 120B based upon the estimated positions for users 110A, 110B, respectively. Also, the artificial reality system 20 can render graphical objects based on the poses of the hand-held devices 114 as determined by the artificial reality system 20.

[0040] In response to an activation condition, HMD 112C of user 110C may render overlay image data having an image 144 representing physical couch 140, and images 146A and 146B representing physical objects 108A and 108B in the physical environment of user 110C. As an example, the activation condition may be detecting that user 110C has pressed button 142 of their hand-held device 114C.

[0041] Each of HMDs 112 concurrently operates within artificial reality system 20. In the example of FIG. 1B, each of users 110 may be a “player” or “participant” in the artificial reality application, and any of users 110 may be a “spectator” or “observer” in the artificial reality application. HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by rendering immersive artificial reality content including virtual objects such as sword 126 based on a determined pose of hand-held device 114C and presenting overlay image data in response to an activation condition.

[0042] HMD 112A and HMD 112B may also operate substantially similar to HMD 112 of FIG. 1A. HMD 112B may receive user inputs from hand-held devices 114A and 144B held by user 110B.

[0043] As shown in FIG. 1B, in addition to or alternatively to image data captured via camera 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, positions and orientations of a user 110, and the locations and orientations of physical objects in the physical environment occupied by a user 110.

……
……
……