Facebook Patent | Virtual user interface using a peripheral device in artificial reality environments

Patent: Virtual user interface using a peripheral device in artificial reality environments

Drawings: Click to check drawins

Publication Number: 20210011556

Publication Date: 20210114

Applicant: Facebook

Abstract

In general, the disclosure describes artificial reality systems and techniques for generating and presenting a virtual user interface with which users may interact using a physical peripheral device. In some examples, an artificial reality system includes an image capture device configured to capture image data; a head-mounted display (HMD) configured to output artificial reality content; a user interface engine configured to detect a peripheral device from the image data, wherein the user interface engine is configured to generate a virtual user interface comprising one or more virtual user interface elements; and a rendering engine configured to render the artificial reality content and to render, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD.

Claims

1. An artificial reality system comprising: an image capture device configured to capture image data; a head-mounted display (HMD) configured to output artificial reality content; a user interface engine configured to detect a peripheral device from the image data, wherein the user interface engine is configured to generate a virtual user interface comprising one or more virtual user interface elements; and a rendering engine configured to render the artificial reality content and to render, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD.

2. The artificial reality system of claim 1, a gesture detector configured to detect a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements, wherein the artificial reality system is configured to perform, in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

3. The artificial reality system of claim 2, further comprising: the peripheral device, wherein the peripheral device comprises a presence-sensitive interface configured to receive one or more inputs by a user, wherein the gesture detector is configured to detect, based on the one or more inputs, the user interface gesture.

4. The artificial reality system of claim 2, wherein to perform the one or more actions, the user interface engine is configured to modify the virtual user interface to generate a modified virtual user interface comprising one or more modified virtual user interface elements, and wherein the rendering engine is configured to render, at the user interface position, the modified virtual user interface for display at the HMD.

5. The artificial reality system of claim 1, wherein to render the virtual user interface the rendering engine is configured to render, overlaid on a surface of the peripheral device, the virtual user interface.

6. The artificial reality system of claim 5, wherein the surface of the peripheral device comprises a presence-sensitive interface.

7. The artificial reality system of claim 1, further comprising: the peripheral device, wherein the peripheral device does not include a display.

8. The artificial reality system of claim 1, further comprising: the peripheral device, wherein the peripheral device includes a display, wherein the virtual user interface engine is configured to send a communication to the peripheral device to cause the peripheral device to deactivate the display.

9. The artificial reality system of claim 1, further comprising: the peripheral device, wherein the peripheral device includes a display, wherein the virtual user interface engine is configured to receive an indication the display is deactivated, and wherein to generate the virtual user interface, the virtual user interface engine is configured to generate the virtual user interface in response to determining, based on the indication, the display is deactivated.

10. The artificial reality system of claim 1, further comprising: the peripheral device, wherein the peripheral device comprises a gesture detector configured to detect a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements, and wherein the HMD is configured to perform, in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

11. The artificial reality system of claim 1, further comprising: a gesture detector configured to detect a drawing gesture performed by a user at positions corresponding to positions on the virtual user interface, wherein the virtual user interface engine is configured to generate, in response to the drawing gesture, a modified virtual user interface comprising virtual markings at the positions on the virtual user interface, wherein the rendering engine is configured to render, at the user interface position, the modified virtual user interface for display at the HMD.

12. The artificial reality system of claim 1, wherein the peripheral device comprises one of a smartphone, smartwatch, or tablet computer.

13. A method comprising: obtaining, by an artificial reality system including a head-mounted display (HMD), image data via an image capture device, the HMD configured to output artificial reality content; detecting, by the artificial reality system, a peripheral device from the image data; generating, by the artificial reality system, a virtual user interface comprising one or more virtual user interface elements; and rendering, by the artificial reality system, the artificial reality content and, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD.

14. The method of claim 13, further comprising: detecting, by the artificial reality system, a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements; performing, by the artificial reality system in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

15. The method of claim 14, further comprising: receiving, at a presence-sensitive interface of the peripheral device, one or more inputs by a user; and wherein the detecting comprises detecting, based on the one or more inputs, the user interface gesture.

16. The method of claim 14, wherein performing the one or more actions comprises modifying the virtual user interface to generate a modified virtual user interface comprising one or more modified virtual user interface elements, the method further comprising: rendering, at the user interface position, the modified virtual user interface for display at the HMD.

17. The method of claim 13, wherein the peripheral device does not include a display.

18. The method of claim 13, further comprising: sending, from the HMD to the peripheral device, a communication to cause the peripheral device to turn off the display.

19. The method of claim 13, further comprising: detecting, by the peripheral device, a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements; and performing, by the artificial reality system in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

20. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system, including a head-mounted display (HMD) configured to output artificial reality content, to: generate a virtual user interface comprising one or more virtual user interface elements; render the artificial reality content and, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD; detect a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements; and perform, in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

Description

TECHNICAL FIELD

[0001] The disclosure generally relates to artificial reality systems, such as augmented reality, mixed reality, and/or virtual reality systems, and more particularly, to user interfaces in artificial reality environments.

BACKGROUND

[0002] Artificial reality systems are becoming increasingly ubiquitous with applications in many fields such as computer gaming, health and safety, industrial, and education. As a few examples, artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks. In general, artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof.

[0003] Typical artificial reality systems include one or more devices for rendering and displaying content to users. As one example, an artificial reality system may incorporate a head-mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include completely-generated content or generated content combined with captured content (e.g., real-world video and/or images). During operation, the user typically interacts with the artificial reality system to interact with virtual reality content in an artificial reality environment.

SUMMARY

[0004] In general, the disclosure describes artificial reality (AR) systems and techniques for generating and presenting a virtual user interface with which users may interact using a physical peripheral device. The AR system renders, for display by an HMD, glasses or other display device, AR content in which the virtual user interface is locked to the peripheral device. That is, the AR system may render the virtual user interface having one or more virtual user interface elements at a position and pose in the artificial reality environment that is based on and corresponds to the position and pose of the physical peripheral device in the physical environment. In this way, the virtual user interface in the artificial reality environment may track the physical peripheral device. The virtual user interface elements can include, in various examples, virtual buttons, a virtual keyboard, a virtual drawing interface, a virtual selectable menu, or other user-selectable virtual user interface elements, which may be context-driven based on the current AR applications engaged by the user. The peripheral device may be a computing device having one or more presence-sensitive surfaces.

[0005] The AR systems may enable the user to interact with the peripheral device through virtual user interface elements of the virtual user interface overlaid on the peripheral device, which may be manipulated and otherwise interacted with by the user to provide input to an AR system through pose tracking of the peripheral device and image-based gesture detection and/or via one or more input devices of the peripheral device, such as a presence-sensitive surface. For example, the user may interact with the virtual user interface rendered on the physical peripheral device to perform user interface gestures with respect to virtual user interface elements. For instance, the user may press their finger at a physical location on the peripheral device corresponding to a position in the artificial reality environment at which the AR system renders a virtual user interface button of the virtual user interface. In this example, the AR system detects this user interface gesture and performs an action corresponding to the detected press of the virtual user interface button. The AR system may also, for instance, animate the press of the virtual user interface button along with the gesture.

[0006] The techniques may provide one or more technical improvements that provide at least one practical application. For example, the techniques can enable the user to provide fine-grained user inputs with respect to user interface elements rendered virtually on a physical peripheral device that provides haptic feedback, in contrast to free-floating virtual user interfaces in the artificial reality environment. This can simplify and improve the precision of gesture detection and provide a more pleasing user experience. In addition, the peripheral device may not display the user interface elements at its own display and may not even include a display. The techniques may therefore additionally reduce power consumption and simplify AR applications by eliminating a separate interface that would otherwise need to be generated and displayed, at a presence-sensitive display of a smartphone or tablet for instance, for receiving precision inputs from a user.

[0007] In some examples, an artificial reality system includes an image capture device configured to capture image data; a head-mounted display (HMD) configured to output artificial reality content; a user interface engine configured to detect a peripheral device from the image data, wherein the user interface engine is configured to generate a virtual user interface comprising one or more virtual user interface elements; and a rendering engine configured to render the artificial reality content and to render, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD.

[0008] In some examples, a method includes obtaining, by an artificial reality system including a head-mounted display (HMD), image data via an image capture device, the HMD configured to output artificial reality content; detecting, by the artificial reality system, a peripheral device from the image data; generating, by the artificial reality system, a virtual user interface comprising one or more virtual user interface elements; and rendering, by the artificial reality system, the artificial reality content and, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD.

[0009] In some examples, a non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system, including a head-mounted display (HMD) configured to output artificial reality content, to: generate a virtual user interface comprising one or more virtual user interface elements; render the artificial reality content and, at a user interface position locked relative to a position of the peripheral device in an artificial reality environment, the virtual user interface for display at the HMD; detect a user interface gesture performed by a user at a position corresponding to one of the virtual user interface elements; and perform, in response to the user interface gesture, one or more actions associated with the one of the virtual user interface elements.

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1A is an illustration depicting an example artificial reality system including a head-mounted display (HMD) and a peripheral device in accordance with techniques of the disclosure.

[0012] FIG. 1B is an illustration depicting an example artificial reality system in accordance with techniques of the disclosure.

[0013] FIG. 2A is an illustration depicting an example HMD and an example peripheral device, in accordance with techniques of the disclosure.

[0014] FIG. 2B is an illustration depicting an example HMD, in accordance with techniques of the disclosure.

[0015] FIG. 3 is a block diagram showing example implementations of a console, an HMD, and a peripheral device of the artificial reality systems of FIGS. 1A, 1B in accordance with techniques of the disclosure.

[0016] FIG. 4 is a block diagram depicting an example in which user interface generation is performed by the HMD of the artificial reality systems of FIGS. 1A, 1B in accordance with techniques of the disclosure.

[0017] FIG. 5 is a flowchart illustrating operations of a peripheral device in accordance with aspects of the disclosure.

[0018] FIG. 6 is an example HMD displays illustrating interacting with artificial reality content in accordance with aspects of the disclosure.

[0019] FIGS. 7A-7C are example HMD displays illustrating a performance of one or more actions in response to a detection of a user interface gesture performed on a peripheral device in accordance with aspects of the disclosure.

[0020] FIGS. 8A-8I are example HMD displays illustrating various interactions with artificial reality content with a virtual laser pointer in accordance with aspects of the disclosure.

[0021] FIG. 9 is a block diagram illustrating an example peripheral device and virtual user interface, according to techniques of this disclosure.

[0022] Like reference characters refer to like elements throughout the description and figures.

DETAILED DESCRIPTION

[0023] FIG. 1A is an illustration depicting an example artificial reality system, in accordance with techniques of the disclosure. In the example of FIG. 1A, artificial reality system 10 includes HMD 112, console 106, peripheral device 136, and may include one or more external sensors 90. As shown, HMD 112 is typically worn by user 110 and comprises an electronic display and optical assembly for presenting artificial reality content 122 to user 110. In addition, HMD 112 includes one or more sensors (e.g., accelerometers) for tracking motion of the HMD 112 and may include one or more image capture devices 138 (e.g., cameras, line scanners) for capturing image data of the surrounding physical environment. In this example, console 106 is shown as a single computing device, such as a gaming console, workstation, a desktop computer, or a laptop. In other examples, console 106 may be distributed across a plurality of computing devices, such as a distributed computing network, a data center, or a cloud computing system. Console 106, HMD 112, and sensors 90 may, as shown in this example, be communicatively coupled via network 104, which may be a wired or wireless network, such as Wi-Fi, a mesh network or a short-range wireless communication medium, or combination thereof. Although HMD 112 is shown in this example as in communication with, e.g., tethered to or in wireless communication with, console 106, in some implementations HMD 112 operates as a stand-alone, mobile artificial reality system.

[0024] In general, artificial reality system 10 uses information captured from a real-world, 3D physical environment to render artificial reality content 122 for display to user 110. In the example of FIG. 1A, user 110 views the artificial reality content 122 constructed and rendered by an artificial reality application executing on console 106 and/or HMD 112. In some examples, artificial reality content 122 may comprise a mixture of real-world imagery (e.g., hand 132, peripheral device 136, walls 121) and virtual objects (e.g., virtual content items 124, 126 and virtual user interface 137) to produce mixed reality and/or augmented reality. In some examples, virtual content items 124, 126 may be mapped (e.g., pinned, locked, placed) to a particular position within artificial reality content 122. A position for a virtual content item may be fixed, as relative to one of walls 121 or the earth, for instance. A position for a virtual content item may be variable, as relative to peripheral device 136 or a user, for instance. In some examples, the particular position of a virtual content item within artificial reality content 122 is associated with a position within the real world, physical environment (e.g., on a surface of a physical object).

[0025] In the example shown in FIG. 1A, virtual content items 124, 126 are mapped to positions on wall 121. The example in FIG. 1A also shows that virtual content item 124 partially appears on wall 121 only within artificial reality content 122, illustrating that this virtual content does not exist in the real world, physical environment. Virtual user interface 137 is mapped to a surface of peripheral device 136. As a result, AR system 10 renders, at a user interface position that is locked relative to a position of peripheral device 136 in the artificial reality environment, virtual user interface 137 for display at HMD 112 as part of artificial reality content 122. FIG. 1A shows that virtual user interface 137 appears on peripheral device 136 only within artificial reality content 122, illustrating that this virtual content does not exist in the real world, physical environment. As used herein, a virtual element locked to a position of peripheral device 136 or other physical object is rendered at a position relative to the position of the physical object so as to appear to be part of or otherwise tied in the artificial reality environment to the physical object.

[0026] In some implementations, artificial reality system 10 generates and renders virtual content items 124, 126 (e.g., GIFs, photos, applications, live-streams, videos, text, a web-browser, drawings, animations, representations of data files, or any other visible media) on a virtual surface. A virtual surface may be associated with a planar or other real-world surface (e.g., the virtual surface corresponds to and is locked to a physical planar surface, such as a wall table, or ceiling). In the example shown in FIG. 1A, the virtual surface is associated with wall 121. In other examples, a virtual surface can be associated with a portion of a surface (e.g., a portion of wall 121). In some examples, only the virtual content items contained within a virtual surface are rendered. In other examples, the virtual surface is generated and rendered (e.g., as a virtual plane or as a border corresponding to the virtual surface). In some examples, a virtual surface can be rendered as floating in a virtual or real-world physical environment (e.g., not associated with a particular real-world surface).

[0027] The artificial reality system 10 may render one or more virtual content items in response to a determination that at least a portion of the location of virtual content items is in the field of view 130 of user 110. For example, artificial reality system 10 may render virtual user interface 137 only if peripheral device 136 is within field of view 130 of user 110.

[0028] During operation, the artificial reality application constructs artificial reality content 122 for display to user 110 by tracking and computing pose information for a frame of reference, typically a viewing perspective of HMD 112. Using HMD 112 as a frame of reference, and based on a current field of view 130 as determined by a current estimated pose of HMD 112, the artificial reality application renders 3D artificial reality content which, in some examples, may be overlaid, at least in part, upon the real-world, 3D physical environment of user 110. During this process, the artificial reality application uses sensed data received from HMD 112, such as movement information and user commands, and, in some examples, data from any external sensors 90, such as external cameras, to capture 3D information within the real world, physical environment, such as motion by user 110 and/or feature tracking information with respect to user 110. Based on the sensed data, the artificial reality application determines a current pose for the frame of reference of HMD 112 and, in accordance with the current pose, renders the artificial reality content 122.

[0029] Artificial reality system 10 may trigger generation and rendering of virtual content items based on a current field of view 130 of user 110, as may be determined by real-time gaze tracking of the user, or other conditions. More specifically, image capture devices 138 of HMD 112 capture image data representative of objects in the real world, physical environment that are within a field of view 130 of image capture devices 138. Field of view 130 typically corresponds with the viewing perspective of HMD 112. In some examples, the artificial reality application presents artificial reality content 122 comprising mixed reality and/or augmented reality. As illustrated in FIG. 1A, the artificial reality application may render images of real-world objects, such as the portions of peripheral device 136, hand 132, and/or arm 134 of user 110, that are within field of view 130 along with virtual objects, such as within artificial reality content 122. In other examples, the artificial reality application may render virtual representations of the portions of peripheral device 136, hand 132, and/or arm 134 of user 110 that are within field of view 130 (e.g., render real-world objects as virtual objects) within artificial reality content 122. In either example, user 110 is able to view the portions of their hand 132, arm 134, peripheral device 136 and/or any other real-world objects that are within field of view 130 within artificial reality content 122. In other examples, the artificial reality application may not render representations of the hand 132 or arm 134 of the user.

[0030] During operation, artificial reality system 10 performs object recognition within image data captured by image capture devices 138 of HMD 112 to identify peripheral device 136, hand 132, including optionally identifying individual fingers or the thumb, and/or all or portions of arm 134 of user 110. Further, artificial reality system 10 tracks the position, orientation, and configuration of peripheral device 136, hand 132 (optionally including particular digits of the hand), and/or portions of arm 134 over a sliding window of time. In some examples, peripheral device 136 includes one or more sensors (e.g., accelerometers) for tracking motion or orientation of the peripheral device 136.

[0031] In accordance with techniques of this disclosure, artificial reality system 10 presents a virtual user interface 137 with which users may interact with using a physical device, referred to as a “peripheral device.” Peripheral device 136 is a physical, real-world device having a surface on which AR system 10 overlays virtual user interface 137. That is, AR system 10 virtually renders virtual user interface 137 at a position and orientation so that virtual user interface 137 appears to be a surface of peripheral device 136 or juxtaposed with the surface of peripheral device 136. In this respect, peripheral device 136 operates as a stage for virtual content, such as virtual user interface 137. Peripheral device 136 may include one or more presence-sensitive surfaces for detecting user inputs by detecting a presence of one or more objects (e.g., fingers, stylus) touching or hovering over locations of the presence-sensitive surface. In some examples, peripheral device 136 may include an output display, which may be a presence-sensitive display. AR system 10 may cause peripheral device 136 to deactivate (i.e., turn off) the output display when rendering virtual user interface 137. AR system 10 may, in some examples, only render virtual user interface 137 when the output display is deactivated. In some examples, peripheral device 136 does not include an output display however.

[0032] In some examples, peripheral device 136 may be a smartphone, tablet computer, personal data assistant (PDA), or other hand-held device. In some examples, peripheral device 136 may be a smartwatch, smartring, or other wearable device. Peripheral device 136 may also be part of kiosk or other stationary or mobile system. Peripheral device 136 may or may not include a display device for outputting content to a screen. Peripheral device 136 may be in communication with HMD 112 and/or console 106 using one or more wired or wireless communications links (e.g., Wi-Fi, near-field communication of short-range wireless communication such as Bluetooth).

[0033] The AR system 10 renders, to HMD, glasses or other display device 112, AR content in which virtual user interface 137 is locked to a surface of peripheral device 136. That is, the AR system 10 may render virtual user interface 137 having one or more virtual user interface elements at a position and orientation in the virtual environment that is based on and corresponds to the position and orientation of the physical peripheral device 136 in the physical environment 130. For example, if the peripheral device 136 is positioned in a vertical position (referred to as “portrait mode”), the AR system 10 may render the virtual user interface in portrait mode and at a location corresponding to the position and orientation of the peripheral device 136. If the peripheral device 136 is positioned in a horizontal position (referred to as “landscape mode”), the AR system 10 may render the virtual user interface in landscape mode and at a location corresponding to the position and orientation of the peripheral device 136. In this way, the virtual user interface being rendered in the virtual environment may track the handheld physical peripheral device 136 such that peripheral device 136 appears, to the user, to be outputting virtual user interface 137 on a surface of peripheral device 136.

[0034] Virtual user interface 137 includes one or more virtual user interface elements. Virtual user interface elements may include, for instance, a virtual drawing interface 142, a selectable menu 144 (e.g., a drop-down menu), virtual buttons 146, a directional pad 148, a keyboard, or other user-selectable user interface elements, glyphs, display elements, content, user interface controls, and so forth. The particular virtual user interface elements for virtual user interface 137 may be context-driven based on the current AR applications engaged by the user.

[0035] When a user performs a user interface gesture in the virtual environment at a location that corresponds to one of the virtual user interface elements of virtual user interface 137 overlaid on the peripheral device 136, the AR system 10 detects the user interface and performs an action. For example, the user may press their finger at a physical location on the peripheral device 136 corresponding to a position in the virtual environment at which the AR system 10 renders a virtual user interface button 146 on virtual user interface 137. In this example, the AR system 10 detects this virtual button press gesture and performs an action corresponding to the detected press of a virtual user interface button 146. The AR system 10 may also, for instance, animate a press of the virtual user interface button along with the button press gesture.

[0036] AR system 10 may detect user interface gestures and other gestures using an inside-out or outside-in tracking system of image capture devices 138 and or external cameras. AR system 10 may alternatively, or in addition, detect user interface gestures and other gestures using a presence-sensitive surface. That is, a presence-sensitive interface of peripheral device 136 may receive user inputs that make up a user interface gesture.

[0037] Peripheral device 136 provides haptic feedback to touch-based user interaction by having a physical surface with which the user can interact (e.g., touch, drag a finger across, grab, and so forth). In addition, peripheral device 136 may output other indications of user interaction using an output device. For example, in response to a detected press of a virtual user interface button 146, peripheral device 136 may output a vibration or “click” noise, or peripheral device 136 may generate and output content to a display.

[0038] In some examples, the user may press and drag their finger along physical locations on the peripheral device 136 corresponding to positions in the virtual environment at which the AR system 10 renders virtual drawing interface 142 of virtual user interface 137. In this example, the AR system 10 detects this drawing gesture and performs an action according to the detected press and drag of a virtual drawing interface 142, such as by generating and rendering virtual markings at the positions in the virtual environment. In this way, AR system 10 simulates drawing or writing on peripheral device 136 using virtual user interface 137.

[0039] In some examples, the user may press their finger at a physical location on the peripheral device 136 corresponding to a position in the virtual environment at which the AR system 10 renders a virtual selectable menu 144. In this example, the AR system 10 detects this user interface gesture and performs an action according to the detected press of a virtual selectable menu 144. The virtual selectable menu 144 may comprise a drop-down menu, a pie menu, a list menu, or any menu including a plurality of selectable items. In some examples, the virtual selectable menu 144 include sub-menus, such that when a user selects an item in a main menu of the virtual selectable menu 144, the AR system 10 would render a sub-menu of one or more items corresponding to the selected item in the main menu, the one or more items further selectable by the user. In this way, AR system 10 simulates interacting with a menu on peripheral device 136 using virtual user interface 137.

[0040] In some examples, the user may press their finger at a physical location on the peripheral device 136 corresponding to a position in the virtual environment at which the AR system 10 renders a virtual directional pad 148. In this example, the AR system 10 detects this user interface gesture and performs an action according to the detected press of virtual directional pad 148. In this way, AR system 10 simulates interacting with a directional pad on peripheral device 136 using virtual user interface 137.

[0041] Accordingly, techniques of the disclosure provide specific technical improvements to the computer-related field of rendering and displaying content by an artificial reality system, which may provide one or more practical applications. For example, artificial reality systems as described herein may provide a high-quality artificial reality experience to a user, such as user 110, of the artificial reality application by generating and rendering virtual user interface 137 locked to a surface of peripheral device 136 and detecting gestures performed by user 110 with respect to peripheral device 136.

[0042] In addition, AR systems as described herein may be configured to detect gestures that provide self-haptic feedback to the user. For example, one or more fingers on each hand 132 of the user 110 may touch or approximately touch peripheral device 136 in the physical world as part an interaction with a particular virtual user interface element of virtual user interface 137 in the artificial reality content. The touch between the one or more fingers of the user’s hand 132 and the peripheral device may provide the user with a simulation of the sensation felt by the user when interacting directly with a physical user input object, such as a button on a physical keyboard or other physical input device.

[0043] As another example, AR systems as described herein can enable the user to provide fine-grained user inputs with respect to virtual user interface elements rendered virtually on a peripheral device 136 that provides haptic feedback, in contrast to free-floating virtual user interfaces in the artificial reality environment. This can simplify and improve the precision of gesture detection. In addition, peripheral device 136 may not display the user interface elements at its own display and may not even include a display. The techniques may therefore additionally reduce power consumption and simplify AR applications by eliminating a separate interface that would otherwise need to be generated and displayed, at a presence-sensitive display of a smartphone or tablet for instance, for receiving precision inputs from a user. In some examples, receiving user inputs with a presence-sensitive surface may provide more precise gesture detection than image-based gesture detection techniques.

[0044] FIG. 1B is an illustration depicting another example artificial reality system 20 in accordance with techniques of the disclosure. Similar to artificial reality system 10 of FIG. 1A, artificial reality system 20 of FIG. 1B may present and control virtual user interface 137 locked to a position of peripheral device 136 in the artificial reality environment.

[0045] In the example of FIG. 1B, artificial reality system 20 includes external cameras 102A and 102B (collectively, “external cameras 102”), HMDs 112A-112C (collectively, “HMDs 112”), controllers 114A and 114B (collectively, “controllers 114”), console 106, physical peripheral device 136, and sensors 90. As shown in FIG. 1B, artificial reality system 20 represents a multi-user environment in which an artificial reality application executing on console 106 and/or HMDs 112 presents artificial reality content to each of users 110A-110C (collectively, “users 110”) based on a current viewing perspective of a corresponding frame of reference for the respective user. That is, in this example, the artificial reality application constructs artificial content by tracking and computing pose information for a frame of reference for each of HMDs 112. Artificial reality system 20 uses data received from cameras 102, HMDs 112, and controllers 114 to capture 3D information within the real world environment, such as motion by users 110 and/or tracking information with respect to users 110 and objects 108, for use in computing updated pose information for a corresponding frame of reference of HMDs 112. As one example, the artificial reality application may render, based on a current viewing perspective determined for HMD 112C, artificial reality content 122 having virtual objects 128A-128B (collectively, “virtual objects 128”) as spatially overlaid upon real world objects 108A-108B (collectively, “real world objects 108”). Further, from the perspective of HMD 112C, artificial reality system 20 renders avatars 120A, 120B based upon the estimated positions for users 110A, 110B, respectively.

[0046] Each of HMDs 112 concurrently operates within artificial reality system 20. In the example of FIG. 1B, each of users 110 may be a “player” or “participant” in the artificial reality application, and any of users 110 may be a “spectator” or “observer” in the artificial reality application. HMD 112C may operate substantially similar to HMD 112 of FIG. 1A by tracking hand 132 and/or arm 134 of user 110C and rendering the portions of hand 132 that are within field of view 130 as virtual hand 136 within artificial reality content 122. HMD 112B may receive user inputs from controllers 114 held by user 110B. In some examples, controller 114A and/or 114B can correspond to peripheral device 136 of FIG. 1A and operate substantially similar to peripheral device 136 of FIG. 1A. HMD 112A may also operate substantially similar to HMD 112 of FIG. 1A and receive user inputs in the form of gestures performed on or with peripheral device 136 by of hands 132A, 132B of user 110A. HMD 112B may receive user inputs from controllers 114 held by user 110B. Controllers 114 may be in communication with HMD 112B using near-field communication of short-range wireless communication such as Bluetooth, using wired communication links, or using other types of communication links.

[0047] In a manner similar to the examples discussed above with respect to FIG. 1A, console 106 and/or HMD 112C of artificial reality system 20 generates and renders virtual user interface 136 at a position locked to a surface of peripheral device 136. That is, console 106 and/or HMD 112C may render virtual user interface 137 having one or more virtual user interface elements at a position and orientation in the virtual environment that is based on and corresponds to the position and orientation of the physical peripheral device 136 in the physical environment.

[0048] As shown in FIG. 1B, in addition to or alternatively to image data captured via one or more image capture devices 138 of HMD 112C, input data from external cameras 102 may be used to track and detect particular motions, configurations, positions, and/or orientations of peripheral device 136 and/or hands and arms of users 110, such as hand 132 of user 110C, including movements of individual and/or combinations of digits (fingers, thumb) of the hand. In this way, AR system 20 can overlay virtual user interface 137 on the surface of peripheral device 136 while tracking peripheral device 136 to ensure virtual user interface 137 moves in the virtual environment along with movements of peripheral device 136 in the physical environment.

[0049] In some aspects, the artificial reality application can run on console 106, and can utilize image capture devices 102A and 102B to analyze configurations, positions, and/or orientations of hand 132B to identify input gestures that may be performed by a user of HMD 112A. Similarly, HMD 112C can utilize image capture device 138 to analyze configurations, positions, and/or orientations of peripheral device 136 and hand 132C to input gestures that may be performed by a user of HMD 112C. In some examples, peripheral device 136 includes one or more sensors (e.g., accelerometers) for tracking motion or orientation of the peripheral device 136. The artificial reality application may render virtual content items and/or user interface elements, responsive to such gestures, in a manner similar to that described above with respect to FIG. 1A.

[0050] Image capture devices 102 and 138 may capture images in the visible light spectrum, the infrared spectrum, or other spectrum. Image processing described herein for identifying objects, object poses, and gestures, for example, may include processing infrared images, visible light spectrum images, and so forth.

[0051] Virtual user interface 137 is mapped to a surface of peripheral device 136. As a result, AR system 10 renders, at a user interface position that is locked relative to a position of peripheral device 136 in the artificial reality environment, virtual user interface 137 for display at HMD 112C as part of artificial reality content 122. FIG. 1B shows that virtual user interface 137 appears on peripheral device 136 only within artificial reality content 122, illustrating that this virtual content does not exist in the real world, physical environment.

[0052] FIG. 2A is an illustration depicting an example HMD 112 and an example peripheral device 136, in accordance with techniques of the disclosure. HMD 112 of FIG. 2A may be an example of any of HMDs 112 of FIGS. 1A and 1B. HMD 112 may be part of an artificial reality system, such as artificial reality systems 10, 20 of FIGS. 1A, 1B, or may operate as a stand-alone, mobile artificial realty system configured to implement techniques described herein.

[0053] In this example, HMD 112 includes a front rigid body and a band to secure HMD 112 to a user. In addition, HMD 112 includes an interior-facing electronic display 203 configured to present artificial reality content to the user. Electronic display 203 may be any suitable display technology, such as liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating visual output. In some examples, the electronic display is a stereoscopic display for providing separate images to each eye of the user. In some examples, the known orientation and position of display 203 relative to the front rigid body of HMD 112 is used as a frame of reference, also referred to as a local origin, when tracking the position and orientation of HMD 112 for rendering artificial reality content according to a current viewing perspective of HMD 112 and the user. In other examples, HMD 112 may take the form of other wearable head mounted displays, such as glasses or goggles.

[0054] As further shown in FIG. 2A, in this example, HMD 112 further includes one or more motion sensors 206, such as one or more accelerometers (also referred to as inertial measurement units or “IMUs”) that output data indicative of current acceleration of HMD 112, GPS sensors that output data indicative of a location of HMD 112, radar or sonar that output data indicative of distances of HMD 112 from various objects, or other sensors that provide indications of a location or orientation of HMD 112 or other objects within a physical environment. Moreover, HMD 112 may include integrated image capture devices 138A and 138B (collectively, “image capture devices 138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. More specifically, image capture devices 138 capture image data representative of objects (including peripheral device 136 and/or hand 132) in the physical environment that are within a field of view 130A, 130B of image capture devices 138, which typically corresponds with the viewing perspective of HMD 112. HMD 112 includes an internal control unit 210, which may include an internal power source and one or more printed-circuit boards having one or more processors, memory, and hardware to provide an operating environment for executing programmable operations to process sensed data and present artificial reality content on display 203.

[0055] In some examples, control unit 210 is configured to, based on the sensed data (e.g., image data captured by image capture devices 138, position information from GPS sensors), generate and render for display on display 203 a virtual surface comprising one or more virtual content items (e.g., virtual content items 124, 126 of FIG. 1A) associated with a position contained within field of view 130A, 130B of image capture devices 138. As explained with reference to FIGS. 1A-1B, a virtual content item may be associated with a position within a virtual surface that is associated with a physical surface within a real-world environment, and control unit 210 can be configured to render the virtual content item (or portion thereof) for display on display 203 in response to a determination that the position associated with the virtual content (or portion therefore) is within the current field of view 130A, 130B. In some examples, a virtual surface is associated with a position on a planar or other surface (e.g., a wall), and control unit 210 will generate and render the portions of any virtual content items contained within that virtual surface when those portions are within field of view 130A, 130B.

[0056] In some examples, in accordance with techniques described herein, control unit 210 is configured to, based on the sensed data, identify a specific gesture or combination of gestures performed by the user and, in response, perform an action. For example, in response to one identified gesture, control unit 210 may generate and render a specific user interface for display on electronic display 203 at a position locked relative to peripheral device 136. For example, control unit 210 can generate and render a user interface including one or more user interface elements (e.g., virtual buttons) on surface 220 of peripheral device 136. As explained herein, in accordance with techniques of the disclosure, control unit 210 may perform object recognition within image data captured by image capture devices 138 to identify peripheral device 136 and/or a hand 132, fingers, thumb, arm or another part of the user, and track movements, positions, configuration, etc., of the peripheral device 136 and/or identified part(s) of the user to identify pre-defined gestures performed by the user. In response to identifying a pre-defined gesture, control unit 210 takes some action, such as selecting an option from an option set associated with a user interface (e.g., selecting an option from a user interface menu), translating the gesture into input (e.g., characters), launching an application, manipulating virtual content (e.g., moving, rotating a virtual content item), generating and rendering virtual markings, generating and rending a laser pointer, or otherwise displaying content, and the like. For example, control unit 210 can dynamically generate and present a user interface, such as a menu, in response to detecting a pre-defined gesture specified as a “trigger” for revealing a user interface (e.g., turning peripheral device to a landscape or horizontal orientation (not shown)). In some examples, control unit 210 detects user input, based on the sensed data, with respect to a rendered user interface (e.g., a tapping gesture performed on a virtual user interface element). In some examples, control unit 210 performs such functions in response to direction from an external device, such as console 106, which may perform, object recognition, motion tracking and gesture detection, or any part thereof.

[0057] As an example, control unit 210 can utilize image capture devices 138A and 138B to analyze configurations, positions, movements, and/or orientations of peripheral device 136, hand 132 and/or arm 134 to identify a user interface gesture, selection gesture, stamping gesture, translation gesture, rotation gesture, drawing gesture, pointing gesture, etc., that may be performed by users with respect to peripheral device 136. The control unit 210 can render a virtual user interface (including virtual user interface elements) and/or a virtual surface (including any virtual content items) and enable the user to interface with the virtual user interface and/or virtual surface based on detection of a user interface gesture, selection gesture, stamping gesture, translation gesture, rotation gesture, and drawing gesture performed by the user with respect to the peripheral device, as described in further detail below.

[0058] In one example, in accordance with techniques described herein, surface 220 of peripheral device 136 is a presence-sensitive surface, such as a surface that uses capacitive, conductive, resistive, acoustic, or other technology to detect touch and/or hover input. In some examples, surface 220 of peripheral device 136 is a touchscreen (e.g., a capacitive touchscreen, resistive touchscreen, surface acoustic wave (SAW) touchscreen, infrared touchscreen, optical imaging touchscreen, acoustic pulse recognition touchscreen, or any other touchscreen). In such an example, peripheral device 136 can detect user input (e.g., touch or hover input) on surface 220. In some examples, surface 220 does not include a display and peripheral device 136 does not include a display.

[0059] In examples with a touchscreen for surface 220, peripheral device 136 may communicate detected user input to HMD 112 (and/or console 106 of FIG. 1A) using wireless communications links (e.g., Wi-Fi, near-field communication of short-range wireless communication such as Bluetooth), using wired communication links (not shown), or using other types of communication links. In some examples, peripheral device 136 can include one or more input devices 222 (e.g., buttons, trackball, scroll wheel) for interacting with virtual content (e.g., to select a virtual user interface element, scroll through virtual user interface elements).

[0060] FIG. 2B is an illustration depicting an example HMD 112, in accordance with techniques of the disclosure. As shown in FIG. 2B, HMD 112 may take the form of glasses. HMD 112 of FIG. 2A may be an example of any of HMDs 112 of FIGS. 1A and 1B. HMD 112 may be part of an artificial reality system, such as artificial reality systems 10, 20 of FIGS. 1A, 1B, or may operate as a stand-alone, mobile artificial realty system configured to implement techniques described herein.

[0061] In this example, HMD 112 are glasses comprising a front frame including a bridge to allow the HMD 112 to rest on a user’s nose and temples (or “arms”) that extend over the user’s ears to secure HMD 112 to the user. In addition, HMD 112 of FIG. 2B includes interior-facing electronic displays 203A and 203B (collectively, “electronic displays 203”) configured to present artificial reality content to the user. Electronic displays 203 may be any suitable display technology, such as liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating visual output. In the example shown in FIG. 2B, electronic displays 203 form a stereoscopic display for providing separate images to each eye of the user. In some examples, the known orientation and position of display 203 relative to the front frame of HMD 112 is used as a frame of reference, also referred to as a local origin, when tracking the position and orientation of HMD 112 for rendering artificial reality content according to a current viewing perspective of HMD 112 and the user.

[0062] As further shown in FIG. 2B, in this example, HMD 112 further includes one or more motion sensors 206, such as one or more accelerometers (also referred to as inertial measurement units or “IMUs”) that output data indicative of current acceleration of HMD 112, GPS sensors that output data indicative of a location of HMD 112, radar or sonar that output data indicative of distances of HMD 112 from various objects, or other sensors that provide indications of a location or orientation of HMD 112 or other objects within a physical environment. Moreover, HMD 112 may include integrated image capture devices 138A and 138B (collectively, “image capture devices 138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. HMD 112 includes an internal control unit 210, which may include an internal power source and one or more printed-circuit boards having one or more processors, memory, and hardware to provide an operating environment for executing programmable operations to process sensed data and present artificial reality content on display 203.

[0063] FIG. 3 is a block diagram showing example implementations of console 106, HMD 112, and peripheral device 136 of artificial reality system 10, 20 of FIGS. 1A, 1B. In the example of FIG. 3, console 106 performs pose tracking, gesture detection, and virtual user interface generation and rendering for HMD 112 in accordance with techniques described herein based at least on sensed data, such as motion data and image data received from HMD 112 and/or external sensors, and in some cases on indications of user input received by peripheral device 136.

……
……
……