Facebook Patent | Self presence in artificial reality

Patent: Self presence in artificial reality

Drawings: Click to check drawins

Publication Number: 20210209854

Publication Date: 20210708

Applicant: Facebook

Abstract

The disclosed artificial reality system can provide a user self representation in an artificial reality environment based on a self portion from an image of the user. The artificial reality system can generate the self representation by applying a machine learning model to classify the self portion of the image. The machine learning model can be trained to identify self portions in images based on a set of training images, with portions tagged as either depicting a user from a self-perspective or not. The artificial reality system can display the self portion as a self representation in the artificial reality environment by positioning them in the artificial reality environment relative to the user’s perspective in the artificial reality environment. The artificial reality system can also identify movements of the user and can adjust the self representation to match the user’s movement, providing more accurate self representations.

Claims

1. A method for providing a self representation of a user in an artificial reality environment, the method comprising: receiving one or more images captured in real time by an artificial reality system; classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user’s own body in an image; and displaying, in the artificial reality environment and as the self representation, the self portion of each of at least one of the one or more images, at a virtual location relative to a user perspective in the artificial reality environment.

2. The method of claim 1 further comprising adjusting at least part of the one or more images to appear to be from a user’s perspective by: obtaining, from multiple cameras on the artificial reality system, multiple contemporaneously taken images; matching features between the multiple contemporaneously taken images; merging the multiple contemporaneously taken images into a single image; determining one or more distances between A) one or more eyes of the user and B) the multiple cameras; and adjusting, based on the determined one or more distances, the single image into one of the one or more images that appear to be from a user’s perspective.

3. The method of claim 1 further comprising adjusting at least part of the one or more images to appear to be from a user’s perspective by: determining one or more distances between A) one or more eyes of the user and B) one or more cameras on the artificial reality system; and adjusting, based on the determined one or more distances, the one or more images to be from a user’s perspective.

4. The method of claim 1, wherein classifying the self portion includes: generating an image mask based on the output of the machine learning model; and applying the image mask to at least part of the one or more images to obtain the self portion of each of the one or more images.

5. The method of claim 1, wherein the machine learning model is trained using a set of images with portions of each image tagged to indicate whether that portion depicts a self portion of a user not.

6. The method of claim 1, wherein classifying the self portion in each of the one or more images includes using the machine learning model to classify each of multiple individual pixels of each image as either depicting or not depicting a part of the user.

7. The method of claim 1, wherein classifying the self portion in each specific image of the one or more images includes classifying parts of the specific image as depicting particular body parts.

8. The method of claim 7 further comprising: receiving, from an application controlling part of the artificial reality environment, an indication of an effect to apply to a depiction of a particular body part of the user; and applying, based on the classified parts of the specific image as depicting particular body parts, the effect to the depiction of the particular body part of the user.

9. The method of claim 1, further comprising receiving, from an application controlling part of the artificial reality environment, an indication of an effect to apply to at least part of the displayed self portion of each of the one or more images, wherein the effect comprises one or more of: a color; a shading; a warp or distortion field; a composite layer to overlay onto at the at least part of the displayed self portion; or any combination thereof.

10. The method of claim 1 wherein, displaying the self portion of each of the one or more images includes overwriting a portion of a frame buffer, written to by an application controlling part of the artificial reality environment, with data for the self portion of each of the one or more images; and the application controlling part of the artificial reality environment does not have access to all of the data for the self portion of each of the one or more images.

11. The method of claim 1 further comprising: identifying a user movement based on identified movement of a controller or a tracked user body part; and warping the self portion of at least one of displayed one or more images to conform to the identified movement.

12. A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for providing a user self representation in an artificial reality environment, the operations comprising: receiving one or more images captured by an artificial reality system; classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user’s own body in an image; and displaying, in the artificial reality environment and as the self representation, the self portion of each of at least one of the one or more images, at a virtual location relative to a user perspective in the artificial reality environment.

13. The computer-readable storage medium of claim 12, wherein the operations further comprise adjusting at least part of the one or more images to appear to be from a user’s perspective by: determining one or more distances relative to one or more cameras on the artificial reality system; and warping, based on the determined one or more distances, the one or more images to be from a user’s perspective.

14. The computer-readable storage medium of claim 12, wherein classifying the self portion includes: generating an image mask based on the output of the machine learning model; and applying the image mask to at least part of the one or more images to obtain the self portion of each of the one or more images.

15. The computer-readable storage medium of claim 12, wherein the machine learning model is a deep neural network trained using a set of images with portions of each image tagged to indicate whether that portion depicts a part of a user or not.

16. The computer-readable storage medium of claim 12, wherein classifying the self portion in each specific image of the one or more images includes classifying parts of the specific image as depicting particular body parts, and wherein the operations further comprise: receiving, from an application controlling part of the artificial reality environment, an indication of an effect to apply to a depiction of a particular body part of the user; and applying, based on the classified parts of the specific image as depicting particular body parts, the effect to the depiction of the particular body part of the user.

17. The computer-readable storage medium of claim 12, wherein displaying the self portion of each of the one or more images includes overwriting a portion of a frame buffer, written to by an application controlling part of the artificial reality environment, with data for the self portion of each of the one or more images.

18. A computing system for providing a user self representation in an artificial reality environment, the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: receiving one or more images captured by an artificial reality system; classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user’s own body in an image; and displaying, in the artificial reality environment and as the self representation, the self portion of each of at least one of the one or more images, at a virtual location relative to a user perspective in the artificial reality environment.

19. The computing system of claim 18, wherein classifying the self portion in each specific image of the one or more images includes classifying parts of the specific image as depicting particular body parts, and wherein the operations further comprise: receiving, from an application controlling part of the artificial reality environment, an indication of an effect to apply to a depiction of a particular body part of the user; and applying, based on the classified parts of the specific image as depicting particular body parts, the effect to the depiction of the particular body part of the user.

20. The computing system of claim 18, wherein displaying the self portion of each of the one or more images includes overwriting a portion of a frame buffer with data for the self portion of each of the one or more images.

Description

TECHNICAL FIELD

[0001] The present disclosure is directed to artificial reality systems with self-views to provide immersive experiences.

BACKGROUND

[0002] Artificial reality systems provide users the ability to experience different worlds, learn in new ways, and make better connections with others. These artificial reality systems can track user movements and translate them into interactions with virtual objects. For example, an artificial reality system can track a user’s hands, translating a grab gesture as picking up a virtual object, and can track a user’s feet to identify when a user kicks a virtual object. When performing such actions in an artificial reality environment, users often feel disconnected from the environment, or can even become physically sick, when they look down and see a computer-generated environment that does not show the user’s body. Some artificial reality systems have addressed this issue by mapping tracked user body parts to a user kinematic model to determine relative orientations of the user’s body parts. These systems can then create an avatar of the user in the artificial reality environment that generally coincides with the user’s movements.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] FIG. 1 is a block diagram illustrating an overview of devices on which some implementations of the present technology can operate.

[0004] FIG. 2A is a wire diagram illustrating a virtual reality headset which can be used in some implementations of the present technology.

[0005] FIG. 2B is a wire diagram illustrating a mixed reality headset which can be used in some implementations of the present technology.

[0006] FIG. 3 is a block diagram illustrating an overview of an environment in which some implementations of the present technology can operate.

[0007] FIG. 4 is a block diagram illustrating components which, in some implementations, can be used in a system employing the disclosed technology.

[0008] FIG. 5 is a flow diagram illustrating a process used in some implementations of the present technology for displaying a self representation in an artificial reality environment based on one or more images of a user.

[0009] FIG. 6 is a flow diagram illustrating a process used in some implementations of the present technology for merging multiple user images and adjusting them to be from a user’s perspective.

[0010] FIG. 7 is a flow diagram illustrating a process used in some implementations of the present technology for obtaining a self portion of an image.

[0011] FIG. 8 is a flow diagram illustrating a process used in some implementations of the present technology for updating a self representation based on an identified user movement.

[0012] FIGS. 9A-9C are conceptual diagrams illustrating an example of extracting a self portion from an image and using it to create a self representation in an artificial reality environment.

[0013] FIGS. 10A-10C are conceptual diagrams illustrating an example of an artificial reality system capturing multiple images, merging them, and warping them to be from the user’s perspective.

[0014] FIGS. 11A and 11B are conceptual diagrams illustrating an example of creating an image mask for an image and using the image mask to extract a self portion from the image.

[0015] FIGS. 12A and 12B are conceptual diagrams illustrating an example of warping a self representation based on an identified user movement.

[0016] The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements.

DETAILED DESCRIPTION

[0017] Embodiments for providing a self representation of a user in an artificial reality environment based on an identified self portion of images of the user are described herein. A self representation is a representation of a user, in an artificial reality environment, that the user can themselves see. For example, when a user looks down or at her hands when in an artificial reality environment and sees a representation of herself, that is the user’s self representation. The self representation may or may not be visible to other users in the artificial reality environment. A self portion of an image is a portion of an image that depicts the user from that user’s perspective, excluding other parts of the image that do not depict that user. For example, in an image taken from near a user’s perspective, including that user’s hands and chest, another user, and a table, the self portion is just the depicted user’s hands and chest. An artificial reality system can generate the self representation by capturing images of the user, in real time, and applying a machine learning model to classify a self portion of each of the images. The artificial reality system can display a version of the self portions as a self representation in the artificial reality environment by positioning the version in the artificial reality environment relative to the user’s perspective view into the artificial reality environment.

[0018] In some implementations, an artificial reality system can contemporaneously capture images, in real time, using one or more cameras. As used herein, contemporaneous means events that occur at the same time or within a threshold time of each other. For example, contemporaneously captured images refers to images captured at the same time or within a threshold time of each. As also used herein, images captured in real time are images that are captured, processed according to the algorithms described herein, and whose results are provided to create a self representation within a particular time limit that permits the self representation to accurately reflect the user’s current body posture to a threshold level. Examples of this time limit are A) a set number of nano-seconds, B) the amount of time to produce one frame of video, or C) other time limits that keep the lag of the self representation below a threshold level. The artificial reality system can merge the contemporaneously captured images into a single image and adjust them to be from the user’s perspective. This can include determining distances between the center of the user’s eye and each of the cameras and using these distances to warp the images to be from the user’s perspective, instead of from the viewpoint of the cameras that captured them. The artificial reality system can also match features between the images to determine overlap and stitch the images together. These two steps, which may be performed in either order, produce a single image of the real-world environment from the user’s perspective. In some instances only a single camera is used, in which case no image stitching is used but perspective warping may still be applied. Depending on the angle of the camera(s), the resulting image may include a self portion depicting at least part of the artificial reality system user.

[0019] The artificial reality system can identify the self portion by applying a machine learning model to the image. This machine learning model can be of various types such as a type of neural network, a support vector machine, Bayes classifier, decision tree, etc. The machine learning model can be trained to identify self portions in images based on a set of training images, with portions (e.g., set areas, pixels, etc.) tagged as either depicting a user from a self-perspective or not. The model can be trained by applying these training images to the model and adjusting the model based on how close the model output is to the correct output for each portion of the image. For example, where the machine learning model is a neural network, parameters or edge weights can be adjusted such that the output of the model more closely matches the correct classifications for the image portions. Once trained, this machine learning model can then be applied to new images to classify which parts of the image depict the user of the artificial reality system.

[0020] The artificial reality system can use the classifications from the machine learning model to create a mask, which then can be applied to the original image to extract the self portion from the image. The artificial reality system can then display this self portion relative to the perspective of the user in the artificial reality environment, e.g., below the user’s perspective, creating a self representation of the user in the artificial reality environment.

[0021] As an example of the disclosed processes and systems in use, a user may be wearing an artificial reality headset of an artificial reality system with five front and side facing cameras. Within a 2 ms timeframe, the cameras can each capture an image, which the artificial reality system can warp to be from the user’s perspective based on the distance of each camera from the user’s eye and can stitch these five images into a single image. The artificial reality system can then identify a self portion of the image that depicts part of the user’s torso, hands, arms, legs and feet by applying a trained machine learning model. The area of the identified self portion can be used as a mask to extract the self portion from the image. The artificial reality system can then display the extracted self portion in the artificial reality system relative to the user’s point of view, thus allowing the user to see a self representation showing her real-world torso, hands, arms, legs and feet in the artificial reality environment.

[0022] The artificial reality system can also identify movements of the user, e.g., by tracking a controller or a body part of the user. Based on this movement, instead of having to capture a new self portion of the user and create a new self representation, the artificial reality system can adjust the self representation to match the user’s movement. This can provide more accurate self representations. For example, a controller may be able to report its position to an artificial reality system headset more quickly than the artificial reality system can capture images and create a new self representation. By warping the existing self representation to match the movement until a new self representation can be created from more current captured self portions of images, the artificial reality system can keep the self representation spatially accurate according to the user’s body position.

[0023] Embodiments of the disclosed technology may include or be implemented in conjunction with an artificial reality system. Artificial reality or extra reality (XR) is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, a “cave” environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0024] “Virtual reality” or “VR,” as used herein, refers to an immersive experience where a user’s visual input is controlled by a computing system. “Augmented reality” or “AR” refers to systems where a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on the screen on the opposite side of the tablet from the camera. The tablet can process and adjust or “augment” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to systems where light entering a user’s eye is partially generated by a computing system and partially composes light reflected off objects in the real world. For example, a MR headset could be shaped as a pair of glasses with a pass-through display, which allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, allowing the MR headset to present virtual objects intermixed with the real objects the user can see. “Artificial reality,” “extra reality,” or “XR,” as used herein, refers to any of VR, AR, MR, or any combination or hybrid thereof.

[0025] Existing artificial reality systems fail to accurately display representations of a user in an artificial reality environment. Representations of an artificial reality system user created by existing artificial reality systems are based on tracking body parts of the user and mapping those to parts of an avatar created in the artificial reality environment. This analysis of captured images to identify parts of a user, determine spatial relationships of the user, and generate an avatar accordingly positioned is a computationally expensive procedure. In addition, such existing systems tend to lag behind the user’s actual movements and/or, due to inaccuracies in body tracking, do not correctly position parts the avatar in the artificial reality environment to match the user’s movements. Further, due to graphic system limitations, computer generated avatars often fail to provide rich detail and can distract users from their artificial reality experience. Yet, users tend to find artificial reality environments without a representation of the user to be disconcerting and can even make some users nauseous.

[0026] The real-world self representation system and processes described herein overcome these problems associated with existing artificial reality systems and are expected to provide self representations that are less computationally expensive, more accurate, and more detailed than those provided by existing systems. Specifically, the process of capturing images, applying a machine learning model to extract a self portion, and displaying the self portion as a self representation can be performed with significantly less computing power than that required by existing systems to track part of a user, map determined body positions into a virtual space, and render an avatar positioned according to the determined body positions. Further, by taking images of the user and using them directly as the self representation, this process more accurately reflects the user’s movements and doesn’t rely on inaccurate position tracking systems, making the disclosed artificial reality system much more accurate than existing artificial reality systems. Finally, by using real world images of the user instead of computer generated avatars, the self representations provided by the disclosed system can be much more detailed than those provided by existing artificial reality systems, while still being malleable, e.g., through the use of filters and composites.

[0027] Several implementations are discussed below in more detail in reference to the figures. FIG. 1 is a block diagram illustrating an overview of devices on which some implementations of the disclosed technology can operate. The devices can comprise hardware components of a computing system 100 that can prove real-world self representations of a user in an artificial reality environment. In various implementations, computing system 100 can include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate over wired or wireless channels to distribute processing and share input data. In some implementations, computing system 100 can include a stand-alone headset capable of providing a computer created or augmented experience for a user without the need for external processing or sensors. In other implementations, computing system 100 can include multiple computing devices such as a headset and a core processing component (such as a console, mobile device, or server system) where some processing operations are performed on the headset and others are offloaded to the core processing component. Example headsets are described below in relation to FIGS. 2A and 2B. In some implementations, position and environment data can be gathered only by sensors incorporated in the headset device, while in other implementations one or more of the non-headset computing devices can include sensor components that can track environment or position data.

[0028] Computing system 100 can include one or more processor(s) 110 (e.g., central processing units (CPUs), graphical processing units (GPUs), holographic processing units (HPUs), etc.) Processors 110 can be a single processing unit or multiple processing units in a device or distributed across multiple devices (e.g., distributed across two or more of computing devices 101-103).

[0029] Computing system 100 can include one or more input devices 120 that provide input to the processors 110, notifying them of actions. The actions can be mediated by a hardware controller that interprets the signals received from the input device and communicates the information to the processors 110 using a communication protocol. Each input device 120 can include, for example, a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptics glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, e.g., an infrared sensor), a microphone, or other user input devices.

[0030] Processors 110 can be coupled to other hardware devices, for example, with the use of an internal or external bus, such as a PCI bus, SCSI bus, or wireless connection. The processors 110 can communicate with a hardware controller for devices, such as for a display 130. Display 130 can be used to display text and graphics. In some implementations, display 130 includes the input device as part of the display, such as when the input device is a touchscreen or is equipped with an eye direction monitoring system. In some implementations, the display is separate from the input device. Examples of display devices are: an LCD display screen, an LED display screen, a projected, holographic, or augmented reality display (such as a heads-up display device or a head-mounted device), and so on. Other I/O devices 140 can also be coupled to the processor, such as a network chip or card, video chip or card, audio chip or card, USB, firewire or other external device, camera, printer, speakers, CD-ROM drive, DVD drive, disk drive, etc.

[0031] Computing system 100 can include a communication device capable of communicating wirelessly or wire-based with other local computing devices or a network node. The communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols. Computing system 100 can utilize the communication device to distribute operations across multiple network devices.

[0032] The processors 110 can have access to a memory 150, which can be contained on one of the computing devices of computing system 100 or can be distributed across of the multiple computing devices of computing system 100 or other external devices. A memory includes one or more hardware devices for volatile or non-volatile storage, and can include both read-only and writable memory. For example, a memory can include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth. A memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory. Memory 150 can include program memory 160 that stores programs and software, such as an operating system 162, artificial reality self-presence module 164, and other application programs 166. Memory 150 can also include data memory 170 that can include, for example, trained machine learning models, user images, extracted self portions, warping models, configuration data, settings, user options or preferences, etc., which can be provided to the program memory 160 or any element of the computing system 100.

[0033] Some implementations can be operational with numerous other computing system environments or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular telephones, wearable electronics, gaming consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, or the like.

[0034] FIG. 2A is a wire diagram of a virtual reality head-mounted display (HMD) 200, in accordance with some embodiments. The HMD 200 includes a front rigid body 205 and a band 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, locators 225, and one or more compute units 230. The position sensors 220, the IMU 215, and compute units 230 may be internal to the HMD 200 and may not be visible to the user. In various implementations, the IMU 215, position sensors 220, and locators 225 can track movement and location of the HMD 200 in the real world and in a virtual environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locators 225 can emit infrared light beams which create light points on real objects around the HMD 200. One or more cameras (not shown) integrated with the HMD 200 can detect the light points. Compute units 230 in the HMD 200 can use the detected light points to extrapolate position and movement of the HMD 200 as well as to identify the shape and position of the real objects surrounding the HMD 200.

[0035] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to a user as dictated by the compute units 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., a display for each user eye). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.

[0036] In some implementations, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and/or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., via light emitted from the HMD 200) which the PC can use, in combination with output from the IMU 215 and position sensors 220, to determine the location and movement of the HMD 200.

[0037] In some implementations, the HMD 200 can be in communication with one or more other external devices, such as controllers (not shown) which a user can hold in one or both hands. The controllers can have their own IMU units, position sensors, and/or can emit further light points. The HMD 200 or external sensors can track these controller light points. The compute units 230 in the HMD 200 or the core processing component can use this tracking, in combination with IMU and position output, to monitor hand positions and motions of the user. The controllers can also include various buttons a user can actuate to provide input and interact with virtual objects. In various implementations, the HMD 200 can also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc. In some implementations, instead of or in addition to controllers, one or more cameras included in the HMD 200 or external to it can monitor the positions and poses of the user’s hands to determine gestures and other hand and body motions.

[0038] FIG. 2B is a wire diagram of a mixed reality HMD system 250 which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other implementations, the mixed reality system 250 includes a headset only, without an external compute device or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a pass-through display 258 and a frame 260. The frame 260 can house various electronic components (not shown) such as light projectors (e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors, MEMS components, networking components, etc.

[0039] The projectors can be coupled to the pass-through display 258, e.g., via optical elements, to display media to a user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projectors to a user’s eye. Image data can be transmitted from the core processing component 254 via link 256 to HMD 252. Controllers in the HMD 252 can convert the image data into light pulses from the projectors, which can be transmitted via the optical elements as output light to the user’s eye. The output light can mix with light that passes through the display 258, allowing the output light to present virtual objects that appear as if they exist in the real world.

[0040] Similarly to the HMD 200, the HMD system 250 can also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system 250 to, e.g., track itself in 3DoF or 6DoF, track portions of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear as stationary as the HMD 252 moves, and have virtual objects react to gestures and other real-world objects.

[0041] FIG. 3 is a block diagram illustrating an overview of an environment 300 in which some implementations of the disclosed technology can operate. Environment 300 can include one or more client computing devices 305A-D, examples of which can include computing system 100. In some implementations, some of the client computing devices (e.g., client computing device 305B) can be the HMD 200 or the HMD system 250. Client computing devices 305 can operate in a networked environment using logical connections through network 330 to one or more remote computers, such as a server computing device.

[0042] In some implementations, server 310 can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers 320A-C. Server computing devices 310 and 320 can comprise computing systems, such as computing system 100. Though each server computing device 310 and 320 is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations.

[0043] Client computing devices 305 and server computing devices 310 and 320 can each act as a server or client to other server/client device(s). Server 310 can connect to a database 315. Servers 320A-C can each connect to a corresponding database 325A-C. As discussed above, each server 310 or 320 can correspond to a group of servers, and each of these servers can share a database or can have their own database. Though databases 315 and 325 are displayed logically as single units, databases 315 and 325 can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.

……
……
……