Facebook Patent | Augmented-Reality Image Processing

Patent: Augmented-Reality Image Processing

Publication Number: 10580218

Publication Date: 20200303

Applicants: Facebook

Abstract

In one embodiment, a computing system accesses a first tracking record of a first user during a first movement session. The first tracking record comprises a plurality of locations of the first user and associated time measurements. During a second movement session, the system determines a current location of a second user and an associated current time measurement. From the plurality of locations in the first tracking record, a first location of the first user in the first movement session is determined based on (1) the associated time measurement relative to a start time of the first movement session and (2) the current time measurement relative to a start time of the second movement session. The system determines a display position of a virtual object on a display screen of the second user based on the first location relative to the current location of the second user.

TECHNICAL FIELD

This disclosure generally relates to image processing, and in particular to systems and methods for augmented reality effects.

BACKGROUND

Augmented Reality (AR) effects are computer-generated visual effects (e.g., images and animation) that are superimposed or integrated into a user’s view of a real-world scene. Certain AR effects may be configured to be placed with objects in the real world. For example, a computer-generated unicorn may be visually placed on a real-world table. The augmented effect may be presented in a variety of ways. For example, the real-world table may be captured by the camera of a mobile smartphone and displayed in real-time through the device’s display. In addition to displaying the table, the computing device may also display the unicorn, integrating it with the video of the table that is being displayed. As another example, a user may be viewing the real-world through AR glasses or viewing optics. Like regular glasses, the AR glasses may have one or more lenses that are translucent so that the user may see real-world objects through the lenses. A computer-generated unicorn may be projected onto the lenses so that the unicorn is seen along with any real-world object that is visible through the lenses.

A mobile computing device, such as a smartphone, AR glasses or headset, tablet computer, or laptop computer, may include functionality for determining its location, direction, or orientation, such as a GPS receiver, compass, gyroscope, or accelerometer. Such a device may also include functionality for wireless communication, such as BLUETOOTH communication, near-field communication (NFC), or infrared (IR) communication or communication with a wireless local area networks (WLANs) or cellular-telephone network. Such a device may also include one or more cameras, scanners, touchscreens, microphones, or speakers. Mobile computing devices may also execute software applications, such as games, web browsers, or social-networking applications. With social-networking applications, users may connect, communicate, and share information with other users in their social networks.

SUMMARY OF PARTICULAR EMBODIMENTS

Movement progress or performance is often measured based on distance as a function of time. For example, a jogger may measure his jogging performance based on how far he has traveled over a period of time. To compare his current performance with his past performance or with someone else’s performance, the jogger typically can only compare metrics that measure the end results: he could compare the amount of time taken to complete a particular jogging distance or route, or he could compare the distance jogged within a particular duration. In either case, the jogger lacks feedback on his progress or performance while he is jogging and only has a sense of how he did after-the-fact.

Embodiments described herein relate to an AR feature where a virtual travel or pacing companion is presented to a user while he is engaged in an activity (e.g., jogging, walking, biking, driving, etc.) in order to provide the user with visual progress comparison in real-time. In particular embodiments, the user’s locations may be tracked and used for positioning a virtual reference object (e.g., an avatar) for display on a user’s AR device while the user is engaged in the activity. That virtual reference object, such as a virtual avatar or character, may be presented to the user as an AR effect that is integrated with real-world scenes. Based on the relative position of that virtual reference object (e.g., the virtual reference object may appear ahead of or behind the user), the user may gauge how well he is currently performing.

One challenge with the above feature relates to how to present the virtual reference object (or “virtual object” for short) so that the visual cues associated with the virtual reference object can accurately reflect and convey the relative position and/or orientation between the user and the reference object. In particular embodiments, a computing system may track the current user’s location with respect to a start time and display a virtual reference object in the user’s field of view based on the relative position between the user and the reference object at corresponding moments in time. For example, if the user has jogged for 3 minutes in the current jogging session, the computing system may ascertain from previous tracking data of the user (or another user) where he was located after jogging for 3 minutes. The relative position between where the user currently is and where he was previously may be used to determine a depth and/or orientation measurement for the virtual reference object with respect to the user. The appearance of the virtual object can be adjusted based on the determined depth (e.g., the object may appear larger if it is closer to the user and smaller if it is farther). Where the virtual object is positioned within the field of view of the user can be determined in accordance with at least the depth and/or orientation of the virtual object, as well as the slope or contour of the surrounding real-world scene (e.g., the streets may slope upwards or downwards). For example, the virtual object can be displayed higher on the user’s AR display screen, such as a pair of AR glasses or a smartphone’s screen, if the user is looking and jogging up a hill and the virtual object is ahead of the user.

In general, embodiments described herein relate to an AR feature that provides real-time visual comparison of a user’s current activity against past recorded activities of the user himself or past or current activities of others. An AR application may track and record a first movement session of a user (or another person, such as a friend of the user). The tracking record may comprise a plurality of locations with respect to a first start time of the first movement session, and each location is associated with a time at which the location may be recorded. Then, the AR application can be invoked again to track the user’s current location in a second movement session with respect to a second start time. Based on the user’s current location, the AR application may determine how the virtual companion should appear in the field of view of the user. The AR application, using the tracking record, may first determine a past location (e.g., GPS coordinates) or travel distance (e.g., half a mile) of the user at a corresponding moment in the first movement session based on the current time in the second session. For example, if the user has jogged for 3 minutes in the current jogging session since the start of the session, the computing system may retrieve the user’s past location or traveled distance when he was 3 minutes into his previous jog. The past location or travel distance of the user in the prior session may be used to determine a relative position between the user’s current location and the virtual companion, which in turn may be used to determine where the virtual companion should be in the real-world scene. The depth (i.e., distance from the current user) and orientation of the virtual companion can be determined based on the relative position, and the position of the virtual companion on the user’s AR display can be determined by the AR application. The appearance of the virtual companion may be adjusted accordingly to be realistically displayed on the user’s AR display (e.g. AR glasses or smartphone screen) based on the determined virtual companion’s position.

In particular embodiments, the application may first track and record a user’s first movement session using a particular tracking method, such as GPS. The AR application may be triggered by a start time, such as when the user presses a “start” button, and start tracking the user’s locations (e.g., GPS coordinates including longitude and latitude) along a route. Each snapshot of the user’s location information may include a GPS coordinate, time, altitude, pace, or any other relevant information related to the instant at which the user’s location was recorded. For example, for each location tracked, a corresponding time may be associated with the location at which the location is recorded. Based on the tracking record, the AR application can figure out a corresponding past location in the tracking record with respect to the user’s current movement, such that if the user has been running for 10 seconds from a start time in the current session, the corresponding past location may be determined when the user had been running for 10 second during the recorded first movement session.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed above. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a snapshot of an example scenario 100 during a first movement session, in accordance with particular embodiments.

FIG. 2 illustrates a snapshot of an example scenario 200 during a second movement session, in accordance with particular embodiments.

FIG. 3 illustrates an example method of creating a virtual reference object by a computing system in particular embodiments.

FIG. 4 illustrates an example network environment associated with a social-networking system in particular embodiments.

FIG. 5 illustrates an example social graph in particular embodiments.

FIG. 6 illustrates an example computer system in particular embodiments.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In particular embodiments, a computing system may be configured to create a virtual “companion” (e.g., for performance comparison or pacing purposes) and integrate the virtual companion as an AR effect into a user’s view of a real-world scenes. Based on the relative position of that virtual companion and the user’s current location, the user be presented with a visual comparison of his current movement (e.g., jogging, walking, biking, or skiing progress) against his own past movement or another user’s past or concurrent movement at a corresponding moment in time with respect to a start time. For example, with a pair of AR glasses or a smartphone screen, the user may be able to see the virtual companion running ahead or behind him in the field of view of the user.

In particular embodiments, the computing system may activate an AR application to track a plurality of user’s locations (e.g., GPS locations) and record the user’s location data in a tracking record. The record may represent a prior running record of the user from a start time, and it may be used in a subsequent activity session (e.g., running) to generate a virtual companion. The virtual companion may be displayed on the user’s AR glasses for visually demonstrating where the user was in the prior activity. The display of the virtual companion may be determined based on the user’s past location in the tracking record, the user’s current location tracked by the computing system, the orientation between those positions, and/or the characteristics of the current real-world environment (e.g., depth information, slope information, road locations, etc.). As an example, the virtual companion can act as a running (or other types of activity, such as driving or sailing) teammate for training or comparison purpose. The virtual companion can represent the user himself or others in a prior activity session, such that the virtual companion can be used to visually show the user how the user is currently doing compared to how he or others did in the prior session. For example, when the user is jogging slower in the current session than he was in a recorded prior session, the virtual companion may be seen jogging ahead of the user on the user’s AR glasses. As another example, when the user is faster in the current session than he was in the recorded prior session, the virtual companion may be seen jogging behind the user when the user turns around.

FIG. 1 illustrates a snapshot of an example scenario 100 during a first movement session, in accordance with particular embodiments. The scenario 100 shows a user 135 jogging along a route 120. Using an AR application, the user 135 may begin recording (e.g., stored locally on the device running the AR application or a mobile device or stored remotely on a server) a movement session at starting location 110. For example, at the beginning of the user’s run, he may have launched the AR application on his mobile device and indicated that a movement session is to be recorded (e.g., by pressing a button, giving a voice command, or making a gesture to indicate a desire to start the recording). The starting location 100 is represented by (x.sub.0, y.sub.0), which may represent the GPS location measured by the user’s mobile device (e.g., longitude and latitude coordinates), and the start time of the movement session is represented by to. Once the movement session begins, the AR application may track and record the user’s movements and the times at which the locations are recorded. In the illustrated snapshot scenario 100, the user 135 may have jogged to location 130, represented by (x.sub.i, y.sub.i), at time t.sub.1. The user 135 may continue to jog until the movement session ends, at which time the recording may cease. The ending location 140, denoted by (x.sub.N, y.sub.N) and associated with time t.sub.N, represents the location at which the movement session ends. The movement session may be terminated by the user 135 (e.g., by pressing a button, giving a voice command, or making a gesture to indicate a desire to end the session) or it may terminate automatically (e.g., upon satisfying one or more predetermined criteria, such as after reaching a predetermined jogging duration, distance, or destination). At the end of the movement session, N number of locations and their associated times may be recorded and associated with that movement session. The recordings may include data of the user’s movement during the movement session (e.g., as the user 135 jogs along the route 120), such as a plurality of locations from the starting location 110 to the ending location 140 and the time at which each of the plurality of locations is recorded.

The user’s movement data may be represented and recorded in a variety of ways. In particular embodiments, as described above, movement recordings may be represented by GPS coordinates. In other embodiments, each movement recording may be represented by the distance traveled since the movement session began. For example, the AR application may use the GPS readings from t.sub.0 to t.sub.i to estimate the distance traveled by the user 135 in that time period and associate that distance with the recording time t.sub.i. In particular embodiments, the AR application may also record the user’s altitude at the recording time t.sub.i. In particular embodiments, the AR application may further record the user’s velocity and acceleration (e.g., based on GPS and/or accelerometer measurements) at the recording time t.sub.i.

The recorded time data associated may also be represented in a variety of ways. For example, the recorded time associated each recording may be an absolute timestamp in Unix time (e.g., the user 135 was at location 130 (x.sub.i, y.sub.i) at timestamp t.sub.i=2018-06-10T11:03:00.00 or an equivalent sequential-number representation). In this case, the user’s running duration d can be computed as d=t.sub.i-t.sub.0. As another example, the recorded time may be a timestamp or duration relative to the start of the movement session. For instance, a relative timestamp, which may also represent a duration since the start of the movement session, may begin counting when the movement session begins, similar to a stopwatch.

The data recorded in the first movement session (e.g., as described with reference to FIG. 1) may be used to generate a virtual companion in a second movement session. In particular embodiments, the second movement session may be invoked by the same user whose movement data were recorded in the first movement session. For example, after the user has recorded his jogging data in the first movement session, he may jog again at a subsequent time and use the virtual companion to see, while jogging, how he is currently doing compared to how he did in the first movement session. In particular embodiments, the user in the second movement session may be different from the user in the first movement session. For example, after user A recorded his movement data in the first movement session, the recorded data may be used by user B in a second movement session so that user B can see how he does compared to user A. In particular embodiments, the movement data recorded in two movement session may be shared in real-time to enable two users to compare their performances as they are both engaged in the activity (e.g., two joggers may race against each other in substantially real-time). For example, the AR applications executing on the users’ respective devices may communicate through a server (e.g., a social networking server). The AR applications may begin respective movement sessions at the same time and share the recorded movement data. For example, at time t.sub.i, user A’s movement data (e.g., distance traveled since the start time) may be sent to user B’s device and user B’s movement data (e.g., distance traveled since the start time) may be sent to user A’s device. Using the movement data, each user’s AR application may render a virtual companion that represents the other user, giving the users a virtual racing experience.

FIG. 2 illustrates a snapshot of an example scenario 200 during a second movement session. The scenario 200 illustrates where a virtual companion 255, generated based on the movement data recorded in the first movement session (e.g., as described with reference to FIG. 1), would conceptually be located relative to a user 235 (who could be the same user 135 who recorded the first movement session or a different user). The example shows the user 235 jogging along a route 220, which could be the same route 120 shown in FIG. 1 or a different one. If the routes are the same, particular embodiments may determine the virtual companion’s 255 relative position to the user 235 based on GPS coordinates or distance measurements. If the routes are different, particular embodiments may determine the virtual companion’s 255 relative position based on distance measurements. The distance measurements may be what was recorded during the first movement session or derived from recordings of GPS coordinates. To derive distance measurements from recorded GPS coordinates, the AR application may translate the coordinates into distance measurements by computing the distance traveled between successive GPS locations.