Facebook Patent | Augmenting Reality with Reactive Programming

Patent: Augmenting Reality with Reactive Programming

Publication Number: 20180357802

Publication Date: 2018-12-13

Applicants: Facebook, Inc.

Abstract

In one or more embodiments, one or more systems, methods, and/or processes may utilize a reactive processing system in an augmented reality (AR) system. The AR system may execute program instructions based on a region of an input video signal and produce an output video signal that includes a first output that includes a video effect from the program instructions. The AR system may determine if the region of the input video signal changes. If so, the program instructions may be executed based on the region of the input video signal and a change of the region of the input video and produce the output video signal that includes the second output that includes the video effect. If not, the output video signal that includes the first output that includes the video effect may continue to be produced.

BACKGROUND 

Augmented reality (AR) can allow virtual objects to be placed in a video stream. For example, AR can augment and/or modify input video with computer-generated sound, video, and/or graphics. In one instance, an AR system can utilize face-tracking software that allows a user to appear to wear a mask that moves with the user's face and/or changes expressions. In another instance, a nod of the user's head can cause the mask to move up and down over the user's face. Consumer-facing offerings like this can drive popularity in AR systems.

Realistic integration of the computer-generated sound, video, or graphics into the input video can be one measure of an AR system. AR systems process various facets of input video, and continually processing the various facets of the input video, regardless of any change or constancy in the input video, can introduce latency into an output video from an AR system. The latency can slow or reduce the integration of the computer-generated sound, video, and/or graphics into the input video.

SUMMARY

In one or more embodiments, one or more systems, methods, and/or processes may utilize a reactive processing system in an augmented reality system. In one example, the reactive system may utilize one or more of a dependency graph and an expression template, among others. In another example, the reactive system may utilize an effects engine that augments one or more video streams with video effects. In one or more embodiments, one or more of facial recognition and object recognition, among others, may provide one or more data bindings (e.g., one or more positions on a person in a video stream, one or more paths along the person in the video stream, etc.) may be utilized in binding functions that produce video effects. For example, the functions that produce video effects may be developed by programmers and uploaded to the reactive system.

In one or more embodiments, data bindings may include time, various parts of a person, a path (e.g., a path along a face, a path along a body, a path along an inanimate object, etc.), and a gyroscope, among others. For example, an application programming interface (API) may permit a programmer to attach a function that produces a video effect to a data binding. For instance, a computer vision process and/or system may provide one or more data bindings to one or more of a face of a person and another portion of the person, among others.

In one or more embodiments, the reactive system may utilize a video input signal, the processed input signal with one or more functions provided by one or more programmers, and provide video output with video effects of the one or more functions provided by the one or more programmers. For example, from video frame to video frame, if the input signal does not change, a function may not be executed, and the output video may remain unchanged. For instance, the reactive system may determine if a portion of the video input signal, that corresponds to a data binding of a function, changes. If the portion of the video input signal that corresponds to the data binding of the function changes, the function may be executed, and one or more of a video effect corresponding to the function may change in an output video signal (e.g., output video stream) and a position of the video effect corresponding to the function may change in the output video signal. If the portion of the video input signal that corresponds to the data binding of the function does not change, a corresponding portion of the output video signal that includes the video effect may not change. In this fashion, less processing in an augmented reality system may be accomplished, according to one or more embodiments. For example, less processing in the augmented reality system may reduce or eliminate latency of the augmented reality system.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. One or more embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. One or more embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed includes not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.