Sony Patent | Building virtual reality (vr) gaming environments using real-world virtual reality maps

Patent: Building virtual reality (vr) gaming environments using real-world virtual reality maps

Drawings: Click to check drawins

Publication Number: 20210275923

Publication Date: 20210909

Applicant: Sony

Abstract

A method for building a gaming environment. The method includes accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The method includes augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The method includes using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

Claims

1. A method, comprising: determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; augmenting the object in the base VR model, wherein the object corresponds to a real-world object in the real-world environment; and stitching in the object that is augmented into the base VR model to generate an augmented base VR model.

2. The method of claim 1, wherein the augmenting the object includes: accessing a previously generated surface model corresponding to the real-world object, wherein the previously generated surface model is used as the object that is augmented when performing the stitching.

3. The method of claim 1, wherein the augmenting the object includes: accessing a plurality of images of the real-world object; generating a complete surface model of the object of the real-world object based on the plurality of images, wherein the complete surface model is used as the object that is augmented when performing the stitching.

4. The method of claim 1, wherein the augmenting the object includes: accessing a filler object, wherein the filler object is used as the object that is augmented when performing the stitching.

5. The method of claim 1, further comprising: detecting a missing portion of the surface model, wherein the augmenting the object includes generating an augmented surface model of the object using a surface segment corresponding to the missing portion of the surface model; and wherein the stitching in the object includes generating an augmented base VR model by stitching the augmented surface model into the base VR model.

6. The method of claim 1, further comprising: detecting the real-world object based on one or more images or videos of the real-world environment using a deep learning engine including a plurality of learned models of objects.

7. The method of claim 1, further comprising: detecting a plurality of real-world objects of the real-world environment based on one or more images or videos of the real-world environment; detecting a gap region in the base VR model including one or more missing surface models corresponding to one or more of the plurality of real-world objects; accessing a filler surface model of a filler object; and stitching in the filler surface model into the base VR model in the gap region to generate the augmented base VR model.

8. A method, comprising: determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; generating an augmented surface model of the object using a surface segment corresponding to a missing portion of the surface model, wherein the object corresponds to a real-world object in the real-world environment; and generating an augmented base VR model by stitching the augmented surface model into the base VR model.

9. The method of claim 8, further comprising: accessing the base VR model of a real-world environment from a third party mapping data store.

10. The method of claim 8, wherein the generating an augmented surface model includes: accessing a plurality of images of the real-world object; building the surface segment from the plurality of images; and augmenting the surface model using the surface segment to generate the augmented surface model.

11. The method of claim 8, further comprising: detecting the real-world object based on one or more images or videos of the real-world environment a deep learning engine including a plurality of learned models of objects.

12. The method of claim 8, wherein the surface segment is a generic surface segment.

13. A non-transitory computer-readable medium storing a computer program, the computer-readable medium comprising: program instructions for determining that a surface model of an object in a base VR model of a real-world environment is incomplete, wherein the base VR model includes a plurality of surface models of a plurality of objects corresponding to real-world objects in the real-world environment; program instructions for augmenting the object in the base VR model, wherein the object corresponds to a real-world object in the real-world environment; and program instructions for stitching in the object that is augmented into the base VR model to generate an augmented base VR model.

14. The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a previously generated surface model corresponding to the real-world object, wherein the previously generated surface model is used as the object that is augmented when performing the stitching.

15. The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a plurality of images of the real-world object; program instructions for generating a complete surface model of the object of the real-world object based on the plurality of images, wherein the complete surface model is used as the object that is augmented when performing the stitching.

16. The non-transitory computer-readable medium of claim 13, wherein the program instructions for augmenting the object includes: program instructions for accessing a filler object, wherein the filler object is used as the object that is augmented when performing the stitching.

17. The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting a missing portion of the surface model, wherein the augmenting the object includes generating an augmented surface model of the object using a surface segment corresponding to the missing portion of the surface model; and wherein the stitching in the object includes generating an augmented base VR model by stitching the augmented surface model into the base VR model.

18. The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting the real-world object based on one or more images or videos of the real-world environment using a deep learning engine including a plurality of learned models of objects.

19. The non-transitory computer-readable medium of claim 13, further comprising: program instructions for detecting a plurality of real-world objects of the real-world environment based on one or more images or videos of the real-world environment; program instructions for detecting a gap region in the base VR model including one or more missing surface models corresponding to one or more of the plurality of real-world objects; program instructions for accessing a filler surface model of a filler object; and program instructions for stitching in the filler surface model into the base VR model in the gap region to generate the augmented base VR model.

Description

CLAIM OF PRIORITY

[0001] This application is a continuation of and claims priority to and the benefit of commonly owned, patent application, U.S. Ser. No. 15/911,780, filed on Mar. 5, 2018, Attorney Docket No. SONYP337, entitled “Building Virtual Reality (VR) Gaming Environments Using Real-World Virtual Reality Maps,” the disclosure of which is incorporated herein in its entirety for all purposes.

TECHNICAL FIELD

[0002] The present disclosure is software development, such as developing 3D virtual reality (VR) gaming worlds of gaming applications. Among other things, this disclosure describes methods and systems for building a VR gaming environments from available base VR models of real-world environments that is enhanced to augment missing surfaces of objects and to fill in gaps.

BACKGROUND OF THE DISCLOSURE

[0003] Video gaming has increasingly become more popular with the advancement of video game technology. For example, high powered graphics processors provide an unbelievably viewing and interactive experience when playing a video game. In addition, displays are being designed with higher and higher resolutions. For example, present technology includes displays having 2K resolution (e.g., 2.2 megapixels over 2048.times.1080 pixels) with an aspect ratio of approximately 19:10. Other displays having 4K UHD (Ultra high definition) resolution (e.g., 8.2 megapixels over 3840.times.2160 pixels) with an aspect ratio of 16:9 are now pushing into the market and is expected to gain traction. Increased graphics processing capabilities along with high resolution displays provide for a heretofore unbelievable viewing experience for the user, especially when playing a video game executed on a gaming engine designed to take advantage of the higher resolution displays.

[0004] Because of the use of high powered graphics, game developers are creating 3D immersive and interactive gaming applications. The 3D content found in these gaming applications include 3D gaming environments that are built by adding increasing number of objects within a 3D space. To be realistic, these objects may follow the normal laws of physics that define object interaction. The creation of the 3D gaming environment involves multiple steps. For example, a 3D gaming environment may be composed of one or more scenes, wherein each scene includes one or more objects. Each of the objects in a particular scene is painstakingly built by a developer and placed into the 3D gaming environment to interact with other objects. This process of building the 3D gaming environment is time-consuming, but necessary for generating an immersive environment that is enjoyable by the end user.

[0005] However, because building the 3D gaming environment is so time consuming and because the developer may be on a time constraint, full development of the 3D gaming environment may be sacrificed, such that the 3D gaming environment may be smaller than intended. In addition, a 3D gaming environment may continually be reused by the same developer in an effort to maximize its use given the cost to develop the gaming environment. However, this leads to staleness in the 3D gaming environment and ultimately to new gaming applications based on the same 3D gaming environment.

[0006] It would be beneficial to streamline the generation of a 3D gaming environment to give developers more time to building the game logic rather than the background upon which the game logic operates.

[0007] It is in this context that embodiments of the disclosure arise.

SUMMARY

[0008] Embodiments of the present disclosure relate to building VR gaming environments using real-world VR models as a base from third party mapping systems, and then augmenting surfaces of objects and filling in objects in the base VR models to create the VR gaming environments. Game developers are able to use the newly augmented and filled VR environments in gaming applications.

[0009] In one embodiment, a method for building a gaming environment is disclosed. The method includes accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The method includes augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications. In particular, the method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The method includes using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

[0010] In still another embodiment, a computer system is disclosed. The computer system includes a processor and memory, wherein the memory is coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for building a gaming environment. The method includes accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The method includes augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications. In particular, the method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The method includes using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

[0011] In another embodiment, a non-transitory computer-readable medium storing a computer program for building a gaming environment is disclosed. The computer-readable medium includes program instructions for accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The computer-readable medium includes program instructions for augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The computer-readable medium includes program instructions for stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The computer-readable medium includes program instructions for storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications. In particular, the computer-readable medium includes program instructions for storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The computer-readable medium includes program instructions for using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application

[0012] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The disclosure may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:

[0014] FIG. 1 illustrates a system configured for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure.

[0015] FIG. 2 illustrates functional components of a VR model builder configured for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure.

[0016] FIG. 3 illustrates an example neural network used for building models of objects, for recognition of objects within images and/or videos, and for recognition of real-world environments, in accordance with one embodiment of the present disclosure.

[0017] FIG. 4A is a flow diagram illustrating steps in a method for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure.

[0018] FIG. 4B is a flow diagram illustrating steps in a method for augmenting incomplete objects in a base VR model with surface segments and filling in missing objects in the base VR model with filler objects, in accordance with one embodiment of the present disclosure.

[0019] FIG. 5A illustrates the use of image capturing devices for building a base VR model of a real-world environment, in accordance with one embodiment of the disclosure.

[0020] FIG. 5B illustrates the identification of incomplete objects and gaps within a VR model of a real-world environment when an insufficient number of image capturing devices are used for building a VR model, in accordance with one embodiment of the present disclosure.

[0021] FIG. 6A is a data flow diagram illustrating the flow of information used for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure.

[0022] FIG. 6B illustrates the selection of an augmented VR model of a real-world environment, and the integration of the augmented VR model into a game logic of a gaming application, in accordance with one embodiment of the present disclosure.

[0023] FIG. 7 illustrates components of an example device that can be used to perform aspects of the various embodiments of the present disclosure.

DETAILED DESCRIPTION

[0024] Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claims that follow this description.

[0025] Generally speaking, the various embodiments of the present disclosure relate to building VR gaming environments using real-world VR models as a base from third party mapping systems. Incomplete objects in the base VR models are augmented with surfaces to create a complete modeled object, and gaps within the base VR model are filled with filler objects to create one or more augmented base VR models. A mapping library contains the augmented base VR models, each of which gaming environment that is based on a real-world environment. As such, the effort used to build gaming environments are much reduced by relying on third party real-world base VR models, and modifying those base VR models to generate augmented base VR models suitable for use within gaming applications (e.g., augmented base VR model interfacing with game logic of a gaming application to implement a gaming environment). In one implementation, these gaming environments are made available to game developers during the development stage of producing gaming applications. In that manner, instead of generating a corresponding VR world or gaming environment for each new gaming application from scratch, which takes significant time and effort, game developers can immediately use the augmented base VR models as VR gaming environments for corresponding gaming applications. Further, each augmented base VR model representing a corresponding real-world environment can be used as gaming environments for multiple gaming applications. That is, once an augmented base VR model is created, that model can be used by many game developers when creating new games that are set in real-world environments. As such, instead of generating one or more unique gaming environments for each gaming application, a gaming environment can be used across multiple gaming applications. In that manner, instead of taking time to generate the VR gaming environments, game developers can focus more on the game logic that interfaces with the VR gaming environments–thereby creating better quality gaming applications. In addition, by using and reusing pregenerated gaming environments, less time is needed to develop each gaming application. Further, because one augmented base VR model can be used across multiple gaming applications, a data store configured for storing augmented base VR models operates more efficiently as it can store a reduced number of augmented base VR models, and need not store different augmented base VR models for each of a plurality of gaming applications. That is, a server system configured to execute a plurality of gaming applications, having gaming environments based on augmented base VR models, is able to efficiently store the gaming environments and quickly access the gaming environments as lesser numbers of augmented base VR models are stored.

[0026] With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various drawings.

[0027] Throughout the specification, the reference to “video game” or “gaming application” is meant to represent any type of interactive application that is directed through execution of input commands For illustration purposes only, an interactive application includes applications for gaming, word processing, video processing, video game processing, etc. Further, the terms video game and gaming application are interchangeable. In addition, the reference to “digital content” is meant to represent 2D or 3D digital content to include, in part, video games, gaming applications, and videos.

[0028] FIG. 1 illustrates a system 100 configured for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure. The server 160 includes an augmented base VR model builder 200 and a deep learning engine 190. The server 160 is configured to build one or more VR gaming environments using real-world mappings or VR models from the third party mapping systems. In particular, the augmented base VR model builder 200 is able to access and/or receive 3D real-world data from the third party mapping systems. Each third party mapping system has their own unique way of building or mapping the real-world environment as represented by base VR models (e.g., indoor and outdoor environments). This data (raw and/or base VR models) is used to build augmented base VR models of the real-world environment, wherein the base VR models are rebuilt by augmenting incomplete surfaces of objects and filling in missing objects. The augmented base VR models provide 3D virtual views of the real-world environment from any location (e.g., not just locations associated with image capture points). In one embodiment, the real-world environment includes a live event, which is represented by the corresponding augmented base VR model. The augmented base VR models of one or more real-world environments can be stored in a library and later accessed by game developers. In that manner, the developers can save time from rebuilding those base VR models, or generating VR models from scratch, by using the pre-generated augmented base VR models. As such, developers can focus more time and energy on developing gaming logic that maximizes the gaming experience of an end user. That is, the developers are freed from building common everyday objects found repeatedly in real-world environments (e.g., buildings, trees, streets, parks, etc.) performed when building the VR gaming environment.

[0029] In particular, the deep learning engine 190 is configured to supplement the abilities of the augmented base VR model builder 200. For instance, the deep learning engine 190 may be configured to build 3D models, to include 3D surface models, of common objects that are found in one or more real-world environments. As illustrative examples, an object may include buildings, furniture, generic persons, animals, trees, bushes, foliage, vehicles, buses, etc. The deep learning engine 190 may be used for recognition of objects based on images, videos, and/or VR models of real-world environments. The 3D models 182 of objects may be stored in data storage 180. The deep learning engine 190 may also be used for recognition of a real-world environment based on images, videos, and/or VR models of the real-world environments. Further, the deep learning engine 190 is configured to recognize objects that are incomplete (e.g., incomplete surface models) from base VR models, and is configured to recognize gaps (e.g., missing objects) within base VR models representing corresponding real-world environments, wherein the gaps are identified by the deep learning engine 190. In that manner, the augmented base VR model builder 200 is able to augment incomplete surface models with additional surfaces based on previously defined object models, and also fill in gaps in the base VR models with filler objects to create augmented VR models. Management of the augmented VR model building process may be performed via a client device 100 providing an interface to the building process.

[0030] The augmented base VR models are stored in the library 181 of data storage 180. In addition, object models (e.g., surface models) 182 are also stored in data storage 180. The augmented base VR models can be later accessed and used as gaming environments that are integrated with gaming logic during development of corresponding gaming applications. In that manner, the game developers need only access a pre-generated gaming environment (e.g., 3D VR models) representing a real world environment for use in gaming applications instead of generating those gaming environment. The game developer may access the augmented VR models library 181 via a client device 170 through the internet 150. In addition, the augmented base VR models can be later accessed by a server or computing resource configured to execute one or more gaming applications in association with game plays of multiple users, wherein during game play of a user game logic of a gaming application creates a corresponding gaming environment as implemented through a corresponding augmented base VR model.

[0031] Client devices 100 (of a VR modeler), 170A (e.g., of a user playing a gaming application), and 170B (e.g., of a game developer) may be any general or special purpose computer known in the art, including but not limited to, a gaming console, personal computer, laptop, tablet computer, mobile device, cellular phone, tablet, thin client, set-top box, media streaming device, etc. In one embodiment, the client devices 100 and 170A-170B can be configured to execute a video game, and output the video and audio from the video game for rendering by an HMD (not shown). The client devices 100 and 170A-170B can also be configured as thin clients in communication over a network with the server 160. For example, the HMD may be worn when editing and/or generating an augmented VR model, in one implementation. In another implementation, the HMD may be worn by a game developer when reviewing a VR gaming environment during game development. The client devices 100 and 170A-170B are not restricted to executing a video game but may also be configured to execute an interactive application, which outputs VR content for rendering by the HMD. In general, an HMD is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a video game from an interactive video game or other content from interactive application, to the corresponding user. The HMD provides a very immersive experience to the user by virtue of its provision of display mechanisms in close proximity to the user’s eyes. Thus, the HMD can provide display regions to each of the user’s eyes which occupy large portions or even the entirety of the field of view of the user.

[0032] System 100 includes a third party mapping system 120 that is configured for providing base VR models. The base VR models 128 are built using image data 126 and/or video data 127, both of which are stored in data storage 125. For example, a base VR model of Paris, France around the vicinity of the Eiffel Tower may be built using captured image data 126 (e.g., from image capture devices of professionals and/or tourists, etc.) and/or captured video data 127. A general discussion on the generation of base VR models 128 is provided below in relation to FIG. 5A.

[0033] For example, the third party mapping system may provide services to users to include a 3D view of a street that acts as a base VR model. Various search companies provide these views of city streets throughout the world to the general public. Basically, the user is able to hop between capture points along a street, and at each capture point rotate the view (e.g., 360 degrees) at that specific capture point. The capture points are typically located at or near the center of a street. Views from points other than capture points are not provided by the third party mapping systems 120. On the other hand, embodiments of the present disclosure are able build upon the base VR models 128 to generate augmented base VR models that are stored in library 181 using the server 160, as will be further described below. In some embodiments, the server 160 is configured to build on the base VR models 128, and in other embodiments, the server 160 is configured to access the raw image data 126 and/or video data 127 in order to build the base VR models 128, which later can be further augmented and filled to create complete VR models of real-world environments, as will be desired further below.

[0034] In addition, system 100 includes the third party image collector 130 that is configured for providing additional image data 136 and/or video data 127, both of which are stored in data storage 135. This additional image and video data may be used for augmenting objects in the base VR models that are either incomplete or missing. For example, the base VR model of Paris, France may be incomplete as the image data 126 and the video data 127 used to create that base VR model may be captured from an insufficient number of capture points. That is, the image data 126 and the video data 127 are also insufficient to build a complete VR model of Paris, France. In that manner, additional and supplemental image data 136 and/or video data 137 may be accessed by the server 160 in order to augment incomplete surface models of objects in the base VR model. For example, incomplete surface models may be completely replaced with corresponding and generic completed surface models, or missing surfaces may be replaced with surface portions from the completed surface models (e.g., built by the deep learning engine 190).

[0035] FIG. 2 illustrates functional components of a server 160 that are configured for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure. In particular, system 160 includes an augmented base VR model builder 200 configured for building augmented base VR models and a deep learning engine 190 supporting builder 200, wherein control and access to system 200 may be through client device 100 either directly with server 160 or through a network, such as internet 150. The components of the system 160 may be implemented in software, hardware, or a combination of software and hardware.

[0036] In particular, augmented base VR model builder 200 includes object/environment recognition detection module 210 that is configured for recognizing objects and/or a local environment based on images and/or video taken within the local environment. The detection module 210 is configured within deep learning engine 190, in one embodiment. In one implementation, the detection module 210 is configured for detecting one or more objects. For example, one or more images and/or a video can be input into the deep learning engine 190 for purposes of object recognition. In particular, the deep learning engine 190 is able to identify objects within an image (including an image of a video) by recognizing that patterns of objects within the image are similar to patterns of objects that have been previously learned or modeled by the deep learning engine 190. For example, the deep learning engine 190 is able to model a house cat, and even more specifically to model a breed or type of house cat (e.g., calico), by providing as input a plurality of images of cats, and having the deep learning engine 190 recognize patterns that are unique to the house cat. A cat recognition (e.g., object recognition for a cat) algorithm can be created that when performed by the deep learning engine can recognize objects within an image having the same patterns for cats. That is, the deep learning engine 190 can recognize when an image includes a cat object. In addition, other objects within the image are also recognized by the deep learning engine 190 using other corresponding object identifier algorithms

[0037] In another implementation, the detection module 210 is configured for detecting the local environment. For example, one or more images and/or a video can be input into the deep learning engine 190 for purposes of environment recognition. That is, the deep learning engine 190 is able to identify the local environment which the image or video is capturing. An environment recognition algorithm (each algorithm specific to a local environment) can be created that when performed by the deep learning engine 190 can recognize the local environment based on captured images and/or video. As such, when the image and/or video is provided as input to the deep learning engine, the corresponding environment recognition algorithm will determine that the input is associated with its local environment.

[0038] In addition, the augmented base VR model builder 200 includes a surface model completeness evaluator 215 that is configured to determine when objects in a base VR model of a real-world environment (e.g., as provided through a third party mapping system) is incomplete. An object model (e.g., surface model) may be incomplete when not enough images are used to create the model. For example, when only one image is used to model the object, the opposing side may not be viewable, and will not be included within the object model, and as such is incomplete. When objects and/or the models of the objects are identified as being incomplete, additional actions may be performed to complete those models, as will be described below.

[0039] Augmented base VR model builder 200 includes an object surface augmentation module 220 that is configured to augment incomplete surface models with information in order to complete those surface models. For example, when a surface model of a corresponding object is identified as being incomplete, portions of the surface that are missing in the model may be accessed and/or generated to be included in the surface model, thereby making the model complete. For example, a generic surface model for that object may have been previously generated, and portions of the surface that are missing from the base VR model can be used to augment the incomplete surface model. Also, the complete generic surface model may be used as a substitute for the incomplete surface model. In addition, if no generic model exists, then the object surface augmentation module 220 is configured to access and/or grab additional images or video of the object (e.g., taken of the object within the corresponding local environment, or taken of a corresponding generic object), and based on the additional information generate portions (or a complete model) of the surface that are missing that are then used to augment the incomplete surface model or build a complete surface model that can be used in substitution.

[0040] Further, augmented base VR model builder 200 includes base VR model gap detector 225 that is configured to analyze a base VR model of a corresponding real-world environment, as provided by a third party source, and detect gaps within the model. A gap in the base VR model is exemplified by one or missing objects within a gap region. For example, when the local environment is recognized by the deep learning engine 190 as being the Eiffel Tower in Paris, France, based on images and/or other information sources it is known that a particular restaurant is within the immediate vicinity. A gap may be identified as not having that particular restaurant in the base VR model. Other missing objects may be identified as missing from the same gap and/or other gaps of a corresponding base VR model.

[0041] The augmented base VR model builder 200 includes a surface modeler for filler objects 230 that is configured to build filler objects to be placed into gaps identified in a corresponding base VR model. For example, the filler objects may have no association with actual objects found in the same space of the corresponding real-world. That is, the filler objects may be any object that may be placed into the gap or gap region of the base VR model. For example, in a base VR model of an office, filler objects may include a credenza, picture on a wall, plant, or plant on a table, etc. These filler object may be pre-generated, or may be generated when augmenting the base VR model. In one implementation, additional images and/or video of a corresponding filler object is accessed (e.g., from a third party image collector 130 of FIG. 1), and used to generate a surface model for the filler object. In another implementation, the filler objects may be associated with actual objects found in the same space of the corresponding real-world. As such, a filler object may be representative of a corresponding real-world object, and the filler object may be used within the base VR model within the corresponding gap region.

[0042] Further, the augmented base VR model builder 200 includes stitching/blending module 235 that is configured to create the augmented VR models of corresponding real-world environments. In particular, additional images and/or videos, as well as augmented surface models of incomplete objects, replacement surface models of incomplete objects, and filler objects can be stitched together to create the augmented VR model that is formed from the base VR model provided by the third party mapping system.

[0043] The augmented base VR model builder 200 includes a software development kit (SDK) builder 240 that is configured to provide access to the generated augmented VR models of corresponding real-world environments. For example, the SDK may include tools that give access to server 200 that manages the library of augmented VR models. In particular, the SDK may give access by game developers to the library of augmented VR models 181, wherein a selected augmented VR model can be layered with game logic to develop a gaming application. The SDK may be specific to a particular gaming platform (e.g., Sony PlayStation.RTM., in one implementation. In another case, the SDK may only provide access to the library of augmented VR models 181, from which a selected VR model may be integrated with game logic playable on one or more gaming platforms.

[0044] The augmented base VR model builder 200 includes object modeler 245 configured for building surface models for one or more generic objects. That is, some objects are found in one or more real-world environments, such as one story buildings, a family home, a two story building, a gas station, a particular animal, a generic person standing, a generic person walking, a vehicle, a motorcycle, a desk, a tree, a semi-trailer, etc. Generic surface models may be generated for those objects. For example, images and/or video for a corresponding object may be input into the object modeler 245, as implemented within the deep learning engine 190, to create a generic model for that object.

[0045] The augmented base VR model builder 200 may also work in conjunction with a deep learning engine 190 that is configured to perform repetitive and compute intensive operations. Specifically, the deep learning engine 190 may include and perform the functions of the previously introduced object modeler 245 and the object/environment recognition/detection module 210. The deep learning engine is described more fully in relation to FIG. 3.

[0046] FIG. 3 illustrates an example neural network used to build surface models for objects, to perform recognition of objects within images and/or videos, and to perform the recognition of local environments that are captured within images and/or videos, in accordance with one embodiment of the present disclosure. Specifically, the deep learning or machine learning engine 190 in the server 160 is configured to receive as input information (e.g., images, videos, base VR models, etc.) related to objects and/or local environments. The deep learning engine 190 utilizes artificial intelligence, including deep learning algorithms, reinforcement learning, or other artificial intelligence-based algorithms to build surface models for objects, as previously described, to recognize objects within images and/or videos, and to recognize local environments within images and/or videos. That is, during learning and/or modeling phases, input data is used by the deep learning engine 190 to create surface models for objects and to create parallel models (e.g., learned algorithms) that can be used to identify/recognize particular predefined objects. For example, the input data may include multiple images of a cat, which when fed into a deep learning engine 190 is configured to create a surface model for a generic cat, and also to create an object recognition algorithm (model) that is used to identify cats within an image and/or video, and within a VR model of a real-world environment.

[0047] A neural network 190 represents an example of an automated analysis tool for analyzing data sets to determine the responses, actions, behavior, wants and/or needs of a corresponding user. Different types of neural networks 190 are possible. In an example, the neural network 190 supports deep learning. Accordingly, a deep neural network, a convolutional deep neural network, and/or a recurrent neural network using supervised or unsupervised training can be implemented. In another example, the neural network 190 includes a deep learning network that supports reinforcement learning. For instance, the neural network 190 is set up as a Markov decision process (MDP) that supports a reinforcement learning algorithm.

[0048] Generally, the neural network 190 represents a network of interconnected nodes, such as an artificial neural network. Each node learns some information from data. Knowledge can be exchanged between the nodes through the interconnections. Input to the neural network 190 activates a set of nodes. In turn, this set of nodes activates other nodes, thereby propagating knowledge about the input. This activation process is repeated across other nodes until an output is provided.

[0049] As illustrated, the neural network 190 includes a hierarchy of nodes. At the lowest hierarchy level, an input layer 191 exists. The input layer 191 includes a set of input nodes. For example, each of these input nodes is mapped to local data 115 collected actively through actuators or passively by sensors during monitoring and/or querying of the user and the environment associated with the user by the autonomous personal companion 100.

[0050] At the highest hierarchical level, an output layer 193 exists. The output layer 193 includes a set of output nodes. An output node represents a decision (e.g., prediction) that relates to one or more components of a local AI model 120, for example. As previously described, the output nodes may identify the predicted or expected responses, actions, behaviors, wants and/or needs of the user for a given set of inputs, wherein the input may define various scenarios (e.g., direct requests, time of day, various patterns of behavior, etc.). These results can be compared to predetermined and true results obtained from previous interactions and monitoring of the user and/or environment in order to refine and/or modify the parameters used by the deep learning engine 190 to iteratively determine the appropriate predicted or expected responses, actions, behaviors, wants and/or needs of the user for a given set of inputs. That is, the nodes in the neural network 190 learn the parameters of the AI model 120 that can be used to make such decisions when refining the parameters.

[0051] In particular, a hidden layer 192 exists between the input layer 191 and the output layer 193. The hidden layer 192 includes “N” number of hidden layers, where “N” is an integer greater than or equal to one. In turn, each of the hidden layers also includes a set of hidden nodes. The input nodes are interconnected to the hidden nodes. Likewise, the hidden nodes are interconnected to the output nodes, such that the input nodes are not directly interconnected to the output nodes. If multiple hidden layers exist, the input nodes are interconnected to the hidden nodes of the lowest hidden layer. In turn, these hidden nodes are interconnected to the hidden nodes of the next hidden layer, and so on and so forth. The hidden nodes of the next highest hidden layer are interconnected to the output nodes. An interconnection connects two nodes. The interconnection has a numerical weight that can be learned, rendering the neural network 190 adaptive to inputs and capable of learning.

[0052] Generally, the hidden layer 192 allows knowledge about the input nodes to be shared among all the tasks corresponding to the output nodes. To do so, a transformation f is applied to the input nodes through the hidden layer 192, in one implementation. In an example, the transformation f is non-linear. Different non-linear transformations f are available including, for instance, a linear rectifier function f(x)=max(0,x).

[0053] The neural network 190 also uses a cost function c to find an optimal solution. The cost function measures the deviation between the prediction that is output by the neural network 190 defined as f(x), for a given input x and the ground truth or target value y (e.g., the expected result). The optimal solution represents a situation where no solution has a cost lower than the cost of the optimal solution. An example of a cost function is the mean squared error between the prediction and the ground truth, for data where such ground truth labels are available. During the learning process, the neural network 190 can use back-propagation algorithms to employ different optimization methods to learn model parameters (e.g., the weights for the interconnections between nodes in the hidden layers 192) that minimize the cost function. An example of such an optimization method is stochastic gradient descent.

[0054] In an example, the training dataset for the neural network 190 can be from a same data domain. For instance, the neural network 190 is trained for learning the patterns and/or characteristics of similar objects, for recognizing a local environment, for building a surface model for a generic object, etc. based on a given set of inputs or input data. In this illustration, the data domain includes image and video data of objects or of a local environment. In another example, the training dataset is from different data domains to include input data other than baseline. As such, the neural network 190 may recognize objects within an image and/or video data, or may recognize a local environment as captured by image and/or video data, or may be configured to generate a surface model for a corresponding object based on input image and video data.

[0055] With the detailed description of the various modules of system 200 including the augmented base VR model builder 200 and deep learning engine 190, flow diagram 400A of FIG. 4A discloses a method for building VR gaming environments using real-world VR models as a base from third party mapping systems, in accordance with one embodiment of the present disclosure. Flow diagram 400A is implemented within system 200, and more particularly within the augmented base VR model builder 200 and/or deep learning engine 190, when building the augmented VR models of corresponding real-world environments, as previously described.

[0056] Though the method of flow diagram 400A is described within the context of building augmented VR models of real-world environments for purposes of providing gaming environments when developing gaming applications, other embodiments are well suited to building augmented base VR models of any type of environment as virtual environments. That is, augmentation of any type of base VR model may be performed to generate one or more augmented VR models, wherein the base VR model may correspond to real, imaginary, virtual, and/or other digitally created worlds.

[0057] At 410, the method includes accessing a base VR model of a real-world environment from a third party mapping data storage. The real-world environment includes a plurality of real-world objects, such that the real-world objects can be found in the environment. Embodiments of the present invention generate VR worlds corresponding to real-world environments, wherein the VR worlds can be used as gaming environments for gaming applications. Rather than beginning from scratch, or from the bottom up, VR worlds from third party mapping entities are used as base VR models. The base VR models are typically very limited in scope and generally incomplete. That is, the base VR models do not provide full coverage of the real-world environment, such that views into the real-world environment are only provided from a predefined set of points or locations within the environment. On the other hand, embodiments of the present disclosure are configured to modify the base VR models to create complete models of the real-world environment that provide full coverage (e.g., the space defined by a corresponding augmented base VR model is viewable from any location within the space). Of course, other embodiments of the present invention are also able to generate the VR worlds, such as those used as gaming environments, in their entireties (e.g., building from the bottom up).

[0058] As previously described, the plurality of real-world objects in the real-world environment may be determined through a deep learning process. That is, images and/or videos of the real-world environment are provided as input into the deep learning or machine learning engine 190 in the server 160. The deep learning engine 190 utilizes artificial intelligence, including deep learning algorithms, reinforcement learning, or other artificial intelligence-based algorithms to recognize objects within the images and/or videos, and to recognize local environments within images and/or videos.

[0059] At 415, the method includes augmenting a first object in the base VR model. The first object corresponds to a first real-world object in the real-world environment. The first object may be a representation (e.g., a surface model) of the first real-world object in the base VR model.

[0060] In one embodiment, the first object may be incomplete in the base VR model, such that the representation is incomplete (e.g., one or more surface segments missing in the surface model). The first object is augmented to generate a more complete representation of the first real-world object within the base VR model. In one embodiment, surface segments may be used to augment an incomplete surface model corresponding to the first real-world object, wherein the surface model is missing one or more surface segments. In another embodiment, a completed surface model of the first real-world object (e.g., a generic or custom surface model) may be substituted for the incomplete surface model and used to represent the first real-world object when augmenting the base VR model. The completed surface model may be pre-generated in one embodiment, and in another embodiment is generated from a plurality of images and/or videos of the first real-world object.

[0061] In other embodiments, the first object may be missing from the base VR model. That is, the augmentation process may be modified to generate a representation of the first real-world object within the base VR model. In one embodiment, a pre-generated completed surface model may be used to represent the first real-world object when augmenting the base VR model. In another embodiment, a completed surface model is generated from a plurality of images and/or videos of the first real-world object and used to represent the first real-world object when augmenting the base VR model.

[0062] The method optionally determines a gap region in the base VR model. The gap region is devoid of at least one object. In one example, the base VR model is missing data for represented regions in the real-world environment. In another example, a representation of an identified second real-world object in the real-world environment may be missing from the base VR model.

[0063] The method includes accessing a filler object to fill in the gap region. In one implementation, the filler object may be any generic object used to fill in the gap region, and may or may not be present within the real-world environment. For example, the filler object may be pre-generated and stored for access, or the filler object may be generated based on images and/or video of the corresponding object and accessed after being generated. For instance, at 420, the method includes generating a filler object, and/or accessing a pre-generated filler object, to fill in the gap region. In another implementation, the filler object may correspond to the second real-world object, previously introduced, that is missing from the base VR model. For instance, at 423, the method includes generating a filler object, and/or accessing a pre-generated filler object, that corresponds to the second real-world object in the real-world environment.

[0064] At 425, the method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model corresponding to the real-world environment. Further, the method optionally includes stitching in the one or more filler objects into the base VR model to generate the augmented base VR model. Stitching algorithms and methods are well known in the art, and are used to integrate objects, images, videos together to create a seamless 3D model of the environment. For example, stitching includes substituting the augmented surface models for the incomplete surface models when generating the augmented VR model. In addition, the one or more filler surface models are placed into the base VR model of the real-world environment when generating the augmented VR model. As such, within the augmented base VR model, objects are viewable from any location. That is, the augmented VR model of the real-world environment is complete, such that views can be generated from any location in the real-world environment. In that manner, objects can be viewable from any perspective as taken from any location in the environment of the augmented VR model.

[0065] At 430, the method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. Each of the augmented base VR models is configured for use as corresponding virtual environments of corresponding gaming applications. In one embodiment, an augmented base VR model can be used as a gaming environment of a gaming application. At 431, the method includes using one or more the plurality of augmented base VR models to define a virtual environment of a gaming application.

[0066] For example, a client device of a game developer may access the one or more augmented base VR models during game development to facilitate the interfacing of the game logic of the gaming application with the augmented base VR model. In particular, the method includes making available a corresponding augmented base VR model from the library data storage through a software development kit (SDK) to a remote device for use in developing game logic of a gaming application configured to interface with the augmented base VR model, the gaming application having a gaming environment based on the real-world environment. In that manner, the game logic is interfaced with the augmented base VR model to provide a gaming environment for the gaming application.

[0067] In another example, a client device in association with game play of a first user playing the gaming application may access the one or more augmented base VR models during execution of the game logic. In that manner, the gaming environment of the game play of the user is implemented through the one or more augmented base VR models of corresponding real world environments. In particular, the method includes receiving a request for the augmented base VR model from a remote device. The augmented base VR model is delivered to the remote device for interfacing with game logic of the gaming application, wherein the game logic is executing on the remote device in association with a game play of the first user. The remote device may be a server configured to execute a plurality of game logics for a plurality of users, in one embodiment. In another embodiment, the remote device may be a console of the first user.

[0068] In that manner, the virtual or gaming environment of a corresponding real-world environment can be generated without having to create all of the graphics from scratch, in embodiments. This is accomplished by using third party mapping data and augmenting it, such that part of the augmented virtual environment is premade, and reusable within the augmented virtual environment. The augmented virtual environment is generated by adding and/or importing other graphics data, as well as portions of other VR worlds. The augmented virtual environment can be used as the background or environment of a gaming application. That is, the augmented virtual environment becomes part of the functional game logic.

……
……
……