Sony Patent | Establishing And Using Virtual Assets On Tangible Objects In Augmented Reality (Ar) And Virtual Reality (Vr)
Publication Number: 20200301553
Publication Date: 20200924
Applicants: Sony
Abstract
A method includes identifying a tangible object, establishing a virtual overlay on the tangible object in a manner that is visible to a user, detecting the user’s interaction with the virtual overlay that is established on the tangible object, and using the user’s interaction with the virtual overlay as a basis for input to a processor based system. A system includes a display and a processor based apparatus in communication with the display. A storage medium storing one or more computer programs is also provided.
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62/822,712, filed on Mar. 22, 2019, entitled “ESTABLISHING AND USING VIRTUAL ASSETS ON TANGIBLE OBJECTS IN AUGMENTED REALITY (AR) AND VIRTUAL REALITY (VR)”, the entire contents and disclosure of which is hereby fully incorporated by reference herein in its entirety.
BACKGROUND OF THE INVENTION
1.* Field of the Invention*
[0002] Embodiments of the present invention relate generally to computer enhancement and simulation technology, and more specifically to augmented reality (AR) and virtual reality (VR) technology.
2.* Discussion of the Related Art*
[0003] Augmented reality (AR) is an interactive experience of a real-world environment in which the view of reality is modified or augmented by computer-generated information. For example, a live view of the real world as seen by a camera on a smartphone can have computer-generated information, features, and/or elements added to it.
[0004] Virtual reality (VR) is a complete immersion interactive computer-generated experience. For example, a VR headset can be used to immerse a user in a fully artificial simulated environment.
[0005] Mixed reality (MR) is a mixture of both AR and VR and can be used to create an environment that allows a user to interact with virtual objects in the real world.
SUMMARY OF THE INVENTION
[0006] One embodiment provides a method, comprising: identifying a tangible object; establishing a virtual overlay on the tangible object in a manner that is visible to a user; detecting the user’s interaction with the virtual overlay that is established on the tangible object; and using the user’s interaction with the virtual overlay as a basis for input to a processor based system.
[0007] Another embodiment provides a system, comprising: a display; and a processor based apparatus in communication with the display; wherein the processor based apparatus is configured to execute steps comprising: identifying a tangible object; establishing a virtual overlay on the tangible object in a manner that is visible on the display; detecting a user’s interaction with the virtual overlay that is established on the tangible object; and using the user’s interaction with the virtual overlay as a basis for input to the processor based apparatus.
[0008] Another embodiment provides a non-transitory computer readable storage medium storing one or more computer programs configured to cause a processor based system to execute steps comprising: identifying a tangible object; establishing a virtual overlay on the tangible object in a manner that is visible to a user; detecting the user’s interaction with the virtual overlay that is established on the tangible object; and using the user’s interaction with the virtual overlay as a basis for input to the processor based system.
[0009] A better understanding of the features and advantages of various embodiments of the present invention will be obtained by reference to the following detailed description and accompanying drawings which set forth an illustrative embodiment in which principles of embodiments of the invention are utilized.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other aspects, features and advantages of embodiments of the present invention will be more apparent from the following more particular description thereof, presented in conjunction with the following drawings wherein:
[0011] FIG. 1 is a diagram illustrating a device viewing a real-world scene in accordance with some embodiments of the present invention;
[0012] FIG. 2 is a flow diagram illustrating a method in accordance with some embodiments of the present invention;
[0013] FIG. 3 is a diagram illustrating a device viewing a real-world scene in accordance with some embodiments of the present invention;
[0014] FIG. 4 is a diagram illustrating a device viewing a real-world scene in accordance with some embodiments of the present invention;
[0015] FIG. 5 is a diagram illustrating a device viewing a real-world scene in accordance with some embodiments of the present invention;
[0016] FIG. 6 is a flow diagram illustrating a method in accordance with some embodiments of the present invention;* and*
[0017] FIG. 7 is a block diagram illustrating a processor based apparatus/system that may be used to run, implement, and/or execute any of the methods, schemes, and techniques shown and described herein in accordance with the embodiments of the present invention.
DETAILED DESCRIPTION
[0018] Some augmented reality (AR) systems include the ability to generate a virtual user interface (UI) that is included in the real world view. Similarly, some virtual reality (VR) systems include the ability to generate a virtual UI that is included in the virtual world view. However, such UIs suffer from the disadvantage that there is no physical experience because the user is just pushing buttons in the air. For example, if the user has a telephone keypad projected in front of his or her face, pushing buttons in the air is not very satisfying.
[0019] Some of the embodiments of the present invention provide methods, systems, and techniques that can be used to improve UIs in AR, VR, and mixed reality (MR) by providing the user with a tangible and physical object that the user can touch and feel when interacting with the UI. Actual tangible and physical objects in the real world are used so the user can hold and feel the actual geometry of the object in his or her hands. It is believed that the ability to touch, hold, and feel a tangible, physical object provides a much more satisfying AR, VR, and MR experience.
[0020] Referring to FIG. 1, there is illustrated a system 100 which is operating in accordance with an embodiment of the present invention. The system 100 includes a smartphone 102 having a display 104 and a camera (not shown) on the backside.
[0021] In some embodiments, for example, a user may be operating the smartphone 102 in an AR mode. The user may decide that he or she needs a keypad, such as for entering a telephone number or for entering a password or passcode for gaining access to some type of secure area. So the user grabs any tangible object that may be nearby, such as for example the tangible object 108, which in this example is a book. The book is something that the user can actually hold. The camera on the back of the smartphone 102 captures the view of the real world which includes the tangible object 108. As such, the tangible object 108 is displayed on the display 104 of the smartphone 102.
[0022] In some embodiments, the user then provides a verbal command to the system 100 by speaking the word “keypad”. The word “keypad” may comprise a keyword that has a UI associated with it, which in this case is a keypad. In some embodiments, in response to the verbal command, a virtual overlay 110 is then established or projected onto the surface of the tangible object 108 in a manner that is visible to the user by being visible on the display 104. In this example the virtual overlay 110 comprises a UI, and in particular a keypad, which is an asset that can be used for many different types of applications. The establishment of the virtual overlay 110 on the tangible object 108 creates a virtual keypad, which comprises a type of virtual asset. By way of example, the keypad may be used as part of a telephone for the entry of a telephone number, or for entry of a password or passcode as stated above. But it should be well understood that the virtual overlay 110 may comprise any type of UI, feature, or other virtual asset.
[0023] With the keypad virtual overlay 110 being established on the surface of the tangible object 108, the user is then able to interact with something tangible such that the user can actually hold and feel a physical object and is not just pushing buttons in the air. For example, as shown the tip of the user’s finger 112 is touching the tangible object 108 in the real world. As shown on the display 104, the user’s finger 112 is interacting with the virtual overlay 110 that is established on the tangible object 108. In this example, by touching the tangible object 108 and viewing the display 104, the user is able to enter a phone number into the keypad that comprises the virtual overlay 110.
[0024] In some embodiments, the processor based system included in the smartphone 100 detects the user’s interaction with the virtual overlay 110 that is established on the tangible object 108. In some embodiments, the detection may be performed by tracking the user’s finger 112. In some embodiments, the tracking may be performed by using one or more cameras, such as the smartphone camera, or any other image capture or depth sensing devices, to detect an intersection of the user’s finger 112 with a geometry of the virtual overlay 110. In some embodiments, the tracking may be performed by using acoustic tracking or sending to listen to the user’s finger 112 tap on the tangible object 108.
[0025] In some embodiments the user’s interaction with the virtual overlay 110 is used as a basis for input to a processor based system, such as the processor based system included in the smartphone 100. For example, the user’s interaction with the virtual overlay 110 may be used as a basis for input to an application, program, process, simulation, or the like. In the illustrated example, the user’s interaction with the virtual overlay 110 is used to enter a telephone number that is used as input to, for example, the telephone included in the smartphone 102.
[0026] Referring to FIG. 2, there is illustrated a method 200 in accordance with an embodiment of the present invention. In some embodiments the method 200 may be performed by the system 100 (FIG. 1). In some embodiments the method 200 may be performed by any other type of AR, VR, or MR system, such as a system employing a headset, a glasses-type user device, head-mounted display (HMD), or the like.
[0027] In step 202 a tangible object is identified. The tangible object may comprise any tangible, physical object in the real world, such as a book as described above, or any other object, such as for example a cup, can, food plate or dish, pad of paper, piece of wood or plastic, rock, brick, table top, floor, wall, cardboard or metal box, etc. In some embodiments, the tangible object may comprise any object on which the user would like a virtual asset, such as a virtual UI or other feature, to appear. In some embodiments, simple props, such as cardboard or plastic props, may be used to provide tangible objects having various shapes and sizes that might be appropriate for virtual UIs or other virtual assets that might be wanted or needed in an AR, VR, or MR experience.
[0028] In some embodiments, the tangible object may initially be identified by a user simply grabbing it or picking it up, such as was described above. The object may then be further identified by a processor based system by way of one or more cameras or other image capture devices.
[0029] More specifically, in some embodiments part of the step of identifying a tangible object is to also identify or determine the geometry, form factor, size, location, and/or position of the tangible object. One way to make such determinations is for a system to use cameras or other image capture devices or sensors to determine the geometry and size of an object. For example, in some embodiments, depth sensing devices such as depth sensing cameras may be used. Depth cameras, stereo cameras, depth sensors, and the like, may be used to determine the absolute geometry of the object, its size, and how far away the object is from the user. In some embodiments, one or more such cameras and/or sensors may be used and may be located on a user’s headset, glasses-type user device, HMD, or elsewhere in an AR, VR, or MR environment or room.
[0030] In some embodiments, another way to determine the size, scale, and geometry of a tangible object is to place it in the view of a camera together with an object of known size, scale, and geometry. That is, take an object of known size and scale along with the unknown object, and hold the two objects up together in front of a camera. The system compares the object of known size and scale to the unknown object to determine the size and scale of the unknown object. By way of example, in some embodiments a system learns and remembers a user’s hand size and then makes judgements to determine the size of other tangible objects by comparing them to the user’s hand.
[0031] In some embodiments, another way that is used to identify or determine the geometry and size of the tangible object is for a system to use cameras or other image capture devices to recognize and/or identify known objects that have known geometries and sizes. Once the geometry and size of an object is known, it will be remembered and known within the network. For example, a standard twelve ounce soda can has a known size and geometry and is easily recognized. Once it is known and learned in the network the system can retrieve its dimensions and geometry as part of determining its location and depth relative to the user.
[0032] In some embodiments, automatic identification of tangible objects may be used. For example, a system can use cameras or other image capture devices to scan a space, such as an entire room, to identify many different tangible objects. Some of the objects will already be of known size and scale, such as for example a soda can as mentioned above, as well as popular consumer products and devices, such as smartphones, tablet computers, etc. The system can first recognize or identify those objects that have a known size and scale. The known objects are then compared to the unknown objects to determine the size, scale, and geometry of the unknown objects. Thus, in some embodiments, all the objects in a room are sized by scanning everything in the room with a camera and then comparing unknown objects to known objects. In this way a system can use auto identification to pre-scale many or even all of the objects in a room.
[0033] Referring again to FIG. 2, in step 204 a virtual overlay is established on the tangible object in a manner that is visible to a user. Establishing a virtual overlay on a tangible object creates a virtual asset. In some embodiments, the virtual overlay may comprise a user interface (UI), which may comprise any type of UI, such as for example a calculator, keyboard, keypad as was described above, etc. The virtual overlay is visible to the user by being visible on the display of a device, such as a smartphone, tablet computer, headset, glasses-type user device, HMD, or the like. By establishing or projecting an interface onto the surface of something the user may have picked up or that is nearby (e.g. a wall), the user is able to interact with a UI that appears to the user to be attached to an object that has a certain form factor and that the user can physically feel. That is, the user can feel the tangible object and the form factor as the user interacts with the virtual UI on the object or other tangible surface.
[0034] In some embodiments, the virtual overlay is established on the tangible object in response to a command from the user. By way of example, the command from the user may comprises an audio command, such as a verbal or spoken command. In some embodiments, the command from the user also specifies or determines a configuration or purpose of the virtual overlay. For example, the command from the user may specify that the virtual overlay comprise a specific type of UI, such as any of the UIs mentioned above.
[0035] As an example, a user may be using an AR, VR, or MR system and be experiencing an AR, VR, or MR environment. While immersed in that environment the user may pick up something from the real world, such as any type of tangible, physical object, and the user may want that object to be a telephone in the virtual environment. The user says the word “telephone”, and in response to that verbal command the system establishes, by projecting or mapping, a virtual telephone interface onto the surface of the object. The object can be anything, such as a brick, piece of wood, or table top surface. The virtual interface is established onto whatever form factor the object comprises.
[0036] In some embodiments, the establishing a virtual overlay on the tangible object comprises scaling the virtual overlay to fit the tangible object. Specifically, as described above a system first detects, identifies, and/or recognizes the tangible object that the user has picked up. As also described above, the system then figures out or determines the size, scale, form factor, and/or geometry of the object. For example, if the object is a box the system determines where the edges are located such as by using depth cameras.
[0037] Depending on the size and geometry of the tangible object, the system may have to scale the virtual overlay in order for it to fit the object correctly. For example, if the virtual overlay comprises a type of UI that is larger than the surface of the object, the system may scale the virtual UI to make it smaller in order to fit the object correctly. Or, if the virtual overlay comprises a type of UI that is generally square in shape, and the surface of the object is generally rectangular in shape, the system may scale, such as by reconfiguring, the virtual UI to make it generally rectangular in shape in order to fit the object correctly. And as another example, the system may scale the virtual UI, asset, or other feature to the geometry and size of the tangible object so that it looks and feels correct to the user and there is no awkwardness or disconnect perceived by the user.
[0038] Referring to FIG. 3, there is illustrated an example of scaling a virtual overlay to fit a tangible object in accordance with an embodiment of the present invention. Specifically, there is illustrated a system 300 which is operating in accordance with an embodiment of the present invention. The system 300 includes a smartphone 302 having a display 304 and a camera (not shown) on the backside.
[0039] In this example the user is operating the smartphone 302 in an AR mode. Similar to the example discussed above in connection with FIG. 1, the user has decided that he or she needs a keypad, such as for entering a telephone number or for entering a password or passcode for gaining access to some type of secure area. As such, the user has picked up or otherwise obtained the tangible object 308, which in this example just happens to comprise a mini sound bar (a type of consumer electronic device). The camera on the back of the smartphone 302 captures the view of the real world which includes the tangible object 308. As such, the tangible object 308 is displayed on the display 304 of the smartphone 302.
[0040] The user would like a keypad UI to be established on a surface of the tangible object 308. Given that the tangible object 308 comprises a sound bar, perhaps the back of the sound bar provides a smoother surface for a keypad. The user can choose whichever surface he or she prefers. In some embodiments, the user then provides a verbal command to the system 300 by speaking the word “keypad”. The word “keypad” may comprise a keyword that has a UI associated with it, which in this case is a keypad. In some embodiments, in response to the verbal command, a virtual overlay 310 is then established or projected onto the surface of the tangible object 308 in a manner that is visible to the user by being visible on the display 304.
[0041] Unlike the example discussed above in connection with FIG. 1, however, the surfaces of the tangible object 308 are rectangular in shape instead of being more square like a typical twelve button keypad. As such, in accordance with an embodiment of the present invention, the keypad that comprises the virtual overlay 310 is scaled in order to correctly fit the rectangular surfaces of the tangible object 308. In this example, the scaling comprises reconfiguring the arrangement of the buttons on the keypad so that it includes two rows of six buttons instead of the more typical four rows of three buttons. By scaling the keypad to include two rows of six buttons, the virtual overlay 310 is a much better fit for the rectangular surfaces of the tangible object 308. The scaling may also comprise making the buttons smaller or larger in order to make them all fit on the surface of the tangible object 308. It should be well understood that the virtual overlay 310 may comprise any type of UI or other feature, and that in some embodiments the scaling may comprise increasing or decreasing its size, reconfiguring or rearranging its inputs and/or outputs, modifying or changing its color for a better match, etc.
[0042] The establishment of the virtual overlay 310 on the surface of the tangible object 308 creates a virtual keypad, which comprises a type of virtual asset. With the keypad virtual overlay 310 having been scaled and established on the surface of the tangible object 308, the user is able to interact with something tangible such that the user can actually hold and feel a physical object and is not just pushing buttons in the air. As shown the tip of the user’s finger 312 is touching the tangible object 308 in the real world. As shown on the display 304, the user’s finger 312 is interacting with the virtual overlay 310 that is established on the tangible object 308. By touching the tangible object 308 and viewing the display 304, the user is able to enter a number, such as a phone number, into the keypad that comprises the virtual overlay 310.
[0043] In some embodiments, the scaling of the virtual overlay to fit the tangible object may happen automatically during the establishing of the virtual overlay on the tangible object. For example, when the user says the word “keypad”, that verbal command is heard and received by the processor based system included in the smartphone 302. In response to the verbal command or keyword “keypad”, the system begins the process of establishing the virtual overlay 310 on the tangible object 308. As part of that process, the system automatically considers the geometry and size of the tangible object 308 and recognizes that the keypad of the virtual overlay 310 will need to be scaled in order for it to fit on the tangible object 308. As such, the system automatically performs such scaling as part of the process of establishing the virtual overlay 310 on the tangible object 308. With this auto scaling feature, the system can resize, reconfigure, and/or rearrange a virtual UI to fit any size tangible object.
[0044] Thus, in some embodiments, a user holds a tangible object, and then says a keyword or command, and in response the system auto scales the virtual overlay (such as a UI) to, or for, the tangible object in order to make it fit the tangible object correctly. The auto scaling feature can be used to automatically make many different types, shapes, and size virtual UIs and other virtual features fit onto the surfaces of many different types, shapes, and size tangible objects. This allows a user to pick up any tangible object to use for a virtual asset because the auto scaling will automatically scale the asset, such as a UI, to fit onto the object.
[0045] For example, a user who is experiencing an AR, VR, or MR environment may want a calculator to use in that environment. A standard calculator would generally fit on a book which is rectangular. But if the user picks up a tangible object having square surfaces, the system will automatically scale the virtual calculator interface to fit on the square surfaces of the object. As such, with the auto scaling feature the user has much flexibility in choosing a tangible object for a UI because the system can auto-scale to associate the virtual overlay to any size object.
[0046] In some embodiments, in implementing the auto scaling feature, the system uses and considers the geometry, shape, and/or size of the tangible object that, in some embodiments, may be determined during the identifying of the tangible object as described above. The system defines the aspect ratio of the object that is picked up, and then matches the virtual UI to the object by automatically scaling it to fit the object correctly. Furthermore, in some embodiments, as the user moves the tangible object around, the system automatically and continuously rescales the virtual overlay to fit the tangible object in its various different positions so that it continues to look correct as viewed by an AR, VR, or MR device. That is, the system dynamically rescales, such as by resizing or reconfiguring, the virtual overlay in order to continue to fit the tangible object in the real world as it moves around. In some embodiments, it can be said that the system dynamically rescales the virtual overlay in the virtual world to fit the tangible object in the real world. Such dynamic rescaling keeps the virtual overlay properly projected and established on the tangible object as it moves around in the real world. In some embodiments, such dynamically rescaling is implemented by tracking the orientation of the tangible object as it moves, and then rescaling to keep everything oriented correctly so that the virtual overlay, such as a virtual UI, stays on the tangible, real object as it moves.
[0047] Thus, in some embodiments, the system automatically scales the virtual asset to the geometry of the tangible object. And in some embodiments, the system automatically continues to dynamically rescale, such as by resizing or reconfiguring, the virtual asset to the geometry of the tangible object.
[0048] In some embodiments, the establishing a virtual overlay on the tangible object comprises mapping one or more tangible input mechanisms on the tangible object to one or more virtual input devices on the virtual overlay. Specifically, some of the tangible objects that might be picked up and/or otherwise available to the user include tangible buttons, switches, and/or other input mechanisms. For example, a TV remote control, landline telephone, calculator, garage door opener, automobile key fob, QWERTY keyboard, microwave oven control panel, etc., all include tangible buttons, switches, and/or other input mechanisms. In some embodiments, the system detects that the tangible object has physical buttons and/or switches on it and maps those physical buttons and/or switches to the virtual buttons or other virtual input devices on the virtual asset. The system then detects that the user has pressed one of the tangible buttons on the tangible object and equates that to an activation of the corresponding mapped virtual input device on the virtual asset. This allows the user feel and press a tangible, physical button when interacting with the virtual UI that is displayed on the user’s AR or VR display. Furthermore, in this way a tangible, real device, such as a TV remote control, can be made into something else, like a calculator, etc., by mapping the buttons.
[0049] Referring to FIG. 4, there is illustrated an example of mapping tangible input mechanisms to virtual ones in accordance with an embodiment of the present invention. Specifically, there is illustrated a system 400 which is operating in accordance with an embodiment of the present invention. The system 400 includes a smartphone 402 having a display 404 and a camera (not shown) on the backside.
[0050] In this example the user is operating the smartphone 402 in an AR mode. The user has decided that he or she needs a calculator to make some calculations involving the number pi (.pi.), which is approximately equal to 3.14159. As such, the user has picked up or otherwise obtained the tangible object 408 in the real world, which in this example comprises a television (TV) remote control. The camera on the back of the smartphone 402 captures the view of the real world which includes the tangible object 408. As such, the tangible object 408 is displayed on the display 404 of the smartphone 402.
[0051] The user would like a virtual calculator to be established on the tangible object 408 and displayed on the display 404. As such, in some embodiments, the user provides a verbal command to the system 400 by speaking the word “calculator”. The word “calculator” comprises a keyword that has a calculator UI associated with it. In some embodiments, in response to the verbal command, a virtual overlay 410 is then established or projected onto the surface of the tangible object 408 in a manner that is visible to the user by being visible on the display 404. That is, a virtual calculator, which comprise a type of virtual asset, becomes visible on the display 404.
[0052] Because the tangible object 408 comprises a TV remote control, it includes several tangible, physical, and real input buttons. In some embodiments, as part of the process of establishing the virtual overlay 410 on the tangible object 408, the system maps one or more of the tangible, physical, and real input buttons of the TV remote control to the virtual inputs that are displayed on the display 404 and that are needed for the virtual calculator. For example, as shown the following button mappings have been established:
[0053] The numerical buttons 1-9 and 0 on the TV REMOTE have been mapped to the numerical buttons 1-9 and 0 on the virtual CALULATOR.
[0054] The “Input” button on the TV REMOTE has been mapped to the decimal point (i.e. “.”) button on the virtual CALULATOR.
[0055] The “VOL+”, “VOL-“, “CH+”, and “CH-” buttons on the TV REMOTE have been mapped to the “+”, “+”, “.times.”, and “-” buttons, respectively, on the virtual CALULATOR.
[0056] The “Guide” and “Mute” buttons on the TV REMOTE have been mapped to the “=” and “Clear” buttons, respectively, on the virtual CALULATOR.
[0057] In addition, in some embodiments, one or more of the tangible, physical, and real input buttons on a tangible object may either be completely eliminated in a virtual overlay, such as by being covered up or masked, or may be replaced by a different asset or feature. For example, as shown the following replacement has been established:
[0058] The “POWER” button on the TV REMOTE has been replaced by an output display screen 414 on the virtual CALULATOR.
……
……
……
