Sony Patent | Using a Portable Device to Interface with a Scene Rendered on a Main Display

Patent: Using a Portable Device to Interface with a Scene Rendered on a Main Display

Publication Number: 20190030441

Publication Date: 2019-01-31

Applicants: Sony

Abstract

A method for interfacing with a scene is provided, including: rendering display data on a main display, the display data defining a scene rendered on the main display, the display data being configured to include a visual cue; capturing the display data by an image capture device incorporated into a portable device; analyzing the captured display data to identify the visual cue; in response to identification of the visual cue, determining additional information that is in addition to the scene that is displayed on the main display, the additional information defining graphics or text to be added to the scene; presenting the additional information on a personal display incorporated into the portable device, the presentation of the additional information being synchronized to the rendering of the display data on the main display.

Background

Augmented reality (AR) attempts to enrich a user’s real environment by adding spatially aligned virtual objects (3D models, 2D textures, textual annotations, etc.) to the user’s environment. The goal is to create the impression that the virtual object is a part of the real environment. The users of the AR system experience the augmented environment through special display devices that are typically worn on the body, e.g., head mounts.

Current augmented reality systems suffer from issues with general registration problems, i.e., getting the computer graphics to stick to a real-world scene. For example, due to the use of multiple sensors, artifacts, such as miss-movement of the computer graphics with the real-world scene may occur. These artifacts are due to the multiple sensors having different speeds. Additionally, delays between the head tracking system and the camera used to capture the real-world scene along with the need to merge the computer graphics into the real-world scene, cause misalignment/registration problems. These registration problems may even become serious enough to cause a user to become uncomfortable when viewing the image. Additionally, the augmented reality systems tend to be complex systems that are not portable. Accordingly, there are no viable consumer applications for these augmented reality systems.

As a result, there is a need to solve the problems of the prior art to provide a method and apparatus for providing a portable device capable of providing an augmented reality experience.

Summary

Broadly speaking, the present invention fills these needs by providing a method and device for providing an enhanced shopping experience using a portable device. It should be appreciated that the present invention can be implemented in numerous ways, including as a method, a system, computer readable media or a device. Several inventive embodiments of the present invention are described below.

In one embodiment, a portable device configured to provide enhanced shopping information is included. The portable device has a display screen and an image capture device and the portable device is configured to access databases through a wireless network. The portable device includes image recognition logic that is configured to perform analysis of an image of an object that includes a bar code associated with a product. The analysis determines if the graphics found on the object corresponds to a bar code and a portion of the image with the bar code is communicated through the wireless network to databases to identify the product. The portable device further includes image generation logic that is configured to obtain product information for the identified product from the databases and present the product information on the display screen of the portable device.

In another embodiment, a method for obtaining product information through a portable device is provided. An image of an object including a bar code associated with a product is captured using an image capture device. The captured image is analyzed to determine if graphics found on the object correspond to a bar code. A portion of the image with the bar code is transmitted to databases through a wireless network and the product associated with the bar code is identified. Product information from the databases is obtained and presented through image generation logic incorporated into the portable device. The presented product information defines multimedia data in a graphical user interface that includes one of an image or text information about the product.

Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.