Sony Patent | Image Generating Apparatus And Image Generating Method

Patent: Image Generating Apparatus And Image Generating Method

Publication Number: 20200162716

Publication Date: 20200521

Applicants: Sony

Abstract

A position and posture acquiring section 250 of an image generating apparatus 200 acquires information related to the position and posture of the user’s head. A view screen control section 252 sets a view screen corresponding to a viewpoint of the user. An original image manipulating section 254 determines displacement of pixels and acquires color values for an image corresponding to the viewpoint of the user such that an object appears fixed according to movement of the viewpoint of the user, and performs synthesis of color values in a color map corresponding to the viewpoint of a preceding frame, to acquire a new color map. A display image generating section 268 determines a value for each of the pixels in the view screen in accordance with a color map for a corresponding position to generate a display image. An output section 270 outputs the display image.

TECHNICAL FIELD

[0001] The present invention relates to an image generating apparatus and an image generating method for generating stereoscopic videos.

BACKGROUND ART

[0002] Three-dimensional display devices such as three-dimensional televisions and head-mounted displays have been utilized that are capable of stereoscopically presenting videos. Devices have also been developed that are capable of stereoscopically presenting videos on portable terminals such as cellular phones and portable game machines. This has led to an increase in opportunities for general users to view stereoscopic videos.

[0003] A three-dimensional display device displaying stereoscopic videos enable a user to stereoscopically view an image by causing the left and right eyes of the user to view respective images with parallaxes. Methods for causing the left and right eyes to view respective images with parallaxes include the use of special optical glasses and the use of a parallax barrier or a lenticular lens instead of the optical glasses.

SUMMARY

Technical Problems

[0004] To cause the user to view undistorted stereoscopic videos, accurate parallax images based on the viewpoint of the user need to be generated. Thus, to present stereoscopic videos while permitting movement of the viewpoint, for example, processing is generally needed in which an object is placed in a virtual three-dimensional space and in which the object is projected with a camera coordinate system changed. However, pursue of the quality and accuracy of images leads to an increase in time needed for the processing. This in turn makes display difficult to follow movement of the viewpoint.

[0005] Additionally, many manipulations are applied to the data of the original parallax image, leading to an increase in the likelihood of degradation of the image.

[0006] In view of these problems, an object of the present invention is to provide a technique capable of generating a high-quality stereoscopic image with reduced delay in spite of displacement of the viewpoint.

Solution to Problems

[0007] A certain aspect of the present invention relates to an image generating apparatus. The image generating apparatus is an image generating apparatus using a pair of original images acquired from left and right different viewpoints to generate an image making an object stereoscopically visible, the image generating apparatus including an original image manipulating section calculating displacement of pixels in the original images according to movement of viewpoints of a user such that the object is fixed in a virtual space with respect to the movement of the viewpoints of the user to acquire a color value for each of pixels in an image corresponding to the viewpoints at a certain point in time, and performing synthesis of color values for pixels expressing an identical image in a color map expressing an image corresponding to the viewpoints at a preceding point in time to generate a new color map at the certain point in time, a display image generating section determining a color value for each of the pixels on a basis of a color value for a position on the color map corresponding to the pixel in a display image, and an output section outputting data of the display image.

[0008] Another aspect of the present invention relates to an image generating method. The image generating method is an image generating method of using a pair of original images acquired from left and right different viewpoints to generate an image making an object stereoscopically visible, the image generating method including a step of calculating displacement of pixels in the original images according to movement of viewpoints of a user such that the object is fixed in a virtual space with respect to the movement of the viewpoints of the user to acquire a color value for each of pixels in an image corresponding to the viewpoints at a certain point in time, and performing synthesis of color values for pixels expressing an identical image in a color map expressing an image corresponding to the viewpoints at a preceding point in time to generate a new color map at the certain point in time, a step of determining a color value for each of the pixels in a display image on a basis of a color value for a position on the color map corresponding to the pixel, and a step of outputting data of the display image.

[0009] Note that valid aspects of the present invention include an optional combination of the above-described components and the expression of the present invention converted between the method, the apparatus, and the like.

Advantageous Effect of Invention

[0010] According to the present invention, a high-quality stereoscopic image can be presented with reduced delay in spite of displacement of the viewpoint.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is an appearance diagram of a head-mounted display according to the present embodiment.

[0012] FIG. 2 is a configuration diagram of an image display system according to the present embodiment.

[0013] FIG. 3 is a diagram illustrating an aspect of display implemented by the image display system of the present embodiment.

[0014] FIG. 4 is a diagram schematically illustrating an image generated by an image generating apparatus of the present embodiment.

[0015] FIG. 5 is a diagram illustrating an image and parallax relationship, the parallax being provided at a stage of acquisition of parallax images according to the present embodiment.

[0016] FIG. 6 is a diagram illustrating a viewpoint and image relationship in a case where parallax images are viewed from an appropriate position.

[0017] FIG. 7 is a diagram illustrating images of the same object expressed in left and right original images, the images overlapping each other.

[0018] FIG. 8 is a flowchart schematically illustrating a processing procedure in which the image generating apparatus according to the present embodiment generates a display image from original images.

[0019] FIG. 9 is a diagram illustrating an internal circuit configuration of the image generating apparatus according to the present embodiment.

[0020] FIG. 10 is a diagram illustrating functional blocks of the image generating apparatus according to the present embodiment.

[0021] FIG. 11 is a diagram illustrating a relationship between moving distance components in a Z-axis direction and an X-axis direction included in the moving distance of a viewpoint and the corresponding moving distances of pixels in original images according to the present embodiment.

[0022] FIG. 12 is a diagram illustrating a relationship between moving distance components in the Z-axis direction and the Y-axis direction included in the moving distance of the viewpoint and the corresponding moving distances of the pixels in the original images according to the present embodiment.

[0023] FIG. 13 is a diagram illustrating a relationship between moving distance components in the Z-axis direction and the X-axis direction included in the moving distance of the viewpoint and the moving distance of each of the pixels in a case where the original image for the left eye is referenced in an image reference vector map for the right eye according to the present embodiment.

[0024] FIG. 14 is a diagram illustrating a method for calculating image reference vectors in extended reference according to the present embodiment.

[0025] FIG. 15 is a diagram schematically illustrating a relationship between displacement vectors and pixel areas according to the present embodiment.

[0026] FIG. 16 is a flowchart schematically illustrating a processing procedure for determining an image reference vector for each of pixels on an image reference vector map according to the present embodiment.

[0027] FIG. 17 is a diagram illustrating a method for interpolating image reference vectors according to the present embodiment.

[0028] FIG. 18 is a flowchart illustrating a processing procedure in which, in S12 in FIG. 8, an original image manipulating section 254 generates an image reference vector map.

[0029] FIG. 19 is a diagram illustrating a procedure for processing of writing, in S22 in FIG. 18, to a Z buffer to generate an image reference vector the reference destination of which is a self image.

[0030] FIG. 20 is a flowchart illustrating a procedure for processing of setting, in S22 in FIG. 18, an image reference vector the reference destination of which is a self image.

[0031] FIG. 21 is a diagram illustrating an AtomicMin instruction used to write Z values to a Z buffer 264 in S36 in FIG. 19.

[0032] FIG. 22 is a diagram illustrating a procedure for processing of integrating the processing in FIG. 19 with the processing in FIG. 20 to efficiently generate the image reference vector map.

[0033] FIG. 23 is a flowchart illustrating a procedure in which, in S14 in FIG. 8, a display image generating section 268 uses the image reference vector map to generate a display image.

[0034] FIG. 24 is a diagram illustrating a position relationship between the display image, the image reference vector map, and the original images.

[0035] FIG. 25 is a diagram illustrating a recursive filter introduced according to the present embodiment.

[0036] FIG. 26 is a diagram schematically illustrating a procedure for generating a color map in a case where the recursive filter is introduced.

[0037] FIG. 27 is a diagram schematically illustrating the procedure for generating a color map in the case where the recursive filter is introduced.

[0038] FIG. 28 is a diagram schematically illustrating the procedure generating a color map in the case where the recursive filter is introduced.

[0039] FIG. 29 is a diagram schematically illustrating a procedure in which the display image generating section according to the present embodiment determines, for pixels to which the recursive filter is not applied, a color value for each of the pixels in the display image.

[0040] FIG. 30 is a diagram schematically illustrating a procedure in which the display image generating section according to the present embodiment determines, for pixels to which the recursive filter is applied, a color value for each of the pixels in the display image.

[0041] FIG. 31 is a diagram illustrating a relationship between moving distance components in the Z-axis direction and the X-axis direction included in the moving distance of the viewpoint and the corresponding moving distance of each of the pixels between image reference vector maps according to the present embodiment.

[0042] FIG. 32 is a diagram illustrating a relationship between moving distance components in the Z-axis direction and the Y-axis direction included in the moving distance of the viewpoint and the corresponding moving distance of each of the pixels between image reference vector maps according to the present embodiment.

[0043] FIG. 33 is a diagram schematically illustrating an example of a change in display image caused by a tilt of the head of a viewer.

[0044] FIG. 34 is a diagram illustrating a method for adjusting the position of a map screen in accordance with the positions of the viewpoints according to the present embodiment.

[0045] FIG. 35 is a flowchart illustrating a processing procedure in which the original image manipulating section of the image generating apparatus according to the present embodiment adjusts the position of an image on an XY plane according to the position of the head of the viewer.