Sony Patent | Generation Device And Generation Method, And Reproduction Device And Reproduction Method

Patent: Generation Device And Generation Method, And Reproduction Device And Reproduction Method

Publication Number: 20200137417

Publication Date: 20200430

Applicants: Sony

Abstract

The present disclosure relates to a generation device and a generation method, and a reproduction device and a reproduction method enabled to provide a mapping that is applicable to an arbitrary mapping method and in which high resolution is set in a viewing direction. The generation device includes a normalization unit that converts a first vector that maps a 360-degree image onto a predetermined 3D model into a second vector of a 3D model of a unit sphere. The present disclosure can be applied to, for example, a generation device that generates a 360-degree image from captured images in six directions, and generates an encoded stream in which high resolution is set in a predetermined direction of the 360-degree image, and the like.

TECHNICAL FIELD

[0001] The present disclosure relates to a generation device and a generation method, and a reproduction device and a reproduction method, and in particular, a generation device and a generation method, and a reproduction device and a reproduction method enabled to provide a mapping that is applicable to an arbitrary mapping method and in which high resolution is set in a viewing direction.

BACKGROUND ART

[0002] Examples of a method of reproducing an omnidirectional image in which it is possible to look around 360 degrees in all directions of top, bottom, left, and right, includes: a method of viewing while changing a display direction by using a controller on a general display; a method of displaying on a mobile terminal screen held in a hand by changing a direction on the basis of posture information obtained from a gyro sensor incorporated in the terminal; a method of displaying an image by a head mounted display mounted on the head reflecting movement of the head; and the like.

[0003] A feature of a reproduction device for one person that reproduces an omnidirectional image by these reproduction methods, is that an image actually displayed is limited to a part of the omnidirectional image since it is not necessary to display data in a direction in which a viewer is not viewing although input image data is prepared for the omnidirectional image. If transmission and decoding processing can be reduced of a part not being displayed, line bandwidth efficiency can be increased.

[0004] However, in a general method of compressing using a standard codec such as Moving Picture Experts Group phase 2 (MPEG2) or Advanced Video Coding (AVC), both the space direction and the time axis direction are compressed using redundancy, and it is difficult to cut out and decode an arbitrary area, and to start reproduction from an arbitrary time.

[0005] Moreover, there is also a problem of server response delay in a case where data is sent for distribution or the like from a server via a network. Therefore, it is very difficult to instantaneously switch data by using a video codec, and even if omnidirectional data is divided into multiple data, when the viewer suddenly changes a direction of viewing, a state occurs of data loss in which there is no data to be displayed, until the switching is actually occurs. Since reproduction quality is extremely degraded when this data loss state occurs, in construction of an actual omnidirectional distribution system, it is necessary to maintain a state in which minimum data for all directions exist to prepare for sudden turning around.

[0006] Therefore, various methods have been devised to make an image high resolution and high quality in a direction in which the viewer is viewing while securing the minimum data for the omnidirectional image within a limited line bandwidth.

[0007] For example, Patent Document 1 devises a so-called divided hierarchical distribution system in which a low-resolution omnidirectional image of equirectangular projection and a high-resolution partially cut-out image are combined and viewed. In the divided hierarchical distribution system, a high resolution partial image in a viewing direction is distributed while being switched depending on a direction of the viewer. In a case of drawing of an area where the high resolution partial image around the visual field is not covered, or in a case where the stream cannot be switched in time due to sudden turning around, a low-resolution omnidirectional image is used, to prevent occurrence of a state in which there is no image to be displayed.

[0008] A problem with such a divided hierarchical method is that, an image is drawn by using the high resolution with good image quality in a portion where the high resolution and the low resolution overlap each other, so that a low resolution side of the overlapping portion is wasted. Double transmission of the high resolution and the low resolution results in loss of line bandwidth.

[0009] There is also a method devised for eliminating the waste of double transmission of this overlapping portion.

[0010] For example, there is a method of using a mapping designed such that directions for the omnidirectional image is included in one image, and an area, in other words, the number of pixels allocated varies depending on the direction, and one specific direction has high resolution.

[0011] This method is described, for example, in the chapter “A.3.2.5 Truncated pyramid” of Working Draft “WD on ISO/IEC 23000-20 Omnidirectional Media Application Format” published at the Geneva meeting in June 2016 of the MPEG meeting (See Non-Patent Document 1) Furthermore, a similar method is also described in Non-Patent Document 2. Facebook, Inc. publishes another mapping based on a similar idea, which is called as “Pyramid Coding”, on the Web (see Non-Patent Document 3)

[0012] According to this method, the number of allocated pixels in the front area is large, and the resolution is high. A plurality of bit streams having different front directions is prepared, and transmission is performed while the bit streams are switched depending on the direction in which the viewer is viewing. Even if there is a difference in resolution, data in all directions are included, so that a sudden change in viewing direction does not cause data loss.

[0013] Such a mapping in which high resolution is set in the viewing direction front, and low resolution is set in the other directions, is called “viewport dependent projection mapping” in the MPEG meeting.

CITATION LIST

Patent Document

[0014] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-15705

Non-Patent Document

[0014] [0015] Non-Patent Document 1: http://mpeg.chiariglione.org/standards/mpeg-a/omnidirectional-media-appli- cation-format/wd-isoiec-23000-20-omnidirectional-media [0016] Non-patent Document 2: https://www.itu.int/en/ITU-T/studygroups/2017-2020/16/Documents/ws/201701- ILE/S2P2-1701-01-19-MPEG_Immersive_Media_Thomas_Stockhammer-.pdf [0017] Non-Patent Document 3: https://code.facebook.com/posts/112635400399553/next generation-video-encoding-techniques-for-360-video-and-vr/

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0018] The above-described method of using a mapping in which high resolution is set in the viewing direction front and low resolution is set in the other directions, is a method using a dedicated mapping in which a mapping that causes a resolution difference is newly defined. The method using the dedicated mapping becomes incompatible with a normal mapping method, in other words, a mapping of displaying an omnidirectional image with uniform pixel density (resolution) in all directions. Therefore, a mapping is desired that can use a mapping designed for a general omnidirectional image as it is, and can generate an image with a high resolution in a specific direction.

[0019] The present disclosure has been made in view of such a situation, and is enabled to provide a mapping that is applicable to an arbitrary mapping method and in which high resolution is set in a viewing direction.

Solutions to Problems

[0020] A generation device of a first aspect of the present disclosure includes a normalization unit that converts a first vector that maps a 360-degree image onto a predetermined 3D model into a second vector of a 3D model of a unit sphere.

[0021] A generation method of the first aspect of the present disclosure includes converting a first vector of a predetermined 3D model onto which a 360-degree image is mapped into a second vector of a 3D model of a unit sphere.

[0022] In the first aspect of the present disclosure, the first vector of the predetermined 3D model onto which the 360-degree image is mapped is converted into the second vector of the 3D model of the unit sphere.

[0023] A reproduction device of a second aspect of the present disclosure includes: a receiving unit that receives a 360-degree image generated by another device; and a normalization unit that converts a first vector that maps the 360-degree image onto a predetermined 3D model into a second vector of a 3D model of a unit sphere.

[0024] A reproduction method of the second aspect of the present disclosure includes receiving a 360-degree image generated by another device, and converting a first vector that maps the 360-degree image onto a predetermined 3D model into a second vector of a 3D model of a unit sphere.

[0025] In the second aspect of the present disclosure, the 360-degree image generated by the other device is received, and the first vector that maps the 360-degree image onto the predetermined 3D model is converted into the second vector of the 3D model of the unit sphere.

[0026] Note that, the generation device of the first aspect and the reproduction device of the second aspect of the present disclosure can be implemented by causing a computer to execute a program.

[0027] Furthermore, to implement the generation device of the first aspect and the reproduction device of the second aspect of the present disclosure, the program to be executed by the computer can be provided by being transmitted via a transmission medium or recorded on a recording medium.

[0028] The generation device and the reproduction device each may be an independent device or an internal block included in one device.

Effects of the Invention

[0029] According to the first and second aspects of the present disclosure, it is enabled to provide a mapping that is applicable to an arbitrary mapping method and in which high resolution is set in a viewing direction.

[0030] Note that, the effect described here is not necessarily limited, and can be any effect described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of a distribution system to which the present disclosure is applied.

[0032] FIG. 2 is a block diagram illustrating a configuration example of a generation device of FIG. 1.

[0033] FIG. 3 is a diagram illustrating setting of the high resolution direction.

[0034] FIG. 4 is a block diagram illustrating a detailed configuration of a mapping processing unit.

[0035] FIG. 5 is a diagram illustrating mappings of the mapping processing unit.

[0036] FIG. 6 is a diagram illustrating a relationship between a texture image and a 3D model in regular octahedron mapping.

[0037] FIG. 7 is a diagram illustrating processing of the mapping processing unit according to the first embodiment.

[0038] FIG. 8 is a diagram illustrating a case where each pixel on the spherical surface of a unit sphere is viewed from an offset position.

[0039] FIG. 9 is a diagram illustrating a correspondence between a two-dimensional texture image and a 3D model by mapping processing of a mapping f.

[0040] FIG. 10 is a diagram illustrating a correspondence between a two-dimensional texture image and a 3D model by mapping processing of a mapping f’.

[0041] FIG. 11 is a diagram illustrating a correspondence between a two-dimensional texture image and a 3D model by mapping processing of a mapping f’ FIG. 12 is a diagram illustrating a configuration example of auxiliary information generated by a table generation unit.

[0042] FIG. 13 is a diagram illustrating an azimuth angle .theta., an elevation angle .phi., and a rotation angle .psi..

[0043] FIG. 14 is a diagram illustrating an example of six omnidirectional images having different high resolution directions.

[0044] FIG. 15 is a flowchart illustrating generation processing.

[0045] FIG. 16 is a block diagram illustrating a configuration example of a distribution server and a reproduction device of FIG. 1.

[0046] FIG. 17 is a diagram illustrating processing of a drawing unit.

[0047] FIG. 18 is a flowchart illustrating reproduction processing.

[0048] FIG. 19 is a diagram illustrating a modification of the first embodiment.

[0049] FIG. 20 is a diagram illustrating the modification of the first embodiment.

[0050] FIG. 21 is a diagram illustrating the modification of the first embodiment.

[0051] FIG. 22 is a diagram illustrating a direction of the mapping of the mapping processing unit of the generation device and the reproduction device.

[0052] FIG. 23 is a diagram illustrating the direction of the mapping of the mapping processing unit of the generation device and the reproduction device.

[0053] FIG. 24 is a block diagram illustrating a configuration example of a mapping processing unit according to a second embodiment.

[0054] FIG. 25 is a diagram illustrating mappings of the mapping processing unit according to the second embodiment.

[0055] FIG. 26 is a diagram illustrating processing of the mapping processing unit according to the second embodiment.

[0056] FIG. 27 is a diagram illustrating a correspondence between a two-dimensional texture image and a 3D model by mapping processing of a mapping g.

[0057] FIG. 28 is a diagram illustrating a relationship between an eccentricity ratio k and a resolution improvement ratio in the first embodiment.

[0058] FIG. 29 is a diagram illustrating a relationship between the eccentricity ratio k and the resolution improvement ratio in the second embodiment.

[0059] FIG. 30 is a diagram illustrating an example of the two-dimensional texture image when the eccentricity ratio k is changed in the first embodiment.

[0060] FIG. 31 is a diagram illustrating an example of the two-dimensional texture image when the eccentricity ratio k is changed in the second embodiment.

[0061] FIG. 32 is a conceptual diagram of mapping processing in the first embodiment.

[0062] FIG. 33 is a conceptual diagram of mapping processing in the second embodiment.

[0063] FIG. 34 is a diagram illustrating a difference between the mapping processing of the first embodiment and the mapping processing of the second embodiment.

[0064] FIG. 35 is a diagram for comparative illustration with a case where a cube model is adopted as the 3D model.

[0065] FIG. 36 is a diagram illustrating processing of a mapping processing unit according to a third embodiment.

[0066] FIG. 37 is a conceptual diagram of mapping processing in the third embodiment.

[0067] FIG. 38 is a diagram illustrating another example of a parameter table as the auxiliary information.

[0068] FIG. 39 is a diagram illustrating a modification in which an encoded stream is an omnidirectional image for a 3D image.

[0069] FIG. 40 is a block diagram illustrating a configuration example of a computer to which the present disclosure is applied.

[0070] FIG. 41 is a block diagram illustrating an example of a schematic configuration of a vehicle control system.

[0071] FIG. 42 is an explanatory diagram illustrating an example of installation positions of a vehicle exterior information detecting unit and an imaging unit.

MODE FOR CARRYING OUT THE INVENTION

[0072] The following is a description of a mode for carrying out a technology according to the present disclosure (the mode will be hereinafter referred to as the embodiment). Note that, description will be made in the following order.

[0073] 1.* First Embodiment*

[0074] 2.* Direction of mapping of mapping processing unit of generation device and reproduction device*

[0075] 3.* Second Embodiment*

[0076] 4.* Relationship between eccentricity ratio k and resolution improvement ratio*

[0077] 5.* Difference between first embodiment and second embodiment*

[0078] 6.* Combination of first embodiment and second embodiment*

[0079] 7.* Conclusion*

[0080] 8.* Modifications*

[0081] 9.* Computer configuration example*

[0082] 10.* Application example*

1.* First Embodiment*

Configuration Example of First Embodiment of Distribution System

[0083] FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of a distribution system to which the present disclosure is applied.

[0084] A distribution system 10 of FIG. 1 includes an imaging device 11, a generation device 12, a distribution server 13, a network 14, a reproduction device 15, and a head mounted display 16. The distribution system 10 generates an omnidirectional image from a captured image captured by the imaging device 11, and displays a display image of a visual field range of a viewer by using the omnidirectional image.

[0085] Specifically, the imaging device 11 of the distribution system 10 includes six cameras 11A-1 to 11A-6. Note that, in the following, in a case where it is not necessary to distinguish the cameras 11A-1 to 11A-6 in particular, they are referred to as cameras 11A.

[0086] Each of the cameras 11A captures a moving image. The imaging device 11 supplies, as captured images, moving images in six directions captured by the respective cameras 11A to the generation device 12. Note that, the number of cameras provided in the imaging device 11 is not limited to six, as long as it is plural.

[0087] The generation device 12 generates an omnidirectional image of 360 degrees around in the horizontal direction and 180 degrees around in the vertical direction, from the captured images supplied from the imaging device 11, with a method using equirectangular projection. The generation device 12 compresses and encodes image data in which the omnidirectional image in which it is possible to look around 360 degrees in all directions of top, bottom, left, and right with the equirectangular projection is mapped onto a predetermined 3D model, with a predetermined encoding method such as Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC)/H.265.

[0088] When mapping the omnidirectional image onto the predetermined 3D model, the generation device 12 sets a plurality of directions corresponding to gaze directions of the viewer, and converts the mapping so that high resolution is set for the resolution in the set directions and low resolution is set in the directions opposite to the set directions, for each direction. Then, the generation device 12 generates an encoded stream in which image data of the omnidirectional image generated by using the converted mapping is compressed and encoded.

[0089] For example, assuming that, with the center of a cube as a viewing position, six directions from the center in directions perpendicular to respective faces of the cube are set as the plurality of directions, the generation device 12 generates six encoded streams in which the image data of the omnidirectional image is compressed and encoded so that the encoded streams correspond to the six directions. The generated six encoded streams become image data of an omnidirectional image having different directions in which high resolution is set (hereinafter, also referred to as high resolution directions).

[0090] The generation device 12 uploads, to the distribution server 13, a plurality of encoded streams respectively having different high resolution directions. Furthermore, the generation device 12 generates auxiliary information for identifying the plurality of encoded streams, and uploads the auxiliary information to the distribution server 13. The auxiliary information is information that defines which direction the high resolution direction is, or the like.

[0091] The distribution server 13 connects to the reproduction device 15 via the network 14. The distribution server 13 stores the plurality of encoded streams and the auxiliary information uploaded from the generation device 12. In response to a request from the reproduction device 15, the distribution server 13 transmits the stored auxiliary information and at least one of the plurality of encoded streams to the reproduction device 15 via the network 14.

[0092] The reproduction device 15 requests the distribution server 13 via the network 14 to transmit the auxiliary information for identifying the plurality of encoded streams stored in the distribution server 13, and receives the auxiliary information transmitted in response to the request.

[0093] Furthermore, the reproduction device 15 incorporates a camera 15A and images a marker 16A attached to the head mounted display 16. Then, the reproduction device 15 detects the viewing position of the viewer in a coordinate system of the 3D model (hereinafter referred to as a 3D model coordinate system) on the basis of the captured image of the marker 16A. Moreover, the reproduction device 15 receives a detection result of a gyro sensor 16B of the head mounted display 16 from the head mounted display 16. The reproduction device 15 determines a gaze direction of the viewer in the 3D model coordinate system on the basis of the detection result of the gyro sensor 16B. The reproduction device 15 determines a visual field range of the viewer positioned inside the 3D model on the basis of the viewing position and the gaze direction.

[0094] Then, on the basis of the auxiliary information and the visual field range of the viewer, the reproduction device 15 requests of the distribution server 13 via the network 14 one of the plurality of encoded streams, and receives one encoded stream transmitted from the distribution server 13 in response to the request.

[0095] In other words, the reproduction device 15 determines, as an encoded stream to be acquired, an encoded stream in a high resolution direction closest to the gaze direction of the viewer among the plurality of encoded streams stored in the distribution server 13, and requests the distribution server 13 to transmit the encoded stream.

[0096] The reproduction device 15 decodes the received one encoded stream. The reproduction device 15 generates a 3D model image by mapping an omnidirectional image obtained as a result of decoding onto a predetermined 3D model.

[0097] Then, the reproduction device 15 generates an image of the visual field range of the viewer as the display image, by perspectively projecting the 3D model image onto the visual field range of the viewer with the viewing position as a focal point. The reproduction device 15 supplies the display image to the head mounted display 16.

[0098] The head mounted display 16 is mounted on the head of the viewer and displays the display image supplied from the reproduction device 15. To the head mounted display 16, the marker 16A imaged by the camera 15A is attached. Thus, the viewer can designate the viewing position by moving while the head mounted display 16 is mounted on the head. Furthermore, the head mounted display 16 incorporates a gyro sensor 16B, and a detection result of angular velocity by the gyro sensor 16B is transmitted to the reproduction device 15. Thus, the viewer can designate the gaze direction by rotating the head on which the head mounted display 16 is mounted.