Sony Patent | Image processing apparatus, image processing method, and program

Patent: Image processing apparatus, image processing method, and program

Drawings: Click to check drawins

Publication Number: 20210400255

Publication Date: 20211223

Applicant: Sony

Abstract

The present technology relates to an image processing apparatus, an image processing method, and a program that enable reproduction-and-editing work for free viewpoint data to be performed easily. Generated is reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, in which, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space. The present technology is applicable to, for example, reproduction and editing of free viewpoint data.

Claims

1. An image processing apparatus, comprising: a reproduction control unit configured to generate reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, wherein, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

2. The image processing apparatus according to claim 1, wherein the reproduction control unit further controls the reproduction of the free viewpoint data in accordance with the reproduction control information.

3. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for start of the reproduction with a frame showing an object selected in the 3D strobe image.

4. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for start of the reproduction with a frame showing an object selected in the 3D strobe image and for end of the reproduction at a frame showing another object selected in the 3D strobe image.

5. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for deletion from a frame showing an object selected in the 3D strobe image to a frame showing another object selected in the 3D strobe image.

6. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for jump of the reproduction from a frame showing an object selected in the 3D strobe image to a frame showing another object selected in the 3D strobe image.

7. The image processing apparatus according to claim 6, wherein the reproduction control unit generates the reproduction control information for jump of the reproduction from the frame showing the object selected in the 3D strobe image of predetermined moving image content to the frame showing the another object selected in the 3D strobe image of another piece of moving image content.

8. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for link of an object selected in the 3D strobe image with predetermined data.

9. The image processing apparatus according to claim 1, wherein the reproduction control unit generates the reproduction control information for application of effect processing to an object selected in the 3D strobe image.

10. An image processing method, comprising: generating reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, wherein, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

11. A computer program for causing a computer to function as a reproduction control unit that generates reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, wherein, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

Description

TECHNICAL FIELD

[0001] The present technology relates to an image processing apparatus, an image processing method, and a program, and particularly relates to an image processing apparatus, an image processing method, and a program that enable, for example, reproduction-and-editing work for free viewpoint data to be performed easily.

BACKGROUND ART

[0002] There has been proposed a method of generating a strobe image showing a subject (image) shot at a plurality of times (see, for example, Patent Document 1). Such a strobe image shows a subject at a plurality of times, and thus the motion and trajectory of the subject can be grasped easily.

CITATION LIST

Patent Document

[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 2007-259477

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0004] In recent years, there has been an increasing demand for easily performing reproduction-and-editing work regarding reproduction and editing for free viewpoint data enabling generation of a free viewpoint image in which the appearance when viewing a three-dimensional space from a virtual viewpoint is recreated.

[0005] The present technology has been made in view of such a situation, and enables reproduction-and-editing work for free viewpoint data to be performed easily.

Solutions to Problems

[0006] An image processing apparatus or a program according to the present technology is an image processing apparatus or a program to cause a computer to function as such an information processing apparatus, and the image processing apparatus includes a reproduction control unit configured to generate reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, in which, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

[0007] An image processing method according to the present technology is an image processing method includes generating reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, in which, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

[0008] In the image processing apparatus, the image processing method, and the program according to the present technology, generated is reproduction control information for control of reproduction of free viewpoint data generated from a plurality of viewpoint images in response to an operation on a three-dimensional (3D) strobe image including a strobe model viewed from a virtual viewpoint, in which, in the strobe model, 3D models of objects at a plurality of times being generated from the plurality of viewpoint images shot from a plurality of viewpoints are disposed in a three-dimensional space.

[0009] Note that the image processing apparatus and a display device each may be an independent apparatus, or each may be an internal block included in one apparatus.

[0010] In addition, the program can be provided by being transmitted through a transmission medium, or by being recorded on a recording medium.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a block diagram illustrating an exemplary configuration of an embodiment of an image processing system with the present technology applied.

[0012] FIG. 2 is a block diagram illustrating an exemplary configuration of a content server 12.

[0013] FIG. 3 is a block diagram illustrating an exemplary configuration of a reproduction-and-editing device 13.

[0014] FIG. 4 is a flowchart explanatorily illustrating exemplary processing performed by the content server 12.

[0015] FIG. 5 is a flowchart explanatorily illustrating exemplary processing performed by the reproduction-and-editing device 13.

[0016] FIG. 6 illustrates an exemplary unnatural three-dimensional (3D) strobe image.

[0017] FIG. 7 illustrates an exemplary natural 3D strobe image.

[0018] FIG. 8 illustrates exemplary frames of a viewpoint image in a strobe section.

[0019] FIG. 9 illustrates exemplary strobe model generation with frames at times t1 to t9 as a strobe section.

[0020] FIG. 10 illustrates exemplary display of a 3D strobe image generated by shooting the strobe model with a virtual camera.

[0021] FIG. 11 explanatorily illustrates an exemplary user operation in search for a predetermined scene (frame) for free viewpoint data.

[0022] FIG. 12 explanatorily illustrates exemplary moving image content of free viewpoint data.

[0023] FIG. 13 illustrates an exemplary strobe model generated with the moving image content of free viewpoint data.

[0024] FIG. 14 explanatorily illustrates a first exemplary user operation on a 3D strobe image.

[0025] FIG. 15 explanatorily illustrates a second exemplary user operation on the 3D strobe image.

[0026] FIG. 16 explanatorily illustrates a third exemplary user operation on the 3D strobe image.

[0027] FIG. 17 explanatorily illustrates a fourth exemplary user operation on the 3D strobe image.

[0028] FIG. 18 is a table indicating an exemplary shooting scenario.

[0029] FIG. 19 explanatorily illustrates exemplary processing jump.

[0030] FIG. 20 explanatorily illustrates exemplary reproduction-and-editing work for assignation of processing link.

[0031] FIG. 21 explanatorily illustrates exemplary reproduction-and-editing work for assignation of effect processing.

[0032] FIG. 22 is a block diagram of an exemplary configuration of an embodiment of a computer with the present technology applied.

MODE FOR CARRYING OUT THE INVENTION

Embodiment of Image Processing System with Present Technology Applied

[0033] FIG. 1 is a block diagram illustrating an exemplary configuration of an embodiment of an image processing system with the present technology applied.

[0034] In the image processing system of FIG. 1, with free viewpoint data generated from live images, generated and displayed is an image viewed when viewing an object as a subject in a three-dimensional space from a predetermined viewpoint, that is, a 2D image acquired by projecting a viewing volume within a three-dimensional space represented by the free viewpoint data onto a projection plane perpendicular in the direction of line-of-sight when viewing an object from a predetermined viewpoint. This 2D image can be generated with a freely determined viewpoint and thus can be called a free viewpoint image. A viewpoint (including the direction of line-of-sight) required for generating a free viewpoint image can be virtually set regardless of the actual viewpoint of the user, and thus such a viewpoint is also referred to as a virtual viewpoint.

[0035] Similarly to the moving image content of the 2D image, the moving image content of the free viewpoint data has a degree of freedom in the direction of time, and has a degree of freedom for a virtual viewpoint, that is, the shooting position and the shooting posture (the direction of shooting) of a virtual camera that shoots (a three-dimensional space represented by) the free viewpoint data. The shooting position of the virtual camera can be represented by coordinates such as the xyz coordinate system as the world coordinate system, and the shooting posture of the virtual camera can be represented by, for example, the rotation angle around each axis in the world coordinate system. The shooting position of the virtual camera can be moved in the x-axis, y-axis, and z-axis directions, and the shooting posture of the virtual camera can be rotated around the x-axis, y-axis, and z-axis. Thus, the shooting position and the shooting posture of the virtual camera has six degrees of freedom (6 DoF). Therefore, free viewpoint data has a total of seven degrees of freedom including one degree of freedom in the direction of time and six degrees of freedom in the shooting position and the shooting posture (of the virtual viewpoint) of the virtual camera. Here, as a free viewpoint image generated by projecting (a viewing volume within a three-dimensional space represented by) free viewpoint data onto a projection plane to a virtual viewpoint, acquired can be a 2D image as if a camera that is virtual shot the free viewpoint data from the virtual viewpoint. Thus, it can be said that projecting the free viewpoint data onto the projection plane to the virtual viewpoint corresponds to shooting the free viewpoint data from the virtual viewpoint with the camera that is virtual. That is, projecting the free viewpoint data onto the projection plane to the virtual viewpoint to generate the free viewpoint image can be compared to shooting the free viewpoint data with the camera that is virtual and disposed in the virtual viewpoint. Therefore, in the present embodiment, generation of a free viewpoint image will be described appropriately by comparing to shooting of free viewpoint data with a camera that is virtual. In addition, such a camera that is virtual and shoots free viewpoint data is also referred to as a virtual camera.

[0036] The image processing system of FIG. 1 includes a shooting device 11, a content server 12, and a reproduction-and-editing device 13.

[0037] The shooting device 11 includes at least a plurality of cameras, and shoots an object from a plurality of viewpoints. For example, the plurality of cameras included in the shooting device 11 is disposed surrounding an object, and each camera shoots the object from the viewpoint as the position where the camera is disposed. The two-dimensional (2D) images shot by the cameras from the positions of the cameras, that is, the (moving images) of the viewpoint images of the plurality of viewpoints as the 2D images shot from the plurality of viewpoints are supplied from the shooting device 11 to the content server 12, on a frame basis.

[0038] Here, the shooting device 11 may be provided with a plurality of distance measuring devices in addition to the plurality of cameras. The distance measuring devices can be disposed one-to-one at the same positions (viewpoints) as the cameras, or can be disposed at the positions different from those of the cameras. The distance measuring devices each measure the distance from the position (viewpoint) where the distance measuring device is disposed to the object, and generates a depth image as a 2D image having, as pixel values, a depth as information regarding the distance. The depth images are supplied from the shooting device 11 to the content server 12.

[0039] Note that in a case where the shooting device 11 is not provided with a distance measuring device, the shooting device 11 can measure the distance to the object by the principle of triangulation with the viewpoint images of two viewpoints among the viewpoint images of a plurality of viewpoints to generate a depth image.

[0040] The content server 12 is, for example, a web server or a server on the cloud. The content server 12 generates the moving image content of the free viewpoint data with the viewpoint images of a plurality of viewpoints and the like supplied from the shooting device 11, for example. The content server 12 further generates, with the free viewpoint data, a strobe model in which the 3D models of the same object (subject) between a plurality of frames (times) of a viewpoint image are disposed in (the 3D model) of the three-dimensional space shot by the shooting device 11. In response to a request from the reproduction-and-editing device 13, the content server 12 transmits the moving image content of the free viewpoint data and the strobe model to the reproduction-and-editing device 13.

[0041] The reproduction-and-editing device 13 is, for example, a client of a personal computer (PC), a television receiver (TV), or a mobile terminal such as a smartphone, or the like. The reproduction-and-editing device 13 requests the moving image content of the free viewpoint data and the strobe model of the content server 12 to acquire them from the content server 12. In addition, in response to a user operation on a 3D strobe image including the strobe model viewed from a virtual viewpoint, the reproduction-and-editing device 13 generates reproduction control information for control of reproduction of the moving image content of the free viewpoint data. Then, the reproduction-and-editing device 13 reproduces and edits the moving image content of the free viewpoint data in accordance with the reproduction control information.

[0042] Note that the content server 12 and the reproduction-and-editing device 13 each can be provided as a separate device as illustrated in FIG. 1, or the content server 12 and the reproduction-and-editing device 13 can be integrally provided as one device (for example, a PC).

[0043]

[0044] FIG. 2 is a block diagram illustrating an exemplary configuration of the content server 12 of FIG. 1.

[0045] The content server 12 includes a free-viewpoint-data generation unit 21, a strobe-model generation unit 22, a storage unit 23, and a communication unit 24.

[0046] The free-viewpoint-data generation unit 21 generates, on a frame basis, free viewpoint data from the viewpoint images of the plurality of viewpoints and the corresponding depth images from the shooting device 11 to generate the moving image content of the free viewpoint data.

[0047] Here, the free viewpoint data broadly means 3D image data (3D data) enabling generation of a free viewpoint image. As the free viewpoint data, for example, a set of the viewpoint images of the plurality of viewpoints and the corresponding depth images from the shooting device 11 can be adopted without any change. In addition, as the free viewpoint data, for example, 3D data having 3D shape models as information regarding a three-dimensional shape and color information, or a set of the 2D images of a plurality of viewpoints and the corresponding depth images can also be adopted.

[0048] In a case where the set of the viewpoint images of the plurality of viewpoints and the corresponding depth images from the shooting device 11 is adopted as free viewpoint data, the free-viewpoint-data generation unit 21 adopts, without any change, the set of the viewpoint images of the plurality of viewpoints and the corresponding depth images from the shooting device 11.

[0049] In a case where 3D data having 3D shape models and color information is adopted as free viewpoint data, the free-viewpoint-data generation unit 21 models by Visual Hull or the like with the viewpoint images of the plurality of viewpoints from the shooting device 11, generates, for example, the 3D shape model of the object showed in each viewpoint image, and adopts, as the viewpoint data, the 3D models together with the viewpoint images of the plurality of viewpoints as textures. Note that, in a case where the depth images from the shooting device 11 are different in viewpoint from the viewpoint images from the shooting device 11, the free-viewpoint-data generation unit 21 can generate, with the depth images of the plurality of viewpoints from the shooting device 11, depth images of the viewpoints of the viewpoint images from the shooting device 11.

[0050] In a case where the set of the 2D the images of the plurality of viewpoints and the corresponding depth images is adopted as free viewpoint data, the free-viewpoint-data generation unit 21 generates, for example, as described above, 3D data having 3D shape models and color information. Then, the free-viewpoint-data generation unit 21 generates, as the free viewpoint data, a set of 2D images and the corresponding depth images including the 3D data viewed from a plurality of viewpoints (may be the same as or different from viewpoints of the cameras included in the shooting device 11).

[0051] As described above, the free-viewpoint-data generation unit 21 generates the moving image content of the free viewpoint data and supplies the moving image content to the strobe-model generation unit 22 and the storage unit 23.

[0052] Note that as free viewpoint data, adoption of the set of the 2D images of the plurality of viewpoints and the corresponding depth images can reduce the amount of free viewpoint data as compared with adoption of 3D data having 3D shape models. A technique for generating and transmitting a set of the 2D images of a plurality of viewpoints and the corresponding depth images as free viewpoint data is described in International Publication No. 2017/082076 previously proposed by the applicant. A set of the 2D images of a plurality of viewpoints and the corresponding depth images as free viewpoint data can be encoded with an encoding scheme for 2D images, such as multiview and depth video coding (MVCD), advanced video coding (AVC), or high efficiency video coding (HEVC).

[0053] Here, the free viewpoint data (expression formats thereof) include a 3D model called view independent (hereinafter, also referred to as VI model) and a 3D model called view dependent (hereinafter, also referred to as VD model).

[0054] The VD model is, for example, 3D data in which information regarding a three-dimensional shape such as a 3D shape model and information regarding an image as a texture are separated. In the VD model, the 3D shape model is colored by mapping an image as a texture (texture mapping). The VD model can express the degree of reflection on the surface of an object that differs depending on the (virtual) viewpoint, for example. The VD model is called view dependent because its appearance depends on the viewpoint.

[0055] The VI model is, for example, 3D data in which polygons and points as components of information regarding a three-dimensional shape have color information. Examples of the VI model include a colored point cloud, and a set of 3D shape model and a UV map as color information of the 3D shape model. In the VI model, the colors of polygons and points are observed from any (virtual) viewpoint. The VI model is called view independent because its appearance is independent from the viewpoint.

[0056] The strobe-model generation unit 22 uses the free viewpoint data from the free-viewpoint-data generation unit 21 to generate a strobe model in which 3D models as the free viewpoint data of the same object between a plurality of frames (different times) of a viewpoint image are disposed in (the 3D model of) the three-dimensional space shown in the viewpoint image.

[0057] That is, the strobe-model generation unit 22 selects a frame for use in generation of a strobe model (hereinafter, also referred to as a generation frame) from frames in a strobe section including the frames for generating the strobe model. Furthermore, the strobe-model generation unit 22 sets at least one object of the objects shown in the generation frame, as a target object for disposing the corresponding 3D model in the strobe model (hereinafter, also referred to as the target object). Then, the strobe-model generation unit 22 generates the strobe model in which the 3D model of the target object shown in the generation frame is disposed, and supplies the strobe model to the storage unit 23.

[0058] Note that the strobe-model generation unit 22 can set, for example, all the moving objects shown in the generation frame, as target objects. In addition, the strobe-model generation unit 22 can set, for example, among the moving objects shown in the generation frame, an object specified by the user of the reproduction-and-editing device 13, as a target object.

[0059] The storage unit 23 stores the moving image content of the free viewpoint data from the free-viewpoint-data generation unit 21 and the strobe model from the strobe-model generation unit 22.

[0060] The communication unit 24 communicates with the reproduction-and-editing device 13. That is, for example, in response to a request from the reproduction-and-editing device 13, the communication unit 24 transmits the moving image content of the free viewpoint data and the strobe model stored in the storage unit 23 to the reproduction-and-editing device 13.

[0061] Note that the free-viewpoint-data generation unit 21 and the strobe-model generation unit 22 can be provided in the reproduction-and-editing device 13.

[0062]

[0063] FIG. 3 is a block diagram illustrating an exemplary configuration of the reproduction-and-editing device 13 of FIG. 1.

[0064] The reproduction-and-editing device 13 includes a communication unit 31, a storage unit 32, a free-viewpoint-image generation unit 33, a display unit 34, a virtual-viewpoint setting unit 35, an operation unit 36, a reproduction control unit 37, and a storage unit 38.

[0065] The communication unit 31 communicates with the content server 12. That is, in response to an operation on the operation unit 36, the communication unit 31 transmits a request for the moving image content of the free viewpoint data and the strobe model, for example. Furthermore, in response to the request, the communication unit 31 receives the moving image content of the free viewpoint data and the strobe model transmitted from the content server 12, and supplies the moving image content of the free viewpoint data and the strobe model to the storage unit 32.

[0066] The storage unit 32 stores the moving image content of the free viewpoint data and the strobe model from the communication unit 31.

[0067] The free-viewpoint-image generation unit 33 generates (renders), as (the data of) a free viewpoint image, for example, a 2D image (2D image acquired by shooting with the virtual camera located at the virtual viewpoint) (including a set of a 2D image for the left eye and a 2D image for the right eye) including (the three-dimensional space represented by) the free viewpoint data stored in the storage unit 32 viewed from the virtual viewpoint supplied from the virtual-viewpoint setting unit 35, and supplies (the data of) the free viewpoint image to the display unit 34.

[0068] The free-viewpoint-image generation unit 33 also generates, as a free viewpoint image, a 3D strobe image (3D computer graphics (CG)) as a 2D image including the strobe model stored in the storage unit 32 viewed from the virtual viewpoint from the virtual-viewpoint setting unit 35, and supplies the free viewpoint image to the display unit 34.

[0069] Here, the strobe image is an image showing at least one same object (image) shot at a plurality of times. A strobe image showing an object shown in each 2D image is also referred to as a 2D strobe image, and a 2D image showing 3D models of the object, that is, a 2D image including the strobe model viewed from a predetermined viewpoint is also referred to as a 3D strobe image. The free-viewpoint-image generation unit 33 generates such a 3D strobe image.

[0070] The display unit 34 displays the 3D strobe image, the other free viewpoint image, or the like from the free-viewpoint-image generation unit 33.

[0071] As the display unit 34, a 2D head-mounted display, a 2D monitor, a 3D head-mounted display, a 3D monitor, or the like can be provided, for example. A 3D head-mounted display or a 3D monitor is a display device that achieves stereoscopic vision by displaying, for example, a 2D image for the left eye and a 2D image for the right eye.

[0072] The virtual-viewpoint setting unit 35 sets a virtual viewpoint for shooting a strobe model by each virtual camera, and supplies the virtual viewpoint to the free-viewpoint-image generation unit 33. The virtual viewpoint can be set in response to a user operation, for example. The user operation includes the state of the user (position and posture) and gestures, in addition to an operation on the operation unit 36 by the user. For example, in a case where a head-mounted display is provided as the display unit 34, the state of the user can be detected by the head-mounted display.

[0073] The operation unit 36 is operated by the user and supplies operation information in accordance with the user operation to a necessary block. Note that the operation unit 36 can be integrally provided with the display unit 34, for example, as a touch panel or the like. Alternatively, the operation unit 36 can be provided as a mouse, a controller, or the like separate from the display unit 34.

[0074] In response to the user operation (on the operation unit 36 or the like) on the 3D strobe image displayed on the display unit 34, the reproduction control unit 37 generates reproduction control information for control of reproduction of the moving image content of the free viewpoint data stored in the storage unit 32. In addition, in accordance with the reproduction control information, the reproduction control unit 37 controls the reproduction of the moving image content of the free viewpoint data stored in the storage unit 32. The control of the reproduction of the moving image content of the free viewpoint data includes control of the position (frame) at which the reproduction starts, control (setting) of a virtual viewpoint for generating a free viewpoint image by the free-viewpoint-image generation unit 33, and the like.

[0075] Examples of the reproduction control information include a timeline scenario and a shooting scenario. The timeline scenario is information indicating a reproduction procedure of reproducing the moving image content of the free viewpoint data. The shooting scenario is information indicating a virtual viewpoint for viewing of the free viewpoint data, that is, information indicating the shooting position and the shooting posture (furthermore, camera parameters such as a zoom ratio) of the virtual camera that shoots the free viewpoint data.

[0076] The storage unit 38 stores the file of the timeline scenario and the shooting scenario as the reproduction control information generated by the reproduction control unit 37.

[0077] Note that it is sufficient if the reproduction-and-editing device 13 has at least a function of communicating with the outside, a function of displaying an image, and a function of accepting a user operation. In FIG. 3, other functions such as the functions achieved by the free-viewpoint-image generation unit 33, the virtual-viewpoint setting unit 35, and the reproduction control unit 37 can be provided in, for example, a server on the cloud.

[0078]

[0079] FIG. 4 is a flowchart explanatorily illustrating exemplary processing performed by the content server 12 of FIG. 2.

[0080] In step S11, the content server 12 acquires, on a frame basis, the viewpoint images of a plurality of viewpoints (and the corresponding depth images) shot by the shooting device 11. The content server 12 supplies the viewpoint images of the plurality of viewpoints to the free-viewpoint-data generation unit 21, and the processing proceeds from step S11 to step S12. Note that the viewpoint images of the plurality of viewpoints shot by the shooting device 11 can be transmitted from the reproduction-and-editing device 13 to the content server 12 through the reproduction-and-editing device 13.

[0081] In step S12, the free-viewpoint-data generation unit 21 generates free viewpoint data on a frame basis, with, for example, the viewpoint images of the plurality of viewpoints, and supplies the free viewpoint data to the strobe-model generation unit 22 and the storage unit 23. The storage unit 23 stores moving image content including the free viewpoint data on the frame basis (moving image content of the free viewpoint data) from the free-viewpoint-data generation unit 21. Thereafter, the processing proceeds from step S12 to step S13.

[0082] In step S13, the strobe-model generation unit 22 generates a strobe model, with the free viewpoint data from the free-viewpoint-data generation unit 21.

[0083] That is, the strobe-model generation unit 22 sets a strobe section including frames for generating a strobe model. For example, all sections of the moving image content of the free viewpoint data, a section from a scene change to the next scene change, a section specified by the reproduction-and-editing device 13 in response to a user operation, or the like are set as the strobe section.

[0084] The strobe-model generation unit 22 also sets an object shown in each viewpoint image as a target object for disposing the 3D model in the strobe model. For example, all the objects shown in the frames in the strobe section are each set as a target object, or an object specified by the reproduction-and-editing device 13 in response to a user operation or the like is set as a target object.

[0085] Furthermore, the strobe-model generation unit 22 selects, from the frames in the strobe section, a frame for use in generation of a strobe model (hereinafter, also referred to as a generation frame).

[0086] Here, if all the frames in the strobe section are used as generation frames to generate a strobe model, the 3D models of the same objects same in number as the frames in the strobe section are superimposed in the strobe model. Thus, the 3D strobe image is difficult to see in some cases.

[0087] Therefore, the strobe-model generation unit 22 can select some frames as generation frames from the frames in the strobe section and can generate a strobe model with (the 3D models of the objects shown in) the generation frames.

[0088] The strobe-model generation unit 22 can select a frame having a level of interference in 3D model not more than a threshold, from the frames in the strobe section, as a generation frame, for example. That is, the strobe-model generation unit 22 calculates the level of interference indicating the level of superimposition between 3D models with the 3D models of the target object shown in the frames in the strobe section disposed in a three-dimensional space. The level of interference is calculated, defined as 100% for the case where the 3D models of any two frames are completely superimposed in a three-dimensional space and 0% for the case where the 3D models of any two frames are not superimposed at all in a three-dimensional space, for example. Then, the strobe-model generation unit 22 selects, as generation frames, frames each having a level of interference not more than a predetermined threshold. As described above, frames each having a level of interference in 3D model not more than a threshold are selected as generation frames from frames in a strobe section, and a strobe model in which the 3D models of the target object shown in the generation frames are disposed is generated. As a result, it can be inhibited that the 3D strobe image is difficult to see due to superimposition of the 3D models in the strobe model.

[0089] Note that in addition to the above, in generation-frame selection, for example, the frames in a strobe section can be simply selected as generation frames for each predetermined number of frames. Alternatively, in generation-frame selection, all frames in a strobe section can be selected as generation frames.

[0090] The strobe-model generation unit 22 uses the free viewpoint data from the free-viewpoint-data generation unit 21, and generates a strobe model in which the 3D models of the target object shown in the plurality of generation frames selected from the frames in the strobe section are disposed on (the 3D model of) the background as the three-dimensional space at the time of shooting of the target object.

[0091] The strobe-model generation unit 22 supplies, to the storage unit 23, the strobe model of the moving image content of the free viewpoint data from the free-viewpoint-data generation unit 21, that is, the strobe model generated with the free viewpoint data from the free-viewpoint-data generation unit 21. The storage unit 23 stores the strobe model from the strobe-model generation unit 22, and the processing proceeds from step S13 to step S14.

[0092] In step S14, for example, in response to a request from the reproduction-and-editing device 13, the communication unit 24 transmits the moving image content of the free viewpoint data and the strobe model stored in the storage unit 23, to the reproduction-and-editing device 13, and then the processing ends.

[0093]

……
……
……