Sony Patent | Video distribution system, video distribution method, and display terminal

Patent: Video distribution system, video distribution method, and display terminal

Drawings: Click to check drawins

Publication Number: 20220239888

Publication Date: 20220728

Applicant: Sony

Assignee: Sony Group Corporation

Abstract

There is provided a video distribution system, a video distribution method, and a display terminal that enable more appropriate display of a video. The video distribution system includes an image acquisition unit that acquires a first image and a second image of a subject captured by a first camera and a second camera, a parameter adjustment unit that adjusts a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired, and a display control unit that displays a video representing the virtual space including the virtual subject corresponding to the adjusted parameter on a display terminal. The present technology can be applied to, for example, a system that distributes a stereoscopic video.

Claims

1. A video distribution system comprising: an image acquisition unit that acquires a first image and a second image of a subject captured by a first camera and a second camera; a parameter adjustment unit that adjusts a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired; and a display control unit that displays a video representing the virtual space including the virtual subject corresponding to the adjusted parameter on a display terminal.

2. The video distribution system according to claim 1, wherein the parameter includes a parameter related to at least one of a first distance between the first camera and the second camera, a second distance between pupils of the user, a distance to the virtual subject, or a size of the virtual subject.

3. The video distribution system according to claim 2, wherein the parameter includes a parameter correlated with a relationship between the first distance and the second distance.

4. The video distribution system according to claim 3, wherein in a case where the first distance and the second distance are different, the parameter adjustment unit adjusts the parameter in such a manner that the virtual subject corresponding to a state where the first distance and the second distance coincide is displayed.

5. The video distribution system according to claim 4, wherein the parameter adjustment unit adjusts the parameter in such a manner that a viewing position of the user is shifted from a center of a spherical surface on which a video is projected.

6. The video distribution system according to claim 5, wherein the parameter adjustment unit brings a position of a virtual camera corresponding to the viewing position of the user close to a projection surface of the spherical surface or away from the projection surface.

7. The video distribution system according to claim 4, wherein the parameter adjustment unit adjusts the parameter in such a manner that, in a state where a viewing position of the user and a position of a center of a spherical surface on which a video is projected coincide, an angle of the video projected on the spherical surface changes.

8. The video distribution system according to claim 7, wherein the parameter adjustment unit rotates the video projected on the spherical surface outward or inward.

9. The video distribution system according to claim 4, wherein the parameter adjustment unit adjusts the parameter in such a manner that a center of a spherical surface on which a video is projected is shifted from a viewing position of the user.

10. The video distribution system according to claim 9, wherein the parameter adjustment unit moves a position of the center of the spherical surface outward or inward with respect to a position of a virtual camera corresponding to the viewing position of the user.

11. The video distribution system according to claim 4, wherein in adjusting the parameter, the parameter adjustment unit performs one method alone or a combination of at least two methods of a first method of shifting a viewing position of the user from a center of a spherical surface on which a video is projected, a second method of changing an angle of the video projected on the spherical surface in a state where the viewing position of the user and the center of the spherical surface coincide, or a third method of shifting the center of the spherical surface from the viewing position of the user.

12. The video distribution system according to claim 11, wherein the parameter adjustment unit shifts the viewing position of the user by bringing a position of a virtual camera corresponding to the viewing position of the user close to a projection surface of the spherical surface or away from the projection surface in a case where the first method is performed, changes an angle of the video projected on the spherical surface by rotating the video projected on the spherical surface outward or inward in a case where the second method is performed, and shifts the center of the spherical surface by moving the position of the center of the spherical surface outward or inward with respect to the position of the virtual camera in a case where the third method is performed.

13. The video distribution system according to claim 1, wherein the first camera is installed at a position on a left side with respect to the subject when the subject is viewed from a front, and the second camera is installed at a position on a right side with respect to the subject when the subject is viewed from the front.

14. The video distribution system according to claim 13, wherein a video representing the virtual space including the virtual subject is displayed by projecting a first video corresponding to the first image captured by the first camera on a first spherical surface centered on a position of a first virtual camera corresponding to a left eye of the user in the virtual space, and projecting a second video corresponding to the second image captured by the second camera on a second spherical surface centered on a position of a second virtual camera corresponding to a right eye of the user in the virtual space.

15. The video distribution system according to claim 14, wherein the first spherical surface and the second spherical surface include a spherical surface corresponding to an entire celestial sphere or a half celestial sphere.

16. The video distribution system according to claim 3, wherein the parameter adjustment unit adjusts the parameter in such a manner that the virtual subject corresponding to a state where the first distance and the second distance are different is displayed in a case where the first distance and the second distance coincide or are different from each other.

17. The video distribution system according to claim 1, wherein when there is a change in the subject as an image-capturing target, the parameter adjustment unit dynamically adjusts the parameter according to an amount of the change.

18. The video distribution system according to claim 1, wherein the display terminal includes a head mounted display.

19. A video distribution method comprising, by a video distribution system: acquiring a first image and a second image of a subject captured by a first camera and a second camera; adjusting a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired; and displaying a video representing the virtual space including the virtual subject corresponding to the adjusted parameter on a display terminal.

20. A display terminal comprising: a display control unit that displays, on a display terminal, a video representing a virtual space including a virtual subject whose parameter is adjusted, the parameter affecting an appearance to a user regarding the virtual subject corresponding to a subject in the virtual space represented by a first image and a second image of the subject captured by a first camera and a second camera.

Description

TECHNICAL FIELD

[0001] The present technology relates to a video distribution system, a video distribution method, and a display terminal, and particularly relates to a video distribution system, a video distribution method, and a display terminal capable of more appropriately displaying a video.

BACKGROUND ART

[0002] In recent years, for example, devices such as head mounted displays have been widely used as display terminals for viewing stereoscopic videos.

[0003] In this type of display terminal, a stereoscopic video is displayed on the basis of video information obtained by image-capturing a subject with a plurality of cameras, and an immersive image is provided to a user wearing the display terminal on the head.

[0004] Furthermore, as a technique for displaying a stereoscopic video, techniques disclosed in Patent Documents 1 and 2 are known.

CITATION LIST

Patent Document

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2003-284093 [0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2014-209768

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0007] Incidentally, when displaying the stereoscopic video on the display terminal, it is desirable to appropriately display a video required by the user who uses the display terminal.

[0008] The present technology has been made in view of such a situation, and is intended to more appropriately display a video.

Solutions to Problems

[0009] A video distribution system according to one aspect of the present technology is a video distribution system including an image acquisition unit that acquires a first image and a second image of a subject captured by a first camera and a second camera, a parameter adjustment unit that adjusts a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired, and a display control unit that displays a video representing the virtual space including the virtual subject corresponding to the adjusted parameter on a display terminal.

[0010] A video distribution method according to one aspect of the present technology is a video distribution method including, by a video distribution system, acquiring a first image and a second image of a subject captured by a first camera and a second camera, adjusting a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired, and displaying a video representing the virtual space including the virtual subject corresponding to the adjusted parameter on a display terminal.

[0011] In the video distribution system and the video distribution method according to one aspect of the present technology, a first image and a second image of a subject captured by a first camera and a second camera is acquired, a parameter that affects an appearance to a user regarding a virtual subject corresponding to the subject in a virtual space represented by the first image and the second image that have been acquired is adjusted, and a video representing the virtual space including the virtual subject corresponding to the adjusted parameter is displayed on a display terminal.

[0012] A display terminal according to one aspect of the present technology is a display terminal including a display control unit that displays, on a display terminal, a video representing a virtual space including a virtual subject whose parameter is adjusted, the parameter affecting an appearance to a user regarding the virtual subject corresponding to a subject in the virtual space represented by a first image and a second image of the subject captured by a first camera and a second camera.

[0013] In the display terminal according to one aspect of the present technology, a video is displayed on a display terminal, the video representing a virtual space including a virtual subject whose parameter is adjusted, the parameter affecting an appearance to a user regarding the virtual subject corresponding to a subject in the virtual space represented by a first image and a second image of the subject captured by a first camera and a second camera.

BRIEF DESCRIPTION OF DRAWINGS

[0014] FIG. 1 is a diagram illustrating an example of a configuration of an embodiment of a video distribution system.

[0015] FIG. 2 is a diagram illustrating an example of a configuration of a workstation.

[0016] FIG. 3 is a diagram illustrating an example of a configuration of a display terminal.

[0017] FIG. 4 is a diagram schematically illustrating a state where a user views a stereoscopic video.

[0018] FIG. 5 is a diagram schematically illustrating a state where a subject is image-captured by two cameras.

[0019] FIG. 6 is a diagram illustrating a camera inter-optical axis distance in a case where a subject is image-captured by two cameras.

[0020] FIG. 7 is a diagram illustrating a user’s interpupillary distance in a case where the user views a stereoscopic video.

[0021] FIG. 8 is a diagram illustrating an example of a functional configuration of the video distribution system to which the present technology is applied.

[0022] FIG. 9 is a flowchart illustrating an overall processing flow of the video distribution system to which the present technology is applied.

[0023] FIG. 10 is a diagram schematically illustrating a state where a user views a stereoscopic video in a case where a relationship of IPD_CAM=IPD_USER occurs.

[0024] FIG. 11 is a diagram illustrating in detail a state where the user views the stereoscopic video in a case where the relationship of IPD_CAM=IPD_USER occurs.

[0025] FIG. 12 is a diagram schematically illustrating a state where the user views the stereoscopic video in a case where a relationship of IPD_CAM>IPD_USER occurs.

[0026] FIG. 13 is a diagram illustrating in detail a state where the user views the stereoscopic video in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0027] FIG. 14 is a diagram illustrating in detail a state where the user views the stereoscopic video when IPD_CAM>IPD_USER in a case where a virtual subject is right in front.

[0028] FIG. 15 is a diagram illustrating in detail a state where the user views the stereoscopic video when IPD_CAM>IPD_USER in a case where the virtual subject is on a right front side.

[0029] FIG. 16 is a diagram illustrating a first example of a state where a first method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0030] FIG. 17 is a diagram illustrating a second example of a state where the first method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0031] FIG. 18 is a diagram illustrating a third example of a state where the first method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0032] FIG. 19 is a diagram illustrating a fourth example of a state where the first method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0033] FIG. 20 is a diagram schematically illustrating a distance to the virtual subject in a virtual space.

[0034] FIG. 21 is a diagram illustrating a state after conversion of the distance to the virtual subject in the virtual space.

[0035] FIG. 22 is a diagram illustrating a first example of a state where a second method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0036] FIG. 23 is a diagram illustrating a second example of a state where the second method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0037] FIG. 24 is a diagram illustrating a third example of a state where the second method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0038] FIG. 25 is a diagram illustrating a state where videos to be attached to entire celestial spheres are rotated outward when IPD_CAM>IPD_USER in a case where the virtual subject is right in front.

[0039] FIG. 26 is a diagram illustrating a state where the videos to be attached to the entire celestial spheres are rotated inward when IPD_CAM>IPD_USER in a case where the virtual subject is right in front.

[0040] FIG. 27 is a diagram illustrating a first example of a state where a third method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0041] FIG. 28 is a diagram illustrating a second example of a state where the third method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0042] FIG. 29 is a diagram illustrating a third example of a state where the third method is applied in a case where the relationship of IPD_CAM>IPD_USER occurs.

[0043] FIG. 30 is a diagram illustrating a state where the entire celestial spheres to which the videos are attached are moved outward when IPD_CAM>IPD_USER in a case where the virtual subject is right in front.

[0044] FIG. 31 is a diagram illustrating a state where the entire celestial spheres to which the videos are attached are moved inward when IPD_CAM>IPD_USER in a case where the virtual subject is right in front.

[0045] FIG. 32 is a diagram illustrating an example when an appearance of a video is changed in time series.

[0046] FIG. 33 is a diagram illustrating a configuration example of a computer.

MODE FOR CARRYING OUT THE INVENTION

[0047] Hereinafter, embodiments of the present technology will be described with reference to the drawings. Note that the description will be made in the following order.

[0048] 1. Embodiments of present technology

[0049] 2. Modification example

[0050] 3. Configuration of computer

[0051] <1. Embodiments of Present Technology>

[0052] (Configuration of Video Distribution System)

[0053] FIG. 1 illustrates an example of a configuration of a video distribution system.

[0054] In FIG. 1, a video distribution system 1 includes a workstation 10, a camera 11-R, a camera 11-L, a video distribution server 12, and display terminals 20-1 to 20-N (N: an integer of 1 or more). Furthermore, in the video distribution system 1, the workstation 10, the video distribution server 12, and the display terminals 20-1 to 20-N are connected to the Internet 30.

[0055] The workstation 10 is an image processing device specialized in image processing. The workstation 10 performs image processing on a plurality of images captured by the cameras 11-L and 11-R, and transmits data obtained by the image processing to the video distribution server 12 via the Internet 30.

[0056] The camera 11-L and the camera 11-R are configured as stereo cameras, and for example, when a subject is viewed from a front, the camera 11-L is installed at a position on the left side with respect to the subject, and the camera 11-R is installed at a position on the right side with respect to the subject.

[0057] The camera 11-L includes, for example, an image sensor such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, and a signal processing unit such as a camera image signal processor (ISP). The camera 11-L transmits data of a captured image (hereinafter, also referred to as a left image) to the workstation 10.

[0058] Similarly to the camera 11-L, the camera 11-R includes an image sensor and a signal processing unit, and transmits data of a captured image (hereinafter, also referred to as a right image) to the workstation 10.

[0059] Note that the camera 11-L and the camera 11-R may be connected to the workstation 10 via a communication line such as a dedicated line (cable), for example, or may be connected by wired communication or wireless communication conforming to a predetermined standard. Furthermore, in the following description, the camera 11-L and the camera 11-R are simply referred to as the camera 11 in a case where it is not particularly necessary to distinguish them.

[0060] The video distribution server 12 is, for example, a web server installed in a data center or the like. The video distribution server 12 receives data transmitted from the workstation 10. In a case where video distribution is requested from any of the display terminals 20-1 to 20-N, the video distribution server 12 transmits a video stream including data from the workstation 10 to the display terminal 20 that is a request source of the video distribution via the Internet 30.

[0061] The display terminal 20-1 is configured as a head mounted display that is worn on the head so as to cover both eyes of the user and allows viewing a moving image or a still image displayed on a display screen provided in front of the eyes of the user. Note that the display terminal 20-1 is not limited to a head mounted display, and may be an electronic device having a display such as a smartphone, a tablet terminal, or a game machine.

[0062] The display terminal 20-1 transmits a request for video distribution to the video distribution server 12 via the Internet 30, for example, according to an operation of the user. The display terminal 20-1 receives and processes a video stream transmitted from the video distribution server 12 via the Internet 30, and reproduces a video. The video includes a moving image such as a virtual reality (VR) moving image distributed (real-time distribution (live distribution) or on-demand distribution) from the video distribution server 12, and content such as a still image.

[0063] Similarly to the display terminal 20-1, the display terminals 20-2 to 20-N include, for example, a head mounted display and the like, and each reproduce videos (for example, moving images, still images, and the like) distributed as video streams from the video distribution server 12. Note that, in the following description, the display terminals 20-1 to 20-N are simply referred to as the display terminal 20 in a case where it is not particularly necessary to distinguish them.

[0064] (Configuration of Workstation)

[0065] FIG. 2 illustrates an example of a configuration of the workstation 10 of FIG. 1.

[0066] In FIG. 2, the workstation 10 includes a processing unit 100, an input unit 101, an output unit 102, a storage unit 103, and a communication unit 104.

[0067] The processing unit 100 includes a processor such as a central processing unit (CPU), a graphic card (video card), and the like. The processing unit 100 is a main processing device that controls operation of each unit and performs various types of arithmetic processing.

[0068] The input unit 101 includes a keyboard, a mouse, physical buttons, and the like. The input unit 101 supplies an operation signal corresponding to an operation of the user to the processing unit 100.

[0069] The output unit 102 includes a display, a speaker, and the like. The output unit 102 outputs video, audio, and the like under control of the processing unit 100.

[0070] The storage unit 103 includes a semiconductor memory including a nonvolatile memory or a volatile memory, a buffer memory, and the like. The storage unit 103 stores various data under the control of the processing unit 100.

[0071] The communication unit 104 includes a communication module compatible with wireless communication or wired communication conforming to the predetermined standard, a video or audio capture card, and the like.

[0072] The communication unit 104 exchanges various data with the video distribution server 12 via the Internet 30 under the control of the processing unit 100. Furthermore, the communication unit 104 receives data from the camera 11-L and the camera 11-R under the control of the processing unit 100.

[0073] Furthermore, the processing unit 100 includes an image acquisition unit 111, an image processing unit 112, and a transmission control unit 113.

[0074] The image acquisition unit 111 acquires (captures) respective image signals of the left image captured by the camera 11-L and the right image captured by the camera 11-R via the communication unit 104, and stores the image signals in the storage unit 103.

[0075] The image processing unit 112 reads image signals of the left image and the right image stored in the storage unit 103, performs predetermined image processing, and supplies data obtained as a result of the image processing to the transmission control unit 113. Note that although details will be described later with reference to FIG. 8 and the like, this image processing includes processing such as conversion processing for video information including image signals of the left image and the right image.

[0076] The transmission control unit 113 controls the communication unit 104 to transmit the data from the image processing unit 112 to the video distribution server 12 via the Internet 30.

[0077] (Configuration of Display Terminal)

[0078] FIG. 3 illustrates an example of a configuration of the display terminal 20 in FIG. 1.

[0079] In FIG. 3, the display terminal 20 includes a processing unit 200, a sensor unit 201, a storage unit 202, a display unit 203, an audio output unit 204, an input terminal 205, an output terminal 206, and a communication unit 207.

[0080] The processing unit 200 includes a CPU and the like. The processing unit 200 is a main processing device that controls the operation of each unit and performs various types of arithmetic processing. Note that, here, a dedicated processor such as a graphics processing unit (GPU) may be provided.

[0081] The sensor unit 201 includes various sensor devices and the like. The sensor unit 201 performs sensing of the user, the surroundings thereof, and the like, and supplies sensor data corresponding to sensing results to the processing unit 200.

[0082] Here, the sensor unit 201 can include a magnetic sensor that detects the magnitude and direction of a magnetic field, an acceleration sensor that detects acceleration, a gyro sensor that detects an angle (posture), an angular velocity, and an angular acceleration, a proximity sensor that detects a nearby object, and the like. Furthermore, a camera having an image sensor may be provided as the sensor unit 201, and an image signal obtained by image-capturing a subject may be supplied to the processing unit 200.

[0083] The storage unit 202 includes a semiconductor memory or the like including a nonvolatile memory or a volatile memory. The storage unit 202 stores various data under the control of the processing unit 200.

[0084] The display unit 203 includes a display device (display apparatus) such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display. The display unit 203 displays a video (a moving image, a still image, or the like) corresponding to the video data supplied from the processing unit 200.

[0085] The audio output unit 204 includes an audio output device such as a speaker. The audio output unit 204 outputs audio (sound) corresponding to audio data supplied from the processing unit 200.

[0086] The input terminal 205 includes an input interface circuit and the like, and is connected to an electronic device via a predetermined cable. The input terminal 205 supplies, for example, an image signal, an audio signal, a command, and the like input from a device such as a game machine (dedicated console), a personal computer, or a reproduction machine to the processing unit 200.

[0087] The output terminal 206 includes an output interface circuit and the like, and is connected to an electronic device via a predetermined cable. The output terminal 206 outputs an audio signal supplied thereto to a device such as an earphone or a headphone via a cable.

[0088] The communication unit 207 is configured as a communication module compatible with wireless communication such as wireless local area network (LAN), cellular communication (for example, LTE-Advanced, 5G, or the like), or Bluetooth (registered trademark), or wired communication.

[0089] The communication unit 207 exchanges various data with the video distribution server 12 via the Internet 30 under the control of the processing unit 200. Furthermore, the communication unit 207 can communicate with an external device including a game machine (dedicated console), a personal computer, a server, a reproduction machine, a dedicated controller, a remote controller, and the like.

[0090] Furthermore, the processing unit 200 includes an image acquisition unit 211, an image processing unit 212, and a display control unit 213.

[0091] The image acquisition unit 211 acquires data included in the video stream distributed from the video distribution server 12, and stores the data in the storage unit 202.

[0092] The image processing unit 212 reads data stored in the storage unit 202, performs predetermined image processing, and supplies data obtained as a result of the image processing to the display control unit 213. Note that this image processing can include processing such as conversion processing for video information in addition to processing such as decoding.

[0093] The display control unit 213 displays a video such as a moving image or a still image on the display unit 203 on the basis of the data from the image processing unit 212.

[0094] The video distribution system 1 is configured as described above.

[0095] (Conventional Problem)

[0096] Next, problems of the prior art will be described with reference to FIGS. 4 to 7.

[0097] In the video distribution system 1, in order to view a stereoscopic video, a subject is image-captured by the cameras 11-L and 11-R configured as stereo cameras, and video is displayed on the immersive display terminal 20 using video information including a left image and a right image obtained by the image-capturing.

[0098] Here, in the conventional non-immersive display terminal (for example, a display apparatus such as a television receiver), regarding perception of the size of the subject, in addition to the size of the subject displayed on the display terminal and the optical size obtained from the distance between the viewing user and the display terminal, an image-captured environment, zoom level, and the like are flexibly adjusted by each individual in consideration of each experience.

[0099] This is based on the recognition that the display surface of the display terminal and the environment to which the user belongs are not continuous and different, and even if the optical size (viewing angle) of the subject changes due to the display terminal, the distance to the display terminal, and other conditions, this does not directly affect the perception of the size of the subject.

[0100] On the other hand, in the immersive display terminal 20, since the display surface and the environment to which the user belongs are felt to be continuous, when the optical size (viewing angle) changes, it is evaluated that the size of the subject itself has changed.

[0101] In the present technology, an expression as illustrated in FIG. 4 is used to conceptually indicate the viewing angle described above. That is, FIG. 4 schematically illustrates a state where a user 50 views the stereoscopic video using the immersive display terminal 20 when seen from above.

[0102] Furthermore, FIG. 5 schematically illustrates a state where a subject 60 is image-captured by the two cameras 11-L and 11-R when seen from above.

[0103] Here, in a case where the user 50 views a stereoscopic image using the display terminal 20 such as a head mounted display, it is common that the user 50 views videos (videos corresponding to a left image and a right image) respectively captured by the camera 11-L and the camera 11-R, such as a video 500-L for the left eye and a video 500-R for the right eye.

[0104] That is, when the subject 60 is viewed from the front, the video 500-L corresponds to the left image captured by the camera 11-L installed at the position on the left side of the image-capturing environment, and the video 500-R corresponds to the right image captured by the camera 11-R installed at the position on the right side of the image-capturing environment.

[0105] Here, a drawing range 501-L in FIG. 4 indicates a drawing range of the subject 60 with respect to the left eye, and corresponds to an imaging range 511-L of the subject 60 captured by the camera 11-L in FIG. 5. Furthermore, a drawing range 501-R in FIG. 4 indicates a drawing range of the subject 60 with respect to the right eye, and corresponds to an imaging range 511-R of the subject 60 captured by the camera 11-R in FIG. 5.

[0106] That is, in a case where the user 50 views the subject 60 (that is, a virtual subject) displayed as the stereoscopic video using the immersive display terminal 20, the user views the subject within the range including the drawing range 501-L from the left eye and the drawing range 501-R from the right eye.

[0107] At this time, in FIG. 4, a point at which a straight line A connecting a right end of the drawing range 501-L and the center of the left eye of the user 50 intersects a straight line B connecting a right end of the drawing range 501-R and the center of the right eye of the user 50 is defined as an intersection X. Furthermore, in FIG. 4, a point at which a straight line C connecting a left end of the drawing range 501-L and the center of the left eye of the user 50 intersects a straight line D connecting a left end of the drawing range 501-R and the center of the right eye of the user 50 is defined as an intersection Y.

[0108] Here, since the intersection X and the intersection Y are points on a straight line connecting the left and right eyes of the user 50 and ends of a portion where (the video of) the virtual subject is projected on projection surface, the intersections X and Y can be regarded as left and right ends of the virtual subject when stereoscopic viewing is performed. Thus, the size of the virtual subject (virtual object) perceived by the user 50 in the virtual space can be expressed as a viewing angle 502.

[0109] FIG. 6 illustrates a distance between the optical axis of the optical system of the camera 11-L and the optical axis of the optical system of the camera 11-R (hereinafter, will be referred to as “camera inter-optical axis distance IPD_CAM”) in a case where the subject 60 is image-captured by the two cameras 11-L and 11-R.

[0110] In FIG. 6, the subject 60 is image-captured by the camera 11-L and the camera 11-R installed at an interval corresponding to the camera inter-optical axis distance IPD_CAM. At this time, there is a case where the camera inter-optical axis distance IPD_CAM cannot be freely determined due to, for example, the sizes of the camera 11 and the lens, other physical limitations, restrictions on the image-capturing environment, and the like.

[0111] FIG. 7 illustrates a distance (hereinafter, referred to as a user’s interpupillary distance IPD_USER) between pupils of left and right eyes of the user 50 in a case where the user 50 wearing the display terminal 20 such as a head mounted display views a stereoscopic video.

[0112] Here, in order to perform stereoscopic viewing, it is necessary to arrange the video 500-L and a video 500-R corresponding to the left image and the right image respectively captured by the camera 11-L and the camera 11-R on the virtual space in accordance with the user’s interpupillary distance IPD_USER.

[0113] In a normal implementation, the video 500-L and the video 500-R corresponding to the captured left image and right image are projected (attached) on an entire celestial sphere for the left eye and an entire celestial sphere for the right eye, respectively, and virtual cameras (virtual cameras corresponding to positions of the left eye and the right eye of the user) are installed at centers of the respective entire celestial spheres, so that the user 50 can view (observe) the videos from the centers of the respective entire celestial spheres at the viewing position.

[0114] Note that, in the normal implementation, in a case where the user 50 wearing the display terminal 20 moves the head back and forth, left and right, and up and down, the entire celestial sphere is implemented to accompany the movement in a similar manner, and thus an appearance of the stereoscopic video from the user 50 does not change.

[0115] Furthermore, in a case where the user 50 rotates the head in the yaw direction or the roll direction (rotation other than vertical rotation, that is, rotation in which the positions of the eyes of the user 50 are shifted from the centers of the entire celestial spheres), parallax deviation occurs, and thus the user 50 cannot correctly view the stereoscopic video. However, as long as the user 50 does not move the eye positions, that is, only moves the eyeballs, the stereoscopic video can be viewed correctly.

[0116] At this time, if the user’s interpupillary distance IPD_USER and the camera inter-optical axis distance IPD_CAM coincide, the display terminal 20 can reproduce the environment at the time of image-capturing including the appearance to the user such as a sense of size (size) and a sense of distance of the virtual subject.

[0117] However, due to restrictions on the sizes of a lens and a camera body in the camera 11, the value of the camera inter-optical axis distance IPD_CAM cannot be made equal to or less than a certain value, and a relationship of IPD_CAM>IPD_USER is inevitable in some cases.

[0118] Note that, in recent years, since downsizing of cameras has progressed, it is possible to select a system in which the value of the camera inter-optical axis distance IPD_CAM can be set to be small, but there are various demands for image-capturing environment, video quality, and usability, and such a system cannot be necessarily selected in all cases.

[0119] Furthermore, conversely, it is also assumed that the camera needs to have a certain size or less depending on the environment in which the subject 60 is image-captured, and in this case, the relationship of IPD_CAM

[0120] If it is assumed to correspond to various image-capturing targets and image-capturing environments in this manner, it is practically difficult to always make the user’s interpupillary distance IPD_USER and the camera inter-optical axis distance IPD_CAM coincide.

[0121] Furthermore, as illustrated in FIG. 7, since the user’s interpupillary distance IPD_USER is generally different for each user, it is difficult to uniquely determine the optimum user’s interpupillary distance IPD_USER to be set at the time of image-capturing. Thus, in order to unify the appearance between individual users, it is necessary to finally perform some adjustment regardless of the image-capturing environment.

[0122] Accordingly, the present technology enables to more appropriately display a video by adjusting a difference in appearance of the stereoscopic video caused due to the difficulty in making the user’s interpupillary distance IPD_USER and the camera inter-optical axis distance IPD_CAM coincide and the presence of variation in the user’s interpupillary distance IPD_USER.

[0123] Note that in the following description, an example of adjusting a parameter correlated with the relationship between the camera inter-optical axis distance IPD_CAM and the user’s interpupillary distance IPD_USER will be mainly described, and the parameter is an example of a parameter that affects the appearance to the user such as a sense of size and a sense of distance of the virtual subject.

[0124] (Functional Configuration of Video Distribution System)

[0125] FIG. 8 illustrates an example of a functional configuration of the video distribution system 1 of FIG. 1.

[0126] In FIG. 8, the video distribution system 1 includes the camera 11 including an imaging unit 120 and an inter-optical axis distance detection unit 130, the display terminal 20 including a reproduction unit 220 and an interpupillary distance detection unit 230, and a conversion processing unit 300.

[0127] The conversion processing unit 300 is included in (the processing unit 100 of) the workstation 10 or (the processing unit 200 of) the display terminal 20, for example. However, the conversion processing unit 300 is not limited to the workstation 10 and the display terminal 20, and may be included in another device such as the camera 11.

[0128] Note that, in the configuration of FIG. 8, only one camera 11 is illustrated for simplification of description, but in practice, two cameras 11-L and 11-R configured as stereo cameras are installed for a subject.

[0129] In the camera 11, the imaging unit 120 image-captures the subject and outputs (transmits) video information obtained by the image-capturing to the conversion processing unit 300.

[0130] Furthermore, the inter-optical axis distance detection unit 130 detects the camera inter-optical axis distance IPD_CAM and outputs a detection result thereof as inter-optical axis distance information.

[0131] Here, the camera inter-optical axis distance IPD_CAM can be detected using a sensor or the like, or can be manually measured or given as a fixed value.

[0132] Thus, the inter-optical axis distance detection unit 130 is not necessarily included in the camera 11, but the camera inter-optical axis distance IPD_CAM is uniquely determined by the installation position of the camera 11-L and the installation position of the camera 11-R, and even in a case where the inter-optical axis distance detection unit 130 is not included, the essential configuration of the present technology does not change.

[0133] In the display terminal 20, the interpupillary distance detection unit 230 detects the user’s interpupillary distance IPD_USER and outputs a detection result as interpupillary distance information.

[0134] Here, the user’s interpupillary distance IPD_USER is detected by, for example, using a detection result by the sensor unit 201 (FIG. 3) or analyzing a captured image at a predetermined timing before the user wearing the display terminal 20 on the head performs an operation of starting reproduction of a video or during reproduction of a video.

[0135] The inter-optical axis distance information (camera inter-optical axis distance IPD_CAM) and the interpupillary distance information (user’s interpupillary distance IPD_USER) are input to the conversion processing unit 300 as conversion information.

[0136] However, the conversion information is not limited to the inter-optical axis distance information and the interpupillary distance information, and can include, for example, information regarding a distance to a virtual subject (main virtual subject among one or a plurality of virtual subjects) and information regarding the size of a virtual subject (main virtual subject among one or a plurality of virtual subjects).

[0137] Then, the conversion processing unit 300 performs conversion processing on the video information from the camera 11 on the basis of the conversion information input thereto, and outputs (transmits) converted video information obtained as a result to the display terminal 20.

[0138] More specifically, the conversion processing unit 300 uses the video information and the conversion information to perform conversion processing according to, for example, any one of the first to third methods or a combination of at least two of the first to third methods.

[0139] In this conversion processing, in order to perform appropriate conversion (correction), it is necessary to appropriately adjust parameters (parameters that affect the appearance to the user regarding the virtual subject) according to each method. In the conversion processing unit 300, a parameter adjustment unit 320 is provided to adjust this parameter. Note that details of the three methods of the first method to the third method will be described later.

[0140] In the display terminal 20, on the basis of the converted video information input thereto, the reproduction unit 220 reproduces video after conversion (stereoscopic video), and displays the video on the display unit 203. Consequently, the user wearing the display terminal 20 on the head can view the stereoscopic video displayed in front of the eyes.

[0141] (Overall Processing Flow)

[0142] Next, an overall processing flow of the video distribution system 1 of FIG. 1 will be described with reference to a flowchart of FIG. 9.

[0143] In step S11, the subject is image-captured by the two cameras 11-L and 11-R configured as stereo cameras.

[0144] In step S12, for example, post-production processing is performed by a distribution side such as a content creator, and a video for distribution is created by (the processing unit 100 of) the workstation 10.

[0145] In this post-production processing, as processing after image-capturing, for example, each of a video corresponding to the entire celestial sphere for the left eye of the user based on the left image captured by the camera 11-L and a video corresponding to the entire celestial sphere for the right eye of the user based on the right image captured by the camera 11-R is generated.

[0146] The video for distribution created here is distributed as a video stream by the video distribution server 12 to the display terminal 20 via the Internet 30.

[0147] In steps S13 to S16, (the processing unit 200 of) the display terminal 20 processes the video stream received via the Internet 30, and performs decoding and rendering processing, for example.

[0148] Specifically, in the display terminal 20, a 3D model and a virtual camera are arranged in the entire celestial spheres for the left eye and the right eye (S13), and processing of moving the arranged 3D model or virtual camera is performed as necessary (S14).

[0149] That is, here, in the virtual space, the virtual camera corresponding to the left eye of the user is arranged at the center of the entire celestial sphere for the left eye, and the virtual camera corresponding to the right eye of the user is arranged at the center of the entire celestial sphere for the right eye (S13). Furthermore, in the virtual space, a 3D model including a virtual subject corresponding to the subject that is image-captured by the stereo cameras is arranged (S13).

[0150] Furthermore, in this example, since the conversion processing unit 300 (FIG. 8) is included in (the processing unit 200 of) the display terminal 20, in a case where the relationship of IPD_CAM>IPD_USER occurs, or the like, the arranged 3D model or virtual camera is moved by performing the conversion processing according to any one of the first method to the third method or a combination of at least two methods of the first method to the third method (S14).

[0151] Subsequently, the display terminal 20 decodes the video (S15), and performs processing of attaching a texture to the 3D model (S16).

[0152] Thus, for example, texture is given to the surface of the 3D model including the virtual subject (S16). Note that, at this time, the conversion processing unit 300 (FIG. 8) rotates and attaches the texture to the 3D model, for example, so that it is possible to support the second method to be described later (that is, although details will be described later, the video to be attached to the entire celestial sphere can be rotated).

[0153] In step S17, it is determined whether the video to be reproduced is a moving image or the adjustment of the parameter is to be dynamically changed.

[0154] In a case where it is determined as affirmative (“Yes”) in the determination processing of step S17, the processing returns to step S14, and the processing of step S14 and subsequent steps is repeated. On the other hand, in a case where it is determined as negative (“No”) in the determination processing of step S17, the processing ends.

[0155] For example, in a case where there is a change in the subject as an image-capturing target, and there is implementation of dynamically adjusting the parameter according to an amount of the change, affirmative determination (“Yes”) is made in the determination processing of step 317, the processing of steps S14 to S16 is repeated, and the conversion processing by the conversion processing unit 300 is performed in the processing of step S14 or S16. Furthermore, the display terminal 20 may (temporarily) store the data of the video subjected to the conversion processing in the storage unit 202. Thus, the user can view the video subjected to the conversion processing later.

[0156] Note that, in the above description, although a case where the parameter adjustment according to the three methods of the first method to the third method is performed at a time of the rendering processing (S14, 316) has been described, the parameter adjustment may be performed not only at the time of the rendering processing but also, for example, at a time of the post-production processing (S12). That is, in this case, the conversion processing unit 300 is included not in (the processing unit 200 of) the display terminal 20 but in (the processing unit 100 of) the workstation 10.

[0157] However, as described with reference to FIG. 9, if it is handled at the time of rendering processing, it is possible to distribute a common video as a video stream from the distribution side and meanwhile display a video unique to each user viewing on the display terminal 20 side (video subjected to conversion processing), and thus there is an advantage that the degree of freedom at the time of distributing the video is increased.

[0158] Furthermore, in FIG. 9, what is distributed as a video stream is not limited to a moving image and may be a still image, and for example, in a case where the display terminal 20 side processes a still image as a video, it is determined as negative (“No”) in the determination processing of step S17 and the processing (loop) of steps S14 to S16 is not repeated, except for a case where parameter adjustment is dynamically performed.

[0159] The overall processing flow of the video distribution system 1 has been described above.

[0160] (Principle of Present Technology)

[0161] Here, the principle of the present technology will be described with reference to FIGS. 10 to 15.

[0162] FIG. 10 schematically illustrates a state where the user 50 wearing the display terminal 20 views the stereoscopic video, when seen from above, in a case where the video 500-L and the video 500-R corresponding to the left image and the right image respectively captured by the camera 11-L and the camera 11-R installed at the positions corresponding to the camera inter-optical axis distance IPD_CAM with respect to the subject are arranged in the virtual space. However, FIG. 10 illustrates when a relationship of IPD_CAM=IPD_USER occurs.

[0163] Note that, in FIG. 10, a direction from a lower side to an upper side in the diagram is a forward direction. Furthermore, this relationship similarly applies to other corresponding drawings.

[0164] As illustrated in FIG. 10, as a representative value characterizing an appearance of a virtual subject (virtual object), in addition to the viewing angle 502, a fusion distance 503 and the like can be exemplified, and an appearance of the virtual subject (virtual object) at this time is an appearance of a reference that looks equal to the real subject (real object).

[0165] More specifically, as illustrated in FIG. 11, in a case where stereo camera image-capturing of a subject is performed with the camera inter-optical axis distance IPD_CAM set to 65 nu, and videos 500-L and 500-R corresponding to the captured left image and right image are attached to the entire celestial spheres for the left eye and the right eye, respectively, it is assumed that the virtual subject is viewed from the centers of the entire celestial spheres for the left eye and the right eye of the user with the user’s interpupillary distance IPD_USER set to 65 mm.

[0166] At this time, a thick line 520 in the diagram corresponding to the distance between the virtual cameras placed at the centers of the entire celestial spheres for the left eye and the right eye coincides with the user’s interpupillary distance IPD_USER. Furthermore, the user’s interpupillary distance IPD_USER also coincides with the camera inter-optical axis distance IPD_CAM.

[0167] In FIG. 11, the range of the stereoscopic video seen in the left eye of the user is represented by a left angle of view 521-L, the range of the stereoscopic video seen in the right eye of the user is represented by a right angle of view 521-R, and the overall angle of view of the stereoscopic video is represented by an angle of view 522. Furthermore, in FIG. 11, a fused video is represented by a fusion video 523, and the angle of view 522 and the fusion video 523 correspond to the viewing angle 502 in FIG. 10.

[0168] Here, since the camera inter-optical axis distance IPD_CAM at the time of image-capturing coincides the user’s interpupillary distance IPD_USER at the time of viewing, a stereoscopic video (captured video) viewed by the user appears equal to that in a case of direct viewing without passing through the cameras 11-L and 11-R. However, here, a description in principle is made in order to make it simple, but in practice it is necessary to consider distortion and the like in image-capturing.

[0169] On the other hand, FIG. 12 schematically illustrates a state where the user wearing the display terminal 20 views the stereoscopic video when seen from above in the case where the relationship of IPD_CAM>IPD_USER occurs.

[0170] As illustrated in FIG. 12, the video displayed for the user 50 is the same as the video illustrated in FIG. 10. At this time, comparing the schematic diagram of FIG. 12 with the schematic diagram of FIG. 10, the viewing angle 502 of FIG. 12 is substantially the same as the viewing angle 502 of FIG. 10, but the fusion distance 503 of FIG. 12 is shorter than the fusion distance 503 of FIG. 10.

[0171] For this reason, under the condition of IPD_CAM>IPD_USER, while the size of the appearance is almost not optically changed, the fusion distance 503 is felt to be close and the virtual subject does not look so large even though the virtual subject is close, and consequently, the user feels that the virtual subject is small.

[0172] More specifically, as illustrated in FIG. 13, it is assumed that in a case where stereo camera image-capturing of the subject is performed with the camera inter-optical axis distance IPD_CAM set to 85 mm, and videos 500-L and 500-R corresponding to the captured left image and right image are attached to the entire celestial spheres for the left eye and the right eye, respectively, the virtual subject is viewed from the centers of the entire celestial spheres for the left eye and the right eye of the user with the user’s interpupillary distance IPD_USER set to 65 mm.

[0173] At this time, the thick line 520 in the diagram corresponding to the distance between the virtual cameras placed at the centers of the entire celestial spheres for the left eye and the right eye coincides with the user’s interpupillary distance IPD_USER, but the user’s interpupillary distance IPD_USER does not coincide with the camera inter-optical axis distance IPD_CAM.

[0174] Here, since the camera inter-optical axis distance IPD_CAM at the time of image-capturing and the user’s interpupillary distance IPD_USER at the time of viewing are in the relationship of IPD_CAM>IPD_USER, the entire celestial spheres to which the left and right videos 500-L and 500-R are attached are arranged inside the position considering the actual image-capturing position, and the overall scale becomes smaller. Thus, the stereoscopic video viewed by the user is seen closer than when directly viewed without passing through the cameras 11-L and 11-R.

[0175] Then, the user feels that the virtual subject is seen nearby even though the overall angle of view 522 (viewing angle 502) of the virtual subject does not change, and thus feels that the virtual subject seems small.

[0176] FIG. 14 illustrates in detail a state where the user views the stereoscopic video when IPD_CAM>IPD_USER in a case where the virtual subject (virtual object) is right in front.

[0177] A of FIG. 14 illustrates a state in the virtual space when it is assumed that the cameras 11-L and 11-R in the real space are installed at positions of black circles (.circle-solid.) at a left end and a right end of the thick line 520 in the diagram respectively as the camera inter-optical axis distance IPD_CAM and the subject is image-captured. On the other hand, B of FIG. 14 illustrates a state in the virtual space when the virtual subject corresponding to the subject that is image-captured in the state of A of FIG. 14 is viewed in a state where the left eye and the right eye (virtual cameras) of the user are located at positions of black circles (.circle-solid.) at a left end and a right end of the thick line 520 in the diagram as the user’s interpupillary distance IPD_USER.

[0178] At this time, in A and B of FIG. 14, the overall angles of view 522 are both approximately 49.degree. and are substantially the same angles, but the positions of the fusion videos 523 of the virtual subject right in front are different from the relationship of IPD_CAM>IPD_USER. That is, in B of FIG. 14, because the position of the fusion video 523 with respect to the thick line 520 in the diagram is closer as compared with that in A of FIG. 14, the user feels that the virtual subject right in front is viewed nearby, and the virtual subject seems small.

[0179] FIG. 15 illustrates in detail a state where the user views the stereoscopic video when IPD_CAM>IPD_USER in a case where the virtual subject (virtual object) is on the right front side.

[0180] In FIG. 15, similarly to FIG. 14 described above, positions of black circles (.circle-solid.) at a left end and a right end of the thick line 520 in the diagram correspond to the installation positions of the cameras 11-L and 11-R at the time of image-capturing (A of FIG. 15) and the positions of the left eye and the right eye of the user (B of FIG. 15), respectively.

[0181] At this time, in A and B of FIG. 15, the overall angles of view 522 are both approximately 440 and are substantially the same angles, but from the relationship of IPD_CAM>IPD_USER, in B of FIG. 15, since the position of the fusion video 523 with respect to the thick line 520 is closer as compared with that in A of FIG. 15, the user feels that the virtual subject on the right front side appears closer and this virtual subject seems small.

……
……
……