Sony Patent | Reproducing device, reproducing method, and program

Patent: Reproducing device, reproducing method, and program

Drawings: Click to check drawins

Publication Number: 20220086366

Publication Date: 20220317

Applicant: Sony

Assignee: Sony Group Corporation

Abstract

The present disclosure relates to a reproducing device, a reproducing method, and a program that enable a wide angle of view video and a planar video that is a part of the wide angle of view video to be switched and viewed without a sense of incongruity. A. video controller determines whether or not a line quality is capable of bearing a display of the wide angle of view video on the basis of the line quality including a line speed of the wide angle of view video supplied. from a server or a remaining amount of buffer data of the wide angle of view video and the partial video. In a case where the line quality is not capable of bearing the display of the wide angle of view video, at a timing immediately before switching the display between. the wide angle of view video and the partial video, after the wide angle of view video together with a white frame indicating the angle of view of the partial video corresponding to the wide angle of view video is displayed, and the display is switched to the partial video. It is applicable to a reproducing device.

Claims

1. A reproducing device comprising: a switching determination unit configured to determine whether or not a switching condition for switching a display between a wide angle of view video and a partial video is satisfied, the partial video corresponding to a partial angle of view of the wide angle of view video; and a video controller configured to selectively display the wide angle of view video and the partial video on a basis of a determination result of the switching determination unit, wherein when switching the display between the wide angle of view video and the partial video on a basis of the determination result of the switching determination unit, the video controller displays the wide angle of view video, to which display information indicating an angle of view of the partial video corresponding to the wide angle of view video has been added.

2. The reproducing device according to claim 1, further comprising video acquisition unit configured to acquire the wide angle of view video and the partial video from a server that is externally provided.

3. The reproducing device according to claim 2, wherein the switching determination unit determines whether or not the switching condition for switching the display between the wide angle of view video and the partial video is satisfied on a basis of a line quality with the server from which the video acquisition unit acquires the wide angle of view video.

4. The reproducing device according to claim 3, wherein the switching condition includes at least one of whether or not the line quality is faster than a predetermined line speed or whether or not a remaining amount of buffer data of the wide angle of view video and the partial video is larger than a predetermined threshold.

5. The reproducing device according to claim 2, further comprising a metadata acquisition unit configured to acquire metadata for identifying a position and a size of the angle of view of the partial video corresponding to the wide angle of view video from the server, wherein the video controller displays the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added on a basis of the metadata.

6. The reproducing device according to claim 5, wherein the metadata is generated on a basis of pan-tilt information including information regarding a video-capturing position and pan-tilt of a partial video capturing unit that captures the partial video.

7. The reproducing device according to claim 5, wherein the metadata acquisition unit acquires the metadata in the state of being adjusted for acquiring the metadata at a same timing with a timing when the video acquisition unit acquires the partial video.

8. The reproducing device according to claim 5, wherein the metadata acquisition unit acquires the metadata including time information when the partial video corresponding is supplied, and the video controller uses the metadata including the time information corresponding to the partial video to display the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added.

9. The reproducing device according to claim 1, wherein the display information indicating the angle of view of the partial video corresponding to the wide angle of view video includes a frame indicating the angle of view of the partial video corresponding to the wide angle of view video.

10. The reproducing device according to claim 2, wherein in addition to the wide angle of view video and the partial video, the video acquisition unit acquires the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added.

11. The reproducing device according to claim 10, wherein. the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added, is generated by adding the display information to the wide angle of view video on a basis of pan-tilt information including a video-capturing position and information regarding pan-tilt of a partial video capturing unit that captures the partial video.

12. The reproducing device according to claim 1, wherein the switching determination unit determines whether or not the switching condition for switching the display between the wide angle of view video and the partial video is satisfied on a basis of a presentation content of a content including the wide angle of view video,

13. The reproducing device according to claim 1, wherein in a case where the wide angle of view video is displayed, the video controller displays the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added, at a timing immediately before switching the display from the wide angle of view video to the partial video on a basis of the determination result of the switching determination unit.

14. The reproducing device according to claim 13, wherein when switching the display from the wide angle of view video to the partial video, the video controller displays a transition video for transitioning from the wide angle of view video to the partial video.

15. The reproducing device according to claim 14, wherein when. switching the display from the wide angle of view video to the partial video, the video controller displays the transition video for transitioning to the partial video while expanding an angle of view of the wide angle of view video.

16. The reproducing device according to claim 1, wherein in a case where the partial video is displayed, the video controller displays the wide angle of view video, to which the display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added, at a timing immediately after switching the display from the partial video to the wide angle of view video on a basis of the determination result of the switching determination unit.

17. The reproducing device according to claim 16, wherein when switching the display from the partial video to the wide angle of view video, the video controller displays a transition video for transitioning from the partial video to the wide angle of view video.

18. The reproducing device according to claim 17, wherein when switching the display from the partial video to the wide angle of view video, the video controller displays the transition. video for transitioning to the wide angle of view video while reducing the angle of view of the partial video.

19. A. reproducing method comprising: a switching determination process for determining whether or not a switching condition for switching a display between a wide angle of view video and a partial video is satisfied, the partial video corresponding to a partial angle of view of the wide angle of view video; and a video control process for selectively displaying the wide angle of view video and the partial video on a basis of a determination result of the switching determination process, wherein when switching the display between the wide angle of view video and the partial video on a basis of the determination result of the switching determination process, the video control process displays the wide angle of view video, to which display information indicating an angle of view of the partial video corresponding to the wide angle of view video has been added.

20. A program for causing a computer to function as: a switching determination unit configured to determine whether or not a switching condition for switching a display between a wide angle of view video and a partial video is satisfied, the partial video corresponding to a partial angle of view of the wide angle of view video; and a video controller configured to selectively display the wide angle of view video and the partial video on a basis of a determination result of the switching determination unit, wherein when switching the display between the wide angle of view video and the partial video on a basis of the determination result of the switching determination unit, the video controller displays the wide angle of view video, to which display information indicating an angle of view of the partial video corresponding to the wide angle of view video has been added.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to reproducing devices, reproducing methods, and programs, and in particular, a reproducing device, a reproducing method, and a program capable of switching between a wide angle of view video and a partial video that is a part of the wide angle of view video without a sense of incongruity for viewing.

BACKGROUND ART

[0002] A technique is proposed in which a wide angle of view video such as an entire celestial sphere video is distributed and displayed on a display device such as a head mounted display (HMD) (see Patent Document 1).

CITATION LIST

Patent Document

[0003] Patent Document 1: Japanese Patent. Application Laid-Open No. 2018-113616

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0004] The distribution of the wide angle of view videos, by the way, enables distribution of videos with realistic feelings. However, depending on a line condition in presentation or a live performance, there are scenes for switching between a wide angle of view video and a partial video (hereinafter, also referred to as a planar video) that is a part of the wide angle of view video.

[0005] In the technique according to Patent Document 1, in switching from a wide angle of view video to a planar video, it is not explicitly displayed at which part in the wide angle of view video the planar video is switched to.

[0006] For this reason, in a case where the wide angle of view video is switched to the planar video, a user who is a viewer cannot recognize that the planar video displayed after switching derives from which part in the wide angle of view video that the user had been viewing until then. Hence, there has been a sense of incongruity in viewing, in some cases.

[0007] The present disclosure has been made in view of such circumstances, and in particular, has an object to enable viewing without a sense of incongruity in switching between a wide angle of view video and a planar video that is a part of the wide angle of view video.

Solutions to Problems

[0008] A reproducing device according to one aspect of the present disclosure includes: a switching determination unit configured to determine whether or not a switching condition for switching a display between a wide angle of view video and a partial video is satisfied, the partial video corresponding to a partial angle of view of the wide angle of view video; and a video controller configured. to selectively display the wide angle of view video and the partial video on the basis of a determination result of the switching determination unit, in which when switching the display between the wide angle of view video and the partial video on the basis of the determination result of the switching determination unit, the video controller displays the wide angle of view video, to which display information indicating an angle of view of the partial video corresponding to the wide angle of view video has been added.

[0009] A reproducing method according to one aspect of the present disclosure corresponds to the reproducing device.

[0010] According to one aspect of the present disclosure, whether or not the switching condition for switching the display between the wide angle of view video and the partial video corresponding to the partial angle of view of the wide angle of view video is satisfied, the wide angle of view video and the partial video is selectively displayed on the basis of a determination result, when switching the display between the wide angle of view video and the partial video on the basis of the determination result, the wide angle of view video, to which display information indicating the angle of view of the partial video corresponding to the wide angle of view video has been added, is displayed.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a diagram illustrating an outline of the present disclosure.

[0012] FIG. 2 is a diagram illustrating a configuration example of a reproducing system in a first embodiment of the present disclosure.

[0013] FIG. 3 is a flowchart illustrating an entire celestial sphere video capture process by an entire celestial sphere camera of FIG. 2.

[0014] FIG. 4 is a flowchart illustrating a planar video capture process by a planar camera of FIG. 2.

[0015] FIG. 5 is a flowchart illustrating a metadata generation. process by a PC of FIG. 2.

[0016] FIG. 6 is a flowchart illustrating a video transmission process by a server of FIG. 2.

[0017] FIG. 7 is a flowchart illustrating a reproduction. process by a reproducing device of FIG. 2.

[0018] FIG. 8 is a flowchart illustrating a display process by an HMD of FIG. 2.

[0019] FIG. 9 is a diagram illustrating an application example of the reproducing system in the first embodiment of the present disclosure.

[0020] FIG. 10 is a flowchart illustrating a metadata generation process by a PC of FIG. 9.

[0021] FIG. 11 is a flowchart illustrating a video transmission process by a server of FIG. 9.

[0022] FIG. 12 is a diagram illustrating a configuration example of a reproducing system in a second embodiment of the present disclosure.

[0023] FIG. 13 is a flowchart illustrating a framed entire celestial sphere video generation process by a PC of FIG. 12.

[0024] FIG. 14 is a flowchart illustrating a video transmission process by a server of FIG. 12.

[0025] FIG. 15 is a flowchart illustrating a reproduction process by a reproducing device of FIG. 12.

[0026] FIG. 16 is a flowchart illustrating a display process by an HMD of FIG. 12.

[0027] FIG. 17 is a diagram illustrating a configuration. example of a general-purpose personal computer.

MODE FOR CARRYING OUT THE INVENTION

[0028] Hereinafter, preferred. embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It is to be noted that in the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate description.

[0029] Hereinafter, modes for carrying out the present technology will be described. The description will be given in the following order.

[0030] 1. Outline of the present disclosure

[0031] 2. First embodiment

[0032] 3. Application example of first embodiment

[0033] 4. Second embodiment

[0034] 5. Example to be performed by software

[0035] 1. Outline of the Present Disclosure

[0036] The present disclosure intends to switch between a wide angle of view video and a planar video that is a part of the wide angle of view video without a sense of incongruity for viewing.

[0037] Therefore, first, the outline of the present disclosure will be described.

[0038] A user wears a head mounted display (HMD) on the head, and a wide angle of view video is displayed according to movements of the user’s head. Consideration is given to a configuration in which, for example, as shown in the left part of FIG. 1, a wide angle of view video P1 including a relayed video of football game is displayed on a display unit C of the head mounted display (HMD).

[0039] In such a configuration, the wide angle of view video P1 in the left part of FIG. 1 is, in some cases, switched to a planar video P2 as shown in the right part of FIG. 1 to enlarge and display an area F’ where two players including a player keeping a ball and a player of the opponent team who is marking the player are present in the wide angle of view video P1.

[0040] It is to be noted that the wide angle of view video P1 is a video that has been captured by a first camera for capturing a wide angle of view video, and the planar video P2 is a video that has an angle of view within the wide angle of view video P1 and that has been captured by a second camera, which is different from the first camera and which captures a video having a narrower angle of view than that of the video P1.

[0041] However, in a case where the wide angle of view video P1 as shown in the left part of FIG. 1 is suddenly switched to the planar video P2 as shown in the right part of FIG. 1 at a predetermined timing, the user who views the video displayed on the display unit D can recognize that the video that the user is viewing has been switched.

[0042] However, the user may not be able to recognize what positional relationship the planar video P2 to be displayed after swtching has with respect to an immediately previous the wide angle of view video P1, and may feel a sense of incongruity.

[0043] Therefore, in the present disclosure, as shown in the central part of FIG. 1, an area corresponding to an area F’, which becomes the planar video P2 to be switched to neat, is displayed as a frame F in the wide angle of view video P1.

[0044] Then, in the present disclosure, the wide angle of view video P1, to which the frame F has been added as shown in the central part of FIG. 1 to the wide angle of view video P1 in the left part of FIG. 1, is displayed, and is then switched. to the planar video P2 shown in the right part of FIG. 1.

[0045] In this manner, at the timing immediately before the wide angle of view video P1 is switched to the planar video P2 to be displayed, a position corresponding to the planar video P2 is displayed as the frame F in the wide angle of view video P1. This configuration makes the user recognize that the planar video P2 to be switched to at the next timing corresponds to a video at which position in the wide angle of view video P1 immediately before switching, and then switches the video.

[0046] Consequently, the user viewing the display unit D recognizes the frame F displayed in the wide angle of view video P1, so as to be able to view the planar video P2 after switching while recognizing the positional relationship between the planar video P2 to be switched to next and the wide angle of view video P1 immediately before the switching.

[0047] As a result, the user can continue viewing without a sense of incongruity when the wide angle of view video P1 is switched to the planar video P2.

[0048] 2. First Embodiment

[0049] Next, a configuration example of a reproducing system in a first embodiment of the present disclosure will be described with reference to a block diagram of FIG. 2.

[0050] In switching between a wide angle of view video and a planar video that is a partial video in the wide angle of view video, the reproducing system according to the present disclosure displays the position of the planar video to be switched to next as a frame in the wide angle of view video, and then switches to the planar video, so as to switch between the videos without a sense of incongruity.

[0051] A reproducing system 1 of FIG. 2 includes an entire celestial sphere camera 11, a planar camera 12, a personal computer (PC) 13, a server 14, a reproducing device 15, a head mounted. display (HMD) 16, and encoding devices 21 and 22.

[0052] The reproducing system 1 of FIG. 2 is a system that relays and distributes contents, for example, sports, plays, and the like from a venue, and the entire celestial sphere camera 11, the planar camera 12, and the personal computer (PC) 13 are configured as venue facilities of a sport facility, a theater, or the like.

[0053] In addition, the reproducing device 15 and the HMD 16 are configured as facilities on a user’s side that receives content distribution.

[0054] Then, the server 14 acquires a video as a content from the venue facilities via a network, and distributes the video to the facilities on the user’s side.

[0055] The entire celestial sphere camera 11 is a camera installed in a fixed. state, and captures an entire celestial sphere video including a wide angle of view video having a range of 180 degrees, a range of 360 degrees, or the like with respect to the horizontal direction and the vertical direction, and outputs the video to the encoding device 21. The encoding device 21 encodes the entire celestial sphere video, and outputs the entire celestial sphere video that has been encoded to the server 14. It is to be noted that the entire celestial sphere camera 11 may have a function of the encoding device 21 so as to encode the captured entire celestial sphere video, and may output the entire celestial sphere video that has been encoded. to the server 14. Further, the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 is encoded by the encoding device 21, and is then output. to the server 14. However, in the following description, the process of encoding, by the encoding device 21, the entire celestial sphere video is omitted as necessary, and, for example, will be described as “the entire celestial sphere camera 11 outputs the entire celestial sphere video that has been captured. to the server 14”. However, only the expression is omitted, and in reality, the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 is encoded by the encoding device 21, and is then output to the server 14.

[0056] The planar camera 12 is a camera provided on a camera platform 12a, and captures a partial range of a range captured by the entire celestial sphere camera 11 as a planar video, for example, with a horizontal vertical ratio of 16:9, while changing a video-capturing position and pan-tilt using the camera platform 12a, and outputs the planar video to the encoding device 22. The encoding device 22 encodes the entire celestial sphere video, and outputs the entire celestial sphere video that has been encoded video to the server 14. It is to be noted that the planar camera 12 may have the function of the encoding device 22 to encode the planar video that has been captured, and may output the planar video that has been encoded to the server 14. Further, the planar video that has been captured. by the planar camera 12 is encoded by the encoding device 22, and is then output to the server 14. However, in the following description, the process of encoding, by the encoding device 22, the planar video is omitted as necessary, and will be simply described as “the planar camera 12 outputs the planar video that has been captured to the server 14”. However, only the expression is omitted, and in reality, the planar video that has been captured by the planar camera 12 is encoded by the encoding device 22, and is then output to the server 14.

[0057] Further, in the present disclosure, the description will be given with regard to an example in which an entire celestial sphere video and a planar video including a partial video of the entire celestial sphere video are respectively captured by the entire celestial sphere camera 11 and the planar camera 12. However, a configuration other than the entire celestial sphere camera 11 and the planar camera 12 may be used, as long as such a configuration is capable of capturing a wide angle of view video and a planar video including a partial video that is a part of the entire celestial sphere video.

[0058] The camera platform 12a outputs information indicating the video-capturing position and a pan-tilt direction of the planar camera 12, as pan-tilt information, to the PC 13.

[0059] The PC 13 acquires the pan-tilt information supplied from the camera platform 12a of the planar camera 12, generates metadata on the basis of the pan-tilt information, and outputs the metadata to the server 14.

[0060] More specifically, the PC 13 includes a controller 31, a data controller 32, and a delay processing unit 33.

[0061] The controller 31 includes a processor, a memory, and the like, and controls the entire operation of the PC 13, and controls the data controller 32 and the delay processing unit 33.

[0062] The data controller 32 generates metadata including the pan-tilt information on the basis of the pan-tilt information supplied from the camera platform 12a, and outputs the metadata to the delay processing unit 33.

[0063] The delay processing unit 33 temporarily stores the metadata supplied from the data controller 32, delays by a predetermined period so as to adjust the timing to match the timing when the planar video that has been captured by the planar camera 12 is output to the server 14, and outputs the metadata to the server 14.

[0064] That is, the planar video output from the planar camera 12 to the server 14 and the metadata including the pan-tilt information of the planar camera 12, which is capturing the planar video, are supplied to the server 14 at the same timing.

[0065] The server 14 receives and stores the entire celestial sphere video supplied from the entire celestial sphere camera. 11, the planar video supplied from the planar camera 12, and the metadata supplied from the PC 13, and in addition, outputs the entire celestial sphere video, the planar video, and the metadata to the reproducing device 15.

[0066] More specifically, the server 14 includes a controller 51, a metadata storage unit 52, a planar video storage unit 53, and an entire celestial sphere video storage unit 54.

[0067] The controller 51 includes a processor, a memory, and the like, and controls the entire operation of the server 14.

[0068] The controller 51 controls the operations of the metadata storage unit 52, the planar video storage unit 53, and the entire celestial sphere video storage unit 54.

[0069] When the metadata is supplied from the PC 13, the controller 51 acquires the metadata, and causes the metadata storage unit 52 to store the metadata. Further, the controller 51 reads the metadata stored in the metadata storage unit 52, and transmits the metadata to the reproducing device 15.

[0070] When the planar video is supplied from the planar camera 12, the controller 51 acquires the planar video, and causes the planar video storage unit 53 to store the planar video. Further, the controller 51 reads the planar video stored in the planar video storage unit 53, and transmits the planar video to the reproducing device 15.

[0071] When the entire celestial sphere video is supplied from the entire celestial sphere camera 11, the controller 51 acquires the entire celestial sphere video, and causes the entire celestial sphere video storage unit 54 to store the entire celestial sphere video. Further, the controller 51 reads the entire celestial sphere video stored in the entire celestial sphere video storage unit 54, and transmits the entire celestial sphere video to the reproducing device 15.

[0072] The reproducing device 15 acquires the entire celestial sphere video, the planar video, and the metadata supplied from the server 14, performs rendering on the basis of head tracking information from the HMD 16 worn by the user, and outputs the videos to the HMD 16 for display.

[0073] The reproducing device 15 causes the HMD 16 to display the entire celestial sphere video in. a case where the line quality is good, and causes the HMD 16 to display the planar video in a case where the line quality is degraded, depending on the line quality of the communication. line with the server 14.

[0074] In such a situation, in switching from the entire celestial sphere video to the planar video, the reproducing device 15 displays the range of the planar video to be switched to next as a frame in the entire celestial sphere video, so as to display that the planar video to be switched to next is at which position in the entire celestial sphere video, and then switches to the planar video and displays the planar video.

[0075] By switching from the entire celestial sphere video to the planar video in this manner, the user can view the video switched from the entire celestial sphere video to the planar video without a sense of incongruity.

[0076] It is to be noted that when the planar video is switched to the entire celestial sphere video, the frame indicating the range of the planar video is displayed in the entire celestial sphere video for a predetermined period at the timing of switching to the entire celestial sphere video.

[0077] By switching from the planar video to the entire celestial sphere video in this manner, the user can view the video switched from the planar video to the entire celestial sphere video without a sense of incongruity.

[0078] More specifically, the reproducing device 15 includes a connection controller 71, a switching determination unit 72, a decoding unit 73, a meta parser unit 74, a video controller 75, a user interface processing unit 76, and a rendering unit 77.

[0079] The connection controller 71 acquires the entire celestial sphere video and the planar video from the server 14, and outputs the videos to the decoding unit 73.

[0080] In such a situation, the connection. controller 71 outputs to the switching determination unit 72, information indicating the line quality of the line for acquiring the entire celestial sphere video from the server 14, for example, information regarding the line speed, information regarding a remaining amount of buffer data of the entire celestial sphere video and the planar video that are being received, and the like.

[0081] On the basis of the information indicating the line quality supplied from the connection controller 71, the switching determination unit 72 supplies the video controller 75 with switching determination information for instructing displaying of the entire celestial sphere video on the HMD 16, in a case where, for example, the line quality such as the line speed is faster than a predetermined line speed and is better than a predetermined quality. Further, in a case where the information indicating the line quality is the information regarding the remaining amount of the buffer data of the entire celestial sphere video and the planar video that are being received, when the remaining amount of the buffer data. is equal to or more than a predetermined threshold, the switching determination unit 72 may determine than he line quality is better than the predetermined quality and may instruct displaying of the entire celestial sphere video on the HMD 16.

[0082] In addition, in a case where the line quality is not better than the predetermined quality, the switching determination unit 72 supplies the video controller 75 with switching determination information for instructing displaying of the planar video on the HMD 16.

[0083] The decoding unit 73 decodes the entire celestial sphere video and the planar video, and outputs the videos to the video controller 75.

[0084] The meta parser unit 74 acquires the metadata supplied from the server 14, and outputs the metadata to the video controller 75.

[0085] The user interface processing unit 76 acquires the head tracking information supplied from the HMD 16, and outputs the head tracking information to the video controller 75.

[0086] The head tracking information is, for example, information regarding the position and the direction of the HMD 16 worn on the user’s head, and is information indicating at which position and in which direction the user is facing.

[0087] The video controller 75 determines a video on the basis of the switching determination information supplied from the switching determination unit 72, the entire celestial sphere video and the planar video supplied from the decoding unit 73, the metadata supplied from the meta parser unit 74, and the head tracking information supplied from the user interface processing unit 76, and outputs the determined video to the rendering unit 77.

[0088] The rendering unit 77 renders the video supplied from the video controller 75, outputs the video that has been rendered to the HMD 16 for display.

[0089] The HMD 16 is a display device worn on the user’s head, and displays the video supplied from the reproducing device 15.

[0090] More specifically, the HMD 16 includes a controller 81, a head tracking information acquisition unit 82, and a display unit 63.

[0091] The controller 81 includes a processor and a memory, and controls the entire operation of the HMD 16.

[0092] The head tracking information acquisition unit 82 includes, for example, a motion sensor or the like, acquires information regarding the position, the direction of the user’s head, and the like as the head tracking information, and transmits the head tracking information to the reproducing device 15.

[0093] The display unit 83 includes a liquid crystal display (LCD) and an organic electro luminescence (EL), is provided in a viewing direction of the user wearing the END 16, and displays the video supplied from the reproducing device 15.

[0094]

[0095] Next, an entire celestial sphere video capture process by the entire celestial sphere camera 11 will be described with reference to a flowchart of FIG. 3.

[0096] In step S11, the entire celestial sphere camera 11 captures an entire celestial sphere video, and outputs the entire celestial sphere video to the encoding device 21.

[0097] In step S12, the encoding device 21 encodes the entire celestial sphere video that has been supplied from the entire celestial sphere camera 11, and transmits the entire celestial sphere video that has bees encoded to the server 14.

[0098] In step S13, the entire celestial sphere camera 11 determines whether or not the end of the process has been instructed. In a case where the end has not been instructed, the process returns to step S11, and the subsequent processes are repeated.

[0099] Then, in step S13, in a case where the end of the process has been instructed, the process ends.

[0100] By the above process, the entire celestial sphere video is captured by the entire celestial sphere camera 11, is encoded by the encoding device 21, and is continuously transmitted to the server 14.

[0101]

[0102] Next, a planar video capture process by the planar camera 12 will be described with reference to a flowchart of FIG. 4.

[0103] In step S31, the planar camera. 12 captures a planar video including a partial video of the entire celestial sphere video captured by the entire celestial sphere camera 11, and outputs the planar video to the encoding device 22.

[0104] In step S32, the camera platform 12a of the planar camera 12 acquires the pan-tilt information including the information regarding the video-capturing position and pan-tilt that is a video-capturing direction of the planar camera 12.

[0105] In step S33, the encoding device 22 encodes the planar video that has been captured, and outputs the planar video that has been encoded to the server 14.

[0106] In step S34, the camera platform 12a outputs the pan-tilt information to the PC 13.

[0107] In step S35, the planar camera 12 determines whether or not the end of the process has been instructed. In a case were the end has not been instructed, the process returns to step S31, and the subsequent processes are repeated.

[0108] Then, in step S35, in a case where the end of the process has been instructed, the process ends.

[0109] By the above process, the planar video is captured by the planar camera 12, is encoded by the encoding device 22, and is transmitted to the server 14, and in addition, the pan-tilt information including the video-capturing position and the information regarding the pan-tilt of the planar camera 12 at the timing when the planar video is captured is output to the PC13.

[0110]

[0111] Next, the metadata generation process by the PC 13 in FIG. 2 will be described with reference to a flowchart of FIG. 5.

[0112] In step S51, the data controller 32 acquires the pan-tilt information supplied from the camera platform 12a of the planar camera 12.

[0113] In step S52, the data controller 32 generates metadata on the basis of the pan-tilt information that has been acquired, and outputs the metadata to the delay processing unit 33.

[0114] In step S53, the delay processing unit 33 acquires the metadata supplied from the data controller 32, stores the metadata until the timing when the corresponding planar video in the planar camera 12 is transmitted to the server 14, and delays the output.

[0115] In step S54, the delay processing unit 33 outputs the metadata to the server 14 at the timing when the planar video corresponding to the pan-tilt information included in the metadata that is stored is transmitted to the server 14.

[0116] In step S55, the controller 31 determines whether or not the end of the process has been instructed. In a case where the end has not been instructed, the process returns to step S51, and the subsequent processes are repeated.

[0117] Then, in step S55, in a case where the end of the process has been instructed, the process ends.

[0118] By the above process, the metadata based on the pan-tilt information including the video-capturing position and the information regarding the pan-tilt at the timing when the planar video is captured by the planar camera 12 is generated, and in addition, the metadata is output to the server 14 at the timing when the corresponding planar video is transmitted to the server 14.

[0119] As a result, the planar video and the metadata including the corresponding pan-tilt information are adjusted at the same timing to be transmitted to the server 14.

[0120]

[0121] Next, a video transmission process by the server 14 of FIG. 2 will be described with reference to a flowchart of FIG. 6.

[0122] In step S71, the controller 51 of the server 14 acquires the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 and that has been encoded by the encoding device 21, and causes the entire celestial sphere video storage unit 54 to store the entire celestial sphere video.

[0123] In step S72, the controller 51 reads the entire celestial sphere video that is stored in the entire celestial sphere video storage unit 54, and outputs the entire celestial sphere video to the reproducing device 15.

[0124] In step S73, the controller 51 acquires a planar video that has been captured by the planar camera 12 and that has been encoded by the encoding device 22, and causes the planar video storage unit 53 to store the planar video.

[0125] In step S74, the controller 51 reads the planar video that is stored in the planar video storage unit 53, and outputs the planar video to the reproducing device 15.

[0126] In step S75, the, controller 51 acquires the metadata supplied from the PC 13, and causes the metadata storage unit 52 Co store the metadata.

[0127] In step S76, the controller 51 reads the metadata that is stored in the metadata storage unit 52, and outputs the metadata to the reproducing device 15.

[0128] In step S77, the controller 51 determines whether or not the end of the process has been instructed. In a case where the end has not been instructed, the process returns to step S71, and the subsequent processes are repeated.

[0129] Then, in step S77, in a case where the end of the process has been instructed, the process ends.

[0130] By the above process, the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 and that has been encoded by the encoding device 21, the planar video that has been captured by the planar camera 12 and that has been encoded. by the encoding device 22, and the metadata that has been generated by the PC 13 are respectively transmitted to the reproducing device 15.

[0131] In such a situation, regarding the entire celestial sphere video, the planar video, and the metadata, respectively corresponding pieces of information at the same timing are supplied to the reproducing device 15.

[0132] Therefore, the processes of steps S71 and S72, the processes of steps S73 and S74, and the processes of steps S75 and S76 in the flowchart of FIG. 6 are substantially parallel processes.

[0133]

[0134] Next, a reproduction process by the reproducing device 15 of FIG. 2 will be described with reference to the flowchart of FIG. 7.

[0135] In step S111, the connection controller 71 acquires the entire celestial sphere video and the planar video that are encoded from the server 14, and outputs the videos to the decoding unit 73.

[0136] In step S112, the decoding unit 73 decodes the entire celestial sphere video and the planar video that are encoded and that have been supplied from the connection controller 71, and outputs the videos that have been decoded to the video controller 75. That is, the video controller 75 acquires the entire celestial sphere video and the planar video that have been decoded.

[0137] In step S113, the meta parser unit 74 acquires the metadata supplied from the server 14, and outputs the metadata to the video controller 75. That is, the video controller 75 acquires the metadata.

[0138] In step S114, the video controller 75 acquires head tracking information from the HMD 16.

[0139] In step S115, the connection controller 71 outputs to the switching determination unit 72, information regarding, for example, the line speed and the remaining amount of the buffer data of the entire celestial sphere video and the planar video that are being received, as the information regarding the line quality of the entire celestial sphere video from the server 14. That is, the switching determination unit 72 acquires the information regarding the line speed and the remaining amount of the buffer data of the videos, as the information regarding the line quality.

[0140] In step S116, the switching determination unit 72 determines whether or not the line speed is faster than a predetermined speed and whether or not the remaining amount of the buffer data is equal to or larger than a predetermined threshold on the basis of the information regarding the line quality including the line speed or the remaining amount of the buffer data of the entire celestial sphere video and the planar video that are being received, so as to determine whether or not the line quality is sufficient to be capable of reproducing the entire celestial sphere video, that is, whether or not to reproduce the entire celestial sphere video.

[0141] In a case where it is determined in step S116 that the line quality is sufficient. to reproduce the entire celestial sphere video, the switching determination unit 72 outputs the switching determination information for instructing reproduction of the entire celestial sphere video to the video controller 75. The process proceeds to step S117.

[0142] In step S117, the video controller 75 selects the entire celestial sphere video that has been decoded on the basis of the switching determination information for instructing the reproduction. of the entire celestial sphere video and the head tracking information, and outputs the entire celestial sphere video that has been selected to the rendering unit 77.

[0143] In step S118, the rendering unit 77 renders the entire celestial sphere video that has been supplied from the reproducing device 15, and outputs the entire celestial sphere video that has been rendered. to the HMD 16 for display.

[0144] In step S119, the video controller 75 determines whether or not the end of the process has been instructed. In a case where the end has not been instructed, the process returns to step S111, and the subsequent processes are repeated.

[0145] Then, in step S119, in a case where it is determined that the end has been instructed, the process ends.

[0146] On the other hand, in a case where it is determined in step S116 that the line quality is not sufficient to reproduce the entire celestial sphere video, the switching determination unit 72 outputs the switching determination information for instructing the reproduction of the planar video to the video controller 75 and the process proceeds to step S120.

[0147] In step S120, the video controller 75 reads the pan-tilt information indicating the video-capturing position and the direction of the planar camera 12 included in the metadata on the basis of the switching determination information, identifies the position of the planar video in the entire celestial sphere video, and sets a display position and a size of a frame indicating the corresponding position in the entire celestial sphere video.

[0148] In step S121, the video controller 75 generates the entire celestial sphere video, to which a frame indicating the position and the size corresponding to the planar video has been added on the basis of the head tracking information, the entire celestial sphere video that has been decoded, and the information regarding the position and the size of the frame, and outputs the entire celestial sphere video with the frame to the rendering unit 77.

[0149] In step S122, the rendering unit 77 renders the entire celestial sphere video with the frame indicating the position and the size corresponding to the planar video, outputs the entire celestial sphere video with the frame to the HMD 16 for display.

[0150] In step S123, the video controller 75 generates a transition video for transitioning from the entire celestial sphere video to the planar video.

[0151] The transition video for transitioning from the entire celestial sphere video to the planar video is, for example, a video in a process of gradually enlarging an area in the planar video displayed as the frame in the entire celestial sphere video and transitioning to the planar video. The entire celestial sphere video and the planar video are used for, for example, interpolation generation.

[0152] In step S124, the rendering unit 77 renders the transition video for transitioning from the entire celestial sphere video to the planar video, and outputs the transition video to the HMD 16 for display.

[0153] In step S125, the video controller 75 outputs the planar video that has been decoded to the rendering unit 77.

[0154] In step S126, the rendering unit 77 renders the planar video, and outputs the planar video to the HMD 16 for display.

[0155] That is, by the processes of steps S120 to S126, in the HMD16, after the frame indicating the corresponding position of the planar video is added and the entire celestial sphere video is displayed, the transition video for transitioning to the planar video is displayed, and is then switched to displaying of the planar video.

[0156] In step S127, the connection controller 71 acquires the entire celestial sphere video and the planar video from the server 14, and outputs the videos to the decoding unit 73.

[0157] In step S128, the decoding unit 73 decodes the entire celestial sphere video and the planar video that have been supplied from the connection controller 71, and outputs the entire celestial sphere video and the planar video that have been decoded to the video controller 75. That is, the video controller 75 acquires the entire celestial sphere video and the planar video that have been decoded.

[0158] In step S129, the meta parser unit 74 acquires the metadata supplied from the server 14, and outputs the metadata to the video controller 75. That is, the video controller 75 acquires the metadata. In step S130, the video controller 75 acquires the head tracking information from the HMD 16.

[0159] In step S131, the connection. controller 71 outputs to the switching determination unit 72, the information regarding, for example, the line speed or the remaining amount of the buffer data of the entire celestial sphere video and the planar video that are being received, as the information regarding the line quality of the entire celestial sphere video from the server 14. That is, the connection controller 71 acquires the information regarding the line speed or the remaining amount of the buffer data of the videos, as the information regarding the line quality.

……
……
……