Sony Patent | Information processing device, method, and program

Patent: Information processing device, method, and program

Drawings: Click to check drawins

Publication Number: 20220109794

Publication Date: 20220407

Applicant: Sony

Abstract

The present technology relates to an information processing device, a method, and a program capable of reducing the visual load of a video. An information processing device includes an input acquisition unit that acquires user input that designates a display range of a free-viewpoint video, and a control unit that controls a virtual camera that determines the display range of the free-viewpoint video on the basis of the user input. When the control unit changes an angle of view of the virtual camera from a first angle of view including a first target to a second angle of view including a second target, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation while maintaining the distance between the virtual camera and the first target. The present technology can be applied to an information processing device.

Claims

1. An information processing device comprising an input acquisition unit that acquires user input that designates a display range of a free-viewpoint video, and a control unit that controls a virtual camera that determines the display range of the free-viewpoint video on a basis of the user input, wherein when the control unit changes an angle of view of the virtual camera from a first angle of view including a first target to a second angle of view including a second target in response to the user input, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation of the virtual camera while maintaining the distance between the virtual camera and the first target.

2. The information processing device according to claim 1, wherein the control unit determines the second angle of view on a basis of the user input.

3. The information processing device according to claim 2, wherein if the angular velocity of at least one of the pan rotation or the tilt rotation of the virtual camera is the predetermined angular velocity larger than a predetermined threshold value, the control unit redetermines the second angle of view so that the angular velocity of the pan rotation and the tilt rotation of the virtual camera is equal to or less than the threshold value.

4. The information processing device according to claim 3, wherein when the control unit redetermines the second angle of view, the control unit performs at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, so that the angle of view of the virtual camera changes from the first angle of view to the redetermined second angle of view.

5. The information processing device according to claim 4, wherein when the control unit redetermines the second angle of view, the control unit moves the virtual camera away from the first target so that the angle of view of the virtual camera changes from the first angle of view to a third angle of view, and then moves the virtual camera so that the angle of view of the virtual camera changes from the third angle of view to the second angle of view.

6. The information processing device according to claim 5, wherein the control unit determines the third angle of view, so that the first target and the second target are included in the third angle of view.

7. The information processing device according to claim 6, wherein the control unit determines the third angle of view on a basis of an amount of movement of corresponding feature points between the free-viewpoint videos at different times.

8. The information processing device according to claim 1, wherein the control unit moves the virtual camera from a position corresponding to the first angle of view to a position corresponding to the second angle of view while keeping the virtual camera away from the first target and the second target by a certain distance or more.

9. The information processing device according to claim 1, wherein when a relationship between the first angle of view and the second angle of view satisfies a predetermined condition, the control unit switches the angle of view of the virtual camera from the first angle of view to the second angle of view, and performs fade processing, so that the free-viewpoint video of the first angle of view gradually changes to the free-viewpoint video of the second angle of view.

10. The information processing device according to claim 9, wherein when a distance from the position corresponding to the first angle of view of the virtual camera to the position corresponding to the second angle of view of the virtual camera is shorter than a predetermined distance and an angle formed by a direction corresponding to the first angle of view of the virtual camera and a direction corresponding to the second angle of view of the virtual camera is larger than a predetermined angle, it is considered that the predetermined condition is satisfied.

11. The information processing device according to claim 2, wherein the control unit determines the second angle of view on a basis of the user input and a movement of the first target.

12. An information processing method comprising acquiring user input that designates a display range of a free-viewpoint video, and when changing an angle of view of a virtual camera that determines the display range of the free-viewpoint video in response to the user input from a first angle of view including a first target to a second angle of view including a second target, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while maintaining the distance between the virtual camera and the first target by an information processing device.

13. A program that causes a computer to perform processing including the steps of acquiring user input that designates a display range of a free-viewpoint video, and when changing an angle of view of a virtual camera that determines the display range of the free-viewpoint video in response to the user input from a first angle of view including a first target to a second angle of view including a second target, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while maintaining the distance between the virtual camera and the first target.

Description

TECHNICAL FIELD

[0001] The present technology relates to an information processing device, a method, and a program, and more particularly to an information processing device, a method, and a program capable of reducing the visual load of a video.

BACKGROUND ART

[0002] For example, by using the free-viewpoint video viewing technology, the user can view a content from a viewpoint from an arbitrary position in a 3D space.

[0003] On the other hand, for contents such as sports where the viewing target and story development are clear, not only can the viewpoint position of the content be directly designated by the user, but also the viewpoint position can be changed according to a camera path generated by the system. With this method, it is possible to present a satisfactory video to the user without any particular operation by the user.

[0004] A camera path indicates a temporal change in the position and imaging direction of a virtual camera when a content video is displayed as if it was captured by the virtual camera. In this case, the position of the virtual camera is the viewpoint position of the content.

[0005] The camera path may be automatically generated by the system, or in a case where an input operation such as designating a target of interest in the content is performed by the user, the camera path may be generated by the system in response to the input operation.

[0006] Here, consider the case where the system generates a camera path in response to a user input operation. For example, when a predetermined target is designated by the user, the system generates a camera path in which the viewpoint position is moved from one viewpoint position to another viewpoint position and the virtual camera is rotated at a constant angular velocity so that the target is kept within the angle of view of the virtual camera.

[0007] However, in this case, while the virtual camera is moving, a state may occur where not only is the target out of the angle of view, but also all other objects are out of the angle of view. In such a case, the user will feel dissatisfied with the presented content video.

[0008] Against this background, there has been proposed a technology for, in a case of generating a free-viewpoint video, for example, limiting the viewpoint position of a virtual camera so that one of multiple objects does not go out of the frame (see Patent Document 1, for example). In Patent Document 1, in FIG. 35, for example, the object is always kept within the frame, that is, within the angle of view by rotating the virtual camera with the predetermined object position as the rotation center.

[0009] Additionally, there has also been proposed a technology for moving a virtual camera in parallel according to the movement of a player who is the subject, so that the virtual camera is always positioned at a certain distance in front of the player even if the player changes his/her position or direction, for example (see Patent Document 2, for example).

[0010] As described above, if some subject is always within the angle of view of the virtual camera, it is possible to curb dissatisfaction with the presented content video.

CITATION LIST

Patent Document

[0011] Patent Document 1: Japanese Patent Application Laid-Open No. 2015-114716 [0012] Patent Document 2: Japanese Patent Application Laid-Open No. 2006-310936

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0013] However, in the above-mentioned technology, the load on the user when the user visually identifies the video is not considered. For this reason, when the system generates the camera path of the virtual camera, the visual load of the video may increase.

[0014] The present technology has been made in view of such a situation, and aims to reduce the visual load of a video.

Solutions to Problems

[0015] An information processing device of one aspect of the present technology includes an input acquisition unit that acquires user input that designates a display range of a free-viewpoint video, and a control unit that controls a virtual camera that determines the display range of the free-viewpoint video on the basis of the user input. When the control unit changes an angle of view of the virtual camera from a first angle of view including a first target to a second angle of view including a second target in response to the user input, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, the control unit performs at least one of the pan rotation or the tilt rotation of the virtual camera while maintaining the distance between the virtual camera and the first target.

[0016] An information processing method or program of one aspect of the present technology includes the steps of acquiring user input that designates a display range of a free-viewpoint video, and when changing an angle of view of a virtual camera that determines the display range of the free-viewpoint video in response to the user input from a first angle of view including a first target to a second angle of view including a second target, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, performing at least one of the pan rotation or the tilt rotation of the virtual camera while maintaining the distance between the virtual camera and the first target.

[0017] In one aspect of the present technology, user input that designates a display range of a free-viewpoint video is acquired, and when changing an angle of view of a virtual camera that determines the display range of the free-viewpoint video in response to the user input from a first angle of view including a first target to a second angle of view including a second target, if at least one of pan rotation or tilt rotation of the virtual camera has a predetermined angular velocity, at least one of the pan rotation or the tilt rotation of the virtual camera is performed while moving the virtual camera away from the first target, and if the angular velocity of the pan rotation and the tilt rotation of the virtual camera is smaller than the predetermined angular velocity, at least one of the pan rotation or the tilt rotation of the virtual camera is performed while maintaining the distance between the virtual camera and the first target.

BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is a diagram illustrating an example of a camera path.

[0019] FIG. 2 is a diagram illustrating generation of a camera path.

[0020] FIG. 3 is a diagram illustrating generation of a camera path.

[0021] FIG. 4 is a diagram illustrating generation of a camera path.

[0022] FIG. 5 is a diagram illustrating generation of a camera path.

[0023] FIG. 6 is a diagram illustrating generation of a camera path.

[0024] FIG. 7 is a diagram illustrating generation of a camera path.

[0025] FIG. 8 is a diagram showing a configuration example of a video viewing system.

[0026] FIG. 9 is a flowchart illustrating camera path generation processing.

[0027] FIG. 10 is a diagram illustrating generation of a camera path.

[0028] FIG. 11 is a diagram illustrating calculation of the pixel difference.

[0029] FIG. 12 is a flowchart illustrating camera path generation processing.

[0030] FIG. 13 is a flowchart illustrating camera path generation processing.

[0031] FIG. 14 is a diagram illustrating correction of a viewpoint position.

[0032] FIG. 15 is a block diagram showing a configuration example of a computer.

MODE FOR CARRYING OUT THE INVENTION

[0033] Hereinafter, an embodiment to which the present technology is applied will be described with reference to the drawings.

First Embodiment

[0034]

[0035] The present technology reduces the visual load of a video by appropriately combining the rotation and translation (parallel translation) of a virtual camera and rotating the virtual camera at a predetermined angular velocity or less when generating a camera path for a free-viewpoint video. The visual load of a video can cause so-called visually induced motion sickness, for example.

[0036] The present technology can be applied to a video viewing system using a head mounted display (HMD), for example, and can also be applied to a video viewing system using a display such as a television or a smartphone.

[0037] The video viewing system to which the present technology is applied assumes presentation of a video (hereinafter also referred to as free-viewpoint video) whose viewpoint position changes over time, such as a free-viewpoint content based on a live-action video and a game content including computer graphics (CG). Additionally, contents presented by the video viewing system include recorded content and real-time content.

[0038] For example, the content of a free-viewpoint video based on a live-action video is a content that allows the viewer to appreciate the video as if it was captured by a virtual camera at an arbitrary position in the space on the basis of videos captured by multiple cameras. That is, the content of a free-viewpoint video is a video content in which the position of the virtual camera is the viewpoint position and the direction in which the virtual camera is directed is the imaging direction.

[0039] The video viewing system may be provided with a device capable of detecting the action (movement) of the user who is the viewer when viewing the content.

[0040] Specifically, in a case where a video viewing system includes an HMD, for example, a position track system that acquires information indicating the orientation and position of the head of the user wearing the HMD, a system that detects the direction of the line-of-sight of the user by a camera, another sensor, or the like, and a system that detects the posture of the user by a camera, a time of flight (TOF) sensor, or the like may be provided.

[0041] In addition, the user’s line-of-sight direction may be detected by a camera, another sensor, or the like attached to a television, for example. Moreover, the video viewing system may be provided with a remote controller or a game controller for transmitting the intention of the user who is the viewer to the video viewing system.

[0042] For example, in the video viewing system, the user can designate a subject (object) of interest by an input operation to a remote controller or a game controller, by the direction of the user’s line-of-sight and the head, the direction of the user’s body, and the like. In this case, the video viewing system moves the viewpoint position of the free-viewpoint video to a position where the target of interest designated by the user can be clearly seen.

[0043] Accordingly, for example, the user can operate a key on a remote controller or the like to move the viewpoint position so that the target is displayed larger, or look at a specific target to designate the target by his/her line-of-sight and move the viewpoint position to a position where the target can be clearly seen.

[0044] Moreover, in a case where the target moves in the free-viewpoint video, the viewpoint position may be moved so that the target is continuously included in the angle of view of the virtual camera. Additionally, in a case where the target is an object that keeps moving like a sports player, the viewpoint position of the free-viewpoint video is not fixed, and even after the target appears large enough in the displayed frame (image), the viewpoint position may continue to move according to the movement of the player.

[0045] Hereinbelow, the present technology will be described more specifically. In particular, in the following, the description will be continued by taking the case of generating a camera path of a free-viewpoint video in a video viewing system as an example.

[0046] A free-viewpoint video is, for example, a video (image) of an arbitrary display range in a space generated on the basis of videos captured by cameras at multiple different viewpoint positions and imaging directions.

[0047] Here, the display range of the free-viewpoint video is the range captured by the virtual camera in the space, that is, the range of the angle of view of the virtual camera, and this display range is determined by the position of the virtual camera in the space which is the viewpoint position, and the direction of the virtual camera, that is, the imaging direction of the virtual camera.

[0048] In a free-viewpoint video, the position (viewpoint position) and imaging direction of the virtual camera change over time.

[0049] For example, the viewpoint position, which is the position of the virtual camera in the space, is represented by the coordinates of a three-dimensional Cartesian coordinate system whose origin is a reference position in the space.

[0050] Additionally, for example, the imaging direction (direction) of the virtual camera in the space is represented by the rotation angle of the virtual camera from a reference direction in the space. That is, for example, the rotation angle indicating the imaging direction of the virtual camera is the rotation angle when the virtual camera is rotated from the state in which the virtual camera is facing the reference direction to the state in which the virtual camera is facing the desired imaging direction.

[0051] Note that more specifically, the rotation angle of the virtual camera includes the yaw angle, which is the rotation angle when the virtual camera is rotated in the horizontal (left-right) direction (pan rotation), and the pitch angle, which is a rotation angle when the virtual camera is rotated in the vertical (up-down) direction (tilt rotation). In the following, when it is described that the virtual camera rotates and the rotation angle changes, it is assumed that at least one of the yaw angle or the pitch angle changes.

[0052] Additionally, the viewpoint position and rotation angle of the virtual camera at a predetermined time are denoted by P0 and R0, and the viewpoint position and rotation angle of the virtual camera at a time after the predetermined time are denoted by P1 and R1.

[0053] At this time, the temporal change of the viewpoint position of the virtual camera and the temporal change of the rotation angle of the virtual camera when the rotation angle of the virtual camera is changed from R0 to R1 while moving the virtual camera from the viewpoint position P0 to the viewpoint position P1 form a camera path of the virtual camera starting from the viewpoint position P0 and ending at the viewpoint position P1.

[0054] More specifically, the temporal change of the viewpoint position of the virtual camera is determined by the movement path of the virtual camera and the movement speed of the virtual camera at each position on the movement path. Additionally, the temporal change of the rotation angle of the virtual camera is determined by the rotation angle and the rotation speed (angular velocity of rotation) of the virtual camera at each position on the movement path of the virtual camera.

[0055] In the following, in particular, the viewpoint position and rotation angle of the virtual camera at the start point of the camera path are denoted by P0 and R0, and the viewpoint position and rotation angle of the virtual camera at the end point of the camera path are denoted by P1 and R1.

[0056] Additionally, the state of the virtual camera having the viewpoint position P0 and the rotation angle R0 is also denoted by a state ST0, and the state of the virtual camera having the viewpoint position P1 and the rotation angle R1 is also denoted by a state ST1.

[0057] Now, assume that when the virtual camera is in the state ST0, a target T0 which is a predetermined subject of interest is included in the angle of view of the virtual camera.

[0058] In such a state ST0, consider a case where the user designates a target T1 which is a new subject of interest, and a camera path in which the state of the virtual camera changes from the state ST0 to the state ST1 is generated, for example. At this time, assume that the target T1 is included in the angle of view of the virtual camera in the state ST1.

[0059] For example, assume that a camera path is generated in which the rotation angle of the virtual camera rotates at a constant angular velocity from R0 to R1 when changing from the state ST0 to the state ST1.

[0060] In this case, for example, as shown in FIG. 1, during the movement of a virtual camera VC11, a timing may occur in which neither the target T0 nor the target T1 is included in the angle of view of the virtual camera VC11.

[0061] In the example shown in FIG. 1, a video of a table tennis match in which the players are the target T0 and the target T1 is displayed as a free-viewpoint video. Additionally, in FIG. 1, the position indicated by an arrow W11 indicates the viewpoint position P0, the position indicated by an arrow W12 indicates the viewpoint position P1, and the dotted line indicates the movement path of the virtual camera VC11.

[0062] In this example, while the virtual camera VC11 moves from the position indicated by the arrow W11 to the position indicated by the arrow W12, the rotation angle of the virtual camera VC11, that is, the imaging direction changes at a constant angular velocity. In other words, the virtual camera VC11 rotates at a constant rotation speed.

[0063] In this case, for example, when the virtual camera VC11 is at the position indicated by an arrow W13, neither the target T0 nor the target T1 is included in the angle of view of the virtual camera VC11. Accordingly, neither the target T0 nor the target T1 appears in the displayed free-viewpoint video, and the user who is the viewer feels dissatisfaction.

[0064] On the other hand, for example, if the target T1 is continuously included in the angle of view of the virtual camera VC11 in the latter half of the camera path as shown in FIG. 2, the dissatisfaction caused in the example shown in FIG. 1 can be solved, and the user’s satisfaction with the free-viewpoint video can be improved. Note that in FIG. 2, the same reference numerals are given to the parts corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate.

[0065] In the example shown in FIG. 2, a midpoint Pm is a predetermined position in the camera path while the virtual camera VC11 moves from the viewpoint position P0 indicated by the arrow W11 to the viewpoint position P1 indicated by the arrow W12. Here, the position indicated by an arrow W21 in the camera path is defined as the midpoint Pm.

[0066] In this case, the rotation angle is determined so that the target T1 is within the angle of view of the virtual camera VC11 at least when the virtual camera VC11 reaches the midpoint Pm. Then, the rotation angle at each position on the movement path of the virtual camera VC11 is determined so that the target T1 is always included in the angle of view of the virtual camera VC11 while the virtual camera VC11 moves from the midpoint Pm to the viewpoint position P1 in the camera path. In other words, the camera path is generated so that the virtual camera VC11 keeps facing the target T1 while moving from the midpoint Pm to the viewpoint position P1.

[0067] As a result, in the latter half of the camera path, that is, at each viewpoint position from the midpoint Pm onward, the user who is the viewer can keep watching the target T1 in the free-viewpoint video until the virtual camera VC11 reaches the viewpoint position P1 which is the end point. As a result, the user can view a satisfactory free-viewpoint video.

[0068] Additionally, in this case, while the virtual camera VC11 moves from the midpoint Pm to the viewpoint position P1, the virtual camera VC11 images the target T1 from various angles. Hence, the user can observe the target T1 from various angles in the free-viewpoint video. As a result, the satisfaction level of the free-viewpoint video can be further improved.

[0069] Here, the determination of the movement path of the virtual camera when generating the camera path will be described.

[0070] For example, as indicated by an arrow Q11 in FIG. 3, assume that a camera path is generated in which the virtual camera VC11 is moved linearly from the viewpoint position P0 indicated by an arrow W31 to the viewpoint position P1 indicated by an arrow W32. In FIG. 3, the same reference numerals are given to the parts corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate.

[0071] In the example indicated by the arrow Q11, a straight line CP11 connecting the viewpoint position P0 and the viewpoint position P1 indicates the movement path included in the camera path of the virtual camera VC11. However, in this example, the straight line CP11 intersects the target T0, and the virtual camera VC11 will collide with the target T0 when the virtual camera VC11 moves.

[0072] Hence, for example, as indicated by an arrow Q12, a camera path is generated by assuming that a repulsion acts on the virtual camera VC11 from an object such as the target T0, that is, the virtual camera VC11 receives a repulsive force from the object such as the target T0.

[0073] In such a case, a model related to the repulsive force received by the virtual camera VC11 is prepared in advance for each object such as the target T0. When generating the camera path, the model related to repulsive force is used to obtain the camera path of the virtual camera VC11, more specifically, the movement path of the virtual camera VC11.

[0074] With this configuration, the movement speed of the virtual camera VC11 at the viewpoint position P0 or the like is appropriately adjusted, and the movement path is adjusted so that the virtual camera VC11 moves at a position a certain distance away from the object such as the target T0. As a result, for example, a movement path CP12 in which the viewpoint position P0 and the viewpoint position P1 are smoothly connected by a curve can be obtained.

[0075] Here, a more specific description will be given of generation of the camera path when the virtual camera VC11 receives a repulsion from an object such as the target T0, that is, when a model related to repulsive force is used.

[0076] For example, as shown in FIG. 4, assume that the target T0 and the target T1 exist in a space, and a camera path for moving from the viewpoint position P0 to the viewpoint position P1 is generated. Note that in FIG. 4, the same reference numerals are given to the parts corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate.

[0077] Here, the position indicated by an arrow ST11 is the viewpoint position P0 which is the start point of the camera path, and at the viewpoint position P0, the rotation angle of the virtual camera VC11 is R0. Additionally, the position indicated by an arrow ED11 is the viewpoint position P1 which is the end point of the camera path, and at the viewpoint position P1, the rotation angle of the virtual camera VC11 is R1.

[0078] Moreover, assume that the virtual camera VC11 moves from the viewpoint position P0 to the viewpoint position P1 while passing through a position at least a distance L away from a main object such as a human being. In this example, the main objects are the target T0 and the target T1.

[0079] In this case, the distance L to be separated from the target T0 and the target T1 is first determined. For example, the distance L may be predetermined, or may be determined from the sizes of the target T0 and the target T1 or the focal length of the virtual camera VC11. Such a distance L corresponds to a model related to repulsive force.

[0080] Next, a straight line connecting the viewpoint position P0 and the viewpoint position P1 is obtained as a path PS1, and the point M0 closest to the target on the path PS1 is searched. Here, of the target T0 and the target T1, the target T0 is located closer to the path PS1, and therefore the point (position) closest to the target T0 on the path PS1 is set as a point M0.

[0081] Moreover, when the point M0 is moved in a direction perpendicular to the path PS1 to a position separated from the target T0 by the distance L, the position after the movement is defined as a position M1. Then, the viewpoint position P0, the position M1, and the viewpoint position P1 are smoothly connected by a curve (path) such as a Bezier curve so that the curvature becomes continuous, and the resulting curve PS2 is defined as the movement path of the virtual camera VC11 that moves from the viewpoint position P0 to the viewpoint position P1. That is, the curve PS2 is the movement path included in the camera path of the virtual camera VC11.

[0082] In this case, the virtual camera VC11 moves from the viewpoint position P0 to the viewpoint position P1 through the position M1 while maintaining the state in which the distance from the object such as the target T0 is a certain distance L or more.

[0083] Note that there are many objects such as the target T0 in the space, and it may not be possible to appropriately determine the movement path by the above method. In such a case, for example, the movement path of the virtual camera VC11 may be determined so that the distance L is maintained with respect to at least the target T0 and the target T1, and in the actual free-viewpoint video, objects other than the target T0 and the target T1 may be displayed semi-transparently.

[0084] By generating the camera path as described above, the virtual camera VC11 can be moved from the viewpoint position P0 to the viewpoint position P1 while maintaining an appropriate distance from the objects such as the target T0 and the target T1. As a result, the virtual camera VC11 can be moved so as to go around the target T0 and the target T1 which are the targets of interest, and the user can observe the target T0 and the target T1 well from various angles in the free-viewpoint video.

[0085] Moreover, with the method shown in FIGS. 5 and 6, for example, the camera path can be generated easier than the case of using the model related to repulsive force. Note that in FIGS. 5 and 6, the same reference numerals are given to the parts corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate. Additionally, in FIG. 6, the same reference numerals are given to the parts corresponding to those in FIG. 5, and the description thereof will be omitted as appropriate.

[0086] In the example shown in FIG. 5, as indicated by an arrow Q21, an intermediate point M0 of a straight line L11 connecting the viewpoint position P0 which is the start point of the movement path of the virtual camera VC11 and the viewpoint position P1 which is the end point of the movement path is obtained.

[0087] Then, the intermediate point M0 is moved in a direction substantially perpendicular to the straight line L11 to a position sufficiently distant from the intermediate point M0, that is, to a position sufficiently distant from the target T0 and the target T1 so that the distance is equal to or greater than a predetermined distance. The position after the movement is defined as the midpoint Pm.

[0088] For example, the midpoint Pm is a position where the target T0 and the target T1 are included in the angle of view of the virtual camera VC11 when the virtual camera VC11 is arranged at the midpoint Pm at a predetermined rotation angle.

[0089] When the midpoint Pm is determined in this way, a curve L12 that smoothly connects the viewpoint position P0, the midpoint Pm, and the viewpoint position P1 is obtained as indicated by an arrow Q22, and the obtained curve L12 is defined as the movement path of the virtual camera VC11 included in the camera path.

[0090] In particular, when obtaining the movement path of the virtual camera VC11, a speed obtained by combining the original movement speed of the virtual camera VC11 and the speed at which the virtual camera VC11 heads for the destination, that is, the speed at which the virtual camera VC11 moves from the viewpoint position P0 to the viewpoint position P1 is defined as the movement speed of the virtual camera VC11 at each position on the moving path.

[0091] Specifically, for example, an arrow MV11 represents the original movement speed of the virtual camera VC11 at the viewpoint position P0, and this movement speed is the speed of the virtual camera VC11 at the viewpoint position P0 when the virtual camera VC11 moves from another position to the viewpoint position P0.

[0092] Additionally, an arrow MV12 represents the speed at which the virtual camera VC11 moves to the viewpoint position P1 which is the destination, and this speed is obtained by the video viewing system on the basis of the viewpoint position P0, the viewpoint position P1, and the like.

[0093] When generating the camera path, the movement speed represented by the arrow MV11 and the movement speed represented by the arrow MV12 as described above are combined, and the combined movement speed is defined as the movement speed of the virtual camera VC11 at the viewpoint position P0 in the camera path. In FIG. 5, an arrow MV13 represents the movement speed obtained by combining the movement speed represented by the arrow MV11 and the movement speed represented by the arrow MV12.

[0094] Additionally, for example, as indicated by an arrow Q31 in FIG. 6, when the subject of interest is switched from the target T0 to the target T1, and the virtual camera VC11 is rotated evenly from the start point to the end point of the camera path, that is, at a constant angular velocity, there is a timing when neither the target T0 nor the target T1 is included in the angle of view of the virtual camera VC11.

[0095] In the example indicated by an arrow Q31, the position indicated by an arrow W11 is the viewpoint position P0, and the position indicated by an arrow W12 is the viewpoint position P1. When the subject of interest is switched from the target T0 to the target T1, the virtual camera VC11 moves from the viewpoint position P0 to the viewpoint position P1. At this time, for example, at the position indicated by an arrow W13, neither the target T0 nor the target T1 is included in the angle of view of the virtual camera VC11.

[0096] Hence, for example, as indicated by an arrow Q32, when the subject of interest is switched from the target T0 to the target T1, a camera path is generated so as to enable viewing of both the old and new subjects of interest, the target T0 and the target T1.

[0097] For example, assume that from a state in which the subject of interest is the target T0, the target T1 is designated as a new subject of interest by a user input operation or the like.

[0098] In this case, assume that the viewpoint position and rotation angle of the virtual camera VC11 are P0 and R0 at the start point of the camera path, and the viewpoint position and rotation angle of the virtual camera VC11 are P1 and R1 at the end point of the camera path. Here, the position indicated by an arrow W41 is the viewpoint position P0, and the position indicated by an arrow W42 is the viewpoint position P1.

[0099] In the example indicated by the arrow Q32, too, the midpoint Pm is determined in a similar manner as in FIG. 5, for example. In particular, here, the position indicated by an arrow W43 is the midpoint Pm.

[0100] The midpoint Pm is a position where the distance from the target T0 and the distance from the target T1 are equivalent, and the target T0 and the target T1 are included in the angle of view of the virtual camera VC11 when the virtual camera VC11 is placed at the midpoint Pm.

[0101] When the midpoint Pm is determined in this way, a camera path is obtained in which the movement path is a curve that smoothly connects the viewpoint position P0, the midpoint Pm, and the viewpoint position P1. In the part indicated by the arrow Q32, a curve L31 represents the movement path of the virtual camera VC11 included in the camera path.

[0102] Here, during the first half of the movement of the virtual camera VC11 according to the camera path, that is, when moving from the viewpoint position P0 to the midpoint Pm, the movement path, movement speed, rotation angle, and rotation speed of the virtual camera VC11 are determined so that at least the target T0 is continuously kept within the angle of view of the virtual camera VC11. In particular, when the virtual camera VC11 is in the vicinity of the midpoint Pm, both the target T0 and the target T1 are included in the angle of view of the virtual camera VC11.

[0103] Additionally, during the latter half of the movement of the virtual camera VC11 according to the camera path, that is, when moving from the midpoint Pm to the viewpoint position P1, the movement path, movement speed, rotation angle, and rotation speed of the virtual camera VC11 are determined so that at least the target T1 is virtually kept within the angle of view of the virtual camera VC11.

[0104] As a result, when the viewpoint is moved, that is, when the virtual camera VC11 is moved, the user who views the free-viewpoint video generated according to the camera path can see the target T0 in the first half of the movement, see both the target T0 and the target T1 in the middle of the movement, and see the target T1 in the latter half of the movement.

[0105]

[0106] Incidentally, if the camera path is generated as described above, when the subject of interest changes from the target T0 to the target T1, the target T0 or the target T1 stays in the field of view, that is, the angle of view of the virtual camera even while the virtual camera is moving. As a result, it is possible to continue to present a meaningful video as a free-viewpoint video.

[0107] However, in a case where the rotation angle of the virtual camera changes significantly in the camera path, that is, in a case where the rotation speed, which is the angular velocity of the rotation of the virtual camera, is large, visually induced motion sickness may occur.

[0108] Specifically, for example, when a user wears an HMD and is viewing a free-viewpoint video, assume that the viewpoint position of the virtual camera is moved independently of the movement of the user’s head. In such a case, the visually induced motion sickness that occurs when the virtual camera is rotated becomes larger than that when the virtual camera is moved in translation. In particular, if the virtual camera rotates largely while the viewpoint position and the target of interest are close to each other, the visually induced motion sickness becomes even more severe.

[0109] Accordingly, when generating a camera path, it is desirable that the virtual camera does not rotate at a certain rotation speed (angular velocity) or higher when the viewpoint is moved.

[0110] Hence, in the present technology, it is possible to reduce visually induced motion sickness by generating a camera path so that the rotation speed of the virtual camera is equal to or less than a predetermined threshold value th. That is, the visual load of the video can be reduced.

[0111] Specifically, assume that the absolute amount of rotation, more specifically, the upper limit of the rotation speed of the virtual camera when changing from the state ST0 to the state ST1 is determined so that the rotation speed becomes equal to or less than the threshold value th, for example. In such a case, a camera path is generated as shown in FIG. 7, for example. Note that In FIG. 7, the same reference numerals are given to the parts corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate.

[0112] For example, as indicated by an arrow Q41 in FIG. 7, assume that the target T0 and the target T1 are in a space, and a camera path is generated in which the angle of view of the virtual camera VC11 changes from the state in which the target T0 is included in the angle of view to the state in which the target T1 is included in the angle of view. In other words, the angle of view of the virtual camera VC11 is changed from the angle of view including the target T0 to the angle of view including the target T1.

[0113] At this time, assume that the movement of the virtual camera VC11 is completed in one second, and the average value of the rotation speed of the virtual camera VC11 is 30 degrees/second at the maximum. That is, assume that the threshold value th=30 degrees/sec. For example, the threshold value th is determined on the basis of whether or not visually induced motion sickness occurs. If the average rotation speed of the virtual camera VC11 is equal to or less than the threshold value th, the camera work is less likely to cause visually induced motion sickness.

[0114] Moreover, assume that in the angle of view of the virtual camera VC11, when the distance from the target T1 to the virtual camera VC11 is L, the target T1 appears in an appropriate size on the free-viewpoint video.

[0115] Moreover, at the start point of the camera path, the virtual camera VC11 is at the position indicated by an arrow W51, and that position is the viewpoint position P0.

[0116] Additionally, assume that when the virtual camera VC11 is at the viewpoint position P0, the rotation angle of the virtual camera VC11 is R0=0 degrees.

[0117] From such a state, when a new target T1 is designated, the state at the end point of the camera path, that is, the viewpoint position P1 and the rotation angle R1 of the virtual camera VC11 after movement are determined, so that the target T1 in an appropriate size is included in the angle of view.

[0118] In the example indicated by the arrow Q41, the position indicated by an arrow W52 is the viewpoint position P1. For example, the viewpoint position P1 is a position separated from the new target T1 by a distance L.

[0119] Additionally, the rotation angle R1 of the virtual camera VC11 at the viewpoint position P1 is, for example, a rotation angle at which the target T1 can be captured (imaged) from substantially the front by the virtual camera VC11. For example, the rotation angle R1 is determined on the basis of the orientation and the like of the target T1. As a specific example, the rotation angle R1 can be determined so that the angle formed by the front direction seen from the target T1 and the optical axis of the virtual camera VC11 is equal to or less than a predetermined threshold value, for example.

[0120] Here, assuming that the determined rotation angle R1 is 60 degrees, the rotation angle of the virtual camera VC11 changes by 60 degrees before and after the movement from the viewpoint position P0 to the viewpoint position P1. That is, the virtual camera VC11 rotates by 60 degrees.

[0121] In this case, if the movement from the viewpoint position P0 to the viewpoint position P1 is to be completed in one second, the average rotation speed of the virtual camera VC11 becomes 60 degrees/second, which is larger than the threshold value th. That is, the camera work is likely to cause visually induced motion sickness.

[0122] Hence, the viewpoint position P1 and the rotation angle R1 after the movement of the virtual camera VC11 are recalculated so that the average rotation speed of the virtual camera VC11 is equal to or less than the threshold value th. In the following, the recalculated, that is, redetermined, viewpoint position P1 and rotation angle R1 of the virtual camera VC11 will be denoted by a viewpoint position P1’ and a rotation angle R1’ in particular.

[0123] When obtaining the viewpoint position P1’ and the rotation angle R1’, the rotation angle R1’ is first obtained so that the average rotation speed of the virtual camera VC11 is equal to or less than the threshold value th. Here, rotation angle R1’=30 degrees, for example.

[0124] Then, as indicated by an arrow Q42, with respect to the rotation angle R1’, a position where the virtual camera VC11 can capture (image) the target T1 from an appropriate angle such as substantially the front and the distance from the target T1 is L is obtained as the viewpoint position P1’, for example. At this time, for example, a position separated from the target T1 by a distance L in the direction opposite to the rotation angle R1’ can be defined as the viewpoint position P1’. In the example indicated by the arrow Q42, the position indicated by an arrow W53 is the viewpoint position P1’.

[0125] According to the viewpoint position P1’ and the rotation angle R1’ redetermined in this way, the rotation of the virtual camera VC11 before and after the movement, that is, the change in the rotation angle is curbed to 30 degrees. As a result, the average rotation speed of the virtual camera VC11 becomes 30 degrees/sec, which is equal to or less than the threshold value th. Thus, it is possible to achieve camera work that is less likely to cause visually induced motion sickness.

[0126] When the viewpoint position P1’ and the rotation angle R1’ after movement are determined, a camera path of the virtual camera VC11 that changes from the viewpoint position P0 and the rotation angle R0 to the viewpoint position P1’ and the rotation angle R1’ is generated. At this time, as described with reference to FIGS. 3, 4, 5, and 6, for example, the movement path and the movement speed of the virtual camera VC11 are determined so that the virtual camera VC11 moves from the viewpoint position P0 to the viewpoint position P1’.

[0127] Note that while it has been described in this example that the movement of the virtual camera VC11 is completed in one second, how many seconds it takes to complete the movement of the virtual camera VC11 may be appropriately determined according to the distance or the like between the target T0 and the target T1, for example.

[0128] Note, however, that it is desirable to complete the movement of the virtual camera VC11 in as short a time as possible. For example, the user designates a new target T1 and instructs the movement of the viewpoint position because there is some event that the user is interested in and he/she wants to see the target T1 related to the event. For example, in an actual sports game, the duration of the event is not long, so it is necessary to complete the movement of the virtual camera VC11 in a short time.

[0129] On the other hand, if the movement speed of the virtual camera VC11 is too fast, the user may not be able to grasp his/her position in the space, that is, the viewpoint position of the virtual camera VC11, or may have visually induced motion sickness. Hence, it is necessary to generate a camera path in which the movement is completed within a certain time, visually induced motion sickness is unlikely to occur, and the user can easily grasp his/her position and movement direction. Against this background, in the present technology, the camera path is generated so that the movement is completed in a short time and the average rotation speed of the virtual camera VC11 is equal to or less than the threshold value th.

[0130]

[0131] Subsequently, a configuration example of a video viewing system that generates a camera path as shown in FIG. 7 will be described. Such a video viewing system is configured as shown in FIG. 8, for example.

[0132] The video viewing system shown in FIG. 8 includes an information processing device 11, a display unit 12, a sensor unit 13, and a content server 14.

[0133] Here, for example, the information processing device 11 may include a personal computer, a game machine body, or the like, and the display unit 12 and the sensor unit 13 may include an HMD, or the information processing device 11 to the sensor unit 13 may be used to form an HMD or a smartphone.

[0134] Alternatively, the display unit 12 can include a television. Moreover, at least one of the display unit 12 or the sensor unit 13 may be provided in the information processing device 11. Note that in the following, the description will be continued assuming that the user who views the free-viewpoint video is wearing the display unit 12 and the sensor unit 13.

[0135] The information processing device 11 acquires content data for generating a free-viewpoint video from the content server 14, and also generates image data of the free-viewpoint video according to the output of the sensor unit 13 on the basis of the acquired content data to supply the image data to the display unit 12.

[0136] The display unit 12 has a display device such as a liquid crystal display, and reproduces a free-viewpoint video on the basis of the image data supplied from the information processing device 11.

[0137] The sensor unit 13 includes, for example, a gyro sensor, a TOF sensor, a camera, or the like for detecting the user’s posture, head orientation, line-of-sight direction, and the like, and supplies the information processing device 11 with output of the gyro sensor or the TOF sensor, an image captured by the camera, or the like as sensor output.

[0138] The content server 14 holds as content data a group of image data of contents imaged from different viewpoints used for generating (constructing) a free-viewpoint video, and supplies the content data to the information processing device 11 in response to a request by the information processing device 11. That is, the content server 14 functions as a server that distributes free-viewpoint video.

[0139] Additionally, the information processing device 11 has a content data acquisition unit 21, a detection unit 22, an input acquisition unit 23, and a control unit 24.

[0140] The content data acquisition unit 21 acquires content data from the content server 14 according to the instruction of the control unit 24 and supplies the content data to the control unit 24. For example, the content data acquisition unit 21 acquires content data from the content server 14 by communicating with the content server 14 through a wired or wireless communication network. Note that the content data may be acquired from a removable recording medium or the like.

[0141] The detection unit 22 detects the posture, head orientation, and line-of-sight direction of the user wearing the display unit 12 and the sensor unit 13 on the basis of the sensor output supplied from the sensor unit 13, and supplies the detection result to the control unit 24.

[0142] For example, the detection unit 22 detects the user’s posture and head orientation on the basis of the output of the gyro sensor or the TOF sensor as sensor output. Additionally, for example, the detection unit 22 detects the user’s line-of-sight direction on the basis of an image as sensor output captured by the camera.

[0143] The input acquisition unit 23 includes, for example, a mouse, a keyboard, buttons, switches, a touch panel, a controller, and the like, and supplies the control unit 24 with a signal corresponding to a user’s operation on the input acquisition unit 23. For example, the user performs a new target T1 designation operation or the like by operating the input acquisition unit 23.

[0144] The control unit 24 includes a central processing unit (CPU), a random access memory (RAM), and the like, for example, and controls the operation of the entire information processing device 11.

[0145] For example, the control unit 24 determines the display range of the free-viewpoint video by controlling the movement and rotation of the virtual camera, and generates image data of the free-viewpoint video according to the determination. Here, determining the camera path of the virtual camera corresponds to controlling the movement and rotation of the virtual camera.

[0146] Specifically, the control unit 24 generates a camera path of the free-viewpoint video on the basis of the detection result supplied from the detection unit 22 and the signal supplied from the input acquisition unit 23. Additionally, for example, the control unit 24 instructs the content data acquisition unit 21 to acquire content data, or generates image data of the free-viewpoint video on the basis of the generated camera path and the content data supplied from the content data acquisition unit 21 to supply the image data to the display unit 12.

[0147]

……
……
……