空 挡 广 告 位 | 空 挡 广 告 位

Sony Patent | Free-viewpoint method and system

小编 映维   |   分类：Sony   |  

Patent: Free-viewpoint method and system

Drawings: Click to check drawins

Publication Number: 20210074023

Publication Date: 20210311

Applicant: Sony

Assignee: Sony Interactive Entertainment Inc.

Abstract

A method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, comprises: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine a camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

Claims

  1. A method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, the method comprising: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine at least one camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; and identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

  2. A method according to claim 1, wherein the data structure is a hierarchical data structure, each successive layer within the hierarchical data structure corresponding to a successive sub-division of the voxels in the voxel representation, at least some of the voxels in the voxel representation being divided into sub-voxels; the method further comprising, identifying, for a sub-voxel within the voxel identified as being within the field of view of the virtual camera, a corresponding sub-voxel surface that is oriented towards the virtual camera; and identifying, based on the accessed data structure, a corresponding camera pair identifier associated with the identified sub-voxel surface, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the camera pair identified for the sub-voxel surface.

  3. A method according to claim 2, comprising iterating over at least some of the layers in the hierarchical data structure so as to identify a camera pair identifier associated with at least some of the sub-voxel surfaces forming the voxel identified as being within the field of view of the virtual camera; wherein the number of iterations is determined based on the intrinsics and/or extrinsics of the virtual camera.

  4. A method according to claim 1, wherein at least some of the voxel surfaces are associated with one or more respective camera pair identifiers and corresponding weightings, the weighting for each camera pair identifier providing an indication of the depth resolution that can be obtained for the corresponding camera pair; and wherein the method further comprises: identifying, for the voxel surface identified as being oriented towards the virtual camera, a corresponding camera pair identifier having the highest weighting relative to the other camera pair identifiers associated with the voxel surface; and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the pair of cameras identified as having the highest weighting associated therewith.

  5. A method according to claim 1, wherein the data structure comprises a look-up table, each entry in the look-up table defining a respective voxel of the voxel representation.

  6. A method according to claim 1, wherein at least some of the voxel surfaces defined in the voxel representation are further associated with a respective parameter indicating at least one of: i. an averaged pose of one more camera pairs providing depth coverage of the corresponding voxel surface; ii. an epipolar diagonal of one or more camera pairs providing depth coverage of the corresponding voxel surface; and iii. the principal points of one or more camera pairs providing depth coverage of the corresponding voxel surface.

  7. A method according to claim 1, further comprising: obtaining the extrinsics and intrinsics of each camera positioned around the scene; determining, based on the obtained extrinsics and intrinsics of the cameras, a depth coverage of respective camera pairs, the depth coverage of a respective camera pair corresponding to the field of view of a depth camera that the depth camera pair corresponds to; and generating the data structure by dividing the scene into a plurality of voxels based on the determined depth coverage, at least some of the voxels corresponding to voxels for which depth data can be obtained for one or more surfaces of those voxels.

  8. A method according to claim 7, wherein generating the data structure comprises: determining, for each voxel, a respective weighting associated with at least one camera pair providing depth coverage of that voxel; wherein the weighting for each camera pair is determined based on at least one of the epipolar diagonal and principal points associated with that camera pair; determining, for respective surfaces of each voxel, a camera pair that is oriented towards that surface; and assigning, to each voxel surface, a camera pair identifier that corresponds to the camera pair determined as having the highest weighting and being oriented towards that voxel surface.

  9. A method according to claim 7, wherein generating the data structure comprises: determining whether the distance between the principal points of a camera pair providing depth coverage of a voxel is less than a dimension of the voxel, and based on a positive determination, sub-dividing the voxel further; and determining for the sub-voxel, a camera pair having the highest weighting and that is oriented towards a surface of the sub-voxel.

  10. A method according to claim 7, wherein dividing the volume into voxels comprises dividing the space in accordance with one or more spherical harmonic functions, the size of each voxel being inversely proportional to the order of the spherical harmonic function used to define that voxel.

  11. A method according to claim 1, comprising: detecting an object as occluding at least some of the scene for at least one of the cameras positioned around the scene; identifying one or more voxel surfaces in the data structure as corresponding to the parts of the scene occluded by the object; determining whether any of the camera pair identifiers associated with the one or more occluded voxel surfaces correspond with the at least one camera for which at least some of the scene is occluded; and in response to a positive determination, filtering from the data structure, for the occluded surfaces, the camera pair identifiers determined as corresponding with the camera for which at least part of the scene is occluded by the object.

  12. A method according to claim 1, comprising displaying the generated reconstruction of the scene from the viewpoint of the virtual camera at a display.

  13. A computer readable medium having computer executable instructions adapted cause a computer system to perform a method of generating a 3D reconstruction of a scene, the scene comprising a plurality of cameras positioned around the scene, the method comprising: obtaining the extrinsics and intrinsics of a virtual camera within a scene; accessing a data structure so as to determine at least one camera pair that is to be used in reconstructing the scene from the viewpoint of the virtual camera; wherein the data structure defines a voxel representation of the scene, the voxel representation comprising a plurality of voxels, at least some of the voxel surfaces being associated with respective camera pair identifiers; wherein each camera pair identifier associated with a respective voxel surface corresponds to a camera pair that has been identified as being suitable for obtaining depth data for the part of the scene within that voxel and for which the averaged pose of the camera pair is oriented towards the voxel surface; identifying, based on the obtained extrinsics and intrinsics of the virtual camera, at least one voxel that is within the field of view of the virtual camera and a corresponding voxel surface that is oriented towards the virtual camera; and identifying, based on the accessed data structure, at least one camera pair that is suitable for reconstructing the scene from the viewpoint of the virtual camera, and generating a reconstruction of the scene from the viewpoint of the virtual camera based on the images captured by the cameras in the identified at least one camera pair.

  14. A free-viewpoint system, comprising: a scene unit configured to obtain the intrinsics and extrinsics of a plurality of cameras arranged to capture a scene and based thereon, generate a voxel representation of the scene; wherein the voxel representation comprises a plurality of voxels and defines for at least some surfaces of the voxels, respective camera pairs that have been identified as being oriented towards those voxel surfaces and that are suitable for obtaining depth data for those voxel surfaces; a virtual camera unit operable to obtain the extrinsics and intrinsics of a virtual camera; and a mapping unit operable to map the extrinsics and intrinsics of the virtual camera to at least one voxel surface defined in a data structure and to identify at least one camera pair suitable for obtaining depth data for that voxel surface based on the corresponding camera pair identifier; and an image generator operable to receive an input from the mapping unit, and in response to said input, obtain images captured by the cameras in the at least one camera pair identified as being suitable for reconstructing the scene from the viewpoint of the virtual camera, and to generate an image of the scene from the viewpoint of the virtual camera based on the obtained images.

  15. A free-viewpoint system according to claim 14, wherein the voxel representation defines, for each camera pair identifier associated with a respective voxel surface, a corresponding weighting indicating the resolution of depth data that can be obtained by the corresponding camera pair.

  16. A free-viewpoint system according to claim 14, wherein the voxel representation corresponds to a hierarchical data structure, each successive layer of the hierarchical data structure corresponding to a sub-division of a respective voxel; wherein at least some of the surfaces of the sub-voxels are associated with respective camera pair identifiers.

  17. A free-viewpoint system according to claim 16, wherein the camera pair identifiers associated with the sub-voxel surfaces are associated with a weighting indicative of the resolution of depth data that can be obtained for that camera pair.

  18. A free-viewpoint system according to claim 16, wherein the mapping unit is configured to map the extrinsics and intrinsics of the virtual camera to at least one sub-voxel surface and to identify, based on the camera pair identifier associated with that sub-voxel surface, a corresponding camera pair suitable for obtaining depth data for the sub-voxel surface.

  19. A free-viewpoint system according to claim 16, wherein the scene unit is configured to determine, based on the obtained extrinsics and intrinsics of the cameras, a depth coverage for each respective camera pair; the scene unit being configured to generate the voxel representation by dividing the scene into voxels based on the determined depth coverage.

  20. A free-viewpoint system according to claim 19, wherein the scene unit is configured to determine, based on the obtained extrinsics and intrinsics of the cameras: i. a weighting associated with each camera pair providing depth coverage of a respective voxel, the weighting providing an indication of the depth resolution of the corresponding camera pair; ii. a directionality of each camera pair providing depth coverage of a respective voxel relative to one or more surfaces of that voxel; and wherein the scene unit is configured to determine, for each voxel surface, a camera pair that is oriented towards that voxel surface and has the highest weighting relative to the other camera pairs providing depth coverage of the corresponding voxel, and to assign to that voxel surface, a corresponding camera pair identifier.

  21. A free-viewpoint system according to claim 14, further comprising: a controller operable to communicate with each of the plurality of cameras, the controller being configured to receive an input from the scene unit, and in response to said input, determine whether the number of voxel surfaces associated with respective camera pair identifiers exceeds a threshold value; and wherein the controller is configured to adjust at least one of the extrinsics and or intrinsics of one or more cameras, based on a determination that the number of voxel surfaces associated with respective camera pair identifiers is less than the threshold value.

  22. A free-viewpoint system according to claim 21, wherein the voxel representation defines, for at least some surfaces of the voxels, an indication of the resolution of depth data that can be obtained for the respective camera pair identifiers associated with at least some voxel surfaces; and wherein the controller is configured to adjust at least one of the intrinsics and extrinsics of one or more cameras, based on a determination that the depth resolution associated with one or more camera pairs is less than a threshold depth resolution.

  23. A free-viewpoint system according to claim 14, comprising a display operable to display images generated by the image generator.

  24. A free-viewpoint system according to claim 23, wherein the display comprises a head-mountable display; and wherein the virtual camera unit comprises a pose detector operable to detect a pose of the head-mountable display, the pose of the head-mountable display corresponding to the pose of the virtual camera for which the reconstruction of the scene from the viewpoint of the virtual camera is to be generated by the image generator.

  25. A free-viewpoint system according to claim 14, comprising: a server operable to communicate with a client device, the server comprising the scene unit, mapping unit and image generator; a client device operable to communicate with the server; wherein the server is operable to communicate with the plurality of cameras and to receive images captured by the cameras; wherein the client device comprises the virtual camera unit and is operable to provide the extrinsics and intrinsics of the virtual camera to the server; wherein the server is configured to generate an image of the scene from the viewpoint of the virtual camera and to transmit the generated image to the client device.

Description

BACKGROUND OF THE INVENTION

Field of the Invention

[0001] The present disclosure relates to a free-viewpoint method and system.

Description of the Prior Art

[0002] The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.

[0003] In recent years, driven at least in part by the improvements made in display technology, there has been an increase in the demand for interactive content that is able to offer an immersive experience to a user. For example, the increase in the number and quality of virtual reality (VR) and augmented reality (AR) devices lends itself to the provision of immersive experiences, while the development of televisions and other display devices that offer increased resolution, refresh rate, and colour reproduction (for example) also contributes to this.

[0004] While video games may be provided that can offer such an experience, the approaches taken to provide immersiveness in video games may not be applicable to captured video content such as movies or sports events. For example, when generating video game content it is common that the locations and properties of all objects in the environment are known and other features, such as lighting information, are also able to be calculated. Such information is often not available for captured video content, and therefore techniques applicable to video games to enable the provision of more immersive content are not considered to be widely applicable.

[0005] One example of captured video content that is adapted for increased immersion of a user is that of three-dimensional video. Consumer devices are available that are operable to display content that may be viewed (often aided by a corresponding set of glasses that are configured to enable the viewing of three-dimensional content) in a manner that causes the user to perceive the content as having significant depth despite the use of a two-dimensional display.

[0006] However, one drawback with such systems is that the viewpoint that is adopted by the user is often pre-defined (such as tied to the camera position in a movie) or severely limited (such as allowing a user to switch between a number of such pre-defined viewpoints).

[0007] This may serve to reduce the level of immersion that is experienced by the user when viewing the content, particularly in a VR context, as despite appearing three-dimensional there is no corresponding motion of the viewpoint as the user moves their head as would be expected were the user to move their head when viewing real-world content. The resulting disconnect between the viewpoint and the user’s motion can lead to a sense of discomfort for the user, in addition to the loss of immersion.

[0008] Similarly, the restrictions placed upon the viewpoint location may be made more noticeable when a user is provided with more immersive content, as the user may be more inclined to try and explore the displayed environment. This can lead to the user attempting to relocate the viewpoint to a desired location in the virtual environment, and becoming frustrated when such a relocation is not possible within the constraints of the provided content.

[0009] It is in view of the above considerations that free viewpoint systems have been developed. The object of such systems is to provide content which a user is able to navigate freely, such that a viewpoint may be selected freely (or at least substantially so) within a virtual environment and a corresponding view is able to be provided to a user. This can enable a user to navigate between any number of viewpoints within the virtual environment, and/or for multiple users to occupy corresponding preferred viewpoints within the virtual environment.

[0010] A number of challenges exist when seeking to provide high-quality image or video content with a free viewpoint. A number of such problems derive from the limitations of the content capturing systems that are used; for example, it may be difficult to capture sufficient image information due to occlusions or the like. In addition to this, information that may be required to generate additional viewpoints (such as lighting information or depth information) may be difficult to derive based upon the captured image information. Similarly, limitations of the image capturing arrangement may lead to noisy data being obtained due to a lack of precision; such data may not be suitable for reproduction.

[0011] While a number of the problems associated with these issues can be mitigated by the inclusion of a greater number of cameras (or other sensors), this can be rather impractical in many cases. Similarly, addressing these issues by simply increasing the amount of processing that is applied can also be problematic, particularly when live content is being provided. It is therefore considered that alternative modifications to the free viewpoint content generating may be advantageous.

[0012] In some free viewpoint systems, a plurality of cameras are positioned around a scene and depth data is obtained from stereo-images captured by respective pairs of cameras. The colour information obtained from images is combined with the obtained depth data and used to generate a 3D reconstruction of the scene. Typically, the colour information provides the x and y coordinates of individual pixels, and the depth information provides the corresponding z-coordinates of those pixels.

[0013] One known technique for generating 3D reconstructions involves iterating over every camera pair so as to extract any meaningful volumetric data from the stereoscopic images captured by the respective camera pairs. However, in some situations, this may result in the comparison of images having no common image data and from which no depth data can be obtained. For example, such may be the case if two cameras are facing each other from opposite ends of a stadium. Moreover, in some situations, it may be that depth data is obtained for a given camera pair, for a given part of the scene, but in actuality, there is a different camera pair better suited for obtaining depth data for that part of the scene. For example, it may be that a different camera pair provides a higher depth resolution for that part of the scene. In any case, the indiscriminate processing of images captured by each camera pair is an inefficient use of time and processing power. The present disclosure seeks to address or at least alleviate these problems.

SUMMARY OF THE INVENTION

[0014] The present disclosure is defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0016] FIG. 1 shows schematically a plurality of cameras arranged to capture images of a scene;

[0017] FIG. 2 shows schematically the epipolar diagonal associated with a pair of cameras;

[0018] FIG. 3 shows schematically an example of a voxel representation of a scene;

[0019] FIG. 4 shows schematically an example of the principal points associated with a pair of cameras;

[0020] FIG. 5 shows schematically an example of a voxel and the averaged pose of a camera pair relative to the voxel;

[0021] FIG. 6 shows schematically an example of a look-up table corresponding to a voxel representation of a scene;

[0022] FIG. 7 shows schematically an example of an octree representing the scene;

[0023] FIG. 8 shows schematically an example of a capture volume divided into voxels in accordance with a kd-tree;

[0024] FIG. 9 shows schematically an example of a hemispherical voxel representation of the scene;

[0025] FIG. 10 illustrates an example of a method for generating a voxel representation of the scene in accordance with the present disclosure;

[0026] FIG. 11 illustrates an example of a method for generating a reconstruction of a scene from the viewpoint of a virtual camera, in accordance with the present disclosure;

[0027] FIG. 12 shows schematically an example of a scene in which one or more voxels representing the scene are occluded by an object;

[0028] FIG. 13 shows schematically an example of a free-viewpoint system for generating a reconstruction of a scene in accordance with the present disclosure; and

[0029] FIG. 14 shows schematically an example of a free-viewpoint system for controlling at least one of the intrinsics and extrinsics of one or more cameras in accordance with the present disclosure.

DESCRIPTION OF THE EMBODIMENTS

[0030] It is known in the art that depth data can be obtained from stereoscopic images, i.e. images captured by a respective pair of cameras having at least partially overlapping fields of view. However, for scenes in which multiple cameras provide coverage, it may not be efficient in terms of time and computing power, to attempt to obtain depth data for every camera pair permutation. In some examples, it may be that there is no overlap between the fields of view of a given camera pair, and so an attempt obtain depth data from the corresponding images is futile. Moreover, in some examples, it may be that there is a given camera pair that is able to obtain depth data for a given portion of the scene at a greater depth resolution relative to other camera pairs, but that depth data is still extracted from the images captured by the other camera pairs. Again, this results in an inefficient use of time and resources when generating a reconstruction of a scene from a desired viewpoint. The present disclosure provides a data structure that may be used to alleviate at least some of these problems.

Pre-Computation of the Data Structure

[0031] A method of generating a data structure for facilitating the 3D reconstruction of a scene, or region thereof, will now be described. In the below-described method, it is assumed that the extrinsics (position and orientation) and intrinsics (e.g. focal length, principal point) are known for the cameras positioned around the scene.

[0032] At a first step, a map corresponding to areas of overlap between the frustums of camera pairs is generated. Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, an example of this is shown in FIG. 1, where three cameras C1, C2, C3 are shown as being positioned around one end of a capture area. The corresponding frustums 101, 102, 103 of the cameras is shown in two dimensions, from a top-down view. It will be appreciated that these frustums will have a volume, corresponding to the volume of scene that falls within the corresponding camera’s field of view. The frustums for the cameras may be determined using the extrinsics and intrinsics associated with the cameras. In FIG. 1, the capture area may correspond to e.g. a football pitch, and the subject may correspond to a football player.

[0033] In FIG. 1, a shaded region 104 is shown; this region corresponds to the area of overlap between the frustrums of cameras C2 and C3. This region corresponds to a volume of space that is within the field of view of each camera forming the camera pair C23.

[0034] Generating the map of overlapping frustums may involve determining, for each camera pair, an epipolar diagonal associated with the images captured by the cameras in that camera pair. An example of this is shown in FIG. 2, which shows the images 201, 202 captured by a pair of parallel cameras C21, C22 separated by a distance, B (baseline). In FIG. 2, the shaded region 203 corresponds to the region of overlap between the images captured by the respective cameras. Each non-shaded region corresponds to an area of an image captured by one camera that is not within the field of view of the other camera, and from which depth data cannot be obtained by stereo-matching. The optical axis of each camera is indicated with arrows 204, 205 respectively.

[0035] The epipolar diagonal provides an indication of the degree of overlap between the images captured by the cameras in a camera pair. In FIG. 2, the epipolar diagonal is shown as corresponding to the distance between opposing corners of the overlapping image region. Generally, the larger the epipolar diagonal associated with a given camera pair, the greater the amount of depth information that can be obtained from the images captured by that camera pair. An epipolar diagonal of zero may indicate that there is no overlapping field of view for the corresponding camera pair.

[0036] The epipolar diagonal may be determined for each camera pair, with each camera pair corresponding to a different combination of two cameras positioned in and around the capture area. In some examples, the images captured by each camera pair are cropped and rectified, and the epipolar diagonal is determined for each camera pair based on the cropped and rectified images. In FIG. 1, there are three different camera pairs (C1 and C2; C2 and C3; and C1 and C3) each having a respective epipolar diagonal value (EPD12, EPD23, EPD31) associated therewith. In FIG. 1, EPD31 may equal zero, since the cameras are facing each other (i.e. have antiparallel optical axes) and will have no common image data from which stereo-matching can be performed. Alternatively, in such a case a flag or other indicator either that no epipolar diagonal exists or that a pair do not overlap may be used.

[0037] Having determined the extent of overlap between the frustums of each camera pair, the method further comprises defining a volume of space that encloses the capture area, or a particular region of interest within the capture area. This volume may be sub-divided into sub-units, for example, in the form of voxels. The voxels may be sub-divided further, as will be described below. In the present disclosure, the volume enclosing the capture area or region of interest is referred to as the capture volume.

[0038] In some examples, the capture volume may correspond to a polyhedron, such as e.g. a rectangular cuboid that encloses the capture area. An example of this is shown in FIG. 3, which shows the capture area 301 (e.g. a football pitch) enclosed by a cuboid 302. In FIG. 3, the cuboid is shown as being divided into voxels 303 of equal size. Each voxel 303 corresponds to a different segment of 3D space within the capture area.

[0039] The method further comprises determining, for each sub-unit (e.g. voxel) of the volume enclosing the capture area, the corresponding camera pairs that provide coverage of that sub-unit. This may be determined using the map of overlapping camera frustums (generated as above). A pair of cameras are said to provide coverage of a given sub-unit if that sub-unit is within the overlapping fields of view of the cameras in the camera pair. In other words, if it is possible to obtain depth data for at least part of the scene within the sub-unit, for a given camera pair.

[0040] For the camera pairs that provide coverage of a given sub-unit, the distance between the principal points of each camera in a respective pair is determined. The principal point corresponds to the point of intersection between the optical axis and image plane of the camera. This point is usually at the centre of the image plane and corresponds to the point of the scene that is in focus. As mentioned above, the principal points for each camera positioned around (and within) the capture area may be known in advance (i.e. as camera intrinsics).

[0041] The method may comprise determining the mid-point between the principal points for each camera pair that provides coverage of a given sub-unit. FIG. 4 shows an example of the principal points associated with each camera in a camera pair. In FIG. 4, a first camera C41 is shown as having a focal length, f1, and principal point, P1; a second camera C42 is shown as having a focal length, f2, and principal point, P2. The midpoint between the principal points is indicated as midpoint MPP. The midpoint between principal points provides a further indication of the degree of overlap between images. The epipolar diagonal may give the depth coverage and the principal point midpoints may give the depth accuracy from minimizing distortion.

[0042] For each camera pair providing coverage of a given sub-unit, a weighting may be determined for that camera pair, with the weighting being dependent on the epipolar diagonal and principal midpoint associated with the camera pair. The weighting may be proportional to the epipolar diagonal and inversely proportional to the midpoint between principal points of the camera pair. That is, the larger the epipolar diagonal associated with a given camera pair, and the smaller the distance between the principal points, the larger the weighting associated with that camera pair. A larger weighting indicates that more depth information can be obtained for the given sub-volume of the space, by the corresponding camera pair.

[0043] As will be appreciated, the direction that each camera is facing relative to the sub-unit will also need to be considered. For example, it may be that only some of the faces of a given sub-unit are facing towards a camera pair, and so it may only be possible to obtain depth data for the parts of the scene within the voxel that are aligned in the same direction. Thus, it may be useful to encode a directionality associated with each camera pair, relative to the one or more sub-units that are within the field of view of that camera pair. Hence, the method may further comprise determining, for each camera pair, a directionality associated with the overall (e.g. averaged) pose of that camera pair. This directionality may be determined relative to one or more surfaces making up the sub-unit. An example of this is shown schematically in FIG. 5.

[0044] FIG. 5 shows schematically an example of a voxel 500 forming part of the volume enclosing the capture area. In FIG. 5, the voxel 500 corresponds to a cuboid having six faces, with the surface normals, {circumflex over (n)}.sub.1, {circumflex over (n)}.sub.2, shown for two of the faces (corresponding to the front 502A and right 502B faces). Two cameras C51 and C52 are shown with each camera being oriented (at least partially) towards the front face of the voxel. The averaged pose of the camera pair is shown as having position

and being pointed in direction {circumflex over (P)}. In FIG. 5, the direction in which each camera is pointed is shown via the corresponding optical axes OX51 and OX52.

[0045] In FIG. 5, it can be seen that the averaged pose of the camera pair is pointed towards the front face 502A of the voxel. Generally, the closer the averaged pose is to being pointed in a direction that is opposite (i.e. antiparallel) to the surface normal of the front face, the more suitable that camera pair is for obtaining depth data for parts of the scene that are front facing. The further the averaged pose deviates from being antiparallel with the surface normal {circumflex over (n)}.sub.1, the less suitable that camera pair is for reconstructing parts of the front-facing parts of the scene.

[0046] For each camera pair, and at least one surface normal of the voxel, it may be determined which, if any camera pairs, are better positioned for obtaining depth data from a respective direction with respect to that surface normal. This may involve determining, for each surface normal, a camera pair having the highest weighting associated therewith (based on the epipolar diagonal and principal points midpoint) and that has an averaged pose that is closest to being antiparallel to the surface normal.

[0047] In the example shown in FIG. 5, this may involve determining for each face of the voxel, a corresponding camera pair having the highest weighting (determined as above) and that is closest to being pointed directly at that face (determined as above). This camera pair may be identified as the camera pair that is suitable for obtaining depth data for the voxel, for that specific surface of the voxel. It may be that each voxel is associated with six camera pairs, with each camera pair corresponding to a different respective face (i.e. surface normal direction) of the voxel. Each camera pair may be associated with a camera pair identifier that enables that camera pair to be identified.

[0048] In some examples, a camera pair may be assigned a weighting that is dependent on the epipolar diagonal, principal point midpoint and direction associated with that camera pair. For example, each voxel surface may be associated with a list of camera pairs and respective weightings, with the weightings for the camera pairs not oriented (on average) towards the voxel surface having lower weightings relative to the camera pairs that are oriented (on average) to the voxel surface. Alternatively, it may be that the weighting is just dependent on the epipolar diagonal and principal point midpoint, and that a camera pair is simply not assigned to a given voxel surface if it is determined that the camera pair is not oriented towards that voxel surface.

[0049] Each voxel as defined in the data structure may correspond to an index in a look-up table (LUT). An example of such a look-up table is shown in FIG. 6, as LUT 600. In FIG. 6, a first column 601 is shown as defining indexes. These indexes may correspond to voxels, e.g. index 1 a first voxel, index 2 a second voxel, and so on and so forth. Each voxel may be associated with six entries, each entry corresponding to a different face of the voxel. For example, in FIG. 6, F1 may correspond to a front face of a voxel, F2 to a left face, F3 to a right face and so on and so forth. Each entry may indicate one or more camera pairs that are suitable for reconstructing depth information within the voxel, with respect to the corresponding face of the voxel. That is, each face may be associated with a camera pair identifier, indicating which (if any) camera pair is oriented towards that face, and can be used for obtaining depth data for corresponding parts of the scene that are facing in the same direction (e.g. front, right, left, back, top, bottom).

[0050] As will be appreciated, depending on the set-up of cameras, it may be that there are no camera pairs suitable for obtaining depth data for a given surface of the voxel, and so it may be that only some of the faces of the voxel are associated with a corresponding camera pair. An example of this is shown in FIG. 5, where it can be seen that there is no camera pair indicated for index 2, F1 (i.e. 0).

[0051] The look-up table may include further entries, such as e.g. for each face of each voxel, the extrinsics, principal points, principal point midpoint, epipolar diagonal, averaged pose associated with the camera pair identified as being suited for obtaining depth data for that surface.

[0052] In some examples, each voxel may be further associated with one or more weightings, indicating the suitability of each camera pair for obtaining depth data for that voxel, for a given direction. For example, it may be that the LUT provides for each face of each voxel, the camera pairs providing depth coverage of that voxel face, and the weightings associated with those camera pairs. The weightings may be determined as described above. In such examples, it may be that the most appropriate camera pair for obtaining depth data for a given face of a given voxel is determined by identifying the camera pair associated with highest weighting. That is, the optimally positioned camera need not be explicitly indicated in the table but can be identified based on a corresponding weighting.

[0053] The above-described step of determining which, if any, of the camera pairs are suited for obtaining depth data from a respective direction may be repeated for each voxel forming the capture volume. This may involve iterating through the steps of determining a weighting associated with each camera pair, and an averaged pose of that camera pair relative to one or more surface normals of each voxel (e.g. each face).

[0054] Returning to FIG. 3, it can be seen that the capture volume has been divided into 36 equal-sized voxels. A first voxel 303, shown in black, may correspond to the voxel shown in FIG. 5. In the example of FIG. 3 it may be that the look-up table corresponds to a table having 36 indexes, with each index corresponding to a different voxel in the capture volume. Each index may be associated with six entries, each entry defining a camera pair that is suited for obtaining depth data from a different respective direction. Using this look-up table, it can then be determined, for a given pose of a virtual camera, which of the voxels will be within the frustum of the virtual camera, and which camera pairs should be used for reconstructing the parts of the scene within those voxels.

Dividing the Capture Volume

[0055] In FIG. 3, the capture volume is shown as being sub-divided uniformly, with each voxel occupying the same volume of space. This corresponds to a relatively arbitrary division of the capture volume. In some examples, it may be that the capture volume is divided up differently, and not necessarily in a uniform manner.

[0056] In some examples, the capture volume may be divided in a hierarchical manner, based on the depth coverage associated with the camera pairs positioned around the scene. In the present disclosure, a pair of cameras are said to provide depth coverage of a part of scene if depth data can be extracted for that part of the scene, from the images captured by the cameras in the corresponding camera pair. In essence, depth coverage is equivalent to the field of view of a depth camera that a given pair of cameras corresponds to. The depth resolution of a camera pair corresponds to the amount of depth data that can be obtained for a given volume of space. For example, the depth resolution of a first camera pair may be said to be higher than a second camera pair, if the first camera pair is able to obtain more depth data for the same volume of space.

[0057] In some examples, the capture volume, as divided into voxels, may be represented as a hierarchal data structure. Each successive layer in the hierarchical data structure may correspond to a further sub-division of the capture volume. An example of this is shown in FIG. 7, where the node at the top of the structure corresponds to the entire capture volume, e.g. the volume enclosing a football field. The capture volume is then split into a second layer of nodes, with each node corresponding to a respective voxel of the capture volume. For example, the nodes N1, N2, N3, N4 may correspond to different quadrants of the capture volume. Each node may then be split again, depending on the granularity with which the voxel space has been divided up. In FIG. 8, the node N1 is shown as being further divided into quadrants. For each node (regardless of layer), one or more camera pairs providing depth coverage from a given direction (e.g. front, back, top, bottom, left, right) may be indicated. In some examples, one or more of the extrinsics, principal points, principal points midpoint, epipolar diagonal and averaged pose may be further defined for each of the camera pairs identified for a given voxel and direction associated therewith.

[0058] In some examples, each voxel (and sub-voxel) is sub-divided based on a determination that there is a sub-volume within a given voxel for which a different camera pair provides depth coverage (from a respective direction) at a higher resolution. The depth coverage may be a higher resolution in the sense that there is more depth data that can be obtained within the sub-voxel, compared with the camera pair providing depth coverage of the coarser parent voxel. In such examples, at least one surface of each voxel (and sub-voxel) may be associated with a corresponding camera pair identifier, indicating the camera pair that is able to obtain depth data for that surface at the highest depth resolution relative to the other camera pairs providing depth coverage of that surface.

[0059] Hence, in some examples, the number and size of the voxels of the voxel representation may be indicative of the depth resolution available from a current camera set-up. For example, a large number of small voxels may indicate that a high depth resolution is available for each voxel (or at least some of the corresponding voxel surfaces); whereas, a few large voxels may be indicative that only a low depth resolution is available for each voxel (or at least some of the corresponding voxel surfaces). A few large voxels being defined in the data structure may occur where e.g. there is a sparse distribution of cameras positioned around the scene. In this way, the data structure acts as a heat map of depth coverage and depth resolution, enabling a user (e.g. camera operator) to determine whether the extrinsics and/or intrinsics of one or more cameras should be adjusted.

[0060] It will be appreciated that, in some examples, the voxels may be sub-divided in a non-uniform manner. For example, some voxels may be sub-divided further than others, depending on the depth coverage (and corresponding depth resolution) associated with those voxels.

[0061] In some examples, the hierarchical data structure may correspond to an octree, with at least some of the voxels in the data structure being sub-divided into octants. Each octant defined in the octree may correspond to a volume of space for which a given camera pair has been identified as providing depth coverage (and optionally, at a higher depth resolution relative to the other camera pairs providing depth coverage of the parent voxel).

[0062] In some examples, it may be determined, for a given voxel, whether the distance between the principal points for a given camera pair providing depth coverage of that voxel is less than a dimension (e.g. width) of the voxel. If the distance between the principal points is less than this, the voxel may be sub-divided further. The steps of determining a weighting for each camera pair, and an averaged pose of each camera relative to a surface normal of the voxel may then be repeated for the sub-divided voxel. That is, the pre-computation stage may be iterative, being performed each voxel and each sub-division thereof.

[0063] In some examples, the capture volume may be divided into voxels according to the midpoints between the principal points of each camera pair. This may be achieved using a k-dimensional tree (kd-tree), for example. An example of this is shown in FIG. 8, which shows a top down view of the capture area 800, where a voxel 802 is shown as being formed of the planes defined by the midpoints MPP1, MPP2, MPP3 between principal points (not shown) of different respective camera pairs. In some examples voxel 802 may be further sub-divided based on midpoints falling within this voxel. Again, for each voxel and sub-division thereof, the camera pairs suitable for reconstructing a given surface of the voxel may be determined as above. In FIG. 8, voxel 802 may correspond to a volume of space for which depth data can be obtained by the camera pairs corresponding to midpoints MPP1, MPP2 and MPP3.

[0064] As mentioned above, the capture volume need not be divided up in a uniform manner. FIG. 9 shows schematically another example of how the capture volume may be divided based on the depth coverage of camera pairs located about the scene. In FIG. 9, the capture area 900 is shown as being encapsulated by a hemisphere 902. A plurality of cameras C91, C92, C93, C94, C95 and C96 are shown positioned around the scene. In FIG. 9, the cameras are shown as being positioned above and oriented towards the hemisphere. It will be appreciated that this just an example of a camera configuration, and that, depending on the type of event being captured, the cameras may be arranged differently.

[0065] In FIG. 9, the first camera pair comprises C91 and C92; the second camera pair comprises C93 and C94; the third camera pair comprises C95 and C96. It will be appreciated that other possible combinations of camera pairs are possible, but for clarity, only three will be described in relation to FIG. 9.
……
……
……

本文链接：https://patent.nweon.com/18087

您可能还喜欢...