Sony Patent | Image processing apparatus and method

Patent: Image processing apparatus and method

Drawings: Click to check drawins

Publication Number: 20210233278

Publication Date: 20210729

Applicant: Sony

Assignee: Sony Corporation

Abstract

The present disclosure relates to an image processing apparatus and method that can prevent an increase in the load of a decoding process for encoded data in a point cloud video-based approach. The parameters related to a plurality of point cloud models of a point cloud are transformed, a two-dimensional plane image onto which the plurality of point cloud models having the transformed parameters is projected is encoded, and a bitstream containing encoded data of the two-dimensional image and transform information that is information regarding the transform of the parameters is generated. The present disclosure can be applied to an information processing device, an image processing apparatus, an electronic apparatus, an information processing method, a program, or the like, for example.

Claims

1. An image processing apparatus comprising: a transform unit that transforms a parameter related to a plurality of point cloud models of a point cloud; and an encoding unit that encodes a two-dimensional plane image onto which the plurality of point cloud models having the parameter transformed by the transform unit is projected, and generates a bitstream containing encoded data of the two-dimensional plane image and transform information that is information regarding transform of the parameter transformed by the transform unit.

2. The image processing apparatus according to claim 1, wherein the transform unit transforms coordinates of the point cloud models as the parameter.

3. The image processing apparatus according to claim 2, wherein the transform unit shifts, rotates, or shifts and rotates the coordinates when transforming the coordinates.

4. The image processing apparatus according to claim 1, wherein the transform unit transforms a time of the point cloud models as the parameter.

5. The image processing apparatus according to claim 1, wherein the transform unit transforms a scale of the point cloud models as the parameter.

6. The image processing apparatus according to claim 1, wherein the transform unit transforms a frame rate of the point cloud models as the parameter.

7. The image processing apparatus according to claim 1, wherein the transform information contains information indicating an amount of change in the parameter transformed by the transform unit.

8. The image processing apparatus according to claim 1, further comprising a packing unit that places patches of the point cloud models in regions different for the respective point cloud models, and packs the patches as a video frame, wherein the encoding unit encodes the video frame in which the patches are packed by the packing unit, and generates the bitstream further containing model information that is information regarding the point cloud models.

9. The image processing apparatus according to claim 8, wherein the regions are independently decodable units of encoding.

10. The image processing apparatus according to claim 8, wherein the model information contains information indicating the number of point cloud models included in the bitstream and the regions in which the patches of the respective point cloud models are placed.

11. An image processing method comprising: transforming a parameter related to a plurality of point cloud models of a point cloud; and encoding a two-dimensional plane image onto which the plurality of point cloud models having the parameter transformed is projected, and generating a bitstream containing encoded data of the two-dimensional plane image and transform information that is information regarding transform of the parameter.

12. An image processing apparatus comprising: a decoding unit that decodes a bitstream, and generates a two-dimensional plane image onto which a plurality of point cloud models is projected, and transform information that is information regarding transform of a parameter of each of the plurality of point cloud models; and a reconstruction unit that reconstructs each of the plurality of point cloud models from the two-dimensional plane image generated by the decoding unit, and, on a basis of the transform information, inversely transforms the parameter of each of the plurality of point cloud models.

13. An image processing method comprising: decoding a bitstream, and generating a two-dimensional plane image onto which a plurality of point cloud models is projected, and transform information that is information regarding transform of a parameter of each of the plurality of point cloud models; and reconstructing each of the plurality of point cloud models from the generated two-dimensional plane image, and, on a basis of the transform information, inversely transforming the parameter of each of the plurality of point cloud models.

14. An image processing apparatus comprising: a projection unit that projects a plurality of partial point cloud models constituting a point cloud model onto projection planes that are set independently of one another; and an encoding unit that encodes a two-dimensional plane image onto which patches of the partial point cloud models projected onto the respective projection planes by the projection unit are projected, and an occupancy map containing projection plane information that is information regarding the projection planes, and generates a bitstream.

15. The image processing apparatus according to claim 14, wherein the projection plane information contains information indicating an amount of rotation of the projection planes.

16. The image processing apparatus according to claim 14, further comprising: a dividing unit that divides the point cloud model into the plurality of partial point cloud models; and a setting unit that sets the projection plane for each of the plurality of partial point cloud models.

17. The image processing apparatus according to claim 16, wherein the dividing unit identifies point-dense clouds from a histogram of the point cloud model, estimates a projecting direction of each of the point-dense clouds on a basis of a normal vector, and sets the point-dense clouds with different projecting directions as the partial point cloud models.

18. An image processing method comprising: projecting a plurality of partial point cloud models constituting a point cloud model onto projection planes that are set independently of one another; and encoding a two-dimensional plane image onto which patches of the partial point cloud models projected onto the respective projection planes are projected, and an occupancy map containing projection plane information that is information regarding the projection planes, and generating a bitstream.

19. An image processing apparatus comprising: a decoding unit that decodes a bitstream, and generates a two-dimensional plane image onto which a point cloud model is projected, and an occupancy map containing projection plane information that is information regarding a projection plane of each of a plurality of partial point cloud models included in the point cloud model; and a reconstruction unit that reconstructs the point cloud model, on a basis of the two-dimensional plane image generated by the decoding unit, and the projection plane information contained in the occupancy map.

20. An image processing method comprising: decoding a bitstream, and generating a two-dimensional plane image onto which a point cloud model is projected, and an occupancy map containing projection plane information that is information regarding a projection plane of each of a plurality of partial point cloud models included in the point cloud model; and reconstructing the point cloud model, on a basis of the generated two-dimensional plane image, and the projection plane information contained in the occupancy map.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method capable of preventing an increase in the load of a decoding process for encoded data in a video-based approach to a point cloud.

BACKGROUND ART

[0002] As a method for encoding 3D data representing a three-dimensional structure such as a point cloud, for example, there has been coding using voxels, such as Octree (see Non-Patent Document 1, for example).

[0003] In recent years, an approach has been suggested for projecting the positional and color information about a point cloud onto a two-dimensional plane on a small-region basis, for example, and encoding the positional and color information by an encoding method for two-dimensional images (hereinafter, this approach will also be referred to as a video-based approach) (see Non-Patent Documents 2 to 4, for example).

[0004] In such coding, when a plurality of point cloud models exists in a point cloud, the point cloud models are encoded independently of one another and are turned into different bitstreams from one another by a conventional method.

CITATION LIST

Non-Patent Documents

[0005] Non-Patent Document 1: R. Mekuria, Student Member IEEE, and K. Blom and P. Cesar., Members IEEE, “Design, Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video”, tcsvt paper submitted february.pdf [0006] Non-Patent Document 2: Tim Golla and Reinhard Klein, “Real-time Point Cloud Compression”, IEEE, 2015 [0007] Non-Patent Document 3: K. Mammou, “Video-based and Hierarchical Approaches Point Cloud Compression”, MPEG m41649, October 2017 [0008] Non-Patent Document 4: K. Mammou, “PCC Test Model Category 2 v0”, N17248 MPEG output document, October 2017

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0009] In the case of this method, however, there is a possibility that the number of instances necessary for the decoder will increase when the number of point cloud models increases.

[0010] The present disclosure has been made in view of such circumstances, and aims to prevent an increase in the number of instances necessary for decoding encoded data of a point cloud in a video-based approach, and prevent an increase in the load of the decoding process.

Solutions to Problems

[0011] An image processing apparatus of one aspect of the present technology is an image processing apparatus that includes: a transform unit that transforms a parameter related to a plurality of point cloud models of a point cloud; and an encoding unit that encodes a two-dimensional plane image onto which the plurality of point cloud models having the parameter transformed by the transform unit is projected, and generates a bitstream containing encoded data of the two-dimensional plane image and transform information that is information regarding transform of the parameter transformed by the transform unit.

[0012] An image processing method of one aspect of the present technology is an image processing method that includes: transforming a parameter related to a plurality of point cloud models of a point cloud; and encoding a two-dimensional plane image onto which the plurality of point cloud models having the parameter transformed is projected, and generating a bitstream containing encoded data of the two-dimensional plane image and transform information that is information regarding the transform of the parameter.

[0013] An image processing apparatus of another aspect of the present technology is an image processing apparatus that includes: a decoding unit that decodes a bitstream, and generates a two-dimensional plane image onto which a plurality of point cloud models is projected, and transform information that is information regarding transform of a parameter of each of the plurality of point cloud models; and a reconstruction unit that reconstructs each of the plurality of point cloud models from the two-dimensional plane image generated by the decoding unit, and, on the basis of the transform information, inversely transforms the parameter of each of the plurality of point cloud models.

[0014] An image processing method of another aspect of the present technology is an image processing method that includes: decoding a bitstream, and generating a two-dimensional plane image onto which a plurality of point cloud models is projected, and transform information that is information regarding transform of a parameter of each of the plurality of point cloud models; and reconstructing each of the plurality of point cloud models from the generated two-dimensional plane image, and, on the basis of the transform information, inversely transforming the parameter of each of the plurality of point cloud models.

[0015] An image processing apparatus of yet another aspect of the present technology is an image processing apparatus that includes: a projection unit that projects a plurality of partial point cloud models constituting a point cloud model onto projection planes that are set independently of one another; and an encoding unit that encodes a two-dimensional plane image onto which patches of the partial point cloud models projected onto the respective projection planes by the projection unit are projected, and an occupancy map containing projection plane information that is information regarding the projection planes, and generates a bitstream.

[0016] An image processing method of yet another aspect of the present technology is an image processing method that includes: projecting a plurality of partial point cloud models constituting a point cloud model onto projection planes that are set independently of one another; and encoding a two-dimensional plane image onto which patches of the partial point cloud models projected onto the respective projection planes are projected, and an occupancy map containing projection plane information that is information regarding the projection planes, and generating a bitstream.

[0017] An image processing apparatus of still another aspect of the present technology is an image processing apparatus that includes: a decoding unit that decodes a bitstream, and generates a two-dimensional plane image onto which a point cloud model is projected, and an occupancy map containing projection plane information that is information regarding a projection plane of each of a plurality of partial point cloud models included in the point cloud model; and a reconstruction unit that reconstructs the point cloud model, on the basis of the two-dimensional plane image generated by the decoding unit, and the projection plane information contained in the occupancy map.

[0018] An image processing method of still another aspect of the present technology is an image processing method that includes: decoding a bitstream, and generating a two-dimensional plane image onto which a point cloud model is projected, and an occupancy map containing projection plane information that is information regarding a projection plane of each of a plurality of partial point cloud models included in the point cloud model; and reconstructing the point cloud model, on the basis of the generated two-dimensional plane image and the projection plane information contained in the occupancy map.

[0019] In the image processing apparatus and method of one aspect of the present technology, the parameter related to a plurality of point cloud models of a point cloud is transformed, a two-dimensional plane image onto which the plurality of point cloud models having the parameter transformed is projected is encoded, and a bitstream containing the encoded data of the two-dimensional plane image and transform information that is information regarding the transform of the parameter is generated.

[0020] In the image processing apparatus and method of another aspect of the present technology, a bitstream is decoded, a two-dimensional plane image onto which a plurality of point cloud models is projected and transform information that is information regarding the transform of the parameter of each of the plurality of point cloud models are generated, each of the plurality of point cloud models is reconstructed from the generated two-dimensional plane image, and the parameter of each of the plurality of point cloud models is inversely transformed on the basis of the transform information.

[0021] In the image processing apparatus and method of yet another aspect of the present technology, a plurality of partial point cloud models constituting a point cloud model is projected onto projection planes that are set independently of one another, a two-dimensional plane image onto which patches of the partial point cloud models projected onto the respective projection planes are projected, and an occupancy map containing projection plane information that is information regarding the projection planes are encoded, and a bitstream is generated.

[0022] In the image processing apparatus and method of still another aspect of the present technology, a bitstream is decoded, a two-dimensional plane image onto which a point cloud model is projected and an occupancy map containing projection plane information that is information regarding the projection plane of each of a plurality of partial point cloud models included in the point cloud model are generated, the point cloud model is reconstructed on the basis of the generated two-dimensional plane image and the projection plane information contained in the occupancy map.

Effects of the Invention

[0023] According to the present disclosure, an image can be processed. Particularly, it is possible to prevent an increase in the load of a decoding process for encoded data in a video-based approach to a point cloud.

BRIEF DESCRIPTION OF DRAWINGS

[0024] FIG. 1 is a diagram summarizing principal features of the present technology.

[0025] FIG. 2 is a diagram for explaining the outline of point cloud encoding and decoding to which the present technology is applied.

[0026] FIG. 3 is a diagram for explaining the outline of point cloud encoding and decoding to which the present technology is applied.

[0027] FIG. 4 is a block diagram showing a typical example configuration of an encoding device.

[0028] FIG. 5 is a flowchart for explaining an example flow in an encoding process.

[0029] FIG. 6 is a flowchart for explaining an example flow in a transform process.

[0030] FIG. 7 is a flowchart for explaining an example flow in a bounding box setting process.

[0031] FIG. 8 is a flowchart for explaining an example flow in a packing process.

[0032] FIG. 9 is a flowchart for explaining an example flow in a BB information generation process.

[0033] FIG. 10 is a block diagram showing a typical example configuration of a decoding device.

[0034] FIG. 11 is a flowchart for explaining an example flow in a decoding process.

[0035] FIG. 12 is a diagram summarizing principal features of the present technology.

[0036] FIG. 13 is a diagram for explaining the outline of point cloud encoding and decoding to which the present technology is applied.

[0037] FIG. 14 is a block diagram showing a typical example configuration of an encoding device.

[0038] FIG. 15 is a block diagram showing a typical example configuration of a patch separation unit.

[0039] FIG. 16 is a flowchart for explaining an example flow in an encoding process.

[0040] FIG. 17 is a flowchart for explaining an example flow in a patch separation process.

[0041] FIG. 18 is a flowchart for explaining an example flow in a dividing process.

[0042] FIG. 19 is a flowchart for explaining an example flow in a packing process.

[0043] FIG. 20 is a block diagram showing a typical example configuration of a decoding device.

[0044] FIG. 21 is a flowchart for explaining an example flow in a decoding process.

[0045] FIG. 22 is a block diagram showing a typical example configuration of a computer.

MODE FOR CARRYING OUT THE INVENTION

[0046] The following is a description of modes for carrying out the present disclosure (these modes will be hereinafter referred to as embodiments). Note that explanation will be made in the following order.

[0047] 1. Transform of point cloud models

[0048] 2. Partial Decoding

[0049] 3. First embodiment (an encoding device)

[0050] 4. Second embodiment (a decoding device)

[0051] 5. Local projection plane control

[0052] 6. Third embodiment (an encoding device)

[0053] 7. Fourth embodiment (a decoding device)

[0054] 8. Notes

1. Transform of Point Cloud Models

[0055]

[0056] The scope disclosed in the present technology includes not only the contents disclosed in the embodiments but also the contents disclosed in the following non-patent documents that were known at the time of filing. [0057] Non-Patent Document 1: (mentioned above) [0058] Non-Patent Document 2: (mentioned above) [0059] Non-Patent Document 3: (mentioned above) [0060] Non-Patent Document 4: (mentioned above) [0061] Non-Patent Document 5: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), “Advanced video coding for generic audiovisual services”, H.264, April 2017 [0062] Non-Patent Document 6: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), “High efficiency video coding”, H.265, December 2016 [0063] Non-Patent Document 7: Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer, and Jill Boyce, “Algorithm Description of Joint Exploration Test Model 4”, JVET-G1001_v1, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 7th Meeting: Torino, IT, 13-21 Jul. 2017

[0064] That is, the contents disclosed in the non-patent documents listed above are also the basis for determining the support requirements. For example, when the Quad-Tree Block Structure disclosed in Non-Patent Document 6 and the Quad Tree Plus Binary Tree (QTBT) Block Structure disclosed in Non-Patent Document 7 are not directly disclosed in the embodiments, those structures are within the scope of the present technology, and satisfy the support requirements of the claims. Further, the technical terms such as parsing, syntax, and semantics are also within the scope of disclosure of the present technology, and satisfy the support requirements of the claims, even when those technical terms are not directly described, for example.

[0065]

[0066] There have been data such as point clouds that represent three-dimensional structures with positional information, attribute information, and the like about point clouds, and meshes that are formed with vertices, edges, and planes, and define three-dimensional shapes using polygonal representations.

[0067] For example, in the case of a point cloud, a three-dimensional structure is expressed as a set of a large number of points (a point cloud). That is, the data of a point cloud is formed with positional information and attribute information (colors and the like, for example) about the respective points in this point cloud. Accordingly, the data structure is relatively simple, and any desired three-dimensional structure can be expressed with a sufficiently high accuracy with the use of a sufficiently large number of points.

[0068]

[0069] A video-based approach has been suggested for projecting positional and color information about such a point cloud onto a two-dimensional plane on a small-region basis, and encoding the positional and color information by an encoding method for two-dimensional images.

[0070] In this video-based approach, an input point cloud is divided into a plurality of segmentations (also called regions), and is projected onto a two-dimensional plane region by region. Note that the data of the point cloud at each position (which is the data of each point) is formed with positional information (Geometry (also referred to as Depth)) and attribute information (Texture) as described above, and is projected onto a two-dimensional plane region by region.

[0071] Each segmentation (also called a patch) projected onto the two-dimensional plane is then placed in a two-dimensional image, and is encoded by an encoding method for two-dimensional plane images, such as Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC), for example.

[0072]

[0073] There are cases where such a point cloud includes a plurality of point cloud model formed with point-dense clouds. For example, when a plurality of persons is scattered in an open space, when the three-dimensional space including the entire open space is turned into a point cloud, a point-dense cloud (a point cloud model) is formed at the portion of each person. That is, a plurality of point cloud models is formed.

[0074] In such a case, if the entire point cloud is regarded as one large point cloud model and is encoded, there is a possibility that the coding efficiency will drop, because there are many portions in which points are sparse. Therefore, a method has been suggested for encoding each of the plurality of point cloud models included in a point cloud independently of one another (for each person, for example). In this case, however, a bitstream is generated for each point cloud model. Therefore, if the number of point cloud models increases, the number of bitstreams will also increase, resulting in a possibility of an increase in the number of instances necessary for the decoder. For example, when there are hundreds of point cloud models in a point cloud, there is a possibility that instances that are hundreds of times larger in number than the number of instances necessary for decoding a single bitstream will be required. Because the load of the decoding process increases in this manner, the cost of the decoder might become higher, and the processing time might become longer.

[0075]

[0076] Therefore, a plurality of point cloud models is stored into one bitstream. For example, as shown in the top row in a table in FIG. 1, a plurality of point cloud models is transformed and is combined into a single group, and the single group is encoded, so that the plurality of point cloud models is stored into a single bitstream.

[0077] By doing so, it is possible to prevent an increase in the number of bitstreams while preventing a decrease in coding efficiency, even when a plurality of point cloud models exists in a point cloud. Accordingly, an increase in the number of instances necessary for decoding can be prevented. That is, an increase in the load of the decoding process can be prevented. Thus, an increase in the cost of the decoder can be prevented. Further, an increase in the processing time of the decoding process can be prevented.

[0078]

[0079] More specifically, as shown in row #1 in the table in FIG. 1, a parameter related to each point cloud model is transformed, and the point cloud models are combined into one group with fewer point-sparse portions (transform of each point cloud model). That is, the group is collectively encoded and is turned into a single bitstream.

[0080] For example, as shown in the leftmost column in FIG. 2, there are point cloud models 11-1 to 11-4 in a point cloud, and bounding boxes 12-1 to 12-4 are set for the respective point cloud models. When the point cloud models 11-1 to 11-4 are not distinguished from one another in the description, these point cloud models are referred to as the point cloud models 11. Also, when the bounding boxes 12-1 to 12-4 are not distinguished from one another in the description, these bounding boxes are referred to as the bounding boxes 12. That is, these point cloud models 11 are at a distance from one another, and a bounding box 12 is set for each of the point cloud models 11.

[0081] At the time of encoding, as shown in the second column from the left in FIG. 2, the parameters of these point cloud models 11 are transformed and gathered, to form a group with fewer point-sparse portions. In the example shown in FIG. 2, a group including the bounding boxes 12-1 to 12-4 is formed, and a bounding box 13 for encoding is set for the entire group.

[0082] The details (the parameter to be transformed) of this transform may be selected as appropriate. For example, as shown in the table in FIG. 1, the coordinates of the point cloud models 11 may be transformed. For example, as the coordinate transformation, the coordinates of the point cloud models 11 may be shifted (the positions of the point cloud models 11 may be moved). That is, each of the positions of the point cloud models 11-1 to 11-4 in the point cloud shown in the leftmost column in FIG. 2 may be spatially separated from the others. By such transform, the positions of the respective point cloud models can be brought closer to one another, for example, and thus, the sparse portions in the group can be reduced.

[0083] Also, the coordinates of the point cloud models 11 may be rotated (the postures of the point cloud models 11 may be rotated). That is, each of the postures (orientations) of the point cloud models 11-1 to 11-4 in the point cloud shown in the leftmost column in FIG. 2 may be different from the others. By such transform, the postures of the respective point cloud models can be made uniform, for example, and thus, the sparse portions in the group can be reduced.

[0084] Also, as shown in the table in FIG. 1, the times (time stumps) of the point cloud models 11 may be transformed, for example. That is, each of the positions of the point cloud models 11-1 to 11-4 in the point cloud shown in the leftmost column in FIG. 2 may be temporally separated from the others (the point cloud models may include a point cloud model existing at a different from time from the others). By such transform, the times of the respective point cloud models can be aligned, for example, and thus, the point cloud models existing at different times can be combined into one group.

[0085] Also, as shown in the table in FIG. 1, the sizes (scales) of the point cloud models 11 may be transformed, for example. That is, each of the scales of the point cloud models 11-1 to 11-4 in the point cloud shown in the leftmost column in FIG. 2 may be different from the others. By such transform, the sizes (scales) of the respective point cloud models can be made uniform, or the resolutions in a spatial direction can be made uniform, for example.

[0086] Further, as shown in the table in FIG. 1, the frame rates of the point cloud models 11 may be transformed, for example. That is, each of the frame rates of the point cloud models 11-1 to 11-4 in the point cloud shown in the leftmost column in FIG. 2 may be different from the others. By such transform, the frame rates (which are the resolutions in a temporal direction) of the respective point cloud models can be made uniform, for example.

[0087] In FIG. 2, at a time of encoding, the group of point cloud models in the bounding box 13 for encoding shown in the second column from the left is projected onto a two-dimensional plane, and is packed in video frames. In this manner, a color video frame 14 containing attribute information (Texture), a geometry video frame 15 containing positional information (Depth), and an occupancy map 16 are generated, as shown in the third column from the left.

[0088] These video frames are then encoded by an encoding method for two-dimensional images, so that a 2D bitstream 17 (including an occupancy map 18 and a header 19) is generated, as shown in the rightmost column in FIG. 2. That is, the encoded data of the point cloud models 11-1 to 11-4 is stored into this single 2D bitstream 17.

[0089] In this manner, an increase in the number of bitstreams can be prevented, and an increase in the load of the decoding process can be prevented.

[0090]

[0091] In the case of decoding, the inverse process of the above is performed. That is, an inverse transform process is performed to return the respective point cloud models from the state of the group shown in the second column from the left in FIG. 2 to the original state shown in the leftmost column in FIG. 2. Therefore, to enable this inverse transform at the time of decoding, transform information indicating the details of the parameter transform performed on the point cloud models at the time of encoding (transform from the state shown in the leftmost column to the state shown in the second column from the left in FIG. 2) is generated as shown in row #1 in the table in FIG. 1, and the transform information (included in a bitstream, for example) is transmitted to the decoding side (that is, signaling of the transform information is performed).

[0092] This transform information may be any information that indicates the details of the transform (the amount of change in the transformed parameter). For example, as shown in the table in FIG. 1, the transform information may be information indicating the ratio of the transformed parameter (Shift, Rotate, Time Stump, Scale, Frame Rate, or the like, for example) between before and after the transform. Alternatively, for example, as shown in the table in FIG. 1, the transform information may be information indicating the difference in the transformed parameter (Shift, Rotate, Time Stump, Scale, Frame Rate, or the like, for example) between before and after the transform.

[0093] Further, any appropriate method can be used for transmitting this transform information. For example, the transform information may be associated with the bitstream containing the encoded data of the point cloud models, and be then transmitted. The transform information may be included in the bitstream, for example. As shown in FIG. 2, such transform information may be included as BB information 21 in the header 19 of the 2D bitstream 17, for example.

[0094] For example, as the transform information, the BB information 21 contains information indicating a shift amount (Shift x/y/z), information indicating an amount of rotation (Rotate x/y/z), information indicating an amount of change in the time stump (Time Stump), information indicating an amount of change in the scale (Scale x/y/z), information indicating an amount of change in the frame rate (Frame Rate), and the like. At the time of encoding, this BB information 21 is generated and is stored into the header 19. At the time of decoding, the BB information 21 is read from the header 19 and is used for inverse transform.

[0095]

[0096] Note that, as shown in row #2 in the table in FIG. 1, when the group of point cloud models shown in the second column from the left in FIG. 2 is projected onto a two-dimensional plane, a projection plane may be set for each point cloud model. In this manner, the respective point cloud models can be projected onto more appropriate projection planes. Thus, a decrease in coding efficiency due to inefficiency in projection of the point cloud models can be prevented (coding efficiency can be increased).

[0097] For example, as shown in the table in FIG. 1, the projection planes in the six orthogonal directions of each point cloud model may be made rotatable. In this manner, the projection planes can be relatively easily made more suitable for the point cloud models (or can be turned into planes on which effective projection can be performed).

[0098] Further, as shown in the table in FIG. 1, any projection plane may be set (added). With this arrangement, the degree of freedom of the projection planes of the respective point cloud models can be increased, and more efficient projection can be expected (or a further increase in coding efficiency can be expected).

[0099]

[0100] Note that, when a projection plane is set for each point cloud model in this manner (when a projection plane common to the group is not set), information about the projection planes is also necessary at the time of decoding (when the point cloud is reconstructed). Therefore, as shown in row #2 in the table in FIG. 1, projection plane information that is the information regarding the projection planes set as described above is transmitted to the decoding side (signaling of projection plane information).

[0101] As shown in the table in FIG. 1, this projection plane information may contain any information that is necessary for identifying the projection planes. For example, as shown in FIG. 2, this projection plane information 22 may be stored into the occupancy map 16 (which is the occupancy map 18).

[0102] For example, the projection plane information 22 contains information indicating amounts of shift in projection planes in six orthogonal directions (Shift x/y/z), information indicating an amount of change in scale (Scale x/y/z), information indicating an amount of rotation (Rotate x/y/z), and the like. The projection plane information 22 may also contain information indicating a projection plane that is added as desired, for example.

[0103] At the time of encoding, this projection plane information 22 is generated and is stored into the occupancy map 16. At the time of decoding, the projection plane information 22 is read from the occupancy map 16, and is used for reconstructing the point cloud.

2. Partial Decoding

[0104]

[0105] Further, as shown in row #3 in the table in FIG. 1, when the patches of each point cloud model are placed in a two-dimensional image and is packed as a video frame, the regions in which the patches are placed may be controlled for each point cloud model. For example, the patches belonging to the same point cloud model may be placed in the same region. This region may be any appropriate region. For example, this region may be an independently decodable unit of encoding. That is, the patches belonging to the same point cloud model may be placed in the same independently decodable unit of encoding.

[0106] This independently decodable unit of encoding may be any appropriate unit. For example, the unit may be a frame, a slice, or a tile, as shown in the table in FIG. 1.

[0107] As shown in A of FIG. 3, a point cloud model 31-1 and a point cloud model 31-2 exist in a point cloud, for example. A bounding box 32-1 is set for the point cloud model 31-1, and a bounding box 32-2 is set for the point cloud model 31-2. That is, these point cloud models are independent of each other. When the patches of the point cloud model 31-1 and the point cloud model 31-2 are placed and packed in a two-dimensional image 33, the patches of the two point cloud models are not divided into separate regions by a conventional method, as shown in B of FIG. 3. In B of FIG. 3, patches 34-1 to 34-4 are the patches of the point cloud model 31-1, and patches 35-1 to 35-5 are the patches of the point cloud model 31-2. In C of FIG. 3, on the other hand, the patches 34-1 to 34-4 of the point cloud model 31-1 are placed in a slice 36-2 of the two-dimensional image 33, and the patches 35-1 to 35-5 of the point cloud model 31-2 are placed in a slice 36-1 of the two-dimensional image 33, for example.

[0108] As the independently decodable units of encoding in which the patches are placed are controlled for each point cloud model (the patches belonging to the same point cloud model are placed in the same independently decodable unit of encoding), it is possible to perform so-called “partial decoding” for decoding and reconstructing only one of the point cloud models. For example, in the case illustrated in C of FIG. 3, the slice 36-1 and the slice 36-2 are units of encoding that can be decoded independently of each other. Accordingly, only the patches 35-1 to 35-5 in the slice 36-1 can be decoded, or only the patches 34-1 to 34-4 in the slice 36-2 can be decoded. That is, only the point cloud model 31-1 can be decoded and reconstructed, or only the point cloud model 31-2 can be decoded and reconstructed.

[0109]

[0110] Note that, to perform such partial decoding, the decoding side needs to understand in which regions (independently decodable units of encoding) the patches of which point cloud models are placed. Therefore, as shown in row #3 in the table in FIG. 1, model information that is information regarding the point cloud models is generated, and is transmitted (signaled) from the encoding side to the decoding side.

[0111] This model information contains any appropriate information. For example, as shown in the table in FIG. 1, the model information may contain information indicating the number of point cloud models to be packed in a video frame. Also, as shown in the table in FIG. 1, the model information may contain information indicating the regions (independently decodable units of encoding) in which the patches of the respective point cloud models are placed, for example.

[0112] Note that this model information is transmitted by any appropriate method. For example, the model information may be associated with the bitstream containing the encoded data of the point cloud models, and be then transmitted. The model information may be included in the bitstream, for example. As shown in FIG. 2, such model information may be included as the BB information 21 in the header 19 of the 2D bitstream 17, for example.

[0113] The BB information 21 contains the model information that is information regarding the number of models and the patch placement regions of the respective point cloud models, for example. At the time of encoding, this BB information 21 is generated and is stored into the header 19. At the time of decoding, the BB information 21 is read from the header 19 and is used for partial decoding.

3. First Embodiment

[0114]

[0115] Next, a configuration for realizing each of the above techniques is described. FIG. 4 is a block diagram showing an example configuration of an encoding device as an embodiment of an image processing apparatus to which the present technology is applied. An encoding device 100 shown in FIG. 4 is a device that projects 3D data such as a point cloud onto a two-dimensional plane and performs encoding by an encoding method for two-dimensional images (an encoding device to which the video-based approach is applied).

[0116] Note that FIG. 4 shows the principal components and aspects such as processing units and the data flow, but does not necessarily show every aspect. That is, in the encoding device 100, there may be a processing unit that is not shown as a block in FIG. 4, or there may be a processing or data flow that is not indicated by arrows or the like in FIG. 4. This also applies to the other drawings for explaining the processing units and the like in the encoding device 100.

[0117] As shown in FIG. 4, the encoding device 100 includes a model transform unit 111, a patch separation unit 112, a packing unit 113, an auxiliary patch information compression unit 114, a video encoding unit 115, a video encoding unit 116, an OMap encoding unit 117, a multiplexer 118, and a BB information generation unit 119.

[0118] The model transform unit 111 performs a process related to parameter transform related to a point cloud model. For example, the model transform unit 111 acquires 3D data (a point cloud, for example) that is input to the encoding device 100 and indicates a three-dimensional structure. The model transform unit 111 also transforms the parameters related to the point cloud models included in the acquired point cloud. At that time, the model transform unit 111 transforms the parameters related to the respective point cloud models by the method described above in and , and combines the plurality of point cloud models into one group. The model transform unit 111 supplies the patch separation unit 112 with the point cloud models whose parameters have been transformed, which are the point cloud including the point cloud models combined into one group.

[0119] The model transform unit 111 also generates transform information regarding the transform, by the method described above in . The model transform unit 111 supplies the generated transform information to the BB information generation unit 119.

[0120] The patch separation unit 112 performs a process related to separation of 3D data. For example, the patch separation unit 112 acquires the point cloud (a point cloud including a plurality of point cloud models combined into one group) supplied from the model transform unit 111. The patch separation unit 112 also separates the acquired point cloud into a plurality of segmentations, projects the point cloud onto a two-dimensional plane for each segmentation, and generates patches of positional information and patches of attribute information. The patch separation unit 112 supplies information regarding each of the generated patches to the packing unit 113. The patch separation unit 112 also supplies auxiliary patch information, which is information regarding the separation, to the auxiliary patch information compression unit 114.

[0121] The packing unit 113 performs a process related to data packing. For example, the packing unit 113 acquires, from the patch separation unit 112, information regarding the patches of positional information (Geometry) indicating the positions of points, and information regarding the patches of attribute information (Texture) such as color information to be added to the positional information.

[0122] The packing unit 113 also places each of the acquired patches in a two-dimensional image, to pack the patches as a video frame. For example, the packing unit 113 arranges the patches of positional information in a two-dimensional image, to pack the patches as a video frame of positional information (also referred to as a geometry video frame). The packing unit 113 also arranges the patches of attribute information in a two-dimensional image, to pack the patches as a video frame of attribute information (also referred to as a color video frame), for example.

[0123] At the time of these packing processes, the packing unit 113 controls the region in which the patches are arranged (independently decodable units of encoding) for each point cloud model, by the method described above in . The packing unit 113 then generates model information by the method described above in . The packing unit 113 supplies the generated model information to the BB information generation unit 119.

[0124] The packing unit 113 also generates an occupancy map associated with these video frames. The packing unit 113 further performs a dilation process on the color video frame.

[0125] The packing unit 113 supplies the geometry video frame generated in this manner to the video encoding unit 115. The packing unit 113 also supplies the color video frame generated in this manner to the video encoding unit 116. The packing unit 113 further supplies the occupancy map generated in this manner to the OMap encoding unit 117. The packing unit 113 also supplies control information regarding such packing to the multiplexer 118.

[0126] The auxiliary patch information compression unit 114 performs a process related to compression of the auxiliary patch information. For example, the auxiliary patch information compression unit 114 acquires data supplied from the patch separation unit 112. The auxiliary patch information compression unit 114 encodes (compresses) the auxiliary patch information included in the acquired data. The auxiliary patch information compression unit 114 supplies the resultant encoded data of the auxiliary patch information to the multiplexer 118.

[0127] The video encoding unit 115 performs a process related to encoding of the video frame of positional information (Geometry). For example, the video encoding unit 115 acquires the geometry video frame supplied from the packing unit 113. The video encoding unit 115 also encodes the acquired geometry video frame by an appropriate encoding method for two-dimensional images, such as AVC or HEVC, for example. The video encoding unit 115 supplies the encoded data obtained by the encoding (encoded data of the geometry video frame), to the multiplexer 118.

[0128] The video encoding unit 116 performs a process related to encoding of the video frame of attribute information (Texture). For example, the video encoding unit 116 acquires the color video frame supplied from the packing unit 113. The video encoding unit 116 also encodes the acquired color video frame by an appropriate encoding method for two-dimensional images, such as AVC or HEVC, for example. The video encoding unit 116 supplies the encoded data obtained by the encoding (encoded data of the color video frame), to the multiplexer 118.

[0129] The OMap encoding unit 117 performs a process related to encoding of the occupancy map. For example, the OMap encoding unit 117 acquires the occupancy map supplied from the packing unit 113. The OMap encoding unit 117 also encodes the acquired occupancy map by an appropriate encoding method such as arithmetic encoding, for example. The OMap encoding unit 117 supplies the encoded data obtained by the encoding (the encoded data of the occupancy map), to the multiplexer 118.

[0130] The multiplexer 118 performs a process related to multiplexing. For example, the multiplexer 118 acquires the encoded data of the auxiliary patch information supplied from the auxiliary patch information compression unit 114. The multiplexer 118 also acquires the control information regarding packing supplied from the packing unit 113. The multiplexer 118 further acquires the encoded data of the geometry video frame supplied from the video encoding unit 115. The multiplexer 118 also acquires the encoded data of the color video frame supplied from the video encoding unit 116. The multiplexer 118 further acquires the encoded data of the occupancy map supplied from the OMap encoding unit 117. The multiplexer 118 also acquires BB information supplied from the BB information generation unit 119.

[0131] The multiplexer 118 multiplexes those acquired pieces of information, to generate a bitstream (Bitstream). The multiplexer 118 outputs the generated bitstream to the outside of the encoding device 100.

[0132] The BB information generation unit 119 performs a process related to generation of BB information. For example, the BB information generation unit 119 acquires the transform information supplied from the model transform unit 111. The BB information generation unit 119 also acquires the model information supplied from the packing unit 113. The BB information generation unit 119 generates the BB information containing the acquired transform information and model information. The BB information generation unit 119 supplies the generated BB information to the multiplexer 118. That is, the BB information is transmitted to the decoding side.

[0133] With such a configuration, the encoding device 100 can transform and combine a plurality of point cloud models of a point cloud into one group, encode the group, and store the result into one bitstream. That is, even when a plurality of point cloud models exists in a point cloud, it is possible to prevent an increase in the number of bitstreams while preventing a decrease in coding efficiency. Accordingly, the encoding device 100 can prevent an increase in the number of instances necessary for decoding. That is, an increase in the load of the decoding process can be prevented. Thus, an increase in the cost of the decoder can be prevented. Further, an increase in the processing time of the decoding process can be prevented.

[0134]

[0135] Next, an example flow in an encoding process to be performed by the encoding device 100 is described, with reference to the flowchart shown in FIG. 5.

[0136] When the encoding process is started, the model transform unit 111 of the encoding device 100 performs a transform process in step S101, to transform and combine a plurality of point cloud models included in a point cloud into one group. The model transform unit 111 also generates the transform information regarding the transform.

[0137] In step S102, the patch separation unit 112 projects (the group of) the plurality of point cloud models combined in step S101 onto a two-dimensional plane, and separates the point cloud models into patches. In doing so, the patch separation unit 112 may set a projection plane for each point cloud model by the method described above in and the like, for example. In this manner, the patch separation unit 112 can project each point cloud model onto more appropriate projection planes. Thus, the decrease in coding efficiency due to inefficiency in the projection of the point cloud models can be prevented (the coding efficiency can be increased).

[0138] The patch separation unit 112 may also generate projection plane information regarding the setting of the projection plane by the method described above in and the like, for example, and transmit the projection plane information to the decoding side. As a result, the decoding side can perform correct decoding (correctly reconstruct the point cloud).

[0139] The patch separation unit 112 also generates the auxiliary patch information regarding the separation.

[0140] In step S103, the auxiliary patch information compression unit 114 compresses (encodes) the auxiliary patch information generated in step S102.

[0141] In step S104, the packing unit 113 performs a packing process, to arrange the respective patches of the positional information and the attribute information generated in step S102 in two-dimensional images, and pack the patches as video frames. The packing unit 113 also generates the model information and the occupancy map. The packing unit 113 further performs a dilation process on the color video frame. The packing unit 113 also generates the control information regarding such packing.

[0142] In step S105, the BB information generation unit 119 performs a BB information generation process, to generate the BB information containing the transform information generated in step S101, the model information generated in step S104, and the like.

[0143] In step S106, the video encoding unit 115 encodes the geometry video frame generated in step S104, by an encoding method for two-dimensional images. In doing so, the video encoding unit 115 encodes the geometry video frame, in accordance with the setting of the unit of encoding that has the respective point cloud model patches arranged therein in step S104 and can be independently decoded (encoding-unit region setting). That is, when slices or tiles are set, for example, the geometry video frame is encoded independently slice by slice or tile by tile.

[0144] In step S107, the video encoding unit 116 encodes the color video frame generated in step S104, by an encoding method for two-dimensional images. In doing so, the video encoding unit 116 encodes the color video frame, in accordance with the setting of the unit of encoding that has the respective point cloud model patches arranged therein in step S104 and can be independently decoded (encoding-unit region setting). That is, when slices or tiles are set, for example, the color video frame is encoded independently slice by slice or tile by tile.

[0145] In step S108, the OMap encoding unit 117 encodes the occupancy map generated in step S104, by a predetermined encoding method.

[0146] In step S109, the multiplexer 118 multiplexes the various kinds of information generated as described above (for example, the encoded data of the auxiliary patch information generated in step S103, the control information regarding the packing generated in step S104, the BB information generated in step S105, the encoded data of the geometry video frame generated in step S106, the encoded data of the color video frame generated in step S107, the encoded data of the occupancy map generated in step S108, and the like), to generate a bitstream containing these pieces of information.

[0147] In step S110, the multiplexer 118 outputs the bitstream generated in step S109, to the outside of the encoding device 100.

[0148] When the process in step S110 is completed, the encoding process comes to an end.

[0149]

[0150] Next, an example flow in the transform process to be performed in step S101 in FIG. 5 is described, with reference to the flowchart shown in FIG. 6.

[0151] When the transform process is started, the model transform unit 111 performs a bounding box setting process in step S131, to set a bounding box for each point cloud model.

[0152] In step S132, the model transform unit 111 transforms a parameter of each bounding box (each point cloud model) set in step S131. For example, the model transform unit 111 transforms a parameter such as shift, rotation, time stump, scale, or frame rate as described above in and the like. By doing so, the model transform unit 111 can prevent an increase in the number of bitstreams while preventing a decrease in coding efficiency, even when a plurality of point cloud models exists in the point cloud. Thus, the model transform unit 111 can prevent an increase in the number of instances necessary for decoding. That is, an increase in the load of the decoding process can be prevented. Thus, an increase in the cost of the decoder can be prevented. Further, an increase in the processing time of the decoding process can be prevented.

[0153] In step S133, the model transform unit 111 generates transform information about the parameter of each bounding box as described above in and the like. For example, the model transform unit 111 generates transform information indicating the ratio, the difference, or the like between before and after the transform. By doing so, the model transform unit 111 can enable the decoding side to perform correct inverse transform.

[0154] When the process in step S133 is completed, the transform process comes to an end.

[0155]

[0156] Next, an example flow in the bounding box setting process to be performed in step S131 in FIG. 6 is described, with reference to the flowchart shown in FIG. 7.

[0157] When the bounding box setting process is started, the model transform unit 111 derives the normal vector of the portion of each point cloud model included in the point cloud in step S141.

[0158] In step S142, the model transform unit 111 sets bounding boxes so that the one having a large distribution in the normal direction derived in step S141 matches the orthogonal projection vector. When the process in step S142 is completed, the bounding box setting process comes to an end, and the process returns to FIG. 6.

[0159]

[0160] Next, an example flow in the packing process to be performed in step S104 in FIG. 5 is described, with reference to the flowchart shown in FIG. 8.

[0161] When the packing process is started, the packing unit 113 sets encoding-unit regions in accordance with the point cloud models in step S151, to allocate different encoding-unit regions to the respective point cloud models.

[0162] In step S152, for each patch of each point cloud model, the packing unit 113 searches for an optimum position in the encoding-unit region allocated to the point cloud model in step S151, and places the patch at the optimum position.

[0163] That is, the packing unit 113 places the patches belonging to the same point cloud model in the same independently decodable unit of encoding, by the method described above in and the like. By doing so, the packing unit 113 generates a geometry video frame and a color video frame. As a result, the decoding side can perform so-called “partial decoding” for reconstruction by only partially decoding the point cloud models.

……
……
……