LG Patent | Point cloud data processing device and processing method

Here, X_length denotes the length along the x-axis, Y_length denotes the length along the y-axis, and Z_length denotes the length along the z-axis. The length along each axis may be secured through content scanning, or may be signaled to the point cloud decoder (or the point cloud data reception device) by the bitstream described with reference to FIGS. 1 to 14.

The data distribution processor 1710 may compare the lengths along the respective axes to determine an axis that may form the basis for splitting. When the axis that is the basis for splitting is determined, the remaining axes that are not the basis maintain the values corresponding to the size of the original node or bounding box. When the lengths along two axes are equal and the length along the other axis is less than those of the two axes, PU splitting is performed based on the two axes at the same time, and the remaining axis maintains the value corresponding to the original node or bounding box size. When the lengths along the three axes or the two axes are less than a preset threshold value, the three axes or the two axes may be split by the same method/reference/mode or the splitting may be skipped.

The split mode determiner 1720 according to the embodiments determines the shape of the PU. The shape of the PU may be determined according to a split mode. In other words, the PU has a shape of a cube, a cuboid, or the like according to the split mode. The split mode determiner 1720 calculates a residual between the reference frame/intra prediction frame and the current frame when the PU is split, finds a split mode with the smallest residual, and determines the mode as the split mode. The determined split mode and residual are transmitted to the motion estimation processor 1730.

According to the embodiments, there are five split modes called Mode 0 to Mode 4. The number and names of the split modes are not limited to this example.

Mode 0: Indicates that the splitting is skipped. This mode is applied when there is no difference in length between the three axes described above or the length do not exceed a preset threshold value. However, Mode 0 is not limited to this example and may be applied in other situations. Accordingly, the box shown in the figure represents one PU. Mode 0 represents a method of finding an MV by setting a searching range without splitting the PU, and the corresponding information (e.g., Mode0 information, etc.) is delivered to the point cloud encoder and the point cloud decoder. When PU splitting is recursively applied, Mode 0 may be applied to a sub PU that is generated from the PU in a split mode (e.g., Mode 1) other than Mode 0.

Mode 1: Indicates 1:1 splitting. The expression X:Y represents a relative difference (or ratio) between the size (or dimensions) of a sub PU close to the origin of the splitting reference axis and the size (or dimensions) of a sub PU close to the origin of the splitting reference axis. According to embodiments, X corresponds to the size of a sub PU close to the origin of the splitting reference axis, and Y corresponds to the size of a sub PU far from the origin of the splitting reference axis. In the case of Mode 1, the difference in size between a sub PU close to the origin (referred to as, for example, a first sub PU) and a sub PU far from the origin (referred to as, for example, a second sub PU) is 1:1, and thus the first PU and the second sub PU have the same size. Mode 1 is applied when there is no difference in length between the two axes or the lengths do not exceed a preset threshold value. However, Mode 1 is not limited to this example and may be applied in other situations as well. A PU is split into halves (½) based on one axis, and each sub PU generated by the splitting is configured as a searching range to find an MV, and then motion compensation is performed. Thereafter, the generated residual value and the split mode value are delivered to the point cloud encoder and the point cloud decoder. The sub PUs split according to the embodiments may have different MVs. Also, each sub PU may be further split or splitting thereof may be skipped according to a different condition or split mode. The MVs may be defined for each PU or may have the same vector information.

Mode 2: Indicates 1:3 splitting. Since the difference in size between a sub PU close to the origin (referred to as, for example, a first sub PU) and a sub PU far from the origin (referred to as, for example, a second sub PU) is 1:3, the second sub PU is larger than the first sub PU. Mode 2 is applied when the length or range along a specific axis is greater or less than the those along the other two axes. However, Mode 2 is not limited to this example and may be applied in other situations. Two sub PUs generated by splitting may have different MVs, may be further split under different conditions or modes, or the splitting thereof may be skipped. The MVs may be defined for each sub PU or may have the same vector information.

Mode 3: Indicates 3:1 splitting. Since the difference in size between a sub PU close to the origin (referred to as, for example, a first sub PU) and a sub PU far from the origin (referred to as, for example, a second sub PU) is 3:1, the second sub PU is smaller than the first sub PU. Mode 3 is applied when the length or range along a specific axis is greater or less than the those along the other two axes. However, Mode 3 is not limited to this example and may be applied in other situations. Two sub PUs generated by splitting may have different MVs, may be further split under different conditions or modes. The MVs may be defined for each sub PU or may have the same vector information.

Mode 2 and Mode 3 are selected according to the data distribution.

Mode 4: 1:2:1 split, indicating that a PU is split into three sub PUs. Since the difference in size between a first sub PU close to the origin, a second sub PU positioned in the center, and a third sub PU farthest from the origin is 1:2:1, the second sub PU is relatively larger than the first sub PU and the third sub PU. Mode 4 may be determined by a split cost and is used when the sub PUs have significantly different MVs. However, Mode 4 is not limited to this example and may be applied in other situations. The three sub PUs generated by the splitting may have different MVs, may be further split under different conditions or modes, or the splitting thereof may be skipped.

The motion estimation processor 1730 according to the embodiments performs motion estimation (or motion vector estimation). The MVs extracted according to the motion estimation, and a cost value for determining additional splitting and/or skip are delivered to the sub PU split determiner 1740.

The sub PU split determiner 1740 checks a cost caused by application of the MVs extracted according to the motion estimation, and determines whether to perform additional splitting into sub PUs or skip the splitting according to the cost. The sub PU split determiner 1740 delivers the corresponding information to the data distribution processor 1710 when additional splitting is performed. The data distribution processor 1710 recursively performs an operation for splitting again. In a case of splitting according to the cost calculation, when splitting is performed based on the remaining axes except for the axis used as the basis of splitting in the previous step without considering data distribution belonging to each sub PU component, the operation of the data distribution processor 1710 is skipped and information related to the additional splitting is delivered to the split mode determiner 1720. The split mode determiner 1720 recursively performs the operation for splitting again.

FIG. 18 illustrates an example of PU splitting according to embodiments.

FIG. 18 illustrates an example of PU splitting along each axis. The left part of the figure shows a PU that is split according to Mode 2 described with reference to FIG. 17 based on the x-axis for the remaining axes. The middle part of the figure shows a PU that is split according to Mode 2 described with reference to FIG. 17 based on the z-axis for the remaining axes. The right part of the figure shows a PU that is split according to Mode 2 with reference to FIG. 17 based on the y-axis for the remaining axes. As described above, according to the splitting, each PU is split into two sub PUs, and the difference in size between the two sub PUs is represented as 1:3.

FIG. 19 illustrates an example of PU splitting according to embodiments.

FIG. 19 sequentially shows examples of PU splitting performed by applying Mode 0 to Mode 4 described with reference to FIG. 17 based on the x-axis. Descriptions of Mode 0, Mode 1, Mode 2, Mode 3, and Mode 4 are the same as those of FIGS. 17 to 18 and thus will be skipped.

FIG. 20 illustrates an example of PU splitting according to embodiments.

FIG. 20 illustrates an example of the PU 2000 corresponding to Mode 0 and examples in which Mode 1, Mode 2, Mode 3, and Mode 4 described with reference to FIG. 17 are applied for each of the x-axis, y-axis, and z-axis. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the x-axis are represented as Mode 1x, Mode 2x, Mode 3x, and Mode 4x, respectively. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the y-axis are represented as Mode 1 y, Mode 2y, Mode 3y, and Mode 4y, respectively. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the z-axis are represented as Mode 1z, Mode 2z, Mode 3z, and Mode 4z, respectively. For the two or three split sub PUs, each sub PU may be split according to any one of Mode 1, Mode 2, Mode 3, and Mode 4.

FIG. 21 illustrates an example of PU splitting according to embodiments.

FIG. 21 illustrates 16 splitting examples in which Mode 1, Mode 2, Mode 3, and Mode 4 described with reference to FIG. 17 are applied, with respect to each of the x-axis, y-axis, and z-axis, to a sub PU 2100 positioned in the middle among the three sub PUs split according to Mode 4 (Mode 4x) based on the x-axis described with reference to FIG. 20. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the x-axis are represented as Mode 4x1x, Mode 4x2x, Mode 4x3x, and Mode 4x4x, respectively. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the y-axis are represented as Mode 4x1y, Mode4x2y, Mode 4x3y, and Mode 4x4y, respectively. Mode 1, Mode 2, Mode 3, and Mode 4 applied based on the z-axis are represented as Mode 4x1z, Mode 4x2z, Mode 4x3z, and Mode 4x4z, respectively.

As shown in the figure, when additional splitting is applied, the axis serving as a reference may be the reference (e.g., the x axis) according to the previous splitting or may be another axis. The axis serving as a basis for applying the additional splitting may be determined by checking and comparing the data distributions for the respective sub PUs or the lengths and/or ranges of the data, or may be preset before PU splitting. Information about the preset axis is signaled to the point cloud encoder described with reference to FIGS. 1 to 14 or signaled to the point cloud decoder through the bitstream described with reference to FIGS. 1 to 14.

FIG. 22 illustrates an example of PU splitting according to embodiments.

The left part of FIG. 22 shows three sub PUs, a first sub PU 2200, a second sub PU 2210, and a third sub PU 2230 which are split according to Mode 4 (Mode 4x) applied based on the x-axis described with reference to FIG. 20. The right part of the figure shows an example in which the first sub PU 2200, the second sub PU 2210, and the third sub PU 2230 are each further split. As shown in the figure, the first sub PU 2200 may be split according to Mode 4 based on the y-axis. Accordingly, the split mode of the first sub PU 2200 is represented as 4x4y (or Mode4x4y). The second sub PU 2210 is split according to Mode 1. Then, the first sub PU may not be split, and the second sub PU may be split according to Mode 1 based on the z-axis. Therefore, the final split mode of the second sub PU 2201 is represented as 4x1x01z (or Mode 4x1x01). The third Sub PU 2230 may be split according to Mode 3 based on the y-axis. Therefore, the final split mode of the third sub PU 2230 is represented as 4x3y (or Mode 4x3y). As shown in the figure, after PU splitting, each sub PU may be split according to the same or different split mode, and thus the shapes of the sub PUs of each sub PU may be different. This splitting configuration may be easily applied to segmentation of point cloud data. FIG. 22 is an example, and PUs having various shapes and sizes may be applied by combining the various split modes described with reference to FIGS. 17 to 21.

FIG. 23 illustrates an example of a split mode determination method.

As described above, the PU may be split in various directions and shapes. The point cloud transmission device and the reception device according to the embodiments perform a process for determining a split mode in order to determine the best mode (or best split mode). As described above, the split mode determiner according to the embodiments may perform PU spitting for a specific node in a frame or octree structure. FIG. 23 is a flowchart 2300 illustrating a method of determining a split mode by a split mode determiner (e.g., the split mode determiner 1720 described with reference to FIG. 17) for a bounding box corresponding to one frame.

The split mode determiner according to the embodiments may split the bounding box for each mode (2310). The split mode determiner selects points corresponding to a window (2320) and extracts points (block) present in a sub PU (2330). The window according to the embodiments represents a region including the sub PU and search range (or node or frame) of a region at the same position in the reference frame. The block according to the embodiments represents a sub PU of a PU in a region (or node, frame) that is a target of the PU in the current frame. The sub PU according to the embodiments represents each element constituting a PU that is split in any split mode. Each PU and sub PU are configured not to overlap each other within the same frame. Since the window includes a search range, it may overlap with another window within the same reference frame. The window according to the embodiments is configured by adding the search range to a region at the same position as the sub PU in order to include a region in which the sub PU may be shifted. The search range may be signaled to the point cloud decoder as preconfigured information by a bitstream, or may be adaptively changed according to the size of the PU.

The split mode determiner according to the embodiments may calculate a cumulated distance between the block and the window for cost calculation for each mode (2340). The distance calculated for each mode is represented as mode A.distance.

The split mode determiner may calculate the cumulated distance for each mode as a value of rate distortion optimization (RDO), and then determine the mode having the lowest cost as the best mode (represented as BestSplit_Mode) (2350).

The following equation represents a method of determining the best mode.

ModeX.distance=∑knumBlock∑lBlock⁢_⁢size∑mwindow⁢_⁢size❘"\[LeftBracketingBar]"blockk,.xl-windowk.xm❘"\[RightBracketingBar]"+❘"\[LeftBracketingBar]"blockk.yl-windowm.ym❘"\[RightBracketingBar]"+❘"\[LeftBracketingBar]"blockk.zl-window.zm❘"\[RightBracketingBar]" BestMode=argminX(RDCos⁢tModeX) RDCostMode=ModeX.distance+λ×R

In the equation, X denotes the mode value (e.g., 0, 1, . . . ), and numBlock denotes the number of blocks constituting modeX. Block_size denotes the total number of points included in a block, and window_size denotes the total number of points included in a window. Also, blockk.x₁, block.y₁, and block.z₁ denote the x, y, z coordinate information related to the 1-th point included in the block, and window.x_m, window.y_m, and window.z_m denote the x, y, z coordinate information related to the m-th point included in the window. Mode_X denotes information corresponding to Modes 0 to 4, λ denotes the Lagrange multiplier, and R denotes the number of bits used to split the mode.

FIG. 24 illustrates an example of a method of determining a split mode.

FIG. 24, shows another example of FIG. 23, is a flowchart 2400 illustrating a method of determining a split mode by a split mode determiner (e.g., the split mode determiner 1720 described with reference to FIG. 17) for applying PU splitting from a specific node size.

When the octree splitting is performed until a specific node size is reached (2410), the split mode determiner according to the embodiments may split the 3D box corresponding to the specific node for each mode (2420). The split mode determiner selects points corresponding to a window (2430) and extracts points (block) present in a sub PU (2440). The windows and blocks according to the embodiments are the same as those described with reference to FIG. 23 and thus a description thereof will be skipped.

The split mode determiner according to embodiments may calculate a cumulated distance between the block and the window for cost calculation for each mode (2450). The distance calculated for each mode is represented as mode A.distance.

split mode determiner may calculate the cumulated distance for each mode as a value of rate distortion optimization (RDO), and then determine the mode having the lowest cost as the best mode (BestSplit_Mode) (2460). The method of determining the best mode is the same as that described with reference to FIG. 24, and thus a detailed description thereof will be skipped.

FIG. 25 shows an example of a window and a block.

The left part of FIG. 25 is an example 2500 illustrating a relationship between a window and a block.

The window according to the embodiments represents a region including the sub PU and search range (or node or frame) of a region at the same position in the reference frame. The block according to the embodiments represents a sub PU of a PU in a region (or node, frame) that is a target of the PU in the current frame. The sub PU according to the embodiments represents each element constituting a PU that is split in any split mode. Thus, a window may include blocks.

The example 2510 in the middle of FIG. 25 shows three sub PUs generated by splitting a PU according to Mode 4 based on the x-axis in order for the split mode determiner to calculate the cost for each mode as described with reference to FIGS. 23 and 24. The sub PUs are blocks, and are represented as block 0, block 1, and block 2, respectively. Window 0, window 1, and window 2, which are windows of each block 0, block 1, and block 2, are determined.

Window 0, window 1, and window 2 are regions obtained by adding a search range to the regions of Block0, Block1, and Block2, respectively, and the points of the reference frame positioned in the regions are included.

An example 2520 on the right side of FIG. 25 shows window 0, which is a window of block 0. As shown in the figure, the region of window 0 is larger than the region of block 0. As described above, the split mode determiner may calculate distances between all points included in the two regions of window 0 and block 0 according to Mode 4.

The point cloud data processing device (e.g., the transmission device described with reference to FIGS. 1, 12 and 14) may transmit the encoded point cloud data in the form of a bitstream. The bitstream is a sequence of bits that forms a representation of the point cloud data (or point cloud frame).

The point cloud data (or point cloud frame) may be partitioned into tiles and slices.

The point cloud data may be partitioned into multiple slices, and is encoded in a bitstream. One slice is a set of points and is expressed as a series of syntax elements representing all or part of the encoded point cloud data. A slice may or may not have dependencies on other slices. In addition, a slice includes one geometry data unit, and may have zero or one or more attribute data units. As described above, since attribute encoding is performed based on geometry encoding, the attribute data units are based on geometry data units in the same slice. That is, the point cloud data reception device (e.g., the reception device 10004 or the point cloud video decoder 10006) may process attribute data based on the decoded geometry data. Accordingly, in a slice, a geometry data unit must precede the associated attribute data units. The data units in the slice are necessarily contiguous, and the order of the slices is not specified.

A tile is a (three-dimensional) cuboid in a bounding box (e.g., the bounding box described with reference to FIG. 5). The bounding box may include one or more tiles. A tile may fully or partially overlap another tile. One tile may include one or more slices.

Accordingly, the point cloud data transmission device may provide high-quality point cloud content by processing data corresponding to a tile according to the importance. That is, the point cloud data transmission device may perform point cloud compression coding with better compression efficiency and appropriate latency on data corresponding to an area important to a user.

The bitstream according to the embodiments contains signaling information and a plurality of slices (slice 0, . . . , slice n). As shown in the figure, the signaling information precedes the slices in the bitstream. Accordingly, the point cloud data reception device may first secure the signaling information and sequentially or selectively process the plurality of slices based on the signaling information. As shown in the figure, slice0 includes one geometry data unit Geom0⁰ and two attribute data units Attr0⁰ and Attr1⁰. Also, the geometry data unit precedes the attribute data units in the same slice. Accordingly, the point cloud data reception device processes (decodes) the geometry data unit (or geometry data), and then processes the attribute data units (or attribute data) based on the processed geometry data. The signaling information according to the embodiments may be referred to as signaling data, metadata, or the like, but is not limited thereto.

According to embodiments, the signaling information includes a sequence parameter set (SPS), a geometry parameter set (GPS), and one or more attribute parameter sets (APSs). The SPS is encoding information about the entire sequence, such as a profile or a level, and may include comprehensive information (sequence level) about the entire sequence, such as a picture resolution and a video format. The GPS is information about geometry encoding applied to geometry included in the sequence (bitstream). The GPS may include information about an octree (e.g., the octree described with reference to FIG. 6) and information about an octree depth. The APS is information about attribute encoding applied to an attribute included in the sequence (bitstream). As shown in the figure, the bitstream contains one or more APSs (e.g., APS0, APS1, . . . in the figure) according to an identifier for identifying the attribute.

The signaling information according to embodiments may further include information about a tile (e.g., tile inventory, a tile parameter set). The information about the tile may include information about a tile identifier, a tile size, and the like. According to embodiments, the signaling information is applied to a corresponding bitstream as information about a sequence, that is, a bitstream level. In addition, the signaling information has a syntax structure including a syntax element and a descriptor describing the same. A pseudo code may be used to describe the syntax. In addition, the point cloud reception device (e.g., the reception device 10004 of FIG. 1, the point cloud decoder of FIGS. 10 and 11, or the reception device of FIG. 13) may sequentially parse and process the syntax elements in the syntax.

Although not shown in the figure, the geometry data unit and the attribute data unit include a geometry header and an attribute header, respectively. The geometry header and the attribute header have the above-described syntax structure as signaling information applied at the slice level.

The geometry header includes information (or signaling information) for processing a corresponding geometry data unit. Therefore, the geometry header appears first in the geometry data unit. The point cloud reception device may process the geometry data unit by first parsing the geometry header. The geometry header has an association with the GPS, which contains information about the entire geometry. Accordingly, the geometry header contains information specifying gps_geom_parameter_set_id included in the GPS. In addition, the geometry header contains tile information (e.g., tile_id), a tile identifier, and the like related to a slice to which the geometry data unit belongs.

The attribute header contains information (or signaling information) for processing a corresponding attribute data unit. Accordingly, the attribute header appears first in the attribute data unit. The point cloud reception device may process the attribute data unit by parsing the attribute header first. The attribute header has an association with the APS, which contains information about all attributes. Accordingly, the attribute header contains information specifying aps_attr_parameter_set_id included in the APS. As described above, attribute decoding is based on geometry decoding. Accordingly, the attribute header contains information specifying a slice identifier contained in the geometry header in order to determine a geometry data unit associated with the attribute data unit.

When the point cloud transmission device performs PU splitting described with reference to FIGS. 15 to 25, signaling information in the bitstream may further include signaling information related to the PU splitting (e.g., information about whether to apply the inter prediction, the split mode, and the split mode determination method as described above). Signaling information related to projection according to embodiments may be included in sequence-level signaling information (e.g., SPS, APS, etc.), a slice level (e.g., an attribute header, etc.), an SEI message, or the like. The point cloud reception device according to the embodiments may perform decoding including reverse projection based on the signaling information related to PU splitting.

The signaling information related to PU splitting according to the embodiments may be included in signaling information of various levels (e.g., a sequence level, a slice level, etc.). The signaling information related to PU splitting is transmitted to the point cloud reception device (e.g., the reception device 10004 of FIG. 1, the point cloud decoder of FIGS. 10 and 11, or the reception device of FIG. 13).

FIG. 26 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 26 shows an example of a syntax structure of an SPS in which signaling information related to PU splitting is included at the sequence level.

pu_coding_flag: Indicates whether inter prediction is performed in a PU. When the value of pu_coding_flag is TRUE, it indicates that inter prediction is performed in the PU. When the value is FALSE, it indicates that inter prediction is not performed in the PU.

frame_pu_split_flag: Indicates whether inter prediction is performed by splitting the entire frame into PUs. When the value of frame_pu_split_flag is TRUE, the entire frame is split into Pus. When the value is FALSE, PU splitting is performed at the slice level or a specific octree level.

split_pu_initial_criteria: Indicates which of the three axes first serves as a reference for the splitting. The axes serving as the reference according to specific values are shown in the table below.

TABLE 1 split_pu_initial_criteria Description 0 Width 1 Height 2 Depth 3-7 Reserved

In the table, the left column shows the values of split_pu_initial_criteria, and the right column shows the reference axis (or the width, height, or depth of the bounding box corresponding to the entire frame or the bounding box corresponding to a specific octree level) according to the value of split_pu_initial_criteria. According to embodiments, the values of split_pu_initial_criteria are changeable.

split_pu_criteria_update_flag: Indicates whether an axis serving as a reference for additional splitting (splitting of the next level) is newly searched for. When the value of split_pu_criteria_update_flag is TRUE, a new axis for the criterion of additional splitting is searched for. When the value is FALSE, the current reference axis is continuously used.

split_pu_threshold: In determining a new reference axis when searching for a reference axis or performing additional splitting, when the difference in length of one or more reference axes is less than a threshold, splitting is performed using the same method, or splitting is performed with respect to the remaining axes. The corresponding value is predefined.

initial_split_mode: Indicates a PU split type or mode. The modes according to the values of initial_split_mode are specified as shown in the table below.

In the table, the left column shows values of initial_split_mode, and the right column shows a ratio indicating the difference in size between sub PUs corresponding to the split mode described with reference to FIGS. 19 to 22 according to the value of initial_split_mode. For example, initial_split_mode equal to 0 may indicate the split mode according to Mode 0 described with reference to FIGS. 19 to 22, that is, no split. Also, initial_split_mode equal to 1 indicates 1:1 according to the split mode according to Mode 1 described with reference to FIG. 19. According to embodiments, the split mode indicated by each value of initial_split_mode may be changed, and the representations are not limited to this example.

initial_length_width, initial_length_height, initial_length_depth: Variables indicating distribution of the entire data in each axial frame of the PU splitting, representing the length, data distribution, or range.

initial_length_width indicates the range of data distributed in the width direction (max−min), initial_length_height indicates the range of data distributed in the height direction (max−min), and initial_length_depth indicates the range of data distributed in the depth direction. The values may be replaced with any values that may be used to calculate and compare the data distribution.

split_pu_max_node_size: Indicates the size of the largest PU when PU splitting is performed in a slice unit or at a specific level (or node corresponding to the level) in the octree structure.

max_split_pu_level: Indicates the maximum number of PU splits. The value of max_split_pu_level may be adaptively changed according to the total number of points, or may be pre-input and predetermined.

The SPS syntax according to embodiments is not limited to the above example, and may further include additional elements or may exclude some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the SPS.

FIG. 27 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 27 shows an example of a syntax structure of a tile parameter set in which signaling information related to PU splitting is included at a tile level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and thus a description thereof will be skipped. The syntax of the tile parameter set according to the embodiments is not limited to the above example. The syntax may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the tile parameter set.

FIG. 28 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 28 shows an example of a syntax structure of a GPS in which signaling information related to PU splitting is included at the sequence level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and thus a description thereof will be skipped. The GPS syntax according to the embodiments is not limited to the above example. The syntax may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the GPS.

FIG. 29 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 29 shows an example of a syntax structure of an APS in which signaling information related to PU splitting is included at the sequence level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and thus a description thereof will be skipped. The GPS syntax according to the embodiments is not limited to the above example. The syntax may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information, for example, a geometry slice data unit other than the APS.

FIG. 30 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 30 shows an example of a syntax structure of a geometry slice header in which signaling information related to PU splitting is included at the slice level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and thus a description thereof will be skipped. The syntax of the geometry slice header according to the embodiments is not limited to the above example. The syntax may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information, for example, a geometry slice data unit other than the geometry slice header.

FIG. 31 shows an example of a syntax structure of signaling information related to PU splitting.

FIG. 31 shows an example of a syntax structure of geometry slice data in which signaling information related to PU splitting is included at the slice level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and thus a description thereof will be skipped. The syntax of the geometry slice data according to the embodiments is not limited to the above example. The syntax may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information (e.g., an attribute header, etc.) or an attribute data unit other than the geometry slice data.

FIG. 32 is a flow diagram illustrating a point cloud processing method according to embodiments.

The flow diagram 3200 of FIG. 32 illustrates a processing method for a point cloud data processing device (e.g., the point cloud data transmission device, the point cloud encoder, the geometry encoder, or the like described with reference to FIGS. 1 to 14) configured to perform the PU splitting described with reference to FIGS. 15 to 31. The point cloud processing method illustrated in the figure may include the method of determining the split mode described with reference to FIGS. 17 and 24 to 25.

The point cloud data processing device (e.g., the geometry encoder) according to the embodiments receives point cloud data as an input and performs data quantization/voxelization to facilitate compression of the geometry of the input point cloud data (3210). The voxelized geometry has an octree structure.

The geometry encoder according to the embodiments may include at least one of the coordinate transformer 40000, the quantizer 40001, the octree analyzer 40002, the surface approximation analyzer 40003, the arithmetic encoder 40004, and the geometry reconstructor 40005 described with reference to FIG. 4. The geometry encoder according to the exemplary embodiments may include at least one of the data input unit 12000, the quantization processor 12001, the voxelization processor 12002, the octree occupancy code generator 12003, and the surface model processor 12004, the intra/inter-coding processor 12005, and the arithmetic coder 12006 described with reference to FIG. 12. Although not shown in the figures, the geometry encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The point cloud data processing device (e.g., the geometry encoder) according to embodiments performs the PU splitting described with reference to FIGS. 15 to 31 (3220). The PU splitting process according to the embodiments is the same as or similar to the method of determining the split mode described with reference to FIGS. 17 and 24 to 25. Also, the PU splitting process 3220 is included in the geometry encoding process described with reference to FIGS. 1 to 15. The PU splitting process 3220 includes the following operations. The point cloud data processing device may determine whether to perform PU splitting for the entire frame or a specific node size in the octree structure, and determine a PU split range (search range). Then, it may identify the characteristics of distribution of the point cloud data in the corresponding region and determine a PU split range (search range) and axis (axes) (3221). Information related to PU splitting may be signaled at various levels such as a sequence level and a tile level through a bitstream. Since the information related to PU splitting is the same as that described with reference to FIGS. 26 to 31, a description thereof will be skipped. When the point cloud data processing device determines an axis serving as a splitting criterion, it calculates the cost required for each mode when splitting is performed according to the split modes (e.g., Mode 0, Mode 1, Mode 2, etc.) described with reference to FIGS. 17 to 24, searches for best split mode with the lowest cost, and perform the splitting (3222). The point cloud data processing device performs motion estimation for each component on each of the split sub PUs (3223). When the motion estimation is completed, the device determines whether to perform additional splitting for each sub PU (3224). The point cloud data processing device may perform or skip the additional splitting. When the additional splitting is performed, the point cloud data processing device may determine the PU split range again according to the dotted arrow, identify the distribution characteristics of the point cloud data in the corresponding region, and determine an axis forming the basis of the splitting. When motion estimation for each PU is completed, the point cloud data processing device performs motion compensation using the obtained MV (3230), and perform attribute encoding based on the reconstructed geometry (or reconstructed point cloud data) (3240). The attribute encoding may be performed by transforming, quantizing, and entropy coding a residual between the source data and the predicted data.

The PU splitting process described with reference to FIGS. 15 to 31 may be applied to attribute encoding. The best mode according to the embodiments is determined based on the cost calculated based on an attribute value of a point present within a search range, rather than a distance between points. In the processing method illustrated in the figure, the respective operations may be performed sequentially, or at least one or more thereof may be performed simultaneously. The processing order of the operations may be changed.

FIG. 33 is a flow diagram illustrating a point cloud processing method according to embodiments.

The flow diagram 3300 of FIG. 33 illustrates a processing method for a point cloud data processing device (e.g., the point cloud data reception device, the geometry decoder, the attribute decoder, or the like described with reference to FIGS. 1 to 14) configured to perform the PU splitting described with reference to FIGS. 15 to 31. The point cloud processing method illustrated in the figure may include the method of determining the split mode described with reference to FIGS. 17 and 24 to 25.

The point cloud data processing device (e.g., the geometry decoder) according to the embodiments may perform entropy decoding on the geometry contained in a received bitstream (3310), perform dequantization (3330) and inverse transformation (3340) to restore a residual, which is a prediction error for each point.

The geometry decoder according to the embodiments may include at least one of the arithmetic decoder 13002, the occupancy code-based octree reconstruction processor 13003, the surface model processor 13004, and the inverse quantization processor 13005 described with reference to FIG. 12. Although not shown in the figure, the geometry decoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The point cloud data processing device (e.g., the geometry decoder) may secure motion vector (MV) information (or MV) signaled for each PU and predict a point value in the PU range of the current frame (3320). The point cloud data processing device outputs the reconstructed point cloud geometry (or reconstructed geometry) by summing the predicted point value and the restored residual, and stores the same in a frame memory (3350). The stored reconstructed point cloud geometry is used for data prediction within the PU range of the next frame. When the reconstruction of the geometry is completed, the point cloud data processing device may inversely transform attributes (e.g., colors) to include the attributes in the position of the point indicated by the reconstructed geometry (3360). Reconstructed point cloud contents based on the reconstructed geometry and attributes are delivered to the renderer. In the processing method illustrated in the figure, the respective operations may be performed sequentially, or at least one or more thereof may be performed simultaneously. The processing order of the operations may be changed.

The point cloud data processing device described with reference to FIGS. 1 to 33 may generate a list of PUs having the same or similar MV in performing inter prediction based on PUs in order to increase the efficiency of compression of point cloud data having one or more frames, and thus perform MV signaling of a plurality of PUs using information about a single MV. As described above, PUs may be generated based on the octree structure. However, when the partitioned octree node is encoded and decoded without considering the distribution and characteristics of data constituting the point cloud content, the PUs may be split without considering the object of the point cloud content, and MV information should be signaled for each PU. Thus, even when the same MV is predicted, MV information should be signaled for each PU, and accordingly unnecessary signaling information is used. In addition, when a PU is split in a split structure (e.g., binary tree, quadtree, etc.) other than the octree structure, compression efficiency and quick data processing may be deteriorated because there is no signaling information for merging between PUs having the same MV.

In this regard, when there are one or more adjacent PUs having the same MV regardless of the PU splitting method, the point cloud data processing device according to the embodiments may merge (or group) the PUs and signal only one MV representing the merged PUs, thereby increasing coding efficiency. According to embodiments, the one MV representing the merged PUs may be referred to as a merged motion vector (MMV). Also, the point cloud data processing device may define one MV in an object unit between frames by merging PUs (standardized PUs or non-standardized PUs) constituting the object. The indexes of the PUs to be merged according to the embodiments are registered in a merge PU list or merge list. The merge list is represented as MMV_list. According to embodiments, the number of merged PU lists is greater than or equal to 1, and each of the PU lists corresponds to a set of PUs constituting an object. Also, each merge list may be referenced in one or more frames. Information on the merge list and the MMV according to the embodiments are transmitted to the point cloud decoder through signaling information related to PU merging signaled through the above-described bitstream. The point cloud decoder (or the geometry decoder described with reference to FIG. 10) according to the embodiments may secure the signaling information related to PU merging from the bitstream and perform inter prediction using the MMV.

FIG. 34 illustrates an example of a PU merge part according to embodiments.

The PU merge part 3400 shown in FIG. 34 is included in the point cloud transmission device or the point cloud encoder described with reference to FIGS. 1 to 14, and may generate and merge PUs. Also, the PU merge part 3400 shown in FIG. 34 may be included in the geometry encoder. The geometry encoder is included in the point cloud encoder.

The geometry encoder according to the embodiments may include at least one of the coordinate transformer 40000, the quantizer 40001, the octree analyzer 40002, the surface approximation analyzer 40003, the arithmetic encoder 40004, and the geometry reconstructor 40005 described with reference to FIG. 4. At least any one or more of the parts 40005 may be included. The geometry encoder according to the exemplary embodiments may include at least one of the data input unit 12000, the quantization processor 12001, the voxelization processor 12002, the octree occupancy code generator 12003, and the surface model processor 12004, the intra/inter-coding processor 12005, and the arithmetic coder 12006 described with reference to FIG. 12. Although not shown in the figures, the geometry encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The PU merge part 3400 includes a PU splitter (PU split) 3410, a motion vector estimation processor 3420, a motion vector comparator (motion vector comparison) 3430, and a merge determiner 3440. In FIG. 34 illustrating an example of the PU merge part 3400, the order of arrangement of the components may be changed, and the PU merge part 3400 may correspond to hardware, software, a process, or a combination thereof.

The PU splitter 3410 may split the point cloud data according to a preset PU shape. Also, the PU splitter 3410 may carry out the PU splitting method described with reference to FIGS. 15 to 33.

The motion vector estimation processor 3420 (e.g., the motion estimation processor 1730 described with reference to FIG. 17) performs motion estimation (or motion vector estimation) according to a preconfigured method (e.g., a preconfigured method in the point cloud encoder and the point cloud decoder) for inter prediction. Although not shown in the figure, an MV extracted according to the motion vector estimation and the cost value for determining whether to perform and/or skip additional splitting are delivered to the sub PU split determiner (e.g., the sub PU split determiner 1740 described with reference to FIG. 17). The sub PU split determiner checks the cost required when the MV extracted according to the motion vector estimation is applied, and determines whether to perform or skip additional splitting into sub PUs according to the cost. When the additional splitting is performed, the corresponding information is delivered to the PU splitter 3410, and the PU splitter 3410 recursively performs an operation for splitting again.

The motion vector comparator 3430 calculates a difference in MV between neighbor PUs to be compared with the MV determined for each PU and each sub PU to calculate a motion vector difference (MVD).

The merge determiner 3440 may select (determine) PUs to be merged based on the calculated MVD and a preconfigured criterion for determination.

FIG. 35 illustrates an example of neighbor PUs.

FIG. 35 illustrates an example of neighbor PUs for the merge described with reference to FIG. 34. The example 3500 on the left side of the figure shows 10 neighbor PUs for the current PU (CPU) 3510 (color-processed blocks around the current PU 3510), and the example 3520 on the right side of the figure shows 26 neighbor PUs for the CPU. That is, a neighbor PU according to the embodiments may be defined as a PU corresponding to a node having the same octree structure depth as the CPU or a parent node. Alternatively, when a PU has a cuboid shape, neighbor PUs may be defined as PUs that share at least one vertex or edge with the current PU. The calculated distance between the best MVs of the neighbor PU is less than a threshold, or the RDO calculated for the best MVs of the neighbor PUs is less than the threshold, or the best MVs of the neighbor PUs are exactly the same, the neighbor PUs are the target to be merged. The neighbor PUs according to the embodiments may be referred to as candidate PUs for merging.

FIG. 36 illustrates an example of neighbor PUs.

The example 3600 of FIG. 36 shows neighbor PUs to be merged among the neighbor PUs around the current PU 3510 described with reference to FIG. 35 (that is, the colored blocks around the current PU 3510 described with reference to FIG. 35, i.e., the block at the center in the figure). As described with reference to FIG. 35, even when a PU corresponding to a specific node meets the criterion for merging and becomes a merge target in the parent node or is a neighbor node to be merged as shown in the figure, it is excluded from the merging target if there are only PUs that are not adjacent to the current PU when the depth is the depth of the child node or is the same as the depth of a node corresponding to the current PU. In addition, the point cloud data processing device according to the embodiments may utilize sibling occupancy or PU index information to determine only neighbor PUs sharing a face with the current PU based on PU index information corresponding to the vicinity of the current PU as targets to be merged.

FIG. 37 illustrates an example of allocation of PU indexes.

According to the embodiments, PU indexes may be sequentially allocated from CC0 to CC7 in this order or may be allocated according to a preconfigured system setting, a Morton code sorting order, other sort order, etc., and are not limited to this example.

FIG. 38 illustrates an example of a neighbor PU.

FIG. 38 shows an example 3800 of six neighbor PUs to be merged with the current PU 3810. When a PU sharing a face with the current PU 3810 and having an MV is selected as described with reference to FIGS. 35 to 37, the point cloud data processing device (e.g., the merge determiner 3440 described with reference to FIG. 34) selects a target to be merged by comparing the best MV determined for each PU with the MV of the current node. The point cloud data processing device may sequentially compare the MVs and update the comparison result to select a target to be merged, or may determine the best MVs for all PUs and compare the determined best MVs. The example in FIG. 38 shows a merge target determined after the best MVs for all PUs are selected. The PUs processed in the same color are neighbor PUs coded before the current PU 3810 or include a neighbor PU that shares at least one face with the current PU 3810.

FIG. 39 illustrates an MV comparison process according to embodiments.

The process of FIG. 39 is an MV comparison process for the example 3800 of neighbor PUs shown in FIG. 38.

The left part of FIG. 39 is an example 3900 illustrating a process of comparing the MV (CMV) of the current PU with the MVs of all neighbor PUs. The point cloud data processing device (or the point cloud encoder or geometry encoder) according to the embodiments calculates a motion vector difference (MVD) between the MV (CMV) of the current PU and the MVs (MVMs) of the PUs which are MV merge targets as described with reference to FIG. 34. For example, suppose that there are 6 neighbor PUs neighboring the current PU. When the order of the indexes of the neighbor PUs is the same as the node search order, the point cloud data processing device registers it as an element of the temporary merged PU index list or array in the corresponding order. The point cloud data processing device compares the best MV (MVMci) found in each PU with the best MV (CMV) of the current PU to calculate each MVD (e.g., MVD1, which is a difference between CMV and MVMc1). The point cloud data processing device selects six calculated MVDs according to a preconfigured criterion (e.g., the MVD is equal to 0 or is less than a threshold). When the MVD is 0 or less than the threshold, the point cloud data processing device selects the corresponding PU as a PU to be merged.

The right side of FIG. 39 is an example 3910 illustrating four neighbor PUs (MVMc1, MVMc2, MVMc3, and MVMc6) selected through the comparison process.

Colored circles correspond to the four selected neighbor PUs. The point cloud data processing device according to the embodiments may signal indexes of the selected neighbor PUs, and there may be one or more PU indexes to be merged. As shown in the figure, when there are one or more PU indexes to be merged, the point cloud data processing device may input the indexes of the PUs to be merged at the end of the merge list such that the indexes of the PUs to be merged may be recorded in the sorting order or neighbor search order.

FIG. 40 shows an example of PUs to be merged and an example of a PU list.

An example 4000 on the left side of FIG. 40 shows four neighbor PUs (MVMc1, MVMc2, MVMc3, MVMc6) that are selected through MVD calculation to be merged as described with reference to FIG. 39 and the current PU.

An example 4010 in the middle of FIG. 40 shows MVs of PUs to be merged and PUs that are not merged, and a list 4020 shown on the right side of FIG. 40 is an example of a merge list (MMV_list). The merge list 4020 according to the embodiments includes an index of the current PU (PU_index_current) and indexes of four PUs (PU_index_c1, PU_index_c2, PU_index_c3, PU_index_c6). As such, when the index of the current PU and the indexes of the four PUs are stored in a separate list or array, the point cloud data processing device may calculate the value of arithmetic mean or geometric mean of the MVs of the four PUs to be merged and may determine the calculated value as a merged motion vector (MMV), which is representative of the four PUs, or determine the MV of the current PU as an MMV. In this case, the point cloud data processing device only needs to signal the MV for the current PU. Therefore, as shown in the example 4010 in the figure, the point cloud data processing device may not signal each of 7 MVs for the current PU and 6 neighbor PUs, but may signal three MVs including the MV of the current PU and MVs (MVMv4, MVMc5) for the two neighbor PUs that are not merged. Accordingly, as the number of MVs to be signaled is reduced, the bitstream size and encoding time may be reduced, thereby increasing coding efficiency.

Thereafter, when a PU to be merged is additionally found through a neighbor PU search for the PUs that are already determined to be merged, the index of the PU is added to the end of the coded merge list (MMV_list), and the indexes of all PUs which are determined to be merged do not overlap in the merge list (MMV_list) or array. As described above, there may be one or more merge lists, and the number of merge lists is equal to the number of MMVs. According to embodiments, two or more merge lists may be integrated, or one merge list may be split. PUs belonging to the same merge list may be configured as a PU set for an object. Also, the merge list may be be shared among one or more frames.

The signaling information related to PU merging according to the embodiments may be included in signaling information of various levels (e.g., a sequence level, a slice level, etc.). The signaling information related to PU merging is transmitted to the point cloud reception device (e.g., the reception device 10004 of FIG. 1, the point cloud decoder of FIGS. 10 and 11, or the reception device of FIG. 13).

FIG. 41 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 41 show an example of a syntax structure of an SPS in which the signaling information related to PU merging is included at the sequence level.

MMV_flag: Indicates whether one MV or merged motion vector (MMV) is signaled for the same or similar MVs when the best MVs for respective PUs are searched for and the difference (MVD) between MVs is calculated.

When the value of MMV_flag is TRUE, it indicates that only one MMV is signaled for PUs having the same or similar MVs. When the value is FALSE, it indicates that PUs having the same or similar MVs are not integrated. num_of_MMV_list: Indicates the number of MMV_list (merge lists) in a slice, a frame, or entire point cloud content.

num_MMV_list_component: Indicates the number of PU components belonging to MMV_list. This value can change until a merge target is found for slice, frame, or entire point cloud content.

MMV: Indicates an MV representing each MMV_list. The value of MMV may be expressed as (x, y, z), (azimuth, radius, laserID) or coordinate values in a format required by the system.

MMV_threshold: Indicates a threshold compared with the calculated MVD to determine whether the PU is to be merged. When only a PU having the same MV is selected as an object to be merged, MMV_threshold may be 0. When a PU having an MVD smaller than some value is selected as an object to be merge, MMV_threshold may be a set value other than 0. When the MVD is less than MMV_threshold, the index of the PU may be added to the MMV_list. The value of MMV_threshold may be predetermined, or may be inferred or calculated in the system.

allow_use_next_frame: A flag for determining whether to allow another slice, tile, or frame to reference the MMV_list. When the value of allow_use_next_frame is TRUE, it indicates that another frame is allowed to reference MMV_list. When the value is FALSE, it indicates that the referencing is not allowed. When the referencing is allowed, the corresponding information is not deleted even after encoding and decoding of another frame is completed, but is delivered when another frame is encoded/decoded.

core_PU_index: Indicates a PU index representing MMV_list when the list is configured. When the PU index representing the MMV_list is positioned at the top in the MMV_list, signaling of the core_PU_index may be skipped.

PU_position: indicates a reference position for indicating the position of a PU in order to signal the position of each PU. An example 4100 on the right side of FIG. 41 shows PU_position given when coordinate information about a PU position is expressed as (x, y, z). PU_position may be a value closest to the origin among the PUs or may be a position indicating the center of the PUs. The coordinate information may vary depending on system requirements.

PU_size: indicates the size of the PU, which is information signaled together with the PU position. PU_size may be expressed as the width, height, and depth of the PU. The information may vary depending on the shape of the PU.

PU_index: An index for referring to (identifying) each PU. The index allocation method may be changed depending on the system-defined sorting method.

MMV_residual: Indicates the residual of each MV when MMV_threshold is set to a non-zero value. The format of the data may depend on the setting of the MMV data.

The SPS syntax according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the SPS.

FIG. 42 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 42 shows an example of a syntax structure of a tile parameter set in which signaling information related to PU merging is included at the tile level. The signaling information related to PU merging is the same as that described with reference to FIG. 41, and thus a description thereof will be skipped. The tile parameter set syntax according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some elements may be signaled through signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the tile parameter set.

FIG. 43 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 43 shows an example of a syntax structure of a GPS in which signaling information related to PU merging is included at the sequence level. The signaling information related to PU merging is the same as that described with reference to FIG. 41, and thus a description thereof will be skipped. The GPS syntax according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some elements may be signaled through other signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the GPS.

FIG. 44 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 44 shows an example of a syntax structure of an APS in which signaling information related to PU merging is included at the sequence level. The signaling information related to PU merging is the same as that described with reference to FIG. 41, and thus a description thereof will be skipped. The APS syntax according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some of the elements may be signaled through signaling information, for example, a geometry slice data unit other than the APS.

FIG. 45 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 45 shows an example of a syntax structure of a geometry slice header in which signaling information related to PU merging is included at the slice level. The signaling information related to PU merging is the same as that described with reference to FIG. 41, and thus a description thereof will be skipped. The geometry slice header syntax according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some elements may be signaled through signaling information other than the geometry slice header, for example, a geometry slice data unit.

FIG. 46 shows an example of a syntax structure of signaling information related to PU merging.

FIG. 46 shows an example of a syntax structure of geometry slice data in which signaling information related to PU merging is included at the slice level. The signaling information related to PU merging is the same as that described with reference to FIG. 41, and thus a description thereof will be skipped. The syntax of the geometry slice data according to the embodiments is not limited to the above example. It may further include additional elements or may not include some of the elements shown in the figure for efficiency of signaling. Some elements may be signaled through other signaling information (e.g., APS, attribute header, etc.) or an attribute data unit other than the geometry slice data.

FIG. 47 is a flow diagram illustrating a point cloud processing method according to embodiments.

The flow diagram 4700 of FIG. 47 illustrates a processing method for a point cloud data processing device (e.g., the point cloud data transmission device, the point cloud encoder, the geometry encoder, or the like described with reference to FIGS. 1 to 14) configured to perform the PU merging described with reference to FIGS. 34 to 46. The point cloud processing method illustrated in the figure may include the method of determining the PU merging method described with reference to FIGS. 34 to 46.

The point cloud data processing device (e.g., the geometry encoder) according to the embodiments receives point cloud data as an input and performs data quantization/voxelization to facilitate compression of the geometry of the input point cloud data (4710).

The geometry encoder according to the embodiments may include at least one of the coordinate transformer 40000, the quantizer 40001, the octree analyzer 40002, the surface approximation analyzer 40003, the arithmetic encoder 40004, and the geometry reconstructor 40005 described with reference to FIG. 4. The geometry encoder according to the exemplary embodiments may include at least one of the data input unit 12000, the quantization processor 12001, the voxelization processor 12002, the octree occupancy code generator 12003, and the surface model processor 12004, the intra/inter-coding processor 12005, and the arithmetic coder 12006 described with reference to FIG. 12. Although not shown in the figures, the geometry encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The point cloud data processing device (e.g., the geometry encoder) splits a PU by a preconfigured method (PU split) (4720). The point cloud data processing device may perform motion estimation on the split PU (4730). The point cloud data processing device may determine whether to perform sub PU splitting by comparing a value of a result of the motion estimation with a target value (4731). When the point cloud data processing device performs sub PU splitting, motion estimation is performed on the split sub PU. When motion estimation is completed, a cost is calculated, and it is determined whether to perform additional splitting or to perform motion compensation for the value, based on the cost. The sub PU splitting 4731 according to the embodiments may be performed or skipped.

The point cloud data processing device according to the embodiments performs PU merging based on a motion vector (MV) obtained by performing motion estimation for each PU and/or sub PU (4740). The PU merging 4740 is the same as the PU merging process described with reference to FIGS. 34 to 40. PU merging 4740 according to embodiments includes the following operations. The point cloud data processing device according to the embodiments performs screening from a PU having the least PU index to select a PU merge target, determines the PU having the least index as a current PU, and configures PUs neighboring the current PU as neighbor PUs to be merged, that is, candidate PUs (CPUs) (4741). The point cloud data processing device calculates a motion vector difference (MVD), which is a difference between the MV of the current PU and the MVs of the CPUs (4742). When the calculated MPD is less than or equal to a preset threshold, the point cloud data processing device selects the c PU as a PU to be merged and adds the index of the PU to MMV_list of the current PU (4744). Otherwise, the point cloud data processing device repeats the same operations for the next candidate PU (CPU) to check the MVD from all candidate PUs (CPUs) and complete the MMV_list. Thereafter, the point cloud data processing device performs the PU merging process 4740 again on a PU having the next index. When the PU having the next index is added to the MMV_list of the PU having the previous index, the MMV_list of the PU is added to the end of the MMV_list of the previous PU and managed. When the MMV_list is completed for all PU indexes, the corresponding information is delivered for motion compensation, and the point cloud data processing device performs the motion compensation (4750). The point cloud data processing device may perform attribute coding based on the reconstructed point cloud data (reconstructed geometry) (4760), and encode the residual between the source data and the predicted data through transform, quantization, and entropy encoding.

In the processing method illustrated in the figure, the respective operations may be performed sequentially, or at least one or more thereof may be performed simultaneously. The processing order of the operations may be changed. For example, the PU merging 4740 may be performed concurrently with the PU splitting 4720 and the motion estimation 4730.

FIG. 48 is a flow diagram illustrating a point cloud processing method according to embodiments.

The flow diagram 4800 of FIG. 48 illustrates a processing method for a point cloud data processing device (e.g., the point cloud data reception device, the point cloud decoder, the geometry decoder, or the like described with reference to FIGS. 1 to 14) configured to perform the PU merging described with reference to FIGS. 34 to 46. The point cloud processing method illustrated in the figure may include the method of determining the PU merging method described with reference to FIGS. 34 to 46.

The point cloud data processing device (e.g., the geometry decoder) according to the embodiments may perform entropy decoding on a geometry included in a received bitstream (4810), perform dequantization (4830), and perform inverse transformation (4840), thereby restoring a residual, which is a prediction error for each point. The geometry decoder according to the embodiments may include at least one of the arithmetic decoder 13002, the occupancy code-based octree reconstruction processor 13003, the surface model processor 13004, and the inverse quantization processor 13005 described with reference to FIG. 12. Although not shown in the figure, the geometry decoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The point cloud data processing device (e.g., the geometry decoder) may secure motion vector (MV) information signaled for each PU or merged PU, and predict a point value in a PU range of the current frame (4820). As described with reference to FIGS. 34 to 46, MV information about each PU may be shared in the MMV_list and merged into one. Accordingly, the point cloud data processing device splits the PU (4821), and determines whether an MMV (or MMV_list) is present for each PU index (4822). When an MMV is present, the point cloud data processing device allocates an MV representing the MMV_list to the PU (4823). When the MMV is not present, the point cloud data processing device may allocate the signaled MV to the PU (4824). The point cloud data processing device outputs reconstructed point cloud geometry (or reconstructed geometry) by securing MVs for all PUs and adding the predicted point values and the restored residual, and stores the same in frame memory (4850). The reconstructed point cloud geometry may be stored together with the MMV_list and may be used as data prediction information within the PU range of the next frame. When the reconstruction of the geometry is completed, the point cloud data processing device may inversely transform attributes (e.g., color) to include the attribute in the position of the point indicated by the reconstructed geometry (4860). Reconstructed point cloud contents based on the reconstructed geometry and attributes are delivered to the renderer. In the processing method illustrated in the figure, the respective operations may be performed sequentially, or at least one or more thereof may be performed simultaneously. The processing order of the operations may be changed.

The PU splitting described with reference to FIGS. 15 to 33 and the PU merging described with reference to FIGS. 34 to 48 may be performed together. The signaling information related to PU splitting and the signaling information related to PU merging according to the embodiments may be included in signaling information of various levels (e.g., a sequence level, a slice level, etc.). The signaling information related to PU splitting and PU merging is transmitted to the point cloud reception device (e.g., the reception device 10004 of FIG. 1, the point cloud decoder of FIGS. 10 and 11, or the reception device of FIG. 13).

FIG. 49 shows a syntax structure of signaling information related to PU splitting and PU merging.

FIG. 49 shows an example of a syntax structure of an SPS in which signaling information related to PU splitting and signaling information related to PU merging are included at the sequence level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and the signaling information related to PU merging is the same as that described with reference to FIG. 41. Thus, a description thereof will be skipped.

FIG. 50 shows a syntax structure of signaling information related to PU splitting and PU merging.

FIG. 50 shows an example of a syntax structure of a tile parameter set in which signaling information related to PU splitting and signaling information related to PU merging are included at the tile level. Although not shown in the figure, the syntax of the tile parameter set may include both the signaling information related to PU splitting and the signaling information related to PU merging shown in FIG. 49. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and the signaling information related to PU merging is the same as that described with reference to FIG. 41. Thus, a description thereof will be skipped.

FIG. 51 shows a syntax structure of signaling information related to PU splitting and PU merge merging.

FIG. 51 shows an example of a syntax structure of a GPS in which signaling information related to PU splitting and signaling information related to PU merging are included at the sequence level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and the signaling information related to PU merging is the same as that described with reference to FIG. 41. Thus, a description thereof will be skipped.

FIG. 52 shows a syntax structure of signaling information related to PU splitting and PU merging.

FIG. 52 shows an example of a syntax structure of an APS in which signaling information related to PU splitting and signaling information related to PU merging are included at the sequence level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and the signaling information related to PU merging is the same as that described with reference to FIG. 41. Thus, a description thereof will be skipped.

FIG. 53 shows a syntax structure of signaling information related to PU splitting and PU merging.

FIG. 53 shows an example of a syntax structure of a geometry slice header in which signaling information related to PU splitting and signaling information related to PU merging are included at the slice level. The signaling information related to PU splitting is the same as that described with reference to FIG. 26, and the signaling information related to PU merging is the same as that described with reference to FIG. 41. Thus, a description thereof will be skipped.

FIG. 54 is a flow diagram illustrating a point cloud data processing method according to embodiments.

The flow diagram 5400 of FIG. 54 illustrates a cloud data processing method for a point cloud data processing device (or point cloud transmission device (e.g., the transmission device or point cloud encoder described with reference to FIGS. 1, 12 and 14). The point cloud data processing device encodes point cloud data including geometry and attributes (5410). The geometry is information indicating positions of points in the point cloud data, and the attributes include at least one of a color and a reflectance of the points. The point cloud data processing device includes a geometry encoder configured to encode a geometry and an attribute encoder configured to encode an attribute based on the encoded geometry. As described with reference to FIGS. 1 to 53, the attribute encoding depends on the geometry encoding. The geometry encoder according to the embodiments may include at least one of the coordinate transformer 40000, the quantizer 40001, the octree analyzer 40002, the surface approximation analyzer 40003, the arithmetic encoder 40004, and the geometry reconstructor 40005 described with reference to FIG. 4. The geometry encoder according to the exemplary embodiments may include at least one of the data input unit 12000, the quantization processor 12001, the voxelization processor 12002, the octree occupancy code generator 12003, and the surface model processor 12004, the intra/inter-coding processor 12005, and the arithmetic coder 12006 described with reference to FIG. 12. Although not shown in the figures, the geometry encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The attribute encoder is included in the point cloud encoder. The attribute encoder according to the embodiments may include at least one of the color transformer 40006, the attribute transformer 40007, the RAHT transformer 40008, the LOD generator 40009, the lifting transformer 40010, the coefficient quantizer 40011, and/or the arithmetic encoder 40012 described with reference to FIG. 4. In addition, the attribute encoder according to the embodiments may include at least one of the color transform processor 12008, the attribute transform processor 12009, the prediction/lifting/RAHT transform processor 12010, and the arithmetic coder 12011 described with reference to FIG. 12. Although not shown in the figures, the attribute encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The geometry encoder and the attribute encoder may be implemented by separate communicatively connected hardware, a single piece of hardware, one or more processors configured to communicate with one or more memories, separate software, or a combination of at least one of the above components.

The geometry encoder according to the embodiments may voxelize a geometry, and split the voxelized geometry into one or more prediction units (PUs) according to the PU splitting described with reference to FIGS. 15 to 33 for inter prediction. Each PU is a unit of inter prediction. The geometry encoder may also determine a split mode for performing additional splitting on at least one split PU, and split the at least one split PU into one or more sub PUs according to the determined split mode. The sub PU is also a unit of inter prediction. The bitstream according to the embodiments may contain signaling information related to PU splitting (the signaling information related to PU splitting described with reference to FIGS. 26 to 31 and the signaling information related to PU splitting described with reference to FIGS. 49 to 53). For example, the bitstream contains information (e.g., pu_coding_flag in FIG. 26) indicating whether inter prediction is performed in a PU, information (e.g., frame_pu_split_flag described with reference to FIG. 26) indicating whether PU splitting into one or more PUs is applied at a specific level among one or more levels constituting the octree structure of geometry or a frame unit of the point cloud data, and information (e.g., initial_split_mode of FIG. 26) indicating a split mode. The PU splitting is the same as that described with reference to FIGS. 15 to 33, and thus a detailed description thereof will be skipped.

Also, the geometry encoder may perform PU merging described with reference to FIGS. 34 to 53. The geometry encoder searches for neighbor PUs for at least one split PU, secures an MV of the PU and MVs of the neighbor PUs, and obtains a difference between the MV of the PU and the MVs of the neighbor PUs (e.g., the above-described MVD) by comparison. When the difference is less than or equal to a preset value, the encoder may a list (e.g., MMV_list) including the indexes of the neighbor PUs, and generate an MV (e.g., the MMV described with reference to FIGS. 34 to 53) representing the generated list. As described above, the bitstream may contain signaling information related to PU merging (e.g., the signaling information related to PU merging described with reference to FIGS. 41 to 46 and the signaling information related to PU merging described with reference to FIGS. 49 to 53). The PU merging is the same as that described with reference to FIGS. 34 to 53, and thus a detailed description thereof will be skipped.

FIG. 55 is a flow diagram illustrating a point cloud data processing method according to embodiments.

The flow diagram 5500 of FIG. 55 illustrates a point cloud data processing method for the point cloud data reception device (e.g., the reception device 10004 or the point cloud video decoder 10006) described with reference to FIGS. 1 to 53.

The point cloud data reception device (e.g., the receiver of FIG. 1, the receiver of FIG. 13, etc.) receives a bitstream containing point cloud data (5510).

The point cloud data reception device (e.g., the decoder of FIG. 10) decodes the point cloud data based on the signaling information (5520). The point cloud data reception device (e.g., the geometry decoder of FIG. 10) decodes a geometry included in the point cloud data. The point cloud data includes the geometry and attributes. The geometry is information indicating positions of points in the point cloud data, and the attributes include at least one of a color and a reflectance of the points. The point cloud decoder may include a geometry decoder (e.g., the geometry decoder of FIG. 10) configured to decode the geometry and an attribute decoder (e.g., the attribute decoder of FIG. 10) configured to decode the attributes based on the decoded geometry. The geometry decoder according to the embodiments may include at least one of the arithmetic decoder 13002, the occupancy code-based octree reconstruction processor 13003, the surface model processor 13004, and the inverse quantization processor 13005 described with reference to FIG. 12. The geometry decoder may secure MV information signaled for the operations of the entropy decoding 3310, the dequantization 3330, and the inverse transform 3340 described with reference to FIG. 33 and each PU, and predict a point value in a PU range of the current frame (3320). In addition, the geometry decoder may secure MV information signaled for each PU or the merged PU described with reference to FIG. 48 and predict a point value in the PU range of the current frame (4820). Although not shown in the figure, the geometry decoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The attribute decoder may include at least one of the arithmetic decoder 13007, the inverse quantization processor 13008, the prediction/lifting/RAHT inverse transform processor 13009 and the color inverse transform processor 13010 described with reference to FIG. 12. Although not shown in the figure, the attribute decoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof.

The geometry decoder and the attribute decoder may be implemented by separate communicatively connected hardware, a single piece of hardware, one or more processors configured to communicate with one or more memories, separate software, or a combination of at least one of the above components.

The bitstream according to the embodiments may contain information related to PU splitting for splitting a geometry into one or more PUs for inter prediction (e.g., the signaling information related to PU splitting described with reference to FIGS. 26 to 31, the signaling information related to PU splitting described with reference to FIGS. 49 to 53). The information related to PU splitting includes information (e.g., pu_coding_flag described with reference to FIG. 26) indicating whether inter prediction is performed in the PU, information (e.g., frame_pu_split_flag described with reference to FIG. 26) indicating whether PU splitting is applied at a specific level among one or more levels constituting the octree structure of the geometry or a frame unit of the point cloud data, and information (e.g., initial_split_mode described with reference to FIG. 26) indicating a split mode of one or more PUs. A detailed description of the signaling information related to the PU splitting described with reference to FIGS. 26 to 31 and the signaling information related to the PU splitting described with reference to FIGS. 49 to 53 will be skipped. The geometry decoder according to the embodiments may split the geometry into one or more PUs. Also, the geometry decoder according to the embodiments may split at least one PU into one or more sub PUs. The PU and the sub PU according to the embodiments are units of inter prediction. The PU splitting according to the embodiments is the same as that described with reference to FIGS. 26 to 33, and thus a detailed description thereof will be skipped.

Also, the geometry decoder according to the embodiments may perform geometry decoding according to the PU merging described with reference to FIGS. 34 to 53. That is, the geometry decoder may split the geometry into one or more PUs. When a merged motion vector for the PUs is present, the decoder may allocate the merged motion vector to the split PU. The bitstream according to the embodiments may include signaling information related to PU merging (e.g., the signaling information related to PU merging described with reference to FIGS. 41 to 46 and the signaling information related to PU merging described with reference to FIGS. 49 to 53). The PU merging and the PU merging operation of the geometry decoder are the same as those described with reference to FIGS. 34 to 48, and thus a detailed description thereof will be skipped.

Point cloud data processing according to the embodiments described with reference to FIGS. 1 to 55. The components of the device may include one or more processors coupled with memory. It may be implemented by including hardware, software, firmware, or a combination thereof. The components of the device according to the embodiments are one chip, for example, one It can be implemented as a hardware circuit. In addition, point clouds according to embodiments Each of the components of the data processing apparatus may be implemented as separate chips. In addition The components of the point cloud data processing device according to the embodiments are at least one or more is one or more capable of executing one or more programs. It may consist of one or more processors, and one or more programs are Among the operations/methods of the point cloud data processing device described with reference to FIGS. To perform or to perform any one or more operations It may contain instructions.

Embodiments have been described from the method and/or device perspective, and descriptions of methods and devices may be applied so as to complement each other.

Although the accompanying drawings have been described separately for simplicity, it is possible to design new embodiments by merging the embodiments illustrated in the respective drawings. Designing a recording medium readable by a computer on which programs for executing the above-described embodiments are recorded as needed by those skilled in the art also falls within the scope of the appended claims and their equivalents. The devices and methods according to embodiments may not be limited by the configurations and methods of the embodiments described above. Various modifications can be made to the embodiments by selectively combining all or some of the embodiments. Although preferred embodiments have been described with reference to the drawings, those skilled in the art will appreciate that various modifications and variations may be made in the embodiments without departing from the spirit or scope of the disclosure described in the appended claims. Such modifications are not to be understood individually from the technical idea or perspective of the embodiments.

Various elements of the devices of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various elements in the embodiments may be implemented by a single chip, for example, a single hardware circuit. According to embodiments, the components according to the embodiments may be implemented as separate chips, respectively. According to embodiments, at least one or more of the components of the device according to the embodiments may include one or more processors capable of executing one or more programs. The one or more programs may perform any one or more of the operations/methods according to the embodiments or include instructions for performing the same. Executable instructions for performing the method/operations of the device according to the embodiments may be stored in a non-transitory CRM or other computer program products configured to be executed by one or more processors, or may be stored in a transitory CRM or other computer program products configured to be executed by one or more processors. In addition, the memory according to the embodiments may be used as a concept covering not only volatile memories (e.g., RAM) but also nonvolatile memories, flash memories, and PROMs. In addition, it may also be implemented in the form of a carrier wave, such as transmission over the Internet. In addition, the processor-readable recording medium may be distributed to computer systems connected over a network such that the processor-readable code may be stored and executed in a distributed fashion.

In this specification, the term “/” and “,” should be interpreted as indicating “and/or.” For instance, the expression “A/B” may mean “A and/or B.” Further, “A, B” may mean “A and/or B.” Further, “A/B/C” may mean “at least one of A, B, and/or C.” Also, “A/B/C” may mean “at least one of A, B, and/or C.” Further, in this specification, the term “or” should be interpreted as indicating “and/or.” For instance, the expression “A or B” may mean 1) only A, 2) only B, or 3) both A and B. In other words, the term “or” used in this document should be interpreted as indicating “additionally or alternatively.”

Terms such as first and second may be used to describe various elements of the embodiments. However, various components according to the embodiments should not be limited by the above terms. These terms are only used to distinguish one element from another. For example, a first user input signal may be referred to as a second user input signal. Similarly, the second user input signal may be referred to as a first user input signal. Use of these terms should be construed as not departing from the scope of the various embodiments. The first user input signal and the second user input signal are both user input signals, but do not mean the same user input signals unless context clearly dictates otherwise.

The terms used to describe the embodiments are used for the purpose of describing specific embodiments, and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular forms “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. The expression “and/or” is used to include all possible combinations of terms. The terms such as “includes” or “has” are intended to indicate existence of figures, numbers, steps, elements, and/or components and should be understood as not precluding possibility of existence of additional existence of figures, numbers, steps, elements, and/or components. As used herein, conditional expressions such as “if” and “when” are not limited to an optional case and are intended to be interpreted, when a specific condition is satisfied, to perform the related operation or interpret the related definition according to the specific condition.

MODE FOR DISCLOSURE

As described above, related contents have been described in the best mode for carrying out the embodiments.

INDUSTRIAL APPLICABILITY

It will be apparent to those skilled in the art that various changes or modifications can be made to the embodiments within the scope of the embodiments. Thus, it is intended that the embodiments cover the modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalents.