Sony Patent | Image Processing Apparatus And Method For Object Boundary Stabilization In An Image Of A Sequence Of Images
Publication Number: 10643336
Publication Date: 20200505
Applicants: Sony
Abstract
Various aspects of an image-processing apparatus and method for object boundary stabilization in an image of a sequence of image frames are disclosed. The image-processing apparatus includes an image processor that receives a depth image of a scene from a first-type of sensor and a color image of the scene from the second-type of sensor. The scene may comprise at least an object-of-interest. A first object mask of the object-of-interest is generated by a depth thresholding operation on the received depth image. Dangling-pixels artifact present on a first object boundary of the first object mask, are removed. The first object boundary is smoothened using a moving-template filter on the color image. A second object mask having a second object boundary is generated based on the smoothening of the first object boundary. The object-of-interest from the color image is extracted based on the generated second object mask.
CROSS-REFERENCE TO RELATED APPLICATIONS/INCORPORATION BY REFERENCE
None.
FIELD
Various embodiments of the disclosure relate to object segmentation and image background substitution technologies. More specifically, various embodiments of the disclosure relate to an image-processing apparatus and method for object boundary stabilization in an image of a sequence of image frames.
BACKGROUND
Recent advancements in the field of video surveillance systems, machine vision systems in the field of robotics and automotive industry, and consumer electronic (CE) devices are largely due to rapid technological development in image processing techniques. Although various object segmentation methods have been known to separate foreground objects from background of an image, the complexity, accuracy, and computational resource requirement varies based on an objective to be achieved. In depth-based object segmentation methods, the use of a depth map for an object segmentation may allow avoidance of many uncertainties in the object delineation process, as compared methods that use a color image alone. Existing depth sensors that provide depth map are still lacking in accuracy and lag to match up with the increasing resolution of RGB cameras. For example, the depth map may contain shadowy areas, where the light from infrared (IR) emitters of depth sensors do not propagate, resulting in areas with unknown depth. In addition, the depth map may be most uncertain at the boundary of an object, where the depth drops sharply, and strongly fluctuates between image frames. The imperfectness in the depth map of modern depth sensors results in significant fluctuations on the boundary of a segmented object, especially visible between frames of a sequence of image frames, for example, a movie or other videos. The resulting artifacts are visually unpleasant to a viewer. Therefore, it may be desirable to reduce the amount of boundary fluctuation and stabilize the object boundary for precise object segmentation and enhanced background substitution.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of described systems with some aspects of the present disclosure, as set forth in the remainder of the present application and with reference to the drawings.
SUMMARY
An image-processing apparatus and method for object boundary stabilization in an image of a sequence of image frames is provided substantially as shown in, and/or described in connection with, at least one of the figures, as set forth more completely in the claims.
These and other features and advantages of the present disclosure may be appreciated from a review of the following detailed description of the present disclosure, along with the accompanying figures in which like reference numerals refer to like parts throughout.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1A is a block diagram that illustrates an exemplary network environment for object boundary stabilization in an image of a sequence of image frames, in accordance with an embodiment of the disclosure.
FIG. 1B illustrates an exemplary color image and a depth image of a scene to depict different artifacts in the depth image processed by an exemplary image-processing apparatus, in accordance with an embodiment of the disclosure.
FIG. 2 is a block diagram that illustrates an exemplary image-processing apparatus, in accordance with an embodiment of the disclosure.
FIGS. 3A to 3M, collectively, illustrate exemplary operations of the image-processing apparatus of FIG. 2 for object boundary stabilization in an image of a sequence of image frames, in accordance with an embodiment of the disclosure.
FIGS. 4A and 4B, collectively, depict a flow chart that illustrates an exemplary method for object boundary stabilization in an image of a sequence of image frames, in accordance with an embodiment of the disclosure.
DETAILED DESCRIPTION
The following described implementations may be found in the disclosed image-processing apparatus and method for object boundary stabilization in an image of a sequence of image frames. Exemplary aspects of the disclosure may include an image-processing apparatus and a method that comprise receipt of a depth image of a scene from a first-type of sensor and a color image of the scene from a second-type of sensor. The first-type of sensor may be different from the second-type of sensor. The scene may comprise at least an object-of-interest. A first object mask of the object-of-interest may be obtained by a depth thresholding operation on the received depth image. Dangling-pixels artifact present on a first object boundary of the first object mask, may be removed. The first object boundary of the first object mask may be smoothened using a moving-template filter on the color image after removal of the dangling-pixels artifact. A second object mask having a second object boundary may be generated based on the smoothening of the first object boundary. The object-of-interest from the color image may be extracted based on the generated second object mask having the second object boundary.
In accordance with an embodiment, the processing of the color image of the scene may be restricted to a field-of-view (FOV) of the first-type of sensor for the extraction of the object-of-interest from the color image. A plurality of depth values greater than a threshold depth value may be excluded by the depth thresholding operation. The threshold depth value may correspond to a maximum depth value associated with pixels of the first object mask of the object-of-interest.
In accordance with an embodiment, zero-depth artifacts may be removed from the depth image. The zero-depth artifacts may correspond to areas with unknown depth values in the depth image. The pixels associated with the unknown depth values may be classified as background pixels or foreground pixels based on specified criteria. Further, an infrared (IR) shadow casted on the first object mask by a portion of the object-of-interest, may also be removed from the depth image. A background region outside the first object mask in the color image, may be dynamically updated for the removal of the IR shadow.
In accordance with an embodiment, the moving-template filter may be positioned on the color image to encompass a boundary pixel of the first object boundary such that the moving-template filter include a first set of pixels located in an interior region of the first object mask and a second set of pixels located in an exterior region outside the first object mask. Pixels with a maximum image gradient along a normal to the first object boundary within the moving-template filter, may be searched. The normal to the first object boundary may define a direction in which image gradients are computed. In accordance with an embodiment, a difference in a color value and a brightness value between the first set of pixels and the second set of pixels, may be computed. A boundary pixel may be identified as a candidate pixel for the smoothening of the first object boundary based on the computed difference in the color value and the brightness value between the first set of pixels and the second set of pixels.
In accordance with an embodiment, the extracted object-of-interest may be embedded into a new image that provides a new background for the object-of-interest. A blending operation may be applied to the second object boundary of the object-of-interest in the new image for a smooth color-brightness blend to the new background. At least one of a color value or a brightness value of a boundary pixel of the second object boundary may be updated. The update may occur based on a difference between the color value or the brightness value of the boundary pixel and a set of pixels adjacent to the boundary pixel. The set of pixels adjacent to the boundary pixel includes a first number of pixels within the second object mask and a second number of pixels in the new background of the new image.
FIG. 1A is a block diagram that illustrates an exemplary network environment for object boundary stabilization in an image of a sequence of image frames, in accordance with an embodiment of the disclosure. With reference to FIG. 1A, there is shown a network environment 100. The network environment 100 may include an image-processing apparatus 102, sensor circuitry 104, a server 106, and a communication network 108. In accordance with an embodiment, the sensor circuitry 104 may be integrated with the image-processing apparatus 102. In accordance with an embodiment, the sensor circuitry 104 may be an external sensor device communicatively coupled to the image-processing apparatus 102. The sensor circuitry 104 may include a first-type of sensor 104a and a second-type of sensor 104b. The image-processing apparatus 102 and the server 106 may be communicatively coupled to each other, via the communication network 108.
The image-processing apparatus 102 may comprise suitable circuitry, interfaces, and/or code that may be configured to receive a depth map of a scene from the first-type of sensor 104a and a color image of the same scene from the second-type of sensor 104b. The depth map and the color image may be received concurrently for processing. The scene, captured by the first-type of sensor 104a, such as a depth sensor, and the second-type of sensor 104b, may comprise one or more objects. Examples of the one or more objects, may include, but are not limited to a human object, an animal, a moving object, a deforming object, or a non-human or inanimate object, such as a robot, or an articulated object. The articulated object refers to an object that have parts which are attached via joints, and can move with respect to one another. The image-processing apparatus 102 may be configured to utilize both the depth map and the color image to accurately identify and refine a boundary of an object-of-interest. Typical artifacts in the depth map, which are characteristic of the depth sensors, such as the first-type of sensor 104a, may be removed by sequential refinement operations by the image-processing apparatus 102. The image-processing apparatus 102 may be configured to execute the sequential refinement operations to reduce an amount of the object boundary fluctuation for the object-of-interest. The image-processing apparatus 102 may be configured to extract the object-of-interest from the color image based on a refined object mask with a refined object boundary. The extracted object-of-interest may be embedded into a new image that provides a new background for the object-of-interest. Examples of the image-processing apparatus 102 may include, but are not limited to, a digital camera, a camcorder, a head-mounted device (HMD), a surveillance equipment, a smartphone, a smart-glass, a virtual reality-, mixed reality-, or an augmented reality-based device, a computing device, and/or other consumer electronic (CE) devices.
The sensor circuitry 104 may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to concurrently capture the depth map and the color image of a same scene. The sensor circuitry 104 may include the first-type of sensor 104a and the second-type of sensor 104b. The first-type of sensor 104a may include a depth sensor and an infrared (IR) emitter. The depth sensor may be an IR depth sensor. The second-type of sensor 104b may be an image sensor, for example, a RGB camera, which may capture the color image, such as an RGB image. The sensor circuitry 104 may be configured to store the depth map and the color image in a local buffer, a memory, and/or the server 106.
The server 106 may comprise suitable circuitry, interfaces, and/or code that may be configured to store a sequence of image frames and depth maps captured by the image-processing apparatus 102. Examples of the server 106 may include, but are not limited to, a database server, a file server, an application server, a cloud server, a web server, or a combination thereof.
The communication network 108 may include a communication medium through which the image-processing apparatus 102 may be communicatively coupled with the server 106. Examples of the communication network 108 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a Local Area Network (LAN), and/or a Metropolitan Area Network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 108, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, or Bluetooth (BT) communication protocols, including variants and/or a combinations thereof.
In operation, the image-processing apparatus 102 may be configured to receive a depth image of a scene from the first-type of sensor 104a and a color image of the scene from the second-type of sensor 104b. The scene may comprise one or more foreground objects, for example, an object-of-interest that is to be segmented. The image-processing apparatus 102 may be configured to restrict processing of the color image of the scene to a field-of-view (FOV) of the first-type of sensor 104a, as shown, for example, in FIG. 1B.
FIG. 1B illustrates an exemplary color image and a depth image of a scene to depict different artifacts in the depth image processed by an exemplary image-processing apparatus, in accordance with an embodiment of the disclosure. With reference to FIG. 1B, there is shown an exemplary color image, such as an RGB image 110a, and an exemplary depth map, such as a depth image 112a, of a scene 114. The RGB image 110a includes a first foreground object 118a, a second foreground object 120a, and a background 116a represented in a RGB color channel. The depth image 112a includes a first depth representation 118b of the first foreground object 118a, a second depth representation 120b of the second foreground object 120a, and a third depth representation 116b of the background 116a. There is also shown certain shadowy areas in the depth image 112a, such as regions 122a, 122b, 122c, 122d, and 124.
The image-processing apparatus 102 may be configured to obtain a first object mask of an object-of-interest, such as the first foreground object 118a, by a depth thresholding operation on the received depth image 112a. The image-processing apparatus 102 may be configured to exclude a plurality of depth values greater than a threshold depth value by the depth thresholding operation. For example, all pixels located less than a certain meter in depth (such as 1.5 depth value) from the sensor circuitry 104 may be considered as belonging to the foreground object(s) and accordingly object mask(s) may be generated. The threshold depth value corresponds to a maximum depth value associated with pixels of the first object mask of the object-of-interest, such as the first foreground object 118a.
In certain scenarios, the depth image 112a may include shadowy areas, for example, as shown in regions 122a,122b,122c, and 122d. The IR light emitted by the IR emitters of the first-type of sensor 104a may not propagate to certain areas of the scene 114 that is captured. Such areas where the light does not propagate effectively, usually appears as shadowy areas in the depth image 112a, and have unknown depth values. The unknown depth values may also be referred to as zero-depth or undefined depth values. For example, the region 122a refers to an area of the scene 114 that is outside the FOV of the first-type of sensor 104a. The region 122a may contain zero-depth values as reported by the first-type of sensor 104a. Thus, to resolve the 0-depth artifact in the region 122a, the image-processing apparatus 102 may be configured to restrict processing of the RGB image 110a of the scene 114 to the FOV of the first-type of sensor 104a, as shown by parallel dotted lines, for example, in the FIG. 1B.
The region 122b may refer to an area in the third depth representation 116b (i.e. which corresponds to background 116a) of the depth image 112a, which may also contain zero-depth values. The region 122b may have boundaries with non-zero depth regions, where a difference between the non-zero depth regions nearby the region 122b may be greater than a threshold depth value. Alternatively stated, the region 122b may indicate a large drop in the depth of the scene 114 as compared to nearby non-zero depth regions that share boundary with the region 122b. The region 122c may refer to a shadowy area in the third depth representation 116b of the depth image 112a, which may also contain zero-depth values. The zero-depth values in the region 122c may be as a result of an IR shadow in the region 122c casted by a foreground object, such as the first foreground object 118a, on the background 116a.
In certain scenarios, a portion of a foreground object, such as the first foreground object 118a, may cast a shadow on itself, as shown by the region 122d. Thus, the region 122d may also contain zero-depth values as IR light emitted by the IR emitters of the first-type of sensor 104a may not propagate to the region 122d. The image-processing apparatus 102 may be configured to remove the zero-depth artifacts from the depth image 112a. The zero-depth artifacts correspond to the areas with unknown depth values, for example, the regions 122a,122b,122c, and 122d, in the depth image 112a. The image-processing apparatus 102 may be configured to classify pixels associated with the unknown depth values as background pixels or foreground pixels based on specified criteria. The classification of pixels may be done to obtain a correct object mask, such as the first object mask, of the object-of-interest. The classification of pixels and the specified criteria are described in detail, for example, in FIG. 3A to 3I.
In accordance with an embodiment, the image-processing apparatus 102 may be configured to remove dangling-pixels artifact present on a first object boundary of the first object mask. The “dangling” or dangling-pixels artifact may be manifested by significant fluctuations at the first object boundary adjacent to the IR shadow areas in the depth image 112a. In those IR shadow areas, such as the region 124, at the first object boundary, the object boundary fluctuation may occur from frame-to-frame and from pixel-to-pixel manner. The region 124 indicates a chaotic depth in the depth image 112a (as reported by the first-type of sensor 104a), which results in the dangling-pixels artifact at and around the first object boundary of the object-of-interest, such as the first foreground object 118a. An example of the dangling-pixels artifact is further shown and described in FIGS. 3A and 3C. The image-processing apparatus 102 may be configured to mark a pixel as a dangling-pixel when the pixel in a 3.times.3 pixels vicinity has at least one depth-undefined pixel (for example, a pixel that contain a zero-depth value).
In accordance with an embodiment, the image-processing apparatus 102 may be configured to smoothen the first object boundary of the first object mask using a moving-template filter on the RGB image 110a after removal of the zero-depth artifacts and the dangling-pixels artifact. The smoothening operations are described in detail in the FIGS. 3A, 3E, 3F, 3G, 3H, and 3I. The image-processing apparatus 102 may be configured to generate a second object mask having a second object boundary based on the smoothening of the first object boundary. The image-processing apparatus 102 may be configured to extract the object-of-interest from the RGB image 110a based on the generated second object mask having the second object boundary. The image-processing apparatus 102 may be configured to embed the extracted object-of-interest into a new image that provides a new background for the object-of-interest. The image-processing apparatus 102 may be configured to apply a blending operation to the second object boundary of the object-of-interest in the new image for a smooth color-brightness blend to the new background. The blending operation is described in detail, for example, in FIGS. 3A and 3M.
In accordance with an embodiment, the object-of-interest may be extracted from each source color image, such as the RGB image 110a, of a sequence of image frames, and blended in each new image frame of a video frame-by-frame in real time or near real time. The image-processing apparatus 102 may be further configured to communicate the video that includes the embedded object-of-interest and the substituted background in the new image frame and the subsequent image frames to the server 106, via communication network 108. The server 106 may be configured to store the modified video.
The disclosed image-processing apparatus 102 for object boundary stabilization in an image of a sequence of image frames, such as a movie or other video, may be implemented in various application areas, such as video surveillance, automatic video editing systems, automatic background substitution systems, or tracking of objects that change position or orientations at different time instances while an input sequence of image frames is captured. The disclosed image-processing apparatus 102 and method may be suited for a real-world tracking application, such as video surveillance of human beings or other articulated objects, object tracking in a gaming system, or other real time or near-real time object segmentation and blending of objects in a new background.
FIG. 2 is a block diagram that illustrates an exemplary image-processing apparatus, in accordance with an embodiment of the disclosure. FIG. 2 is explained in conjunction with elements from FIGS. 1A and 1B. With reference to FIG. 2, there is shown the image-processing apparatus 102. The image-processing apparatus 102 may include one or more circuits, such as an image processor 202, a memory 204, and an object blending processor 206, an I/O device 208, and a network interface 210. The I/O device 208 may include the sensor circuitry 104 and a display 208A. The sensor circuitry 104 is shown as an integrated unit of the image-processing apparatus 102, in an example. The image processor 202 may be communicatively coupled with the memory 204, the object blending processor 206, the I/O device 208, the network interface 210, and the sensor circuitry 104. The network interface 210 may facilitate communication with the server 106, via communication network 108.
The image processor 202 may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to smoothen the first object boundary of the first object mask using a moving-template filter on the received color image after removal of different types of artifacts, such as zero-depth and dangling-pixels artifacts. The image processor 202 may be configured to generate a second object mask having a second object boundary based on the smoothening of the first object boundary. Thereafter, the object-of-interest may be extracted from the color image based on the generated second object mask having the second object boundary, which is the refined object boundary. The image processor 202 may be configured to execute a set of instructions stored in the memory 204. The image processor 202 may be implemented based on a number of processor technologies known in the art. Examples of the image processor 202 may be a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC), a hardware processor, a central processing unit (CPU), and/or other processors or control circuits.
