Sony Patent | Image processing apparatus, image processing method, and image processing program

Patent: Image processing apparatus, image processing method, and image processing program

Drawings: Click to check drawins

Publication Number: 20210192680

Publication Date: 20210624

Applicant: Sony

Abstract

An image processing apparatus (100) according to the present disclosure includes: a learning-data creation unit (132) configured to perform projection transformation on image data including a target as a subject, the learning-data creation unit (132) being configured to create learning data including the target as correct data; and a model generation unit (133) configured to generate, based on the learning data created by the learning-data creation unit (132), a learned model for detecting the target included in input data that includes a wide angle view image and is input to the learned model, the wide angle view image being created by projection transformation identical in scheme to the projection transformation by which the learning data is created.

Claims

1. An image processing apparatus comprising: a learning-data creation unit configured to perform projection transformation on image data including a target as a subject, the learning-data creation unit being configured to create learning data including the target as correct data; and a model generation unit configured to generate, based on the learning data created by the learning-data creation unit, a learned model for detecting the target included in input data that includes a wide angle view image and is input to the learned model, the wide angle view image being created by projection transformation identical in scheme to the projection transformation by which the learning data is created, wherein the learning-data creation unit changes an angle of the subject from an initial state of the subject, performs projection transformation on image data corresponding to the changed angle, and creates the learning data.

2. (canceled)

3. The image processing apparatus according to claim 1, wherein the learning-data creation unit changes respective angles in a pitch direction, a roll direction, and a yaw direction of the subject, performs projection transformation on image data corresponding to the changed angles, and creates the learning data.

4. The image processing apparatus according to claim 3, wherein the learning-data creation unit changes the angle in the yaw direction of the subject within a range in which the subject is partially in contact with an image frame of the image data, performs projection transformation on image data corresponding to the changed angle, and creates the learning data.

5. The image processing apparatus according to claim 1, further comprising: a detection unit configured to detect the target from the input data with the learned model generated by the model generation unit.

6. The image processing apparatus according to claim 5, wherein the model generation unit generates a plurality of learned models on a type basis of the projection transformation, the plurality of learned models being different from each other, and the detection unit selects a learned model corresponding to the type of the projection transformation performed on the input data, from among the plurality of learned models, and detects the target with the selected learned model.

7. The image processing apparatus according to claim 5, wherein the model generation unit generates, based on the learning data created by the learning-data creation unit, a replacement model for replacing entirely or partially the target detected by the detection unit.

8. The image processing apparatus according to claim 7, wherein the learning-data creation unit changes an angle of the subject from an initial state of the subject, performs projection transformation on image data corresponding to the changed angle, and creates the learning data, and the model generation unit generates, based on learning data created on an angle basis by the learning-data creation unit, a plurality of replacement models on an angle basis, the plurality of replacement models being different from each other.

9. The image processing apparatus according to claim 8, wherein the detection unit selects a replacement model corresponding to the target detected from the input data, from among the plurality of replacement models generated by the model generation unit, and replaces the target, with the selected replacement model.

10. The image processing apparatus according to claim 9, wherein the detection unit replaces a portion included in the target that is detected from the input data.

11. The image processing apparatus according to claim 1, wherein the model generation unit generates a learned model for detecting the target included in at least any of spherical content, semi-spherical content, or a panoramic image, as the wide angle view image.

12. The image processing apparatus according to claim 1, wherein the learning-data creation unit creates the learning data from image data including a subject of a human face as the target.

13. The image processing apparatus according to claim 12, wherein the learning-data creation unit creates the learning data from the image data in which position information of each part included in the human face is set.

14. The image processing apparatus according to claim 1, wherein the learning-data creation unit performs at least any of equirectangular projection, cube map, or fisheye projection, as the scheme of the projection transformation, and generates the learning data.

15. The image processing apparatus according to claim 10, wherein the model generation unit generates a learned model for detecting the target included in at least any of spherical content, semi-spherical content, or a panoramic image, as the wide angle view image, and the detection unit detects the target from the input data with the learned model for detecting the target included in the at least any of the spherical content, the semi-spherical content, or the panoramic image.

16. The image processing apparatus according to claim 14, wherein the learning-data creation unit creates the learning data from image data including a subject of a human face as the target.

17. The image processing apparatus according to claim 4, wherein the learning-data creation unit performs at least any of equirectangular projection, cube map, or fisheye projection, as the scheme of the projection transformation, and creates the learning data.

18. An image processing method, by a computer, comprising: performing projection transformation on image data including a target as a subject and creating learning data including the target as correct data; and generating, based on the created learning data, a learned model for detecting the target included in input data that includes a wide angle view image and is input to the learned model, the wide angle view image being created by projection transformation identical in scheme to the projection transformation by which the learning data is created, wherein the performing projection transformation and the creating learning data includes changing an angle of the subject from an initial state of the subject, performing projection transformation on image data corresponding to the changed angle, and creating the learning data.

19. An image processing program for causing a computer to function as: a learning-data creation unit configured to perform projection transformation on image data including a target as a subject, the learning-data creation unit being configured to create learning data including the target as correct data; and a model generation unit configured to generate, based on the learning data created by the learning-data creation unit, a learned model for detecting the target included in input data that includes a wide angle view image and is input to the learned model, the wide angle view image being created by projection transformation identical in scheme to the projection transformation by which the learning data is created, wherein the learning-data creation unit changes an angle of the subject from an initial state of the subject, performs projection transformation on image data corresponding to the changed angle, and creates the learning data.

Description

FIELD

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program. For details, the present disclosure relates to processing of detecting a subject in video data.

BACKGROUND

[0002] With the widespread use of image capturing equipment, images have been widely used, for example, many users capture still images and moving images and upload the captured images on the Web. Use of the images may need deleting an unspecified number of persons or a predetermined target inappropriate for the subject captured in the images.

[0003] As a technique relating to such images, there has been proposed a technique of detecting a target unintended for the image capturing person from an image with the target captured therein, and deleting or replacing the target with another object (e.g., Non Patent Literature 1). Note that images that are present are not limited to images captured by a normal camera or the like, and thus include images each having an angle of view wider than that displayed on a display (hereinafter, collectively referred to as “wide angle view image”), such as spherical content (spherical images or spherical moving images) and panoramic images to be used in, for example, virtual reality (VR) technology (e.g., Non Patent Literature 2).

CITATION LIST

Non Patent Literature

[0004] Non Patent Literature 1: “Image Inpainting for Irregular Holes Using Partial Convolutions”, Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro.

[0005] Non Patent Literature 2: “Information technology-Coded representation of immersive media-Part 2: Omnidirectional media format”, ISO/IEC FDIS 23090-2

SUMMARY

Technical Problem

[0006] However, with the above conventional technique, a target fails to be detected accurately in some cases. Typically, target detection processing to be performed includes generating a learned model obtained by learning the feature amount of a target to be detected (correct data), inputting data of an image or the like into the generated learned model, and determining whether or not the target is included in the image.

[0007] However, a wide angle view image such as spherical content is recorded in a format projection-transformed with an equirectangular projection scheme or the like, and thus a target is stretched or a target is in contact with each edge portion of the image. In such a case, the feature amount of an object to be detected (e.g., shape) is significantly different from the correct data, and thus the target may not be detected with a learned model corresponding to two-dimensional images obtained by typical perspective projection.

[0008] Therefore, the present disclosure proposes an image processing apparatus, an image processing method, and an image processing program that enables accurate detection of a target.

Solution to Problem

[0009] To solve the problem described above, an image processing apparatus includes: a learning-data creation unit configured to perform projection transformation on image data including a target as a subject, the learning-data creation unit being configured to create learning data including the target as correct data; and a model generation unit configured to generate, based on the learning data created by the learning-data creation unit, a learned model for detecting the target included in input data that includes a wide angle view image and is input to the learned model, the wide angle view image being created by projection transformation identical in scheme to the projection transformation by which the learning data is created.

BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 illustrates the overview of image processing according to a first embodiment of the present disclosure.

[0011] FIG. 2 is a diagram illustrating an example of an image processing system according to the first embodiment of the present disclosure.

[0012] FIG. 3 illustrates an example of an image-data storage unit according to the first embodiment of the present disclosure.

[0013] FIG. 4 illustrates an example of a learning-data storage unit according to the first embodiment of the present disclosure.

[0014] FIG. 5 illustrates exemplary content projection-transformed with an equirectangular projection scheme.

[0015] FIG. 6 illustrates a relationship between a wide angle view image and the eye point of a user.

[0016] FIG. 7 is an explanatory illustration (1) of creation processing according to the first embodiment of the present disclosure.

[0017] FIG. 8 is an explanatory illustration (2) of the creating processing according to the first embodiment of the present disclosure.

[0018] FIG. 9 explanatorily illustrates generation processing according to the first embodiment of the present disclosure.

[0019] FIG. 10 explanatorily illustrates a flow of the creation processing and the generation processing according to the present disclosure.

[0020] FIG. 11 is a flowchart (1) illustrating a processing procedure according to the first embodiment of the present disclosure.

[0021] FIG. 12 is a flowchart (2) illustrating the processing procedure according to the first embodiment of the present disclosure.

[0022] FIG. 13 is a flowchart (3) illustrating the processing procedure according to the first embodiment of the present disclosure.

[0023] FIG. 14 illustrates the overview of image processing according to a second embodiment of the present disclosure.

[0024] FIG. 15 is a hardware configuration diagram of an example of a computer that achieves the functions of an image processing apparatus.

DESCRIPTION OF EMBODIMENTS

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that in each of the following embodiments, the same reference signs are given to the same parts, and duplicate description thereof will be omitted.

1. First Embodiment

1-1. Use of Wide Angle View Image

[0026] Prior to the description of image processing according to the present disclosure, there will be described problems in using a wide angle view image (e.g., reproduction of a wide angle view image or distribution thereof to a user), as a premise of the image processing of the present disclosure.

[0027] Note that the wide angle view image according to the present disclosure is an image having an angle of view wider than that displayed on a display, such as spherical content and a panoramic image. In the present disclosure, spherical content will be described as an example of the wide angle view image.

[0028] The spherical content is generated by being shot with a spherical camera capable of shooting in 360.degree. in all directions. The spherical content has an angle of view wider than that of a typical display (e.g., head mounted display (HMD) mounted on the user), and thus only a partial area trimmed in accordance with the size of the display (in other words, a field angle of view of the user). For example, the user views the spherical content while changing the display position by operating a touch display or while changing the display location by applying change of the line of sight or posture of the user via the mounted HMD.

[0029] Here, the spherical content is generated by being shot in 360.degree. in all directions, and thus a situation occurs easily that an unintended subject is reflected in the spherical content, particularly when shooting outdoors. For example, in a case where the subject includes an unspecified number of persons, an issue such as portrait rights may occur. Furthermore, for example, even in a case where the subject includes a company’s logo mark, trademark, or the like, the use of the spherical content with the subject captured therein may be restricted.

[0030] In order to avoid the above issues, the content creator artificially specifies a specific target (e.g., an unspecified number of persons or logo mark) captured in the image data, and performs effect processing such as mosaicing on the specified target. The performance of the effect processing such as the mosaicing, however, may impair the immersive feeling, which is one of the features of the spherical content.

[0031] In response to the above problems, on typical still images and moving images, in some cases, there is performed processing including automatically detecting a target by machine learning the feature amount of a specific target, deleting (masking) the detected target, and then replacing the deleted target with a substitute that can be shown naturally.

[0032] However, a wide angle view image such as spherical content is recorded in a format projected, for example, with an equirectangular projection scheme, and thus a target is stretched near the poles or a target is located at each edge portion of the image. Thus, in the case of executing the above approach, it needs repeating that the content creator designates a partial area at a predetermined angle of view, performs detection and replacement processing after transforming the designated area into perspective projection, and performs similar processing on another area. Thus, such an approach has an issue that the processing load is increased and image quality is deteriorated due to repeating the projection transformation.

[0033] Therefore, in order to solve the above problems, the image processing according to the present disclosure includes performing projection transformation corresponding to input data to be input into a learned model (hereinafter, simply referred to as “model”), on image data including a predetermined target as a subject, and creating learning data to be used for learning the model. Furthermore, the image processing according to the present disclosure includes generating a model for detecting the predetermined target included in the input data that is spherical content, on the basis of the created learning data. With the image processing according to the present disclosure, the detection is performed with the model learned by the learning data corresponding to a projection scheme for the spherical content (e.g., the equirectangular projection scheme), so that the target can be detected without performing perspective projection transformation on the spherical content. Hereinafter, the overview of the image processing according to the present disclosure will be described with FIG. 1. Note that in the present disclosure, there will be described an example in which one example of the target is “the faces of an unspecified number of persons”.

1-2. Overview of Image Processing According to First Embodiment

[0034] An image processing apparatus 100 illustrated in FIG. 1 is an information processing apparatus that executes the image processing according to the present disclosure. The image processing apparatus 100 is, for example, an image server that provides a user with spherical content.

[0035] In the example illustrated in FIG. 1, the image processing apparatus 100 holds a detection model 150 and a replacement model 160 generated on the basis of learning data 145 stored in a learning-data storage unit 122. As will be described below in detail, the learning data 145 is an image set obtained by performing projection corresponding to a projection scheme (format) for spherical content, onto each face image set for learning (images obtained with a typical perspective projection scheme). For example, for one face image for learning, the image processing apparatus 100 generates images with the equirectangular projection scheme, the images having respective angles different in a pitch direction, a roll direction, and a yaw direction in the equirectangular projection scheme. Hereinafter, each of the generated face images may be referred to as a “distorted face image” because the face is distorted by the equirectangular projection.

[0036] That is, the image processing apparatus 100 performs learning with the learning data 145 that is a set of the distorted face images, instead of performing learning with a face image set for learning with a typical perspective projection scheme. Then, the image processing apparatus 100 generates the detection model 150 for determining whether or not the input data includes a feature amount indicating a “human face”. Furthermore, the image processing apparatus 100 generates a replacement model 160 for replacing a detected face image.

[0037] In the example of FIG. 1, the image processing apparatus 100 accepts spherical content 50 stored with the equirectangular projection scheme (step S1). The spherical content 50 includes faces 60 of an unspecified number of persons.

[0038] The image processing apparatus 100 inputs the accepted spherical content 50 into the detection model 150 (step S2). As a result, the image processing apparatus 100 detects each face 60 included in the spherical content 50. Furthermore, the image processing apparatus 100 inputs the spherical content 50 into the replacement model and replaces the face 60 with face data having a feature amount similar to that of the face 60. Note that the image processing apparatus 100 may replace only a part of the face 60 (e.g., eyes), instead of replacing the whole face 60. As an example, the image processing apparatus 100 replaces a part of the face 60 on the basis of the mean value of the feature amounts of parts in a plurality of images within the learning data 145 having angle information similar to that of the face 60.

[0039] Then, the image processing apparatus 100 generates spherical content 51 with the face 60 replaced with a face 61. Then, the image processing apparatus 100 outputs the spherical content 51 (step S3). For example, as spherical content distributed to the user, the image processing apparatus 100 distributes the spherical content 51 with the face 61 replaced, instead of the spherical content 50. As a result, the image processing apparatus 100 can prevent a situation in which the user views the face 60 mistakenly captured in the spherical content 50. Furthermore, the image processing apparatus 100 performs the detection and replacement processing on the spherical content 50 stored with the equirectangular projection scheme, without time and effort, for example, performing perspective projection transformation on the spherical content 50. As a result, the image processing apparatus 100 can prevent deteriorating the image quality of the spherical content 50 and can perform processing speedily.

1-3. Configuration of Image Processing System According to First Embodiment

[0040] Subsequently, the image processing apparatus 100 according to the present disclosure and an image processing system 1 according thereto will be described in detail with FIG. 2 and the subsequent figures.

[0041] FIG. 2 is a diagram illustrating an example of the image processing system 1 according to the first embodiment of the present disclosure. As illustrated in FIG. 2, the image processing system 1 includes an HMD 10, a data server 200, and the image processing apparatus 100.

[0042] The HMD 10 is a display apparatus to be mounted on the user’s head and is a so-called wearable computer. The HMD 10 achieves display processing in response to the orientation and movement of the user’s body, the moving speed of the user, and the like.

[0043] The data server 200 is information equipment connected to the image processing apparatus 100 and the HMD 10 by a wired or wireless network. The data server 200 is a storage server that accumulates image data for learning and provides the accumulated image data to the image processing apparatus 100. Furthermore, the data server 200 may be a service server that provides content such as moving images to a user who uses the HMD 10.

[0044] As described above, the image processing apparatus 100 is the information processing apparatus that creates learning data corresponding to the spherical content to be the target of the detection processing and generates a model on the basis the created learning data. The image processing apparatus 100, for example, inputs spherical content that is about to be distributed to the user into a detection model, detects the faces of an unspecified number of persons included in the spherical content, and replaces or deletes each detected face. Then, the image processing apparatus 100 distributes the spherical content having subjected to the replacement processing and the like to the user.

[0045] As illustrated in FIG. 2, the image processing apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the image processing apparatus 100 may include an input unit (e.g., a keyboard or a mouse) that accepts various operations from an administrator or the like who administers the image processing apparatus 100, and a display unit (e.g., a liquid crystal display) for displaying various types of information.

[0046] The communication unit 110 is achieved by, for example, a network interface card (NIC). The communication unit 110 is connected to a network N (e.g., the Internet) in a wired or wireless manner, and exchange information between the data server 200, the HMD 10, and the like via the network N.

[0047] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes an image-data storage unit 121, a learning-data storage unit 122, and a model storage unit 123. Hereinafter, each of the storage units will be described in the above order.

[0048] The image-data storage unit 121 stores image data as the original of learning data created by the image processing apparatus 100. The original of image data is, for example, face image data to be distributed as a sample of correct data for machine learning. Such face image data for machine learning is, for example, data in which a label serving as a correct answer of the image (e.g., information indicating that the subject of the image is a person’s face) is added to the image data. Note that the correct label may include attribute information of the person as the subject (e.g., the race, age, or gender of the person as the subject).

[0049] FIG. 3 illustrates an example of the image-data storage unit 121 according to the first embodiment. FIG. 3 is the example of the image-data storage unit 121 according to the first embodiment of the present disclosure. The example illustrated in FIG. 3, the image-data storage unit 121 has items such as “image ID”, “image data”, and “part information”.

[0050] The “image ID” indicates identification information for identifying an image. The “image data” indicates actual data of an image including the image. For example, the actual data includes the number of pixels including an image, color information of each of the pixels, and the like. Note that in FIG. 3, the item of image data is represented by a concept such as “B01”; however, actually, the item of image data stores detained data regarding the pixels.

[0051] The “part information” indicates information regarding each part included in a face image. For example, the part information includes position information indicating the position of the eyes, nose, or mouth of a subject (information indicating which position (pixels) in the image indicates the part), color information, and the like. Note that in FIG. 3, the respective pieces of data of items of the part information indicating the eyes, the nose, and the mouth are represented by concepts such as “C01”, “D01”, and “E01”; however, actually, each of the items stores specific information corresponding to the item. For example, the item of eyes stores position information of pixels corresponding to the eyes of a person (e.g., information of an area assumed to indicate the eyes in an image), color information of each of the pixels, and the like.

[0052] Furthermore, in the example of FIG. 3, the information regarding the eyes, the nose, and the mouth is indicated as the part information; however, the part information is not limited to the example. For example, the image-data storage unit 121 may store position information, color information, and the like of the eyebrows, the ears, and the like of a human face. In addition, information stored in face image data distributed in advance for learning may be used as part information, or part information may be information set artificially.

[0053] That is, as the example of the data stored in the image-data storage unit 121, FIG. 3 illustrates that an image identified by an image ID “A01” has image data “B01”, and “C01” for the eyes, “D01” for the nose, and “E01” for the mouth as the pieces of the part information. Note that although not illustrated in FIG. 3, the face image may include attribute information such as the race, age, gender, or skin color of the person as the subject.

[0054] Next, the learning-data storage unit 122 will be described. The learning-data storage unit 122 stores learning data created with an original image stored in the image-data storage unit 121 and corresponding to the projection scheme for a wide angle view image to be input into a model. In other words, the learning-data storage unit 122 stores, as learning data, an image data group obtained with the projection scheme identical to that for a wide angle view image to be input into a model.

[0055] FIG. 4 illustrates an example of the learning-data storage unit 122 according to the first embodiment of the present disclosure. In the example illustrated in FIG. 4, the learning-data storage unit 122 has items such as “image ID”, “learning data ID”, “image data”, “angle information”, and “post-transformation part information”.

[0056] The “image ID” indicates identification information for identifying an image and corresponds to the same item illustrated in FIG. 3. The “learning data ID” indicates identification information for identifying each learning data. The “image data” indicates the actual data of constituent images included in learning data. Such items one-to-one correspond to the same items illustrated in FIG. 3.

[0057] The “angle information” indicates information regarding angles set in each learning data. As illustrated in FIG. 1, learning data includes images each obtained by projection-transforming an original face image with the equirectangular projection scheme, and, for example, created by changing by 1.degree. for each of pitch, roll, and yaw elements. Therefore, each piece of learning data has angle information indicating how much the angles deviate from those of the original face image (e.g., a state having a pitch of 0.degree., a roll of 0.degree., and a yaw of 0.degree.). For example, learning data having a pitch of 45.degree. is an image in which a face is shifted by 45.degree. in the pitch direction (upward in the image) from the original face image. In addition, the learning data includes a face distorted by projection-transforming the original face shifted from the center of the image. For example, with the equirectangular projection scheme, as the original face closer to the upper or lower edge portion of the image (as the pitch angle approaches 90.degree. or -90.degree.), the transformed face is stretched laterally. As a result, the learning data includes a distorted face image.

[0058] The “pitch” “roll”, and “yaw” each indicate a direction in which the subject of an original image is shifted. The pitch indicates the longitudinal direction in a two-dimensional image. For example, regarding the pitch direction, with the center in the longitudinal direction of the image defined as 0.degree., the upper edge portion of the image indicates 90.degree. and the lower edge portion thereof indicates -90.degree.. In addition, the roll indicates rotation in a two-dimensional image. For example, regarding the roll direction, with the face image in an initial state defined as 0.degree. and 360.degree. defined as one rotation, the angle increases as the amount of shifting in clockwise rotation increases. In addition, the yaw indicates the lateral direction in a two-dimensional image. For example, regarding the yaw direction, with the center in the lateral direction of the image defined as 0.degree. and 180.degree. defined as one round, the angle increases as the amount of right shifting increases. For example, when the face located at the center reaches the opposite state (a state in which the face image is equally divided laterally), the yaw angle is 180.degree..

[0059] The “post-transformation part information” indicates, for example, position information of each part of the face after projection transformation. FIG. 4 illustrates the post-transformation part information as one item; however, the post-transformation information includes, for example, respective pieces of position information for the eyes, the nose, and the mouth. Note that if part information of the original face image is known, the post-transformation part information is obtained by calculating numerical values (coordinates) indicating, for example, to which positions original pixels are shifted after the face information is projection-transformed.

[0060] That is, as an example of the data stored in the learning-data storage unit 122, FIG. 4 illustrates that the pieces of learning data one-to-one identified with learning data IDs “F01”, “F02”, and “F03” are created from the image identified with the image ID “A01”. Furthermore, learning data identified with a learning data ID “F01” indicates image data “G01”, a pitch of “0.degree.”, a roll of “0.degree. “, a yaw of “0.degree.”, and post-transformation part information “H01”. Note that although not illustrated in FIG. 4, the learning data may include attribute information such as the race, age, gender, or skin color of the person as the subject, similarly to the original image data.

[0061] Next, the model storage unit 123 will be described. The model storage unit 123 stores the learned model generated by the image processing apparatus 100. For example, the model storage unit 123 stores the detection model 150 and the replacement model 160.

[0062] Note that the model storage unit 123 may store a plurality of types of detection models 150 and a plurality of types of replacement models 160. For example, the model storage unit 123 stores a plurality of different detection models 150 and a plurality of different replacement models 160 in accordance with the format (projection scheme) of a wide angle view image to be input into such models. Furthermore, as will be described below in detail, when replacing a face, the model storage unit 123 performs replacement with an image corresponding to an angle of the detected face, and thus stores a plurality of replacement models 160 generated on an angle basis.

[0063] Referring back to FIG. 2, the explanation will be continued. The control unit 130 is achieved by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (e.g., image processing program according to the present disclosure) stored in the image processing apparatus 100, in the random access memory (RAM) or the like as a working area. Furthermore, the control unit 130 is a controller, and, for example, may be achieved by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0064] As illustrated in FIG. 2, the control unit 130 includes an acquisition unit 131, a learning-data creation unit 132, a model generation unit 133, an acceptance unit 134, a detection unit 135, and a transmission unit 136, and achieves or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to that illustrated in FIG. 2, and thus may be another configuration if the image processing described below is performed with the other configuration.

[0065] The acquisition unit 131 acquires various types of information via, for example, a wired or wireless network. For example, the acquisition unit 131 acquires image data as the original of learning data, from the data server 200.

[0066] For example, the acquisition unit 131 acquires image data including a person as a subject. More specifically, the acquisition unit 131 acquires image data including a person’s face as the subject. At this time, the acquisition unit 131 may acquire face image data including, for example, information regarding an area occupied by the face in the image, an area occupied by each part of the face in the image, and attribute information such as the race, gender, or age of the person. For example, the acquisition unit 131 acquires an image data group in which a plurality of face images are set, from the data server 200 or the like that provides a material for machine learning. Alternatively, the acquisition unit 131 may collect face image data by crawling the network and may acquire the collected face image data.

[0067] The learning-data creation unit 132 performs a predetermined projection transformation on image data having a predetermined target as a subject, and creates learning data including the predetermined target as correct data. Specifically, the learning-data creation unit 132 performs projection transformation corresponding to input data to be input into a model on image data including a predetermined target as a subject, and creates learning data to be used for learning the model.

[0068] For example, the learning-data creation unit 132 creates learning data from image data including a subject of a human face as a predetermined target. Note that the learning-data creation unit 132 may create learning data from image data in which position information of each part included in the human face is set. As a result, the learning-data creation unit 132 can create the learning data for detecting the human face from the image data and generating a model that can detect the positions of parts such as the human eyes and nose.

[0069] For example, the learning-data creation unit 132 accepts a setting regarding a projection scheme for a wide angle view image targeted by a model to be generated, transforms original image data with the projection scheme identical to the accepted projection scheme, and creates learning data.

[0070] Specifically, the learning-data creation unit 132 changes an angle of the subject from an initial state thereof, performs projection transformation on image data corresponding to the changed angle, and creates learning data. As a result, the learning-data creation unit 132 can create the learning data for generating a model directly executable detection processing even on input data including a distorted person’s face, without performing transformation processing such as perspective projection before the detection processing.

[0071] More specifically, the learning-data creation unit 132 changes the respective angles in the pitch direction, the roll direction, and the yaw direction of the subject, performs projection transformation on image data corresponding to the changed angle, and creates learning data. For example, the learning-data creation unit 132 changes respective angles in the pitch direction, the roll direction, and the yaw direction by 1.degree., and creates an image group corresponding to the changed angles. As a result, the learning-data creation unit 132 can obtain a learning data set of 180 pieces different in pitch direction from one piece of original image, a learning data set of 180 pieces different in roll direction for each image of the first learning data set, and a learning data set of 180 pieces different in yaw direction for each image of those learning data sets.

[0072] Note that regarding the yaw direction, the learning-data creation unit 132 may create learning data including only a range in which the subject is in contact with the image frame. This is because the feature amount of the subject simply shifted parallelly (with the yaw angle changed) without contact with the image frame is duplicated in the equirectangular projection scheme. In other words, the learning-data creation unit 132 may create learning data including only an image with the subject divided into two and in contact with the left and right edge portions. That is, the learning-data creation unit 132 does not need to create learning data with only the yaw angles changed, if the subject is not in contact with the image frame. Specifically, the learning-data creation unit 132 may create learning data while changing the angle in the yaw direction within a range from a state where the subject starts contacting the image frame to a state where the subject is away from the image frame. As a result, the learning-data creation unit 132 can reduce the processing load of creating the learning data.

[0073] Note that if the area occupied by the subject (object recognized as the subject) in the image is known, the learning-data creation unit 132 can determine whether the subject is in contact with the image frame (at the left and right edge portions of the image) or is away from the image frame.

[0074] As above, the learning-data creation unit 132 changes the angle of the subject from the initial state thereof, performs the projection transformation on the image data corresponding to the changed angle, and creates the learning data. That is, the learning-data creation unit 132 creates the subject-angle-based learning data set. As an example, when creating learning data from 10000 pieces of original image data, the learning-data creation unit 132 can create 10000 pieces of learning data having, for example, a pitch of 90.degree., a roll of 45.degree., and a yaw of 160.degree.. The learning-data creation unit 132 can also create 10000 pieces of learning data at a pitch angle of 91.degree., a roll angle of 45.degree., and a yaw angle of 160.degree.. The model generation unit 133 described below generates replacement models on an angle basis, with the angle-based learning data in such a manner.

[0075] Here, the details of the learning data created by the learning-data creation unit 132 will be visually described with reference to FIGS. 5 to 8. FIG. 5 illustrates exemplary content projection-transformed with the equirectangular projection scheme.

[0076] Content 70 illustrated in FIG. 5 is an image captured by, for example, an omnidirectional camera. As illustrated in FIG. 5, the content 70 has a spherical shape and such a state is unsuitable for distribution to the HMD 10 or reproduction on the HMD 10. Thus, typically, the content 70 is projection-transformed with a predetermined projection scheme, and transformed into a format suitable for distribution or reproduction.

[0077] Content 71 illustrates in FIG. 5 is an image obtained by projection-transforming the content 70 with the equirectangular projection scheme. In the equirectangular projection scheme, the latitude (vertical direction) and longitude (horizontal direction) of the original image are, respectively, directly transformed into the latitude (vertical direction) and longitude (horizontal direction) of the transformed image. Thus, the transformed image has an equidistance on the centerline in the vertical direction and at the position in the vertical direction (latitude); however, the object in the image is stretched (distorted) near the spherical poles.

[0078] The content 70 and the content 71 each include human faces. Note that although the original image to be actually used by the image processing apparatus 100 includes one face in one image, the example of FIG. 5 illustrates the content 70 and the content 71 each including the plurality of faces for the sake of explanation.

[0079] As illustrated in FIG. 5, a face 72 located at the center of the latitude of the content 70 maintains an almost accurate shape on the content 71. On the other hand, in the content 70, a face 73 located at a higher latitude than the face 72 has a shape of which portion is more distorted as the portion is located at a higher latitude on the content 71. Furthermore, in the content 70, a face 74 located at a higher latitude (closer to the pole) than the face 73 has a more distorted shape than that of the face 73 on the content 71.

[0080] The learning-data creation unit 132 learns a face having a distorted shape as above, and thus creates a distorted face image as learning data. As described above, the learning-data creation unit 132 changes the angles set to the original image, and obtains a distorted face image by projection-transforming the changed image.

[0081] Here, a relationship between a wide angle view image like the content 70 and the eye point of the user will be described with reference to FIG. 6. FIG. 6 illustrates the relationship between the wide angle view image and the eye point of the user. Note that in the example of FIG. 6, there will be described spherical content is taken as an example of the wide angle view image.

[0082] As illustrated in FIG. 6, the eye point of the user at the spherical content is represented by using the elevation angle .PHI., the rotation angle .PSI., and the azimuth angle .theta.. The elevation angle .PHI. is an angle in the longitudinal direction with the XZ plane of the 3D model coordinate system illustrated in FIG. 6 defined as a reference plane. Furthermore, the rotation angle .PSI. is an angle indicating a rotation direction around the user’s line of sight in the 3D model coordinate system illustrated in FIG. 6. Still furthermore, the azimuth angle .theta. is an angle from a predetermined reference axis on the XZ plane that is the horizontal plane of the 3D model coordinate system illustrated in FIG. 6. In the present disclosure, the elevation angle .PHI., the rotation angle .PSI., and the azimuth angle .theta. correspond to an angle in the pitch direction, an angle in the roll direction, and an angle in the yaw direction, respectively.

[0083] Subsequently, distorted face images created by the learning-data creation unit 132 will be described with reference to FIG. 7. FIG. 7 is an explanatory illustration (1) of the creation processing according to the first embodiment of the present disclosure.

[0084] FIG. 7 visually illustrates angles set to each distorted face image created by the learning-data creation unit 132 and what kinds of distorted face images are created actually. First, FIG. 7 illustrates an example in which the learning-data creation unit 132 creates each distorted face image while changing an angle in the pitch direction and an angle in the roll direction.

[0085] For example, the learning-data creation unit 132 reads an image having an angle of 0.degree. in the pitch direction and an angle of 0.degree. in roll direction as an initial state, and creates a distorted face image 77. The face included in the distorted face image 77 has a shape with almost no distortion.

[0086] Thereafter, the learning-data creation unit 132 sequentially creates distorted face images while changing the angle in the pitch direction of the original image every 1.degree.. A distorted face image 78 illustrated in FIG. 7 is a distorted face image created from an image having an angle of 45.degree. in the pitch direction and an angle of 0.degree. in the roll direction. The shape of the face included in the distorted face image 78 corresponds to, for example, the shape of the face 72 illustrated in FIG. 5, and has larger distortion as the face is closer to the pole. On the other hand, a distorted face image 79 illustrated in FIG. 7 is a distorted face image created from an image having an angle of -45.degree. in the pitch direction and an angle of 0.degree. in the roll direction. The shape of the face included in the distorted face image 79 has larger distortion as a portion of the face is located in the opposite direction (downward direction) to the shape of the face included in the distorted face image 78 (location further closer to the lower pole).

……
……
……