Microsoft Patent | Image Processing For Person Recognition

Patent: Image Processing For Person Recognition

Publication Number: 20200380245

Publication Date: 20201203

Applicants: Microsoft

Abstract

An image processing system is described which has a memory holding at least one image depicting at least one person previously unseen by the image processing system. The system has a trained probabilistic model which describes a relationship between image features, context, identities and a plurality of names of people, wherein at least one of the identities identifies a person depicted in the image without an associated name in the plurality of names. The system has a feature extractor which extracts features from the image, and a processor which predicts an identity of the person depicted in the image using the extracted features and the probabilistic model.

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is a continuation application of and claims priority to U.S. patent application Ser. No. 15/723,144, entitled “IMAGE PROCESSING FOR PERSON RECOGNITION,” filed on Oct. 2, 2017, the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

[0002] Image processing systems which recognize faces in images and videos typically operate by extracting facial features from the images and applying template matching or classification. In the case of template matching a library of templates is available with each template being annotated as representing the face of a specified person. When extracted facial features from an incoming image match a particular template the system recognizes the face of the person associated with the template. In the case of classification an automated classifier such as a neural network is trained in advance using huge quantities of images depicting faces of specified people. In order to annotate the templates or annotate the training images significant time and expense is involved. These types of face recognition systems work well in controlled environments where the lighting is good and the person is facing the camera but are often not robust where lighting changes, occlusion, and different camera viewpoints occur.

[0003] Existing face recognition systems do not behave or operate in the same way as a human does. As a result the functionality of such face recognition systems is limited as compared with a human who is trying to recognize individuals. Also, because existing face recognition systems do not behave or operate in the same way as a human does the existing face recognition systems are not intuitive to use or integrate with other automated systems.

[0004] The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known image processing systems for person recognition.

SUMMARY

[0005] The following presents a simplified summary of the disclosure in order to provide a basic understanding to the reader. This summary is not intended to identify key features or essential features of the claimed subject matter nor is it intended to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0006] An image processing system is described which has a memory holding at least one image depicting at least one person previously unseen by the image processing system. The system has a trained probabilistic model which describes a relationship between image features, context, identities and a plurality of names of people, wherein at least one of the identities identifies a person depicted in the image without an associated name in the plurality of names. The system has a feature extractor which extracts features from the image, and a processor which predicts an identity of the person depicted in the image using the extracted features and the probabilistic model.

[0007] Many of the attendant features will be more readily appreciated as the same becomes better understood by reference to the following detailed description considered in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

[0008] The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein:

[0009] FIG. 1 is a schematic diagram of an image processor for person recognition.

[0010] FIG. 2 is a flow diagram of a method of operation at an image processor such as that of FIG. 1;

[0011] FIG. 3 is a flow diagram of part of the method of FIG. 2 in more detail;

[0012] FIG. 4 is a schematic diagram of another example of an image processor;

[0013] FIG. 5 is a schematic diagram of clusters computed by an image processor;

[0014] FIG. 6 is an example of a probabilistic model for use in the image processor of FIG. 1;

[0015] FIG. 7 is an example of another probabilistic model for use in the image processor of FIG. 1;

[0016] FIG. 8 is a flow diagram of a method of removing an identity or a person from the image processor of FIG. 1;

[0017] FIG. 9 is a flow diagram of another method of removing an identity or a person from the image processor of FIG. 1;

[0018] FIG. 10 is a flow diagram of a method of switching on or off a familiar stranger functionality of the image processor of FIG. 1;

[0019] FIG. 11 illustrates an exemplary computing-based device in which embodiments of an image processor are implemented.

[0020] Like reference numerals are used to designate like parts in the accompanying drawings.

DETAILED DESCRIPTION

[0021] The detailed description provided below in connection with the appended drawings is intended as a description of the present examples and is not intended to represent the only forms in which the present example are constructed or utilized. The description sets forth the functions of the example and the sequence of operations for constructing and operating the example. However, the same or equivalent functions and sequences may be accomplished by different examples.

[0022] The technology described herein uses images and videos of people and stores names of people and identifiers of people. Users of the technology, and those people observed in the images and videos may give their consent to the use of the technology in advance and are made aware of the type of data which will be stored. Privacy concerns and sensitivity of data is fully taken into account.

[0023] The methods herein, which involve the observation of people in their daily lives, are enacted with utmost respect for personal privacy. Accordingly, the methods presented herein are fully compatible with opt-in participation of the persons being observed. In embodiments where personal data is collected on a local system and transmitted to a remote system for processing, that data is encrypted in a known manner. In other embodiments, personal data is confined to a local system, and only non-personal, summary data transmitted to a remote system.

[0024] Consider a scenario where a partially or fully sighted user moves to a new city or goes to a cocktail party. The user meets a person and often experiences difficulty naming the person or recalling whether he or she has met the person before. If the user commutes by public transport to work he or she encounters other commuters on a regular basis but typically does not know their names. If the user enters a medical clinic he or she encounters a person that was encountered on the previous visit to the same clinic, but the user does not know the name of that person. The user is often able to use reasoning to infer the names of people, such as by using context. The context includes things like the situation the user is in such as a lunchtime context, a garden context, a swimming pool context and has associated information about what people are likely to be encountered in each of the different contexts.

[0025] The present technology recognizes the problem that existing face recognition systems are not human-like in a variety of ways, such as the ability to make inferences using context and/or the ability to recognize a familiar stranger. A familiar stranger is a person who has been encountered before but for whom no name is known. The ability to make inferences is very powerful and people are able to use environment context such as time of day and/or the geographical location. Environment context also includes the type of environment such as medical clinic, public transport, office, garden.

[0026] In order to enable the present technology to make inferences a probabilistic model is used so that inference is able to be computed using the probabilistic model. A probabilistic model comprises a plurality of variables represented using probability distributions. Some of the variables take values which are observed empirically and some of the variables are unobserved, latent variables that take values learnt during training. The probabilistic model is carefully designed so that identities are represented separately from names; and this enables familiar strangers to be taken into account. An identity is a unique identifier assigned by the image processing system to a person. The identifier is a number or other identifier. In contrast, a name is one of a plurality of person names stored in the image processing system. By separating identities and names in this way the image processing system is able to make inferences about familiar strangers and so become more human like. The term “person recognition” is used to refer to computing a prediction of a unique identifier and/or name of a person depicted in an image.

[0027] Usually in machine learning it is assumed that labels are error-free, such as the name labels of the people depicted in the images in the present case. However, the present technology explicitly models that the user may make errors, and some users may make more errors then others. This is done using so a naming model with a noise parameter that is either constant over time or slowly changes over time.

[0028] FIG. 1 is a schematic diagram of an image processor 100 which is computer implemented. The image processor receives as input one or more images 102 such as video, color camera images, depth images, or other images. The image processor optionally receives as input sensed environment data 104 such as light sensor data, global positioning sensor data, pressure sensor data, accelerometer data, touch sensor data, time data, or other sensed environment data 104. The image processor optionally receives user input 106 comprising names of people in particular images, but this is not essential as the image processor is able to operate in an unsupervised manner.

[0029] The image processor comprises a probabilistic model as mentioned above and which is described below in more detail. The image processor uses the inputs it receives to update observed variables of the probabilistic model and to make inferences to update unobserved variables of the probabilistic model. The unobserved variables are referred to as learnt latent variables 108 and these are available as output of the image processor 100 as indicated in FIG. 1. The probabilistic model is used to compute predictions. The predictions are predictions of any of the variables of the model and this includes one or more of: identities of people depicted in the images 102, names of people depicted in the images 102, contexts, names of people likely to be encountered next or in particular contexts, identities of people likely to be encountered next or in particular contexts. The predictions are provided to the user using audio and/or graphical indications rendered onto a display of the images 102 as illustrated in FIG. 1 or onto an augmented reality display.

[0030] In the example of FIG. 1 the red circle 122 indicates a face which is detected by the image processor 100 but for which the image processor identifies that not enough detail is available to compute an accurate identity and/or name of the person (because the face is occluded by a hand). The green circle 122 indicates a face which is detected by the image processor and for which context data is available in the model. The yellow square 114 within the green circle 122 indicates a region of interest computed by the image processor and from which image features are extracted for processing by the probabilistic model. Using the image features from region of interest 114 the image processor 100 computes that the name of the depicted person is Adam with a probability of 97.5%. The image processor 100 also computes region of interest 118 and extracts image features from that region of interest. The features from region of interest 118 are input to the probabilistic model which predicts that the person depicted in region of interest 118 is a new person not previously encountered by the image processor 100 with a probability of 100%.

[0031] Suppose that the person in the red circle 122 does not have a hand occluding the face. In this case a region of interest is detected over the face in the red circle 122 and features are extracted from the region of interest. The features are input to the probabilistic model which predicts that the face is the second of two familiar strangers (people who have been encountered before by the image processor 100 but for whom no name is known with high certainty by the image processor 100). In this case the image processor displays the wording “familiar stranger 2” next to the face in red circle 122.

[0032] In the example of FIG. 1 face detection is used. However, the image processor uses body detection and body recognition in some examples, either in addition to face detection or as an alternative to face detection.

[0033] In some examples the image processor of FIG. 1 is deployed at a server which is in communication with one or more client computing devices, such as smart phones, personal computers, augmented reality head worn computing devices and others. The functionality of the image processor is at the server or is shared between the server and the client computing device.

[0034] The image processor of FIG. 1 is deployed in a user device in some examples, such as in an augmented reality head worn computing device or in a smart phone.

[0035] Alternatively, or in addition, the functionality of the image processor described herein is performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that are optionally used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

[0036] FIG. 2 is a flow diagram of a method of operation at an image processor such as that of FIG. 1. Captured images 102 are received such as web camera video images and/or depth images from a depth camera. The image processor detects one or more regions of interest (ROIs) in the images. This is done by using a face detector or a body detector. Known face detectors and known body detectors are available. A region of interest is thus a part of an image which is likely to depict a face or a body of a person.

[0037] The image processor checks the quality of the detected regions of interest and discards any which have criteria below a specified threshold. For example, this is done by determining the effective resolution of the detected region of interest which is comprised of the pixel resolution and the amount of imaging artifacts present, such as imaging noise and blur. In the case of regions related to face detection additional information such as whether the face is facing towards the camera are utilized to retain regions with enough visible face.

[0038] For a given region the image processor computes features. In an example this is done by inputting the region to a neural network which reduces the dimensionality of the image region and outputs a vector of specified length. The neural network has been trained in advance to compute an embedding of an image region into a space of a specified number of dimensions, using known technology.

[0039] The image processor also receives sensed environment data 104 associated with the captured images 102 in some cases. That is, the sensed environment data 104 is optional. The image process optionally receives names 212 annotated on one or more of the images 102. Note that the name input 212 is optional as the probabilistic model is able to train using unsupervised training.

[0040] The image processor updates 206 observed variables of the probabilistic model using the computed features 204 and where available the sensed environment data 104 and where available the name(s) 212. This is done by incorporating the observed data into the probabilistic model by adding new observed variables to the probabilistic model. Once the observed variables have been updated 206 inference is carried out to compute updates 208 to the latent variables of the probabilistic model. The inference is computed using a Bayesian update process and is achieved through one or more of: message passing algorithms, Markov chain Monte Carlo procedures such as Gibbs sampling or Metropolis-Hastings,* variational inference or others*

……
……
……