Google Patent | High Speed, High-Fidelity Face Tracking

Patent: High Speed, High-Fidelity Face Tracking

Publication Number: 20180356883

Publication Date: 2018-12-13

Applicants: Google

Abstract

An electronic device estimates a pose of a face by fitting a generative face model mesh to a depth map based on vertices of the face model mesh that are estimated to be visible from the point of view of a depth camera. A face tracking module of the electronic device receives a depth image of a face from a depth camera and generates a depth map of the face based on the depth image. The face tracking module identifies a pose of the face by fitting a face model mesh to the pixels of a depth map that correspond to the vertices of the face model mesh that are estimated to be visible from the point of view of the depth camera.

Background

The present disclosure relates generally to imagery capture and processing and more particularly to face tracking using captured imagery.

Face tracking allows facial expressions and head movements to be used as an input mechanism for virtual reality and augmented reality systems, thereby supporting a more immersive user experience. A conventional face tracking system captures images and depth data of the user’s face and fits a generative model to the captured image or depth data. To fit the model to the captured data, the face tracking system defines and optimizes an energy function to find a minimum that corresponds to the correct face pose. However, conventional face tracking systems typically have accuracy and latency issues that can result in an unsatisfying user experience.