Magic Leap Patent | Augmented Reality Identity Verification

Patent: Augmented Reality Identity Verification

Publication Number: 20200184201

Publication Date: 20200611

Applicants: Magic Leap

Abstract

An augmented reality device (ARD) can present virtual content which can provide enhanced experiences with the user’s physical environment. For example, the ARD can detect a linkage between a person in the FOV of the ARD and a physical object (e.g., a document presented by the person) or detect linkages between the documents. The linkages may be used in identity verification or document verification.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of U.S. application Ser. No. 15/611,398, filed on Jun. 1, 2017, entitled “AUGMENTED REALITY IDENTITY VERIFICATION,” and claims the benefit of priority under 35 U.S.C. .sctn. 119(e) to U.S. Provisional Application No. 62/345,438, filed on Jun. 3, 2016, entitled “AUGMENTED REALITY IDENTITY VERIFICATION,” the disclosure of which is hereby incorporated by reference herein in its entirety.

FIELD

[0002] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems and more particularly to various authentication techniques in an augmented reality environment.

BACKGROUND

[0003] Modern computing and display technologies have facilitated the development of systems for so called “virtual reality”, “augmented reality”, or “mixed reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR”, scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR”, scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user; a mixed reality, or “MR”, related to merging real and virtual worlds to produce new environments where physical and virtual objects co-exist and interact in real time. As it turns out, the human visual perception system is very complex, and producing a VR, AR, or MR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements is challenging. Systems and methods disclosed herein address various challenges related to VR, AR and MR technology.

SUMMARY

[0004] Various embodiments of an augmented reality system for detecting linkages among objects/people in a user’s environment or authenticating the objects/people are disclosed.

[0005] In one embodiment, an augmented reality (AR) system for detecting a linkage in an AR environment is disclosed. The augmented reality system comprises an outward-facing imaging system configured to image an environment of the AR system; an AR display configured to present virtual content in a three-dimensional (3D) view to a user of the AR system; and a hardware processor. The hardware processor is programmed to: obtain, with the outward-facing imaging system, an image of the environment; detect a first face and a second face in the image, wherein the first face is the face of a person in the environment and wherein the second face is a face on an identification document; recognize the first face based on first facial features associated with the first face; recognize the second face based on the second facial features; analyze the first facial features and the second facial features to detect a linkage between the person and the identification document; and instruct the AR display to present a virtual annotation indicating a result of the analysis of the first facial features and the second facial features.

[0006] In another embodiment, a method for detecting a linkage in an augmented reality environment is disclosed. The method can be performed under control of an augmented reality device comprising an outward-imaging imaging system and a hardware processor, the augmented reality device configured to display virtual content to a wearer of the augmented reality device. The method can comprise: obtaining an image of the environment; detecting a person, a first document, and a second document in the image; extracting first personal information based at least partly on an analysis of the image of the first document; accessing second personal information associated with second document; extracting third personal information of the person based at least partly on an analysis of the image of the person, wherein the first personal information, the second personal information, and the third personal information are in a same category; determining a likelihood of match among the first personal information, the second personal information, and the third personal information; and displaying a linkage of among the first document, the second document, and the person in response to a determination that the likelihood of match exceeds a threshold condition.

[0007] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects, and certain physical objects viewed by a person.

[0009] FIG. 2 schematically illustrates an example of a wearable system.

[0010] FIG. 3 schematically illustrates aspects of an approach for simulating three-dimensional imagery using multiple depth planes.

[0011] FIG. 4 schematically illustrates an example of a waveguide stack for outputting image information to a user.

[0012] FIG. 5 shows example exit beams that may be outputted by a waveguide.

[0013] FIG. 6 is a schematic diagram showing an optical system including a waveguide apparatus, an optical coupler subsystem to optically couple light to or from the waveguide apparatus, and a control subsystem, used in the generation of a multi-focal volumetric display, image, or light field.

[0014] FIG. 7 is a block diagram of an example of a wearable system.

[0015] FIG. 8 is a process flow diagram of an example of a method of rendering virtual content in relation to recognized objects.

[0016] FIG. 9 is a block diagram of another example of a wearable system.

[0017] FIG. 10 is a process flow diagram of an example of a method for determining user input to a wearable system.

[0018] FIG. 11 is a process flow diagram of an example of a method for interacting with a virtual user interface.

[0019] FIG. 12A illustrates an example of identity verification by analyzing linkages between a person and a document.

[0020] FIG. 12B illustrates an example of identity verification by analyzing linkages between two documents.

[0021] FIG. 13 is a flowchart of an example process for determining a match between a person and an identification document presented by the person.

[0022] FIG. 14 is a flowchart of an example process for determining a match between two documents.

[0023] FIG. 15 is a flowchart of example process for determining a match between a person and a plurality of documents.

[0024] Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure. Additionally, the figures in the present disclosure are for illustration purposes and are not to scale.

DETAILED DESCRIPTION

Overview

[0025] An augmented reality device (ARD) can present virtual content which can enhance a user’s visual or interaction experiences with the user’s physical environment. The user can perceive the virtual content in addition to the physical content seen through the ARD.

[0026] For example, at an airport security checkpoint, a traveler usually presents his or her identification document (e.g., a driver’s license or passport) to an inspector who may wear the ARD. The driver’s license can include identifying information such as the traveler’s name, photo, age, height, etc. The traveler may also present a ticket which can include travel information such as the traveler’s name, destination, carrier, etc. The inspector may view the traveler (as well as other persons in the traveler’s environment) and the traveler’s documents through the ARD. The ARD can image the traveler and the traveler’s documents and detect linkages among the traveler’s documents and the traveler (or others in the environment, such as traveling companions).

[0027] For example, the ARD can image the traveler’s passport to detect a photograph of the traveler and compare it to an image of the traveler obtained by an outward-facing camera on the ARD to determine whether the passport photograph is that of the traveler. The ARD may image the traveler’s ticket and determine the name on the ticket and compare it to the name on the traveler’s passport. The ARD can provide a visual focus indicator showing information about the linkages found among the documents or between a document and the traveler. For example, the ARD may display a border around the passport photograph and around the traveler, and a virtual graphic showing the likelihood of a match between the traveler and the person shown in the photograph (e.g., the facial characteristics of the traveler match the photo on the passport). The inspector can use the virtual information displayed by the ARD to pass the traveler through security (in the event of a high degree of match for the linkage between the photo and the traveler) or take further action (in the event of a low degree of match for the linkage).

[0028] The ARD can additionally or alternatively determine that the traveler is the same person to whom the ticket was issued by verifying that the information on the ticket matches the information on the identify document (e.g., name or address).

[0029] Advantageously, the ARD can ameliorate the problem of degraded visual analysis and judgment in repeated tasks (e.g., repeating an identity verification task on a large number of individuals) and increase accuracy of identity verification if the identity verification were to be conducted by a human inspector (rather than by programmatic image comparison by the ARD). However, using the ARD for identity verification can also present challenges unique to the device because the ARD may not be equipped with human cognition to recognize and compare human characteristics by, for example, identifying faces and comparing facial features. Furthermore, the ARD may not know what to look for during an identity verification process because the ARD may not be able to identify the person or the document that needs to be verified. To address these challenges, the ARD may use its imaging system to obtain an image of the document and the person presenting the document. The ARD can identify information on the document (e.g., an image of the face of the person who was issued the document) and identify relevant features of the person (e.g., facial or other body features). The ARD can compare the information from the document with the features of the person and calculate a confidence level. When the confidence level is higher than a threshold, the ARD may determine that the person presenting the document is indeed the person described by the document. The ARD may also extract other identifying information for the document (e.g., age, height, gender) and compare the extracted information to the corresponding characteristic estimated from the person. The ARD may present an annotation of showing a match (or non-match) to the wearer of the ARD. For example, an image on the driver’s license can be highlighted and linked to the person’s face to show a match or a non-match. Additional details related to identity verification by an ARD are further described with reference to FIGS. 12A-15.

[0030] As another example of providing an enhanced user experience with physical objects in the user’s environment, the ARD can identify linkages of physical objects in the user’s environment. Continuing with the example in the preceding paragraph, a traveler may present multiple documents to an inspector. For example, an airline passenger may present a driver’s license (or passport) as well as an airline ticket. The ARD can analyze linkages of such multiple documents by obtaining an image of the documents. The ARD can compare the information extracted from one document with information extracted from another document to determine whether the information in the two documents is consistent. For example, the ARD can extract a name from a driver’s license and compare it to a name extracted from the airline ticket to determine whether the airline ticket and the driver’s license likely were issued to the same person. As described above, the ARD can identify a facial match of an image from the driver’s license to an image of the person to determine that the person, the driver’s license, and the airline ticket are associated with each other. In some embodiments, the ARD may extract information from one of the documents (e.g., a bar code) and retrieve additional information from another data source. The ARD can compare the retrieved information with information extracted from the image of the document. If the information between the two documents is inconsistent, the ARD may determine either or both documents are falsified. In some embodiments, the ARD may conduct additional analyses or require a user of the ARD to manually verify the information when the information between the two documents appears to be inconsistent. On the other hand, if the ARD determines that information in both documents is consistent, the ARD may find that either document is valid or both documents are valid. Further, by matching identifying information extracted from the documents with identifying information extracted from an image of the person, the ARD can determine whether the person was likely issued one or both documents.

[0031] Although the examples are described with reference to an ARD, the systems and methods in the present disclosure are not required to be implemented by the ARD. For example, the systems and methods for identity and document verification may be part of a robotic system, security system (e.g., at a transportation hub), or other computing systems (such as an automatic travel check-in machine). Further, one or more features and processes described herein are not required to be performed by the ARD itself. For example, the process of extracting information from an image may be performed by another computing device (e.g., a remote server).

[0032] Also, the devices and techniques described herein are not limited to the illustrative context of security at a travel hub but can be applied in any context where it is desirable to extract information from documents, make comparisons among documents or persons, identify persons in the environment of the device, enhance security, etc. For example, a ticket taker at an amusement park or entertainment venue could use embodiments of the techniques and devices described herein to admit (or deny admittance) to patrons entering the park or venue. Similarly, a guard at a secure facility (e.g., a private laboratory or warehouse, an office building, a prison, etc.) or a police officer could use the ARD to image a person and an identification document. In yet other applications, a person viewing a number of documents through the ARD (e.g., an accountant viewing invoices, receipts, and account ledgers) can use the ability of the ARD to identify or highlight information that may exist on the documents being viewed (e.g., the accountant’s ARD can highlight documents that include a particular person’s name or expense so that the accountant can more readily reconcile a receipt to an invoice, etc.) to expedite tasks.

……
……
……