Microsoft Patent | Annotation Using a Multi-Device Mixed Interactivity System

Patent: Annotation Using a Multi-Device Mixed Interactivity System

Publication Number: 20190004684

Publication Date: 2019-01-03

Applicants: Microsoft

Abstract

In various embodiments, methods and systems for implementing a multi-device mixed interactivity system are provided. The interactivity system includes paired mixed-input devices for interacting and controlling virtual objects. In operation, a selection profile associated with a virtual object is accessed. The selection profile is generated based on a selection input determined using real input associated with a selection device and virtual input associated with a mixed-reality device. The selection device has a first display and the mixed-reality device has a second display that both display the virtual object. An annotation input for the virtual object based on a selected portion corresponding to the selection profile is received. An annotation profile based on the annotation input is generated. The annotation profile includes annotation profile attributes for annotating a portion of the virtual object. An annotation of the selected portion of the virtual reality object is caused to be displayed.

Background

Virtual experience computing systems (e.g., virtual reality or augmented reality devices) receive input from input devices to select objects in virtual environments and augmented reality environments (i.e., collectively virtual experiences). For example, a virtual reality controller can be used to make a selection in a virtual experience based on tracking motion of the virtual reality controller, where the motion is communicated to a virtual reality computing system of the virtual experience to indicate the selection. Selecting virtual reality objects or augmented reality objects (“virtual objects”) can further be complimented with annotation inputs to associate additional information with the selections. As such, an interactivity system for selection and annotation is an integral part of interacting with virtual experience computing systems.

Summary

Embodiments of the present invention are directed to a multi-device mixed interactivity system for selection and annotation. The multi-device mixed interactivity system (“interactivity system”) includes paired mixed-input devices for interacting with and controlling virtual objects using displays, real inputs and virtual inputs, sensors, and passive and active haptic feedback associated with the paired mixed-input devices. Real device space tracker data and virtual device space tracker data is accessed via the paired mixed-input devices, respectively, to determine real input and virtual input that are integrated and processed to determine selection inputs and annotation inputs. The real device space tracker data and virtual device space tracker data are also used in different interaction contexts for determining selection inputs and annotation inputs.

The interactivity system includes an auxiliary mixed-input device (e.g., a selection device or annotation device) and a virtual mixed-input device (e.g., a mixed-reality device). The auxiliary mixed-input device is configured to determine real input based on real device space tracker data associated with the auxiliary mixed-input device. The auxiliary mixed-input device is a mobile device having a first display that displays a virtual object. The virtual mixed-input device is configured to determine virtual input based on virtual device space tracker data associated with the virtual mixed-input device. The virtual mixed-input device is a head-mounted display having a second display that displays the virtual object. The virtual object is an augmented reality object via the second display.

An integrated processing component is configured to determine a selection input or annotation input based on the real input and the virtual input. The selection input is used to determine a selected portion of the virtual object. The annotation input is used to determine an annotation for the virtual object. The integrated processing component includes a selection manager configured to process selection inputs and interfaces and an annotation manager configured to process annotation inputs and interfaces. The integrated processing component is further configured to cause display on the first display associated with the auxiliary mixed-input device the selected portion or the annotation of the virtual object. The integrated processing component is further configured to cause display on the second display associated with virtual mixed-input device the selected portion or the annotation of the virtual object.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in isolation as an aid in determining the scope of the claimed subject matter.