Magic Leap Patent | Viewing system with interpupillary distance compensation based on head motion
Patent: Viewing system with interpupillary distance compensation based on head motion
Drawings: Click to check drawins
Publication Number: 20210341996
Publication Date: 20211104
Applicants: Magic Leap
Assignee: Magic Leap
Abstract
The invention provides a viewing system including an augmented reality system that generates a visual presentation to a user based at least in part on an IPD of the user, and an IPD compensator that adjusts the visual presentation based on an IPD compensation factor.
Claims
1. A viewing system method: detecting, with an interpupillary distance (IPD) detector, a plurality of IPD's of a user to generate IPD data that includes a plurality of IPD values; detecting, with a head movement detector device, movement of a head of a user to generate head movement data that includes a plurality of head positions based on movement of the head of the user; generating, with a correlator connected to the IPD detector and the head movement detector device, a correlation between the IPD data and the head movement data such that different head positions have different associated respective IPD compensation factors; storing, with a storing system, being a software component executable by a processor, connected to the correlator, the correlation; generating, with an augmented reality system having a light projector, a visual presentation to a user based at least in part on an IPD of the user; calculating, with an IPD compensation factor calculator connected to the head movement detector device, a select IPD compensation factor based on the head movement data such that different head positions have different associated respective IPD compensation factors; and adjusting, with an IPD compensator, the visual representation based on the select IPD compensation factor.
2. The method of claim 1, further comprising: securing an apparatus frame securable to a head of the user, the IPD detector and head movement detector device being secured to the apparatus frame.
3. The method of claim 2, wherein the IPD detector is a camera with a field of capture oriented towards eyes of the user.
4. The method of claim 2, wherein the head movement detector device includes one or more accelerometers, gyroscopes, inertial measurement units (IMU's) or cameras.
5. The method of claim 2, wherein the head movement detector device determines a least one of rotation and position of the head of the user.
6. The method of claim 2, further comprising: having the user bite on a mouth bit interface to fixedly attach the apparatus frame to the head of the user.
7. The method of claim 6, wherein the user can accelerate their head while the IPD data is collected.
8. The method of claim 1, further comprising: calculating, with an IPD compensation factor calculator, an IPD compensation factor based on the correlation.
9. A viewing method, comprising: generating, with an augmented reality system having a light projector, a visual presentation to a user based at least in part on an interpupillary distance (IPD) of the user; detecting, with a head movement detector device, movement of a head of a user and generate head movement data based on movement of the head of the user; detecting, with an IPD detector, an IPD of a user and generate IPD data; generating, with a correlator connected to the IPD detector and the head movement detector device, a correlation between the IPD data and the head movement data such that different head positions have different associated respective IPD compensation factors; storing, with a storing system, being a software component executable by a processor, connected to the correlator to store the correlation; calculating, with an IPD compensation factor calculator connected to the head movement detector device, and operable to calculate a select IPD compensation factor based on the head movement data and the correlation_such that different head positions have different associated respective IPD compensation factors; and adjusting, with an IPD compensator, the visual presentation based on the select IPD compensation factor.
10. The method of claim 9, wherein the head movement detector device includes a pitch angle detector that detects pitch angle of a head of the user, wherein the IPD compensation factor is dependent on the pitch angle by the pitch angle detector.
11. The method of claim 9, further comprising: guiding, with a viewing calibration system, including at least a software component executable by a processor, the user through a series of viewing exercises to determine one or more IPD compensation factors.
12. The method of claim 9, further comprising: securing an apparatus frame to a head of the user, the IPD detector and head movement detector device being secured to the apparatus frame.
13. The method of claim 12, wherein the IPD detector is a camera with a field of capture oriented towards eyes of the user.
14. The method of claim 12, wherein the head movement detector device includes one or more accelerometers, gyroscopes, inertial measurement units (IMU's) or cameras.
15. The method of claim 12, wherein the head movement detector device determines at least one rotation and position of the head of the user.
16. The method of claim 12, further comprising: having the user bite on a mouth bit interface to fixedly attach the apparatus frame to the head of the user.
17. The method of claim 16, wherein the user can accelerate their head while the IPD data is collected.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is continuation of U.S. patent application Ser. No: 16/530,776, filed Aug. 2, 2019, which claims priority from U.S. Provisional Patent Application No. 62/714,056, filed on Aug. 2, 2018, all of which is incorporated herein by reference in its entirety.
FIELD OF THE INVENTION
[0002] This invention is related to connected mobile computing systems, methods, and configurations, and more specifically to mobile computing systems, methods, and configurations featuring at least one wearable component which may be utilized for virtual and/or augmented reality operation.
BACKGROUND
[0003] It is desirable that mixed reality, or augmented reality, near-eye displays be lightweight, low-cost, have a small form-factor, have a wide virtual image field of view, and be as transparent as possible. In addition, it is desirable to have configurations that present virtual image information in multiple focal planes (for example, two or more) in order to be practical for a wide variety of use-cases without exceeding an acceptable allowance for vergence-accommodation mismatch.
SUMMARY OF THE INVENTION
[0004] The invention provides a viewing system. Including an interpupillary distance (IPD) detector that is positionable to detect an IPD of a user and generate IPD data, a head movement detector device that generates head movement data based on movement of a head of the user, a correlator connected to the IPD detector and the head movement detection device to generate a correlation between the IPD data and the head movement data and a storing system connected to the correlator to store the correlation.
[0005] The viewing device may further include an apparatus frame securable to a head of the user, the IPD detector and head movement device being secured to the apparatus frame.
[0006] The viewing device may further include that the IPD detector is a camera with a field of capture oriented towards eyes of the user.
[0007] The viewing device may further include that the head movement detector includes one or more accelerometers, gyroscopes, inertial measurement units (IMU's) or cameras.
[0008] The viewing device may further include that the head movement detector determines a least one rotation and position of the head of the user.
[0009] The viewing device may further include a mouth bit interface for the user to bite on to fixedly attach the apparatus frame to the head of the user.
[0010] The viewing device may further include that the user can accelerate their head while the IPD data is collected.
[0011] The viewing device may further include an IPD compensation factor calculator that calculates an IPD compensation factor based on the correlation.
[0012] The viewing device may further include an augmented reality system that generates a visual presentation to a user based at least in part on an IPD of the user and an IPD compensator that adjusts the visual representation based on the IPT compensation factor.
[0013] The invention also provides a viewing system, including an augmented reality system that generates a visual presentation to a user based at least in part on an IPD of the user, and an IPD compensator that adjusts the visual presentation based on an IPD compensation factor.
[0014] The viewing system may further include a pitch angle detector that detects pitch angle of a head of the user, wherein the IPD compensation factor is dependent on the pitch angle by the pitch angle detector.
[0015] The viewing system may further include a viewing calibration system that guides the user through a series of viewing exercises to determine one or more IPD compensation factors.
[0016] The viewing system may further include an IPD detector that is positionable to detect an IPD of a user and generate IPD data, a head movement detector device that generates head movement data based on movement of a head of the user, a correlator connected to the IPD detector and the head movement detection device to generate a correlation between the IPD data and the head movement data and a storing system connected to the correlator to store the correlation.
[0017] The viewing system may further include an apparatus frame securable to a head of the user, the IPD detector and head movement device being secured to the apparatus frame.
[0018] The viewing system may further include that the IPD detector is a camera with a field of capture oriented towards eyes of the user.
[0019] The viewing system may further include that the head movement detector includes one or more accelerometers, gyroscopes, inertial measurement units (IMU's) or cameras.
[0020] The viewing system may further include that the head movement detector determines a least one rotation and position of the head of the user.
[0021] The viewing system may further include a mouth bit interface for the user to bite on to fixedly attach the apparatus frame to the head of the user.
[0022] The viewing system may further include that the user can accelerate their head while the IPD data is collected.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The invention is further described by way of example with reference to the accompanying drawings, wherein:
[0024] FIG. 1 is a schematic drawing illustrating an augmented reality viewing system;
[0025] FIG. 2 is a schematic drawing of a user, illustrating various head movements by the user and changes in an interpupillary distance (IPD) of the user;
[0026] FIG. 3 is a view similar to FIG. 2 with the user tilting their head in an upward direction;
[0027] FIGS. 4A and 4B are perspective views illustrating a user with an experimental apparatus that is used to detect IPD compensations based on head motions;
[0028] FIG. 5 is a graph illustrating IPD compensating factors relative to head pitch;
[0029] FIG. 6A is a flow chart illustrating IPD head rotation compensation;
[0030] FIG. 6B is a flow chart illustrating IPD head rotation compensation;
[0031] FIG. 6C is a flow chart illustrating IPD head rotation compensation;
[0032] FIG. 7 is a partial top plan view and partial block diagram of an augmented reality system; and
[0033] FIG. 8 is a top plan view of the augmented reality system illustrating IPD compensating features thereof.
DETAILED DESCRIPTION
[0034] Referring to FIG. 1, an augmented reality system is illustrated featuring a head-worn viewing component (2), a hand-held controller component (4), and an interconnected auxiliary computing or controller component (6) which may be configured to be worn as a belt pack or the like on the user. Each of these components may be operatively coupled (10, 12, 14, 16, 17, 18) to each other and to other connected resources (8) such as cloud computing or cloud storage resources via wired or wireless communication configurations, such as those specified by IEEE 802.11, Bluetooth.RTM., and other connectivity standards and configurations. As described, for example, in U.S. patent application Ser. Nos. 14/555,585, 14/690,401, 14/331,218, 15/481,255, and 62/518,539, each of which is incorporated by reference herein in its entirety, various aspects of such components are described, such as various embodiments of the two depicted optical elements (20) through which the user may see the world around them along with visual components which may be produced by the associated system components, for an augmented reality experience. In various embodiments, such as the many described in the aforementioned incorporated by reference patent applications, one or more of the components may feature devices or subcomponents, such as accelerometers, gyroscopes, potentiometers, integrated inertial measurement units ("IMU"), and cameras, which may be utilized to determine or estimate the position and/or orientation of an intercoupled user body part (such as the position or orientation of a user's head when coupled to an instrumented head-worn viewing component (2)), as well as assist in determining velocities and/or accelerations thereof, linearly and/or angularly. In various embodiments, such as the many described in the aforementioned incorporated by reference patent applications, it may be valuable for the system to utilize as at least one input the inter-pupillary distance ("IPD") of the individual user in presenting such user with visual information pertaining to the augmented or virtual reality experience. In various embodiments, it may be convenient to simply measure a user's IPD before use of a pertinent system and provide this information to the system as a user input; in other embodiments, it may be that the system is configured to utilize inward-directed (i.e., toward the eyes of the user) devices such as cameras to automatically determine the user's IPD information before and/or during runtime of various applications or presented information. As is discussed in further detail below, while utilizing various embodiments of augmented reality systems and associated testing apparatuses, we have determined that various users may benefit from a compensation or adjustment in the positioning of presented augmented reality information as such users rotate or re-orient their heads in various ways relative to the rest of their bodies and the environments around them. For example, in one embodiment it may be valuable to have a compensation factor that slightly varies the z-axis position (i.e., straight out from the plane of the user's face) of presented augmented reality information with the pitch position of the user's head. This may be related to at least some of these users experiencing an actual or functional change in IPD as they change the pitch of their head, yaw their head to the side, or even roll their head (i.e., such as about a z-axis extending perpendicularly from their nose). In one embodiment, a variation in the IPD information inputted into the pertinent calculations, which may be correlated with head orientation (such as in the form of an equation or lookup table correlating IPD adjustment factor or compensation with head orientation), may be utilized as a compensating variable to generate the presented augmented reality information to the user in such configurations.
[0035] Referring to FIG. 2, a representation of a user's body (30) with attached head (32) is shown, the user being positioned within a room comprising fixed walls (22, 24), floor (28), and ceiling (26), which together may be associated with a global coordinate system for such room featuring X,Y,Z cartesian coordinates (40, 42, 44, respectively), for example. Another coordinate system may be associated with the head (32) of the user, such that the Z axis (38) is approximately straight out from the face, and the X (34) and Y (36) axes are orthogonal to the Z axis (38) as shown in FIG. 2. The user's head (32) is oriented such that the Z axis (38) is approximately parallel with the Z axis of the room global coordinate system (44), and the gaze vectors (52, 54) from the eyes of the user are focused on a target (46), which may be virtual or actual, at the position of the left wall (22), at a position that causes level eye gaze (52, 54) to be approximately parallel to the Z axis (38) of the user's head, which, as noted above for this example, is approximately level with the floor, or approximately parallel with the Z axis (44) of the room coordinate system. From such a position, the user may pitch their head down toward the floor, or up toward the ceiling. This depicted position may be deemed a zero rotation position, for example, with pitch down toward the floor up to about -90 degrees, and up toward the ceiling by about +90 degrees, capable by the typical person. In the zero rotation position, the IPD (50) may be measured manually and/or automatically using aspects of the augmented reality system wearable component (2).
[0036] Referring to FIG. 3, the same user (30, 32) is illustrated, with the user's head rotated up (56) to a pitch of about +50 degrees relative to the plane of the floor (28; or the Z axis of the room coordinate system 44), and the gaze of the eyes (52, 54) of the augmented reality system wearable component (2) user are directed toward a second target (48). In such a rotated configuration, the IPD may be measured manually and/or automatically using aspects of the augmented reality system wearable component (2). In laboratory experiments using such a configuration on various subject users, we have found variation in the IPD (50) as the head pitch angle is varied.
[0037] Using an experimental apparatus such as that depicted in FIGS. 4A and 4B, we have gathered data from the repositioning of the heads of various users as regards linear and/or rotational position (i.e., relative to the room or surrounding environment), linear and/or rotational velocity (i.e., relative to the room or surrounding environment), and linear and/or rotational acceleration (i.e., relative to the room or surrounding environment). The depicted apparatus comprises a high resolution camera (62) with a field of capture oriented toward the eyes (70) of the user such that the user's IPD may be measured from video information captured by an intercoupled (64) computing system and may also comprise one or more angular or linear motion measurement devices, such as one or more accelerometers, gyroscopes, IMU's, or cameras which may be operatively coupled to the apparatus frame (68) and configured to be utilized to determine rotation/position based upon captured imagery from surroundings (i.e., such as "head pose" determination based upon computer vision techniques). The apparatus frame (68), to which the camera device (62) is fixedly coupled, is removably coupled to the head of the user using a mouth bit interface (66) for the user to bite on such that the user can move and accelerate his head about relatively easily while data is acquired pertaining to his eyes and IPD. Referring to FIG. 5, a chart (72) of sample data from a group of user subjects is illustrated, featuring a plot (76) of diopter error versus head pitch angle; also shown is a polynomial equation mathematically fit through this sample data (74), which may be utilized as an IPD compensating factor (with zero pitch being as shown in FIG. 2; pitch in degrees; -90 being user looking approximately straight down at the floor; +90 being user looking approximately straight up at the ceiling). One can see that in the sample experimental data depicted in FIG. 5, there is a general slight increase in diopter error as the users' heads were pitched from -90, incrementally toward 0, and then up toward +90. The associated IPD compensating factor (74) developed for this sample experimental data, may be utilized as an input to the augmented reality system such that focus is maintained during pitch rotation of the head of a user from this sample, for example.
[0038] An apparatus such as that depicted in FIGS. 4A and 4B, or a virtual or augmented reality system with appropriate componentry, such as that illustrated in FIGS. 1 and 2, may be utilized to not only gain information between the relationship of measured IPD and head pitch angle position, but also linear and/or angular velocity in pitch relative to surroundings, and linear and/or angular acceleration in pitch relative to surroundings. Further, such relationships may be determined for other axes, such as an orthogonal yaw axis, or an orthogonal roll axis. We have experimentally seen variations in eye positioning associated with position, velocity, and acceleration changes about each of these axes.
[0039] Referring to FIG. 6A, a configuration is illustrated without IPD-head-rotation compensation, wherein a user is wearing a calibrated (i.e., with initial input or determination of IPD) augmented reality system (80), for example. As the user gazes at a first target in space (82), the system is configured to generate a visual presentation or portion thereof pertaining to that user's gaze at the first target, based at least in part upon that user's IPD (84). The user may change gaze to a second target (86), and the system may be similarly configured to generate a visual presentation or portion thereof pertaining to that user's gaze at the second target, based at least in part upon, again, that user's IPD (88) (i.e., which has not been compensated for head position or rotation related variables).
[0040] Referring to FIG. 6B, a compensated configuration is illustrated, wherein after initial calibration (80) and gaze at a first target (82), the system is configured to generate visual presentation or portion thereof pertaining to the user's gaze at the first target, based at least in part upon an IPD compensated for head orientation (such as head pitch angle when viewing the first target, the pitch angle being determined by the system). Then if the user changes gaze to a second target (86), the system is configured to generate visual presentation or portion thereof pertaining to the user's gaze at the second target, based at least in part upon an IPD compensated for head orientation (such as head pitch angle when viewing the second target, the pitch angle being determined by the system).
[0041] Referring to FIG. 6C, the system itself may be utilized to develop one or more compensating relationships for the particular user. As shown in FIG. 6C, a user may be wearing calibrated augmented reality system (for example, with IPD determined by system at level head pitch angle, such as that associated with gaze to infinity over substantially level horizon) (100). To determine any variation in user's vision system in actual or functional IPD with various positions, angular or linear velocities, or angular or linear accelerations, a viewing calibration system may guide the user through a series of viewing exercises (i.e., wherein he positions, accelerates his head and wherein the system is configured to capture data pertaining to actual and/or functional IPD) (102). The system may be configured to determine IPD compensation configuration (such as a lookup table or one or more mathematical relationships) for the user that is variable with the various positions, angular or linear velocities, or angular or linear accelerations (104), completing the user's IPD compensation configuration (106). Then when the user gazes at a first target in space (82), the system may be configured to generate visual presentation or portions thereof pertaining to the user's gaze at the first target, based at least in part upon the user's IPD compensation configuration for the position, velocity, and/or acceleration (angular and/or cartesian) of the user's head for the gaze at the first target (108). Then when the user gazes to a second target (86), the system may be configured to generate visual presentation or portions thereof pertaining to the user's gaze at the second target, based at least in part upon the user's IPD compensation configuration for the position, velocity, and/or acceleration (angular and/or cartesian) of the user's head for the gaze at the second target (110).
[0042] FIG. 7 illustrates an augmented reality system 142 of the in more detail. The system 142 includes a stereoscopic analyzer 144 that is connected to the rendering engine 130 and forms part of the vision data and algorithms.
[0043] The system 142 further includes left and right projectors 166A and 166B and left and right waveguides 170A and 170B. The left and right projectors 166A and 166B are connected to power supplies. Each projector 166A and 166B has a respective input for image data to be provided to the respective projector 166A or 166B. The respective projector 166A or 166B, when powered, generates light in two-dimensional patterns and emanates the light therefrom. The left and right waveguides 170A and 170B are positioned to receive light from the left and right projectors 166A and 166B, respectively. The left and right waveguides 170A and 170B are transparent waveguides.
[0044] In use, a user mounts the head mountable frame 140 to their head. Components of the head mountable frame 140 may, for example, include a strap (not shown) that wraps around the back of the head of the user. The left and right waveguides 170A and 170B are then located in front of left and right eyes 220A and 220B of the user.
[0045] The rendering engine 130 enters the image data that it receives into the stereoscopic analyzer 144. The image data is projected onto a plurality of virtual planes. The stereoscopic analyzer 144 analyzes the image data to determine left and right image data sets based on the image data for projection onto each depth plane. The left and right image data sets are data sets that represent two-dimensional images that are projected in three-dimensions to give the user a perception of a depth.
[0046] The stereoscopic analyzer 144 enters the left and right image data sets into the left and right projectors 166A and 166B. The left and right projectors 166A and 166B then create left and right light patterns. The components of the system 142 are shown in plan view, although it should be understood that the left and right patterns are two-dimensional patterns when shown in front elevation view. Each light pattern includes a plurality of pixels. For purposes of illustration, light rays 224A and 226A from two of the pixels are shown leaving the left projector 166A and entering the left waveguide 170A. The light rays 224A and 226A reflect from sides of the left waveguide 170A. It is shown that the light rays 224A and 226A propagate through internal reflection from left to right within the left waveguide 170A, although it should be understood that the light rays 224A and 226A also propagate in a direction into the paper using refractory and reflective systems.
[0047] The light rays 224A and 226A exit the left light waveguide 170A through a pupil 228A and then enter a left eye 220A through a pupil 230A of the left eye 220A. The light rays 224A and 226A then fall on a retina 232A of the left eye 220A. In this manner, the left light pattern falls on the retina 232A of the left eye 220A. The user is given the perception that the pixels that are formed on the retina 232A are pixels 234A and 236A that the user perceives to be at some distance on a side of the left waveguide 170A opposing the left eye 220A. Depth perception is created by manipulating the focal length of the light.
[0048] In a similar manner, the stereoscopic analyzer 144 enters the right image data set into the right projector 166B. The right projector 166B transmits the right light pattern, which is represented by pixels in the form of light rays 224B and 226B. The light rays 224B and 226B reflect within the right waveguide 170B and exit through a pupil 228B. The light rays 224B and 226B then enter through a pupil 230B of the right eye 220B and fall on a retina 232B of a right eye 220B. The pixels of the light rays 224B and 226B are perceived as pixels 134B and 236B behind the right waveguide 170B.
[0049] The patterns that are created on the retinas 232A and 232B are individually perceived as left and right images. The left and right images differ slightly from one another due to the functioning of the stereoscopic analyzer 144. The left and right images are perceived in a mind of the user as a three-dimensional rendering.
[0050] As mentioned, the left and right waveguides 170A and 170B are transparent. Light from a real-life object such as the table 116 on a side of the left and right waveguides 170A and 170B opposing the eyes 220A and 220B can project through the left and right waveguides 170A and 170B and fall on the retinas 232A and 232B.
[0051] FIG. 8 shows further details of the device 142 as it relates to IPD compensation as hereinbefore described. The device further includes an IPD camera 302 that serves as an IPD detector, a world camera 304 and an IMU 306 that detect head motion, correlator 308, a storing system 310, an IPD compensation factor calculator 312, an IPD compensator 314 and a viewing calibration system 316. The correlator 308 is connected to the IPD camera 302, the world camera 304 and the IMU 306. The correlator 308 correlates head movement data from the world camera 304 and the IMU 306 with IPD data from the IPD camera 302. The storing system 310 is connected to the correlator 308 and stores the correlation that is generated by the correlator 308. The IPD compensation factor calculator 312 calculates an IPD compensation factor. The IPD compensator 314 is connected to the IPD compensation factor calculator 312 and the rendering engine 130 is connected to the IPD compensator 314. The IPD compensator 314 modifies the visualization created by the rendering engine 130 based on the IPD compensation factor calculator 312.
[0052] The viewing calibration system 316 prompts the user through a series of vision tests to generate one or more IPD compensation factors such as the IPD compensation factor calculator 312.
[0053] While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the current invention, and that this invention is not restricted to the specific constructions and arrangements shown and described since modifications may occur to those ordinarily skilled in the art.
