Sony Patent | Head mounted display, control method for head mounted display, information processor, display device, and program
Drawings: Click to check drawins
Publication Number: 20210124174
Publication Date: 20210429
Applicant: Sony
Assignee: Sony Corporation
Abstract
The present disclosure relates to an HMD, a control method for the head mounted display, an information processor, a display device, and a program that make it possible to estimate camera parameters of a camera mounted in the HMD without the awareness of a user wearing the HMD. At a timing when the HMD is displaying content including a VR image or an AR image, the camera mounted in the HMD captures an image of a calibration pattern displayed on an external display, executes calibration on the basis of the captured image of the calibration pattern, and estimates the camera parameters. At this time, the calibration pattern displayed on the external display is displayed with dimensions according to a positional relationship between the HMD and the external display. The present disclosure is applicable to the HMD.
Claims
-
A head mounted display comprising: a display unit; an imaging unit that images an outside; and a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
-
The head mounted display according to claim 1, wherein the calibration processing unit generates the calibration image on a basis of a positional relationship between the head mounted display and the external display equipment.
-
The head mounted display according to claim 2, wherein the calibration processing unit generates the calibration image on a basis of positional information of the head mounted display and positional information of the external display equipment.
-
The head mounted display according to claim 3, wherein the calibration processing unit adjusts a size of the calibration image and generates the calibration image in accordance with a distance between the head mounted display and the external display equipment.
-
The head mounted display according to claim 4, wherein the calibration processing unit adjusts the size of the calibration image to a large extent when the distance between the head mounted display and the external display equipment is greater than a predetermined distance.
-
The head mounted display according to claim 2, wherein the calibration processing unit generates the calibration image on a basis of orientation information of the head mounted display and orientation information of the external display equipment.
-
The head mounted display according to claim 6, wherein the calibration processing unit adjusts a size of the calibration image and generates the calibration image in accordance with an angle formed by an imaging direction of the imaging unit and a surface direction of the external display equipment.
-
The head mounted display according to claim 7, wherein the calibration processing unit adjusts the size of the calibration image with respect to a direction in which the formed angle is formed.
-
The head mounted display according to claim 8, wherein when the formed angle is larger than a predetermined angle, the calibration processing unit adjusts the size of the calibration image to a large extent with respect to the direction in which the formed angle is formed.
-
The head mounted display according to claim 2, wherein the calibration processing unit estimates camera parameters of the imaging unit by using the calibration image captured by the imaging unit and displayed by the external display equipment.
-
The head mounted display according to claim 10, wherein in a case where the calibration processing unit estimates the camera parameters on a basis of a plurality of images of the calibration images captured by the imaging unit and displayed by the external display equipment, the calibration processing unit causes the display unit to display an image that guides a user wearing the head mounted display so that imaging is performed from a direction in which imaging has not been performed by the imaging unit among imaging directions of the imaging unit with respect to a surface direction of the external display equipment when the imaging directions of the imaging unit with respect to the surface direction of the external display equipment are imbalanced for the plurality of images.
-
The head mounted display according to claim 11, wherein in a case where the calibration processing unit estimates the camera parameters of each of the imaging units of a plurality of the head mounted displays, the calibration processing unit causes the display unit to display an image that guides the user wearing the head mounted display so that the calibration image displayed by the external display equipment is able to be captured without overlapping of each of the plurality of the imaging units.
-
The head mounted display according to claim 2, wherein the calibration processing unit generates the calibration image on a basis of brightness of the calibration image captured by the imaging unit and displayed by the external display equipment.
-
The head mounted display according to claim 13, wherein the calibration processing unit reduces the brightness of the calibration image captured by the imaging unit and displayed by the external display equipment in a case where there are more than a predetermined number of pixels with saturated brightness in the calibration image displayed by the external display equipment, and increases the brightness of the calibration image captured by the imaging unit and displayed by the external display equipment in a case where there are more than a predetermined number of pixels with the lowest brightness in the calibration image displayed by the external display equipment.
-
The head mounted display according to claim 1, wherein the calibration processing unit generates the calibration image suitable for the calibration and causes the external display equipment to display the calibration image at a timing when the external display equipment is detected in an image captured by the imaging unit, at a timing when a predetermined time has elapsed, at a timing when a discrepancy occurs between an image to be originally imaged by the imaging unit and an image actually imaged by the imaging unit on a basis of positional information of the head mounted display and on a basis of camera parameters of the imaging unit, or at a timing when a user requests the calibration.
-
The head mounted display according to claim 1, wherein the calibration image is an image including a marker in a predetermined shape or an image including a marker colored in a predetermined color arrangement.
-
The head mounted display according to claim 1, wherein the calibration image is an image displayed at a predetermined timing in content including a moving image.
-
A control method for a head mounted display including a display unit and an imaging unit that images an outside, the method comprising a calibration process of generating a calibration image suitable for calibration and causing external display equipment to display the calibration image.
-
A program for causing a computer controlling a head mounted display, which includes a display unit and an imaging unit that images an outside, to function as a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
-
An information processor for supplying content to a head mounted display including a display unit and an imaging unit that images an outside, the information processor comprising a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
-
A display device for displaying a calibration image of an imaging unit that images an outside in a head mounted display including a display unit and the imaging unit, the display device comprising a calibration processing unit that generates the calibration image suitable for calibration and causes the display device to display the calibration image.
Description
TECHNICAL FIELD
[0001] The present disclosure relates to a head mounted display, a control method for the head mounted display, an information processor, a display device, and a program, and relates to a head mounted display, a control method for the head mounted display, an information processor, a display device, and a program that make it possible to perform calibration without the awareness of a user and appropriately estimate camera parameters especially in a home environment.
BACKGROUND ART
[0002] In a setup including a camera, distortion of an optical system, deviation in an imaging direction, and the like occur, and parameters for dealing with the distortion and deviation are measured (estimated) in advance, whereby an appropriate image can be obtained by application of the parameters to a captured image.
[0003] The existing methods of parameter estimation in this setup often assume only one initial measurement. Among those, a known pattern on a display or the like is imaged by a camera, and parameters are estimated on the basis of the imaging result.
[0004] For example, there has been proposed a technology in which a known pattern is displayed on a display device, and a captured image obtained by imaging the displayed known pattern is used to achieve camera calibration (see Patent Document 1.).
[0005] However, when a deformation or change occurs in the setup thereafter, calibration is required again in those methods.
[0006] That is, in a case where calibration is performed in a factory or the like, when a camera setup is deformed due to an impact or the like in a home environment such as a user’s home after shipment, re-shipment to the factory is necessary in order to perform the calibration again. To avoid such a situation, it is necessary for the user to perform calibration in a home environment (in a home-based environment) so that these parameters can be estimated again.
CITATION LIST
Patent Document
Patent Document 1: Japanese Patent Application Laid-Open No. 2000-350239
SUMMARY OF THE INVENTION
Problems to be Solved by the Invention
[0007] However, some existing methods assuming the home environment have required special objects for parameter estimation to be prepared separately. In addition, with many methods requiring a process including special procedures, there has been a possibility that the user dislikes executing these processes and does not perform calibration itself.
[0008] Further, in Patent Document 1, there has been a possibility that a change in the imaged known pattern may not be handled due to a situation such as a positional relationship between the user and the display, and the camera parameters may not be able to be estimated appropriately.
[0009] The present disclosure has been made in view of such circumstances, and calibration is performed without the awareness of the user, and camera parameters are estimated appropriately especially in a home environment.
Solutions to Problems
[0010] A head mounted display of one aspect of the present disclosure is a head mounted display including: a display unit; an imaging unit that images an outside; and a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
[0011] A control method for a head mounted display of one aspect of the present disclosure is a control method for a head mounted display including a display unit and an imaging unit that images an outside, the method including a calibration process of generating a calibration image suitable for calibration and causing external display equipment to display the calibration image.
[0012] A program of one aspect of the present disclosure is a program for causing a computer controlling a head mounted display, which includes a display unit and an imaging unit that images an outside, to function as a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
[0013] An information processor of one aspect of the present disclosure is an information processor for supplying content to a head mounted display including a display unit and an imaging unit that images an outside, the information processor including a calibration processing unit that generates a calibration image suitable for calibration and causes external display equipment to display the calibration image.
[0014] A display device of one aspect of the present disclosure is a display device for displaying a calibration image of an imaging unit that images an outside in a head mounted display including a display unit and the imaging unit, the display device including a calibration processing unit that generates a calibration image suitable for calibration and causes the display device to display the calibration image.
[0015] In one aspect of the present disclosure, a calibration image suitable for calibration is generated and displayed on external display equipment.
Effects of the Invention
[0016] According to one aspect of the present disclosure, calibration can be performed without the awareness of the user, and camera parameters can be appropriately estimated especially in a home environment.
BRIEF DESCRIPTION OF DRAWINGS
[0017] FIG. 1 is an exemplary view for explaining a configuration example of a calibration system of the present disclosure.
[0018] FIG. 2 is a view for explaining an overview of a calibration process according to a first embodiment of the present disclosure.
[0019] FIG. 3 is a view for explaining an example of display of a calibration pattern according to the distance between an HMD and a display device.
[0020] FIG. 4 is a view for explaining an example of display of a calibration pattern according to a positional relationship between the HMD and the display device.
[0021] FIG. 5 is a diagram for explaining a configuration example of the HMD of the first embodiment.
[0022] FIG. 6 is a diagram for explaining a configuration example of an information processor of the first embodiment.
[0023] FIG. 7 is a diagram for explaining a configuration example of the display device of the first embodiment.
[0024] FIG. 8 is a flowchart for explaining the calibration process of the first embodiment.
[0025] FIG. 9 is a flowchart for explaining a calibration pattern display state confirmation process according to the first embodiment.
[0026] FIG. 10 is a view for explaining an overview of a first modification.
[0027] FIG. 11 is a flowchart for explaining a calibration pattern display state confirmation process according to the first modification.
[0028] FIG. 12 is a view for explaining an overview of a second modification.
[0029] FIG. 13 is a flowchart for explaining a calibration process of the second modification.
[0030] FIG. 14 is a view for explaining an overview of a third modification.
[0031] FIG. 15 is a flowchart for explaining a calibration pattern display state confirmation process according to the third modification.
[0032] FIG. 16 is a diagram for explaining a configuration example of an information processor according to a second embodiment.
[0033] FIG. 17 is a flowchart for explaining a calibration process of the second embodiment.
[0034] FIG. 18 is a diagram for explaining a configuration example of a display device of a third embodiment.
[0035] FIG. 19 is a flowchart for explaining a calibration process of the third embodiment.
[0036] FIG. 20 is a diagram for explaining a configuration example of a general-purpose personal computer.
MODE FOR CARRYING OUT THE INVENTION
[0037] Preferred embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that in the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions thereof will be omitted.
[0038] Modes for carrying out the present technology will be described below. The description will be performed in the following order:
[0039] 1. Overview of calibration system of present disclosure
[0040] 2. First Embodiment
[0041] 3. First modification
[0042] 4. Second modification
[0043] 5. Third modification
[0044] 6. Second Embodiment
[0045] 7. Third Embodiment
[0046] 8. Example of execution by software
-
Overview of Calibration System of Present Disclosure
[0047]
[0048] Referring to FIG. 1, an overview of a calibration system to which the technology of the present disclosure is applied will be described.
[0049] FIG. 1 shows a configuration example of a calibration system of the present disclosure. A calibration system 11 shown in FIG. 1 is a system made up of a head mounted display (HMD) 31, an information processor 32, and a display device 33 including a display, in which camera parameters of a camera 31a provided in the HMD 31 is estimated (measured) by calibration.
[0050] The calibration system 11 of FIG. 1 functions as a device for displaying content, such as a game machine, for example, in normal operation More specifically, the information processor 32 outputs content including a virtual reality (VR) image or an augmented reality (AR) image to the HMD 31 worn by the user 21 for display.
[0051] Further, the calibration system 11 performs calibration of the camera 31a mounted in the HMD 31 without the awareness of the user, while the user views an image displayed on the mounted HMD 31, for example, while the user enjoys a game, and the calibration system 11 estimates (measures) the camera parameters of the camera 31a.
[0052] That is, the calibration system 11 executes a calibration process in the background while achieving the original function as a game machine.
[0053] More specifically, for example, as shown by a state St1 in FIG. 2, at the time of executing the calibration process, the calibration system 11 displays, for example, an image of a plurality of square-shaped markers or an image of a checker flag pattern as a calibration pattern which is for performing calibration on the display device 33 and in which where to display at what position on the display device 33 is known. Note that the shape of the marker may be a predetermined shape except for a square, or may be a known color arrangement pattern, for example, as long as being a known shape.
[0054] At this time, the camera 31a of the HMD 31 captures an image of the calibration pattern displayed on the display device 33, and the HMD 31 estimates (measures) the camera parameters of the camera 31a on the basis of the image of the calibration pattern captured by the camera 31a.
[0055] Also, at the time of executing the calibration process, the HMD 31 of the calibration system 11 simultaneously displays and presents, to the user 21, a VR image as shown by a state St2 in FIG. 2 and an AR image as shown by a state St3 in FIG. 2.
[0056] Thus, the calibration system 11 executes the calibration process in the background while functioning as a game machine and estimates (measures) the camera parameters of the camera 31a.
[0057] The camera parameters to be obtained here are, for example, the information of the position and orientation of the setup of the camera 31a starting from the display device 33, information of internal parameters (distortion, focal length, optical center, etc.) of the camera 31a, the information of the position and orientation between the cameras 31a in a case where there is a plurality of cameras 31a, color information including the white balance of the camera 31a, and the like. However, in a case where the camera parameters of the color information including the white balance are estimated (measured), the calibration pattern needs to be, for example, an RGB image.
[0058] As a result, while the user 21 wears the HMD 31 and enjoys the game using the information processor 32, the calibration can be performed with the user 21 being unaware of the calibration, and the camera parameters can be estimated (measured) appropriately.
[0059] Note that as shown by the states St2 and St3 in FIG. 2, a state in which a VR image is displayed or a state in which an AR image is displayed shows the image of an image to be viewed by the user wearing the HMD 31 and is different from the actual display state.
[0060]
[0061] The camera 31a provided in the HMD 31 images the front direction of the face of the user 21 wearing the HMD 31, for example. Therefore, as shown in FIG. 1, when the user 21 turns the face to the direction of the display device 33 with the HMD 31 worn on the head of the user 21, the display device 33 is imaged.
[0062] Further, a display (not shown) is provided in the HMD 31 so as to cover the eyes of the user 21 when the HMD 31 is put on the head of the user 21, and a VR image or an AR image as content of a game or the like is displayed on the display.
[0063] With such a configuration, the operation in which the user 21 views the VR image or the AR image displayed on the HMD 31 and the operation in which the camera 31a images the front direction of the user 21 are completely independent of each other, but the viewing direction and the imaging direction change in conjunction with the operation of the user 21.
[0064] Furthermore, with the configuration described above, the user 21 may not be able to recognize the positional relationship between the viewing direction of the user 21 and the display device 33 while wearing the HMD 31, and hence the positional relationship between the camera 31a and the display device 33 is assumed to be in various states.
[0065] For example, in a case where the front direction of the face of the user 21 wearing the HMD 31 and the display surface of the display device 33 face each other, the calibration pattern displayed on the display device 33 imaged by the camera 31a varies between when the HMD 31 and the display device 33 are present at a position closer to each other than a predetermined distance and when the HMD 31 and the display device 33 are present at a position farther from each other than the predetermined distance.
[0066] That is, in a case where the front direction of the face of the user 21 wearing the HMD 31 and the front direction of the display surface of the display device 33 face each other and when the HMD 31 and the display device 33 are present at the predetermined distance from each other, the dimensions of the marker constituting the calibration pattern imaged by the camera 31a are an appropriate size, and the boundaries of a plurality of markers can be recognized appropriately.
[0067] Note that although the calibration pattern will be described as an image in which a plurality of square-shaped markers is arranged, the calibration pattern may be another image and may, for example, be an image of a checker flag pattern in which a plurality of square-shaped markers of white and black is arranged alternately in a densely packed state.
[0068] In contrast, in a case where the front direction of the face of the user 21 wearing the HMD 31 and the front direction of the display surface of the display device 33 face each other and the HMD 31 and the display device 33 are present at positions farther from each other than the predetermined distance, the dimensions of the marker constituting the calibration pattern imaged by the camera 31a become smaller than when the HMD 31 and the display device 33 are present at positions at the predetermined distance from each other, and there is a possibility that the position of the marker cannot be appropriately recognized because, for example, the boundary of the marker is collapsed, or the like.
[0069] When the position of the marker (the position of the boundary, especially in the case of the square-shaped marker) comes into a state not appropriately recognizable, there is a possibility that appropriate camera parameters may not be able to be estimated (measured) by calibration.
[0070] Therefore, in the calibration system 11 of FIG. 1, the size of the marker constituting the calibration pattern displayed on the display device 33 is changed and displayed in accordance with the distance between the HMD 31 and the display device 33.
[0071] That is, as shown in the upper portion of FIG. 3, in a case where the distance between the HMD 31 and the display device 33 is the predetermined distance D1, the image of the display device 33 captured by the camera 31a becomes, for example, an image P1. In the case of the image P1, it is assumed that the square-shaped markers constituting the calibration pattern are arranged in an array, and the horizontal size of the markers is a default size d and appropriate for the estimation (measurement) of the camera parameters.
[0072] In contrast, in a case where the distance between the HMD 31 and the display device 33 is greater than the predetermined distance, there is a possibility that the horizontal size of the marker constituting the calibration pattern may become smaller than the size d and may become an inappropriate size for estimating (measuring) the camera parameters.
[0073] Therefore, in the calibration system 11 of FIG. 1, the size of the marker constituting the calibration pattern is changed in accordance with the distance between the HMD 31 and the display device 33. That is, for example, as shown in the lower portion of FIG. 3, in a case where the distance between the HMD 31 and the display device 33 is a distance D2 that is greater than the predetermined distance D1, an image captured by the camera 31a is displayed with a larger horizontal size of the marker constituting the calibration pattern so as to be, for example, an image P2.
[0074] In this way, the size d of the marker constituting the calibration pattern in the image P2 becomes a similar size d to the size when the distance between the HMD 31 and the display device 33 is the distance D1, as shown in the upper portion of FIG. 3, and can be set to a size appropriate for estimating (measuring) the camera parameters.
[0075] Thus, the size of the marker constituting the calibration pattern can be made an appropriate size regardless of the distance between the HMD 31 and the display device 33, so that the calibration pattern of an appropriate size can be displayed regardless of the distance between the HMD 31 and the display device 33.
[0076] As a result, the camera parameters can be estimated (measured) appropriately.
[0077]
[0078] Further, for example, in a case where the front direction of the face of the user 21 wearing the HMD 31 and the front direction of the display surface of the display device 33 do not face each other, and, for example, the camera 31a of the HMD 31 images the display device 33 from either the right or left oblique directions with respect to the front direction of the display device 33, a change occurs in the calibration pattern imaged by the camera 31a and displayed on the display device 33 in accordance with the positional relationship between the imaging direction of the HMD 31 and the display device 33.
[0079] That is, in a case where the display device 33 is imaged from either the right or left oblique direction as the front direction of the face of the user 21 wearing the HMD 31 images the display device 33 with respect to the front direction of the display surface of the display device 33, the horizontal size of the marker constituting the calibration pattern is smaller than in a case where the imaging is performed from the front direction.
[0080] As a result, the horizontal dimension of the marker constituting the calibration pattern imaged by the camera 31a becomes smaller, and it is conceivable that the position of the marker cannot be appropriately recognized because, for example, the boundary of the marker of the imaged calibration pattern is collapsed, or the like.
[0081] When the position of the marker cannot be appropriately recognized, there is a possibility that appropriate camera parameters may not be able to be obtained by the calibration process.
[0082] Therefore, in the calibration system 11 shown in FIG. 1, the horizontal size of the marker constituting the calibration pattern displayed on the display device 33 is changed in accordance with the positional relationship between the front direction of the face of the user 21 wearing the HMD 31 and the front direction of the display surface of the display device 33.
[0083] That is, as shown by an image Pt11 in the left portion of FIG. 4, in a case where the calibration pattern displayed on the display device 33 is an image of a checker flag pattern, when the image is captured in a state where the front direction of the face of the user 21 wearing the HMD 31 coincides with the front direction of the display device 33, the white or black square part of the calibration pattern of the checker flag pattern functions as a marker, and the image is captured in a state where the markers are arranged regularly. Therefore, in the left portion of FIG. 4, the calibration can be appropriately performed, and as a result, the camera parameters can be obtained appropriately.
[0084] However, when a calibration pattern displayed on the display device 33 is imaged by the camera 31a, for example, from a right oblique direction with respect to the front direction of the display device 33, an image as shown by an image Pt11’ in the center of FIG. 4 is imaged, for example.
[0085] That is, as shown by the image Pt11’, there is a possibility that the horizontal size of the marker in the checker flag pattern of the imaged calibration pattern is smaller than the horizontal size of the marker in the image Pt11 when imaged from the front direction, and may not be an appropriate marker size.
[0086] Therefore, in the calibration system 11 of FIG. 1, in a case where the imaging is performed with the imaging direction of the HMD 31 being from the right oblique direction with respect to the front direction of the display device 33, as shown by an image Pt12 in the right portion of FIG. 4, the horizontal size of the marker constituting the calibration pattern is increased, and the calibration pattern is displayed on the display device 33 such that the horizontal size of the marker becomes appropriate in the calibration process.
[0087] Thus, the size of the marker constituting the calibration pattern can be made an appropriate size regardless of the positional relationship between the imaging direction of the HMD 31 and the display device 33. As a result, the calibration can be appropriately performed regardless of the positional relationship between the imaging direction of the camera 31a in the HMD 31 and the display device 33, and the camera parameters can be (measured) estimated appropriately.
-
First Embodiment
[0088]
[0089] Next, a configuration example of the HMD 31 constituting the calibration system 11 of FIG. 1 will be described with reference to FIG. 5.
[0090] The head mounted display (HMD) 31 is worn so as to be wound around the head of a user 21, and a display unit 54 provided so as to cover the right and left eyes of the user 21 displays content including a VR image or an AR image supplied from the information processor 32 and causes the user 21 to view the content.
……
……
……
