空 挡 广 告 位 | 空 挡 广 告 位

Sony Patent | Information processing apparatus, information processing method, and program

小编 映维   |   分类：Sony   |  

Patent: Information processing apparatus, information processing method, and program

Drawings: Click to check drawins

Publication Number: 20210005167

Publication Date: 20210107

Applicant: Sony

Abstract

To improve the quality of rendering or display of a virtual image based on a sensing result relating to a real space. Provided is an information processing apparatus including a rendering control unit configured to switch a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space, and a display control unit configured to cause a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

Claims

  1. An information processing apparatus comprising: a rendering control unit configured to switch a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space; and a display control unit configured to cause a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

  2. The information processing apparatus according to claim 1, wherein the rendering control unit controls the rendering frequency of the virtual image to the first rendering frequency when the virtual image based on the sensing result has a first change amount, and the rendering control unit controls the rendering frequency of the virtual image to the second rendering frequency when the virtual image has a second change amount larger than the first change amount.

  3. The information processing apparatus according to claim 2, wherein re-rendering of the virtual image is prohibited at the first rendering frequency.

  4. The information processing apparatus according to claim 2, further comprising a correction unit configured to correct a display position of the virtual image, wherein the correction unit corrects the display position of the virtual image while the rendering control unit reduces re-rendering of the virtual image when the virtual image has the first change amount.

  5. The information processing apparatus according to claim 4, wherein the correction unit two-dimensionally corrects the display position of the virtual image rendered by the rendering control unit on the basis of position and attitude relating to the display device.

  6. The information processing apparatus according to claim 2, wherein the virtual image is re-rendered at the second rendering frequency.

  7. The information processing apparatus according to claim 2, wherein the first change amount is a change amount smaller than a predetermined threshold, the second change amount is a change amount larger than the predetermined threshold, and the rendering control unit changes the predetermined threshold to switch the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency.

  8. The information processing apparatus according to claim 1, wherein the rendering control unit controls a frame rate to switch the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency.

  9. The information processing apparatus according to claim 1, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of a positional relationship between a user of the display device, the user visually recognizing the virtual image, and the virtual image.

  10. The information processing apparatus according to claim 9, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of a distance between the user and the virtual image.

  11. The information processing apparatus according to claim 10, wherein the rendering control unit controls the rendering frequency of the virtual image to the first rendering frequency when the distance between the user and the virtual image is a first distance, and controls the rendering frequency of the virtual image to the second rendering frequency when the distance between the user and the virtual image is a second distance smaller than the first distance.

  12. The information processing apparatus according to claim 11, wherein the rendering control unit reduces the rendering frequency of the virtual image as the distance between the user and the virtual image becomes larger.

  13. The information processing apparatus according to claim 1, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of an accuracy of the sensing result relating to the real space.

  14. The information processing apparatus according to claim 13, wherein the accuracy of the sensing result relating to the real space includes a recognition accuracy of position and attitude relating to the display device, and the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of the recognition accuracy.

  15. The information processing apparatus according to claim 14, wherein the rendering control unit controls the rendering frequency of the virtual image to the first rendering frequency when the recognition accuracy is a first recognition accuracy, and controls the rendering frequency of the virtual image to the second rendering frequency when the recognition accuracy is a second recognition accuracy higher than the first recognition accuracy.

  16. The information processing apparatus according to claim 1, wherein a characteristic or a state of the virtual image is determined on the basis of the sensing result relating to the real space, and the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of the characteristic or the state of the virtual image.

  17. The information processing apparatus according to claim 16, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of at least one of complexity of a shape, pattern, or color, a size, or a state of an operation of the virtual image.

  18. The information processing apparatus according to claim 1, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of a resolution of the display device.

  19. The information processing apparatus according to claim 1, wherein the rendering control unit switches the rendering frequency of the virtual image between the first rendering frequency and the second rendering frequency on the basis of a positional relationship between a line of sight of a user of the display device and the virtual image.

  20. The information processing apparatus according to claim 1, wherein the rendering control unit makes the first rendering frequency lower than a frame rate of the virtual image displayed by the display device.

  21. An information processing method executed by at least one processor, the method comprising: switching a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space; and causing a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

  22. A program for causing a computer to execute: switching a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space; and causing a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

Description

FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

BACKGROUND

[0002] In recent years, with the advancement of information processing technologies, technologies that display virtual images, such as virtual reality (VR) and augmented reality (AR), have been used in various apparatuses, systems, and services.

[0003] For example, Patent Literature 1 discloses a technology that enables a distant limit of a rendering range to be set to the farthest possible limit by changing the frequency of updating scenery with a virtual image attached according to, for example, the distance between a user and the virtual image or a moving speed of the user in a virtual space.

CITATION LIST

Patent Literature

[0004] Patent Literature 1: JP 2013-214162** A**

SUMMARY

Technical Problem

[0005] In prior art including Patent Literature 1, the rendering of a virtual image based on a sensing result relating to a real space is not sufficiently taken into consideration. Thus, there is room for improvement in the quality of the rendering or display of a virtual image.

[0006] The present disclosure has been made in view of the above, and provides a new and improved information processing apparatus, a new and improved information processing method, and a new and improved program that enable improvement in the quality of rendering or display of a virtual image based on a sensing result relating to a real space.

Solution to Problem

[0007] According to the present disclosure, an information processing apparatus is provided that includes: a rendering control unit and a display control unit. The rendering control unit switches a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space. The display control unit causes a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

[0008] Moreover, according to the present disclosure, an information processing method executed by at least one processor is provided that includes: switching a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space; and causing a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

[0009] Moreover, according to the present disclosure, a program is provided that causes a computer to execute: switching a rendering frequency of a virtual image between a first rendering frequency and a second rendering frequency higher than the first rendering frequency on the basis of a sensing result relating to a real space; and causing a display device to display the virtual image on the basis of the sensing result and either the first rendering frequency or the second rendering frequency.

Advantageous Effects of Invention

[0010] As described above, the present disclosure enables improvement in the quality of rendering or display of a virtual image based on a sensing result relating to a real space.

[0011] Note that the effects of the present disclosure are not necessarily limited to the above effect. The present disclosure may achieve, in addition to or instead of the above effect, any effect described in the specification or another effect that can be grasped from the specification.

BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 is a diagram describing an outline of the operation of an HMD displaying a virtual image.

[0013] FIG. 2 is a diagram describing 2D correction.

[0014] FIG. 3 is a diagram describing the difference between the 2D correction and 3D rendering.

[0015] FIG. 4 is a diagram describing parallax in a case where the distance between a user and a virtual image is small.

[0016] FIG. 5 is a diagram describing the parallax in the case where the distance between the user and the virtual image is small.

[0017] FIG. 6 is a diagram describing parallax in a case where the distance between a user and a virtual image is large.

[0018] FIG. 7 is a diagram describing the parallax in the case where the distance between the user and the virtual image is large.

[0019] FIG. 8 is a diagram illustrating a 2D correction threshold according to the distance between the user and the virtual image.

[0020] FIG. 9 is a block diagram illustrating a functional configuration of an HMD according to a first embodiment.

[0021] FIG. 10 is a flowchart illustrating the operation of the HMD according to the first embodiment.

[0022] FIG. 11 is a diagram illustrating an application example of the first embodiment.

[0023] FIG. 12 is a diagram illustrating an application example of the first embodiment.

[0024] FIG. 13 is a block diagram illustrating a functional configuration of an HMD according to a second embodiment.

[0025] FIG. 14 is a flowchart illustrating the operation of the HMD according to the second embodiment.

[0026] FIG. 15 is a diagram illustrating an application example of the second embodiment.

[0027] FIG. 16 is a block diagram illustrating a hardware configuration of the HMD.

DESCRIPTION OF EMBODIMENTS

[0028] Hereinbelow, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in the specification and drawings, elements having substantially the same functional configuration are designated by the same reference sign to omit redundant description.

[0029] Note that the description will be made in the following order. [0030] 1. Background [0031] 2. Outline [0032] 3. First Embodiment [0033] 3-1. Functional Configuration [0034] 3-2. Operation [0035] 3-3. Application Example [0036] 4. Second Embodiment [0037] 4-1. Functional Configuration [0038] 4-2. Operation [0039] 4-3. Application Example [0040] 5. Hardware Configuration [0041] 6. Conclusion

  1. Background

[0042] First, the background of the present disclosure will be described.

[0043] As described above, in recent years, with the advancement of information processing technologies, technologies that enable various user experiences by displaying virtual images, such as AR and VR, have been developed actively. Typically, VR can be regarded as a technology that displays an image in a space (virtual space) representing a location different from a location of a user in a real space while changing the image according to changes in the position and attitude of a display device. In this case, the image in the virtual space can include an image in a real space at a distant location. On the other hand, AR can be regarded as a technology that adds additional information such as a virtual image to an image in a real space representing a location substantially the same as a location of a user present in the real space. In this case, the image in the real space includes not only an image in the real space directly visually recognized optically by the user, but also an image in the real space acquired by an imaging device in real time.

[0044] For example, there has been developed a technology that changes a virtual image to be displayed on a head-mounted display (hereinbelow, referred to as “HMD”) in conjunction with the position and attitude or movement of a user wearing the HMD. More specifically, the HMD performs a process for estimating the position and attitude of the HMD using a sensor (e.g., an outward stereo camera, an acceleration sensor, or a gyro sensor) mounted on the HMD and predicting the position and attitude of the HMD after an elapse of a certain time. Further, the HMD renders a virtual image according to a result of the process and displays the virtual image on a display. For example, the HMD renders the virtual image at a certain frame rate (e.g., 60 [Hz]) in conjunction with the movement of a user so that the virtual image can be displayed following changes in the user’s view.

[0045] At this time, the virtual image displayed on the HMD may flicker. The flicker is caused by various factors. For example, rendering the virtual image on the basis of a sensing result relating to a real space as describe below may cause the flicker.

[0046] For example, the flicker of the virtual image may be caused by a slight change in the position and attitude of the HMD. More specifically, when the position and attitude of the HMD slightly changes, the HMD renders a virtual image with a slight change at high speed (at the frame rate frequency) reflecting the change in the position and attitude of the HMD, which many result in flicker of the virtual image. Further, the HMD may render a virtual image with lighting (indicating processing for showing the image as if light is applied to the image from a virtual light source disposed in a three-dimensional space) applied. In this case, even when a change in the position and attitude of the HMD is slight, a change in the virtual image caused by the change becomes larger. Thus, the flicker of the virtual image becomes larger. Further, the slight change in the position and attitude of the HMD causes a larger change in a three-dimensional virtual image than in a two-dimensional virtual image. Thus, the flicker of the virtual image becomes larger in the three-dimensional virtual image even when no lighting is applied.

[0047] Note that various factors cause the slight change in the position and attitude of the HMD. For example, in estimation or prediction of the position and attitude by the HMD using a result of imaging by the outward stereo camera which is mounted on the HMD, the accuracy of a result of the estimation or prediction of the position and attitude by the HMD is reduced when the imaging environment is dark or when there are many flat areas with small change in the captured image. Thus, even when the position and attitude of the HMD hardly changes actually, the estimation result or the like of the position and attitude may slightly change.

[0048] Further, when the HMD corrects the position and attitude of the HMD using output of various sensors (e.g., the gyro sensor and the acceleration sensor) mounted on the HMD, accumulation of calculation errors of the sensors with an elapse of time may increase an error in the estimation result or the like of the position and attitude. Further, an output error of the various sensors may increase as the temperature of the sensors themselves and the peripheral temperature become higher, which may increase the error in the estimation result or the like of the position and attitude. Further, the various sensors operate independently from each other. Thus, the error of the estimation result or the like of the position and attitude may further increase by the above errors of the various sensors acting on each other.

[0049] Further, for example, when the virtual image is standing still (or when the virtual image is moving at low speed), when the virtual image has a complicated shape, pattern, or color, or when the distance between the user and the virtual image is small, the flicker of the virtual image becomes larger. For example, when the user observes a virtual image of an art object with complicated processing applied near the virtual image, the user more sensitively feels flicker of the virtual image.

[0050] Note that, in addition to the slight change in the position and attitude of the HMD, the flicker of the virtual image becomes larger (or the user is more likely to recognize the flicker), for example, when the resolution of the display is low. In particular, a wearable device, such as an HMD, may have a reduced resolution of the display in order to reduce power consumption or heat generation. In this case, the user may recognize flicker of the virtual image.

[0051] The inventor of the present disclosure has created the technology according to the present disclosure in view of the above circumstances. Hereinbelow, an embodiment of the present disclosure will be described in detail.

  1. Outline

[0052] The background of the present disclosure has been described above. Next, the outline of the present disclosure will be described.

[0053] The present disclosure is applicable to various apparatuses or systems. Further, the present disclosure may be applied to any apparatus or system capable of displaying some image. For example, the present disclosure may be applied to an apparatus or system capable of implementing AR or VR. Hereinbelow, as an example, a case where the present disclosure is applied to an optical transmission type HMD (hereinbelow, referred to as “HMD 100”) which is an information processing apparatus configured to display an AR image will be described.

[0054] Here, the HMD 100 to be used may be of any type and any shape. For example, the HMD 100 may be of a head band type (a type attached to the head with a band worn around the entire circumference of the head or a band worn around not only the side of the head, but also the crown of the head) or a helmet type (a visor part of the helmet corresponds to a display).

[0055] The HMD 100 is provided with various sensors including an outward stereo camera which captures an image in the direction of the line of sight of the user, or the outward direction when the HMD 100 is worn, an inward stereo camera which captures an image of the eyes of the user when the HMD 100 is worn, an acceleration sensor, and a gyro sensor. Note that the sensors mounted on the HMD 100 are not limited to these sensors.

[0056] Hereinbelow, an outline of a series of operations of the HMD 100 displaying a virtual image will be described with reference to FIG. 1. First, in “SENSING DATA ACQUISITION” in FIG. 1, the HMD 100 acquires sensing data output by the various sensors including the outward stereo camera. In “POSITION AND ATTITUDE ESTIMATION”, the HMD 100 estimates the position and attitude of the HMD 100 on the basis of the sensing data. In “POSITION AND ATTITUDE PREDICTION (FOR RENDERING)”, the HMD 100 predicts the position and attitude of the HMD 100 when a virtual image is displayed on a display unit 150 (e.g., a display) on the basis of the estimated position and attitude of the HMD 100. Note that the HMD 100 may predict not only the position and attitude of the HMD 100, but also, for example, the position and attitude of an object in a real space visually recognized by the user by analyzing an image captured by the outward stereo camera.

[0057] In “RENDERING”, the HMD 100 renders a virtual image on a frame buffer 134 on the basis of the prediction of the position and attitude. In “POSITION AND ATTITUDE PREDICTION (FOR DISPLAY POSITION CORRECTION)”, the HMD 100 predicts the position and attitude of the HMD 100 at a time point when the virtual image is displayed on the display unit 150. The prediction is performed at a time point closest possible to the time point when the display on the display unit 150 is performed. In “DISPLAY POSITION CHANGE AMOUNT CALCULATION”, the HMD 100 calculates a change amount of the display position of the virtual image on the basis of the prediction result of the position and attitude. In “DISPLAY POSITION CORRECTION”, the HMD 100 corrects the display position of the virtual image on the basis of the change amount of the display position. In “DISPLAY UNIT LIGHTING”, the HMD 100 displays the virtual image with the corrected display position on the display unit 150. Note that the processing details illustrated in FIG. 1 are merely an example, and can be appropriately modified. For example, not only the correction of the display position of the virtual image, but also correction for deforming the virtual image may be performed.

[0058] Hereinbelow, “DISPLAY POSITION CORRECTION” will be described in more detailed with reference to FIG. 2. When a virtual image 10a is rendered on the frame buffer 134, and the HMD 100 corrects the display position of the virtual image 10a to the upper left of the display unit 150 on the basis of the prediction result of the position and attitude of the HMD 100, the HMD 100 changes a reading start position to the lower right of the frame buffer 134 as illustrated in FIG. 2. This enables the HMD 100 to correct the display position of a virtual image 10b displayed on the display unit 150 to the upper left of the display unit 150. Hereinbelow, such two-dimensional correction of the virtual image is referred to as “2D correction”.

[0059] In this manner, the HMD 100 predicts the position and attitude of the HMD 100 at the time point of the display on the display unit 150 at the time point closest possible to the time point when the display is performed and performs the 2D correction on the basis of the prediction result. Accordingly, a display delay of the virtual image felt by the user can be reduced.

[0060] Further, the 2D correction can reduce flicker of the virtual image compared to re-rendering (hereinbelow, referred to as “3D rendering”) of the three-dimensional virtual image. Hereinbelow, the difference between the 2D correction and the 3D rendering will be described with reference to FIG. 3. 3a illustrates the 2D correction of the virtual image which has been described with reference to FIGS. 2, and 3b illustrates an example in which the display position of the virtual image is changed to the upper left of the display unit 150 by the 3D rendering of the virtual image.

[0061] As illustrated in FIG. 3, the comparison between the virtual image 10b with the 2D correction applied and a virtual image 10d with the 3D rendering applied shows that the virtual image 10d with the 3D rendering applied has a larger change amount. Thus, in the case where the 3D rendering is applied, the change amount of the virtual image based on a slight change in the position and attitude of the HMD 100 becomes larger than that in the case where the 2D correction is applied, and the virtual image is thus more likely to flicker.

[0062] Thus, the HMD 100 according to the present disclosure can reduce flicker of the virtual image by controlling switching between a plurality of rendering frequencies in the rendering process of the virtual image. Note that the image frequencies can include at least a first rendering frequency and a second rendering frequency which is higher than the first rendering frequency. Here, it is only required that the first rendering frequency be a frequency that reduces re-rendering of the virtual image, and the first frequency may be a value larger than zero. For example, the HMD 100 can reduce flicker of the virtual image by setting a threshold of the change amount for which only the 2D correction is performed without performing the 3D rendering (hereinbelow, referred to as “2D correction threshold”) and controlling the threshold when the display position of the virtual image changes on the display unit 150. More specifically, when the change amount of the display position of the virtual image is smaller than the 2D correction threshold (when the virtual image has a first change amount), the HMD 100 performs only the 2D correction on the virtual image without performing the 3D rendering to correct the display position of the virtual image. In other words, when the change amount of the display position of the virtual image is smaller than the 2D correction threshold, the HMD 100 may control the rendering frequency to zero and prohibit the re-rendering. On the other hand, when the change amount of the display position of the virtual image is equal to or larger than the 2D correction threshold (when the virtual image has a second change amount), the HMD 100 performs not only the 2D correction, but also the 3D rendering to change the display position and angle of the virtual image. Note that, in this manner, the HMD 100 can reduce flicker of the virtual image by setting the 2D correction threshold compared to the case where the display of the virtual image is constantly controlled by the 3D rendering.

[0063] Here, a method for controlling the 2D correction threshold is not particularly limited to any method. For example, the HMD 100 may control the 2D correction threshold on the basis of the distance between the user and the virtual image to reduce flicker of the virtual image. Hereinbelow, the principle of the method for controlling the 2D correction threshold based on the distance between the user and the virtual image will be described with reference to FIGS. 4 to 8.

[0064] First, a case where the distance between the user and the virtual image is small will be described with reference to FIGS. 4 and 5. As illustrated in FIG. 4, in the case where the distance between a user 20 and a virtual image 10 is relatively small, and the user 20 is located at the front of the virtual image 10 (the position of a user 20a in FIG. 4), the virtual image 10 (indicated as a virtual image 10a in FIG. 5) is displayed on the center of the display unit 150 as illustrated in 5a of FIG. 5.

[0065] When the user moves to a right position (the position of a user 20b in FIG. 4) from the front of the virtual image 10, the display position of the virtual image 10 changes to the left on the display unit 150 as illustrated in 5b (indicated as a virtual image 10b in FIG. 5). The parallax in the case where the distance between the user and the virtual image is small is larger than that in the case where the distance is large. Thus, the user is highly likely to feel something strange as illustrated in 5b unless the display angle of the virtual image 10 is also changed in addition to the display position thereof.

[0066] On the other hand, when the user moves to a left position (the position of a user 20c in FIG. 4) from the front of the virtual image 10, the display position of the virtual image 10 changes to the right on the display unit 150 as illustrated in 5c (indicated as a virtual image 10c in FIG. 5). Also in this case, the user is highly likely to feel something strange as illustrated in 5c unless the display angle of the virtual image 10 is also changed in addition to the display position thereof.

[0067] Next, a case where the distance between the user and the virtual image is large will be described with reference to FIGS. 6 and 7. As illustrated in FIG. 6, in the case where the distance between the user 20 and the virtual image 10 is relatively large, and the user 20 is located at the front of the virtual image 10 (the position of a user 20a in FIG. 6), the virtual image 10 (indicated as a virtual image 10a in FIG. 7) is displayed on the center of the display unit 150 as illustrated in 7a of FIG. 7.

[0068] When the user moves to a right position (the position of a user 20b in FIG. 6) from the front of the virtual image 10, the display position of the virtual image 10 changes to the left on the display unit 150 as illustrated in 7b (indicated as a virtual image 10b in FIG. 7). The parallax in the case where the distance between the user and the virtual image is large is smaller than that in the case where the distance is small. Thus, as illustrated in 7b, the user is less likely to feel something strange even when only the display position of the virtual image 10 is changed without changing the display angle thereof.

[0069] On the other hand, when the user moves to a left position (the position of a user 20c in FIG. 6) from the front of the virtual image 10, the display position of the virtual image 10 changes to the right on the display unit 150 as illustrated in 7c (indicated as a virtual image 10c in FIG. 7). Also in this case, as illustrated in 7c, the user is less likely to feel something strange even when only the display position of the virtual image 10 is changed without changing the display angle thereof.

[0070] Based on the above, the HMD 100 sets the 2D correction threshold to a smaller value in the case where the distance between the user and the virtual image is small than that in the case where the distance is large, and, on the other hand, sets the 2D correction threshold to a larger value in the case where the distance is large than that in the case where the distance is small. In other words, the HMD 100 may control the rendering frequency of the virtual image to the first rendering frequency in the case where the distance between the user and the virtual image is a first distance, and control the rendering frequency of the virtual image to the second rendering frequency in the case where the distance between the user and the virtual image is a second distance which is smaller than the first distance. Further, in a case where the distance between the user and the virtual image is extremely large (e.g., a case where the virtual image is located at infinity from the user), the parallax is extremely small. Thus, the HMD 100 may set the 2D correction threshold to an extremely large value to omit the 3D rendering. Note that the HMD 100 may reduce the rendering frequency of the virtual image in a continuous manner or in stages as the distance between the user and the virtual image becomes larger.

[0071] Next, the method for controlling the 2D correction threshold will be described in more detail with reference to FIG. 8. In FIG. 8, 8a illustrates a 2D correction threshold in a case where the distance between the user and the virtual image is relatively large, and 8b illustrates a 2D correction threshold in a case where the distance between the user and the virtual image is relatively small.

[0072] In 8a, when a virtual image 10d displayed on the center of the display unit 150 moves to the right, the HMD 100 performs only the 2D correction until the virtual image 10d moves beyond a boundary 30a, and performs not only the 2D correction, but also the 3D rendering when the virtual image 10d has moved beyond the boundary 30a. At this time, the 2D correction threshold is represented by a change amount 40a corresponding only to the 2D correction. Similarly, when the virtual image 10d moves to the left, the HMD 100 performs only the 2D correction until the virtual image 10d moves beyond a boundary 30b, and performs not only the 2D correction, but also the 3D rendering when the virtual image 10d has moved beyond the boundary 30b.

[0073] Then, in 8b, when a virtual image 10k displayed on the center of the display unit 150 moves to the right, the HMD 100 performs only the 2D correction until the virtual image 10k moves beyond a boundary 30c, and performs not only the 2D correction, but also the 3D rendering when the virtual image 10k has moved beyond the boundary 30c. At this time, the 2D correction threshold is represented by a change amount 40b corresponding only to the 2D correction. Similarly, when the virtual image 10k moves to the left, the HMD 100 performs only the 2D correction until the virtual image 10k moves beyond a boundary 30d, and performs not only the 2D correction, but also the 3D rendering when the virtual image 10k has moved beyond the boundary 30d.

[0074] As illustrated in FIG. 8, the HMD 100 sets the 2D correction threshold to a larger value in the case where the distance between the user and the virtual image is large than that in the case where the distance is small. This enables the HMD 100 to reduce flicker of the virtual image without making the user feel something strange by reducing the frequency of the 3D rendering and covering the reduction by the 2D correction.

[0075] Note that the method for controlling the 2D correction threshold illustrated in FIG. 8 is merely an example, and can be appropriately modified. For example, the 2D correction threshold may be set to different values on the basis of the direction in which the display position of the virtual image changes. More specifically, the 2D correction threshold may be set to different values between a case where the display position of the virtual image changes in the horizontal direction and a case where the display position changes in the vertical direction. Further, the 2D correction threshold may be set to different values on the basis of the display position before changed. More specifically, the 2D correction threshold may be set to different values between a case where the virtual image is displayed on the center of the display unit 150 and a case where the virtual image is displayed on an end of the display unit 150.

[0076] Further, the 2D correction threshold may be controlled on the basis of an element other than the distance between the user and the virtual image.

[0077] For example, the HMD 100 may switch the rendering frequency of the virtual image on the basis of the accuracy of a sensing result relating to a real space. The sensing result relating to the real space includes the recognition accuracy of the position and attitude of the HMD 100 (display device), and the 2D correction threshold may be controlled on the basis of the recognition accuracy of the position and attitude of the HMD 100. For example, the HMD 100 may control the rendering frequency of the virtual image to the first rendering frequency when the recognition accuracy is a first recognition accuracy, and control the rendering frequency of the virtual image to the second rendering frequency when the recognition accuracy is a second recognition accuracy which is higher than the first recognition accuracy. More specifically, the HMD 100 may control the 2D correction threshold on the basis of a residual which is the difference between the prediction of the position and attitude of the HMD 100 and an actual result thereof. As described above, the HMD 100 predicts the position and attitude of the HMD 100 using the various sensors (the outward stereo camera, the acceleration sensor, and the gyro sensor) mounted on the HMD 100. When the residual in the prediction of the position and attitude of the HMD 100 is large, the operations of the various sensors are considered to be unstable. Thus, the flicker of the virtual image may become larger if the rendering of the virtual image is performed on the basis of the prediction of the position and attitude of the HMD 100. Thus, the HMD 100 may set the 2D correction threshold to a larger value as the residual in the prediction of the position and attitude of the HMD 100 becomes larger.

[0078] Further, as described above, the output error of the various sensors may increase as the temperature of the sensors themselves and the peripheral temperature become higher. Thus, the flicker of the virtual image is likely to increase. Thus, the HMD 100 may set the 2D correction threshold to a larger value as the temperature of the sensors themselves or the peripheral sensor become higher.

[0079] Further, the HMD 100 may control the 2D correction threshold on the basis of the resolution of the display unit 150 (e.g., a display). More specifically, as described above, when the resolution of the display unit 150 is low, the flicker of the virtual image is likely to increase. Thus, the HMD 100 may set the 2D correction threshold to a larger value as the resolution of the display unit 150 becomes lower.

[0080] Further, as described above, when the virtual image itself is standing still, the flicker of the virtual image is likely to become larger. Thus, the HMD 100 may set the 2D correction threshold to a larger value when the virtual image is standing still than that when the virtual image is not standing still.

[0081] Further, when the virtual image has a more complicated shape, pattern, or color, the flicker of the virtual image becomes larger. Thus, the HMD 100 may set the 2D correction threshold to a larger value when the virtual image has a more complicated shape, pattern, or color.

[0082] Further, as the virtual image displayed on the display unit 150 becomes larger, the user is more likely to recognize flicker of the virtual image. Thus, the HMD 100 may set the 2D correction threshold to a larger value when the virtual image displayed on the display unit 150 becomes larger.

[0083] Further, when the user is not gazing at the virtual image, the user is less likely to feel something strange even if the frequency of the 3D rendering is reduced, and the reduction is covered by the 2D correction. Thus, the HMD 100 may set the 2D correction threshold to an extremely large value when the user is not gazing at the virtual image. Further, the HMD 100 may set the 2D correction threshold to a larger value as the gaze point by the user becomes more distant from the virtual image.

[0084] The above methods are merely examples, and the method for controlling the 2D correction threshold is not limited to the above.

……
……
……

本文链接：https://patent.nweon.com/16574

您可能还喜欢...