Sony Patent | Information processing apparatus, information processing method, and program

Patent: Information processing apparatus, information processing method, and program

Drawings: Click to check drawins

Publication Number: 20210056765

Publication Date: 20210225

Applicant: Sony

Assignee: Sony Corporation

Abstract

There is provided an information processing apparatus, an information processing method, and a program capable of further diversifying a method of implementing a virtual object using the AR technology, the information processing apparatus including: a display control unit configured to perform a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

Claims

1-20. (canceled)

21. An information processing apparatus comprising: a display control unit configured to determine whether a real object having a flat surface is positioned within a range in accordance with a distance between the real object and a mobile terminal, perform, when it is determined that the real object is positioned within the range, a first process of displaying a virtual object, wherein a display of the virtual object is changed and the virtual object is superimposed on the flat surface of the real object based on a user operation in the first process, and perform, when it is determined that the real object is not positioned within the range, a second process of displaying the virtual object superimposed in midair in front of the mobile terminal based on the user operation in the second process, wherein the display control unit is implemented via at least one processor.

22. The information processing apparatus according to claim 21, wherein the user operation includes a drag operation.

23. The information processing apparatus according to claim 21, wherein, in the first process, a shape of the virtual object is transformed.

24. The information processing apparatus according to claim 23, wherein, in the first process, the virtual object is rotated by an angle formed by a display region of the flat surface and a line-of-sight direction.

25. The information processing apparatus according to claim 23, wherein the shape of the virtual object is transformed in accordance with a shape of the flat surface.

26. The information processing apparatus according to claim 21, wherein the flat surface is a horizontal surface.

27. The information processing apparatus according to claim 21, wherein, in the first process, the virtual object is superimposed in a same plane region as the flat surface.

28. The information processing apparatus according to claim 21, wherein, in the first process, the virtual object is superimposed in a plane region that is separated from a plane region of the flat surface.

29. An information processing method comprising: determining whether a real object having a flat surface is positioned within a range in accordance with a distance between the real object and a mobile terminal; performing, when it is determined that the real object is positioned within the range, a first process of displaying a virtual object, wherein a display of the virtual object is changed and the virtual object is superimposed on the flat surface of the real object based on a user operation in the first process; and performing, when it is determined that the real object is not positioned within the range, a second process of displaying the virtual object superimposed in midair in front of the mobile terminal based on the user operation in the second process.

30. A non-transitory computer-readable storage medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising: determining whether a real object having a flat surface is positioned within a range in accordance with a distance between the real object and a mobile terminal; performing, when it is determined that the real object is positioned within the range, a first process of displaying a virtual object, wherein a display of the virtual object is changed and the virtual object is superimposed on the flat surface of the real object based on a user operation in the first process; and performing, when it is determined that the real object is not positioned within the range, a second process of displaying the virtual object superimposed in midair in front of the mobile terminal based on the user operation in the second process.

Description

CROSS REFERENCE TO PRIOR APPLICATION

[0001] This application is a continuation of U.S. patent application Ser. No. 15/577,113 (filed on Nov. 27, 2017), which is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2016/056666 (filed on Mar. 3, 2016) under 35 U.S.C. .sctn. 371, which claims priority to Japanese Patent Application No. 2015-120186 (filed on Jun. 15, 2015), which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

[0002] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

BACKGROUND ART

[0003] In recent years, technology called augmented reality (AR) that presents a user with additional information superimposed on the real world has attracted public attention. In the AR technology, information presented to the user is also called annotation and can be visualized using various forms of virtual objects such as text, icons, or animation For example, a technique in which a virtual object of the AR is displayed in association with a real object, for example, a virtual object is displayed on a surface of a real object, or a virtual object is caused to float above a real object and be displayed is disclosed in Patent Literature 1.

CITATION LIST

Patent Literature

[0004] Patent Literature 1: WO 2014/162823

DISCLOSURE OF INVENTION

Technical Problem

[0005] It has not been long since the AR technology proposed in the Patent Literature 1 or the like was developed, and it is difficult to say that techniques for utilizing the AR in various situations have been sufficiently proposed. For example, diversification of a method of implementing a virtual object using the AR technology is also one of techniques which are not sufficiently proposed.

[0006] In this regard, the present disclosure proposes an information processing apparatus, an information processing method, and a program, which are novel and improved and capable of further diversifying a method of implementing a virtual object using the AR technology.

Solution to Problem

[0007] According to the present disclosure, there is provided an information processing apparatus including: a display control unit configured to perform a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

[0008] Further, according to the present disclosure, there is provided an information processing method including: performing, by a processor, a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

[0009] Further, according to the present disclosure, there is provided a program causing a computer to function as: a display control unit configured to perform a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

Advantageous Effects of invention

[0010] As described above, according to the present disclosure, it is possible to further diversify methods of implementing a virtual object using the AR technology. Note that the effects described above are not necessarily limitative. With or in the place of the above effects, there may be achieved any one of the effects described in this specification or other effects that may be grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a diagram illustrating an example of an external configuration of a display device according to the present embodiment.

[0012] FIG. 2 is a block diagram illustrating an example of an internal configuration of a display device according to the present embodiment.

[0013] FIG. 3 is a diagram illustrating an overview of a process executed in a display device according to the present embodiment.

[0014] FIG. 4 is a diagram for describing a display control process performed by a display device according to the present embodiment.

[0015] FIG. 5 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0016] FIG. 6 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0017] FIG. 7 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0018] FIG. 8 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0019] FIG. 9 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0020] FIG. 10 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0021] FIG. 11 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0022] FIG. 12 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0023] FIG. 13 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0024] FIG. 14 is a diagram for describing a display example of a virtual object according to the present embodiment.

[0025] FIG. 15 is a flowchart illustrating an example of a flow of pre-processing performed in a display device according to the present embodiment.

[0026] FIG. 16 is a flowchart illustrating an example of a flow of a rendering process performed in a display device according to the present embodiment.

[0027] FIG. 17 is a flowchart illustrating an example of a flow of a rendering process performed in a display device according to the present embodiment.

[0028] FIG. 18 is a flowchart illustrating an example of a flow of a display control process according to an interaction performed in a display device according to the present embodiment

[0029] FIG. 19 is a block diagram illustrating an example of a hardware configuration of an information processing apparatus according to the present embodiment.

MODE(S)* FOR CARRYING OUT THE INVENTION*

[0030] Hereinafter, (a) preferred embodiment(s) of the present disclosure will be described in detail with reference to the appended drawings. In this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.

[0031] In addition, there are cases in the present specification and the diagrams in which constituent elements having substantially the same functional configuration are distinguished from each other by affixing different letters to the same reference numbers. For example, a plurality of constituent elements having substantially the same functional configuration are distinguished, like imaging units 10A and 10B, if necessary. However, when there is no particular need to distinguish a plurality of constituent elements having substantially the same functional configuration from each other, only the same reference number is affixed thereto, For example, when there is no particular need to distinguish imaging units 10A and 10B, they are referred to simply as imaging units 10.

[0032] The description will proceed in the following order.

1. Configuration example 1.1. External configuration example 1.2. Internal configuration example 2. Technical features 2.1. Overview of process 2.2. Variation of display process 2.3. Display region setting criteria 2.4. Display control based on interaction 2.5. Processing according to display region 2.6. Display control according to user 2.7. Other display control 3. Operation process example 4. Hardware configuration example

2. Conclusion

<<1. Configuration Example>>

<1.1. External Configuration Example>

[0033] First, an example of an external configuration of an information processing apparatus according to one embodiment of the present disclosure will be described with reference to FIG. 1.

[0034] FIG. 1 is a diagram illustrating an example of an external configuration of a display device 1 according to the present embodiment. The display device 1 illustrated in FIG. 1 is a device also called smart glasses or a head mounted display (HMD). The display device 1 including a mounting unit 50 of a frame structure that surrounds, for example, the head of the user and is fixed to the head of the user through the mounting unit 50. The display device 1 has a configuration in which a pair of display units 20A and 20B for the left eye and the right eye are arranged in front of the eyes of the user in the mounted state illustrated in FIG. 1. For example, a transmissive type display is used as the display unit 20, and the display device 1 is able to enter a through state, that is, a transparent or semi-transparent state, by controlling transmittance of the transmissive type display. When the display unit 20 is in the through state, there is no difficulty in normal life even if the user constantly wears the display device 1 as with glasses.

[0035] The display unit 20 is able to display a virtual object of the AR superimposed on the landscape of a real space by displaying images such as text or drawings in the transparent or semi-transparent state. n other words, the display device 1 may be implemented as a transmissive type HMD. Note that, in the transmissive type HMD, preferably, the display device 1 is fixed to the head of the user through the mounting unit 50, and a relative positional relation between the two eyes of the user and the display unit 20 is fixed as illustrated in FIG. 1. This is because if the relative positional relation changes, a position on the display for displaying the virtual object may change.

[0036] Further, the display unit 20 is also able to display the virtual object superimposed on captured images of the real space captured by imaging units 10A and 10B while displaying the captured images of the real space. The display unit 20 is also able to display the virtual object superimposed on the image of the virtual space while displaying images similar to images obtained by imaging the virtual space through the imaging units 10A and 10B. In other words, the display device 1 may be implemented as an immersive (video through type) HMD.

[0037] Alternatively, the display unit 20 may be implemented as an LED light source or the like that projects an image directly onto the retina of the user. In other words, the display device 1 may be implemented as a projection type HMD.

[0038] Various types of content may be displayed on the display unit 20 as the virtual object. For example, content may be data such as moving image content including a movie or a video clip, still image content imaged by a digital still camera or the like, an electronic hook, or the like. Further, such content may all be data to be displayed such as computer use data such as image data, text data, or spreadsheet data which is generated by the user using a personal computer or the like, a game image based on a game program, or the like

[0039] The imaging units 10A and 10B are arranged to perform imaging using a real space in a direction in which the user views as an imaging range in a state in which the user wears the display device 1. Each of the imaging units 10A and 10B may be implemented as a stereo camera capable of acquiring information indicating a distance in a direction in which the user views (hereinafter also referred to as “depth information”). In a case in which each of the imaging units 10A and 10B is implemented as a stereo camera, the display device 1 is able to recognize a shape and a posture of the real object in the real space with a high degree of accuracy. Hereinafter, each of the imaging units 10A and 10B is also referred to as an “outward facing stereo camera 10.”

[0040] On the other hand, imaging units 10C and 10D are arranged to perform imaging using the direction of the user, more specifically, both eyes of the user, as the imaging range in the state in which the user wears the display device 1. Each of the imaging units 10C and 10D may be implemented as a stereo camera capable of acquiring the depth information in the direction of both eyes of the user. In a case in which each of the imaging units 10C and 10D is implemented as a stereo camera, the display device 1 is able to recognize an eyeball position, a pupil position, a line-of-sight direction, or the like of the user with a high degree of accuracy. Hereinafter, each of the imaging units 10C and 10D is also referred to as an “inward facing stereo camera 10.”

[0041] Further, although not illustrated in FIG. 1, the display device 1 may have a speaker or an earphone speaker. Further, the display device 1 may include a microphone for acquiring an external sound.

[0042] The external appearance of the display device 1 illustrated in FIG. 1 is an example, and various structures in which the user wears the display device 1 can be considered. The display device 1 may be constituted by a mounting unit which is generally considered to be of an eyeglass type or head mounted type, and at least in this embodiment, it is sufficient that the display unit 20 is arranged close to the eyes of the user. Further, the display unit 20 may be configured such that a pair of units corresponding to both eyes are disposed or a single unit corresponding to one eye may be disposed.

[0043] Similarly, two speakers or two earphone speakers corresponding to the left. and right ears may be disposed, or one speaker or one earphone speaker corresponding to one ear may be disposed. Further, one or more microphones may be disposed, and this arrangement is arbitrary.

[0044] The example of the external configuration of the display device 1 according to the present embodiment has been described above. Next, an example of an internal configuration of the display device 1 according to the present embodiment will be described with reference to FIG. 2.

[0045] Note that, in the following description, as an example, the display device 1 is assumed to be implemented as a transmissive type HMDI). Hereinafter, an image displayed on the display unit 20 (transmission type display) of the display device 1 (including a transparently visible background and a superimposedly displayed virtual object) is also referred to as a “real space image.”

<1.2. Internal Configuration Example>

[0046] FIG. 2 is a block diagram illustrating an example of an internal configuration of the display device 1 according to the present embodiment. As illustrated in FIG. 2, the display device 1 includes an imaging unit 10, the display unit 20, a storage unit 30, and a control unit 40.

[0047] The imaging unit 10 has a function of imaging the real space and outputting a captured image. For example, the imaging unit 10 is able to be implemented as the outward facing stereo camera 10 and the inward facing stereo camera 10 illustrated in FIG. 1.

[0048] The display unit 20 has a function of displaying an image. For example, the display unit 20 may be implemented as a transmissive display arranged at both eyes of the user illustrated in FIG. 1.

[0049] The storage unit 30 has a function of storing various information. For example, the storage unit 30 stores information related to the virtual object displayed on the display unit 20. For example, the storage unit 30 stores content such as an image or text serving as a source of the virtual object.

[0050] The control unit 40 functions as an operation processing device and a control device and has a function of controlling an overall operation of the display device 1 according to various kinds of programs, The control unit 40 is implemented by an electronic circuit such as a central processing unit (CPU), a microprocessor, or the like. Note that the control unit 40 may include a read only memory (ROM) that stores programs, operation parameters, and the like to be used and a random access memory (RAM) that temporarily stores parameters which appropriately change and the like. Further, the control unit 40 may include a graphics processing unit (GPU) and a video RAM (VRAM) for image processing, and the like.

[0051] As illustrated in FIG. 2, the control unit 40 functions as a detecting unit 41, a storage control unit 43, and a display control unit 45, The detecting unit 41 has a function of detecting various kinds of information from the captured image output from the imaging unit 10. The storage control unit 43 has a function of storing information in the storage unit 30 and acquiring information stored in the storage unit 30. The display control unit 45 has a function of performing display control for causing images to be displayed on the display unit 20. Specifically, the display control unit 45 controls the display unit 20 such that the virtual object is displayed superimposed on the real space (hereinafter, this is also described as “the display device 1 superimposedly displays the virtual object”),

[0052] The example of the internal configuration of the display device 1 according to the present embodiment has been described above.

<<2. Technical Features>>

[0053] Next, technical features of the display device 1 according to the present embodiment will be described in order.

<2.1. Overview of Process>

[0054] First, an overview of a process performed in the display device 1 will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating an overview of a process performed in the display device 1 according to the present embodiment.

(1) Imaging Process

[0055] As illustrated in FIG. 3, the imaging unit 10 first performs an imaging process and acquires a captured image.

(2) Space Recognition Process

[0056] Next, as illustrated in FIG. 3, the detecting unit 41 performs a space recognition process on the basis of the captured image output from the imaging unit 10. Spatial information indicating the position and the posture of the display device 1 in the real space is acquired through the space recognition process.

[0057] For example, the spatial information may be an environment recognition matrix recognized by a publicly known image recognition technique such as a structure from motion (SfM) technique or a simultaneous localization and mapping (SLAM) technique. For example, the environment recognition matrix indicates a relative position and a posture of a coordinate system of a reference environment (real space) relative to a device-specific coordinate system of the display device 1. For example, in a case in which the SLAM method is used, a state variable including a position, a posture, a velocity, and an angular velocity of a device and a position of at least one feature point included in a captured image is updated for each frame of a captured image on the basis of a principle of an extended Kalman filter. Accordingly, it is possible to recognize the position and posture of the reference environment based on the position and posture of the device using an input image input from a monocular camera. Note that, for detailed description of the SLAM technique, see, for example, “Real-Time Simultaneous Localization and Mapping with a Single Camera” (Andrew J. Davison, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2, 2003, pp. 1403-1410).

[0058] In addition, the spatial information may be any information as long as it indicates the relative position and the posture of the imaging unit in the real space. For example, the environment recognition matrix may be recognized on the basis of depth data obtained from a depth sensor that may be installed in the imaging unit. The environment recognition matrix may also be recognized on the basis of output data output from an environment recognition system such as an infrared ranging system or a motion capture system. An example of such a technique is described in, for example, S. Izadi, et al, KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera, ACM Symposium on User Interface Software and Technology, 2011, but the technique is not limited thereto, and various publicly known techniques may be used for generation of the spatial information.

[0059] Alternatively, the spatial information may be generated by specifying a relative positional relation of each frame image through a stitching analysis for a series of frame images obtained by imaging the real space. In this case, the stitching analysis may be two-dimensional stitching analysis of attaching frame images onto a base plane or three-dimensional stitching analysis of attaching frame images onto arbitrary positions in a space.

[0060] Further, the spatial information may be acquired using an inertial sensor such as an acceleration sensor or a gyro sensor included in the display device 1 together. In this case, the spatial information can be estimated at a higher speed. Further, it is possible to acquire the spatial information with a certain degree of accuracy even in a case in which it is difficult to acquire the spatial information on the basis of the captured image due to a motion blur or the like caused by fast movement of the user (camera position).

(3) Real Object Recognition Process

[0061] Further, as illustrated in FIG. 3, the detecting unit 41 performs a real object recognition process on the basis of the captured image output from the imaging unit 10. For example, a real object included in the real space image near the user who wears the display device 1 is detected through the real object recognition process. Specifically, real object information indicating a position and a shape of the real object, more accurately, a position and a shape of the real object in the real space image (that is, unevenness of the real space viewed from the imaging unit 10), is acquired through the real object recognition process.

[0062] For example, the real object information may be the depth information of each pixel and a degree of reliability of the depth information based on the image obtained by the stereo camera. For example, the detecting unit 41 acquires the real object information on the basis of a difference (a binocular parallax) in the real object in a plurality of captured images in which the same real space is an imaging target from different viewpoints. Note that, due to a characteristic of stereo image recognition, the degree of reliability of the depth information related to a region in which a change in a color tone or the like is small may be low. Further, the depth information may be acquired using an arbitrary technique such as a depth sensor of a time of flight (ToF) scheme.

[0063] Further, the real object information may be acquired by comparing a feature quantity calculated from the captured image output from the imaging unit 10 with a feature quantity of the real object stored in the storage unit 30 in advance. In this case, the real object information may include identification information of a target real object and information indicating a position and a posture thereof For example, the feature quantity may be calculated by a publicly known feature quantity calculation technique such as a SIFT technique or a random ferns technique.

[0064] In addition, the real object information may be acquired by recognizing a known diagram or symbol, an artificial marker (for example, a bar code or a QR code (registered trademark)), or a natural marker.

(4) Gesture Recognition Process

[0065] Further, as illustrated in FIG. 3, the detecting unit 41 performs a gesture recognition process on the basis of the captured image output from the imaging unit 10. Gesture information indicating content of a gesture performed by the user wearing the display device 1 or another user nearby is acquired through the gesture recognition process. The gesture information may be acquired using, for example, a technique similar to that for the real object recognition process.

(5) Viewpoint Position Estimation Process

[0066] Next, as illustrated in FIG. 3, the display control unit 45 performs a viewpoint position estimation process. Viewpoint position information indicating a position of the eye of the user wearing the display device 1 is acquired through the viewpoint position estimation process. For example, the display control unit 45 estimates the viewpoint position information on the basis of the spatial information acquired through the space recognition process. The display control unit 45 may estimate the viewpoint position information further on the basis of the eyeball position, the pupil position, the line-of-sight direction, or the like of the user recognized from the image acquired by the inward facing stereo camera 10.

(6) Shielding Object Determination Process

[0067] Further, as illustrated in FIG. 3, the display control unit 45 performs a shielding object determination process. Shielding object information indicating overlapping of the real objects viewed by the user wearing the display device 1 is acquired through the shielding object determination process. For example, the display control unit 45 recognizes a positional relation between the real objects, a region which is invisible to the user due to overlapping, and the like on the basis of real object information.

(7) Display Region Setting Process

[0068] Further, as illustrated in FIG. 3, the display control unit 45 performs a process of setting a display region for the virtual object. A display region on which the virtual object is displayed (superimposed) is set in one region on the real space through the display region setting process.

[0069] For example, the display control unit 45 detects a plane region on the basis of the real object information. The plane region refers to a flat surface (having no or little unevenness) in the real object. Further, the display control unit 45 calculates, for example, an area, a normal line, a color, and a color variation degree of the plane region on the basis of the depth information and the captured image. Then, the display control unit 45 sets a region suitable for displaying the virtual object in the plane region as the display region. The display control unit 45 may set a plane region calculated on the basis of the depth information with a higher degree of reliability as the display region preferentially over the plane region calculated on the basis of the depth information with a lower degree of reliability. The display region may have various shapes such as a rectangle, a triangle, a polygon, or a circle. For example, the suitable region is a flat surface of the real object and indicates a surface having a sufficient area, and moreover, a small angle between a vector from the imaging unit 10 to a plane and a normal line of a plane (that is, a surface facing the user). Further, a suitable region may be a region whose color change is flat. As described above, when the display region is set in the region suitable for the display, collision between real world information and the virtual object is prevented or reduced.

[0070] On the other hand, the display control unit 45 may set, for example, an arbitrary region (for example, a region in the air) on the space other than the surface of the real object as the display region. For example, in a case in which there is no flat surface in the real object, in a case in which there is no surface having a sufficient area, or in a ease in which the angle between the vector from the imaging unit 10 to the plane and the normal line of the plane is large (that is, in a case in which it does not face the user, for example, a line-of-sight direction is substantially parallel to the plane), the display control unit 45 sets a region in the air as the display region.

[0071] Further, the display control unit 45 may set the display region on the basis of the gesture information. For example, the display control unit 45 enlarges, reduces, or moves the display region according to the instruction of the user.

[0072] The display control unit 45 lays out content in accordance with the set display region and renders the laid-out content into the texture of the virtual object.

(8) Virtual Object Rendering Process

[0073] Then, as illustrated in FIG. 3, the display control unit 45 performs a virtual object rendering process. For example, the display control unit 45 decides a viewpoint for rendering on the basis of the viewpoint position information and the spatial information. Specifically, the display control unit 45 performs projective transform of the display region set in the display region setting process into coordinates on the display unit 20 on the basis of the viewpoint position information and the spatial information. Hereinafter, the display region which has undergone the projective transform is also referred to simply as a “display region.” Then, the display control unit 45 transforms the texture of the virtual object through the GPU and writes resulting information in a VRAM corresponding to the display region which has undergone the projective transform. Note that the display control unit 45 may cause a shielded portion of the virtual object to disappear on the basis of the shielding object information.

(9) Display Process

[0074] Finally, as illustrated in FIG. 3, the display unit 20 performs a display process. For example, the display unit 20 performs display based on pixel information read from the VRAM through a display controller (for example, the display control unit 45). Accordingly, the virtual object is displayed superimposed on the real space image.

<2.2. Variation of Display Control Process>

[0075] The display device 1 (for example, the display control unit 45) according to the present embodiment is able to simultaneously or selectively perform a plurality of types of superimposing processes. For example, the display device 1 is able to perform a first process of displaying the virtual object superimposed on the real space at a position which is based on the real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space. Hereinafter, a variation of the display control process performed by the display device will be described with reference to FIG. 4.

[0076] FIG. 4 is a diagram for describing a display example of the virtual object according to the present embodiment. As illustrated in FIG. 4, the user wearing the display device 1 is looking at a table. Virtual objects 110A and 110B are displayed by the display device 1 and visible to the user. For example, in the first process, the display device 1 displays the virtual objects in accordance with a position, a posture, a shape, or the like of the real object. For example, in the example illustrated in FIG. 4, the virtual object 110A is displayed as if the virtual object 110A stuck to a top surface of a table through the first process. On the other hand, in the second process, the display device 1 displays the virtual object regardless of the real object. For example, in the example illustrated in FIG. 4, the virtual object 110B is displayed as if the virtual object 110B floated while moving forward along the line of sight of the user through the second process.

[0077] Here, in this specification, a “position” may mean a position in the real space perceived by the user (that is, real space coordinates) or may mean a position on the display unit 20 (for example, a transmissive type display) (that is, coordinates on a screen). However, the display of the virtual object may be different depending on what it means. For example, even when there is a sense of perspective as the position perceived by the user, it may be displayed at the same position on the display unit 20.

[0078] The virtual object may be operation information which is a target operated by the user. For example, the user is able to input information, and the like by touching the virtual object. In a case in which the virtual object floats in the air, it has been difficult for the user to perform an operation while receiving feedback indicating that an operation is performed by touching the virtual object. On the other hand, in a case in which the virtual object is displayed on the surface of the real object, the user is able to touch the virtual object and touch the real object, and thus the user is able to perform an operation while receiving feedback.

[0079] The first process and the second process will be described below in detail.

(1) First Process

(1a) Basic Policy

[0080] For example, the display device 1 may change the display of the virtual object on the basis of the shape of the detected real object in the first process. Accordingly, the virtual object is more naturally superimposed on the real space when viewed by the user.

[0081] Specifically, the display device 1 changes the display of the virtual object by changing the relative posture of the virtual object to the detected real object and/or by transforming the shape of the virtual object, For example, in a case in which the plane set as the display region in the detected real object does not faces the user, that is, in a case in which the display region is not orthogonal to the line-of-sight direction of the user, the display device 1 rotates the virtual object by an angle formed by the display region and the line-of-sight direction. Further, for example, in a case in which the shape of the plane sot as the display region does not match the original shape of the virtual object, the display device 1 transforms the shape of the virtual object in accordance with the shape of the display region. Accordingly, the virtual object is viewed to be combined more integrally with the real object when viewed by the user. As a technique for changing the relative posture of the virtual object with respect to the real object, for example, a publicly known technique such as a technique described in JP 2012-221249A may be applied.

(1b) Superimposed Display on Surface of Real Object

[0082] The display device 1 may display the virtual object superimposed on the surface of the detected real object. For example, the display device 1 sets the surface of the real object as the display region, and maps the virtual object to the surface of the real object, and displays the virtual object, Accordingly, the virtual object looks to stick to the real object when viewed by the user. In a case in which the virtual object is displayed to be superimposed on the surface of the real object, visibility is improved as compared with a case in which the virtual object is displayed in the air which may overlap with the background. Note that the display device 1 may display the virtual object superimposed on the same surface as the surface of the real object (that is, on the surface of the real object) or may display the virtual object superimposed on a surface separated from the surface (for example, a plane region horizontally spaced apart from a plane region of the real object by several millimeters or several centimeters).

[0083] Here, the display device 1 may display the virtual object superimposed on a continuous surface among the surfaces of the detected real object. For example, the continuous surface is a region which has no shielding object or is not divided by a shielding object or the like in the plane region. Further, for example, the continuous surface may be regarded as a surface which has a flat slope change and/or a flat color tone change. Accordingly, the virtual object is displayed to be more easily visible when viewed by the user. Here, the present display example will be described with reference to FIGS. 5 and 6.

[0084] FIG. 5 is a diagram for describing a display example of the virtual object according to the present embodiment. In an example illustrated in FIG. 5, in a real space image 120, an envelope 122 is placed on the table. For example, the display device 1 sets a surface of envelope 122 which has a flat slope change and a flat color tone change as the display region and displays a virtual object 124 on the envelope 122.

[0085] FIG. 6 is a diagram fir describing a display example of the virtual object according to the present embodiment. In an example illustrated in FIG. 6, in a real space image 130, an envelope 132 is placed on the table, and a cable 134 and a keyboard 136 are further placed on the envelope 132. For example, the display device 1 sets a portion which is not blocked by the cable 134 and the keyboard 136 serving as the shielding object in the surface of the envelope 132 which is the continuous surface having a flat slope change and a flat color tone change as the display region, and causes a virtual object 138 to be displayed on the set region.

[0086] Further, the display device 1 may display the virtual object superimposed on the surface determined to be horizontal or the surface determined to be vertical among the surfaces of the detected real object, The display device 1 may determine a horizontal or vertical surface on the basis of the spatial information, For example, the display device 1 displays the virtual object on a ground or a top surface of a table or the like determined to be horizontal, or a wall determined to be vertical, or the like. Accordingly, the virtual object is displayed to be easily visible when viewed by the user.

[0087] Note that it is determined whether or not it is the around on the basis of an acceleration direction, a size of a plane, a relative position to the display device 1, whether or not the plane is lowest among the horizontal planes, or the like. In a case in which the ground is set as the display region, the display device 1 is able to perform navigation according to a terrain.

(1c) Superimposed Display in Air

[0088] Further, the display device 1 may display the virtual object superimposed on a position spaced apart from the detected real object, For example, the display device sets the display region in various spaces such as a surface on an extension of the surface of the real object, a space above the real object, or a space between the real object and the user. More simply, the display device 1 displays the virtual object as if it is floating in the air. Accordingly, it is possible to display the virtual object in association with the real object, for example, even when the surface of the real object has no suitable display region. Note that, preferably, the display region set in the air faces the user.

……
……
……