Sony Patent | Information Processing Apparatus, Information Processing Method, And Program

Patent: Information Processing Apparatus, Information Processing Method, And Program

Publication Number: 20200202161

Publication Date: 20200625

Applicants: Sony

Abstract

[Problem] An information processing apparatus, an information processing method, and a program are provided. [Solution] An information processing apparatus includes a display control unit configured to display a first virtual object having a visual salience corresponding to color information of a real space, and having substantially a same shape as a real object existing in the real space, in the real space so as to be viewed at substantially a same position as the real object.

FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

BACKGROUND

[0002] In recent years, there has been widely used an augmented reality (AR) technology that augments a reality environment perceived by human, by superimposing additional information onto a real space, and the AR technology has been applied to information presentation. In the AR technology, additional information presented to the user is also referred to as an annotation, and can be visualized using virtual objects in various forms, such as text, icons, images, or 3D models.

[0003] For example, Patent Literature 1 discloses a technology of transparently displaying an annotation (virtual object) with being superimposed on an image in a real space that is being viewed by the user, using a so-called see-through display (transmissive display unit). In addition, Patent Literature 1 describes displaying a notification (e.g. shadow of annotation) indicating the existence of an annotation, in a case where a part or all of the annotation falls outside a visible range.

CITATION LIST

Patent Literature

[0004] Patent Literature 1: WO 2014/162825** A**

SUMMARY

Technical Problem

[0005] Nevertheless, in the above-described AR technology, when a virtual object is displayed on a see-through display or the like, it sometimes becomes hard to see a real object existing in a real space. In this case, for example, it might become difficult for the user to feel as if a virtual object existed in a real space, or even though a real object existing in the real space is desired to attract attention, the user might feel hard to see the real object.

[0006] In view of the foregoing, the present disclosure proposes an information processing apparatus, an information processing method, and a program that can enhance the visibility of a real object.

Solution to Problem

[0007] According to the present disclosure, an information processing apparatus is provided that includes: a display control unit configured to display a first virtual object having a visual salience corresponding to color information of a real space, and having substantially a same shape as a real object existing in the real space, in the real space so as to be viewed at substantially a same position as the real object.

[0008] Moreover, according to the present disclosure, an information processing method is provided that includes: by a processor, displaying a first virtual object having a visual salience corresponding to color information of a real space, and having substantially a same shape as a real object existing in the real space, in the real space so as to be viewed at substantially a same position as the real object.

[0009] Moreover, according to the present disclosure, a program is provided that causes a computer to implement a function of: displaying a first virtual object having a visual salience corresponding to color information of a real space, and having substantially a same shape as a real object existing in the real space, in the real space so as to be viewed at substantially a same position as the real object.

Advantageous Effects of Invention

[0010] As described above, according to the present disclosure, it is possible to enhance the visibility of a real object.

[0011] Note that the effects described above are not necessarily limitative. With or in the place of the above effects, there may be achieved any one of the effects described in this specification or other effects that may be grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 is a diagram describing an overview of an information processing apparatus 1 according to the present embodiment.

[0013] FIG. 2 is an explanatory diagram for describing display control for displaying a content object with being appropriately shielded.

[0014] FIG. 3 is a block diagram illustrating a configuration example of the information processing apparatus 1 according to the embodiment.

[0015] FIG. 4 is a flowchart illustrating an operation example of the information processing apparatus 1 according to the embodiment.

[0016] FIG. 5 is an explanatory diagram for describing an example in which a masking object has luminance corresponding to luminance of a real object.

[0017] FIG. 6 is an explanatory diagram for describing an example in which a masking object has texture corresponding to texture of a real object.

[0018] FIG. 7 is an explanatory diagram for describing an example in which a masking object has luminance corresponding to information regarding brightness of a real space.

[0019] FIG. 8 is an explanatory diagram for describing an example in which a content object has a color corresponding to information regarding a color temperature of a real space.

[0020] FIG. 9 is an explanatory diagram describing an example of displaying a shadow of a masking object and a shadow of a content object in a real space.

[0021] FIG. 10 is an explanatory diagram describing an example of a case of displaying a shadow of a masking object and a shadow of a content object in a real space.

[0022] FIG. 11 is an explanatory diagram for describing an example of controlling a visual salience of a masking object on the basis of positional relationship between a real object and a content object.

[0023] FIG. 12 is an explanatory diagram for describing an example of controlling a visual salience of a masking object on the basis of whether the entire masking object is displayable.

[0024] FIG. 13 is an explanatory diagram for describing an example in which a focus object has texture corresponding to texture of a real object.

[0025] FIG. 14 is an explanatory diagram for describing an example of controlling a visual salience of a masking object on the basis of visual line information.

[0026] FIG. 15 is an explanatory diagram for describing an example of controlling a visual salience of a masking object on the basis of visual line information.

[0027] FIG. 16 is a block diagram illustrating an example of a hardware configuration.

DESCRIPTION OF EMBODIMENTS

[0028] Hereinafter, (a) preferred embodiment(s) of the present disclosure will be described in detail with reference to the appended drawings. Note that, in this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.

[0029] Note that the description will be given in the following order.

[0030] <<1. Overview>>

[0031] <<2. Configuration>>

[0032] <<3. Operation>>

[0033] <<4. Example of Display Control>>

[0034] <4-1. Example of Displaying Masking Object and Content Object>

[0035] <4-2. Example of Displaying Focus Object>

[0036] <<5. Hardware Configuration Example>>

[0037] <<6. Conclusion>>

[0038] <<1. Overview>>

[0039] First of all, an overview of an information processing apparatus according to a first embodiment of the present disclosure will be described. FIG. 1 is a diagram describing an overview of an information processing apparatus 1 according to the present embodiment. As illustrated in FIG. 1, the information processing apparatus 1 according to the present embodiment is implemented by a glasses-type head mounted display (HMD) attached to a head portion of a user U, for example. A display unit 13 corresponding to eyewear lens portions positioning in front of eyes of the user U when the user U wears the information processing apparatus 1 may be a so-called optical see-through display having optical transmissivity. The information processing apparatus 1 can present a virtual object within a vision range of the user U by displaying the virtual object on the display unit 13. In addition, an HMD being an example of the information processing apparatus 1 is not limited to a display that presents an image to both eyes, and may be a display that presents an image only to one eye. For example, an HMD may be a one-eyed display including the display unit 13 that presents an image to one eye.

[0040] In addition, the information processing apparatus 1 is provided with an outward camera 110 that captures an image in a visual line direction of the user U, that is to say, a vision range of the user when the user wears the information processing apparatus 1. Furthermore, the information processing apparatus 1 may be provided with an inward camera that captures an image of an eye of the user U when the user U wears the information processing apparatus 1, and various sensors such as a microphone, which are not illustrated in FIG. 1. A plurality of outward cameras 110 and a plurality of inward cameras may be provided.

[0041] Note that the shape of the information processing apparatus 1 is not limited to an example illustrated in FIG. 1. For example, the information processing apparatus 1 may be a headband type HMD (attached by a band wrapping around the whole circumference of a head portion, or a band passing through not only the side of the head but also the top of the head is sometimes provided), or a helmet-type HMD (a visor portion of the helmet corresponds to the display unit 13).

[0042] Here, for example, in a case where the display unit 13 has optical transmissivity, the user U can view information displayed on the display unit 13 while viewing a real space through the display unit 13. Thus, a virtual object displayed on the display unit 13 can be said to be displayed in the real space.

[0043] Furthermore, control that causes the user U to feel as if a virtual object existed in the real space can be performed. For example, the arrangement, the shape, or the like of a virtual object can be controlled on the basis of information regarding the real space that is obtained by image capturing of the outward camera 110, such as information regarding the position or the shape of a real object existing in the real space, for example.

[0044] A various types of virtual objects can be displayed on the display unit 13. For example, a virtual object may be a virtual object (hereinafter, will be sometimes referred to as a content object) indicating various types of content dependent on an application provided by the information processing apparatus 1. Alternatively, a virtual object may be a virtual object (hereinafter, will be sometimes referred to as a focus object) for highlighting a real object (hereinafter, will be sometimes referred to as a focus target) desired to attract attention of the user U, among real objects existing in the real space.

[0045] Note that various types of focus targets are used in accordance with applications provided by the information processing apparatus 1, and a focus target may be a real object indicating information required by the user in the real space, for example. Alternatively, in a case where an application that can perform a manipulation using a real object is provided, a focus target may be a real object that can be used for the manipulation.

[0046] Meanwhile, in a case where a virtual object is displayed on the display unit 13, the user U sometimes feels hard to see a real object. For example, by the display unit 13 displaying a virtual object more brightly (with higher luminance) than a real object, in some cases, the real object is felt relatively dark and it becomes hard to see the real object. In addition, if a focus object is displayed so as to be overlapped with a focus target for highlighting the focus target, in some cases, it becomes rather hard to see the focus target. Hereinafter, specific examples of these two cases will be described.

[0047] First of all, the description will be given of a case where, by the display unit 13 displaying a virtual object more brightly than a real object, the real object is felt relatively dark and it becomes hard to see the real object. When the display unit 13 having optical transmissivity displays a virtual object, if the virtual object is displayed darkly (with low luminance), it becomes hard to see the virtual object due to an overlap with the real space existing anteriorly to the display unit 13. Thus, for making the virtual object clearly visible, it is desirable to display the virtual object brightly (with high luminance).

[0048] Nevertheless, in a case where a virtual object is displayed brightly so as to make the virtual object clearly visible, the user U sometimes feels that light for the display unit 13 displaying the virtual object is more intense than light in the real space. In addition, as compared with the virtual object, the real object is felt relatively dark and it becomes hard to see the real object in some cases. In addition, a boundary between the virtual object and the real space sometimes stands out due to contrast between the virtual object and the real space.

[0049] Furthermore, if it becomes hard to see a real object or a boundary between a virtual object and the real space stands out, the user might become less likely to feel as if the virtual object existed in the real space. Particularly in a case where display control that is based on positional relationship between a virtual object and a real object is performed for causing the user to feel as if the virtual object existed in the real space, it is hard to see the real object and influence of an even is large.

[0050] Here, the display control that is based on positional relationship between a virtual object and a real object will be described. For causing the user U to feel as if a virtual object existed in the real space, for example, it is desirable to display a real object and a virtual object in such a manner that a feeling of strangeness is not felt at a boundary between the real object existing in the real space and the virtual object. Thus, for example, on the basis of information regarding a position and a shape of a real object, display control can be performed in the following manner for appropriately representing positional relationship between the real object and a virtual object. More specifically, the virtual object can be displayed in such a manner that a portion existing on the rear side of the real object when viewed from the user U is shielded. Note that because such display control is effective in the case of displaying the above-described content object, for example, the following description will be given using an example case where display control related to the content object is applied. Nevertheless, the display control can be similarly applied in the case of displaying another virtual object, and the technology to be described below using a content object as an example can also be similarly applied to another virtual object.

[0051] An example of display control for displaying a content object with being appropriately shielded on the basis of positional relationship between a real object and the content object will be described with reference to FIG. 2. FIG. 2 is an explanatory diagram for describing display control for displaying a content object with being appropriately shielded.

[0052] In the example illustrated in FIG. 2, a real object R1 exists in a real space. A real space image RS1 illustrated in FIG. 2 is a captured image of a real space obtained by capturing an image of the real space by the outward camera 110 illustrated in FIG. 1, for example. In addition, in the example illustrated in FIG. 2, a content object V1 is displayed on the basis of content data VD1 that has been prestored or acquired from an external device.

[0053] Here, display control is performed in the following manner for appropriately representing positional relationship between the real object R1 and the content object V1. More specifically, the content object V1 is displayed in such a manner that a portion existing on the rear side of the real object R1 when viewed from the user U is shielded.

[0054] First of all, a virtual object (hereinafter, will be referred to as a masking object) M1 for performing shielding (masking) in accordance with the shape and the position of the real object R1 is arranged in a virtual space together with the content object V1. Then, a virtual space image VS1 obtained by performing rendering at a position in the virtual space that corresponds to a viewpoint position of the user U is displayed on the display unit 13. Note that the position and the shape of the content object V1 may be controlled on the basis of information regarding the real space, or may be preset.

[0055] Here, in a case where the content object V1 is shielded by the masking object M1, it is desirable to perform display control in such a manner that the real object R1 is visible in the shielded region. By the display control, it becomes possible to cause the user U to feel as if the content object V1 existed in the real space. In this example, because the real object R1 seems to shield the content object V1, the user U feels as if the content object V1 existed on the rear side of the real object R1 in the real space.

[0056] For causing the above-described effect, the masking object M1 has black color as illustrated in FIG. 2, for example. Note that, in the present disclosure, the description will be given using an example case where the display unit 13 has a specification of performing display in such a manner that a portion corresponding to a black color region included in the image lets light through. Nevertheless, the specification can vary depending on a display device. For example, some display devices display a region with another color indicating transparency so as to let light through, or display a transparent region so as to let light through on the basis of information indicating the transparent region. In this case, in accordance with the specification of the display device, the other color indicating transparency or the information indicating the transparent region may be used in place of the black color.

[0057] In addition, for causing the above-described effect, it is desirable that the masking object M1 has the same shape as the real object R1, and is arranged at a position in the virtual space that corresponds to the position of the real object R1. In other words, in a case where the virtual space image VS1 is displayed on the display unit 13, it is desirable that the masking object M1 is displayed so as to be overlapped with the real object R1 at exactly the same position.

[0058] Nevertheless, the shape and the position of the masking object M1 need not be strictly the same as the shape and the position of the real object R1, and may be substantially the same. For example, the shape of the masking object M1 may be a shape closely analogous to the shape of the real object R1. Also in this case, the shape of the masking object M1 is interpreted as substantially the same as the shape of the real object R1 in this specification.

[0059] FIG. 2 illustrates a vision range AR1 of the user U that is obtained through the display unit 13 when the virtual space image VS1 is displayed on the display unit 13. As described above, in a region in the vision range AR1 that corresponds to a black color region in the virtual space image VS1, the real space is assumed to be viewed. In other words, in a case where the masking object M1 has black color as described above, among regions in the display unit 13, a region in which the masking object M1 is displayed lets light through, and the user is assumed to visually recognize that the real object R1 shields the content object V1.

[0060] Here, as described above, if the content object V1 is displayed with high luminance for making the content object V1 clearly visible, the user U feels the real space including the real object R1, relatively dark. As a result, as in the vision range AR1 illustrated in FIG. 2, it becomes hard for the user U to see the real object R1.

[0061] As described above, because the content object V1 seems to be shielded by the real object R1, it becomes possible for the user U to feel as if the content object V1 existed in the real space. Nevertheless, if it becomes hard to see the real object R1, the content object V1 does not seem to be shielded by the real object R1, and the user might become less likely to feel as if the content object V1 existed in the real space.

[0062] Subsequently, the description will be given of a case where, if a focus object is displayed so as to be overlapped with a focus target for highlighting the focus target, it becomes rather hard to see the focus target.

[0063] As a method for focusing attention on a focus target, for example, it is considered to perform display control of changing the color of the focus target. In such display control, a focus object having substantially the same shape as the focus target is displayed at substantially the same position as the focus target. That is to say, the focus object is displayed so as to be overlapped with the focus target. For example, by displaying, on the display unit 13, a focus object having a predetermined color so as to be overlapped with the focus target, light in the real space and light emitted by the display unit 13 are added and the color of the focus target seems to change.

[0064] Meanwhile, depending on the colors of a focus target and a focus object, by displaying the focus object so as to be overlapped with the focus target, it might become rather hard to see the focus target. For example, if a focus object with color largely different from the color of a focus target is displayed, the focus target viewed through the display unit 13 seems to have color with low visibility, and it might become hard to see the focus target. In addition, if the focus target seems to have color totally different from the color of the focus target that is normally recognized by human, the focus target is felt unnatural and it might become hard to see the focus target.

[0065] Thus, from a viewpoint of the above-described circumstances, an embodiment of the present disclosure has been eventually devised. Similarly to the above-described example, the information processing apparatus 1 according to the present embodiment displays a first virtual object (masking object or focus object in the above-described example) having substantially the same shape as a real object, in a real space so as to be viewed at substantially the same position as the real object. At this time, the information processing apparatus 1 according to the present embodiment can enhance visibility of the real object by performing display control in such a manner that the first virtual object has a visual salience corresponding to color information of the real space. Hereinafter, a configuration of the information processing apparatus 1 according to the present embodiment that has such an effect will be described in detail.

[0066] <<2. Configuration>>

[0067] FIG. 3 is a block diagram illustrating a configuration example of the information processing apparatus 1 according to the present embodiment. As illustrated in FIG. 3, the information processing apparatus 1 includes a sensor unit 11, a control unit 12, the display unit 13, a speaker 14, a communication unit 15, a manipulation input unit 16, and a storage unit 17.

[0068] (Sensor Unit 11)

[0069] The sensor unit 11 has a function of acquiring (sensing) various types of information regarding the user or a surrounding environment. For example, the sensor unit 11 includes the outward camera 110, an inward camera 111, a microphone 112, a gyro sensor 113, an acceleration sensor 114, an orientation sensor 115, a position measuring unit 116, and a biological sensor 117. Note that specific examples of the sensor unit 11 listed here are examples, and the present embodiment are not limited to these. In addition, a plurality of sensors may be provided as each type of the sensors.

[0070] The outward camera 110 and the inward camera 111 each include a lens system including an imaging lens, a diaphragm, a zoom lens, a focus lens, and the like, a drive system that causes the lens system to perform a focusing operation or a zoom operation, a solid-state image sensor array that generates an imaging signal by photoelectrically converting imaging light obtained by the lens system, and the like. The solid-state image sensor array may be implemented by a charge coupled device (CCD) sensor array or a complementary metal oxide semiconductor (CMOS) sensor array, for example.

[0071] Note that, in the present embodiment, it is desirable that a field angle and orientation of the outward camera 110 are set so as to capture an image of a region corresponding a vision range of the user in the real space. In addition, a plurality of outward cameras 110 may be provided. Furthermore, the outward camera 110 may include a depth camera that can acquire a depth map by sensing.

[0072] The microphone 112 collects voice of the user and ambient environmental sound, and outputs the collected voice and sound to the control unit 12 as voice data.

[0073] The gyro sensor 113 is implemented by a triaxial gyro sensor, for example, and detects angular speed (rotational speed).

[0074] The acceleration sensor 114 is implemented by a triaxial gyro sensor, for example, and detects acceleration in movement.

[0075] The orientation sensor 115 is implemented by a triaxial geomagnetic sensor (compass), for example, and detects an absolute direction (azimuth direction).

[0076] The position measuring unit 116 has a function of detecting a current position of the information processing apparatus 1 on the basis of a signal acquired from the outside. Specifically, for example, the position measuring unit 116 is implemented by a global positioning system (GPS) positioning unit, detects a position at which the information processing apparatus 1 exists, by receiving radio waves from a GPS satellite, and outputs the detected position information to the control unit 12. In addition, the position measuring unit 116 may be a sensor that detects a position by Wi-Fi (registered trademark), Bluetooth (registered trademark), transmission and reception with a mobile phone, a personal handyphone system (PHS), a smartphone, or the like, near-field communication, or the like, for example, aside from the GPS.

[0077] The biological sensor 117 detects biological information of the user. Specifically, for example, the biological sensor 117 can detect heartbeat, body temperature, perspiration, blood pressure, pulsebeat, aspiration, palpebration, an eyeball movement, a gaze time, a size of a pupil diameter, blood pressure, brain waves, body motion, a bodily position, cutaneous temperature, electrical skin resistance, micro vibration (MV), muscle potential, blood oxygen saturation level (5P02)), or the like.

[0078] (Control Unit 12)

[0079] The control unit 12 functions as an arithmetic processing unit and a control device, and controls the overall operations in the information processing apparatus 1 in accordance with various programs. In addition, as illustrated in FIG. 6, the control unit 12 according to the present embodiment functions as a recognition unit 122 and a display control unit 124.

[0080] The recognition unit 122 has a function of performing recognition (including detection) of information regarding the user or information regarding a real space around the user, on the basis of various types of sensor information (sensing result) sensed by the sensor unit 11.

[0081] For example, on the basis of a captured image of the real space that has been obtained by image capturing performed by the outward camera 110, the recognition unit 122 may three-dimensionally recognize the real space around the user and further recognize a real object existing in the real space.

[0082] In the three-dimensional recognition of the real space that is performed by the recognition unit 122, in a case where the outward camera 110 includes a plurality of cameras, for example, a depth map obtained by performing stereo matching on a plurality of captured images obtained by the plurality of cameras may be used. In addition, the three-dimensional recognition of the real space may be performed by the recognition unit 122 by associating feature points detected from captured images between frames on the basis of time-series captured images. In addition, in a case where the outward camera 110 includes a depth camera, the three-dimensional recognition of the real space may be performed on the basis of a distance image obtained by sensing of the depth camera.

[0083] In addition, the recognition unit 122 may recognize the shape of a real object. The shape of a real object that is recognized by the recognition unit 122 may be a three-dimensional shape in the real space or may be a two-dimensional shape in a captured image. A three-dimensional shape of a real object in the real space may be recognized by separating real objects on the basis of a three-dimensional recognition result of the real space, for example. In addition, a two-dimensional shape of a real object in a captured image may be recognized using a known region segmentation technology, for example.

[0084] In addition, the recognition unit 122 may recognize an AR marker being a real object used for displaying a virtual object in the AR technology, and the shape of the AR marker. For example, on the basis of information regarding an AR marker stored in the storage unit 17, the recognition unit 122 can recognize an AR marker and the shape of the AR marker from a captured image obtained by image capturing performed by the outward camera 110. Because a known technology can be used for the recognition of an AR marker, the detailed description is omitted here.

[0085] In addition, on the basis of a captured image of the real space that has been obtained by image capturing performed by the outward camera 110, the recognition unit 122 may recognize a light source in the real space and acquire light source information regarding the light source. The light source information may include information regarding brightness or color temperature of the real space, for example.

[0086] In addition, the recognition unit 122 may recognize a visual line of the user as information regarding the user, and acquire visual line information regarding the visual line of the user. For example, the recognition unit 122 can recognize a visual line of the user by analyzing an image of an eye of the user that has been obtained by image capturing performed by the inward camera 111. The visual line information may include information regarding a gaze point of the user, for example. For example, in a case where a visual line of the user stays in a fixed range for a predetermined time or more, the recognition unit 122 may detect a point existing on the visual line of the user as a gaze point. Note that the information regarding a gaze point of the user may be information regarding a three-dimensional position in the real space, or may be information regarding a two-dimensional position in a captured image of in the real space that is obtained by image capturing performed by the outward camera 110. In addition, a method of detecting a gaze point of the user by the recognition unit 122 is not limited to the example, and the detection may be performed using known various methods.

[0087] The display control unit 124 controls display performed by the display unit 13 having optical transmissivity, and displays a virtual object in the real space. Various virtual objects are displayed by the display control unit 124 in the real space, and the display control unit 124 may display a content object, a masking object, and a focus object, which have been described above, and the like, for example. In addition, as described above, the display control unit 124 may display a virtual object in the real space by arranging the virtual object in a virtual space and displaying, on the display unit 13, a virtual space image obtained by performing rendering at a position in the virtual space that corresponds to a viewpoint position of the user.

[0088] As described above, the display control unit 124 can display various content objects (an example of a second virtual object to be described later) dependent on an application provided by the information processing apparatus 1, for example. The display control unit 124 may identify a position of a content object in the real space on the basis of the setting of the application and information regarding the real space recognized by the recognition unit 122, for example. The display control unit 124 displays a content object in the real space so as to be viewed at a position of the content object in the real space.

[0089] In addition, as described above, for causing the user to feel as if a virtual object (for example, content object) existed in the real space, the display control unit 124 performs display control in such a manner that positional relationship between a real object and the virtual object is appropriately represented. Thus, the display control unit 124 displays a masking object (an example of a first virtual object to be described later) having substantially the same shape as a real object, in the real space so as to be viewed at substantially the same position as the real object. Because the display control has already been described with reference to FIG. 2, the detailed description is omitted here. Nevertheless, the masking object displayed by the display control unit 124 according to the present embodiment differs from the masking object M1 described with reference to FIG. 2 in that the masking object has a visual salience corresponding to color information of the real space.

[0090] In addition, as described above, the display control unit 124 displays a focus object (an example of a first virtual object to be described later) for highlighting a focus target desired to attract attention of the user, in the real space. As described above, the display control unit 124 displays a focus object having substantially the same shape as the focus target, so as to be viewed at substantially the same position as the focus target, in the real space. Nevertheless, the focus object displayed by the display control unit 124 according to the present embodiment differs from the above-described focus object in that the focus object has a visual salience corresponding to color information of the real space.

[0091] In this manner, the masking object and the focus object that are displayed by the display control unit 124 have similar features at least partially. Hereinafter, the masking object and the focus object will be sometimes collectively referred to as a first virtual object in a case where there is no need to make a distinction between the masking object and the focus object.

[0092] The display control unit 124 may acquire color information of the real space, from a captured image of the real space that has been obtained by image capturing performed by the outward camera 110. For example, the color information of the real space may include light source information acquired by the recognition unit 122 performing recognition related to a light source on the basis of a captured image.

[0093] In addition, the color information of the real space may include color information regarding a real object, for example. The color information regarding a real object may include luminance information regarding luminance of the real object, hue information regarding hue, saturation information regarding saturation, texture information regarding texture, and the like, for example. In addition, luminance information, hue information, and saturation information of the real object each may be statistically-processed information, and each may be information regarding average luminance, average hue, or average saturation of the real object, for example. The color information regarding a real object can be acquired from a captured image.

[0094] The visual salience of the first virtual object may include luminance corresponding to luminance of the real object, for example. In other words, the display control unit 124 may display the first virtual object so as to have luminance corresponding to luminance of the real object, on the basis of luminance information regarding luminance of the real object. With this configuration, because a first virtual object having luminance corresponding to luminance of a real object is displayed to be superimposed on the real object when viewed from the user, the visibility of the real object is enhanced.

[0095] Here, in a case where the display control unit 124 displays a second virtual object different from the first virtual object in the real space, luminance of the first virtual object is desirably lower than luminance of the second virtual object for blending each virtual object with the real space. Thus, the display control unit 124 may display the first virtual object and the second virtual object in such a manner that luminance of the first virtual object becomes lower than luminance of the second virtual object.

[0096] Note that the second virtual object may be the above-described content object, for example. In other words, as described above, the display control unit 124 displays a masking object (an example of first virtual object) and a content object (an example of second virtual object) on the basis of positional relationship between a real object and the content object.

[0097] Note that, with reference to FIG. 5, the description will be specifically given later of an example in which the display control unit 124 displays a masking object as the above-described first virtual object and displays a content object as the second virtual object.

[0098] In addition, the visual salience of the first virtual object is not limited to the above-described example. For example, the visual salience of the first virtual object may include texture corresponding to texture of a real object. In other words, the display control unit 124 may display the first virtual object in such a manner that the first virtual object has texture corresponding to texture of a real object, on the basis of texture information of the real object. Note that an example in which the display control unit 124 displays a masking object as the first virtual object will be specifically described later with reference to FIG. 6. In addition, an example in which the display control unit 124 displays a focus object as the first virtual object as the first virtual object will be specifically described later with reference to FIG. 13.

[0099] In addition, the visual salience of the first virtual object may include luminance corresponding to light source information. For example, the display control unit 124 may display the first virtual object in such a manner that the first virtual object has luminance corresponding to brightness of the real space, on the basis of light source information. Specifically, in a case where the real space is darker, the display control unit 124 may display the first virtual object in such a manner that the first virtual object has higher luminance. Note that an example in which the display control unit 124 displays a masking object as the first virtual object will be specifically described later with reference to FIG. 7.

[0100] Note that the use of light source information by the display control unit 124 is not limited to the example. The display control unit 124 may display the second virtual object so as to have color corresponding to light source information. For example, the display control unit 124 may change the color of a content object (an example of second virtual object) that has been originally set by an application, in accordance with information regarding color temperature that is included in light source information. Note that an example in which the display control unit 124 displays a content object as the second virtual object will be specifically described later with reference to FIG. 8.

[0101] In addition, the display control unit 124 may display a shadow of the first virtual object and a shadow of the second virtual object in the real space. The shadow may be a shadow generated (rendered) on the basis of a virtual light source set in a virtual space. Note that the display control unit 124 may set a virtual light source on the basis of light source information of the real space, or may set a virtual light source independently of light source information of the real space. The example will be specifically described later with reference to FIGS. 9 and 10.

[0102] Heretofore, the first virtual object and the second virtual object displayed by the display control unit 124 have been described. The display control unit 124 displays the first virtual object so as to have any of the above-described visual saliences or a combination of a plurality of visual saliences.

[0103] The display control unit 124 may dynamically control a visual salience of the first virtual object among the above-described visual saliences. Hereinafter, several examples of control of a visual salience of the first virtual object will be described.

[0104] The display control unit 124 may control a visual salience of a masking object on the basis of positional relationship between a real object and a content object.

[0105] The display control unit 124 can determine whether a masking object shields at least part of a content object, on the basis of positional relationship between a real object and the content object. Then, in a case where the display control unit 124 displays a masking object and a content object in such a manner that the masking object shields at least part of the content object, the display control unit 124 may perform control in such a manner that a visual salience varies for each region in the masking object.

[0106] For example, the display control unit 124 may control a visual salience of a masking object in such a manner that a first region of the masking object that is shielded by a content object has a first visual salience, and a second region of the masking object that excludes the first region has a second visual salience different from the first visual salience. Note that the example will be specifically described later with reference to FIG. 11.

[0107] In addition, the display control unit 124 may control a visual salience of a masking object depending on whether the masking object shields at least part of a content object. For example, the display control unit 124 may perform control in such a manner that a visual salience of a masking object varies between a case where the masking object shields at least part of a content object, and a case where the masking object does not shield the content object. For example, in a case where a number of real objects are recognized by the recognition unit 122, if the visibility of all the real objects is enhanced, the visibility of a content object might decrease relatively. In this case, for example, the display control unit 124 controls a visual salience of a masking object that does not shield a content object, in such a manner that the visibility of a real object related to the masking object does not change or is not enhanced so much. Then, the display control unit 124 controls a visual salience in such a manner that the visibility of a real object related to a masking object that shields a content object is enhanced more. With this configuration, it becomes possible to cause the user to feel as if a content object existed in the real space, without decreasing the visibility of the content object.

[0108] In addition, the display control unit 124 may control the visual salience of the first virtual object on the basis of whether the entire first virtual object can be displayed in such a manner that the first virtual object has substantially the same shape as a real object, and is viewed at substantially the same position as the real object. The example will be specifically described later with reference to FIG. 12.

[0109] In addition, the display control unit 124 may control a visual salience of a first virtual object on the basis of visual line information. For example, the display control unit 124 may identify a gaze region of the user in the real space on the basis of information regarding a gaze point that is included in visual line information, and control a visual salience of a first virtual object on the basis of the gaze region.

[0110] For example, the display control unit 124 may control a visual salience of a first virtual object in such a manner that a third region of the first virtual object that corresponds to the gaze region has a third visual salience, and a fourth region of the first virtual object that excludes the third region has a fourth visual salience different from the third visual salience. Note that the example will be specifically described later with reference to FIGS. 14 and 15.

[0111] (Display Unit 13)

[0112] The display unit 13 has optical transmissivity, and is implemented by a lens unit that performs display using a hologram optical technology, a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, or the like, for example.

[0113] (Speaker 14)

[0114] The speaker 14 reproduces a voice signal in accordance with control of the control unit 12.

[0115] (Communication Unit 15)

[0116] The communication unit 15 is a communication module for performing transmission and reception of data with another device in a wired/wireless manner. The communication unit 15 wirelessly communicates with an external device directly or via a network access point, using a method such as, for example, a wired local area network (LAN), a wireless LAN, Wireless Fidelity (Wi-Fi, registered trademark, infrared communication, Bluetooth (registered trademark), or near field communication/contactless communication.

[0117] (Manipulation Input Unit 16)

[0118] The manipulation input unit 16 is implemented by a manipulation member having a physical structure, such as a switch, a button, or a lever.

[0119] (Storage Unit 17)

[0120] The storage unit 17 stores programs and parameters for the above-described control unit 12 executing functions. For example, the storage unit 17 may store information regarding an AR marker that is to be used for the recognition unit 122 recognizing an AR marker, and information regarding a content object to be displayed by the display control unit 124.

[0121] Heretofore, the configuration of the information processing apparatus 1 according to the present embodiment has been specifically described, but the configuration of the information processing apparatus 1 according to the present embodiment is not limited to the example illustrated in FIG. 3. For example, at least part of functions of the control unit 12 of the information processing apparatus 1 may exist in another device connected via the communication unit 15.

[0122] <<3. Operation>>

[0123] Heretofore, a configuration example of the information processing apparatus 1 according to the present embodiment has been described. Subsequently, an operation example of the information processing apparatus 1 according to the present embodiment will be described with reference to FIG. 4. FIG. 4 is a flowchart illustrating an operation example of the information processing apparatus 1 according to the present embodiment.

[0124] As illustrated in FIG. 4, first of all, sensing is performed by the sensor unit 11, and for example, the outward camera 110 acquires a captured image by capturing an image of the real space (S12). Subsequently, on the basis of a result of sensing performed in Step S12, the recognition unit 122 of the control unit 12 performs recognition of information regarding the user and information regarding the real space around the user (S14).

[0125] Subsequently, the display control unit 124 arranges a virtual object in a virtual space and performs rendering at a position in the virtual space that corresponds to a viewpoint position of the user (S16). Then, the display control unit 124 displays a virtual space image obtained by performing rendering in Step S16, on the display unit 13 (S18).

[0126] In Step S16, the display control unit 124 according to the present embodiment arranges, in the virtual space, a first virtual object having a visual salience corresponding to color information of the real space, and having substantially the same shape as a real object. In addition, a position at which the first virtual object is arranged corresponds to a position of the real object in the real space. As a result, if the virtual space image obtained by performing rendering at a position in the virtual space that corresponds to a viewpoint position of the user is displayed on the display unit 13 in Step S18, the first virtual object is viewed at substantially the same position as the real object.

[0127] <<4. Example of Display Control>>

……
……
……