Sony Patent | Information processor, information processing method, and storage medium

3. Conclusion1. CONFIGURATION EXAMPLE<1-1. Configuration Example of Information Processor>

A basic configuration of an information processor 10 according to an embodiment of the present disclosure will be first described below. FIG. 1 is a block diagram illustrating an example of the basic configuration of the information processor 10 according to an embodiment of the present disclosure. As illustrated in FIG. 1, the information processor 10 includes a control unit 100, a communication unit 110, a camera 120, an operation input unit 130, a sensor unit 140, a display unit 150, a speaker 160, and a storage unit 170.

The information processor 10 according to the present embodiment mainly performs control to display a captured image of a real space on the display unit 150 and displays a virtual object superimposed according to the position of an object in the real space (hereinafter referred to as a real object) included in the captured image. The configurations of the information processor 10 will be described below.

(1) Communication Unit 110

The communication unit 110 is connected to an external device and transmits and receives data to and from the external device via wire or radio communications. For example, the communication unit 110 is connected to a network to transmit and receive data to and from a server on the network. For example, the communication unit 110 may receive, from the server, data on a virtual object to be superimposed on a captured image of a real space or various kinds of data about superimposition. The communication unit 110 is connected to an external device or a network to perform communications via, for example, a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a mobile communication network (LTE (Long Term Evolution), 3G (third-generation mobile communication system), 4G (fourth-generation mobile communication system), and 5G (fifth-generation mobile communication system)).

(2) Camera 120

The camera 120 is an example of an imaging unit having the function of capturing an image of a real space. An image of a real space captured by the camera 120 is displayed on the display unit 150. The captured image on the display unit 150 is displayed as a so-called through image corresponding to the view of a user of the information processor 10. The through image may be regarded as a captured image that is displayed in real time. Alternatively, the through image may be regarded as the latest one of images captured by the camera 120. An image of a real space captured by the camera 120 may be used for recognizing the real space.

In the present specification, a displayed image that corresponds to a user's view is also referred to as a view image. The camera 120 aimed at acquiring a view image is desirably oriented in the direction of the line of sight of the user who uses the information processor 10. It is assumed that the user views the display unit 150 when using the information processor 10. Thus, if the information processor 10 is implemented by, for example, an HMD that is mounted on the head of the user and is configured with the display unit 150 placed immediately in front of user's eyes when being mounted, the camera 120 is oriented in the same direction as the head of the user.

The camera 120 may be a single camera or a plurality of cameras. Alternatively, the camera 120 may be configured as a so-called stereo camera.

(3) Operation Input Unit 130

The operation input unit 130 has the function of receiving a user operation. The operation input unit 130 outputs information about a received operation to the control unit 100. The operation input unit 130 may be implemented by an input device, e.g., a touch panel or a button.

(4) Sensor Unit 140

The sensor unit 140 has the function of sensing a real space, for example, a position (user position) and a motion of the information processor 10 and the surrounding circumstances. The sensor unit 140 includes, for example, a position measuring unit, an acceleration sensor, an angular velocity sensor, and a terrestrial magnetism sensor. The sensor unit 140 may further include a camera for capturing an image used for recognizing a real space (a camera for recognition), the camera being different from the camera 120 for capturing a view image. In this case, the angle of view of the camera for recognition may include at least the angle of view of the camera 120 for capturing a view image. Furthermore, the sensor unit 140 may include a sensor for measuring a distance from an object existing in a real space. The sensor for measuring a distance may be a sensor for measurement based on a stereoscopic image captured by a stereo camera or an infrared sensor.

The position measuring unit has the function of calculating an absolute or relative position of the information processor 10. For example, the position measuring unit may detect a current position on the basis of a signal acquired from the outside. Specifically, a GNSS (Global Navigation Satellite System) that detects the current position of the information processor 10 by receiving, for example, radio waves from an artificial satellite. In addition to the GNSS, Wi-Fi (registered trademark), Bluetooth (registered trademark), transmission and reception to and from a cellular phone, a PHS, and a smartphone, or short-distance communications may be used for a method of detecting a position. The position measuring unit may estimate information indicating a relative change, on the basis of the detection result of an acceleration sensor or an angular velocity sensor or the like.

(5) Display Unit 150

The display unit 150 is implemented by a so-called video see-through display. A video see-through display provides an image of a real space for the user by displaying, on the display, a moving image of the real space captured by an imaging device fixed relative to the display, that is, a through image in real time. In the case of a typical video see-through display, light in a real space is blocked by the casing of the video see-through display and does not directly reach user's eyes.

The display unit 150 may be switchable between a video see-through display and an optical see-through display. An optical see-through display is a display that can directly transmit light in a real space to user's eyes. An optical see-through display can be used in known modes including a half-mirror mode, a light-guide plate mode, and a retina direct-drawing mode. The outer surface of an optical see-through display has a configuration, for example, a dimming element that dynamically blocks light in a real space, enabling switching between the optical see-through display and a video see-through display.

Moreover, a video see-through display for implementing the display unit 150 may be a hand-held display for a smart phone or a mobile terminal for a wearable display. A mobile terminal may be connected wirelessly or via a cable to a computer separated from the mobile terminal. A video see-through display for implementing the display unit 150 can be provided in various mobile objects including an automobile. The function of controlling the display of a video see-through display may be performed by a standalone terminal or may be performed by a plurality of information processors via a wireless network or wired connection.

(6) Speaker 160

The speaker 160 has the function of outputting sound. For example, the speaker 160 may be configured as headphones, earphones, or a bone conduction speaker.

(7) Storage Unit 170

The storage unit 170 is implemented by ROM (Read Only Memory) in which programs used for the processing of the control unit 100 and arithmetic parameters or the like are stored, and RAM (Random Access Memory) in which optionally changed parameters or the like are temporarily stored.

(8) Control Unit 100

The control unit 100 acts as an arithmetic processing unit and a controller and controls all operations in the information processor 10 in accordance with various programs. The control unit 100 is implemented by, for example, a CPU (Central Processing Unit) and electronic circuits such as a microprocessor. The control unit 100 may further include ROM (Read Only Memory) in which programs to be used and arithmetic parameters or the like are stored, and RAM (Random Access Memory) in which optionally changed parameters or the like are temporarily stored.

The control unit 100 according to the present embodiment also acts as a recognition unit 101, a virtual object drawing unit 102, a view drawing unit 103, and a display processing unit 104.

(Recognition Unit 101)

The recognition unit 101 recognizes various kinds of data inputted to the control unit 100. Specifically, the recognition unit 101 recognizes a real space on the basis of a captured image of the real space. Alternatively, the recognition unit 101 may recognize its own position or orientation on the basis of sensing data or the like inputted from the sensor unit 140. When the control unit 100 superimposes a virtual object on a captured image of a real space such that the virtual object corresponds to a real object, the position and orientation of the real object or the position and orientation of the camera 120 (in other words, the information processor 10 including the camera 120 or a user wearing the information processor 10) are used.

Recognition of Real Space

The recognition unit 101 recognizes a real space on the basis of a captured image of the real space. In the recognition of the real space, for example, an object (real object) in a captured image of the real space is recognized. The algorithm of object recognition is not particularly limited. For example, three-dimensional object recognition or a bone estimation algorithm may be used. Moreover, recognition may be performed by using a plurality of recognition algorithms at the same time, the recognition algorithms including an algorithm for recognizing humans and an algorithm for recognizing objects other than humans. The recognition of an object includes at least the position or orientation of the object. A captured image used for recognition may be an image captured by a camera that captures an image (through image) to be displayed on the display unit 150 or may be an image captured by a camera that is different from the camera and is provided for recognition. The recognition unit 101 may acquire depth data from the camera 120 or the sensor unit 140 and use the data for the recognition of a real space. The recognition unit 101 can acquire a distance from an object in a real space.

Recognition of Own Position

The recognition unit 101 recognizes at least one of the position and orientation of the information processor 10 (more specifically, the camera 120) on the basis of the detection result of the sensor unit 140. The recognition by the recognition unit 101 may include processing performed by the position measuring unit or may be processing for acquiring, from the position measuring unit, information indicating its own position. For example, the recognition unit 101 recognizes its own position or orientation as information indicating a relative change, on the basis of the detection result of an acceleration sensor or an angular velocity sensor or the like. The information indicating its own position may include a moving velocity.

Alternatively, the recognition unit 101 may recognize the position of the information processor 10 (in other words, a user) by comparison with a space map generated in advance by a technique of SLAM (Simultaneous Localization and Mapping) or recognize a real-space positional relationship with a real object in a captured image. If the information processor 10 is an HMD to be mounted on a user's head, the recognition unit 101 can recognize the position, orientation, inclination, and a moving velocity or the like of the user's head as the recognition of its own position. In a specific example, the recognition unit 101 detects components in a yawing direction, a pitching direction, and a rolling direction as motions of the user's head, thereby recognizing a change of at least one of the position and orientation of the user's head.

(Virtual Object Drawing Unit 102)

The virtual object drawing unit 102 draws a virtual object to be superimposed on a captured image (view image) in a real space, into a corresponding buffer. The buffer is at least a part of the storage area of a storage unit, e.g., flash memory or RAM for temporarily or permanently retaining various kinds of data. Moreover, the buffer is a so-called frame buffer for storing the display contents of a screen. At the completion of recognition by the recognition unit 101 on the basis of a captured image in a real space, the virtual object drawing unit 102 draws a virtual object at a proper position or with proper orientation relative to the position or orientation of a real object on the basis of the result of recognition (the position or orientation of the real object). Hereinafter, the buffer in which a virtual object to be superimposed on a view image is drawn will be also referred to as a virtual information buffer.

(View Drawing Unit 103)

The view drawing unit 103 draws an image (view image) captured by the camera 120 in a real space, into a corresponding buffer. As an example, the buffer for drawing a view image and the buffer for drawing a virtual object are assumed to be separate storage areas. The separate storage areas may be different storage areas allocated in one storage unit or may be obtained by logically dividing the same storage area. Alternatively, different storage units may be allocated. Hereinafter, the buffer in which a captured image (view image) in a real space is drawn will be also referred to as a view information buffer.

(Display Processing Unit 104)

The display processing unit 104 performs control (display control) to read information drawn in the buffer and output the information to the display unit 150. Specifically, for example, the display processing unit 104 performs control such that a virtual object read from the virtual information buffer is superimposed on a view image read from the view information buffer and is displayed on the display unit 150.

The configuration of the information processor 10 was specifically described above. The configuration of the information processor 10 according to the present disclosure is not limited to the example of FIG. 1. For example, the information processor 10 does not need to include all the configurations in FIG. 1.

A device for implementing the information processor 10 is, for example, a head-mounted display (HMD). The information processor 10 according to the present embodiment is implemented by, for example, a video see-through HMD. A video see-through HMD is configured to cover user's eyes when being mounted on the head or face of a user. A display unit, e.g., a display (the display unit 150 in the present embodiment) is held immediately in front of the user's eyes. Moreover, the video see-through HMD includes an imaging unit (the camera 120 in the present embodiment) for capturing an image of scenery around the user and displays, on the display unit, an image of scenery captured by the imaging unit ahead of the user. With this configuration, a user wearing a video see-through HMD may have difficulty in directly viewing scenery outside but can confirm the scenery through an image displayed on the display unit. If the information processor 10 is implemented by a video see-through HMD, the display unit 150 includes left and right screens fixed at positions corresponding to the left and right eyes of a user and displays a left-eye image and a right-eye image. The display unit provided for a video see-through HMD for implementing the information processor 10 according to the present embodiment may allow switching to an optical see-through display. The information processor 10 may be a terminal held with a user's hand, e.g., a smartphone, a cellular phone, or a tablet or a wearable device on a user's body.

The information processor 10 may be implemented by a plurality of devices. For example, the information processor 10 may be a system configured with a display device including at least the camera 120, the sensor unit 140, and the display unit 150 in FIG. 1 and a controller including at least the control unit 100. The display device and the controller are connected to each other via wire or radio communications and can transmit and receive data to and from each other. The display device and the controller may communicate with each other though, for example, Wi-Fi (registered trademark), a LAN (Local Area Network), or Bluetooth (registered trademark).

At least some of the functions of the control unit 100 in the information processor 10 may be implemented by a server provided on a network, a dedicated terminal disposed in the same space as the user, a smartphone, a tablet, or a PC.

<1-2. Organization of Problems>

The following describes problems when a captured image in a real space is displayed in real time and a virtual object is superimposed thereon by using the video see-through display.

To obtain a view as if a virtual object is actually present in a real space, the virtual object is desirably displayed at the position of a real object. Also when the video see-through display is used, a real object is recognized on the basis of a captured image in a real space. At the completion of the recognition, a virtual object is drawn at a proper position or with proper orientation according to the position or orientation of the real object on the basis of the result of recognition.

The drawing of a virtual object requires the recognition of a real space as a preceding step and thus may be delayed from the drawing of a captured image in a real space. For example, if the user moves to change the view, processing for drawing a virtual object may stay behind processing for drawing a captured image in a real space. FIG. 2 is an explanatory drawing illustrating unnaturalness caused by a display delay of a virtual object. A displayed image in FIG. 2 is an example of an image displayed on the display unit 150. The displayed image includes a captured image (view image) in a real space and a virtual object 30. For example, if the user turns right (the camera 120 turns right), an image captured by the camera 120 is displayed in real time on the display unit 150 (for example, about 15 milliseconds from imaging to display) as illustrated in a display image 200 and a display image 201 in FIG. 2, so that the view changes. If the view is covered with the display unit 150 and scenery in a real space does not directly come into sight, the user hardly notices a displacement (display delay) between a real view and a view image displayed on the camera 120, which does not interfere with naturalness. In contrast, a display delay of the virtual object 30 relative to the view image tends to be noticeable even if the displacement is relatively small, leading to unnaturalness. Specifically, as indicated in the display image 201 of FIG. 2, the latest (current) image is displayed as a view image. At this point, the recognition of the latest (current) view image is not completed, so that the display position of the virtual object 30 is displaced from a desk (real object), leading to unnaturalness. Thereafter, at the completion of recognition and drawing based on the recognition, the display position of the virtual object 30 is updated to a proper position corresponding to the position of the desk as indicated in a display image 202.

Referring to FIG. 3, the principle of a display delay of the virtual object 30 will be specifically described below. FIG. 3 is a timing chart indicating the flow of a series of processing to explain a display delay of the virtual object 30.

As indicated in FIG. 3, a series for displaying a virtual object superimposed on a real object includes “imaging” of a real space, “recognition” of a real object by analyzing a captured image, “drawing (virtual object drawing)” of a virtual object to be superimposed, “drawing (view drawing)” of a captured image, and “display (output)” of the drawn virtual object and the captured image. A captured image is drawn when being acquired. A virtual object is drawn after the completion of the recognition of the captured image. In the recognition, processing requiring a processing time (an example of a load of recognition), for example, three-dimensional object recognition can be performed. The view drawing, the recognition, and the virtual object drawing are performed in parallel, but the completion of the virtual object drawing is delayed from the completion of a captured image.

In the example of FIG. 3, a captured image used as a view image and a captured image used for recognition are both indicated to be obtained from a single step of “imaging” (imaging unit). “Imaging” (imaging unit) may be provided in multiple steps. Specifically, “imaging” (imaging unit) for acquiring a captured image used as a view image and “imaging” (imaging unit) for acquiring a captured image used for recognition may be provided. The processing of display may be typically updated at 90 Hz. A time for recognition may be assumed to be, for example, about 12 ms to 13 ms depending upon the hardware performance, the kind of recognition algorithm, and a target of recognition.

As indicated in FIG. 3, when imaging I1 is performed by the camera 120, the recognition unit 101 analyzes a captured image (1) acquired by the imaging I1 and performs recognition R1 for recognizing a real object. The virtual object drawing unit 102 then performs drawing W_v1 of a virtual object on the basis of the recognition result of the recognition R1. Since the camera 120 continuously performs imaging, a captured image (3) acquired by imaging I3 is the latest view image at the completion of the drawing W_v1 by the virtual object drawing unit 102, and then the view drawing unit 103 performs drawing W_f1 of the captured image (3). Subsequently, the display processing unit 104 performs display processing O1 for displaying, on the display unit 150, the virtual object drawn by the drawing W_v1 on the basis of the recognition result of the captured image (1) and the latest captured image (3) that is drawn in the drawing W_f1. Specifically, when the latest captured image (3) drawn in the drawing W_f1 is displayed, recognition R2 of the latest captured image (3) is not completed and the virtual object drawn by the drawing W_v1 based on the recognition R1 is stored in a buffer for reading, so that the virtual object drawn by the drawing W_v1 is read and is displayed while being superimposed.

In the display processing O1, a display delay of a view and a display delay of the virtual object may occur as indicated in FIG. 3. The display delay of a view is a displacement between real outside scenery and a displayed view image. Specifically, for example, a time from the imaging I3 to the display (output) of the captured image (3) acquired by the imaging I3. The display delay of the virtual object is a displacement between a view image and the virtual object. Specifically, a time between a time when the imaging I3 is performed to acquire the view image (captured image (3)) and a time when the imaging I1 is performed to acquire the captured image (1) based on the recognition result (recognition R1) that is referenced in the drawing of the virtual object. In other words, the display position or orientation of the virtual object outputted in the display processing O1 is based on the captured image (1) acquired by the imaging I1, and the view image is the captured image (3) acquired by the imaging 3. If the view changes before the drawn virtual object is displayed, a displacement may occur in the relative positional relationship between the position of the real object of the view image and the position of superimposition of the drawn virtual object.

As described above, if the view is covered with the display unit 150 and scenery in a real space does not directly come into sight, a displacement between a real view (outside scenery) and a view image displayed on the camera 120 is hardly noticed in the display delay of the view, which does not interfere with naturalness. In contrast, as described with reference to FIG. 2, even a small display delay of the virtual object tends to be noticeable, leading to unnaturalness. In the example of FIG. 3, the virtual object to be superimposed on the view image has two frames of display delay (two imaging processes).

Thus, the present disclosure proposes the information processor 10 that can reduce a displacement of the relative positional relationship between a real object and a virtual object (that is, a display delay when the virtual object is superimposed on a view) in a video see-through display and perform more suitable display control.

2. TECHNICAL FEATURES

The technical features of the information processor 10 according to the present embodiment will be described below.

<2-1. Reduction in Display Delay Time of Virtual Object>

FIG. 4 is a timing chart indicating the flow of a series of processing to explain display control according to the present embodiment. In the present embodiment, as indicated in FIG. 4, a view image displayed (outputted) in display processing O11 is not the latest (current) captured image (a captured image acquired by the imaging I3) but a past captured image (for example, a captured image acquired by the imaging I2). Specifically, the display processing unit 104 performs control to display a virtual object superimposed on a view image, the virtual object being drawn by the drawing W_v1 on the basis of the recognition result of the recognition R1 in which the captured image (1) acquired by the imaging I1 is analyzed and recognized, the view image being drawn by the drawing W_f1 on the basis of the captured image (2) acquired by the imaging I2. In this case, the display of the virtual object is delayed by one frame relative to the view image, so that the display delay is shorter than that in the example of FIG. 3. In contrast, the display delay time of the view is longer than that in the example of FIG. 3. However, “the display delay of the view” is hardly noticed by a user, so that even if “the display delay of the view” is extended, unnaturalness is reduced by preferentially reducing “the display delay of the virtual object superimposed on the view image,” enabling more suitable display control. Moreover, image deformation or prediction is not performed, thereby reducing the artificiality of a position and orientation when a virtual object is superimposed.

In the example of FIG. 4, imaging is continuously performed after the imaging I3. The subsequent imaging is omitted. Moreover, in display processing immediately before the display processing O11, a captured image that is acquired by imaging performed immediately before the imaging I1 is used as a view image. The captured image is omitted. Furthermore, display processing in the past is also omitted. In the present embodiment, the imaging speed of a view and the frequency of display update can be maintained. The display processing unit 104 uses, as a view image to be displayed, a captured image prior to a current (latest) captured image while keeping the frequency of display update, thereby reducing the display delay of a virtual object to be superimposed on the view image. In the example of FIG. 4, a captured image of an immediately preceding frame is selected as a view image to be displayed. The selection is merely exemplary. A captured image in the second frame prior to the current frame may be selected instead, and the frame of an image to be selected is not particularly limited.

<2-2. Operation Processing>

FIG. 5 is a flowchart indicating an example of a flow of display control performed by the information processor according to the present embodiment. As indicated in FIG. 5, first, the camera 120 of the information processor 10 performs imaging to capture an image of a real space (step S103). If an additional camera is provided for recognition, the camera for recognition also performs imaging to capture an image of the real space in parallel with the camera 120.

The recognition unit 101 of the information processor 10 then analyzes a captured image of the real space and recognizes a real object (step S106). The recognition is performed on the basis of the latest captured image with respect to the start time of the recognition.

Subsequently, the control unit 100 measures a time for the recognition (step S109). In the present embodiment, “time” required for recognition is described as an example of a load of recognition. A time required for recognition means a time from the start of recognition of a captured image in a real space by the virtual object drawing unit 102 to the completion of the recognition. The time required for recognition may vary depending on the feature of a recognition algorithm used for analyzing a captured image and the number of used algorithms. Furthermore, the time required for recognition also varies depending on the target and level of recognition. For example, in the case of an algorithm with a high processing load, e.g., three-dimensional object recognition or bone estimation, a longer time is necessary as compared with two-dimensional object recognition or the like. Moreover, a plurality of recognition algorithms may be used, the recognition algorithms including an algorithm for recognizing a change (difference) from a past captured image. Alternatively, a plurality of recognition algorithms may be used for different objects to be recognized. If a plurality of recognition algorithms are used, a recognition time to be measured is a time until the completion of recognition of all the algorithms.

Subsequently, the control unit 100 determines whether the recognition time is longer than a view drawing time (step S112). The view drawing time means a time from the start of drawing of a view image in the buffer by the virtual object drawing unit 102 to the completion of the drawing. The recognition can be processed in parallel with the view drawing. The control unit 100 may determine whether the recognition is completed after the view drawing time. In an example of a first load of the present disclosure, a recognition time is equal to or shorter than a view drawing time. In an example of a second load greater than the first load of the present disclosure, a recognition time is longer than a view drawing time. In other words, the first load corresponds to the case where a time required for recognition is equal to or shorter than a time for drawing a captured image. The second load corresponds to the case where a time required for recognition is longer than a time for drawing a captured image.

Subsequently, if a recognition time is longer than a view drawing time (that is, if a recognition time corresponds to the second load) (step S112/Yes), the view drawing unit 103 changes an object of view drawing to a previously captured image (step S115). In other words, the view drawing unit 103 selects a past captured image instead of the current (latest) captured image (that is, a through image) as an object of view drawing. Which one of past captured images is to be selected is not particularly limited. For example, the selected image may be a captured image that is acquired at the same timing as a captured image that is used in recognition for drawing a virtual object, or a more newly captured image. Moreover, the imaging time of the past captured image may be within a tolerance from the current time. The tolerance is desirably set at a level where the user does not feel unnaturalness about a difference (the display delay of a view) between real outside scenery and a view image. “A level where the user does not feel unnaturalness” may be set at a predetermined value (the upper limit of the display delay of a view) or may be flexibly changed depending on the situation. The situation indicates a motion range of the user and a change around the user. As the display delay of a view is extended, the display delay of a virtual object can be shortened to reduce unnaturalness. It is desirable to suppress a difference between real outside scenery and a view image from the user in view of safety.

As described above, in a typical video see-through display, the current (latest) captured image is displayed as real-time as possible. In the present embodiment, a displacement of display between a virtual object and a view image is suppressed by delaying view display. In the present specification, first processing for drawing the current (latest) captured image (through image) as a view image is processing with a small delay in view display and is referred to as “low-delay processing.” Furthermore, according to the present embodiment, second processing for drawing a past captured image as a view image and delaying view display is processing with a larger delay in view display than “low-delay processing” and is referred to as “high-delay processing” in the present specification.

In the present embodiment, a time (a time required for recognition) is described as an example of a load of recognition. The present disclosure is not limited thereto. For example, a load of recognition may be based on the kind of recognition algorithm or a frame rate.

Subsequently, the control unit 100 causes the view drawing unit 103 to draw the view image into the view information buffer and causes the virtual object drawing unit 102 to draw the virtual object into the virtual information buffer (step S118). These drawing processes can be performed in parallel.

Subsequently, if a recognition time is equal to or shorter than a view drawing time (that is, if a recognition time corresponds to the first load) (step S112/No), the view drawing unit 103 does not change an object of view drawing. In other words, the view drawing unit 103 draws a captured image at the latest imaging time (that is, a through image) in the view information buffer as usual.

The control unit 100 then causes the display processing unit 104 to read information drawn in each of the buffers and display the information on the display unit 150 (step S121). Specifically, the display processing unit 104 performs control to display the virtual object superimposed on the view image.

The processing of steps S103 to S121 is repeated until the completion of the display control of the display unit 150 (step S124).

The flow of display control according to the present embodiment was specifically described above. The order of processing according to the present disclosure is not limited to the flow of FIG. 5. For example, the imaging in step S103 and the recognition in step S106 may be performed in parallel. At the completion of the recognition, the recognition unit 101 acquires the latest captured image as a subsequent object of recognition and starts recognition.

<2-3. Support for Safety>

The information processor 10 according to the present embodiment performs, optionally depending on the situation, switching from the high-delay processing (second processing) for displaying a past captured image as a view image to the low-delay processing (first processing) for displaying the current (latest) captured image (through image) as a view image, thereby supporting safety.

For example, when the user of the information processor 10 is moving at a specified speed or higher, the information processor 10 performs control to switch the high-delay processing to the low-delay processing in view of safety. Upon switching (switching from a display state of a past captured image to a display state of the current captured image), switching can be achieved with reduced unnaturalness and artificiality by using measures such as the use of a view image deformed over a predetermined number of frames according to the speed (gradually bringing the view image to the current/past captured image). The details will be described later.

Upon switching to the low-delay processing, the information processor 10 may shorten the recognition as much as possible by changing the kind of recognition algorithm to be used or increasing a frame rate. This can reduce the display delay of a virtual object during the low-delay processing. The details will be described later.

When a mobile object is present near the user, the information processor 10 may perform control to switch the high-delay processing to the low-delay processing. If the user is likely to collide with a real object, the information processor 10 may stop the display of a superimposed virtual object (a view image is displayed by the low-delay processing).

The support for each item of safety will be specifically described below.

(1) Switching control between high-delay processing and low-delay processing First, switching control between the high-delay processing and the low-delay processing according to the speed of the user will be described below. The speed of the user means a speed of the user who wears or holds the information processor 10 on the head or the like. The speed may be also referred to as the speed of the information processor 10 (at least the camera 120). In addition to the switching control between the high-delay processing and the low-delay processing, a threshold value is set for stopping the superimposition of a virtual object. The threshold value for stopping the superimposition of a virtual object is set by, for example, a distance. Such a threshold value is also referred to as a superimposition stop line.

FIG. 6 is an explanatory drawing of a superimposition stop line and a judgement criterion element for switching control. As illustrated in FIG. 6, the information processor 10 acquires an own position P, an own speed s, and a distance d from a moving object q on the basis of sensing by the sensor unit 140 and recognition by the recognition unit 101. The moving object q is assumed to be an object present in a real space and an object being relocated (moved). For example, the object is assumed to be a human, a bicycle, an automobile, a self-propelled robot, or a drone.

The information processor 10 sets a superimposition stop line (distance D) for stopping the superimposed display of a virtual object according to, for example, the own speed s. The display position of a virtual object obj may include depth information. The information processor 10 performs non-display control when the display position of the virtual object obj is closer than the superimposition stop line (near the own position P), and performs display control when the display position of the virtual object obj is more remote than the superimposition stop line. Thus, in the example of FIG. 6, a virtual object V-obj1 located more remote than the superimposition stop line is displayed, whereas a virtual object V-obj2 located closer than the superimposition stop line is hidden.

Moreover, the information processor 10 optionally performs control to switch between the high-delay processing and the low-delay processing according to the own speed s and the distance d from the moving object q.

Referring to FIGS. 7 and 8, an example of a flow of switching between low delay/high delay will be described below. Table 1 indicates threshold values specified for the own speed s used in FIGS. 7 and 8.

TABLE 1 Low← →High s1 > s2 > s3 > s4 > s5 > s6

Table 2 indicates threshold values specified for the own position P and the distance d from the moving object q, the own position P and distance d being used in FIGS. 7 and 8. In Table 2, “Near” indicates a short distance from the own position P while “Remote” indicates a long distance from the own position P.

TABLE 2 Near ← →Remote d1 > d2 > d3 >

Table 3 indicates threshold values (the distance D from the own position P) specified for the superimposition stop line used in FIGS. 7 and 8. In Table 3, “Near” indicates a short distance from the own position P while “Remote” indicates a long distance from the own position P. The relationship of d3

TABLE 3 Near← →Remote D4 > D5 > D6 >

Switching to High Delay During Low Delay

FIG. 7 is a flowchart indicating an example of a flow of switching during low delay according to the present embodiment. During the low delay, control is performed with a small delay in view display. In other words, control is performed to draw the current (latest) captured image as a view image and display the image on the display unit 150. FIG. 7 describes switching to high delay in this case.

As indicated in FIG. 7, first, if the own speed s is larger (higher) than s6 (step S203/Yes), the control unit 100 sets the superimposition stop line at D6 (step S206). Subsequently, if the own speed s is larger (higher) than s5 (step S209/Yes), the control unit 100 sets the superimposition stop line at D5 (step S212).

Thereafter, if the own speeds is larger (higher) than s4 (step S215/Yes), the control unit 100 sets the superimposition stop line at D4 (step S218).

The display processing unit 104 performs processing such that a virtual object located closer than (near the user) the set superimposition stop line is displayed without being superimposed. Processing for switching from low delay to high delay according to the own speed s and the distance d from the moving object q will be described below.

If the own speed s is smaller (lower) than s2 (step S221/No), the control unit 100 performs switching to high-delay processing (high-delay setting) according to the conditions.

Specifically, if the moving object q is not present between the user and the superimposition stop line (step S224/No), the control unit 100 sets view drawing at high delay (step S227). In other words, the control unit 100 sets view drawing at a mode for performing delay control on view display according to the present embodiment, in which a past view image is drawn in the buffer. In this case, the following condition is set:

The own speed s

(Note that the moving object q is not present between the user and the superimposition stop line)