HTC Patent | Virtual image display device and object selection method for virtual image thereof

Patent: Virtual image display device and object selection method for virtual image thereof

Patent PDF: 加入映维网会员获取

Publication Number: 20230107586

Publication Date: 2023-04-06

Assignee: Htc Corporation

Abstract

A virtual image display device and an object selection method for virtual image thereof are provided. The object selection method for virtual image includes the following: a virtual image display is enabled to capture image information of a hand of a user; whether an image of an auxiliary tool is present in the image information is determined; position information of the auxiliary tool is generated according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information; the virtual image display is enabled to generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information; and the virtual image display is enabled to display an auxiliary indicator for selecting object according to the position information.

Claims

What is claimed is:

1.An object selection method for virtual image, comprising: enabling a virtual image display to capture image information of a hand of a user; determining whether an image of an auxiliary tool is present in the image information; generating position information of the auxiliary tool according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information; enabling the virtual image display to generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information; and enabling the virtual image display to display an auxiliary indicator for selecting an object according to the position information.

2.The object selection method for virtual image according to claim 1, wherein when the image of the auxiliary tool is not present in the image information further comprises: enabling the virtual image display to send a demand signal to the auxiliary tool; and enabling the auxiliary tool to initiate an inertial measurement action according to the demand signal and to generate the inertial measurement information.

3.The object selection method for virtual image according to claim 1, wherein enabling the virtual image display to display the auxiliary indicator for selecting the object according to the position information comprises: enabling the virtual image display to display the auxiliary indicator in a virtual image at a position corresponding to the hand of the user.

4.The object selection method for virtual image according to claim 1 further comprising: enabling the auxiliary tool to send a selection command to the virtual image display; and enabling the virtual image display to perform a selection action on a virtual object image pointed to by the auxiliary indicator according to the selection command.

5.A virtual image display device, comprising: a virtual image display; and an auxiliary tool electrically coupled to the virtual image display, wherein the virtual image display is configured to: capture image information of a hand of a user; determine whether an image of the auxiliary tool is present in the image information; generate position information of the auxiliary tool according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information; generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information; and display an auxiliary indicator for selecting an object according to the position information.

6.The virtual image display device according to claim 5, wherein when the image of the auxiliary tool is not present in the image information, the virtual image display sends a demand signal to the auxiliary tool, and the auxiliary tool initiates an inertial measurement action according to the demand signal and generates the inertial measurement information.

7.The virtual image display device according to claim 6, wherein the auxiliary tool comprises: a button configured to generate a selection command, wherein the auxiliary tool sends the selection command to the virtual image display, and the virtual image display performs a selection action on a virtual object image pointed to by the auxiliary indicator according to the selection command.

8.The virtual image display device according to claim 7, wherein the auxiliary tool further comprises: an inertial measurement device activated according to the demand signal to generate the inertial measurement information; and a communication interface configured to transfer information with the virtual image display.

9.The virtual image display device according to claim 5, wherein the virtual image display displays the auxiliary indicator in a virtual image at a position corresponding to the hand of the user.

10.The virtual image display device according to claim 5, wherein the virtual image display comprises: a camcorder configured to capture the image information of the hand of the user; a controller coupled to the camcorder to generate the position information; a display coupled to the controller to display the auxiliary indicator; and a communication interface coupled to the controller to transfer information with the auxiliary tool.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of U.S. Provisional Application Serial No. 63/249,543, filed on Sep. 28, 2021. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.

BACKGROUNDTechnical Field

The disclosure relates to a virtual image display device and an object selection method for virtual image, in particular to a virtual image display device providing an auxiliary tool and an object selection method for virtual image by means of the auxiliary tool.

Description of Related Art

With the advancement of electronic technology, virtual image displays that allow for virtual reality and augmented reality are becoming more and more mainstream. In today’s technology, to enhance the user’s experience in the virtual image world, it can also be paired with the user’s accessories to provide a mechanism for the user to interact with the images of objects displayed in the virtual world.

In particular, when the user selects an object in the virtual environment, the human body is accustomed to use the index finger to point to the object to make the selection as a direct response. However, such an action may cause the hand to obscure the user’s worn accessories, and the object pointing and selection action cannot be performed effectively. As a result, a conventional virtual image display device may limit the user to apply specific but uncomfortable movements for object selection, reducing the user’s comfort and experience in the virtual reality experience.

SUMMARY

The disclosure provides a virtual image display device and an object selection method for virtual image, capable of effectively enhancing a user’s experience.

The object selection method for virtual image of the disclosure includes the following: a virtual image display is enabled to capture image information of a hand of a user; whether an image of an auxiliary tool is present in the image information is determined; position information of the auxiliary tool is generated according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information; the virtual image display is enabled to generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information; and the virtual image display is enabled to display an auxiliary indicator for selecting an object according to the position information.

The virtual image display device of the disclosure includes a virtual image display and an auxiliary tool. The auxiliary tool is electrically coupled to the virtual image display. The virtual image display is configured to: capture image information of a hand of a user; determine whether an image of the auxiliary tool is present in the image information; generate position information of the auxiliary tool according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information; generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information; and display an auxiliary indicator for selecting an object according to the position information.

Based on the above, the virtual image display of the disclosure displays the auxiliary indicator for selecting the object by capturing the image of the auxiliary tool, and displays the auxiliary indicator for selecting the object according to the inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not available. In this way, the user may perform a selection action on a virtual object without the restriction of selection pose, which may effectively enhance the experience of virtual reality images.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a flowchart illustrating an object selection method for virtual image according to an embodiment of the disclosure.

FIGS. 2A to 2C are schematic diagrams illustrating implementation of an object selection action according to embodiments of the disclosure.

FIG. 3 is a flowchart illustrating an object selection action for virtual image according to an embodiment of the disclosure.

FIG. 4 is a schematic diagram of a virtual image display device according to an embodiment of the disclosure.

DESCRIPTION OF THE EMBODIMENTS

Referring to FIG. 1, FIG. 1 is a flowchart illustrating an object selection method for virtual image according to an embodiment of the disclosure. The object selection method for virtual image according to this embodiment is applicable to virtual display devices, such as head-mounted display devices for generating virtual reality or augmented reality. In step S110, a virtual image display is enabled to capture image information of a hand of a user. Also, in step S120, whether an image of an auxiliary tool is present in the image information is determined. Here, the auxiliary tool may be worn on the hand of the user and used as an auxiliary prop for the user to perform a selection action on a virtual object. According to this embodiment, the auxiliary tool may be a ring and be worn on the user’s finger.

In step S130, according to the determination action of step S120, the virtual image display is enabled to generate position information of the auxiliary tool according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information. In contrast, in step S140, the virtual image display is enabled to generate the position information of the auxiliary tool according to inertial measurement information of the auxiliary tool when the image of the auxiliary tool is not present in the image information. Also, in step S150, the virtual image display is enabled to display an auxiliary indicator for selecting an object according to the position information. According to this embodiment, the auxiliary indicator may be a beam emitted by the auxiliary tool worn on the hand of the user to provide the user with a beam to point to a selected virtual object and to perform the selection action on the virtual object.

Referring to FIG. 1 and FIG. 2A to FIG. 2C, FIG. 2A to FIG. 2C are schematic diagrams illustrating implementation of an object selection action according to embodiments of the disclosure. In FIG. 2A, corresponding to step S110 to step S130, the user wears a virtual image display device 110 as a head-mounted display, and captures image information of a hand HD through the virtual image display device 110. In addition, the virtual image display device 110 generates position information of an auxiliary tool 120 by analyzing the image information of the hand HD according to an image of the auxiliary tool 120 on the image information of the hand HD. Further, corresponding to step S150, the virtual image display device 110 may display an auxiliary indicator LB for selecting an object OBJ according to the position information of the auxiliary tool 120. The object OBJ may be a virtual object in the virtual image.

Next, referring to FIG. 2B, when the hand HD (e.g. palm or fist) of the user blocks the auxiliary tool 120 due to a change in the user’s hand posture, and the virtual image display 110 cannot capture the image of the auxiliary tool 120, i.e., the image of the hand HD of the user captured by the image display 110 cannot be determined to contain the image of the auxiliary tool 120, the virtual image display 110 may send a demand signal RQS to the auxiliary tool 120 as shown in FIG. 2C. At this time, when the auxiliary tool 120 receives the demand signal RQS, the auxiliary tool 120 may activate an internal inertial measurement unit (IMU) to perform an inertial measurement action, and transfer inertial measurement information IMF generated during the inertial measurement action to the virtual image display 110. In this way, the virtual image display device 110 may estimate the position information of the auxiliary tool 120 on the hand HD of the user according to the inertial measurement information IMF, and further display the auxiliary indicator LB according to the obtained position information. The user may then apply the auxiliary indicator LB to point to a desired object and perform an object selection action.

That is to say, according to this embodiment of the disclosure, the user may effectively perform the object selection action without being restricted by posture. Thus, the user may, for example, use an index finger to point to the selected object to perform the object selection action under the premise of the human body’s habitual movement, effectively enhancing the experience in the virtual reality (or augmented reality) environment.

According to this embodiment, the virtual image display device 110 may continuously detect whether the image of the auxiliary tool 120 can be effectively interpreted in the image of the hand HD of the user. Once the image of the hand HD of the user has a valid image of the auxiliary tool 120, the virtual image display 110 may stop sending the demand signal RQS and cause the auxiliary tool 120 to stop performing the inertial measurement action to save unnecessary power consumption.

Referring to FIG. 3, FIG. 3 is a flowchart illustrating an object selection action for virtual image according to an embodiment of the disclosure. In step S310, through a camcorder (e.g., a camera) of the head-mounted display (the virtual image display device), the head-mounted display device may recognize a position of a finger of the user and display a beam (the auxiliary indicator) sent by the finger. The head-mounted display may determine the position of the finger of the user according to an image of the finger of the user captured by the camcorder. In step S320, the head-mounted display is used to determine whether its camcorder can capture the ring on the finger of the user. When the camcorder of the head-mounted display can capture the ring on the finger of the user, step S330 can be performed. In step S330, the head-mounted display may continuously display the beam sent by the finger, and provide the user with the beam to point to a virtual object. In contrast, if a determination result of step S320 is no, step S340 can be performed. In step S340, the head-mounted display may receive the inertial measurement information from the ring to estimate a movement of the finger (and the ring) and continuously generate the beam emitted from an estimated position of the finger.

In step S350, when the user points the beam to the virtual object and presses a button on the ring, the virtual object pointed to by the beam is a selected object.

Referring to FIG. 4, FIG. 4 is a schematic diagram of a virtual image display device according to an embodiment of the disclosure. A virtual image display device 400 includes a virtual image display 410 and an auxiliary tool 420. The virtual image display 410 and the auxiliary tool 420 are electrically coupled to each other. The virtual image display 410 includes a camcorder 411, a controller 412, a display 413, and a communication interface 414. The auxiliary tool 420 includes a button 421, an inertial measuring device 422, and a communication interface 423. The camcorder 411 may be configured to capture motion images or still images.

According to this embodiment, the camcorder 411 is configured to capture the image information of the hand of the user. The controller 412 may analyze the image information of the hand of the user and determine whether the image of the auxiliary tool is present in the image information. Moreover, the controller 412 may generate the position information of the auxiliary tool according to the image of the auxiliary tool when the image of the auxiliary tool is present in the image information. Next, the controller 412 may drive the display 413 to display an auxiliary indicator at a position corresponding to the auxiliary tool according to the position information.

On the other hand, when the controller 412 determines that the image of the auxiliary tool is not present in the image information, then the demand signal RQS may be sent through a communication interface 414. The auxiliary tool 420 may receive the demand signal RQS through a communication interface 423. In addition, the auxiliary tool 420 may activate the inertial measurement device 422 according to the demand signal RQS to perform the inertial measurement action, and transfer the inertial measurement information IMF obtained by the inertial measurement action to the virtual image display 410 through the communication interface 423.

The virtual image display 410 may receive the inertial measurement information IMF through the communication interface 414. The controller 412 may estimate the position information of the auxiliary tool according to the inertial measurement information IMF, and may drive the display 413 to display an auxiliary indicator at the position corresponding to the auxiliary tool according to the estimated position information.

On the other hand, the button 421 on the auxiliary tool 420 may serve as a medium for sending a command signal. According to this embodiment, when the user decides on the object to be selected, the user may send a command signal to the virtual image display 410 by pressing the button 421. The controller 412 of the virtual image display 410 may perform the selection action on a corresponding object according to the command signal and the position of the auxiliary indicator.

Regarding the details of the hardware architecture, the display 413 may be any form of display without any special restrictions. The communication interfaces 414 and 423 may be wired or wireless signal transmission interface circuits in any format. The inertial measurement device 422 may be any form of inertial measurement unit sensor known to those skilled in the art. In addition, the controller 412 may be a processor with computing capabilities. Alternatively, the controller 412 may be a hardware circuit designed by means of a hardware description language (HDL) or any other digital circuit design method well known to those skilled in the art, and implemented by means of a field programmable gate array (FPGA), Complex Programmable Logic Device (CPLD) or Application-specific Integrated Circuit (ASIC).

In summary, the virtual image display of the disclosure uses the determination of whether the auxiliary tool on the hand of the user can be effectively captured to obtain the position information of the auxiliary tool according to the image of the auxiliary tool, or the position information of the auxiliary tool according to the inertial measurement information transferred by the auxiliary tool. Further, the virtual image display may display the auxiliary indicator according to the position information of the auxiliary tool, which may be used as a basis for object selection by the user. In this way, the virtual image display does not restrict the user’s movements during object selection, which enhances the comfort and experience of the user’s movements in the virtual environment.