Apple Patent | Method and device for detecting a touch between a first object and a second object

Patent: Method and device for detecting a touch between a first object and a second object

Drawings: Click to check drawins

Publication Number: 20210117042

Publication Date: 20210422

Applicant: Apple

Abstract

The present disclosure is related to a method and device for detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object. The method includes providing at least one thermal image of a portion of the second object, determining in at least part of the at least one thermal image a pattern which is indicative of a particular value or range of temperature or a particular value or range of temperature change, and using the determined pattern for detecting a touch between the at least part of the first object and the at least part of the second object.

Claims

1-30. (canceled)

31. A system comprising: a thermographic camera; a display; one or more processors; and one or more computer readable media comprising computer readable code executable by the one or more processors to: receiving, from the thermographic camera, a thermal image of a portion of a first object and a portion of a second object, wherein the thermal image is presented on the display; determine, based on the thermal image, a first temperature for the first object and a second temperature for the second object; detect a touch between the first object and the second object in response to detecting pixels in the thermal image having a third temperature where the first object meets the second object; determine a position on the display associated with the touch; and provide user input in accordance with the determined position.

32. The system of claim 31, wherein the position on the display is determined in accordance with a calibration between the thermographic camera and the display;

33. The system of claim 31, wherein the computer readable code to determine a position on the display further comprises computer readable code to: detect a first display region associated with the position, wherein the display comprises a plurality of regions; determine a first class associated with the first region; and assign the first class to the touch, wherein the user input is further provided in accordance with the first class.

34. The system of claim 33, wherein the plurality of regions comprises a second region associated with a second class, and where in the first class and the second class are associated with different actions when utilized as user input.

35. The system of claim 31, wherein the touch is detected in accordance with the pixels comprising a number of pixels that satisfies a size threshold.

36. The system of claim 31, wherein the user input comprises 3D coordinates associated with the touch and 2D coordinates associated with the position on the display.

37. The system of claim 31, wherein detecting a touch comprises detecting a part of a user touching at least a part of the second object at a place where virtual information is displayed to the user, wherein upon detecting the touch the virtual information is manipulated.

38. The system of claim 31, wherein a human-computer-interface handles a touch detected in the thermal image according to at least one of the following: a position of the touch relative to a real object, relative to a virtual object, depending on a global position of the thermographic camera.

39. The system of claim 31, wherein a human machine interface stores a history of detected touches.

40. The system of claim 31, wherein the first object comprises a portion of a user, the system further comprising computer readable code to: determine whether the portion of the user is associated with a left side or a right side of the user, wherein the user input is provided in accordance with the determined side of the user.

41. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to: receive, from a thermographic camera of a system, a thermal image of a portion of a first object and a portion of a second object, wherein the thermal image is presented on a display of the system; determine, based on the thermal image, a first temperature for the first object and a second temperature for the second object; detect a touch between the first object and the second object in response to detecting pixels in the thermal image having a third temperature where the first object meets the second object; determine a position on the display associated with the touch; and provide user input in accordance with the determined position.

42. The non-transitory computer readable medium of claim 41, wherein the position on the display is determined in accordance with a calibration between the thermographic camera and the display.

43. The non-transitory computer readable medium of claim 41, wherein the computer readable code to determine a position on the display further comprises computer readable code to: detect a first display region associated with the position, wherein the display comprises a plurality of regions; and determine a first class associated with the first region; and assign the first class to the touch, wherein the user input is further provided in accordance with the first class.

44. The non-transitory computer readable medium of claim 43, wherein the plurality of regions comprises a second region associated with a second class, and where in the first class and the second class are associated with different actions when utilized as user input.

45. The non-transitory computer readable medium of claim 41, wherein the touch is detected in accordance with the pixels comprising a number of pixels that satisfies a size threshold.

46. The non-transitory computer readable medium of claim 41, wherein the user input comprises 3D coordinates associated with the touch and 2D coordinates associated with the position on the display.

47. The non-transitory computer readable medium of claim 41, wherein detecting a touch comprises detecting a part of a user touching at least a part of the second object at a place where virtual information is displayed to the user, wherein upon detecting the touch the virtual information is manipulated.

48. A method comprising: at an electronic device with a display and a thermographic camera: receiving, from a thermographic camera of a system, a thermal image of a portion of a first object and a portion of a second object, wherein the thermal image is presented on a display of the system; determining, based on the thermal image, a first temperature for the first object and a second temperature for the second object; detecting a touch between the first object and the second object in response to detecting pixels in the thermal image having a third temperature where the first object meets the second object; determining a position on the display associated with the touch; and providing user input in accordance with the determined position.

49. The method of claim 48, wherein determining a position on the display further comprises: detecting a first display region associated with the position, wherein the display comprises a plurality of regions; determining a first class associated with the first region; and assigning the first class to the touch, wherein the user input is further provided in accordance with the first class.

50. The method of claim 49, wherein the plurality of regions comprises a second region associated with a second class, and where in the first class and the second class are associated with different actions when utilized as user input.

Description

[0001] This application is a continuation-in-part of PCT/EP2014/053017 filed Feb. 17, 2014, which is herein incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

[0002] The present disclosure is related to a method and device for detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object. The disclosure is also related to a computer program product comprising software code sections which are adapted to perform such method.

2. Background Information

[0003] A natural way for humans to interact with (real) objects is to touch them with their hands. For example, in current Augmented Reality (AR) applications, interaction with real and virtual objects usually involves the user’s hand and a screen displaying an image of the real object, instead of interacting with real objects directly. Such screens that allow detecting and localizing touches on their surface are commonly known as touch screens and are nowadays common part of, e.g., smartphones and tablet computers. A current trend is that displays for AR are becoming smaller and/or they move closer to the retina of the user’s eye. This is for example the case for head-mounted displays, and makes using touch screens difficult or even infeasible.

[0004] One possible solution in this case is to use occlusion-based interaction methods, such as described in references PCT Patent Publication No. WO 2013/016104 A1 and “Occlusion Based Interaction Methods for Tangible Augmented Reality Environments”, Lee et al., VRCAI ‘04 Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry, pages 419-426, 2004; (hereinafter “Lee”). In these methods, an interaction event is triggered if a certain area of a real object is occluded from the viewpoint of a camera. Note that the camera’s pose (i.e. position and orientation) with respect to the real object needs to be known to being able to identify such occlusions. This pose can either be determined once offline or continuously during runtime which allows for motion of the real object and/or the camera. Such occlusion-based interaction is, for example, part of the Vuforia SDK under the name “virtual buttons”. Occlusion-based virtual buttons have the following shortcomings: they cannot distinguish if an object (e.g. fingertip) actually touches the virtual button or if it only occludes it and they cannot distinguish if the occlusion (or touch) is caused by a finger(tip) on purpose or by any other object (e.g. a sleeve hem) by accident.

[0005] The following common approaches exist to detect a touch between at least part of a human body and an object. The most common approach is to physically equip the object or the human body (e.g. fingertip) with a sensor capable of sensing touch. This could be anything from a simple mechanical switch to a touch-pad or touch screen. It could, for example, also be based on electrical voltage applied to the body and closing a circuit when touching a real object. The limitation of such kinds of approaches is that they require modifications of the object or the human body.

[0006] A touch can also be detected if the pose of the part of the body, e.g. the hand, is known relative to the object. There are many approaches aiming at tracking the pose of a finger or a hand. These can be based on one or more cameras that sense visible light and/or depth, e.g. using a time-of-flight camera, or active stereo based on infrared structured light. There are also approaches that equip a user’s hand with sensors, e.g. inertial sensors, to sense the pose of the hand. The limitation of all these approaches is that the determined pose of a hand or finger is too inaccurate to reliably tell if a fingertip touches a real object or if it is only very close to it, e.g. 2 mm apart from the object.

[0007] Other approaches; as described in PCT Patent Publication No. WO 2013/016104 A1 and Lee, such as the virtual buttons in the Vuforia SDK mentioned above, do not aim at detecting a touch but at detecting an occlusion resulting in many limitations. Virtual buttons need to have certain size that allows to robustly identify if the button is occluded or not. It would, for example, not be feasible to subdivide an A4 sheet of paper into a grid of 297.times.210 virtual buttons with a size of 1.times.1 mm each. This limits the application of virtual buttons to tasks that do not require precise and continuous positional input, but only discrete button triggers. Furthermore, these virtual buttons need to have a visual appearance different from that of a finger, so that occlusions can be identified. The fact that these methods detect occlusions instead of touches results in another severe limitation, which will be discussed in the following at the example of a number pad. With virtual buttons it is impossible to trigger a button, e.g. No. 5 out of an array of adjacent buttons on a number pad, without triggering any other button before, because the button No. 5 cannot be reached without occluding any of the surrounding buttons. This puts heavy constraints on the layout of virtual buttons. Furthermore, while touching or occluding the button No. 5 on a number pad, the hand will additionally occlude other buttons at the same time. In Lee, the authors propose to solve this issue by only considering the top-left button in case multiple buttons are occluded, but this is a very heuristic and unreliable method.

[0008] In PCT Patent Publication No. WO 2012/039836 A1, a blow tracking user interface system and method is described, wherein embodiments thereof are directed to user interfaces for control of computer systems, and more specifically to user interfaces that track the blowing of a user’s breath to provide control input to a computer program. This blow tracking is done based on thermal infrared imaging.

[0009] A user interface system and method using thermal imaging is described in PCT Patent Publication No. WO 2012/040114 A1. It describes a user interface based on obtaining one or more thermal infrared images of one or more objects with one or more thermographic cameras, analyzing the thermal infrared images, identifying characteristics of the objects from the thermal infrared images and using the characteristics as a control input in the computer program. The objects are users and the characteristics are vital signs.

[0010] U.S. Patent Publication No. US 2011/0050643 A1 discloses a passive infrared sensing user interface and device using the same. A device includes a passive infrared sensor for user interface. When a user places his finger over the infrared sensor, the sensor generates a digital signal indicating the time, position, or movement of the user finger over the sensor. The user finger does not need to touch or press the infrared sensor, but it can be used for touchless user interfaces.

[0011] “Omnitouch: Wearable Multitouch Interaction Everywhere”, Harrison et a;., Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST’11, pages 441-450, 2011 (hereinafter referred to as “Harrison”), uses a depth camera to detect touches between a fingertip and a surface. In a first step fingers are detected in the depth image and the second step then determines if a continuous connection between the depth values of a detected finger and the depth values of a surface exists. If so, the respective finger is considered to touch the surface. This approach has the major limitation that it cannot reliably distinguish between a physical contact between a finger and a surface (i.e. a touch) and proximity between the two (without contact). In Harrison, a fingertip needs to be at least 2 cm apart from a surface such that it can be robustly classified as not touching the surface.

[0012] “HeatWave: Thermal Imaging for Surface User Interaction`, Larson et al., Proc. CHI 2011 (hereinafter referred to as “Larson”) discloses using a thermographic camera to detect touches between fingers and a table-top that is rigidly connected to the thermographic camera. After calibration of the static setup, the method disclosed in Larson performs background subtraction in the thermal image followed by a segmentation of hands and localization of fingertips based on this segmentation. In the next step, a classifier determines for all pixels that were in the vicinity of detected fingertips in the current frame or in at least one of the preceding frames, if the pixel captures heat residual as a result of a touch or not. The employed classifier is based on smoothed temperature, temporal derivative of temperature with respect to time (i.e. multiple thermal images captured at different time are required), and background-subtracted temperature Finally the method fits geometric primitives, such as lines, into the pixels classified as touched pixels accumulated over a number of frames.

[0013] “Dante vision: In-air and touch gesture sensing for natural surface interaction with combined depth and thermal cameras”, Saba, et al., IEEE International Conference on Emerging Signal Processing Applications (ESPA), 2012 (hereinafter “Saba”) discloses segmentation of a hand based on the image of a thermographic camera and subsequently tries to detect touches between the segmented hand a surface based on a depth camera similar to Harrison but using a classifier. Optionally, the approach disclosed in Larson is used to detect touches after the finger is released in order to re-train the depth-based classifier. This approach allows to detect a touch when it happens and not only afterwards.

SUMMARY OF THE INVENTION

[0014] It is an object of the invention to provide a method and device for detecting a touch between a first object and a second object which may be implemented without using a touch screen and does not provide the drawbacks as mentioned above.

[0015] According to an aspect, there is provided a method of detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object, comprising the steps of providing at least one thermal image of a portion of the second object, determining in at least part of the at least one thermal image a pattern which is indicative of a particular value or range of temperature or a particular value or range of temperature change, and using the determined pattern for detecting a touch between the at least part of the first object and the at least part of the second object.

[0016] More particularly, with determining the pattern thermal energy is detected which is transferred from one of the first or second object to the respective other of the first or second object. Advantageously, in this way thermal energy, such as thermal radiation, is detected that is emitted by a surface of one of the first or second object after it was touched by the respective other object (e.g., a human body such as a finger), instead of trying to detect the object (i.e. body or finger) itself as in the prior art.

[0017] For example, the first object is at least part of a human body. The second object may be, in principle, any object in a real environment.

[0018] With the present invention at least one touch between at least part of a human body and at least one object may be determined by detecting radiation in the infrared range emitted by the at least one captured object at those parts of the surface where the at least one touch occurred. For capturing a thermal image, a thermal camera, such as an infrared thermographic camera, may be used.

[0019] The invention, for instance, enables many useful applications in the field of Augmented Reality (AR) and other areas dealing with human computer interfaces, particularly tangible user interfaces.

[0020] Advantageously, with the present invention it can be accurately determined if a surface of a real object was actually touched or only occluded (or approached but not touched). It can distinguish touches or occlusions caused by a human body (that happened on purpose) from touches or occlusions by anything else (that might happen by accident). It is capable of precisely localizing a touch and can identify touch events after they occurred (i.e. even if no camera, computer or whatsoever was present at the time of touch).

[0021] According to an embodiment, the at least part of the second object and the portion of the second object may or may not overlap. The touching or touched part of the second object may not be captured in the at least one thermal image, as it may be occluded by the first object (as e.g. shown in FIG. 3 described in more below). Thus, the term “at least part of the second object” represents the touching or touched part of the second object and the term “portion of the second object” represents the imaged part of the second object. For example, the at least part of the second object could be on the front side of a thin object while the portion of the second object is located on the back side of the thin object. This also becomes evident in more detail below.

[0022] According to an embodiment, the method further comprises determining a position, a size, an orientation, a direction, a trajectory, or a shape of the detected touch in the thermal image.

[0023] According to a further embodiment, the method further comprises providing the detected touch as an input to a machine interface program, wherein the detected touch changes a state in the machine interface program. For example, the machine interface program is part of a human machine interface which may be used, for example, in a mobile device such as a smartphone, a tablet computer, a wearable computer, or a head-mounted device.

[0024] According to an embodiment, determining the pattern comprises determining one or more differences between temperatures measured at different positions in the at least one thermal image.

[0025] According to another embodiment, determining the pattern comprises computing a first or second derivative of temperature in the thermal image with respect to position.

[0026] For example, determining the pattern comprises computing a first or second derivative of temperature in the thermal image with respect to time.

[0027] According to an embodiment, determining the pattern comprises determining a temperature distribution between at least two temperature intervals which are indicative of a respective temperature of the first and second object.

[0028] For example, the method further comprises determining whether a first of the intervals shows a first increase in temperature followed by a second increase, which is steeper than the first increase, and whether a second of the intervals shows a first descent in temperature followed by a second descent, which is less steep than the first descent.

[0029] The method may further comprise calculating a histogram of temperatures in the at least one thermal image and using the histogram as a basis to define at least one of the first and second intervals and an interval between the first and second intervals that is determined for detecting a touch.

[0030] The method may further comprise obtaining a measurement of the temperature of the air or of the environment using a thermometer as a basis to define at least one of the first and second intervals and an interval between the first and second intervals that is determined for detecting a touch.

[0031] The method may further comprise obtaining information on the current weather situation (e.g. from a server, a database, or a distributed web service) as a basis to define at least one of the first and second intervals and an interval between the first and second intervals that is determined for detecting a touch.

[0032] According to an embodiment, determining the pattern comprises determining a temperature distribution of at least one sample line in the at least one thermal image which can have any orientation within the thermal image.

[0033] According to a further embodiment, determining the pattern comprises determining a cluster (such as a blob) in the thermal image which satisfies one or more constraints on its size and/or average temperature.

[0034] According to an embodiment, the method comprises providing a sequence of thermal images which comprises at least two thermal images of a portion of the second object.

[0035] For example, determining the pattern comprises determining a change of temperature between the at least two thermal images and determining whether the change is above a first defined threshold and/or below a second defined threshold.

[0036] According to a further embodiment, determining the pattern comprises determining a derivative of temperature between the at least two thermal images and determining whether the derivative is above a defined first threshold and/or below a second defined threshold.

[0037] According to an embodiment, determining the pattern comprises determining a first change of temperature between the at least two thermal images and a second change of temperature between the at least two thermal images, and using the first and second changes and derivatives of the first and second changes for detecting a touch.

[0038] According to an embodiment, the method further comprises imaging a portion of the second object by a visible light camera and a thermal camera providing the at least one thermal image, providing a first spatial transformation between the visible light camera and the thermal camera, providing a second spatial transformation between the visible light camera and the imaged portion of the second object, concatenating the first and second spatial transformations resulting in a third spatial transformation between a coordinate system of the imaged portion of the second object and a coordinate system of the thermal camera, and determining a position and orientation of the thermal camera in the coordinate system of the imaged portion of the second object based on the third spatial transformation.

[0039] For instance, the method may further comprise determining a position of a touch in the at least one thermal image, wherein the position of the touch in the coordinate system of the imaged portion of the second object is determined by intersecting a ray originating from an origin of the thermal camera transformed to the coordinate system of the imaged portion of the second object and pointing towards the location of the detected touch on the image plane of the thermal camera with a model of the imaged portion of the second object, wherein the intersection is used to trigger a touch event at that position.

[0040] Advantageously, the method is applied as part of a human machine interface in an Augmented Reality application. For example, detecting a touch comprises detecting a part of a user touching at least a part of the second object at a place where virtual information is displayed to the user, wherein upon detecting the touch the virtual information is manipulated.

[0041] The method may be used within an application using a video-see-through setup, an optical-see-through setup, or a projective AR setup. Particularly, the method is used with a hardware setup that does not include a touch screen interface.

[0042] According to another aspect, there is disclosed a device for detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object, comprising a processing device adapted to receive image information of at least one thermal image of a portion of the second object, the processing device configured to determine in at least part of the at least one thermal image a pattern which is indicative of a particular value or range of temperature or a particular value or range of temperature change, and the processing device configured to use the determined pattern for detecting a touch between the at least part of the first object and the at least part of the second object.

[0043] According to an embodiment, the processing device is communicating with a thermal camera for providing the at least one thermal image, wherein at least one of the processing device and the thermal camera is implemented in or associated with a head-mounted display or a projector for performing projector-based Augmented Reality.

[0044] All embodiments and examples described herein with respect to the method can be equally implemented by the processing device being configured (by software and/or hardware) to perform the respective steps. Any used processing device may communicate via a communication network, e.g. via a server computer or a point to point communication, with a thermal camera and/or other components, such as a visible light camera, or with a server computer.

[0045] For example, the processing device (which may be a component or a distributed system) is at least partially comprised in a mobile device which is associated with the thermal camera, and/or in a computer device which is adapted to remotely communicate with the thermal camera, such as a server computer adapted to communicate with the thermal camera or mobile device associated with the thermal camera. The system according to the invention may be comprised in only one of these devices, or may be a distributed system in which one or more processing tasks are distributed and processed by one or more components which are communicating with each other, e.g. by point to point communication or via a network.

[0046] According to another aspect, the invention is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention. Particularly, the software code sections are contained on a computer readable medium which is non-transitory. The software code sections may be loaded into a memory of one or more processing devices as described herein. Any used processing devices may communicate via a communication network, e.g. via a server computer or a point to point communication, as described herein.

[0047] In contrast to Harrison, the present invention can clearly distinguish between the situations that a finger gets very close to a surface and that it actually touches the surface. In the latter case much more thermal energy is transferred from the finger to the surface, which an embodiment of this invention may detect based on thermography. Besides that, the method disclosed in Harrison puts a variety of constraints on the pose of the fingers such that they can be reliably detected. For example, their technique is sensitive to approach angle and it requires fingers to be outstretched. This invention does not require any finger detection and is therefore invariant to the pose of any involved fingers. Further, Harrison does not motivate to use thermal images or temperature information to detect the touch.

[0048] In contrast to Larson, the present invention can detect touches in a dynamic setup, where both the thermographic camera and the object to interact with may freely move. One embodiment of the present invention could determine a touch position (i.e. 3D position) in a coordinate system of the object in addition to the touch position (i.e. 2D image position) in the images of thermographic camera. The pose, i.e. position and orientation, of an object relative to the thermographic camera is determined for multiple captured thermal images individually. Such object tracking approach could be based on information obtained from the thermal image, it could use information captured with a second camera imaging at least part of the object or the thermal camera, or it could be based on any other mechanic, electromagnetic, acoustic, or optical tracking system. The pose of an object relative to the thermographic camera, as obtained from an object tracking approach, allows for determining the 3D position of a touch in the coordinate system of the object if the 2D position in the thermal camera image is known. Object tracking approaches as exploited in an embodiment of this invention, further allow for sampling the temperature of the same point on the object at different points in time even if the camera or object move which for example enables computing the temporal derivative of a point on the object despite motion.

[0049] There are also embodiments of this invention which are based on a single thermal image and therefore are particularly well suited for dynamic scenes and these embodiments may detect touches with less delay than the approach disclosed in Larson which is based on information of many subsequent frames of a thermographic camera which inheritably introduces a delay. Furthermore, an embodiment of this invention is capable of detecting a touch while it happens while Larson can only detect it afterwards. Additionally, the present invention supports objects of any size and shape, including generic 3D objects, while the approach in Larson is limited to planar (and static) objects.

[0050] Another positive aspect of an object tracking approach is that it provides the distance between the camera and any point on the surface of the object in metric units. Thereby a correlation between a physical area or distance on the surface, e.g. 1 cm.sup.2 or 1 cm, and the corresponding area or distance in the image, e.g. 123 pixels.sup.2 or 12 pixels, can be established. Such correlation allows for determining the physical area of a heat residual on the object imaged by the thermal camera. Thereby, an embodiment of this invention is capable of detecting heat residuals which have an area that is similar to the area of a typical fingerprint. In contrast to the method disclosed in Larson, this invention is thereby for example capable of excluding the heat residuals caused by a thenar and to only focus on fingertip-sized heat residuals. The size of the touch may be defined by an area or a distance. The distance may refer to the distance between the two most distant points of the touch area.

[0051] The method disclosed in Saba has the same limitations as Harrison. Most importantly, this approach does not provide robust differentiation between a finger touching a surface and a finger hovering over a surface very closely. As mentioned above, our invention is capable of this distinction in a reliable way and it enables detection of a touch while it happens.

BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Aspects and embodiments of the invention will now be described with respect to the drawings, in which:

[0053] FIG. 1 shows a flowchart of a method according to an embodiment of the invention.

[0054] FIG. 2 shows an example of a thermal image mapped to grayscale for visualization which may be used in connection with the present invention,

[0055] FIG. 3 shows another embodiment of the present invention to determine a touch between two objects based on the thermal energy transferred from one object to the other sensed with infrared thermography,

[0056] FIG. 4 shows a thermal image similar as the one shown in FIG. 2, but which has been discretized and visualized with isolines,

[0057] FIG. 5 shows a sequence of infrared thermal images according to an embodiment of the invention,

[0058] FIG. 6 shows an exemplary embodiment of the invention in which a real object is imaged by a visible light camera and an infrared thermal camera,

[0059] FIG. 7 shows an embodiment of the invention which could be used as a human machine interface in an Augmented Reality application,

[0060] FIG. 8 shows a sequence of infrared thermal images where the first object moves over the surface of the second object during a touch according to an embodiment of the invention,

[0061] FIG. 9 shows two exemplary hardware setups in the context of the invention,

[0062] FIG. 10 shows an embodiment of a human-computer-interface according to an embodiment of the invention

[0063] FIG. 11 shows another embodiment of a human-computer-interface according to another embodiment of the invention.

[0064] Although various embodiments are described in the following with reference to certain components, any other configuration of components, as described herein or evident to the skilled person, can also be used when implementing any of these embodiments.

DETAILED DESCRIPTION

[0065] In the following, embodiments and exemplary scenarios are described, which shall not be construed as limiting the invention.

[0066] In an exemplary scenario, a touch between at least part of a human body and an object results in thermal energy being transferred from the at least part of a human body to the object. This thermal energy then results in the emission of radiation in the infrared range. This can be sensed using an infrared thermographic camera and be used as a human computer interface, in particular to detect a touch and as a result trigger a touch event.

[0067] FIG. 1 shows a flowchart of a method according to an embodiment of the invention. In a first step 101, at least one thermal image (e.g. infrared thermal image) of an object or environment is provided. In a second step 102, radiation resulting from at least one touch between at least part of a human body and the object or environment is automatically detected if present. If radiation resulting from at least one touch could be detected (step 103), at least one touch event is triggered (step 104). Otherwise, the method exits (step 105).

[0068] The present invention is capable of distinguishing multiple situations. The general assumption for this embodiment of the invention is that the real object has a temperature different from the temperature of a human, which is usually around 36.5.degree. C. Only if a part of the human body, for example the fingertip, actually touches a real surface, then it will transfer sufficient thermal energy to the real object at the area where they touch, such that the resulting radiation emitted by the object in the infrared range is clearly measurable with a thermographic camera. Optionally this invention can be implemented, such that only if the entity that touches the real object has a temperature similar to 36.5.degree. C., a touch will be detected. Thermographic cameras are available at low-cost and ubiquitous presently and in the near future, see e.g. FLIR ONE, Personal thermal imaging device for your iPhone5 and iPhone5s, www.flir.com/flirone.

[0069] FIG. 2 shows an example of a thermal image which may be used in connection with the present invention. Particularly, FIG. 2 shows a hand through a thermal image. The hand can be clearly distinguished from the background, because it has a higher temperature. When a fingertip touches a surface 202, it transmits thermal energy to the surface. This becomes visible in the thermal camera image as a warm fingerprint (spot 204) on the surface once the fingertip moved away. FIG. 2 shows a thermal image 201 which is mapped to grayscale for visualization. In this case, the scene (surface) 202 has a lower temperature than a hand 203. There is also visible a spot 204 with an increased temperature relative to the remaining scene, which is indicative of a position where the hand recently touched the scene. A bar 208 visualizes the mapping from temperatures in degrees Celsius to grayscale for better understanding. By detecting such warm fingerprints, it can be identified that a touch between the hand and a surface happened recently. Furthermore the location of this touch can be accurately determined in the image of the thermographic camera. It is further possible to determine a touch at the time it happens by detecting heat that is transferred to the surface close to the touch, see e.g. FIG. 3.

[0070] As described above, an infrared thermographic image allows detecting a touch by a user on arbitrary and unmodified real objects, which can then be used as (tangible) user interaction devices. An exemplary application of this method would, in addition to the thermographic camera, use a visual light camera enabling an Augmented Reality experience as shown in FIG. 6 and FIG. 7.

[0071] For example, a user holds a real object in his or her hands which is localized in the visual light camera image. On this object there are virtual buttons which were printed as gray areas onto the real object in this example. The thermal image of the same scene reveals a warm fingerprint caused by a part of a human body (a finger tip) touching the object. The position of this touch can be localized in the thermal camera image. Because the visual light camera is calibrated with respect to the thermal camera, and the shape, position and orientation of the real object is known with respect to the visual light camera, the position of the touch can be transformed to the coordinate system of the real object.

[0072] Any desired action with respect to the 3D position of this touch event may be performed, such as triggering a virtual button located on the real object at the position of the touch. The touch event can then affect the Augmented Reality application that displays virtual contents spatially registered overlaid on the live image feed of the visible light camera. In the described example, the touch would change the visualization of the button and also change the state of a virtual object which is rigidly attached to the real object and represents a virtual display.

[0073] The present invention can be used in any Augmented Reality application that requires interaction beyond changing the position and orientation of a camera or of a real object. It is particularly useful if the application requires the selection of one or more positions on the surface of a real object. It can be used for video-see-through, optical-see-through, or projective AR setups. The invention makes sense for handheld AR applications, but it is particularly interesting for hardware setups that do not include a touch screen interface, such as a head-mounted display or projector-based AR. It could be used in many different applications, for example all applications based on tracking printed materials such as magazines or books to directly click on advertisements or images for example to start a video. It could also be used in a maintenance application where a worker marks defects in paint work on a car by simply touching them with their fingers for later inspection. The invention enables a very intuitive and tangible way of human machine interaction without the need to modify the real object to interact with and without the need to attach hardware to the user’s hands.

[0074] There are different approaches according to embodiments of the present invention to determine thermal energy an object has as a result of a touch (i.e. direct contact) with an object having different temperature.

[0075] One approach, which enables detection of a touch while the two objects still touch each other is based on heat distribution gradients measured according to the thermal camera image. While the transition between two objects with different temperatures that do not touch is abrupt and therefore has a strong gradient, the transition in the vicinity of a touch, i.e. physical contact between the two objects, becomes smoother resulting in less strong gradients in a larger area. This is further explained in connection with FIG. 3.

……
……
……