Ultraleap Patent | Ultrahapticons

Patent: Ultrahapticons

Drawings: Click to check drawins

Publication Number: 20220083142

Publication Date: 20220317

Applicants: Ultraleap

Abstract

Described herein are "Ultrahapticons," which are a set of tangible and recognizable mid-air haptic icons that have been derived from research study participants' metaphorical associations with car infotainment features. In line with semiotic theory (the study of signs), data from the study was analyzed to identify key characteristics that when realized in mid-air haptic form, would enable a user to "feel" the feature they are interacting with. Their use is not limited to an automotive context, they can be instrumented to any application that exhibits the same feature functionality i.e. home entertainment system, laptop UI's, digital communication, Extended Reality (XR), and the like.

Claims

1. A method comprising: detecting movement of a human hand issuing an activation command for an automotive function; to confirm the activation command, creating tactile sensations on a target on a human hand by using a phased array of ultrasound transducers to transmit ultrasound waves to exert an acoustic radiation force on the human skin in mid-air; adjusting a phase emitted by each ultrasound transducer such that the ultrasound waves arrive concurrently at the target in order to maximize the acoustic radiation force exerted; wherein the maximize the acoustic radiation force exerted is directed at least one finger of the human hand by adjusting threshold value, frequency, and pattern propagation in view of mechanoreceptor density of the human hand.

2. The method as in claim 1, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in single focal point that travels along a path rendered at around a 35-degree angle from right to left in three sequential locations going up an index finger towards the tip.

3. The method as in claim 2, wherein the at least one finger is two adjacent fingers.

4. The method as in claim 3, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in pulsing sensation at fingertip ends of the two adjacent fingers.

5. The method as in claim 3, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in a two bar sensation that starts in between a middle finger and an index finger that increases in size as it travels up the index finger and the middle finger.

6. The method as in claim 3, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in focal points jumping randomly from one location on one finger to another on another finger with equal intervals.

7. The method as in claim 3, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in: (1) a double bar effect that accelerates up an index finger and a middle finger and decelerates as it reaches fingertips where it pauses momentarily; (2) the double bar effect accelerates down the human hand and then decelerates as it reaches a bottom of a palm where the double bar effect pauses momentarily again; and (3) the double bar effect slides back to its original position.

8. A method comprising: detecting movement of a human hand issuing an activation command for an automotive function; to confirm the activation command, creating tactile sensations on a target on a human hand by using a phased array of ultrasound transducers to transmit ultrasound waves to exert an acoustic radiation force on the human skin in mid-air; adjusting a phase emitted by each ultrasound transducer such that the ultrasound waves arrive concurrently at the target in order to maximize the acoustic radiation force exerted; wherein the maximize the acoustic radiation force exerted is directed to a thumb of the human hand by adjusting threshold value, frequency, and pattern propagation in view of mechanoreceptor density of the human hand.

9. The method as in claim 8, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in a solid double bar effect actuating on the thumb.

10. The method as in claim 8, wherein adjusting the threshold value, the frequency, and the pattern propagation in view of the mechanoreceptor density of the human hand results in: (1) a double bar haptic sensation at tips of an index finger and a middle finger; (2) the double bar haptic sensation accelerates down the index finger and the middle finger and decelerates as it reaches a base of a thumb where it pauses momentarily; (3) the double bar haptic sensation then accelerates up towards the thumb where it decelerates at a thump tip and pauses momentarily; and (4) the double bar haptic sensation accelerates down to a base of the thumb, pauses momentarily and then accelerates back up the index finder and the middle finger and then decelerates.

Description

PRIOR APPLICATIONS

[0001] This application claims the benefit of the following application, which is incorporated by references in its entirety:

[0002] Serial No. 63/079,708, filed Sep. 17, 2020.

FIELD OF THE DISCLOSURE

[0003] The present disclosure relates generally to improved techniques in establishing useful hand gestures within ultrasonic fields to activate features.

BACKGROUND

[0004] There are situations when receiving haptic feedback before touching the surface would be beneficial. These include when vision of the display is restricted, such as while driving, and when the user doesn't want to touch the device, such as when their hands are dirty. Providing feedback above the surface would also allow for an additional information channel alongside the visual.

[0005] A mid-air haptic feedback system creates tactile sensations in the air. One way to create mid-air haptic feedback is using ultrasound. A phased array of ultrasonic transducers is used to exert an acoustic radiation force on a target. This continuous distribution of sound energy, which will be referred to herein as an "acoustic field", is useful for a range of applications, including haptic feedback.

[0006] It is known to control an acoustic field by defining one or more control points in a space within which the acoustic field may exist. Each control point is assigned an amplitude value equating to a desired amplitude of the acoustic field at the control point. Transducers are then controlled to create an acoustic field exhibiting the desired amplitude at each of the control points.

[0007] Tactile sensations on human skin can be created by using a phased array of ultrasound transducers to exert an acoustic radiation force on a target in mid-air. Ultrasound waves are transmitted by the transducers, with the phase emitted by each transducer adjusted such that the waves arrive concurrently at the target point in order to maximize the acoustic radiation force exerted.

[0008] By defining one or more control points in space, the acoustic field can be controlled. Each point can be assigned a value equating to a desired amplitude at the control point. A physical set of transducers can then be controlled to create an acoustic field exhibiting the desired amplitude at the control points.

[0009] As a result, mid-air gesture interaction in the automotive domain has been explored to counteract some drawbacks associated with increasingly ubiquitous touchscreen interfaces (e.g. added complexity and increased eyes-off-the-road time) yet remains a relatively fledgling modality. Naturally, gestures have their own limitations, namely potential cultural semantic nuances, the learning associated with more intricate gestures, and a lack of a sense of agency. Focused ultrasound can be used to create haptic sensations without requiring physical contact which creates the opportunity to bind tangible sensations to mid-air gestures; thus, haptic sensations can provide confirmatory cues thereby increasing one's feeling of control over their gesture. The advent of this technology also provides an alternative to auditory feedback as the latter could plausibly displease vehicle occupants by interrupting conversations and audible media. Furthermore, mid-air haptic icons could allow more expressivity than previous haptic technologies as result of information encoded via novel spatial and temporal interplay. This could provide higher resolution in relaying feature semantics and therefore reduce the onus on having intuitive gestures, learning the meaning of haptic signals or visually attending to a touchscreen for information transfers. From a user experience standpoint, the refinement of mid-air haptics for an automotive gesture interface could also improve user engagement and aesthetic appeal as seen in applications in public digital signage.

[0010] Preliminary investigations have compared mid-air haptic gesture (MAHG) interfaces with industry trending touchscreen solutions, and have notionally combined existing gesture sets with stock haptic sensations. These studies demonstrated clear benefits associated with MAHG interfaces, such as reduced eyes-off-the-road times and increased task performance. However, beyond their scope, was using semiotic and psychophysical principles to ground original holistic interaction designs by mapping gestures and haptics with specific tasks as well as optimizing haptic intensity, transience, pattern, location on the hand etc. All of these factors are likely to play a key role in augmenting the opportunities that mid-air haptics present, therefore, the aim of this ongoing program of research is to build and evaluate an exemplar set of robust, function-associated haptic gestures--based on human-centered design--to support an in-vehicle infotainment system (IVIS).

[0011] Current automotive mid-air gesture interfaces provide only visual and audible feedback as confirmation of a successfully executed interaction with an IVIS. Visual feedback presents the complication of providing potential for additional driver distraction from the roadway which is the third most common cause of car crashes; this is one of the primary problems that gesture interfaces aim to solve. Audible feedback is a reasonable alternative to this problem; however, it can be intrusive to the driving experience by interrupting music, radio, conversations or even sleeping occupants.

[0012] Using an arbitrary mid-air haptic sensation that actuates onto the driver's hand after they have performed an appropriate gesture/hand-pose can let the driver know their input has been acknowledged by the system which improves the driver's sense of agency (control) in their interaction. However, they still cannot be absolutely certain that the system has correctly distinguished their performed gesture from others in a gesture set without this information relayed through visual or auditory feedback. The premise behind this invention is therefore to transfer this information through the haptic channel using function-associated and user-centered mid-air haptic icons or "Ultrahapticons". In this way, the driver can recognize the sensation for the intended infotainment feature and be confident they have made the right selection without removing their eyes from the road. Conversely if they recognize an Ultrahapticon that does not represent the feature they intended to select, they know there has been a false-positive in the system and they can perform their gesture again without affecting the system status. Additionally, Ultrahapticons hold potential for enabling an improved learning curve of the gesture set; this is a barrier to user acceptance of the interaction. Unfamiliarity with a novel interaction paradigm has also been linked to detrimental lane-keeping ability.

[0013] There have been a few research efforts to explore the augmentation of gestures with mid-air haptic feedback in the automotive sector as well as studies to investigate the human's ability to detect and recognize mid-air haptic sensations. Orestis Georgiou, Valerio Biscione, Adam Harwood, Daniel Griffiths, Marcello Giordano, Ben Long, and Tom Carter. 2017. Haptic In-vehicle gesture controls. 233-238. DOI: https://doi.org/10.1145/3131726.3133045 (Georgiou) demonstrated a mid-air haptic gesture (MAHG) interface in this 2017 conference paper. Their interface gave an example of how mid-air haptic sensations can be assigned to match the behavior of the gestural actions. This prototype demonstrated a proof-of-concept that mid-air haptic sensations can be mapped onto a driver's gesture in real-time, however, the sensations are intended to represent the gestures and not the features themselves.

[0014] David R. Large, Kyle Harrington, Gary Burnett, and Orestis Georgiou. 2019. Feel the noise: Mid-air ultrasound haptics as a novel human-vehicle interaction paradigm. Appl. Ergon. (2019). DOI: https://doi.org/10.1016/j.apergo.2019.102909 (Large) conducted a study where mid-air haptic buttons are seemingly suspended above the gear shifter in a driving simulator. The driver can "tap" these buttons with their palm to select and then move their palm sideways to adjust. During the adjustment interaction the driver feels haptic clicks on the palm to represent increments. This study aimed at evaluating the human factors issues when gestures are augmented with haptics in terms of driver safety and user experience, they found there were genuine advantages over in-vehicle touchscreen and non-haptic gesture input methods. This study provides evidence for the utility of mid-air haptics from an automotive human factors stand-point however the interface was function agnostic and the tasks were purely arbitrary.

[0015] Gozel Shaken, John H. Williamson, and Stephen Brewster. 2018. May the force be with you: Ultrasound haptic feedback for mid-air gesture interaction in cars. Proc.--10th Int. ACM Conf. Automot. User Interfaces Interact. Veh. Appl. AutomotiveUI 2018 July (2018), 1-10. DOI: https://doi.org/10.1145/3239060.3239081 (Shaken) ran an experiment to establish the effects of combining 3 forms of feedback including peripheral visual, auditory and mid-air haptics. The sensations they used followed similar logic to the Georgiou et al prototype in that they mimicked the behavior and nature of the gesture and not the feature itself i.e. a gesture performed with a clockwise circular motion to increase volume was paired with a haptic that displayed a clockwise circular motion. In this study the participants were not always able to feel the sensations or distinguish between them which caused them to ignore the mid-air haptics entirely in some cases.

[0016] Davide Rocchesso, Francesco Saverio Cannizzaro, Giovanni Capizzi, and Francesco Landolina. 2019. Accessing and selecting menu items by in-air touch. (2019), 1-9. DOI: https://doi.org/10.1145.335195.3352053 (Rocchesso) demonstrate a methodology for using arbitrary shapes as icons for different items in a multi-level menu. They show that certain shapes are easily confused with one another and should not be included coincidentally in a menu design. The icons used in this paper were arbitrary with no direct application for a specific use as this was not the intention of the research.

SUMMARY

[0017] Ultrahapticons are a set of tangible and recognizable mid-air haptic icons that have been derived from research study participants' metaphorical associations with car infotainment features by detecting movement of a human hand to operate an automotive function. In line with semiotic theory (the study of signs), data from the study was analyzed to identify key characteristics that when realized in mid-air haptic form, would enable a user to "feel" the feature they are interacting with. Their use is not limited to an automotive context, they can be instrumented to any application that exhibits the same feature functionality i.e. home entertainment system, laptop UI's, digital communication, Extended Reality (XR) etc.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, serve to further illustrate embodiments of concepts that include the claimed invention and explain various principles and advantages of those embodiments.

[0019] FIGS. 1-3 show Ultrahapticons for telephone calls.

[0020] FIGS. 4-5 show Ultrahapticons for audio.

[0021] FIGS. 6-8 show Ultrahapticons for cabin temperature.

[0022] FIGS. 9-10 show Ultrahapticons for seat temperature.

[0023] FIG. 11 shows Ultrahapticons for fan speed.

[0024] FIGS. 12-14 show Ultrahapticons for navigation.

[0025] FIGS. 15-16 show Ultrahapticons for a home screen.

[0026] FIG. 17 shows semiotic components drawn from fan speed sketches.

[0027] FIG. 18 shows a diagram of Ultrahapticon concepts.

[0028] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.

[0029] The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent.

DETAILED DESCRIPTION

I. Variable Haptic Sensitivity within the Hand

[0030] Various parts of the hand are susceptible to haptic effects in different ways. For example:

[0031] 1. Threshold Value. The fingertip has a lower threshold value than the palm. Since the maximum output of our device is also limited, that means that there will be more values of intensity that can be discriminated from within the fingertips.

[0032] 2. Frequency. The frequency range relevant to touch (0-500 Hz) is common through the body. But frequency resonance is achieved at different value of frequency through the hand. This means that greater displacement, and therefore greater perceived strength, can be achieved by designing Ultrahapticons with the right frequency for the right body part.

[0033] 3. Mechanoreceptor Density. The density of each mechanoreceptor through the hand is not constant. Discrimination between waveform will be greater at the fingertips.

[0034] 4. Pattern Propagation. Surface wave propagates through the skin and can affect positively or negatively our perception of mid-air haptics (e.g., speed optimization). This is specifically true for the palm, which is a large continuous skin part, but it is less true on a finger where skin parts are interrupted by the joints crease. It doesn't apply between fingers since this involves different skin parts. In addition, 2-point discrimination threshold (the ability to perceive two points as distinct and not one) is higher on the palm than on the fingertips. This means that multiple patterns, such as parallel lines or concentric circles, cannot be simultaneously render on the palm without being confused as one large fuzzy buzz. However, multiple patterns can be rendered on bones of a same finger.

II. Suite of Ultrahapticons

[0035] FIGS. 1-16 show 17 sensations that were generated for 7 infotainment features in a participatory design process (without limitation, the features shown in FIG. 13 can also be performed in reverse). The categories of features for which Ultrahapticons have been designed include: (a) telephone calls; (b) audio; (c) cabin temperature; (d) seat temperature; (e) fan speed; (f) navigation; and (g) home screen (i.e., a user interface landing page). Although specific time and space parameters are described herein, other parameters may similarly be operative to produce similar effects.

[0036] In addition, Ultrahapticons may be classified by where on the hand the effect is felt.

[0037] A. Ultrahapticons for Telephone Calls

[0038] FIG. 1 shows a sequence 100 for a metaphor: "Coiled Wire". This Ultrahapticon intends to match with the metaphor of coiling the wire of a traditional telephone around the finger. The user receives a single focal point (dark dot) 110 that travels along a path (dark lines) rendered at around a 35-degree angle from right to left in three sequential locations 120, 130, 410 going up the index finger towards the tip.

[0039] This is a "single finger only" Ultrahapticon.

[0040] FIG. 2 shows a sequence 200 for a metaphor: "Bouncing Handset". This Ultrahapticon is intended to emulate the handset of an old telephone, bouncing in its cradle as it rings, the focal rings are supposed to represent the speaker and microphone sections of the handset. The user receives a focal ring 210 "bouncing" between 4 locations 220, 230, 240, 250 to the temporal rhythm of a retro telephone ring. The full sensation consists of a 1 second loop of 6 taps over the 4 locations (with 2 taps at the final location-base of thumb). After the first 3 taps there is a small pause before the next 3 follow.

[0041] This is a "palm only" Ultrahapticon.

[0042] FIG. 3 shows a sequence 300 for a metaphor: "Rotary Dial". This Ultrahapticon aims to emulate the holes of a rotary dial on an old-fashioned telephone. When accepting an incoming phone call, the user shall receive 5 sequential, equally spaced haptic circles 310, 320, 330, 340, 350 that are rendered from the bottom left of the palm in a semi-circular path to the base of the little finger knuckle; each haptic circle is sequentially stronger intensity than the last. If the user were rejecting an incoming phone call, they would receive the above sensation however this time it would be flipped 180 so that the sensation starts at the base of the little finger, follows the medial edge of the palm and ends at the base of the thumb. The sequence of intensity is also reversed so that each haptic circle is 0.1 less than the previous intensity, i.e., 1-0.6.

[0043] This is a "palm only" Ultrahapticon.

[0044] B. Ultrahapticons for Audio

[0045] FIG. 4 shows a sequence 400 for a metaphor: "Bass Speaker". For this Ultrahapticon, the driver receives a continuous pulsing sensation 410, 420, 430, 440, 450 at the ends of the fingertips that intends to resemble the users' metaphor of bass emanating from a speaker.

[0046] This is a "two finger only" Ultrahapticon.

[0047] FIG. 5 shows a sequence 500 for a metaphor: "Sound Waves". The intention with this Ultrahapticon is to signify the users' metaphor of sound waves. The user receives a small two bar sensation that starts in between the middle and index finger knuckle 510 that increases in size as it travels up the fingers 520, 530. The small two bar sensation then resumes in between the middle and index finger knuckle 540.

[0048] This is a "two finger only" Ultrahapticon.

[0049] C. Ultrahapticons for Cabin Temperature

[0050] FIG. 6 shows a sequence 600 for a metaphor: "Flames". The user receives a continuous sensation of focal points that intend to represent the metaphor of flames licking the index and middle fingers 610, 620, 630, 640, 650, 660 to signify the hot aspect of temperature. The focal points will jump randomly from one location on one finger to another on the other finger with equal intervals. The locations will be dictated by the 3 joints on the fingers.

[0051] This is a "two finger only" Ultrahapticon.

[0052] FIG. 7 shows a sequence 700 for a metaphor: "Ice". The user shall receive a solid double bar on the thumb actuating at 4 pulses per second (BPS) 710, 720, 730, 740, 750. This Ultrahapticon represents the metaphor of ice in order to signify the cool aspect of temperature and can be used in conjunction with Flames to create a representation of a temperature scale.

[0053] This is a "thumb only" Ultrahapticon.

[0054] FIG. 8 shows a sequence 800 for a metaphor "Thermometer". This Ultrahapticon aims to convey the rising and falling mercury in the metaphor of a thermometer and subsequently represent temperature. The user feels a "double bar" haptic sensation, initially, halfway up the palm; this "double bar" then accelerates up the index and middle fingers and decelerates as it reaches the fingertips where it pauses momentarily 810. Once at the fingertips, the double bar then accelerates all the way down the hand and then decelerates as it reaches the bottom of the palm where it pauses momentarily again 820. After this, the double bar slides back to the original position. Within this haptic animation there will be notches at, for example, 6 equidistant locations in the travel envelope. Thus, the user will feel a drop in intensity between these locations to accentuate the metaphor of a thermometer's increments. After the initial animation, to indicate increasing temperature, the double bar starts focused on the middle of the hand then it will travel to the index and middle fingertips over 2 seconds. Conversely, to indicate a decrease in temperature, the double bar will travel down from the middle of the hand to the bottom of the palm over 2 seconds.

[0055] This is a "palm and two finger" Ultrahapticon.

[0056] D. Ultrahapticons for Seat Temperature

[0057] FIG. 9 shows a sequence 900 for a metaphor: "Seat Profile". This Ultrahapticon aims to convey the profile of a seat that is prevalent metaphor for the seat heater and cooler. The user will feel a "double bar" haptic sensation, initially, at the tips of the index and middle fingers. This "double bar" then accelerates down the index and middle fingers and decelerates as it reaches the base of the thumb/bottom of palm where it where it pauses momentarily 910. Once at the base of the thumb, the double bar then accelerates up towards the thumb where it decelerates at the tip and pauses momentarily again 920. After this, it accelerates down to the base of the thumb, pauses momentarily 930 and then accelerates back up the index and middle fingers and culminates with another subtle deceleration 940.

[0058] This is a "thumb, palm and two finger" Ultrahapticon.

[0059] FIG. 10 shows a sequence 1000 for a metaphor: "Heating Elements". This Ultrahapticon aims to convey snaking heating elements embedded in a car seat which are associated with the seat temperature feature. The user feels a focal ring at the base of the thumb 1010; this sensation then travels in an S-shape path up the palm 1020 to finish at the base of the middle finger knuckle 1030.

[0060] This is a "palm only" Ultrahapticon.

[0061] E. Ultrahapticons for Fan Speed

[0062] FIG. 11 shows a sequence 1100 for a metaphor: "Propeller Fan". The user receives a continuous anti-clockwise rotation of 3 haptic focal points 1110, 1120, 1130 to represent the users' metaphor of a propeller fan turning.

[0063] This is a "palm only" Ultrahapticon.

[0064] F. Ultrahapticons for Navigation

[0065] FIG. 12 shows a sequence 1200 for a metaphor: "T Junction". This Ultrahapticon represents a T-junction on a road which was commonly elicited as participants' mental model for navigation. The user receives a dynamic sensation that starts with a single focal point at the middle-bottom of the palm 1210 then travels up the palm 1220. Once the haptic focal point reaches the center of the knuckles 1230, it splits into two separate focal points 1240, 1250; one travels left across the knuckles and the other follows right along the knuckles.

[0066] This is a "palm only" Ultrahapticon.

[0067] FIG. 13 shows a sequence 1300 for a metaphor: "Compass Out". This Ultrahapticon aims to coincidentally represent three separate metaphors: "the compass", "crossroads" and "`X` marks the spot". A single focal point starts in the center of the palm 1310, from which splits off four focal points of equal size 1320. These four focal points move in synchrony 1330 towards equidistant locations at the distal, proximal, lateral, and medial extremes of the palm 1340. An alternative to the "Compass Out" is the "Compass In" which works in the dynamic inverse of the of the above sensation.

[0068] This is a "palm only" Ultrahapticon.

[0069] FIG. 14 shows a sequence 1400 for a metaphor: "Waypoint Blip". This Ultrahapticon represents the metaphor of waypoint visual on a GPS interface to signify navigation. A pulse in the center of the palm 1410 spreads outwards 1420, 1430 and fades 1440.

[0070] This is a "palm only" Ultrahapticon.

[0071] G. Ultrahapticons for Home Screen

[0072] FIG. 15 shows a sequence 1500 for a metaphor: "House Roof". A focal point renders an equilateral triangle to represent the roof of a house which was a construct extracted by participants to represent the "Home" function. The focal point starts at the base of the thumb 1510 and completes three separate strokes as sides of the triangle. At each point of the triangle there is a momentary pause before the rendering continues 1520, 1530 as this is known to improve recognition.

[0073] This is a "palm only" Ultrahapticon.

[0074] FIG. 16 shows a sequence 1600 for a metaphor: "Sofa Cushion". A mid-air haptic ring starts in the center of the palm 1610, it expands in thickness and diameter 1620, 1630 at which point it shrinks back to its original dimensions 1640, 1650. This Ultrahapticon replicates the compression of a cushion as one sits on a sofa; this is a semi-abstract metaphor that study participants had for the home screen function.

[0075] This is a "palm only" Ultrahapticon.

III. Study of Ultrahapticons Creation

[0076] A. Feature and Participant Selection

[0077] It was evident in the related work that the non-driving related task (NDRT) and User Interface (UI) design is principle in the success of MAHGs. Results from Large indicated that MAHGs appear superior to a touchscreen interface and mid-air gestures (without haptics), with regard to visual demand and task performance, for tasks that involve incrementally adjusting a setting. However, the results indicate that the touchscreen is more appropriate for the selection of a non-goal orientated 4.times.4 button grid when compared with MAHGs in a car following paradigm simulation. Arguably, the button selection condition might yield different results if the UI reflected the smaller button proportions inherent in some contemporary production vehicles. Hence, an expert user experience appraisal was conducted in a Tesla Model X to identify ecologically valid features and interactions in the touchscreen interface that could benefit from actuation using MAHGs. These consisted of discrete selection and continuous adjustment interactions for fan speed, cabin and seat temperature, navigation and audio volume; discrete shortcut interactions for telephone calls and the landing page (home) as well as response to telephone call notifications.

[0078] A participatory design study was subsequently conducted with a sample of seventeen participants (Male n11, Female n6, Age Range 19-65 yrs, mean 30 yrs) comprising members of the Nottingham Electric Vehicle Owners Club, staff and students at the University of Nottingham and non-technical employees at Ultraleap Ltd. Understanding cultural difference was considered important therefore multiple nationalities were recruited for the sample (UK [n7], Mexico [n3], Malaysia [n3], Hungary [n1], Spain [n2] and India [n1]).

[0079] B. Participatory Design Study Procedure

[0080] The procedure encompasses an amalgamation of learnings from related literature. Six of the participants were involved in individual face-to-face sessions where they experienced the mid-air haptic technology (the exposed group); eleven were organized remotely as a result of Covid-19 lockdown measures (the non-exposed group), the technology was therefore comprehensively demonstrated to them via remote video call.

[0081] C. Cognitive Mapping

[0082] Following a practice word-association task, participants verbalized the mental models they associated with each infotainment feature. Specifically, participants were asked "For the words [infotainment feature] what would you associate [tactually i.e. physical sensations; visually, as mental images or objects and auditorily i.e. sounds.]?". Participants were encouraged to consider the words themselves and not the features within context based on findings that structuring the questioning in this way led to less bias yet still yielded concrete metaphors.

[0083] D. Mental Model Visualization.

[0084] The next stage involved asking the participants to sketch the visual, auditory and tactual elements they had previously mentioned. In consideration of differing sketching abilities, the participants were encouraged to follow a "think-aloud" protocol as they sketched; this would enable the investigator to review video footage to understand the participants' thought processes if the sketch wasn't sufficiently communicative. The investigator demonstrated with an arbitrary example of a radio metaphor illustrated as a retro "boombox" radio, and then directed the participants to render a conceptual sketch for each feature. The "exposed" group were then demonstrated examples of sensations via the Ultraleap STRATOS explore array and the non-exposed group had the technology and types of sensations thoroughly described to them with aid from a graphical visualizer. The participants were then informed of twelve tunable, mid-air haptic parameters that could be manipulated to create different sensations. Using this information the participants highlighted elements of each sketch they thought most embodied the metaphor (i.e. the antenna on the radio example); they were then encouraged by the investigator to elicit how they would use the parameters, along with a nominal open palm gesture, to encapsulate these characteristics as their personal mid-air haptic icons, or "Ultrahapticons".

[0085] E. Ultrahapticon Refinement

[0086] The next step guided the participants to extend their designs to include how specific dimensions of the sensation would adapt to reflect a user-manipulated change in the feature setting (real-time interactional feedback). This time the participants were asked to consider that the feature will be adjusted using a more realistic "index finger and thumb pinch" hand pose and that they should elicit what axis this hand pose should move along to influence the function. This gesture was selected based on current design guidelines for automotive gesture interfaces generated by the array manufacturer-Ultraleap.

[0087] F. Semiotic Decomposition

[0088] The Ultrahapticon study elicited 119 total sketched designs which were analyzed for their semiotic components to determine the most intuitive designs for each feature (referent). First the participants' mental model sketches were classified into distinct prevailing styles (proposals). Although not specifically instructed to, many of the participants proposed multiple styles for a single referent. To account for this, the proposals were analyzed for "Max Consensus" (MC: percent of participants eliciting the most popular proposal) and "Consensus-Distinct Ratio" (CDR: the spread of participants displaying the most popular proposal-the closer the value is to 1, the smaller the spread and the more agreement there is among participants). Singular incidences of proposals were eliminated resulting in a shortlist of 23 Ultrahapticon styles for the 7 features.

[0089] The next level of analysis involved breaking down the styles into exemplar level semiotic components; their feature "constructs" and "intents". Constructs are physical characteristics of a feature that, collectively, comprise the holistic mental model (e.g. the rails of a rocking chair); an intent is a symbolic construct that is used by the designer to express meaning or behavior (e.g. blurred lines indicating movement of the rocking chair).

[0090] FIG. 17 shows one example as the process map 1800 for the fan speed feature with a CDR of 0.319 and MC of 59%. "Styles" included propeller 1810, air vents 1820, hair blowing 1830 and car HMI 1840. "Constructs and Intents" for: hair blowing 1830 included hair strands 1832, air movement 1831; for Car HMI included LED indicators 1843, turning dial 1842, air movement 1841; for propeller 1810 included rotation 1811, blades, 1812, clunky buttons 1813, whirring 1814, grid cover 1815, air movement 1816; and for air vents 1820 included grid 1821, air movement 1822 and directional toggle 1823.

[0091] Derived from the 23 styles were 88 distinct semiotic features; these were analyzed further for consensus which indicated that 65 were commonly occurring (appearing at least twice). 32 of these were adapted by the participants into their Ultrahapticon designs.

[0092] These Ultrahapticons were subsequently decomposed into their value level components to understand the participants' expectation of the real-time interactional feedback. This included understanding any consensus regarding construct rendering; what spatial, temporal and spatiotemporal parameters were used to signify feature intents; location of the sensations on the hand, axial direction of the pinch gesture and the dynamic adaptation of the sensation to reflect the feature adjustment.

[0093] Limited consensus was observed among the participants during the technical "value level" part of the study and sometimes the concepts were not feasible. Therefore, some results were adapted based on literature heuristics and where high disagreement occurred, all variations were considered for that icon style. Additionally, frequently reoccurring constructs and intents from popular styles that weren't selected for haptification by participants were reimagined as sensations by the investigator on the basis that these may have been discarded purely due to the participants' partial understanding of feasibility. This was exacerbated by the language barrier in some cases which was the only cultural difference observed in the study. To refine the resultant 30 user-centered Ultrahapticons, a remote workshop was conducted with four mid-air haptic experts. The Ultrahapticon design process was described to the attendees after which they were asked to rate each concept on a five-point Likert scale pertaining to feature appropriateness, expected salience, naturalism, instant recognizability, perspicuity, and technical feasibility. They then provided expert consultation on how to adapt the designs to improve the aforementioned aspects. The data from the workshop was used to hone the concepts and the result was a shortlist of 17 sensations for 7 features.

[0094] FIG. 18 shows a process map 1700 for these 7 IVIS features and their associated chosen 17 Ultrahapticons.

[0095] For IVIS feature Telephone Calls 1720, the chosen Ultrahapticons were "Rotary Dial" 1722, "Coiled Wire" 1724, "Bouncing Headset".

[0096] For IVIS feature Audio 1760, the chosen Ultrahapticons were "Sound Waves" 1762 and "Bass Speaker" 1764.

[0097] For IVIS feature Cabin Temperature 1730, the chosen Ultrahapticons were "Fire" 1732, "Ice" 1734, and "Thermometer" 1736.

[0098] For IVIS feature Seat Temperature 1770, the chosen Ultrahapticons were "Heating Elements" 1772 and "Seat Profile" 1774.

[0099] For IVIS feature Navigation 1740, the chosen Ultrahapticons were "Compass" 1742 (both "Compass In" and "Compass Out"), "Road Systems" 1744, "T-Junction" 1746, "Waypoint Blip" 1748.

[0100] For IVIS feature Fan Speed 1750, the chosen Ultrahapticon was "Propeller" 1752.

[0101] For IVIS feature Home 1710, the chosen Ultrahapticons were "Sofa Cushion" 1714 and "Triangular Roof" 1712.

[0102] G. Conclusions and Future Work

[0103] The next phase in this research may determine the most articulate icons from the shortlist; the icons will be prototyped along with synchronous hand poses based on established psychophysical principles and then evaluated in a salience study. An initial objective will be to understand the "articulatory directness" of the icons, that is the strength of the link between feature and metaphor and whether the icon rendering reflects the intended symbolism. This includes identifying the optimal way of manipulating haptic spatio-temporal dimensions to reflect the dynamic interaction with a specific feature.

[0104] The next study may also focus on perceptual optimization of the icon set by testing the icons' salience under simulated workload similar to that of a driving task. This will determine whether certain icons are masked through cognitive load and whether some sensations are confused with others within the set. When eventually tested in a driving simulator, this design process will improve the likelihood of validating distraction and task time measures associated with this MAHG concept without perception confounds.

IV. Conclusion

[0105] The various features of the foregoing embodiments may be selected and combined to produce numerous variations of improved haptic-based systems.

[0106] In the foregoing specification, specific embodiments have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.

[0107] The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.

[0108] Moreover in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has", "having," "includes", "including," "contains", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises . . . a", "has . . . a", "includes . . . a", "contains . . . a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms "a" and "an" are defined as one or more unless explicitly stated otherwise herein. The terms "substantially", "essentially", "approximately", "about" or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art. The term "coupled" as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

[0109] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.