Ultraleap Patent | Mid-air haptic generation analytic techniques

where R is the radius of the circle, t is time, and co is the STM angular frequency. The amplitude, which would need to be supplied in tandem, is a fixed value in this case. A particularly useful mathematical generator for STM curves is given by

x⁡(t)=A⁢ cos⁢ (a⁢ω⁢t+δ)⁢ cos⁢ (k⁢ω⁢t), y⁡(t)=B⁢ cos⁢ (b⁢ω⁢t)⁢ cos⁢ (k⁡(ω⁢t+π2)). z⁡(t)=0.

where t is time and all other values are adjustable constants. Adjusting δ to zero and k to 1 for instance, results in a series of curves named Rose curves. Adjusting δ to π/2 and k to 0 similarly results in a class of curves named Lissajous curves. Other adjustments to the constants result in circles and lines. Generating the paths using trigonometric functions such as the above yield paths without sharp angles or large spacing between points which is known to create unwanted noise in mid-air haptic devices.

Primitives expressed as generating functions have the added benefit of being easily adjustable in real-time. If the primitive parameters are exposed to the programmer, this gives a sophisticated designer the ability to adjust the primitive in controlled, and often smooth, ways. It must be noted that the time variable here can be evaluated at arbitrary time or linked to discrete points, such as once per ultrasonic acoustic cycle.

After a primitive is selected a series of transforms are applied to modify the shape, size and location of the points being generated to create more sophisticated sensations.

In one embodiment of this invention, this takes place in the form of a series of cascaded affine transforms which are applied through a dot product with the coordinate vector. Affine transforms are implemented as a n×n matrix multiplication where n is one more than the dimension of the input coordinates. For instance, an affine transform on a 2-dimensional set of coordinates is a 3×3 matrix with the input coordinates padded with a 1 as their 3^rd coordinate before multiplication. Likewise, 3-dimensional coordinates use 4×4 element transformation matrices. Transformation matrices do not need to be square and can increase or decrease dimensionality if needed. Affine transformations are capable of but not limited to translation, reflection, scaling, rotation, shear, as well as simultaneous combinations of these.

As with primitive parameters, transforms can be implemented in the host or on the device. If implemented on the device, the host could modify exposed coefficients intermittently when things need to be changed, or on a regular basis. It must be noted that the primitive output coordinate system needs to be controlled in order for the transformations to be easily understood. For instance, a standard scaling transformation given by

T=[200010001],

which in this example would be interpreted as scaling the x-dimension by a factor of 2 while leaving the y-dimension alone. For this transform to operate as intended (scaling x symmetrically by 2), the primitive curve to be transformed by this needs to be centered at the origin. If, for example, a circle were centered at a coordinate other than zero, this transform would result in both a scaling but also a translation of the geometric center of the primitive. This could interfere with subsequent translation and targeting of the sensation. Therefore, for consistent functioning of transformations, a best practice of locating the geometric center of primitives at the origin should be practiced.

Primitives with more than one simultaneous point specified will need to have each point run through the transformation pipeline independently. One method to accomplish this is to represent each point as column in a location matrix and use the standard definition of dot product to apply each affine transform. In that case the output transformation will produce the transformed coordinates in their respective columns.

Acoustic solvers which take the output of this invention to generate acoustic fields can take as input both location and amplitude. More sophisticated solvers will attempt to achieve the desired parameters within the constraints of the transducers available. Amplitude information can take the form of simple acoustic pressure (measured in pascals), or can be more sophisticated if allowed by the solver. This includes, but is not limited to, squared pressure, acoustic intensity, particle velocity, particle velocity in a particular direction, nonlinear acoustic force, and nonlinear acoustic force in a particular direction. Amplitude information must be specified during the generation of the primitive. This amplitude can be modified post-primitive by scaling that is synched with or independent of transforms. Similar to the primitive generator, the amplitude could be represented as a mathematical function with adjustable variables. Scaling above the maximum output value can be managed through clipping, scaling, dynamic scaling, or other methods common to signal processing.

One particular primitive of note is one which contains points with a specified amplitude of zero. When a zero amplitude goal is given, a standard acoustic solver will attempt to formulate a field with zero (or very low) pressure at that location. A primitive which only contains points with zeros could therefore be seen as a “quiet” region in the field and placed in such a way as to shield potentially sensitive devices or animals. This quiet primitive can be transformed just as other primitives in order to track the sensitive device or animal. A primitive with zero or “null” points needn't have all its points specified at low pressure—a mix of zero and non-zero is also possible.

Turning to FIG. 1, shown is a block diagram 100 of the basic transform pipeline. After primitive generation 110 the output point proceeded through a series of transforms (T1 120, T2 130, . . . Tn 140) before being passed to an acoustic solver 150 which drives the mid-air haptic device.

Turning to FIG. 2, shown is a block diagram 200 of the transformation pipeline starting with primitive generation 201 including animation transforms (T1a₁ . . . T1a_m 202 to T1 203; T2a₁ . . . T2 a_m 212 to T2 213; . . . Tna₁ . . . Tna_m 222 to Tn 223) and stored transforms (T1s 205, T2s 215, . . . Tns 225). Animation transforms are applied to their associated base transform at regular intervals dictated by logic not shown. At any point the base transform may be reset to a stored value in the stored transform before being sent to the acoustic solver 240.

The introduction of animation transforms in FIG. 2 provide additional examples. These transformations take the same form as the base transforms (T1 120, T2 130, . . . , Tn 140) but are applied to the base transformation repeatedly, at regular intervals. In one embodiment of this invention, these take the form of affine transforms. For example, if T1a₁ were active and designed to be applied after every update of the primitive, before T1 is applied to the output of the primitive generator, T1 203 is modified through the dot product of (T1a₁ T1) and the output stored into T1 203. This repeated application of the same transform can produce progressive animations such as steady rotation or translation. Multiple animation transforms can be associated with each transform slot. Each can be used independently, with logic designed to dictate when each should be applied—for example, run the first one for 1 second, then switch to the next animation transform, and so on.

Alternatively, multiple animation transforms could be cascaded with each evaluation, or some combination of both methods. This allows the system to independently apply transformations which change the output of the system without constant changes being dictated by the host. As an example, a line primitive can be made to rotate about its center by using a rotation transform in an animation transform slot. Upon each update interval, this animation transform applies a fixed rotation into the base transform, and since repeated applications are cumulative, the resulting base transform progressively rotates the output at a rate dictated by the animation transform and the frequency of animation application. Because the base transforms are continually modified, it is useful to provide storage for each transform to give the ability to return back to a known point. These are shown illustrated by T1s 205, T2s 215, and so on.

Transforms can be fixed, modified by some internal logic, or addressed externally. In the context of mid-air haptic feedback, it is useful to categorize one transform, usually at the end of the chain, as a “tracking transform”. Transforms up to this point typically assume the geometric center of the primitive is located at the origin of the coordinate system. Using this best practice, values in the transforms can be more easily interpreted as scale, sheer, rotation, or other simple operations. But the haptic sensation at some point needs to be located on a user's hand or other body part which exists in a coordinate system separately provided by some kind of tracking device such as a camera, time of flight sensor or the like. Transforming from the internal device-coordinate-system to the tracking-coordinate-system can be accomplished in one or more base transforms. Producing this transform can be done via a host or internally to the array system with feedback from an external or internal tracker.

Generating points using methods presented in this invention which are fed to an acoustic solver which adjusts the output of a mid-air haptic device can occur at different levels in the control chain with different benefits. Traditionally, a mid-air haptic device expects point locations with desired amplitudes from a controller device, delivered in a timely manner. The controller device could take the form of a computer running a display or some other device integrated into user-tracking of some kind. It is possible to implement this invention on this host device which would address and operate the mid-air haptic device like before, with a stream of focus locations or more granular phases and amplitudes or similar information.

In another embodiment of this invention, the haptic device itself generates primitive coordinate locations and amplitudes and accepts from the host options to modify the primitive such as generating constants in a functional primitive as presented above and/or transform coefficients (both base and animation) as well as other logic to control timing and amplitude (not shown). In this arrangement, primitives and transforms can operate on the device, generating haptically effective acoustic field values independently while waiting for updates from a host. This improves the consistency of haptic effects as possible communication interruptions from the host will be largely invisible to the user experiencing the haptic effect.

In the legacy arrangement where points were streamed to the haptic device, missing updates for even a few milliseconds could result in uneven, noisy, or even completely absent haptic sensations. With primitives and transforms being generated on the devices, consistent output is maintained, even when the host is unresponsive.

Changing the primitives or transforms can be done from an external device (such as a host, as described above) or from within the mid-air haptic device. One method of organizing these changes is by means of a sequencer. This takes the form of a set of haptic parameters (primitive specification and transforms) which have a playback time and a pointer to another set of haptic parameters (with their own associated playback time). Upon activation, the system would produce the first set of parameters for the specified time period and then switch to the next as specified by the pointer. That next set of parameters would have its own playback time and pointer to another set. That pointer could point to a unique set of parameters or to one previously used.

In addition, any one set of parameters could also include logic to change parameters in other sets. For example, a group of 4 sets could proceed from 1 to 4 in order, but after getting to 4, that set changes pointers in 2 and 3 so that the order is reversed (3 points to 2 and 2 points to 1). Upon getting back to 1, that set returns the pointers to the original ordering sequence. This creates a sophisticated, repeating pattern without any direct changes necessary. During this playback, the tracking transform would be continuously modified as needed to maintain targeting of the haptic on the user. In this arrangement, a control (or host) device could modify primitives, transforms, pointers, or playback time found in any of the parameter sets, instead of directly modifying currently used transforms.

Switching between sets of parameters can be done immediately or over time via a smooth transition. Small changes, such as changing the translation speed in an animation transform, for instance, can be done immediately without unwanted noise. Some changes, however, such as switching between very different primitives, need to be smoothed to produce a better user experience.

In one embodiment, a transition period can be incorporated using interpolation or low-pass filtering. This can be implemented at the final point output before going into the acoustic solver or at various points in between base transforms. In this case, the x, y, z locations and amplitudes for each point would all be separately interpolated or low pass filtered for a certain period until the filtering or interpolating is deemed unnecessary. Another way to implement the transition is to interpolate or low-pass filter all primitive and transform values from one set of parameters to another.

In some implementations of this invention, more than one primitive with associated, independent transforms will need to be produced simultaneously. For example, a two-handed interaction needs separate haptics to be produced on each hand with each hand moving independently. As discussed above some acoustic solvers have the ability to produce complex pressure fields with more than one high-pressure point at the same time. For these systems, one solution is to maintain at least two independent primitive/transformation pipelines where the output of each is joined into a single request to the acoustic solver at regular intervals. This does have some drawbacks, however, as acoustic energy will be split among all of the desired high-pressure points in the system.

Another solution to produce multi-primitive sensations is to alternate between them. In systems with a multipoint acoustic solver, this can be achieved by only having the amplitude of one primitive on at any one time, with all others off. At a specified interval (for instance, after one complete primitive interval for a repeated path), the amplitude of one primitive cross-fades to another primitive, which is driven for a time before crossfading to another and eventually looping back to the first and repeating. This may reduce the total acoustic energy delivered to any one primitive but can increase the peak pressure for the time each primitive is being driven.

Another method to produce at least two primitives simultaneously is to alternate between them at regular intervals. In other words, produce one for a certain interval, and then switch to the other, then the next, and so on. While this will accomplish the desired sensations, large jumps between individual points (such as between primitives) can produce unwanted audible noise.

A solution to this is to include a set of transition points between each primitive. This transition will include an amplitude ramp towards zero during and/or immediately preceding the transition. New points, not part of any primitive, will then be fed to the acoustic solver which proceed from the last point produced by the previous primitive to the first desired point in the next one. The path for the transition can be a straight line path, or a more complex curve generated by smoothing methods such as filtering or polynomial curves. At or just preceding the arrival at the new primitive, amplitude can ramp back to the desired value. The intervening path to join multiple primitives together will be a compromise between speed and audibility. The faster the transition, the more audible noise will be produced, while the slower the transition, the less time will be spent drawing the haptics which is likely to reduce haptic feel. Ramps can be linear, polynomial, trigonometric, or other suitable curve. Depending on the headroom of acoustic energy available, creating multiple, simultaneous mid-air haptics can change the feel of an individual primitive and care must be taken to accommodate the possible change.

FIG. 3 shows a block diagram 300 of an example implementation of the invention. In this case, a primitive generation 302 interfaces with two independent base transforms (including up to 2 animation transforms T1a₁, T1a₂ 304 to T1 305 and T2a₁, T2a₂ 314 to T2 315, and storage T1s 304, T2s 317), which interfaces with T3 320. This is implemented as a tracking transforms to tilt and translate output points to a user's tracked palm.

Further, FIG. 3 shows an example implementation of this invention. In this case there are 3 base transforms with the first two having 2 animation transforms with associated storage. The third transform T3 320 is reserved as a tracking transform. In this arrangement, all transforms are implemented as 4×4 affine transformation matrices and the primitive generator outputs in 3 dimensions. This arrangement of transforms is capable of a wide variety of animated mid-air haptics with a minimum of computation.

FIG. 4 shows a block diagram 400 of an example implementation of the invention. In this case, the primitive generation 401 outputs only 2-dimensional points as shown in graph form 402. Output of the primitive generation is fed into a base transform with 2 associated animation transforms (including animation transform T1a₁, T1a₂ 403 to T1 404 and storage T1s 405) all still in 2 dimensions 406.

Tilt T2 410 is designed to take the 2D information and output a 3-dimensional set of points which matches the tilt of a tracked user's palm 412. T3 414 then translates the points onto the user's hand 416. The 3D points then interface to the acoustic solver 420.

Further, in this case the output of the primitive generator is 2-dimensional, simplifying the haptic design to a plane which corresponds to the palm of a tracked user's hand. The first base transform and its associated animation and storage transforms are all 3×3 affine transformation matrices, operating on 2-dimensional coordinates. More of these transformations can be cascaded if needed (not shown in the figure). The tilt transform (T2 410) takes the 2-dimensional information and adds a 3^rd coordinate through the following matrix multiplication:

u=norm⁢ (nˆ×[0,0,1]) c=nˆz s=sqrt⁢ (nˆx2+nˆy2) T=[c+ux2(1-c)ux⁢uy(1-c)ux⁢uy(1-c)c+uy2(1-c)uyS-ux⁢s],

where {circumflex over (n)} is the normal of the tracked palm.

Depending on the definition of normal, a reversing of the normal may be required (multiplying {circumflex over (n)} by −1). After the tilt transform, a regular offset transform (T3 414) is applied to translate from array coordinates to tracking coordinates. This formulation of the invention reduces the amount of computation required for the initial cascade of base transforms, as they are only operating on 2 dimensions, rather than 3. It is possible to group the tilt and offset transform into one transform but is broken out into two in this case for readability.

The above description includes language where an “acoustic solver” drives a system to produce a desired acoustic field. This can take the form of a phased array of ultrasonic transducers where each transducer can be independently controlled. In this case the acoustic solver produces phase and amplitude parameters for driving each transducer. Other arrangements of an acoustic solver could include the ability to move a given transducer or groups of transducers. In this case the solver would produce phase, amplitude and location information. Regardless of actual implementation, this invention will be applicable to any mid-air haptic device which can flexibly produce acoustic fields with specified locations and associated amplitudes.

Turning to FIG. 5, shown is an illustrative example 500 of four tracking reference frames 520 525 530 535, each in this example pinned to the palm of the hand of a tracked hand, a single global reference frame (center) 502 and two emitter reference frames 505, 510. A ‘pin’ transform transforms spatial haptic information from a local tracking reference frame into which the haptic effect is initially composited, into the global reference frame. An ‘emitter’ transform then transforms this spatial haptic information from a global reference frame to a local emitter reference frame in which its local processing is defined. This allows individual discrete devices to manage their own transducer locations, so the devices may be agnostic to their location in the global reference frame.

B. Transformations as Buffered States

Transformations applied to the control point data can be classified into three broad categories, that are applied in a specific order. First, as described, chains of composition transforms take points and other primitives and transform them to build local haptic effects. Second, pinning transforms change a local control point spatial frame of reference in which the effect is built into a global frame of reference.

Finally, emitter transforms change the global spatial frame of reference into a frame of reference with respect to the tile of transducers. These are applied in sequence to take a control point specified to a final control point for a system in control of a transducer array device to solve for. Equally, in some cases these may be omitted, for example if tracking is not necessary, or if a single system reference frame in which the transducer positions are defined is functionally equivalent to a global reference frame.

Each of these can be supplied as a data stream to the mid-air haptic device to realize a further light-weight approach to generating haptic effects. A number of streams of keyframed pinning transformations may be chosen between to allow different local reference frames to be transformed into the global frame. For example, each joint of each finger on the hand may be assigned a reference frame and control points may be moved to a reference frame by selecting that stream and thus pinning the control point to the reference frame selected. This allows for interactions for multiple hands, or multiple joints to be naturally created.

As tracking the user may be folded into the process of generating the pinning transformation stream, this can provide an abstract interface that can allow the process to consume transformations without regard for the origins of the data, so this allows for the use of any tracking system to provide source data. Equally, some compositional transforms may be merged with the tracking to generate a more primitive definition reference frame for the haptic effect, depending on the intent of the haptic designer.

C. Recycling Buffered States

An alternate method for generating haptic effects may be realized by recycling buffered keyframes of the control points through time as shown in FIG. 7, as an alternate mode of device processing when a stream is not being supplied from an external source as in the ring buffer operations shown in FIG. 6.

Turning to FIG. 6, shown is a diagram 600 illustrating states of the device moving through time that are constructed by interpolating together known states at known instants or ‘timepoints’ 610. The clocks 640 and times are given for illustration purposes, as in a real implementation the queue shown would generally be consumed in a period many orders of magnitude shorter. This diagram 600 shows the structure of the ring buffer 670 for the queue and how this maps onto the behavior of the array or arrays along the timeline when the user is actively streaming data to the device. Future states marked as ‘fresh’ (leftward slashes and horizontal lines 601g 601h 601i 601j) are added incrementally to the ring buffer at the write index 630, incrementing it and defining the future behavior of the device.

When the read pointer reaches a state obtained from a user 655, as it is currently in the future, it is mixed with the previous state that is known to be in the past 620 (rightward slashes 601a 601b 601c 601d) to create the state that is to be used now (these two states are shown shaded 601e 601f, straddling the dashed ‘now’ line 645). This is achieved by interpolating the keyframes represented by the two states.

Turning to FIG. 7, shown is a diagram 700 shows the structure of the ring buffer 795 for the queue and how this maps onto the behavior of the array or arrays along the timeline 710 when the user is not streaming data to the device and state date from the user is unavailable 720. The clocks 745 and times are given for illustration purposes, as in a real implementation the queue shown would generally be consumed in a period many orders of magnitude shorter. Three states are shown shaded 701e 701f, 701g straddling the dashed ‘now’ line 740.

In this example, future states that can no longer be written to 755 (horizontal lines) are locked because there is not enough time to replace them before they are used. If the unlocked future states (leftward slashes in FIG. 6) are exhausted before fresh states are obtained from the user, instead of locking and thus using one of the stale states (wide rightward slashes that contain invalid data 701a 701b 701c 701d), policies will be used to copy data from past states 760 (narrow rightward slashes), often amalgamating timing information (shown as the two adjacent states 791) with other state data (shown as the other outlined states 792 793) to generate new state data that extrapolates into the future the desired behavior of the array. As such, the ring buffer structure 795 may be used to create simple repeating haptic effects that may be composited with data from other interfaces and/or buffers, where these other types of data may include spatial transformations that are applied later in the haptic effect render pipeline.

When the stream of externally supplied haptic effect primitive data (such as for example, control points or parametric haptic curves) is exhausted, policies describe how to recycle past data to generate repeating effects. This is realized by modifying the times associated with the control point states in the past while copying into the future section of the input buffer. In this way haptic effects comprising repeating cycles of the same control point data may be generated. Parallel queues of transformations that run alongside and modify these states through the application of the transformations can modify the haptic effects, so as to allow them to follow tracking data. Each control point may be generated from separate individual queues of states, or a combined queue—by singling out control points in this way, individual applications may apply bespoke repeating patterns as input to the mid-air haptic system. By recycling the control point states, but allowing the composition and pin or tracking transformations associated with them to change, different effects may be generated. Without loss of generality, recycling data in queues may be performed on queued transformation data with or without recycling control point data, providing palettes of effects such as for example running the two cycles of control point data and transformation data out of phase to produce a rose curve or Lissajous figure. Equally, a further example could be realized as small circles created with the control point data while allowing the transformation data to change to provide a brush that is moved by pinning the small circles to a transformation stream.

As the input queue is being fed by recycled data, further policies may be used to control how the haptic effect changes if an external haptic effect primitive data stream source is resumed. This may involve for example, blending the transform position or created minimum curvature paths so that the haptic effect of the recycling process may proceed smoothly into the haptic effect created by the resumed external stream.

Keyframe states of the haptic points as shown in FIGS. 6 and 7 may also have tangent information in order to construct Bezier and other types curves in the interval between the keyframes. Bezier curves are especially effective however, as by using the de Casteljau algorithm to generate Bezier curves by applying a recursive linear interpolation, a linear interpolation or other interpolation scheme that is already present may be reused to allow the number of keyframe states required to generate complex repeating haptic effects to be minimized.

While this particular approach has advantages for fixed-function hardware pipelines, the recycling of input buffers or queues of haptic effect primitive data may be implemented in any number of different ways, in both software and hardware systems, such as FPGAs or microcontrollers that contain sufficient buffering and queueing capabilities.

II. Distributed Time-Domain Phased-Array Solver for Applications in Ultrasonics

A. Introduction

As described earlier, the system may be split into many pieces which are responsible for local output on each local transducer array. Noting that for ‘small’ groups of transducer elements that are close in space and thus must be ‘close’ in time-of-flight, solving for the group does not incur noticeable time-of-flight artefacts or errors due to the locality of the group. Given this, solving independently for ‘small’ groups of transducer elements will each produce output that is largely free of time-of-flight error.

Therefore, splitting the system into many smaller solvers which are each responsible for a portion of the output is possible, but the independent nature of the groups is a problem. The inputs to each “solver” must be modified to enable the system to work as a whole, allowing these disparate units to cooperate on producing the acoustic field.

These are divided for the purposes of illustration into three categories, although the physical implementations of such devices may fall into more than one adjacent category.

Writing the problem definition in mathematics, a_q(X_j) may be used to describe a complex-valued scalar linear acoustic quantity a measured at a position offset from the transducer element q by the translation vector x_j, which may evaluate to be acoustic pressure or an acoustic particle velocity in a direction chosen for each j, the matrix A may be written:

A=[α1(?1)…αN⁢(?1)⋮⋱⋮α1⁢(?m)…αN⁢(?m)],

As this is matrix A is not square, and the degrees of freedom number more than the constraints, this is termed a ‘minimum norm’ system. It is ‘minimum norm’ because as there are infinitely many solutions, the most expeditious solution is the one which achieve the correct answer using the least ‘amount’ of x—the solution x with minimum norm. To achieve this, some linear algebra is used to create a square system from the minimum norm system Ax=b:

A^HAx=A^Hb,