In the last post, Dr. Marissé Masis and I built a vestibular sensory substitution system to help people with equilibrium problems. Our idea was to measure head tilt via a headpiece; to relay tilt information to a receiver via RF; and to communicate head tilt information on the palm via haptics embedded in a glove. You nudge your head to the right, and you get a jolt on the right side of your palm; tilt your head forward, and you receive a jolt at the front of the palm, near the fingers; etc. This kind of sensory substitution has been shown to radically improve equilibrium in people with bilateral vestibular damage (BVD).

We got the sensors working, as well as the calibration, RF communication, and actuation. But when it came to actually feel the head orientation through the glove, the system fell short. All you felt was a dull vibration whose location was very hard to determine.

We messed around haphazardly to try and improve the system, experimenting with different stimulation protocols, and increasing the strength of vibration. Eventually, we realized that the glove was vibrating sympathetically with the haptic motors, and communicated the vibration all over the palm.

A faster feedback loop

We had a hypothesis for why the haptics weren’t working, and we had a couple of ideas about how we might reduce the sympathetic vibrations; but what we really needed was some hard data. Without hard data, it’s easy to make some tweaks and fool yourself into thinking the tweaks are better. With hard data, we can instead iterate assuredly, picking up a 10% improvement in performance here and another 5% there, until we reach a much better design than the original. For the data scientists reading this blog, this should sound very familiar: A/B testing.

I had originally intended to evaluate the system end-to-end; try the system on real people with vestibular problems, and measure the improvement as we vary a handful of design parameters. Putting your design in front of people is a great idea, but it requires a lot of resources. We therefore looked for much cheaper experiments we could run on ourselves to test and improve isolated parts of the design. That way, we could iterate faster and cheaper, I recently heard about an experiment that run about the 3D Pens and the results were amazing.

Psychophysics, qu’est-ce que c’est?

We turned to psychophysics. What is it?

Psychophysics quantitatively investigates the relationship between physical stimuli and the sensations and perceptions they produce. Source: wikipedia

To make the correct compensatory head movements in response to the physical sensations in the palm, you have to correctly tell which part of the palm is vibrating. Thus, we systematically mapped how difficult it is to tell apart two feedback directions under different conditions.

On a given trial, one motor in the glove is activated for a short period, say, the one on the left; there’s a one second wait, then another motor on the glove is turned on, say, at the front. The subject must determine whether the stimuli are the same or different. For a good haptics device, it should be easy to tell apart two well-separated stimuli.

How do you set it so that people can’t cheat, and always say, “yes, it’s different”? On half the trials, the stimuli were indeed the same. Therefore, to perform optimally, the subject’s responses should be split equally between “same” and “different”.

Different enclosures

There were a few factors we wanted to test that we knew they could influence accuracy. First, there was the issue of sympathetic vibrations. ERM motors vibrate along the x and y (axial) directions; therefore, any sort of enclosure that is rigid in the x and y directions should exacerbate sympathetic vibrations.

We therefore designed and 3d printed two enclosures that were designed to better isolate vibrations. First, an enclosure with the coin ERMs rotated on their side, to rotate the vibration from the x/y plane to the z plane:

Secondly, we tested an enclosure with the 4 motors in their natural orientation joined by four spring-like segments to absorb the vibration in the x/y plane:

We secured the enclosures with 4 pieces of double-sided tape underneath the motors, and ran 256 trials in a factorial design. Marissé patiently judged whether the stimuli were the same or different, answers which I duly transcribed in the Arduino serial monitor — the psychophysics took about 3 hours, half of which was cutting the double-sided tape. We analyzed the results in R in about 10 minutes: stat_summary() does quick work of this kind of data.

Here’s the repo with the experiment code, the csv files, and the analysis.

Because we suspected that the enclosures might have a bit of leftover sympathetic vibration, after running each experiment, we cut the plastic segments joining the motors, and ran the psychophysics protocol again.

 

Cutting the segments turned out to be quite important for the flat enclosure:

Indeed, we found a large increase in performance (about 6% in absolute terms) after cutting the arms connecting the flat enclosure, which means we had still not fully resolved the issue of sympathetic vibrations. By contrast, for the rotated enclosure, there was no significant change in performance after removing the adjoining arms.

Of all the variants we tried, the best performing one was also the one with the simplest geometry: disconnected motors lying flat within the palm held with adhesive. In the next iteration of our prototype, we’ll stick with the flat configuration; we’ll use very thin connecting segments  between the pads to confer it the minimal amount of structural integrity to be easy to handle.

Location on the palm

We then looked at how the direction of the reference stimulus (0 degrees corresponding to the right of the hand, 90 degrees to the top of the hand, etc.) influenced accuracy:

Here, the 45 deg, 135 deg, etc. positions correspond to driving two motors simultaneously at a lower voltage, to simulate an intermediate direction between two motors. These intermediate positions turned out to be a bottleneck in our design; it’s a lot easier to tell apart stimuli when one is located in a cardinal direction. If we extrapolate the performance along cardinal directions with a sine wave, we get a prediction of what the performance might be like with 8 motors instead of 4:

I estimate that we could get a sizable 6% absolute increase in discrimination with 8 motors instead of 4. Another thing to notice is that out of all the cardinal directions, 0 degrees, on the right side of the left palm, is noticeably less sensitive. By moving this motor a bit to the left, it might be possible to squeeze more performance out of the system.

Strength of the vibration

You might recall that I built a boosting circuit on the receiver to increase the strength of the drive to the motors from 3.3V all the way to 5V. Was it worth it?

The results unambiguously say no. In fact, the discriminability goes does down slightly as we increase the strength of the vibration. I must admit, when I saw this, I was floored.

The vibration receptors in the palm — Pacinian corpuscules — are most sensitive to frequencies in the 200-300Hz range. Driven at high voltages (5V), ERMs can vibrate at frequencies past this optimum. By bringing the voltage down, ERMs vibrate less, but at frequencies the skin is more sensitive to. Precision Microdrives has excellent whitepapers on the subject of haptics actuators.

The whole boosting circuit, which took ages to solder, was pretty useless. As Donald Knuth famously quipped, premature optimization is the root of all evil.

Single vibrations versus motion vectors

There’s some evidence in the literature that it’s easier to tell apart spatiotemporal sequences of vibrations — a sweep from bottom to top vs. a sweep from right to left — than it is to tell apart single vibrations, or taps.

We tested this in our setup and found that, in our setup, performance was worse with sweeps than with taps:

How do we square this with the previous literature? In the Eagleman paper, the vibration motors were very close compared to the size of receptive fields at the stimulation location (lower back). Furthermore, they compared the performance of these sweeps to complex patterns actuated by several motors simultaneously. Given that the motors were so close together that they likely drove the same afferents, it’s not surprising that patterns underperformed compared to sweeps: the separation in the temporal dimension could compensate for the lack of separation in the spatial dimension.

Another potential reason is that our coin-cell ERMs have sluggish temporal responses, which make them ill-suited for communicating fine-grained temporal information. While we have nice, monotonic psychometric functions for the single tap experiments, in the sweep experiments, the performance at 180 degrees is similar to 90 degrees apart:

That means it’s hard to tell apart an upward sweep from a downward sweep. If a person makes a compensatory movement that is 180 degrees away from the correct direction, they’ll end up in a worse position than they started; performance at 180 degree should be close to 100% for the design to work.

When I tested the impulse response function of these coin ERMs, and found that they really get going after ~50 ms, with some jitter; given that the haptic pulses here are only 100ms, it’s not surprising that the direction of the felt motion could be misestimated because at that jitter.

Conclusions

We learned a lot from this simple series of experiments:

• a flat enclosure can work well as long as it deals with sympathetic vibrations propery
• some parts of the palm are more sensitive than others
• 8 motors should work better than 4 motors
• stimulation voltage is a minor parameter; a 3V drive is sufficient
• undirected stimulation works better than directed sweeps

In our baseline condition — a flat, connected enclosure, directed stimulation, strong vibration — which was most similar to the stimulation within the glove, accuracy was at about 63%. With the various tweaks that we found — as well as the addition of another 4 motors — we expect to increase the accuracy to about 83%. While A/B testing and psychophysics can feel an awful lot like bean counting when you’re doing it, the cumulative effect of small tweaks can lift a design from clunky to polished.

There is one parameter of the design that we didn’t try to tweak: changing the actuator type. Linear resonant actuators (LRAs) are more temporally precise than ERMs, and they vibrate in the z direction rather than in the x-y direction, so they don’t have as many issues of sympathetic vibrations. On the other hand, they require dedicated drivers and are quite a bit more expensive.

For now, we think that simply packing more ERMs in the glove is a more promising direction than switching to LRAs, but we don’t know yet. In the next installment, we’ll pitch an LRA-based prototype against an improved, ERM-based one.

Follow me on Twitter or Facebook to get notified of next week’s installment.