Tag: ultrasound haptics

Ultrasound Haptics

Introduction

Ultrasound haptic technology creates mid-air haptic sensations using focused ultrasound waves, so that people can experience haptic feedback against the hands without any physical contact with a device. There are numerous benefits of this technology, especially for touchless user interface design. To keep this introduction short, I recommend reading our recent survey paper of this technology [1] if you want more details about the technology and its applications.

I have been involved in the ultrasound haptics community for a long time (since 2012) and have sporadically done research in this area. On this page, I give a brief overview of some of my research into ultrasound haptics.

ultrasound array
An ultrasound haptics device consists of an array of ultrasound emitters. These are typically, although not always, in a rectangular layout.

Perception

One of my research interests is the perception of ultrasound haptic sensations. Since this is an emerging haptic technology, we still only have a basic understanding of how its tactile sensations are perceived by the body. In addition to this, it is so radically different from existing haptic technologies (e.g., vibrotactile) that little existing knowledge can be transferred. Research is needed to better understand the relationships between rendering characteristics and perception, so that we can make the most of the technology.

Research highlights:

Applications

One promising application for ultrasound haptic technology is in giving feedback for touchless mid-air gesture interaction. Touchless gestures have great potential but their adoption has been hampered by numerous usability issues [4]. Ultrasound haptic feedback can potentially help to alleviate these issues. A key benefit of this technology is that interaction feedback can be spatially coupled with the user’s input actions: in other words, haptic feedback can be given directly to the user’s hands whilst they gesture.

My work has explored various ways of integrating ultrasound haptic feedback in touchless user interface design, including using ultrasound haptic feedback to help users find a good hand position for touchless interaction [5] and giving confirmatory feedback about input gestures [6].

Photo of the HaptiGlow system. An Ultrahaptics UHEV1 device with a strip of LEDs around the front edge and left and right sides. The LEDs are green, indicating that the user has their hand in a good position.
One application of ultrasound haptics is helping guide hand movements in mid-air, e.g., like we did in the HaptiGlow project.

Research highlights:

Miscellaneous

Research highlights:

References

[1] A Survey of Mid-Air Ultrasound Haptics and Its Applications
I. Rakkolainen, E. Freeman, A. Sand, R. Raisamo, and S. Brewster.
IEEE Transactions on Haptics, vol. 14, pp. 2-19, 2020.

 PDF       DOI       Website      [Bibtex] 

@article{ToHSurvey,
    author = {Rakkolainen, Ismo and Freeman, Euan and Sand, Antti and Raisamo, Roope and Brewster, Stephen},
    title = {{A Survey of Mid-Air Ultrasound Haptics and Its Applications}},
    year = {2020},
    publisher = {IEEE},
    journal = {IEEE Transactions on Haptics},
    volume = {14},
    issue = {1},
    pages = {2--19},
    pdf = {http://research.euanfreeman.co.uk/papers/IEEE_ToH_2020.pdf},
    doi = {10.1109/TOH.2020.3018754},
    url = {https://ieeexplore.ieee.org/document/9174896},
    issn = {2329-4051},
}

[2] Perception of Ultrasound Haptic Focal Point Motion
E. Freeman and G. Wilson.
In Proceedings of 23rd ACM International Conference on Multimodal Interaction – ICMI ’21, 697-701. 2021.

 PDF       DOI       Website       Data      [Bibtex] 

@inproceedings{ICMI2021Motion,
    author = {Freeman, Euan and Wilson, Graham},
    booktitle = {{Proceedings of 23rd ACM International Conference on Multimodal Interaction - ICMI '21}},
    title = {{Perception of Ultrasound Haptic Focal Point Motion}},
    year = {2021},
    publisher = {ACM},
    pages = {697--701},
    doi = {10.1145/3462244.3479950},
    url = {http://euanfreeman.co.uk/perception-of-ultrasound-haptic-focal-point-motion/},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2021_Motion.pdf},
    data = {https://zenodo.org/record/5142587},
}

[3] Enhancing Ultrasound Haptics with Parametric Audio Effects
E. Freeman.
In Proceedings of 23rd ACM International Conference on Multimodal Interaction – ICMI ’21, 692-696. 2021.

 PDF       DOI       Website       Data      [Bibtex] 

@inproceedings{ICMI2021AudioHaptic,
    author = {Freeman, Euan},
    booktitle = {{Proceedings of 23rd ACM International Conference on Multimodal Interaction - ICMI '21}},
    title = {{Enhancing Ultrasound Haptics with Parametric Audio Effects}},
    year = {2021},
    publisher = {ACM},
    pages = {692--696},
    doi = {10.1145/3462244.3479951},
    url = {http://euanfreeman.co.uk/enhancing-ultrasound-haptics-with-parametric-audio-effects/},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2021_AudioHaptic.pdf},
    data = {https://zenodo.org/record/5144878},
}

[4] Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems – CHI ’16, 2319-2331. 2016.

 PDF       DOI       Website       Video      [Bibtex] 

@inproceedings{CHI2016,
    author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
    booktitle = {{Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems - CHI '16}},
    title = {{Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems}},
    year = {2016},
    publisher = {ACM},
    pages = {2319--2331},
    doi = {10.1145/2858036.2858308},
  pdf = {http://research.euanfreeman.co.uk/papers/CHI_2016.pdf},
  url = {http://euanfreeman.co.uk/gestures/},
  video = {{https://www.youtube.com/watch?v=6_hGbI_SdQ4}},
}

[5] HaptiGlow: Helping Users Position their Hands for Better Mid-Air Gestures and Ultrasound Haptic Feedback
E. Freeman, D. Vo, and S. Brewster.
In Proceedings of IEEE World Haptics Conference 2019, the 8th Joint Eurohaptics Conference and the IEEE Haptics Symposium, TP2A.09. 2019.

 PDF       DOI       Website       Video       Data      [Bibtex] 

@inproceedings{WHC2019,
    author = {Freeman, Euan and Vo, Dong-Bach and Brewster, Stephen},
    booktitle = {{Proceedings of IEEE World Haptics Conference 2019, the 8th Joint Eurohaptics Conference and the IEEE Haptics Symposium}},
    title = {{HaptiGlow: Helping Users Position their Hands for Better Mid-Air Gestures and Ultrasound Haptic Feedback}},
    year = {2019},
    publisher = {IEEE},
    pages = {TP2A.09},
    doi = {10.1109/WHC.2019.8816092},
    url = {http://euanfreeman.co.uk/haptiglow/},
    pdf = {http://research.euanfreeman.co.uk/papers/WHC_2019.pdf},
    data = {https://zenodo.org/record/2631398},
    video = {https://www.youtube.com/watch?v=hXCH-xBgnig},
}

[6] Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the International Conference on Multimodal Interaction – ICMI ’14, 419-426. 2014.

 PDF       DOI       Website       Video      [Bibtex] 

@inproceedings{ICMI2014,
    author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
    booktitle = {Proceedings of the International Conference on Multimodal Interaction - ICMI '14},
    pages = {419--426},
    publisher = {ACM},
    title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2014.pdf},
    doi = {10.1145/2663204.2663280},
    year = {2014},
    url = {http://euanfreeman.co.uk/projects/above-device-tactile-feedback/},
    video = {{https://www.youtube.com/watch?v=K1TdnNBUFoc}},
}

[7] Textured Surfaces for Ultrasound Haptic Displays
E. Freeman, R. Anderson, J. Williamson, G. Wilson, and S. Brewster.
In Proceedings of 19th ACM International Conference on Multimodal Interaction – ICMI ’17 Demos, 491-492. 2017.

 PDF       DOI      [Bibtex] 

@inproceedings{ICMI2017Demo,
    author = {Freeman, Euan and Anderson, Ross and Williamson, Julie and Wilson, Graham and Brewster, Stephen},
    booktitle = {{Proceedings of 19th ACM International Conference on Multimodal Interaction - ICMI '17 Demos}},
    title = {{Textured Surfaces for Ultrasound Haptic Displays}},
    year = {2017},
    publisher = {ACM},
    pages = {491-492},
    doi = {10.1145/3136755.3143020},
  url = {},
  pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2017_Demo.pdf},
}

[8] UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound
R. Morales Gonzalez, A. Marzo, E. Freeman, W. Frier, and O. Georgiou.
In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction – TEI ’21, Article 1. 2021.

 PDF       DOI       Website      [Bibtex] 

@inproceedings{TEI2021,
    author = {Morales Gonzalez, Rafael and Marzo, Asier and Freeman, Euan and Frier, William and Georgiou, Orestis},
    booktitle = {{Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '21}},
    title = {{UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound}},
    year = {2021},
    publisher = {ACM},
    pages = {Article 1},
    doi = {10.1145/3430524.3440620},
    pdf = {http://research.euanfreeman.co.uk/papers/TEI_2021.pdf},
    url = {http://euanfreeman.co.uk/ultrapower-powering-tangible-wearable-devices-with-focused-ultrasound/},
}

UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound

Overview

Wireless power transfer creates new opportunities for interaction with tangible and wearable devices, by freeing designers from the constraints of an integrated power source. In a paper published at the 15th ACM International Conference on Tangible, Embedded, and Embodied Interaction (TEI ’21), we explored the use of focused ultrasound as a means of transferring power to a distal tangible device. We call this UltraPower. This has the radical effect of transforming passive props into dynamic active objects.

Schematic outline of an UltraPower system. An array of ultrasound transducers creates a sound pressure field. This is received by a circuit containing ultrasound transducers, rectified into energy, and used to power a load circuit.

We analysed the ability to transfer power from an ultrasound array commonly used for mid-air haptic feedback and investigated the practical challenges of ultrasonic power transfer (e.g., receiving and rectifying energy from sound waves).

We also explored the ability to power electronic components and multimodal actuators such as lights, speakers and motors. In the paper, we described numerous exemplar wearable and tangible device prototypes that are activated by UltraPower, illustrating the potential applications of this novel technology.

    UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound
    R. Morales Gonzalez, A. Marzo, E. Freeman, W. Frier, and O. Georgiou.
    In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction – TEI ’21, Article 1. 2021.

     PDF       DOI       Website      [Bibtex] 

    @inproceedings{TEI2021,
    author = {Morales Gonzalez, Rafael and Marzo, Asier and Freeman, Euan and Frier, William and Georgiou, Orestis},
    booktitle = {{Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '21}},
    title = {{UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound}},
    year = {2021},
    publisher = {ACM},
    pages = {Article 1},
    doi = {10.1145/3430524.3440620},
    pdf = {http://research.euanfreeman.co.uk/papers/TEI_2021.pdf},
    url = {http://euanfreeman.co.uk/ultrapower-powering-tangible-wearable-devices-with-focused-ultrasound/},
}

Ultraleap Logo
Logo for the Universidad Publica de Navarra
University of Glasgow logo

Enhancing Ultrasound Haptics with Parametric Audio Effects

Overview

Ultrasound haptic devices can create parametric audio as well as contactless haptic feedback. In a paper at the 2021 ACM International Conference on Multimodal Interaction, I investigated if multimodal output from these devices can influence the perception of haptic feedback. I used a magnitude estimation experiment to evaluate perceived roughness of an ultrasound haptic pattern.

We rendered a circular haptic pattern using a focal point moving at one of three frequencies (a): 50, 70, 90 Hz. This was accompanied by a parametric audio effect (b): noise, tone, or none. Participants moved their hand back and forth across the haptic pattern then rated how rough it felt.

Results suggest that white noise audio from the haptics device increased perceived roughness and pure tones did not, and that lower rendering frequencies may increase perceived roughness.

Scatterplot showing the mean roughness estimates with 95% confidence intervals. X-axis shows rendering frequency with three levels: 50 Hz, 70 Hz, 90 Hz. Y-axis shows normalised roughness estimates on a scale from 0 to 1. The two key trends in the figure are that roughness estimates are higher for the white noise audio condition, and higher for the 50 Hz rendering frequency.

Our results show that multimodal output has the potential to expand the range of sensations that can be presented by an ultrasound haptic device, paving the way to richer mid-air haptic interfaces.

    Enhancing Ultrasound Haptics with Parametric Audio Effects
    E. Freeman.
    In Proceedings of 23rd ACM International Conference on Multimodal Interaction – ICMI ’21, 692-696. 2021.

     PDF       DOI       Website       Data      [Bibtex] 

    @inproceedings{ICMI2021AudioHaptic,
    author = {Freeman, Euan},
    booktitle = {{Proceedings of 23rd ACM International Conference on Multimodal Interaction - ICMI '21}},
    title = {{Enhancing Ultrasound Haptics with Parametric Audio Effects}},
    year = {2021},
    publisher = {ACM},
    pages = {692--696},
    doi = {10.1145/3462244.3479951},
    url = {http://euanfreeman.co.uk/enhancing-ultrasound-haptics-with-parametric-audio-effects/},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2021_AudioHaptic.pdf},
    data = {https://zenodo.org/record/5144878},
}

Acknowledgements

This research has received funding from the 🇪🇺 European Union’s Horizon 2020 research and innovation programme under grant agreement #737087. This work was completed as part of the Levitate project.

Perception of Ultrasound Haptic Focal Point Motion

Overview

Ultrasound haptic patterns can be rendered by continuously moving an ultrasonic focal point. It is not known how this focal point motion affects haptic perception. In a paper at the 2021 ACM International Conference on Multimodal Interaction, we describe two psychophysical experiments investigating the perception of an ultrasound haptic focal point moving along a circular path.

Mid-air haptic patterns can be created by rapidly moving ultrasonic focal points, e.g., along a circular path. In this work, we investigated how such motion is perceived.

Our first experiment finds that a sensation of motion is perceived at speeds up to 17 revolutions per second (17 Hz rendering frequency), similar to the so-called ‘flutter’ sensation associated with low frequency vibrations and movements.

Plot showing the mean threshold render frequencies with 95% confidence intervals. The x-axis shows circle diameter, from 4 to 7 centimetres. The y-axis shows focal point render frequency, from 0 to 18 revolutions per second. The plot shows a mean of approximately 17 revolutions per second for all circle sizes.

Our second experiment found a mostly linear relationship between movement speed and perceived intensity up to this speed.

Plot showing mean intensity estimates for both sized circles with 95% confidence intervals. The x-axis shows rendering frequency, from 5 to 19 revolutions per second. The y-axis shows normalised intensity estimates, from 0 to 1. Plot shows that magnitude estimates increase with frequency, and there are higher magnitude estimates for the larger circle.

Haptic circles are widely used in ultrasound haptic interfaces: e.g., for spherical virtual objects or to give feedback about mid-air gestures. Our results can inform the design of ultrasound haptic interfaces, so that designers can create or avoid the sensation of tactile motion. Motion may be desirable for dynamic feedback: e.g., using below the 17 revolutions per second threshold to create moving patterns to indicate changing values or to accompany animated visual icons. Conversely, designers may wish to emphasise the contiguous outline of a virtual shape by rendering significantly above 17 revolutions per second. Since perceived intensity scales with circle size and rendering frequency, our results can also be used to create perceptually similar haptic objects: i.e., balancing size and frequency to yield similar intensity.

    Perception of Ultrasound Haptic Focal Point Motion
    E. Freeman and G. Wilson.
    In Proceedings of 23rd ACM International Conference on Multimodal Interaction – ICMI ’21, 697-701. 2021.

     PDF       DOI       Website       Data      [Bibtex] 

    @inproceedings{ICMI2021Motion,
    author = {Freeman, Euan and Wilson, Graham},
    booktitle = {{Proceedings of 23rd ACM International Conference on Multimodal Interaction - ICMI '21}},
    title = {{Perception of Ultrasound Haptic Focal Point Motion}},
    year = {2021},
    publisher = {ACM},
    pages = {697--701},
    doi = {10.1145/3462244.3479950},
    url = {http://euanfreeman.co.uk/perception-of-ultrasound-haptic-focal-point-motion/},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2021_Motion.pdf},
    data = {https://zenodo.org/record/5142587},
}

Acknowledgements

This research has received funding from the 🇪🇺 European Union’s Horizon 2020 research and innovation programme under grant agreement #737087. This work was completed as part of the Levitate project.

University of Glasgow logo.
University of Strathclyde logo.

ICMI ’14 Paper Accepted

Posted on by euan

My full paper, “Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions”, was accepted to ICMI 2014. It was also accepted for oral presentation rather than poster presentation, so I’m looking forward to that!

Tactile Feedback for Above-Device Interaction.
Tactile Feedback for Above-Device Interaction.

In this paper we looked at tactile feedback for above-device interaction with a mobile phone. We compared direct tactile feedback to distal tactile feedback from wearables (rings, smart-watches) and ultrasound haptic feedback. We also looked at different feedback designs and investigated the impact of tactile feedback on performance, workload and preference.

ultrasound array
Array of Ultrasound Transducers for Ultrasound Haptic Feedback.

We found that tactile feedback had no impact on input performance but did improve workload significantly (making it easier to interact). Users also significantly preferred tactile feedback to no tactile feedback. More details are in the paper [1] along with design recommendations for above- and around-device interface designers. I’ve written a bit more about this project here.

Video

The following video (including awful typo on the last scene!) shows the two gestures we used in these studies.

References

[1] Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the International Conference on Multimodal Interaction – ICMI ’14, 419-426. 2014.

 PDF       DOI       Website       Video      [Bibtex] 

@inproceedings{ICMI2014,
    author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
    booktitle = {Proceedings of the International Conference on Multimodal Interaction - ICMI '14},
    pages = {419--426},
    publisher = {ACM},
    title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2014.pdf},
    doi = {10.1145/2663204.2663280},
    year = {2014},
    url = {http://euanfreeman.co.uk/projects/above-device-tactile-feedback/},
    video = {{https://www.youtube.com/watch?v=K1TdnNBUFoc}},
}