Tag: wearable

A New Smart-Watch Design Space?

Posted on by euan

Almost exactly a year ago I wrote about my first impressions of Pebble and concluded that “I have to wonder if smart-watches even need a display“. As a smart-watch, I found Pebble most useful for remotely controlling my phone (through its physical buttons) and for promoting awareness of notifications on my phone (through its vibration alerts); its “clunky and awkward user interface” was even detrimental to its other, more important, function as an ordinary watch.

With that in mind, I was excited by Yahoo! Labs recent paper at Tangible, Embedded, and Embodied Interaction (or TEI): Shimmering Smartwatches. In it, they present two prototype smart-watches which don’t have a screen, instead using less sophisticated (but just as expressive and informative) LEDs.

One of their prototypes, Circle, used a circular arrangement of twelve LEDs, each in place of an hour mark on the watch-face. By changing the brightness and hue of the LEDs, the watch was able to communicate information from smart-watch applications, like activity trackers and countdown timers. Their other prototype used four LEDs placed behind icons on the watch-face. Again, brightness and hue could be modulated to allow greater information to be communicated about each of the icons.

I really like the ideas in this paper and its prototypes. High resolution displays are more expensive than simple LED layouts, require more power and are not necessarily more expressive. Hopefully someone builds on the new design space presented by Shimmering Smartwatches, which can certainly be expressive but also lower cost. Also, everything is better with coloured LEDs.

Above-Device Tactile Feedback

Introduction

My PhD research looks at improving gesture interaction with small devices, like mobile phones, using multimodal feedback. One of the first things I looked at in my PhD was tactile feedback for above-device interfaces. Above-device interaction is gesture interaction over a device; for example, users can gesture at a phone on a table in front of them to dismiss unwanted interruptions or could gesture over a tablet on the kitchen counter to navigate a recipe. I look at above-device gesture interaction in more detail in my Mobile HCI ’14 poster paper [1], which gives a quick overview of some prior work on above-device interaction.

Tactile Feedback for Above-Device Interaction

In two studies, described in my ICMI ’14 paper [2], we looked at how above-device interfaces could give tactile feedback. Giving tactile feedback during gestures is a challenge because users don’t touch the device they are gesturing at; tactile feedback would go unnoticed unless users were holding the device while they gestured. We looked at ultrasound haptics and distal tactile feedback from wearables. In our studies, users interacted with a mobile phone interface (pictured above) which used a Leap Motion to track two selection gestures.

Gestures

An illustration of the count gesture. The user has extended four fingers on their right hand, thus selecting the fourth target.

Our studies looked at two selection gestures: Count (above) and Point (below). These gestures were from our user-designed gesture study [1]. With Count, users select from numbered targets by extending the appropriate number of fingers. When there’s more than five targets, we partition targets into groups. Users can select from a group by moving their hand. In the image above, the palm position is closest to the bottom half of the screen so we activate the lower group of targets. If users moved their hands towards the upper half of the screen, we would activate the upper group of four targets. Users had to hold a Count gesture for 1000 ms to make a selection.

Illustration of the point gesture. A hand with an extended index finger is selecting one of the on-screen targets, using a circular cursor.

With Point, users controlled a cursor which was mapped to their finger position relative to the device. We used the space beside the device to avoid occluding the screen while gesturing. Users made selections by dwelling the cursor over a target for 1000 ms.

For a video demo of these gestures, see:

Tactile Feedback

In our first study we looked at different ways of giving tactile feedback. We compared feedback directly from the device when held, ultrasound haptics (using an array of ultrasound transducers, below) and distal feedback from wearable accessories. We used two wearable tactile feedback prototypes: a “watch” and a “ring” (vibrotactile actuators affixed to a watch strap and an adjustable velcro ring). We found that all were effective for giving feedback, although participants had divided preferences.

A photograph of an ultrasound haptics device.

Some preferred feedback directly from the phone because it was familiar, although this is an unlikely case in above-device interaction because an advantage of this interaction modality is that users don’t need to first lift the phone or reach out to touch it. Some participants liked feedback from our ring prototype because it was close to the point of interaction (when using Point) and others preferred feedback from the watch (pictured below) because it was a more acceptable accessory than a vibrotactile ring. An advantage of ultrasound haptics is that users do not need to wear any accessories and participants appreciated this, although the feedback was less noticeable than vibrotactile feedback. This was partly because of the small ultrasound array used (similar size to a mobile phone) and partly because of the nature of ultrasound haptics.

Tactile Watch Prototype

In a second study we focused on feedback given on the wrist using our watch prototype. We were interested to see how tactile feedback affected interaction using our Point and Count gestures. We looked at three tactile feedback designs in addition to just visual feedback. Tactile feedback had no impact on performance (possibly because selection was too easy) although it had a significant positive effect on workload. Workload (measured using NASA-TLX) was significantly lower when dynamic tactile feedback was given. Users also preferred to receive tactile feedback to no tactile feedback.

A more detailed qualitative analysis and the results of both studies appear in our ICMI 2014 paper [2]. A position paper [3] from the CHI 2016 workshop on mid-air haptics and displays describes this work in the broader context of research towards more usable mid-air widgets.

Tactile Feedback Source Code

A Pure Data patch for generating our tactile feedback designs is available here.

References

[1] Towards Usable and Acceptable Above-Device Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Mobile HCI ’14 Posters, 459-464. 2014.

 PDF       DOI      [Bibtex] 

@inproceedings{MobileHCI2014Poster,
  author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
  booktitle = {Mobile HCI '14 Posters},
  pages = {459--464},
    publisher = {ACM},
  title = {Towards Usable and Acceptable Above-Device Interactions},
  pdf = {http://research.euanfreeman.co.uk/papers/MobileHCI_2014_Poster.pdf},
  doi = {10.1145/2628363.2634215},
  year = {2014},
}

[2] Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the International Conference on Multimodal Interaction – ICMI ’14, 419-426. 2014.

 PDF       DOI       Website       Video      [Bibtex] 

@inproceedings{ICMI2014,
    author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
    booktitle = {Proceedings of the International Conference on Multimodal Interaction - ICMI '14},
    pages = {419--426},
    publisher = {ACM},
    title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2014.pdf},
    doi = {10.1145/2663204.2663280},
    year = {2014},
    url = {http://euanfreeman.co.uk/projects/above-device-tactile-feedback/},
    video = {{https://www.youtube.com/watch?v=K1TdnNBUFoc}},
}

[3] Towards Mid-Air Haptic Widgets
E. Freeman, D. Vo, G. Wilson, G. Shakeri, and S. Brewster.
In CHI 2016 Workshop on Mid-Air Haptics and Displays: Systems for Un-instrumented Mid-Air Interactions. 2016.

 PDF      [Bibtex] 

@inproceedings{MidAirHapticsWorkshop,
    author = {Freeman, Euan and Vo, Dong-Bach  and Wilson, Graham and Shakeri, Gozel and Brewster, Stephen},
    booktitle = {CHI 2016 Workshop on Mid-Air Haptics and Displays: Systems for Un-instrumented Mid-Air Interactions},
    title = {{Towards Mid-Air Haptic Widgets}},
    year = {2016},
    pdf = {http://research.euanfreeman.co.uk/papers/MidAirHaptics.pdf},
}

ICMI ’14 Paper Accepted

Posted on by euan

My full paper, “Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions”, was accepted to ICMI 2014. It was also accepted for oral presentation rather than poster presentation, so I’m looking forward to that!

Tactile Feedback for Above-Device Interaction.
Tactile Feedback for Above-Device Interaction.

In this paper we looked at tactile feedback for above-device interaction with a mobile phone. We compared direct tactile feedback to distal tactile feedback from wearables (rings, smart-watches) and ultrasound haptic feedback. We also looked at different feedback designs and investigated the impact of tactile feedback on performance, workload and preference.

ultrasound array
Array of Ultrasound Transducers for Ultrasound Haptic Feedback.

We found that tactile feedback had no impact on input performance but did improve workload significantly (making it easier to interact). Users also significantly preferred tactile feedback to no tactile feedback. More details are in the paper [1] along with design recommendations for above- and around-device interface designers. I’ve written a bit more about this project here.

Video

The following video (including awful typo on the last scene!) shows the two gestures we used in these studies.

References

[1] Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the International Conference on Multimodal Interaction – ICMI ’14, 419-426. 2014.

 PDF       DOI       Website       Video      [Bibtex] 

@inproceedings{ICMI2014,
    author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
    booktitle = {Proceedings of the International Conference on Multimodal Interaction - ICMI '14},
    pages = {419--426},
    publisher = {ACM},
    title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
    pdf = {http://research.euanfreeman.co.uk/papers/ICMI_2014.pdf},
    doi = {10.1145/2663204.2663280},
    year = {2014},
    url = {http://euanfreeman.co.uk/projects/above-device-tactile-feedback/},
    video = {{https://www.youtube.com/watch?v=K1TdnNBUFoc}},
}

What Is Around-Device Interaction?

Posted on by euan

One of my biggest research interests is gesture interaction with mobile devices, also known as around-device interaction because users interact in the space around the device rather than on the device itself. In this post I’m going to give a brief overview of what around-device interaction is, how gestures can be sensed from mobile devices and how these interactions are being realised in commercial devices.

Why Use Around-Device Interaction?

Why would we want to gesture with mobile devices (such as phones or smart watches) anyway? These devices typically have small screens which we interact with in a very limited fashion; using the larger surrounding space lets us interact in more expressive ways and lets the display be utilised fully, rather than our hand occluding content as we reach out to touch the screen. Gestures also let us interact without having to first lift our device, meaning we can interact casually from a short distance. Finally, gesture input is non-contact so we can interact when we would not want to touch the screen, e.g. when preparing food and wanting to navigate a recipe but our hands are messy.

Sensing Around-Device Input

Motivated by the benefits of expressive non-contact input, HCI researchers have developed a variety of approaches for detecting around-device input. Early approaches used infrared proximity sensors, similar to the sensors used in phones to lock the display when we hold our phone to our ear. SideSight (Butler et al. 2008) placed proximity sensors around the edges of a mobile phone, letting users interact in the space beside the phone. HoverFlow (Kratz and Rohs 2009) took a similar approach, although their sensors faced upwards rather than outwards. This let users gesture above the display. Although this meant gesturing occluded the screen, users could interact in 3D space; a limitation of SideSight was that users were more or less restricted to a flat plane around the phone.

Abracadabra (Harrison and Hudson 2009) used magnetic sensing to detect input around a smart-watch. Users wore a magnetic ring which affected the magnetic field around the device, letting the watch determine finger position and detect gestures. This let users interact with a very small display in a much larger area (an example of what Harrison called “interacting with small devices in a big way” when he gave a presentation to our research group last year) – something today’s smart-watch designers should consider. uTrack (Chen et al. 2013) built on this approach with additional wearable sensors. MagiTact (Ketabdar et al. 2010) used a similar approach to Abracadabra for detecting gestures around mobile phones.

So far we’ve looked at two approaches for detecting around-device input: infrared proximity sensors and magnetic sensors. Researchers have developed camera-based approaches for detecting input. Most mobile phone cameras can be used to detect around-device gestures within the camera field of view, which can be extended using approaches such as Surround-see (Yang et al. 2013). Surround-see placed an omni-directional lens over the camera, giving the phone a complete view of its surrounding environment. Users could then gesture from even further away (e.g. across the room) because of the complete field of view.

Others have proposed using depth cameras for more accurate camera-based hand tracking. I was excited when Google revealed Project Tango earlier this year because a mobile phone with a depth sensor and processing resources dedicated to computer vision is a step closer to realising this type of interaction. While mobile phones can already detect basic gestures using their magnetic sensors and cameras, depth cameras, in my opinion, would allow more expressive gestures without having to wear anything (e.g. magnetic accessories).

We’re also now seeing low-powered alternative sensing approaches, such as AllSee (Kellogg et al. 2014) which can detect gestures using ambient wireless signals. These approaches could be ideal for wearables which are constrained by small battery sizes. Low-power sensing could also allow always-on gesture sensing; this is currently too demanding with some around-device sensing approaches.

Commercial Examples

I have so far discussed a variety of sensing approaches found in research; this is by no means a comprehensive survey of around-device gesture recognition although it shows the wide variety of approaches possible and identifies some seminal work in this area. Now I will look at some commercial examples of around-device interfaces to show that there is an interest in moving interaction away from the touch-screen and into the around-device space.

Perhaps the best known around-device interface is the Samsung Galaxy S4. Samsung included features called Air View and Air Gesture which let users gesture above the display without having to touch it. Users could hover over images in a gallery to see a larger preview and could navigate through a photo album by swiping over the display. A limitation of the Samsung implementation was that users had to be quite close to the display for gestures to be detected – so close that they may as well have used touch input!

Nokia also included an around-device gesture in an update for some of their Lumia phones last year. Users could peek at their notifications by holding their hand over the proximity sensor briefly. While just a single gesture, this let users check their phones easily without unlocking them. With young smartphone users reportedly checking their phones more than thirty times per day (BBC Newsbeat, 4th April 2014), this is a gesture that could get a lot of use!

There are also a number of software libraries which use the front-facing camera to detect gesture input, allowing around-device interaction on typical mobile phones.

Conclusion

In this post we took a quick look at around-device interaction. This is still an active research area and one where we are seeing many interesting developments – especially as researchers are now focusing on issues other than sensing approaches. With smartphone developers showing an interest in this modality, identifying and overcoming interaction challenges is the next big step in around-device interaction research.

Round-faced Smart Watches

Posted on by euan

A photo of a Motorola 360 smart watch.

When Motorola announced their Moto 360 watch (above) recently, many hailed it as a great moment in wearable design – finally, someone designed a watch which actually looks like a watch! It’s a big step forward from the uninspiring square-faced designs which have come before it. The timing of the announcement was perfect because people were still awing over a concept design which appeared a few days earlier, also featuring a circular display which imitates traditional watch design.

At the same time, Google also announced their Android Wear platform for wearables (used by the Moto 360 amongst others). At the moment the platform seems focused on three areas: card-based notification design, and touch-gesture and speech input. I’d love to see if Google eventually follow up on their patent for on-arm input and provide future platform support for novel input on the body or in the space around the watch.

TEDx Demos

Posted on by euan

TEDx badge

A couple of days ago our group showed off some of our research during TEDx Glasgow University. It was a fun experience. We don’t often get to engage with a non-academic audience so it was refreshing to chat about future technology with non-computing scientists. I came away from the demo session feeling inspired and with some fresh ideas about where to take my research.

I presented a gesture interface for mobile phones, using around-device gestures as input. People seemed to find this modality particularly attractive for use in the kitchen, when hands are often full, wet or messy. I was also showing how wearables could be used alongside mobile phones. People seemed to enjoy the novelty of this, although understandably there was some doubt about having to wear another accessory (alongside fashion items like watches or bracelets). I definitely feel that future wearables need to be designed with fashion in mind so people want to wear them as accessories first and interfaces second.

Inevitably someone mentioned interfaces found in Minority Report and Iron Man – I hope HCI finds ways to inspire people’s imaginations about interfaces of the future in the same way that Hollywood has.

New Wearable Tech: On-Body Input and Pop-out Earpieces

Posted on by euan

This week I’ve seen two wearable device concepts which I really like: extending the input space using the arm and detachable ear-pieces letting the device be used for phone calls. The first is demonstrated in the following tweet, showing an excerpt from a patent application attributed to Google:

Here the arm is used for extra input space, keeping the small display entirely visible during interaction. This style of input is similar to SideSight [1], a research prototype which used proximity sensors on the side of a phone to detect pointer input beside the device. I like the idea of interacting on the arm rather than in the space around the watch (e.g. Abracadabra [2]) because tactile cues from pressing against your own body could make it easier to interact [3].

Huawei’s new wearable wristband features a pop-out earpiece (the display) which lets you use your phone without taking it out of your pocket. While this is hardly a groundbreaking idea (it’s basically a bluetooth headset that you don’t wear on your head), it at least justifies using a wearable to give you incoming call alerts. While Pebble, for example, gives call notifications on the watch display, you’d have to take your phone out of your pocket to use it anyway.

[1] Butler, A., Izadi, S. and Hodges, S.: SideSight: Multi-“touch” Interaction Around Small Devices. In Proc. of UIST ’08 (2008), p. 201-204.
[2] Harrison, C. and Hudson, S.: Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proc. of UIST ’09(2009), p. 121-124.
[3] Gustafson, S., Rabe, B. and Baudisch, P.: Understanding Palm-Based Imaginary Interfaces: The Role of Visual and Tactile Cues when Browsing. In Proc. of CHI ’13 (2013), p. 889-898.

First Impressions of Pebble

Posted on by euan

pebble-watch-5-colors-w

 

Smart-watches seem to be quite trendy at the moment, no doubt thanks to the Kickstarter success of Pebble showing that there’s growing interest in wearable computers. I’m quite interested in wearable technology and have thrown together really low-fidelity prototypes of wearables in the past for research projects, using elastic bands, velcro and old watch straps. To bring my research back to the twenty-first century we picked up a Pebble, mostly just to see what it can do. In this post I ramble about my first impressions of Pebble and talk about what I want from a “smart”-watch.

I’ve been wearing it for the past few days now – pretty much from when I wake up to when I go to sleep. My initial impression is that the watch needn’t even have a display – the things I find it most useful for (and the things it does the best) are all non-visual.

I find Pebble to be of limited use as an input device; however, the hardware buttons are a nice size and are easy to use without looking at the watch. I use my phone for music whilst driving and have never been able to skip tracks, although Pebble now makes this possible. It’s easy to keep one hand on the steering wheel and use the other to press the “next track” button on the watch, without taking my eyes off the road.

The other thing I find Pebble useful for is knowing when I have notifications on my phone to attend to. This is largely thanks to the vibrotactile alerts – the display itself is of limited use because of its small size. While the vibrotactile notifications are useful, they lack customisation (although I hope this is something which changes in future iterations of the Pebble software). All notifications seem to have the same vibrotactile pattern, so it’s impossible to tell the difference between a text and an email without looking at the display.

With these two uses – hardware buttons to control another device and vibrotactile notifications – in mind I have to wonder if smart-watches even need a display. I’m completely unconvinced by the clunky and awkward user interface the watch provides and would much prefer a “normal” watch which connects to my phone (or other devices) so I can remotely control them and receive notifications from them.

mens-proximity-watch-citizen

Which brings me to Citizen’s Proximity watch (above). This is the first “smart-watch” I’ve seen which still looks like a normal watch. It connects to smartphones and delivers notifications using vibrotactile feedback and simple visual feedback on the watch-face. When a notification is delivered, one of the watch-hands jumps to a text label on the display, showing the notification type. If only those two chunky buttons could also be put to good use!