Gesture interfaces let users interact with technology using hand movements and poses. Unlike touch input, gestures can be performed away from devices in the larger space around them. This allows users to provide input without reaching out to touch a device or without picking it up. We call this type of input above-device interaction, as users gesture over devices which are placed on a flat surface, like a desk or table. Above-device gestures may be useful when users are unable to touch a device (when their hands are messy, for example) or when touching a device would be less convenient (when wanting to interact quickly from a distance, for example).
Our research focuses on above-device interaction with mobile devices, such as phones. Most research in this area has focused on sensing gesture interactions. Little is known about how to design above-device gestures which are usable and acceptable to users, which is where our research comes in. We ran two studies to look at above-device gesture design further: we gathered gesture ideas from users in a guessability study and then ran an online survey to evaluate some of these gestures further. You can view this survey here.
The outcomes of these studies are a set of evaluated above-device gestures and design recommendations for designing good above-device interactions. This work was presented at Mobile HCI ’14 as a poster .
We selected two gestures for each mobile phone task from our first study. Gestures were selected based on popularity (called agreement by others) and consistency. Rather than select based on agreement alone, we wanted gestures which could be combined with other gestures in a coherent way. Agreement alone is not a good way of selecting gestures, as our online evaluation actually found that some of the most popular gestures were not as socially acceptable as their alternatives.
We now describe our gestures and link to videos describing them. See our paper  for evaluation results. Click on the gesture names to see a video demonstration.
Swipe: User swipes quickly over the device. Can be from left-to-right or from right-to-left.
Draw Rectangle: User extends their finger and traces a rectangle over the device. Imitates the envelope icon used for messages.
Move Left and Right
Note: We did not look at any specific mapping of gesture direction to navigation behaviour. This seems to be a controversial subject. If a user flicks their hand to the left, should the content move left (i.e. navigate right) or should the viewport move left (i.e. navigate left)?
Scrunch: User holds their hand over the device then makes a fist, as though scrunching up a piece of paper.
Draw X: User extends their finger and draws a cross symbol, as though scoring something out.
Place Phone Call
Dismiss / Close Item
Answer Incoming Call
Ignore Incoming Call
Place Call on Hold
One Moment: User extends their index finger and holds that pose, as though signalling “one moment” to someone.
Lower Hand: User lowers their hand with their fingers fully extended, as though holding something down.
End Current Call
Check Calendar / Query
Accept and Reject
Give non-visual feedback during interaction
Feedback during gestures is important because it shows users that the interface is responding to their gestures and it helps them gesture effectively. However, above-device gestures take place over a phone so visual feedback will not always be visible. Instead, other modalities (like audio or tactile ) should be used.
Make non-visual feedback distinct from notifications
Some participants suggested that they may be confused if feedback during gesture interaction was like feedback used for other mobile phone notifications. Gesture feedback should be distinct from other notification types. Continuous feedback which responds to input would let users know that feedback is being given for their actions.
Emphasise that gestures are directed towards a device
Some participants in our studies were concerned about people thinking they were gesturing at them rather than at a device. Above-device interactions should emphasise gesture target by using the device as a referent for gestures and letting users gesture in close proximity.
Support flexible gesture mechanics
During our guessability study, some participants gestured with whole hand movements whereas others performed the same gestures with one or two fingers. Gestures also varied in size; for example, some participants swiped over a large area and others swiped with subtle movements over the display only. Above-device interfaces should be flexible, letting users gesture in their preferred way using either hand. Social situation may influence gesture mechanics. For example, users in public places may use more subtle versions of gestures than they would at home.
Enable complex gestures with a simple gating gesture
Our participants proposed a variety of gestures, from basic movements with simple sensing requirements, to complex hand poses requiring more sophisticated sensors. Always-on sensing with complex sensors will affect battery. Sensors with low power consumption (like the proximity sensor, for example) could be used to detect a simple gesture which then enables more sophisticated sensors. Holding a hand over the phone or clicking fingers, for example, could start a depth camera which could track the hand in greater detail.
Use simple gestures for casual interactions
Casual interactions (such as checking for notifications) are low-effort and imprecise so should be easy to perform and sense. Easily sensed gestures lower power requirements for input sensing and allow for variance in performance when gesturing imprecisely. Users may also use these gestures more often when around others so allowing variance lets users gesture discreetly, in an acceptable way.
 Towards Usable and Acceptable Above-Device Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Mobile HCI ’14 Posters, 459-464. 2014.
 Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
E. Freeman, S. Brewster, and V. Lantz.
In Proceedings of the International Conference on Multimodal Interaction – ICMI ’14, 419-426. 2014.
This research was part funded by Nokia Research Centre, Finland. We would also like to thank everyone who participated in our studies.