Levitating object displays are a novel type of display where content is composed of levitating objects. To find out more about these displays, read my introduction to the Levitate project. This page provides more information and some video demonstrations of the Point-and-Shake interaction technique, described in my CHI 2018 paper .
Point-and-Shake is a technique for selecting levitating objects. Selection is an important interaction, because it needs to happen before users can manipulate or interact with content in a levitating particle display. Users cannot always directly touch objects (e.g., because it disrupts the acoustic levitation forces), so we developed a mid-air gesture technique. All feedback is given through the appearance and behaviour of the levitating objects.
We used ray-cast pointing, enabling users to select “that one there”, by pointing an extended finger towards the target object. Feedback is important, to help users understand how the system is interpreting their actions. The only visual elements in a levitating object display are the objects themselves, so we manipulate the appearance of the objects to give feedback. When the user targets an object, we shake it from side to side as a means of giving feedback. Thus, Point-and-Shake is the combination of pointing gestures with object shaking as feedback. The following video demonstrates this.
Selecting Occluded Objects
A limitation of ray-cast pointing is that users might have trouble selecting occluded objects: i.e., an object hidden behind another. This is because the user cannot directly point at the object without first pointing at others in the way. We implemented two versions of the Lock Ray technique (Grossman et al.), to allow selection of occluded objects. This breaks selection into two stages: 1) aiming towards the intended target; then 2) disambiguating the selection. The following video demonstrates this.
We evaluated Point-and-Shake through two user studies. In these studies, we asked users to select one of two objects using our technique. Object shaking was a successful way of giving feedback about selection. Users completed 94-96% of tasks successfully, within the given task time limits. The mean selection times were 3-4 seconds. Detailed results are described in my CHI 2018 paper .
This research has received funding from the 🇪🇺 European Union’s Horizon 2020 research and innovation programme under grant agreement #737087.
 Point-and-Shake: Selecting from Levitating Object Displays
E. Freeman, J. Williamson, S. Subramanian, and S. Brewster.
In Proceedings of the 36th Annual ACM Conference on Human Factors in Computing Systems – CHI ’18, Paper 18. 2018.