# Heuristics for smarter Leap Motion tracking

Leap Motion is quite easy to develop for but I find that hand and finger tracking can be tricky – especially for more precise gesture control. I’ve been doing some development with Leap that requires precise positioning (~mm accuracy needed) with multiple fingers. In this post I run through a few strategies I use to try and improve my Leap applications.

###### 1) Track hand and finger identifiers

Each Finger and Hand object has a unique identifier which should remain consistent between frames, so long as the object continues to be tracked. If you can make a reasonable assumption about which finger(s) the user is intending to use for interaction, you can look for a specific identifier to find the finger(s) of interest, ignoring anything else in the frame. I find that when extending their index finger for interaction, most users tend to hold their thumb out unwittingly; which Leap detects, of course.

I use a simple heuristic to try address this issue: pick the finger with the smallest z-coordinate (furthest forward) and track its identifier in future frames. When I want to track multiple fingers for interaction, I make sure those fingers are attached to the hand whose identifier I’m tracking. This rejects other unattached fingers in the field of view, e.g. the cylindrical lights in my office at university! (Seriously…)

###### 2) Check finger lengths

Sometimes Leap sees your knuckles in a closed fist and reports them as fingers. The looser the grip, the worse the problem. Checking that the length of each finger is above a threshold length can help you reject misclassified knuckles and clenched fingers. A threshold of 2 cm or thereabouts should be fine; any longer and you risk rejecting small fingers or fingers which are fully extended but Leap just isn’t tracking properly.

###### 3) Look at direction vectors for unlikely values

The biggest problem I have with Leap is that it picks up fingers in a closed fist. Looking at the debug visualiser, I tend to see fingers pointing upwards through the fist, rather than an accurate representation of a finger in a clenched fist. These “phantom” fingers tend to have extreme y values in the direction vector (> 0.6). Through observation of some of my users I’ve also noticed that when using a certain hand pose for input, “inactive” fingers are left to hang down; which Leap picks up perfectly. These fingers typically point downwards, with a high y value in the direction vector (< -0.6).

A heuristic I use to reject “phantom” fingers and fingers the user probably doesn’t want to use for input is to take the absolute y value from the direction vector and ignore the finger if this value exceeds a threshold. I use a threshold around 0.4. A limitation of this heuristic is that the user has to keep their hand quite flat. To address this, you could instead measure finger direction relative to the hand orientation and then apply the threshold.

###### Summary

Sometimes Leap gives crap data and we also can’t expect our users to keep a perfectly clenched fist. A few simple checks on each frame update can help us reject fingers which either don’t exist or weren’t intended to be part of the interaction.

# Leaping into motion

My Leap Motion arrived today. I’ve been working with a development version of the hardware for a couple of months now but I was excited to see what improvements the retail version brought. Other than the obvious aesthetic differences (I’m a fan of the brushed aluminium look!), the retail version has a wider field of view. While this doesn’t affect use of the device much, it does mean that it’s less likely to lose sight of your hands when you move too far away whilst close to the desk. Based on the diagnostic visualiser, it also seems to be more accurate and deals with hand rotation better (although this is likely a software rather than hardware improvement).

Having tried some of the apps in the Airspace app store I’m surprised at the large variety of pointing and selection mechanisms being used by developers. Using the finger as a pointer to control the cursor seems easy enough (although it quickly becomes fatiguing after extended periods!), and most developers seem consistent in how they implement this. Getting the control:display gain correct is going to be a key challenge, as some apps require only the most subtle of finger movements to send the cursor flying accidentally. Smoothing the cursor position (e.g. using a Kalman filter) would help to deal with cursor jitter and unexpected movement.

There are a variety of selection methods being used, including finger poke and tap (two gestures recognised by the base SDK), dwell, and opening the hand. The latter selection gesture is used in the Jazz Hands game and works surprisingly well. While pointing to control a cursor, the user can make a selection by opening their hand up. While there is a little inadvertent movement of the cursor as a result of the gesture, it doesn’t require any direct movement of the pointing finger.

I’m not a fan of the poke and tap gestures but that might just be because I’ve yet to try a good implementation of them. I’ve implemented my own recogniser for tap gestures a couple of weeks ago and know how challenging it can be to get it feeling “right”.

Dwell selection is generally very accurate and I think that this is going to be one of the best selection methods for Leap going forward. This is largely because it allows the user to leap into an app (a terrible pun, I know) without having to figure out how to make selections. The immediacy of the visual feedback makes it obvious that by remaining over a button, something is taking place.

I’m excited about the future of Leap and other gesture interfaces because they offer the potential to create new ways of interacting with computers. It’s not the future, but it’s a step in the right direction.