Home › How do I…? › How to use hand-tracking gestures as input

How to use hand-tracking gestures as input

Advanced Framework Pro recognises hand-tracking gestures from data assets, with no code: finger curl ranges, bone distances and angles, a hold time and a priority. When a gesture is recognised the framework presses a synthetic input key, so a gesture binds in Enhanced Input exactly like a controller button. By the end of this you will have your own gesture driving any input action, on a Quest or any OpenXR headset with hand tracking.

At a glance

DA_GestureDef_<Name>   AFSHandGestureDefinition data asset
├─ finger curl rules, bone distance rules, bone angle rules   (all must pass)
├─ Min Hold Time, Priority
└─ Left Key, Right Key   (synthetic input keys)
Input mapping context › your Input Action ← those keys
Shipped: DA_GestureDef_Pinch, _Point, _Fist, _Grasp, _OpenPalm, _Drag

Steps

  1. Use a VR pawn whose motion controllers carry the hand meshes (BP_MotionController_Hands_Left and _Right). OpenXR Hand Tracking is enabled with the plugin.
  2. Look at the shipped gesture definitions in AFS_Core_MotionController/Data/Gestures: pinch, point, fist, grasp, open palm, drag. Use them directly, or duplicate the closest one.
  3. For a new gesture, create an AFSHandGestureDefinition data asset and add finger curl rules, bone distance rules and bone angle rules. All of them must pass at the same time.
  4. Set Min Hold Time so the gesture does not flicker, and Priority so it wins over gestures that match a similar hand shape.
  5. Assign the left and right synthetic input keys on the definition.
  6. Bind those keys to an input action in your input mapping context, the same way as a controller button.
A gesture definition with curl and distance rules
AFS_Core_Pawn_gesture_dataasset.pngA gesture definition with curl and distance rules
A gesture definition with curl and distance rules

Notes

Abilities never know gestures exist: they react to input actions, so a pinch, a trigger pull and a mouse click can all drive the same thing.

Gesture evaluation runs per hand per frame while hand tracking is active. Keep the set of defined gestures small and let priority and hold time do the work.

A gesture never fires? Its rules cannot all be satisfied at once. Widen a curl range first; those are the most common cause.

Hands do not appear? The motion controller classes are not assigned on the pawn, or OpenXR Hand Tracking is not enabled.

Map_Examples_VRHands shows hand tracking, custom poses and gesture-driven interaction.