Hand Gesture Tracking for Musical Performance
An technical reference describing the real-time hand gesture recognition system used to control the Ranat Ek — a traditional Thai xylophone — within the Flow3D gesture-controlled synthesizer environment.
Contents
- 1. System Overview
- 2. Hand Tracking Pipeline
- 3. Gesture Detection: Finger Verticality
- 4. The Three Performance Modes
- 5. Pitch, Tempo & Dynamics Mapping
- 6. Musical Scale Integration
- 7. Audio Engine & Sample Playback
- 8. Summary of Mappings
- 9. Nang Yai Shadow Puppetry: Cross-Domain Gesture Mapping
1. System Overview
The Flow3D Ranat Ek module translates bare-hand gestures captured via a standard webcam into expressive musical performance on a sample-based virtual instrument. No wearable sensors or controllers are required — the performer's hands are the sole interface. The system operates in real time, mapping spatial hand position, finger orientation, and inter-hand distance to discrete musical events at a constant rhythmic clock of approximately four notes per second.
The architecture consists of three layers: a vision layer (MediaPipe Hands) that extracts 21 3D landmarks per hand, a geometry layer that converts landmarks into directional finger lines and computes spatial relationships, and a synthesis layer (RanatEkSynth) that interprets these geometric features as musical control parameters and triggers corresponding audio samples.
2. Hand Tracking Pipeline
Hand skeleton tracking is performed by Google's MediaPipe Hands library, which provides 21 three-dimensional landmarks per detected hand at video frame rate. Each landmark is normalized to the camera frame: x (horizontal, 0–1), y (vertical, 0–1), and z (relative depth). The system supports up to two hands simultaneously.
For each hand, five finger lines are generated from the fingertip landmark to the proximal joint of the same finger, using the following landmark pairs:
| Finger | Tip Landmark | Joint Landmark |
|---|---|---|
| Thumb | #4 | #3 |
| Index | #8 | #7 |
| Middle | #12 | #11 |
| Ring | #16 | #15 |
| Pinky | #20 | #19 |
These landmark coordinates are re-projected into a centered 3D space (×10 scale) to form directional line segments, each characterized by a start point, an end point, an angle, a length, and a verticality measure.
3. Gesture Detection: Finger Verticality
The core gesture classification metric is verticality — a dimensionless value indicating how upright a finger line is relative to the horizontal camera plane. It is computed as:
where Δx and Δy are the horizontal and vertical components of the finger line projected onto the camera plane. A verticality near 1.0 means the finger points straight up (vertical); a value near 0.0 means the finger lies flat (horizontal).
The system averages the verticality across all five fingers for each hand, producing a single per-hand verticality score. Two thresholds classify each hand's orientation:
- Flat / Horizontal: average verticality < 0.4
- Upright / Vertical: average verticality > 0.6
- Ambiguous zone (0.4–0.6): the current gesture mode is held to prevent rapid oscillation
The combination of both hands' orientations determines which of three performance modes is active (detailed in the next section).
4. The Three Performance Modes
The Ranat Ek supports three discrete gesture modes, each triggered by a distinct spatial configuration of the performer's hands. Mode switches occur in real time as the performer reorients their fingers.
Step Mode (Direct Tracking)
Trigger: Both hands with fingers held upright / vertical (verticality > 0.6)
The most direct mapping mode. The note is determined continuously and directly from the performer's hand height — there is no sequential stepping or advancement. As the hand moves up, the pitch rises; as it moves down, the pitch falls. The note at each rhythmic tick is whatever note corresponds to the current hand height, allowing the performer to 'draw' a melody with vertical hand motion.
Skip Mode (Alternating Thirds)
Trigger: Both hands with fingers held flat / horizontal (verticality < 0.4)
An alternating-note pattern that produces a melodic 'skip' interval. The system oscillates between a base note (determined by hand height) and the note two scale degrees above it, producing a 1–3–1–3–1–3... pattern. As the performer raises or lowers their hand, the base note pair shifts accordingly, allowing the alternating interval to traverse the full range of the instrument while maintaining its characteristic leap pattern.
Tremolo Mode (Dual-Hand Split Control)
Trigger: Hands held perpendicular — one hand flat/horizontal, the other upright/vertical
A split-control mode that divides pitch and speed across the two hands. The flat (horizontal) hand dictates pitch via its height (Y axis) — raising the flat hand selects higher notes, lowering it selects lower notes. The vertical (upright) hand controls the tremolo speed via its height (Y axis) — raising the vertical hand accelerates the tremolo, lowering it slows the tremolo. The note is rapidly retriggered at double speed, producing the characteristic trembling effect.
5. Pitch, Tempo & Dynamics Mapping
Pitch
Pitch is derived from the performer's hand position, but the spatial axis used depends on the active mode:
- Step & Skip modes: pitch is mapped from the average vertical (Y) height of both hands.
- Tremolo mode: pitch is mapped from the flat (horizontal) hand's height (Y) position.
The continuous hand position is normalized to a 0–1 range and then quantized to the nearest available note in the selected musical scale. The vertical detection range is compressed (a sub-range of the full camera frame) so that the performer can comfortably reach the entire pitch range without extreme hand movements.
Tempo (Rhythmic Clock)
The internal clock fires at a base rate of approximately four notes per second. This rate is modulated by the distance between the two hands: closer hands produce a faster tempo (shorter inter-note interval), while hands farther apart produce a slower, more spacious tempo. The distance is computed in 3D space using all three spatial axes.
In Tremolo mode, the clock runs at double speed and is further modulated by the vertical hand's height, allowing the performer to independently control tremolo rate. Both Step and Skip modes run at roughly 1.8× the base clock speed for a livelier, more dynamic feel.
Volume / Dynamics
Amplitude is inversely related to inter-hand distance: hands close together produce louder notes (up to ~0.9 gain), while hands far apart produce softer notes (down to ~0.4 gain). This mirrors the physical intuition of a performer leaning into an instrument with intensity versus playing from a relaxed distance.
A hands-lost diminuendo provides a natural phrase-ending gesture across all three modes. When one or both hands leave the camera frame, the engine repeats the last-played note at the current rhythm while fading its amplitude to silence over 2 seconds, then stops entirely. This mirrors the natural decay of a struck Ranat Ek bar and gives the performer a simple, reliable way to end a musical phrase — simply withdraw the hands — without relying on fist-detection, which proved unreliable across hand orientations.
Velocity / Articulation
The system supports multiple velocity layers in the sample library (mapped at MIDI velocities 60, 90, and 127). Hand movement speed and openness are tracked frame-to-frame to influence velocity selection, enabling dynamic articulation from the gesture itself.
6. Musical Scale Integration
All pitch mappings are constrained to a user-selected musical scale, ensuring that every gesture-triggered note is harmonically valid regardless of hand position. The available scales and their interval structures (in semitones from the root) are:
| Scale | Intervals (semitones) |
|---|---|
| Pentatonic | 0, 2, 4, 7, 9 |
| Major | 0, 2, 4, 5, 7, 9, 11 |
| Minor | 0, 2, 3, 5, 7, 8, 10 |
| Chromatic | 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 |
| Phrygian | 0, 1, 3, 5, 7, 8, 10 |
| Lydian | 0, 2, 4, 6, 7, 9, 11 |
| Blues | 0, 3, 5, 6, 7, 10 |
| Persian | 0, 1, 4, 5, 6, 8, 11 |
The pitch-to-note quantization function filters the available sample notes to those belonging to the selected scale, sorts them ascending, and maps the normalized hand position to an index within that filtered set. If no samples match the selected scale, the full available note set is used as a fallback.
7. Audio Engine & Sample Playback
The Ranat Ek is a sample-based instrument. Audio samples are uploaded by the user and organized by MIDI note number and velocity layer, parsed from the filename (e.g., C4_v90.aif). Both AIFF/AIF and standard audio formats are supported via a custom decoder for AIFF files and native browser decoding for WAV/MP3.
On each rhythmic tick, the synthesis engine:
- Determines the active gesture mode from finger verticalities.
- Computes the target pitch from the appropriate hand axis (mode-dependent).
- Selects the best available velocity layer for articulation.
- Stops any currently playing voice (monophonic playback).
- Creates a new
AudioBufferSourceNodefrom the matched sample. - Routes it through a gain node (for dynamics) into the master output and effects chain.
- Cleans up finished sources to manage memory.
The audio signal passes through a global effects engine offering delay (time, feedback, wet/dry mix) and reverb (room size, decay, density, wet/dry mix) before reaching the output, enabling spatial sculpting of the Ranat Ek timbre.
8. Summary of Mappings
The following table consolidates all gesture-to-music mappings across the three performance modes:
| Parameter | Step (Vertical) | Skip (Horizontal) | Tremolo (Perpendicular) |
|---|---|---|---|
| Pitch source | Hand height (Y) | Hand height (Y) | Flat hand height (Y) |
| Note pattern | Direct tracking | Alternating 1–3–1–3 | Rapid repeat (tremolo) |
| Tempo control | Inter-hand distance | Inter-hand distance | Distance + vert. hand Y |
| Volume | Inter-hand distance | Inter-hand distance | Inter-hand distance |
| Clock rate | ~7 notes/sec (1.8× base) | ~7 notes/sec (1.8× base) | ~8 notes/sec (double) |
| Hands-lost diminuendo | Repeat last note, fade to silence over 2 s when hands leave frame | ||
9. Nang Yai Shadow Puppetry: Cross-Domain Gesture Mapping
Nang Yai (หนังใหญ่, "large shadow puppet") is a centuries-old Thai shadow-puppet tradition in which puppeteers manipulate life-sized, intricately carved leather figures against an illuminated screen, accompanied by a live piphat ensemble that includes the Ranat Ek. The puppeteers' movements — the angularity, height, speed, and openness of the figures — are choreographed in tight synchrony with the music. This section explores how the gesture-tracking vocabulary of this synthesizer could map to and from the physical choreography of Nang Yai performance.
Shared Gesture Vocabulary
Both the digital gesture system and Nang Yai puppetry rely on the performer's upper-body geometry — hand height, finger orientation, arm distance, and movement speed — as expressive primitives. The table below draws conceptual parallels:
| Synth Gesture | Musical Result | Nang Yai Choreographic Analogue |
|---|---|---|
| Vertical hand (Step) | Direct pitch tracking | Puppeteer raises/lowers the puppet figure to mark melodic contour |
| Horizontal hand (Skip) | Alternating interval leaps | Side-to-side sweeps of the puppet, evoking character dialogue turns |
| Perpendicular hands (Tremolo) | Rapid repeat, split pitch/speed control | One puppet holds steady posture while the other trembles/shimmers |
| Hand closing | Diminuendo (volume fade) | Puppeteer draws the puppet toward the body, diminishing its screen presence |
| Inter-hand distance | Tempo & volume | Distance between two puppets on stage — closer = more intense scene |
Bidirectional Mapping Potential
The mapping can flow in both directions. Forward (gesture → music): a puppeteer wearing motion-tracked gloves could drive the Ranat Ek synthesizer directly, so the choreographed dance of the puppets becomes a live musical performance — the puppets "play" the instrument. Reverse (music → gesture): a pre-recorded or live Ranat Ek performance could be analysed in real time and used to animate virtual Nang Yai puppets, translating pitch into puppet height, rhythmic density into movement speed, and dynamic swells into puppet proximity to the screen.
This bidirectionality suggests a hybrid performance paradigm: a single performer controls both the visual choreography and the musical accompaniment through one unified gesture language, dissolving the traditional boundary between puppeteer and musician. Just as the Nang Yai puppeteer and the piphat ensemble are locked in mutual responsiveness, the digital system treats visual motion and sonic output as two facets of a single expressive gesture.
Further Reading
Flow3D — Gesture-Controlled Musical Synthesizer · Ranat Ek Module
Documentation generated for academic reference purposes.