Camera & FX

Put yourself on the stage — and the stage over your own piano

Draw a live webcam or a looping video file onto the stage in any corner, key out a green backdrop, and turn on hand tracking: Keyflow reads your hands straight out of the camera frame with classical computer vision — no model, no download, nothing leaves your machine. The AR overlay goes further and draws the falling notes over your real keyboard, calibrated from four marked corners.

Live camera or video file4 corners · up to 60 fpsGreen key, 0–100 toleranceAR overlay on the real keysHand tracking, no ML model
The live stage where the camera overlay is drawnCamera & FX
The live stage: the camera overlay is composited into a corner of this same frame, the hand marker sits on the keys themselves, and the AR overlay puts this whole picture over your own keyboard.
60fps the camera can reach the stage at
4corners to mark for the AR calibration
0–300 mscamera delay, remembered per camera
0models to download

Webcam overlay

A camera window on the stage, keyed if you want it

  • Camera overlay draws a live camera or a looping video file onto the stage as a small window. The list is the machine's own: one line per camera with its real name (never “Camera 1”), kind, port and VID/PID — and Automatic prefers colour cameras, ranks virtual ones after them and infrared (Windows Hello) cameras last.
  • Plugging a camera in or out, or starting and stopping OBS Virtual Camera, updates the list and reopens the camera when it has to — but never while a take is being recorded, the delay is being measured or the keys are being marked.
  • For a live camera the app picks the mode nearest 720p at the highest rate the camera offers — up to 60 fps, MJPG preferred — reads at the camera's own pace instead of a fixed 30 fps cap, shows the real mode in the status line (`720p · 60 fps · MJPG`) and passes frames through a three-buffer ring so a slow draw cannot tear a picture.
  • Corner places it in any of the four corners, Size sets 15–60% of the stage width, Opacity and Mirror do what they say, and Key tolerance goes from 0 (key off) to 100 (widest) to remove pure green — exactly the colour of the Green Screen preset.
  • The status line under the rows says plainly which camera is open, which file is playing, or why it cannot run: no camera on this machine, or no Media Foundation.
A stage with a picture behind the keys, ready for an overlayBackground
The same stage that the overlay is composited onto — here with a picture background instead of a shader one.

AR overlay

The notes fall onto your keyboard, not onto a piano roll

  • The AR OVERLAY card turns the camera into a full-screen background and puts the stage on top of it. Mark the four corners of your keyboard's key strip once, and the app solves the camera's field of view and a full 3D pose for the keyboard from those marks.
  • Calibrate has a Unity-style Move / Rotate / Scale gizmo plus numeric translation and uniform-scale sliders, so a pose that is nearly right can be nudged instead of marked again. Calibration version 5 stores the complete transform, the FOV and the baseline; versions 1–4 still load — with zero translation and unit scale, v4 keeping its rotation, v3/v4 their saved FOV.
  • Notes in perspective turns the falling notes into camera-facing quads drawn from that pose, and their travel has its own aiming: a preview arrow is projected over the camera picture with Screen-up and key-depth buttons plus independent Note direction tilt and azimuth controls — none of which move the keyboard transform.
  • Quality offers Lite, Standard and Ultra. Learn / Perform, look-ahead, opacity and key tint stay available in AR exactly as they do on the flat stage.
The dock page the AR OVERLAY card lives onCamera & FX · AR
The AR OVERLAY card sits on Camera & FX, right beside the webcam overlay and hand tracking.

AR playback

Press bars, a damper strip and a press light that fires once

  • Live press bars are an independent toggle and off by default; they do not disable key tint. A chevron-hidden Bar coverage slider (25–300% of the keys' depth) lets a bar's leading edge run onto the keys — 100 stops at the keys, 200 reaches a white key's front edge, Keysight's full coverage — and the flat bars and the 3D route honour it alike.
  • Bar fade follows lifetime (also off by default) makes a bar's opacity walk with how far it has stretched: Fade at birth and Fade at full length, 60% → 100% by default, so bars brighten as they grow — and the fade walks backwards across the release.
  • Every bar is retinted by the look's velocity colour through the press's true strength, and the press light fires exactly once per press: the finger is the only source, so the falling head of a live bar no longer re-flashes at touchdown on either note direction. The flash blooms in over about 55 ms, brightness and size ride velocity over a wider band, and a long-held key breathes.
  • From the AR Standard tier up, a glowing damper strip hugs the keys' back edge; the sustain pedal (level 0..1) reaches it out over the fallboard and brightens it in the look's halo colour, with its own on/off toggle and a Follow look / Custom colour switch in the AR PLAYBACK card.
The notes and the effects, drawn over the camera in ARGPU stage · Electric Storm
The same engine, the same effects — the AR overlay only changes where the keyboard is on screen.

Staying aligned

The tools that tell you the tripod moved

  • Camera delay is stored per camera — the same key the calibration file uses, in ar-delay.json beside the settings folder. A USB webcam is late by its own amount (roughly 60–150 ms) and a virtual camera or a phone-as-webcam is very different, so each camera brings its own 0–300 ms value back to the slider, and dragging it or accepting a MEASURE DELAY result writes to the camera that is open.
  • Alignment check redraws the saved key frame with octave lines over the live camera picture — calibrated only, on every tier — so a bumped tripod or a keyboard that shifted shows up the moment the lines drift off the real keys. Five high-contrast frame colours, machine-local, never carried by presets or share codes.
  • A drift watch, on by default, samples the key area about every two seconds and compares it with the saved calibration; three straight readings past the threshold put a re-mark nudge on the AR status line. Playing hands and lighting changes stay silent, and it never edits the calibration itself.
  • EXPORT CALIBRATION gathers every saved alignment of the camera in use — every view, picture size and keyboard size — into one portable keyflow-ar-calibration JSON, and drops a snapshot PNG of the camera's current frame beside it. IMPORT CALIBRATION restores them under the backup's own camera link, pushing every row through the checks a fresh calibration passes: rows that fail are counted and skipped, and no setting is touched.
The in-app keyboard and shortcuts cardKeyboard & shortcuts
The same F1 card that tells you how to play the stage — the AR tools all live in the dock next to it.

Inside Calibrate, and under the hood

One line of camera tips answers the classic first-time mistakes — hang the camera over your shoulder looking down the keys, avoid a very wide lens, keep your hands out of the picture while marking — and a live Y′CbCr skin-lobe lookout samples the still it shows: a skin-coloured share before the corners are down (anywhere in the picture) or after (only inside the marked keys) quietly asks you to move your hands. It is advice, not a gate.

The look retunes itself per camera view: on the leaned cameras the paper-thin effect lifts stay at the catalogue's heights and the stand-up beams and arcs keep drawing, while the 90° overhead view shrinks the lifts to 0.05 cm / 0.1 cm and leaves the stand-up quads home, because they squash to a noise-thin hairline from straight above. If the GPU runs over budget, ArQualityBudget drops effect families in a fixed order — particles, pulses, note lights, light bars, note style, impact glow — and restores them as the pressure passes; keys, notes and keypresses are never dropped.

Hand tracking

Which key is your hand over? The stage knows

  • Hand tracking reads your hand out of the camera frame itself: skin pixels are found by colour (YCbCr) and the largest group of neighbouring skin cells is taken as the hand. No model, no download, no network.
  • It counts the fingers you raise and marks the key your hand is over with a light bar on that key, a position dot, and one dot per finger.
  • The camera stays open for this layer even when the camera picture itself is hidden — so you can have the markers without your face on the stage.
  • Skin sensitivity is a slider, and the status line reports the finger count, the key and the percentage of the frame your hand fills.
The session dialog, where the score and the stage layers meetSession
Hand tracking draws on the stage itself; the session dialog is where the song and the stage layers are chosen.

Two recipes

The “player cut out of the background” look, and the “stage on my piano” look

  1. Film against greenUse a green backdrop behind you; the key tolerance removes exactly that green.
  2. Or key the stage insteadChoose the Green Screen preset and composite the whole stage over your own footage in OBS.
  3. Mark four corners onceHang the camera over your shoulder, mark the four corners of the key strip, and the AR overlay draws the notes onto your real keys from then on.
  4. Keep it honestLEAVE Alignment check on so the frame lines tell you the moment the tripod moves, and export the calibration before you move the camera to another room.

Camera & FX page

Everything on one page of the dock

CardControls
Camera & FXParallax, zoom, frame, saturation, contrast and bloom for the stage itself.
WEBCAM OVERLAYCamera overlay switch, the machine's own camera list with REFRESH CAMERAS, USE CAMERA / CHOOSE VIDEO…, Corner, Size, Opacity, Mirror, Key tolerance and a status line with the real mode.
AR OVERLAYAR overlay switch, camera view, Calibrate (four corners, Move/Rotate/Scale gizmo, numeric translation and scale, camera tips and the skin lookout), Notes in perspective, the note-direction controls, Quality (Lite / Standard / Ultra) and Camera delay.
AR PLAYBACKLive press bars, Bar coverage (25–300%), Bar fade follows lifetime with Fade at birth and Fade at full length, and the damper strip with its colour choice.
KEYBOARD ALIGNMENTAlignment check with five frame colours, the drift watch, EXPORT CALIBRATION and IMPORT CALIBRATION — all of it machine-local.
HAND TRACKINGHand tracking switch, Skin sensitivity, and a status line with the finger count, the key and the share of the frame the hand fills.

Honest about what classical vision and a webcam can do

Hand tracking is classical computer vision, not a machine-learning model. With two hands in view, or a backdrop close to skin colour, only the biggest hand is followed; the finger count is only trustworthy when the hand is large enough in the frame; and it wants a simple, reasonably lit background.

The AR overlay is the newest part of the app and it says so: it has automated regression coverage (the render path is re-projected on WARP on every build), but it has not been tried on a real webcam yet, full setting/value parity with the flat stage is not claimed, and the ambient layers — petals, shooting stars — stay flat screen-space rather than following the perspective. Stage-space falling trails are suppressed over the camera, and the camera is the machine's: applying a preset or pasting a share code never changes the camera you picked.

Nothing is uploaded — frames are read locally, drawn locally and never leave the machine.

One camera, one cable, one stage

The overlay, the AR calibration and hand tracking are in the same free download as everything else.