Putting the mouse down: a shop and a music player you use with your hand
- MediaPipe
- Three.js
- GLSL
- Web Audio
- GSAP
- React
- Role
- Solo, end to end
- Design and code, both projects
- Stage
- Exploration, personal
- Both built and live on Vercel
- Tracking
- MediaPipe
- Hand landmarks from the webcam, in the browser
- Order
- Gesture Shop first
- Started Nov 29, 2025, my first with MediaPipe
The idea
I was exploring OpenCV when the idea came, almost by accident: what if you could control the web with the camera instead of the mouse? Gesture Shop, started on November 29, 2025, was the first answer, and Aura followed. Both follow your hand through the webcam, right in the browser, and turn what it does into clicks, drags and scrolls.
Gesture Shop
Gesture Shop was the first thing I made with MediaPipe. A cursor follows your hand across the screen. Pinch to open, select or click; close your hand to grab a product, move it, and let go over the cart to add it. You can get around the whole store without touching the keyboard or the mouse.
A hand in front of a webcam never sits still, and the raw positions jitter. I smoothed them before they reach the cursor, so it glides instead of trembling, and changed its look between hovering over a product and holding one, so you always know what your hand is doing. A toggle switches back to the mouse at any time.
Aura
Aura plays music and music videos, and your hand runs it: play and pause, skip tracks and change the look, with a mode switch for when you'd rather click. Its themes, including Nebula, Kaleidoscope, Cyber Glitch, Vortex and Liquid Gold, are drawn with Three.js and my own GLSL shaders.
The visuals listen to the track. The Web Audio API reads its frequencies as it plays, and those numbers drive the shaders' scale, colour and speed, so the picture pulses with the music.
Tracking a hand, analysing the audio and running full-screen shaders all at once is a lot for a laptop. Drawing the effects into smaller buffers and keeping the shaders lean keeps it running smoothly.
What happened
I haven't posted either of them online. I showed them to friends instead, who were amazed to shop and play music without touching anything. Both are live on Vercel, linked above.
These taught me that an interface isn't only what you tap or click: sometimes it's how you move.
- What I learnedDesigning for a hand in the air means designing for imprecision, with a cursor that doesn't shake, a clear sign of what your hand is holding, and a way back to the mouse