I built a fun Max/MSP patch that utilises a webcam to interact with a computer-generated fly. The “fly” is created using optical flow-based tracking, which was produced through Jean-Marc Pelletier’s cv.jit externals. They’re great for doing anything with Computer Vision in Max/MSP/Jitter.
As a sound designer, I am obviously more interested in using Computer Vision to affect sound. Inspired by Andy Farnell’s fantastic book on procedural audio Designing Sound, I built a fly sound generator to give life to the fly.
If you want to play with the Computer Vision Fly application yourself and you’re on a Mac:
Download Computer Vision Fly Application (9.4MB)
I mapped the X-axis to pitch (left being low, and right being high). To make it feel a bit more natural, I added a ‘drunk’ function so that the pitch always changes slightly, even if the fly is stagnant. I also mapped the X-axis to the left and right pan to give it movement and create the slight illusion of a Doppler effect when the fly whizzes past.
The Y-axis is mapped to volume to give the illusion of distance. When the fly is farther up the screen the volume is softer and when the fly is lower, the volume is louder.