Interested in using webcams to translate on-screen motion to MIDI? Want x, y, and z 3D tracking? Ben Tan writes to let us know about his in-development software project called Peripheral MIDI Controller (pmidic) which does just that. The current build is still a work in progress, but has added enough stability and features that it should be worth a look.

Peripheral MIDI Controller

Grab your pen light and start waving it around for filter cutoff and resonance — whoo!

Right now, it’s Windows-only, but the libraries on which it’s built are cross-platform and could be ported to both Linux and Mac. (He’s using Intel’s OpenCV, which is the most popular, open platform for computer vision — odds are if you’ve seen slick webcam tricks, OpenCV was involved, because writing these algorithms from scratch requires a heavy-duty math and computer science background. And the app itself is built with the superb super-cross-platform wxWidgets library.)

If you’ve been looking to experiment with webcam control, this could be a good start — practicing interpreting the control data, experimenting with lighting, and experimenting with inputs all take time. And if you’re a coder, the project is open, so all of us C++ beginners can slog a little collectively.

Tonight here in New York, I’ll be presenting a free workshop on detecting and visualizing motion from camera inputs, which may be of interest not only for those of you eagerly anticipating the new Create Digital Motion site, but also anyone who’d like to use cameras as controllers for music. Full details after the jump.

If you’re not in New York, don’t fret; I plan to organize this stuff and have online examples/tutorials in the near future. But if you are there, say hi! And yes, I finally plan to deliver on my New York CDM get-together promises in June; my schedule will finally allow that.

Still from a recent performance with Eric Dunlap and Mare Hieronimus at Eyewash, the visual performance series in NYC.

Cycling ‘74 has released a new collection of plug-ins I’ve been eagerly anticipating, developed by a team from Electrotap. These aren’t your run-of-the-mill audio effects and virtual instrument: think spectral and granular sound, wild filter/delay stuff, morphing color-palette interfaces, and input from gamepads and webcams:

Cycling ‘74 Hipno “Subversive” Plug-in Collection

US$199 for a whole mess of plug-ins; works with everything. (Mac RTAS/VST/AU, Windows RTAS/VST.) More after the jump.

If you haven’t yet seen commentary on this at Music Thing or (originally) WWMNA, David Merrill of the MIT Media Lab has a project for controlling guitar effects with his face. (It’s not new, incidentally — dates to 2003, and you can look forward to more of this sort of thing at the annual New Instruments for Musical Expression aka NIME, due next in Paris in June 2006.)

So, uh, aside from being weird, could this ever actually be useful and not just freaky? Possibly: gestural mapping gets especially interesting, as David uses a nod to trigger an event, for instance. As video processing gets less intensive relative to computer speed, and gestural processing gets more intelligent, you could eventually interact with your tech like a musician. Hell, if your musician friends ignore you the way mine do, you may have even more luck with the tech.

What’s mysterious here is the software David mentions — check the PDF for more project details. The software is a modification to something called FaceSense, running on Linux. Don’t think this is something you can just go download, though. See also:

Vicon, providers of pro-level motion capture and control systems

Visual Tracking of Movements for a Gesture-Based Interface, a detailed paper by Daniel Hunnisett

Mouthesizer, which reads mouth positions (also from NIME 2003, demoed mapping to guitar effects)

Motion capture does promise to ultimately provide performance and expression without the aid of physical objects, using full-body motion as performance. Now go build something that lets you control your synth with your eyebrows, okay?

Regine reports from Cybersonica on a “manual input” method for controlling sound and image — no, really “manual,” as in hands — on Near Near Future. Golan Levin and Zachary Lieberman have developed a system that analyzes hand movements above an overhead projector. Projected imagery combines the direct output of the overhead (i.e., they shine the overhead at a screen) with digitally-modified projections. Their use of the technology is strikingly simple and elegant — check the photos and videos on their site. I’m looking forward to Regine’s report!

Cycling 74’sMax/MSP and Jitter offer awesome power for developing interactive multimedia, but building everything from scratch can get time-consuming fast. Part of the draw of reusable objects is the ability to incorporate time-saving pre-built tools rather than reinventing the wheel with each new project. That’s the idea behind the insanely-cool bundle of Max goodies, Electrotap’s Tap.Tools. Newly-released version 2.0 has over 150 “externals” (objects for Max) for tasks like:

Audio effects processing (reverb, pitch shift, dynamics vocoder, new delays and envelope substitution)

Audio filtering (including new filters with LFO-driven FFTs)

Signal analysis, number-crunching, and conversion

Helpers for building your own plug-ins

Jitter graphics processing, including motion tracking

Still want more? How about ADSR envelope generators, buffer processing for loop recording and playback, XML file utilities, MIDI mapping, random number generators, AppleScript loading (on Mac), and fancy, reusable interfaces for envelope generation and parameter control, among others?

Here’s more good news: Tap.Tools is cheap enough for artists, with a US$65-99 license. Pay the US$99 and you can even build your own collectives, standalones, and plug-ins, and distribute your creations to whomever you want. (Definitely worth the extra $34!) You even have access to Source Code. Since my C++ skills are nonexistent, I’m planning to print it out and use it in my decorating, but programmers will be glad. As for the non-programmers, we’ll be building new plug-ins with motion tracking! (Check out Electrotap’s other stuff, too, from sensor hardware to its performance-oriented Jade software.)