Bless Nintendo for making the Wii controller: inexpensive, lots of internal sensor data (motion sensing, tilt sensing, buttons), elegant design, and standard Bluetooth support allowing it to be used with Mac, Windows, and Linux.

Now there’s free and open source software for making the most of your Wiimote as a musical instrument. First up: Wiinstrument, a multi-purpose percussion instrument, now available for all three operating systems (a Windows version was recently added).

• Plays percussion / drums with gestures
• Use an (in-development) internal sampler with WAV files, or trigger other software via MIDI
• Use tilt for control changes
• Supports tilt, velocity (how much force you use when you move your Wiimote), with acceleration from both the Wiimote and nunchuk
• It works with Mac, Windows, and Linux, via a standard OpenGL-based interface, thanks to the awesome 2D OpenGL library Gosu. (Programmers, take note.)

Of course, drums are just the beginning — you could use this to trigger clips, grooves, visualist videos and animations, whatever. And it comes with demos, tutorials, source code, the lot.

Wiinstrument Release Information
GarageBand tutorial (relevant to other apps, too)
Support information for Windows, Mac OS X Leopard, Linux
Via thread with the creator on our forums

Here is in action.

But, you say, that’s all well and good, but it’s not … air guitar. Today is your lucky day:

There’s a simple problem: sound is invisible, and sound synthesis concepts don’t have any physical reality. Knobs, faders, patch cords, keyboards, infrared sensors, touchpads, and the like all work quite nicely for synthesizing sounds. But take a closer look at Bjork’s use of the reacTable, an interactive multimedia interface that uses a camera to track the movements of blocks on a surface. They really are using it to make sounds, those sounds really are visualized in a nice new way (watch the waveforms connecting the blocks), and while the result is some swoopy synthy sounds, the interface does make making them a lot of fun.

It helps that Bjork pulls out some of her synthiest, electronicilicious-est tracks, like Pluto:

and Hyperballad:

And, of course, part of what happens is that the computer screen here has become the interface. When it works — when the visuals match the sounds, and suggest some new ways of constructing music — it really does show potential for this kind of instrument. (Even if you don’t buy into the blocks, the way the visualization itself works has a lot of promise.)

That’s the idea behind Microsoft’s Surface, too … but sometimes the gimmick can be a solution in search of a problem. Well, actually, maybe your computer of the future really will be “a big-ass table.” (Thanks, SarcasticGamer.com, for making me laugh so heartily.)

The good news: Microsoft is taking multi-touch, camera tracking, and gestural technologies seriously, and they have what looks like a very nice implementation that will be one of the first commercial implementations. The bad news: it’ll cost US$10,000 out of the gate. That high price will mean you’ll see at places like T-Mobile stores and Sheraton hotel lobbies first. But what you need to know: you can build your own version, thanks to available open source tools, with is likely to be more useful for music.

Good sources of commentary:
New Media Initiatives Blog at Walker Arts Center, which notes this could be museum-friendly tech.

Chris O’Shea @ Pixelsumo, who has built a device something like this himself.

The video does show what’s cool about Surface — and it’s easy to imagine these same techniques being applied to live visual and music performance. (People have already tried experiments in that, and I think there’s a lot more to be done — once you’re talking music rather than just digital snapshots, you get into deeper questions about how to model the interface.)

But let’s get a few things out of the way:
1. Enough about Minority Report, already!

Before Nintendo imagined mass-market gamers waving around a wireless remote to conduct music, digital music pioneer Max Mathews devised the Radio Baton:

Radio Baton at the Electronic Music Foundation site

Radio Baton image, in a rather cool presentation on alternative controller ideas

Max Mathews page at CSounds.com, which contains extensive photos, documentation, and even software for the Radio Baton (unfortunately, several of those links appear to be broken)

In contrast to Nintendo’s Wii remote, the Radio Baton uses very basic technology: FM transmitters in two batons coordinate X, Y, and Z position relative to a surface. It’s an idea that’s indebted in some ways to the Theremin, and like the Theremin creates the challenge of performing without physical resistance or feedback on your location (well, unless you count air and looking at the surface, respectively). The payoff, though, is freedom of gesture and expression. Unlike the Wii, too, the two-baton system lets you control balance of instruments; the Wii does tempo only.

There was a time when skeptics thought mice would never catch on. “People will never give up their QWERTY keyboards,” they said. They were half right: now we take both for granted.

Now, more experiments in multi-touch interfaces are appearing by the day. Aside from mysterious Apple patents, we have, via We Make Money Not Art, new research in multi-touch interactions from a team led by Jefferson Han. (Demos pictured.) This isn’t just any touchscreen: not only does it recognize multiple fingers as inputs, but it projects whatever imagery you want in response, enabling new, fluid interfaces, and even responds to force feedback.

With the Web abuzz about Apple’s latest patent, filing, it’s worth reading the actual patent, 0060026536. Like all patent filings, this research may never translate to a shipping product. But it does make for good reading, and it clears up some issues — the most important one being this is about gestures, not specific hardware. Oh, and yes, Apple is working on a touchscreen music mixer:

Okay, wannabe cyborgs, you know you want it: no matter the price, you have to have a robotic armature on your body that sends MIDI data. The latest, via Engadget: the Gypsy MIDI controller. (Wait a second, the gypsy MIDI controller? Now, that doesn’t sound very cyborg. Marketing department, please?) It’ll cost you US$855 an arm, or US$1,675 for the whole suit (best value, as the marketing people would say). Sound pricey? No, that’s about typical in the history of these kind of mechanisms. Speaking of which, despite their claims, this is not the first device of this kind. But it can perform wirelessly, and comes configured out of the box for DJs — now that’s new. Let’s take a look at this latest entry, and see which other attempts I can remember . . .

Sometimes the musical instrument industry seems to evolve slowly, but not when it comes to creating wild new guitars. Here’s a quick roundup of just-released technology, the sort of things that get introduced at this week’s NAMM show; hopefully I’ll get up close and personal with these and others on the floor. (See last year’s NAMM Oddities.)

Guitars with optical pickups? Bowed guitars? Weird body-less “frame” electro-acoustics with MIDI? Effects you control with your finger? Why, sure. Read on.

You wear it on your shoulder, but it’s not a guitar. It’s a US$425 MIDI controller, powered by SONAR and photocell sensors for Theremin-like control in the air. It could be none other than the latest creation from instrument builder Tony Amendolare, aka ElectroKraft.

You might have seen this instrument in November (see post and discussion on Music thing from when I was on vacation), but since then Tony has posted some video files. (Still more discussion: GetLoFi) It’s a lot easier to follow how the thing works in the video, and as for the spacey lab coats and welder’s masks, well, costumes just help add atmosphere:

Performance demonstrations by Dr. Modulus [QuickTime videos]

Audio sample [MP3]

The sound file is particularly stunning: the genius of the instrument is that it sends cascades of synth notes and percussion, either triggered as discrete events by tapping the, erm, globe, or as continuous series of notes by moving your hand through the air relative to the SONAR sensor. [Read the full description]

It’s a great demonstration of how changing the interface is more than just a gimmick: it can change the way your music sounds. You might be able to play something like this on a keyboard or MIDI guitar, but you’d be limited in live control and it would require a lot of trickery. And, of course, this model sends MIDI — so it’d be a blast to hook up to Logic’s Sculpture, Max/MSP, or Reaktor (among others). As usual, more is coming: Tony promises a model with rhythmic sequencing features.

Previously:

ElectroKraft Lunar Module: Spacey Handmade “Optical Theremin” Photocell Controller

Space Invaders Invades Synth, Guitar
Space Box: Theremin + Effects Box
Axe*Synth Theremin Guitar

If you haven’t yet seen commentary on this at Music Thing or (originally) WWMNA, David Merrill of the MIT Media Lab has a project for controlling guitar effects with his face. (It’s not new, incidentally — dates to 2003, and you can look forward to more of this sort of thing at the annual New Instruments for Musical Expression aka NIME, due next in Paris in June 2006.)

So, uh, aside from being weird, could this ever actually be useful and not just freaky? Possibly: gestural mapping gets especially interesting, as David uses a nod to trigger an event, for instance. As video processing gets less intensive relative to computer speed, and gestural processing gets more intelligent, you could eventually interact with your tech like a musician. Hell, if your musician friends ignore you the way mine do, you may have even more luck with the tech.

What’s mysterious here is the software David mentions — check the PDF for more project details. The software is a modification to something called FaceSense, running on Linux. Don’t think this is something you can just go download, though. See also:

Vicon, providers of pro-level motion capture and control systems

Visual Tracking of Movements for a Gesture-Based Interface, a detailed paper by Daniel Hunnisett

Mouthesizer, which reads mouth positions (also from NIME 2003, demoed mapping to guitar effects)

Motion capture does promise to ultimately provide performance and expression without the aid of physical objects, using full-body motion as performance. Now go build something that lets you control your synth with your eyebrows, okay?