These strange glyphs represent the dictionary of hand gestures Geco and LEAP can turn into music control.

These strange glyphs represent the dictionary of hand gestures Geco and LEAP can turn into music control.

Here we go again. Touchless hand gestures have been part of electronic musical performance ever since the Theremin first hummed to life almost 100 years ago. And those gestures embody the same challenges and promise. We have the ability as humans to think spatially, in three dimensions, and to have a tight sense of control via our muscles of gestures in space. We use gestures to communicate and to manipulate our world. Those same expectations can be disappointed in electronic systems, however, as they lack tangible physical feedback and may misinterpret our intentions.

It’s easier to play with these ideas and experiment with them than talk about them, though. And for everyone who’s turned off by the idea, someone else is enthused.

What the US$79.99 Leap Motion may do for gestures in music is to lower the bar for entry – and up the bar for performance. Leap is affordable hardware, there are already lots of developer units out in the world, and there’s an easy-to-program SDK. We’ve already seen Microsoft’s Kinect open up gestural control to lots of new music projects. Leap may do more: it’s cheaper, it’s faster and operates with vastly lower latency, and it’s more precise for individual hand gestures. It also offers a platform for developers to share their work, in an app store full of stuff you can use, so that the hardware theoretically won’t become a paperweight in the cubicles of the digerati.

Latency alone could make a big difference for musical applications. It’s not the only challenge in motion control, but it has been the showstopper, particularly with the hefty lag you get using something like Kinect. Leap is different, offering latencies low enough to satisfyingly control music applications.

The unobtrusive Leap Motion hardware. Courtesy the manufacturer.

The unobtrusive Leap Motion hardware. Courtesy the manufacturer.

That doesn’t mean you should run out and buy one. Healthy skepticism is always good practice. So, I actually agree with some of Chris Randall’s complaints about Leap, as discussed on Twitter. I think anyone experimenting with novel control schemes, though, may learn something from successes and failures alike.

If you’re ready for the adventure, though, Leap will make it immediately easy to start mucking about with music. Leading the charge is Geert Bevin and his Geco (originally Gesture Control) app. I’m testing it now, but here’s a quick look at what it does.

By making a virtual MIDI port, and using a library of gestures and mappings, Geco allows a wave of your hand to control any music tool that works with MIDI.

  • Using two hands, create up to 40 different streams of control messages.
  • 16 MIDI channels.
  • Mappings with MIDI Control Change or (with greater data precision) pitch bend.
  • Manipulate different streams using “open” and “close” gestures of your hands.
  • Low-latency control, with visual feedback on both MIDI and movement analysis.
  • Send MIDI on the Mac using a virtual MIDI port you can then connect to other applications – or, on either platform, to physical MIDI ports.
  • Graphical UI with color/graphical customization, information on gestures and so on.
  • Thin out your MIDI data to work with old gear that can’t respond to all those messages.

The intro price will be US$9.99. It should launch with the Leap Motion app store – dubbed Airspace – when the controller launches on May 13.

MIDI is useful, but it’s too bad there’s no higher-precision control implementation here. (OSC would be one option; it seems apps that do that are a likely addition.) There is a whole lot of detail and thought that has gone into how the UI works, and Geert promises that the whole engine is low on system resources and approaches “zero latency” (at least, it’s very, very fast).

Updated: Geert fills us in on that high-resolution data question and OSC. From comments:

Yes, there will be OSC in a next version and I plan to add direct hosting of AU/VST also. I’m also thinking of making an AU/VST version of Geco itself so that it can perfectly be integrated into any DAW and process the audio that’s flowing through.

It’s worth taking a look at the draft documentation for more detail:

Here’s another experiment showing VST and AU control:

Nor is Leap Motion the only game in town. On Create Digital Motion yesterday, I wrote about another project that is using crowd funding to launch an open source rival. I can imagine developer APIs that let you work across each. The advantage of open hardware would be that people can understand how the device works, and modify it for specific applications (both code and hardware form factor.)

DUO is a DIY 3D Sensor – Like Leap, But Open Source, From Gesture and Vision Veterans

I’m clarifying the details of their licensing plan. At least one of this team has come under criticism in the past for the approach to open source releases and Kinect hacking – you can read the discussion in both directions, though I’m encouraged that developer AlexP was ultimately responsive to some of those concerns. We’ll see how this project is structured.

It does seem that people will continue to develop this thread in motion control. We’ll be watching.

As I do have a Leap, let me know if there’s anything you’d like tested or developed (summer project!), or questions you may have.

  • Geert Bevin

    Thanks a lot for the article Peter, here’s a quick follow-up about higher precision data.

    Yes, there will be OSC in a next version and I plan to add direct hosting of AU/VST also. I’m also thinking of making an AU/VST version of Geco itself so that it can perfectly be integrated into any DAW and process the audio that’s flowing through.

    I purposely kept this first version very focused though since a lot of the time went into experimenting with the gestures that worked for musical performance and how the configuration process would work best.

  • Dave

    Brilliant. I can’t wait to get my leap and try this out.

    • fapjacks

      In three months…

  • Andy

    Hmm, don’t know if it’s just that software – but seems to be a similar lag to the Kinect ….

    • James Britt / Neurogami

      I’ve worked with both the Kinect and the Leap, and the latency on the Leap is much, much lower. That said, software that is interpreting gestures and doing calculation swill add some lag, but in my own experience it’s much better than with the Kinect.

  • James Britt / Neurogami

    Very cool. I’ve written something like this myself (though it’s more focused on sending OSC than MIDI). One concern I have is that Leap Motion has said that apps sold through the Leap Air Space store must only capture gestures if the app is the active window. I’ve been controlling Renoise using the Leap, but I have the Leap app as a background program since I want Renoise to be fully visible. How does Geco get around this restriction?

    • Geert Bevin

      Geco doesn’t get around it. Leap SDK up until 0.7.6 allowed for background execution. However with SDK 0.7.7 and 0.7.8 that has been disabled. Leap Motion promised to release a supported background app solution for a next SDK in the coming days.

    • James Britt / Neurogami

      Interesting. I noticed that after they announced that the SDK was enforcing it I still had no trouble running background apps with the then-released SDK, but I think that’s been “fixed” in a later (i.e. the most current) update (which I’ve been negligent in grabbing).

  • Sylvain Poitras

    Personally, I’m waiting for MYO:

    Taking the camera out of the equation makes more sense to me from a performance standpoint.

  • Michael

    There are several open-source Leap Motion to OSC libraries/scripts in development and in fact an email thread was started just this morning to discuss how we might all share a similar OSC message format, such that OSC receivers can work with more than one OSC sender.

    I would add a bunch of project links here but that might be rejected as spam, so let Google and Github search be your guide.

  • Ezmyrelda

    I think you could have left out that twitter interaction.. it contributed almost nothing to the article.

  • Hogo

    I’ll get this on day one just to support this type of tech. I would love to see this integrated to work with an ipad through the cck or other USB interface.