It started as some compelling demos or proof of concept, but it’s plenty real now: the tools for translating movement, gesture, and dance from the body to interactive music march forward. Empowered by Microsoft’s Kinect and an artist-friendly toolchain, even a single, clever developer can do a lot. Sound designer, music producer, and Max/MSP developer Chris Vik of Melbourne has been one of those busy early pioneers, with an incredible tool called Kinectar.

So, the tech is cool and shiny and impressive: what about the actual music? And, even more importantly, what if all the hand waving and moving about could be meaningful? That’s the next step. For his part, Chris is teaming up with a dancer and choreographer to combine his compositional ideas with someone who knows how to move. The Dubstep-y demos (all below) are impressive, true, but the early tests of the work with the choreographer are simply beautiful, and demonstrate that wobble bass isn’t the limit of what this can do. They also turn the arbitrary arm-waggling into a part of the art.

And as for you: the software’s alpha, but you can fire up your copy of software like Ableton Live and grab this software for Mac or Windows and try it yourself. So if you don’t like the results – be the gesture-controlled basslines too wobbly, be they not wobbly enough – you can put your music, and your movement, where your mouth is.

At top, Chris shows off an early test of the dance collab. (There’s more to come.) Below, a tutorial that shows how this works with Ableton. And read on for more from Chris on what the work with the dancer is about, and what the tool can do.

Chris writes:

Since April 2011 I’ve been working solidly with the Microsoft Kinect, developing my software, Kinectar, to enable its use as a MIDI controller for performing music live. I’ve done a number of performances around Australia since I started the project, however, it’s safe to say that, although I would consider myself an electronic musician, I’m certainly no dancer. Enter, Paul…

Dancer, Paul Walker and I have joined forces to bring the Kinect controlled music concept into the world of contemporary dance. Recently we obtained a residency at PACT theatre (centre for emerging artists), where we spent the week developing different ways of implementing my Kinect music control system in a dance context.

My system is developed in Max and uses OpenNI drivers, OSCeleton and Ableton Live.

via Chris’ blog

CDM will check back in with Chris soon, because:

I’ve got some more videos to release over the coming weeks from a range of my different Kinect music performance applications, including controlling/conducting the Melbourne Town Hall Organ and a 100+ speaker Kinect-controlled diffusion performance. I’ll keep you posted when they’re released!

More on the software:

Kinectar Performance Platform is a toolkit developed by music producer Chris Vik to allow the use of Microsoft’s Kinect motion tracking sensor in computer-based music. The software is designed for electronic musicians to expand the way they control their music in a futuristic and extremely expressive way, using only the waving of hands and a small amount of creativity. It can be used to control the simplest of parameters like a filter or LFO, play notes and chords on a sampler or synthesizer, or be programmed to control an entire live-set through nothing more than gesture.

Key Features:

Movement Tracking UI allows manipulation of the Kinect’s human tracking capabilities, displaying all relevant data extracted from the hands location in 3d-space

Instrument Builder lets the user build virtual ‘instruments’ by outputting MIDI notes in three modes:

  • Static – Produces a single note value. Useful for drum triggers, turning on/off effects within a DAW or feed that trigger into Kinectar to switch between presets using your gesture
  • Solo – Do sweeping solos by selecting from over 40 musical scale presets or click the notes on the UI to make your own
  • Chord – Create a progression of up to 8 chords per preset to play live

Global Flags lets you turn on/off Kinectar’s instruments using a MIDI note sent from your DAW, external MIDI controller or Kinectar itself

MIDI Preset Control lets you switch between Kinectar’s presets and instruments using a single MIDI note

Value Editor enables many more MIDI/OSC outputs, for controlling device values

Visual Metronome popout window sits on top of all programs to make it easy to see if you’re in-time when the music gets messy

It’s labeled “rough alpha,” so don’t expect a finished tool here, but you can go download it and give it a try (or learn more about what’s possible):

http://kinectar.org/download

And now, the obligatory (but quite awesome, Chris) Dubstep demo videos:

  • Tohm

    great interface design… very user friendly yet flexible.

    I am still fascinated how we are using this data… most often, I have seen mostly hands and full body tracking (single or multiple users) while we leave out the data for the hips, knees, shoulders.  I have a feeling that, mostly likely, it is because people want to see what is happening.  I call it the “wiimote syndrome”… does the piece still hold water if no one knows you are using alternate controllers?

    As an aside, I have been dying to find a dancer who is interested in the subtlety of these other points…

    • Patrice

      I am the female dancer/Max programmer you are looking for! Hit me up! pscanlon at mills dot edu

  • Tohm

    great interface design… very user friendly yet flexible.

    I am still fascinated how we are using this data… most often, I have seen mostly hands and full body tracking (single or multiple users) while we leave out the data for the hips, knees, shoulders.  I have a feeling that, mostly likely, it is because people want to see what is happening.  I call it the “wiimote syndrome”… does the piece still hold water if no one knows you are using alternate controllers?

    As an aside, I have been dying to find a dancer who is interested in the subtlety of these other points…

  • six fingered deformity

    !!! is there any software similar to this that can use a webcam?? this is amazing!

    • God

       yes if you gor Max4Live
      check the website

    • six fingered deformity

      thank you very much!

  • Patrice

    Thank you for sharing this Chris! I’ve been watching your videos since October, which is when I started working with the kinect.  It is amazing to see all the progress you have made with this device! Truly inspiring and encouraging!

  • http://www.facebook.com/people/Marco-Donnarumma/526602563 Marco Donnarumma

    here my provoking 2cents.

    The real problem which does not emerge at first sight, but it’s crucial to the development of new instruments is creating expressive sound color and forms that present nuances, like traditional instruments.
    Playing loops is rather limited, sound forms are repeating all over. It’s good to watch for three minutes, but then you already know what to expect and how it’s going to sound; your attention fails, and your expectations of playfulness fly away.
    This is the perfect reflection of the conservative approach to technology that corporations push into our mind: the magic of the computer is to allow you to do things without effort.

    But this is a very dangerous viewpoint for us working in music and arts. Music and arts are mostly created by effort, any kind of effort; it’s in our DNA and historical heritage. It’s that almost not perceivable nuance that makes a musical or dance performance enjoyable and captivating, or fascinating. A virtuoso is most of the time defined so according to his ability of playing subtlety, which other can’t.We don’t really want effortless music, because behind the one-to-one mickeymouse interaction (sorry about that), it lacks of expressivity (I’m obviously not pointing to this work, just generalizing the discourse).Controlling a synth or the trigger of a loop can be done in thousand ways… you can also set up the computer to recognize your whistle and trigger things without moving at all. You don’t need the kinect to do it, and, above all, you don’t need the kinect to be able to share more widely the technology, what about _making playful, expressive, multi-faceted music_?
    How many people here enjoy home listening of the same 10 loops in different sequences for ten minutes? Would you wish to attend a concert of this kind for the sake of the waving hand interaction or for the sake of music?

    I’m being purposely provocative to spicy up the issue, because I believe we really need to stop and think for a second about this phenomena. 
    “Human-Computer Interaction” is not only about finding the 10000th way of controlling a trigger.
    And it would be very sad to limit ourselves to this kind of approach in year 2012.

    • http://www.facebook.com/profile.php?id=100001138444109 Mutis Mayfield

      As I said you at FB, I’m agree with you in “focus” but… well, MickeyMouse is great for kids and less expertise people. If you are going to “show” a kid about electronics what do you think it could be better… arduino or lego?
      I don’t believe that Corps are trying to create “one push button” people, they are trying to find the most common use or better user experience to make something significant and don’t fail at sales (I think you know how rules the development process).

      For experimental re-searchers your POV is a must have and usually they are funded to do that. For regular user or non “cutting the edge” people, maxforlive and kinect are the door to new tools.

      Then is our work, talk about the full possibilities, pure data, computer vision (with cheap IR cameras) and so… or talk about “standarized” solutions (with good stability and so…). It depends on the “listener”…

      but the last word is on them. Don’t blame the tool, don’t argue against “not so cute” develops.

      All are the same, forms of expressing oneself. Choose your one, spread your work and let the time put the pieces on the right place.
      ;)

      Vitor Joaquim talked me about your work with kinestesic sensors at Trendelemburg Festival (Gijon). Maybe it will be interesting know a bit more about these works and help us to understand better your point.

      -m!

    • http://www.facebook.com/people/Marco-Donnarumma/526602563 Marco Donnarumma

      Hi Mutis, I’m happy you followed up on this.

      I love to see kids playing with Arduino nowadays :)  http://vimeo.com/4187008

      As you say, Corps _don’t_ fail at sales they rather stop selling. And there lies the real issue.It’s not only about creating a monopole for creative tools, but it’s also about injecting into the minds of most unexperienced individuals (and often also academics and working artist) that their innovation are the best and only.
      You mentioned the point of engaging beginners, which leads us to the issue of empowerment.

      Let me make a little cost analysis in this sense. Which can hopefully let us understand the actual degree of accessibility for beginners.

      This video was recorded at the Microsoft’s REMIX11 conference using “kinect/ableton/max”. The ticket to the conference costed $310 [1], Kinect costs, say, about £100, Ableton Suite costs about £383, Max for Live is an additional £191.
      So to attend this performance/talk and try to build something similar with these technologies one needs about £870.
      Without considering that Kinectar is free but is not open, so I need a licensed version of MS or OSX to run it. That is a relevant additional cost.

      We are easily around £1000 to play loops and drive one synth parameter.
      Do you find it really encouraging for beginners?
      On top of that, when a beginner spent all that money and got the thing working, he’s done with it. There’s very little room for personal exploration of this interaction, because the tools are closed. You cannot open, hack and extend them. Game over.

      On the other hand, we have the open community efforts.
      Friend and colleague Will Brent developed a library for Pd that enables you to deploy data not only from a Kinect, but also from a Wii, from a camera and from your keyboard, all in one collection of objects which don’t need elaborate programming skills [2]. He did that initially for his students.

      Jaime Oliver developed one single object for Pd/GEM that run a quite complex computer vision system, and produce an incredibly rich amount of data to be applied to whatsoever [3]. A related instrument of him won the Guthman New Musical Instrument Competition at Georgia Tech, USA.

      In the case of my own work, I developed a biophysical system called Xth Sense, that is capable of capturing the sound of human muscle contractions [4]. You can use muscle sounds as musical material, and control data at the same time. 
      That’s to say, you can create music with your body and control whatever (your own sounds, other sounds, music and video software, lights, a robot, …) lives on your machine with a tiny bracelet sensor. 
      You can forget about the computer, I usually hide it in a corner of the stage, because the whole system sense your kinestetic behaviour, adapt to it, and respond creatively. You don’t need to stop the performance and push a button to change the preset in use. The Xth Sense won this year the same Guthman Competition mentioned above.

      Pure Data is free and open, all these libraries and software for Pd are free and open, they can run on Linux which is free and open, and you don’t even need an arduino.
      (which is still one of the most beautiful thing we have, but there are plenty of emerging technologies that are worth exploration).
      The total cost of doing a performance similar to the one at the top (and possibly better), with these open tools provided by the community is close to £5/10 (the former is the parts cost of my sensors, or the cost of an external webcam).

      Me and Will also gave a talk at the Pd Convention last year, and the convention registration is free.

      So, a fair comparison shows that it’s about £1000 to access and deploy Corps tech, against £10 for open tools with the same and possibly alternative, better capabilities.
      Besides, when you are done with your basic project, you can hack everything you want, change it, extend it, etc.. which is what really gives you pragmatical and conceptual knowledge about something.

      Which approach has a true potential to engage beginners, providing accessibility to and knowledge about technologies?

      [1] http://www.noisetosignal.com.au/remix/
      [2] http://williambrent.conflations.com/pages/research.html
      [3] http://www.jaimeoliver.pe/recipes/pdgem-objects
      [4] http://marcodonnarumma.com/works/xth-sense/

    • http://www.facebook.com/profile.php?id=100001138444109 Mutis Mayfield

        Right point and completely agree.

      In addition to this we have the middle point that I usually defend and empower people as a “startpoint” is about “Blackout” way like 1975 explosion[1]. The godfathers of the HipHop and other creative movements based them on “steal” these tools more than “buy” them.

      Our actual “blackout” is called piracy and most of the actual children are playing with these tools without investing a penny. That’s a fact. We could argue and discuss about morality or ethics but in our actual society, children are growing with “free” and “hacking” even with closed tools.

      The point is, taking these potential and show them “the open way” as a true alternative (I work with all the sides, open source artist and Commercial corporations trying to fill the gaps between them and believing in a utopic world where Corps mature into something less agressive and more friendly like Korg with Monotron[2] project ie). Obviously it is a market bet (which create internal friction inside these Corps too) and they want to see money return from this kind of investment. We must positionate ourselves and try to do at our best (and pay the bills over the travel…).

      So I understand your point and I’m only show “other” intermediate grey in the scale from black to white and from the POV of the non-power user which usually rejects use pure data due to its UI and sharp learning curve.

      I will give you an example:
      We have Live as a standard for Live-act (another fact in this market target) and there is maxforlive. Ok.

      We have also LiveControl[3] and Pure data. They are free and open source and one beautiful door to show “regular” user the benefits of open source and the ethics of non piracy options.

      Then, if they want, they can make the next step into whole open source solutions skipping Live. But if they don’t, blame Corps instead of making Pd more user friendly is not an option for people which usually buys a Mac (IMHO). I think (based on my experience of course) it could be more useful (and realistic with they relative reality) show the possibilities and left them try. These is the true value of freedom: Let people choose.
      Our goal must be working hard to show them why and how use these solutions to keep them free to make these choice.

      In other case we are making the same as Corps creating a “cliché” and fighting against money forever and ever blabla…

      “The problem is not the tool, is the use and abuse of these tools in junction with the fault of knowledge of the beautiful alternatives around us” like your amazing work.

      Once again Thanks Marco for this stimulant discussion because in the middle I hope they were a lot of people learning from and with us.
      ;)

      [1]http://www.youtube.com/watch?v=aJmaH3r03nA
      [2]http://www.adafruit.com/blog/2010/11/06/hackers-react-to-korg-monotron-synth-schematic/
      [3]http://livecontrol.q3f.org/

  • pepezabala

    what a pity, it’s not available for download anymore. I have been doing some experiments with Synapse, another software that gives you straight Max4Live integration for the kinect. Most annoying is the big latency, for rythmic stuff it’s just too slow. But if you use the kinect to control filters and modulation it’s big fun. And I did some stuff with Robert Henke’s Grain-freeze device and the kinect which is absolutely mindblowing. You basically can walk through a soundfile, and when you stand still you get incredible sounding drones, slight movements make subtle changes in sound, fantastic experience, really.

  • http://profiles.google.com/trebtid Polite Society

    Is this the same Chris who is also Synaecide?  A quick google proves it is. Wow.

    His productions are *amazing* check out his soundcloud if you haven’t already.