As it happens, hunching over your computer does not center your body and mind. So, drawing from yoga and other practices, Adriano Clemente is getting his whole body into the act of making music. While Kinect is not a perfect solution for every vision application, either in tracking capability or latency, it is stunningly good at following your skeleton through space. And here, using moderated, slow-moving motion, the body can navigate musical worlds with applomb.

With apologies to everyone staying up late at night working on tracks in your undies, it’s also a convincing excuse to perform music without shirt or pants. (Given some of the world is now at 43 degrees C / 110 F or so, I can think of no better time.)

Kin-Hackt is Adriano’s Kinect-controlled performance rig, combining the ingredients of Ableton, Kinect, OSC and TUIO (an open protocol for vision based on OSC), and make-it-easy Mac control tool Osculator.

Pants-Off Dance-Off Ableton Live, indeed.


This is one of the Body Instruments designed by Adriano Clemente for his project Kin-Hackt.
This performance is one extract from the “Composition for Five Bodies and Kinect”.

This is achieved with a 3D sensor (Kinect) able to map the joints of a human body, then tracking their movements which are translated to musical impulses.

My art project is based on the idea of the body itself as the ultimate instrument of expression and communication, without the use of any other objects to hold, wear or touch. Interacting directly with a virtual sound-space, the essence of the expression is contained in the movement.

The person performing is the composer, the music player, the instrument and even the orchestra.

The possibilities of this new “instrument” are limited only by one’s imagination.

This is, of course, a different take on what we saw yesterday. Marco Donnarumma’s body control system is similarly about using your entire physique to make music. But while it does require additional gear, the payoff is a interface that works directly from muscle impulses; Kinect, while allowing the body to be naked of other equipment (pants or no), relies on watching the body from a distance. Each approach has some unique advantages.

Check out New York City-based artist Adriano’s site for more information:!__kin-hackt/ableton-trainer

For more Ableton Kinect goodness, see this previous effort by Adriano, with more layers of clothes (ideal for Ableton’s hometown of Berlin, which is cold and rainy as always in July sunny, warm, and beautiful).

  • PaulDavisTheFirst

    sad to say but i find this incredibly uninteresting. it appears to connect a sensing system with low resolution, low precision and few parameters to a sound generation system in which most of the work has already been done.  this claim:

    My art project is based on the idea of the body itself as the
    ultimate instrument of expression and communication, without the use of
    any other objects to hold, wear or touch. Interacting directly with a
    virtual sound-space, the essence of the expression is contained in the

    seems to me like a cover for “yeah, so i set up most of the music before hand, and by dancing around a bit you get to tweak a few aspects of it”.

    i appreciate that the field of controlling music synthesis requires a lot experimentation, since we’re clearly far from understanding all the possibilities yet, alone the right ones. but this sort of game-driven conception, in which the human simply fiddles with a few preset parameter options within a world created previously, just doesn’t seem to me as having anything to do with forward progress.

    • Paul

       Hmmm.  Sort of like how a luthier sets up a world of musical possibility and exposes it as a few strings and tuning knobs for the musician to fiddle with and tweak?  I’m not sure I understand the objection here, unless it is to the whole sample/loop/midi-controller portion of digital music.  Maybe the diy x synth is more to your liking with the analog mic retaining more bits of information than the Kinect CV? 

    • PaulDavisTheFirst

      no, entirely the opposite of that. the luthier sets up a world of possibilities in which very very subtle changes in motion and pressure will change the qualities of the sound, to quote eno (again) “in ways that the body can learn and the mind cannot”. so subtle and hard to learn that it typically takes years of practice to link desire with results. this ends up creating an instrument capable of very delicate expression. now, you don’t need this if you want to make 4 to the floor dance music, certainly. but nobody points to such music as the result of a fine instrumental performance – its great dance music, nobody wants to watch you make it (at least not without drugs).

      this, on the other hand, purports to be both (a) an instrument capable of modulating between the desire of the performer and the production of sound, and (b) to have some performance element to it. i think it does a very simplistic and not very learnable job of (a) and isn’t a lot better at (b).

      so: more bits, more use of the senses of the performer, more ability to be subtle on purpose.

    • Paul Rose

      I totally agree. Been there, done that. You can basically replace the body with, let’s say, a moving curtain. or a camera tracking traffic on the street in front of your house. The whole piece would be substantially the same. There is nothing in it that is somehow specific to the fact that there is a body being tracked.

    • Peter Kirn

      I couldn’t disagree more with your analysis of the description as a cover.

      I absolutely believe that’s what he’s trying to do. It’s an admirable and extraordinarily ambitious goal – one that may prove unachievable. But I think it’s worth setting audacious goals.

      I think judging the work is fair; I’d question whether you can read other people’s intentions.

      Parameter fiddling – now, that’s really the heart of the problem, which I think is what Marco is trying to avoid with his work, which is why I raised it. 

      If you’ll excuse me, though, I have to get back to parameter fiddling. Forward progress? Well, you know you have to engage in your own process of experimentation.

      What are you suggesting as the alternative?

    • PaulDavisTheFirst

      simple: music via touch, just like its always been. i’m entirely open to changing what we touch – you know my enthusiasm for jones’ work on high resolution multitouch coupled to physical modelling synthesis – but i don’t think motion is really viable for the creation of music beyond sound fields/ambient abstraction.

      and before anyone says it – yes, i know about tap. but what matters in tap is your feet (shoes) touching the surface … the body motions are just a means to make that happen, much as bowing a cello requires some movement beyond the immediately required touching of the strings.

  • PaulDavisTheFirst

    On re-reading my comment above, I feel that I want to amend it a little with a more fundamental observation.

    The essence of all music making except singing has been touch – a person touching a thing in a way that makes it produce sound waves.

    The essence of all dance is movement. Even when dancers touch, movement and motion remain at the core of what makes dance interesting. Even when a dancer stops moving, how they moved before and how they will/may move in the future is what makes the stationary moment(s) what they are.

    Perhaps there is an interesting project still to be done in which music is created through dance. Perhaps technology is making this possible in ways that never have been before. But gross (large scale) motion of the human body is a very very very difficult thing for most of us to control – for everyone, in fact, except the truly great dancers. The history of instrument design is a history of adapting physical objects to the capabilities of human touch (and vice versa), to take advantage of the latter’s precision. I am skeptical that much will ever come of trying to blend large-scale motion of the body with the performance/creation of music.

    • Peter Kirn

      Well, fair enough. I can at least say that in years of accompanying dance with music, we tended to try things like this:

      1. Stop playing and move. (I forced an improv group I was doing to put down their instruments and dance for a while, then go back to playing.)

      2. Avoid watching the dancers. (Literally.)

      Now, curiously, while musicians ignoring the dancers often worked, the dancers’ relationship to the musicians was always the opposite. They were responding to what we were playing – sometimes even really out-there experimental sounds – no matter how focused they were on movement.

      I can’t recall where I read it, but was hearing someone describing music and rhythm, and suggesting that you have the sensation of moving as you hear music, even if you don’t respond directly.

      I should say, at the same time, all of this is all the more reason to try experiments like this, even if others – even if you yourself – deem them a failure. 

    • PaulDavisTheFirst

      absolutely to everything you just said. but you were the musician(s) and the dancers were the dancers. you can make music to accompany all kinds of sensory input (even touch), but this is in no way equivalent to trying to make music via motion.

      incidentally, i strongly suspect that if this kind of thing were to work, it would most likely be on much more ambient (in the eno-sense) pieces. there’s a piece of music i love called La Voz Del Rio … can’t find the artist … massive, big ambient pads that sweep and swirl. its not hard to imagine playing a piece like this entirely through body motions – winding, unwinding, spinning, reaching, falling. i still don’t think i would call it an instrument, and the harmonic structure would still be largely pre-determined. but it would be closer to a music-through-motion experience than i think this video demonstrates.

    • Peter Kirn

      I see what you mean – you need the movement itself to be generative. And it may or may not be in an instrumental sense, but you want it to somehow navigate the musical structure rather than stand in for a knob.

    • PaulDavisTheFirst

      re: the knob. bingo! that’s precisely it.  or nearly precisely. yes, i want it to be generative. i don’t want it to navigate an existing structure, at least no more than playing the piano can be said to be navigating a given set of tones.

    • Paul Rose

       well, what about the theremin then? The kinect is in fact a multi-gesture-theremin-type-controller. People have been able to gain virtuosity on a theremin, so I believe that this should be possible with a kinect-driven patch. Although the latency that the kinect has proven in my own experiments doesn’t allow much more than ambient music at the time being.
      Also there has always been something like conducting an orchestra – I believe that we should be able to see someone conducting a number of max-patches or similar by using a kinect in a similar way.

    • PaulDavisTheFirst

      re: the theremin. i knew that someone would raise this :) the theremin is an interesting example. its also the sort of example that proves the rule: hard to play, not played by very many people, very difficult to play the same kind of tone sequences that other instruments make easy. i’d be interested to know if performers like clara rockwell ever felt that their bodies really “knew” the subtleties of how the theremin field interacted with their hands. if not, i’d happily accept two kaos pads as an entirely acceptable alternative 😉

      the orchestra/conductor thing is another pet issue of mine. i believe that in the discussion of music performance, the focus on composer vs. performer misses that key third category of conductor, and that an awful lot of electronic music production today occurs by way of a process much, much more similar to that followed by an orchestral conductor than either the composer or performer model. so yes, i agree with you, but i also think that the distinction has long been important and should continue to be so. the strategies for creating an interest max patch or an expression-rich csound instrument or a sonically loaded physically modelled synth are *very* different from the ones you need to conduct such things during performance. but they are also not the same as those of a luthier or a piano maker …

  • Joel

    Thought I would just throw up another interesting Kinect project.

    PS. It’s tied into an advertising campaign in New Zealand however it’s still nicely done.  

  • Eric Rieper

    I didn’t read every comment here, but noticed a lot of “unimpressive” and “it’s just minor tweaking” themes, and while the latter is true, I think this opinion is kind of… wrong.

    We’re all a bunch of Tech Headzz and Digital Artists and as such I think many of the people here are looking to be impressed by or moved by the technicality of a piece.

    Personally, I find digital art that largely focuses on the digital aspects, the technical aspects, and is more enamored by the tools rather than superseding them, to be “unimpressive”– regardless of the complexity and scale.

    I would think, especially because this is about movement and in some ways, dance, that CDM comment folks would evaluate it on it’s success as a piece of art / performance, not as a tech demo (as often is the case.)

    • MyxtMedia

      Eric, the artistic/creative aspect is what everyone seemed to be critiquing to me. Or rather, the inability to truly be expressive musically with many of these new creations (not yet instruments). Since everyone here is a Tech Head, we can all analyze and understand the reason the “artist” is not impressing us and allow our understanding of this to provoke new ideas and suggestions to improve the tools being used. I think we can all agree that the technologies being developed are, indeed, impressive and something to appreciate. But the point of them is to create. The tools are now limited in their expressive range and the output is not attractive as “art” to many. This is all still mostly a science experiment. 

    • MyxtMedia

      Some may consider this guy, waving his arms around in his underwear, an artist. I am leaning more towards “knob”, as others have previously commented. Less of an instrument, more of a tool, at this point.

  • Napoleon Dynamite