It started as some compelling demos or proof of concept, but it’s plenty real now: the tools for translating movement, gesture, and dance from the body to interactive music march forward. Empowered by Microsoft’s Kinect and an artist-friendly toolchain, even a single, clever developer can do a lot. Sound designer, music producer, and Max/MSP developer Chris Vik of Melbourne has been one of those busy early pioneers, with an incredible tool called Kinectar.

So, the tech is cool and shiny and impressive: what about the actual music? And, even more importantly, what if all the hand waving and moving about could be meaningful? That’s the next step. For his part, Chris is teaming up with a dancer and choreographer to combine his compositional ideas with someone who knows how to move. The Dubstep-y demos (all below) are impressive, true, but the early tests of the work with the choreographer are simply beautiful, and demonstrate that wobble bass isn’t the limit of what this can do. They also turn the arbitrary arm-waggling into a part of the art.

And as for you: the software’s alpha, but you can fire up your copy of software like Ableton Live and grab this software for Mac or Windows and try it yourself. So if you don’t like the results – be the gesture-controlled basslines too wobbly, be they not wobbly enough – you can put your music, and your movement, where your mouth is.

At top, Chris shows off an early test of the dance collab. (There’s more to come.) Below, a tutorial that shows how this works with Ableton. And read on for more from Chris on what the work with the dancer is about, and what the tool can do.

Chris writes:

Since April 2011 I’ve been working solidly with the Microsoft Kinect, developing my software, Kinectar, to enable its use as a MIDI controller for performing music live. I’ve done a number of performances around Australia since I started the project, however, it’s safe to say that, although I would consider myself an electronic musician, I’m certainly no dancer. Enter, Paul…

Dancer, Paul Walker and I have joined forces to bring the Kinect controlled music concept into the world of contemporary dance. Recently we obtained a residency at PACT theatre (centre for emerging artists), where we spent the week developing different ways of implementing my Kinect music control system in a dance context.

My system is developed in Max and uses OpenNI drivers, OSCeleton and Ableton Live.

via Chris’ blog

CDM will check back in with Chris soon, because:

I’ve got some more videos to release over the coming weeks from a range of my different Kinect music performance applications, including controlling/conducting the Melbourne Town Hall Organ and a 100+ speaker Kinect-controlled diffusion performance. I’ll keep you posted when they’re released!

More on the software:

Kinectar Performance Platform is a toolkit developed by music producer Chris Vik to allow the use of Microsoft’s Kinect motion tracking sensor in computer-based music. The software is designed for electronic musicians to expand the way they control their music in a futuristic and extremely expressive way, using only the waving of hands and a small amount of creativity. It can be used to control the simplest of parameters like a filter or LFO, play notes and chords on a sampler or synthesizer, or be programmed to control an entire live-set through nothing more than gesture.

Key Features:

Movement Tracking UI allows manipulation of the Kinect’s human tracking capabilities, displaying all relevant data extracted from the hands location in 3d-space

Instrument Builder lets the user build virtual ‘instruments’ by outputting MIDI notes in three modes:

• Static – Produces a single note value. Useful for drum triggers, turning on/off effects within a DAW or feed that trigger into Kinectar to switch between presets using your gesture
• Solo – Do sweeping solos by selecting from over 40 musical scale presets or click the notes on the UI to make your own
• Chord – Create a progression of up to 8 chords per preset to play live

Global Flags lets you turn on/off Kinectar’s instruments using a MIDI note sent from your DAW, external MIDI controller or Kinectar itself

MIDI Preset Control lets you switch between Kinectar’s presets and instruments using a single MIDI note

Value Editor enables many more MIDI/OSC outputs, for controlling device values

Visual Metronome popout window sits on top of all programs to make it easy to see if you’re in-time when the music gets messy

It’s labeled “rough alpha,” so don’t expect a finished tool here, but you can go download it and give it a try (or learn more about what’s possible):

http://kinectar.org/download

And now, the obligatory (but quite awesome, Chris) Dubstep demo videos:

The latest in a long tradition of transforming hand-drawn graphics into music (see, in particular, Iannis Xenakis and UPIC), we see a computer-vision-powered pen-and-paper music generator. Kovacs Balazs writes:

This is a manual sounddrawer. Doesn’t need any sensors, but a camera, paper, colored pens. Doesn’t need sensor glove or reactable as well.

What I love about this, though, is that the resulting sounds are utterly crazy, a big collision of notes and sound.

By the way, UPIC lives on here in a very advanced program descended from the original tool:
http://www.iannix.org/en/index.php

From credits: Magyar Eötvös Ösztöndíj Alapítvány, UCSB-MAT, CSALÁD

More:
http://soundsofpictures.blogspot.com/2012/01/17.html

Perhaps you’ve seen the demo videos, as people do astounding things by moving their body around and using the Kinect camera to make music. Now, a set of Max for Live devices makes it reasonably easy to access your body as input inside Ableton Live.

FaceOSC Mapper

Pictured at top, this builds on Kinect superstar coder Kyle McDonald’s face-tracking tool and lets you use your face – position and even facial movements – to control Ableton Live parameters.

Kinect Camera

For use with the V-Module and vizzABLE systems, you can plug in one or more Kinect cameras, and get tilt, distance filtering (to remove backgrounds), depth maps, RGB and IR modes, and plug in your depth-sensing camera for more goodness.

Kinect – OSCeleton

The home run: look at skeletal tracking for extremely precise human control of parameters, as seen in the video. It only gives your left and right hand, but stay tuned for further developments. See also this example patch.

Example video shows how to “track hand positions and translate to volume and send levels.” Not impressed? Remember, it’s a proof of concept: you can assign to other parameters, practice your movements, change the musical content, and even modify the patch to make it work better.

It’s Electronic Body Music! (Sorry, couldn’t resist. Kids, ask your … parents jeez we’re getting old, aren’t we?)

Across series of colored bars, sounds warp and mutate. Vines entangle as organic threads of music. Fingers and objects traverse sonic landscapes in surprising, mysterious ways. Welcome to the worlds of BricK, the musical table interface by Jordan Hochenbaum and Owen Vallis, which, charged with software by Dimitri Diakopoulos, Jim Murphy, and Memo Akten, explores new musical frontiers. The tool uses a combination of open source tools for tracking fingers and objects on a table, then feeds those into sound and music environments.

Just following the landmark, long-awaited release of Processing 1.0, BricK demonstrates the expressive potential of the open-source platform. Processing allows quick and elegant development of stunning visual interfaces, while other tools (ChucK and Reaktor, for instance) serve as sonic engines. Sometimes the sounds themselves are not revolutionary, but by simply replacing the visuals and interaction – just as with changing the look of a score – the music is transformed, too. (At top: experiments with different interfaces for music using the platform they’ve built.)

CDM got to talk to Owen and Jordan about the projects. And now’s a perfect time – the gorgeous Roots is looking for a home, in case we have any curators / galleries / other interested parties in our audience. First, a review of what these platforms are:

Spaces, Multi-Touch Music

Spaces Multi-Touch Music Environment from BricK Table on Vimeo.

Jordan tells CDM about Spaces, their latest creation, which premiered alongside a performance by Daedelus in LA:

Spaces is the latest interactive multi-touch musical application for the Brick Table. Designed as a minimalist interface to free musicians from traditional compositional markers such as frets and keys, the environment enables musicians to compose intuitively through immediate visual and sonic feedback.

In this video, Spaces mediates a spontaneous composition and performance of a slow-moving ambient soundscape.

Spaces was developed by Jordan Hochenbaum, Owen Vallis, Dimitri Diakopoulos, and Jim Murphy.  It was recently used in a performance at the REDCAT lounge at the Walt Disney Theatre, Los Angeles, and further developments are currently underway.

Roots, an Organic Installation

Roots Multi Touch Tangible Installation Teaser from BricK Table on Vimeo.

Roots has been impressive in Web videos, but it’s looking to make the transition to the real world, after a shipping mishap prevented what was supposed to be its premiere showing at New York’s Minitek Festival earlier this fall:

“Roots” is an interactive installation for the Brick Table tangible and multi-touch interface, where multiple people can collaborate in making music in a dynamic & visually responsive environment.  Users use their fingers and tangible objects to create and interact with virtual branch-like vines that move around the screen, allowing users to create either entirely generative, semi-generative, or sudo-composed arrangements and compositions. 

Roots is truly a unique and expressive interactive installation which came together through an internet collaboration between Brick Table’s creators (Jordan Hochenbaum and Owen Vallis), and the super-talented London-based designer/developer Memo Akten.  It was recently selected as a featured Processing Exhibition on Processing.org and we feel it is time to release Roots into the wild…

So! We are calling out to all of you lovely CDM readers out there to get Roots out and into the public.

For more information on how Roots works, please see What is Roots?

Please use the contact on the BricK Table website if you are interested.

Behind the Scenes

CDM: How do the visuals relate to the sound?

BricK: The nature of the vine-like branches in Roots lends itself to creating music with what is — in our opinion — an organic and open feeling. We felt that the music should both sound and feel as if it is coming out of the visuals, and vice-versa, and so we did our best to stay true to this relationship in the overall musical aesthetic of the sounds produced.

The Spaces environment expands on the theme of unconventional visual representations of sound manipulation. Each column is an open space connecting an idea with a musical parameter. Combined with the visual feedback, we decided Spaces would work best with slow-moving ambient soundscapes, although it is certainly possible to experiment with other musical styles..

What sorts of relationships did you experiment with before settling on something you liked?

BricK: With Roots, we first worked with Memo to develop the visual elements before even attempting the musical side of things.  We discussed various approaches to its visual and musical relationships. Did we want it to be completely generative? Did we want a more direct and repeatable relationship between your finger and the resulting sound?  We really liked both ideas, and so we made it all inclusive– making it able to create completely generative, semi-generative, or directly manipulated/composed musical outcomes by the use of finger pressing, sliding, and tangible object interaction.  This really makes Roots unique in comparison to other environments which enable generative musical arrangements.  Each performer can exert as much or as little control over the relationship between physical, visual, and musical interaction as they want at any given moment.

In Spaces, we discussed a few different ideas about the layout and design of the interface. Ultimately, we decided on Spaces being able to control four different instruments, each with four parameters (volume, and three others). We toyed with different methods for visually representing the value of each column without turning them into a traditional slider. We felt the cool-to-hot color morph in each column was fitting: the user has to rely more specifically on the sonic result rather than exact value, veering from more traditional musical interface paradigms.

How did you deal with timing relative to the visuals?

BricK: In Roots, it was necessary to have the generative data play in a relatively synchronized manner to maintain a degree of musicality. As the vines move around the environment, the musical outcomes are quantized to various beats. [Ed.: The quantization all happens in ChucK.] That being said, continuous finger movement scrubs audio in a direct 1:1 relationship that gives the user the feeling of direct manipulation when that is wanted.

Spaces has no generative movement (at the moment) which means timing is always completely synchronous with finger movement, both sonically and visually. We tried to make sure that the way in which the colors morph feel as free and smooth as the slowly evolving musical outcome.

Can you talk a bit about how the sound is generated?

BricK: Roots uses audio buffers as its underlying sound source (although the musical outcome is VERY different than the original material). Each vine gets assigned an audio buffer which is then ‘scrubbed’ through as the vine generatively maneuvers around the screen.  The audio and buffer manipulation is done using the ChucK audio programming language. By simply changing its source material, Roots will produce vastly differing musical results.

Spaces generates sounds in a number of different ways, all using Reaktor. Each of the four instruments employs a selection of synthesis methods. Some columns control pitch, other columns control combinations of filters and effects. The clicky percussive sounds are generated from an audio loop which is granulized and re-synthesized with altered delay rate, etc.

What are your future plans for these pieces?

BricK: Roots is ready to go, but in our free-time, Owen and I play with using it as a sequencing device in other ways — using movement and vine-location to pluck notes, control effects and filters, etc.

Exploring Roots along these other avenues will probably create the need for a new GUI interface, which means perhaps Roots will have a new little cousin sometime in the future.  That being said, we are really happy with Roots as is (we reached our specific goals), and we are more interested in giving it the proper debut it deserves, rather than changing the way it works.  We had a great time working with Memo, whose work I actually first came across here on CDM, and would love to work with him again in the future.

Spaces was developed in a very short timeframe for a performance at the REDCAT Lounge at the Walt Disney Concert Hall in downtown Los Angeles, and so we are absolutely looking to expand the possibility of what the Spaces interface is capable of. First, we would like to expand the number of instruments capable of being performed. Secondly, we would also like the interface to be “physics”-enabled, for example, using a flick motion to send a bouncing ball down a column to automate a parameter as the user concentrates on other instruments.

The Software

Just to review, here’s the software powering BricK:

tbeta (“The Beta”): finger tracking. tbeta is an open-source, cross-platform computer vision and multi-touch sensing platform. It’s the successor to the former touchlib, which wasn’t as cross-platform or quite as awesome. More on tbeta on Create Digital Motion.

reacTIVision: fiducial marker tracking for objects. (Fiducial markers are these funny, cellular-looking patterns pictured at right that allow you to track specific objects manipulated on the table. reacTIVision is the open-source library developed by the folks who did reactable. Sounds as though we might get fiducial tracking in the other library, though.)

ChucK: a strongly-timed, quick-to-code sound and synthesis language. It’s elegant enough that it’s used for real-time programming – as in, onstage, in laptop ensembles like PLOrk and (its West Coast descendent we just saw here on CDM) SLOrk.

Native Instruments Reaktor: The modular sequencer, instrument, and effect builder, which we cover regularly on our Kore minisite. It’s the only commercial / non-open-source choice here, though it may actually replace ChucK on Roots in the future.

More Info

Brick website