Perhaps you’ve seen the demo videos, as people do astounding things by moving their body around and using the Kinect camera to make music. Now, a set of Max for Live devices makes it reasonably easy to access your body as input inside Ableton Live.

FaceOSC Mapper

Pictured at top, this builds on Kinect superstar coder Kyle McDonald’s face-tracking tool and lets you use your face – position and even facial movements – to control Ableton Live parameters.

Kinect Camera

For use with the V-Module and vizzABLE systems, you can plug in one or more Kinect cameras, and get tilt, distance filtering (to remove backgrounds), depth maps, RGB and IR modes, and plug in your depth-sensing camera for more goodness.

Kinect – OSCeleton

The home run: look at skeletal tracking for extremely precise human control of parameters, as seen in the video. It only gives your left and right hand, but stay tuned for further developments. See also this example patch.

Example video shows how to “track hand positions and translate to volume and send levels.” Not impressed? Remember, it’s a proof of concept: you can assign to other parameters, practice your movements, change the musical content, and even modify the patch to make it work better.

It’s Electronic Body Music! (Sorry, couldn’t resist. Kids, ask your … parents jeez we’re getting old, aren’t we?)

You might imagine sound in space, or dream up gestures that traverse unexplored sonic territory. But actually building it is another matter. Kinect – following a long line of computer vision applications and spatial sensors – lets movement and gestures produce sound. The challenge of such instruments has long been that learning to play them is tough without tactile feedback. Thereminists learn their instrument through a the extremely-precise sensing of their instrument and sonic feedback.

In AHNE (Audio-Haptic Navigation Environment), sonic feedback is essential, but so, too, is feel. Haptic vibration lets you know as you approach sounds — essential, as they’re invisible. The work of Finland-based DJ/VJ Matti Niinimäki, aka MÅNSTERI (“Mons-te-ri”), the project is part of research undertaken at SOPI Research Group at Media Lab Helsinki. Like some sort of sound sorcerer, the user is entirely dependent on movement, feel, and sound as they move unseen sound sources through space. (More technical details below.)

It’s labeled, as always, “proof of concept.” The creator promises more videos to come; we’ll be watching as this evolves, as it looks terribly promising.

Below, “Tension” is a fair bit simpler, in which users walk through a space and control synth parameters. (“You are the knob,” one might say, though I don’t suggest shouting that at someone you don’t know. They could take it the wrong way.)

More descriptions:

AHNE

This is a demonstration video of AHNE – Audio-Haptic Navigation Environment.

It is an audio-haptic user interface that allows the user to locate and manipulate sound objects in 3d space with the help of audio-haptic feedback.

The user is tracked with a Kinect sensor using the OpenNI framework and OSCeleton (github.com/​Sensebloom/​OSCeleton).

The user wears a glove that is embedded with sensors and a small vibration motor for the haptic feedback.

This is just the first proof-of-concept demo. More videos coming soon.

HEI Project 2011
SOPI Research Group
sopi.media.taik.fi/

Aalto University School of Art and Design

AHNE – Sound and Physical Interaction

Tension

A brief video showing Tension. An interactive spatial sound installation for multiple users.

A person enters the space and a generative sound is assigned to that person. The sound pans around in the 6-channel speaker system following the user in the space.

Up to 5 users can use the installation at the same time. Each person modifies the other sounds based on the distance to the other users. The closer you are to other people the more the tension in the sound increases.

Tension – Sound and Physical Interaction

Side note: watching these two videos makes me want to consult with someone on non-verbal expression, posture, and stage presence. That criticism is mounted at myself – I could use it. Perhaps we need an all-physical, unplugged music event for laptopists, controllerists, and electronic musicians. And I can at least say I’ve had some experience in this, working in the dance program at my undergraduate alma mater, Sarah Lawrence. Anyone game? (Sounds like something we could do while CDM is in Berlin in the fall.)

For their part, the Finnish research facility is working with dancers, along with Nokia Research Center. (Sadly, I can’t find documentation.) But I think interesting things happen when us non-dancers learn movement technique, too.

Music making with your face? It’s just the latest novel way of manipulating your computer with movement, thanks to a revived interest in camera-based interaction spurred by Microsoft’s Kinect and hackers making it work, and other computer vision libraries. One original work: FaceOSC, which uses custom tracking code and a standard computer webcam (no additional hardware required) and free code to send control information for applications like live music performance. Kyle McDonald may have already wowed you with his face-tracking wizardry, but it’s easy to want to know more. Sure, it’s cool, but, um, what is it for? How do you get started? Is the timing quick enough for this to work in music? And what can we expect in the future?

I spoke with Kyle, educator, artist, and coder, about those questions and more. He’s also got some examples of what people are already doing just days after the release of his software – there’s some serious viral quality to open source code.

You’ve done work with both Kinect and now, in FaceOSC, your own camera-tracking software. How is working with Kinect for musical applications, in terms of latency? How have you found latency in your own FaceOSC application?

The only number I’ve heard regarding the Kinect’s latency is from Synthtopia, where they give 100 ms. That seems a little high to me. In my experience, the depth camera seems to have an extra frame of latency compared to the color camera. So I’d put the latency somewhere between 30 ms and 80 ms. In other words: don’t expect it to be a precision tool for live percussion, but for everything else I think there’s just
as much to explore as with any other camera.

FaceOSC feels like adding an extra frame or two of latency on top of what you’re getting from your camera. So you shouldn’t expect to beatbox or do percussion with it, but for controlling parameters in a musical context, you should be set.

How do you imagine this being used? I mean, obviously, in some ways this is (very) experimental, if good, clean fun — is there a
practical application? (EyeWriter, the eye-tracking application that improves accessibility, comes to mind as one possibility.)

I imagine FaceOSC being used to prototype ideas surrounding face-based interaction. I created it because Jason Saragih, the researcher behind FaceTracker, uses an open source non-commercial license for his code. He asks that anyone who wants to use the code email him directly, as a way to keep track of the usage. This is great, but I know that one of the fastest ways to get cool stuff happening is to make new tools and research accessible to a wide audience. So I asked him if it would be ok to make a standalone app for people to prototype their ideas — even if they don’t have access to the code. Everyone already “speaks” OSC so I thought this would be the easiest way to get the technology out there. Eventually, if people need to integrate it into a single application, they can contact Jason directly and use my ofxFaceTracker addon to get started:
https://github.com/kylemcdonald/ofxFaceTracker

And if they need to go the commercial route, there’s FaceAPI
http://www.seeingmachines.com/product/faceapi/

As far as a practical applications, I could imagine it augmenting the way the computer understands us. I’ve been thinking a lot about this
recently. Your computer has a microphone to listen to you, an accelerometer to know when you drop it, a camera to watch you, an
ambient light sensor to know how bright the screen should be. I have to wonder if it makes sense to respond to our pose and facial
expressions.

That said, here are the few experiments I’ve seen so far:

http://jeffwinder.blogspot.com/2011/07/face-gestures-faceosc-and-flash.html

Ingredients above:

FaceOSC
+
RoboFab’s Glyph Math
robofab.org
+
Vanilla
code.typesupply.com
+
Ideal Sans
http://www.typography.com/fonts/font_overview.php?productLineID=100042

Want to talk at all about your approach to developing this — particularly as you’ve been teaching others?

I think everyone learns differently, but for me I learn by playing. So I try to make it easy for other people to play by providing interfaces like FaceOSC (or, with 3d scanning, via my structured light work).

Anything else musicians might want to know about your work?

I haven’t spent enough time recently making music, but I’m always thinking about things in musical terms. My older work has a lot of
musical interfaces and ideas scattered through it, if you dig through http://kylemcdonald.net you might find some inspiration there.

Thanks, Kyle! If you’re in NYC, as Kyle and I are — or, may be, when we’re not traveling to opposite ends of the globe — Kyle has a couple of recommendations. There’s a “no-more-than-monthly” Kinect meetup organized by Sean Kean:
http://www.meetup.com/volumetric/

Also, there’s an amazing “summer school” meetup on July 21. Wish I could be there myself, but I’ll send regards from Berlin. Hope one of the other New Yorkers can report back.
http://eyebeam.org/events/meetup-demo-day

For more on Kinect, we’ve got loads of coverage on Create Digital Motion:
http://createdigitalmotion.com/tag/kinect/

It is, after all, Motion!

Got a creation of your own, or a meetup in your area? Let us know!

Speaking of sound design, here’s a nice weekend diversion for you: “140 dB” is an experimental short film, combining sonic acrobatics with digitally-synthesized motion spectacle, that meditates on the “interaction between sound and physical bodies. The main idea is to show how sound can change objects form and structure including human mental state.”

Thanks to the flexibility and media-agnosticism of the computer, all the work – art direction, animation, and sound design – come from one person, Tadas Svilainis. It’s a tantalizing teaser; I’d love to see the idea expanded. Tadas, you’re one multi-talented maker.

Of course, as for the content, one man’s torture is apparently another man’s party. I can see a few friends seeing these kinds of sounds as a nice way to unwind. Richard Devine, looking at you.

Found via @TheFilmo and @SoundPlusDesign.

America, enjoy your 4th of July holiday weekend. (I’m in the Netherlands; hopefully I don’t create any fireworks of my own, like accidentally plugging in a non-universal power adapter… again.)

Ben X Tan writes to let us know he’s working with hacks for Microsoft’s Kinect 3D camera system for Xbox to perform MIDI control. Result: depth-sensing, gestural musical manipulations! It’s just a prototype, but since today I cover the larger landscape of what’s happening with Kinect, it’s well worth teasing. From the description:

Coded in C#.net using this: http://codelaboratories.com/nui

Very hacky ugly, yucky, alpha prototype, source code available here:
http://benxtan.com/temp/pmidickinect.zip

Next project is making a version of pmidic that uses Kinect. Then, you can control Ableton Live or any other MIDI software or hardware with you limbs. Isn’t that amazing!!!

If you are interested, you should also check out:
http://pmidic.sourceforge.net/
http://benxtan.com

He tells me, “It’s just proof of concept for now, but want to make something nicer in the long run The community on IRC for this stuff is great. Very supportive. I love all the hacking going on at the moment. Especially after watching the Social Network. Very inspiring!”

So, what’s this whole Kinect thing about, why should you care, why might it be useful to artists and musicians and designers generally, and where do you go to find the code? I’ve rounded up various hackers working on the project to answer those questions on Create Digital Motion (in this case, Capture Digital Motion):
Kinect Hacking and Art Round Table: Why it Matters, What You Need to Know

I enjoy a little April Foolery now and then – see Google.com at the moment. But I had a funny revelation about some of the April Fools’ jokes out there today: they could be real. Some even already are.

For starters, consider the EyeDJ. The video is hilarious, a spot-on parody of the hype level in digital DJing at the moment. And they tune directly into the kind of breathless sense of innovation with no real purpose, with lines like “There is no need to cart around heavy turntables or expensive MIDI controllers – all you need is a computer and a webcam and you’re ready to mix with the best.” It taps directly into the reinvent-the-wheel zeitgeist in which we sometimes find ourselves and, yes, which to which I’ll admit I sometimes (often?) contribute.

But that’s the funny thing about design challenges and unusual interfaces: often the more seemingly off-the-wall, the more you learn. When Kokoromi organized a one-button game challenge, accompanied by a gallery show I helped assemble, some of the immediate feedback was that one-button games could be essential to people who require one button interaction for accessibility reasons. And sure enough, the EyeDJ is not already possible, but already implemented. EyeWriter is an in-development project using the open source C++ cousin of Processing, OpenFrameworks, which allows legendary graffiti artist TEMPTONE (Tony Quan) to overcome ALS and continue creating tags. Check out the video:

The Eyewriter from Evan Roth on Vimeo.

It’s clear the technology could have musical applications, too. None of us knows what our physical capabilities will be tomorrow, or next week. It’s time to start thinking about different interactions, and different humans. Compose Xenakis worked with architect Le Corbusier’s Modulor, a design unit that related the Golden Ratio and the human body. I’m a fan of both thinkers, but the problem with the Modulor is that it assumes there’s an “ideal” human proportion. In fact, we’re all different.

Don’t get me wrong – and I hope I didn’t make the EyeDJ guys feel insensitive. In fact, far from it: sometimes it’s the kind of apparently-absurd thinking you get from humor or parody that can free you from the constraints of everyday thinking.

Other “jokes” got me thinking, too. Topping the list:

TechCrunch, still sore about their Crunchpad project, suggest a DIY alternative:

Introducing The Do-It-Yourself CrunchPad Kit [Video]

They may have gotten a bit overexcited, as well, as they posted it with a dateline of March 31. A kit you can buy that converts existing laptops to touchscreens? Sign me up! In fact, it’s been done.

Touchscreens.com is one of a number of vendors. Nicely enough, you can spend a small amount of pocket change and convert even a PowerPC-based Mac. It’s single touch only, but it’s not too expensive, and you even get true stylus input, something the iPad doesn’t do.

http://www.touchscreens.com/products-addon.html

On the multi-touch front, research and artist group NOR_/D has their own kits, called, nicely enough, TouchKit. I’ve been working to get in touch with that crew to talk more about it. They’ll set you back ($1500, $2500 is the ballpark), but, again, truth is stranger than fiction.

http://labs.nortd.com/touchkit/

Generally, I love the idea of DIY touch and gestural input kits – let’s make them.

The Moog Music Moogerfooger MF-401 “Auto De-Tune” is another Moog April Fools’ classic, especially with presets like “Original Star Trek cast member attempts to make pop music.” (Ah, but which one – with Shatner, Nimoy, and TNG’s Spiner, Frakes…)

But the idea is sort of interesting, seen abstractly: tune incoming audio, then manipulate it. There’s a Pd patch in here somewhere. Go make it.

Also, I briefly thought Moog was making some sort of vocal processor, which would also have been cool.

Bonus round: In a late-night IM session with James Grahame of Retro Thing, we began to wonder what a one-knob synth might be like. (Having just done a one-button instrument, and thinking I may actually work on it more, we could have a whole suite of stuff here.)

James also has my favorite April Fools’ joke of the year. Catch it while you can.

And the grand prize for truth simply being stranger than fiction, Novation’s faux 10×8 controller, seen on Synthtopia can’t possibly compete with the absurdity of seeing the monome 512, which is apparently not an April Fools’ joke. For the current low bid of US$2800, you can get a 32 x 16 matrix of buttons. (Brian and Kelli write that “several close friends encouraged the endeavor.” I hope there was a bet involved.)

It is very, very pretty. Time to start scratching off lotto tickets.

But what April Fools’ jokes would you like to see become reality – and anyone want to take up the challenge? (Remember, Saturday midnight is the deadline for a contest to tell us your music tech dreams.)

Neurosonics Audiomedical Labs Inc. from Chris Cairns on Vimeo.

Marvin Suggs and his Amazing Muppaphone was just way, way ahead of his time. But if you haven’t already seen it making the rounds, you owe yourself a little video watching break to check out Neurosonics Audiomedical Labs, Inc., an audiovisual dreamscape in which disembodied heads form electronic drum heads and spin on turntables. The work is produced by Chris Cairns of Partizan Lab, who has a striking resume of commercial spots and worked with folks like Lady Sovereign.

The good folks of Motionographer get the scoop on the production background, and interestingly note that the music is scratching away in Scratch Perverts’ weapon of choice, Serato. Be sure to spot that story, as well as the official film site:

Neurosonics Audiomedical Labs Inc. [Motionographer]
http://www.neurosonicsaudiomedical.com/

If you have any interest in video turntablism, you won’t want to miss dj rndm’s detailed review of the Video-SL by Serato, which allows fluid scratching of video from the Serato digital DJ solution. Thanks to Todd, Josh Randall, and everyone else who sent this our way.

And yet no one has really produced modern Muppaphone technology. Shame. (Hint: get some friends, some socks, and don’t forget googly eyes.)

From the time we’re kids, we use gestures to make music – shaking, tapping, moving our bodies around, and connecting physical movement to sound. The idea of using these kinds of gestures to control digital music has been something researchers have worked on for many years. But with increasingly smart phones, equipped with mics, tilt and acceleration sensors, cameras, and other inputs, it’s possible to actually deliver these tools to average users.

The latest entry in the field is ZooZBeat. Its life as a mobile app is just a matter of months, but the research behind it involves years of work at Georgia Tech (which recently opened the Georgia Tech Center for Music Technology). The work comes from Gil Weinberg and and co-designers/programmers Andrew Beck and Mark Godfrey. We’ve followed Gil’s work with smart music apps for some time. I got the chance to talk to him about ZooZBeat.

ZooZBeat Website

Georgia Tech Center for Music Technology

Shake it Like a Polaroid

The idea behind ZooZBeat is to use gestures to build up music ideas. Shake and tilt, touchscreen taps, and (Nokia) keypad presses add rhythmic and melodic lines, as seen in the video. Now, if this seems to lack some of the precision of a musical instrument, it’s not just you: the early apps are primarily built to be friendly to novices.

“You can go and you can practice and be much better,” says Weinberg. “But … it helps you get started, even if you’re a novice.”

The free ZooZBeat Lite version already lets you play individually with up to 2 beats running in the background and 10 instrument sounds, and a full-blown version adds voice recording (minus the iPod touch), song saving, more customization, and more sounds. A “Pro” version is coming, too, for more serious use.

If you have an iPhone, an iPod touch, or a Nokia N95, you can try this out for yourself. (Interestingly, the Symbian-based N95 actually trumps the iPhone when it comes to wireless sharing.) The Apple-platform app is available now, with the Nokia app coming within the next few days.

Lowering the Floor, Raising the Ceiling

I talked to Gil about the development process and the ideas behind the project.

“The main issue is how to create low floor and high ceiling — how to allow everyone, kids to [older people] to make music they like and have a meaningful beginning,” says Gil. “People try a cello and it sounds terrible and they drop it. I’m trying to make it easier [to] connect to sound.”

That idea is a familiar one, of course, and something that comes up regularly in new digital instrument design. (In fact, one might wonder if it causes people to neglect the potential of design with instruments intended for more depth.) But the interesting thing is always just how you go about it. Gil says this is the culmination of about ten years of research. For ZooZBeat, it involved doing a lot of testing and development, including interviews, surveys, and user testing.

“Sometimes I did it with musicians, but with the cellphones we focused on novices,” says Gil. “We have kids — friends of my kids from school, a group of them played with [the instrument], and also students at Georgia Tech. observations were very useful, just watching as people used it.”

And the idea wasn’t just to focus on making the design novice-friendly. “The low floor is easy if you just care about the low floor,” Gil observes. “The trick is how to make a high ceiling — once you start, you can also grow up in the house, become better musically.”

As it happens, working with testing and allowing novices and kids to try the instrument yielded some surprises. “The way I played it was tapping. I took it with one hand and tapped on the other hand, the way I thought it would be expressive. Kids came and preferred to shake it.”

With shaking the primary interface, the question of how to accurately measure shakes becomes important. I note some of the challenges of using this as a input, as witnessed by early game development on the Nintendo Wii; recently Nintendo even announced it was adding additional hardware to allow the Wii remote to be more accurate. Gil answers that Georgia Tech is working with providers that may be able to add additional data.

Buzz around the iPhone aside, Gil had a lot of success working cross-platform. Both apps share a common engine for gesture recognition. Building specifics for the platforms wasn’t such a major challenge, thanks to the work both Apple and Nokia have done. “We did it pretty quickly,” says Gil. “We started with the Nokia, believe it or not.” After Apple released the 2.1 SDK for its iPhone and iPod touch, Gil says the team got the work done in under a couple of months. They’re examining other platforms, as well. (By the way, another reason to be interested in Nokia as a development platform: Nokia Labs has already completed a Symbian mobile library for computer vision applications — read, easy camera analysis. Hear that, Gil and programmers?)

Gil promises more developments soon, including that Pro app. We’ll be watching – and it’ll be interesting to hear your feedback.

Previous Research

Mobile software is one delivery platform, but it’s worth looking at some of Gil’s previous research to see where this came from. I suspect some people may actually prefer the tangible objects to mobile phones.

For an overview of what Gil has done:

Music Shapers: These squeezable balls created soft, squishable musical inputs

Beatbugs: Networked physical objects for kids, the Beatbugs are intelligent “rhythm computers” – handheld percussion for the digital age

iltur: Inventing is one thing – and some point, composition and performance matter, actually using those inventions. iltur is a series of compositions realizing musical applications of the Beatbugs.

Obviously, this is not a comprehensive guide to gestural music research, just Gil’s own contributions. Doing that kind of round-up wouldn’t be a bad idea, so if you have suggestions, I’m all ears (or squeeze-ready fingers).

Stay tuned; more soon.

What does music look like? With new sounds and new technologies, the question is more apt than ever. Tom of Music thing points, via his Twitter feed, to this interesting post regarding Ligeti’s Artikulation:

Visualizing Artikulation [Bad Assembly]

Music notation takes on a different meaning in the age of computers. After all, the essential divide in notation – between sound representation and realization – is blurred in the digital domain, in which we move between visual and sonic information seamlessly and a sound can be reproduced exactly. But, perhaps in that fluid context and without the musical conventions that grew up with notation, the importance of notation becomes that much clearer.

In this case, the classic experimental electronic composition Artikulation by composer György Ligeti has already had a visual score associated with it. Rainer Wehinger created the visuals above after the fact as an “aural score,” intending visuals to present a visible “reading” of the sounds of the piece. That makes the score itself closer to the digital visualizations we see as motion graphics works all over the Web (and on our sister site Create Digital Motion). The point isn’t to create a set of instructions by which you can perform a piece, but a visual counterpart that allows you to (presumably) hear it differently.

To be honest, I’m not always certain what to make of these results. Does this score really help you hear the piece? I’m curious to hear different reactions. But I wonder if the real holy grail comes back to software and interface. Seeing a pre-composed score is already interesting. But make that score interactive, and, in short, you have music creation software. Perhaps we’ll get beyond simple sequencers and step sequencers and start to see a growing number of interactive software designs that play around with that concept. (See Tom’s other thoughts on that today as he looks to Audio Damage’s new Automaton plug-in.)

Side Note: Twittering

If you want to follow us music bloggers on Twitter, I’m (uncreatively) peterkirn; Tom Whitwell is tombola. FriendFeed for me is the same. I haven’t made a CDM Twitter account; if for some reason that interested you, let me know, but otherwise I’m inclined to think RSS is just fine.

And if you have Twitters/FriendFeeds you think I should follow, please do holler.

In addition to the week’s worth of discussions and skillsharing classes, there will be two “showcase” nights, Saturday Jan. 19th and Sunday Jan. 20th. For those interested, I will be exhibiting two audiosculptural pieces, Octophonopod and Snowy Day during the event on Saturday. There’s a riduculous amount of talent on both nights, amounting to some of the most fresh and innovative people working in sound and motion graphics today.

[- Michael Una]

A full week of buzzing digital media creation will be showcased starting Tuesday, January 15th through Monday, January 21st 2008, at several venues downtown Chicago.

MGFest 08 features two world-premier theater screenings, multiple post-production studio tours, audio/visual art exhibits, a/v performances, six days of motion design / sound design / and motion programming classes.

These events celebrate creative minds and bright ideas within the emerging media landscape. Whether your a hard-core geek, a dedicated composer, a designer, film-maker, artist, engineer, programmer, educator, aficionado, prosumer, hobbyist, or just someone who likes being close to the creative community, mgFest is for you. The festival has become a destination for creative professionals from all over the Midwest and the nation by attracting the meshwork of companies and individuals that surround creative motion-picture design.

Socialize at one or all of the night events while experiencing multimedia art, video and sound in synch. Watch some of the past years most creative shorts or travel thru alternate dimensions in video art. Come mingle with local art directors and advertising executives, or discuss the technical operations of a professional edit bay and graphics shop with seasoned veterans of the industry. Listen to panels on the newest advancements in delivery technology affecting every media professional. Attend cutting edge classes of your choice within the realms of motion design, sound design and programming. Awaken your creativity at the Imagination College.

( Design | Sound | Code ) .: in motion

Check out mgFest.com for the most current festival information.