The latest in a long tradition of transforming hand-drawn graphics into music (see, in particular, Iannis Xenakis and UPIC), we see a computer-vision-powered pen-and-paper music generator. Kovacs Balazs writes:

This is a manual sounddrawer. Doesn’t need any sensors, but a camera, paper, colored pens. Doesn’t need sensor glove or reactable as well.

What I love about this, though, is that the resulting sounds are utterly crazy, a big collision of notes and sound.

By the way, UPIC lives on here in a very advanced program descended from the original tool:
http://www.iannix.org/en/index.php

From credits: Magyar Eötvös Ösztöndíj Alapítvány, UCSB-MAT, CSALÁD

More:
http://soundsofpictures.blogspot.com/2012/01/17.html

Compare the complex model of what a computer can use to control sound and musical pattern in real-time to the visualization. You see knobs, you see faders that resemble mixers, you see grids, you see – bizarrely – representations of old piano rolls. The accumulated ephemera of old hardware, while useful, can be quickly overwhelmed by a complex musical creation, or visually can fail to show the musical ideas that form a larger piece. You can employ notation, derived originally from instructions for plainsong chant and scrawled for individual musicians – and quickly discover how inadequate it is for the language of sound shaping in the computer.

Or, you can enter a wild, three-dimensional world of exploded geometries, navigated with hand gestures.

Welcome to the sci fi-made-real universe of Portland-based Christian Bannister’s subcycle. Combining sophisticated, beautiful visualizations, elegant mode shifts that move from timbre to musical pattern, and two-dimensional and three-dimensional interactions, it’s a complete visualization and interface for live re-composition. A hand gesture can step from one musical section to another, or copy a pattern. Some familiar idioms are here: the grid of notes, a la piano roll, and the light-up array of buttons of the monome. But other ideas are exploded into spatial geometry, so that you can fly through a sound or make a sweeping rectangle or circle represent a filter.

Ingredients, coupling free and open source software with familiar, musician-friendly tools:

• Two projectors
• A monome
• Processing, the elegant and artist-savvy free software for visual code
• Ableton Live and Cycling ’74′s Max for Live, acting as the interactive glue with the sound world
• Drumaxx, Image-Line’s tasty physical-modeled drum synth
• Native Instruments Battery, the sampled drum engine
• Eclipse, the free IDE, for Java coding in this case
• Community Core Vision and reacTIVision (based on our previous info, at least), free and open source community-based projects for making the interfaces you see in movies happen in real life.

Another terrific video, which gets into generating a pattern:

Now, I could say more, but perhaps it’s best to watch the videos. Normally, when you see a demo video with 10 or 11 minutes on the timeline, you might tune out. Here, I predict you’ll be too busy trying to get your jaw off the floor to skip ahead in the timeline.

At the same time, to me this kind of visualization of music opens a very, very wide door to new audiovisual exploration. Christian’s eye-popping work is the result of countless decisions – which visualization to use, which sound to use, which interaction to devise, which combination of interfaces, of instruments – and, most importantly, what kind of music. Any one of those decisions represents a branch that could lead elsewhere. If I’m right – and I dearly hope I am – we’re seeing the first future echoes of a vast, expanding audiovisual universe yet unseen.

Previously:
Subcycle: Multitouch Sound Crunching with Gestures, 3D Waveforms

And lots more info on the blog for the project:
http://www.subcycle.org/

Perhaps you’ve seen the demo videos, as people do astounding things by moving their body around and using the Kinect camera to make music. Now, a set of Max for Live devices makes it reasonably easy to access your body as input inside Ableton Live.

FaceOSC Mapper

Pictured at top, this builds on Kinect superstar coder Kyle McDonald’s face-tracking tool and lets you use your face – position and even facial movements – to control Ableton Live parameters.

Kinect Camera

For use with the V-Module and vizzABLE systems, you can plug in one or more Kinect cameras, and get tilt, distance filtering (to remove backgrounds), depth maps, RGB and IR modes, and plug in your depth-sensing camera for more goodness.

Kinect – OSCeleton

The home run: look at skeletal tracking for extremely precise human control of parameters, as seen in the video. It only gives your left and right hand, but stay tuned for further developments. See also this example patch.

Example video shows how to “track hand positions and translate to volume and send levels.” Not impressed? Remember, it’s a proof of concept: you can assign to other parameters, practice your movements, change the musical content, and even modify the patch to make it work better.

It’s Electronic Body Music! (Sorry, couldn’t resist. Kids, ask your … parents jeez we’re getting old, aren’t we?)

Ed.: Hacking Web databases to search sounds, remixing tools to automatically create dub tunes, cameras to sequence and analyze images in new ways, Montréal hackers have been busy. Trevor Knight writes from the event with full coverage from Canada, latest outpost of this global music coding phenomenon:

Music Hack Day made its first appearance in Canada at the end of September, painting the event with a Montréal flavour, complete with bilingualism, Montréal-style bagels, and even an appearance of Stephen Harper in a hack. Over the Saturday-Sunday event, musicians, programmers, and hackers scramble to create any sort of new music project.

With the help of dozens of laptops, gallons of coffee, several APIs and staff from such companies as The Echonest, Soundcloud, and Grooveshark, the assembled hackers churned out and presented 24 hacks in 24 hours.

Bruno Angeles took home first prize for his hack, FaCeQuencer, which uses computer vision and a webcam to control a squencer/looper and at the same time, outfit the user appropriately to the style of music.

Free Assembly, the hack from Graham Boyes, took second prize and recreates a target sound using a database of samples. It uses The Echonest Remix API for the analysis and finds target sounds sourced using Freesound.org’s API. The power of this hack was clear when Graham demonstrated using a drum and bass track as the target sound and a recording of a dog playing in water as the sample.

With a heavy presence of students and researchers from the Centre for Interdisciplinary Research in Music Media Technology (CIRMMT) and McGill University, several of the hacks incorporated data-mining. For example, David Weigl, Hannah Robertson, and Andrew Hankinson created wuzhear, a venue-based database of historical concerts in Montreal from the Montreal Jazz Festival website and last.fm’s API, including set information and playable with a Grooveshark widget.

The Wub Machine 2.0, from Peter Sobot, automatically creates Dub or Electrohouse remixes of an audio sample, while The Beatbox Machine allows one to record beatboxing and return a drum sequence replaced with actual drum samples.

For a complete list of hacks, check out http://wiki.musichackday.org/index.php?title=Montreal_2011_Hacks

Now that Canada has tasted the sweet Music Hack Day nectar, there’s already buzz for a hack day in Toronto.

Dac Chartrand (Renoise) writes with more details. Dac has his own set of hacks, but I’m excited enough about it that I’ll put that in a separate post -PK:

Other "Renoisers" were on site, but used the 24 hour session to try new ideas, but not present them. For example Steve Sinclair (Radarsat1) tried to port Mark Zadel’s Different Strokes to Android. Different Strokes resembles a freehand drawing application. The drawn strokes create animated figures whose motion is mapped to sample playback. The musician performs by assembling networks of strokes live, generating audio patterns. Steve got drawing and particles working but not enough time to hook into the Android audio subsystem, so he didn’t bother presenting.

Longtime CDM readers Studioimaginaire were also on site hacking away at their multi-touch table. They were there the full two days but also didn’t bother presenting. I tried to talk them into it several times, saying that the crowd would obviously vote for them just on cool factor alone and that they would walk away with a prize, but they stuck to their principles. Something to be said about the vibe of the event. Hackers were there to have fun.

David Viens of Plogue made am appearance for the demo session Sunday afternoon.

Good times had by all.

You might imagine sound in space, or dream up gestures that traverse unexplored sonic territory. But actually building it is another matter. Kinect – following a long line of computer vision applications and spatial sensors – lets movement and gestures produce sound. The challenge of such instruments has long been that learning to play them is tough without tactile feedback. Thereminists learn their instrument through a the extremely-precise sensing of their instrument and sonic feedback.

In AHNE (Audio-Haptic Navigation Environment), sonic feedback is essential, but so, too, is feel. Haptic vibration lets you know as you approach sounds — essential, as they’re invisible. The work of Finland-based DJ/VJ Matti Niinimäki, aka MÅNSTERI (“Mons-te-ri”), the project is part of research undertaken at SOPI Research Group at Media Lab Helsinki. Like some sort of sound sorcerer, the user is entirely dependent on movement, feel, and sound as they move unseen sound sources through space. (More technical details below.)

It’s labeled, as always, “proof of concept.” The creator promises more videos to come; we’ll be watching as this evolves, as it looks terribly promising.

Below, “Tension” is a fair bit simpler, in which users walk through a space and control synth parameters. (“You are the knob,” one might say, though I don’t suggest shouting that at someone you don’t know. They could take it the wrong way.)

More descriptions:

AHNE

This is a demonstration video of AHNE – Audio-Haptic Navigation Environment.

It is an audio-haptic user interface that allows the user to locate and manipulate sound objects in 3d space with the help of audio-haptic feedback.

The user is tracked with a Kinect sensor using the OpenNI framework and OSCeleton (github.com/​Sensebloom/​OSCeleton).

The user wears a glove that is embedded with sensors and a small vibration motor for the haptic feedback.

This is just the first proof-of-concept demo. More videos coming soon.

HEI Project 2011
SOPI Research Group
sopi.media.taik.fi/

Aalto University School of Art and Design

AHNE – Sound and Physical Interaction

Tension

A brief video showing Tension. An interactive spatial sound installation for multiple users.

A person enters the space and a generative sound is assigned to that person. The sound pans around in the 6-channel speaker system following the user in the space.

Up to 5 users can use the installation at the same time. Each person modifies the other sounds based on the distance to the other users. The closer you are to other people the more the tension in the sound increases.

Tension – Sound and Physical Interaction

Side note: watching these two videos makes me want to consult with someone on non-verbal expression, posture, and stage presence. That criticism is mounted at myself – I could use it. Perhaps we need an all-physical, unplugged music event for laptopists, controllerists, and electronic musicians. And I can at least say I’ve had some experience in this, working in the dance program at my undergraduate alma mater, Sarah Lawrence. Anyone game? (Sounds like something we could do while CDM is in Berlin in the fall.)

For their part, the Finnish research facility is working with dancers, along with Nokia Research Center. (Sadly, I can’t find documentation.) But I think interesting things happen when us non-dancers learn movement technique, too.

While we’re on the subject of making music with Kinect, the 3D computer vision camera with depth-sensing, here are some other experiments into how music might work. As with the classic Theremin, those musical gestures tend to be mapped against two-dimensional axes in space.

And from there, things become wide open. Johannes Kreidler, a musician and artist known for irreverent and inventive experiments in music, shares his studies for the Kinect, which he terms “conceptual music.” A solo “for violin” can involve literally waving a violin around. “House music” can mean making music whilst ironing a shirt. Any gesture in space becomes musical. Without tangible feedback, that can be challenging, and since these are just gestures in air, precision and nuance may not be a strong suit. But it’s a fascinating look into what’s possible, a set of thought experiments in music with a camera.

Composer Johannes Kreidler’s other works have included provocative ideas like making a performance of a short piece with 70,200 quotations of other “sampled” works, tunes from wildly-gyrating stock market quotes, entire bodies of work (like the Beatles) compressed into seconds, and pieces from avant-garde happenings to more conventional electro-acoustic scores. See his site for more.

And he’s got a killer book on using Pd, too.

In a whimsical proof of concept, artist and inventor Dan Wilcox harnesses the depth-sensing powers of the Kinect camera to turn a room full of drifting balloons into music. It occurs to me that the basic spatial model can be seen as descended directly from the Theremin – way to go, Leon, still relevant today. The sounds are simple, but it seems something you could continue to develop musically – to say nothing of what it could do for the ball pit at Chuck E. Cheese’s. (Slogan: Where a Kid Can Be a Kid Who Gets Obsessed With Skeeball Prizes / Get Scared Out of a Kid’s Mind By the Other Kids in the Ball Pit. Sorry, it’s an American suburban thing of a certain age, for the more than half of you who have no idea what I’m talking about.)

Full description from Dan:

Experiments in balloon motion and sound using an MS Kinect depth sensing camera.

Created for the Carnegie Mellon 1st & 2nd year MFA Graduate show entitled “Fresh Baked Goods” at Bakery Square, April 2011.

A machine stands in a room surrounded by balloons. Circulating fans blow the balloons over the machine which creates sound based on their movements.

Mode 1: Tones

Balloon height and x/y position control the pitch and panning of a treble and bass voice. The tones can be quantized into a certain key or a glisssando can be employed for a theremin-style effect.

Mode 2: 99 Luftballons

The playback speed of Nena’s 99 Luftballons is controlled by balloon height. The balloons must be kept in the air for the song to play. Feed the machine.

Built using Open Frameworks, ofxKinect, and Open CV for balloon tracking and Pure Data for sound generation/playback.

See danomatika.com/​blog/​balloon-project for more info.

Dan has a master plan with a robotic music-playing suit and other ideas, so I can’t wait to see where this goes.

You may have already seen FaceOSC, free software that eases the use of facial tracking from a computer camera for use as a controller, here with music software (top). Synthtopia picked up the story in July, featuring artist and engineer Kyle McDonald. But one FreeKa Tet has done his own implementation (second from top), and while the video is a bit grainy, he sounds wonderfully terrifying, as if his face is trying to slip out of The Matrix.

Sometimes, I’m rendered entirely silent (no, really, it happens), and it’s best to let videos speak for themselves. So here, after the jump, are some whimsical and wild prosthetic sound light-up … hell, I don’t have any idea what’s going on, but I’m enjoying it.

I dare you to start some conversation about musicality. Just don’t be surprised, comment trolls, if you find yourself abducted by a glowing and oddly glitchy-sounding creature with long, monstrous fingers. I’d watch what I say, frankly. Remember the old saying “on the Internet, no one knows if you’re a dog?” I expect that extends to space aliens, too.

Plenty more where those came from.

Oh, look, I could have done my research and seen there’s a bio for Mr. FreeKa Tet, aka Bacon ClapCLAP.
:

Extreme violent bursts, silence, speech, hard rock samples, strange atmospheres, Burgers , American Idol Icon, crackles, retardation, puking static, rocking a gabber party, cutting a worm in half and watching both parts moving, get a watch tatoo on his wrist, confusing videogames with music, drawing little puke characters on friends faces …

Right. Okay. That cleared everything up.

Enjoy your weekend, folks, and stay high and dry, those of you here on the Eastern seaboard of the US.

Music making with your face? It’s just the latest novel way of manipulating your computer with movement, thanks to a revived interest in camera-based interaction spurred by Microsoft’s Kinect and hackers making it work, and other computer vision libraries. One original work: FaceOSC, which uses custom tracking code and a standard computer webcam (no additional hardware required) and free code to send control information for applications like live music performance. Kyle McDonald may have already wowed you with his face-tracking wizardry, but it’s easy to want to know more. Sure, it’s cool, but, um, what is it for? How do you get started? Is the timing quick enough for this to work in music? And what can we expect in the future?

I spoke with Kyle, educator, artist, and coder, about those questions and more. He’s also got some examples of what people are already doing just days after the release of his software – there’s some serious viral quality to open source code.

You’ve done work with both Kinect and now, in FaceOSC, your own camera-tracking software. How is working with Kinect for musical applications, in terms of latency? How have you found latency in your own FaceOSC application?

The only number I’ve heard regarding the Kinect’s latency is from Synthtopia, where they give 100 ms. That seems a little high to me. In my experience, the depth camera seems to have an extra frame of latency compared to the color camera. So I’d put the latency somewhere between 30 ms and 80 ms. In other words: don’t expect it to be a precision tool for live percussion, but for everything else I think there’s just
as much to explore as with any other camera.

FaceOSC feels like adding an extra frame or two of latency on top of what you’re getting from your camera. So you shouldn’t expect to beatbox or do percussion with it, but for controlling parameters in a musical context, you should be set.

How do you imagine this being used? I mean, obviously, in some ways this is (very) experimental, if good, clean fun — is there a
practical application? (EyeWriter, the eye-tracking application that improves accessibility, comes to mind as one possibility.)

I imagine FaceOSC being used to prototype ideas surrounding face-based interaction. I created it because Jason Saragih, the researcher behind FaceTracker, uses an open source non-commercial license for his code. He asks that anyone who wants to use the code email him directly, as a way to keep track of the usage. This is great, but I know that one of the fastest ways to get cool stuff happening is to make new tools and research accessible to a wide audience. So I asked him if it would be ok to make a standalone app for people to prototype their ideas — even if they don’t have access to the code. Everyone already “speaks” OSC so I thought this would be the easiest way to get the technology out there. Eventually, if people need to integrate it into a single application, they can contact Jason directly and use my ofxFaceTracker addon to get started:
https://github.com/kylemcdonald/ofxFaceTracker

And if they need to go the commercial route, there’s FaceAPI
http://www.seeingmachines.com/product/faceapi/

As far as a practical applications, I could imagine it augmenting the way the computer understands us. I’ve been thinking a lot about this
recently. Your computer has a microphone to listen to you, an accelerometer to know when you drop it, a camera to watch you, an
ambient light sensor to know how bright the screen should be. I have to wonder if it makes sense to respond to our pose and facial
expressions.

That said, here are the few experiments I’ve seen so far:

http://jeffwinder.blogspot.com/2011/07/face-gestures-faceosc-and-flash.html

Ingredients above:

FaceOSC
+
RoboFab’s Glyph Math
robofab.org
+
Vanilla
code.typesupply.com
+
Ideal Sans
http://www.typography.com/fonts/font_overview.php?productLineID=100042

Want to talk at all about your approach to developing this — particularly as you’ve been teaching others?

I think everyone learns differently, but for me I learn by playing. So I try to make it easy for other people to play by providing interfaces like FaceOSC (or, with 3d scanning, via my structured light work).

Anything else musicians might want to know about your work?

I haven’t spent enough time recently making music, but I’m always thinking about things in musical terms. My older work has a lot of
musical interfaces and ideas scattered through it, if you dig through http://kylemcdonald.net you might find some inspiration there.

Thanks, Kyle! If you’re in NYC, as Kyle and I are — or, may be, when we’re not traveling to opposite ends of the globe — Kyle has a couple of recommendations. There’s a “no-more-than-monthly” Kinect meetup organized by Sean Kean:
http://www.meetup.com/volumetric/

Also, there’s an amazing “summer school” meetup on July 21. Wish I could be there myself, but I’ll send regards from Berlin. Hope one of the other New Yorkers can report back.
http://eyebeam.org/events/meetup-demo-day

For more on Kinect, we’ve got loads of coverage on Create Digital Motion:
http://createdigitalmotion.com/tag/kinect/

It is, after all, Motion!

Got a creation of your own, or a meetup in your area? Let us know!

Digital musician and artist Josh Silverman began the Synplode process with something familiar – a checkerboard. Play a game of checkers on its computer vision-equipped playing field and beats and loops triggered in Ableton Live generated a responsive soundtrack for the game. But as it’s evolved, Synplode has become a general-purpose musical grid. Whether with little robotic insects (the Hexbugs here) or full-sized human persons, the grid can turn any space into a dynamic, interactive dance floor. (I think I may actually prefer those cute little bugs to the people and dancers and whatnot. Robot rave, anyone?)

I prodded Josh to write up more description of what’s going on, so he’s created lots of documentation on the project Website.

The basic interaction:

At the start of the Synplode demo video, it is easy to see that a wave passes over the basic projected grid, flashing one column at a time, each containing 8 trigger regions. When a participant (or microbot) is present on a region, it is activated. When the wave intersects with an activated region, it causes a Synplosion, expressed through a splash of color and a distinctive sound. In the grid, each row represents a distinctive color and pitch or audio sample.

The basic ingredients:
1. Computer vision in OpenFrameworks, the fully open-source, artist-friendly C++ toolkit inspired by Processing.
2. Ableton Live, triggering clips in Set Mode and modulating them with MIDI effects and racks.

For more detail:
How it Works (details, in particular, of what’s happening in Ableton)
Why it Works (some of the thinking behind the interaction)
Synplode Project Page

Josh first demonstrated this system publicly at our Handmade Music series here in New York, and this is just the kind of experimentation and iteration I like to see. Here’s the original, checkerboard version: