Neurosonics Audiomedical Labs Inc. from Chris Cairns on Vimeo.

Marvin Suggs and his Amazing Muppaphone was just way, way ahead of his time. But if you haven’t already seen it making the rounds, you owe yourself a little video watching break to check out Neurosonics Audiomedical Labs, Inc., an audiovisual dreamscape in which disembodied heads form electronic drum heads and spin on turntables. The work is produced by Chris Cairns of Partizan Lab, who has a striking resume of commercial spots and worked with folks like Lady Sovereign.

The good folks of Motionographer get the scoop on the production background, and interestingly note that the music is scratching away in Scratch Perverts’ weapon of choice, Serato. Be sure to spot that story, as well as the official film site:

Neurosonics Audiomedical Labs Inc. [Motionographer]
http://www.neurosonicsaudiomedical.com/

If you have any interest in video turntablism, you won’t want to miss dj rndm’s detailed review of the Video-SL by Serato, which allows fluid scratching of video from the Serato digital DJ solution. Thanks to Todd, Josh Randall, and everyone else who sent this our way.

And yet no one has really produced modern Muppaphone technology. Shame. (Hint: get some friends, some socks, and don’t forget googly eyes.)

From the time we’re kids, we use gestures to make music – shaking, tapping, moving our bodies around, and connecting physical movement to sound. The idea of using these kinds of gestures to control digital music has been something researchers have worked on for many years. But with increasingly smart phones, equipped with mics, tilt and acceleration sensors, cameras, and other inputs, it’s possible to actually deliver these tools to average users.

The latest entry in the field is ZooZBeat. Its life as a mobile app is just a matter of months, but the research behind it involves years of work at Georgia Tech (which recently opened the Georgia Tech Center for Music Technology). The work comes from Gil Weinberg and and co-designers/programmers Andrew Beck and Mark Godfrey. We’ve followed Gil’s work with smart music apps for some time. I got the chance to talk to him about ZooZBeat.

ZooZBeat Website

Georgia Tech Center for Music Technology

Shake it Like a Polaroid

The idea behind ZooZBeat is to use gestures to build up music ideas. Shake and tilt, touchscreen taps, and (Nokia) keypad presses add rhythmic and melodic lines, as seen in the video. Now, if this seems to lack some of the precision of a musical instrument, it’s not just you: the early apps are primarily built to be friendly to novices.

“You can go and you can practice and be much better,” says Weinberg. “But … it helps you get started, even if you’re a novice.”

The free ZooZBeat Lite version already lets you play individually with up to 2 beats running in the background and 10 instrument sounds, and a full-blown version adds voice recording (minus the iPod touch), song saving, more customization, and more sounds. A “Pro” version is coming, too, for more serious use.

If you have an iPhone, an iPod touch, or a Nokia N95, you can try this out for yourself. (Interestingly, the Symbian-based N95 actually trumps the iPhone when it comes to wireless sharing.) The Apple-platform app is available now, with the Nokia app coming within the next few days.

Lowering the Floor, Raising the Ceiling

I talked to Gil about the development process and the ideas behind the project.

“The main issue is how to create low floor and high ceiling — how to allow everyone, kids to [older people] to make music they like and have a meaningful beginning,” says Gil. “People try a cello and it sounds terrible and they drop it. I’m trying to make it easier [to] connect to sound.”

That idea is a familiar one, of course, and something that comes up regularly in new digital instrument design. (In fact, one might wonder if it causes people to neglect the potential of design with instruments intended for more depth.) But the interesting thing is always just how you go about it. Gil says this is the culmination of about ten years of research. For ZooZBeat, it involved doing a lot of testing and development, including interviews, surveys, and user testing.

“Sometimes I did it with musicians, but with the cellphones we focused on novices,” says Gil. “We have kids — friends of my kids from school, a group of them played with [the instrument], and also students at Georgia Tech. observations were very useful, just watching as people used it.”

And the idea wasn’t just to focus on making the design novice-friendly. “The low floor is easy if you just care about the low floor,” Gil observes. “The trick is how to make a high ceiling — once you start, you can also grow up in the house, become better musically.”

As it happens, working with testing and allowing novices and kids to try the instrument yielded some surprises. “The way I played it was tapping. I took it with one hand and tapped on the other hand, the way I thought it would be expressive. Kids came and preferred to shake it.”

With shaking the primary interface, the question of how to accurately measure shakes becomes important. I note some of the challenges of using this as a input, as witnessed by early game development on the Nintendo Wii; recently Nintendo even announced it was adding additional hardware to allow the Wii remote to be more accurate. Gil answers that Georgia Tech is working with providers that may be able to add additional data.

Buzz around the iPhone aside, Gil had a lot of success working cross-platform. Both apps share a common engine for gesture recognition. Building specifics for the platforms wasn’t such a major challenge, thanks to the work both Apple and Nokia have done. “We did it pretty quickly,” says Gil. “We started with the Nokia, believe it or not.” After Apple released the 2.1 SDK for its iPhone and iPod touch, Gil says the team got the work done in under a couple of months. They’re examining other platforms, as well. (By the way, another reason to be interested in Nokia as a development platform: Nokia Labs has already completed a Symbian mobile library for computer vision applications — read, easy camera analysis. Hear that, Gil and programmers?)

Gil promises more developments soon, including that Pro app. We’ll be watching – and it’ll be interesting to hear your feedback.

Previous Research

Mobile software is one delivery platform, but it’s worth looking at some of Gil’s previous research to see where this came from. I suspect some people may actually prefer the tangible objects to mobile phones.

For an overview of what Gil has done:

Music Shapers: These squeezable balls created soft, squishable musical inputs

Beatbugs: Networked physical objects for kids, the Beatbugs are intelligent “rhythm computers” – handheld percussion for the digital age

iltur: Inventing is one thing – and some point, composition and performance matter, actually using those inventions. iltur is a series of compositions realizing musical applications of the Beatbugs.

Obviously, this is not a comprehensive guide to gestural music research, just Gil’s own contributions. Doing that kind of round-up wouldn’t be a bad idea, so if you have suggestions, I’m all ears (or squeeze-ready fingers).

Stay tuned; more soon.

What does music look like? With new sounds and new technologies, the question is more apt than ever. Tom of Music thing points, via his Twitter feed, to this interesting post regarding Ligeti’s Artikulation:

Visualizing Artikulation [Bad Assembly]

Music notation takes on a different meaning in the age of computers. After all, the essential divide in notation – between sound representation and realization – is blurred in the digital domain, in which we move between visual and sonic information seamlessly and a sound can be reproduced exactly. But, perhaps in that fluid context and without the musical conventions that grew up with notation, the importance of notation becomes that much clearer.

In this case, the classic experimental electronic composition Artikulation by composer György Ligeti has already had a visual score associated with it. Rainer Wehinger created the visuals above after the fact as an “aural score,” intending visuals to present a visible “reading” of the sounds of the piece. That makes the score itself closer to the digital visualizations we see as motion graphics works all over the Web (and on our sister site Create Digital Motion). The point isn’t to create a set of instructions by which you can perform a piece, but a visual counterpart that allows you to (presumably) hear it differently.

To be honest, I’m not always certain what to make of these results. Does this score really help you hear the piece? I’m curious to hear different reactions. But I wonder if the real holy grail comes back to software and interface. Seeing a pre-composed score is already interesting. But make that score interactive, and, in short, you have music creation software. Perhaps we’ll get beyond simple sequencers and step sequencers and start to see a growing number of interactive software designs that play around with that concept. (See Tom’s other thoughts on that today as he looks to Audio Damage’s new Automaton plug-in.)

Side Note: Twittering

If you want to follow us music bloggers on Twitter, I’m (uncreatively) peterkirn; Tom Whitwell is tombola. FriendFeed for me is the same. I haven’t made a CDM Twitter account; if for some reason that interested you, let me know, but otherwise I’m inclined to think RSS is just fine.

And if you have Twitters/FriendFeeds you think I should follow, please do holler.

In addition to the week’s worth of discussions and skillsharing classes, there will be two “showcase” nights, Saturday Jan. 19th and Sunday Jan. 20th. For those interested, I will be exhibiting two audiosculptural pieces, Octophonopod and Snowy Day during the event on Saturday. There’s a riduculous amount of talent on both nights, amounting to some of the most fresh and innovative people working in sound and motion graphics today.

[- Michael Una]

A full week of buzzing digital media creation will be showcased starting Tuesday, January 15th through Monday, January 21st 2008, at several venues downtown Chicago.

MGFest 08 features two world-premier theater screenings, multiple post-production studio tours, audio/visual art exhibits, a/v performances, six days of motion design / sound design / and motion programming classes.

These events celebrate creative minds and bright ideas within the emerging media landscape. Whether your a hard-core geek, a dedicated composer, a designer, film-maker, artist, engineer, programmer, educator, aficionado, prosumer, hobbyist, or just someone who likes being close to the creative community, mgFest is for you. The festival has become a destination for creative professionals from all over the Midwest and the nation by attracting the meshwork of companies and individuals that surround creative motion-picture design.

Socialize at one or all of the night events while experiencing multimedia art, video and sound in synch. Watch some of the past years most creative shorts or travel thru alternate dimensions in video art. Come mingle with local art directors and advertising executives, or discuss the technical operations of a professional edit bay and graphics shop with seasoned veterans of the industry. Listen to panels on the newest advancements in delivery technology affecting every media professional. Attend cutting edge classes of your choice within the realms of motion design, sound design and programming. Awaken your creativity at the Imagination College.

( Design | Sound | Code ) .: in motion

Check out mgFest.com for the most current festival information. 

I purchased an AVM02 when it was released in Feb, and have been testing it out in preparation for 2 months touring Australia with a rock star.

While I’m very excited about my AVM02, there seems to be some definite resistance from the VJ community as it doesn’t include some features we’ve come to rely on (MIDI, easily accessible effects parameters). However, for DJs wanting to expand their performance to include visuals this may be just the right mix of audio and video.

Unfortunately the lack of MIDI will probably turn off live musicians who are too busy actually playing their instruments to mess with manually selecting video channels and crossfading. Let’s hope Numark release a MIDI capable AVM03 soon.

In the next couple of years I see this product lineup in the video market expanding similarly to DJ mixers, with different layouts, effects and options available for “turntablist” style highly-technical VJing, live looping with onboard sampling and effects, battle-style VJ mixers allowing you to mount your DVD player sideways and tag over your DVD labels like the hiphop kids do… In the meantime the AVM02 is really the only A/V mixer available at anywhere near this price point, and it does a superb job too. If you’re a visualist branching out into the frightening world of audio, a DJ looking to add video to your set, or just someone looking for a solid, competitively priced vision mixer, you should definitely give the AVM02 a try.

Read the full article on CreateDigitalMotion.

Tonight here in New York, I’ll be presenting a free workshop on detecting and visualizing motion from camera inputs, which may be of interest not only for those of you eagerly anticipating the new Create Digital Motion site, but also anyone who’d like to use cameras as controllers for music. Full details after the jump.

If you’re not in New York, don’t fret; I plan to organize this stuff and have online examples/tutorials in the near future. But if you are there, say hi! And yes, I finally plan to deliver on my New York CDM get-together promises in June; my schedule will finally allow that.

Still from a recent performance with Eric Dunlap and Mare Hieronimus at Eyewash, the visual performance series in NYC.

DETECTING AND VISUALIZING MOTION: Mechanics and Design Possibilities
by Peter Kirn

THIS THURSDAY, May 11, 6:30pm
Harvestworks
596 Broadway, #602
New York, NY 10012
Broadway at Houston Street

Please note that space is limited to approx. 15 people – please come early!

PETER KIRN presents methods of translating live camera inputs into digital visuals and control data, using Adobe Flash 8 and Max/MSP/Jitter (applicable to some other tools, as well). He will use examples from his own work, following a residency with choreographer Christopher Williams at Dance Theater Workshop and a performance via DTW and Boston CyberArts at Ideas in Motion in Cambridge. Looking behind the scenes at the mechanics of analyzing video and infrared photography, he’ll explore both the mechanics of these methods and how they might be applied to other projects, and how attendees can learn and explore some of these evolving techniques on their own.

Digging through to the actual product page for this, there are some interesting features. It is sending only MIDI, which is a little disappointing given the accuracy of data required for real motion capture; I wonder if they’re using the full data range of MIDI or just 127 values per sensor. But it’s a wireless device; many previous designs were wired. Its software looks very intuitive, rather than the homebrew Max/MSP patches that usually accompany such devices. There are presets for everything from Pro Tools to Traktor DJ. And, maximizing your value, it can moonlight as a motion capture device with Autodesk Motion Builder, in case you happen to be developing a computer game or shooting a sci-fi motion picture.

Be sure to check out the videos on the homepage to see this in action.

Now, it’s time to play “Where have we seen this before?” In fairness, the Gypsy MIDI folks claim only to be the first “commercial” product. Depending on how broadly we define commercial, though, that would include the Body Synth, developed in 1994 by Chris Van Raalte and Ed Severinghaus. I’ve seen several people using these, usually dancers, but the most “famous” user is probably singer/electronic musician Pamela Z (pictured here looking a bit like Edward Scissorhands). Another device that has made the rounds (not commercially, but I believe used by more than one artist) is the MIDI Dancer, developed and used by the Troika Ranch dance company. Like the Gypsy MIDI, this device sends data wirelessly.

So, yet again, what’s old is new again. These devices are looking like a bargain buy compared to the US$2500 Lemur multitouch touchscreen and, let’s face it, also cost less than a lot of high-end synths from Roland, Yamaha, and Korg. And body sensors are arguably more fun, since they require you to move about to produce sound. But while the hardware designs have been repeated, I think there’s a lot more progress to be made in this area.

First, these designs are all fairly intrusive. Portable — well, sort of, but they’re not comfortable to wear. And they look a bit like a pile of electrical cords fell on you; hardly wearable high-tech fashion. (The artsy shot here is the choreographer of Troika Ranch, tangled in her sensor suit.)

The next, and more important problem, is software. While we’ve come up with clever ways of capturing motion capture data for animation, it’s not at all clear how to best use body mappings for sound and music. I still believe that a great instrument should sound terrific, and sound unique, even if you don’t see how the sounds are being produced and controlled. I’d be really interested to see how they mapped the presets for Gypsy MIDI here. The Traktor DJ settings work well, but it’s mostly for show — the performance in the video is cool, but mapping a filter and crossfader is a little too pedestrian to have you screaming “I have to own this!”

To me, these are exciting problems, however. Sensor tech is obviously readily available: now the interesting question is, how can we make them musically successful?

Come on, cyborgs, let’s get on this.

(And readers, I’m sure I’ve missed other body control systems, so let’s put together our collective intelligence and flesh out my list!)