luanna is a beautiful new application out of Tokyo-based visual/sound collective Phontwerp_. Amidst a wave of audiovisual iPad toys, luanna is notable for its elegance, connecting swirling flurries of particles with gestures for manipulation. I imagine I’m not alone when I say I have various sample manipulation patches lying around, many in Pd, lacking visualization, and wonder what I might use in place of a knob or fader to manipulate them. In the case of luanna, these developers find one way of “touching” the sound.

As the developers put it:

luanna is an audio-visual application designed for the iPad
that allows you to create and control music through the manipulation of moving images.

The luanna app has been designed to be visually simple and intuitive, whilst retaining a set of rich and comprehensive functions. Through hand gestures you can touch, tap and manipulate the image, as if you were touching the sound. The image changes dynamically with your hand movements, engaging you with the iPad’s environment.

The interface is multi-modal, with gestures activating different modes. This allows you to select samples, play in reverse, swap different playback options, mute, and add a rhythm track or crashing noises. It’s sort of half-instrument, half-generative.

Phontwerp_ themselves are an interesting shop, descibed as a “unit” that will “create tangible/intangible products all related to sound.” Cleverly naming each as chord symbols, ∆7, -7, add9, and +5 handle sound art, merch, music performance / composition / sound design, and code, respectively. That nexus of four dimensions sounds a familiar one for our age.

Sadly, this particular creation is one of a growing number of applications that skips over the first-generation iPad and its lower-powered processor and less-ample RAM. Given Apple can make some hefty apps run on that hardware, though, I hope that if independent developers find success supporting the later models, they back-port some of their apps.

See the tutorial for more (including a reminder that Apple’s multitasking gestures are a no-no).

US$16.99 on the App Store. (Interested to see the higher price, as price points have been low for this sort of app – but I wonder if going higher will eventually be a trend, given that some of the audiovisual stuff we love has a more limited audience!)

Readers request Audio Copy and sample import right away. I think sample import, at least, could easily justify a higher price, by making this a more flexible tool.

Find it on our own directory, CDM Apps:
http://apps.createdigitalmusic.com/apps/luanna

http://phontwerp.jp/luanna/

Very similar in its approach is the wonderful Thicket, well worth considering:
http://apps.createdigitalmusic.com/apps/thicket

See our recent, extensive profile of that application’s development:
Thicket for iOS Thickens; Artists Describe the Growth of an Audiovisual Playground

See also, in a similar vein, Julien Bayle’s recent release US$4.99 Digital Collisions:

http://julienbayle.net/2012/04/07/digital-collisions-1-1-new-features/

http://apps.createdigitalmusic.com/apps/digital-collisions-hd

While we’re on the subject of making music with Kinect, the 3D computer vision camera with depth-sensing, here are some other experiments into how music might work. As with the classic Theremin, those musical gestures tend to be mapped against two-dimensional axes in space.

And from there, things become wide open. Johannes Kreidler, a musician and artist known for irreverent and inventive experiments in music, shares his studies for the Kinect, which he terms “conceptual music.” A solo “for violin” can involve literally waving a violin around. “House music” can mean making music whilst ironing a shirt. Any gesture in space becomes musical. Without tangible feedback, that can be challenging, and since these are just gestures in air, precision and nuance may not be a strong suit. But it’s a fascinating look into what’s possible, a set of thought experiments in music with a camera.

Composer Johannes Kreidler’s other works have included provocative ideas like making a performance of a short piece with 70,200 quotations of other “sampled” works, tunes from wildly-gyrating stock market quotes, entire bodies of work (like the Beatles) compressed into seconds, and pieces from avant-garde happenings to more conventional electro-acoustic scores. See his site for more.

And he’s got a killer book on using Pd, too.

Whatever the platform, you can expect musicians to do weird things and make noise. (It is, after all, our raison d’être.) In this case, while the input controller is an iPad or iPhone, the emphasis remains on running sound from additional gear, whether an inexpensive hardware synth in the form of a Korg monotron or the ubiquitous Ableton Live.

From the iOS inbox this week…

Top: nonstandard sequencing. Developer Franz Keller’s Radarhead 19 is one of the more unusual creations I’ve seen on iOS, a somewhat inexplicable (in a good way) alternative sequencer. It’s available now on iTunes.

Give it a try, after a short while some rather unique situations can emerge!
(In the hands of a real musician, or an untrained explorer.)
More complicated apps for musical experimentation are in the works now.

Interestingly, many control applications use Ableton Live as the sound source.

Via Vincent (something-wicked @ SoundCloud, Denkitribe has an Ableton control rig with two iPhones connected to Live wirelessly and gestural control via his n-forcer app, apparently not yet available but coming soon. Seen here in a demo at the Apple Store Ginza, and previously on ever-vigilant MATRIXSYNTH.

Perhaps more practical, Bjorn of Covert Operators and International Feel Recordings add their own custom iPad control layout app for Ableton Live to the various entries competing for your Ableton-controlling attention (including some on the JazzMutant Lemur). MB Control resembles some of the Lemur layouts, focusing on touch-augmented renditions of standard control schemes for triggering, mixing, and parameter adjustment. I still imagine this will fail to impress some people who enjoy tactile hardware controllers, but it does look practical, and there’s no reason you can’t put an iPad next to a box of physical faders.

No other details for now, so I think this qualifies as a “teaser.”

Whether we see similar competition from the Android, Chrome, and Linux and Windows camps I think remains dependent on what touch technology new devices use, and whether it works for control. We should know more as the fall and winter stretch on and some of those devices appear.

We’ve got further compelling evidence Apple doesn’t really own multi-touch and multi-touch gestures — and that other devices and interfaces will press forward (which is a good thing for everyone). Lest you think I’m straying too far from creating digital music, by the way, I think this means lots of new music apps – as musicians have devoured multi-touch more than any other group (and certainly have used it for the coolest stuff).

I am concerned about how multi-touch innovation will wrangle with over-zealous intellectual property legal wrangling. But hopefully I made it clear that, even with my concerns about Apple, the report that Google had pulled multi-touch capabilities to please Apple was full of question marks.

Luke Hutchison is more of an expert in this field than any Silicon Valley rumor reporter. Luke pulled off the kernel module hack that turns the Android G1 into a multi-touch gesture-capable device (with, incidentally, some examples that have tantalizing possibilities for musical applications). He has detailed instructions on that, if you’re interested – and his familiarity with the code gives him a compelling argument that Google did not cave to Apple – and Apple may not even have relevant patents in this case.

It’s worth reading his whole story as it’s full of technical details as far as multi-touch’s future on G1, but here’s the executive summary as far as Apple blocking multi-touch on Android:

(1) The G1 was simply never intended to be a multi-touch device.
(2) Apple’s multitouch patent may not even cover the pinch gesture.
(3) Google *is* interested in multitouch capabilities, it’s just nowhere near the top of their priority list.
(4) Google will deal with legal issues if and when they come up, but that hardly stops them doing something they think should be done.
(5) Apparently the driver for a resistive MT-capable/iPhone-like touchscreen was checked into the git kernel tree after the 1.0 release, so we now have (at least?) two MT-capable drivers in the tree.

In other words, if you make an iClone, expect to hear from Apple legal. If you just want to use or develop multi-touch devices and interfaces, rest easy – because even if Apple decides to make trouble, they’re likely facing even more multi-touch gesture-controlled devices and law teams to back them up.

Definitely worth reading, at Luke’s blog:
The Android Multi-Touch Conspiracy… and more tinfoil hats

Zoom-Zoom-Zoom — Get Multi-Touch Zooming Support on your T-Mobile G1 TODAY (and by the way, you can hack the kernel on new G1s without the Android developer unit?)

Before the iPhone, before HP computers and Windows 7 touch features and Apple trackpad gestures, the Jazz Mutant Lemur multi-touch interface was ahead of its time. Today, it’s still unique, in that it’s one of the few commercially-available devices to support OpenSoundControl, it’s a luxuriously-large multi-touch screen, and it has exceptional precision and low latency with its tracking. Of course, it has also been subject to two primary complaints: one, that the software options for creating onscreen interfaces is two simple, and two, that it costs too much.

Well, the Lemur and its more conventional DAW-controlling Dexter sibling address each of those. The Lemur has gotten a significant software upgrade, and both have gotten a steep price cut.

First, Lemur V2 is the biggest set of improvements since the Lemur’s unveiling. New in the upgraded firmware:

• Tabs for containers
• Breakpoint object for envelopes
• Alias controllers to save memory and (your) time
• Control the mouse cursor or keyboard shortcuts directly (that’s actually a huge deal right there)
• Pinch, rotate, trace gestures
• Pop-up menu object
• New JazzEditor, scripting powers, and instant Ableton Live reactivity – yep, doing stuff even the new Akai APC can’t.

The only bad news is, even with price drops, these devices aren’t cheap, though arguably they can make up for that with longevity. The Dexter has undergone a permanent price drop to a much more competitive US$1519. The Lemur has temporarily dropped to US$1769 for a 60-day promotion. No official word yet on what happens after that offer expires mid-March; stay tuned.

I do think this makes a pretty significant adjustment on value. I’m a big fan of the iPhone/iPod touch apps, but the input area is extremely small; there’s no real comparison.

Lemur V2
Jazz Mutant

And even if you decide not to get a Lemur yourself or can’t afford them, they remain a compelling example of what’s possible in the future of music hardware – and how powerful OSC can be. Look for a hands-on with Lemur V2 and more on OSC in other applications (many completely free) over the coming months.

To close out, here’s a nice, if simple, video demo of a Max/MSP step sequencer controller with the Lemur. What’s lovely about this is that it shows how building both software and hardware interface from the ground up can really give you control over how you’re playing. (Meaning, even if you hate this, you can go create something for youself that’s exactly what you love.)

this is a sequencer i programmed in max using the lemur as a control interface. it sends midi info to whatever program you use. i also forgot to mention that each track can have independent timings, so varitions can span longer than just 1 bar of music.

lemur users can download it off the jazzmutant website. user name – Andrew Graham

Got Lemur creations of your own? We’d love to see them. (I really appreciated seeing the amazing work Bryant Place was doing with LemurV2 and Ableton Live while I was out in LA.)

From the time we’re kids, we use gestures to make music – shaking, tapping, moving our bodies around, and connecting physical movement to sound. The idea of using these kinds of gestures to control digital music has been something researchers have worked on for many years. But with increasingly smart phones, equipped with mics, tilt and acceleration sensors, cameras, and other inputs, it’s possible to actually deliver these tools to average users.

The latest entry in the field is ZooZBeat. Its life as a mobile app is just a matter of months, but the research behind it involves years of work at Georgia Tech (which recently opened the Georgia Tech Center for Music Technology). The work comes from Gil Weinberg and and co-designers/programmers Andrew Beck and Mark Godfrey. We’ve followed Gil’s work with smart music apps for some time. I got the chance to talk to him about ZooZBeat.

ZooZBeat Website

Georgia Tech Center for Music Technology

Shake it Like a Polaroid

The idea behind ZooZBeat is to use gestures to build up music ideas. Shake and tilt, touchscreen taps, and (Nokia) keypad presses add rhythmic and melodic lines, as seen in the video. Now, if this seems to lack some of the precision of a musical instrument, it’s not just you: the early apps are primarily built to be friendly to novices.

“You can go and you can practice and be much better,” says Weinberg. “But … it helps you get started, even if you’re a novice.”

The free ZooZBeat Lite version already lets you play individually with up to 2 beats running in the background and 10 instrument sounds, and a full-blown version adds voice recording (minus the iPod touch), song saving, more customization, and more sounds. A “Pro” version is coming, too, for more serious use.

If you have an iPhone, an iPod touch, or a Nokia N95, you can try this out for yourself. (Interestingly, the Symbian-based N95 actually trumps the iPhone when it comes to wireless sharing.) The Apple-platform app is available now, with the Nokia app coming within the next few days.

Lowering the Floor, Raising the Ceiling

I talked to Gil about the development process and the ideas behind the project.

“The main issue is how to create low floor and high ceiling — how to allow everyone, kids to [older people] to make music they like and have a meaningful beginning,” says Gil. “People try a cello and it sounds terrible and they drop it. I’m trying to make it easier [to] connect to sound.”

That idea is a familiar one, of course, and something that comes up regularly in new digital instrument design. (In fact, one might wonder if it causes people to neglect the potential of design with instruments intended for more depth.) But the interesting thing is always just how you go about it. Gil says this is the culmination of about ten years of research. For ZooZBeat, it involved doing a lot of testing and development, including interviews, surveys, and user testing.

“Sometimes I did it with musicians, but with the cellphones we focused on novices,” says Gil. “We have kids — friends of my kids from school, a group of them played with [the instrument], and also students at Georgia Tech. observations were very useful, just watching as people used it.”

And the idea wasn’t just to focus on making the design novice-friendly. “The low floor is easy if you just care about the low floor,” Gil observes. “The trick is how to make a high ceiling — once you start, you can also grow up in the house, become better musically.”

As it happens, working with testing and allowing novices and kids to try the instrument yielded some surprises. “The way I played it was tapping. I took it with one hand and tapped on the other hand, the way I thought it would be expressive. Kids came and preferred to shake it.”

With shaking the primary interface, the question of how to accurately measure shakes becomes important. I note some of the challenges of using this as a input, as witnessed by early game development on the Nintendo Wii; recently Nintendo even announced it was adding additional hardware to allow the Wii remote to be more accurate. Gil answers that Georgia Tech is working with providers that may be able to add additional data.

Buzz around the iPhone aside, Gil had a lot of success working cross-platform. Both apps share a common engine for gesture recognition. Building specifics for the platforms wasn’t such a major challenge, thanks to the work both Apple and Nokia have done. “We did it pretty quickly,” says Gil. “We started with the Nokia, believe it or not.” After Apple released the 2.1 SDK for its iPhone and iPod touch, Gil says the team got the work done in under a couple of months. They’re examining other platforms, as well. (By the way, another reason to be interested in Nokia as a development platform: Nokia Labs has already completed a Symbian mobile library for computer vision applications — read, easy camera analysis. Hear that, Gil and programmers?)

Gil promises more developments soon, including that Pro app. We’ll be watching – and it’ll be interesting to hear your feedback.

Previous Research

Mobile software is one delivery platform, but it’s worth looking at some of Gil’s previous research to see where this came from. I suspect some people may actually prefer the tangible objects to mobile phones.

For an overview of what Gil has done:

Music Shapers: These squeezable balls created soft, squishable musical inputs

Beatbugs: Networked physical objects for kids, the Beatbugs are intelligent “rhythm computers” – handheld percussion for the digital age

iltur: Inventing is one thing – and some point, composition and performance matter, actually using those inventions. iltur is a series of compositions realizing musical applications of the Beatbugs.

Obviously, this is not a comprehensive guide to gestural music research, just Gil’s own contributions. Doing that kind of round-up wouldn’t be a bad idea, so if you have suggestions, I’m all ears (or squeeze-ready fingers).

Stay tuned; more soon.

Facial music, indeed. Daito Manabe has made the music the controller and his face the controlled in a new project. (As several commenters were quick to note, I got this exactly backwards when I posted this a few minutes ago: his face is the output, and the music the input.)

Daito Manabe is a Japanese-based composer, media artist, and DJ who does strange and wonderful things with inputs. This time, he’s hooked up electrical outputs to his face, so multimedia software Max/MSP, his usual tool of choice, can sequence muscle movements via electrical pulses transmitted directly to the surface of his face.

This is just one tech demo, but as an artist he does work in more fully-formed, full-length works — and happens to be coming to New York in November as part of a multimedia show at the Japan Society. Daito, if you’re out there reading I hope to catch up with you then, and perhaps buy you a Kirin, my honorary beer since it’s typically how my name is misspelled. (Peter Kirin is my typo Japanese alterego; the more popular Peter Kim my typo Korean doppelganger.)

Here’s another video; thanks to oscillateur and cptn for catching my jet lagged error.

Bless Nintendo for making the Wii controller: inexpensive, lots of internal sensor data (motion sensing, tilt sensing, buttons), elegant design, and standard Bluetooth support allowing it to be used with Mac, Windows, and Linux.

Now there’s free and open source software for making the most of your Wiimote as a musical instrument. First up: Wiinstrument, a multi-purpose percussion instrument, now available for all three operating systems (a Windows version was recently added).

• Plays percussion / drums with gestures
• Use an (in-development) internal sampler with WAV files, or trigger other software via MIDI
• Use tilt for control changes
• Supports tilt, velocity (how much force you use when you move your Wiimote), with acceleration from both the Wiimote and nunchuk
• It works with Mac, Windows, and Linux, via a standard OpenGL-based interface, thanks to the awesome 2D OpenGL library Gosu. (Programmers, take note.)

Of course, drums are just the beginning — you could use this to trigger clips, grooves, visualist videos and animations, whatever. And it comes with demos, tutorials, source code, the lot.

Wiinstrument Release Information
GarageBand tutorial (relevant to other apps, too)
Support information for Windows, Mac OS X Leopard, Linux
Via thread with the creator on our forums

Here is in action.

But, you say, that’s all well and good, but it’s not … air guitar. Today is your lucky day:

Wii Guitar, Drum Kit

Evan told us earlier about his Wii Drum Kit. To that nicely-crafted (Windows-only) software, he’s added a Wii Guitar. Evan’s drum kit isn’t quite as sophisticated as the Wiinstrument, though it could still be useful as a simple solution. But the Wii Guitar does something the Wiinstrument doesn’t:

It is a basic wiimote guitar that can be played similar to the way you strum a real guitar. Hold a direction on the d-pad to select a chord, and swing your wiimote up and down to strum.

What? You think you might look a little … silly … doing this? Heck, yeah! We wouldn’t have it any other way.

Groovy. And if you’re a Windows-based .NET programmer, you may find his site a good starting place for your own Wii-based projects. (Cross-platform developers will want to stick to the Wiinstrument for wiinspiration.)

Wiimote Guitar v0.1
Wiimote Drum Kit

Previously:

Wii Controller as Complete Audiovisual Musical Instrument, and How Less is More

It helps that Bjork pulls out some of her synthiest, electronicilicious-est tracks, like Pluto:

and Hyperballad:

And, of course, part of what happens is that the computer screen here has become the interface. When it works — when the visuals match the sounds, and suggest some new ways of constructing music — it really does show potential for this kind of instrument. (Even if you don’t buy into the blocks, the way the visualization itself works has a lot of promise.)

That’s the idea behind Microsoft’s Surface, too … but sometimes the gimmick can be a solution in search of a problem. Well, actually, maybe your computer of the future really will be “a big-ass table.” (Thanks, SarcasticGamer.com, for making me laugh so heartily.)

The good news: Microsoft is taking multi-touch, camera tracking, and gestural technologies seriously, and they have what looks like a very nice implementation that will be one of the first commercial implementations. The bad news: it’ll cost US$10,000 out of the gate. That high price will mean you’ll see at places like T-Mobile stores and Sheraton hotel lobbies first. But what you need to know: you can build your own version, thanks to available open source tools, with is likely to be more useful for music.

Good sources of commentary:
New Media Initiatives Blog at Walker Arts Center, which notes this could be museum-friendly tech.

Chris O’Shea @ Pixelsumo, who has built a device something like this himself.

The video does show what’s cool about Surface — and it’s easy to imagine these same techniques being applied to live visual and music performance. (People have already tried experiments in that, and I think there’s a lot more to be done — once you’re talking music rather than just digital snapshots, you get into deeper questions about how to model the interface.)

But let’s get a few things out of the way:
1. Enough about Minority Report, already! Minority Report was a terrific movie from an interface design perspective. (Slightly less so from a … well, movie perspective, but that’s another story.) But this has nothing to do with Minority Report‘s free-form, table-free, gestures in 3D space with gloves interface, unless all vaguely futuristic interfaces will now be compared to that movie. Actually, this thing is more practical, unless you want to share gloves with people at the Sheraton. (Ewwww … oh, and incidentally, there’s no logical reason why in the future we need gloves to do tracking.)

How about other comparisons? My apartment looks kinda like Blade Runner. This ATM is exactly like Lost in Space. If I start selling giant rubber costumes with fins, I can be the first to bring commercially-viable Doctor Who monster technology to a mainstream public. Oh, never mind.

2. “First commercial implementation” is sometimes meaningless. Since the dawn of time, pricey, first-to-market versions have often failed. Look at Apple: Lisa? Newton? Even the Mac initially lost out to the Apple II. That’s just one company. Now, there’s the additional reality of readily-available tools allowing people to take concepts like this and do whatever they want, freely experimenting without market restrictions. The commercial implementations may be more successful in that case — but the independent efforts could be sexier.

3. Tables take up space. Part of the reason you’ll be seeing this in hotel lobbies is that the tracking here presumably requires a large physical object in order to work, much like other interactive tables we’ve seen. If you want gestural or multi-touch technologies to be portable, or work with much smaller computer form factors, you need a different design. I’m still intrigued by what cameras may be able to do with enhanced computing resources.

None of this is intended to criticize Microsoft. I’m personally excited this stuff is catching on. I just want to make sure we remain tethered to the larger realities here.

That said, I’ll be eager to try out the demo version in New York in a couple of weeks, which just happens to coincide with a conference of people interested in musical interface design. Heck, maybe we can convince the Microsoft engineers to come over for a few drinks.

DIY Surface Computing

Be Your Own Bjork: If you want to build your own device like this, the reacTable software is available open source. It’s even available as a library for the free tool Processing, which is an ideal environment in which to learn how to code this stuff. reacTable is a little different from Surface, in that reacTable has objects on the surface to manipulate. That rules out some gestures you might make with your hands, but it does add additional tactile feedback, and, hey, playing with blocks is fun. Actual touch might be more difficult, though other computer vision tools that enable tracking are available.

Chris O’Shea (who comments on today’s announcement on his blog Pixelsumo) had also begun an open source software and hardware project using blocks on tables, called Sonic Forms. That project ultimately didn’t take off, but Chris has gone on to write about, speak publicly about, and do other wonderful things, like a table installation with music boxes.

Of course, my significant other doesn’t live in the world of blogs, and says, wisely, “but would you really want to do that?” That’s not a bad question to answer. It’s worth weighing this against other interface possibilities, and, for that matter, balancing the time you spend on the interface with the time you spend on music. More on both those topics soon.

And yes, obviously whoever shoots photos of these things doesn’t really get this concept. Here’s a hot tip to save you US$10,000:squeeze someone you love right now, huddle next to your computer in front of Google Maps, and you, too, can have “collaborative computing.” (Make sure they point meaningfully at the screen and laugh with delight at … the … directions … to the airport.) Since you can do this with a normal laptop, you don’t have to put a giant, clunky digital table in your living room. Which is good, because those coffee table books might screw up the tracking.