From the time we’re kids, we use gestures to make music – shaking, tapping, moving our bodies around, and connecting physical movement to sound. The idea of using these kinds of gestures to control digital music has been something researchers have worked on for many years. But with increasingly smart phones, equipped with mics, tilt and acceleration sensors, cameras, and other inputs, it’s possible to actually deliver these tools to average users.

The latest entry in the field is ZooZBeat. Its life as a mobile app is just a matter of months, but the research behind it involves years of work at Georgia Tech (which recently opened the Georgia Tech Center for Music Technology). The work comes from Gil Weinberg and and co-designers/programmers Andrew Beck and Mark Godfrey. We’ve followed Gil’s work with smart music apps for some time. I got the chance to talk to him about ZooZBeat.

ZooZBeat Website

Georgia Tech Center for Music Technology

Shake it Like a Polaroid

The idea behind ZooZBeat is to use gestures to build up music ideas. Shake and tilt, touchscreen taps, and (Nokia) keypad presses add rhythmic and melodic lines, as seen in the video. Now, if this seems to lack some of the precision of a musical instrument, it’s not just you: the early apps are primarily built to be friendly to novices.

“You can go and you can practice and be much better,” says Weinberg. “But … it helps you get started, even if you’re a novice.”

The free ZooZBeat Lite version already lets you play individually with up to 2 beats running in the background and 10 instrument sounds, and a full-blown version adds voice recording (minus the iPod touch), song saving, more customization, and more sounds. A “Pro” version is coming, too, for more serious use.

If you have an iPhone, an iPod touch, or a Nokia N95, you can try this out for yourself. (Interestingly, the Symbian-based N95 actually trumps the iPhone when it comes to wireless sharing.) The Apple-platform app is available now, with the Nokia app coming within the next few days.

Lowering the Floor, Raising the Ceiling

I talked to Gil about the development process and the ideas behind the project.

“The main issue is how to create low floor and high ceiling — how to allow everyone, kids to [older people] to make music they like and have a meaningful beginning,” says Gil. “People try a cello and it sounds terrible and they drop it. I’m trying to make it easier [to] connect to sound.”

That idea is a familiar one, of course, and something that comes up regularly in new digital instrument design. (In fact, one might wonder if it causes people to neglect the potential of design with instruments intended for more depth.) But the interesting thing is always just how you go about it. Gil says this is the culmination of about ten years of research. For ZooZBeat, it involved doing a lot of testing and development, including interviews, surveys, and user testing.

“Sometimes I did it with musicians, but with the cellphones we focused on novices,” says Gil. “We have kids — friends of my kids from school, a group of them played with [the instrument], and also students at Georgia Tech. observations were very useful, just watching as people used it.”

And the idea wasn’t just to focus on making the design novice-friendly. “The low floor is easy if you just care about the low floor,” Gil observes. “The trick is how to make a high ceiling — once you start, you can also grow up in the house, become better musically.”

As it happens, working with testing and allowing novices and kids to try the instrument yielded some surprises. “The way I played it was tapping. I took it with one hand and tapped on the other hand, the way I thought it would be expressive. Kids came and preferred to shake it.”

With shaking the primary interface, the question of how to accurately measure shakes becomes important. I note some of the challenges of using this as a input, as witnessed by early game development on the Nintendo Wii; recently Nintendo even announced it was adding additional hardware to allow the Wii remote to be more accurate. Gil answers that Georgia Tech is working with providers that may be able to add additional data.

Buzz around the iPhone aside, Gil had a lot of success working cross-platform. Both apps share a common engine for gesture recognition. Building specifics for the platforms wasn’t such a major challenge, thanks to the work both Apple and Nokia have done. “We did it pretty quickly,” says Gil. “We started with the Nokia, believe it or not.” After Apple released the 2.1 SDK for its iPhone and iPod touch, Gil says the team got the work done in under a couple of months. They’re examining other platforms, as well. (By the way, another reason to be interested in Nokia as a development platform: Nokia Labs has already completed a Symbian mobile library for computer vision applications — read, easy camera analysis. Hear that, Gil and programmers?)

Gil promises more developments soon, including that Pro app. We’ll be watching – and it’ll be interesting to hear your feedback.

Previous Research

Mobile software is one delivery platform, but it’s worth looking at some of Gil’s previous research to see where this came from. I suspect some people may actually prefer the tangible objects to mobile phones.

For an overview of what Gil has done:

Music Shapers: These squeezable balls created soft, squishable musical inputs

Beatbugs: Networked physical objects for kids, the Beatbugs are intelligent “rhythm computers” – handheld percussion for the digital age

iltur: Inventing is one thing – and some point, composition and performance matter, actually using those inventions. iltur is a series of compositions realizing musical applications of the Beatbugs.

Obviously, this is not a comprehensive guide to gestural music research, just Gil’s own contributions. Doing that kind of round-up wouldn’t be a bad idea, so if you have suggestions, I’m all ears (or squeeze-ready fingers).

Stay tuned; more soon.

With all this talk of mobile music creation, it’s time to get a little scientific. Which mobile digital platforms do you actually own? Which do you want to read about on CDM? We have, of course, lots of interesting stuff happening with actual mobile computers – think UMPC, Eee, and OLPC – but then, those fit nicely with other computing platforms since that’s what they are. Other handheld game systems, PDAs, and phones require real, specific attention for musicians. And naturally, this is about making music on mobile systems, not necessarily playing Mario Kart.

We need to know what you think. Your feedback will help us direct the site. Don’t worry, we still love things that no one else does, so fear not if you’re in a more obscure category – though you will want to get your votes in.

This is also a chance to sign up for our soon-to-launch email list, which we’ll use for human-created, exclusive dispatches from team CDM in a form that makes sense for our overburdened inboxes. (It won’t duplicate anything else, and it won’t be too often, and it won’t be sold to anyone else – it’s just another way for me and the team to chat with you.)

So, here goes – you’ll need to click through to the actual site to finish the survey, or head straight to:

http://www.surveygizmo.com/s/61300/yvwf9

Respond now; poll closes Monday 8/18

Photo credit: Yesrobot’s Game Boy rig, captured by Alícia

Some good news, some bad news for iPhone/iPod Touch owners. (For everyone who doesn’t care, we’ll be consolidating iPhone news from here on out so you can safely ignore it.)

Good news: iZotope’s mobile version of iDrum is here (seen above). It’s a nifty $5 toy, though some restrictions, including the lack of audio export, may keep it from being more than that.I Correction: you can exchange both samples and project files with the desktop iDrum, and use ringtone bounce (including, apparently, on iPod touch) to export audio. That could make this very useful as a mobile addition to your workflow.

I do also think it’s inspiring in the way that it uses touch interfaces, something that could bode well for what touch-enabled computer music apps might look like.

Better news: BtBx is a fun-looking US$3.99 beat machine with drum sounds and (at last!) real-time synthesizers from the creator of PSP Rhythm. Unfortunately, it doesn’t let you use your own samples, and it can’t quite stand up to the cooler PSP Rhythm – even if hacking a PSP is kind of a pain. But it is a good sign.

But bad news for Apple owners, good news for owners of other gadgets: the generative music studio Mixtikl will hit those platforms first because of Apple is tying its developers’ hands with technical and legal restrictions. It’s not a deal killer for everyone – we’ve seen developers write special client apps to get around file exchange issues, and obviously a number of developers aren’t concerned with legal terms because they’re releasing apps anyway. (Jobs is justifiably proud of their 60 million-download count.) But there’s no question that part of why the iPhone is more a mobile toy and less a mobile computer is in fine print and legalese, not silicon. That could be mobile carriers’ fault – but either way, it could also demonstrate that shrinking computers and not more powerful mobiles are the future for mobile music creation.

iDrum is In

iZotope have released their first app for iPhone, a mobile counterpart to the iDrum drum machine. (Thanks to Richard Lawler for the tip!)

iDrum for iPhone/iPod touch comes in two editions, each costing US$4.99 – a “Hip Hop” and “Club” version. (If you buy both, bizarrely, you get two apps; Richard speculates this may be due to how Apple sandboxes their apps.)

Good:

• An elegant interface, showing what touch-enabled apps in general can do
• Ring tone creation
• Some sound design names we enjoy (Goldbaby, Matt Simmers, Art Gillespie, Sable Gray)
• Round-trip work with the iDrum desktop app

Bad:

• Can’t export audio of your creations directly, but you can use the ringtone bounce
• Ring tone creation requires a sync app on the desktop

iZotope iDrum for iPhone and iPod Touch

Update/correction: I’m, fairly I think, called out by iDrum developer Art Gillespie on two points:

1. I missed the most important feature here, which is that the mobile iDrum works with round-trip co

mpatibility with the desktop iDrum. As Art points out in comments:

“you can do full round-trip editing of beats–including sending samples back and forth–with iDrum (desktop) for Windows/Mac.”

This obviously would fundamentally change the workflow of using the mobile app. If you’re not an iDrum user, you might stick to the rival drum machine for iPhone, Intua Beatmaker. But if you are an iDrum desktop fan, this could be a real killer app.

2. His experience with Apple developer relations has been positive, meaning me blurring the description of iDrum with some other criticisms of Apple’s platform and developer relations is unfair.

In my defense, there’s actually no explicit mention of the ability to share project files between desktop and mobile iDrum. So, let’s say that right now, as that’s very, very cool. (In fact, it’s cool enough that this is worthy of a separate aside!)

As for developer relations, I think that’s fair – and it’s absolutely in keeping with what we’ve been hearing. Some people are happy, some are unhappy, and some are simultaneously happy and unhappy. That’s what one would expect on any developer platform. I don’t think it’s unreasonable to point out some of the weaknesses alongside the strengths. It would be far easier to do so if Apple hadn’t placed an NDA over everything having to do with development, so that does mean I’m often ranting in the dark. But without violating an NDA, I think we can very safely say Art is happy, and there are a number of happy developers putting out great apps. There are other developers who are less happy, which has the side effect of ensuring we’ll have mobile apps on other platforms (and the jailbroken Apple platform) to look forward to, alongside these apps.

BtBx (“BeatBox”), From PSP Rhythm’s Creator

Louis Iturzaeta and Billy, the talented developers of the way-awesome PSP Rhythm on Sony’s gaming handheld, has launched their first iPhone / iPod Touch app, using a modified version of their RHYTHM engine.

The good:

• Real-time synths, with some great, acid-style sounds
• Fairly impressive features in a compact space
• Pattern-auto save, online docs
• Real-time sound modification via their engine

The bad:

• No custom samples
• No audio export
• Kinda silly-looking interface (I prefer PSP Rhythm’s look – but Louie promises there’s a new skin coming soon)

At this absurdly low price, I can’t complain. Don’t let the baby toy interface fool you: the underlying sound engine means this could be a seriously fun soundmaker.

That said, I have to say, I’d choose the PSP app over this. I’ll add a major caveat, though: hacking a PSP is a pain. (More on that soon. Short version: buy a used unit on which someone has done the hard work for you.) It’s too bad Sony doesn’t have some outlet for homebrew developers like this to sell through the Sony PSP store. I think they could do great, iPhone-killing work.

Full specs from Billy:

Hey Peter and team, the drum machine/synthesizer BtBx that Billy and I (from PSP Rhythm) created is now available in the app store!

Below is are the features/specs of the app. It was written with a modified version of our RHYTHM audio engine. Our plan is to create a synth application and a full studio application as well as release "Lite" versions of each app.

BtBx is available for $3.99 and the "Lite" version will be available for $0.99 when it is released.
-Louie

http://www.pureprofitcorp.com/btbx
http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=287398151&mt=8

BtBx ("BeatBox") is a music sequencer for the iPhone or iPod Touch.
BtBx gives you instant access to the world of electronic music with big drum sounds and acid-style synthesizers.

Features:

• 8 drum sounds
• 2 instrument sounds
• 2 real-time synthesizers
• 16 step drum machine style sequencer
• 16 patterns
• Keyboard with +/- 3 Octave Range
• Realtime Mutes
• Tempo Adjustment (40-240 BPM)
• Low Pass Filter with cutoff frequency and filter resonance
• Auto-saves patterns so no data is lost
• Instruction manual is built into the application

BtBx utilizes a custom audio engine and sequencer which enables you to:

• Program song melodies with any sound
• Play any drum or instrument sound forward or in reverse on any pattern step
• Modify a Low Pass Filter on each sound (12 total LFPs running at the same time!)
• Add accent notes on the synthesizers
• Add Distortion to any sound
• Add Delay to any sound

BtBx Product Page

Mixtikl Bails on Apple for Now

One of the most exciting upcoming iPhone apps is delayed for the forseeable future. That’s Apple’s loss, but a gain for Windows Mobile and Symbian as they gain the developers’ focus. The tool is Mixtikl, a mobile edition of an innovative music creation platform with:

• a generative music engine
• synths and samplers
• effects network

Is the problem that the iPhone isn’t powerful enough? Absolutely not. The problem is what happens after you add in other restrictions. Bottom line:

We were (and still are) very excited by the potential of our software running on these Apple devices, and we love Apple products and all our other software products run on Mac OS X (and Windows XP of course).

So, we have decided not to press ahead with development until Apple can:

• relax a number of (as we see it) overly restrictive terms in the iPhone Developer Program License Agreement
• allow apps to share/exchange data/files between themselves and an attached PC/Mac

The developers can’t talk about specifics because of the NDA covering the agreement, but they do point back to some of the issues I’ve discussed here.

Of course, this isn’t the end of the road for Mixtikl, necessarily. If Apple could relax or even better clarify the terms of their agreement, this app could be back. But this further illustrates the problems with the NDA. It’ll be even harder for developers to share these restrictions with one another, and for those issues to be addressed, if no one can even talk about it.

We have some wonderful mobile toys at the moment, but I do look forward to the day when cool mobile platforms don’t come with gag orders attached (cough, Apple) or require elaborate hacking (ahem, DS and PSP) just to use. Windows Mobile and Symbian remain valid and should have better hardware behind them soon. As for Linux platforms, basically, we’re just waiting for more to actually ship.

It’s well worth reading the full story:

Mixtikl for iPhone / iPod touch – Intermorphic postpones development for now

Mixtikl Product Page

Previously:

Generative iPod? Deep Modular, Generative Music System Bound for iPhone, Phones, Windows, Mac

Aside from being toy-like mini-computers, could mobile devices take on a musical usefulness all their own?

At the Electronic Music Foundation’s 10th Anniversary Symposium in 2004, Morton Subotnik and fellow panelists imagined an iPod that, instead of playing canned music from your music library, would actually generate music for you on the spot. Believe it or not, commercial demand aside, that might soon be reality.

We saw Intermorphic’s fascinating generative music engine noatikl at the end of last year. It’s the “spritual successor” to the Koan generative system used by Brian Eno in 1996. Read up and see the videos here:

noatikl: New Generative Music Engine, So You Can Rock Out Like Eno

They’ve got various videos showing off what the results can be like, including the one at top, which combines noatikl and Apple’s Logic 8 synths. If you’re interested in learning more, Intermorphic has a page with some background on generative music with comments from pioneer Eno:

generative music @ Intermorphic

Enter iPhone, Mobile

Here’s where this all goes mobile. Wonderful mobile music site Palm Sounds notes that Mixtikl will allow on-the-go music production for a variety of platforms. You’ll be able to work on your Mac and Windows PC VST/AU host, but you’ll also be able to support:

• Windows Mobile 5, 6
• iPhone, iPod Touch
• Symbian Series 60 V2/3 smartphone
• Antix Game Player

The basic idea is a music tool that blends generative music tools and playback with access more traditional loops and patterns.

Mobile functions will include:

• Generative music playback and creation alongside loops, MIDI, and modular synths and effects
• Quick mixing with selectable sources/loops and effects, and preset generative players
• Cell-based performance mixers
• Synth sound editing
• Effect and “network” editing
• Packaging for export

Intermorphic’s Peter Cole also tells us that non-commercial licenses for noatikl are automatically being upgraded to commercial-use launch — and price cuts are coming on all the products.

Now, I have heard lots of skepticism about generative music in general, in everything from games to composition to live performance. But I’d remember, too, that this approach to music is really new. Eno’s 1996 iteration was revolutionary at the time, and few have followed. The enabling technologies have only recently fallen into place. And whereas most musical creation systems have plenty of existing precedents from linear analog tape and mixing decks to musical notation, generative music requires new ways of thinking. So I’m very interested to see what happens. And while this didn’t get a demo onstage at WWDC’s iPhone bash today, it could wind up meaning deeper things for music in the future than even iPod.

mixtikl Product Page; product due later this year