In the thrilling conclusion of our chat with Boing Boing’s Xeni Jardin, Matt Ganucheau and I explore deep thoughts about the roles of interactivity and adaptivity in music and game design — then attempt to dance in giants Katamari Damacy hats. (Note the use of the word attempt — those things were more than a bit tricky to move in. Hilarity ensues.)

Prior to leading a dance dance Revolución, we talk a bit about the ways in which game design relate to gesture in musical interface and how musical scores could become non-linear. The gesture issue really goes well beyond games to the fundamental question of how to relate to music physically — and, in a way, awkwardly-dancing musicians may be a fitting metaphor. Or parable. Or something or other.

We do it all for you.

I really did find this a fascinating way to promote discussion, so if you’ve got suggestions for future broadcasts, I’d love to hear them. You can even think of new silly things for us / guests to do.

See also, related:
Troels Folmann on the boiled waterphone-style instrument, sound design inspiration
Troels on “micro-scoring” adaptive music

For part the first:
GDC: Music, Video Games, and Interactivity – Chat with Boing Boing Video

Matt Ganucheau and I got to sit down with Xeni Jardin of Boing Boing Video during the Game Developer Conference to discuss some of the potential for interactive music in games. Matt is a composer, sound designer, and educator, talking about how he’s encouraging his own students to think about adaptive music in new ways, combining Max/MSP and a Space Invaders clone built in the Unity Game Engine. (See our story from earlier this week.)

I talk a little about my sense that new tools could expand the range of possibilities in game music. Right now, the two major game engines are the AudioKinetic Wwise and Firelight fmod engines, each of which do have potential of their own – and continue adding features for more interactive sound scores. Each got some significant, flashy new features announced at GDC. But I was especially impressed by the use of Pure Data (Pd) in a custom implementation inside the game Spore. That allowed the compositional team to produce a truly generative musical score (led by legendary composer Brian Eno, with EA’s Kent Jolly and composer Aaron McLeran). I hope we see more of that in the future. Starting of students doing it themselves (with Max in this case) is not a bad way to start.

Boing Boing has more video of us they’ll be posting soon – including the embarrassing but diverting footage of us dancing around in Katamari costumes.

I’m new to this speaking live thing, but hope you enjoy. My favorite part was getting questions going. Livecasting is something we’ll try here soon.

A big thanks to Xeni and the talented Boing Boing TV crew for inviting us on and running a great show! And thanks to those of you who came on the chat rooms to talk to us – actually a lot of terrific questions and comments we weren’t able to address.

Music in Video Games, a conversation with Peter Kirn and Matt Ganucheau [Boing Boing, with lots of download options for YouTube, MP4, iTunes, etc.]

Dreaming of a future in which music, instead of just being rendered audio files, arrives in fully generative, interactive form? Albums might “listen” to the world around you, and listeners could record their own alternate versions of music and share with others.

RjDj, the generative mobile music platform for Apple devices, realizes that future right now, instead of at some nebulous time in the future. In addition to the iPhone, you can make use of a second-generation iPod to use it. (You’ll need a headset with a mic; I have one by Griffin I’m testing.) And the RjDj folks have a whole bevy of significant updates to share:

• Free downloads (limited time): All three RjDj releases are available now for free. That includes the RjDj app itself (from which you can now grab and share releases), as well as RjDj Album (with a selection of generative/interactive/reactive releases) and the new RjDj shake.
• Download “scenes”: From the beginning, we knew that RjDj was imagined as a platform for other people to release interactive music. Now you can download scenes for free or fee. (Paid scenes currently redirect to the browser, but with iPhone SDK 3.0, you’ll be able to buy right from the app.)
• Share recordings: Because RjDj-generated music is controlled by the user and often records from the environment, the music may sound different each time. You can now share recordings with others from the device and the new social site.
• RjDj.me community: The RjDj folks have built a little community where you can share your favorite scenes and upload recordings, and keep track of scenes coming out from other artists.

By the way, with all of these releases, I know there’s some confusion. RjDj is the main app — the platform from which you’ll be able to grab scenes in the future. RjDj Album is a collection of “player”-style RjDj scenes — ones we have seen released previously. RjDj Shake is a newer, accelerometer-powered set of scenes. It comes with the amusing admonition that you should “PLEASE TAKE CARE, DON’T HURT ANYONE AND DON’T SMASH YOUR DEVICE.” If you missed Shake before, it now also has the recording sharing features. But the main RjDj app is the big release going forward.

We’re also awaiting news on more “sprints,” community-driven development efforts for making new scenes. And if you’re a fan of this sort of thing, RjDj isn’t the only game in town. RjDj itself is powered on Pure Data, the open source patching cousin of Max/MSP. Pd is making its way to other mobile devices; I even have it working on the BUG Labs gadget. At the same time, I’m curious to see if the RjDj gang can succeed in building a platform for lots of people doing this sort of work.

Regardless of how you look at it, more mobile generative / responsive music is most definitely in your future. Stay tuned.

Elsewhere:
Free trip into the rainbow vein: reality-enhancing iPhone app RjDj free with social update [Boing Boing Offworld]

RjDj.me community site
RjDj 0.7 out now [RjDj News]

RjDj [iTunes]
RjDj Album [iTunes]
RjDj Shake [iTunes]

Be sure to see our previous interview with the creators:
Exclusive RjDj Interview: Interactive Music Listening, Everywhere You Go

Also, expect more iPod/iPhone news and hands-on’s soon — I’m way behind, but let’s assume that means the best stuff will rise to the top.

It’s something we take for granted: listen to a track, and it starts at the beginning and goes to the end in a fixed length of time. Wonderful things can be done with music that way, and it’s the traditional model of composition and recording. But the equally old, if not older, tradition of improvisation suggests that music doesn’t always have to be linear. It can be specific to a place, a time, a mood.

Now that the technologies that power music creation can fit on a standard mobile device, listeners could have music that’s as pliable when they listen through headphones as it is in a studio when it’s created. Music could respond to the environment you’re in, and sound different each time you plug in your earbuds. That presents new challenges for the people making the music, but it could be an entirely new medium.

The team behind RjDj, a reactive and interactive music platform for mobile devices, don’t just want to wait around for this to happen. They’ve got it up and running right now, in a just-released application for iPhone. I spoke via Skype to the team in Vienna as a crowd of enthusiastic programmers and volunteers hacked away in a massive patching and music-making fest they call a “sprint.” More sprints are planned around the world, and the entire project is being built with the open-source visual patching environment for multimedia, Pd (Pure Data), cousin to Max/MSP.

If you’re ready to geek out with Pd, in fact, you can have at the patches yourself. But even if you’re just an interested musician, there’s plenty to watch here. It’s about more than just the software (Pd) or device (iPhone) – indeed, this app alone is likely to extend to other devices. What it’s really about is a new approach to how to listen to music, how to develop musical tools, and how communities own and share that work.

And, oh, by the way, team members have been behind everything from the port of Pd to Linux to the launch of Last.fm – the latter sold to CBS as one of the hottest musical properties on the Web, and a personal fave among the CDM team. So don’t doubt for a second that this group can drive some serious change.

If you watch just one video, check out the one above – especially about halfway in, as it starts to get juicy. Even for someone who’s been doing this for a while, watching a tiny device respond to the environment is magical.

Gunter Geiger is a technologist and advocate of free software. He puts his code where his mouth is: he ported the multimedia tools Pd and GEM to Linux a decade ago, helping launch the free community around them. Now he’s harnessing Pd again – but it’s not just about the software, he says.

Gunter: It’s not about if it’s Pd or not. The idea is to be able to create music in a different way. Instead of doing a fixed track, you do something interactive. These kinds of programs have been around for ages, but it really didn’t catch up on the music market.

The important thing is to get momentum behind it — not just one guy doing this thing. [And] it’s not only having people to create things, but [expanding] the audience, which is very small. What we really want to create is some momentum, and a scene. We hope that we get artists who make new [work].

You start to create different forms of music. Some of them are more like classical interactive things. Others are using the sound input a lot. It’s really a very open world, and the good thing about using Pd in there is that basically you can do everything. It’s really so open that we don’t know what’s coming out of it. We’re just trying to improve the things, and all the people working here are constantly changing their scenes and making them better.

I asked specifically about whether they were working to standardize these interactive structures, but Gunter emphasized they’re mainly keeping it open. And that’s important to note here – the actual “scenes” are completely open-ended, limited only by what you can do with the target hardware and the objects in Pd approved for the project.

You have a sort of chicken and egg problem. It’s really hard to make a structure before you know what these things look like.

What he could promise was growth – and on more devices than just the iPhone.

Now it’s the iPhone. In a year, I hope … more. There are sprints happening everywhere.

Michael Breidenbruecker initiated the project, now joined by a team of musical and technological thinkers and coders, with a select group of backers with experience in new Web projects for music. As one of the original co-founders of Last.fm, Michael is familiar with what a platform can do for music listening. He’s committed not only to the free, open source model for the project, but to transforming the way people think about music making – even those who aren’t musicians themselves.

Michael: I think we are all just starting at this, in a way. The scenes that we have right now have a [deep] effect. If you’re producing music, maybe you remember the first time you played with an echo or with a delay. At least for me, I spent ages pushing the button and going "poo, poo." For many people on the street, or what I experienced at Burning Man [with the RjDj], people were really going crazy because it was the first time they had this interactive or reactive experience of music. Music was not just something fixed or something they could consume, but something they could influence.

Ever since Burning Man, I’ve known we have a reason to be on the planet, to do what we’re doing.

When you write about this or talk about this, it’s really hard for people to understand what it is. As soon as you put headphones on them, they actually get it.

Michael says that to make that connection with listeners, they first have to connect with artists – which means their challenge is not only evangelizing interactive and reactive music, but on the tool side, making Pd’s power more accessible.

Michael: The big task now for us … the couple of sprints we’ve had, and the people we have involved already, is just blowing my mind. And that’s something that we really actively want to push. In the next couple of months, we’ll have to do a lot of work on the composing interface. Pd is a bit abstract for people who are used to other production software. So that’s our job in the end.

There are a lot of people standing behind Pd, but in the art scene it’s totally … inadequate. If RjDj can bring the whole idea of Pd and interactive music closer to the market, that would be really great.

We are trying to keep it as free as possible. It makes a lot of sense to use and reuse things. All the stuff that’s done should be provided to the community. We have it all on a public SVN [Subversion, a free, standard server tool for tracking changes to code and collaborating on projects]. All we can say to the artists is, if you don’t want to share it, don’t put it up there now.

Artists selling RjDj scenes could be very possible in the future – and wouldn’t necessarily conflict with providing open-source patches for those savvy enough to run Pd. But so far, Michael says the project is driven by imagining a new shift in music more than a new business model. And, interestingly, the ideas behind RjDj predate the now wildly-successful Last.fm, which was acquired last year by CBS.

I had this idea for a project ages ago. I started to work on this thing in 99. In 2000/2001, I started up Last.fm. When I saw what was happening on the iPhone, I said maybe it’s time to start [this concept] up again. I tried to get a bit of structure, all of our investors.

Michael: To be honest with you, including the investors we haven’t yet said, this is our business model, not at all. We just know we’re working on something new which we think has potential for the future. We’d [be happy to] manage to get the idea of reactive music booted, in two, three, four years even, to see a shift in the music market. So people who are now listening to MP3 songs could also be listening to reactive music, and something that’s customizable, highly dynamic, and personal. We would certainly try to be the driving force in that development, that market. Right now, all we can do is try to make the product as good as possible, that the person from the street would be able to listen to it and enjoy it, and artists would enjoy doing scenes.

I can tell you how the idea was born. It was actually one of these stupid things. In the 90s, people started to wear earplugs to raves because they were so loud. They had to protect their ears. Then I saw people who actually had microphones on their ears, and I thought, wow, that’s crazy. They have a microphone and a headphone, so what they hear is filtered. I found out that’s not what it was; it was a binaural microphone. I thought it was like sound glasses. I thought that was great. Eyeglasses for your ears.

Changing the medium, Michael notes, does transform what music can be – for musicians, as well. They have hooked up RjDj to a P.A. at parties, taking care to avoid feedback since RjDj scenes often make use of the microphone as an input. Even networking is potentially on the table, for collaborative scenes, though no development has taken place yet. (Pd supports networking, so that’s definitely something that could happen, with control data beamed between different devices running RjDj.) In the meantime, RjDj poses problems you might not even have imagined.

There is another interesting topic which we haven’t solved yet. You have the RjDj scene, and your sound experience is in the boundaries of that scene, but what you’re actually hearing is totally individual. That’s something that you can record on the rjdj. What do the artists — if a listener makes a recording of his scene which is very private, it’s his voice, his environment, what about that? Who’s the owner of that?

It’s scalable uniqueness — the RjDj scene, you can copy it a trillion times, it’s still the same, it’s a copy, but the individual experience listening to it. and tha’ts something traditional music is fighting. You have a digital copy of a recorded track. The musical industry wanted that scalable; that’s why they made that digital format, the digital CD. So they had this tremendous scalability, but then they started to realize that the uniqueness [is lost]. That was one reason why we did Last.fm at that time.

[Then] people started to realize they make music with objects. An instrument, it’s an object. But with digital music, music in a way became totally objectless. Look at the iPhone – in the end, it’s so miniaturized. RjDj is really bringing it back to the object. You know how this glass sounds [if you strike it], but with RjDj it sounds different. People begin to experience objects in a different way.

RjDj received its first official release today on the iTunes App Store. Software is available for free, or as an “album” for US$2.99.

Where to go:

RjDj Site / About / Blog

How to Create Scenes (And incidentally, you can work on scenes with a laptop even if you don’t own an iPhone. Testing on the device is, of course, very nice – fellow iPod touch users, I’m working on finding out how that mic solution is coming for us!)

A new generation of mobile devices is changing the way we hear music. Now we can say that, and not just be speculating are talking theory. The apps are here. Brian Eno had released a generative music album as computer software in the 90s. But this week, that idea hit a bigger audience when we saw his app Bloom, created with Peter Shilvers, running on any iPod touch or iPhone. Tap your ‘Pod, and you can add your own patterns, then let them “evolve.” (No more putting Music for Airports on repeat!)

RjDj goes one step further. Instead of just evolving and allowing inputs, it makes use of the iPhone microphone and sensors to respond to your environment. Not only does the music change, but it changes because of where you are and what you’re doing. The creators describe the effect:

The world around you will sound different or suddenly become part of a song. Some scenes sound best in certain situations like walking through the city, being alone, or making music with your friends. You can also record your mind twisting hearing sensations and listen to them later just like a normal music title.

RjDj affects the perception of your reality. It is the soundtrack to your life.

Don’t believe them? Well, iPhone users, now you can try it for yourself, on the cheap. The “single” version is free, with one scene in it. An “album” with multiple scenes is US$2.99.

This also reflects a new model for how to make, own, and price music. The scenes that drive RjDj are written in Pd (Pure Data), an open source environment for modular multimedia patching and cousin to the proprietary Max/MSP. The artists are generally releasing the code itself as open-source. That creates a really different ownership model. The music actually is software, in a more direct sense than even conventional computer music. The software itself is then priced approximately like a music track would be online, except that once you buy it, you make it your own, and it’ll sound different to you than it will to anyone else.

Each “scene” is an interactive musical creation. There’s even embedded album art for each scene, and information on the artist, so the creators can use this as a promotional tool.

Will we be covering this development a whole lot more? Yes. Yes, we will. In the meantime, go play.

RjDj single

RjDj album

Unfortunately, the app is not compatible with the iPod touch, because it lacks a microphone. There are third-party mics, but none seems to be compatible with the new 2.x firmware. That means I’m personally out of luck. Anyone got a first-generation iPhone they want to sell me, seriously? Or even better, anyone got any news about hacking those (unsupported) mics so they do work with the 2.x firmware? Support is promised, but no breakthrough yet, it seems.