Brainpipe is a psychedellic journey down the neural pathways, a long, strange trip into the minds of an unusual band of independent game designers. And while some games demand muscular graphics cards or brilliant flat panels, this is one that requires playing with headphones. The immersive sense of the descent down this brain’s pathway is entirely dependent on its sound. While even big development houses often license sound engines, the band of hard-core designers at Digital Eel also rolled their own interactive audio code to make the sounds fully seamless.

Designers and developers Iikka Keränen (the primary coder) and Rich Carlson spoke to me about their work. (They make reference to artist Bill “Phosphorous” Sears, as well.) In the process, they have a lot to say about the design process, about ambient sound design and composition, that goes well beyond just the gaming world. This isn’t just about gaming: it’s truly about digital music.

Digital Eel has won three excellence in audio awards over the past six years from the Independent Games Festival, including, most recently, a nomination for the psychedellic hit “Brainpipe” at the Game Developer Conference this spring. Incredibly, though, says Digital Eel’s Brainpipe, in that time no one has interviewed them about the sound in their games. Independent of the interview, Rich concede to me the challenge of getting people to focus on sound:

People are focused on graphics –and gameplay– and, you know, sound always gets the short shrift, even at game companies.  Sound and music are always the smallest slice of the development budget pie.

But not so at Digital Eel.  Sound and music are integral and integrated with design from the first moment we have something happening on the screen.  We feel it must be, and not just sfx but music, especially music which so often sounds like something….like dressing, something painted on, like makeup or apartment paint to help cover up the picture holes on the walls.

Brainpipe Game Page (with Mac/Windows download links – demos available so if you hate this, you’ll find out!)

Brainpipe on Steam (Windows only)

At a glance:

Engine: Custom

Favorite inspiration: demoscene, The Dig, Star Control II, Stockhausen, Varese, Morton Subotnick, Ussachevsky

Special acheivements: hiding loop points, creating a seamless acoustic descent, tapping into your subconscious

Peter: Let’s talk about the game mechanic. Some of it feels familiar – this descent through a cylindrical pipe – but there’s something quirky and unique about your take on it. How did you settle on the interaction mechanic?

Iikka: This was quite literally the first thing I programmed for Brainpipe. We were trying to come up with a new "short" game after putting another larger project on the back burner because we didn’t have enough free time to work on it. Within a few hours I had the basic control scheme and the moving pipe running on the screen. This is similar to how some of our other short games (Plasmaworm, Dr. Blob’s Organism) got started; the first prototype is something you can play with. After that there were tweaks of course, but the feel stayed much the same.

Rich: Everybody likes the "wormhole effect" you see in space shows and movies, and we do, too, so we wanted to do something like that.  Iikka got the pipe happening and we began to play with it as a prototype. Originally, we just wanted the player to fly down the pipe having a kind of zen experience as the speed slowly increased, and that’s all.  Not much of game there, though.

We were talking about music right away and how the sound, the intensity of the patterns and colors on the pipe walls, and the speed of traveling through the pipe should all work together. [We wanted] a kind of triple whammy to suck the player in deeper and deeper — a strong, cumulative effect.

We did add obstacles and specials, things to scoop up, and plenty of things to avoid that look pretty but are lethal.  But the blend of music, color and pattern complexity, and speed remained as we’d originally intended — this began very early on in the game’s development.

Making sure each obstacle has a sustained sound so you can hear it coming in the distance in front of you and then hear it pass by and recede with Doppler shift certainly adds to the audio illusion.
I think the kicker is the way the intensity ramps in the game.  It’s sort of like a rising sawtooth waveform-shaped thing.  During each level, the intensity, the speed increases, Then, between each level, the intensity drops to give you a breather before the next level begins.  Each time the intensity drops, it is still at a higher intensity level than during the previous level break, and all of this ramps upward.

It kind of coaxes you along.  You might not realize that you’re actually moving faster and faster each level for a few levels.  It’s a good training system.  Eventually you’ll get it as the game approaches its highest intensity levels and speeds.  Anyway, I still think it’s really cool.

The sensation of synesthesia is something a handful of game designers have tried to achieve. What are some of the games that have inspired you? Are there games you feel have reached that fusion of sound and visuals?

Iikka:  My personal influence is the "demoscene" that I was a part of when I was younger; it’s a subculture of programmers and artists using computers to create non-interactive but real time audio-visual experiences.

Rich:  For me, LucasArts’ adventure game, The Dig, with its seamless looping of various Wagner themes and so on. The music would morph as scenes changed.  It was an amazing piece of work. 

The music from Star Control II innovated with music and visuals, and it directly inspired the music for Strange Adventures in Infinite Space and Weird Worlds: Return to Infinite Space.  The idea that each alien race should have their own theme music came from there (though this kind of thing is less unusual now than it was when SC2 was originally released), as did the idea to attach separate and distinctly different music to each thing, category of thing, item, window, pop up announcement –every action in the game and every flick of the interface … like a toddler’s "busy box" of sound.

Back to Brainpipe, other areas of music outside of games inspired us as well.  Aleatoric, musique concrete, avant garde — stuff Bill just naturally creates and stuff I’ve always loved since I was a kid. [I checked] out the LP’s at the library by Stockhausen, Varese, Morton Subotnick, Ussachevsky, all these wonderful pre-synthsizer electronic sound and found sound composers. And the records were awesome because they were always in pristine condition — relatively few others ever checked them out.

[It’s] mindblowing stuff to listen to while you’re listening to Steppenwolf on your Japanese transistor radio and playing John Phillips Sousa in your Junior High band.  Liberating.  Of course this stuff scares some people and some people react to it negatively –all strongly– but if you listen to it, put together by someone who pays attention to details while intuitively knowing what they’re doing, you can hear the music in the sighing of pond reeds, or on the heavy end, the music within industrial clamor and the beauty in the beast.

That seemed perfect for Brainpipe which really demanded a whole different musical approach and completely different kinds of music produced in ways that are not normal — not typical at all.

I love that you talk about sound being integral with the design process. Even for a musician, though, thinking in more than one medium can be a challenge. How do you approach this in terms of design; how do you make it part of the process in practice?

Rich:  When we make a game, music and sound are in right away.  From the first couple of hours, the basic prototype is on the screen, so they began to shape the sonic style of the game immediately.

Because sound and music are growing up at the same time as the art and programming is, all these elements influence each other pretty equally, so you don’t get music and sound that sound "separate" or tacked-on.  You get sound you can’t turn off, and you don’t want to, because it’s actually part of the game.

Sounds can also influence and inspire and change things.  You might be after a certain sound effect, but then you stumble across something else that’s much cooler, so the animation of a visual effect is changed to match the sound.

What was the compositional process like on this game? The sound design / sound score clearly fuse – with these recurrent "whooshing" sounds as an added layer. How were these assembled?

Rich:  Basically what you’re hearing is a series of loops.  Most of them are 16-second loops.

I knew right away that "music" with beats wasn’t the way to go.  The music had to create a soundscape, something that supports a mindscape, really — pun intended — rather than making you want to tap your foot.  It had to smoothly transition just as the "art" on the pipe wall and the speed of traveling through the pipe smoothly transition in the game.

I also knew that the music had to have a kind of primal power and evoke a sense of mystery about what is supposed to be going on and what is being revealed.  Bill was very much into this too.

At the same time, we wanted it to reflect the random thoughts floating through and bouncing around inside your brain.  One of the best ways to accomplish this was to leave conventional music behind, which is what Bill and I ended up doing.

It was important that the loops be seamless.  If you’re working with beats and grooves, that’s a very easy thing to do — it starts on one and ends on four.  You simply loop that, attaching the end to the beginning and it sounds fine because, for the most part, that’s how a bass/drums/guitar combo plays.

On top of that I knew we needed loops that didn’t sound like loops.  Loops gamers wouldn’t notice were loops, with no obvious "breaks" where the end of a loop would be obviously attached to the beginning, or the beginning of another.  The loops had to have no beginning or end!

The sources for the loops were varied. There are very successful loops in the game that are extremely simple, comprised of only two or three tracks or elements.  [But] some of them are monsters mapping out to 32 tracks or more.  Again, the idea was to create loops that don’t sound like loops with a range that would reach an orchestral level of density.

Finally, the soundtrack loops had to blend seamlessly with each other while increasing in intensity. One way to do this, of course, is to cross fade them, but that wasn’t going to be enough. The intensity and the components of each loop needed to be gauged so a dramatic and appropriate intensity ramp was reached.  I think we came very close to nailing it, but I want to keep experimenting with this.  We can go farther now, having only scratched the surface.

Can you talk about some of the found sounds that are collaged into the result? (I’m hearing the TARDIS materializing…)

Rich:  I don’t want to spoil the magic trick but, like most who do music and audio, I collect sounds from all kinds of sources.  I’ve been doing it for a long time, so if you listen carefully you’ll hear things from old TV shows, records, radio shows, interviews, sound effects records, and God knows what else folded in there. 

The soundtrack is meant to represent the background music of your own brain so references to "real life" should resonate –especially, we hoped, on an unconscious level.

I don’t want to list all the sources — some of them are in the credits –  because what is going on is also a sound trivia game.  It’s the Mystery Science Theater of game music, but the gamer is provoked to make guesses and speculate.

You noted that part of why you embarked on building your own sound engine was that OpenAL [a standard, open, cross-platform API for spatial audio] wound up being inadequate. What were some of the obstacles you encountered? Have you found other independent game creators dealing with the same issues?

Iikka:  We had to switch away from OpenAL because it made repeating clicking sounds on common integrated audio hardware. The lack of features is not terribly important, as you can always just use OpenAL as the output channel for your own sound mixing system. My sound code would be perfectly happy living on top of OpenAL if it was universally supported.

Sound is a rather underappreciated and underdeveloped area in games. To many game developers, especially smaller ones, it’s enough that it "makes a sound when something happens". The focus of development is very much on the visuals.

It seems like game audio lacks a functioning standard. OpenAL is promising but lacks some of the maturity of, say, the OpenGL API which game visuals can use. What’s your take on the landscape? Is there hope that a new standard or engine could address these issues, and result perhaps in better sound and music design in games?

Iikka:  I think it’s possible that OpenAL will mature to a point that it will work reliably on all common hardware some day, and at least form a standard foundation for people to base their sound engines on so they won’t need to learn a new API for each operating system they support.

As for workflow and design, I don’t view these as dependent on what is under the hood; they are the result of the mindset among the team members. Certainly one could imagine development tools that allow an audio artist to work more directly with the game, but a good first step is just making sure that everybody involved in the project is involved in designing the work flow.

Can you describe your custom sound engine? What functionality did you find you wanted to build into it? What would you want to put in the next iteration?

Iikka: I call it "eelmix". It’s a modular sound system in which sound sources, filters and mixers are arranged in a tree structure, like a scene graph of sorts. It’s analogous to musical instruments wired together, eventually converging to a master output to speakers.

The main goal was the modularity, the system makes it easy to make a "box" that takes a sound output (from any source), mangles it in some interesting way, and then feeds it to where it was originally going, without modifying either the source or the destination. We haven’t really used the full capabilities of this yet but the modular system is also useful for things like separating UI ("2d") sounds from the game ("3d") sounds that makes balancing them easier. And it eliminates the need to conform to some preset number of "channels".

There are other lesser goals, like eliminating clipping by using 32-bit precision internally and simulating a non-linear response curve when rendering the final output. This is a very simple and useful bit of code that really improves sound quality when there’s tons of sounds being played.
Going into the future, what’s left is mostly just filling in some blanks like including basic prefab filters, making sure that every kind of sound source can use every kind of sound sample, that sort of stuff.

Indie games it seems, like larger games, have struggled a bit on sound and music – perhaps because of the lack of better tools. But what are some smaller, experimental, or independent titles you feel have done good things with their soundtracks?

Rich:  A couple of indie games have had sound and music that was really special, I thought.  I loved the music from Saints & Sinners Bowling a few years ago.  Just great stuff that nestled right in there so you didn’t want to turn off, and that’s the true test.

Musaic Box which I encountered at this year’s IGF [Independent Games Festival] uses conventional music ingeniously.  You solve musical puzzles by ear, assembling melodies to reach certain goals. I think music is actually more integral to this game than it is in Brainpipe because it’s directly a part of gameplay.  You couldn’t play Musaic Box without it.

Flow stood out to me for doing something really quite gentle and tasteful and, well, flowy — even the soundtrack lived up to the game’s name.  That’s important I think, and that gets back to some of the things I’ve already said here.

Sounds to Hear

To head deeper into the strange sonic world the Digital Eels inhabit, Rich sent along some additional sonic resources:

Weird Worlds stuff
http://www.digital-eel.com/files/vault/the_single.mp3
http://www.digital-eel.com/files/vault/mfbtpv2_320Kbps.mp3

Misc. stuff from different games old & new:
http://www.digital-eel.com/files/voidprobe.mp3
http://www.digital-eel.com/files/vault/blok.mp3
http://www.digital-eel.com/files/vault/drblob.mp3
http://www.digital-eel.com/files/vault/forest.mp3
http://www.digital-eel.com/files/vault/plasmaworm.mp3
http://www.digital-eel.com/files/vault/haircut.mp3

An interview (Omaha Sternberg interviewing Bill, "Phosphorous") with a couple snippets of Brainpipe music in it:
http://www.digital-eel.com/files/omahaphos_intermix1_160.mp3

Enjoy. The game is really unusual, so I look forward to hearing what CDM readers think of the experience. And if you have other games (or other interactive experiences) about which you’d like to learn more or get an interview, let us know.

GPS Beatmap from Jesse Stiles on Vimeo.

“Locative art,” the idea that somehow location will feed into music and visuals, has eluded culture. We have the technology, in the form of sophisticated databases of location information and highly accurate, publicly-available GPS satellites. But it’s one of those solutions in search of a problem, and begs the question, why?

That is, until you unleash a nearly 6-liter V8 Ford LTD Crown Victoria on the legendary Bonneville Salt Flats, and your driving gets translated to music. Now it makes sense. And sweeping through the salty dust in one of America’s greatest action-car-chase cars of all time, manipulating music on a Max/MSP software patch, all becomes right with the world. (That’s how it is in my head, anyway.)

The planet is your control surface.

Such is the project sent by co-creator Jesse Stiles, who worked with Rich Pell (and editor/documentarian Olivia Robinson) under the name Face Removal Services to perform this vehicular musical production. (Thank, as well, The Center for Land Use Interpretation / GPS Expo 2006. PS – I think we now know what to do with all those clunkers Americans are turning in for Cash for Clunkers.)

Now, this covers only X and Y axis. I think we need to add the Z-axis, for base jumpers. (I had a dream last night in which I was hang gliding from the rim of the Grand Canyon to the Colorado River below, a reminder that the Earth – and computer interfaces – do not have to be flat.)

Bored with mouse pushing and knob twiddling? The d-touch tangible sequencer / drum machine makes a cheap interface (with free downloadable software) for assembling sequences. Make some (attractive) blocks, set up a webcam, and plug into your computer. I took a first look at this tool last month, and noted its use in sequencing walnuts. (Yes, the ones that fall from trees.) Since then, the developers have been hard at work on updates. Enrico writes:

We just released the d-touch sequencer, a new, more advanced, audio application. In the sequencer you can record your own samples in real time.

We also have few updates for the drum machine, which should solve the activation problems we were having at the beginning.

Go grab the markers and the software, and you have your own webcam-based drum machine.

Should you decide to go beyond their free instrument, the underlying system is really quite sophisticated. Part of what makes it beautiful is that you can design your own markers rather than settling for predefined patterns, as with most similar marker-tracking systems. There’s even a tool for correcting problems in your design. The freely-downloadable analysis software is written in C/C++, but if you use another environment (like Max or Processing or Reaktor), you can simply pipe data to your tool of choice.

The drum machine and sequencer are available now, so go download them and let you know how you fare! System requirements: a printer, a webcam, and a PC/Mac. Enjoy!

http://www.d-touch.org/

For some hands-on impressions of working with these things, the excellent PC Music Guru has a great description of the experience. Or, if you read the language, there’s a Japanese-language hands-on blog entry.

Float away with Mujik… from The Amazing Rolo on Vimeo.

Musical technology is often designed to be “hard” in character. Interfaces are cold and technological-looking, futuristic like spaceships, or made to replicate antique gear to make guitarists feel nostalgic. Musical interfaces consciously avoid anything “childish” – calling something a “toy” being the worst possible insult – and they’re certainly never whimsical.

That’s why the real news about Mujik isn’t that it’s a new iPhone app, or that, after a few weeks of teasers, you can download it today on the iTunes store. (The app is free for a limited time.) The news is that it’s a musical interface with lightning bugs.

Mujik teaser… from The Amazing Rolo on Vimeo.

The creators, led by Yann Seznec, clearly had this goal in mind:

Do you want an iPhone app to make beats? And produce like totally awesome tracks with a zillion channels and plugins and automation and 808 simulators and stuff yeah? Oh. Well, this is not for you. Mujik is more than a music app – get away from fake sliders and buttons and enter the world of Mujik, with charming graphics, lovely music, and a wonderfully tactile interface. It’s not rocket science, it’s just fun.

Mujik is brought to you by Lucky Frame, a small team of awesome people based in Edinburgh, Scotland. It’s lovely here, you should come visit.

You can’t save yet, but I rather like that. And while you can’t yet add your own sounds (which they’re also working on), I actually can imagine this would be something you’d share with your friends who are intimidating by the arcane, unreadable tracker you have installed on your hacked Game Boy.

On the team: Yann Seznec is also known as The Amazing Rolo, who we’ve seen previously on CDM making a free Wii looping machine, and who recently did a three-part series just for CDM on the Maker Faire. (one, two, three)

Thanks to Chris Hahn on Twitter for the tip!

Sonoridades Líquidas [Liquid Sounds] from Rui Penha on Vimeo.

Wonderful work in sculpting sound into beautiful handmade electronic-acoustical instruments is pouring out of Portugal. Now we get to see more of that work – and if you are in Portugal, you can share in person at a new event.

The Handmade Music project and community has found its second home – Porto, Portugal, at the massive Casa da Música! The first event will be held Tuesday, July 21. (Full announcement, English/Português).

De hardware a software feito em casa até circuit bending, kits personalizados ou instrumentos acústicos originais, todos estão convidados a aparecer na Casa da Música pelas 21h30 para montagem de instrumentos. Pelas 22h abrimos o evento ao público geral – a entrada é livre e recomenda-se -, ocupando a Digitópia e a zona do bar do Foyer Sul. Contamos convosco!

To celebrate the new party, organizer Rui Penha sends over a survey of some of the instruments and sculpture that are being homegrown in Porto. And yes, even first-time-creator high school students can make fantastic work (see after the jump). At top:

An interactive installation made for Casa da Música – Porto, Portugal – with João Ricardo de Barros Oliveira and Luís Girão. Shown here is the algorithmic composition and synthesis patch I made in Max/MSP and the Arduino-based sensor system, co-developed with Luís Girão.

Lula · robotic Hang player from Rui Penha on Vimeo.

This robotic Hang player is called Lula (Squid).
It was, designed, built and its algorithm was created with a group of teenagers participating in a four-day workshop called OrCA – Orquestra de Computadores e Autómatos (OrCA – Orchestra of Computers and Robots), at Casa da Música – Porto.
It was made with picaxe, servos and ultrasonic sensors.

Correntes Sonoras from Serv. Educativo Casa da Música on Vimeo.

João Ricardo constrói instrumentos-esculturas a partir de materiais das mais diversas origens: pedaços de metal, bocados de madeira corroídos pelo tempo, objectos usados e danificados, provenientes de contentores do lixo, da rua, da natureza. As esculturas têm sons complexos, poéticos, muito diferentes do habitual e são um desafio à imaginação, aos sentidos e às convenções daquilo que é um instrumento ou daquilo que é música.

A ExposiSom [Correntes Sonoras] explora diversos espaços da Casa da Música e é concebida como um percurso de descoberta da arquitectura e da relação do som com o espaço que ele habita.

Sonoridades Líquidas from Serv. Educativo Casa da Música on Vimeo.

A ExposiSom [Sonoridades Líquidas] estabelece uma ponte entre a escultura sonora, a música e a arte digital e permite a descoberta do edifício da Casa da Música através de um percurso onde o público interage com o conteúdo sonoro, transformando assim a Casa num mega-instrumento. A ExposiSom [Sonoridades Líquidas] usa um conjunto de cerca de 50 objectos idênticos [esculturas sonoras] que servem de «interface» com um computador. Explora-se a ideia de elicitar sons ou algoritmos que geram som, a partir de um estímulo muito simples: a vibração causada por uma gota de água que cai sobre uma pele de um instrumento de percussão.

We hope to send out some resources for folks working to plan Handmade Music in other places, too. Here’s the running list of locales we know have expressed interest – and yes, I’ll be following up with you folks later today!

• Machester, UK
• (city TBD) North Carolina, USA
• Toronto, Canada
• Argentina
• Chicago, USA
• Berlin, Germany
• San Francisco, USA
• Joinville, Brazil
• Greece
• Los Angeles, USA
• Washington, DC, USA
• Austin, TX, USA
• Detroit, USA
• Boulder, CO, USA

Not on the list and interested in organizing an event? Fill out the form on Google Docs.

Stay tuned for coverage of the upcoming NYC and Portugal events here.

Matt Ganucheau and I got to sit down with Xeni Jardin of Boing Boing Video during the Game Developer Conference to discuss some of the potential for interactive music in games. Matt is a composer, sound designer, and educator, talking about how he’s encouraging his own students to think about adaptive music in new ways, combining Max/MSP and a Space Invaders clone built in the Unity Game Engine. (See our story from earlier this week.)

I talk a little about my sense that new tools could expand the range of possibilities in game music. Right now, the two major game engines are the AudioKinetic Wwise and Firelight fmod engines, each of which do have potential of their own – and continue adding features for more interactive sound scores. Each got some significant, flashy new features announced at GDC. But I was especially impressed by the use of Pure Data (Pd) in a custom implementation inside the game Spore. That allowed the compositional team to produce a truly generative musical score (led by legendary composer Brian Eno, with EA’s Kent Jolly and composer Aaron McLeran). I hope we see more of that in the future. Starting of students doing it themselves (with Max in this case) is not a bad way to start.

Boing Boing has more video of us they’ll be posting soon – including the embarrassing but diverting footage of us dancing around in Katamari costumes.

I’m new to this speaking live thing, but hope you enjoy. My favorite part was getting questions going. Livecasting is something we’ll try here soon.

A big thanks to Xeni and the talented Boing Boing TV crew for inviting us on and running a great show! And thanks to those of you who came on the chat rooms to talk to us – actually a lot of terrific questions and comments we weren’t able to address.

Music in Video Games, a conversation with Peter Kirn and Matt Ganucheau [Boing Boing, with lots of download options for YouTube, MP4, iTunes, etc.]

Dreaming of a future in which music, instead of just being rendered audio files, arrives in fully generative, interactive form? Albums might “listen” to the world around you, and listeners could record their own alternate versions of music and share with others.

RjDj, the generative mobile music platform for Apple devices, realizes that future right now, instead of at some nebulous time in the future. In addition to the iPhone, you can make use of a second-generation iPod to use it. (You’ll need a headset with a mic; I have one by Griffin I’m testing.) And the RjDj folks have a whole bevy of significant updates to share:

• Free downloads (limited time): All three RjDj releases are available now for free. That includes the RjDj app itself (from which you can now grab and share releases), as well as RjDj Album (with a selection of generative/interactive/reactive releases) and the new RjDj shake.
• Download “scenes”: From the beginning, we knew that RjDj was imagined as a platform for other people to release interactive music. Now you can download scenes for free or fee. (Paid scenes currently redirect to the browser, but with iPhone SDK 3.0, you’ll be able to buy right from the app.)
• Share recordings: Because RjDj-generated music is controlled by the user and often records from the environment, the music may sound different each time. You can now share recordings with others from the device and the new social site.
• RjDj.me community: The RjDj folks have built a little community where you can share your favorite scenes and upload recordings, and keep track of scenes coming out from other artists.

By the way, with all of these releases, I know there’s some confusion. RjDj is the main app — the platform from which you’ll be able to grab scenes in the future. RjDj Album is a collection of “player”-style RjDj scenes — ones we have seen released previously. RjDj Shake is a newer, accelerometer-powered set of scenes. It comes with the amusing admonition that you should “PLEASE TAKE CARE, DON’T HURT ANYONE AND DON’T SMASH YOUR DEVICE.” If you missed Shake before, it now also has the recording sharing features. But the main RjDj app is the big release going forward.

We’re also awaiting news on more “sprints,” community-driven development efforts for making new scenes. And if you’re a fan of this sort of thing, RjDj isn’t the only game in town. RjDj itself is powered on Pure Data, the open source patching cousin of Max/MSP. Pd is making its way to other mobile devices; I even have it working on the BUG Labs gadget. At the same time, I’m curious to see if the RjDj gang can succeed in building a platform for lots of people doing this sort of work.

Regardless of how you look at it, more mobile generative / responsive music is most definitely in your future. Stay tuned.

Elsewhere:
Free trip into the rainbow vein: reality-enhancing iPhone app RjDj free with social update [Boing Boing Offworld]

RjDj.me community site
RjDj 0.7 out now [RjDj News]

RjDj [iTunes]
RjDj Album [iTunes]
RjDj Shake [iTunes]

Be sure to see our previous interview with the creators:
Exclusive RjDj Interview: Interactive Music Listening, Everywhere You Go

Also, expect more iPod/iPhone news and hands-on’s soon — I’m way behind, but let’s assume that means the best stuff will rise to the top.

Game Audio: Selected Student Works from Matt Ganucheau on Vimeo.

In the early days of game sound, musical soundtracks were all largely adaptive and interactive, fused with the sound effects of the game and the logic of gameplay. Scores were less Alfred Newman or John Williams, more Spike Jones. Today, game music has the potential to reinvent composition itself, to help us reimagine what makes a musical score as on-screen user action drives musical ideas. But with a few, notable exceptions, most modern titles have opted for big, Hollywood-style soundtracks – and the linear composition that goes with them, as though someone just took a film score CD and hit play.

It’s one thing to talk about that in theory. Better yet: give it a shot yourself. So why not teach game music as its own discipline?

Matt Ganucheau, a composer, sound designer, and interactive developer/artist, is teaching just that, working with students at Expression College in Emeryville, California. The accelerated course works with the elegant Unity game engine and a clone of the legendary Space Invaders arcade game, adding music built in Max/MSP. If Max seems an unlikely choice, its open source cousin Pure Data (Pd) is actually integrated with the game engine for Electronic Arts’ Spore, with music by Brian Eno working with EA’s Kent Jolly and contributor Aaron McLeran. So, this could be the wave of the future. The first problem: figuring out how to actually compose.

The results are astonishing, given that the students were just learning Max and had extremely limited amounts of time. I asked Matt to write up for CDM how the coursework evolved; he shares his process and what he learned as a teacher. We’re also working on open sourcing the coursework content and the patches, which we’ll soon provide both for Pd and Max/MSP. I’m doing some work on the game side so that you can play with game mechanics in Processing. Stay tuned for more on that.

We spoke a bit about this process – and interactive music in general – with Xeni Jardin and Boing Boing in their Game Developer Conference livecast a week ago Friday. Edited video of that coming soon.

Here’s Matt on the coursework itself:

When faced with the challenge of updating our Game Audio course at Expression College, we wanted to create a course that reflected the increase of interest in adaptive and interactive audio in the current game industry. To do this successfully, we had to make sure our students had an understanding of how audio engines have evolved in the past eight years. Since our terms are only five weeks and our student body is comprised of non-programmers, this seemed like quite a daunting task. But having carefully fine-tuned the details, we feel we have a good recipe.

First, we begin by having the students build simple environments and place audio emitters inside the Unreal 2k environment. This shows them the restrictions of audio functionality in a proprietary engine. After a few labs with Unreal, the students are then introduced to the concepts of a middleware platform, using Audiokinetic’s WWise connected to the game Cube. Here, they are able to explore more interactive audio such as real-time control parameters and dynamic music changes. Finally, the students are introduced to Max/MSP. Lead through labs comprised of synthesis, sampling, basic programming concepts and sound design, we are able to arm the students will all of the information needed to create their own generative audio engine inside Max/MSP. By hacking away at a recreation of Space Invaders posted to the Unity3d forums (thank you, Eric Haines), we are able to pipe all of the real-time game data to Max/MSP via the UDP transport (with help from Bjerre).

Inside Max/MSP, the game data is received in our Unity2Max patch. With this initial infrastructure in place, the students are able to use the real-time events to remix the classic arcade game with their own audio engine. Piece by piece, we recreate the original audio engine through tasks such as creating the alternating pitched footsteps for the invaders, and a UFO spaceship noise with a flanger and a sine-wave, as well as mapping invader’s proximity to the music’s speed. For their final project, the students are allowed to use these tools to go in any stylistic direction they wish, as long as the music is adaptive.

We did not give students access to all of the game events because we didn’t want them to become overwhelmed with options. To our surprise, these restrictions created the opposite reaction. Students were frustrated by not having a message saying that the “UFO was destroyed”, so they hacked their own ways to find this out by deducing the change in points. In another example a student wanted the missile explosion to sound when the bunker was hit, so he placed a threshold on the missile flight time to be able to see if a bunker was hit. Hacks like these began to appear all over the students projects. This may seem like basic programming techniques to some, but to see this development come from a class of audio engineers is quite amazing.

Although this new course design has only been active for 4 months, we have seen a dramatic increase of interest from our students. Once a cultural standard like Space Invaders is deconstructed, the students become extremely excited to explore a new direction for the classic game. It still amazes me just how far students can go with only 3 weeks of Max/MSP instruction.

Unity Game Engine (recently updated to 2.5, and now both on Mac and Windows)

Cycling ‘74, Makers of Max/MSP

Expression College for Digital Arts

And the bits for this game, specifically:

Unity Invaders on the Unity Community Forum (the Space Invaders game used in the class)
Unity Invaders Site with downloadable, playable versions of the game
Discussion of UDP communication between Max and Unity, with the patch solution by Bjerre

Also, don’t miss the fantastic Pd-based book Designing Sound (well worth a read for Max users, as well). It’s an entire textbook built on the idea of doing interactive sound design in Pd, useful for games but other live and interactive sound, too – and while the emphasis is sound design rather than music per se, it remains a great reference on learning to patch and learning about audio synthesis.

Andy Farnell

I’m going to make this a minimalist post because I’ve said what I’ll say about Kodu, the one really cool part of Microsoft’s keynote yesterday, on Create Digital Motion. (Am I the only person who wishes Sparrow had just done the whole keynote?)

But have a look at the shot above. One of the complaints about generative and algorithmic music software (and music software in general) is that the interface has been so complex. Clearly, there are many other ways to design these interfaces, and in turn, to shape the way we use these to compose and perform music. Forget for a moment that games are “games,” and this this thing is “for kids,” and I think you’ll agree – there are lots of areas to explore, and lots of potential.

It doesn’t even require some futuristic music software. Imagine more complex rules in Ableton Live’s follow actions, made graphically.

Excuse me, I’m going to pick up some Tinker Toys to think about interactive design.

You Know, For Kids: Game Design, World Creation as Microsoft Research Previews Kodu [Create Digital Motion]

PS, I believe now more than ever that Music and Motion deserve separate sites, but have a look and I think you will find some overlap.

For one area in which forward-thinking digital music types are doing innovative work in game and interactive audio, look no further than the Interactive Audio SIG. They’re doing really interesting stuff in looking at how tools can support future interactive music. And if you are going to GDC, this is another one you’ll want to catch. It’s worth noting that the “interactive” in their title really is just that: this is about all forms of interactive music, not just games per se. Given what we’ve seen with generative music apps on desktops and mobiles alike over the past year, that really brings this point home. In fact, unconstrained by the harsh business realities of big-budget game development, individual composers and small teams experimenting with the future of music is really where I expect to see progress.

That doesn’t make the challenge any less for formats like IASIG’s iXMF interactive spec – if game developers are primarily interested in keeping their proprietary engines humming and focus on fairly non-interactive assets, iXMF may not really catch on. But then, I like a good challenge, and anyone who knows the game and interactive space knows that sometimes real innovation takes time.

Watch CDM for more coverage and even some tutorials on these topics all year long – think we’ve got some good stuff coming.

Brad Fuller of IASIG writes: “I didn’t see a mention of the IASIG on your blog. Maybe I missed it.”

Well, Brad, sometimes I just think things and forget to write about them, which admittedly makes it hard to read. So here’s what Brad sends along on the IASIG meeting:

Each year at GDC we have our annual IASIG Town Hall meeting where we gather to discuss progress in our working groups and also allow others to discuss what they would like the IASIG to address in an open mic session.

We also have an annual mixer where we share drinks and a raffle of great audio prizes from sponsors such as: Creative Labs, Garritan, Numark, MixMeister, O’Reilly Media and Wave Arts. IASIG members and those in the audio business are welcome (a business card, or other identification is recommended.)

Here’s a recap of last year’s party and Town hall meeting:

http://iasig.org/gdc/2008gdc.shtml

This year will be no different, a mixer and a town meeting will be held at GDC. Check your bag for a flyer that will indicate the time and location.

Hope to see you there!

More info: http://iasig.org/

The IASIG exists to allow developers of audio software, hardware, and content to freely exchange ideas about improving the performance of interactive applications by influencing hardware, software, and tool design. The IASIG provides resources in the form of standards creation and maintenance, research reports, and recommended practices. Anyone with a commercial interest in audio for interactive entertainment is encouraged to become a member <http://iasig.org/join.shtml> of the IASIG.

Thanks, Brad! I’m working on getting out there myself, so maybe see you there.