Wired has a great mini-documentary on the score for the videogame Infamous. It’s chock full of sound design ear candy, not only served by the chops of composer Amon Tobin but the team at Sony Music and Sony’s entertainment division, as well. Curiously, Jonathan Mayer, Music Manager at SCEA, says explicitly that he doesn’t want composers writing interactive music. He’d prefer to have them write a conventional score and then adapt it to the interactive engine. Now, of course, around these parts we like the idea of composers finding ways to write genuinely generative and interactive scores. But in this case, Mayer is acting as a kind of remix artist for the game realm, sampling Tobin’s compositions and reconceiving them in the game world. That kind of collaboration could be powerful.

Chuck Doug, SCEA music director, overstates things a bit by claiming this game has a unique aesthetic. The visuals are a burnt-out, post apocalyptic city – yeah, been there quite a few times. The music involves lots of ethnic percussion-y instruments and bowed metal and deep booming sounds. (Let me get this straight: we’ll hear a plucky stringy thing, then a bowedy metally thing, then there will be a big boom!) So, generally, not some radical new departure from game and motion soundtracks. But regardless of its novelty, I’d be an utter killjoy to complain: it sounds utterly gorgeous.

Previously:

I got to listen in on a lot of gems regarding sound design from composer Troels Folmann. He doesn’t just bow metal instruments – he boils them.

GDC: Boiling Waterphones and Other Sonic Inspirations from Composer Troels Folmann

And on the subject of getting composers to write interactively, Matt Ganucheau has been teaching that way:

Teaching Adaptive Music with Games: Unity + Max/MSP, Meet Space Invaders!

Game Audio: Selected Student Works from Matt Ganucheau on Vimeo.

In the early days of game sound, musical soundtracks were all largely adaptive and interactive, fused with the sound effects of the game and the logic of gameplay. Scores were less Alfred Newman or John Williams, more Spike Jones. Today, game music has the potential to reinvent composition itself, to help us reimagine what makes a musical score as on-screen user action drives musical ideas. But with a few, notable exceptions, most modern titles have opted for big, Hollywood-style soundtracks – and the linear composition that goes with them, as though someone just took a film score CD and hit play.

It’s one thing to talk about that in theory. Better yet: give it a shot yourself. So why not teach game music as its own discipline?

Matt Ganucheau, a composer, sound designer, and interactive developer/artist, is teaching just that, working with students at Expression College in Emeryville, California. The accelerated course works with the elegant Unity game engine and a clone of the legendary Space Invaders arcade game, adding music built in Max/MSP. If Max seems an unlikely choice, its open source cousin Pure Data (Pd) is actually integrated with the game engine for Electronic Arts’ Spore, with music by Brian Eno working with EA’s Kent Jolly and contributor Aaron McLeran. So, this could be the wave of the future. The first problem: figuring out how to actually compose.

The results are astonishing, given that the students were just learning Max and had extremely limited amounts of time. I asked Matt to write up for CDM how the coursework evolved; he shares his process and what he learned as a teacher. We’re also working on open sourcing the coursework content and the patches, which we’ll soon provide both for Pd and Max/MSP. I’m doing some work on the game side so that you can play with game mechanics in Processing. Stay tuned for more on that.

We spoke a bit about this process – and interactive music in general – with Xeni Jardin and Boing Boing in their Game Developer Conference livecast a week ago Friday. Edited video of that coming soon.

Here’s Matt on the coursework itself:

When faced with the challenge of updating our Game Audio course at Expression College, we wanted to create a course that reflected the increase of interest in adaptive and interactive audio in the current game industry. To do this successfully, we had to make sure our students had an understanding of how audio engines have evolved in the past eight years. Since our terms are only five weeks and our student body is comprised of non-programmers, this seemed like quite a daunting task. But having carefully fine-tuned the details, we feel we have a good recipe.

First, we begin by having the students build simple environments and place audio emitters inside the Unreal 2k environment. This shows them the restrictions of audio functionality in a proprietary engine. After a few labs with Unreal, the students are then introduced to the concepts of a middleware platform, using Audiokinetic’s WWise connected to the game Cube. Here, they are able to explore more interactive audio such as real-time control parameters and dynamic music changes. Finally, the students are introduced to Max/MSP. Lead through labs comprised of synthesis, sampling, basic programming concepts and sound design, we are able to arm the students will all of the information needed to create their own generative audio engine inside Max/MSP. By hacking away at a recreation of Space Invaders posted to the Unity3d forums (thank you, Eric Haines), we are able to pipe all of the real-time game data to Max/MSP via the UDP transport (with help from Bjerre).

Inside Max/MSP, the game data is received in our Unity2Max patch. With this initial infrastructure in place, the students are able to use the real-time events to remix the classic arcade game with their own audio engine. Piece by piece, we recreate the original audio engine through tasks such as creating the alternating pitched footsteps for the invaders, and a UFO spaceship noise with a flanger and a sine-wave, as well as mapping invader’s proximity to the music’s speed. For their final project, the students are allowed to use these tools to go in any stylistic direction they wish, as long as the music is adaptive.

We did not give students access to all of the game events because we didn’t want them to become overwhelmed with options. To our surprise, these restrictions created the opposite reaction. Students were frustrated by not having a message saying that the “UFO was destroyed”, so they hacked their own ways to find this out by deducing the change in points. In another example a student wanted the missile explosion to sound when the bunker was hit, so he placed a threshold on the missile flight time to be able to see if a bunker was hit. Hacks like these began to appear all over the students projects. This may seem like basic programming techniques to some, but to see this development come from a class of audio engineers is quite amazing.

Although this new course design has only been active for 4 months, we have seen a dramatic increase of interest from our students. Once a cultural standard like Space Invaders is deconstructed, the students become extremely excited to explore a new direction for the classic game. It still amazes me just how far students can go with only 3 weeks of Max/MSP instruction.

Unity Game Engine (recently updated to 2.5, and now both on Mac and Windows)

Cycling ‘74, Makers of Max/MSP

Expression College for Digital Arts

And the bits for this game, specifically:

Unity Invaders on the Unity Community Forum (the Space Invaders game used in the class)
Unity Invaders Site with downloadable, playable versions of the game
Discussion of UDP communication between Max and Unity, with the patch solution by Bjerre

Also, don’t miss the fantastic Pd-based book Designing Sound (well worth a read for Max users, as well). It’s an entire textbook built on the idea of doing interactive sound design in Pd, useful for games but other live and interactive sound, too – and while the emphasis is sound design rather than music per se, it remains a great reference on learning to patch and learning about audio synthesis.

Andy Farnell

For one area in which forward-thinking digital music types are doing innovative work in game and interactive audio, look no further than the Interactive Audio SIG. They’re doing really interesting stuff in looking at how tools can support future interactive music. And if you are going to GDC, this is another one you’ll want to catch. It’s worth noting that the “interactive” in their title really is just that: this is about all forms of interactive music, not just games per se. Given what we’ve seen with generative music apps on desktops and mobiles alike over the past year, that really brings this point home. In fact, unconstrained by the harsh business realities of big-budget game development, individual composers and small teams experimenting with the future of music is really where I expect to see progress.

That doesn’t make the challenge any less for formats like IASIG’s iXMF interactive spec – if game developers are primarily interested in keeping their proprietary engines humming and focus on fairly non-interactive assets, iXMF may not really catch on. But then, I like a good challenge, and anyone who knows the game and interactive space knows that sometimes real innovation takes time.

Watch CDM for more coverage and even some tutorials on these topics all year long – think we’ve got some good stuff coming.

Brad Fuller of IASIG writes: “I didn’t see a mention of the IASIG on your blog. Maybe I missed it.”

Well, Brad, sometimes I just think things and forget to write about them, which admittedly makes it hard to read. So here’s what Brad sends along on the IASIG meeting:

Each year at GDC we have our annual IASIG Town Hall meeting where we gather to discuss progress in our working groups and also allow others to discuss what they would like the IASIG to address in an open mic session.

We also have an annual mixer where we share drinks and a raffle of great audio prizes from sponsors such as: Creative Labs, Garritan, Numark, MixMeister, O’Reilly Media and Wave Arts. IASIG members and those in the audio business are welcome (a business card, or other identification is recommended.)

Here’s a recap of last year’s party and Town hall meeting:

http://iasig.org/gdc/2008gdc.shtml

This year will be no different, a mixer and a town meeting will be held at GDC. Check your bag for a flyer that will indicate the time and location.

Hope to see you there!

More info: http://iasig.org/

The IASIG exists to allow developers of audio software, hardware, and content to freely exchange ideas about improving the performance of interactive applications by influencing hardware, software, and tool design. The IASIG provides resources in the form of standards creation and maintenance, research reports, and recommended practices. Anyone with a commercial interest in audio for interactive entertainment is encouraged to become a member <http://iasig.org/join.shtml> of the IASIG.

Thanks, Brad! I’m working on getting out there myself, so maybe see you there.

Digging through recent entries on Troels’ blog is definitely a source of weekend inspiration. I’m fond of found samples, but I tend to record sound making things around the house up close with a mobile recorder for a more intimate sound. Troels drags them over to a concert hall and uses the natural reverb to turn a candle light holder and Coke bottle into something that sounds like massive, tribal percussion. To keep himself disciplined, he limited himself to objects in a random photo. Here’s what it sounds like:

To add to the ambience, he uses the Timefreezer plug-in ($99 for Mac, Windows, Mac Intel, the lot). As the name implies, it “freezes” samples of sound as an effect or instrument. I’ve done some similar things as DIY patches, but it sounds like they’ve done a nice job of implementation.

This approach to sampling percussion with natural reverb, and making an art of the samples, is part of why they pay Troels the big bucks. Be sure to hear his percussion demo for more of the sounds. Little wonder that he blogs the meditation on autism that’s been making the YouTube rounds: sampling sounds requires an almost extrasensory focus on the world around us that we spend most of our time shutting out.

So there you have some fiddling with household objects. What about this “future of adaptive music” business?

Our own W. Brent Latta covered this in some detail in a 2006 interview:

CDM Interview: Tomb Raider: Legend Composer Troels Brun Folmann on Adaptive “Micro-Scoring”

Troels follows up on that topic in a new interview with Game Soundtracks, as reproduced in total on his blog. He has this to say about “micro-scoring” — and notes the use of sliced samples, manipulated live:

So a part of my Ph.D. studies in game music related to developing new methodologies for advanced types of application of music in the game. One of my main focal points was – and still is – the development of something I call: “Micro-scoring”. Micro-scoring is essentially about breaking the score into a variety of small components that are assembled in real-time according to player action and/or interaction. The micro-scores are made in such a way that they adapt to player action or interaction. You have to imagine that there are thousands of things going on in the game environment — the idea behind micro-scoring is to support the major elements in the environment. An example can be a 3-second score for breaking pillars or falling stones, which is scored in the same key as the main ambient background score. We also have more detailed types of micro-scores which are based on slices samples like REX and other sliced sample formats. This allows us to fully adjust pitch- and timing based on player interaction with the game. An example of this is adjusting beat to footsteps and increasing tempo when she starts running. A good example of micro-scoring application relates to chopping up a score in multiple components. So essentially composing a score in 15 different steps and cutting each step up, so it can seemingly integrate into any of the other 15 steps. The system then blends the steps in real-time and you have a much more varied and versatile score – made from micro-scores. This allows you to adjust mood in music with using basic cross-fades, but actually have adaptive types of compositions. Needless to say it’s a fairly complicated effort.

Game scores had at one time been more interactive than they are now, because for at least a period around the early 1990s, PC games used MIDI scores instead of more-rigid audio. Scores are making progress back towards interactivity at those levels and beyond, aided by more powerful game systems that start to resemble the computers we use for live music production. Troels also speaks to where this future is going (and there have been some interesting developments since October 2006 when we spoke to him last):

It depends on whether studios are willing to commit to investing properly into game music. The commitment involves a variety of factors, including prioritizing audio in the production planning and a willingness to invest properly in the scoring. I doubt we will see a huge leap within the next ten years, but we will see more adaptive types of music based on principles similar to the micro-scoring methodology I described. We will also see some real-time DX/VST-based FX plugins like the integration of Waves plugins in Halo 3.

We will not see true adaptive music, since the next-next generation consoles won’t have the processing power to play a 50 GB orchestral sample library playing in real-time with 5 high-end convolution reverbs and an advanced AI that translate player action into music.

We will see more ties between motion picture, television and games – and most likely a larger degree of score usage between the media. But we also see a billion mediocre game scores and they will retain game music in a space it doesn’t need to be. Bleep, Bleep. Blob.

So, there you go — to any of you who are skeptical as far as quality, most game composers are, too! (They’re the ones, after all, most invested in seeing their field progress, while having to wrangle with mediocrity in their area, tight budgets, and tight deadlines.)

I am curious, though, about those 50 GB orchestral samples. Given that adaptive music in games could extend to all kinds of game play that doesn’t need massive orchestras in the background, let alone other venues for adaptive music beyond gaming (from live performance to installation), I’m more optimistic. This year, we’ll see Spore from EA powered by a sound engine built in Pure Data, and I expect other games from developers big to indie. The sound of 8-bit is making a comeback, as well, which will hardly tax game consoles.

With composers like Troels around, that may be a 20 MB sample of a Mexican Coke bottle that sounds better than the 50 GB orchestra, anyway. Troels, give us a ring if you decide to release your Things On My Coffee Table Sample Collection.