In the thrilling conclusion of our chat with Boing Boing’s Xeni Jardin, Matt Ganucheau and I explore deep thoughts about the roles of interactivity and adaptivity in music and game design — then attempt to dance in giants Katamari Damacy hats. (Note the use of the word attempt — those things were more than a bit tricky to move in. Hilarity ensues.)

Prior to leading a dance dance Revolución, we talk a bit about the ways in which game design relate to gesture in musical interface and how musical scores could become non-linear. The gesture issue really goes well beyond games to the fundamental question of how to relate to music physically — and, in a way, awkwardly-dancing musicians may be a fitting metaphor. Or parable. Or something or other.

We do it all for you.

I really did find this a fascinating way to promote discussion, so if you’ve got suggestions for future broadcasts, I’d love to hear them. You can even think of new silly things for us / guests to do.

See also, related:
Troels Folmann on the boiled waterphone-style instrument, sound design inspiration
Troels on “micro-scoring” adaptive music

For part the first:
GDC: Music, Video Games, and Interactivity – Chat with Boing Boing Video

Matt Ganucheau and I got to sit down with Xeni Jardin of Boing Boing Video during the Game Developer Conference to discuss some of the potential for interactive music in games. Matt is a composer, sound designer, and educator, talking about how he’s encouraging his own students to think about adaptive music in new ways, combining Max/MSP and a Space Invaders clone built in the Unity Game Engine. (See our story from earlier this week.)

I talk a little about my sense that new tools could expand the range of possibilities in game music. Right now, the two major game engines are the AudioKinetic Wwise and Firelight fmod engines, each of which do have potential of their own – and continue adding features for more interactive sound scores. Each got some significant, flashy new features announced at GDC. But I was especially impressed by the use of Pure Data (Pd) in a custom implementation inside the game Spore. That allowed the compositional team to produce a truly generative musical score (led by legendary composer Brian Eno, with EA’s Kent Jolly and composer Aaron McLeran). I hope we see more of that in the future. Starting of students doing it themselves (with Max in this case) is not a bad way to start.

Boing Boing has more video of us they’ll be posting soon – including the embarrassing but diverting footage of us dancing around in Katamari costumes.

I’m new to this speaking live thing, but hope you enjoy. My favorite part was getting questions going. Livecasting is something we’ll try here soon.

A big thanks to Xeni and the talented Boing Boing TV crew for inviting us on and running a great show! And thanks to those of you who came on the chat rooms to talk to us – actually a lot of terrific questions and comments we weren’t able to address.

Music in Video Games, a conversation with Peter Kirn and Matt Ganucheau [Boing Boing, with lots of download options for YouTube, MP4, iTunes, etc.]

Award-winning composer Troels Folmann has made a name as a video game composer on the likes of the Tomb Raider series, as well as espousing new ideas about adaptive music for games like his “micro-scoring” methodology. But speaking to a roomful of composers and sound designers at the recent Game Developer Conference, he turned to the topic of reinvention. Even having perfected signature sounds that keep him in demand on jobs like blockbuster feature trailer soundtracks, Troels challenged attendees to get out of their usual habits and comfort zones.

And that means torturing some instruments. No, really torturing them: breaking sticks, destroying drums, warping instruments, and boiling waterphones (putting the whole instrument on a stove).

Human beings, of course, shouldn’t be tortured – to get the best sound of them, you want to get them drunk. (I want the Drunken Eastern European Choir sample library, Troels!)

Speaking excitedly in run-on sentences that clipped one another – a bit like sample in and out points were set wrong – Troels revealed some of his latest sampling explorations and sonic secrets. It was, truly, one of the best talks I saw at GDC – and unquestionably the highest idea and inspiration – to – time ratio, even if you weren’t into sound. Here are some of the gems from that conversation, along with some of the lists of bizarrely-combined sampled instruments in recent compositions.

I was looking over my notes and wondering if I should polish them. But then, I realized that I had transcribed all the things Troels said that interested me. If I put them all in a jar, I could take any one idea out on a day when my musical reserves were dry and be inspired. So I’ll share them with you in exactly that form.

The Right Wrong

Pipe organ, kalimba, baby toys, didgeridoo, conga, claps, IKEA stopwatch, church bell, vocals, ambience

One of the things I’m playing with is trying to do the right thing wrong — I call it the right wrong.

Some of these instruments [I sample] suffered through [the sampling process]. When you sample, you have to take it one step further. When it gets into the computer, it dies a bit. I don’t know what it is, there’s a translation issue. You have to push it further.

tonehammer_lakeside_organ_demo_4_dressed.mp3

Naked Ear

Kalimba, hang-drum, IKEA flower vase, Coke Bottle, public domain vocals – girl’s choir

We have certain ways we get stuck as composers — certain harmonic progressions and so forth. What I’m trying to do is more of a naked ear. I disregard any kind of theory. If it sounds right, it is right.

It’s an awesome practice, because it allows you to step out of theory.

This is a $19 kalimba. I don’t buy the most expensive instrument — I get 90% out of this instrument. And I can torture it through sampling. IKEA is the best music store; I don’t know if you know that.

Sometimes we get super caught in [the idea that ] it needs to be pristine, it needs to be high quality — it doesn’t matter. You get it in the mix, you can totally make something wonderful out of it.

I never have anything 24-bit …. It doesn’t really matter.

tonehammer_kalimba_demo_1.mp3

No Fear

Propane drum, flower vase, Coke bottle, kalimba, monkey balls, harmonica, vocals

[On eBay], I found this wonderful drum. I have a hang drum, this super-expensive crazy drum. This one was way better, and it’s like $300.

tonehammer_propanium_demo_1_dressed.mp3

Twist and Tweak

Didgeridoo, soda tabs, water cooler ensemble, hang drum

[On working with a Dr. Pepper soda.] You can … tap it to become percussion, you can also talk into it, sing into it … I multisampled [the taps] into an entire instrument.

tonehammer_didge_demo_1_dressed.mp3

It Doesn’t Matter

Things don’t matter so much. I was playing a 7-string guitar, and it wasn’t nasty enough. I took all the strings and drop tuned them to the same note … so it didn’t make a sound any more. I got this nasty sound to it. I’m starting more and more to let go of these conventions …how it should be.

I took a 5 string bass and again I couldn’t get it nasty enough — I’m not a great musician by any means. Put it down on the table, let the surgery begin. I put towels down to mute the sound. I played it with drumsticks, and got this tight sound that I was looking for.

Especially in the low frequencies of instruments, you get these … amazing, fat sounds. There’s so much you can do.

Sampling a Restroom

One of the best songs — I went to a restroom. I always use the handicapped restroom because there’s more space and you can be alone. I hate American restrooms – European restrooms are closed, you can’t see in to see what people are doing.

[On the result -- multi-sampling the metal bar next to the toilet in a handicapped restroom.] You expand your palette when you do that. There are so many sounds out there.

Boiled [and timestretched) Waterphone

There's so much you can do in terms of torture to get more out of it. Of course you can strum it, you can play it sort of percussively. But then you can boil it.

It was totally ruined in the end. But at least someone has boiled a waterphone.

We recorded it at different temperatures. It started spinning, as well, as you got to higher temperatures.

[In a separate experiment, timestretching:] As you know, the waterphone is impossible to control tonally. [I tried] timestretching a single note — [Native Instruments’ sampler] Kontakt has a harmonizer — putting some other notes on top of it to make a more strange, otherworldly sound to it.

Hybrid [Stacked] Orchestras

Unfortunately game composers are asked to do epic scores all the time. The main elements in it — it’s really about stacking. It needs several different libraries; you can’t stack the same library or it starts phasing. I like to stack until it starts phasing. You can also stack until it starts clipping.

There is no less — there’s only more.

I have synths for the bases, I have drones that line underneath the basses. Arpeggiators are almost mandatory for strings, so when you have stacatto notes — which is also stacked, at least two or three libraries — you also have arpeggiators under that.

It’s the art of adding, epic music.

tbf_epic_orchestral_demo_2009.mp3

The Future of Music

i think the future of music is partly all of us exploring more textures. We all want to do epic music and trailers …. and everyone is sounding a lot alike now. Especially in games; I never hear things that sound all that unique. We have to find ways to differentiate ourselves.

I’m a super commercial composer … I force myself to step out of that.

There are many many ways that we can stand apart. The best thing ever is the Zoom [H4 portable digital] recorder. I use it for everything, for the handicapped recording. There are sounds all over. You can break the convention, break the theory.

Successfully Sampling Choirs

The sampling is incredibly demoralizing. So you have to actually have them play a melody. If you get a performance that is not emotional, it totally dies.

We got an entire Eastern European orchestra drunk. It was a huge help. …They were half drunk, so they could still play.

Successfully Sampling Drums

Percussion is its own science. It’s important when you do recording sessions to dent the drums. If you don’t dent the drum …it won’t work. A mistake a lot of people make is …they only use one stick. Always use two sticks. The sound may flange .. it doesn’t matter. And those sticks need to break, if you want “triple-X” percussion.

Favorite Tools

Timefreezer is just incredible — you have to sculpt it in realtime, don’t just make a drone. Put it in multisamplers, map to velocity and really sculpt that tone. Put them in a sampler and assign it to a mod wheel — anything you have to do to get more control.

LA Scoring Strings is coming out — it’s the first library that’s really nailed legato. [with legato for different tempi] …solo instruments, divisi, full section.

The Wizoo W2 reverb plug-in…[now distributed through M-Audio / part of the Advanced Instruments Research group at Digidesign]

Compositional Process

Daily Exercises:

1. Watch YouTube

2. Chat and forums

3. Listen

4. Network

5. Talent = time = fun

I listen more than I compose these days. I listen two or three hours a day consciously. For me the process of listening is as important as composing.

Troels also listed some of his own inspirations, which included YouTube videos seen on this site:

Video Mashed Kutiman Funk: What if All of YouTube Played a Song?

Depressing Project of the Day: Stock Market, Set to Music with Microsoft Songsmith

What’s interesting about this is that he took these not simply as worktime distractions but inspiration for his own work – to try to analyze the thought process behind the videos and do something similar in his own work.

Here’s an example of his own: what’s the sound of one hand clapping? Well, here’s one hand clapping, made into an entire composition:

One sound composition

For more on Troels’ own sample house:

tonehammer

And everything on Troels himself:

http://www.troelsfolmann.com/

Previously, right here on CDM:

Weekend Inspiration: Coke Bottle as Tribal Percussion, and the Future of Adaptive Music

CDM Interview: Tomb Raider: Legend Composer Troels Brun Folmann on Adaptive “Micro-Scoring”

There’s plenty to process here, so I hope we’ll talk to Troels again soon.

A real highlight for me at the Game Developer Conference was getting to hear Satoru Iwata deliver the keynote. Aside from being CEO of Nintendo as they have launched their most successful console ever, Iwata-san has left a sizable development legacy as a veteran of HAL Laboratory (Balloon Fight, Kirby). In the game community, I think the reception to his keynote was mixed – mostly, it introduced long-overdue storage solutions for Wiiware titles, along with some relatively minor game titles. But as a person interested in design and development – and what innovative interfaces could do for music and not just games – I found the rare insight into Nintendo’s development process inspiring.

The surprise: despite their enormous resources, Nintendo is moving to ever-smaller development teams. And they’re taking dance classes to work on their musical rhythm.

Any developer with limited resources is familiar with what Iwata described as the “development death spiral”: financial pressure means rushed titles with poorer quality, resulting in fewer sales, resulting in greater financial pressure. Oddly, Iwata didn’t quite explain how do navigate out of the death spiral, explicitly. “Once you enter the death spiral, it is difficult to escape,” Iwata acknowledged. But the implication of his presentation was that you could do more with less, by focusing on process – not necessarily adding resources, but focusing on humans and fun. (The analog for music, perhaps, would be as much “expressivity” as fun.)

To illustrate, Iwata spoke mainly of Nintendo’s chief designer, Shigeru Miyamoto, creator of … um, nearly everything, in fact. Now, some of Miyamoto’s habits have been widely published, like his tendency to turn hobbies (gardening and puppies) into games. But to hear a normally-secretive Japanese company talking frankly about its process is something special.

Miyamoto’s Way is what Iwata called an “Upward Spiral.”

He observes people everywhere having fun. He thinks about how the core of that fun might come into games. But even as one project starts, he is still observing other people having fun – more ideas are born. Most developers prepare a thick design document to explain their intension to their teams. Mr. Miyamoto almost never writes one.

His first goal is always the same – a [prototype,] very limited and very clear. The amount of time being spent on the game’s appearance is zero.

Mr Miyamoto always has multiple projects in this stage at the same time.

What I find most important is how in each phase of overall development, he can clearly distinguish which details must be perfectly finished in that phase, and separate them from the parts that can be tentatively prepared.

You can see just how primitive some of these prototypes are in the example below from Punch Out. I think this is actually an important issue, as many beginning developers of games and audiovisual works don’t get primitive when doing early drafts, thus making it harder to make changes later.

Miyamoto is also notorious for randomly kidnapping employees for playtesting – playtesting without focus groups or statistics collection, but more qualitative evaluations of how people like a creation. Again, this isn’t unheard of in the industry, but it seems not to happen enough. And Miyamoto looks very fetching in his Cowboy / Outlaw getup.

Now, prototyping is nothing unique to Nintendo. But remarkably, Miyamoto’s prototype phase can last “more than two years.” And while no other developer has Miyamoto working for them, I expect that this is unusual:

I make it a point not to ask how [the project]’s doing. I believe this could make the team cut corners, or settle for less than their desired outcome.

This is not very good for my mental health. This is because of Mr. Miyamoto’s tendency to … upend the tea table.

“Upending the tea table” – also known as the “Miyamoto Test” – is a signature Miyamoto move by which the designer scraps a development process in mid-stream in order to make corrections. Again, this happens in the game industry, though perhaps not as often as it should – and certainly, no one has the leeway Miyamoto does.

Mr. Miyamoto is never an … angry man. He resets the dishes he had scattered, explaining just how they should be arranged on the tray.

I know many developers and critics are increasingly becoming frustrated with the dogma of fun, believing it forces the industry into a narrow range of expression. But, then, I enjoy depressing movies. Defined as enjoyment, Nintendo’s philosophy of fun is more a kind of commitment to its users. As Iwata puts it:

We create entertainment, and entertainment is meant to be enjoyed. If it can’t be enjoyed, it’s not the consumer’s fault – the fault belongs to us.

(At this moment in the presentation, in fact, Iwata bent forward slightly and halted, as if to consider the shame of such a potential situation.)

The musical connection to all of this is the rhythm game, “Rhythm Heaven.” We were lucky enough to get a copy for DS as we left the presentation; more on how it works soon. The game has already had a life as a Japanese-only Game Boy Advance title, but is now a worldwide release on DS. Several revelations were interesting to me in this presentation:

• Nintendo is turning to increasingly-smaller teams – as few as five on the GBA game and three on the DS. That says a lot about the way the videogame titan views effective development, and should give hope to penny-pinching indie developers and publishers, as well as us musical / visual experimenters toying with developing new interfaces.
• Rhythmic theory: The impetus for the game was designer/developer Tsunku’s new “rhythmic theories,” and ideas about how to teach and play with rhythm.
• Dance instruction: To help developers learn better rhythm themselves, Nintendo turned not to music lessons but dance movements – Tsunku bet that movement would help hone the programmers’ rhythmic skills. 

As Iwata explained:

How can a good rhythm game be created if the developers’ themselves don’t have much rhythm?

The quickest way to learn rhythm, [Tsunku] believes, is to dance. So the developers danced. Maybe they’re like winners of the Japanese ‘Dancing with the Stars.’

This was the first time as a game producer that I had to approve a budget for dance lessons.

For those of you wondering what the future of platforms for gaming or music are, Iwata had other juicy stats, as well. In 2008, female usage of the DS was up sharply to 47%. That brings hope for less male dominance of music technology. And anyone betting the iPhone would obliterate the DS as a gaming platform ought to think again. The DSi – the latest DS model with downloadable titles and a built-in camera – set a new advance-order record on Amazon for game systems. Some 90% of WiiWare titles are independent, so that makes me imagine that we could see creative new music and visual creations on both WiiWare and the DSi download service soon – a nice change from the current situation, which requires you to hack your system just to get real music apps. It’s nowhere near as open as the iPhone, though, so installed base aside, I think the iPhone / iPod touch remains a friendlier development platform.

Iwata closed with a nice sentiment for all of us:

Remember, in the Great Depression American inventors invented the jet engine, television, and even the chocolate chip cookie. As a developer, I believe anything is possible. The future of video games is in your hands.

I’m always a fan of “off-the-fovea” thinking, which was part of why I went to GDC. Hearing game developers tackle these problems I think has a lot of lessons for development of other creative projects – and I certainly believe a lot of these lessons are applicable to audiovisual makers, even if you don’t intend to release an iPhone – erm, DS — music game. Prototyping, testing and observation, small teams, using movement to make music and rhythm more powerful – all of these have great lessons not only relative to the game industry’s norms but for everyone else, too. I’m curious to hear what you think. But, if you’ll excuse me, I’m going to take a DS break.