Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

Game lovers may lap up anything the title Portal touches as though it’s covered in powdered sugar, but resident Valve Software composer Mike Morasky deserves special mention. His music for Portal, and now Portal 2, is dead-on: chilly, atmospheric, dystopian, but also pulsing with energy and able to capture the gaming blockbuster’s strange combination of diabolical cerebral puzzles with wit. It’s all the more impressive, as Morasky has straight-up parodied musical styles in his whimsical Team Fortress or horror movie-cinematic Left 4 Dead scores.

Developer Valve quietly released 22 instrumental tracks from Portal 2 as “Soundtrack Volume 1: Songs to Test By,” free in 320 kpbs MP3 form. “Music to Code By” could be just as appropriate. Even if you ignore this post, know that this score will be racking up Last.fm playcount as it pipes into the headphones of nerd boys and girls.

GamesRadar published an interview with Morasky. Interestingly, while this is being released in soundtrack form, Portal 2 is in fact adaptive in the game. The system is in the foreground only in a few scenes, but there, multiple layers give a sense of progression. Any musicians who have been … erm … sucked into this game no doubt found these scenes highlights already, but here’s Morasky’s explanation to GamesRadar:

There are several cases where the music adds channels and complexity as you successfully solve portions of the puzzle, with each additional piece of music actually coming from the device that is participating in the activated game play mechanic. Obviously, this can heighten the sense of achievement as one completes the puzzle but also turns the mechanics of the puzzle into a sort of interactive music instrument that you can explore by selectively triggering the different channels of music with differing timings and configurations. Most of the interactive music is also positional so that as you move through the space you also change the mix and volume of the music you are hearing, which invites explorations of the space as well.

Morasky and Valve have been at the forefront of adaptive music in games – an area still left surprisingly unexplored – in particular in Left 4 Dead’s use of cinematic cues to heighten suspense and integrate with actual gameplay.

But that doesn’t make the soundtrack to me any less satisfying. With nods to spooky scifi and electronica convention alike, it nonetheless emerges with a distinctive voice – much like the game itself. In a world of cookie-cutter mainstream gaming, at least in the triple-A territory, the success of the title could be encouraging. Find it here:
http://www.thinkwithportals.com/music.php

Found via the superb gaming blog Rock, Paper, Shotgun; who also note two additional volumes are on the way. Follow CDM’s Twitter feed and we’ll let you know when those hit.

And if you’d like to see more covered than in the interview above, let us know.

In the thrilling conclusion of our chat with Boing Boing’s Xeni Jardin, Matt Ganucheau and I explore deep thoughts about the roles of interactivity and adaptivity in music and game design — then attempt to dance in giants Katamari Damacy hats. (Note the use of the word attempt — those things were more than a bit tricky to move in. Hilarity ensues.)

Prior to leading a dance dance Revolución, we talk a bit about the ways in which game design relate to gesture in musical interface and how musical scores could become non-linear. The gesture issue really goes well beyond games to the fundamental question of how to relate to music physically — and, in a way, awkwardly-dancing musicians may be a fitting metaphor. Or parable. Or something or other.

We do it all for you.

I really did find this a fascinating way to promote discussion, so if you’ve got suggestions for future broadcasts, I’d love to hear them. You can even think of new silly things for us / guests to do.

See also, related:
Troels Folmann on the boiled waterphone-style instrument, sound design inspiration
Troels on “micro-scoring” adaptive music

For part the first:
GDC: Music, Video Games, and Interactivity – Chat with Boing Boing Video

Matt Ganucheau and I got to sit down with Xeni Jardin of Boing Boing Video during the Game Developer Conference to discuss some of the potential for interactive music in games. Matt is a composer, sound designer, and educator, talking about how he’s encouraging his own students to think about adaptive music in new ways, combining Max/MSP and a Space Invaders clone built in the Unity Game Engine. (See our story from earlier this week.)

I talk a little about my sense that new tools could expand the range of possibilities in game music. Right now, the two major game engines are the AudioKinetic Wwise and Firelight fmod engines, each of which do have potential of their own – and continue adding features for more interactive sound scores. Each got some significant, flashy new features announced at GDC. But I was especially impressed by the use of Pure Data (Pd) in a custom implementation inside the game Spore. That allowed the compositional team to produce a truly generative musical score (led by legendary composer Brian Eno, with EA’s Kent Jolly and composer Aaron McLeran). I hope we see more of that in the future. Starting of students doing it themselves (with Max in this case) is not a bad way to start.

Boing Boing has more video of us they’ll be posting soon – including the embarrassing but diverting footage of us dancing around in Katamari costumes.

I’m new to this speaking live thing, but hope you enjoy. My favorite part was getting questions going. Livecasting is something we’ll try here soon.

A big thanks to Xeni and the talented Boing Boing TV crew for inviting us on and running a great show! And thanks to those of you who came on the chat rooms to talk to us – actually a lot of terrific questions and comments we weren’t able to address.

Music in Video Games, a conversation with Peter Kirn and Matt Ganucheau [Boing Boing, with lots of download options for YouTube, MP4, iTunes, etc.]

Game Audio: Selected Student Works from Matt Ganucheau on Vimeo.

In the early days of game sound, musical soundtracks were all largely adaptive and interactive, fused with the sound effects of the game and the logic of gameplay. Scores were less Alfred Newman or John Williams, more Spike Jones. Today, game music has the potential to reinvent composition itself, to help us reimagine what makes a musical score as on-screen user action drives musical ideas. But with a few, notable exceptions, most modern titles have opted for big, Hollywood-style soundtracks – and the linear composition that goes with them, as though someone just took a film score CD and hit play.

It’s one thing to talk about that in theory. Better yet: give it a shot yourself. So why not teach game music as its own discipline?

Matt Ganucheau, a composer, sound designer, and interactive developer/artist, is teaching just that, working with students at Expression College in Emeryville, California. The accelerated course works with the elegant Unity game engine and a clone of the legendary Space Invaders arcade game, adding music built in Max/MSP. If Max seems an unlikely choice, its open source cousin Pure Data (Pd) is actually integrated with the game engine for Electronic Arts’ Spore, with music by Brian Eno working with EA’s Kent Jolly and contributor Aaron McLeran. So, this could be the wave of the future. The first problem: figuring out how to actually compose.

The results are astonishing, given that the students were just learning Max and had extremely limited amounts of time. I asked Matt to write up for CDM how the coursework evolved; he shares his process and what he learned as a teacher. We’re also working on open sourcing the coursework content and the patches, which we’ll soon provide both for Pd and Max/MSP. I’m doing some work on the game side so that you can play with game mechanics in Processing. Stay tuned for more on that.

We spoke a bit about this process – and interactive music in general – with Xeni Jardin and Boing Boing in their Game Developer Conference livecast a week ago Friday. Edited video of that coming soon.

Here’s Matt on the coursework itself:

When faced with the challenge of updating our Game Audio course at Expression College, we wanted to create a course that reflected the increase of interest in adaptive and interactive audio in the current game industry. To do this successfully, we had to make sure our students had an understanding of how audio engines have evolved in the past eight years. Since our terms are only five weeks and our student body is comprised of non-programmers, this seemed like quite a daunting task. But having carefully fine-tuned the details, we feel we have a good recipe.

First, we begin by having the students build simple environments and place audio emitters inside the Unreal 2k environment. This shows them the restrictions of audio functionality in a proprietary engine. After a few labs with Unreal, the students are then introduced to the concepts of a middleware platform, using Audiokinetic’s WWise connected to the game Cube. Here, they are able to explore more interactive audio such as real-time control parameters and dynamic music changes. Finally, the students are introduced to Max/MSP. Lead through labs comprised of synthesis, sampling, basic programming concepts and sound design, we are able to arm the students will all of the information needed to create their own generative audio engine inside Max/MSP. By hacking away at a recreation of Space Invaders posted to the Unity3d forums (thank you, Eric Haines), we are able to pipe all of the real-time game data to Max/MSP via the UDP transport (with help from Bjerre).

Inside Max/MSP, the game data is received in our Unity2Max patch. With this initial infrastructure in place, the students are able to use the real-time events to remix the classic arcade game with their own audio engine. Piece by piece, we recreate the original audio engine through tasks such as creating the alternating pitched footsteps for the invaders, and a UFO spaceship noise with a flanger and a sine-wave, as well as mapping invader’s proximity to the music’s speed. For their final project, the students are allowed to use these tools to go in any stylistic direction they wish, as long as the music is adaptive.

We did not give students access to all of the game events because we didn’t want them to become overwhelmed with options. To our surprise, these restrictions created the opposite reaction. Students were frustrated by not having a message saying that the “UFO was destroyed”, so they hacked their own ways to find this out by deducing the change in points. In another example a student wanted the missile explosion to sound when the bunker was hit, so he placed a threshold on the missile flight time to be able to see if a bunker was hit. Hacks like these began to appear all over the students projects. This may seem like basic programming techniques to some, but to see this development come from a class of audio engineers is quite amazing.

Although this new course design has only been active for 4 months, we have seen a dramatic increase of interest from our students. Once a cultural standard like Space Invaders is deconstructed, the students become extremely excited to explore a new direction for the classic game. It still amazes me just how far students can go with only 3 weeks of Max/MSP instruction.

Unity Game Engine (recently updated to 2.5, and now both on Mac and Windows)

Cycling ’74, Makers of Max/MSP

Expression College for Digital Arts

And the bits for this game, specifically:

Unity Invaders on the Unity Community Forum (the Space Invaders game used in the class)
Unity Invaders Site with downloadable, playable versions of the game
Discussion of UDP communication between Max and Unity, with the patch solution by Bjerre

Also, don’t miss the fantastic Pd-based book Designing Sound (well worth a read for Max users, as well). It’s an entire textbook built on the idea of doing interactive sound design in Pd, useful for games but other live and interactive sound, too – and while the emphasis is sound design rather than music per se, it remains a great reference on learning to patch and learning about audio synthesis.

Andy Farnell