Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

  • tom h

    ahhhhh, stockwell skate park – brings back good memories of scuffed knees & red stripe :)  

  • David

    Going by the reviews there are apparently in-app purchases akin to subscription required? That would be a harsh change in tune from the Inception app and put the price point well above a latte…? Can anybody elaborate?

  • http://www.edisonsdemo.tumblr.com edison

    annnnnnndroooooiiiiiiidddddd!!!!!!!

  • Leslie

    Siri tells me that this app is crap… ;)

  • Bob Bell

    I’ll look forward to this. I downloaded the Inception app straight after watching the film and will never forget going to the toilet whilst plugged in. Mad piss! I walked round the house making noises for a good while after too, which was nice.

  • kconnor9000

    Anyone that wants an Android version of music app X, Y or Z, go to
    http://code.google.com/p/android/issues/detail?id

    and click on the star, left hand side top, just beside " issue 3434".

    Basically, Android audio latency is SOOO bad that everyone who tries developing audio games or anything requiring real-time audio processing ends up hitting this fundamental bug/issue/whatever. The audio latency in Android is simply awful. Anything we can do to improve the visibility of this bug will help. I hope.

    • peterkirn

      Okay, you do know that entire bug report is a joke, right? The same reporter goes down a list of Linux audio APIs in common use and claims none of them is any good.

      There are only two issues that matter: if an individual developer is able to get performance they need on Android, and whether there's a business market for them. Starring a page filled with complaining developers saying things that are technically inaccurate is about as good as taking your head and banging it against a wall.

      I'll assume that since Dimensions didn't ship for Android, they didn't see that the business and technical equation made sense, and that's it.

      Google is aware of complaints from us developers about what we'd like to see improved, and at this point, those complaints are more specific and more technically informed than issue 3434.

      In fact, go to ANY project, anywhere, and try filing a long litany of specific design characteristics you'd like to see as a "bug" and see what kind of response you get.

  • Jamsire Ernoir

    huh?

  • DBM

    Damn kids these days . We used to just eat mushrooms and go camping for the weekend .lol seriously though do my pupils have to be as dilated as the FAS baby in the ad ? 

  • Grumpy

    Interesting but do we really need another app to distract people from what is going on around them? They already have phones, messaging and mp3 player (and thats not counting day dreaming). I am losing count of the number of people I see nearly having a road accident because they are playing with some hand held device when they should be paying attention.

  • James Levine

    I like her ears, but the ad would make more sense in terms of its overhanded delivery if they grew larger while she talked to the camera and her eyes bulged out a bit more.

  • http://www.jonbro.tk JonBro

    re: rj_readsf

    While it would obviously be better if this was opensourced, I was able to get at least the compressed audio loading working by using this code:&nbsp ;https://github.com/dichodaemon/flatland/blob/master/src/flatland/SoundBuffer.cpp

    I don't have it in a pd external yet, I was using it for another application, but if anyone is really itching for this, that is a nice place to start.

  • Toddbert

    The Quantum Cells accumulate waaaaayyy too slowly, and frankly, I'm disappointed in what would be a really groovy idea. I'm upset that I am asked to pay additional dough after forking over for the game itself. 

  • Toddbert

    And I'm not against In App Purchases, I bought some of the RJDJ stuff– but those things are not *needed* to enjoy it. The Quantum Cells are needed. 

  • http://rjdj.me Joe

    We're still tweaking the in-game ecomonics, please bear with us.

  • http://rjdj.me Joe

    Pun intended? ;)