You might not expect a handheld game console, the gadget kids use to play Pokemon, to prove much worth as a musical instrument. But even in the age of readily-available computer plug-ins and iPhone apps, the DS holds its own. In the hands of two sets of artists, we find music that stands alone, independent of the gimmick of the device on which it was made. For these artists, the limitations of a fold-up touchscreen – entirely independent of doubling as a phone, or a computer, or a Facebook-browsing engine, or a powerful 64-bit DAW – apparently prove enticing. Beginning with Korg’s DS-10 cartridge, they use a stylus-operated software synth with its own unique character.

On some level, I almost hesitate to wax poetic about the fact that these were made with a Nintendo DS at all, because what these are, really, is love letters to synthesis.

And as it happens, both are available as free downloads from Bandcamp.

First up: AuxPulse is the duo of Rutger Muller and Michael Vultoo, based in Amsterdam and Kockengen, Netherlands, respectively. Late last year, they debuted their first album at Amsterdam’s prestgious Stedelijk Museum of modern art, playing a big set (two and a half hours) on small devices. Primarily employing the Nintendo DS, they nonetheless produce sounds that are rich and layered, sometimes even tending to the ambient exploration, not just the rawer chip-music sounds regularly associated with Nintendo handhelds.

Their music is trippy but danceable, unapologetically electronic, fully exploiting the DS-10′s idiosyncratic sonic character, one that’s slightly lower-fidelity than many soft synths (or even iPhone apps), without being “chippy” in the sense of retro devices. Dark textures collide with precise, clockwork rhythms, in sounds that sometimes tend to acid techno and sci fi game realms. (Lo-acid-fi, anyone?)

As you watch them live, you also see the value of the interface compositionally, both in terms of its pattern banks and its more conventional synth controls, all manipulated with the added precision of a stylus.

As they put it:

We aim to bring experimentation back to the dancefloor by expressing a psychedelic atmosphere through the use of a variety of rhythms and moods. Some of our inspirations are analogue synthesizers, acid, IDM, hardcore, gabber, ambient and oldschool electro.

Right now we mainly use the KORG DS-10 synthesizer for Nintendo DS to compose and improvise our music. When playing live we fuck with the synths as much as we can, trying to surprise ourselves with new sounds.

Our first album was recently released in Stedelijk Museum Amsterdam! Now we perform regularly, trying to open up some minds and move some feet.

The album, on Bandcamp:

And on SoundCloud:
Dream Stages (FREE ALBUM!) by AuxPulse

Bonus: an interview with them (in Dutch, naturally)

In a very different direction, Princeton, New Jersey-based DJ and producer Christian Montoya (love and tonic records) produces music on the DS-10 that’s drier and more exposed, as he programs intricate bass music on the unprocessed Nintendo cart. Christian works as a game designer by day, and channels some of the DS-10′s game music and so-called “chip music” heritage. The results, though, are a perfect marriage of game chip-waveform rawness, nude bass and synth and percussion sounds, and carefully-concocted grooves. For anyone concerned that game systems could hinder moving your butt out of the seat, this album is required listening. It’s utterly stripped-bare dance goodness – and it turns out the DS bass sounds fantastic.

Grab the record for free:

DS-10 users, got any tips for us on getting the most out of a Nintendo handheld and this KORG synth? Let us know.

Also, from comments but worth pointing out, Rutger directs us to good resources for getting the most out of DS-10:

If you’re interested in making DS-10 music you can check out http://www.ds10forum.com

I (Rutger, DS-10 Dominator, 1/2 of AuxPulse) run it with Harley (http://harleylikesmusic.com, superb DS-10 composer!) and we try to help out beginner’s and advanced users as much as we can.

Truly generative musical scores in games have been few and far between, and “music games” has traditionally meant arcade-style rhythm games in which you repeat phrases or whole songs as accurately as possible. Pugs Luv Beats breaks those molds. Part of a vanguard of new gaming creations that generate dynamic music on the fly, it marries grid-based sequencing and resource-gathering gaming, as music making and gameplay blur together. The interactively-produced music could itself become a new way of delivering a musical signature with sound packs.

And beneath it all lurks a free and open source library, libpd – the embeddable version of tried-and-true free graphical music environment Pure Data. (That library is now on GitHub, and vastly updated, by the way, and we’re expecting a book soon from the library’s principle author Peter Brinkmann.)

Oh, yeah, and don’t forget about some seriously addictive gameplay and adorable pugs. I’m suddenly not concerned about the 15 hours Europe-to-North-America travel I’m doing tomorrow.

Here’s what the gameplay looks like, since it’s much easier to see:

Pugs Luv Beats was just approved on the iTunes App Store for iPhone and iPad.

Co-creator Yann Seznec (The Amazing Rolo) is a terrific musician; I just caught up with him in Edinburgh and Berlin and watched him play a homebrewed pig gut instrument with Matthew Herbert for the performance piece “One Pig,” on tour at Berghain. Working with Pd allowed Yann to focus on those musical impulses and not just engineering, and to let him try things he otherwise would never have imagined on a mobile title. So I asked Yann to walk us through how the project was built. He responded with an exhaustively-detailed examination of the evolution of this title, right down to the Pd patches. (Click through for high-res versions.) If your New Year’s Resolution is doing something with patching, you might want to hang onto these answers. Here’s Yann:

The origins of Pugs Luv Beats date back about two years. After making [musical iPhone game] Mujik, Jon (Jonathan Brodsky, aka jonbro) and I were trying to think of other approaches to music mobile app design, and we started thinking more and more about games. Music games, as a whole, are an oddly passive and traditionalist experience – you play along with a premade track, and you are judged on your accuracy and flair (which is strangely reminiscent of music conservatory mindset…). Obviously there are exceptions (RjDj’s Dimensions, Elektroplankton, etc.), but there you go.

Particularly interesting to me was the idea that game mechanics are often very similar to compositional techniques. So for example, when Sonic runs at a normal speed he collects rings at one rate. However when he powers up and goes super fast, he collects rings at a much higher rate. This could be compared to introducing a melody and then speeding it up – and when there are two players, doing this with two melodies. Instant fugue!

We started looking at how we could make a music game where the music and the game elements were fully intertwined and augmented by each other. So Jon prototyped a space shooter drum machine. It was awesome.

To make a (very very very long and boring) story short, our idea and prototype landed us some funding from Channel 4 and Creative Scotland to work on games that focus on musical creativity and composition.

For various reasons, we decided to put aside the space shooter drum machine for a while, and start from scratch. After going through several full prototyping iterations we eventually settled on a core game mechanic that turned out to be in many ways similar to a Tenori-on [Yamaha grid instrument]/Boiingg-style [monome hardware patch] music generation system – in our final prototype, you controlled a series of little dots that moved around the screen, creating loops. This is super fun from a musical perspective because it’s easy and rewarding within a few seconds, and when you have several loops going it can gain some pretty serious rhythmic and melodic depth.

The key from there for us was turning this into a game. We had been using free Internet graphics packs up until then (we hadn’t hired our artist Sean yet) which featured a ladybug, so we had been referring to the main characters as ‘bugs’. During some discussion one of us accidentally said ‘pugs’, and the game idea was born. We constructed a story about pugs and their love for beets (like the vegetables) which create beats (ha!), and how their love turned into greed and got out of control, destroying their world. The game, therefore, is about helping the pugs rebuild their lost civilization by guiding them to create beats. You grow your galaxy by collecting beats, which you do most efficiently when you dress your pugs up in costumes. What’s not to like?

To get to the part that I imagine CDM readers are most interested in, the app development was done by Jon using openFrameworks, [lightweight language] Lua, our own game engine called Blud, and the audio is all done in Pure Data using libpd (through ofxPd). In hindsight we started using libpd really late in the game, just at the very end of the prototyping stage, which was rather silly. Our adoption of libpd basically made our dev cycle about a million times more efficient. My background is as a musician and sound designer, and I have very little coding knowledge. I do, however, have lots of knowledge of Max/MSP, so picking up Pure Data was pretty easy. This allowed Jon to completely pass off all the audio processing (not to mention aesthetic sound design choices) to me, saving him loads of time, giving me direct control over the sound, and letting me test and prototype different approaches to audio within an environment that I knew would be recreated in the game. Also, as Jon mentioned to me recently, by using PD we are able to take advantage of 20 years of audio DSP research and development. Pretty amazing.

How it all works:

The entire audio engine is contained within this patch. Pardon the messiness.

The simplest part of the patch is the “sounds” section, which is used to playback simple sound effects, for the most part linked with interface actions in the game. I did this by creating a very simple patch which plays a sound when it receives a bang. Which sound it plays is dictated by the argument (in this case, the sound of discovering a new capsule). The process for adding a new sound, then, is as simple as adding the sound file to the /assets/sounds/ folder, and making a new instance of “sounds.pd” and naming it the same as the new sound. Jon, in the project code, created a list called “sounds” which is sent into Pure Data. When that list contains “capsule”, a bang is sent into that subpatch, and the sound is played.

A more complex version of what could be done with this type of data is seen in the voice of Mr Puggles, who helps you learn how to play the game. Mr Puggles pops on and off the screen to guide you through the first few worlds, and when he does he send Pure Data a “puggleShow” and “puggleHide” signal. I wanted to give Puggles a funny synthesizer voice that was different every time – dead simple in PD. To do that, I take the puggleShow bang and use it to trigger five more bangs, spaced out over a second. Each of these bangs triggers a random number which is translated into a MIDI note. This note controls the pitch of two oscillators (a sine and a sawtooth), one of which is slightly modified to make them slightly different pitches. These are played through a short volume envelope and a filter which is also controlled by a random number generator. Result? Hilarious beeping boopy Mr Puggles voice, all coming from one bang.

Every time a player buys or selects a planet, a short list is sent to Pure Data comprised of the planet BPM and a random number seed. The BPM is used to calculate delay times and such, and the random number seed is used to create a sort of musical identity for the planet. This is done by choosing a “beat library” and a musical mode.

The mode is created by building a lookup table that chooses the notes from a chromatic scale that would be used in a particular mode. For example, a major scale (ionian mode) uses notes 1, 3, 5, 6, 8, 10, and 12. Each melodic sound library I used is comprised of a full chromatic octave, and the notes that are played on any given planet are controlled by this table. This ensures not only that all of the different sound libraries being played on a planet will be in the same key, but also that a planet will have a strong melodic identity.

The sound libraries in the game are all controlled by the pugs on the planets. As they run around, each time they land they will trigger a sound. The type of sound is dependent on what terrain they are on – thus, if they run through the snow they play a toy piano, if they run through lava a distorted guitar, etc. There are two states of playing the sound, one if the player deliberately tells the pug to go to that tile, and the second if the pug is traveling over that tile to get somewhere else. It’s super easy to do that kind of thing in Pd; just set up two different ‘play sound’ envelopes, maybe a little extra delay or reverb, and you’re done!

The final piece of the puzzle for making the pugs running around into music is to make each tile be a different note. The terrain of each planet is created by making a sort of height map, where different heights correspond with different terrain types (grass, water, snow, etc). This also means that each tile has a unique number between 0 and 1. When the player buys or selects a planet, a giant random number table is generated in Pure Data which creates a number between 1 and 13 for each possible value between 0 and 1. That value is what is used to pick the note of the mode. This somewhat convoluted approach again lets us make sure that each planet will have a unique, but fully reproducible, musical character.

The actual playing of the sounds is probably the messiest part of the patch structure. Purists look away now.

I wanted to make sure this part of the patch was as flexible as possible, so I ended up using the soundfiler and tabread~ objects, rather than tabplay~, which is great in practice though does look rather uncouth. Additionally, I had some limitations imposed upon the structure of the patch – namely, I had to keep the number of tables down as much as possible, to save on memory. So each sound bank has two voice polyphony – there are many sound banks, and the beats and sound effects aren’t counted in this, so that limitation is not really heard in the final product at all. It did mean I had to work out a decent voice allocation system though!

I think my memory issues were probably my only problem with using PD in this project – though only indirectly. As I mentioned, they were hardly a problem artistically, however it took me a while to get used to the idea that not everything I patched on a computer would work on an iPhone. Similarly, I had to be very careful about things like relative volumes. In a generative music game like Pugs Luv Beats, the player could quite easily send 15 pugs running around making sound, which mounts up pretty quickly. It means that all of the patches and sound need to be designed to withstand lots of triggering without distorting. None of these things are problems, really, all they require is regular testing on devices and simulators – something that every mobile developer is already used to.

That’s the Pure Data audio engine in a nutshell. The end result is a flexible and powerful audio engine that sounds really great and is fully integrated into Pugs Luv Beats. The game is a great combination of music, silliness, and strategy – there’s a bit of something in there for everyone. You can definitely just play with the game to make beats, or you can try and collect all of the costumes, or you can try and make the most efficient planet ever. You can also explore the galaxies being made by your Game Center friends, to hear what they’re up to.

The background story:

And, just for fun, a silly promo featuring real pugs. Anyone traumatized by the sight of Pd patches, these should relax you.

More information at the developer site:
http://luckyframe.co.uk/pugsluvbeats/game.html

Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

What happens when you meld the most futuristic Microsoft technology with the most futuristic Apple technology with the most ColecoVision-esque graphics as built in Jitter? Or you create gameplay that couples physical human contortion with the step sequencing rhythms of music? A different take on music games, that’s what.

Developers Matt (“M@tt”) Boch and Ryan Challinor work, in their day jobs, on the music game as most people know it, at Harmonix. Harmonix’s roots remain in the rhythm game, so that music play, even at its most serious, is still about musical timing accuracy. Pxl Pusher is a very different mechanic: imagine a step sequencer grid on an iPad, presenting blocks that, true to the classic game Twister, require another player to balance and stretch their bodies to match.

I caught up with Matt and Ryan over the summer at New York’s Museum of Modern Art. Before throngs of crowds swarmed the game – easily one of the most popular of the night – I managed to get some quick footage of the creators doing last-minute patching and trying out gameplay. (The quiet there is atypical; we got to shoot before the doors opened.) An insane travel schedule kept me from publishing sooner, but here, Matt and Ryan share their process.

Another interesting twist: Max/MSP and Jitter allowed extremely rapid prototyping with Kinect, something of interest to anyone doing this sort of work. (And dig those “3D” images … if they don’t blind you.)

CDM: How did you work together on this project?

Matt: Jamin from Kill Screen had asked me to make a project for the Pop Rally, and I was looking for someone to team up with to make it happen. Around the same time, Ryan was developing Synapse, and I got super excited about that tech. Having worked with him closely on Dance Central 2, I was positive we’d make a killer team, so I proposed we team up and make it happen. He got the ball rolling immediately.

Max/MSP is a pretty ideal environment for the both of us to work in, as it lets us evaluate ideas really quickly, and completely obliterate the lines between disciplines that we’re used to in traditional game development. Each of us designed, coded, sound designed, and made art for the game. It was awesome to be able to share in the creation of all parts of the game.

Ryan: We riffed on ideas with each other until it snowballed into the final product. Max allowed us both to work in the same space, as opposed to the traditional programmer/designer arrangement, where I would work in code and Matt would tweak values that I expose. This allowed us to work much faster and blurred the line between programmer and designer.

Any roadblocks, discoveries along the way?

Ryan: The whole project was driven by discovery: discover a fun interaction, then figure out how to leverage it. There were no real roadblocks, other than Matt being on painkillers for the entirety of the project!

Matt: I fell, knocked out four teeth, and fractured my jaw, so I was on heavy doses of painkillers for much of the development. I’m doing much better now, almost fully recovered.

The greatest discovery for me was more of a rediscovery. I remembered how freeing it is to design a game for a very specific context. Before working at Harmonix, I made arcade games in a contemporary art context. Getting back into that mindset was a blast.

Tell us a bit how you work with Max for rapid prototyping. How did you set it up? Any specific tools you made use of? Any tips, either related to Max or Kinect specifically?

Ryan: No real tips for Max, other than do a lot of commenting when you’re working on a file with someone else. As far as Kinect tips, my main advice would be to explore and discover what fun you can have with he input, other than starting with a specific goal. The only tools we used outside of Max were Synapse for the Kinect data, Adobe Premiere for the background movie, and some web app Matt used to generate some of the sound effects.

Matt: We ended up using Beanstalkapp & Versions for version control, which was huge for us in the late stages of development when we both needed to be working simultaneously. It was great that we structured our patch in such a way that we could easily turn elements of the patch into standalone patches, so simultaneous work could happen. You lose some flexibility, but it’s worth it to be able to work in tandem.

How did you conceive the idea for this game?

Matt: Largely through iteration and idea exchange. I VJ around Boston a fair amount and often used TouchOSC to VJ with my iPhone, so I’m not stuck behind my laptop for hours at a time, so when Ryan brought up the idea of using TouchOSC with MAX and Synapse, I was totally sold. We quickly arrived at the mashup between Twister and Step Sequencer, and every after that was polishing mechanics, arriving at a unified aesthetic, and then finding the best way to ramp difficulty.

Ryan: Matt approached me to work on a game for the PopRally right around when I just finished up Synapse, so I wanted to leverage that in the project. I had also just learned about TouchOSC, so I thought it could be fun to incorporate that. So, the initial concept was born out of experimenting with interactions that combined the Kinect and iPad.

How did things go at MOMA? Any surprises in terms of reception? It was effectively rapid playtesting — anything that you would now incorporate into the game? Or things that worked well, for that matter?

Matt: It was a phenomenal success. I never imagined we’d have a line of people waiting to play the game all night, never mind a line full of people cheering, laughing, and clapping as they watched others play. We’ve talked a bit about ways to modify the existing game to make it a bit deeper and more broadly compatible, but I’m also wary of upsetting the balance we arrived at.

Ryan: I was pleased with the reception at MoMA, it was very satisfying to work on a project and then immediately have people playing it, literally 10 minutes after it was finished (thanks to a last-minute feature request from Matt’s friend Christina)

What does it mean to you to have this game in this landmark art museum?

Ryan: I’m really grateful that we were able to show it off in such an amazing space. It feels a little unreal to me, my brain hasn’t really processed it yet.

Matt: It was hugely validating, especially because my Dad was in town and was able to make it to the event. He’s never seen crowds of people play the games I’ve worked on so to have him there, seeing how much fun everyone was having, was awesome in and of itself. That it was at the MoMA? Still haven’t totally processed it. It was the stuff of dreams when I was getting my art degree; it’s going to take a bit for me to internalize that that dream was realized. Not only was I able to show work at the MoMA, but it was shown in the context of so many great games: Bit.Trip Beat, Limbo, B.U.T.T.O.N., QWOP, the list goes on…

How about the connection to Kill Screen Magazine?

Matt: I think Kill Screen is far and away the best game writing happening right now. Both their daily updates on the web and their themed print issues are staples in my life. To be fair, I might be biased as I’m good friends with [editor-in-chief] Jamin, but nevertheless, I think what they’ve accomplished is undeniably phenomenal. What other publication could convince the MoMA to have a video game night?

Can you tell us a bit about your roles at Harmonix, and how you came to be there?

Ryan: I’m a programmer at Harmonix, I’ve worked on The Beatles: Rock Band and the Dance Central series. I was a CS major at the University of Texas, I got an internship at a game studio in Austin that turned into a job, then a few years later a coworker talked me up to someone he met from Harmonix, and I got recruited to work there.

Matt: I am a Project Lead at Harmonix, though I’ve occupied many roles previously, most recently Senior Designer. I started at Harmonix as a Production Assistant while I was finishing my thesis at Harvard. My friend and VJ partner, Josh Randall, is the Creative Director of Harmonix. We’d been VJing together for a bit while I was in school. He gave me a call one day, suggesting I apply for a Production Assistant role. One of my jobs was to track all the prototype hardware, but I ended up spending most of my time repairing it, as my art practice of building arcades gave me a fair amount of experience about mechanical engineering, electronic engineering, and coding. That landed me a role as hardware designer, and, many Rock Band iterations and trips to China later, I was tasked with investigating tech for a potential dance game. That lead to me heading up the development of a game prototype that would become Dance Central, and I’ve been designing for the franchise ever since. A circuitous path for sure, but I’m really happy with where I ended up, and deeply grateful to Harmonix for allowing me to occupy such varied roles throughout my 4 years tenure.

Synapse is something other folks can use, too, correct? What does it do, and where might they begin?

Ryan: Yep, Synapse is a freely released toolset. The primary focus of the project was for controlling Ableton Live with Kinect, but it can be used with anything that receives OSC input. You can find more information at http://synapsekinect.tumblr.com/

Any future plans for this game, or other Synapse-powered stuff?

Ryan: In the immediate future, we’ll be showing off PXL PUSHR at Indiecade in LA on October 8th. Beyond that, I’m not sure what the future is. At the very least, we’re going to keep the ideas of the gameplay mechanic in mind to be incorporated into future projects. As far as Synapse, I’m currently exploring other ideas so I don’t have any Synapse stuff in the works at the moment, but I know of a few other people working on Synapse-powered projects currently.

Matt: As Ryan stated, we’re showing it at Indiecade, and are hoping we’ll get more opportunities to show the game publicly. I deeply enjoyed working on the game with Ryan and I hope that TeamPXL, the moniker we’ve adopted, will develop additional games with the same working methodology. For now, given that we both just wrapped development on Dance Central 2, we’re taking a much needed break to recharge. I know Ryan’s got some awesome audio manipulation patches in the works, and I’m excited to see what comes of those. In time, I’m sure an awesome idea for a game will cross one of our minds and we’ll have no choice but to dig in and make something new!

http://teampxl.tumblr.com/

The gaming industry has made their bet, and it’s that touchscreens go better with tactile controls. Might digital musicians reach the same conclusion?

A funny thing has happened on the way to the touch era. The vision of a device like the iPad is minimalist to the extreme: an uninterrupted, impossibly-slim metal slate, as impenetrable as some sort of found alien scifi object. The notion is that by reducing physical controls, the software itself comes to the fore. It’s beautiful conceptually … and then you find yourself tapping and stroking a piece of undifferentiated glass. For navigating interfaces – and even, I’d argue, exploring sound design and composition – it works brilliantly. But for live digital performance (what to game lovers is called “gaming”), for anything that wants tactile feedback, it can be imprecise or unsatisfying, or both.

Watching this shake out as a design problem is fascinating, especially coming from the perspective of music. Digital musicians were exploring alternative interfaces since before it was cool. Given the ability to make any sound we can possibly imagine, the question of how you design an interface around sound is compositional, philosophical, essential.

Whatever winds up working in the marketplace, there are some fascinating ideas for combining touch with tactile. Since both are good at certain tasks, why not do both?

We’ve seen several examples among musicians and researchers exploring how to augment the touchscreen with physical input:

Mike Kneupfel’s research at NYU’s ITP program, in the video at top, investigates adding additional inputs. See: Extending the Touchscreen.

We saw that kind of extensibility in an iPad dock concept by Livid Instruments.

While it lacks additional tangible controls, I/O extensibility is featured in a still-as-yet-unreleased “pro” dock by Alesis, and most recently in a DIY dock by circuit bending pioneer Reed Ghazala.

Now, game vendors are moving in the same direction – even with prototypes that look quite a lot like the research project above. (Sometimes, arriving at the obvious conclusion is necessary for a great design.)

Sony’s PlayStation Vita, successor to the PSP mobile game platform, augments touch input with tactile controls in much the same way as Michael Knuepfel’s work does. Notably, it also proposes how these inputs can coexist in a form factor that’s larger than a phone, but smaller than a tablet – scaled roughly to a comfortable holding distance between your two hands. (Microsoft and Apple each unveiled standard split keyboards on Windows 8 and iOS 5, respectively. The era of thumb ergonomics is now fully underway.)

Nintendo’s Wii U controller combines a lot of sensing capabilities into one device. Like Sony’s effort, the centerpiece is the combination of the interactive touch display with analog controls. But true to its Wii heritage, Nintendo is packing other sensing technology, too. While its evolution has been more piecemeal, the same is true of the Xbox 360 in the Kinect era. The Kinect camera is really a bundle of mic and stereoscopic camera sensing with software intelligence for motion analysis and even speech analysis via a variety of methods. While Kinect is touchless, the conventional gamepad still plays a role.

What’s the relevance of all of this evolution to music? Digital music’s demands parallel gaming, requiring precision, accessibility, scalability from beginners to hardcore experts, and real-time interaction. Also, music research has often been at the forefront of experimentation with a variety of means of translating sensory data to expression. And since musical practice itself is roughly as old in human evolution as language, if not older, it’s a key way of glimpsing how ubiquitous interfaces can become meaningful.

Let me put that another way: the stuff game companies are doing now looks a heck of a lot like what computer musicians have been doing for years.

While much of the acclaim for platforms like the iPad has been for their transparency and unadorned interfaces – and while I believe those are valuable concepts – bundles of capabilities for interacting with the world can become powerful. That means efforts like Apple’s addition of USB MIDI connectivity to the iPad, or Google’s nascent work to standardize USB host mode and open hardware development (based on Arduino), take on new meaning. Add to this additional connectivity via Bluetooth and wifi, and it may be that we only really see what these platforms do when, like the PC, they start geting sociable with a range of other gear.

This could also mean that communities like the music community have a chance to prove that the “post-PC era” is a little different than it’s been described in the mainstream press – and maybe a little less a radical departure. The “post PC era,” we’re told, is less about being a hub for a lot of hardware. But as people look for tactile feedback, some of the coolest applications of these platforms may not be in the mainstream use as “consumption” devices, but at the fringe.

I’ve just come from the launch of the Samsung Galaxy Tab 10.1 in New York. You’re not missing much; there were a handful of people snapping up the tablets. (I think the 10.1, and a few other Honeycomb-based tablets, do have a bright future, though their growth may be a bit slow at first as developers get their hands on them and give people a reason to buy them.)

What was most compelling to people at the launch, though, was a planned appearance by pop star Ne-Yo (at least according to some staffers to whom I spoke).

But the connection was, at best, tenuous. It may be when devices like these tablets are made more viable for musicians onstage that that connection starts to make sense. And that may mean that Apple and Google/Android vendors alike need to start to think more aggressively about the larger ecosystem and hardware applications. Remember all those futuristic promises from Apple about hardware accessories? Right now, the most significant hardware is the Square payment add-on, and it uses a hack to make it work through the audio jack. Both Apple and Google can do more work to open up hardware development.

It’s all well and good for the tablet to be a “post PC” device, to be different from PCs, to be better. But they may simultaneously need some of the openness to other gadgets that made the PC age so revolutionary.

Load up an Ableton set or mix samples, and you’re already in the domain of interactive music. With joysticks and arcade buttons and other controls, the blending of game and musical interface into generative compositional fusion is even clearer. It’s little wonder many electronic musicians take an interest in the nexus of gaming and music.

Any discussion of interactive music scores for games would be incomplete without Jonathan Mak. His self-produced title Everyday Shooter used classic top-down space combat as a musical experience: not only do sound effects in the game act as musical elements, but even the flow of the game itself fits into a generated song structure. Mak even imagined the title as an album. Playing through it, once you get into the groove of the action, the roles of gamer and listener merge into a single flow. (See video, at end.) Another nice feature – breaking from cliche, it’s a music game that employs guitar lacks in place of, say, a pounding trance soundtrack.

Now, Mak turns his attentions from space shooter to platformer, with SoundShapes, on the console Sony announced this week, the PlayStation Vita. It’s best to watch the video to see what’s going on. It’s just a taste; we’ll have to see the final title.

There’s more good news, too: on of our favorite artists, the inventive i am robot and proud, is the musical collaborator. And you can now look to Toronto as a hotbed of indie game action – take that, Montreal and New York – with i am robot and proud and the team behind iPad album-as-game-as-album Swords & Sworcery.

The effects of making ever game event musical can be cartoonish at times – though, perhaps in a game, that’s part of the pleasure and aesthetic. But in Mak’s best moments, it was as though he was composing with gameplay – canonical gameplay forms as a modern, digital parallel to musical forms like a sonata.

We’ll be watching. Thanks to Metehan Korkmazel for the tip!

Via GameSetWatch, who have some nice analysis. (One point of disagreement – they speculate this will be onstage “in the hands of at least one chiptunes performer.” I think that’s actually relatively unlikely; chip music artists continue to prefer dedicated music tools, not games. Laptop artists would occasionally feature an ElectroPlankton cameo. But I’ve been fiddling with some Processing sketches that try to make game interfaces for music. It’s fun, if really hard; I’ll keep trying. Seems a good airport layover project.)

Previously on CDM:
Portal 2′s Musical World, Available Free, in Non-Adaptive Form “For Testing” [ Also makes use of interactive musical accompaniment ]

Game Meets Album: Behind the Music and Design of the iPad Indie Blockbuster Swords & Sworcery

Game lovers may lap up anything the title Portal touches as though it’s covered in powdered sugar, but resident Valve Software composer Mike Morasky deserves special mention. His music for Portal, and now Portal 2, is dead-on: chilly, atmospheric, dystopian, but also pulsing with energy and able to capture the gaming blockbuster’s strange combination of diabolical cerebral puzzles with wit. It’s all the more impressive, as Morasky has straight-up parodied musical styles in his whimsical Team Fortress or horror movie-cinematic Left 4 Dead scores.

Developer Valve quietly released 22 instrumental tracks from Portal 2 as “Soundtrack Volume 1: Songs to Test By,” free in 320 kpbs MP3 form. “Music to Code By” could be just as appropriate. Even if you ignore this post, know that this score will be racking up Last.fm playcount as it pipes into the headphones of nerd boys and girls.

GamesRadar published an interview with Morasky. Interestingly, while this is being released in soundtrack form, Portal 2 is in fact adaptive in the game. The system is in the foreground only in a few scenes, but there, multiple layers give a sense of progression. Any musicians who have been … erm … sucked into this game no doubt found these scenes highlights already, but here’s Morasky’s explanation to GamesRadar:

There are several cases where the music adds channels and complexity as you successfully solve portions of the puzzle, with each additional piece of music actually coming from the device that is participating in the activated game play mechanic. Obviously, this can heighten the sense of achievement as one completes the puzzle but also turns the mechanics of the puzzle into a sort of interactive music instrument that you can explore by selectively triggering the different channels of music with differing timings and configurations. Most of the interactive music is also positional so that as you move through the space you also change the mix and volume of the music you are hearing, which invites explorations of the space as well.

Morasky and Valve have been at the forefront of adaptive music in games – an area still left surprisingly unexplored – in particular in Left 4 Dead’s use of cinematic cues to heighten suspense and integrate with actual gameplay.

But that doesn’t make the soundtrack to me any less satisfying. With nods to spooky scifi and electronica convention alike, it nonetheless emerges with a distinctive voice – much like the game itself. In a world of cookie-cutter mainstream gaming, at least in the triple-A territory, the success of the title could be encouraging. Find it here:
http://www.thinkwithportals.com/music.php

Found via the superb gaming blog Rock, Paper, Shotgun; who also note two additional volumes are on the way. Follow CDM’s Twitter feed and we’ll let you know when those hit.

And if you’d like to see more covered than in the interview above, let us know.

In the blurring areas between gaming and creation, toys and tools, there’s certainly a lot of action, spurred on by platforms for sharing software.

Pulse is a new title for the iPad, an ambient rhythmic gaming experience with a unique interface centering around a series of concentric circles. The graphic design looks gorgeous in its abstraction, as much music visualization and animation as game UI. The developer, Cipher Prime, has done this kind of terrific work before – their work includes the ambient streams of colored particles in Auditorium, the Flash-based browser game, followed by the Mac + PC game Fractal. Items of note here:

• The game combines melodic and rhythmic gameplay elements.
• Pulse is as much interactive album as game, accompanied by a release of songs (including the single below).
• Gameplay can be collaborative, not just single player.
• The title is built in the awesome Unity engine, which means, by the way, Android development isn’t ruled out. Ahem. Let’s hope those OEMs get their tablets straightened out – I repeat my mntra, choice is good.
• The developers credit their community of geeks and musicians in Philadelphia, PA.
• In addition to the existing tracks, the developers are looking for indie musicians in Philly looking to get in on the action. Game developers: the new record labels.

Pulse: Volume One

The title is already earning praise and recognition, including topping the charts and getting featured as iPad game of the week.

Interestingly, as the iPad morphs into game platform, that hasn’t stopped people from reconsidering game platforms as venues for music creation tools. So, by way of contrast and comparison – and in case your Xbox is feeling lonely with all the iPad news – it seems only right to counterpoint Pulse with a new Xbox 360 title also released last week.

Music Box is a Tenori-On-inspired music sequencer for Xbox Live Arcade. It’s fairly simple in conception, but makes clever use of the spare controls on an Xbox game controller, and at only a buck, it’s almost certainly a must-buy for music lovers with an Xbox.

Grab the 99-cent title from the Xbox Live Marketplace.

Developer Vadim of Facetious Creations built Music Box with Microsoft’s XNA toolset, which opens up the possibility of Windows Phone, too. He says the response so far has been terrific. I find it fun to play with – and an interesting diversion for a game console.

For all our complaints about iOS and even Android, game consoles remain the most closed platforms out there. (Indeed, some of the anxiety over iOS I believe stems from concerns the game consoles locked-down model will spread to other computing hardware.) That said, Microsoft arguably does more than any other console vendor to promote indie game titles; amidst some noise, there are some real gems on the Xbox Live Arcade.

So, there you have it – two very different models for two different platforms. Let us know what you think.

Addendum: Many, many games have taken on the idea of games as albums, or at least with strong musical dimensions. There’s a nice list of inspiration listed in the sidebar of the blog for Cipher Prime, just to name a few that offer indie and ambient goodness:
Aether
Blueberry Garden
Eufloria
flOw
Knytt Stories
Machinarium
Osmos
Passage
Samarost 2
Windosill

illucia is a project at the intersection of lots of forms of goodness and imagination. The physical interface – what you see first in the image and video here – is a DIY modular controller, complete with Buchla-style patch cords and arcade buttons and pretty knobs. But while that might lead you to believe we’re in analog synthesis land, the physical controls are just a tangible cockpit for what the artist calls “code-bending.” Connect the USB controller to software, and it modifies audiovisual games in modular, interactive fashion.

Since you probably don’t have this particular physical controller lying about your home, you can also use these games directly with whatever software or hardware interface you like, thanks to OpenSoundControl (OSC) interfacing.

The result: free, Creative-Commons licensed audiovisual “playgrounds.” The video at top is just a teaser, but there’s more to come, says creator Chris Novello. Chris tells CDM:

In a quick sentence: I make games that can play other games, and I’ve designed a physical console to interact with them.

The vimeo video’s accompanying text offers some explanation, but here is some extra info as well:

“Codebending?”

Codebending is the mutant daughter of live coding, circuit bending, and modular synthesizers. It gives software “patch points,” and allows for connections between otherwise-unrelated computer programs. Anything can control anything, and computer software tropes collapse into strange generative art. In terms of implementation, these patch points are really just OSC addresses, which allow for easy transmission between programs. http://www.paperkettle.com/codebending has more general information on codebending.

“Instrument?”

illucia is a physical instrument for making these connections – it gives the software patch points physical counterparts, and is a real life console for routing information between programs. The act of building and breaking connections (of exploring generative emergences via patching) becomes a form of play. In terms of hardware, I use Atmega mircocontrollers to encode the state of the device (connections between jacks, knob positions, and button presses) into a tiny byte array, which then gets sent to my computer over USB. This state is used to update a software model of illucia on the computer, which determines the routing between several games / programs. I use Max/MSP a lot for rapid development of routing utilities.

All of my games are written in Processing. They utilize classic arcade mechanics, but are more like modules in a modular synthesizer. I also add some twists, like:

“Soviet Life Sequencer,” which is a Tetromino step sequencer that is remixable by Conway’s Game of Life. I remember a CDM post about Conway sequencers, as well as one featuring the game Chime, but I think I’m the first person to combine Cellular Automata, falling Tetrominos, and a Step Sequencer : 0

also, “Pile of Secrets,” which is a codebendable text editor. Office software is a surprisingly fun companion to video game mechanics.

I’m in the process of releasing the first wave of info right now, but I’ve got a ton on the way as well!

For more information:
http://vimeo.com/22732435
paperkettle.com/​
twitter.com/​paperkettle
facebook.com/​paperkettle

Chris also says he’s sending over some more images, so I’ll add those to the story.

This should start the conversation, but if you have questions, fire away and I’m sure we can get Chris to answer them!

More images (all photos courtesy Chris Novello; used by permission):

A music score is, in essence, a way of making space into time: traversing notation from left to right and top to bottom, you move through a series of events. So, why not make that spatial map an actual map, as in the familiar, isometric interactive cityscape popularized by Will Wright’s classic game Sim City?

Isle of Tune does just that: lay out trees, houses, and city streets, and you sequence musical patterns as virtual islands. It’s available right now on the Web, powered by Flash – Chrome users can even get a one-click install via the Chrome Web Store. If you prefer to use a phone or tablet, mobile versions are coming, too, beginning with an imminent iPad release, seen in a video below.

Isle of Tune must have hit a nerve in this surreal pairing of imaginary landscapes and simple sequenced songs, because the YouTube nation has responded en masse.

I spoke to developer Jim Hall, who gives CDM the latest update and some background on himself:

A little bit about me: I’ve worked as an animator, flash developer and art director over the last 10 years in London, UK – mainly for digital ad agencies (some previous work here: www.happylander.co.uk). Around about June last year I pretty much quit advertising work (before my soul was lost completely in the desolate mire of micro-sites and adver-games!) and took some time out to make stuff for myself – mainly based around music and playful interaction.

Isle of Tune came from a desire to make a different kind of music sequencer along the lines of http://lab.andre-michelle.com/tonematrix or the Reactable – but with less of an abstract feel and a little more game-like and social. Since it went live in December there has been over 60,000 islands made
with some properly creative tunes in there eg. http://isleoftune.com/?id=63320 or http://isleoftune.com/?id=48359

Over the last few months I’ve been making updates from user requests – mainly to the way you can browse, search and view rated islands, it seems a lot of visitors come by just to check out the islands other people have made!

I’m currently working with a talented chap – Stuart Middleton – a developer since the days of the Spectrum (and now runs a small app dev company at www.abstractalien.com) who is converting the Isle of Tune codefor various mobile platforms and also helping me take it beyond what I could online with the limitations of flash. My far-fetched dream with the app is to be able to fund further development and allow me to make more stuff like this in the future without having to go back to the world of advertising!

For those of you with iPads, Jim says you should be able to look forward to the iOS release some time around mid-May if all goes to plan.

But for anyone, you can use this right now with Flash:
http://isleoftune.com/

And yes, even Requiem for a Dream gets the Island of Tune treatment. Hmmmm… this island seems a bit too sunny and cheery, no? Maybe the title needs a dreary, overcast Coney Island mode.