You might not expect a handheld game console, the gadget kids use to play Pokemon, to prove much worth as a musical instrument. But even in the age of readily-available computer plug-ins and iPhone apps, the DS holds its own. In the hands of two sets of artists, we find music that stands alone, independent of the gimmick of the device on which it was made. For these artists, the limitations of a fold-up touchscreen – entirely independent of doubling as a phone, or a computer, or a Facebook-browsing engine, or a powerful 64-bit DAW – apparently prove enticing. Beginning with Korg’s DS-10 cartridge, they use a stylus-operated software synth with its own unique character.

On some level, I almost hesitate to wax poetic about the fact that these were made with a Nintendo DS at all, because what these are, really, is love letters to synthesis.

And as it happens, both are available as free downloads from Bandcamp.

First up: AuxPulse is the duo of Rutger Muller and Michael Vultoo, based in Amsterdam and Kockengen, Netherlands, respectively. Late last year, they debuted their first album at Amsterdam’s prestgious Stedelijk Museum of modern art, playing a big set (two and a half hours) on small devices. Primarily employing the Nintendo DS, they nonetheless produce sounds that are rich and layered, sometimes even tending to the ambient exploration, not just the rawer chip-music sounds regularly associated with Nintendo handhelds.

Their music is trippy but danceable, unapologetically electronic, fully exploiting the DS-10′s idiosyncratic sonic character, one that’s slightly lower-fidelity than many soft synths (or even iPhone apps), without being “chippy” in the sense of retro devices. Dark textures collide with precise, clockwork rhythms, in sounds that sometimes tend to acid techno and sci fi game realms. (Lo-acid-fi, anyone?)

As you watch them live, you also see the value of the interface compositionally, both in terms of its pattern banks and its more conventional synth controls, all manipulated with the added precision of a stylus.

As they put it:

We aim to bring experimentation back to the dancefloor by expressing a psychedelic atmosphere through the use of a variety of rhythms and moods. Some of our inspirations are analogue synthesizers, acid, IDM, hardcore, gabber, ambient and oldschool electro.

Right now we mainly use the KORG DS-10 synthesizer for Nintendo DS to compose and improvise our music. When playing live we fuck with the synths as much as we can, trying to surprise ourselves with new sounds.

Our first album was recently released in Stedelijk Museum Amsterdam! Now we perform regularly, trying to open up some minds and move some feet.

The album, on Bandcamp:

And on SoundCloud:
Dream Stages (FREE ALBUM!) by AuxPulse

Bonus: an interview with them (in Dutch, naturally)

In a very different direction, Princeton, New Jersey-based DJ and producer Christian Montoya (love and tonic records) produces music on the DS-10 that’s drier and more exposed, as he programs intricate bass music on the unprocessed Nintendo cart. Christian works as a game designer by day, and channels some of the DS-10′s game music and so-called “chip music” heritage. The results, though, are a perfect marriage of game chip-waveform rawness, nude bass and synth and percussion sounds, and carefully-concocted grooves. For anyone concerned that game systems could hinder moving your butt out of the seat, this album is required listening. It’s utterly stripped-bare dance goodness – and it turns out the DS bass sounds fantastic.

Grab the record for free:

DS-10 users, got any tips for us on getting the most out of a Nintendo handheld and this KORG synth? Let us know.

Also, from comments but worth pointing out, Rutger directs us to good resources for getting the most out of DS-10:

If you’re interested in making DS-10 music you can check out http://www.ds10forum.com

I (Rutger, DS-10 Dominator, 1/2 of AuxPulse) run it with Harley (http://harleylikesmusic.com, superb DS-10 composer!) and we try to help out beginner’s and advanced users as much as we can.

Truly generative musical scores in games have been few and far between, and “music games” has traditionally meant arcade-style rhythm games in which you repeat phrases or whole songs as accurately as possible. Pugs Luv Beats breaks those molds. Part of a vanguard of new gaming creations that generate dynamic music on the fly, it marries grid-based sequencing and resource-gathering gaming, as music making and gameplay blur together. The interactively-produced music could itself become a new way of delivering a musical signature with sound packs.

And beneath it all lurks a free and open source library, libpd – the embeddable version of tried-and-true free graphical music environment Pure Data. (That library is now on GitHub, and vastly updated, by the way, and we’re expecting a book soon from the library’s principle author Peter Brinkmann.)

Oh, yeah, and don’t forget about some seriously addictive gameplay and adorable pugs. I’m suddenly not concerned about the 15 hours Europe-to-North-America travel I’m doing tomorrow.

Here’s what the gameplay looks like, since it’s much easier to see:

Pugs Luv Beats was just approved on the iTunes App Store for iPhone and iPad.

Co-creator Yann Seznec (The Amazing Rolo) is a terrific musician; I just caught up with him in Edinburgh and Berlin and watched him play a homebrewed pig gut instrument with Matthew Herbert for the performance piece “One Pig,” on tour at Berghain. Working with Pd allowed Yann to focus on those musical impulses and not just engineering, and to let him try things he otherwise would never have imagined on a mobile title. So I asked Yann to walk us through how the project was built. He responded with an exhaustively-detailed examination of the evolution of this title, right down to the Pd patches. (Click through for high-res versions.) If your New Year’s Resolution is doing something with patching, you might want to hang onto these answers. Here’s Yann:

The origins of Pugs Luv Beats date back about two years. After making [musical iPhone game] Mujik, Jon (Jonathan Brodsky, aka jonbro) and I were trying to think of other approaches to music mobile app design, and we started thinking more and more about games. Music games, as a whole, are an oddly passive and traditionalist experience – you play along with a premade track, and you are judged on your accuracy and flair (which is strangely reminiscent of music conservatory mindset…). Obviously there are exceptions (RjDj’s Dimensions, Elektroplankton, etc.), but there you go.

Particularly interesting to me was the idea that game mechanics are often very similar to compositional techniques. So for example, when Sonic runs at a normal speed he collects rings at one rate. However when he powers up and goes super fast, he collects rings at a much higher rate. This could be compared to introducing a melody and then speeding it up – and when there are two players, doing this with two melodies. Instant fugue!

We started looking at how we could make a music game where the music and the game elements were fully intertwined and augmented by each other. So Jon prototyped a space shooter drum machine. It was awesome.

To make a (very very very long and boring) story short, our idea and prototype landed us some funding from Channel 4 and Creative Scotland to work on games that focus on musical creativity and composition.

For various reasons, we decided to put aside the space shooter drum machine for a while, and start from scratch. After going through several full prototyping iterations we eventually settled on a core game mechanic that turned out to be in many ways similar to a Tenori-on [Yamaha grid instrument]/Boiingg-style [monome hardware patch] music generation system – in our final prototype, you controlled a series of little dots that moved around the screen, creating loops. This is super fun from a musical perspective because it’s easy and rewarding within a few seconds, and when you have several loops going it can gain some pretty serious rhythmic and melodic depth.

The key from there for us was turning this into a game. We had been using free Internet graphics packs up until then (we hadn’t hired our artist Sean yet) which featured a ladybug, so we had been referring to the main characters as ‘bugs’. During some discussion one of us accidentally said ‘pugs’, and the game idea was born. We constructed a story about pugs and their love for beets (like the vegetables) which create beats (ha!), and how their love turned into greed and got out of control, destroying their world. The game, therefore, is about helping the pugs rebuild their lost civilization by guiding them to create beats. You grow your galaxy by collecting beats, which you do most efficiently when you dress your pugs up in costumes. What’s not to like?

To get to the part that I imagine CDM readers are most interested in, the app development was done by Jon using openFrameworks, [lightweight language] Lua, our own game engine called Blud, and the audio is all done in Pure Data using libpd (through ofxPd). In hindsight we started using libpd really late in the game, just at the very end of the prototyping stage, which was rather silly. Our adoption of libpd basically made our dev cycle about a million times more efficient. My background is as a musician and sound designer, and I have very little coding knowledge. I do, however, have lots of knowledge of Max/MSP, so picking up Pure Data was pretty easy. This allowed Jon to completely pass off all the audio processing (not to mention aesthetic sound design choices) to me, saving him loads of time, giving me direct control over the sound, and letting me test and prototype different approaches to audio within an environment that I knew would be recreated in the game. Also, as Jon mentioned to me recently, by using PD we are able to take advantage of 20 years of audio DSP research and development. Pretty amazing.

How it all works:

The entire audio engine is contained within this patch. Pardon the messiness.

The simplest part of the patch is the “sounds” section, which is used to playback simple sound effects, for the most part linked with interface actions in the game. I did this by creating a very simple patch which plays a sound when it receives a bang. Which sound it plays is dictated by the argument (in this case, the sound of discovering a new capsule). The process for adding a new sound, then, is as simple as adding the sound file to the /assets/sounds/ folder, and making a new instance of “sounds.pd” and naming it the same as the new sound. Jon, in the project code, created a list called “sounds” which is sent into Pure Data. When that list contains “capsule”, a bang is sent into that subpatch, and the sound is played.

A more complex version of what could be done with this type of data is seen in the voice of Mr Puggles, who helps you learn how to play the game. Mr Puggles pops on and off the screen to guide you through the first few worlds, and when he does he send Pure Data a “puggleShow” and “puggleHide” signal. I wanted to give Puggles a funny synthesizer voice that was different every time – dead simple in PD. To do that, I take the puggleShow bang and use it to trigger five more bangs, spaced out over a second. Each of these bangs triggers a random number which is translated into a MIDI note. This note controls the pitch of two oscillators (a sine and a sawtooth), one of which is slightly modified to make them slightly different pitches. These are played through a short volume envelope and a filter which is also controlled by a random number generator. Result? Hilarious beeping boopy Mr Puggles voice, all coming from one bang.

Every time a player buys or selects a planet, a short list is sent to Pure Data comprised of the planet BPM and a random number seed. The BPM is used to calculate delay times and such, and the random number seed is used to create a sort of musical identity for the planet. This is done by choosing a “beat library” and a musical mode.

The mode is created by building a lookup table that chooses the notes from a chromatic scale that would be used in a particular mode. For example, a major scale (ionian mode) uses notes 1, 3, 5, 6, 8, 10, and 12. Each melodic sound library I used is comprised of a full chromatic octave, and the notes that are played on any given planet are controlled by this table. This ensures not only that all of the different sound libraries being played on a planet will be in the same key, but also that a planet will have a strong melodic identity.

The sound libraries in the game are all controlled by the pugs on the planets. As they run around, each time they land they will trigger a sound. The type of sound is dependent on what terrain they are on – thus, if they run through the snow they play a toy piano, if they run through lava a distorted guitar, etc. There are two states of playing the sound, one if the player deliberately tells the pug to go to that tile, and the second if the pug is traveling over that tile to get somewhere else. It’s super easy to do that kind of thing in Pd; just set up two different ‘play sound’ envelopes, maybe a little extra delay or reverb, and you’re done!

The final piece of the puzzle for making the pugs running around into music is to make each tile be a different note. The terrain of each planet is created by making a sort of height map, where different heights correspond with different terrain types (grass, water, snow, etc). This also means that each tile has a unique number between 0 and 1. When the player buys or selects a planet, a giant random number table is generated in Pure Data which creates a number between 1 and 13 for each possible value between 0 and 1. That value is what is used to pick the note of the mode. This somewhat convoluted approach again lets us make sure that each planet will have a unique, but fully reproducible, musical character.

The actual playing of the sounds is probably the messiest part of the patch structure. Purists look away now.

I wanted to make sure this part of the patch was as flexible as possible, so I ended up using the soundfiler and tabread~ objects, rather than tabplay~, which is great in practice though does look rather uncouth. Additionally, I had some limitations imposed upon the structure of the patch – namely, I had to keep the number of tables down as much as possible, to save on memory. So each sound bank has two voice polyphony – there are many sound banks, and the beats and sound effects aren’t counted in this, so that limitation is not really heard in the final product at all. It did mean I had to work out a decent voice allocation system though!

I think my memory issues were probably my only problem with using PD in this project – though only indirectly. As I mentioned, they were hardly a problem artistically, however it took me a while to get used to the idea that not everything I patched on a computer would work on an iPhone. Similarly, I had to be very careful about things like relative volumes. In a generative music game like Pugs Luv Beats, the player could quite easily send 15 pugs running around making sound, which mounts up pretty quickly. It means that all of the patches and sound need to be designed to withstand lots of triggering without distorting. None of these things are problems, really, all they require is regular testing on devices and simulators – something that every mobile developer is already used to.

That’s the Pure Data audio engine in a nutshell. The end result is a flexible and powerful audio engine that sounds really great and is fully integrated into Pugs Luv Beats. The game is a great combination of music, silliness, and strategy – there’s a bit of something in there for everyone. You can definitely just play with the game to make beats, or you can try and collect all of the costumes, or you can try and make the most efficient planet ever. You can also explore the galaxies being made by your Game Center friends, to hear what they’re up to.

The background story:

And, just for fun, a silly promo featuring real pugs. Anyone traumatized by the sight of Pd patches, these should relax you.

More information at the developer site:
http://luckyframe.co.uk/pugsluvbeats/game.html

Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

Think back to playing a simply childhood game like Musical Chairs. The actual gameplay depends only on auditory clues – something you take for granted as a kid, but something apparently lost on game engineers who insist exclusively on advanced 3D rendering engines for visuals. And because you get your body involved, the game becomes dynamic. That musical cue isn’t just off in the background: in the dizzying run around the chairs, the soundtrack can become the singular focus of your brain, an urgent score to the — DIVE, got the chair!

As the scene around game experimentation grows richer, there’s a rekindled interest in how game mechanics can play to different senses. In some cases, it can be a source of whimsy; in others, it’s the only way to design games for people who are absent one of those senses. And an ongoing exploration of music and sound as gameplay mechanic – not just gameplay accompaniment – ought to interest composers and sound designers. When you look at a conventional arcade game, tuning your reflexes to the graphics is key, even if sounds provide reward and ambience. In these games, the sound is where the play is.

Johann Sebastian Joust has a lot in common with Musical Chairs. The game input is the lovely Sony PlayStation Move motion controller, which – yep, you guessed it, is where the jousting comes in. (An earlier version used the Wiimote.) But in place of graphics, listening to the music itself tells you when to act, just as in the childhood game:

When the music plays in slow-motion, the controllers are extremely sensitive to changes in acceleration. When the music speeds up for, this threshold becomes less strict, giving the players a small window to dash at their opponents. If the player’s controller is ever moved beyond the allowable threshold, that player loses.

Little wonder that the game resembles some of those kids’ games: the designers reveal that they got the idea after improvising “folk” games with friends. Now, there is some concession to adding additional feedback – the controllers use the light-up ball on the end and rumble feedback just to make absolutely clear what’s going on; some “sound games” are more pure in their all-sonic interface. But the idea remains the same.

The game is the work of the Copenhagen Game Collective. They describe themselves as “multi-gender, multi-national, non-profit”; I would add to that “blazing hot stuff.” CGC’s games have earned some serious accolades; for one, B.U.T.T.O.N., a group party game, was the runaway hit of the Kokoromi GAMMA party in 2010, and also showed up wowing crowds again at the same Kill Screen / Museum of Modern Art Show at which we saw Pxl Pusher, covered yesterday. (CDM and myself were also involved in that Gamma party, and co-organized a one-button art show at San Francisco’s GAFFTA art space.) But the group has in no small sense put Copenhagen on the map.

The team for this title:
Douglas Wilson: concept, programming, and video
Nils Deneken: graphics and announcer voice
Nicklas “Nifflas” Nygren: music and sound

Composer Nicklas Nygren is a triple threat: game designer, coder, and composer. Check out some of his music on SoundCloud:

Latest tracks by Nifflas

There’s plenty more to say about sound games and music games and interactive music for games. I gave a presentation at the Indie Gaming Showcase in Berlin on the topic at an event hosted by arts network Platoon – see their write-up. I’ll pull those notes together; if you have any nominees of game work you’d like to see covered, let us know in comments.

But for now, I’ll leave you with the image of Johann Sebastian Joust and Musical Chairs. After all, composition and ensemble playing themselves can be seen as games with musical mechanics. They certainly can even have “win” and “fail” mechanics – ask your local orchestra player.

http://www.copenhagengamecollective.org/johann-sebastian-joust/

What happens when you meld the most futuristic Microsoft technology with the most futuristic Apple technology with the most ColecoVision-esque graphics as built in Jitter? Or you create gameplay that couples physical human contortion with the step sequencing rhythms of music? A different take on music games, that’s what.

Developers Matt (“M@tt”) Boch and Ryan Challinor work, in their day jobs, on the music game as most people know it, at Harmonix. Harmonix’s roots remain in the rhythm game, so that music play, even at its most serious, is still about musical timing accuracy. Pxl Pusher is a very different mechanic: imagine a step sequencer grid on an iPad, presenting blocks that, true to the classic game Twister, require another player to balance and stretch their bodies to match.

I caught up with Matt and Ryan over the summer at New York’s Museum of Modern Art. Before throngs of crowds swarmed the game – easily one of the most popular of the night – I managed to get some quick footage of the creators doing last-minute patching and trying out gameplay. (The quiet there is atypical; we got to shoot before the doors opened.) An insane travel schedule kept me from publishing sooner, but here, Matt and Ryan share their process.

Another interesting twist: Max/MSP and Jitter allowed extremely rapid prototyping with Kinect, something of interest to anyone doing this sort of work. (And dig those “3D” images … if they don’t blind you.)

CDM: How did you work together on this project?

Matt: Jamin from Kill Screen had asked me to make a project for the Pop Rally, and I was looking for someone to team up with to make it happen. Around the same time, Ryan was developing Synapse, and I got super excited about that tech. Having worked with him closely on Dance Central 2, I was positive we’d make a killer team, so I proposed we team up and make it happen. He got the ball rolling immediately.

Max/MSP is a pretty ideal environment for the both of us to work in, as it lets us evaluate ideas really quickly, and completely obliterate the lines between disciplines that we’re used to in traditional game development. Each of us designed, coded, sound designed, and made art for the game. It was awesome to be able to share in the creation of all parts of the game.

Ryan: We riffed on ideas with each other until it snowballed into the final product. Max allowed us both to work in the same space, as opposed to the traditional programmer/designer arrangement, where I would work in code and Matt would tweak values that I expose. This allowed us to work much faster and blurred the line between programmer and designer.

Any roadblocks, discoveries along the way?

Ryan: The whole project was driven by discovery: discover a fun interaction, then figure out how to leverage it. There were no real roadblocks, other than Matt being on painkillers for the entirety of the project!

Matt: I fell, knocked out four teeth, and fractured my jaw, so I was on heavy doses of painkillers for much of the development. I’m doing much better now, almost fully recovered.

The greatest discovery for me was more of a rediscovery. I remembered how freeing it is to design a game for a very specific context. Before working at Harmonix, I made arcade games in a contemporary art context. Getting back into that mindset was a blast.

Tell us a bit how you work with Max for rapid prototyping. How did you set it up? Any specific tools you made use of? Any tips, either related to Max or Kinect specifically?

Ryan: No real tips for Max, other than do a lot of commenting when you’re working on a file with someone else. As far as Kinect tips, my main advice would be to explore and discover what fun you can have with he input, other than starting with a specific goal. The only tools we used outside of Max were Synapse for the Kinect data, Adobe Premiere for the background movie, and some web app Matt used to generate some of the sound effects.

Matt: We ended up using Beanstalkapp & Versions for version control, which was huge for us in the late stages of development when we both needed to be working simultaneously. It was great that we structured our patch in such a way that we could easily turn elements of the patch into standalone patches, so simultaneous work could happen. You lose some flexibility, but it’s worth it to be able to work in tandem.

How did you conceive the idea for this game?

Matt: Largely through iteration and idea exchange. I VJ around Boston a fair amount and often used TouchOSC to VJ with my iPhone, so I’m not stuck behind my laptop for hours at a time, so when Ryan brought up the idea of using TouchOSC with MAX and Synapse, I was totally sold. We quickly arrived at the mashup between Twister and Step Sequencer, and every after that was polishing mechanics, arriving at a unified aesthetic, and then finding the best way to ramp difficulty.

Ryan: Matt approached me to work on a game for the PopRally right around when I just finished up Synapse, so I wanted to leverage that in the project. I had also just learned about TouchOSC, so I thought it could be fun to incorporate that. So, the initial concept was born out of experimenting with interactions that combined the Kinect and iPad.

How did things go at MOMA? Any surprises in terms of reception? It was effectively rapid playtesting — anything that you would now incorporate into the game? Or things that worked well, for that matter?

Matt: It was a phenomenal success. I never imagined we’d have a line of people waiting to play the game all night, never mind a line full of people cheering, laughing, and clapping as they watched others play. We’ve talked a bit about ways to modify the existing game to make it a bit deeper and more broadly compatible, but I’m also wary of upsetting the balance we arrived at.

Ryan: I was pleased with the reception at MoMA, it was very satisfying to work on a project and then immediately have people playing it, literally 10 minutes after it was finished (thanks to a last-minute feature request from Matt’s friend Christina)

What does it mean to you to have this game in this landmark art museum?

Ryan: I’m really grateful that we were able to show it off in such an amazing space. It feels a little unreal to me, my brain hasn’t really processed it yet.

Matt: It was hugely validating, especially because my Dad was in town and was able to make it to the event. He’s never seen crowds of people play the games I’ve worked on so to have him there, seeing how much fun everyone was having, was awesome in and of itself. That it was at the MoMA? Still haven’t totally processed it. It was the stuff of dreams when I was getting my art degree; it’s going to take a bit for me to internalize that that dream was realized. Not only was I able to show work at the MoMA, but it was shown in the context of so many great games: Bit.Trip Beat, Limbo, B.U.T.T.O.N., QWOP, the list goes on…

How about the connection to Kill Screen Magazine?

Matt: I think Kill Screen is far and away the best game writing happening right now. Both their daily updates on the web and their themed print issues are staples in my life. To be fair, I might be biased as I’m good friends with [editor-in-chief] Jamin, but nevertheless, I think what they’ve accomplished is undeniably phenomenal. What other publication could convince the MoMA to have a video game night?

Can you tell us a bit about your roles at Harmonix, and how you came to be there?

Ryan: I’m a programmer at Harmonix, I’ve worked on The Beatles: Rock Band and the Dance Central series. I was a CS major at the University of Texas, I got an internship at a game studio in Austin that turned into a job, then a few years later a coworker talked me up to someone he met from Harmonix, and I got recruited to work there.

Matt: I am a Project Lead at Harmonix, though I’ve occupied many roles previously, most recently Senior Designer. I started at Harmonix as a Production Assistant while I was finishing my thesis at Harvard. My friend and VJ partner, Josh Randall, is the Creative Director of Harmonix. We’d been VJing together for a bit while I was in school. He gave me a call one day, suggesting I apply for a Production Assistant role. One of my jobs was to track all the prototype hardware, but I ended up spending most of my time repairing it, as my art practice of building arcades gave me a fair amount of experience about mechanical engineering, electronic engineering, and coding. That landed me a role as hardware designer, and, many Rock Band iterations and trips to China later, I was tasked with investigating tech for a potential dance game. That lead to me heading up the development of a game prototype that would become Dance Central, and I’ve been designing for the franchise ever since. A circuitous path for sure, but I’m really happy with where I ended up, and deeply grateful to Harmonix for allowing me to occupy such varied roles throughout my 4 years tenure.

Synapse is something other folks can use, too, correct? What does it do, and where might they begin?

Ryan: Yep, Synapse is a freely released toolset. The primary focus of the project was for controlling Ableton Live with Kinect, but it can be used with anything that receives OSC input. You can find more information at http://synapsekinect.tumblr.com/

Any future plans for this game, or other Synapse-powered stuff?

Ryan: In the immediate future, we’ll be showing off PXL PUSHR at Indiecade in LA on October 8th. Beyond that, I’m not sure what the future is. At the very least, we’re going to keep the ideas of the gameplay mechanic in mind to be incorporated into future projects. As far as Synapse, I’m currently exploring other ideas so I don’t have any Synapse stuff in the works at the moment, but I know of a few other people working on Synapse-powered projects currently.

Matt: As Ryan stated, we’re showing it at Indiecade, and are hoping we’ll get more opportunities to show the game publicly. I deeply enjoyed working on the game with Ryan and I hope that TeamPXL, the moniker we’ve adopted, will develop additional games with the same working methodology. For now, given that we both just wrapped development on Dance Central 2, we’re taking a much needed break to recharge. I know Ryan’s got some awesome audio manipulation patches in the works, and I’m excited to see what comes of those. In time, I’m sure an awesome idea for a game will cross one of our minds and we’ll have no choice but to dig in and make something new!

http://teampxl.tumblr.com/

In the blurring areas between gaming and creation, toys and tools, there’s certainly a lot of action, spurred on by platforms for sharing software.

Pulse is a new title for the iPad, an ambient rhythmic gaming experience with a unique interface centering around a series of concentric circles. The graphic design looks gorgeous in its abstraction, as much music visualization and animation as game UI. The developer, Cipher Prime, has done this kind of terrific work before – their work includes the ambient streams of colored particles in Auditorium, the Flash-based browser game, followed by the Mac + PC game Fractal. Items of note here:

• The game combines melodic and rhythmic gameplay elements.
• Pulse is as much interactive album as game, accompanied by a release of songs (including the single below).
• Gameplay can be collaborative, not just single player.
• The title is built in the awesome Unity engine, which means, by the way, Android development isn’t ruled out. Ahem. Let’s hope those OEMs get their tablets straightened out – I repeat my mntra, choice is good.
• The developers credit their community of geeks and musicians in Philadelphia, PA.
• In addition to the existing tracks, the developers are looking for indie musicians in Philly looking to get in on the action. Game developers: the new record labels.

Pulse: Volume One

The title is already earning praise and recognition, including topping the charts and getting featured as iPad game of the week.

Interestingly, as the iPad morphs into game platform, that hasn’t stopped people from reconsidering game platforms as venues for music creation tools. So, by way of contrast and comparison – and in case your Xbox is feeling lonely with all the iPad news – it seems only right to counterpoint Pulse with a new Xbox 360 title also released last week.

Music Box is a Tenori-On-inspired music sequencer for Xbox Live Arcade. It’s fairly simple in conception, but makes clever use of the spare controls on an Xbox game controller, and at only a buck, it’s almost certainly a must-buy for music lovers with an Xbox.

Grab the 99-cent title from the Xbox Live Marketplace.

Developer Vadim of Facetious Creations built Music Box with Microsoft’s XNA toolset, which opens up the possibility of Windows Phone, too. He says the response so far has been terrific. I find it fun to play with – and an interesting diversion for a game console.

For all our complaints about iOS and even Android, game consoles remain the most closed platforms out there. (Indeed, some of the anxiety over iOS I believe stems from concerns the game consoles locked-down model will spread to other computing hardware.) That said, Microsoft arguably does more than any other console vendor to promote indie game titles; amidst some noise, there are some real gems on the Xbox Live Arcade.

So, there you have it – two very different models for two different platforms. Let us know what you think.

Addendum: Many, many games have taken on the idea of games as albums, or at least with strong musical dimensions. There’s a nice list of inspiration listed in the sidebar of the blog for Cipher Prime, just to name a few that offer indie and ambient goodness:
Aether
Blueberry Garden
Eufloria
flOw
Knytt Stories
Machinarium
Osmos
Passage
Samarost 2
Windosill

As a guitar, the Rock Band 3 Mustang is a bit unusual – there are strings, but an array of buttons replaces the frets, and it is intended as a game controller. But with all those buttons, strings, and sensors, it makes a remarkably flexible, surprisingly inexpensive controller. Our friend nay-seven puts it to good use with Sensomusic Usine. And talk about a budget-minded setup – one that could put platforms like the iPad to shame. Usine costs just EUR90 for a full license, with discounted educational pricing and a version you can try for free. The Mustang runs just over US$100, which could put it in the category of must-buy for anyone who loves experimenting with alternative controllers – guitarist or not. Add an inexpensive Windows laptop and audio interface, and you’ve got a pretty terrific setup.

Previously, with the help of Harmonix engineers, we documented how the MIDI spec works:
Exclusive Details: How the Rock Band 3 Fender Mustang Works as a MIDI Guitar

nay-seven writes:

I’ve discover this guitar with this article Peter, so thanks for this ! it’s a quiet cool guitar and cheap for the possibilities . I’ve made a little patch in Usine to add some features like open tuning, x/y visualization, and you can also use most of the buttons to run effects or samples.

Your mileage may vary, but it sure looks promising. Keep in mind, this is the cheaper Rock Band controller with buttons. As such, it’s a good choice for people wanting some cheap experimentation and people who aren’t guitarists. As for the real guitar controller for Rock Band 3, the Fender Squier Rock Band Controller, I have a writer working on documenting its more advanced MIDI features. It’s more complicated, and I can’t endorse it just yet – wait for the full review and details.

While the object of both vigorous praise and criticism, music games have helped people see music and musical interfaces in new ways. Masaya Matsuura (PaRappa the Rapper) is known as the father of that genre. With a new title, you can try out his latest music gaming concept for free, on iPod touch and iPhone. The game is the product of a collaboration with Dutch developer Triangle Studios (makers of the iPhone port of augmented reality browser Layar). And it’s all for a good cause: the OneBigGame effort is using proceeds from track sales for this title and other gaming revenues to support Save the Children and Starlight Children’s Foundation.

More than any other designer, Matsuura has thought of plenty of novel ideas for how to vary the music game model. At its simplest, of course, music games owe some of their legacy to titles like the toy Simon: copy a sequence of events in time. With WINtA, the gameplay has been varied again. Here, the digital vocals become the object of the player’s input. There are also clever visuals unique to each music track, making the game as much music visualization and interactive toy as game.

The creators say the new approach will present a “new way of involving players in music and lyrics,” partly because the lyrics and vocals become an active part of the title. The free track comes from Matsuura himself; other titles are in-app purchases. (Free tracks will be released weekly, too, according to the publishers.)

Also, I don’t doubt that in these simple, geometric graphics, there might be for someone out there the seed for a musical interface that isn’t a toy.

Part of what fascinates me about Matsuura’s work – aside from the fact that he inspired game house Harmonix – is that it is so very different from conventional musical interface work. It does need to function more simply, as a toy, and with clear states of success and failure (arguably not the case in traditional music performance). This should be, at the very least, worth a quick play-through. And you get a taste of Matsuura’s whimsical musical style, as well.

WINtA on the iTunes App Store

http://wintagame.com
http://support.ngmoco.com/
http://onebiggame.com

“HALO: REACH” Game Audio Profile from Michael Coleman on Vimeo.

Our friends at Designing Sound have been rocking out on that site, with extraordinary original and linked content for sound designers, ranging from work on games, broadcast, and films to sonic exploration for the curious field recorder or producer. (Designing Sound is hosted by CDM and Noisepages – hence the new template, which will benefit from some corrections we’re making over the coming days.)

Now, they’ve launched a second site just to pull together video content.

If you love sound design of any kind, get ready to curl up on the couch with this one. I could say more, but instead I’ll simply give you a moose and some Halo above, and let every minute you have be spent watching sound design videos:
http://designingsound.tv

Checkerboard Step Sequencer V2 from Josh Silverman on Vimeo.

Shall we play a game?

Have your checkers chops ready, because Josh Silverman’s Checkerboard Step Sequencer, a tangible interface for music, will test both your game mettle and your grooves.

Built with the open source coding tool OpenFrameworks and Ableton Live as sound source, the checkerboard fuses computer vision technology and … well, some beats.

This video should make obvious the relationship between the position of the checkers pieces and the noises they represent and trigger. It’s still a work in progress, but for now I won’t subject you to the cacophony that is the sound of an actual game of checkers.

Aside from the kick drum, which just keeps pace on every beat, all other drum samples are triggered off the board.

In this version, I’ve implemented a Mute Region that surrounds the board. When the application sees activity in the mute region, it disables the updating of the sequencer. This way, my hand doesn’t trigger a cacophony when I move the pieces.

More technical explanation on Josh’s blog:

How it works [prettyextreme]

As it happens, you can meet this project in person if you’re in the NYC area. We’ll be hosting Josh on Sunday night at Handmade Music, at Culturefix’s Lower East Side. 4:30-6p is an open lab, a chance to check out this project and others (including MeeBlip!), followed by cacophonous demos and raucous music starting at 7p.

Detailed Lineup; Handmade Music site
Event on Facebook

Historical precedent: It’s fairly hard to top John Cage and Marcel Duchamp playing chess, with or without sonification, but apparently sonified they were:
John Cage Playing Chess [Uncertain Times]

No information on what the chess game sounded like, however. Backgammon, anyone? Thanks, SkyRon, for the tip! Also, from 1997, a grid game with Toshio Iwai and Ryuichi Sakamoto; thanks, Ctrlsave. (Interesting to reflect on how much easier this is to do in 2010, thanks to more accessible software and greater, cheaper horsepower.)