Turning music and sound into three-dimensional worlds often yields something that fields like a trip through space. But this feels like a real trip. Through pulsing, glowing starfields, “Versum”‘s audiovisual movements are brain-bendingly transformative. Artist Tarik Barri has created an integrated world of sound and image that makes the interface and the compositional realms seamless. It seems as though this really is a musical universe, through whose harmonies of the spheres you can fly like. Boldly going, indeed.

Ingredients: Max/MSP/Jitter, Processing, Java, SuperCollider, GLSL [the 3D shading language], and … some serious skill and time, I imagine.

The work has been in development for some years (not surprisingly, given the results). But it surfaced again as we brought up the 3Dconnexion SpaceNavigator hardware as a practical controller for 3D. See Create Digital Motion:
Look at Me, I’m Flying: SpaceNavigator Hardware + Blender

Tarik’s work resurfaced after a presentation in the UK. Reader janklug writes:

I’m just back from the M4_u Max/MSP/Jitter conference in Leicester (was great, btw), where Tarik Barri presented his project ‘Versum’, both as an installation and as a performance.
The user (and in case of the performance, Tarik) navigates through this incredible 3D-space-sequencer-universum with the help of a SpaceNavigator; glowing objects floating in this space produce sound, and as you approach them, they even give this nice doppler effect…
It was totally amazing to be able to float between pulsing rhythm-planet-objects and shiny drone-beams; navigation was easy and natural. Tarik uses a combination of Processing and Max/MSP; don’t know which one the SpaceNavigator is connected to.
Having tried this, I immediately ordered one; I think it also could be a great interface for M4L…

More information:
http://tarikbarri.nl/projects/versum
PDF documentation [2009]

Significantly, it’s really the act of flying that controls the music. That remains interactive, but it’s the movement through the three-dimensional space that determines what you hear. As the artist explains:

This virtual world is seen and heard from the viewpoint of a moving virtual camera with virtual microphones attached. This camera, controlled in realtime by means of a joystick (or any other kind of controller) moves through space, similar to how first person shooter games work. Within this space, I place objects that can be both seen and heard, and like in reality, the closer the camera is to them, the louder you hear them. So when the camera moves past several visual objects, you simultaneously hear several sounds fading in and out. Consequently, the way the camera travels past them actually causes melodies and compositional structures to be seen and heard.

The visual position of each object coincides with the panning of its sound: objects to the right of the camera will also be heard on the right, and those behind the camera will be heard from behind in case a surround speaker setup is used. This principle also applies to the Z-axis, meaning that sounds can be heard coming from above and below if the speaker setup supports it.

That’s the essential question, to me, when looking at 3D environments for music. What about the dimensionality will interact with the music? Is it something spatial, or will there be other sorts of interactions? (New Zealander-turned-Berliner Julian Oliver worked extensively with game engines, for instance. One solution for him was modifying the “gun” in those games to be an implement for doing things in the space, turning swords into plowshares after a fact by making the gun produce music rather than kill virtual entities.)

So, now you’ve seen some of the technical demonstration. But Tarik uses his work as an environment in which to make audiovisual performances. Here’s what some actual live playing looks like, in a beautiful, meditative piece called “Eleven”:

In fact, the biggest challenge to me of a piece this awesome is that you want an immersive environment, not just the small, rectangular screens that are often all festivals and venues can afford.

Holodeck, anyone?

More:

Look at a music software interface – particularly a tracker-style interface – and you might easily see something resembling a spreadsheet.

So, why not gaze into the cells of a spreadsheet and begin to imagine music?

Karlsruhe-based electronic artist and programmer Patrick, cappel:nord, had just such a flight of fancy about office software. He explains:

A spreadsheet could be used as a music sequencer. If you know your spreadsheet software well, the built-in functions can be used. I don’t! I also struggle with the interface The last time I used spreadsheets is 10 years ago or so.

This was the second time trying this, so I make a lot of mistakes. It’s more a proof of concept. This was a 3-hour hack so don’t expect much from the source code. But here it is:

http://www.cappel-nord.de/files/libre-jam.zip

You have to figure out how it works for yourself. I don’t give any support .

I did it for the lulz.

Lots of similar brilliance, sounds, and geekery – like a pixel matrix for Processing, audio players, code, and music – at his blog (not recently updated, but worth plumbing anyway):

http://blog.cappel-nord.de/

Thanks to headchant for the tip!

As it happens, in the first few months of CDM’s existence, I wrote up a little story on what people were doing with Microsoft Excel:
Microsoft Excel for Music: Applications Bizarre and Useful

Applications: building a drum set, a drum machine with sequencer, databases of music, music library tracking, and even a tuning calculator. I’m sure there are more.

I hear these spreadsheets also do something with numbers and finance, but where’s the fun in that?

What an extraordinary thing an interface can be, a map to making music.

A new community-generated album from users of the now-legendary monome grid instrument yields a variety of musical outcomes. The results are instrumental and lovely, breaking off on lots of different stylistic vectors, but glued together by the notion of key and pitch. Let’s let contributor Joshua Saddler explain this – and the holiday album – as well as share some of the music. If you celebrate Orthodox Christmas or more generally the idea of “Holidays” (ahem), or if you just like good music, you can overlook the fact that the latter arrives a bit late on the Western calendar. But both albums are terrific, and I suspect the approach to the music in key, to sharing samples and field recordings, could well be an inspiration in your own music-making endeavors. Sometimes rules are liberating.

If you want to get a jump start on musical New Year’s resolutions, I can think of nothing better. Joshua writes:

The monome community has released not one, but two albums for the holidays. Both are freely available at http://mcrpmusic.bandcamp.com

The first, MCRPv11 (Monome Community Remix Project, volume 11), was released mid-November, five months after the MCRPv10 album (which CDM has previously covered).

http://mcrpmusic.bandcamp.com/album/mcrpv11-all-keyed-up-edition

As with all MCRP albums, there are guidelines and a theme. Participants submitted a field recording and a short instrumental sample in the key of G/E-minor. The participants then chose as many samples as they wished from the shared pool (though they couldn’t use their own samples), and had a couple of weeks to assemble their tracks. Sounds ranged from falling rocks to ocean waves to modular synthesizers to toy ukeleles and dogs barking. From this pool emerged fifteen startlingly diverse tracks.

Have a listen, and head to Bandcamp for downloads in any format you desire:

I appreciate the chance to see Joshua’s process in video:

I’m pretty pleased with how my contribution, “mnml autmn,” turned out:

mnml autmn by ioflow

I sequenced bits and pieces from four samples with Renoise (in some cases using single-cycle waveforms…so it still counts, even if it sounds nothing like the original!), exported sections to loops, and performed them live with rove (http://docs.monome.org/doku.php?id=app:rove) on a monome 128. I recorded and rearranged the resulting segments using Ardour3‘s timeline view. The tracker and the traditional DAW actually worked well together. As I’m the sole Linux musician on the album, composing and arranging takes much longer using free software than more common tools like Ableton Live. Things that took me hours are probably three-click operations in Live. Still, by having to strike out on my own, I learn so many new things each time I sit down to create…it’s worth the occasional frustration at not being able to do things the easy way, using the same process as everyone else.

The second release is the annual Monome Community Christmas Album volume 2, made available on December 21.

http://mcrpmusic.bandcamp.com/album/monome-community-christmas-album-volume-2

This project had much more leeway; no hard-and-fast rules about samples or themes. I ended up forgoing the monome entirely for this album, instead improvising an original acoustic piano piece:

gloria by ioflow

http://soundcloud.com/ioflow/gloria

There were fewer participants for MCXAv2, since it began immediately after MCRPv11, but the quality of the tracks is still extraordinary. Warm neo-retro-loungetronica. I’ll be listening to it year-round, not just in December.

Me, too. And perhaps you, as well:

Thanks, monome-ers!

http://monome.org

Come on – you know that occasionally you want technology to respond when you slap it.

As my sister watched an episode of the television show Quantum Leap, I thoroughly enjoyed watch Dean Stockwell’s character Al give his pocket computer, looking for all the world like a 7″ tablet, little helpful smacks.

SmackTop does that for music. Yes, we hear, ad infinitum, the complaint that laptop musicians simply stare inertly at blue glowing laptops as if checking their email. Now they get to put a little skin in the game, literally. And a version 0.3 update makes this humorous novelty genuinely useful:

Imagine your laptop as a MIDI drum kit. SmackTop is an open-source application for Mac laptops which translates physical motion into MIDI messages. Through real-time analysis of the built-in accelerometer’s output, SmackTop is able to classify four different ‘smacks’. Now you can control your favorite DAW by simply tapping your computer. Slap samples, ping notes and hit record – SmackTop is the MIDI controller you already own.

Try it yourself, free:

http://code.google.com/p/smacktop/ (they miss the obvious name, “SlapTop,” but…)

Got another motion-sensing laptop that’s not a Mac and feel jealous? Maybe someone can port this.

In January, we also expect to catch up in person with developer Raymond Weitekamp and Interface LA, the awesome live performance collective in southern California. Stay tuned.

Slap that laptop, make it free!

Now, a tribute to slaps we love…

Have an uncommon yule with tools and music from the Commons.

That’s the pitch (so to speak) of the Ultimate Christmas Songbook, an iPad app built with 50 Christmas songs and a fully free and open source notation engine. Making use of public domain songs, the number of songs available continues to grow as the community contributes tunes. (Those contributors got the app for free.)

As notation proliferates on tablets, the app also suggests that “commercial” doesn’t have to mean “closed.” The scores themselves are available in open, cross-platform formats (MIDI, MusicXML, MuseScore, and PDF). But by generating revenues, the app can support further development – something that’s often been missing in open source music software projects.

And if you’re looking for a way to help family and friends play music, and they have iPads, the score reading features are quite reasonable. You get lovely display of scores, audio playback, tempo change, transpose, and the all-important font resize with reflow so you don’t have to squint.

The app is on iOS now, but other platforms are planned; an Android version is already in testing. And we hear lots more is coming from MuseScore, too, hot on the heals of a release that earned half a million downloads:
A Christmas update from MuseScore

More resources:
Open source code for mscore at SourceForge
Contributed scores to download
Ultimate Christmas Songbook, US$1.99 at iTunes
http://musescore.com/, software and community, including the desktop software for Mac, Windows, and Linux

For reference, here’s a look at how the desktop software works:

Truly generative musical scores in games have been few and far between, and “music games” has traditionally meant arcade-style rhythm games in which you repeat phrases or whole songs as accurately as possible. Pugs Luv Beats breaks those molds. Part of a vanguard of new gaming creations that generate dynamic music on the fly, it marries grid-based sequencing and resource-gathering gaming, as music making and gameplay blur together. The interactively-produced music could itself become a new way of delivering a musical signature with sound packs.

And beneath it all lurks a free and open source library, libpd – the embeddable version of tried-and-true free graphical music environment Pure Data. (That library is now on GitHub, and vastly updated, by the way, and we’re expecting a book soon from the library’s principle author Peter Brinkmann.)

Oh, yeah, and don’t forget about some seriously addictive gameplay and adorable pugs. I’m suddenly not concerned about the 15 hours Europe-to-North-America travel I’m doing tomorrow.

Here’s what the gameplay looks like, since it’s much easier to see:

Pugs Luv Beats was just approved on the iTunes App Store for iPhone and iPad.

Co-creator Yann Seznec (The Amazing Rolo) is a terrific musician; I just caught up with him in Edinburgh and Berlin and watched him play a homebrewed pig gut instrument with Matthew Herbert for the performance piece “One Pig,” on tour at Berghain. Working with Pd allowed Yann to focus on those musical impulses and not just engineering, and to let him try things he otherwise would never have imagined on a mobile title. So I asked Yann to walk us through how the project was built. He responded with an exhaustively-detailed examination of the evolution of this title, right down to the Pd patches. (Click through for high-res versions.) If your New Year’s Resolution is doing something with patching, you might want to hang onto these answers. Here’s Yann:

The origins of Pugs Luv Beats date back about two years. After making [musical iPhone game] Mujik, Jon (Jonathan Brodsky, aka jonbro) and I were trying to think of other approaches to music mobile app design, and we started thinking more and more about games. Music games, as a whole, are an oddly passive and traditionalist experience – you play along with a premade track, and you are judged on your accuracy and flair (which is strangely reminiscent of music conservatory mindset…). Obviously there are exceptions (RjDj’s Dimensions, Elektroplankton, etc.), but there you go.

Particularly interesting to me was the idea that game mechanics are often very similar to compositional techniques. So for example, when Sonic runs at a normal speed he collects rings at one rate. However when he powers up and goes super fast, he collects rings at a much higher rate. This could be compared to introducing a melody and then speeding it up – and when there are two players, doing this with two melodies. Instant fugue!

We started looking at how we could make a music game where the music and the game elements were fully intertwined and augmented by each other. So Jon prototyped a space shooter drum machine. It was awesome.

To make a (very very very long and boring) story short, our idea and prototype landed us some funding from Channel 4 and Creative Scotland to work on games that focus on musical creativity and composition.

For various reasons, we decided to put aside the space shooter drum machine for a while, and start from scratch. After going through several full prototyping iterations we eventually settled on a core game mechanic that turned out to be in many ways similar to a Tenori-on [Yamaha grid instrument]/Boiingg-style [monome hardware patch] music generation system – in our final prototype, you controlled a series of little dots that moved around the screen, creating loops. This is super fun from a musical perspective because it’s easy and rewarding within a few seconds, and when you have several loops going it can gain some pretty serious rhythmic and melodic depth.

The key from there for us was turning this into a game. We had been using free Internet graphics packs up until then (we hadn’t hired our artist Sean yet) which featured a ladybug, so we had been referring to the main characters as ‘bugs’. During some discussion one of us accidentally said ‘pugs’, and the game idea was born. We constructed a story about pugs and their love for beets (like the vegetables) which create beats (ha!), and how their love turned into greed and got out of control, destroying their world. The game, therefore, is about helping the pugs rebuild their lost civilization by guiding them to create beats. You grow your galaxy by collecting beats, which you do most efficiently when you dress your pugs up in costumes. What’s not to like?

To get to the part that I imagine CDM readers are most interested in, the app development was done by Jon using openFrameworks, [lightweight language] Lua, our own game engine called Blud, and the audio is all done in Pure Data using libpd (through ofxPd). In hindsight we started using libpd really late in the game, just at the very end of the prototyping stage, which was rather silly. Our adoption of libpd basically made our dev cycle about a million times more efficient. My background is as a musician and sound designer, and I have very little coding knowledge. I do, however, have lots of knowledge of Max/MSP, so picking up Pure Data was pretty easy. This allowed Jon to completely pass off all the audio processing (not to mention aesthetic sound design choices) to me, saving him loads of time, giving me direct control over the sound, and letting me test and prototype different approaches to audio within an environment that I knew would be recreated in the game. Also, as Jon mentioned to me recently, by using PD we are able to take advantage of 20 years of audio DSP research and development. Pretty amazing.

How it all works:

The entire audio engine is contained within this patch. Pardon the messiness.

The simplest part of the patch is the “sounds” section, which is used to playback simple sound effects, for the most part linked with interface actions in the game. I did this by creating a very simple patch which plays a sound when it receives a bang. Which sound it plays is dictated by the argument (in this case, the sound of discovering a new capsule). The process for adding a new sound, then, is as simple as adding the sound file to the /assets/sounds/ folder, and making a new instance of “sounds.pd” and naming it the same as the new sound. Jon, in the project code, created a list called “sounds” which is sent into Pure Data. When that list contains “capsule”, a bang is sent into that subpatch, and the sound is played.

A more complex version of what could be done with this type of data is seen in the voice of Mr Puggles, who helps you learn how to play the game. Mr Puggles pops on and off the screen to guide you through the first few worlds, and when he does he send Pure Data a “puggleShow” and “puggleHide” signal. I wanted to give Puggles a funny synthesizer voice that was different every time – dead simple in PD. To do that, I take the puggleShow bang and use it to trigger five more bangs, spaced out over a second. Each of these bangs triggers a random number which is translated into a MIDI note. This note controls the pitch of two oscillators (a sine and a sawtooth), one of which is slightly modified to make them slightly different pitches. These are played through a short volume envelope and a filter which is also controlled by a random number generator. Result? Hilarious beeping boopy Mr Puggles voice, all coming from one bang.

Every time a player buys or selects a planet, a short list is sent to Pure Data comprised of the planet BPM and a random number seed. The BPM is used to calculate delay times and such, and the random number seed is used to create a sort of musical identity for the planet. This is done by choosing a “beat library” and a musical mode.

The mode is created by building a lookup table that chooses the notes from a chromatic scale that would be used in a particular mode. For example, a major scale (ionian mode) uses notes 1, 3, 5, 6, 8, 10, and 12. Each melodic sound library I used is comprised of a full chromatic octave, and the notes that are played on any given planet are controlled by this table. This ensures not only that all of the different sound libraries being played on a planet will be in the same key, but also that a planet will have a strong melodic identity.

The sound libraries in the game are all controlled by the pugs on the planets. As they run around, each time they land they will trigger a sound. The type of sound is dependent on what terrain they are on – thus, if they run through the snow they play a toy piano, if they run through lava a distorted guitar, etc. There are two states of playing the sound, one if the player deliberately tells the pug to go to that tile, and the second if the pug is traveling over that tile to get somewhere else. It’s super easy to do that kind of thing in Pd; just set up two different ‘play sound’ envelopes, maybe a little extra delay or reverb, and you’re done!

The final piece of the puzzle for making the pugs running around into music is to make each tile be a different note. The terrain of each planet is created by making a sort of height map, where different heights correspond with different terrain types (grass, water, snow, etc). This also means that each tile has a unique number between 0 and 1. When the player buys or selects a planet, a giant random number table is generated in Pure Data which creates a number between 1 and 13 for each possible value between 0 and 1. That value is what is used to pick the note of the mode. This somewhat convoluted approach again lets us make sure that each planet will have a unique, but fully reproducible, musical character.

The actual playing of the sounds is probably the messiest part of the patch structure. Purists look away now.

I wanted to make sure this part of the patch was as flexible as possible, so I ended up using the soundfiler and tabread~ objects, rather than tabplay~, which is great in practice though does look rather uncouth. Additionally, I had some limitations imposed upon the structure of the patch – namely, I had to keep the number of tables down as much as possible, to save on memory. So each sound bank has two voice polyphony – there are many sound banks, and the beats and sound effects aren’t counted in this, so that limitation is not really heard in the final product at all. It did mean I had to work out a decent voice allocation system though!

I think my memory issues were probably my only problem with using PD in this project – though only indirectly. As I mentioned, they were hardly a problem artistically, however it took me a while to get used to the idea that not everything I patched on a computer would work on an iPhone. Similarly, I had to be very careful about things like relative volumes. In a generative music game like Pugs Luv Beats, the player could quite easily send 15 pugs running around making sound, which mounts up pretty quickly. It means that all of the patches and sound need to be designed to withstand lots of triggering without distorting. None of these things are problems, really, all they require is regular testing on devices and simulators – something that every mobile developer is already used to.

That’s the Pure Data audio engine in a nutshell. The end result is a flexible and powerful audio engine that sounds really great and is fully integrated into Pugs Luv Beats. The game is a great combination of music, silliness, and strategy – there’s a bit of something in there for everyone. You can definitely just play with the game to make beats, or you can try and collect all of the costumes, or you can try and make the most efficient planet ever. You can also explore the galaxies being made by your Game Center friends, to hear what they’re up to.

The background story:

And, just for fun, a silly promo featuring real pugs. Anyone traumatized by the sight of Pd patches, these should relax you.

More information at the developer site:
http://luckyframe.co.uk/pugsluvbeats/game.html

There are times when something happens that reminds you why you make the sacrifices to do what you do. A real highlight of 2011 for me was Gwydion ap Dafydd appearing with the MeeBlip, our open source synthesizer, baked into a cookbook.

I knew Gwydion had gotten creative in making a housing for his MeeBlip kit, and I knew that it was a book. But then, he opened it up to reveal the MeeBlip’s controls popping out of a cookbook page, with I/O ports conveniently located on the side, and even the ability to remove the panel to get at the board. And then… the pig’s eye lit red to indicate MIDI messages and power, and I was floored.

Synth in a Book [Konkreet Labs, also here in Berlin]

I can share some good news from the MeeBlip project: we’re now shipping a US$39.95 MeeBlip micro kit, an ultra-compact variation of the MeeBlip. It’s in stock in Canada, in transit within 48 hours. I’m especially excited, because the MeeBlip micro is designed to allow MeeBlip users to surprise us. With pins for analog and digital input, you could connect any arbitrary number of controls in any layout. You could have one giant filter knob if you wanted, or turn it into a wearable project in a purse. Or you could just make a nice, little housing and control it via MIDI. (We now provide full MIDI control of all of the controls.) And we’re excited that it’s forty bucks, because it makes a kit highly accessible to hobbyists. A fully-assembled version will be available soon.

Meet the MeeBlip micro: Small, Hackable Project Synth, Shipping Now, $39.95

http://meeblip.noisepages.com/get-one/

We can’t wait to see what you do with it. We think the simplicity of the MeeBlip’s design, its low cost, and its straightforward MIDI operation could mean people will turn the micro into things we can’t yet imagine. (At least, that’s why we designed it that way!)

In 2012, we’ll again be offering MeeBlips in quantity that come with cases, and affordable MeeBlips (and now MeeBlip micros) that you can get pre-assembled, so the need to solder something together won’t keep your synth on a shelf. Now, with MIDI input in place, we’re ready to get back to MeeBlipping and playability. James Grahame has worked feverishly on the engineering of the current MeeBlip generation, and I look forward to us getting to share the work he did, and how he did it. He’s also been working on how to make the thing easier to manufacture and ship, so we’re ready to share that, too!

It’s also been fascinating to watch people use MIDI – and even iPads – to control the MeeBlip.

Gwydion’s MeeBlipBook (MeeBook?) made another surprise appearance (it sure as heck surprised me) in a teaser video for Lemur on the iPad. In case you missed it:

I had been playing the MeeBlip, when my keyboard or laptop weren’t handy, with the excellent Little MIDI Machine for iPad. I’m looking forward to playing with some other non-computer MIDI solutions, too, not just the iPad.

I’ve seen at least one Lemur template in the works, too; will link to those – and perhaps Pd patches and such – when they’re ready.

In the meantime, the fact that the MeeBlip is out in the world doing things and making sounds, and that we’re at last shipping them again, means I actually have a reason to be festive this holiday season. And, MeeBlip, what are you doing, New Year’s, New Year’s Eve?

meeblip.com

If you liked the generative, probability-based sequencing seen earlier this week, here’s another example – and it’s free and open source, so if you do want to pick it apart and you own a copy of Max/MSP or Max for Live, you can. Co-creator Giuseppe Sorce points us to the work:

This is a simple generative music synthesizer built in Max/MSP created by Diego Caponera, Nicolò Paternoster and Giuseppe Sorce. It involves 5 FM generators which play notes randomly based on a root key and intervals defined by the user. It’s an university project made for an exam for Sound’s Science degree ( Math Department of Roma Tor Vergata ).

The software is distributed “as is”, without any warranty, under a GPL license.

Not bad for an exam project. A heck of a lot more fun than a paper (and, believe me, I say that wholeheartedly having done a bit of university teaching).

Grab it:
http://code.google.com/p/smilesynth/

Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

First revealed last month, PreenFM is an open source hardware synth. As the name implies, it’s an FM synth, with some very serious specs: up to six-operator FM synthesis with some nine algorithms, up to 4-voice polyphony (depending on algorithm), glide, selectable LFOs, modulation matrix, and preset banks with SysEx support. It’s all usable via a display and MIDI support.

It’s also fully open source hardware; whereas early efforts often had commercial restrictions attached, PreenFM is free for use under the GPLv3 and Creative Commons. And it’s got a unique platform under the hood: the open source LeafLabs 32-bit development platform gives this some serious horsepower. It’s very much in contrast to the ultra-inexpensive 8-bit brain of our own MeeBlip synth; think of the MeeBlip as an exercise in what you can do with a little two-stroke engine versus the V8 muscle in this. (The creator says the MeeBlip helped inspire his creation – yes, synths are multiplying!)

You may have glimpsed the PreenFM making the rounds online, but I got creator Xavier Hosxe to tell us more of the gory details and share some sounds.

CDM: So this is all based on the Leaf platform?

Xavier: Yes it’s built around a LeafLab board.
I coded a first version on the Maple [development board]; then when they announced their “Maple Mini,” I realized it was going to be very easy to plug it into a PCB.
I’m not directly connected to [LeafLabs]; I participated in the forum and learnt many things from the team.

They are very friendly and helpful.

What was it like working with the Leaf?

The LeafLabs boards uses an ARM Cortex-M3 microcontroller.

It’s a 32-bit chip runing at 72Mhz that can do 32-bit multiplication in 1 clock cycle, has 128Kb of flash [memory] and 20Kb of RAM. That seems very few but it’s not, PreenFM software uses 92Kb for the moment.

LeafLabs provide a Linux/gcc toolchain that allows to develop in your IDE of choice… Eclipse in my case, which is very confortable.

They also provide a strong bootloader and some libraries that worked perferfeclty for my needs : Usart (Midi), I2C (EEPROM), LiquidCrystal (LCD).

What will you get in the PreenFM kit?

All you need to build yourself a complete synth: PCB, screws, resistors, ICs, audio/midi jack, box, 20×4 LCD, encoders, knobs, buttons… even an USB cable [for power].

You’ll also get a Maple Mini board with PreenFM soft preloaded. The Maple Mini is easily updatable, and you can experiment lots of different things with it.

PreenFM C++ source code is available on GitHub. It’s easy to read and modifiable. If you want to see your name to welcome you on the boot screen, go ahead

To build the kit, you only need a soldering iron and some solder.

There will be 2 differences with the photos you can see on the site: the final PCB will be blue (I should receive them next week). [Ed.: See the photo in blue at top; the orange one is included here for variety! -PK]

Xavier also sends along some welcome news:

Here is a sound I can get with my soon-to-be-released StepSequencer feature in PreenFM.

This is a single voice of a simple 3-oscillator voice. 1 very slow LFO + 2 * step sequencer routed to the modulation indices.

PreenFMStepSeq by cdm

Here’s a sound sample:
PreenFM 1 by cdm

http://www.preenfm.net