Delicate and dense, melodies and sounds from church contexts, found sounds of bells and voices, are set against crisp, sharply-solid, forward-driving electronic beats. And then, there are the visuals: an archaic architecture of mystical symbols and three-dimensional, evolving forms interpret the music in visual form.

Swedish-born artist and technologist Eric Wahlforss, in other words, has been busy. As the artist Forss, his album is an app, appropriately for someone who is the co-founder and CTO of SoundCloud. Eric showed me an early build over cheeseburgers. It’s reactive, perhaps, more than interactive, but there’s still a chance to use your hands to rotate both visuals and music, a bit like picking up a sculpture and viewing it from different angles – though with the added element of sound. What you get is a sense of an interwoven visual and musical world, and an aesthetic vision that Wahlforss has pulled together.

From the man who built the world’s largest online recording business, it’s little surprise that recording features prominently, in two threads:

Recordings of strings, choirs, organs and ambient noise from church concerts which have been cut up into fragments and rearranged into a new mosaic of music, and recordings of wooden, stone and metal objects which make up the beats and percussion. These are the plosive, rhythmical noises that provide the link between the traditional to modern electronica.

That musical combination sounds to me familiar, though also clearly comfortable to Mr. Wahlforss. The collaboration is especially intriguing, though, as a Viennese graphic designer (Leonard Lass) and German computer graphics artist (Marcel Schobel, Untouch) collaborate to produce an audiovisual experience. Berghain, that cavernous church of techno (and occasionally more experimental sounds), seems an appropriate setting in the city that also played home to SoundCloud’s founding. (The fact that the former power station has the acoustics of a church doesn’t hurt, either – even if it’s ill-suited to denser music for the same reason.) Ecclesia will get its launch across media: live show in Berlin, app on iPad, album. For now, you can hear the tracks streamed via – of course – SoundCloud, even shared directly from Ableton Live.

The live show premieres May 2 in Berlin at Berghain/Panorama Bar, with the app out the same day. The album itself releases on June 12.

http://forssmusic.com/

Visuals come from Untouch (Marcel Schobel):
http://www.untouch.fm/
…and Leonard Lass:
http://depart.at

One perhaps unexpected impact of technology has been to change the way we think about ourselves and our experience. Recording equipment – from photography to phonograph – has given us a new sense that memory itself might be fixed, unchanging, an accurate record of an unmoving truth.

Except, of course, neither the recorded object nor the thing it is recording ever quite seems to work out that way. (Ask your local theoretical physicist, or for a more localized, humanized, sociological view, any loved one.)

UNRAVEL is an installation that uses just those sorts of technologies to construct a narrative, and push and tug at that narrative. And if you don’t like it, well, that’ll impact the video, too. (Just complain via Twitter, and you’ll make the narrator “increasingly insecure.” As a blogger, I can relate.)

Combining record playback, a robotic band contributing incidental music, and a set of interactive dials, the installation recounts a story with mechanically-reproduced soundtrack, as the audience adjusts what happens. It’s all clear in the extended video:

More information:

UNRAVEL opens to the public on 20 April – 7 May as part of Glasgow International Festival of Visual Art at Arch 24/ SWG3.

UNRAVEL is a collection of devices making up a gallery-based, reactive sound installation, through which the audience will attempt to unravel the truth about The Narrator’s life by playing records from his collection.

When we tell the story of a memory, how much of it is true and how much is shaped by who we are talking to? Once we’ve told the story many times, how do we even know what is true any more – what is constructed and what actually happened?

The installation is the work of Edinburgh based arts collective / experimental pop band FOUND, whose members include Ziggy Campbell, Simon Kirby and Tommy Perman and Glasgow-based author and musician, Aidan Moffat best known as one half of the band Arab Strap. FOUND and Aidan Moffat are signed to Glasgow record label Chemikal Underground.

At the heart of the installation is a vinyl record player and ten 7” records of familiar singles from pop music’s heyday. Visitors to the gallery are encouraged to select a record from the collection to be played. As soon as they drop the needle on to the record the installation springs to life. The vinyl controls a series of acoustic, self-playing musical instruments positioned throughout the gallery which soundtrack the story as the narrator recounts a memory he associates with that record. Each 7” record represents a different memory, but unlike conventional vinyl recordings they sound different each time they are played.

Just as a real narrator alters the way they tell a story depending on their mood, audience and context, the memories embodied in the installation will distort, evolve and warp depending on external influences: the time of day, the size of #UNRAVEL’s audience, the local weather, and what people are writing about the installation on twitter from moment to moment.

A year in the making, #UNRAVEL is the first collaboration between FOUND and Aidan Moffat and represents a major new body of work for both. The project required Aidan to write 10 short stories with multiple variations of each, to be soundtracked by a total of 160 new musical compositions by FOUND.

With Investment from Creative Scotland’s Vital Spark programme and New Media Scotland‘s Alt-w Fund with the support of the Centre for Design Informatics, Glasgow International Festival of Visual Art, SWG3 and the University of Edinburgh.

Funny side note: I recall some evenings out drinking with Scottish people that also questioned the boundaries of what is real and not real and the imperfection of memory, though more in a performative, real-time sense than in an interactive installation. (I was a willing and active participant, so I’ll not hold this experience against the fine countrymen and women of Scotland. Indeed, I hope to toast with these chaps next time I’m in Glasgow. I, of course, do not condone such behavior, and you will find that by contrast, this particular interactive installation has no ill health effects that I know of.)

For something completely different, here’s a beautiful set of instrument robotic solos incorporating acoustic instruments, plants, and bamboo:

A composition for plants, yangqin, bamboo robot and robotic chimes, Three Pieces is designed as a collaboration between robots, traditional instruments, and living things, housed in Victorian Palm House of the Royal Botanic Garden in Edinburgh. A traditional Chinese dulcimer is played by a robot with many bamboo fingers while the surrounding foliage hides an ensemble of robotic chimes. Despite being separate individuals, the robots communicate and perform together. The robot performers are conducted by all the living things in the Palm House. The moisture content of the soil changes slowly as the plants absorb water, while on a much faster timescale, the temperature changes in the building as animals, including humans, move about. The installation detects this living presence in the Palm House and the music changes accordingly. The robots react to humans, but their mood alters with the plants. For more info visit… foundcollective.com

Previously:

Plant-Reactive Robots Play Bamboo, Chinese Instruments at Royal Botanic Garden, Scotland

Reconceived Acoustic Music on an Interactive Table: Etiquette in Edinburgh

Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

Sometimes, to quote Depeche Mode, words are very unnecessary. Instead, lose yourself for a few minutes in the vibrating mathemagical lands of numbercult, audiovisual immersions in which sound and geometry fuse in a strange, abstract dance.

Their most recent creation, found via Richard Devine’s prolific Facebook wall and posted earlier this summer, explores an actual audiovisual sequencer. See it at top:

Connected is a graphical/musical sequencer system. a three way flow of information, between graphics, sound and external triggers shape the composition. Recorded in real-time.

But actual functioning interfaces aside, I’ll leave you with some other video clips that traverse similar territory, these syncing up separate visual and audio systems.

These folks make music, too – have a listen to their album, at bottom. And that shifts to body candy, as in, for your butt, with danceable grooves.

All three of the above videos combine vvvv – the Windows-only, graphical patching environment for powerful 3D effects – with Ableton Live for sound.

But lest you think it’s all abstraction, have a listen to their excellent dance release on Bandcamp. Downloading:

http://www.numbercult.com/

By the way, ever wondered what visual software people are using? So did we. Don’t miss this look on our sister site, Create Digital Motion, including where vvvv fits on the spectrum:
What Visual Software Readers Use: Some Clear Favorites, Plenty of Diversity, in Census Results

“The hills are alive /
with the sound of browsers”

Ever thought you’d make sounds in a browser, or have new ways of visualizing music playback? It’s happening, with builds of Firefox anyone can download.

Work to make browsers rich with sound synthesis and visualization continues. “Compatibility” isn’t really an advantage yet, because Firefox is the only browser with support, and only in the next version, though that could change in the future. And yes, Flash is capable of some of this, too (though not real 3D), with 90-95% saturation, conservatively, of computers. But if not compatibility, what these experiments do represent is what happens when someone working on a tool (Firefox, in this case) really commits to making sound a priority, and supporting free standards and developer tools (an emerging standard API, WebGL, Processing.js, etc.).

In fact, it’d be great if this occurred everywhere: if you’re making a platform, make sound a priority, and people will do mind-blowing stuff with your platform.

Among the latest fruits:

1. 3D eye candy. Charles Cliffe has a psychedelic visualization of sound playback. The JavaScript nuts are also proceeding to do more things with their language than most would deem possible, even moving DSP calculations to JavaScript code. I remain a bit skeptical there: the question to me isn’t whether JavaScript is “fast enough,” but whether native code is faster or simply the better tool for some jobs. Details below.

2. Patching in a browser – with a Pd clone. Chris McCormick is porting a subset of basic Pd objects to the browser. Now, one side of me wonders whether Pd is the best choice; it’s a somewhat idiosyncratic, if powerful, language for describing sound patching. But on the other hand, I could see this being fantastic in teaching and sharing: put basic patches up in a browser, let people play with them live, then build more advanced tools (with greater hardware access and external support than is possible in a browse) in the traditional Pd tool. As I keep saying, I think there’s far too much partisanship in the discussion (“Browsers for everything!” / “Browsers are useless!”), far too little thinking about how the browser and the desktop tool are more powerful together.

Web Audio Data API – Pure Data and Processing.js from David Humphrey on Vimeo.

Check out:
mccormick.cx/dev/webpd/
wiki.mozilla.org/Audio_Data_API

Also — heck, I may try this out in workshops as soon as next week. The browser could build a basic language for music and visuals in Processing and Pd, then robust performance tools could be built in the native tools, with quite a lot of compatibility between the two.

3. Actual standards. The W3C, the standards body behind HTML, has added this discussion to an Audio Incubator group. (It’s been incubating for some time, but maybe this will help something actually hatch.) Now I’d just like to see these things in Chrome/Chromium, too – I wonder if anyone’s up to a test build, as the standards adoption discussion continues. A number of readers have pointed out that MPEG4 had a specification that included, wholesale evidently, Csound. But this process seems more organic to me – you need actual tools and real-world experiments to evaluate the validity of something, not just standards on paper.

Putting the Awesomeness in Context: An Appeal

A side rant, though: why do Web geeks only care about what happens in the browser? It’s funny to me it seems that outlets like Slashdot jump on stories like browser-based tools, but ignore exactly the same ideas if they’re in a separate app. That’s not a criticism of the Mozilla crew or these brilliant hackers – this is what development is all about, pushing your tools to the limits. But if there isn’t a broader recognition of the value of what you’re doing or why you’re doing it in the first place, there’s a danger that unsustainable tool fetish will miss the point. That is, synthesis in the browser is excellent, but if people don’t understand the value of the synthesis itself, we have a lot more work to do.

Even the tools themselves need a context. It also JavaScript is amazing, but so are tools in Python, Java, Scala, and so on… and some of the enduring power of C still shows here. Browser powers are cool, but the OS is just as important – performance of Firefox would be heavily dependent on support for OS-native, low-latency audio outputs, like JACK on Linux. (Yes, it’s open source, so you can go do it yourself. No, I have no idea how to build Firefox for JACK – maybe a reader does?)

I’ve still yet to see a compelling explanation of what the browser really is, and what’s possible with its interface paradigm. That should be a fascinating discussion, actually, especially with the radical transformation of the browser, particularly as players like Google make it the central aspect of TV-watching or tablet experiences. But the discussion is only really interesting if you don’t start out with the value as a given. For instance, if browsers become a bigger part of what we do, is its simplistic tab metaphor really sufficient? If browsers simply bundle a set of native tools, are there ways “standalone” apps might adopt similar, standards-based approaches?

David Humphrey argues that part of the value here is the view source concept, but the Web has had the same empowering influence on sharing, collaboration, and reuse with platforms other than just JavaScript. The browser itself is a largely misunderstood piece of technology, partly because users (understandably) focus on their experience, and doesn’t pay attention to which aspects are delivered by the browser, the OS, or some other piece of code.

Oh, side note: this isn’t about “the cloud.” The cool stuff here is happening on your local hardware, period. That’s what makes it fast, and that’s what makes it work for audio, and your local machine is getting cheaper, cooler, and less power-hungry all the time. New DSP and floating-point capabilities in devices like tablets could make sound more powerful and flexible than ever before – provided people work out how to maximize, not squander, those capabilities.

So, here’s what I’d like to ask: what form will the standards discussion take? And how can these larger discussions – many of which transcend the discussion of any one tool or standard – find a forum?

Behind the Scenes, More Info

While you ponder that (and I’m open to suggestions), here’s more reading for you:
Experiments with audio, part X [Dave Humphrey's increasingly-awesome blog]

Previously:
Real Sound Synthesis, Now in the Browser; Possible New Standard?

More details on the first example, and how it was built (Minefield is Firefox 3.7):

All runs in real-time with Javascript, WebGL and HTML5 only (uses Minefield Audio build) — no browser plugins are used.

This demo combines the CubicVR 3D engine on WebGL (www.cubicvr.org) with the Mozilla HTML5 Audio API (hacks.mozilla.org), Processing.js (www.processingjs.org) and BeatDetektor.js (www.beatdetektor.com)

Mozilla Audio API is used to sample the HTML5 audio tag on the page, this information is processed by BeatDetektor.js which produces timing information for the Processing.js real-time canvas textures and the CubicVR.js procedurally generated WebGL scene using them.

The camera is set to free roam a simple chase pattern with a probability to follow a nearby cube (fully automated).

Available online at:

http://cubicvr.org/CubicVR.js/bd3/BeatDetektor3HD.html

or if you have a Float32Array enabled Minefield build:

http://cubicvr.org/CubicVR.js/bd3/Bea…

you can find more info about audio api-enabled Minefield builds at:

https://wiki.mozilla.org/Audio_Data_API

You can also feel free to chat with us about the Audio API via the #audio channel on irc.mozilla.org

Enjoy! And yes, I’ll have to work out a more beginner-friendly, here’s how to do this post.

For many musical artists, the frontier of reactive, interactive music has been a long time coming. RjDj, an app which we first saw as a series of interactive musical scenes on the iPhone, is now being expanded by its developers into a mini-ecosystem of interactive music tools for creation and distribution. I don’t think it’s likely to work for everyone – some artists may have their own ideas about how to distribute such work, or may take this concept in different directions for performance. But it’s nothing if not stimulating to watch.

Here’s the basic formula:

1. Authoring: The RJC1000 looks like an MPC-style drum pad, but it’s actually a tool for assembling scenes. (Currently Mac-only, but I believe built in Python so it could show up on other platforms.) In “expert mode,” you can write your own modules in the free patching environment Pd (Pure Data).

2. Playback: The RjDj on the iPhone and iPod touch, and now the Voyager app on iPad (top) provide “player” mechanisms – whether for your own performance, or as a way to distribute your work as “interactive albums” to listeners/users. Check out the RjDj apps for more. (Little Boots and AIR each did their own, exclusive artist app based on the same tech.) Lest you think this is all about Apple platforms, there’s also an “RjDjroid” Android app.

3. Network of music: The RjDj network is a means by which artists, for free, can get their work in the hands of users.

4. Open-source tools and patches you can steal use: Not everything the RjDj crew have done has been open sourced, which I have seen generate some disagreement. But there is some very nice stuff in the developer sandbox, a public API, and best of all, a brilliant Composers’ Pack chock full of Pd goodness. In fact, it’s probably the most useful set of Pd patches I’ve ever seen, a whole mess of useful macros for building usable instruments.

Check out the Voyager app, in particular. I love that it breaks out of the traditional interface paradigms — even those RjDj themselves are exploring with the MPC-style authoring tool. Voyager reimagines music listening not in lists of tracks or grids of beats, but in the liquid, alien landscapes of your dreams.

RjDj isn’t enough for you? With the open source tooling behind RjDj, there’s the possibility for an “ecosystem” beyond just the RjDj universe. (Pardon the use of that ecosystem word again. Let’s translate to “other good stuff could be happening,” or “general hoopla is involved,” or “see also: awesomeness.”) It means if you like the idea but not the implementation, you can try your own ideas. And it means, thanks to Pd’s ability to run just about anywhere (thanks to support for ARM architectures and Linux and not just the narrow world of x86 on Mac and Windows), the future isn’t dependent on one company’s vision. It can depend on yours.

I do think RjDj and the Pd development team that worked with them deserve some credit here, though. For the first time, we’ve seen an interactive “label” that’s devoted to making music dynamic and changing for the listener. If they’ve got it right, that means there’s far more to come.

And seriously. Go “steal” those patches. That’s the whole idea.

Here’s a look at the authoring app, in video demo form, running on the Mac:

Michael Breidenbrücker of RjDj has more to share with CDM. He writes us:

Scenes produced with the RJC1000 can currently be distributed into two different apps that have different interfaces and use cases. This is a strategy that we will increasingly apply. I think with the RJC1000 we have a very powerful authoring tool for reactive music. Music which is produced with the RJC1000 can be listened to or consumed in different ways on different RjDj apps. For example we implemented a very simple paging interface in our iPhone app because we think that is a good interface for the device. On the iPad though we did implement a drag and drop and twist and push interface for scene playback. The important point for the producer is that he only produces once and cans distribute his creation into several apps. The next app we are working on is a music game…

With the RJC1000 we are releasing a free authoring tool which is the tool to get your music into the RjDj network http://more.rjdj.me/make-music/ . We will continue development on new apps even on different platforms and the RJC1000 will increasingly be our main tool for that.

The RJC1000 runs on Pd (like everything we do which means it is a very powerful tool. You know we have started off with Pd as our main authoring tool but soon discovered that it is way too abstract for many musicians and their approach to music. I think with the RJC1000 we have found an elegant way to get musicians and producers on board and at the same time keep Pd very close.

Got questions? Made anything cool with those Pd patches and macros? Let us know.

In the thrilling conclusion of our chat with Boing Boing’s Xeni Jardin, Matt Ganucheau and I explore deep thoughts about the roles of interactivity and adaptivity in music and game design — then attempt to dance in giants Katamari Damacy hats. (Note the use of the word attempt — those things were more than a bit tricky to move in. Hilarity ensues.)

Prior to leading a dance dance Revolución, we talk a bit about the ways in which game design relate to gesture in musical interface and how musical scores could become non-linear. The gesture issue really goes well beyond games to the fundamental question of how to relate to music physically — and, in a way, awkwardly-dancing musicians may be a fitting metaphor. Or parable. Or something or other.

We do it all for you.

I really did find this a fascinating way to promote discussion, so if you’ve got suggestions for future broadcasts, I’d love to hear them. You can even think of new silly things for us / guests to do.

See also, related:
Troels Folmann on the boiled waterphone-style instrument, sound design inspiration
Troels on “micro-scoring” adaptive music

For part the first:
GDC: Music, Video Games, and Interactivity – Chat with Boing Boing Video

Matt Ganucheau and I got to sit down with Xeni Jardin of Boing Boing Video during the Game Developer Conference to discuss some of the potential for interactive music in games. Matt is a composer, sound designer, and educator, talking about how he’s encouraging his own students to think about adaptive music in new ways, combining Max/MSP and a Space Invaders clone built in the Unity Game Engine. (See our story from earlier this week.)

I talk a little about my sense that new tools could expand the range of possibilities in game music. Right now, the two major game engines are the AudioKinetic Wwise and Firelight fmod engines, each of which do have potential of their own – and continue adding features for more interactive sound scores. Each got some significant, flashy new features announced at GDC. But I was especially impressed by the use of Pure Data (Pd) in a custom implementation inside the game Spore. That allowed the compositional team to produce a truly generative musical score (led by legendary composer Brian Eno, with EA’s Kent Jolly and composer Aaron McLeran). I hope we see more of that in the future. Starting of students doing it themselves (with Max in this case) is not a bad way to start.

Boing Boing has more video of us they’ll be posting soon – including the embarrassing but diverting footage of us dancing around in Katamari costumes.

I’m new to this speaking live thing, but hope you enjoy. My favorite part was getting questions going. Livecasting is something we’ll try here soon.

A big thanks to Xeni and the talented Boing Boing TV crew for inviting us on and running a great show! And thanks to those of you who came on the chat rooms to talk to us – actually a lot of terrific questions and comments we weren’t able to address.

Music in Video Games, a conversation with Peter Kirn and Matt Ganucheau [Boing Boing, with lots of download options for YouTube, MP4, iTunes, etc.]

Dreaming of a future in which music, instead of just being rendered audio files, arrives in fully generative, interactive form? Albums might “listen” to the world around you, and listeners could record their own alternate versions of music and share with others.

RjDj, the generative mobile music platform for Apple devices, realizes that future right now, instead of at some nebulous time in the future. In addition to the iPhone, you can make use of a second-generation iPod to use it. (You’ll need a headset with a mic; I have one by Griffin I’m testing.) And the RjDj folks have a whole bevy of significant updates to share:

• Free downloads (limited time): All three RjDj releases are available now for free. That includes the RjDj app itself (from which you can now grab and share releases), as well as RjDj Album (with a selection of generative/interactive/reactive releases) and the new RjDj shake.
• Download “scenes”: From the beginning, we knew that RjDj was imagined as a platform for other people to release interactive music. Now you can download scenes for free or fee. (Paid scenes currently redirect to the browser, but with iPhone SDK 3.0, you’ll be able to buy right from the app.)
• Share recordings: Because RjDj-generated music is controlled by the user and often records from the environment, the music may sound different each time. You can now share recordings with others from the device and the new social site.
• RjDj.me community: The RjDj folks have built a little community where you can share your favorite scenes and upload recordings, and keep track of scenes coming out from other artists.

By the way, with all of these releases, I know there’s some confusion. RjDj is the main app — the platform from which you’ll be able to grab scenes in the future. RjDj Album is a collection of “player”-style RjDj scenes — ones we have seen released previously. RjDj Shake is a newer, accelerometer-powered set of scenes. It comes with the amusing admonition that you should “PLEASE TAKE CARE, DON’T HURT ANYONE AND DON’T SMASH YOUR DEVICE.” If you missed Shake before, it now also has the recording sharing features. But the main RjDj app is the big release going forward.

We’re also awaiting news on more “sprints,” community-driven development efforts for making new scenes. And if you’re a fan of this sort of thing, RjDj isn’t the only game in town. RjDj itself is powered on Pure Data, the open source patching cousin of Max/MSP. Pd is making its way to other mobile devices; I even have it working on the BUG Labs gadget. At the same time, I’m curious to see if the RjDj gang can succeed in building a platform for lots of people doing this sort of work.

Regardless of how you look at it, more mobile generative / responsive music is most definitely in your future. Stay tuned.

Elsewhere:
Free trip into the rainbow vein: reality-enhancing iPhone app RjDj free with social update [Boing Boing Offworld]

RjDj.me community site
RjDj 0.7 out now [RjDj News]

RjDj [iTunes]
RjDj Album [iTunes]
RjDj Shake [iTunes]

Be sure to see our previous interview with the creators:
Exclusive RjDj Interview: Interactive Music Listening, Everywhere You Go

Also, expect more iPod/iPhone news and hands-on’s soon — I’m way behind, but let’s assume that means the best stuff will rise to the top.

It’s something we take for granted: listen to a track, and it starts at the beginning and goes to the end in a fixed length of time. Wonderful things can be done with music that way, and it’s the traditional model of composition and recording. But the equally old, if not older, tradition of improvisation suggests that music doesn’t always have to be linear. It can be specific to a place, a time, a mood.

Now that the technologies that power music creation can fit on a standard mobile device, listeners could have music that’s as pliable when they listen through headphones as it is in a studio when it’s created. Music could respond to the environment you’re in, and sound different each time you plug in your earbuds. That presents new challenges for the people making the music, but it could be an entirely new medium.

The team behind RjDj, a reactive and interactive music platform for mobile devices, don’t just want to wait around for this to happen. They’ve got it up and running right now, in a just-released application for iPhone. I spoke via Skype to the team in Vienna as a crowd of enthusiastic programmers and volunteers hacked away in a massive patching and music-making fest they call a “sprint.” More sprints are planned around the world, and the entire project is being built with the open-source visual patching environment for multimedia, Pd (Pure Data), cousin to Max/MSP.

If you’re ready to geek out with Pd, in fact, you can have at the patches yourself. But even if you’re just an interested musician, there’s plenty to watch here. It’s about more than just the software (Pd) or device (iPhone) – indeed, this app alone is likely to extend to other devices. What it’s really about is a new approach to how to listen to music, how to develop musical tools, and how communities own and share that work.

And, oh, by the way, team members have been behind everything from the port of Pd to Linux to the launch of Last.fm – the latter sold to CBS as one of the hottest musical properties on the Web, and a personal fave among the CDM team. So don’t doubt for a second that this group can drive some serious change.

If you watch just one video, check out the one above – especially about halfway in, as it starts to get juicy. Even for someone who’s been doing this for a while, watching a tiny device respond to the environment is magical.

Gunter Geiger is a technologist and advocate of free software. He puts his code where his mouth is: he ported the multimedia tools Pd and GEM to Linux a decade ago, helping launch the free community around them. Now he’s harnessing Pd again – but it’s not just about the software, he says.

Gunter: It’s not about if it’s Pd or not. The idea is to be able to create music in a different way. Instead of doing a fixed track, you do something interactive. These kinds of programs have been around for ages, but it really didn’t catch up on the music market.

The important thing is to get momentum behind it — not just one guy doing this thing. [And] it’s not only having people to create things, but [expanding] the audience, which is very small. What we really want to create is some momentum, and a scene. We hope that we get artists who make new [work].

You start to create different forms of music. Some of them are more like classical interactive things. Others are using the sound input a lot. It’s really a very open world, and the good thing about using Pd in there is that basically you can do everything. It’s really so open that we don’t know what’s coming out of it. We’re just trying to improve the things, and all the people working here are constantly changing their scenes and making them better.

I asked specifically about whether they were working to standardize these interactive structures, but Gunter emphasized they’re mainly keeping it open. And that’s important to note here – the actual “scenes” are completely open-ended, limited only by what you can do with the target hardware and the objects in Pd approved for the project.

You have a sort of chicken and egg problem. It’s really hard to make a structure before you know what these things look like.

What he could promise was growth – and on more devices than just the iPhone.

Now it’s the iPhone. In a year, I hope … more. There are sprints happening everywhere.

Michael Breidenbruecker initiated the project, now joined by a team of musical and technological thinkers and coders, with a select group of backers with experience in new Web projects for music. As one of the original co-founders of Last.fm, Michael is familiar with what a platform can do for music listening. He’s committed not only to the free, open source model for the project, but to transforming the way people think about music making – even those who aren’t musicians themselves.

Michael: I think we are all just starting at this, in a way. The scenes that we have right now have a [deep] effect. If you’re producing music, maybe you remember the first time you played with an echo or with a delay. At least for me, I spent ages pushing the button and going "poo, poo." For many people on the street, or what I experienced at Burning Man [with the RjDj], people were really going crazy because it was the first time they had this interactive or reactive experience of music. Music was not just something fixed or something they could consume, but something they could influence.

Ever since Burning Man, I’ve known we have a reason to be on the planet, to do what we’re doing.

When you write about this or talk about this, it’s really hard for people to understand what it is. As soon as you put headphones on them, they actually get it.

Michael says that to make that connection with listeners, they first have to connect with artists – which means their challenge is not only evangelizing interactive and reactive music, but on the tool side, making Pd’s power more accessible.

Michael: The big task now for us … the couple of sprints we’ve had, and the people we have involved already, is just blowing my mind. And that’s something that we really actively want to push. In the next couple of months, we’ll have to do a lot of work on the composing interface. Pd is a bit abstract for people who are used to other production software. So that’s our job in the end.

There are a lot of people standing behind Pd, but in the art scene it’s totally … inadequate. If RjDj can bring the whole idea of Pd and interactive music closer to the market, that would be really great.

We are trying to keep it as free as possible. It makes a lot of sense to use and reuse things. All the stuff that’s done should be provided to the community. We have it all on a public SVN [Subversion, a free, standard server tool for tracking changes to code and collaborating on projects]. All we can say to the artists is, if you don’t want to share it, don’t put it up there now.

Artists selling RjDj scenes could be very possible in the future – and wouldn’t necessarily conflict with providing open-source patches for those savvy enough to run Pd. But so far, Michael says the project is driven by imagining a new shift in music more than a new business model. And, interestingly, the ideas behind RjDj predate the now wildly-successful Last.fm, which was acquired last year by CBS.

I had this idea for a project ages ago. I started to work on this thing in 99. In 2000/2001, I started up Last.fm. When I saw what was happening on the iPhone, I said maybe it’s time to start [this concept] up again. I tried to get a bit of structure, all of our investors.

Michael: To be honest with you, including the investors we haven’t yet said, this is our business model, not at all. We just know we’re working on something new which we think has potential for the future. We’d [be happy to] manage to get the idea of reactive music booted, in two, three, four years even, to see a shift in the music market. So people who are now listening to MP3 songs could also be listening to reactive music, and something that’s customizable, highly dynamic, and personal. We would certainly try to be the driving force in that development, that market. Right now, all we can do is try to make the product as good as possible, that the person from the street would be able to listen to it and enjoy it, and artists would enjoy doing scenes.

I can tell you how the idea was born. It was actually one of these stupid things. In the 90s, people started to wear earplugs to raves because they were so loud. They had to protect their ears. Then I saw people who actually had microphones on their ears, and I thought, wow, that’s crazy. They have a microphone and a headphone, so what they hear is filtered. I found out that’s not what it was; it was a binaural microphone. I thought it was like sound glasses. I thought that was great. Eyeglasses for your ears.

Changing the medium, Michael notes, does transform what music can be – for musicians, as well. They have hooked up RjDj to a P.A. at parties, taking care to avoid feedback since RjDj scenes often make use of the microphone as an input. Even networking is potentially on the table, for collaborative scenes, though no development has taken place yet. (Pd supports networking, so that’s definitely something that could happen, with control data beamed between different devices running RjDj.) In the meantime, RjDj poses problems you might not even have imagined.

There is another interesting topic which we haven’t solved yet. You have the RjDj scene, and your sound experience is in the boundaries of that scene, but what you’re actually hearing is totally individual. That’s something that you can record on the rjdj. What do the artists — if a listener makes a recording of his scene which is very private, it’s his voice, his environment, what about that? Who’s the owner of that?

It’s scalable uniqueness — the RjDj scene, you can copy it a trillion times, it’s still the same, it’s a copy, but the individual experience listening to it. and tha’ts something traditional music is fighting. You have a digital copy of a recorded track. The musical industry wanted that scalable; that’s why they made that digital format, the digital CD. So they had this tremendous scalability, but then they started to realize that the uniqueness [is lost]. That was one reason why we did Last.fm at that time.

[Then] people started to realize they make music with objects. An instrument, it’s an object. But with digital music, music in a way became totally objectless. Look at the iPhone – in the end, it’s so miniaturized. RjDj is really bringing it back to the object. You know how this glass sounds [if you strike it], but with RjDj it sounds different. People begin to experience objects in a different way.

RjDj received its first official release today on the iTunes App Store. Software is available for free, or as an “album” for US$2.99.

Where to go:

RjDj Site / About / Blog

How to Create Scenes (And incidentally, you can work on scenes with a laptop even if you don’t own an iPhone. Testing on the device is, of course, very nice – fellow iPod touch users, I’m working on finding out how that mic solution is coming for us!)