By the 1990s, the notion that computer software could be a means of delivering interactive digital art more personally was enjoying a Renaissance. This was the age of the Voyager CD-ROM, which catered to new multimedia PCs and Macs with titles from the likes of Laurie Anderson and Morton Subotnick, the decade in which Brian Eno released Generative Music as software and Monolake – before Ableton – included a Max/MSP patch with an album. But the reach of these experiments was doomed to be relatively limited.

Now, of course, things are different. First, we saw some widely-available audiovisual toys, coinciding in particular with the debut of the iTunes App Store. But now, those fairly one-dimensional experiments are beginning to blossom into something else. When these particular gadgets and app stores are forgotten, the question is whether those aesthetic adventures, the personalization of the digital art experience, will endure.

Joshue Ott, co-creator of Thicket for iOS, points to a review of that application on Apple’s App Store. “I always want to touch the masterpieces in museums,” a user says in that review. “I’ll use Thicket instead of getting arrested!”

“It’s the democratization of our own performance works,” muses Ott. “It’s a way people can play along with us,” he says. “We’re constantly creating processes to create sound and music; it’s what we’ve done for ten years or so,” chimes in Ott’s creative partner, Morgan Packard. “Now people can own the processes, not just the results.”

Ott and creative partner Packard have long each been visual and music performers, respectively. That meant what it has traditionally meant: the artist gets up in front of an audience, the real work hidden behind an onstage laptop. With Thicket, by contrast, the raw materials of that performance became embodied in the software itself, and thus in the hands of the audience, who can double as performer. At first, this software included only a simple mode or two, each with a specific sound, musical ambience, and visual look. Even in those versions, Thicket made some appearances in an occasional gallery show or performance – the app you download could also be the art.

As Thicket has added modes, though, it has evolved in a kind of platform of its own. Ott and Packard produce new works that can be distributed as in-app purchases (more on how they contend with that in a bit). The sum total of those modes has created a massive audiovisual playground, a compendium of ideas and aesthetic.

A new version released this week adds three new modes, seen in the video at top here, building atop modes added in late December. For the first time, you can use Thicket on an iPhone and not just an iPad; it’s a Universal app. Screenshot sharing is available, too. But the addition of all these modes, unveiled with a “reboot” of the app at the end of last year, represents a shift in thinking as these artists and developers reevaluated what it was they were doing.

“We found that the modes were becoming so different, so much deeper,” says Ott:

We were having such fun using it as a big sketchbook that we decided to ditch the ‘rotate to change modes’ system so that we could handle lots of modes, rather than just four or five. The modes in Thicket reboot are completely new, and each one is a lot more complex than the older modes. They’re all very different, and each have separate methodologies behind how you control them. We’re playing with different concepts in user interaction design, searching for the right intuitive feel to make a true audiovisual instrument (among quite a few other things).

In other words, if you haven’t played with Thicket lately, it’s a different animal. It’s a Long Play album to the first version’s single cut. The work is immersive, too; you can transmit video output via HDMI or VGA on the iPad, and get up to 1920×1200 HD output, with no menu intervening. (One of the many significant current drawbacks of Android for the moment for artists: the move to a soft menu on Android tablets means menu detritus that never goes away. Artists were intensely relieved this week when Apple’s new iPad kept its signature, dedicated hardware menu button.)

Morgan Packard says he has some strong feelings about why this kind of experience has value in his work:

I’d say where we both overlap is our shared interest in how abstract sound and picture, plus interactivity, all can work together. Thicket is a bit of a research sketchbook for us. There’s something very magical about just twiddling your fingers and having sound and visuals spring to life. Frankly, we don’t entirely understand this medium yet. But we like not knowing, trying to understand it in different ways.

The gestural thing is huge with us, and is at the core of what thicket is. It’s partly why I’m a bit resistant to the idea of layering features on to Thicket. Of all the different people who give us feedback, I get the most gratification from parents of special needs kids.The non-fiddly, large-motor interaction style is very accessible to a huge range of minds and hands. I want to explore this more, to give people new ways of feeling expressive and creative with movement and gesture. In my mind, that’s what’s really special about what we’re doing.

The duo did get a chance to try the app with people with different user needs. Ott explains:

Morgan and I actually toured a special needs school earlier this year and observed autistic kids using Thicket. A very special music teacher is using Thicket (among a couple of other technologies) to teach the kids music, and had found that it seemed to really empower them. He offered to let us visit and we happily agreed… really really amazing experience.

As Subotnick hoped years ago in “All My Hummingbirds Have Alibis” for Voyager, the distribution of art as software can create a new kind of “chamber” art, in which the work is personal, enjoyed by a few people. It can be a family or a couple of friends on a couch.

Of course, somewhere in all of this, these artists are looking for revenue in order to be able to devote the massive amounts of development and testing time the application demands. (Neither has quit day jobs, which means finding a way to devote resources to development.) Thicket easily climbed in download counts, but only after the application was made free. In-app purchases have been a tough mountain to climb, but have at least allowed some revenue to trickle in; the challenge was finding a way to make them appealing to users, says Ott:

I think in general people hate In-App Purchasing (IAP), because, in general, I think IAP is usually not handled so well. We have thought a lot about how to show people exactly what they are buying before they buy it, and I’m really pleased with what we’ve come up with. Every mode in the new Thicket has a pre-recorded “demo” of one of us playing the mode. Before you buy a mode you can watch this demo, learn what the mode can do, watch someone use it in an interesting way, and decide if that’s something you’re interested in or not. You can of course watch the demos even after you’ve purchased the mode (and the free Sinemorph mode also includes a demo as well). The demos are a great way for us to show users different tricks and techniques.

So the reboot is really about making Thicket a platform rather than just a single art piece. Something that we can keep adding to (with a financial structure that makes sense for us to keep adding to). Something that we can start collaborating with other artists on – we are talking to a couple of different people about releasing modes within the Thicket system. So yeah, that’s what the platform part is. We’re really excited about it, and what it will become in the future.

But these concerns aside, the developers aren’t just creating Thicket for users; they’re building something they use themselves. As Josh explains:

I’ve performed with Thicket now a couple of times, once at the excellent SONiC festival, and another at Issue Project Room in a program curated by Ryan Lott (AKA Son Lux), and have started to really feel like it has the potential to be a new form of audiovisual instrument. I want to see more stuff like it- things that generate graphics and audio intertwined, and I want to continue to explore these relationships in different ways. I’m actually pretty excited about performing with Thicket more, and I think doing so will push it even further in that direction.

“That’s really what an audiovisual instrument is to me,” says Ott. “It’s something that you can bang on and make something interesting, but you can touch it subtly, as well, to shape it, to express with it. That’s what I want to make. We’re right at the beginning of that exploration, and I think we have something that is a promising vehicle for it.”

You can try out the new Thicket now, as seen in CDM Apps:

Thicket @ CDM Apps
[Says iPad, is actually now Universal. PS - music and beauty flow from my fingers all the time - no app needed - but I'm glad now the rest of you get the chance.]

Turning music and sound into three-dimensional worlds often yields something that fields like a trip through space. But this feels like a real trip. Through pulsing, glowing starfields, “Versum”‘s audiovisual movements are brain-bendingly transformative. Artist Tarik Barri has created an integrated world of sound and image that makes the interface and the compositional realms seamless. It seems as though this really is a musical universe, through whose harmonies of the spheres you can fly like. Boldly going, indeed.

Ingredients: Max/MSP/Jitter, Processing, Java, SuperCollider, GLSL [the 3D shading language], and … some serious skill and time, I imagine.

The work has been in development for some years (not surprisingly, given the results). But it surfaced again as we brought up the 3Dconnexion SpaceNavigator hardware as a practical controller for 3D. See Create Digital Motion:
Look at Me, I’m Flying: SpaceNavigator Hardware + Blender

Tarik’s work resurfaced after a presentation in the UK. Reader janklug writes:

I’m just back from the M4_u Max/MSP/Jitter conference in Leicester (was great, btw), where Tarik Barri presented his project ‘Versum’, both as an installation and as a performance.
The user (and in case of the performance, Tarik) navigates through this incredible 3D-space-sequencer-universum with the help of a SpaceNavigator; glowing objects floating in this space produce sound, and as you approach them, they even give this nice doppler effect…
It was totally amazing to be able to float between pulsing rhythm-planet-objects and shiny drone-beams; navigation was easy and natural. Tarik uses a combination of Processing and Max/MSP; don’t know which one the SpaceNavigator is connected to.
Having tried this, I immediately ordered one; I think it also could be a great interface for M4L…

More information:
http://tarikbarri.nl/projects/versum
PDF documentation [2009]

Significantly, it’s really the act of flying that controls the music. That remains interactive, but it’s the movement through the three-dimensional space that determines what you hear. As the artist explains:

This virtual world is seen and heard from the viewpoint of a moving virtual camera with virtual microphones attached. This camera, controlled in realtime by means of a joystick (or any other kind of controller) moves through space, similar to how first person shooter games work. Within this space, I place objects that can be both seen and heard, and like in reality, the closer the camera is to them, the louder you hear them. So when the camera moves past several visual objects, you simultaneously hear several sounds fading in and out. Consequently, the way the camera travels past them actually causes melodies and compositional structures to be seen and heard.

The visual position of each object coincides with the panning of its sound: objects to the right of the camera will also be heard on the right, and those behind the camera will be heard from behind in case a surround speaker setup is used. This principle also applies to the Z-axis, meaning that sounds can be heard coming from above and below if the speaker setup supports it.

That’s the essential question, to me, when looking at 3D environments for music. What about the dimensionality will interact with the music? Is it something spatial, or will there be other sorts of interactions? (New Zealander-turned-Berliner Julian Oliver worked extensively with game engines, for instance. One solution for him was modifying the “gun” in those games to be an implement for doing things in the space, turning swords into plowshares after a fact by making the gun produce music rather than kill virtual entities.)

So, now you’ve seen some of the technical demonstration. But Tarik uses his work as an environment in which to make audiovisual performances. Here’s what some actual live playing looks like, in a beautiful, meditative piece called “Eleven”:

In fact, the biggest challenge to me of a piece this awesome is that you want an immersive environment, not just the small, rectangular screens that are often all festivals and venues can afford.

Holodeck, anyone?

More:

Graphics are good. Graphics are shiny. But when it comes to reality-bending, emotionally-immersive, perception-shifting power, look to sound and music.

At least that’s the feeling you could get after playing Dimensions. Following their reactive music tools and Inception dream states for iOS, RjDj have turned their mind-altering sonics to gameplay. As with previous releases, these tools are powered by the open source visual development environment Pure Data. Pd engineering wizardry here meetings the compositional and sound design prowess of Hans Zimmer.

You can see a bit of how the musical world works in the teaser video above, and the music sound design video below.

But we wanted quite a lot more information. So, CDM got RjDJ’s Rob, Joe, and Martin to share some detailed thoughts on how the game experience is put together and how it works.

The App

RjDj Team: Most games require your full attention when you play them. You either live your life or play the game. Dimensions is different. It’s designed to be played in parallel with your normal life.

Gameplay is intertwined deeply into your daily life. Some dimensions unlock if you are physically active and others unlock if you are quiet. The app automatically detects what you are doing and syncs the game to it making use of every possible sensor on the iPhone.

You stay immersed in the game by listening to augmented sound and the voice of Emily from Mission Control. She guides you through many exciting challenges like collecting Artifacts and avoiding the dreaded Nephilim.

With Dimensions we are very interested in creating a gameplay experience which is between the device based focus of a casual game and the passive use of listening to music. Its a game which you play by listening – a game that place in parallel to your everyday life.

The Tech: Reading Files

We built our own version of readsf, rj_readsf, in order to be able to read compressed audio and make the samples available for processing in Pd. One advantage of readsf is that possibly lengthy audio assets do not need to be loaded into memory. If memory is limited, especially when Pd may be running in the background, limiting exposure to system memory warnings helps keep the app running and the music playing. Given that compressed audio is roughly ten times smaller in size than uncompressed audio, and that audio assets make up the majority of the size of the entire app, it is a huge benefit to be able to deliver and read compressed audio assets directly, without the need to decompress in memory or onto disk. Dimensions requires that several dozen such players be open and viable at any time, and special consideration was given to concurrent behaviour. rj_readsf can loop a file when it gets to the end, and it indicates with a bang when a file has been loaded (an asynchronous operation) or the end as been reached (in the non-looping case). rj_readsf is built on iOS standard APIs and can read any file format that iOS can.

Ed.: I’m waiting to hear if rj_readsf will be open-sourced. The issue of reading files is one we’ve had around libpd recently. While their rj_readsf sounds great, my sense is the best long-term solution will be a similar object that is independent of the APIs of any one OS, so this same set of problems may need a different solution for the open source community more generally. (Building such a tool is absolutely possible, though it might require more effort.)

The Music, and How the Music Plays with You

The music of Dimensions uses various different techniques from straight sample playback to audio analysis and synthesis:

Realtime manipulation of audio input from the mic:

This is perhaps the most recognisable technique we use. We process audio from the iPhone microphone live in many different ways. It’s kinda like a feeling of being inside the music.

The key thing we do with effects is attempting to analyse the environment of the player / listener and then making appropriate things happen within the effect. For instance, the Flux Dimension features a filterbank on the mic input. We analyse the incoming audio from the players environment and make the filter frequencies change as events occur ( either due to pitch changes or onsets ) this gives the impression that objects and activity around the player is somehow “playing” the music.

In the Ghost Dimension there is an effect which records audio whenever it detects an event, then scrubs repeatedly forwards and backwards through the sample using granular techniques stretching it out in time. This manipulation accentuates the textural and pitch based qualities of the sample as it repeats and works well with the atmospheric music Hans Zimmer composed.

Dynamically-controlled stems:

All the Dimensions use stems and hits from a conventional sequencer in some way, re-arranged live on the device relative to how the player is interacting. These stems were mainly composed in Cubase and Logic.

For example, in the Kinetic Dimension we feed accelerometer data from the device into Pd and drive the music from that. The player hears more energetic beats when they go for a run, but if they stop at the lights to cross the road, the drums immediately drop away. This was achieved with a large number of hits with all the rhythmic sequencing happening in a hybrid reactive / generative way live on the device.

In the Tranquil Dimension, the music introduces more stems the longer the player is quiet. If they make too much noise the music “shrinks away” from them and becomes quieter. If they stay in a Zen like peaceful state, the music grows into a kind of crescendo of serenity.

Reactive synthesis:

We often control parts of the music by doing a frequency analysis of incoming microphone audio from the device and then using those frequencies to determine the notes synths will play within the music. The Travelling Dream in Inception the App uses this extensively. Tranquil Dimension in Dimensions also uses onset and frequency changes to trigger synth melodies in the music.

The synths we use range in complexity from very simple additive synthesis to some great synth patches from the rjlib by Frank Barknecht and Andy Farnell.

Generative approaches:

There are some sections within Dimensions which are generative. These play back prepared samples as well as triggering onboard synthesis. They also feed the results of this through various live sampling and glitching patches. They are governed by various sets of rules which have various long term parameters, like adjusting to the intensity of the audio environment of the listener, or how dense areas of music have been around the present time.

Sample triggering:

Ghost Dimension uses a simple but effective technique of triggering samples from the music on onsets in the environment. This can cause some real jump out of your skin moments. We combined this section with a randomised very short delay on the mic which acts almost like a resonator, turning the mic sounds into creepy atonal pitched noises.

Sound Design

The main hub section in Dimensions, called the Launch screen, acts as a entry point to your augmented adventures. It also displays all available Dimensions via the floating tile icons.

Visually, these represent a snapshot of your previous experience using your location at that time. Sonically we wanted them to have an aura or energy from the Dimensions themselves.

SoundCloud examples:
Dimensions Sound Design Example by rjdjme

Example of using mixture of synthesis and samples to create user feedback when interacting with Dimension icons in the game.

Map Tile Down: several recordings of a synth in Pure Data that is played when the tiles are touched. Each one is slightly different due using two detuned oscillators.

Map Tile Open Only: a sample from Logic Pro for the woosh sound when showing the information view.

Map Tile Click: a sample from Logic Pro for touch events.

Map Tile Open: recording of how it sounds when put together.

Sound is a mixture of samples and real-time synthesis. The energy sound is made using two oscillators (one detuned) to create some modulation for a glowing effect. Added to some harmonics to make it more of a beam sound and some chorus and reverb. The open tile is made in logic, when closed it’s the same sound but reversed and pitched down in Pd.

We wanted the tiles you tap on to feel like each Dimension has some sort of energy radiating out.

Here’s a sense of how the sound design works in the game:

It’s an ambitious app, and the whole cost is US$2.99. I guarantee it’ll change your world more than a latte. (Well … unless we’re talking a really crazy latte. And that might not be legal.) As sometimes-CDM contributor Jaymis Loveday notes, there are terrific choices in coloring Google Maps, and how modes change based on ambient sound and motion.

Requires an iPhone 3GS or better, or third-generation iPod Touch or better, or an iPad.

Dimensions @ iTunes Store

More reading:
Dimensions Augments Reality Purely Through Sound [Leigh Alexander, one of my favorite game writers, for Gamasutra
The Roundabout Tapes – RjDj now plans to game reality with sound [TCTV] [Techcrunch EU]

Drasko Vucevic, Toronto- and Santa Monica (California)-based sound designer and artist/composer, is apparently not only interested in playing alone. His upcoming interactive installation at Toronto’s Royal Music Conservatory will have an audience jamming along live via Twitter. And the artistry is crowd-sourced, too – with a range of artists already onboard, Drasko is calling on musical and visual artists (read: you) to be involved with sounds and visuals.

Drasko has sent along extensive notes, so I’m going to let him speak for himself:

I am working through both Drastic Music and Eksperimental (my companies) to create an interactive installation experience for this year’s TEDx Toronto conference taking place at the Royal Music Conservatory.

I am also doing an interactive music performance – an audio visual performance with a complete 3d/visual journey, which allows the audience (physical and digital) to collaborate with me by triggering audio and video elements in real time through their tweets.
So far, I have a few great artists contributing their time to create some beautiful visuals for this first of its kind interactive real-time jam.

Installation [Call for Audio]:
The installation concept is based around redefining collaboration. We are doing this by using real-time data (motion, color, sound, light) and tweets relating to TEDx to trigger, control, manipulate and compose audio elements on our back-end audio library (ableton). This is all happening through Processing, Max 5, Arduino, Ableton.

The massive back-end sound library contains loops, melodies, soundscapes, fx, you name it – produced by some great artists. The beauty of this is also that artists which have never before collaborated, will be remixed and mashed up solely by the audience, in relation to key words, discussions, movement, etc.

So far, the artists contributing audio content are:

Yoko K
Trifonic
Richard Devine
Drumcell
Audioandroid
David Della Santa
Darrin Wiener
Audionerve
Box of Toys
Lodewijk Vos
Matt Davis
Adrian Ellis
Andrew Lauzon
Drasko V

Performance [Visual Call]:

As I mentioned, my performance will be very interactive, musically and visually. Both audio and video elements will be triggered based on tweets in real time. I have some great visual artists contributing their time, such as Murat Pak, Yongsub, Charlie Vicetto, etc, but am looking for more, to create elements for the performance. They would of course get the great exposure of TEDx brand, be mentioned everywhere online, and will be in the final video spread throughout blogs once we launch the digital music version.

So how does the call for works … work?

The TEDxToronto conference is on September 23. Here’s how all the pieces come together for that and how to submit:

Musical system uses a massive library of sound structures – loops, melodies, fx, soundscapes and more – triggered and manipulated solely by tweets (relating to TEDxToronto) and motion, color, sound and light within the RCM venue.

Over 12 compositions will be recorded on the day of the conference. The arrangement, structure and sounds used will depend only on the key words used in tweets, the types of emotional replies, and physical interactivity within the venue.

Beauty of having some great artists be remixed and mashed up by the general public, in a very subliminal way. (again – Through their emotional replies, and physical movement)

This posting is a call to artists who may wish to apply to contribute their audio content and be a part of the soundtrack we will create that day. They should contact drasko (at) drasticmusic (dot) com with a link to their portfolio and we will take it from there.

Our installation progress may be followed on my personal site (drasko-v.com) or via Drastic Music or Eksperimental blogs.

We plan to expand the installation idea and bring it online for an ever-changing musical universe manipulated by truly organic methods (digitally and physically).

Interested to see how this will all come together. We’ll be watching. If you submit, and if you attend, let us know how it goes.

More on Drasko:
http://drasko-v.com/

You’ve heard the lamentations before: album art died with the move from the large canvas of the LP vinyl record to the CD. Well, eat your heart out, LPs: album art is back, it’s interactive, and it’s trippier than ever. Just ask Flying Lotus.

Fieldlines is a free Mac and Windows application that creates an “augmented reality” experience for your computer. Based on the exquisite cover art by Leigh J. McCloskey for Flying Lotus’ upcoming album “Cosmogramma”, the organic visual pattern animates to movement tracked from your computer webcam, accompanied by the sounds of the record. (Harpist Rebekah Raff’s delicate textures dance through the air as it moves.)

Interactive artist Aaron Meyers did the programming. The work combines imagined magnetic fields with free code framework OpenFrameworks. I asked Aaron about how he put the work together. He responds:

Late last year, I was writing a piece of software with Processing that I was using to make images and animation using simulated magnetic forces. The magnetic force algorithm was actually based on some code I discovered in Toxi’s onedotzero identity project. So anyway, I was playing around with that stuff and making stuff like this:

So a little earlier this year, I was talking to the Warp guys about some of the stuff they wanted to do for the Cosmogramma release. They showed me the cover artwork and at the time, I was looking some pictures of solar wind and the Earth’s magnetosphere (good images at the wikipedia pages for solar wind and magnetosphere) and it kind of clicked for me that the magnetic field line stuff I’d be playing with before related to the cover artwork in what I thought was a really intuitive way. So I hacked together a quick prototype in OpenFrameworks and they were really into it and we just went from there.

More on FlyLo’s record as we get closer to launch. I can’t bear to tease the thing any longer. In the meantime, you can grab Fieldlines and play right now, free:

http://flying-lotus.com/fieldlines

I have to say, too, for all the hype about tablets and mobile devices (some of which I propagate myself), here’s a gorgeous, free toy that takes advantage of the larger, brighter screen and greater processor muscle on your laptop or desktop. (Now, if there were just easier ways to get these kinds of creations on big-screen HDTVs.)

The album launches Tuesday, with a full-length preview available on NPR’s fantastic All Songs Considered blog Monday.

As an addendum to the last story, Ivica Ico Bukvic sends along an example of the [myu] Max/MSP + Unity game engine combination in action. Here’s the surprise: Unity isn’t generating visuals. Instead, Unity simulates ripples created by movement in the space, and builds physical models that are sonified and spatialized by Max/MSP.

Speaking of work involving art museums and the combination of Max and Unity, VJ Anomolee notes in comments his own work with the pairing. Lightbent Synth is an in-progress piece with alternative controllers and sensors that produces sound with a novel visual representation (sound’s very quiet in this preview — more hopefully once it progresses):

Lightbent Synth from VJ Anomolee on Vimeo.

Ivica explains the top work:

This past fall [myu] had seen its first real-world implementation in an exhibit that was a part of the grand opening of the Taubman Museum of Art in Roanoke, VA (http://www.taubmanmuseum.org/). The exhibit utilized [myu] as part of an interactive aural installation titled "elemental." An online tech
demo video of the installation, including written synopsis is available also via Youtube at http://www.youtube.com/watch?v=PA-9BOgc1gk. Below is a brief synopsis of the installation:

"elemental" interactive communal soundscape premiered in November 2008 as part of the Revo:oveR collection commissioned for the grand opening of the Taubman Museum of Art in Roanoke, VA. The Youtube video focuses primarily on the technical aspects of the installation. Using Max/MSP/Jitter, a homebrew IR webcam with fish eye lens and a LED-based IR spotlights, entire 24×36-foot exhibit space is converted into an aural sandbox giving visitors an opportunity to generate and shape the
ensuing soundscape. Positional data of up to 20 visitors is forwarded to Unity3d using [myu] Max-Unity interoperability toolkit developed at DISIS (http://disis.music.vt.edu). Unity is used for physical simulation of ensuing ripples and the resulting data is sent back to Max for spatialization across a 12-channel (4×3) ceiling-mounted speaker array. Driven by communal interaction, virtual ripples refract from each other spawning an algorithmically generated aural fireworks. The exhibit ran non-stop for approximately 5 months until March 2009.

Bonus video below: an early prototype that did include visuals. After days of looking at emulated knobs and faders, it certainly does speak to some of the possibilities for musical interface and expression.