Musicians and artists now have the power to fuse visuals, sound, and interaction, to make a spectacle, an album, and a game all at once. But with the blank canvas of three different media before you, what form should that fusion take?

Space shooters with pounding electronic beats behind them have cleared some of the way. Now it’s ambient music’s turn. In the game Osmos, you become a mysterious particle, floating amongst gravity wells in various fields of material. By carefully navigating, applying just the right vector force to move through the shifting landscape, you merge with other particles and escape to safety.

http://www.hemispheregames.com/osmos/

The move from “shoot stuff” to “move” or “eat” seems to be rising in popularity, with games like fl0w and Spore’s initial “cell stage” encouraging nonviolent navigation. To me, there’s something happening to the zeitgeist, perhaps a renewed awareness of cosmic (micro- or macroscopic) being, and of movement that draws on free-floating physics.

Even amongst a wave of games in this mode,when you actually play Osmos, you realize that it is something different and special. The design makes ingenious use of different kinds of movement and pacing through its different modes, at one point calling upon you to hurtle around a black hole, then move at nearly imperceptible speeds through a seemingly impossible-to-traverse petri dish of massive particles. No less than a shooter, it connects to the id and survival instinct. Pac-Man, the most successful arcarde game of all time, and one of the few that sucked in men and women in equal measure, was noted for its emphasis on eating as the mechanic. Consuming stuff appeals to everyone.

Of course, this is on a music site, and with good reason: what makes Osmos work is that Osmos is musical. It’s immediately beautiful and delicate, a perfect aesthetic union of the texture of the music and the on-screen arrangements of particles. More importantly, the music is woven directly into game play, providing subtle cues for dangers, and underscoring the pace of gameplay. You can only solve a level by managing speed and motion, and the music helps provide both the literal indications of speed and help your head get into the right zone to lose yourself in the world. If blips in early arcade games helped create a zone of play trance, now we have spectacular ambient soundtrack of music by Loscil, Gas/High Skies [Microscopics], Julien Neto, and Biosphere.

The music isn’t simply a beautiful soundtrack to the game. The game really feels like an extension of the world of the music. Put it all together, and something magical happens in this $10 game: you hear the music in a new way.

I spoke to the lead designer behind the game, programmer/animator Eddy Boxerman, along with musical-sonic collaborator Mat Jarvis aka Gas aka High Skies.

Ten minutes. Four or five kids (or adults). Make a song. Go.

That’s the idea behind the Youth Music Box, developed by Silent Studios and Chris O’Shea. (Our friend Chris you may recall from various interactive projects and the blog pixelsumo; he sends this project our way.) The software is build in openFrameworks, the C++-based creative coding environment for artists.

With keys, drums, and yes, even a scratching DJ-style interface, the music box brings together kids for quick music making, inspired by the phenomenon of musical games. The experience is guided by genre, with some effort to make sure whatever they do sounds good, but it’s extraordinary how effective it is at conveying the experience of the successful jam. It’s a bit of a confidence builder, in other words, for a group musical experience, perhaps more so than those ear-splitting, cheap plastic recorder consorts I recall from my youth.

And oh yeah, those kids look super cute once they get rocking out. (See video below.)

Youth Music Box Experience from Silent Studios | Resonate on Vimeo.

All of this raises some fascinating questions, and not always with the answers you might expect. In a normal musical ensemble, you begin sounding like crap, amp up difficulty, and eventually sound something like this – at least as far as coherence goes, assuming you’re not aiming for experimental free jazz. But with the addition of technology, whether musical games or the presets on our favorite synths or the quantization and beat-synced loops of our sequencers, it goes something in reverse. You start out sounding like this, pull apart the mechanisms that make you sound a certain way, and eventually find your way to your own personal approach. (And at some point, you get some of the readers on this site, writing code to produce their own sounds and musical structures line by line.) In fact, one could imagine scaling difficulty of even this particular setup, gradually adding greater musical freedom and taking away the “training wheels” of all the rules-based restrictions that make the results sound a particular way.

It’s still anyone’s guess exactly what fruit parent technology maker Stantum may soon ship, but the JazzMutant Lemur touchscreen is getting a component update soon. Nat Lecaude points to a quiet MySpace post from JazzMutant with the details of a coming manufacturing change.

“…the next batch of Lemur will feature the latest generation of our multi-touch technology: better optical performances, higher precision, greater accuracy and responsiveness. It will be clearer and have brighter colors. We plan on launching the new Lemur in early October, and of course we will keep you updated as we get closer to launch date. We once again thank you for your patience, and look forward to sharing the excitement early October!”

It’s actually quite remarkable to me that JazzMutant remains alone in this market – and with Stantum focused on the mass market, that could be the case in the future, too. The issue is that doing multi-touch well still costs some money. There are basic implementations on computers that are cheaper, but that restricts you to a few computer models, because slapping multitouch overlays on displays remains pricey. So HP can get a few computers to the mass market, but not without cutting some corners and not even on that company’s full range. The iPhone has brilliant multi-touch control, but a mobile form factor makes this much easier.

I’ve got some videos demonstrating what’s possible with the Lemur coming soon, as well as some notes on how the software has evolved since I first saw it in its initial release. Even if you don’t want or can’t afford a Lemur, it’s a fascinating demonstration of interaction design and OSC, with lessons (inspiring and tough alike) for other interfaces.

Photo by Rainer Knobloch for CDM.

The step sequencer. The sixteen-pad drum machine. The piano roll. The step sequencing piano roll. The waveform editor. The multi-track recording. Live music is a dynamic and changing phenomenon, but much of our technology assumes fairly predictable interfaces with time. Elysium, which we saw early this week, breaks out of that mold by defining generative systems that live on a hexagonal grid or “honeycomb.” There’s lots of great reader feedback on that story, and Elysium’s creator wrote in to talk a bit about what influenced him.

I want to highlight two sequencers that you play as if they’re games. (Just don’t play a Vulcan – they always win.)

Robots on a Grid

Al-Jazari is named for a 13th-Century scholar and musician who apparently invented an entire band of water-powered hydraulic robotic musicians with more than fifty facial and body movements per song. (Okay, that clearly deserves a separate post later. So, our Western education is so eager to avoid the achievements of Arabs that we skipped over the fact that he basically invented Disneyland in the Middle Ages.)

Al-Jazari in the 21st Century iteration takes the idea of robotic agents and builds a sequencer around them. Creator Dave built a grid on which you can give the robots symbolic instructions (like up, right, down, left), selected from a gamepad. Each grid square represents a note, with pitch modulated by moving bricks up and down. Like Elysium, the music is generated as events are triggered on the grid. And like Microsoft Research’s (non-musical) game Kodu, the gamepad and a set of symbols make what is essentially scripting easy and transparent. (Few would likely call this “programming” because it doesn’t look scary, but that’s what it actually is.)

Al-Jazari is open source, built in the elegant coding language Scheme (a Lisp dialect) atop a game engine called Fluxus. Dave has extensive documentation on its development, and not only the code but even the textures and models. You can use this yourself on Mac and Linux, but it’ll require some messy compiling. (Thanks for this link, MattH – this is layered with things that blow my mind!)

Al Jazari [pawful.org]

reacTogon

Mark Burton’s reacTogon was the influence for Elysium. It’s a “chain reactive performance arpeggiator” – that is, it takes the usual, static, repeating patterns of an arpeggiator and turns them into something altogether different, by allowing events to transform dynamically in two dimensions across a hexagonal grid. The interface is a multi-touch controller with physical objects, so there’s a tangible element, as well.

Looking at reacTogon alongside Al-Jazari really demonstrates some of the advantages of a hexagonal grid versus the more traditional square grid. (And if you think about most musical applications, most of what we have is relatively non-dynamic right-angle grids. There’s movement, but only left to right, with start/stop or loop points. One exception: Follow Actions in Ableton Live.)

Al-Jazari requires movement only to tiles with adjacent edges. reacTogon, since it tiles hexagons, has six adjacent tiles instead of four. It can also map a harmonic table, as other musical hexagonal grids do. Now, that’s not to say reacTogon is better than the other – on the contrary, it demonstrates that just one choice – a grid of squares or a grid of hexagons – can create very different musical possibilities. So even if you’re not musically impressed by these examples just yet, think about the possibilities here. We’re still early in software design and musical interface, so early that something as simple as a simple geometric pattern can become an entire composition.

That’s something to ponder on the eve of the music manufacturers’ trade show.

(If anyone has more documentation on Mark or his creation, let me know.)

pauk on pages from Pauk on Vimeo.

Part of what sets apart the open source monome controller instrument (cdm tag | site) is that, despite its minimalist grid of pads, it really behaves like an extension of software. That grid can be thought of as touch-ready pixels. Using Pages, an app developed by phortran that allows you to overlay different, switchable functions on the same controls, you can navigate through different capabilities or move through the structure of a composition.

Pauk has put together a video showing what this feature is all about. He’s got quite a bag of tricks taking advantage of Pages, including:

• Skipping clips in Ableton Live
• Sequencer controlling effects
• Sequencer controlling a RhythmBox, with two levels of velocity, a la vintage Roland gear
• As a MIDI keyboard controlling effects or a soft synth instrument
• Running external applications, including a Tenori-On-inspired “Boing” app

So, that’s the technology involved. But watching the video got me thinking – what is he really doing? That’s a deeper question raised by a lot of what’s going on in computer music. In some ways, there’s a parallel to DJing, in that he’s manipulating music at a large scale. But, of course, the musical result is different, because of the array of techniques – well beyond what most DJs would use. It’s sort of like live composition, at a higher level. It is a performance, but it’s not a performance at the level traditionally associated with playing an instrument – we’re controlling more elements at once, by making them as a set of interactive compositions.

In other words, this is “interactive music,” the term most associated with game music.

Food for thought. (Hey, people did beg for something to take their mind off that demo video.)

For more monome video goodness, here’s the awesome Edison: