You might imagine sound in space, or dream up gestures that traverse unexplored sonic territory. But actually building it is another matter. Kinect – following a long line of computer vision applications and spatial sensors – lets movement and gestures produce sound. The challenge of such instruments has long been that learning to play them is tough without tactile feedback. Thereminists learn their instrument through a the extremely-precise sensing of their instrument and sonic feedback.

In AHNE (Audio-Haptic Navigation Environment), sonic feedback is essential, but so, too, is feel. Haptic vibration lets you know as you approach sounds — essential, as they’re invisible. The work of Finland-based DJ/VJ Matti Niinimäki, aka MÅNSTERI (“Mons-te-ri”), the project is part of research undertaken at SOPI Research Group at Media Lab Helsinki. Like some sort of sound sorcerer, the user is entirely dependent on movement, feel, and sound as they move unseen sound sources through space. (More technical details below.)

It’s labeled, as always, “proof of concept.” The creator promises more videos to come; we’ll be watching as this evolves, as it looks terribly promising.

Below, “Tension” is a fair bit simpler, in which users walk through a space and control synth parameters. (“You are the knob,” one might say, though I don’t suggest shouting that at someone you don’t know. They could take it the wrong way.)

More descriptions:

AHNE

This is a demonstration video of AHNE – Audio-Haptic Navigation Environment.

It is an audio-haptic user interface that allows the user to locate and manipulate sound objects in 3d space with the help of audio-haptic feedback.

The user is tracked with a Kinect sensor using the OpenNI framework and OSCeleton (github.com/​Sensebloom/​OSCeleton).

The user wears a glove that is embedded with sensors and a small vibration motor for the haptic feedback.

This is just the first proof-of-concept demo. More videos coming soon.

HEI Project 2011
SOPI Research Group
sopi.media.taik.fi/

Aalto University School of Art and Design

AHNE – Sound and Physical Interaction

Tension

A brief video showing Tension. An interactive spatial sound installation for multiple users.

A person enters the space and a generative sound is assigned to that person. The sound pans around in the 6-channel speaker system following the user in the space.

Up to 5 users can use the installation at the same time. Each person modifies the other sounds based on the distance to the other users. The closer you are to other people the more the tension in the sound increases.

Tension – Sound and Physical Interaction

Side note: watching these two videos makes me want to consult with someone on non-verbal expression, posture, and stage presence. That criticism is mounted at myself – I could use it. Perhaps we need an all-physical, unplugged music event for laptopists, controllerists, and electronic musicians. And I can at least say I’ve had some experience in this, working in the dance program at my undergraduate alma mater, Sarah Lawrence. Anyone game? (Sounds like something we could do while CDM is in Berlin in the fall.)

For their part, the Finnish research facility is working with dancers, along with Nokia Research Center. (Sadly, I can’t find documentation.) But I think interesting things happen when us non-dancers learn movement technique, too.

Digital musician and artist Josh Silverman began the Synplode process with something familiar – a checkerboard. Play a game of checkers on its computer vision-equipped playing field and beats and loops triggered in Ableton Live generated a responsive soundtrack for the game. But as it’s evolved, Synplode has become a general-purpose musical grid. Whether with little robotic insects (the Hexbugs here) or full-sized human persons, the grid can turn any space into a dynamic, interactive dance floor. (I think I may actually prefer those cute little bugs to the people and dancers and whatnot. Robot rave, anyone?)

I prodded Josh to write up more description of what’s going on, so he’s created lots of documentation on the project Website.

The basic interaction:

At the start of the Synplode demo video, it is easy to see that a wave passes over the basic projected grid, flashing one column at a time, each containing 8 trigger regions. When a participant (or microbot) is present on a region, it is activated. When the wave intersects with an activated region, it causes a Synplosion, expressed through a splash of color and a distinctive sound. In the grid, each row represents a distinctive color and pitch or audio sample.

The basic ingredients:
1. Computer vision in OpenFrameworks, the fully open-source, artist-friendly C++ toolkit inspired by Processing.
2. Ableton Live, triggering clips in Set Mode and modulating them with MIDI effects and racks.

For more detail:
How it Works (details, in particular, of what’s happening in Ableton)
Why it Works (some of the thinking behind the interaction)
Synplode Project Page

Josh first demonstrated this system publicly at our Handmade Music series here in New York, and this is just the kind of experimentation and iteration I like to see. Here’s the original, checkerboard version:

There’s a peculiar false controversy going on at the moment over music notation. First, the blog for online (Flash-based) browser notation editor Noteflight introduced a manifesto:
Music Notation Today, Part 1: A Brief Manifesto

The essay by president Joe Berkovitz is a good read, but it oddly makes the comparison between notation and recorded sound, which is a bit like saying a telephone is better than a DVD. One is interactive and intended for human conversation; one is not. So, go ahead and enjoy the copy of Inception that arrived from Netflix — just don’t take it as an excuse not to call your mother. It’s an argument notation will win, to be sure; it’s just not really a very fair fight.

That is, of course, the implication of Berkovitz’s argument, but the failure to state it overtly prompts Synthtopia to run with the comparison:
Does Music Notation Matter For Electronic Music?

Synthtopia’s James Lewin then goes on to make the following argument:

While Berkovitz argues in favor of “looser” communication of music, an over-arching trend in electronic music has been to give you greater and more immediate control over sound.

I’ve heard this before, and it’s worth asking. But I think if you really ask the question, you’ll find that notation isn’t less relevant: it’s profoundly more relevant.

Yes, indeed, electronic music does give composers direct control over sound for solo work. Lewin goes on to say, “For example, it’s fairly routine for composers to create large scale works, such as soundtracks, without the use of traditional notation.” True — so long as they don’t hire any musicians.

Involve more than yourself, and you’re back where you started. Let’s assume, for instance, you want turntablists, samplists, or controllerists. Great! Oh, wait – you might need to tell them what to do. Now, you could try to explain it to them, but the moment you want to provide any kind of structure to the improvisation, odds are you’ll need some sort of picture.

“Some sort of picture” has always been the core element of music notation. The issue of whether this follows traditional 19th century engraving practice is irrelevant – and entirely inappropriate to many forms of music. But if you draw a picture, whether you use a computer to make that picture or not, it’s a score.

Even working alone, these kinds of representations become critical. We might assume that it “marginalizes” notation because computers facilitate solo work. But as we remember the contributions of Max Mathews this week, it’s important to note that from his first pioneering digital synthesis system over half a century ago, there was always the notion of some sort of musical structure. (In Csound to this day, it’s called a “score,” and not by accident.) Whether you notate on a staff, in pictures, or in code, you create a representation of musical structure in time. In a conventional score, that representation is interactive and open to interpretation. Computer programming languages and graphical patching environments give us new ways of doing this. Sharing that code or graphical patch lets us share our ideas with others. And the moment you want someone to perform a physical gesture to make your music, you return to the same set of needs that have driven music notation for millennia.

There are fancy solutions — see this paper, with lots of pretty images of “spectromorphology,” for one — but how fancy it is doesn’t matter. You’ll need something, even if you scrawl on napkins.

In fact, the moment you want to think about the musical structure, you’re likely to use some sort of visual or representational metaphor. Open up any music software program, and these representations are ubiquitous. Waveforms and spectra are also accompanied by piano rolls, graphs, blocks, colors, and symbols. The Ableton Live Session View has LEGO-style colored blocks. Drum machines represent rhythmic subdivision in units derived from centuries of notation; take away even the handy notes and flags Roland added to theirs, and you still see a grid that you could quickly explain to someone who fell through a wormhole from the 16th Century.

If you want to take any one of those patterns and give it to another musician, then you will certainly translate it into a picture. If traditional notation is the most appropriate, you’ll use that. If graphical notation gets the point across more clearly, you’ll do something non-traditional. But that question has everything to do with intention and communication. You might need to adapt the notation to the technology, but that’s always the case. The turntable requires some specialized symbols, but so, too, do fingerings on a woodwind or plucking technique on a harp.

Speaking as a composer, what frustrated many composers in the 20th century with notation was actually the same criticism typically levied against the computer: notation was too precise, too limiting, too entrenched in certain expectations about measuring time and tune. If you really only wish to organize sound in the privacy of your own home, never involving another human being, you might find these attributes of the computer appealing. But if anything, the computer has given us the potential to be freed from these same limitations, by allowing us to quickly create new graphical and textual languages for representing music, and by reassigning time, tune, and timbre to anything we can possibly imagine. In doing so, they present new frontiers for other human beings to improvise and perform live, whether they’re working with another digital machine, their own voice, or a kazoo.

What has electronic music done for music notation? Simple: it’s expanded its necessity, broadened its meaning and applications, facilitated its storage, transmission, and sharing, simplified its production, exploded its possibilities in everything from graphics to interactivity, and freed it from centuries of accumulated restrictions.

My prediction: if you want to look for the growth area in music technology, it’ll be in notation. We’ll see more of what we already have (conventional notation), and a broader category of what qualifies as musical notation – a greater spectrum of notational systems:

• More kinds of visual musical notation. New interactive systems will facilitate explosive exploration of the connection of visual symbols to sound.
• The display becomes a blank page. Tablets of all kinds – the iPad being only the beginning – will adapt computer displays to forms usable in performance. That’ll be a huge boon to conventional notation and new graphical notational systems alike.
• More connected. The ongoing growth of the Web will mean new ways to edit, share, and view notation. Case in point: guitar tab is massively popular as a a search term online.
• More possibilities. Whereas engraving systems restricted notational practice to certain (largely Western) traditions, open-ended computer notation will make it easier than ever to use alternative notations and non-Western systems.
• More people. People will continue to play instruments. And they’ll need to notate gestures for new instruments as they’re invented.
• More improvisation. Written notation and improvisation aren’t necessarily at odds. Any culture with writing will typically make some annotation, no matter how simple, on a score, even if only squiggles on a sheet of lyrics.

The only way recorded sound would make this go away is if recording makes people stop making live music. But recording, for all the times it threatened to do that, hasn’t succeeded yet in making that happen.

In fact, the potential of digital technology for notation is so broad, so diverse, that it almost does it a disservice to put it in one post. So don’t look at this as a manifesto: look at it, instead, as a challenge, to look at new ideas in electronic music in terms of how they use design, visuals, and textual representation to communicate ideas.

Viewing the world of sound through the grand staff is limiting, and for certain sounds, anachronistic. But to cease to view music through any kind of representation whatsoever would mean abandoning musical thought itself.

I love this definition of music notation on Wikipedia: “Music notation or musical notation is any system that represents aurally perceived music, through the use of written symbols.”

The word “written” doesn’t really fit; if it did, engraving killed musical scores and writers stopped “writing” when they bought typewriters. Music notation, like language itself, is fundamentally symbols.

Oh, and by the way – editing and sharing scores in your browser? Pretty darned cool. And if you think Internet access isn’t capable of making revolutions happen? Well…

More exhibits:

What about the vision impaired? Using notation does not require having sight; computers have been a boon to expanding access to notation. The late Ray Charles was a Sibelius user; sadly, it seems Dancing Dots no longer supports Sibelius, but there are other options. The GPL-licensed open source Freedots continues to work with MusicXML scores for compatibility with many tools. Dancing Dots continues a variety of software and hardware tools for varying degrees of vision impairment from low vision to blindness. These also include interfaces that enable other music software, notably Cakewalk’s SONAR. A 2006 overview from the Texas School for the Blind and Vision Impaired discusses some of the research and tools.

What about rote learning? None of this is to take away the power of rote musical learning. But that’s independent from the computer question; rote musical transmission is perhaps the most direct means of communicating a musical idea between people, and illustrates how significant human communication is to musical process. And even through rote learning, I would think you might come to understand certain patterns of mode or rhythm, which means internalizing those patterns as some kind of mental representation or symbol.

Where these cultures have writing, they tend to have some form of notation. So, for instance, in India – even in a culture in which oral transmission is common – notation has been found as early as 200 BC.

The quickest route to expressing an idea remains the gesture of a hand. That gesture may be crudely interpreted through today’s touch displays, but the immediacy remains. Presumably because of some of the device’s limitations, a lot of the experiments with the iPad have involved controllers that operate independently from sound software, like a remote control. Those interfaces, while useful, largely simulate existing hardware controls in a more flexible form, rather than introduce new ideas. But it seems the long-term potential for touch devices is in designs that unite touch, graphic, and sound in a single piece of software, exploring new paradigms for interaction along the way.

Usine is one of music creation’s most surprising secrets: it’s powerful sound software that incorporates creative touch interfaces as a core design principle. And in the video above, it’s running on a relatively cheap PC two-touch display from Packard Bell. Nay-Seven is one of the founders of the Usine community, all while lecturing internationally, and has been pushing the Usine software to its limits.

Here, he tells us about some of his latest experiments, and the potential they hold.

Always looking for a way to use the computer as a real musical instrument, my latest works try to combine graphics and music using a touchscreen interface. The software Usine from sensomusic gives me the freedom to build my own interfaces. Some examples:

Drawing pitch and pan

Here [at top], the purpose is to draw directly some pitch information on the waveform display of a sample. I’ve also added an LFO [low frequency oscillator for modulation]; this way, the drawing can move slowly according to different speed presets.

[At bottom], I play with pan and volume: the x position of the black ball on the lines gives pan information and y the volume. As I’m on working with a dualtouch screen, I can quickly draw some speed changes. Note that this panel is not only for pan and volume; I can also send this drawing to others parameters like delay and filters, here with the << button.

Geometry …or not

This workspace is also dedicated to drawing. I’ve built four layers, each one with its own color and its own sound. The XY position gives the pitch value of the notes and other parameters, like velocity or pan. The geometry provides sequences; lines give a kind of glissando.

Vertical sequencers and Pads

Using the new Matrix module (thanks to Martin Fleurent), I’ve built this vertical sequencer [seen at top]. I like the idea that notes fly under my hands this way. [At bottom], I‘ve built pads for tablet surfing on the “iPad” mode, adding also a drone option.

Multitouch gestures

On the same idea of movement, here are two screenshots of a video illustrating a new patch made by Olivier Sens (the Usine developer). This patch provides multitouch gesture recognition, opening new doors to ways in which we use our computers and touchscreen. We can easily imagine some new symbols or alphabets, and new forms of interactions in our musical practice. You draw a ‘V,’ you play with volume, you draw a ‘P,’ you play with pitch…

For more on the display, check out the Packard Bell Viseo 200T. It was previewed by Engadget last year and carried a street price – impressively – of only about US$300, all for a 20-inch screen and low latency. I’m gathering either something happened or it was re-branded for distribution outside the UK; anyone with more information, let us know in comments and I’ll update the story.

More on nay-seven’s Flickr:
http://www.flickr.com/photos/usine/

All screen images courtesy nay-seven. Used by permission.

Musicians have long made pictures to represent musical ideas, share those ideas, and allow others to participate. Before computers, we created scores. Now, we can create interfaces, too. Of course, just because you’re using a digital interface doesn’t mean the pencil as prototyping tool has to go anywhere. It’s the quickest way to sketch out an idea. And if your hand is steady, it just might become a lovely, personal interface.

OtoBlock by Tsubasa Naruse is a hand-drawn music sequencer. The basic interface is nothing new, dropping blocks into sequence to make sounds, but the charm is the rough edges on the pencil-made buttons, and the whimsical hand-drawn characters that live on them.

OtoBlock @ iTunes
Tsubasa Naruse website, in Japanese, but don’t miss the other adorable sketches
Via Matrixsynth, by way of Palm Sounds

Here is a 2009 experiment in “sonic interaction” by the same artist, also exploring
some of these ideas. (link) I’m not sure I could even describe it, but the relationship of minimal electronic sounds to handmade animation is utterly irresistible.

So, with mobile music tools like iPhones and portable amps from the likes of Roland and KORG, can you actually go out and make music in a subway? I recall people mocking an old M-Audio ad in which someone was doing laptop production on the subway platform. But when it comes to mobile busking, the same videographer who shot the hands-on video at top also captures an impromptu TB-303 jam in the Tokyo subway. (Apparently, this young woman did not inspire love from the police. Sadly, the app she’s using is entangled in some sort of contract issues.)

More great iOS videos on perfumepod’s channel; it’s a great way to explore different user interfaces:
http://www.youtube.com/user/perfumepod

And yes, I’ve been tipped off to “Tokyo Techno Girl” before; I have to find out more about her.

This post, by definition, overlaps with the worlds of Create Digital Music and Create Digital Motion, so I’m cross-posting — absolutely not one you want to miss, both because of the event in New York, and because the landscape of works here engages issues about which readers here I know are passionate.

Music and visuals are each themselves endless wells of potential; put them together, and “infinite possibility” probably isn’t an overstatement. This July, label Ghostly International is working with researchers at New York’s Eyebeam research center to do a free, one-week intensive on dynamically-generated visuals for sound. Before you read on, that deadline is the end of today NYC time, via a fairly simple online application form. Check out the full details and application form.

The event is led by artists Aaron Meyers (Flying Lotus’ Fieldlines) and Aaron Koblin (Daisy Bell). I asked Mr. Meyers for a round-up of the kind of work that he’s done…

Read the full story on Create Digital Motion

FINE COLLECTION OF CURIOUS SOUND OBJECTS from Georg Reil on Vimeo.

Commentary would spoil the video above, but to me it’s a reminder of the power of theater in all we do with technology and musical objects.

The arrangement includes six exceptional exhibits from the world of sounds and acoustics. At first sight looking trivial, each object incorporates a very unique ability.

The magical character of each object is accompanied with a little story, almost completely concealing the existence of technical components such as speakers or sensors. Only small connection ports as well as the uniform black finishing point to thier unusual abilities.

In form and functionalty all these exhibits pursue John Maeda’s „Simplicity“. They are enjoying to use, they are surprising and one wants to explore and investigate them.

The designs are a project by Georg Reil and Kathy Scheuring, University of Applied Sciences Würzburg-Schweinfurt. Built with Processing and Arduino (naturally).

“Press play” … “button-mashing” … the very criticism of digital music is often directed at the button or switch, even as the cult hit monome spreads arrays of buttons like a virus.

Well, we’re still interested in what you can do with a button, so to fully focus you, we’re only giving you one button with which to play. The challenge of limiting interaction to one button has already spawned an explosion of entries from game designers, who have fought their way through intense competition for the legendary Gamma indie/experimental game competition. We’ll see the winners at the Game Developer Conference next week.

But we want to see what people can do with a single button and sound. Friday, March 12, sonic (and game-based, and other) objects involving a single button will converge at San Francisco’s Gray Area Foundation, in the midst of GDC.

The deadline is officially today, March 1, but as I follow up on entries, we’re extending that to Friday, March 5, by the end of the day NYC time. There are already some terrific-looking submissions, but we’re willing to entertain the possibility of more, at least for a few more days. (if you have something you want to share online but can’t ship, let us know that, too)

How to enter – simplified rules:

It doesn’t have to be a game. (But it could be.)

It does need to do something – make noise, make lights, move, or otherwise interact.

It needs to operate on its own. We have to be able to plug it in and have it function, without the addition of a computer, etc.

It needs to be shipped to California for March 10 arrival, to be ready for the opening Friday, March 12. It will then be shipped back to you.

Send submissions, as detailed as you can, to:
onebuttonobject@kokoromi.org

Party + Hack

Part of the beauty of the one-button limitation is that it encourages quick hardware hacks and simplified designs. It’s a design you can make even if you’re out of time. We’ll be having a party to finish off creations in NYC on Saturday afternoon, building last-minute creations for Handmade Music Monday night (details forthcoming) and the One Button Objects show in San Francisco. If you want to get your own little hackday going between now and then and join us online from your local hackerspace / studio (anywhere in the world), let us know in comments. Here are details for those of you near NYC – plus some music for everyone to listen to while you solder/code/build:

Handmade Game Objects Hackday + Party
SATURDAY OPEN HOURS @ 2PM – 6PM
L TRAIN – 915 WYCKOFF AVE ( SILENT BARN )

Map
Facebook

Babycastles teams up with Handmade Music Night for a hacking afternoon. Come make crazy new video game objects, art, and controllers with us! We’ll have a jungle of fun stuff like dentures and gloves (high five to play!), but you should bring some fun things too. No experience at all necessary! (Bring soldering irons, tools, etc. if you’ve got them.)

with music by CHEESE’N'BEER CHIP MUSIC COLLECTIVE MATINÉE

ADAMGETSAWESOME »>
http://myspace.com/adamgetsawesome
Adam uses a gameboy with LSDJ and a lot of alcohol! We assure you that his name is not just all talk, he does in fact “get awesome.”

Zen Albatross »>
http://8bitcollective.com/members/Zen+Albatross/
Zen Albatross make stuff with pixels, Game Boys and ancient spirit magick. He also blogs about art, airships, bleeps, bloops and other swell things.

Goatslacker »>
http://music.goatslacker.com/
Goatslacker is Florida’s Josh Perez who promises to fill you up with high octane chip music.

and curry by chef Syed Salahuddin

And yes, that music lineup includes Goatslacker, who did the MUSE covers in 8-bit. It’s the sort of high-energy music that goes well with trying to keep your brain on hardware hacking.

Seriously, if anyone wants to switch on a webcam or IRC chat while you work on your submission, let us know and we can co-hack internationally.

For all the focus on clever little music apps on your phone, it’s the slate/tablet form factor that seems to hold the greatest promise for live performance. Thanks to a larger screen area, these devices look far more usable for control – equipped with multi-touch, they could be reasonable substitutes for hardware control surfaces, a la the Lemur.And with greater horsepower under the hood, you might not need to use them as a controller – you could run an entire live gig off them.

With this week’s Consumer Electronics Show (CES), many onlookers expected news on these devices, particularly as industry buzz anticipated a big announcement during Microsoft chairman Steve Ballmer’s keynote last night. And we got that news – sort of. Unfortunately, manufacturers teased “concepts” and prototypes, without much in the way of details – a repeat performance of 2009′s fuzzy glimpse at this device category.

That said, having been wrong about when it’ll happen, I’m still convinced we’re about to see a flood of new PC devices with interesting potential for music performance. Here’s what we’ve got so far:

Dell has a tablet “concept.” Dell’s own keynote included a brief mention of a five-inch tablet. That could make a nice form factor to stow on a keyboard or music stand as a controller. That’s about all it’ll do, as the pictures show only an audio output jack. But it will evidently have multitouch. This is only a “concept,” with no details publicly released; I’ll be following up with Dell if they announce an actual product. Photos:

Dell Tablet Concept [Flickr]

More on Dell’s new lineup (the rest of it is shipping, and may interest you more, anyway — Dell is taking advantage of wildly cheaper PC component prices to deliver some amazing machines under $1000)

HP’s Slate: Seen briefly in Ballmer’s CES keynote, the Slate is a “consumer notebook” in a slate form factor. The only good news relative to Dell’s model is that this is supposedly hardware that will ship. The bad news is, HP isn’t saying much else. The device does have a nice, sizable screen, at at least 10″ or larger (if my ability to tell the scale of things relative to Steve Ballmer’s torso is correct). That could make this an appealing alternative to other devices and form factors.

And, oh yeah, Apple: Here’s the power of Apple: PC makers, who have been shipping tablets for years, and who have shipped alternative form factors for years more, are accused of ripping off an Apple product that isn’t yet public, and about which most of us know nothing about (including, indeed, if it actually exists in the form we think it does). Not only that, but sight unseen, I’ve heard many people who assume that the Apple model of this currently-nonexistent product category will be superior, even though they don’t know what OS it’ll run, what it’ll do, what it’ll look like, what size it’ll be, or what it’ll cost. PC vendors, of course, had the opportunity to provide a clear alternative, and instead made their picture somewhat murky, too.

I don’t just mean Mac fanboys, either, who could be excused the pre-emptive positive review. Even The New York Times got in the act. Ashlee Vance of the NYT Bits blog wrote in advance of Ballmer’s speech:

It could be one of Steve Ballmer’s riskiest trade-show moves in years.

On Wednesday, Mr. Ballmer, Microsoft’s chief executive, will unveil a novel take on a slate-type computer … This product better be good because Apple is expected to unveil its take on the slate/tablet form factor later this month … The last thing Mr. Ballmer wants to hold up is a me-too device.

Yes, how dare he … announce … a product … that might compete with a product that no one has seen yet? What a risky move! (Deep thought: can a product be “me-too” even before there’s a “me,” or in this case, an “i”?)

That said, yes, most industry analysts expect an Apple announcement later this month. I’m skeptical about whether such an announcement will be useful to our audience, however. If Apple chooses its relatively locked-down iPhone-style operating system over the Mac OS, and if there’s no hardware input and output, and if the focus is buying magazines and books from iTunes, I think I’ll pass. Of course, some PC vendors may go a similar route.

And, in fairness, I’m sure part of what has prompted PC makers to unveil prototypes of non-shipping products is fears of what happens if Apple gets there first. It’s too bad Apple doesn’t leak a secret plan to solve global warming, or give away chocolates.

Android is a big winner. Murky as the slate announcements were, the one message that has been clear out of CES is that we’re going to see more of Android.
HP may even ship a version of HP Slate running the OS, says TechCrunch. Ordinarily, this would be relatively bad news; on Windows, you can run any music software, whereas Android is relatively limited. But I think that could improve, with open source controllers and work on porting free multimedia tools like Pd (Pure Data) and Processing.

Just keep in mind…

Don’t forget “traditional” tablets for multi-touch. Sure, these smaller slates are interesting, apparently an attempt to blend the appeal of e-readers like Kindle and Nook with handhelds like the iPhone. But why carry a tablet and a laptop when a multitouch laptop could be both? Yep, tablet PCs are back, now with multi-touch input as well as pen. And their convertible form factor means you could have multi-touch control without your arms getting tired.

Case in point: Dell’s Latitude XT2 joins entries from Lenovo and HP. HP’s TouchSmart tm2 (as seen on Engadget) finally improves on HP’s previous, somewhat underpowered entry; I’ll be looking more closely at it. Also appealing: the HP is the first of these devices I’ve seen to pack discrete graphics, which could give you a machine with enough graphics muscle to do live visuals and video, plus music, all with multi-touch control and the I/O ports you’d expect on a laptop. It could be an all-in-one live performance beast if it pans out; I hope to check it out soon.

More analysis of the options – and why the upcoming battles could be a battle for computing’s soul:

Gotta Be Mobile has long been a stalwart analyst of, and advocate for, the tablet PC. Here’s the surprise: even die-hard Tablet PC fans are skeptical about just what the new “tablet” or “slate” means. And the bigger surprise: even outside of the world of music and visualist sites like CDM, people are asking the question about whether the future of slate/tablet computing is passive consumption. Here’s Tablet PC MVP Warner Crocker writing for the blog:

Then there’s the question of what do we do with these things now that we seem to be on the threshold of seeing them everywhere? That boils down to content and in most cases that means consuming it, not creating it.

Floating on a Sea of Tablet Paradoxes

Christophe Stoll of precious, the Hamburg-based design firm responsible for everything from familiar soft synth user interfaces to rock band graphic looks, has similar skepticism. His take is even more far-reaching: in the midst of rabid gadget consumption, what about affordability, ecological impact, and truly open, community development? His first story looked at some of the shiny possibilities in the future:

The tablet innovation race: Three commented examples of what Tablet Computers could look like in the near future.

A follow-up story, however, responding to comments by me and others, wondered if a more open, sustainable, hype-free future could apply more intelligent design:
Tablet innovation race II: Some more critical thoughts regarding the ongoing hype around tablet computers.

Bottom line: by this time next year, I do expect that we’ll have some powerful, new, affordable solutions for multi-touch control and portable music and visual performance. Just what form that will take, though, isn’t much clearer now than it was this time last year. I hope that situation will change soon – and I hope Apple doesn’t prove to be the only company able to articulate a vision for the category.

Musicians and artists now have the power to fuse visuals, sound, and interaction, to make a spectacle, an album, and a game all at once. But with the blank canvas of three different media before you, what form should that fusion take?

Space shooters with pounding electronic beats behind them have cleared some of the way. Now it’s ambient music’s turn. In the game Osmos, you become a mysterious particle, floating amongst gravity wells in various fields of material. By carefully navigating, applying just the right vector force to move through the shifting landscape, you merge with other particles and escape to safety.

http://www.hemispheregames.com/osmos/

The move from “shoot stuff” to “move” or “eat” seems to be rising in popularity, with games like fl0w and Spore’s initial “cell stage” encouraging nonviolent navigation. To me, there’s something happening to the zeitgeist, perhaps a renewed awareness of cosmic (micro- or macroscopic) being, and of movement that draws on free-floating physics.

Even amongst a wave of games in this mode,when you actually play Osmos, you realize that it is something different and special. The design makes ingenious use of different kinds of movement and pacing through its different modes, at one point calling upon you to hurtle around a black hole, then move at nearly imperceptible speeds through a seemingly impossible-to-traverse petri dish of massive particles. No less than a shooter, it connects to the id and survival instinct. Pac-Man, the most successful arcarde game of all time, and one of the few that sucked in men and women in equal measure, was noted for its emphasis on eating as the mechanic. Consuming stuff appeals to everyone.

Of course, this is on a music site, and with good reason: what makes Osmos work is that Osmos is musical. It’s immediately beautiful and delicate, a perfect aesthetic union of the texture of the music and the on-screen arrangements of particles. More importantly, the music is woven directly into game play, providing subtle cues for dangers, and underscoring the pace of gameplay. You can only solve a level by managing speed and motion, and the music helps provide both the literal indications of speed and help your head get into the right zone to lose yourself in the world. If blips in early arcade games helped create a zone of play trance, now we have spectacular ambient soundtrack of music by Loscil, Gas/High Skies [Microscopics], Julien Neto, and Biosphere.

The music isn’t simply a beautiful soundtrack to the game. The game really feels like an extension of the world of the music. Put it all together, and something magical happens in this $10 game: you hear the music in a new way.

I spoke to the lead designer behind the game, programmer/animator Eddy Boxerman, along with musical-sonic collaborator Mat Jarvis aka Gas aka High Skies.

Peter: I found it amusing that some of the game press have stumbled around looking for a name for a new genre here, the "eat other stuff" category. To me, the basic game controls could be traced back to early titles like Asteroids. What are the games that have inspired you, either specifically or generally?

Eddy: Some people are dubbing the gameplay type as "eat ‘em up", as opposed to the classic "shoot ‘em up". But it’s true, the physics/controls have more in common with classic games such as Asteroids and Gravitar. The only "modern" game that otherwise influenced me was Katamari Damacy. It’s funny: when I sent out the first prototype to friends, I was concerned they would think the game was too similar to Katamari. That’s when I heard about flOw, which I’ve tried to steer stylistically clear of ever since. I guess it’s an archetype though, and the comparisons have been inevitable. As for Spore, Osmos was about one year into development when I saw the first preview footage of its "cell stage". On top of that, I learned that Will Wright had hired Brian Eno to do the music! I definitely had a "why should I bother?" moment at that point. But here we are, a couple years later, and I think we’ve managed to put out a unique game with a great mix of gameplay, sound, and visuals.

Peter: Ha – well, I’m glad you “bothered”! One thing that I think is really unique about Osmos is the gravitational mechanic. How did this come about?

Eddy: I was (and probably still am) a sci-fi kid. And when I read about real phenomena like colliding galaxies, retrograde planets, and the like, my mind goes to stange and wonderful places. That said, the Spacecraft Dynamics course I took during my engineering degree definitely played a large part in putting those mechanics and levels together. Thanks, Professor Misra!

Peter: How are the physical mechanics of the game constructed? This is essentially two-dimensional vector math, yes? Were there refinements you needed to make in order to keep performance up, or to keep the levels playable?

Eddy: That’s a big question! But yes, essentially it’s 2D vector math, though we make use of some 3D math (including quaternions) on the particle systems. The majority of the physics is quite simple though — first year bachelor’s stuff: F = ma, conservation of mass and momentum, gravitational laws, basic time integration, etc. We follow the spirit of the laws, but not always the letter. For instance, if you look at the Earth’s solar system in its entirety, you wouldn’t even see most of the planets; that doesn’t make for a very playable game though. As such, we took many liberties when it comes to scale, constants and exponents in the equations. The majority of the work was in the tweaking of those values to make the game /feel /good.

Peter: How did you prototype the game? What was the process of evolution like? Were there any failed attempts along the way?

Eddy: I put together the first prototype starting from a NeHe OpenGL tutorial. I added some basic physics, mouse controls, some highly unoptimised collision detection code, and voila, I had something to play with. After that, I immediately integrated [cross-platform 3D audio API] OpenAL, as the game was just begging for sound and music. I then hunted for some good sounds (discovering freesound.org in the process), put together a simple looping song (my first rough and humbling attempt to create digital music), and sent it out to a few friends. I’ll make that version available for free someday… when I’m feeling brave.

As for failed attempts, there were tons of shelved experiments and features that were cut. One interesting problem was the difficulty curve of the gravitational levels. In fact, the first gravity levels looked a great deal like the "Warped Chaos" levels do today: several free-floating Attractors in a sea of regular motes. I found it incredibly difficult, however, to make a "beginner" version of those levels. If the gravity was too weak, the Attractors had little apparent effect; too strong, and the level became incredibly difficult. I stuggled with this for a long time, until I hit on the idea of putting one, immobile Attractor in the center, with everything else orbiting around it. This allowed for the creation of systems with strong gravity effects, but long-term stability. It seems so obvious now, especially given the example of our solar system, but it wasn’t back then. In the end, these are some of the coolest levels in Osmos — practically its "signature" level type. Moral of the story: necessity really is the mother of invention.

Peter: What is the visual engine like? The sound engine?

Eddy: As I mentioned, I bootstrapped from a NeHe tutorial, but that’s all been replaced at this point. It’s now a homegrown engine using OpenGL. It doesn’t make any use of pixel/vertex shaders, so it runs on a wide variety of hardware. The majority of our texture work is done in Photoshop.

[The sound engine is] a wrapper we’ve built around OpenAL. At a basic level, all we ever do is mix and adjust the gain and pitch of our sounds and music. This allows us to run on generic sound hardware. But we do make extensive use of those "effects", and it’s possible to get a great deal of variety and feedback with them. In the end, it’s all about the quality of our sound sources, and the algorithms that adjust and smooth the gain and pitch of each.

Peter: For pitch shifting, were you able to do that natively in OpenAL?

Eddy: Yes, OpenAL supports pitch shifting at the software level. I ported our sound engine to use DirectSound at one point, and found that it’s pitch shift range is rather limited. We reverted back to OpenAL.

Peter: One thing that really strikes me about the game that is its use of pacing. Even without the obvious technique of allowing the user to control speed, there’s this extraordinary contrast between levels that require you to move quickly, that require you to anticipate the rate of movement along a vector of other objects, or that require you to move almost imperceptibly slowly. Was this an early goal, or something that evolved out of developing the game concepts?

Eddy: It evolved from the game concepts themselves. We tried to fully explore the fundamental controls and game space of Osmos, and converged on a number of interesting regions and structures that were fun to play with/in. Some people actually "complain" that the pacing of the game isn’t consistent across levels — some are fast, some slow — but that wasn’t our goal. In a sense, we’re merely presenting the results of a great deal of research and exploration into the space, with our sole requirement being that the levels are fun and interesting. We wanted it to be a game of concepts and understanding, not speed. The addition of time-warping allowed us to break free from that, expand our scope, and include a much larger variety of levels, as some have both fast and slow moments in them; we wanted the player to be able to control the pace and difficulty to their level of comfort.

Peter: Connecting the action to music, how important is the use of music in conveying time?

Eddy: I think pitch-shifting the music and sound effects helps players track time-warping in an inutitive way. Without these aural cues, the mental connection would be incomplete. That said, I sometimes test the game without sound, and it’s totally playable — just much less immersive.

Peter: At what stage did you involve the musical element? How was that relationship established?

Mat: Eddy first contacted me two or three years ago about using one of my tracks on Osmos (“Gas – Discovery”). He sent me a basic alpha version where the game was literally flat circles on a plain background, but even back then with the simple graphics and no sound it was compelling. I think it was at this early stage that I sent him a copy of another High Skies track, “The Shape of Things to Come,” which he also used a section from. It wasn’t too long later that he sent me a version of Osmos with most of the music that made it to the release version and it all just complimented each other beautifully.

Eddy has done a great job choosing music that compliments the game and graphics, and they also compliment the music. I think he’s done a great job overall as well, keeping his vision consistent for the look and feel of the Osmos world. Even though some of the levels are quite varied; some are quite chilled, others fast paced, nothing seems out of place. He has created a new world that’s both microscopic and macroscopic.

Peter: The musical and sound effect elements really do blend effortlessly. Mat, I know you made substitute sounds – how did this help blend the aesthetic?

Mat: I made the rebound noise when you bounce off the walls/ boundary. The original sound was quite hard and metallic with a lot of reverb, and I suggested a softer echoey sound. There was nothing wrong with the original, but it made the boundary feel quite hard and cold like being in a metal tank. I sent Eddy some analogue samples, using a [vintage Roland keyboard] Jupiter 6 with varying delays/ echoes. Because the tones are so pure and tuned, Eddy pitched them slightly to fit with the various keys of the different music.

Eddy: In general, it was a matter of searching and experimentation for the "right" set of sounds that fit well together — which can be a lengthy but rewarding process. Once integrated, minor gain and pitch adjustments helped fit it all together.

Mat provided the bounce sound, as well as the continuous absorb/being-absorbed sounds. The other sounds were created/shared on freesound by the users Elektrocell, fran_ky and Jovica. (Check out the Osmos credits in the readme.html for details.)

Peter: I recently spoke to Crystal Method, and while they’ve done some notable game scores, they actually said they wouldn’t want their music to be placed in a game context at which the speed changed, that it’d lose something. This seems like the opposite of that. Now, maybe it’s easy to take for granted with ambient music as opposed to something beat-driven like Crystal Method, but do you think there’s something special that allows this music to be adaptive in that way?

Mat: I think that in the right circumstances music with drums could work well and may even highlight the time-shifting more. Of course with the wrong game and wrong music it could sound terrible too.

Eddy: It’s a delicate issue. Out of the context of the game, I think it would seem wrong. There were also some pieces of music I tried which didn’t lend themselves well to this kind of manipulation, and which didn’t get included as a result. But with these songs, and everything working together in-game, I feel the whole experience is stronger as a result. I’m just glad Mat and the other artists agreed — or at least… didn’t object.

Peter: Hey, I think if you can write music that sounds good played at a different speed, that’s nice.

Osmos seems to me a musical experience, in that the sense of the music is transformed by the visuals and the interaction with them. Do you think it’s possible that audiovisual interfaces could go even further in terms of their impact on the music?

Mat: Yes, it almost seems as though Osmos creates a world for the music to exist.

I’ve always liked music visualisers like the Processing and Cymatics stuff, they’re quite compelling to watch how they react to the music, so it would be interesting to go the other way; by manipulating/ sculpting abstract shapes which then modify or even create sounds and music, especially using the new controllers like the Wii, [Microsoft’s] Project Natal and Sony’s Motion Controller instead of the mouse.

Eddy: I totally agree with Mat. Yes, please! More feedback between music, visuals and interactivity — in all directions. I’d love to experiment more on all of these fronts. In general, I think games such as Audiosurf and Auditorium, while great, have only begun to scratch the surface of what’s possible. Interactive musical "toys" such as [Nintendo/Toshio Iwai’s ]ElectroPlankton or Dub Selector are another great direction; and Rock Band Network is about to expand the market in a huge way. It’s exciting!

Peter: Along those lines, this is obviously a new take on music distribution in a way that goes well beyond what titles like Rock Band have done. Are there other cases in game history that to you have done that? Is there potential in these new outlets, outlets that are more accessible to independent developers (Steam, Direct2Drive, Xbox Live Arcade, iTunes App Store), to really change game development?

Eddy: Wow, that’s another big question. Really, I just felt the music was such an important part of the experience in Osmos that we should show the artist and track names during the game. But now that you mention it, I can’t think of another game that has done this, besides games like Rock Band where the song /is/ the game/level. That said, one way I’d love to push this evolution is to "augment" music videos with interactivity: ie. into music video-games. (The term "game" could be used rather loosely here.) Everyday Shooter does this, though Jon Mak actually created the music /for/ the game in that case — which rocks — but collaborations could lead to some really amazing things in this direction.

Peter: Eddy, Mat, thank you. Whether it’s in the form of a game or an audiovisual performance, we’ll certainly continue to explore these areas. And – hint, hint – Osmos 2, with multiplayer? Just a thought.

Osmos Trailer from hemisphere games on Vimeo.