Omer Yosha has created a beautiful, elegant interface that uses infrared sensors to control music applications. Touch-free interfaces, of course, date back to the Theremin, but Omer is trying some new things here, creating an invisible matrix of controls in the air. And I love the way the physical object looks. He writes to tell us about the details:

I’m an Interface Design student from the FH Potsdam (near Berlin), i have a musical background, and the idea to create an AirPiano developed as i was playing around with the Arduino board, Processing and some IR sensors in my free time. It was fun controlling MIDI through moving my hands in the air, so i eventually found a way to set it all up in a way that makes sense and that is easy to control.
The concept behind the AirPiano is having a matrix in the air, with virtual keys & faders. The location of each key must be very clear for the user and easily learnt. The AirPiano is therefore only one example of an application that could adopt this concept. Since it is only the first prototype i built, it features at the moment a matrix with 3 layers, 8 keys for each layer. As long as a key is triggered, a note plays and an LED underneath the virtual key turns on (unfortunately it is hard to see it on the videos). The LEDs give the user additional feedback. The device is connected through USB and communicates with the AirPiano Software, which allows the user to assign each key/fader with a Note/Controller number, Channel and Velocity as well as transpose and save/load presets. The AirPiano Software can communicate with any MIDI instrument/sequencer. It is of course a polyphonic controller.
The AirPiano is not only fun to play, it also invites to experiment, to explore endless arrangements and develop new playing techniques. It might be useful for DJ performance, as a music therapy instrument or as a toy.
I’m at the moment trying to look for investors and people that could help me take this idea further. I presented the prototype two months ago in the Hannover Messe and received very good feedback. The concept is protected as a Provisional U.S. Patent Application.

If you can help him, chime in! I’d love to see what develops.

Here it is controlling Ableton Live:

More photos and another video to give you a sense of how this works (it’s particularly clear once you see the software interface):

Here’s a brief video snippet I discovered someone took at a talk I did at this year’s South by Southwest, with interaction design pioneer Joy Mountford (formerly Yahoo, Apple). We were talking about the idea of “data as art”, which happened to coincide neatly with the Design and the Elastic Mind show at MOMA, featuring several works from Joy’s recently-disbanded Design Innovation Group team at Yahoo.

The audience response to the work Joy showed was really overwhelming, as search activity danced around the globe and photos came to life in three dimensions. And it was nice to be able to show them the tool used to create these projects, Processing, and encourage people to try it out for free, even if they hadn’t tried programming before.

But I was surprised by how people reacted to a quick musical demo I closed with. Using Java, I wrote a simple program that checked my Gmail account using IMAP, then translated the time spam messages arrived into MIDI notes. I’m still developing a more advanced real-time version, so I threw the resulting SMF file into Ableton Live.

I’ll actualy be showing a newer version of this for Internet Week at an event sponsored by Make Magazine; more on that in a few days. (I’ll also use that as an opportunity to post some updated code.)

We spend so much time talking about how visualization can make data more expressive that we sometimes overlook other media. The spam “musicification” made sense to people partly because even the untrained ear is sensitive to musical timing, I think. Sonification of data isn’t always the right choice; the results can be abstract, though perhaps there’s value in that, too. But it’s worth remembering that people are sensitive to sound as they are to visuals. Since it’s not an either/or choice, necessarily, it’s too bad that so often designers neglect aspects of sound and timing while focusing only on what something looks like. It’s a challenge, certainly — there’s a reason most of us mute annoying sound feedback on computer interfaces — but I think it’s an area in which we’ll see a lot more discussion.

Now, data in smell-o-vision — that’s a story for another day.

Weird Fishes: Arpeggi from flight404 on Vimeo.

Plenty of bands have jumped on the "remix generation" bandwagon, releasing music to be remixed and sampled and visualized as a publicity stunt. But, then, plenty of bands aren’t Radiohead. Readers here may have been disappointed that our favorite superband didn’t embrace Creative Commons sharing when announcing their iTunes-only stems. But a number of the artists we follow came up with some brilliant work.

In Visuals: Robert Hodgin, aka Flight404, has the enviable job of exploring new visual expressions as his day gig of sorts. Working primarily in code developed in the open-source, Java-powered Processing, he develops a technique and then iterates and iterates on it until it goes from computer gimmick to refined artistry. He blogs that process, as well, pushing forward the rest of the Processing community. His video above uses abstract, generative processes to visualize Radiohead’s "Weird Fishes", but is developed enough to become organic. It’s a voyage under the sea. Via our sister site, Create Digital Motion.

In Music: A number of readers tackled the Radiohead remix contest. Here’s my favorite: our friend Alan Molina created a sparse string accompaniment that spotlights Tom Yorke’s vocal part. He explains:

Thanks for listening!  I actually recorded and mixed all the strings.  

They are all a violin (just lots of layers of me).  My profession is an orchestral violinist–this remix was an outlet to do something different!

I used Ableton Live 7 for the effects, and used  the kind of mic that clips on behind the bridge of a violin.  Done on my couch in front of my computer!

Of course, the other direction to go is stretching the tune past the point of recognizability, with strange bizarro-universe remixes pulling the tune to experimental glitch and faux-punk. Here are a few of the more unusual takes on their music:

Everything has an “API” these days, but what that means in practice is often not so exciting. You can make little widgets for Facebook, or post recent Twitter messages, or do other simple developer tricks. Echo Nest’s “Musical Brain” API is more far-reaching: it’s an API for music. All music online. The first of a series of developer tools, “Analyze” is designed to describe music the way you hear it, figuring out tempo, beats, time signature, song sections, timbre, key, and even musical patterns. More developer tools will follow.

Twelve years in development, the Musical Brain is a bit like a digital music blogger. It’s been crawling the Web while you sleep, reading blog posts, listening to music to extract musically-meaningful attributes, and even predicting music trends. It’s like almost like a robotic, algorithmic Pitchfork. (And I’m serious — it may be April Fools’ Day, but this is real. That’s what the “brain” claims to do, backed by research at UC Berkeley, Columbia, and MIT.)

What does all of this mean? The Musical Brain may not be replacing your friendly local music blog any time soon, but what it can do is infuse some musical intelligence into applications like music visualization. The Matmos album at top used the Analyze algorithm to map the timbral profiles of songs on an upcoming album; that graph was then rendered by an artist in watercolor, translating the digital into traditional paint media.

The Analyze API could also enable everything from Web music apps and mash-ups to live audiovisual performance tools, or even smarter music games. That’s the reason co-creators Brian Whitman and Tristan Jehan, both with PhDs from the MIT Media Lab, chose to open up development to a broad audience. I got to speak to Brian a bit about the project.

Processing is an open-source coding tool, built in Java, designed specifically to be versatile for artists and friendly to non-coders. Code is elegant and simple, but can take advantage of all the potential power and performance (no, really) of Java. Java really can be fast enough to use in live performance situations, though its one Achilles’ heal is that automatic memory management — the very thing that makes coding easier, via something called a garbage collector — can make sound glitchy at lower latencies. (JavaSound seems worst on Mac OS X, as implementation of the sound API by Apple hasn’t kept pace with improvements in Java audio on other platforms. It is possible to build a real-time-ready Java implementation that performs as well as languages like C++ for audio, but right now there’s not yet a mainstream implementation of this type.)

That doesn’t mean Processing isn’t useful for musical applications. With experimentation, sound libraries like Minim can perform quite well, especially if extreme low-latency is unnecessary. (see Processing’s libraries page for more.) And you can always use Processing as a visual front end, while sound comes from elsewhere (Max, Pd, Reaktor, or even Ableton Live or a plug-in.)

There’s plenty of incentive to work with the environment as an artist. People who never coded before are able to build entire projects in Processing, not just uber-programmer-geeks. Even experienced coders can find it a fast way of experimenting with ideas — sometimes better-suited to tasks that are more difficult with patching environments. Despite all the hype around Flash/AIR/Flex and Silverlight, I find Processing easier to develop in, and you have far more robust development options, free and open source tools and libraries, and genuine OpenGL 3D capabilities.

I put out a call for people working with Processing for music, and we’ve already got a handful of interesting examples. Because of the open community around Processing, code is available for a couple of the ideas here, so you can have a peek and learn from fellow Processing coders.

nodeSeq from Jared Arave on Vimeo.

Chris O’Shea sends along his latest project, a collaboration with sound designer and composer Owen Lloyd called Drop Spin Fade. Part of the Future of Sound tour, Drop Spin Fade allows users to position, sculpt, and play with sound in a 3D environment using gestural control.

Drop Spin Fade

The music/sound environment: Through a series of iterations, Chris and Owen have started simple and built increasingly-sophisticated sonic control using the setup, manipulating granular samples by spinning and bouncing them around the space. It’s not just positioning at work here: you can actually shape the sounds you’re hearing by interacting with the geometric forms in the environment. Eventually, the designers hope to give users more compositional control, making this into a kind of 3D sequencer.

The guts behind the scenes: The work was built to showcase the Illustrious positional sound system, which can use positioning data to create 3D sound environments. For control, the project uses the Gametrak game controller hardware, which you may have seen used in inexpensive golf and other sport games. It happens to be a very nice gestural controller, as well, with extremely low latency when compared to video camera tracking solutions. Visuals and hardware interface are performed in Adobe Director, routing positional control to Illustrious via MIDI and playing a live sound patch built in Max/MSP via OpenSoundControl data. There will be yet another piece as work proceeds on support for the Nintendo Wii controller.

I’m actually quite surprised that more work hasn’t been done with 3D interfaces — though I suppose I shouldn’t be too surprised, as it’s extremely labor intensive! 3D has focused largely on positioning, but with powerful hardware and software capabilities bring 3D to the masses, 3D interfaces are surely next. Iterations and shared research are vitally important to any medium advancing, so I hope we’ll have more projects in this area. (I happen to be working on something different but related in the 3D space in my own research, which I’ll share when I’m ready.)

Previously from Chris O’Shea land: Muon Speakers, with Processing Visuals

All rational arguments for development on mobile phones aside, some phone owners are driven by the irrational, purely personal need to tap into their phones and do stuff. Our friend Sebastian Tomczak reports back on a first experiment with the mobile version of Processing, the Java-based “coding for artists” tool.

Simple Live Synth for Phone

This is nothing fancy, but that’s the point: simple is the best way to get started. It’s the sound version of “Hello, World.” In programming, I find, even simpler than I think is necessary is often the best place to start. Erm, especially if you’re me and screw things up really quickly.

Mobile Processing is an example of what the future could look like if we could program phones with standard Java libraries. After all, your phone is a computer — often a pretty expensive computer, considering. And then, when you upgrade, it becomes useless without service. Recycling phones into something useful is only natural. And because they’re simple computers, there’s an opportunity to return to the days when programming was actually considered a killer app — for “end users.”

And yes, this does fit in the category of “things you can’t do with Ajax on iPhone and can do with zillions of other, cheaper phones.” I’m glad to be over the “what might actually impact iPhone sales” category and into the “completely useless phone applications that appeal only to insanely geeky musicians” area again. Phew! Much better.

Non-Processing/Java geeks can stop reading here, but I should elaborate a bit.

Looks can be a powerful agent for changing how we think about sound. Pairing liquid, organic speakers with equally fluid and dynamic visualizations, the launch of Muon last month in Italy made this principle readily apparent. I’m all about lo-fi, cheap gear here on CDM, but if you absolutely must launch luxurious aluminum speakers with spectacular animated visuals at a posh party in an Italian salon, I sure won’t complain. Pass the prosecco, please?

Muon Project Page, documentation videos at chrisoshea.org
See coverage at ze | d | esign, toxi’s project blog, MoCo Loco, elsewhere. (Yeah, CDM’s motto is: cover things last. Was a bit busy with Maker Faire!)
Created by Moving Brands

Details on the installation and how it was done:

Hello, coming to you live from San Mateo and the Maker Faire hosted by Make Magazine. If you stopped by the CDM booth today, I hope you stick around a while. This site, for musicians, DJs, composers, and soundmakers, covers the audible side. Create Digital Motion covers VJing and live and interactive visuals (for an audience we like to call visualists). We cover store-bought hardware and software as well as DIY, and love having everyone from beginners to advanced readers around.

Here are some links to more information on some of what I was showing:

Sensors and MIDI: Working with sensors and physical inputs is what some term “physical computing.” We’ve covered lots of physical computing projects here on CDMusic.

I wrote up an introduction to MIDI for Make 07 which would be a good place to start if you’re relatively new to MIDI.

The MIDI sock monkey and infrared and touch sensors are powered by Lady Ada’s MIDIsense, as introduced here:
Sensors-to-MIDI with the MIDIsense Board. See also the Arduino board, which we’ve covered mostly on the motion side.

Barcode scanner to music converter, powered by Processing, a Java-based development language that makes programming friendly for artists. Check out the official Processing site, Processing Blogs aggregator, and Processing coverage on our sister site, Create Digital Motion.

Live visuals: The live video-processing “mirror” via the MacBook was built in Max/MSP/Jitter. Check out our ongoing Jitter coverage on CDMotion. Nope, no “source code” / source patch available yet — it’s a bit too messy, the reason you don’t see more open-sourced Max patches!

And totally off-subject: The keyboard I was using is a ReMOTE 37SL from Novation. Nothing to do with Maker or Maker Faire but very handy to have around … and well worthy of combining with other DIY hardware.