That doesn’t mean Processing isn’t useful for musical applications. With experimentation, sound libraries like Minim can perform quite well, especially if extreme low-latency is unnecessary. (see Processing’s libraries page for more.) And you can always use Processing as a visual front end, while sound comes from elsewhere (Max, Pd, Reaktor, or even Ableton Live or a plug-in.)

There’s plenty of incentive to work with the environment as an artist. People who never coded before are able to build entire projects in Processing, not just uber-programmer-geeks. Even experienced coders can find it a fast way of experimenting with ideas — sometimes better-suited to tasks that are more difficult with patching environments. Despite all the hype around Flash/AIR/Flex and Silverlight, I find Processing easier to develop in, and you have far more robust development options, free and open source tools and libraries, and genuine OpenGL 3D capabilities.

I put out a call for people working with Processing for music, and we’ve already got a handful of interesting examples. Because of the open community around Processing, code is available for a couple of the ideas here, so you can have a peek and learn from fellow Processing coders.

nodeSeq from Jared Arave on Vimeo.

nodeSeq is a fluid music making tool that maps melodic materials to nodes, floating as particles in an interactive 2D interface. The physics-y goodness comes from my favorite Processing physics library, traer.physics. In this case, Max/MSP is providing the sound, but any application that supports OSC would work. Full source code; it’s worth just checking out the libraries used as they’re all essential downloads for this kind of work. ControlP5 is particularly useful — it maps variables to controls that you can use for live performance or troubleshooting (and they can be hidden, as well). In “early development,” so it’ll be interesting to see where Jared takes this.

“Jug Hero” by Shawna Hein (looking on in the background) and Kevin Linn, students at University of California Berkeley, uses jugs as a tangible interface. Blow over the jugs, in the tradition of American jug band music, or clink glasses for extra points. Processing provides the visual interface for the project. The physical interface itself comes from Arduino, which is an ideal companion to Processing, as it uses similar syntax and the same development environment.

What I particularly like about this project, as well, is the creators worked to have more open-ended musical cooperation and not just some of the rigid gameplay mechanics introduced by Guitar Hero. Nothing against Guitar Hero, mind you — but why not let loose a bit if you’re building your own game, and try something new?

Again, more versions are planned, so we’ll watch this as it develops.

genPunk is a wonderfully glitchy sounding music tool built entirely in Processing — sounds and all. It may not be a show piece for Java’s audio quality (well, at least they chose to make it sound this way), but it is a fantastic example of how you can use Processing to build a lightweight tool around your own personal musical ideas and style. And, being braver than me, they’ve put up all their code, warts and all, in “alpha mode.” Argentinean code madman and TOPLAP advocate Ivan Ivanoff is the scientist behind this.

If you’ve got more projects, we’d love to see them.

Later this month, too, I hope to give some background on how to really get audio and MIDI working right now on different platforms. There are some tricks to it, definitely. Stay tuned.

The fantastic cat illustration at top comes from Christer Carlsson, designer, and I think embodies the evolutionary nature of life on planet Processing.

A new patching environment called Lily, inspired by tools like Max/MSP, works its magic using JavaScript inside a browser. So turning your browser into a music tool becomes more practical. And Lily supports the network-savvy OpenSoundControl (motto: “it’s not MIDI!”), so you can hook up an OSC controller like the Monome and jam with Firefox and the New York Times.

Finally found a use for the NY Times from Bill Orcutt on Vimeo.

How does it work? Get prepared for some Web technospeak, kids!

When the patch starts, the browser enters a DOM inspection mode and mousing over a DOM element highlights the node. Clicking on a node writes the element’s data (its innerHTML value if it’s a text element or the binary data if it’s an image) as a sound file and the file is then loaded in a quicktime player in the patch. The sounds can then be triggered using OSC messages.

Hey, where’d everyone go?

If DOM models don’t exactly get your pulse racing, here’s a strange and elegant physics-based sequencer hooked up to Reaktor. Fans of Processing, that environment is also capable of similar stuff; this is even modeled on the Processing-compatible traer physics library.

SVG Midi Sequencer from Bill Orcutt on Vimeo.

Ready to get going with this yourself? Lily is now in public beta, ready to run for free on Mac, Windows, or Linux. Browser not included.

Lily Public Beta 1

Gobs more examples and documentation on the blog

Previously: Browser Beatboxes and the Rebirth of Max-Like Patching

Looks can be a powerful agent for changing how we think about sound. Pairing liquid, organic speakers with equally fluid and dynamic visualizations, the launch of Muon last month in Italy made this principle readily apparent. I’m all about lo-fi, cheap gear here on CDM, but if you absolutely must launch luxurious aluminum speakers with spectacular animated visuals at a posh party in an Italian salon, I sure won’t complain. Pass the prosecco, please?

Muon Project Page, documentation videos at chrisoshea.org
See coverage at ze | d | esign, toxi’s project blog, MoCo Loco, elsewhere. (Yeah, CDM’s motto is: cover things last. Was a bit busy with Maker Faire!)
Created by Moving Brands

Details on the installation and how it was done:

Liquid-y Speakers: The speakers themselves were beautiful enough. Designed by UK speaker research center KEF Audio and Ross Lovegrove, a champion of organic, 21st Century design and one of the most respected designers on the planet, the key to the design is super-formed aluminum. The process does for metal something like what vacuum forming does for plastic: you heat sheets of aluminum so they can be molded into unique forms. The speakers themselves are formed out of single, 6-foot pieces of metal, into an acoustically-conceived, flowing form. I haven’t heard them, but we’ve already discussed (at a radically lower price point) why speakers really don’t have to be — or even shouldn’t be — rectangular.

Liquid-y Visualization: And that’s just the speakers. Part of the beauty of digital media is that they can make the invisible and the impossible visible in a dynamic way. So Muon creators employed London’s responsive media firm Moving Brands, who in turn brought in two of our favorite people — responsive media guru Chris O’Shea (see his blog Pixelsumo, and artist and Processing ninja Toxi (aka Karsten Schmidt). Working with creative director David Eveleigh-Evans, the team created a dynamic animation on a huge LED screen that could visualize the sound coming from the speakers and reflect in motion what the speakers do in product design.

How They Did It

The animation isn’t just a pretty visualization; it organically reflects what’s happening with the sound. Performing a spectral analysis of the sound (via a Fast Fourier Transform or FFT), the software uses amplitude levels in different zones of frequencies to produce particle objects, which spring and bob based on polarity, turning the peaks in amplitude in sound into a pulsating pool of fluid. The model itself is actually 3D, but it’s squashed into 2D space (or you can imagine looking at the 3D space from above). The other essential element is that the software looks at a history of amplitudes over time, so that overall changes can be adjusted (a bit like the simple “peak” meter on a consumer stereo EQ).

If you imagine an EQ meter using a pool of mercury instead of simple bars, that’s the basic idea.

The implementation is, as I’d expect from this team, simple and elegant — a few basic elements are tweaked to produce a maximal effect. Here’s the gear (software and hardware) used to pull it off:

Processing, the open-source, Java-based, simple coding environment for graphics and multimedia. (Trust me. You can code in it. Even a 10-line sketch can often be interesting, though sound-related stuff tends to get a lot more involved fast.)

Ess, which is one of a few competing sound libraries for Processing based on the Java sound API. (See also the JSyn-based Sonia, though there seem to be some compatibility issues with that one, and the newer, tongue-twisting Mimin.)
traer.physics, a wonderful and easy-to-use physics library. (I’ve used it in a few projects — a must-download.)

Toxi’s own libraries.

ControlP5 for debugging, a library that places on-screen controls in your sketch. (I imagine it’d also be hugely useful for performance, and can’t believe I haven’t tried it before!)

And don’t forget the ginormous LED screen from Creative Technology, “containing 73,728 full colour LEDs over a 10 x 5 metre floor, using the Barco MiTrix system.” Yum. 73,000 LEDs.

More stuff from Chris on the way. If you’ve got a Processing project for sound, we’d love to hear about it. It’s better-known on the visual side (and a regular subject on Create Digital Motion), but there are still MIDI and audio tasks at which it excels, even if you have access to tools like Max/MSP/Jitter. Right tool for the right job, and whatnot.

Here’s my personal favorite. (YouTube has some other configurations; one even includes beat juggling.)

DIY Interactive Blocks: TileToy

Ever wish you could go beyond oohing and ahing on the Web and experiment with technology like this yourself? You’re in luck: one project has released extensive open source documentation. TileToy is a similar concept in that it uses blocks and radio frequency-to-serial connection to a computer. (It adds a nice extra: interactive LED matrices on top of the blocks let you display different patterns.) The system works via a patch built in Max/MSP. (It could also be easily ported to Pure Data or another free environment if you’re on a budget.) Even if just part of this project appeals to you — like the RF aspect, or the grooving LED lights — it could be well worth checking out both the project and the “do it yourself” section:

TileToy.org

The DIY section includes a parts list, assembly guide, complete guide to controlling the hardware, software for Max/MSP, and software for communicating with the PIC. Might want to cut your teeth on something a little simpler if you’re a beginner, but file this away for when you’re ready to make the jump! (Thanks for the reminder of this goes to Chris at Pixelsumo.)

TileToy is the latest of a series of similar projects. We Make Money Not Art covered Digital Cubes by Simon Schiessel way back in January 2005, and from 2003, the “father” of these kinds of projects is Ryoto Kuwakubo’s Block Jam for Sony (thanks, again, Chris!):

Block Jam [Sony Design]

Other good tangible computing resources out there? Let us know!

Previously:
Babies Making Electronic Music on Video, and More on Traditional Japanese Instruments (Interactive teething ring, also built with Max/MSP)

[tags]DIY, hardware, alternative-interfaces, design, how-to, physical-computing, tangible, electronics[/tags]