Visual programming or “patching”, producing custom software by connecting on-screen objects with patch cords, until recently had only niche appeal. The domain largely of academic computer musicians, patching was scoffed at by computer science departments and unknown to everyone else. Lately, though, something very strange has been happening: this technique, popularized by experimental music synthesists, is being reborn in the Web age.

Patching for the Web

Patching software goes back to the 1980s, best known in its incarnation in Max (later Max/MSP, later Max/MSP/Jitter and Pure Data), software for making music and multimedia. Max is well known in these parts, not so well among the general public. But its basic patching metaphor, itself inspired by early hardware synthesizers like the Moog and Buchla, has filtered into other software.

First, Apple quietly acquired the developer of a little-known live visual/VJ app called Pixelshox, transformed it into a new app called Quartz Composer, made it part of the Mac OS X developer tools, and made it central to their UI efforts. One day, a tiny VJ app with a cult following, the next, central to Cupertino’s OS strategy? Interesting.

Quartz Composer didn’t exactly take the world by storm, but it did update Max’s taking on patching with a pretty, zoomable patching interface. Someone must have noticed, because more recently mighty Yahoo unveiled Yahoo Pipes, an online tool for creating Web remixes, with an almost pixel-perfect, color-exact clone of the QC interface. (Imitation is the sincerest form of Web 2.0, apparently.) Microsoft’s Popfly tool is a bit different, but even it uses an object and patch-cord metaphor.

Lily, JavaScript Patching

Enter Lily: Lily is a JavaScript-powered patching environment. Lily-created software can run standalone, as a Firefox plug-in, or even in a browser. Much of the functionality is Web-focused, as you’d expect: modules for mashing-up data from Amazon, Flickr, Wikipedia, Yahoo, and the like, widget support for popular JavaScript library elements, SQLite database storage and file system access — all good stuff, but primarily Web-based.

Lily

Where things get especially interesting is that Lily has some multimedia support:

• Graphics: SVG, canvas elements.
• Multimedia: Audio and video file support.
• Connectivity: OpenSoundControl support, which in turn could be used to connect to tools like Max/MSP, Flash, Processing, Reaktor, Traktor DJ, and others.
• Custom modules: Code your own modules in JavaScript.

There’s even a demo of Lily being used as a Beatbox.