“The hills are alive /
with the sound of browsers”

Ever thought you’d make sounds in a browser, or have new ways of visualizing music playback? It’s happening, with builds of Firefox anyone can download.

Work to make browsers rich with sound synthesis and visualization continues. “Compatibility” isn’t really an advantage yet, because Firefox is the only browser with support, and only in the next version, though that could change in the future. And yes, Flash is capable of some of this, too (though not real 3D), with 90-95% saturation, conservatively, of computers. But if not compatibility, what these experiments do represent is what happens when someone working on a tool (Firefox, in this case) really commits to making sound a priority, and supporting free standards and developer tools (an emerging standard API, WebGL, Processing.js, etc.).

In fact, it’d be great if this occurred everywhere: if you’re making a platform, make sound a priority, and people will do mind-blowing stuff with your platform.

Among the latest fruits:

1. 3D eye candy. Charles Cliffe has a psychedelic visualization of sound playback. The JavaScript nuts are also proceeding to do more things with their language than most would deem possible, even moving DSP calculations to JavaScript code. I remain a bit skeptical there: the question to me isn’t whether JavaScript is “fast enough,” but whether native code is faster or simply the better tool for some jobs. Details below.

2. Patching in a browser – with a Pd clone. Chris McCormick is porting a subset of basic Pd objects to the browser. Now, one side of me wonders whether Pd is the best choice; it’s a somewhat idiosyncratic, if powerful, language for describing sound patching. But on the other hand, I could see this being fantastic in teaching and sharing: put basic patches up in a browser, let people play with them live, then build more advanced tools (with greater hardware access and external support than is possible in a browse) in the traditional Pd tool. As I keep saying, I think there’s far too much partisanship in the discussion (“Browsers for everything!” / “Browsers are useless!”), far too little thinking about how the browser and the desktop tool are more powerful together.

Web Audio Data API – Pure Data and Processing.js from David Humphrey on Vimeo.

Check out:
mccormick.cx/dev/webpd/
wiki.mozilla.org/Audio_Data_API

Also — heck, I may try this out in workshops as soon as next week. The browser could build a basic language for music and visuals in Processing and Pd, then robust performance tools could be built in the native tools, with quite a lot of compatibility between the two.

3. Actual standards. The W3C, the standards body behind HTML, has added this discussion to an Audio Incubator group. (It’s been incubating for some time, but maybe this will help something actually hatch.) Now I’d just like to see these things in Chrome/Chromium, too – I wonder if anyone’s up to a test build, as the standards adoption discussion continues. A number of readers have pointed out that MPEG4 had a specification that included, wholesale evidently, Csound. But this process seems more organic to me – you need actual tools and real-world experiments to evaluate the validity of something, not just standards on paper.

Putting the Awesomeness in Context: An Appeal

A side rant, though: why do Web geeks only care about what happens in the browser? It’s funny to me it seems that outlets like Slashdot jump on stories like browser-based tools, but ignore exactly the same ideas if they’re in a separate app. That’s not a criticism of the Mozilla crew or these brilliant hackers – this is what development is all about, pushing your tools to the limits. But if there isn’t a broader recognition of the value of what you’re doing or why you’re doing it in the first place, there’s a danger that unsustainable tool fetish will miss the point. That is, synthesis in the browser is excellent, but if people don’t understand the value of the synthesis itself, we have a lot more work to do.

Even the tools themselves need a context. It also JavaScript is amazing, but so are tools in Python, Java, Scala, and so on… and some of the enduring power of C still shows here. Browser powers are cool, but the OS is just as important – performance of Firefox would be heavily dependent on support for OS-native, low-latency audio outputs, like JACK on Linux. (Yes, it’s open source, so you can go do it yourself. No, I have no idea how to build Firefox for JACK – maybe a reader does?)

I’ve still yet to see a compelling explanation of what the browser really is, and what’s possible with its interface paradigm. That should be a fascinating discussion, actually, especially with the radical transformation of the browser, particularly as players like Google make it the central aspect of TV-watching or tablet experiences. But the discussion is only really interesting if you don’t start out with the value as a given. For instance, if browsers become a bigger part of what we do, is its simplistic tab metaphor really sufficient? If browsers simply bundle a set of native tools, are there ways “standalone” apps might adopt similar, standards-based approaches?

David Humphrey argues that part of the value here is the view source concept, but the Web has had the same empowering influence on sharing, collaboration, and reuse with platforms other than just JavaScript. The browser itself is a largely misunderstood piece of technology, partly because users (understandably) focus on their experience, and doesn’t pay attention to which aspects are delivered by the browser, the OS, or some other piece of code.

Oh, side note: this isn’t about “the cloud.” The cool stuff here is happening on your local hardware, period. That’s what makes it fast, and that’s what makes it work for audio, and your local machine is getting cheaper, cooler, and less power-hungry all the time. New DSP and floating-point capabilities in devices like tablets could make sound more powerful and flexible than ever before – provided people work out how to maximize, not squander, those capabilities.

So, here’s what I’d like to ask: what form will the standards discussion take? And how can these larger discussions – many of which transcend the discussion of any one tool or standard – find a forum?

Behind the Scenes, More Info

While you ponder that (and I’m open to suggestions), here’s more reading for you:
Experiments with audio, part X [Dave Humphrey's increasingly-awesome blog]

Previously:
Real Sound Synthesis, Now in the Browser; Possible New Standard?

More details on the first example, and how it was built (Minefield is Firefox 3.7):

All runs in real-time with Javascript, WebGL and HTML5 only (uses Minefield Audio build) — no browser plugins are used.

This demo combines the CubicVR 3D engine on WebGL (www.cubicvr.org) with the Mozilla HTML5 Audio API (hacks.mozilla.org), Processing.js (www.processingjs.org) and BeatDetektor.js (www.beatdetektor.com)

Mozilla Audio API is used to sample the HTML5 audio tag on the page, this information is processed by BeatDetektor.js which produces timing information for the Processing.js real-time canvas textures and the CubicVR.js procedurally generated WebGL scene using them.

The camera is set to free roam a simple chase pattern with a probability to follow a nearby cube (fully automated).

Available online at:

http://cubicvr.org/CubicVR.js/bd3/BeatDetektor3HD.html

or if you have a Float32Array enabled Minefield build:

http://cubicvr.org/CubicVR.js/bd3/Bea…

you can find more info about audio api-enabled Minefield builds at:

https://wiki.mozilla.org/Audio_Data_API

You can also feel free to chat with us about the Audio API via the #audio channel on irc.mozilla.org

Enjoy! And yes, I’ll have to work out a more beginner-friendly, here’s how to do this post.

  • http://www.jonbro.tk Jonbro

    I spent some time yesterday trying to get the audio patch working (not much), and I couldn't get it… ah well, hopefully someday soon there will be a mac build?

  • http://andrew.hicox.com plurgid

    Slashdot cares about browser stuff because they are a site about "internet tech culture". They don't care about when the latest version of Acid comes our or PD or MAX/MSP, etc. … because really their motto, "news for nerds, stuff that matters" isn't really accurate. It's just a more hard-core "Wired" with a serious Linux fixation … and I don't say that to trash them. I read slashdot and have almost since the beginning.

    Anyhow, this stuff is neat. Innovations around sound (and synthesis) in the browser take me back to the 90's. Back when bandwidth was a limitation, and we had to worry about how to generate audio on the client side. Like when MIDI files got support. There were some other interesting file formats doing rompler type stuff, if I recall correctly but they never gained browser support (wish I could remember them, there were several I believe).

    How to use audio effectively on web-pages (beyond flash things that will play an mp3 for you) is a design problem that has yet to be tackled in my opinion … and an area where there's still innovation to be had. Exciting stuff!

  • brrr

    intersting browser patching:

    (FF/JS based) http://www.lilyapp.org/

    and:

    (Air/Flash based) http://ffnnkk.org/launched/

  • Mudo

    Cubic player and Demoscene forever!

  • scott mc laughlin

    @jonbro, I have it working just fine on snow leopard using the Minefield version of FF from the Audio API link. Some bugginess but generally fine.

    https://wiki.mozilla.org/Audio_Data_API

    http://ftp.mozilla.org/pub/mozilla.org/firefox/tr

  • THE CLOOOOUD

    SUP DAWG, WE HERD YOU LIKE BROWSERS SO WE BUILT A JAVASCRIPT-BASED BROWSER SO YOU CAN BROWSE WHILE YOU BROWSE

    I mean, seriously, how long until someone does this?

    Cross-compatibility is nice, but other than that, what's the point of running everything inside a browser? Why not just make a cross-platform application platform so that code works identically on any OS, without tons of layers between the code and the CPU? Pd has enough trouble running on my 4-year-old machine. How is it going to run with another javascript layer in between?

  • http://www.musicvita.com music social network

    Hi guys,

    I'd like to invite all of you to become part of this exciting and fun music hub ever, Music Vita. You'll get to read the latest news, updates, and even band announcements in this site. All you have to do is to register for free, and then you'll get to enjoy the benefits of being in an interactive Social Networking

    Music hub you will surely not regret.

    So see ya all there guys!

  • anechoic

    SAOL:
    http://www.cs.berkeley.edu/~lazzaro/sa/

    I was interested in this when I was working in SV 10 years ago…not sure what ever became of it…

  • Pingback: HTML 5: BeatDetektor | MY FUTURE ME