What does music look like? With new sounds and new technologies, the question is more apt than ever. Tom of Music thing points, via his Twitter feed, to this interesting post regarding Ligeti’s Artikulation:

Visualizing Artikulation [Bad Assembly]

Music notation takes on a different meaning in the age of computers. After all, the essential divide in notation – between sound representation and realization – is blurred in the digital domain, in which we move between visual and sonic information seamlessly and a sound can be reproduced exactly. But, perhaps in that fluid context and without the musical conventions that grew up with notation, the importance of notation becomes that much clearer.

In this case, the classic experimental electronic composition Artikulation by composer György Ligeti has already had a visual score associated with it. Rainer Wehinger created the visuals above after the fact as an “aural score,” intending visuals to present a visible “reading” of the sounds of the piece. That makes the score itself closer to the digital visualizations we see as motion graphics works all over the Web (and on our sister site Create Digital Motion). The point isn’t to create a set of instructions by which you can perform a piece, but a visual counterpart that allows you to (presumably) hear it differently.

To be honest, I’m not always certain what to make of these results. Does this score really help you hear the piece? I’m curious to hear different reactions. But I wonder if the real holy grail comes back to software and interface. Seeing a pre-composed score is already interesting. But make that score interactive, and, in short, you have music creation software. Perhaps we’ll get beyond simple sequencers and step sequencers and start to see a growing number of interactive software designs that play around with that concept. (See Tom’s other thoughts on that today as he looks to Audio Damage’s new Automaton plug-in.)

Side Note: Twittering

If you want to follow us music bloggers on Twitter, I’m (uncreatively) peterkirn; Tom Whitwell is tombola. FriendFeed for me is the same. I haven’t made a CDM Twitter account; if for some reason that interested you, let me know, but otherwise I’m inclined to think RSS is just fine.

And if you have Twitters/FriendFeeds you think I should follow, please do holler.

Labels and artists are only now catching on to the idea of letting fans remix their music, and are even slower to give those fans access to individual stems. But where musicians have embraced this idea, they’ve gotten surprisingly big outpourings of support — thank a culture that’s gotten savvy with digital music tools and consumes more music than ever.

While that change continues to spread slowly, though, audiovisual remixing could already have a jump start.

Radiohead: Big news for fans of data visualization, the coding tool Processing, and Creative Commons: Radiohead have "shot" their latest video using only 3D scanning devices in place of cameras, and they’ve made source code and the data (in friendly CSV files) free. The whole thing is released under a non-commercial / ShareAlike CC license, which is well-suited to remixes in general.  So, to anyone who was disappointed that Radiohead didn’t use a Creative Commons license for their remix contest, now you’ve gotten something you didn’t even ask for — three-dimensional, animated data of Tom Yorke’s face. And because this is essentially raw data, it’s unusually open to interpretation.

Visual stems? By total coincidence, Create Digital Motion’s Jaymis wonders aloud if the entire A/V scene couldn’t be given a jump start by two obvious (but strangely elusive) decisions: 1. release video "stems" for music videos to give people free access to them, and 2. go get a real visualist. Some artists have done #1, of course, but there wasn’t a specific name given to the result, and they’ve more often than not released full videos — so here you go.

Both stories are covered today on Create Digital Motion:

Radiohead Makes House of Cards Video with 3D Plotting, Processing; Gives You the Data

To The Next Level of AV Remix Culture: It’s Time to Release Music Video “Stems”

But I think it’s well worth asking readers here on CDMusic, too. Music sampling and even remixing may be old news — even if copyright protection remains the norm. But could opening up visual remixes and free visual interpretation re-energize how people think about music?

Of course, this isn’t just for the sake of doing it. Jaymis launched his discussion partly because he wanted something more expressive at a performance, and Radiohead’s CC decision allows them to take an experience that would be pretty limited (a few minutes of cool video) and make it far less so (live data and code remixed by especially-savvy fans). Likewise, the CC license is essential in the latter case; there’s far less incentive to fans to code their own visual software if they can’t share ownership of the result, or — just as importantly — share the resulting code with each other. (The tool the band’s video used, too, wouldn’t even exist without the open source community that created it.)

So, what’s next — particularly if you’re not as famous as Radiohead?

The audience response to the work Joy showed was really overwhelming, as search activity danced around the globe and photos came to life in three dimensions. And it was nice to be able to show them the tool used to create these projects, Processing, and encourage people to try it out for free, even if they hadn’t tried programming before.

But I was surprised by how people reacted to a quick musical demo I closed with. Using Java, I wrote a simple program that checked my Gmail account using IMAP, then translated the time spam messages arrived into MIDI notes. I’m still developing a more advanced real-time version, so I threw the resulting SMF file into Ableton Live.

I’ll actualy be showing a newer version of this for Internet Week at an event sponsored by Make Magazine; more on that in a few days. (I’ll also use that as an opportunity to post some updated code.)

We spend so much time talking about how visualization can make data more expressive that we sometimes overlook other media. The spam “musicification” made sense to people partly because even the untrained ear is sensitive to musical timing, I think. Sonification of data isn’t always the right choice; the results can be abstract, though perhaps there’s value in that, too. But it’s worth remembering that people are sensitive to sound as they are to visuals. Since it’s not an either/or choice, necessarily, it’s too bad that so often designers neglect aspects of sound and timing while focusing only on what something looks like. It’s a challenge, certainly — there’s a reason most of us mute annoying sound feedback on computer interfaces — but I think it’s an area in which we’ll see a lot more discussion.

Now, data in smell-o-vision — that’s a story for another day.

Everything has an “API” these days, but what that means in practice is often not so exciting. You can make little widgets for Facebook, or post recent Twitter messages, or do other simple developer tricks. Echo Nest’s “Musical Brain” API is more far-reaching: it’s an API for music. All music online. The first of a series of developer tools, “Analyze” is designed to describe music the way you hear it, figuring out tempo, beats, time signature, song sections, timbre, key, and even musical patterns. More developer tools will follow.

Twelve years in development, the Musical Brain is a bit like a digital music blogger. It’s been crawling the Web while you sleep, reading blog posts, listening to music to extract musically-meaningful attributes, and even predicting music trends. It’s like almost like a robotic, algorithmic Pitchfork. (And I’m serious — it may be April Fools’ Day, but this is real. That’s what the “brain” claims to do, backed by research at UC Berkeley, Columbia, and MIT.)

What does all of this mean? The Musical Brain may not be replacing your friendly local music blog any time soon, but what it can do is infuse some musical intelligence into applications like music visualization. The Matmos album at top used the Analyze algorithm to map the timbral profiles of songs on an upcoming album; that graph was then rendered by an artist in watercolor, translating the digital into traditional paint media.

The Analyze API could also enable everything from Web music apps and mash-ups to live audiovisual performance tools, or even smarter music games. That’s the reason co-creators Brian Whitman and Tristan Jehan, both with PhDs from the MIT Media Lab, chose to open up development to a broad audience. I got to speak to Brian a bit about the project.

Machines that Get Music

“We’ve had it internally in code for a long time — but we want other people to make cool stuff with this,” Brian explains. “We have friends who make visualizers, VJ-type things, music mash-ups. So some of the things we’ve seen are visualizers. Video games is another one. We have a lot of friends who make electronic music, making Max patches and stuff, making chop-up kind of things [mash-up tools] now.”

Brian’s background was natural language processing. Tristan had worked in acoustic analysis. Both had seen research in automated music listening and machine listening, as well as new abilities to train computers to understand musical properties and human language. What’s been missing, says Brian, is a way to bring together these different research developments — mature after years of academic research — and put them out to a broader audience, not only of programmers, but artists, as well. Echo Nest is a way to do this.

Mash-up Machine

Right now, you can upload songs to the Musical Brain and try out analysis, and you can read up on the documentation and try some stuff yourself.

But if you’re anxious to see where this may be going, there’s also a first Web application so you can see how these tools could work in online development. It’s called “thisismyjam”, and it uses the API to create “micromixes”, instant mixes of your music. Type in music (it was able to handle a wide range, though it missed some more obscure examples I threw at it), and it will mash them together into a consistent mix.

Here’s my first experiment. I intentionally chose some disparate stuff that I liked to see how it would handle it, and let the algorithm decide automatically on ordering to try to match the music.

Because thisismyjam knows something about music, it was able to match my tune to something related.

Now, thisimyjam is mostly just a curiosity and demonstration for the moment. The attention deficit-style mixing of tunes was intentional, says Brian. The project grew out of their frustration with licensing rules, which limit the length of audio excerpts on sites like iTunes. “What we were trying to do is, if you want to hear an entire album in a minute, do it our way,” he says.

That said, it does suggest some of the possibilities of working with large-scale musical information. As opposed to something like Pandora, this is not manually-entered musical information; it’s all an algorithm. (See, by way of contrast, our interview with Pandora’s founder.) As such, even this basic example demonstrates what machines may be able to do if given some additional human-style intelligence. Mash-ups aside, these additional analysis tools could go beyond the basic audio-analysis and beat detection tools in current music software and create new audio-processing possibilities, even with audio you’ve recorded yourself.

Get Developing; More Information

Of course, the nice thing about all of this is is, like Matmos, you can get with the happy music picture-making straight away for free, using tools like Flash and Processing.

Developer Site (you’ll want to request an API key)

Analysis API Documentation

And if you don’t want to brave coding, there are plenty of examples to play around with:

Analyze API Examples

thisismyjam.com (free mash-up community site)

The Echo Nest

We’ll be watching this as it develops — and I’ll be messing with Processing, the free, artist-friendly coding tool for Mac, Windows, and Linux, so I hope to post some examples over on CDM Labs soon.

If you’re involved in related research, we’d love to hear about that, too.

In addition to the week’s worth of discussions and skillsharing classes, there will be two “showcase” nights, Saturday Jan. 19th and Sunday Jan. 20th. For those interested, I will be exhibiting two audiosculptural pieces, Octophonopod and Snowy Day during the event on Saturday. There’s a riduculous amount of talent on both nights, amounting to some of the most fresh and innovative people working in sound and motion graphics today.

[- Michael Una]

A full week of buzzing digital media creation will be showcased starting Tuesday, January 15th through Monday, January 21st 2008, at several venues downtown Chicago.

MGFest 08 features two world-premier theater screenings, multiple post-production studio tours, audio/visual art exhibits, a/v performances, six days of motion design / sound design / and motion programming classes.

These events celebrate creative minds and bright ideas within the emerging media landscape. Whether your a hard-core geek, a dedicated composer, a designer, film-maker, artist, engineer, programmer, educator, aficionado, prosumer, hobbyist, or just someone who likes being close to the creative community, mgFest is for you. The festival has become a destination for creative professionals from all over the Midwest and the nation by attracting the meshwork of companies and individuals that surround creative motion-picture design.

Socialize at one or all of the night events while experiencing multimedia art, video and sound in synch. Watch some of the past years most creative shorts or travel thru alternate dimensions in video art. Come mingle with local art directors and advertising executives, or discuss the technical operations of a professional edit bay and graphics shop with seasoned veterans of the industry. Listen to panels on the newest advancements in delivery technology affecting every media professional. Attend cutting edge classes of your choice within the realms of motion design, sound design and programming. Awaken your creativity at the Imagination College.

( Design | Sound | Code ) .: in motion

Check out mgFest.com for the most current festival information. 

Take the modular, patchable sound-making capabilities of Reaktor and (at the other end of the scale) Reason, and combine it with a graphical, zoomable, nodal interface with patch cords showing actual signal, as on the reacTable interactive table interface, and you should get something like Mod271. (Pronounced “mode.”)

The software is in pre-pre-alpha phase, but it’s freely downloadable for Windows users if you’re adventurous, and the developer promises more progress and other operating systems soon.

Features:

• ASIO/MIDI support, VSTi version coming
  
• Everything is full audio-rate, meaning you can mix and match MIDI and DSP
  
• Powerful nodes: “every node can be automated with unlimited control points and automation takes place right in the 3D environment.” … “nodes can influence any amount of other nodes or switched into a singular state.”, with radial and linear modes for the nodes
  
• Interface is 3D hardware-accelerated, and the signals even (optionally) draw at audio rate for realistic previews
  
• sample-accurate envelopes and motion, “meaning you can make an oscillator out of an envelope.”
  
• 25 node types and growing
  
• Make your own nodes with Python

Crazy stuff! It’ll be interesting to see how this one develops. And I hope the Reaktor developers are paying close attention; there’s a lot here that could inspire a future Reaktor version.

Full description, lots of background, and that bleeding-edge super-pre-alpha-at-your-own-risk:

Mod271 @NuDSP

Thanks to Ronnie of rekkerd.org for the tip!

Bridging the gap between music visualizer and game, Masaya Matsurra’s musika brings a unique experience to Apple’s iPod.

Matsurra, best known as the creator of PaRappa the Rapper for the original PlayStation, has long used Apple’s tools as a means of creative expression. He sees the iPod as a natural for interactive music visualization.

“The iPod is a device that revolutionized music and it is now poised to be a progressive gaming platform,” said Masaya Matsuura, President of NanaOn-Sha. “Many years ago Apple’s tools first opened my eyes to the power of music and multimedia, so it’s exciting to release my first game for this device.”

Like Apple’s own Music Quiz, which comes with the iPod, musika uses your own music library as the basis for the game. musika can be a passive experience — just kick back and enjoy the trippy visuals — or a more active one in which you press the iPod’s center button as letters in the song’s title appear. As you score points and advance to higher levels, you are rewarded with additional visual effects.

musika is almost brain-dead simple to play. If it weren’t for the superior eye candy, you’d tire of it very quickly. Let’s face it, despite Matsurra’s enthusiasm, the iPod isn’t much of a gaming platform. But as a quick diversion, it’s perfect. Musika is played with a single button and its use of your own music library is inspired. Just as Snake captured the imagination of an earlier generation of mobile phone users, musika is set to revolutionize mobile gaming.

musika is available for purchase and download from the iTunes Music Store for USD$5. It requires a fifth-generation iPod with iPod software 1.2 or later and won’t work on any other iPod or the iPhone.

Masaya Matsuura (Parappa) Releases musika For iPod [Gamastura]

Click To Play

Max/MSP visionary David Zicarelli is fond of saying that Max/MSP is really about numbers. You might hear music, but it’s number crunching that makes it all happen. Understand how to make the numbers work, and you can make your music and visuals do what you want. (Happily, this does not require a whole lot of math acuity, or I wouldn’t be able to do it. Instinct and imagination seem to be the best hallmark of Max masters.)

Lest you believe numbers can’t really make music, though, there are always bizarre and unusual examples of sources for Max projects. The latest comes to us from reader Stanley Ruiz:

Here is a clip of my audio-visual work presented at the 4th Asia-Europe Art Camp in Helsinki, Finland (June 2006).

I used the Philippines’ Gross Domestic Product (GDP per capita) as source to create sounds and manipulate video. GDP values were converted to MIDI data using a gesture-based sensor interface (the data is being sent as I move my hand). Converted MIDI values are then processed in a custom program written in Max/MSP. I used MIDIsense as sensor interface.

The output is an algorithmically composed music, as well as manipulated video (in this instance the video’s frame rate and contrast were manipulated).

… from Stanley’s blog.

(For more on sensor interfaces and MIDIsense, see our previous story.)

Okay, you can’t quite sing along, but Stanley is working on sonifying the GDP of other countries. Eventually, it should make the differences in affluence come alive in a way they might not on a bar chart. Got some unusual ways of working with numbers for music and motion? Let us know.

Looks can be a powerful agent for changing how we think about sound. Pairing liquid, organic speakers with equally fluid and dynamic visualizations, the launch of Muon last month in Italy made this principle readily apparent. I’m all about lo-fi, cheap gear here on CDM, but if you absolutely must launch luxurious aluminum speakers with spectacular animated visuals at a posh party in an Italian salon, I sure won’t complain. Pass the prosecco, please?

Muon Project Page, documentation videos at chrisoshea.org
See coverage at ze | d | esign, toxi’s project blog, MoCo Loco, elsewhere. (Yeah, CDM’s motto is: cover things last. Was a bit busy with Maker Faire!)
Created by Moving Brands

Details on the installation and how it was done:

Liquid-y Speakers: The speakers themselves were beautiful enough. Designed by UK speaker research center KEF Audio and Ross Lovegrove, a champion of organic, 21st Century design and one of the most respected designers on the planet, the key to the design is super-formed aluminum. The process does for metal something like what vacuum forming does for plastic: you heat sheets of aluminum so they can be molded into unique forms. The speakers themselves are formed out of single, 6-foot pieces of metal, into an acoustically-conceived, flowing form. I haven’t heard them, but we’ve already discussed (at a radically lower price point) why speakers really don’t have to be — or even shouldn’t be — rectangular.

Liquid-y Visualization: And that’s just the speakers. Part of the beauty of digital media is that they can make the invisible and the impossible visible in a dynamic way. So Muon creators employed London’s responsive media firm Moving Brands, who in turn brought in two of our favorite people — responsive media guru Chris O’Shea (see his blog Pixelsumo, and artist and Processing ninja Toxi (aka Karsten Schmidt). Working with creative director David Eveleigh-Evans, the team created a dynamic animation on a huge LED screen that could visualize the sound coming from the speakers and reflect in motion what the speakers do in product design.

How They Did It

The animation isn’t just a pretty visualization; it organically reflects what’s happening with the sound. Performing a spectral analysis of the sound (via a Fast Fourier Transform or FFT), the software uses amplitude levels in different zones of frequencies to produce particle objects, which spring and bob based on polarity, turning the peaks in amplitude in sound into a pulsating pool of fluid. The model itself is actually 3D, but it’s squashed into 2D space (or you can imagine looking at the 3D space from above). The other essential element is that the software looks at a history of amplitudes over time, so that overall changes can be adjusted (a bit like the simple “peak” meter on a consumer stereo EQ).

If you imagine an EQ meter using a pool of mercury instead of simple bars, that’s the basic idea.

The implementation is, as I’d expect from this team, simple and elegant — a few basic elements are tweaked to produce a maximal effect. Here’s the gear (software and hardware) used to pull it off:

Processing, the open-source, Java-based, simple coding environment for graphics and multimedia. (Trust me. You can code in it. Even a 10-line sketch can often be interesting, though sound-related stuff tends to get a lot more involved fast.)

Ess, which is one of a few competing sound libraries for Processing based on the Java sound API. (See also the JSyn-based Sonia, though there seem to be some compatibility issues with that one, and the newer, tongue-twisting Mimin.)
traer.physics, a wonderful and easy-to-use physics library. (I’ve used it in a few projects — a must-download.)

Toxi’s own libraries.

ControlP5 for debugging, a library that places on-screen controls in your sketch. (I imagine it’d also be hugely useful for performance, and can’t believe I haven’t tried it before!)

And don’t forget the ginormous LED screen from Creative Technology, “containing 73,728 full colour LEDs over a 10 x 5 metre floor, using the Barco MiTrix system.” Yum. 73,000 LEDs.

More stuff from Chris on the way. If you’ve got a Processing project for sound, we’d love to hear about it. It’s better-known on the visual side (and a regular subject on Create Digital Motion), but there are still MIDI and audio tasks at which it excels, even if you have access to tools like Max/MSP/Jitter. Right tool for the right job, and whatnot.