Remember the music industry? We used to talk about radio play and record deals. Now, we’re talking developers, APIs, and analytics. Of course, the test, now as then, is whether there’s actually substance for music listeners and artists. On Friday, we looked at Apple’s Ping and how, via TuneCore, artists who aren’t Lady Gaga can get their own pages. We also saw some vigorous discussion of TuneCore, which helps you get your music into “big bucket” sites like Amazon and iTunes, and SoundCloud, who together offer integrated sharing and distribution.

The Echo Nest is an unusual animal in this landscape, an “intelligent platform” for other tools built on machine listening. Via algorithms that analyze music and try to absorb human knowledge about music, Echo Nest powers apps that do everything from matching beats to workouts to tracking data for artists and labels. They even apply natural language algorithms to try to absorb text about music, meaning a robot may be … listening to … everything I say. Okay, moving on… (This leads to some interesting sci fi scenarios… No! Robot! Obey your master! You will NOT harm Katy Perry!) Check out how it works.

The latest news from Echo Nest: they’re partnering with online store 7Digital to connect all that data with some actual music sales opportunities. The result: a DIY online music store platform. Developers can create their own applications and music sales websites, drawing on all of the intelligent data on music from Echo Nest – from machine-analyzed news feeds to musical attributes of individual tracks – with the 7Digital store. 7Digital is now up to 10 million songs, with art, previews, and sales. And whereas a lot of deals (like Ping) are closed, in this case the “product” is actually a toolkit that relies on developers to do interesting things. Whether or not this particular idea gets traction, that’s an interesting shift in models.

If you’re skeptical about those machine algorithms, you’ll be gratified to know even Echo Nest’s own Brian Whitman conceded around the Ping launch that he has “a strong aversion to music recommenders and music similarity services.” And Whitman knows his stuff – he’s an MIT Media Lab PhD and co-founder of the company. But there’s no saying this has to be limited to recommendations – as those jogging apps that analyze tempo illustrate.

I spoke to Echo Nest to find out more about the new offering, and to see what they think about other trends, like Apple’s much-hyped – then much-criticized – Ping.

Jim Lucchese of Echo Nest and developer Paul Lamere respond.

CDM: How do the two APIs fit together? That is, given that The Echo Nest and 7digital each have their own open APIs to begin with, what’s unique here about the joint offering?

Paul: We’ve incorporated 7Digital Artist and song IDs into our Rosetta stone system. [Ed. As the name implies, that's a mechanism for translating all the different ID numbers used by music databases for tracking songs, since there's no universal numbering system for music.]

This means that developers can use 7Digital Artist and Song IDs with our APIs, and can instruct our APIs to return 7Digital IDs.

In addition, all Echo Nest results can be limited to the 7Digital ID space. This is useful for tasks such as search and artist similarity. For instance, we can limit the results of an artist similarity query to include just artists that are in the 7Digital catalog.

We have also applied the Echo Nest analyzer to each of the 10 million or so 7Digital tracks. This means that we have a very detailed understanding of what every song in the 7Digital catalog sounds like. We know the tempo, mode, key, time signature. We have a detailed understanding of the loudness profile of the song and of the rhythm structure of the song, we know where all the bars, beats and tatums are. We have a detailed understanding of the harmonic and timbral content of each of the songs. With the Echo Nest / 7Digital partnership, developers can get access to this detailed analysis for any of the 7Digital tracks and use this analysis for all sorts of apps.

The Analysis data allows developers to create interesting playlists using the new Echo Nest Playlist API based on the 7Digital catalog. For instance, a developer could create a playlist of songs by artists similar to Lady Gaga with a tempo between 120 and 135 BPMs. You can read more about the playlisting API on my blog [Music Machinery]:

Is that a million songs in your pocket, or are you just glad to see me?
Echo Nest Playlist API Methods

Is this something that might become relevant to other stores, too — provided, of course, they were as open as 7digital is with their data?

Jim: We expect so, though we’ve found that 7Digital is definitely the most forward-thinking around open API access and developer offerings.

You mentioned that monetization for developers could be a big draw. How does that work — is it simply a 7digital affiliate program, essentially?

Jim: Yes. It’s like an uber affiliate network of developers. Sell tracks through your app and get a cut of each sale.

What sorts of applications do you imagine this might encourage?

Paul: I see a wide range of possibilities here:

• Music discovery apps – given the deep data that Echo Nest has about music coupled with the content that 7Digital can provide will make it much easier for developers to create new music discovery and exploration apps – think of a next generation of Ishkur’s guide to electronic music
  [4], or musicovery [mood-based radio search]
• Music listening apps – Applications that help people listen to music – for instance I can imagine a pandora-style app that gives the user more control over the listening experience – ‘more music like this but with more energy’ or ‘give me music by hair metal bands from the 70s that are from the UK’
• Music Interaction apps – since developers have access to a detailed analysis of each track, it would be possible for developers to create interesting music visualizers that respond to the music
• Apps that we can’t even think of – never underestimate the creativity of a developer – someone will create something that we can’t think of around music that we can’t even imagine now, but will become a big part of our music experience.

Artists are becoming increasingly excited about engaging these platforms themselves, particularly as many of them get more savvy about development and use independent distribution to get their work out there. Is there a place for music makers to do something with this kind of technology?

Paul: That’s a big question – some areas where platforms like 7Digital/Echo Nest platform will be relevant to artists are:

• Interactive media – the recent Arcade Fire interactive video ‘The Wilderness Downtown‘ is an excellent example of the type of interactive app that artists will be able to create on top of platforms like 7Digital/Echo Nest/
• Analytics about fans – artists are spending more time figuring out how to market their music. APIs around data about who is listening to, going to shows, or writing about an artist will be very useful for artists

What about the end user (that is, the listener) – what will this allow a music lover to do that they couldn’t before?

Paul: As the music app space developers, we will see better applications for music discovery and exploration. A music lover will be able to find more interesting music that they will like.

Music listening apps can be greatly improved with access to all of the contextual information about the artist and the song, along with recommendations for other artists And you can see context-dependent playlisting – a user could create a ‘jogging’ playlist that has songs at just 135 BPM, or a high school reunion playlist of pop songs from 1985.

One of the things that struck me about Apple’s Ping announcement was that – while we’re waiting on information about any potential API – the general sense was of something pretty walled-off, specific to just their store, in the client software, and not connected to the Web and across platforms. Do you see a contrast in philosophy here?

Also, the initial reaction of the press surprised me; the first response was that Apple’s installed user base would just wipe everyone else out. I disagree, but how do you see open offerings like this fitting into (or providing alternatives to, or both) that Apple ecosystem?

Paul: There’s no question that Apple’s installed base makes their foray into social music a significant event. Millions of people who would never think of visiting a social music site like Last.fm will now become part of a social music network. That will be good. Music is very social, and one of the best ways to discovery new music is through one’s friends. However, social music systems can have some problems: Feedback loops, decreasing diversity in listening, susceptibility to hacking, schilling and spamming, and the cold start problem (where new artists have no way to break into the listening rotation) are all issues that plague social recommender systems.

Systems that can draw on multiple sources of data – social data, plus other data like curated reviews, blog posts and the content of the music itself can provide a way to avoid the problems inherent in social-only systems. I think that music discovery applications that are built on broad sets of data (like we are building here at the Echo Nest) will ultimately do a better job in helping people finding new music.

The most interesting possibility I see with the opening up of all the different music-oriented APIs is the emergence of a thriving music application developer community – (like we are starting to see coming out of the Music Hack Days). This developer community is ultra creative and they love music. They love to listen to it, and they love to create it. I am convinced that these developers are going to have as much of an impact on the future of music as the iPod did. These developers will be creating the applications that will help all of us to create, to discover, to interact with and to listen to music.

Shimon, an adaptive, improvisational, percussion-playing robot, is getting smarter – and more famous, with appearances in places like the Stephen Colbert show. Now, humans have been known to get a big head under such circumstances. Shimon’s head has gotten “more social” – gestural intelligence helps the robot relate to fellow players and nod its head in time to the music.

I got a chance to talk more to project creator Dr. Gil Weinberg, director of the Georgia Tech Center for Music Technology. He’s also taken some of this technology and built it into mobile app ZOOZBeat which you can spot in one of the videos here alongside the (much more expensive, no you can’t have one) robot.

What might surprise you about the Shimon project is that it isn’t just about copying what humans can do with a robot. In fact, if anything, says its creator, it’s about human-robotic relations. “The project was always aimed at creating new and inspiring interactions between humans and robots, with the goals of creating new and exciting musical outcome that cannot be created otherwise,” says Weinberg. He emphasizes that, while the robot assimilates human listening, it has a musical style all its own.

And yes, if there’s any doubt that Georgia Tech students can kick our ass in smarts and drum chops, even the ones who aren’t advanced robots, that’s Caity in the new promo video at top, an architecture major with LEED certification and drum line experience right up to the pros (having drummed for the Atlanta Falcons). That’s their fight song at the end. In fact, the only thing Georgia Tech students can’t do, evidently, is sing. (Though, based on my undergrad alma mater, I really, really, really can’t make fun.)

Shimon on Colbert from Georgia Tech on Vimeo.

Talking Machine Music

CDM: You talk in the NPR interview about not wanting the robot to play just like a human. So, I wonder – would you think of the robot as an expression of the musical taste and instincts of its designers? In some ways, it does sound different from a human; could we think of Shimon as having its own style, or being interesting because of its non-humanness?

Weinberg: Our motto is “listen like a human, play like a machine”. The idea behind it is that in order to connect with humans, Shimon has to understand music the way we humans do. For this purpose we developed perceptual modules based on music perception research for concepts such as tension and release, stability, similarly, etc. When Shimons responds, however, we want him to be surprising and inspiring, introducing new ideas that humans are not likely to use, whether by using mathematical processes that humans cannot process in real time or just through mechanical abilities. For this purpose we developed algorithms that utilize concepts such as genetic algorithms, fractals, morphing of HMM-based improvisation, etc. So for example, Shimon can respond by morphing the styles of Monk, Coltrane and his human co-player, in a way that humans probably will never use. In that sense he has its own musical style. For each different piece, though, Shimon may have a different style or “taste” based on the algorithm we use. I assume that one could say that his “taste” is a combination of all of his styles, which are inspired by the designers’ input.

How has the robot’s algorithm evolved since we first talked about it? What sorts of modifications have you found useful?

An important recent addition to the project is Shimon’s social head, developed with the help of my former post doc – Guy Hoffman. In order to create the connection with humans, we explored ideas of embodiment and gestures, as instrumental aspects of expressive musical group play. Currently Shimon can detect to the beat and nod his head accordingly, which helps humans get into the groove. He would look at what he finds interesting (if one player is playing something different than before, or different than the other players, Shimon would look at her rather than the other players). If Shimon plays something sophisticated, he is more likely to look at his own arms etc. We are also working on anticipating and coordination. We installed a camera in his head, and are currently working on letting Shimon use the visual input to anticipate and coordinate his playing with humans, along with the auditory information it currently processes.

Did the Colbert Bump go to Shimon’s head?

First Shimon was somewhat insulted, for himself and for the idiom of Jazz in general. Then he realized that they spelled his name correctly, so he started bragging about it. He also started to play in other styles to distance himself from the genre

But seriously, I am wiring a new piece for Shimon in an African Marimba Band Play style, which will help show its versatility in genres. (And here is an old clip where he plays an Indian Raga.)

Any other work we should know about, or other research growing out of the Shimon project?

To validate the importance of visual cues in music group play (i.e, validating the importance of Shimon’s embodiment and physically of the robot in comparison to interacting with computer generated music) we conducted this research: Visual Cues:
The Effect of the Visual Modality on Musical Ensemble Synchronization

Also, check out this paper, which won the Best Cognitive Paper in ICRA 2010. [That's the IEEE International Conference on Robotics and Automation, for those of you not in the know. And IEEE originated as the Institute of Electrical and Electronics Engineers. Whew. -Ed.]
“Gesture-Based Human-Robot Jazz Improvisation” by Guy Hoffman and Gil Weinberg [PDF link]

Shimon in Videos

Georgia Tech Center for Music Technology

It’s an old piece (Christmas 2004), but if you find your ears are ringing with retailers playing Christmas tracks on endless loop for the past few weeks, I can think of no better time for this. A Singular Christmas involved sixteen processors working for two weeks to compress the essence of Christmas music into sixteen singles. The results: tracks of droning, glistening sonic ice sculptures, like an ethereal pipe organ got caught in a wormhole.

Confused? See the easy diagram below. Now, didn’t that make that make a lot more sense?

The endless drones may put you in a sleepy trance, but that could be just what you need to recover from another holiday season. (Well, that or possibly dreaming about using Processing to code up A/V-synced Christmas lights next year.)

Best of all? The titles, like “Radiant bells,” “Hail the shining star” and “Berries sleeping.”

A Singular Christmas

An interview about what it was all about

Creator Brian Whitman: current site

As it happens, Brian Whitman hasn’t been sitting idly. He took all that machine listening knowledge applied to this project and went on to found the Echo Nest, conceived as an API for all of music. I need to catch up and revisit this project soon, but here was our first look, including an interview with Brian:

Musical Brain API: An API for Music on the Web – And it Makes Pretty Pictures