LaDiDa Demo from khush on Vimeo.

There’s no question iPhone/iPod touch development – really, just clever mobile development – has gotten a bit overhyped lately. But that’s all the more reason to do a round-up of genuinely interesting stories, real innovation happening on the platform. So, I’m clearing out my inbox with some of the more creative tools appearing recently on Apple’s mobile gadgets. There’s no better way to kick off today’s festivities than with this unusual “reverse karaoke” creation.

Sure, people may think they’re tone-deaf. But even the layperson has extraordinary powers of musical perception. So how could you train your iPhone to perceive and respond to music? That’s the question asked by LaDiDa for iPhone, the first of a new line of “intelligent” music applications for mobile devices. A “reverse karaoke” tool, the idea is to listen to singing and fake accompaniment, rather than having you sing along to canned backing tracks. Nothing is pre-programmed; everything is generated on the fly on the device.

It’ll even make up a Bollywood accompaniment to your singing:

LaDiDa Bollywood Duet from khush on Vimeo.

Of course, to me, it’s interesting not only what the iPhone is able to musically, but also what these algorithms are unable to make sound musical. Both reveal a whole lot about how we hear and conceptualize music. I think the team deserves real credit for making this fun, though, and on constrained hardware.

The app’s creator Khush follows in the footsteps of Smule in that it takes hard-core academic music research and uses mobile devices as a vessel for getting that tech in the hands (literally) of the general public. (See my interview with Smule founder and ChucK originator Dr. Ge Wang.)

Parag Chordia, developed at professor at the Georgia Institute of Technology and the gentleman you see in the video, spoke to CDM about what’s happening behind the scenes. He tells us about how this application was developed, and how the intelligent algorithms work (or at least try to work, as music analysis and auto-accompaniment remain at early stages).

First, an explanation of the app.

Khush CEO Prerna Gupta explains how it works:

1. You sing into the phone, and LaDiDa will compose music to match.
2. LaDiDa’s patent-pending technology analyzes the pitch and structure of the melody to compose a unique accompaniment for each recording.
3. To be clear, we do not query a database of pre-recorded songs. That is, LaDiDa has been designed to work with any music.
4. After recording your song, you can hear it with different styles. LaDiDa comes with three styles — E Piano Pop, Rhythm Synth Pop and Dub Tone — each of which has been developed using high-quality instrumentation to work specifically with our algorithm.
5. We will be launching new styles every month that will be made available through in-app purchases.
6. LaDiDa also works on rap! This month we’ll be adding three new rap styles.
7. After choosing your style, you can save the song and share it on Facebook, Twitter and email.
8. LaDiDa also has a Discover page, where you can hear songs recorded by other users from all over the world.
9. Khush was founded by music technology enthusiasts from the Georgia Tech Music Intelligence Lab. You can read about us here and also find out more about the research at our lab here.
10. LaDiDa went live in the iTunes store last week and is currently priced at $0.99.

Prena, the woman you see in the video, has some Web experience to boot, too, including founding a popular Indian dating site. Oh, and she’s a better singer than the music researcher, but, hey, that’s why we all went into computer music, right?

In case you’re wondering how you take a research idea and make it run on the iPhone – or how the algorithm works (and might get smarter in the future) – I turned to Parag for those details:

The initial code was developed in my lab in c++. Since the core algorithms are basically mathematical, that portion was relatively easy to port. However, we spent significant time thinking about how to optimize for the iphone and every aspect of the app, from the interface to sound design, has been built with the iphone in mind. For example, there are significant limits on sampler performance — samples have to be short and effects are more or less out — but we thought it was important for our styles to have a rich sound. So we
put great effort into designing light styles that sound realistic.

Another significant challenge was making the analysis robust to external noise; iphone recordings are lo-fi and corrupted with tons of background noises, which makes robust (and again computationally efficient) pitch detection essential.

Our approach to reverse karaoke is somewhat different than what’s been done before. A significant limitation of previous work was a lack of fine-grained key estimation, a problem that we felt was critical to successful vocal accompaniment (most people are not anywhere near a piano or an instrument with fixed tuning when singing into the app).

We also worked on trying to give some larger structure to the
accompaniment, which can often sound locally reasonable but notably lacking in direction. Again, a difficult problem particularly when people are singing snippets. Still it is sometimes possible to detect phrases, and we have tried to incorporate that information as well.

Auto-accompaniment is an endlessly fascinating and deep problem. As we learn more about human perception and cognition of music, as well as improve our tools for machine listening, our systems will become more musical. While we still have a ways to go, we believe that, with LaDiDa, we’ve created a product that is engaging and allows regular people to express themselves creatively.

If all of this talk about musical perception recalls the questions about how culture and background versus neurology can be used to explain music – as seen at the Notes & Neurons conference – that’s no coincidence. Researcher Parag played sarod with a fascinating ensemble at that same conference. Bobby McFerrin sings a really beautiful solo with the ensemble.

In fact, it’s absolutely worth contrasting the elegance and beauty of these all-human musical responses to the somewhat clumsy (sorry, Khush) iPhone responses. That’s not to say the iPhone creation is any less human – it’s a computation model programmed by humans, and is capable of some impressive feats made possible by their musical instincts and training. As such, we really can hear the gap between what advanced musicians can do intuitively and what we can model computationally, atop the restrictions of the device’s ability to sense the world around it.

World Science Festival 2009: Notes & Neurons, Part 5 of 5 from World Science Festival on Vimeo.

World Science Festival 2009: Bobby McFerrin Demonstrates the Power of the Pentatonic Scale from World Science Festival on Vimeo.

At the World Science Festival in June here in New York, specialists – including musical specialist Bobby McFerrin – gathered to ask what in music we humans hear universally, versus what is culturally specific.

Is our response to music hard-wired or culturally determined? Is the reaction to rhythm and melody universal or influenced by environment? Join host John Schaefer, Jamshed Barucha, scientist Daniel Levitin, Professor Lawrence Parsons and musical artist Bobby McFerrin for live performances and cross cultural demonstrations to illustrate music’s note-worthy interaction with the brain and our emotions.

You can watch a series of five video highlights, but the one above is perhaps the most striking. (I believe it’s already more than made the rounds around the Interwebs, but, well, we can say we were all busy creating digital music.)

Notes and Neurons videos

It’s funny just how low the average person’s opinion of their musical ability can be. Ask an average “non-musician,” and they’ll often claim to be deaf to rhythm and pitch. Push the issue, though, and typically you’ll discover quite the opposite. Listen as the crowd laughs at discovering they all share some basic intuition about how pitch works. These are, after all, science and neurology types, not musicians.

Ah, you say, but this is just a crowd in New York. And most of us interact only with people in our own cultural circles. For me, that means people surrounded by pop music, Western harmony and counterpoint, chord changes derived from Protestant hymns — the lot.

What’s wonderful is that certain basic rhythmic and pitch elements – belying rich complexities of psychoacoustic phenomena underneath – do indeed seem to be universal. To me, that profound universality says something about what we share as human beings. At the same time, it makes me even more interested in all of the local details. When playing Balinese gamelan, some Western-trained musicians literally turned up their noses because they said the results sounded “out of tune.” Like a pungent flavoring in a foreign food, they discovered something unfamiliar. (I wonder if they would have the same reaction to sambal.) Of course, the underlying pitch systems are related to pentatonic (and heptatonic) pitch collections. And the same thing that disturbed one person has excited other musicians – not simply because it’s exotic, but because it can speak to something deeper in our hearing that we don’t get from other music.

Anyway, thanks to (noou) for this story, via IRCAM’s Eric Boyer; it really made my day. And it should certainly spark (ahem) some interest in neurology and the brain. Or, as I’m going to start saying whenever coming across something like this,

“Larry, what the hell just happened here?”

“I picked up some music,” Deborah wrote, “and held it open for Clive to see. I started to sing one of the lines. He picked up the tenor lines and sang with me. A bar or so in, I suddenly realized what was happening. He could still read music. He was singing. His talk might be a jumble no one could understand but his brain was still capable of music. . . . When he got to the end of the line I hugged him and kissed him all over his face. . . . Clive could sit down at the organ and play with both hands on the keyboard, changing stops, and with his feet on the pedals, as if this were easier than riding a bicycle. Suddenly we had a place to be together, where we could create our own world away from the ward. Our friends came in to sing. I left a pile of music by the bed and visitors brought other pieces.”

Neurologist Oliver Sacks has offered insights into the mind and consciousness before; here, he gives us a glimpse from an extreme world of just how important our experience is in all our minds. And if this doesn’t make you want to practice — in fact, realize what a profound experience that practice is — nothing will.

A Neurologist’s Notebook: The Abyss – Music and amnesia. [The New Yorker]

Thanks, Wally, who adds: “music is about being in the present – that seemed to be the thrust of things, and it was beautiful to read how this man who lives only in the present, with no remembrance of what happened just moments ago, was able to build his life around that. Ironically, after reading that story, i thought to myself — I will always think of this story whenever i listen to music or work on music.”