mysong Microsoft Research has done some amazing work; it doesn’t always move me to tears, but there’s some fantastic stuff that deserves real recognition. And MySong is … well, technologically impressive, if musically painful. It’s a sort of collision between AutoTune and Band-in-a-Box: it recognizes a melody as input, then harmonizes that melody.

The vocal input goes well, and illustrates the number of different inputs beyond the mouse you can expect in The Future. Here’s the problem: harmony is extraordinarily difficult to model on a computer because of the number of variables, the amount that’s driven by instinct and art. And let’s be blunt: it doesn’t work right.

In short: if you’re planning to build a Jerome Kern robot, the technology may not be there just yet.

If you’ve got a strong stomach, you can watch the application lay waste to “The Way You Look Tonight.” Speaking of tears: composer Kern actually drove the lyricist, Dorothy Field, to tears with the original. MySong might make you cry … in a different way. It chooses chords that fit a key and fit the melody, but completely unravels when it comes to making chords work horizontally with each other with the melody — which, when you think about it, isn’t all that easy even for experienced musician. The funny thing is, the harmonic structure of the song isn’t that complex (well, until MySong gets cranked to its avant-post-bop setting later in the demo). Harmony is perhaps just harder than the technologists may realize.

The researchers do compare their tool to Band-in-a-Box’s automatic harmony selection module, and this works better than that — but that’s not saying much.

I also have to admit, I’m getting a little fatigued of all these tools that want to dumb down music, as if somehow it’s music’s obligation to be push-button easy. Do we build giant robotic armatures so people can play basketball without practicing? Isn’t it the struggle that makes it fun? The researchers in this point seem to have missed the point: all those hours you spend sitting with an instrument working out chords are perhaps what music is about. There’s not some musical secret the experts are keeping from everyone else. The songwriter with the guitar very likely received very little training. All of that tweaking of melody and harmony is part of the process that eventually yields things like, well, “The Way You Look Tonight.” Jerome Kern and Cole Porter and Richard Rogers did it very quickly; amateurs may do it more slowly. But it may not be possible to reduce to rules in a way that the current generation of computing intelligence can even understand — and even if it does, it may require more than one or two sliders to adjust.

The best part of the video is the editable parameters: sliders for Jazz factor and Happy factor settings. (Theory fans: the approach seems to be for Happy factor to lobotomize to major I/V chords and Jazz factor to eventually turn everything into sus13.) I’d like to suggest a few additional settings for reproducing a broader variety of music:

  • Emo angst factor
  • Tone deaf factor
  • Pretentious techno chords factor
  • Stoned factor
  • Saccharine-sweet triteness factor
  • Community theater audition accompanist factor
  • Went to a liberal arts college where everyone on my floor played Ani DiFranco way too much factor

It’s well worth watching the demo. And, of course, this is the reason to tackle artificial intelligence — even if you’re unsuccessful, you’re learning. My guess is, we’ll need genuine AI before we can successfully harmonize melodies.

MySong, from Microsoft Research, makes your singing sound a lot better than it really does  [istartedsomething]

MySong: Automatic Accompaniment for Vocal Melodies [Explanation, Demos, Academic Paper]

  • chris

    One comment: "Hi. I wanna write a song for my girlfriend, but I can't play any instruments. Good thing I have mysong!" How pathetic he is. A real man would learn guitar for his girlfriend!

  • Wallace Winfrey

    How about the "Black Metal suicide is a solution factor"? Wait, the" Happy factor" is just a re-branding of that setting.

  • Johnny Horizon

    OMG – that video is hilarious. This is like Karaoke times ten. How much fun would this be at a party?

    Is the blog that video comes from serious or tongue-in-cheek? Tell me they're not serious.

  • http://www.createdigitalmusic.com Peter Kirn

    @Johnny: I think *parts* of this were a joke (like the girlfriend thing), but I'm not sure they meant this to be quite as comical as the MySong accompaniments are. And the blog is really serious.

    @Wally: lol.

  • http://www.myspace.com/wilsontherocker Evan

    Didn't watch the video, but "The Way You Look Tonight" is a tear-worthy song, I'd say. However, the only recording I own of the song is a Ella Fitzgerald/Nelson Riddle version that is painfully slow, and even the Queen of Song has to take some awkward breaths right smack in the middle of a phrase.

    "Just thinking oooooooooooooooooooooooooof *GASP* you, and the way you loooook tooonight. *GASP*"

    And how about the "Off-Key Hip-Hop Hook factor"?

  • http://pagancrow.spaces.live.com/blog ikyouCrow

    i can't see this being the final implementation of this technology. surely, they have other plans in store for the engine behind the recognition.

    and whatever happened to Monaco?

  • http://itp.nyu.edu/blogs/jao305 Jose Olivares

    This reminds me of the auto harmonize feature in Finale and a proposed search engine where you could hum the song you were looking for and it would try its best to find the correct song.

  • Pingback: Create Digital Music » The Real AI Jazz Factor: Think Different

  • martron

    @Chris: Seems to be the reason why it turned into a breakup song.

    @Everyone: The sample set of 300 pieces is probably too small and/or they haven't refined the model well enough to infer harmonic characteristics. The closest example to this that I can find at the moment is David Cope's work with Experiments in Musical Intelligence where he had EMI create songs in the style of certain composers based on analyses of their body of work ( http://arts.ucsc.edu/faculty/cope/experiments.htm <- mp3s). I consider this a lot more aesthetically successful.

  • martron

    I gave the wrong link for the EMI mp3s… The following is correct: http://arts.ucsc.edu/faculty/cope/mp3page.htm

    ;)

  • http://www.createdigitalmusic.com Peter Kirn

    @martron: see, this is why I shoot my mouth off; I always learn something. I have to admit, I know some of David Cope's work but hadn't heard the musical examples. They're really mind blowing. For instance, I can't imagine being fooled into thinking the faux Chopin was real, but it actually becomes, strangely, brilliant parody.

    The example above is anything but musical "intelligence." It actually sounds like bad beginning theory exercises — though that's also kind of interesting. It sounds like, aside from lacking a sample set large enough (which maybe also could be said of many beginning music students), the harmonic model is overly vertical. Just having proper voice leading doesn't necessarily make harmonies coherent against a forward moving melody.

    I also find it amusing that we got a spam trackback from a deaf spamblog.

  • http://www.retrothing.com/ James Grahame

    @Peter – It sounds like they've almost nailed the "Community theater audition accompanist" sound. All they need to add are a few ferocious vamps and the ability to push so far ahead of the beat that the vocalists rush into the final few bleating bars at around twice the original tempo.

    The reason people write songs with instruments is that nothing beats the immediacy of being able to try a handful of chord combinations in quick succession to find underlying magic that makes the hair on the back of your neck stand up. There's an incredibly fluid interplay between melody and harmony that occurs in real time, and which develops with experience. You can't access any of that by croaking monotonically into a microphone like a dying goat.

    The trouble with predictive music software is that it always looks backwards, incorporating progressions that have worked in previous hits, along with styles which were popular in the past. There's nothing new, unless you accidentally set the sliders to channel both George Gershwin and Frank Zappa in the same horrific tune.

  • Pingback: DeafPulse.com - the one-stop pulse for all Deaf-related news and blogs.

  • http://www.creativebump.com Myles de Bastion

    Hmm, what do you think of http://www.improvisator.de/en/
    and
    http://www.chordspace.com/ ?

    Pretty automated musical tools but the difference is that your decisions and feedback shape the overall outcome and some musical ability is needed to make something pleasing. It does arguable "dumb down" the theory side of music though..

  • Greg

    I think that the software is pretty successful, if only in the sense that most of the chords don't crash with the voices. And you have to agree that the result sounds better than what the users would have come up with sitting down at a piano for the first time.

    Sometime in the future (although not in our lifetimes) there will be software capable of composing tonal music at the same level as humans, or better. This would not make music less human, per se, because human involvement would still be needed to create algorithms, which is basically what composition is anyway.

  • mariano latorre

    It should have support for plugins so you can choose/create/download/publish music styles.

    Then you sing, choose parameters and a style, burn CD and become rich….hahaha

  • Paul van Dinther

    I think the pretentious slider is way up on you because you miss the point completely. This is not about dumbing down music production. It is about understanding music production. I agree they are miles away from their goal but as they learn they will get there.

    "Isn’t it the struggle that makes it fun?"

    Nonsense, Does better video editing software make video montage less fun? Is Photoshop spoiling graphics editing since it was so much fun in MS Paint? Tools are there to make things, having the tools is never a guarantee that you can actually create the thing the tool is for.

    I think taking music creation to the people is a good thing because rather than being passive CD buyers or MP3 pirates people get involved and entertain themselves making music. It doesn't have to be hard to be fun be it that easy gained results will never provide a high level of personal satisfaction.

  • Teresa Devlin

    hello iam deaf so i could like music drum

    cd and mp3

  • Pingback: Create Digital Music » Microsoft Research’s Songsmith Will Sell for $30, Match Accompaniment to Your Singing

  • jay Luttes

    where do you buy this softwear

  • ZZZ

    It is a FAKE OF Microsoft SO U COULD NOT GET ANY DOWNLOAD LINK