LaDiDa Demo from khush on Vimeo.

There’s no question iPhone/iPod touch development – really, just clever mobile development – has gotten a bit overhyped lately. But that’s all the more reason to do a round-up of genuinely interesting stories, real innovation happening on the platform. So, I’m clearing out my inbox with some of the more creative tools appearing recently on Apple’s mobile gadgets. There’s no better way to kick off today’s festivities than with this unusual “reverse karaoke” creation.

Sure, people may think they’re tone-deaf. But even the layperson has extraordinary powers of musical perception. So how could you train your iPhone to perceive and respond to music? That’s the question asked by LaDiDa for iPhone, the first of a new line of “intelligent” music applications for mobile devices. A “reverse karaoke” tool, the idea is to listen to singing and fake accompaniment, rather than having you sing along to canned backing tracks. Nothing is pre-programmed; everything is generated on the fly on the device.

It’ll even make up a Bollywood accompaniment to your singing:

LaDiDa Bollywood Duet from khush on Vimeo.

Of course, to me, it’s interesting not only what the iPhone is able to musically, but also what these algorithms are unable to make sound musical. Both reveal a whole lot about how we hear and conceptualize music. I think the team deserves real credit for making this fun, though, and on constrained hardware.

The app’s creator Khush follows in the footsteps of Smule in that it takes hard-core academic music research and uses mobile devices as a vessel for getting that tech in the hands (literally) of the general public. (See my interview with Smule founder and ChucK originator Dr. Ge Wang.)

Parag Chordia, developed at professor at the Georgia Institute of Technology and the gentleman you see in the video, spoke to CDM about what’s happening behind the scenes. He tells us about how this application was developed, and how the intelligent algorithms work (or at least try to work, as music analysis and auto-accompaniment remain at early stages).

First, an explanation of the app.

Khush CEO Prerna Gupta explains how it works:

1. You sing into the phone, and LaDiDa will compose music to match.
2. LaDiDa’s patent-pending technology analyzes the pitch and structure of the melody to compose a unique accompaniment for each recording.
3. To be clear, we do not query a database of pre-recorded songs. That is, LaDiDa has been designed to work with any music.
4. After recording your song, you can hear it with different styles. LaDiDa comes with three styles — E Piano Pop, Rhythm Synth Pop and Dub Tone — each of which has been developed using high-quality instrumentation to work specifically with our algorithm.
5. We will be launching new styles every month that will be made available through in-app purchases.
6. LaDiDa also works on rap! This month we’ll be adding three new rap styles.
7. After choosing your style, you can save the song and share it on Facebook, Twitter and email.
8. LaDiDa also has a Discover page, where you can hear songs recorded by other users from all over the world.
9. Khush was founded by music technology enthusiasts from the Georgia Tech Music Intelligence Lab. You can read about us here and also find out more about the research at our lab here.
10. LaDiDa went live in the iTunes store last week and is currently priced at $0.99.

Prena, the woman you see in the video, has some Web experience to boot, too, including founding a popular Indian dating site. Oh, and she’s a better singer than the music researcher, but, hey, that’s why we all went into computer music, right?

In case you’re wondering how you take a research idea and make it run on the iPhone – or how the algorithm works (and might get smarter in the future) – I turned to Parag for those details:

The initial code was developed in my lab in c++. Since the core algorithms are basically mathematical, that portion was relatively easy to port. However, we spent significant time thinking about how to optimize for the iphone and every aspect of the app, from the interface to sound design, has been built with the iphone in mind. For example, there are significant limits on sampler performance — samples have to be short and effects are more or less out — but we thought it was important for our styles to have a rich sound. So we
put great effort into designing light styles that sound realistic.

Another significant challenge was making the analysis robust to external noise; iphone recordings are lo-fi and corrupted with tons of background noises, which makes robust (and again computationally efficient) pitch detection essential.

Our approach to reverse karaoke is somewhat different than what’s been done before. A significant limitation of previous work was a lack of fine-grained key estimation, a problem that we felt was critical to successful vocal accompaniment (most people are not anywhere near a piano or an instrument with fixed tuning when singing into the app).

We also worked on trying to give some larger structure to the
accompaniment, which can often sound locally reasonable but notably lacking in direction. Again, a difficult problem particularly when people are singing snippets. Still it is sometimes possible to detect phrases, and we have tried to incorporate that information as well.

Auto-accompaniment is an endlessly fascinating and deep problem. As we learn more about human perception and cognition of music, as well as improve our tools for machine listening, our systems will become more musical. While we still have a ways to go, we believe that, with LaDiDa, we’ve created a product that is engaging and allows regular people to express themselves creatively.

If all of this talk about musical perception recalls the questions about how culture and background versus neurology can be used to explain music – as seen at the Notes & Neurons conference – that’s no coincidence. Researcher Parag played sarod with a fascinating ensemble at that same conference. Bobby McFerrin sings a really beautiful solo with the ensemble.

In fact, it’s absolutely worth contrasting the elegance and beauty of these all-human musical responses to the somewhat clumsy (sorry, Khush) iPhone responses. That’s not to say the iPhone creation is any less human – it’s a computation model programmed by humans, and is capable of some impressive feats made possible by their musical instincts and training. As such, we really can hear the gap between what advanced musicians can do intuitively and what we can model computationally, atop the restrictions of the device’s ability to sense the world around it.

World Science Festival 2009: Notes & Neurons, Part 5 of 5 from World Science Festival on Vimeo.

jumping beans & .tape. from la bisogno on Vimeo.

What might a jar full of Mexican jumping beans sound like if composing their own ambient music? Scott Worley points us to a musical experiment by his labelmate Daniel Romero aka .tape, on netlabel yo.yo.pang!.
.tape programmed a sound environment in the free multimedia patching environment Pd (Pure Data). Contact microphones listen for the beans to jump, then use Pd’s onset detection (an analysis for transients) to trigger the sounds. Daniel reports the technique is “easy, but wholly effective.”

I’ll say – the music winds up being quite lovely, and rather than having a boring software-based random event generator, there’s something mesmerizing about watching the beans. You can download a free MP3/OGG file of the track, as well (and it sounds as though more projects may be coming):

pet-o-matic [asociación cultural la bisogno]

Descripción original en Español:

empezamos esta serie con la picante unión entre el músico Daniel Romero (aka .tape. ) y Pancho, Emiliano y Marcos, tres frijoles saltarines mexicanos

Sonidos y programación por .tape. secuenciación en directo por los 3 frijoles saltarines mexicanos micrófono de contacto + un “onset detection” en pd para disparar los sonidos. fácil pero rotundamente efectivo.

In other Pd news, the creators of the RjDj interactive/generative iPhone music app, which employs Pd patches, will be holding another sprint. This one will be located in London October 2-4.

Osmos Trailer from hemisphere games on Vimeo.

Osmos is a glorious glimpse of the fusion of electronic sound and game design many of us anticipate. It is built around a (challenging!) physics-based gameplay style – in the same vein as float-around-the-world games like flOw (and long before that, Asteroids) with procedural design and a perfect, liquid electronic soundtrack. Osmos became available DRM-free yesterday on Valve’s terrific Steam service for US$10 (on sale for a little less this week), and you can buy worldwide from the developer on Direct2Drive. (The developer has temporarily suspended direct sales from their site. Updated: the game is available again direct from the developer. It’s worth buying direct if you want to get a free coupon for the Mac and Linux licenses when they become available. Steam and D2D are great stores that support indie developers, though, and in the case of Osmos you don’t get any DRM with either one.) There’s also a free demo available. Versions for Mac and Linux are coming soon.

The roster of artists working on the music is simply a dream, including Loscil, Gas/High Skies, Julien Neto, and Biosphere.

I’ve already lost myself in the opening levels, and can’t wait to get deeper. CDM will have an interview with the creators by next week, once they recover from the launch.

And yes, indie gaming looks like a very fertile ground for digital artists and musicians, indeed.

Now, I’m not going to say anything else, because I want some time to play the game.

http://www.hemispheregames.com/osmos/

Sonoridades Líquidas [Liquid Sounds] from Rui Penha on Vimeo.

Wonderful work in sculpting sound into beautiful handmade electronic-acoustical instruments is pouring out of Portugal. Now we get to see more of that work – and if you are in Portugal, you can share in person at a new event.

The Handmade Music project and community has found its second home – Porto, Portugal, at the massive Casa da Música! The first event will be held Tuesday, July 21. (Full announcement, English/Português).

De hardware a software feito em casa até circuit bending, kits personalizados ou instrumentos acústicos originais, todos estão convidados a aparecer na Casa da Música pelas 21h30 para montagem de instrumentos. Pelas 22h abrimos o evento ao público geral – a entrada é livre e recomenda-se -, ocupando a Digitópia e a zona do bar do Foyer Sul. Contamos convosco!

To celebrate the new party, organizer Rui Penha sends over a survey of some of the instruments and sculpture that are being homegrown in Porto. And yes, even first-time-creator high school students can make fantastic work (see after the jump). At top:

An interactive installation made for Casa da Música – Porto, Portugal – with João Ricardo de Barros Oliveira and Luís Girão. Shown here is the algorithmic composition and synthesis patch I made in Max/MSP and the Arduino-based sensor system, co-developed with Luís Girão.

Lula · robotic Hang player from Rui Penha on Vimeo.

This robotic Hang player is called Lula (Squid).
It was, designed, built and its algorithm was created with a group of teenagers participating in a four-day workshop called OrCA – Orquestra de Computadores e Autómatos (OrCA – Orchestra of Computers and Robots), at Casa da Música – Porto.
It was made with picaxe, servos and ultrasonic sensors.

Correntes Sonoras from Serv. Educativo Casa da Música on Vimeo.

João Ricardo constrói instrumentos-esculturas a partir de materiais das mais diversas origens: pedaços de metal, bocados de madeira corroídos pelo tempo, objectos usados e danificados, provenientes de contentores do lixo, da rua, da natureza. As esculturas têm sons complexos, poéticos, muito diferentes do habitual e são um desafio à imaginação, aos sentidos e às convenções daquilo que é um instrumento ou daquilo que é música.

A ExposiSom [Correntes Sonoras] explora diversos espaços da Casa da Música e é concebida como um percurso de descoberta da arquitectura e da relação do som com o espaço que ele habita.

Sonoridades Líquidas from Serv. Educativo Casa da Música on Vimeo.

A ExposiSom [Sonoridades Líquidas] estabelece uma ponte entre a escultura sonora, a música e a arte digital e permite a descoberta do edifício da Casa da Música através de um percurso onde o público interage com o conteúdo sonoro, transformando assim a Casa num mega-instrumento. A ExposiSom [Sonoridades Líquidas] usa um conjunto de cerca de 50 objectos idênticos [esculturas sonoras] que servem de «interface» com um computador. Explora-se a ideia de elicitar sons ou algoritmos que geram som, a partir de um estímulo muito simples: a vibração causada por uma gota de água que cai sobre uma pele de um instrumento de percussão.

We hope to send out some resources for folks working to plan Handmade Music in other places, too. Here’s the running list of locales we know have expressed interest – and yes, I’ll be following up with you folks later today!

• Machester, UK
• (city TBD) North Carolina, USA
• Toronto, Canada
• Argentina
• Chicago, USA
• Berlin, Germany
• San Francisco, USA
• Joinville, Brazil
• Greece
• Los Angeles, USA
• Washington, DC, USA
• Austin, TX, USA
• Detroit, USA
• Boulder, CO, USA

Not on the list and interested in organizing an event? Fill out the form on Google Docs.

Stay tuned for coverage of the upcoming NYC and Portugal events here.

Gestural Music Sequencer from Unearthed Music on Vimeo.

Something as simple as remapping a single knob can give you new musical ideas. So expand that to entire gestures and live video input, and you can help push your performance in new directions and out of old habits. That’s why it’s always great to see projects like the Gestural Music Sequencer.

Built entirely in free tools – tools fairly friendly even to non-coders – the GMS lets composer and musician John Keston explore new ideas through gestures captured in a video stream. It’s easier to see than to talk about, so check out the just-completed documentary short by Josh Klos, with the aid of Julie Kistler and Brian Smith. (And yes, documentation makes a huge difference; we’d love to see more of this stuff!)

The ingredients:

• Processing, the free, multiplatform coding environment [site | cdmu tag | cdmo tag]
• controlP5, a lovely, light, quick-and-dirty library for UI controls
• Ableton Live – though you could substitute other software via MIDI, Live makes a nice, familiar interactive music engine

Lots more information on John Keston’s wonderful Audio Cookbook blog, which is fast becoming one of my favorite reads:

http://audiocookbook.org/category/gms/

And here’s a really lovely video that demonstrates what you can do with video. It uses a string of lights in a jar as the source. Yes, in a way, it’s almost like having a very focused random generator, but I think there’s nothing wrong with that. There’s an almost analog approach to seeing the source, and using that to organically create music.

GMS: Chromatic Currents Part II from Unearthed Music on Vimeo.

I have to observe, while this works reasonably well with MIDI, it reveals why standardizing on networked communication, as OSC does, makes more sense. In a world of software, “controller” can really mean anything you like. Control is increasingly about software talking to software – including when devices are involved, since they generally have a software layer of their own. Also, because sometimes it’s easier to code this with Processing than with Max, I can see some powerful uses of the Python-based Live API, which we expect to mature later this year. (Yes, the project called Live API seems to be in a holding pattern, but we may be able to work up a more complete, Live 8-ready alternative.)

By the way, our goal is to make noisepages a platform and collection of tools for people doing this sort of work (or anything creative with music and motion), even if you host your blog elsewhere. Stay tuned for the details on that.

Eno had Music for Airports. It’s fitting that Monolake would do Music for Jet Lag. Robert Henke writes about this month’s free download:

Since I also have been flying a lot recently, I named it after one of the most annoying side effects of modern transportation and mixed it in a way that reflects that dizzy feeling of being hyper active and totally asleep at the same time. ( "Last call for mister Robert Henke, flying to Berlin, please come to gate B 154 IMMEDIATELY or we will unload your luggage !!!!!!!!!" )

I am myself recovering from jetlag on the way to Portugal, so the timing is perfect. In a way, I have to say I sometimes oddly enjoy the disorienting feeling. I don’t think it’d be terribly addictive, but it’s a physical, profound reminder of traveling a great distance, something you could otherwise ignore in the age of absurdly-fast jet travel.

Grab the download here:

Free Downloads of the Month [yetlag, May 2009 – should be archived if you’re catching this late]

Installation details:

http://monolake.de/installations/lufthansa.html

The installation is fascinating in itself: a Max/MSP-powered, interactive sound score for a giant flight simulator, a model of the presence of jets, travel, and air traffic control. Robert did the sound; Christopher Bauder of white void was the concept and very elegant visual design. (See also Aaron Koblin’s striking Processing-based visual piece Flight Patterns, which seems to have embedded itself on a certain airborne digital zeitgeist. The United States becomes a feathery web of connections and flying traffic. You can imagine how this might continue to be mined in sound.)

As we work to keep our creative process flowing, I especially love the idea of focusing on a feeling to get a production started, as Monolake did here. So often, it’s too easy to get caught up in something technical or some very particular idea, then lose that in the process. By focusing on a feeling or deeper sentiment, it’s possible to remain connected to the ethos of what the track really means to us.

Of course, travel too much, and that may just wind up being … well, jet lag.

Meanwhile, as I listen to more music piped through airport terminals and even Metro stations, I wish Eno’s original idea had caught on.

Dreaming of a future in which music, instead of just being rendered audio files, arrives in fully generative, interactive form? Albums might “listen” to the world around you, and listeners could record their own alternate versions of music and share with others.

RjDj, the generative mobile music platform for Apple devices, realizes that future right now, instead of at some nebulous time in the future. In addition to the iPhone, you can make use of a second-generation iPod to use it. (You’ll need a headset with a mic; I have one by Griffin I’m testing.) And the RjDj folks have a whole bevy of significant updates to share:

• Free downloads (limited time): All three RjDj releases are available now for free. That includes the RjDj app itself (from which you can now grab and share releases), as well as RjDj Album (with a selection of generative/interactive/reactive releases) and the new RjDj shake.
• Download “scenes”: From the beginning, we knew that RjDj was imagined as a platform for other people to release interactive music. Now you can download scenes for free or fee. (Paid scenes currently redirect to the browser, but with iPhone SDK 3.0, you’ll be able to buy right from the app.)
• Share recordings: Because RjDj-generated music is controlled by the user and often records from the environment, the music may sound different each time. You can now share recordings with others from the device and the new social site.
• RjDj.me community: The RjDj folks have built a little community where you can share your favorite scenes and upload recordings, and keep track of scenes coming out from other artists.

By the way, with all of these releases, I know there’s some confusion. RjDj is the main app — the platform from which you’ll be able to grab scenes in the future. RjDj Album is a collection of “player”-style RjDj scenes — ones we have seen released previously. RjDj Shake is a newer, accelerometer-powered set of scenes. It comes with the amusing admonition that you should “PLEASE TAKE CARE, DON’T HURT ANYONE AND DON’T SMASH YOUR DEVICE.” If you missed Shake before, it now also has the recording sharing features. But the main RjDj app is the big release going forward.

We’re also awaiting news on more “sprints,” community-driven development efforts for making new scenes. And if you’re a fan of this sort of thing, RjDj isn’t the only game in town. RjDj itself is powered on Pure Data, the open source patching cousin of Max/MSP. Pd is making its way to other mobile devices; I even have it working on the BUG Labs gadget. At the same time, I’m curious to see if the RjDj gang can succeed in building a platform for lots of people doing this sort of work.

Regardless of how you look at it, more mobile generative / responsive music is most definitely in your future. Stay tuned.

Elsewhere:
Free trip into the rainbow vein: reality-enhancing iPhone app RjDj free with social update [Boing Boing Offworld]

RjDj.me community site
RjDj 0.7 out now [RjDj News]

RjDj [iTunes]
RjDj Album [iTunes]
RjDj Shake [iTunes]

Be sure to see our previous interview with the creators:
Exclusive RjDj Interview: Interactive Music Listening, Everywhere You Go

Also, expect more iPod/iPhone news and hands-on’s soon — I’m way behind, but let’s assume that means the best stuff will rise to the top.

As an addendum to the last story, Ivica Ico Bukvic sends along an example of the [myu] Max/MSP + Unity game engine combination in action. Here’s the surprise: Unity isn’t generating visuals. Instead, Unity simulates ripples created by movement in the space, and builds physical models that are sonified and spatialized by Max/MSP.

Speaking of work involving art museums and the combination of Max and Unity, VJ Anomolee notes in comments his own work with the pairing. Lightbent Synth is an in-progress piece with alternative controllers and sensors that produces sound with a novel visual representation (sound’s very quiet in this preview — more hopefully once it progresses):

Lightbent Synth from VJ Anomolee on Vimeo.

Ivica explains the top work:

This past fall [myu] had seen its first real-world implementation in an exhibit that was a part of the grand opening of the Taubman Museum of Art in Roanoke, VA (http://www.taubmanmuseum.org/). The exhibit utilized [myu] as part of an interactive aural installation titled "elemental." An online tech
demo video of the installation, including written synopsis is available also via Youtube at http://www.youtube.com/watch?v=PA-9BOgc1gk. Below is a brief synopsis of the installation:

"elemental" interactive communal soundscape premiered in November 2008 as part of the Revo:oveR collection commissioned for the grand opening of the Taubman Museum of Art in Roanoke, VA. The Youtube video focuses primarily on the technical aspects of the installation. Using Max/MSP/Jitter, a homebrew IR webcam with fish eye lens and a LED-based IR spotlights, entire 24×36-foot exhibit space is converted into an aural sandbox giving visitors an opportunity to generate and shape the
ensuing soundscape. Positional data of up to 20 visitors is forwarded to Unity3d using [myu] Max-Unity interoperability toolkit developed at DISIS (http://disis.music.vt.edu). Unity is used for physical simulation of ensuing ripples and the resulting data is sent back to Max for spatialization across a 12-channel (4×3) ceiling-mounted speaker array. Driven by communal interaction, virtual ripples refract from each other spawning an algorithmically generated aural fireworks. The exhibit ran non-stop for approximately 5 months until March 2009.

Bonus video below: an early prototype that did include visuals. After days of looking at emulated knobs and faders, it certainly does speak to some of the possibilities for musical interface and expression.

Take a powerful game engine (for animation, 2D and 3D graphics, physics, and on-screen interaction). Add the flexibility of a visual development environment for programming with virtual patch cords, for rich sonic and musical capabilities plus easy interaction with data and input. That’s the idea of combining something like Unity 3D with Max/MSP. In the example from earlier today, the solution simply routed basic data from a Unity-based game to a responsive music engine in Max.

In the case of [myu] – the Max Unity Interoperability Toolkit – that integration goes further still. Developed at the DISIS (Digital Interactive Sound & Intermedia Studio) at Virginia Tech, [myu] allows bi-directional integration of the Unity engine with Max or Pd. The two tools use netsend/netreceive to send data via TCP and glue the two together.

For visualists using Jitter, you can even exchange texture data, which offers some mind-blowing powers for live visuals.

Download at Virginia Tech — bonus, an extension of the aka.wiiremote object so you can use the lovely Wii Fit controller, among various other projects
Discussion on the Unity Community Forums
Discussion on the Cycling ‘74 forum
Virginia Tech DISIS

As an interactive prototyping tool, this should have a lot of potential for lovers of patch-style programming.

Thanks to Dr. Ivica Ico Bukvic, DISIS Director and researcher, for sending in his project. I’ll be curious to see what other people might do with this.

Sequencers by definition traditionally lock musical patterns into repetitive, unchanging blocks of time. But a new generation of generative sequencers can instead form organic patterns that change and transform.

Nodal is a totally free-as-in-beer (closed-source) sequencer for composing music. (A license is needed for commercial use.) As the name implies, it uses a matrix of nodes to represent musical structure. The best way to understand what that means exactly is to check out the examples and give the app a shot, but is good fun – and capable of creating some lovely, unusual musical textures.

The good news now is that if you’re on Windows XP/Vista, you’re no longer left out of the fun: the app now runs Universal on Mac and on Windows, as well.

Aside from Windows support, also new in version 1.1:

• New, more polished UI
• Keyboard shortcuts
• Internal MIDI synth support on Windows

It does sound as though Nodal may not remain free-as-in-beer, but with some significant updates coming later this year will move into the cheap-as-in-beer territory. Stay tuned.

Composer and co-developer Peter Mcilwain sends along the video here with a live improvisation made in the software. It’s a bit Minimalist-influenced, but shows how you can use Nodal to drive some musical inspiration. Peter also explains just what Nodal means musically to him and the small but growing collection of users taking advantage of Nodal’s paradigm:

Interest in Nodal probably centres around the fact that the software enables sophisticated, or “deep”, generative approaches within an intuitive graphical user interface that is simple and easy to use. Many users say that the program can be learnt within a short space of time and that is fun, inspiring and is fascinating to use. Once a little experience is gained people come to recognize the rich possibilities that the network approach gives them. Nodal offers a wealth of compositional possibilities that enable users to explore transformations of musical ideas quickly and easily.

Much music has been made using processes such as looping. Nodal does this too, but with a number significant additions. For example, it is possible to have loops within loops. The musical diversity that is possible takes Nodal beyond from the concept of a loop into a territory that is more like creating maps of musical pathways with which musical possibilities can be navigated. Musical materials can be played forwards and backwards, recombined or played with different rhythms. All of these transformations can happen in real-time making it a rich tool for computer assisted improvisation.
In fact working with Nodal can be very similar to more traditional composition processes. Here a small musical cell might be created which is then developed, elaborated or transformed into new material. Seen in this way, composing does not necessarily involve making a string of new ideas but instead it is the exploration of different aspects of a single idea. By limiting a composition to a small number of ideas the music maintains coherence and a sense of unity. While composers have done this manually, and in some cases masterfully, Nodal enables the transformation of idea to be automated and extended to transformational processes that would be very difficult to achieve with pen and paper.

Nodal: Generative Music Software

Previously:

Nodal: Generative Music Software for Mac (Free for Non-Commercial Use)

Tiction: Animated, Nodal Generative Music App in Progress, in Processing

Alternative Sequencers: Elysium Generative Mac App and the Joy of Hex

A Mutating Drum Step Sequencer, New MIDI Library for Processing