Yesterday, I talked about two complaints of music developers writing applications for the iPhone. These come from developers who are really iPhone fans, who just want to get their software released and (for many music devs) better categorized on Apple’s store. Pajamahouse Studios, maker of the new Sonorasaurus remix application, follow up with a more detailed explanation of their situation.

These are not rejections; at least rejections are generally accompanied with some sort of suggestion of what would need to be changed. They represent the dreaded iPhone developer “limbo,” in which an application is neither rejected nor approved. For Sonorasaurus, that’s been the state of affairs for over two months. As the developers explain, there seems to be nothing unusual about their app:

• Library access: It doesn’t access the iPhone/iPod music library. (no application is allowed to do that, which incidentally limits a lot of the DJ app possibilities of the device) Clarification: The status of the music API itself is unclear; some developers report just this sort of approval delay when trying to use it. [Source] Also, access to files inside the media library is not directly possible, which can be compared to the status of Android.
• File access: A separate http server is provided, with a parallel library, for users to store their own tracks – again, something found on numerous other approved applications. This doesn’t use the included library.
• Included music / music distribution: Five included songs are for testing only – something found in a number of other, similar applications that have been approved. The application is not an alternative to iTunes for distribution.
• Media decoding: Custom MP3 decoding technology – something not provided on the iPhone – was separately licensed. Clarification: This was not meant to imply that you can’t do MP3 decoding; the developers meant to make the point that they were not violating patents or licensing by using their own decoding, which presumably they did for the purposes of building a DJ app.

Of course, whatever the reason, we’ve seen in past applications suddenly approved after weeks or months, so who knows what will actually happen with this app.

Read the full explanation:

In Limbo Pt. 1 [Sonorasaurus]

While reading that, though, I also have to observe how significant these workarounds are. Without launching into an Android versus iPhone debate – believe me, there are many, many things to criticize about the Android as a platform, especially relative to music –  none of these is an issue on the Android. Forget platform wars or fanboys. Alternatives are good. I’d hope that we do have more than one approach to how to do this. These approaches should have to compete with one another, as they offer different tradeoffs and advantages.

If music is becoming an application, this kind of freedom is important.

Point by point:

• Library access: Android’s standard, supported APIs provide access to the media library and user playlists. Clarification: this includes direct access to the files and the ability to read from the buffer of these files (with some effort), in public, documented, approved APIs, with no chance of having an app rejected for the use of these APIs. My understanding is that this is not exactly the case on the iPhone.
• File access: Users are free to put files on their SD card over USB, and off-load those files – neither possible on iPhone. And yes, these will be integrated with the media library; iTunes-style sync isn’t necessary.
• Included music / music distribution: Including songs is actually a bit of a challenge, but you can freely download content and store it on the SD card. Because Google doesn’t have an equivalent of iTunes, that includes creating your own alternative distribution methods – meaning a label or music store can do make their own outlet.
• Media decoding: Decoding technology is included on the phone, including the ability to decode the open OGG Vorbis format. Clarification: Some folks read this to mean that the iPhone can’t decode MP3s, which was not what I intended; the key point here is that Android has in-box support for free formats and byte-level access to the audio buffers they give you, by default, straight out of the user’s media library. That is not entirely the case on iPhone.

Beneath all of this is the major fundamental difference, which is that you can install applications for Android whether or not they’ve been approved for Android. There’s actually a checkbox in the Market that allows you to opt into installing other apps you’ve downloaded directly from a developer or from another source.

Again, I don’t mean to make a pro-Android argument. In fact, I believe many of these items are also true on Windows Mobile, Symbian, and upcoming Linux platforms; I just happen to be working on Android now, so I’m more familiar with it.

What’s important is that this represents an alternative approach to how to provide music as an application, one in which the user is free to load content on and off the device.

Specifically, this paragraph jumped out at me:

Another problem would be that Apple could see this as a means to circumvent iTunes as a means to sell and distribute music. This we also addressed. These songs can only be used within the App. They can not be removed from the app / device for use elsewhere (iTunes on the desktop, burning to a CD, etc).

Now, we hear from many developers that the iTunes integration is something that attracts them to the platform. Likewise, many content creators will want just these sorts of restrictions.

But what if you want fewer restrictions? Let’s say you’re an artist releasing Creative Commons-licensed tracks, and you want to encourage remixing, sampling, modification, and free use of your tracks. Or what if you’re a label or artist collective, and want to experiment with new ways of using mobile for distribution, beyond what’s possible with iTunes and Apple’s stores? The same qualities that may attract someone else should, I think, concern you. I don’t think that necessarily means you shouldn’t write an iPhone application with your music, but perhaps you should also consider trying an alternative platform.

There seems to be a growing sense that the iPhone Way is The Only Way. Obviously, that’s not the case. This very debate demonstrates just how much room for interpretation the distribution of content can produce.

Part of the beauty of the iPhone from its launch date was the notion of a Mac you could fit in the palm of your hand. That makes it ironic that, for so many developers, mobile platforms in general have turned into a way to fragment software, to make it run fewer places instead of more. There’s something to be said for designing to a specific mobile device, but on the other hand, how many developers would want to restrict where their creations run? And particularly in music, isn’t the appeal of mobile creation the ability to have your tools work in more places? Maybe targeting just one gadget is the right choice for a given tool, but it shouldn’t be the only choice.

On the iPhone, the application Thump has plenty in common with a range of mobile music making tools. It’s simple but fun, a groove box with sequencing, subtractive synths, drums, and a set of basic effects, plus the ability to load your own samples and export songs.

Thump also demonstrates how simplifying sonic capabilities can produce musically-beautiful results, by focusing on the essentials and creating something with personality. Here’s a track by its creator showing off its sounds:
thump soundreel by mazbox

Well worth checking out the app on your platform of choice:
http://www.mrkbrz.com/thump/

What might not be immediately apparent is that under the hood, Thump makes use of the open source environment openFrameworks. As a result, the same code runs on iPhone, Mac, and Windows, as well as Linux. (It’s not distributed for Linux, but it could be. Hint, hint.) Creator Marek Bereza gives back, too – he ported the openFrameworks audio library to the iPhone, where it’s available to anyone.

Updated Marek notes in comments just what this means. The video below is, essentially, the same app. In place of the iPod screen, he has used a massive lattice of physical controls. A separate installation at the same show used a large touchscreen and simplified interface. And this really demonstrates what cross-platform means. Guy Kawasaki in the 80s mocked “ports” as a cheap wine. The idea is not to simply dump your code on a different platform and hope for the best – in fact, in this case, the changes from one platform to another were radical. The key is maximizing what’s essential, what really is not specific to a single device.

Physical Sequencer from Marek Bereza on Vimeo.

Creative Applications has a detailed write-up of the installation with more documentation.

If you’re interested in creating your own projects, oF has an elegant syntax based on Processing, but adapted to C/C++ coding paradigms and libraries.

I would recommend trying desktop development first as you’re learning, but when you do want to go mobile, setup is absurdly easy:

Setup instructions for iPhone

Having this as a foundation means more freedom for users and developers alike, the ability to run where you want, and a shared community of artists helping one another out. It does not mean sacrificing revenue, either – which is important, because revenue is often what enables (and encourages) developers to make new things. Here, we see a commercial mobile tool and distribution via the iTunes store, with free-as-in-beer distribution of desktop apps. But even the Free Software Foundation actually encourages charging for distribution – and to many people, the FSF is the most extreme example.

The bottom line is, you’re free to create what you want, where you want – and it doesn’t even have to have a price of free.

The use of openFrameworks is possible because the iPhone allows the execution of native code. Apple does not allow the Java virtual machine, which locks out many useful platforms and tools — including Processing. Naturally, that should not be the case with forthcoming non-Apple platforms based on Linux; more on that soon.

Thanks to Memo Akten for the tip — and for his exceptional documentation of iPhone openFrameworks development.

GPU-tiful and Cross-Platform

Updated Another framework I left out is the lovely libnui. The tagline says it all: “C++ API, multiplatform (Mac OS X, iPhone, Win32, Win64, Linux), 3D hardware accelerated GUI and more…”

On the iPhone, Intua’s elaborate Beatmaker software and David Wallin’s bleep!box have both been made with libnui. I want to single out bleep!box because of its UI – note how UI elements are simplified and scaled to the size of your fingertip. That’s something that, speaking of multiple platforms, could be equally important with touch hardware coming to Mac, Windows, and Linux platforms (some of it here already). I saw a huge fan of Windows tablets loving Propellerhead Reason on a tablet. He probably would be less enthused running Reason or Record on a touch interface; the UI elements are way too small.

bleep!box is also worth a look for the same reason as Thump: elegant UI, simple functions, and a lot of fun.

Pd, too

Pure Data, the open source patching cousin of Max/MSP, can also run on mobile devices. It’s currently powering the application RjDj, but because Pd is free, that could be just the beginning. In May, I wrote about the work being done by Hans-Christoph Steiner, who has been porting Pd even to ancient PDAs and iPods – including those, notably, without native floating-point capabilities (the kind of number crunching usually associated with sound).

Well, now Hans-Christoph is looking toward other, current-generation devices, too, kicking off with a workshop on making interactive sound on the iPhone using Pure Data. Here’s a taste:

OK, kids, you won’t believe how easy this is. Once you have Pure Data installed on your your iPhone or iPod-Touch, creating and uploading your own sound-processing and sound-generating patches is as trivial as copying a text file to your device! You’ll still need an Apple Developer License, but because Pd patches are treated as media files, they don’t have to go thru Apple’s elaborate code review — so, you can just dive right in, turning your phone into a pocket synth within minutes. In this 3-hour hands-on workshop, interactive sound whiz and Pure Data developer Hans-Christoph Steiner will lead you through all the steps.

Hans-Christoph will be talking Pd on the iPhone at a conference next month at Carnegie Mellon University in Pittsburgh. That will also include Memo Akten and Zach Gage talking about openFrameworks. And for nearly-entirely open source hardware, there’s a tantalizing-looking workshop combining Android with Arduino, too.

http://artandcode.ning.com/

There’s quite a lot more to say about this, but I hope this whets your appetite.

And, of course, riding the train to Pittsburgh or your nearest university or open source workshop, now you’ll have Thump which which to make some thumping tracks on your iPhone.

LaDiDa Demo from khush on Vimeo.

There’s no question iPhone/iPod touch development – really, just clever mobile development – has gotten a bit overhyped lately. But that’s all the more reason to do a round-up of genuinely interesting stories, real innovation happening on the platform. So, I’m clearing out my inbox with some of the more creative tools appearing recently on Apple’s mobile gadgets. There’s no better way to kick off today’s festivities than with this unusual “reverse karaoke” creation.

Sure, people may think they’re tone-deaf. But even the layperson has extraordinary powers of musical perception. So how could you train your iPhone to perceive and respond to music? That’s the question asked by LaDiDa for iPhone, the first of a new line of “intelligent” music applications for mobile devices. A “reverse karaoke” tool, the idea is to listen to singing and fake accompaniment, rather than having you sing along to canned backing tracks. Nothing is pre-programmed; everything is generated on the fly on the device.

It’ll even make up a Bollywood accompaniment to your singing:

LaDiDa Bollywood Duet from khush on Vimeo.

Of course, to me, it’s interesting not only what the iPhone is able to musically, but also what these algorithms are unable to make sound musical. Both reveal a whole lot about how we hear and conceptualize music. I think the team deserves real credit for making this fun, though, and on constrained hardware.

The app’s creator Khush follows in the footsteps of Smule in that it takes hard-core academic music research and uses mobile devices as a vessel for getting that tech in the hands (literally) of the general public. (See my interview with Smule founder and ChucK originator Dr. Ge Wang.)

Parag Chordia, developed at professor at the Georgia Institute of Technology and the gentleman you see in the video, spoke to CDM about what’s happening behind the scenes. He tells us about how this application was developed, and how the intelligent algorithms work (or at least try to work, as music analysis and auto-accompaniment remain at early stages).

First, an explanation of the app.

Khush CEO Prerna Gupta explains how it works:

1. You sing into the phone, and LaDiDa will compose music to match.
2. LaDiDa’s patent-pending technology analyzes the pitch and structure of the melody to compose a unique accompaniment for each recording.
3. To be clear, we do not query a database of pre-recorded songs. That is, LaDiDa has been designed to work with any music.
4. After recording your song, you can hear it with different styles. LaDiDa comes with three styles — E Piano Pop, Rhythm Synth Pop and Dub Tone — each of which has been developed using high-quality instrumentation to work specifically with our algorithm.
5. We will be launching new styles every month that will be made available through in-app purchases.
6. LaDiDa also works on rap! This month we’ll be adding three new rap styles.
7. After choosing your style, you can save the song and share it on Facebook, Twitter and email.
8. LaDiDa also has a Discover page, where you can hear songs recorded by other users from all over the world.
9. Khush was founded by music technology enthusiasts from the Georgia Tech Music Intelligence Lab. You can read about us here and also find out more about the research at our lab here.
10. LaDiDa went live in the iTunes store last week and is currently priced at $0.99.

Prena, the woman you see in the video, has some Web experience to boot, too, including founding a popular Indian dating site. Oh, and she’s a better singer than the music researcher, but, hey, that’s why we all went into computer music, right?

In case you’re wondering how you take a research idea and make it run on the iPhone – or how the algorithm works (and might get smarter in the future) – I turned to Parag for those details:

The initial code was developed in my lab in c++. Since the core algorithms are basically mathematical, that portion was relatively easy to port. However, we spent significant time thinking about how to optimize for the iphone and every aspect of the app, from the interface to sound design, has been built with the iphone in mind. For example, there are significant limits on sampler performance — samples have to be short and effects are more or less out — but we thought it was important for our styles to have a rich sound. So we
put great effort into designing light styles that sound realistic.

Another significant challenge was making the analysis robust to external noise; iphone recordings are lo-fi and corrupted with tons of background noises, which makes robust (and again computationally efficient) pitch detection essential.

Our approach to reverse karaoke is somewhat different than what’s been done before. A significant limitation of previous work was a lack of fine-grained key estimation, a problem that we felt was critical to successful vocal accompaniment (most people are not anywhere near a piano or an instrument with fixed tuning when singing into the app).

We also worked on trying to give some larger structure to the
accompaniment, which can often sound locally reasonable but notably lacking in direction. Again, a difficult problem particularly when people are singing snippets. Still it is sometimes possible to detect phrases, and we have tried to incorporate that information as well.

Auto-accompaniment is an endlessly fascinating and deep problem. As we learn more about human perception and cognition of music, as well as improve our tools for machine listening, our systems will become more musical. While we still have a ways to go, we believe that, with LaDiDa, we’ve created a product that is engaging and allows regular people to express themselves creatively.

If all of this talk about musical perception recalls the questions about how culture and background versus neurology can be used to explain music – as seen at the Notes & Neurons conference – that’s no coincidence. Researcher Parag played sarod with a fascinating ensemble at that same conference. Bobby McFerrin sings a really beautiful solo with the ensemble.

In fact, it’s absolutely worth contrasting the elegance and beauty of these all-human musical responses to the somewhat clumsy (sorry, Khush) iPhone responses. That’s not to say the iPhone creation is any less human – it’s a computation model programmed by humans, and is capable of some impressive feats made possible by their musical instincts and training. As such, we really can hear the gap between what advanced musicians can do intuitively and what we can model computationally, atop the restrictions of the device’s ability to sense the world around it.

World Science Festival 2009: Notes & Neurons, Part 5 of 5 from World Science Festival on Vimeo.

The creators of Fruity Loops aka FL Studio (Image Line software) have just announced a new DJ application called DeckaDance, and this thing looks like the answers to our prayers. We here at CDM love applications that don’t tie you into a specific controller/hardware/control vinyl solution. DeckaDance seems to have everything we’ve ever wanted in a DJ application, and not only does it support several different types of control vinyl, it actually has a learn mode to adapt itself to new types! Add the fact that it’s a VST host capable of hosting both effects AND instruments, and is capable of operating as a VST plugin as well, and I think we have something that will truly change the face of DJing (unlike some other recent products). Here’s a run-down of some of the more notable features:

1. Works as standalone or VSTi plugin
2. Hosts VSTi soft synths & FX in 8 slots
3. 8-slot sampler that samples from the decks
4. 6 inputs & 12 outputs
5. Supports Final Scratch, Serato Scratch, MixVibes and (our favorite) Ms Pinky vinyl
6. Capable of learning new vinyl control systems


7. Auto-sync

Most surprisingly, it says that a Mac / Audio Units version is planned, in addition to the current Windows version. With the availability of Image-Line’s Poizone and Toxic III soft synths for OS X, perhaps it’s not as big a surprise as it would have been a few years ago, but it’s nice to see ImageLine pushing forward with multiple OS support (I’m guessing that a Linux/Jack port is further down the line :-).

The only glaring omission I see so far is a lack of support for FLAC files (although it does support MP3, WAV and Ogg), but I’m sure we’ll see that in a future version. No release date has been set, although Image-Line says it’s coming “real soon”. If the performance aspect is there, and if this thing has a similarly powerful workflow and rock-solid stability like it’s sequencer brethren Fruity Loops, I think Traktor (and DJ Decks, and Torq, and everything else) is in for some serious competition.

A demo is already available.

DeckaDance Sneak Preview

Ed.: This is just the sort of thing we’ve been waiting for. A full-featured DJ plug-in that works inside other apps means you can add DJ sets to your existing live sets, scratch during a performance … the lines between DJing and performance continue to blur. And an open DJ app could finally free the full-time DJs from reliance on hardware/software platforms, at just the right moment as Final Scratch customers got burned by the Native Instruments – Stanton divorce. We’ll watch this closely, for sure. -PK