Day 32, Project 365 - 11.21.09

Touch DJ, one of the early DJ entries on iOS. These apps could tie more closely into iTunes libraries on the device, broadening their appeal. Photo (CC-BY-ND) William Brawley.

Following mobile music making means keeping up with technical details that are complex and changing. And because the Internet is open, when you post a story on iOS DJ apps, odds are it may be read by some of the Apple audio engineers, third-party developers, and a casual DJ with little understanding of what’s behind the scenes. But I’ll say this: the behind the scenes stuff matters, and it’s a great gift that we’re all able to compare notes.

Last week, I noted that, via the developers of a scratch app called Flare, there was evidence that Apple had broadened developers’ ability to work with music from the iTunes library in iOS 4. That led to an explosion of comments, including many from developers. The issue is still one we’re exploring, but I’d like to clear up some questions.

Also, I want to invite developers interested in comparing notes and sharing code to join a new group on Noisepages, the community site we’re finishing developing this summer:
Next-gen Mobile Music + Visual Dev Hack Group

The group has forum-style discussion features, the ability to connect with other users, and a group blog which could be useful for posting code snippets and the like. That means if you want to share something you’ve discovered with other developers or ask a question, you can. We’re not trying to replace other resources, but to make it easier to find them – so you can point to a stackoverflow thread, or your Git repository, or your commercial developer blog. Part of the reason to invite you is that this July and August is when we’re establishing other features for launch, so if there’s something you wish the site did, just suggest it. You only need to sign up for a Noisepages account, which is free and now open to anyone. I kept this platform agnostic partly because I hope we can all learn more about different operating systems, and because there’s plenty of overlap.

But first, let’s clear up some of those questions users and developers might have about Friday’s story.

Why all this fuss over having access to the iTunes library? Aren’t there DJ apps already?

For a lot of DJ end users, the ability to manage music with iTunes but DJ with a mobile app is important. In the past, developers have had to keep a separate set of files for use in the DJ app, synchronized separately, which for many end users isn’t good enough.

Wasn’t this added before? This isn’t news.

Actually, this is news. iOS 3 added very primitive access to the library — that is, you could see what was in the library on the device and play and stop, but nothing else. DJ apps will naturally want to be able to do things like change pitch or scratch – or even play the volume – which weren’t available in iOS 3. That means access to the buffer data for the audio in order to process it directly. (Those are the numbers filling your head Prince was recently complaining about.)

So, what’s the story? You can access audio buffers in iOS 4?

Here’s where things got a bit confusing. Because there’s a new set of APIs, music developers on iOS are still working out what’s possible and how to implement it. As a developer, you really don’t know a platform feature until you’ve successfully used it in your own code. Even with a documented feature, you may discover subtle issues – and new features often aren’t fully documented, if at all.

Async Games, makers of Flare, did manage to pull audio files from a device library and scratch them. So far, so good. But to get there, they required an intermediate step; what developers really want is direct, full-blown access to the Library. (And yes, I got a few messages in my inbox from developers who thought I was overstating what Apple had provided.)

To see the discussion of what’s there, what isn’t there, and how to use it, complete with code samples, see comments on Friday’s post.

Async Games and some of our regular iOS developers readers have been helpful to me in untangling this. But while Flare was the first to officially scratch an iTunes track, here’s some good news – it does sound out like the makers of DJ Mixer have figured out how to build a whole DJ app. There’s still that intermediate step, however. DJ Mixer developer Aviv Eyal tells me:

It works by copying the audio file into the App’s sandbox. MP3, M4A, AIF and WAV are supported.
Once there, the App has full access to the audio data and it calculates the BPM, beats, bars, extracts cover image and other meta-data that helps in mixing.
After this one-time import, the music is mixable and can be played without additional importing.

http://djmixer.fm/

And as I said on Friday, even for non-DJ apps, this could be significant. You could make a sample-based performance app, for instance, that allowed your user to manage samples via iTunes on the desktop instead of a separate conduit. So, for instance, an artist might take all of her albums, load it as an iTunes playlist, and then remix her back catalog in a mobile app.

It could also mean creative visualizer applications for the iTunes library. Stay tuned; hopefully we can document how to actually do this. (Join the group above.)

(Yes, some readers say they prefer a separate file store for their DJ tracks; if that’s you, you don’t have to worry about any of this! As a developer, though, you have to cater to the users who want to use iTunes.)

Here’s a new video of the upcoming 2.1 version of DJ Mixer with the iTunes Library functionality; it’s currently awaiting approval from Apple as I write this.

What about Android? You said the situation was different on Android. This is all about different philosophies of openness, Apple’s lockdown, blah, blah…

Well, yes and no. I brought up Android on Friday to provide some context – and not because I wished to make a black-and-white comparison about which was “better,” but on the contrary, as a way of highlighting the unique challenges all mobile developers face.

Here’s the deal. First, Android does allow developers freer access to files loaded on the device. That is a partly philosophical difference; since before the iPhone, Apple has maintained a fair degree of control over media files synced to their iPod. For instance, when you mount an Android device on your desktop, you can freely drag and drop files into the music database. That’s actually been a cause for complaint among some reviewers, but I will say, even a lot of less-sophisticated / less-techie Android owners I talk to like it.

In this case, though, these issues are less relevant to the task at hand. The issue with a DJ app is that you want to get buffer access not just to any file, but very possibly a compressed file. Part of the reason neither Android nor iOS has made this very obvious may even be that both platforms are using hardware acceleration for some decoding tasks, and they’d need to get the data back from the decoder.

Anyway, this isn’t a platform discussion, and it isn’t philosophy – it’s good, old-fashioned, nuts-and-bolts, hands-dirty development.

For the record, when I said it was possible to directly access files on Android, I wasn’t incorrect, but I was simplifying matters in regards to decoded files. There isn’t a software-side decoder that comes out-of-box with the platform. It is possible to decode MP3s, however; you just need to add your own library and dig into the NDK. The MAD MP3 library would be a way to do that, and it’s free. I’m going to actually look into this, because I need for a project. But yes, assuming we do work out how to use them, it looks like the iOS platform will be a path of less resistance thanks to these recent changes. (In-box decoding could be an issue on new platforms, too, like Chrome and MeeGo.)

The bottom line? If you’re building a DJ app right now on mobile platforms, be prepared to roll up your sleeves, because some things you take for granted on desktops simply aren’t there. On the other hand, there’s such interest in the problem, you won’t be alone – and I suspect we’ll be getting code examples going in relatively short order.

So, definitely do join up the Noisepages group if you’d like to learn from fellow coders; otherwise, as always, stay tuned here. And if you’re an end user, of course, we’ll explain what’s available to you as these apps ship. Having a window into development simply means you’ll know what’s coming a little earlier.

  • Dj Hombre

    I'm hoping the guys behind sonorasaurus Rex will get onto this asap, as this is a topic they've been battling with apple for a while now.

  • http://the-palm-sound.blogspot.com/ PalmSounds

    What I'm really interested in is the ability for recorder apps like MultiTrack to use the iPod Library too.

    For me that makes life a lot easier than bringing lots of tracks in one at a time.

    See this post:

    http://the-palm-sound.blogspot.com/2010/07/ipod-l

  • Dj Hombre

    Yes, I'm totally interested to see what the multitrack guys can achieve too. Makes sense to me that all my content can be quickly synched via the fairly robust method of iTunes rather than a manual workaround into a shared location….will also save duplication of files I guess, where you have the same track required in several apps.

  • http://async-games.com/flare.html Rhythmic Fistman

    Re Android. Who said libMAD was free? It's GPLed and if you want a commercial license you have to sign an NDA before they'll even tell you how much it costs.

    Then there's the small problem of libMAD not being written in Java, so you'd have to delve into the NDK and limit the number of devices you can target.

    Plus I can't even get paid by the Android Marketplace in my country.

    Android's a mess.

  • http://www.createdigitalmusic.com Peter Kirn

    @Rhythmic Fistman: Note that libmad is "a way," not "the way." You're correct about it being GPL (v2), but that's actually not the worst of it — the bigger issue is patent and licensing, because MP3 the codec isn't free.
    http://en.wikipedia.org/wiki/MP3#Licensing_and_pa

    That's not strictly an Android problem, of course; it's more of an MP3 problem. Apple solves this by shipping their own codec, and apparently that includes software-side decoding NDK … uh, not sure what you're talking about. You can target any device with the NDK, unless you mean 1.5 vs 1.6 etc., though that's becoming less of an issue. I'm not sure I'd want a pure Java decoder, anyway.

    But no, the problem is the GPL and moreover the fact that with Apple, Apple is paying the license fee; with Android, you have to deal with it. And that's a problem with MP3.

  • http://async-games.com/flare.html Rhythmic Fistman

    Hi, @Peter Kirn – Android phones not having an MP3 license is a problem with Android phones, and a problem that iDevices don't have. Sorry if that seems unfair.

    Here are the current obstacles to writing decent Android DJ apps

    1) No MP3 decoder

    2) max 3 finger multitouch (actual numbers may be lower)*

    3) google checkout less than ubiquitous

    4) Java**

    * consider that Flare Scratch uses 5 fingers, and Baby Decks 10.

    ** now I'm just being mean

    So I guess theoretically you _could_ write a DJ app for Android (after you decide how you want to pay your licensing fees, percentage or flat rate?) but you'd need to have a fanatical devotion to the platform. These problems are not present on iDevices, which favours its DJ app ecosystem.

  • http://www.createdigitalmusic.com Peter Kirn

    @Rhythmic Fistman:

    It's not correct to say Android doesn't have an MP3 decoder. MP3 is a supported format on Android. There's just not buffer access using the in-box decoder. You can add another decoder using the NDK. No, that's not ideal.

    Yes, multitouch support on Android handsets right now is very poor. That's a handset issue; the physical digitizers are currently limited to two touch. (The API could support more than 3 theoretically; there just isn't handset to justify that right now.)

    I think, though, it's ironic to count Market availability against Apple's iTunes. You'll recall iTunes didn't launch with all the countries it has now. And unlike on iOS, you're free to sell software installers through other sources – not optimal, indeed, but still a possibility. (This also opens up things like supporting some interesting Chinese markets if you can do localization.) There's a pretty big marketplace out there for both iOS and Android.

    If Apple really wants to supports its DJ app "ecosystem," they'll keep working on these APIs and come up with something better-supported and documented. Yes, absolutely, they have a head start.

  • Gavin@FAW

    I think though that the elephant in the room is the single audio output on the iPhone. Visual mixing, mono output headphone splitter cables, beat detection…while kind of work are still a hack for having a real monitor output, which I why we won't see a pro dj whipping out an iPhone anytime soon.

  • http://www.createdigitalmusic.com Peter Kirn

    @Gavin: yes and no. It's a fair point, and of course the ability to support USB devices is useless since the API is limited to those two outs.

    I think people are using the device with a mixer and another source. So, yes, ironically, you wind up with a setup that's less portable, but…

    Then again, while "DJ" is the word that keeps coming up, it may be the recording apps integrating with iTunes that wind up being what people actually use.

    Anyway, it's interesting navigating this brave new world.

  • Gavin@FAW

    I'm not sure even with dual outputs if the iPhone would be a professional solution. Looking at the pacemaker, I have yet to see it on stage so to speak. Think the iPhone has loads of potential for people to enjoy music in new ways, but not so sure about the DJ paradigm, as you can see from some of the app store reviews. It ends up frustrating the general users who doesn't know how to beat mix with pro's staying away as they know better and are aware of the limitations.

    The iPad is a different story though, I could easily see an app that would replace say a single Pioneer CD-J being used in a pro situation.

  • Gavin@FAW

    just to clarify, by DJ paradigm I mean non quantized full tracks that you need to beat match on the iPhone ;)

  • http://async-games.com/flare.html Rhythmic Fistman

    Hi @Gavin, there's no elephant – the DJ just has to buy two iDevices, just as they used to buy two turntables.

  • http://async-games.com/flare.html Rhythmic Fistman

    @Peter Kirn, I'm sure you're right, but right now Android not a very compelling platform for creating digital music apps. Flare Scratch started its life in the beginning of 2009, on the extremely limited iOS 2.2, yet in the 2nd half of 2010 Android is still nowhere near as capable aurally speaking. That's a pretty big gap.

    I'll check back in a couple of years.

    Cheers,

    RF.

  • Gavin@FAW

    @Rhythmic Yeah, but still, its never going to be a professional solution.

    I think Flarescracth is great, it is what it is in the same way Touch Mix is.

    The thing is when DJ apps claim to replace $1000 pro systems. I've been DJing for 18 years and if asked could probably quote the technical specs of the japanese record cartridge that David Mancuso's used in his loft parties during the 70s and from my point of view I can never see any pro dj using an iPhone.

  • http://www.createdigitalmusic.com Peter Kirn

    @Gavin: nope, agreed; I think they can't. Of course, if you added all the things they're missing, you might simply have … a computer. A $500 laptop these days is perfectly capable of running any "pro" DJ software. Add a $200 hardware interface and you're pretty much good to go. (Oh, wait, I guess that does add up to $1000 total… but, still, it's a *total*, which isn't bad.)

  • http://traxpad.physipop.com Joel Pryde

    This is a great discussion, I've definitely been looking into the issue of iTunes music loading myself.

    I've been working on a professional quality iPad DJ app (currently on version 1.1 and working on a ton of features for 1.2). The app is still in a fairly early stage but I think it will satisfy a lot of people looking for a pro-level dj app. Please check it out and let me know what you think!

    http://traxpad.physipop.com

    -joel

  • someobody

    hy joel i write you 2 mails but you not aswered. do you finnished your updates and ad time display pitch numbers etc. i buy your app and waiting for updates