Developers for iPhone, iPod touch, and now the iPad have long complained about lack of access to the iTunes library, the file store and metadata for uploaded files. While version 3.0 of the OS provided limited playback capabilities (play/pause/stop), it was a far cry from what you’d need to build a DJ or other music app that made use of a user’s content. You couldn’t, for instance, adjust volume control, cueing, no mixing, or scratching.

Update/clarification: I may have exaggerated how much access is available. Obviously, Flare is working, as you can see, but the question of the exact mechanism by which it works and the documentation of the updated APIs is something that remains to be confirmed, beyond Asyn’c experience. I’m actively researching this issue. I’ve changed some wording to reflect the fact that what we do know about access to this functionality in the SDK, and how it works in the real world, is based on one shipping application for iOS 4 and the experiences of the developer of that app. The other issue is what mediates between the file in the database and playback; that is, unlike on desktop, you don’t necessarily get direct buffer access to files without an additional step. But the upshot is still that you can scratch tunes from iTunes, at least! -PK

Executive summary: It appears what has opened up is the ability to export assets from the library on the device, and following that extra step, use a local copy for access to buffers. See comments for more discussion. This would fall well short of being able to manipulate buffers from the file directly, but it’s a step. Stay tuned for further info next week as I confirm with developer sources.

With iOS 4.0, currently available for the iPhone and iPod touch, that changes, at least according to what we’re hearing from one developer – and seeing demonstrated in a shipping app. CDM has confirmed with developer Async Games that a new public API (meaning a fully supported developer tool) provides something closer to full-blown access to the iTunes library, in terms of not only metadata about tracks but also manipulation of audio data during playback. Result: you can now, for instance, scratch audio from songs uploaded from iTunes on a Mac or PC to the device. If correct, that would be likely to mean a coming flood of DJing on the iPhone and iPod touch, soon to be followed by the iPad whenever the 4.0 OS becomes available for Apple’s tablet. (Nor is this even limited in applicability to DJ apps – iTunes could become a more convenient way for syncing your own tracks and samples, or loading a playlist of backing tracks or other musical content onto your Apple mobile.)

The caveat appears to be that you need to “import” tracks from the library first; I’m confirming what that means and how long it takes. That would be shy of having direct, buffer-level access to tracks as seen in desktop DJ apps.

So far, the first app to actually ship with support for the feature is Flare Scratch, a simple simulation of a turntable with touch scratch support. You can scratch any song in your iTunes library, meaning this app developer has figured out a way to access audio buffers as they’re played. Apparently, the API isn’t entirely perfect, but it sounds like a step in the right direction, based on the impressions we hear from this developer. The developers of Flare Scratch also make a more full-featured DJ tool called Baby Decks DJ for the iPad, which could mean that, with iPad support, the tablet could become an all-in-one DJ solution. (Of course, a MacBook still has one significant edge – far, far greater storage capabilities.)

Flare Scratch @iTunes
Flare site

Incidentally, this isn’t an issue with Google’s Android platform, which provides open access to any file stored on the user’s SD card, and complete buffer access to that media (video, audio, the lot). Note that there may be some issues even on Android with using compressed assets; I’m writing some tests. Android developers, however, face a different set of challenges, like inconsistent handset audio drivers that can interfere with crackle-free, low-latency audio performance. (That situation looks brighter in the future, but it’s a whole other can of worms.)

Anyway, if you’re all alone on a park bench and want to scratch softly to yourself, you now have a solution. (Other videos, ranging from the useful tutorial to the somewhat frightening demo, at iflarescratch’s YouTube channel.)

I hereby open commenting to a bunch of people complaining about how this will ruin DJing and continue the slow decay of civilization itself. (Hey, I’m just the messenger. Leave me out of it.)

  • BirdFLU

    Using the video out capability of the Ipad, DJs could let the audience see what they're doing on the app, so maybe it's not a bad thing.

  • http://www.createdigitalmusic.com Peter Kirn

    @BirdFLU: That's actually a good idea. I think the app would need to support it, but that opens up some interesting possibilities.

  • Mastah Lee

    I downloaded this app about a week ago and I have to say this exceeded my (admittedly very low) expectations. The built in scratch sounds cover pretty much everything off of the SuperSeal vinyl and other classics, and given some practice I've been able to do some scratching that I wouldn't be embarrassed to perform live. Would I replace my Ms. Pinky DVS setup with it? No way, but if I did a set with just my laptop and a controller and wanted do some scratching on top I'd use this. My only complaint is that they went a little over the top in trying to have a rad-looking interface and all the non-scratch related operations are pretty unintuitive.

    Honestly, I'd like someone to use the touchscreen paradigm to explore a better approach to sound manipulation than just emulating the analog mechanisms. Why use the screen to just show a graphic of a piece of vinyl when you can skip the unnecessary metaphor and just display the waveform? "Scratching" a sample in the traditional sense with a touchscreen will never be better than vinyl if you play by the old rules. We need to throw out the baggage of what manipulating the speed and direction of a piece of audio is and construct a new interface from the ground up if there is ever any hope of audio manipulation on an iOS device ever being considered anything but a toy.

  • marc

    "Using the video out capability of the Ipad, DJs could let the audience see what they’re doing on the app, so maybe it’s not a bad thing."

    it would be as boring as watching someone putting a cd into a cd player…

    dj do the music , vj and lightjays do the show.

  • KVJ

    I couldn't find anything in the iphone developer documentation describing how to get access to the files ? Does anyone have any pointers as to where I should be looking for more info. on this ?

  • http://ezmyrelda.com Ezmyrelda

    About time.. Of course I got rid of my Touch to go to a party.. I guess I will have something to look forward to when I get another, or maybe even an iPad.

  • RichardL

    @KVJ iOS 4 augments AV Foundation Framework. This allows access iPod library metadata. I don't believe there's any way for an app to get access to the files.
    http://bit.ly/9jrM4G

  • KVJ

    @RichardL Thanks a lot.

  • http://www.createdigitalmusic.com Peter Kirn

    @KVJ/@RichardL: the developers of Flare have found some way to do this, and when asked, said it was via public APIs. I also have gotten a couple of second-hand reports from developer contacts suggesting this is correct.

    However, I've also been unable thus far to absolutely confirm the presence and form of this apparatus, or what it's called. ;) I'm researching. Obviously, the developers of Flare have *something* working, so it's a matter of working out how they did it and confirming that it is a public API with documentation.

  • Kevin Connor

    Access to the playout stream == graphic equalizer app I've been wanting since iPod came out in 2002. Finally comparable to my Akai walkman c.1987. Realtime pitch and time-shift apps for my itunes content. A million 'tube warmer' apps (ugh). I'll believe it when I see the API though.

  • http://blog.tapsquare.com Art Gillespie

    What, Peter, I don't rate as an iOS developer source? ;)

    Apple added a new property to MPMediaItem called MPMediaItemPropertyAssetURL

    You can only use this NSURL* with APIs in AVFoundation—notably, to create an AVURLAsset instance.

    The only way I've figured out how to get access to raw samples is to use AVAssetExportSession to create a local copy of the file. The problem with this method is it takes 1:08 for a 3-minute file—in my opinion, much too slow to avoid users showing up at the office with pitchforks. But Flare can import the same file in six seconds, so obviously they've found another way to do it.

    I'll happily post the code I have if anyone's interested.

  • http://blog.tapsquare.com Art Gillespie

    @Kevin We're not granted access to the device's output stream samples, we just have extremely mediated access to the files in the iPod Library themselves.

  • http://www.createdigitalmusic.com Peter Kirn

    @Art: Thanks for this. I've forwarded your comment to Async; hopefully they can clarify.

    I apologize for jumping the gun. I'll issue a separate correction / retraction / clarification with further information once I hear more.

    It does seem that Flare is pulling off some tricks we didn't previously think possible, at least. So to whatever extent they wish to share what those are, I'll pass it along. At least hopefully we can clarify what is and isn't there.

    Incidentally, this isn't an issue with desktop DJ development, or on platforms like Android; there you do have direct access to files and therefore to buffer data. Of course, the iOS devices are more protective about the files loaded onto them.

  • KVJ

    @Art Gillespie – It would be great if you could post that code somewhere.

  • http://www.createdigitalmusic.com Peter Kirn

    I had heard mixed signals on this. That would obviously not be ideal. Hold, and maybe I can work with Richard, Art, and Async and get one answer. And again, apologies if I was out there before I had solid information. (Welcome to the Web…)

  • http://async-games.com/flare.html Rhythmic Fistman

    Hi guys, sounds like speculation is rife!

    I don't think there's any doco for this stuff,

    you just need to read the header files.

    Good luck!

    And WRT to Android, there's no memory based MP3/AAC decoder, so you've got to license your own (pricey).

    Cheers,

    RF.

  • http://the-palm-sound.blogspot.com/ PalmSounds

    I think this is a mixed bag. I kind of like the fact that when I use TouchDJ I have a separate library of tracks just in that app.

    That way I can't inadvertently play a Fennesz track!

  • drumwell

    without turning this into a stackoverflow thread, has anyone looked into AVCaptureAudioDataOutputSampleBuffer? this looks promising, though i haven't played with it directly and it's unclear if it would work with an AVAsset obtained from an MPMediaItem.

    my company produces music apps for the iphone, and we had to do quite a bit of obstacle-engineering to get our applications (which focus on guitar instruction and play-along) to play along with mp3s. having access to the raw samples would have made this much easier.

  • http://blog.tapsquare.com Art Gillespie

    @KVJ

    Here's the XCode project with my method—I'll be sure to update if I hear of a faster way to pull it off:

    @drumwell

    The problem there is that as far as I know, the only inputs you can specify for an AVCaptureSession are microphone, camera, etc. (i.e., the devices returned from AVCaptureDevice:devices)

  • http://blog.tapsquare.com Art Gillespie

    Oops! Right, then, let's try html for that link ;)

    iPodLibraryAccess.zip

    @Peter Would be great if we could preview and/or edit our comments.

  • KVJ

    @Art Thanks a ton !

  • Pingback: DJ Apps on the rise » Apple Opens Access to iTunes Library on iOS

  • http://tommytench.com tmd

    hahahahahahaha sh*ts so weak

  • http://www.harmonicdog.com pwnified
  • http://www.harmonicdog.com pwnified

    MultiTrack DAW now has iPod library access, finally apple has approved it. Thanks Art for the initial ideas! MP3 exporting is slow because it has to be recoded as an m4a, but wav and aif are passthrough without re-coding.

  • http://jens.mooseyard.com Jens Alfke

    Wow, this is pretty big news! I'd read through the new API docs during WWDC but hadn't seen the part that gives you URL-based access to library files.

    Note that copying the audio to a local file may not be necessary for all uses, since you can do a fair bit just with the AV objects you obtain from the library. For example, you should be able to do basic mixing by playing multiple tracks at once and ramping their volume envelopes.

  • http://www.createdigitalmusic.com Peter Kirn

    Actually, now I need to look into those Android APIs. ;) The structure is fairly similar.

    I'm not sure about the MP3 decoding, though; so long as the decoder is built into the handsets (which for MP3, almost everything is), I think you're able to pass buffers to AudioTrack. I'll have to give it a try. You certainly have – as on iOS 3 – access to metadata and the file store.

    I haven't tried manipulating buffers from an MP3, though, so I'll have to give that a shot.

  • Pingback: iPod???????????????DJ Apps(Flare Scratch) : monogocoro ?????

  • http://www.jorgebarrientos.com Jorge from Madrid

    @ Mastah Lee:

    Scratching on a waveform:

    Try Protein DS Scratch.

    Is it possible to implement something like this on another device (the Nintendo DS has a little too crunchy sound sometimes)

  • Pingback: Ganalot! » Blog Archive » Apple Wish List 2010 (MacMost Now 340)

  • angstrom

    wow, the linkbacks in here are DEAFENING!

    Can we have more malformed javascript in here please, it's not quite unintelligible or enough.

    Damn, forget checking the comments for updates on this issue eh?

  • http://musiquenontalent.blogspot.com Robman84

    No idea how it works, but I can say Flare does indeed do what they say it does. I can now scratch away with Kraftwerk and the series of guitar lessons I have in my iTunes library…

    Maybe I need to finally load some more tunes onto my idevices…

    Roll on iOS4 support for iPad so babydecks can access the itunes library too.

    Or roll on VirtualDJ for iPad :o )

    Of course USB camera-kit support for multiple audio outputs becomes ever more urgent!

  • Josh Pressnell

    Thanks for the tips Art!! I've played with your project a bit and it APPEARS that if you set the preset as AVAssetExportPresetPassthrough and the output file type as AVFileTypeAIFF, that I can export MP3 files in near real time. The test I used in your app resulted in 1 second completion or less.

    I'm going to try enabling this in my own app and see if I can get it to work. More soon.

  • http://notes.variogr.am/ brian

    josh– i guarantee that aiff it generated will have 0 length.

  • http://notes.variogr.am/ brian

    (to clear up my statement, i meant that using Passthrough and writing to aiff definitely does not work properly– it generates a 0 length file for aiff and wav and caf and a empty header for a .mov with no data for any of them. i'd love to know what flare is doing but i don't think it's using avexportsession.

  • arjun

    That Flare demo video is from back in February.

    Why are you saying this is something new in iOS 4?

    This seems like a lot of hype about nothing.

  • Josh Pressnell

    Okay… as an update, while setting the passthrough option and the aiff output type works, according to the export session, an AVPlayer gives me a very strange and cryptic error when trying to open and play the exported file… so I'm guessing that something isn't working in the export.

    The error is: Error creating audio player(The operation couldn’t be completed. (OSStatus error 2003334207.))

  • http://createdigitalmusic.com Peter Kirn

    arjun:

    What's new is that you're pulling tracks from the iTunes Library on the device, and NOT having to keep a separate store of files for DJing.

    I'm a little tired of hearing the word "hype," though; does it mean something else than what I think it means? Just because I'm posting about something doesn't mean I think it's earth-shaking news, just that it's something worth posting. ;)

    But yes, I initially got this wrong — I thought there was the ability to pull buffer data directly from the file, which would have been a bigger deal (and maybe even worth a little hype). That's clearly not the case, but it is still more possible in the case of Flare for a user to pull files from their library, which is something.

  • http://notes.variogr.am/ brian

    josh– you get OSStatus because that file is 0 bytes long. NSLog the attributesOfItemAtPath, you'll see.

  • http://notes.variogr.am/ brian

    i looked into Flare's documents folder (did a backup and used Backup Extractor) and it appears they are somehow directly copying the mp3 over, no wav or m4a coding or anything. The file is the same except for there is no ID3 tag information in their version (but it is named Artist – Title.mp3) — i.e. no transcoding at all goes on. I am flummoxed on how apple would allow this, and more flummoxed that it seems to have nothing to do with AVExportSession.

  • arjun

    Sorry, Peter, but this article does read like hype.

    First, you say that iOS 4 is going to open the floodgates for DJ apps and that it's going to turn the iPad into an all-in-one DJ platform.

    Which got me pretty interested.

    But then you back it up with a demo of a cheesy music game, running on iOS 3. And it sounds like iOS 4 will let us play the cheesy music game with songs in the iTunes library.

    Is that it? Or am I missing something?

  • Josh Pressnell

    @Brian

    Well… just because they named the files artist – title.mp3 doesn't mean they're not using some form of AVEXportSession. It would be easy enough to grab the artist and title from the MPMediaItem and use that to create a file name.

    I've been digging all day today and I still can't come up with anything. I'm almost willing to bet that they're doing something "private" that Apple hasn't caught yet.

  • http://notes.variogr.am/ brian

    it's the same file, the same bytes. it would be a pretty amazing export/import if they transcoded an mp3 to another mp3 with the exact same bytes, especially an mp3 i had encoded in 2002. (let alone the fact that the API has no MP3 encoding preset!) they are obviously doing something private as there is no public api for pulling files out of the media library– the question is if this is private in the "back door method using existing public APIs" or "actual private APIs"

  • http://blog.tapsquare.com Art Gillespie

    @Everyone

    After much head-beating-against-the-desk I've figured this out so you can get speedy imports (about 3 seconds) with any file type. I'll post updated code soon.

    In the meantime, AVAssetExportPresetPassthrough only works if the destination filetype can act as a container for the source file's encoding. If you set AVAssetExportPresetPassthrough and AVFileTypeAIFF and you pass in an AVURLAsset pointing to an mp3, you won't get any errors, but you will get a zero-length aif.

    What's bizarre here is that AVFileTypeCoreAudioFormat (.caf) isn't a valid output file type for the passthrough preset—this file format seems perfect for the passthrough preset because it's a valid container for almost *any* audio encoding, but alas, if you try to use it you'll get an exception complaining about invalid output file type.

    The trick is to set the export session's preset to AVAssetExportPresetPassthrough and the output file type to AVFileTypeQuicktimeMovie—since .mov is a valid container for any audio encoding you can stuff m4a, mp3, aiff, wav, whatever without any decode/encode step. Fast!

    Of course, now you have a .mov file in your app's directory instead of an mp3. The AudioFile/ExtAudioFile APIs don't understand the .mov container, so you have to extract the relevant chunk yourself (mdat) and write it to an .mp3 before you can pass it to ExtAudioFile to decode for your app.

    Like I said, I'll post code that demonstrates this as soon as possible, but for anyone that's familiar with these APIs, that should be enough to get you going.

  • http://notes.variogr.am/ brian

    art — great, but i am having no luck doing exactly that. please do post your code! when i set Passthrough and output a .mov w/ FileTypeQuickTimeMovie I get a 136 byte empty .mov file no matter what the input is. Is there anything else you're doing in setup?

  • http://notes.variogr.am/ brian

    art– i got it. i had the AVURLAssetPreferPreciseDurationAndTimingKey on which was messing with it. hilarious, because i had the same idea as you and decided it didn't work. now it does. this has to be what flare are doing, nice work!

  • KVJ

    @Art We're not worthy!

    Incidentally, I was wondering what device you tested your code on (the version which did the transcoding). I tried it out on a Iphone 4 and it was pretty fast – a 12 minute mp3 took maybe 5 seconds or so.

  • http://blog.tapsquare.com Art Gillespie

    @Brian – Sometimes I think the only way to deal with these APIs is to create a table and try _every_ single permutation, which is what I finally did here.

    @KVJ – My slow transcoding was on iPhone 4, too. Are you sure you're getting valid, playable m4a's from those transcodes?

  • Christopher Penrose

    Is there someone that can recommend a troll filter for Safari, Chrome or Firefox, which prevents me from seeing comments labeling stories that I am quite interested in as "hype"? It would be very helpful for me. Otherwise, the comments are very informative…

  • http://leisuresonic.com/ Christopher Penrose

    @Art, are the slow transcodes coming from native m4a files? Or are they mp3s or some other format?

  • KVJ

    @Art I'll have to check it in a couple of days. I'd only tried this out on someone else's device, and I don't have access to it now.

  • http://leisuresonic.com/ Christopher Penrose

    @Art, ah saw your update… speedy transcodes for any file type… excellent work!

  • Zach

    How is Apple handling DRM issues? Are the exported files stripped of their protection?

  • http://createdigitalmusic.com Peter Kirn

    @Zach: My guess is that these are not DRMed.

    Oh, and for the record, since a couple of people complained that I wasn't correct regarding Android, we're sort of … both right. There's a GPLed MP3 decode library.
    http://stackoverflow.com/questions/2494499/mp3-dehttp://stackoverflow.com/questions/2336866/decodihttp://www.underbit.com/products/mad/

    So I gather that what iOS does for you is include in its frameworks MP3 decoding. (Presumably you could go the other way, and add a library for missing OGG Vorbis/FLAC decode support, right?)

    This comparison is perhaps work making, since the Engadget readers are turning this into a platform war that doesn't actually exist. It's more a matter of, you hit mobile and you reinvent everything. Sometimes it's a good thing (as noted by recent praise for the simplicity and stability and usability of iOS versus conventional desktop OSes). Sometimes s*** ain't finished yet.

  • http://notes.variogr.am/ brian

    zach: i haven't tried on an old style m4p file (you know that iTMS stopped selling DRM-encrypted files awhile ago, yes?) but my guess is if you give this process one it will gladly copy/"import" yet but ExtAudioFile will not work on it. So any PCM reading/manipulation won't work, which is the whole idea behind this. You can of course always play, mix, timescale etc both m4p and normal files using the higher level APIs.

  • http://www.harmonicdog.com Hamilton Feltman

    Zach, you can't access the DRM'ed files because the MPMediaQuery returns a (null) URL for them. Without that file url, there is nothing to send to AVAssetExportSession.

  • http://www.harmonicdog.com Hamilton Feltman

    Just wanted to say thanks to Art again for the tips, I've reimplemented MultiTrack DAW with the idea on using a mov as a container and passthrough everything, and submitted to apple. Wrote a quick and dirty mov parser to find the "mdat" chuck. It's working great and much faster in the case of mp3 export.

  • Josh

    @Art

    Brilliant! I had tried that as well, but fell into the same hole that Brian did. I'll be looking forward to seeing your code post. I'm not all that familiar with the .mov parsing. THANKS!!!

  • http://djmixer.fm djplayer

    I can confirm this is working for all audio file types and the soon to be approved by Apple version of DJ Mixer will support direct iPod lib mixing

  • Pingback: Disquiet » Tangents: Listening Day, iOS Thawing, Creative Commons, …

  • http://async-games.com/flare.html Rhythmic Fistman

    Good work guys, you figured it out. A couple of weeks late, but whatever.

    Now we all need to go and log bugs against AVAssetExportSession because pass through is incorrectly causing audio interruptions.

    Audio interruptions would make sense for a conversion, but we're not doing conversions so they just unnecessarily stop our RemoteIO units

    making the iPod library access less than seamless.

    Cheers,

    RF.

  • http://blog.tapsquare.com Art Gillespie

    @Everyone

    I've posted an MIT-licensed open-source class that handles quickly importing mp3/wav/aif/m4a files from users' iPod libraries.

    Tapsquare blog: TSLibraryImport

    Hope someone finds it useful!

  • Pingback: Create Digital Music » Follow-up: iTunes Library Access on iOS, Developers, and iPhone, iPad DJ Apps

  • Brad Tritone

    @Art

    Awesome contribution, Art! Seems to work as-advertised for mp3/wav/m4a (what I've tried so far). As we might expect it doesn't support "Protected AAC audio file" (.m4p) at least, not yet. I never paid much attention before, but only about half of my iTunes purchases show up as "protected". The others are mostly "Purchased AAC audio file", and the Import works on those just fine.

    Haven't dug through the code, but one question… do you need to bundle "test.mov" with the .zip? It's rather large.

  • Paul P

    Hey Art, great piece of work there.

    Just a quick question for yourself and others. Are you checking that there's enough free space before the importing or leaving that to user?

  • Josh

    @Art

    I love it! I made some minor tweaks to allow the class to choose whether or not an existing file is a failure. In my app, an existing file just means that it was imported already and I don't want to do the import again. It's working well and cleanly and now I'm operating with live FX. Well done!

  • Pingback: iOS 4 adds direct access to iTunes library, iPhone DJ apps about to get crazy : wizteq.com

  • Another84

    Hi guys! is this possibile to convert .mp3 to .m4a on the fly, when exporting ?