Critics frequently attach the phrase “lock-in” to Apple’s iTunes Store – iTunes – iPod/iPhone combination. But, in the post-DRM age, what does that mean, exactly?

First, you have to recall that while for many of us the manual drag-and-drop music management is appealing, it isn’t so for many average consumers. They want sync. That means that music will be stored in iTunes and synced to Apple devices and nothing else. Apple is serious about locking you to their store and their devices, enough so that they frequently update their software with special keys that prevent the use of devices. iTunes is “free,” but Apple determines which mobile devices you can use and which you can’t. And Apple has gone after anyone who dares give you the ability to use your own music software or own devices, including efforts (ironically) to make their iPhone and iPod work with Linux and open source players.

These efforts don’t protect the music or prevent privacy – they protect users of Apple’s software and mobile devices from using anything but Apple’s tools. Yet Apple has used the Digital Millenium Copyright Act to take legal action over anyone who dares to even talk about how to use legally-purchased music and hardware:

OdioWorks v Apple

Perhaps suspecting their case was too thin to defend, Apple eventually backed off that particular claim — after, says the Electronic Frontier Foundation, “7 months of censorship and a lawsuit.”

Apple Withdraws Threats Against Wiki Site

But the software and hardware locks are unchanged. And Apple has won, in my view, an even more important battle: they have a monopoly over mindshare.

Here’s an example from a recent review by Gizmodo of the Android 2.0 mobile operating system from Google, as implemented on the Verizon-distributed Motorola Droid. They have some fair points about Android’s maturity and strong and weak points. But note what they say about music sync:

The only way to get your music and videos on the phone is to manually drag and drop the files. There is no syncing, no easy way to get your music library onto your phone. How are normal people supposed to figure this out? Verizon reps actually joked about how putting music on the Droid is sure to make for a lovely Saturday afternoon. What. The. Shit.

In fact, this is technically accurate, to my knowledge, only if you’re using iTunes. That incompatibility is engineered specifically by Apple. It’s a “feature”: other vendors could make other devices sync with iTunes, but Apple engineers regular updates to prevent them from doing so. In fact, while Apple was conceding defeat in its efforts to censor the Web over its iTunes lock, it was simultaneously busy blocking the Palm Pre from working with iTunes. This should be especially sad to long-time Mac watchers, who saw a Mac community railing against Microsoft’s effective office software and operating system monopolies in the 90s. Those Mac historians should also recall the early development of iTunes and shareware predecessor SoundJam, both of which worked with a variety of hardware. Now, some members of the same Mac community cheer market share numbers and anti-competitive practices by Apple.

But, engineering aside, it’s really the mindshare battle that’s most impressive. Gizmodo, in saying the Android “doesn’t sync,” really means that it “doesn’t sync with iTunes.” And given iTunes’ massive market share, Gizmodo is not alone – I’ve seen similar complaints from other press outlets and, anecdotally, many, many users.

In fact, Android sync is supported by a variety of applications. In my tests, it works with the open-source players Songbird (Mac, Windows, Linux), Banshee (Mac, Linux), Rhythmbox (Linux), Winamp (Windows), Media Monkey (Windows), and yes, even Microsoft’s own Windows Media Player. Microsoft may restrict the use of its Zune media player, but ironically its music playback software is far more open than Apple’s.

By “sync,” incidentally, I mean automatically – it’s no harder to use these applications with Google Android than Apple’s iTunes and iPhone/iPod. I personally find most of them more flexible and intuitive than iTunes. And I can show someone in a couple of minutes how to manage their device via the file system, too – even “normal people.” (I definitely don’t count as “normal,” so no argument there. But presumably “normal people” can learn to use the Mac Finder, right? Apple certainly argues they can – then locks users out of that tool when they connect an Apple mobile player.)

This is not a pro-Android argument, despite the screenshot. Any music player or phone that supports normal disk mounting will work the same way.

Why should all of this matter to musicians? The reasons monopolies are a concern in the first place has to do with pricing, and media monopolies add to that control of culture and speech. Even if your music isn’t distributed through iTunes, pricing and consumption patterns, and even the kinds of music people listen to and where they discover it are now being deeply impacted by Apple. Apple, in turn, by convincing users that there are no other options and engineering interoperability out of their products protect that control, just as digital music is growing by leaps and bounds. (For statistical evidence of the resulting trends, see today’s other story, linked below.)

I spoke to the Electronic Frontier Foundation’s Senior Staff Attorney Fred von Lohmann in April about the paper-thin (literally) arguments from Apple, when Apple was trying to prevent websites from talking about the database lock between iTunes and mobile devices:

All Apple has told us about this is in the letter they sent to us in December, as posted on the website as an exhibit to our complaint. Apple simply cites the fact that the iTunesDB page authors said that the obfuscation mechanisms used to create the iTunesDB has “may reside” in the FairPlay DRM code.

…The important thing here is that the iTunesDB pages were simply discussions about what might need to be done to reverse engineer the iTunesDB hashing. There was nothing to indicate that the efforts had succeeded. So even if understanding the iTunesDB hashing mechanism somehow magically unlocked all of FairPlay (which would seem to be far fetched), nothing on the pages suggests that the authors were anywhere near that goal.

Note that at the time, the EFF did not claim Apple lacked the right to make these kind of locks. The EFF told CDM at the time, “They have every right to do – to try to block it. Apple can certainly try to block it. What they can’t do is use inapplicable federal law to use legal threats to get them to stop.” And Apple backed off those claims.

The issue is whether you should invest in a product that limits your freedoms to use it. And the issue for musicians is whether this kind of a behavior from a company with an effective monopoly is limiting the potential power of digital music listeners in the future.

This is not to say that there aren’t reasons to choose to use an Apple device or its iTunes software. As reader “low resolution sunset” says in comments on the previous story:

This is pure conjecture: but I tend to think that slick interface design, trust, and loyalty for the Apple brand identity is what’s winning them the dominant market share of downloads.

Indeed. So, why not rely on that design, trust, and natural loyalty? Why force loyalty through engineering? And even given these qualities, isn’t there a danger when one company becomes so dominant that people don’t so much as consider alternatives? What’s to keep Apple competitive on good design if they have no competitors?

I certainly can’t answer those questions. And in the meantime, I’m looking to other alternatives, alternatives that have made me quite happy.

More on what this can actually mean:

Digital Sales Up, But is Apple Monopoly the Price? NPD, Mint Data, Editorial Analysis

Commercial developers are now releasing music creation apps for mobile game systems, in the form of the KORG DS-10 for Nintendo DS and Rockstar’s Beaterator for PSP. But some of the best ideas still come from the homebrew community.

What’s most impressive about ToneSynthDS is not so much what it does as its interface, fitting all its functionality into the DS’ two compact screens. Its minimal interface finds an elegant arrangement of everything you most urgently need, with a sequencer screen on one DS screen and basic virtual analog synth parameters on the other. A 4 x 4 matrix next to the main sequencer grid lets you switch between patterns, in a step sequencer reminiscent of the monome and Tenori-On. There isn’t a whole lot of depth to event editing in this early version, but it could be a lovely way to sketch melodic patterns. (And some of those limitations come from the DS itself. Note, though, that this app gets a full 16 real-time channels on the original DS hardware to the Korg DS-10’s paltry two.)

Developer Fanta/Hotelsinus Sound Design has been posting mock-ups, demos, and now builds as he goes. That means that he gets feedback from an audience of readers and incorporates those as he develops the app – another key difference between the DIY/homebrew scene and conventional commercial development.

More good news: this DS app should also run as a PC VST in a forthcoming version, opening up the fun to folks using netbooks and laptops instead of the DS and creating a nice mobile-to-computer workflow.

http://ndscomposer.blogspot.com/

In related Nintendo DS news: If you’re thinking about getting the new DS-10 Plus Limited Edition of the KORG DS-10, you’ll need to get it for the region coding of your DS. (In other words, you probably won’t want to import it.) The “Dual Mode” functions are region-locked, so North American and European users can’t use the Japanese DS-10. That’s not such a big deal, as North American distribution was announced, and other regions are expected to follow, but it’s good to know. See details on the All Things KORG DS-10 blog. (Thanks, DS-10 Dominator!)

Check out some demo videos and a quick run-down on specs, and if you’ve got the capability to run homebrew, you can give this a try. Thanks to Art/toitoy for the tip!

• Two oscillators with fixed oscillators, ADSHR envelopes
• Filter section (in development)
• Ring modulation and “cross” mixing (cross-fading between oscillators)
• 16×16 step sequencer, 4×4 pattern selection

Clearly, there’s a lot of work still to be done, so test this build at your own risk. But it’s already good fun, and the coming roadmap looks logical, with filter options, disk rendering, song mode, and more in store. It’s also an interesting read as far as wrangling with emulators and some of the challenges of DS development.

The iPhone’s glass touchscreen may be a thing of beauty, but despite its multi-touch capabilities, it would seem this device is incapable of responding to how hard you tap it. But the developers at Wave Machines Labs apparently didn’t want to take no for an answer.

The iGOG drum suite for iPhone provides drum pads and sample triggering in unique ways, most notably in its velocity-sensitive VelAUcity. How do you get velocity response from a device that’s supposedly not pressure-sensitive? Presumably there’s additional data in the touch events that makes this possible, but for now Wave Labs aren’t saying:

iGOG’s proprietary VelAUcity technology does the unthinkable and turns the iPhone’s screen into touch sensitive drum pads. Play loud, play soft, or play a full-blown crescendo on a crash cymbal, iGOG will capture every nuance of your performance. Just plug in your headphones and start playing.

Here’s the interesting twist: generally, when any of us say “iPhone,” what we really mean is “iPhone or iPod touch.” That’s not true in this case: “NOTE: VelAUcity is only available on iPhone devices. if you’re using an iPod Touch, VelAUcity is disabled.” That seems to suggest that the trick is the built-in mic, or at the very least some private API that’s iPhone-specific. (Audio triggering is most likely, as this app comes from a developer with drum replacement experience.) That would also suggest to me that you might be able to pull this off with non-Apple mobile devices and controllers in the future.

As a result, though, I can’t test it – I have only the iPod touch.

Unconvinced or uninterested? iGOG has some other approaches to how the small Apple handheld can be made more useful as a set of pads:

• Multi-positional pads that assign articulation based on location
• A mic trigger mode — which may be more useful than the VelAUcity. This mode assigns any nearby sound (like rapping a table) as a trigger. (That’s possible, of course, with a variety of software, but having the device itself be portable could be handy, so to speak.)
• Multi-samples, file management, and sequencing for doing more with the samples.
• No, real multi-sampling — 384 multisamples per drum!

Even if you’re skeptical of yet another iPhone app, I think there are some great ideas here for music software (mobile and otherwise). Here’s a look at the mic trigger mode in action:

Oh yes, and speaking of all this audio triggering, the same developer makes the superb Drumagog drum replacing plug-in. With pricing at US$199-379, you have to be pretty serious about drum replacement, but I know people who are who swear by it. There’s really nothing stopping you from whipping up your own solution, but Drumagog has a lot of niceties that make the process easier, quicker, and more powerful.

http://www.drumagog.com

If you are a Drumagog user, you can bring those files into iGOG. That makes this app all the more interesting – it can be both a fun toy for someone new to the idea, and a companion to a more serious (and more expensive tool).

More on iGOG:
http://www.wavemachinelabs.com/igog/

Today was full of good news for people interested in carrying pads in the palm of their hand.

Fans of the Nintendo DS in North America, the Korg DS-10 Plus synthesizer for Big N’s game system is now coming to your side of the Pacific Ocean. (That also bodes well, I think, for other parts of the world.) The DS-10 I think really deserves some credit for making a straight-up music title a hit on gaming platforms, and its success certainly surpassed my own expectations. It’s not a game, it’s not an interactive experience, it’s not a music game – it’s actually a synth and music workstation that happens to run on a game platform. The DS-10 Plus beefs up the original’s features, though it now has a commercially-available rival in the form of Rockstar’s Beaterator for PSP.

In Plus for both the DS and DSi:

• MUTE/SOLO built into the SONG mode
• EDIT/PLAY enabled for all modes within the SONG mode

Apparently DSi-exclusive (as I had speculated in the original story on the new edition):

• Twice the analog synths (4 of them, instead of 2)
• Twice the drum machines (8 instead of 4)
• Twice the tracks (12 instead of 6)
• Expanded song mode: programmable track mute, realtime editing (that is, edit parameters inside the song mode
• Two effects layers instead of just the usual effects routing (the equivalent of running two instances of DS-10)

(Previously: Korg DS-10 Plus Coming, with Beefed-Up Features for Nintendo DSi)

I’m also pleased that, if the Joystiq story confirming North American distribution is correct, only the extra effects layers require the newer-model Nintendo DSi. It sounds as though the rest of this functionality works just fine on other DS models.

Correction: As Liam notes in comments, and as I’ve clarified above, many of the new features are indeed DSi-exclusive. That means this is probably worth upgrading if you have a DSi, and a reasonable purchase if you don’t already have DS-10, but something you’ll ignore if you have a pre-DSi system and the earlier DS-10 title. Joystiq apparently mis-interpreted the press release, which is easy enough to do; it’s confusingly written.

XSEED press release

Via Joystiq’s David Hinkle:
XSEED bringing Korg DS-10 Plus to North America

It’s drumming, the multi-player game. The Drummer is an open-source application for the Nintendo DS handheld, developed by Andrea Bianchi and Woon Seung Yeo and presented alongside a paper earlier this year at the NIME Conference (The International Conference on New Interfaces for Musical Expression). As with any Nintendo homebrew software, you’ll need a special DS cartridge capable of loading software from flash memory – though if this app were developed more, it could make a terrific DSi app.

The idea is this: while making a handheld game system into an instrument, why not take advantage of its networking features? Grab a friend (or friends) with the Nintendo DS, whip up a drum kit that’s to your liking, then play along.

Oddly, while we live in a networked, Internet age, the client-server model rarely gets applied to music. One of the things I try to explain about the protocol OSC (OpenSoundControl), aside from the fact that it doesn’t have to be about sound, is that it’s really a collection of best practices in open networking communication. It’s not simply about connecting devices to one another in serial fashion, as with MIDI, but forming a network – an idea familiar to anyone who uses instant messaging online. True, latency considerations and other complications can add another dimension of challenge. But I think there’s plenty to explore when it comes to networking devices, and it promises to make computer music a less solitary experience. The Drummer is a good step in that direction.

The Drummer project page + NIME paper
Google Code page

Thanks to Andrea for sending this our way!

Part of the beauty of the iPhone from its launch date was the notion of a Mac you could fit in the palm of your hand. That makes it ironic that, for so many developers, mobile platforms in general have turned into a way to fragment software, to make it run fewer places instead of more. There’s something to be said for designing to a specific mobile device, but on the other hand, how many developers would want to restrict where their creations run? And particularly in music, isn’t the appeal of mobile creation the ability to have your tools work in more places? Maybe targeting just one gadget is the right choice for a given tool, but it shouldn’t be the only choice.

On the iPhone, the application Thump has plenty in common with a range of mobile music making tools. It’s simple but fun, a groove box with sequencing, subtractive synths, drums, and a set of basic effects, plus the ability to load your own samples and export songs.

Thump also demonstrates how simplifying sonic capabilities can produce musically-beautiful results, by focusing on the essentials and creating something with personality. Here’s a track by its creator showing off its sounds:
thump soundreel by mazbox

Well worth checking out the app on your platform of choice:
http://www.mrkbrz.com/thump/

What might not be immediately apparent is that under the hood, Thump makes use of the open source environment openFrameworks. As a result, the same code runs on iPhone, Mac, and Windows, as well as Linux. (It’s not distributed for Linux, but it could be. Hint, hint.) Creator Marek Bereza gives back, too – he ported the openFrameworks audio library to the iPhone, where it’s available to anyone.

Updated Marek notes in comments just what this means. The video below is, essentially, the same app. In place of the iPod screen, he has used a massive lattice of physical controls. A separate installation at the same show used a large touchscreen and simplified interface. And this really demonstrates what cross-platform means. Guy Kawasaki in the 80s mocked “ports” as a cheap wine. The idea is not to simply dump your code on a different platform and hope for the best – in fact, in this case, the changes from one platform to another were radical. The key is maximizing what’s essential, what really is not specific to a single device.

Physical Sequencer from Marek Bereza on Vimeo.

Creative Applications has a detailed write-up of the installation with more documentation.

If you’re interested in creating your own projects, oF has an elegant syntax based on Processing, but adapted to C/C++ coding paradigms and libraries.

I would recommend trying desktop development first as you’re learning, but when you do want to go mobile, setup is absurdly easy:

Setup instructions for iPhone

Having this as a foundation means more freedom for users and developers alike, the ability to run where you want, and a shared community of artists helping one another out. It does not mean sacrificing revenue, either – which is important, because revenue is often what enables (and encourages) developers to make new things. Here, we see a commercial mobile tool and distribution via the iTunes store, with free-as-in-beer distribution of desktop apps. But even the Free Software Foundation actually encourages charging for distribution – and to many people, the FSF is the most extreme example.

The bottom line is, you’re free to create what you want, where you want – and it doesn’t even have to have a price of free.

The use of openFrameworks is possible because the iPhone allows the execution of native code. Apple does not allow the Java virtual machine, which locks out many useful platforms and tools — including Processing. Naturally, that should not be the case with forthcoming non-Apple platforms based on Linux; more on that soon.

Thanks to Memo Akten for the tip — and for his exceptional documentation of iPhone openFrameworks development.

GPU-tiful and Cross-Platform

Updated Another framework I left out is the lovely libnui. The tagline says it all: “C++ API, multiplatform (Mac OS X, iPhone, Win32, Win64, Linux), 3D hardware accelerated GUI and more…”

On the iPhone, Intua’s elaborate Beatmaker software and David Wallin’s bleep!box have both been made with libnui. I want to single out bleep!box because of its UI – note how UI elements are simplified and scaled to the size of your fingertip. That’s something that, speaking of multiple platforms, could be equally important with touch hardware coming to Mac, Windows, and Linux platforms (some of it here already). I saw a huge fan of Windows tablets loving Propellerhead Reason on a tablet. He probably would be less enthused running Reason or Record on a touch interface; the UI elements are way too small.

bleep!box is also worth a look for the same reason as Thump: elegant UI, simple functions, and a lot of fun.

Pd, too

Pure Data, the open source patching cousin of Max/MSP, can also run on mobile devices. It’s currently powering the application RjDj, but because Pd is free, that could be just the beginning. In May, I wrote about the work being done by Hans-Christoph Steiner, who has been porting Pd even to ancient PDAs and iPods – including those, notably, without native floating-point capabilities (the kind of number crunching usually associated with sound).

Well, now Hans-Christoph is looking toward other, current-generation devices, too, kicking off with a workshop on making interactive sound on the iPhone using Pure Data. Here’s a taste:

OK, kids, you won’t believe how easy this is. Once you have Pure Data installed on your your iPhone or iPod-Touch, creating and uploading your own sound-processing and sound-generating patches is as trivial as copying a text file to your device! You’ll still need an Apple Developer License, but because Pd patches are treated as media files, they don’t have to go thru Apple’s elaborate code review — so, you can just dive right in, turning your phone into a pocket synth within minutes. In this 3-hour hands-on workshop, interactive sound whiz and Pure Data developer Hans-Christoph Steiner will lead you through all the steps.

Hans-Christoph will be talking Pd on the iPhone at a conference next month at Carnegie Mellon University in Pittsburgh. That will also include Memo Akten and Zach Gage talking about openFrameworks. And for nearly-entirely open source hardware, there’s a tantalizing-looking workshop combining Android with Arduino, too.

http://artandcode.ning.com/

There’s quite a lot more to say about this, but I hope this whets your appetite.

And, of course, riding the train to Pittsburgh or your nearest university or open source workshop, now you’ll have Thump which which to make some thumping tracks on your iPhone.

The novelty of the iPhone or [your favorite device here] may fade. But part of what matters in mobile design is thinking about how to create interfaces and uses that can scale to the size of your palm. That can mean embracing radical simplicity, and reducing an interactive, digital musical object down to its essential noise-making functions. In acoustic instrument design, that means economizing sound production in a form. In the digital world, it means finding the interactive role you’d want to bring with you onstage, in the length roughly equivalent your fingertips to your wrist.

I’m a few weeks overdue actually writing about it, but one design I really admire is Star6, developed by Jason Forrest and Agile Partners. There are no awkward, gimmicky emulations of hardware interfaces here; it’s clear this was an interface that was illustrated in two-dimensions. It has funky nerdster chic color combos, with neon pink atop wood grain. It demonstrates that, in the space of a grid, you can fit triangles. It makes use of computer wifi capability to easily load samples without mucking around with over-designed clients – or record right on the iPhone. And it’s – surprisingly – one of the few apps to make heavy use of the accelerometer, which means rather than looking like you’re trying to text message someone, you can move it around. There’s a “grain” mode so that you can randomize sounds and not have everything synced all the time. I also enjoy the “reset” button. These are all design decisions that could make sense in more commercial software – and our own home-brewed Max/Pd patches and such, too.

Apparently Agile Partners were also influenced by the brightly-colored, handheld fun of the Buddha Machine, too; see their interview with the creator.

Star6
A lovely lineup of free samples, including the Buddha Machine

It’s not a perfect app (no mobile app really can be – that’s the fun of it), and it doesn’t do everything, but I find Star6’s personality rather irresistible. The real test of all of this is whether you can use it in real music-making. And, while my inbox is full of cheezy bands trying to ride the iPhone wave, I love the offbeat Star6 music launch party from Berlin, as documented in the video below. It ranges from Jason’s own work to Warp Records artist Jackson and ex-Chicks on Speed Kiki Moorse. And there’s a crazy iPhone + banjo + accordion cover of Katy Perry’s “I Kissed a Girl.” There are even some genuinely experimental sounds – not the sort of thing you’d expect at a launch event, sadly. (I wish we could have more of that.)

An Evening With Star6 – Berlin (Compilation) from Star6 on Vimeo.

More on the artists, and some of Star6 creator Jason Forrest’s own unique work:

Jason’s own artistic aesthetic, as seen in this video for “War Photographer,” does have this quirky efficiency to it, the sense of cut-out animation (in both visuals and music, I’d argue), and saturated, rich, retro colors.

The eclectic Berlin launch.

Jackson (Warp, FR)
Kiki Moorse (ex-Chicks On Speed,DE)
Song Band (US)
Jason Forrest (CRD, US)
Guido Mobius (Karaoke Kalk, DE)
Ben Butler & Mousepad (SCT/DE)
DJ’s: Finkobot & Marius Reisser

Jacki Terrasse / Joseph (@ Maria)
An Der Schilling Brücke
10243 Berlin

For more on the artists:
myspace.com/moorse
myspace.com/jacksonand
myspace.com/benbutlerandmousepad
myspace.com/guidomoebius
myspace.com/jason_forrest
myspace.com/songbandmyspace
myspace.com/finckobot
myspace.com/mariusreisser

Video shot by Martin Sulzer
Photos by Marco Macrobi

Complete sets:
Ben Butler and Mousepad
Guido Mobius
Kiki Moorse
Jason Forrest

LaDiDa Demo from khush on Vimeo.

There’s no question iPhone/iPod touch development – really, just clever mobile development – has gotten a bit overhyped lately. But that’s all the more reason to do a round-up of genuinely interesting stories, real innovation happening on the platform. So, I’m clearing out my inbox with some of the more creative tools appearing recently on Apple’s mobile gadgets. There’s no better way to kick off today’s festivities than with this unusual “reverse karaoke” creation.

Sure, people may think they’re tone-deaf. But even the layperson has extraordinary powers of musical perception. So how could you train your iPhone to perceive and respond to music? That’s the question asked by LaDiDa for iPhone, the first of a new line of “intelligent” music applications for mobile devices. A “reverse karaoke” tool, the idea is to listen to singing and fake accompaniment, rather than having you sing along to canned backing tracks. Nothing is pre-programmed; everything is generated on the fly on the device.

It’ll even make up a Bollywood accompaniment to your singing:

LaDiDa Bollywood Duet from khush on Vimeo.

Of course, to me, it’s interesting not only what the iPhone is able to musically, but also what these algorithms are unable to make sound musical. Both reveal a whole lot about how we hear and conceptualize music. I think the team deserves real credit for making this fun, though, and on constrained hardware.

The app’s creator Khush follows in the footsteps of Smule in that it takes hard-core academic music research and uses mobile devices as a vessel for getting that tech in the hands (literally) of the general public. (See my interview with Smule founder and ChucK originator Dr. Ge Wang.)

Parag Chordia, developed at professor at the Georgia Institute of Technology and the gentleman you see in the video, spoke to CDM about what’s happening behind the scenes. He tells us about how this application was developed, and how the intelligent algorithms work (or at least try to work, as music analysis and auto-accompaniment remain at early stages).

First, an explanation of the app.

Khush CEO Prerna Gupta explains how it works:

1. You sing into the phone, and LaDiDa will compose music to match.
2. LaDiDa’s patent-pending technology analyzes the pitch and structure of the melody to compose a unique accompaniment for each recording.
3. To be clear, we do not query a database of pre-recorded songs. That is, LaDiDa has been designed to work with any music.
4. After recording your song, you can hear it with different styles. LaDiDa comes with three styles — E Piano Pop, Rhythm Synth Pop and Dub Tone — each of which has been developed using high-quality instrumentation to work specifically with our algorithm.
5. We will be launching new styles every month that will be made available through in-app purchases.
6. LaDiDa also works on rap! This month we’ll be adding three new rap styles.
7. After choosing your style, you can save the song and share it on Facebook, Twitter and email.
8. LaDiDa also has a Discover page, where you can hear songs recorded by other users from all over the world.
9. Khush was founded by music technology enthusiasts from the Georgia Tech Music Intelligence Lab. You can read about us here and also find out more about the research at our lab here.
10. LaDiDa went live in the iTunes store last week and is currently priced at $0.99.

Prena, the woman you see in the video, has some Web experience to boot, too, including founding a popular Indian dating site. Oh, and she’s a better singer than the music researcher, but, hey, that’s why we all went into computer music, right?

In case you’re wondering how you take a research idea and make it run on the iPhone – or how the algorithm works (and might get smarter in the future) – I turned to Parag for those details:

The initial code was developed in my lab in c++. Since the core algorithms are basically mathematical, that portion was relatively easy to port. However, we spent significant time thinking about how to optimize for the iphone and every aspect of the app, from the interface to sound design, has been built with the iphone in mind. For example, there are significant limits on sampler performance — samples have to be short and effects are more or less out — but we thought it was important for our styles to have a rich sound. So we
put great effort into designing light styles that sound realistic.

Another significant challenge was making the analysis robust to external noise; iphone recordings are lo-fi and corrupted with tons of background noises, which makes robust (and again computationally efficient) pitch detection essential.

Our approach to reverse karaoke is somewhat different than what’s been done before. A significant limitation of previous work was a lack of fine-grained key estimation, a problem that we felt was critical to successful vocal accompaniment (most people are not anywhere near a piano or an instrument with fixed tuning when singing into the app).

We also worked on trying to give some larger structure to the
accompaniment, which can often sound locally reasonable but notably lacking in direction. Again, a difficult problem particularly when people are singing snippets. Still it is sometimes possible to detect phrases, and we have tried to incorporate that information as well.

Auto-accompaniment is an endlessly fascinating and deep problem. As we learn more about human perception and cognition of music, as well as improve our tools for machine listening, our systems will become more musical. While we still have a ways to go, we believe that, with LaDiDa, we’ve created a product that is engaging and allows regular people to express themselves creatively.

If all of this talk about musical perception recalls the questions about how culture and background versus neurology can be used to explain music – as seen at the Notes & Neurons conference – that’s no coincidence. Researcher Parag played sarod with a fascinating ensemble at that same conference. Bobby McFerrin sings a really beautiful solo with the ensemble.

In fact, it’s absolutely worth contrasting the elegance and beauty of these all-human musical responses to the somewhat clumsy (sorry, Khush) iPhone responses. That’s not to say the iPhone creation is any less human – it’s a computation model programmed by humans, and is capable of some impressive feats made possible by their musical instincts and training. As such, we really can hear the gap between what advanced musicians can do intuitively and what we can model computationally, atop the restrictions of the device’s ability to sense the world around it.

World Science Festival 2009: Notes & Neurons, Part 5 of 5 from World Science Festival on Vimeo.

Smule, the iPhone/iPod touch development house that has released an Ocarina and a Leaf Trombone to the iPhone, has now partnered with T-Pain and Auto-Tune to bring a T-Pain-branded app to the mobile platform. I interviewed Ge Wang earlier; he gets exceptional music geek cred for the creation of the real-time synthesis language ChucK.

But this app goes further. It isn’t just called Auto-Tune for iPhone, or T-Pain Presents Auto-Tune by Smule or something like that. It actually promises to turn you into T-Pain, sort of like the toys that make you sound like Darth Vader. And that means it has exactly one application – one essential application, I’d say. It means you can do amazing covers of “I’m on a Boat.”

Something has jumped a shark here, but I’m not sure which. Maybe it jumped over the shark onto the boat.

Two more serious observations:

One, this theoretically could be a useful addition to your mobile arsenal. Mike Una uses Auto-Tune to map the continuously-varying pitch of his “Beep-It” optical Theremin to a scale. Of course, the problem is that the iPhone lacks an audio input jack, though maybe someone has an idea for how to solve that.

Two, a question: just when, exactly, will we get basic audio DSP coding on a platform that’s not the iPhone? Sony’s PSP is arguably more powerful, but requires you to take your game system into a back alley to modify it to run homebrewed software. Google’s Android has more powerful hardware in the pipeline, at least, but there’s still not really official support for running native audio code (even though that’s how the phone’s own audio system was built). People are starting to simply say “iPhone app” when what they mean is “mobile app,” and that’s a shame.

As you may know, the creators of games like Grand Theft Auto have collaborated with Timbaland to bring a mobile music studio to Sony’s PSP (and later, the iPhone), based on an ambitious free Flash experiment on their Website. Now, it’s my impassioned belief that you shouldn’t need lots of canned loops or celebrity endorsements to make music fun, so normally I might actually run the opposite direction of any story starting with that line. But here’s the surprise: underneath, the app is more powerful than I expected.

I’ve gotten an early preview of the title in person at Rockstar’s offices here in New York, and was also able to grill their developers on geeky details of how the sound engine is put together. A test copy isn’t yet available so I can’t properly review the app, but I am at least able to talk about some of what lies beneath the PSP screens and marketing.

For some time, a select few have known that the Sony PSP’s secret is that it’s a powerful handheld computer, ideal for mobile music. Brilliant-but-underground apps like PSPSEQ and PSP Rhythm capitalized on this potential, but required you hack your PSP in order to run them, because Sony restricts launching non-authorized applications from memory.

Beaterator is the first full-featured app that can be run directly on the PSP. Some people may not look past the fact that it comes from a game company, past its (admittedly) thick layer of marketing glitz and celebrity endorsement. But based on a first look, I believe Beaterator is the most powerful music app ever released through game channels, surpassing in functionality even the recent cult hit Korg DS-10 for the Nintendo DS.

In the interest of disclosure, I have a confession: I didn’t expect to have any interest in Beaterator at all. I was concerned that the musical experience would be watered down (though more on what I actually discovered below). The fact that this game had one artist – Timbaland – literally dancing around the screen talking about how it’s his game I thought would be a deal killer. And, of course, it’ll be impossible to talk about this game without the shadow of the “Acidjazzed Evening” controversy, which aleges Timbaland plagiarized music by Finnish composer Janne Suni – made worse by a lawsuit and a glib interview in which the artist responded, “It’s from a video game, idiot.” Timbaland is by no means the first artist to get into trouble with an uncleared sample, but the fact that it was a much lesser-known artist and that the situation was handled less than gracefully certainly created a credibility issue in the enthusiast community.

I bring those issues up front, because I know readers will bring them up. But what intrigues me about Beaterator is that it has an essentially split personality. At one moment, Beaterator is an animated Timbaland talking to you while you trigger canned loops with game buttons, neither game nor, really, a music creation app. At another moment, though, it’s a full-blown music sequencer you can carry around on your PSP, with some retro design and sound features that might actually make it appealing. And I think it’d be unfair to cover one side without the other.

The marketing for Beaterator focuses on the thousands of loops assembled by Timbaland and Rockstar. But Beaterator isn’t limited to those loops. What you likely won’t see emphasized in the gaming press:

• Sampling: You can make your own samples using the mic (or, if you can find it, you can add a PSP microphone)
• Audio import: You can import WAV files from a computer, as easily as dropping them onto a MemoryStick.
• Audio export: You can save your work as an audio file. Rockstar will have its own site for exchanging your music with other users, but that will be limited to the built-in effects (I’m guessing so they don’t have to police piracy). But that won’t stop you from exporting audio on your MemoryStick and using it however you like.
• MIDI import and export: While even many serious iPhone games lack this functionality, you can use Beaterator as a mobile MIDI editing workstation.
• Grown-up interface and effects: Beaterator has real audio effects, with real labels. The Compressor has labels like Gain and Ratio, instead of, you know, “Phatness” or “AMPMEUP.” It’s a clue that this really is a tool and not a game.
• It’s-a-me … not! Game cartoon character heads never appear in the interface – though I do have to admit, those Mario Paint Mario noteheads were cute. (For the PSP, might I suggest semidemiquavers with the face of Kratos?)

I love my desktop sequencers, but having these kinds of features in a comfortable-to-hold mobile device you take anywhere, being able to fly through settings with the PSP buttons, and lots of little details added by the Rockstar developers like confining pattern editing to scales and keys make Beaterator look like something that’ll be fun to use. After a couple of minutes, I was ready to charge up my PSP and fire up Beaterator alongside PSPSEQ.

Inside Beaterator’s Engine

Beaterator began several years ago on Rockstar’s website. Before making full-blown music production tools with Flash was popular, before the idea of “cloud editing” had become a buzzword, a side project at Rockstar yielded a free Flash game, which you can still play. The interface is loop-based and reminiscent of tools like ACID, GarageBand, and Fruity Loops. But it’s surprisingly minimal, capable of full-blown pattern and loop editing, includes real-time effects, and comes with a selection of loops from some of my all-time favorite producers – A.VEE & 3D, Juan Atkins, King Britt, Matthew Dear, and Steinski. Side note: please, Rockstar, can we have a custom version of Beaterator with those producers?

http://beaterator.rockstargames.com/beaterator.html

Having talked to mobile developers both big and indie, I was curious about the technical details of Beaterator’s implementation – especially after being impressed in a short demo by capabilities that went beyond what I had expected. Rockstar replied with some very particular details from the developers. I think the answers say a lot about what’s possible on the PSP – even with that iPhone version in the works – and how a handheld sequencer on mobile hardware can be put together.

PK: So, I see eight tracks, some effects – what are the capabilities of the underlying audio engine in Beaterator?

Rockstar: Our engine runs at 16-bit stereo, 22.05kHz throughout. As you say, there are 8 tracks, each of which each can have up to two insert effects* summed into a stereo mix. Each track also has a stereo pre-fader Send to a dedicated reverb buss which runs a global reverb unit which is also added into the output. The channel level, pans, aux send and all effects parameters can be automated to 1-bar resolution, as can the final mix output level, pan, and the global reverb parameters. At any given time, each of the 8 tracks can be playing either a sample-based Melody/Drum loop (with 8 channel polyphony), a monophonic synth melody loop, or a mono/stereo timestretched audio loop.

Our sequencer also supports 8 channels, at 4ppqn. The maximum song length is 240 bars, and we are fixed to 4/4 time. BPM ranges from 60-300, and there is a simple 16th-note swing control as well.

The insert effects we support are: Compressor, Chorus, Delay, Distortion, 3-band EQ, Multimode Resonant Filter, Flanger, Noise Gate, Phaser & Tremolo.

A lot of the editing (aside from vocals, really) appears to be in MIDI pattern editing. Is this always triggering a sampled instrument, or is there live synthesis / DSP going on under the hood at all? (I know the PSP is capable of such things.) Do you have any control over the timbre of the instruments, or just the musical patterns?

We trigger either our pitched sample playback code or our own DSP synthesiser on a channel. The sample playback code is a fairly simple 8-voice (which means a theoretical maximum polyphony of 64 if you are playing 8 samples on each of the 8 channels) pitchshifter (no multisampling), with no real control of timbre (no filters). But it does support velocity sensitivity (although this has to be sequenced as the PSP buttons aren’t velocity sensitive) and a full ADSR amplitude envelope.

The inbuilt synth is monophonic (but you can run one per channel, so up to 8 in theory). It’s a simple 3-oscillator Virtual Analog design. Each oscillator has a smoothly-morphable waveshape from Sine, through Triangle, Sawtooth, Square and finally to 10% Pulse) and has independent +/-2 octave pitch and +/-1 semitone detune controls. There is also a separate white-noise generator. The synth also has its own multimode resonant filter (24dB/octave Low Pass, 12dB/octave Low Pass, 12dB/octave Band Pass, 12dB/octave High Pass and 24dB/octave Low Pass modes) with controllable keyboard tracking. There are two ADSR envelope generators, one locked to amplitude controls, and the other freely assignable to the modulation matrix. There are also two LFOs with multiple shapes and speeds. These three modulation sources can be freely assigned to any of the 14 modulation destinations, which amounts to 42 modulation slots. Unfortunately, none of the parameters of the synth can be automated.

Pattern editing in Beaterator does not use MIDI internally – The game is locked to 4ppqn on a five-octave keyboard, and the melody notes use a unique pitch bend/portamento technique which doesn’t map directly onto MIDI pitch bend events. However, we can import and export MIDI files with a reasonable degree of accuracy, which is a feature we think will be particularly useful.

I’m already impressed by some of the editing capabilities – I was surprised to see a brief glimpse of envelope rubber-banding. Any other specifics you want to talk about in terms of editing possibilities?

As explained above, we have ADSR envelope control throughout, and we are particularly proud of our Synthesizer. Our sound waveform editor supports most of the normal editing tools you would expect (trim, insert silence, normalisation), and also allows you to set up sustain regions for the sample playback engine, timestretch and slice the waveform.

Okay, so let’s imagine I’ve got a bunch of audio loops on my drive and want to do, effectively, what Timbaland did with his loops. Do you have as much control over authoring loops for Beaterator as Rockstar? Are there limitations on this feature?

You can freely import and edit any 44.1kHz or 22.05kHz mono or stereo uncompressed 16-bit WAV into Beaterator (we have 9MB of sample memory available in any one song). You can edit these however you like (indeed we use these tools ourselves to generate content).

Do you need to prepare your loops at a fixed bpm? If I bring in a 120 bpm loop and change the tempo to 144, does my loop stretch? (If it’s doing stretching, the audio warping engine sounds really fantastic!)

We do timestretch loops to match the songs tempo (but this can be disabled if desired). You can also over-stretch sounds for that early-90s “granular timestretch” effect. Thanks for the compliments on our warping engine too!

I’ve got a soft spot for 90s digital, I think. What was the emphasis of the Rockstar team in terms of their samples and musical genres?

The focus was on loops that could work well with most genres. We like to think you can create songs from Rock to Rap with the selection we present.

Is it (You) Live, or is it Timbaland?

Beaterator gets at the heart of what’s happening with the way music creation is packaged in a digital age. Rockstar contrasted their title with things that are “games,” like the Rock Band/Guitar Hero mold that was refined by Harmonix. Rockstar emphasizes creative music creation and deemphasizes gaming. At the same time, they pack the title with pre-made loops and are concerned about whether what you do will easily sound good, or whether you’ll be overwhelmed by the genuinely powerful tools underneath. Beaterator, unlike Rock Band, is not about playing along with your favorite artists – but clearly Rockstar is betting that Timbaland is a favorite artist of prospective buyers, and that you will try to sound like him and use his loops. Rock Band makes no such attempt to be creative. The Beatles, coming out next week, is about playing along with the Beatles. (I recall singing along with my sister when we were kids, and I was someone who grew up taking weekly piano lessons.) Yet the “game” in this case does just the opposite – instead of trying to be easier, the whole selling point is a ramped difficulty curve. Maybe the reason studies are showing people graduating from Rock Band to real instruments is that, eventually, if you seek out difficulty, you need to go beyond the game. (Actual instruments: they’re the ultimate expansion pack.)

It’s a paradox, but it’s not a paradox restricted to gaming. You can take the conflict above and apply it to the way all music technology is marketed. On one hand, you have software that’s almost comically complex – sometimes offering so many options that it’s hard even for people with doctoral-level training in digital signal processing to make actual music. On the other, from many of the same vendors, you have pre-built loop libraries and presets with push-button simplicity, requiring less musical coordination or rhythm than, well, Rock Band on Easy mode.

All of us, like our technology, have a split personality when we use digital tech. But maybe the ultimate question is a simple one: can you make something?

I’ll reserve that question in regards to Beaterator until I get a final version. I was set to take a development build home with me, but I do have to wait until the final release. I do think these are interesting questions, though.

Oh, and say what you will about Timbaland, but animated Timbaland has some sort of nuclear control panel that he uses to DJ from. I want a real one. Surplus shopping, anyone?

Stay tuned for a look at the finished product. If fans of mobile game music were willing to use Mario Paint to get an extremely basic song editor, I think Beaterator could be a revelation. And in addition to looking at music production in Beaterator well beyond what might qualify as “Rock” or “Rap,” I think it’s long overdue for a guide to the other independent apps for the PSP. (In fact, I just got a new one in my inbox), so watch for that, too, in a separate story.

Videos from Rockstar (all images and videos are of the PSP version, not the iPhone)…

Now, I talk too much. The best way to make my argument? Listen to the kids. Their favorite feature: recording their own voice.

I rest my case.