Well, fine, Jamie Lidell. Now you go and ruin it for the rest of us. See, none of us playing with a Nintendo DSi will possibly look as good as you do.

I jest, of course. Jamie Lidell, the wildly-talented vocalist, picks up the new, online-savvy take of the Nintendo DS and breathes cool into it. This is what Sony ads tried to do, but Jamie does masterfully. And, okay, don’t expect the built-in sound app on the DSi to do as much as it appears to be doing here – there’s quite a lot of non-real-time, non-DSi remixing going on, even though what he does do with the simple app is genius.

Thanks to Liz Revision for finding this one.

This does bring us to a burning question: Nintendo and Sony, I’m looking at you. When will we be able to run eccentric and niche music creation apps as official software on your machine? Imagine NitroTracker on the DSi download store or PSPSEQ and PSPRhythm on the Sony Store.

AR scratching from vanderlin on Vimeo.

“Augmented Reality” is a fancy term for describing ways of using computer vision to overlay digital intelligence on images. In other words, you can, for instance, scratch a vinyl record using a camera – plus a tag for identifying the object’s position in 3D space.

Cambridge-based designer Todd Vanderlin put together an elegant demonstration of the possibilities here, and his video has accordingly been making the rounds. (See: Synthtopia – and I actually heard about it this morning from a high school friend. The power of the Internet.)

Todd has more details on his site, which includes all kind of wonderful projects, like laser sound fountains and, always favorite around here, creepy circuit-bent baby dolls.

AR Scratching [Todd Vanderlin]

There’s actually some work to this: you need to figure out how the album is spinning. And of course, because this is augmented reality and not reality, there’s real potential here to imagine a new kind of vinyl DJing in which normal physics don’t apply.

From the video description:

I was playing around with some AR markers the other day and came up with this idea. taking just a plain old vinyl record and attaching an AR marker to the label you can track the record in 3D space. The next question was, can you scratch the record?

So by figuring out the velocity of the records rotation and applying it to the payback of the audio you can scratch. There is some digital noise that needs to bee worked out, but sounds pretty good. Its still really hard to scratch, it takes some practice but is super fun. The next step is to figure out some nice triggers for different modes. I like the idea of not needing a turntable but the actual spinning of the record helps with the scratching and playback. I made a couple modes, one where the record is paused and you can just scratch through the song. The other looks for zero velocity for x time and then continues on with the song. If there is velocity you then are scratching and the audio is affected. I think that this project has some legs can’t wait to play more.

I Want My Augmented Reality TV

So, this has sufficiently inspired you and you want more augmented reality? We’ve got more for you.

Take a powerful game engine (for animation, 2D and 3D graphics, physics, and on-screen interaction). Add the flexibility of a visual development environment for programming with virtual patch cords, for rich sonic and musical capabilities plus easy interaction with data and input. That’s the idea of combining something like Unity 3D with Max/MSP. In the example from earlier today, the solution simply routed basic data from a Unity-based game to a responsive music engine in Max.

In the case of [myu] – the Max Unity Interoperability Toolkit – that integration goes further still. Developed at the DISIS (Digital Interactive Sound & Intermedia Studio) at Virginia Tech, [myu] allows bi-directional integration of the Unity engine with Max or Pd. The two tools use netsend/netreceive to send data via TCP and glue the two together.

For visualists using Jitter, you can even exchange texture data, which offers some mind-blowing powers for live visuals.

Download at Virginia Tech — bonus, an extension of the aka.wiiremote object so you can use the lovely Wii Fit controller, among various other projects
Discussion on the Unity Community Forums
Discussion on the Cycling ‘74 forum
Virginia Tech DISIS

As an interactive prototyping tool, this should have a lot of potential for lovers of patch-style programming.

Thanks to Dr. Ivica Ico Bukvic, DISIS Director and researcher, for sending in his project. I’ll be curious to see what other people might do with this.

A real highlight for me at the Game Developer Conference was getting to hear Satoru Iwata deliver the keynote. Aside from being CEO of Nintendo as they have launched their most successful console ever, Iwata-san has left a sizable development legacy as a veteran of HAL Laboratory (Balloon Fight, Kirby). In the game community, I think the reception to his keynote was mixed – mostly, it introduced long-overdue storage solutions for Wiiware titles, along with some relatively minor game titles. But as a person interested in design and development – and what innovative interfaces could do for music and not just games – I found the rare insight into Nintendo’s development process inspiring.

The surprise: despite their enormous resources, Nintendo is moving to ever-smaller development teams. And they’re taking dance classes to work on their musical rhythm.

The iPhone 3.0 SDK is a fantastic update, bringing a lot of what was on developer wish lists for the device. But some of the early speculation – that the so-called “library access” would enable music games and DJ apps — may have been premature. Jordan Balagot writes to let us know that, at least in the current SDK, access to media is very limited.

The “library access” in the 3.0 SDK is only a player control API similar to that of the iPod; there is not even read only file access for MP3s nor any way to modify the output from the library. So no iPhone DJing, no BPM detection, no interactive PD or Reaktor patches with your library.

Unfortunately, this seems consistent with Apple’s desire to be the one and only media player on the device. I’m hoping that this is still something Apple plans to add – imagine the ability to add effects or run games based on the library (a la the PC game Audiosurf) or create DJ apps. I know many people who use iPhone or iPod as sample players or backups for live sets; having a custom player app could also be useful.

By comparison, Google’s Android has no such limitations on its MediaPlayer class – the fundamental difference being that you aren’t limited from playing media on your device. Unfortunately, Android has its own limitations: no real audio buffer access, which means it’s not possible to build effects or DJ apps or games on Android, either.

And that’s typical of the sort of situation the newest mobile devices present. We have the iPhone, more sophisticated technically, but limited, apparently, by design in order to protect Apple control over certain functions. Then we have the Android, philosophically unlimited but technically limited by some key missing capabilities.

My question is, which device will evolve first to give us the freedom to make use of its full potential?

No file or output access to iPhone MP3 library – 3.0 SDK still too restrictive

If we’re lucky, perhaps the 3.1 SDK? (Or something we’ll still see in 3.0 that isn’t done yet?)