The “space bassoon” Eigenharp seems to have landed from another planet. Today, I’ve got good news: it’s bringing alien gifts with it. By next year, both the software and the high-performance protocol the instrument uses will be open source. Taken together with other advancements in the open source community and with protocols like OSC, that could mean we’re at the vanguard of a golden age for more open, more intelligent, more expressive digital instruments.

Genuinely new music controllers made available commercially don’t come along very often. So this week’s news of a strange but wonderful-looking instrument shaped like a bassoon with customizable key controls turned many heads. With high-resolution, high-frequency data and reliance on the computer for everything from sound generation to mapping the keys to different tunings, the computer connection matters. Eigenharp’s chairman, John Lambert, sets the record straight for CDM on the software, the way it talks to your computer and other gear, and how open the tools and protocol will be.

I’ll be talking more with John next week, but I want to bring you this news now. Part of blogging means that you don’t hold back – you share that first reaction and then learn more. I’m pleased to say I was dead wrong on the Eigenharp. What looked on the spec sheet like MIDI-only communication and proprietary software turns out to be just the opposite. Sometimes, being wrong is great. Here are all the details:

What protocol do the instruments use?

The instruments talk to their base station by a dedicated, high speed, differential, transformer-coupled, error-correcting digital protocol designed (and tested) for use in noisy stage environments by ourselves. The wire used is a 4 core, standard star quad mic cable and can be up to 40M long. The base station then talks to the control software running on the host PC (Mac at the moment) via a dedicated protocol over USB2. In the host system, the native protocol used between agents (our software components) is a network protocol called Eigentalk. This can run over Ethernet wires (with slightly increased latencies due to some smart jitter buffering we do). We will be making this public and freely available (when we open-source our main software) in Febuary next year. There is an open API, to be published at the same time, for our equivalent of AU’s or VST’s, hosted in our software, EigenD. This is rather richer than the AU and VST API’s, for reasons connected with the expressiveness and configure-ability (is that a word?) of our system. A couple of interesting AU vendors are looking at producing native EigenD instruments at the moment, and I think there will be many more in time. We try to avoid MIDI wherever possible as it has so many serious performance limitations.

Had you considered OSC?

We did have a look at OSC but there are a bunch of things that our protocol does that OSC didn’t (not least of which, for example, is clock syncronisation for jitter removal, utterly vital over ethernet for live performance) so we decided to grow our own and release it. Absolutely no reason whatsoever that we can’t interoperate fuly with it though, I suspect that an OSC EigenD Agent will probably come quite soon after the open source release.

Ed.: Actually, with OSC (OpenSoundControl) moving increasingly toward broader input and assimilating other standards, I can easily imagine that a future implementation of OSC could begin to implement these features. More on both protocols in the coming months. -PK

Is it possible to use this as a controller for MIDI hardware? MIDI software?

Yes, EigenD has in and out MIDI gateway programs. The incoming [MIDI] is mainly used for pedalboards at the moment (to control scale, key, tempo, etc.), although there is no reason you couldn’t use a midi keyboard to play notes. We’re using the MIDI out to do a wide variety of things, including playing MIDI sound modules, and controlling video DJ software and lighting systems. There is a huge reduction in effective bandwidth when ‘downsampling’ to MIDI, but for a lot of things it still works very well. And of course, the AU and VST standards (which we fully support) are a kind of MIDI interface.

Is it possible to access its native protocol for custom software / Max-Pd patches and the like?

Well, we’re open sourcing the whole thing early next year, and I think there’ll be a lot of work done to glue all sorts of external stuff in then. Max/MSP has attracted a lot of interest in this area. The whole of EigenD is kind of built for this really. We’re an Open Source company: my main personal desktop is Linux and I have a strong personal commitment to open standards. I offer a number of ‘programming bounties’ for Linux improvements on my personal website at http://johnhenrylambert.com.

Tell us more about open-sourcing the software. What tools did you use to build the software.

It’s written in a mixture of C++ and Python. It’s quite scary code, but very clean and pretty, with little ‘cruft’. I think the open source community are going to have fun with it. Its a big system too, 8 years of R&D in one huge release. I don’t think that there’s been anything quite like this open sourced before in the music world – it is a bit as if Logic or Ableton were GPL’d tomorrow.

Thanks, John! Stay tuned for more details, and let me know if you have specific questions for the creators.

Part of the beauty of the iPhone from its launch date was the notion of a Mac you could fit in the palm of your hand. That makes it ironic that, for so many developers, mobile platforms in general have turned into a way to fragment software, to make it run fewer places instead of more. There’s something to be said for designing to a specific mobile device, but on the other hand, how many developers would want to restrict where their creations run? And particularly in music, isn’t the appeal of mobile creation the ability to have your tools work in more places? Maybe targeting just one gadget is the right choice for a given tool, but it shouldn’t be the only choice.

On the iPhone, the application Thump has plenty in common with a range of mobile music making tools. It’s simple but fun, a groove box with sequencing, subtractive synths, drums, and a set of basic effects, plus the ability to load your own samples and export songs.

Thump also demonstrates how simplifying sonic capabilities can produce musically-beautiful results, by focusing on the essentials and creating something with personality. Here’s a track by its creator showing off its sounds:
thump soundreel by mazbox

Well worth checking out the app on your platform of choice:
http://www.mrkbrz.com/thump/

What might not be immediately apparent is that under the hood, Thump makes use of the open source environment openFrameworks. As a result, the same code runs on iPhone, Mac, and Windows, as well as Linux. (It’s not distributed for Linux, but it could be. Hint, hint.) Creator Marek Bereza gives back, too – he ported the openFrameworks audio library to the iPhone, where it’s available to anyone.

Updated Marek notes in comments just what this means. The video below is, essentially, the same app. In place of the iPod screen, he has used a massive lattice of physical controls. A separate installation at the same show used a large touchscreen and simplified interface. And this really demonstrates what cross-platform means. Guy Kawasaki in the 80s mocked “ports” as a cheap wine. The idea is not to simply dump your code on a different platform and hope for the best – in fact, in this case, the changes from one platform to another were radical. The key is maximizing what’s essential, what really is not specific to a single device.

Physical Sequencer from Marek Bereza on Vimeo.

Creative Applications has a detailed write-up of the installation with more documentation.

If you’re interested in creating your own projects, oF has an elegant syntax based on Processing, but adapted to C/C++ coding paradigms and libraries.

I would recommend trying desktop development first as you’re learning, but when you do want to go mobile, setup is absurdly easy:

Setup instructions for iPhone

Having this as a foundation means more freedom for users and developers alike, the ability to run where you want, and a shared community of artists helping one another out. It does not mean sacrificing revenue, either – which is important, because revenue is often what enables (and encourages) developers to make new things. Here, we see a commercial mobile tool and distribution via the iTunes store, with free-as-in-beer distribution of desktop apps. But even the Free Software Foundation actually encourages charging for distribution – and to many people, the FSF is the most extreme example.

The bottom line is, you’re free to create what you want, where you want – and it doesn’t even have to have a price of free.

The use of openFrameworks is possible because the iPhone allows the execution of native code. Apple does not allow the Java virtual machine, which locks out many useful platforms and tools — including Processing. Naturally, that should not be the case with forthcoming non-Apple platforms based on Linux; more on that soon.

Thanks to Memo Akten for the tip — and for his exceptional documentation of iPhone openFrameworks development.

GPU-tiful and Cross-Platform

Updated Another framework I left out is the lovely libnui. The tagline says it all: “C++ API, multiplatform (Mac OS X, iPhone, Win32, Win64, Linux), 3D hardware accelerated GUI and more…”

On the iPhone, Intua’s elaborate Beatmaker software and David Wallin’s bleep!box have both been made with libnui. I want to single out bleep!box because of its UI – note how UI elements are simplified and scaled to the size of your fingertip. That’s something that, speaking of multiple platforms, could be equally important with touch hardware coming to Mac, Windows, and Linux platforms (some of it here already). I saw a huge fan of Windows tablets loving Propellerhead Reason on a tablet. He probably would be less enthused running Reason or Record on a touch interface; the UI elements are way too small.

bleep!box is also worth a look for the same reason as Thump: elegant UI, simple functions, and a lot of fun.

Pd, too

Pure Data, the open source patching cousin of Max/MSP, can also run on mobile devices. It’s currently powering the application RjDj, but because Pd is free, that could be just the beginning. In May, I wrote about the work being done by Hans-Christoph Steiner, who has been porting Pd even to ancient PDAs and iPods – including those, notably, without native floating-point capabilities (the kind of number crunching usually associated with sound).

Well, now Hans-Christoph is looking toward other, current-generation devices, too, kicking off with a workshop on making interactive sound on the iPhone using Pure Data. Here’s a taste:

OK, kids, you won’t believe how easy this is. Once you have Pure Data installed on your your iPhone or iPod-Touch, creating and uploading your own sound-processing and sound-generating patches is as trivial as copying a text file to your device! You’ll still need an Apple Developer License, but because Pd patches are treated as media files, they don’t have to go thru Apple’s elaborate code review — so, you can just dive right in, turning your phone into a pocket synth within minutes. In this 3-hour hands-on workshop, interactive sound whiz and Pure Data developer Hans-Christoph Steiner will lead you through all the steps.

Hans-Christoph will be talking Pd on the iPhone at a conference next month at Carnegie Mellon University in Pittsburgh. That will also include Memo Akten and Zach Gage talking about openFrameworks. And for nearly-entirely open source hardware, there’s a tantalizing-looking workshop combining Android with Arduino, too.

http://artandcode.ning.com/

There’s quite a lot more to say about this, but I hope this whets your appetite.

And, of course, riding the train to Pittsburgh or your nearest university or open source workshop, now you’ll have Thump which which to make some thumping tracks on your iPhone.

LaDiDa Demo from khush on Vimeo.

There’s no question iPhone/iPod touch development – really, just clever mobile development – has gotten a bit overhyped lately. But that’s all the more reason to do a round-up of genuinely interesting stories, real innovation happening on the platform. So, I’m clearing out my inbox with some of the more creative tools appearing recently on Apple’s mobile gadgets. There’s no better way to kick off today’s festivities than with this unusual “reverse karaoke” creation.

Sure, people may think they’re tone-deaf. But even the layperson has extraordinary powers of musical perception. So how could you train your iPhone to perceive and respond to music? That’s the question asked by LaDiDa for iPhone, the first of a new line of “intelligent” music applications for mobile devices. A “reverse karaoke” tool, the idea is to listen to singing and fake accompaniment, rather than having you sing along to canned backing tracks. Nothing is pre-programmed; everything is generated on the fly on the device.

It’ll even make up a Bollywood accompaniment to your singing:

LaDiDa Bollywood Duet from khush on Vimeo.

Of course, to me, it’s interesting not only what the iPhone is able to musically, but also what these algorithms are unable to make sound musical. Both reveal a whole lot about how we hear and conceptualize music. I think the team deserves real credit for making this fun, though, and on constrained hardware.

The app’s creator Khush follows in the footsteps of Smule in that it takes hard-core academic music research and uses mobile devices as a vessel for getting that tech in the hands (literally) of the general public. (See my interview with Smule founder and ChucK originator Dr. Ge Wang.)

Parag Chordia, developed at professor at the Georgia Institute of Technology and the gentleman you see in the video, spoke to CDM about what’s happening behind the scenes. He tells us about how this application was developed, and how the intelligent algorithms work (or at least try to work, as music analysis and auto-accompaniment remain at early stages).

First, an explanation of the app.

Khush CEO Prerna Gupta explains how it works:

1. You sing into the phone, and LaDiDa will compose music to match.
2. LaDiDa’s patent-pending technology analyzes the pitch and structure of the melody to compose a unique accompaniment for each recording.
3. To be clear, we do not query a database of pre-recorded songs. That is, LaDiDa has been designed to work with any music.
4. After recording your song, you can hear it with different styles. LaDiDa comes with three styles — E Piano Pop, Rhythm Synth Pop and Dub Tone — each of which has been developed using high-quality instrumentation to work specifically with our algorithm.
5. We will be launching new styles every month that will be made available through in-app purchases.
6. LaDiDa also works on rap! This month we’ll be adding three new rap styles.
7. After choosing your style, you can save the song and share it on Facebook, Twitter and email.
8. LaDiDa also has a Discover page, where you can hear songs recorded by other users from all over the world.
9. Khush was founded by music technology enthusiasts from the Georgia Tech Music Intelligence Lab. You can read about us here and also find out more about the research at our lab here.
10. LaDiDa went live in the iTunes store last week and is currently priced at $0.99.

Prena, the woman you see in the video, has some Web experience to boot, too, including founding a popular Indian dating site. Oh, and she’s a better singer than the music researcher, but, hey, that’s why we all went into computer music, right?

In case you’re wondering how you take a research idea and make it run on the iPhone – or how the algorithm works (and might get smarter in the future) – I turned to Parag for those details:

The initial code was developed in my lab in c++. Since the core algorithms are basically mathematical, that portion was relatively easy to port. However, we spent significant time thinking about how to optimize for the iphone and every aspect of the app, from the interface to sound design, has been built with the iphone in mind. For example, there are significant limits on sampler performance — samples have to be short and effects are more or less out — but we thought it was important for our styles to have a rich sound. So we
put great effort into designing light styles that sound realistic.

Another significant challenge was making the analysis robust to external noise; iphone recordings are lo-fi and corrupted with tons of background noises, which makes robust (and again computationally efficient) pitch detection essential.

Our approach to reverse karaoke is somewhat different than what’s been done before. A significant limitation of previous work was a lack of fine-grained key estimation, a problem that we felt was critical to successful vocal accompaniment (most people are not anywhere near a piano or an instrument with fixed tuning when singing into the app).

We also worked on trying to give some larger structure to the
accompaniment, which can often sound locally reasonable but notably lacking in direction. Again, a difficult problem particularly when people are singing snippets. Still it is sometimes possible to detect phrases, and we have tried to incorporate that information as well.

Auto-accompaniment is an endlessly fascinating and deep problem. As we learn more about human perception and cognition of music, as well as improve our tools for machine listening, our systems will become more musical. While we still have a ways to go, we believe that, with LaDiDa, we’ve created a product that is engaging and allows regular people to express themselves creatively.

If all of this talk about musical perception recalls the questions about how culture and background versus neurology can be used to explain music – as seen at the Notes & Neurons conference – that’s no coincidence. Researcher Parag played sarod with a fascinating ensemble at that same conference. Bobby McFerrin sings a really beautiful solo with the ensemble.

In fact, it’s absolutely worth contrasting the elegance and beauty of these all-human musical responses to the somewhat clumsy (sorry, Khush) iPhone responses. That’s not to say the iPhone creation is any less human – it’s a computation model programmed by humans, and is capable of some impressive feats made possible by their musical instincts and training. As such, we really can hear the gap between what advanced musicians can do intuitively and what we can model computationally, atop the restrictions of the device’s ability to sense the world around it.

World Science Festival 2009: Notes & Neurons, Part 5 of 5 from World Science Festival on Vimeo.

Bored with mouse pushing and knob twiddling? The d-touch tangible sequencer / drum machine makes a cheap interface (with free downloadable software) for assembling sequences. Make some (attractive) blocks, set up a webcam, and plug into your computer. I took a first look at this tool last month, and noted its use in sequencing walnuts. (Yes, the ones that fall from trees.) Since then, the developers have been hard at work on updates. Enrico writes:

We just released the d-touch sequencer, a new, more advanced, audio application. In the sequencer you can record your own samples in real time.

We also have few updates for the drum machine, which should solve the activation problems we were having at the beginning.

Go grab the markers and the software, and you have your own webcam-based drum machine.

Should you decide to go beyond their free instrument, the underlying system is really quite sophisticated. Part of what makes it beautiful is that you can design your own markers rather than settling for predefined patterns, as with most similar marker-tracking systems. There’s even a tool for correcting problems in your design. The freely-downloadable analysis software is written in C/C++, but if you use another environment (like Max or Processing or Reaktor), you can simply pipe data to your tool of choice.

The drum machine and sequencer are available now, so go download them and let you know how you fare! System requirements: a printer, a webcam, and a PC/Mac. Enjoy!

http://www.d-touch.org/

For some hands-on impressions of working with these things, the excellent PC Music Guru has a great description of the experience. Or, if you read the language, there’s a Japanese-language hands-on blog entry.

Software doesn’t have to mean virtualizing everything and letting go of physical objects. On the contrary, it can create all sots of imaginative, new ways of mapping musical ideas to the physical world. And that’s how we wind up with a walnut drum sequencer.

There’s something about virtual drum machines and snacks. We’ve seen bubblegum and Skittles, beer bottle caps, soda bottles, and now walnuts. Don’t stop now: someone has to do Cheetos, even if it means dealing with orange stuff all over your fingers.

That said, it’s not walnuts that make d-touch an important project. Built by Enrico Costanza back in 2003, the project is now available for free download as an open source library, a server (in case you don’t want to get into the C++ code but might want to use this in your own projects), a free, usable drum machine, and a set of documentation that can help you make your own stuff easily. Enrico worked on the original reacTable prototype and has done some really important work in this field. Right now, Enrico and co are looking for feedback, but if you’re ready to just be a tester and play with this – and see what you can do musically – now’s your chance.

d-touch also combines high levels of computer readability for accurate tracking with the ability to make your own tags. Instead of using ugly-looking glyphs, you can make patterns that make sense to human beings as well as computers. Oh, yeah – and mobile fans, this runs at a full 14 fps even on S60 phones.

For more, check out the d-touch site:
http://d-touch.org/ [Register first to make the download available]
and follow them on Twitter:
http://twitter.com/audiodtouch

Thanks to Martin (of reacTable, which is moving toward a commercial product) for sending this our way. Thanks, too, to Ben, who’s working on tangible interfaces with special needs students. I really look forward to hearing how that’s going.

Want to interact with your computer musically using physical objects and other fancy-schmancy, science-fiction-y interfaces? Don’t want to rely on Microsoft or wait until 2019? You’re in luck. It’s like Christmas for DIYers and interactive futurists.

Enter LusidOSC, a set of protocols, libraries, and useful code, and Trackmate, a clever and cheap-to-build system for tangible interfaces. Together, you’ve just got a bunch of tools to help you start playing with blocks — erm, experiencing new spatial interfaces.

• LusidOSC, the library maps “spatial input devices” – really, any tangible devices or sensors in the real world – to applications, like live music or live visuals. It uses the network-savvy OpenSoundControl protocol. The library is available now for Processing, with Java, Python, Flash, and C++ in the works.
• LusidOSC, the apps/code: a set of Processing utilities to get you started includes a basic example app, command launcher, playlist mixer, and a MIDI sequencer. You can build on these for your own Processing apps.
• Your own input device: Tags? Sensors? Markers? Regions? Fingertips? If it’s in physical space, you can map it via LusidOSC. Or, if you want a place to start, try:
• Trackmate: A project of MIT’s Tangible Media Group, Trackmate lets you affix tags to any objects you want and use them as an interface.

LusidOSC is just the protocol; it’s physical applications like Trackmate that get really interesting. In these economically-challenged times, Trackmate gives you tangible interfaces for next to nothing. All you need is a computer (Mac, Windows, or Linux), and a camera (even a webcam will do). Print out randomly-generated tags in the free and open source software, stick them on stuff you want to use, and go to town.

You can make your own Trackmate surfaces out of hardwood or plexiglass for as little as $40 in parts. There’s even a Processing-based simulator app so you can experiment without the physical objects. (Good for troubleshooting on the bus or plane, I imagine.)

Trackmate is just one project, though; LusidOSC promises to support other interface ideas, too.

LusidOSC at Sourceforge, the work of “many research institutions and companies.” (Which, I wonder?)
Trackmate at Sourceforge, a project of the MIT Media Lab Tangible Media Group

Alternatives: TUIO, the protocol used in the Bjork-endorsed Reactable project marches forward, as well. And as both projects are open source, there could be some cross-pollination. I hear a revised TUIO is coming, and in the meantime, LusidOSC acts as a gateway to TUIO.

The idea of the Reactable is to make music tangible, with control of sound mapped to physical objects you move around on a table. But that hasn’t stopped the Psychosynth from creating a virtual version. (Upside: it’s a lot more portable.)

Psychosynth

Watch the video, but they seem to have made the opening minutes as dull as possible to thin out the non-believers. Skip past the generation of the white noise oscillator (wow, white noise!), and somewhere around halfway through, it becomes laugh-out-loud funny, with trance-style vocals about freeing your mind with free software. (Seriously – it’s awesome.)

While it’s in alpha stage, the software is fully free and open source (binaries available for Debian and OpenSUSE) – and that means this could be a good project to snoop around in or code through, those of you who are eager hackers.

At its heart, it’s simply an interactive modular synth inspired by Reactable. Drag objects around and connect them to make sounds and patterns. Underneath is a powerful C++ synth library, a 3D synth, and even a server version – so even if you aren’t sold on this interface, there are pieces here that could be useful. It’s all virtual now, but that’s not to say you couldn’t add input; support for the input library is planned later.

That said, I don’t think they went far enough with the virtual thing. Next, why not simulate virtual players for the virtual Reactable inside the computer. They could even behave like Sims, requiring regular stimulation and bathroom breaks. Eventually, you could unlock Bjork.

Enjoy. If anyone gets this up and running even in alpha state, let us know.

Previously: tables and tangible, like the lovely acoustic sounds of Etiquette and, of course, Spaces / Roots