Yesterday, we saw a bit of the idea of making mobile gadgets more sociable with each other. The idea is, through network/wireless and cabled connectivity, you extend possibilities for expression, control, and collaboration with yourself and others. It’s the same thing that makes a recording studio useful: tools work together to let people work together to create music. It’s absolutely not a new idea; the engineering question is just making it work well on new platforms.

On iOS, we’re already seeing some of this: apart from MIDI connectivity, developer like KORG have even tried using wireless features intended largely for multiplayer gaming to connect multiple beatboxes and synths over a network.

Continuing a look at the Android side of things, Andrea Belloni sends an open source project that gets a sequencer working over a network with a computer and Arduino-based hardware project. It’s really a proof of concept – let’s call it a hack of concept – so they say specifically they weren’t concerned with making it pretty. It’s at an earlier stage of the process, but it’s nice to see that. The basic idea is to control the hardware sequencer, built on Arduino, with a Processing sketch running on the phone, all over a WiFi network. Take the same basic idea and streamline the hardware a bit (the Arduino and shield get a bit clunky), and this could actually be practical.

Android Arduino Sequencer [Italian + English]

But I think as an illustration of what I was saying in words yesterday, it’s worth covering this just so I have an excuse to use this picture:

Yep, that’s the idea. Add in some other operating systems to that graph, too, if you like!

The history of music and the history of music notation are closely intertwined. Now, digital languages for communicating musical ideas between devices, users, and software, and storing and reproducing those ideas, take on the role notation alone once did. Notation has always been more than just a way of telling musicians what to do. (Any composer will quickly tell you as much.) Notation is a model by which we think about music, one so ingrained that even people who can’t read music are impacted by the way scores shape musical practice.

All of this creates a special challenge. Musical notational systems had traditionally evolved over centuries. Now, we face the daunting question of how to build that language overnight.

This question has been a topic I’ve visited in a couple of talks, first here in New York at in/out fest last December, then most recently for a more general audience at RSVP, a new conversation series in Hamburg, Germany hosted by the multi-disciplinary design studio Precious Forever. (See photo at top, by which we can prove that the event happened. Check out more on the event and how the Precious gang hope this will inspire new interchange of ideas in Hamburg – something perhaps to bring to your town.)

What I’ve learned in talking to people at those events is, music notation matters. It’s more relevant to broad audiences than even those audiences might instinctively think. The most common lingua franca we have for digital music storage, MIDI, is woefully inadequate.

But perhaps most importantly: replacing MIDI’s primitive note message is far from easy. The more you try to “fix” MIDI, the more you appreciate its relative simplicity. And engineering new solutions could take re-examining assumptions Western music notation has made for centuries.

Musical notation and culture

Explaining the importance of notation to expert musicians is easy. But to convey its importance to lay people, you need look no further than the game interface developed by Harmonix for the hit titles Guitar Hero and Rock Band (and in turn descended from a similar interface paradigm used in the Japan-only Konami GuitarFreaks). These games demonstrate that, even among non-musician gamers, certain received wisdoms from Western notation endure. (In fairness, many of the designers of music games have a fair bit of musical experience, but the fact that their work is received by audiences in the way it is nonetheless speaks volumes.)

The Guitar Hero interface actually is a Western musical score, rotated 90 degrees to make it easier to see how the events on-screen are matched to game play input. (For visual effect, the “track” is also rotated away from the screen, so that events further in the future recede into the background – a bit of visual flair that helped differentiate Harmonix from flatter-looking Japanese games.)

Whatever the rotation, the assumptions of the game screen itself are rooted in notation. Pitch is displayed along lines and spaces, just as on a score. Rhythm is displayed along a metrical grid, which reads as a linear track. Not coincidentally, I believe, when Harmonix has deviated from this formula, their titles have tended to be less successful. More sophisticated interactions in titles like Amplitude and Frequency (and the iPod game Phase) were big hits among gamers, but less so among the general public, perhaps in part because they require a more abstract relationship to the music.

Games are just one example, of course. Musical scores reflect basic cultural expectations, and in turn shape the music that people in that culture produce. As with most Western languages, text flows from left to right and top to bottom. Ask people to describe pitch in any culture that uses this notational system, and they’ll use the notions of “up” and “down,” “higher” and “lower” – even though these metaphors are meaningless in terms of sound. (Indonesian culture, for instance, gets it more physically correct, by describing what we call higher pitches as “smaller” and deeper pitches as “larger,” as they are in gongs.) And music in Western cultures are also deeply rooted on a grid, on 4/4 time and equal subdivisions. It wasn’t always so: even in the West, prior to the advent of notation of these meters, metrical structures flowed more freely.

It’s little surprise, then, that some of the biggest successes in electronic musical instruments have adopted the same conventions. From the Moog sequencer to the Page R editor on the Fairlight CMI sampler to the array of buttons on Roland’s grooveboxes, rhythmic sequencers that follow the grids devised in Western music notation are often the most popular. Even if the paradigm of the interface is one degree removed from the notation, the assumptions of how rhythms are divided – and thus the kinds of patterns you produce – remain.

Nowhere is this more true than in MIDI. MIDI is itself a kind of notational system, around which nearly all interfaces in software and hardware have been based over the past two and a half decades since its introduction.

MIDI, keyboards, and piano rolls: An incomplete “standard”

The first thing to understand about MIDI is that it began life as a keyboard technology. A complete history of MIDI should wait for another day, but even as its early history is told by the MIDI Manufacturing Association, it’s a technology for connecting keyboard-based synthesizers, not a solution to the broader question of how to represent music in general.

Many of the tradeoffs in MIDI, though, were made long before the 1980s or the invention of digital technology. When the 19th Century creators of the player piano needed not only standardization but reproduce-ability – before the advent of recording, the power to recreate entire musical performances – they turned to the piano as a way of modeling musical events. Indeed, the first player pianos quite literally reproduced the process of playing a piano, using wooden, mechanical fingers to strike notes on the keys just as a human would, before that mechanism was replaced with the internal players familiar to us today. What these inventors found in the piano was an instrument that, in the name of accessibility, aligned pitch to a simple grid.

The piano is a beautiful instrument, but its great innovation – the grid of its black and white keys – is also its greatest shortcoming. That grid is an imperfect model even of Western musical pitches, let alone other cultural systems. The 12-tone equal-tempered tuning used on modern pianos makes tuning multiple keys easier, but only by way of compromises. Even a modern violinist or singer may differentiate between the inflection of a G flat and an F sharp, based on context, but to the piano, these pitches are the same. And tuning is only the beginning. Piano notes begin with a note being “switched” on and end with it being “switched” off – no bending or other events within that pitch as on most other instruments.

It’s little wonder, given MIDI’s origins as a protocol for communicating amongst keyboards, that the editing view most common in music software is the piano roll, labeled as such. The piano roll is the perfect paradigm for sequencing events played on a keyboard, but that doesn’t mean it’s the best language for describing all music. And the obligation of a digital protocol is actually greater than that of musical notation, because there’s no human being at the other end to fill in missing expression and context.

Consider what’s missing in MIDI:

• Pitch reference: By convention, MIDI note 60 is C4. However, musical practice internationally lacks a consistent standard for what the tuning of C4 is, and any number of variables can interfere, from independent tuning tables to the use of the pitch bend to the activation of an octave transpose key.
• Pitch meaning: MIDI note values use an arbitrary pitch range from 0 to 127, a hypothetical 128-key piano, which itself makes no sense. 4? 8? 15? 16? 23? 42? The numbers themselves don’t mean anything.
• Pitch resolution: Because of the 0-127 resolution constraints, to get notes in between the pitches, you need a series of separate messages like pitch bend, giving you two values with only an incidental relationship to one another. Since pitch range is kept in yet another message, the results are confusing and un-musical, far more complex than they need to be. (Why wouldn’t 60.5 be a half-tone higher than 60?)
• Real expression: Events between note on and note off are represented independently as control change values. But that causes problems, because it means there’s no standard way to represent something as simple as a musical glissando. On a synth, making an expression (like twisting a knob or turning a wheel) separate from a note (pressing a key) makes sense. But that doesn’t make musical sense, and it doesn’t match most non-keyboard instruments. Only aftertouch is currently available, and that again assumes a keyboard and doesn’t expose pitch relationships created by adding the data.
• Musical representations of tuning and mode: The MIDI Tuning extensions require that you dump tuning information in fairly unstructured System Exclusive binary dumps. The standard itself is in some flux, and at best, its reliance on byte messages means that it’s not something a human being can read. And it still must be aligned with 128 otherwise arbitrary values. It’ll work, but it only makes sense on keyboards, and even then, it’s not terribly musical. Looking at number 42 in your sequencer, you’d have no idea of the tuning behind it, or the position in a mode – something any rational musical notational system would make clear.

Ironically, it was this very set of constraints that early innovators on the Buchla and Moog synthesizers hoped to escape. They were fully aware that the very genius of the keyboard was restricting musical invention. Analog control voltage, the basic means of interconnecting equipment prior to digital tech, was more open ended than MIDI, which replaced it. But that’s not to say it was better. Standardization is an aid in communication, as is the ability to describe messages. The question is, how can you do both? How can you be open ended and descriptive at the same time?

How do you build a new system?

Deconstructing is easy; constructing is hard. We certainly have the ability to send more open-ended messages and higher-resolution data; that’s not a problem. (Even by the early 80s when MIDI was introduced, its tiny messages and slow transmission speeds were conservative.) We also have OpenSoundControl (OSC), which has some traction and popularity, including near-viral use on mobile devices and universal support in live visual applications. It’s telling that that protocol is itself not really an independent protocol in the sense that MIDI is, but built on existing standards like TCP/IP and UDP. 2010 is, after all, not 1984.

The hold-up, I think, is simply the lack of a solid proposal for how to handle musical notes. And there are plenty of distractions. It’s tempting to throw out the simplicity of MIDI’s note on and note off schema, but it’s partly necessary: with a live input, you won’t know the duration of a pressed key until that key is released. It’s equally tempting to cling to Western musical pitches, even though those pitches themselves lack solid standardization and don’t encompass musical practices in the rest of the world. (12-tone equal temperament is a recent invention even in the Western world, and one that doesn’t encompass all of our musical practice. World tunings should best be described not by majority, but plurality, anyway – have a look at the current demographics of Planet Earth.)

One solution is simply to express musical events by frequency. That’s not a bad lowest common denominator, or a way to set the frequency of an oscillator. As a musical representation, though, it’s inadequate. It’s simply not how we think musically. The numbers are also unpleasant, because we perceive pitch roughly logarithmically. Pop quiz:

Can you do logarithms in your head? Yes or no?

Can you count?

MIDI gets it half right by using numbers, but then it’s hard to see octave equivalence, another essential concept for perceiving pitch. MIDI note 72 is probably equivalent to MIDI note 60… assuming 12 steps per octave. Or it might not be.

If you need a common denominator that covers a variety of musical traditions, mode (or more loosely, pitch collection) and register aren’t a bad place to start. I don’t think a system needs to be terribly complex. It could simply be more descriptive than MIDI is – while learning from the things MIDI does effectively.

Consider a new kind of musical object, described over any protocol you choose. It would ideally contain:

• Mode/pitch collection: As with MIDI and the MIDI tuning tables, tuning would need to be defined independently, but it can be done in a musical, human-readable way. It then becomes possible even to define modes that have different inflections based on context, as with pitches that are slightly different in ascending and descending gestures (common in many musical systems).
• Relative degree: a notation like “1 1 2 3 5 6″ can work in any musical language. You simply need to know the active mode or pitch collection.
• Register: Instead of conflating register and scale degree, you could simply define an octave register and starting frequency. This retains modal identities and octave equivalence, and makes relative transposition easy to understand. (A “transposition” message could be defined as an actual message, which is more musically meaningful.)
• Standardized inflections, connected to pitch: Pitch bends and glissandi should be relative to a specific note, because notes can have pitches that bend around their relative scale degree. (Think of a singer bending just below a note and into the actual pitch. These aren’t independent events.) A trombonist would never have invented MIDI notes. They would likely have immediately turned to the question of how to universally describe bending between notes.
• Yes, frequency: There will be times when directly referring to frequency makes sense, and that should be possible, as well.
• Relative duration: Musical notation, regardless of musical culture, uses some kind of relative indication of duration. Only machines use raw clock values. The result is that it’s possible to make musically meaningful changes in tempo and have durations respond accordingly. And whereas note on and note on events make sense on input, a musical event would not logically separate these events; there’s some notion of an event with a beginning, middle, and end. If you sing an ‘A,’ that’s one event, with a duration, not an independent beginning of the note and end of the note.

Far from replacing existing standards for music notation, this kind of standard could interchange more gracefully with printed notation. If you import a standard MIDI file into notation software, you get results that are typically full of errors, because the SMF lacks musical information about the events it contains. With more of that information stored, and stored in standard ways, translating to paper would become vastly more effective.

I’m sure attempts to model this in OSC have been attempted before, but it’s worth compiling those ideas and resurrecting the discussion.

What about input?

Ah, you say, but then, let’s go back to the keyboard. None of these events makes sense on a keyboard. You don’t know when a note is pressed how long it’ll last. You don’t know the modal degree of a particular, arbitrarily-played note.

I was stuck on the same problem, until I realized what I had been taking for granted: MIDI conflates two very separate processes. It makes input and output the same. Musical notational systems have never done that. When you look at a score, it’s a set of musical ideas, given meaning and context. If you record a series of events from an input, those events don’t immediately have meaning or context. It’s confusing the mechanical with the musical. It’s the reason MIDI is not just like a player piano – it is a digital player piano.

Separate out the issue of recording mechanical input events, and you can have a system that’s more flexible. That system should fit whatever the input is. An organ, a shakuhachi, a didgeridoo, and an electric guitar aren’t the same thing. Why would they be represented with the same set of input events? That’s pretty daft.

Look at it this way: imagine if instead of being invented by synthesizer people, Aeolian Harp players had invented MIDI. (It’s not so far-fetched: the Aeolian Harp has a millenia-long history and was once quite popular.) An Aeolian Harp sequencer would feature elaborate, high-resolution data recording for wind pressure relative to different strings. It might measure, even, wind direction. In fact, it’d look a lot more like meteorological data than musical data per se. It certainly wouldn’t involve integers from 0 to 127.

This should lead to a simple conclusion with profound consequences:

Physical input and musical output should not be the same thing.

One of the advantages of a protocol like OSC (or any open, networked, self-described protocol) is that it can be open-ended and descriptive, meeting our earlier challenge. For instance, using a hierarchy of meta-data attached to the message, you could describe a set of variables relevant to wind input. If you wanted to transcribe the results in musical terms, you could then use a musical notation, as above – one that used musical identity attached to the resulting frequencies, as in relative modal pitch and rhythmic duration. But the input would be a separate problem. That’s a far piece from MIDI, which is adequate neither as a complete description of the input device, nor of any kind of resulting musical system.

But wait a minute – how is there a standard? How do you standardize something that could include an Aeolian Harp, a vuvuzela, and a bagpipe? Welcome to the problem of music. Music is by its very nature resistant to standardization, because the possibilities of the physical world are so broad. This also suggests how input protocols (and output protocols) can go beyond musically-exclusive data. Again, we can turn back to MIDI as a model. MIDI was intended with specific applications in mind, with messages that referred to MIDI notes and filter cutoff. But that didn’t stop it from being warped to accommodate tasks well outside the standard, ranging from triggering videos to controlling amusement park robotic characters (literally). This suggests to me that what defines a standard protocol of this kind is not what is most strictly standardized, but what is most flexible.

The real challenge with something like OSC, then, is to come up with standardized ways of defining non-standardized events, and using some kind of reflection or remote invocation to allow devices or software that have never communicated before to handle unexpected messages intelligently. At the very least, they should give users clear, understandable options about the data they send and receive. This independent question has been one the OSC community has raised for some time. To me, all that remains is to make some compelling implementations and let the most effective solution evolve and win out. Recent reading on the topic (though this absolutely deserves a separate post, which I’ll get to soon):
Best Practices for Open Sound Control
Minuit : Propositions for a query system over OSC

That’s a separate problem from how to make events musically meaningful. But that to me is the central revelation, and something MIDI completely misses: these are two separate problems, not one problem. Handle input events as input. If it makes sense in a sequencer to record them as musical events (like scale degree pitches), do that. If it makes sense to record them as a series of time-stamped, physical events, do that – but with actual information relative to what was recorded, so that the wind across an Aeolian Harp is recorded in a way that makes sense for that input. And when describing musical events, describe them in musical ways.

This isn’t relevant only to music communities, either: it’s relevant to anyone recording events in time. It’s part of the reason the “sound” needs to be dropped from OSC. MIDI is as specific as it is partly because the specification has messages too small to contain information describing what the events mean. We now have standard network protocols that do that, so they can include information about other kinds of events. There’s no reason someone monitoring water levels in their herb garden and someone recording a sousaphone solo couldn’t use some of the same underlying protocols. There’s also every reason they’d record different kinds of data content.

Promising venues and a call to action

There’s really no need to try to “replace” or “fix” MIDI – if MIDI has endured for a specific application, maybe it actually is well-suited to that application. I think it’s time, instead, to think about how new systems can encompass more musical meaning from our own traditions and traditions around the world, and how we can standardize broad ranges of events instead of trying to fit everything into narrow, rigid boxes.

There is every reason to believe new things can happen now, too. Whereas hardware standardization once was a slow process, requiring the involvement of major manufacturers, we now carry around programmable computers inside our pockets as “phones” and learn to write embedded code in Freshman college classes using $30 Arduino boards. If you want new hardware standards, you can literally make them yourself. We have the ability to share musical notation directly in a Web browser using standard descriptions, as covered here recently. Because browsers in general are demanding newly distributed, networked applications, communicating in standard ways – as Web APIs do naturally – is becoming imperative.

But there’s one thing that makes me especially optimistic: you. Via the Web, we have instant access to your collective knowledge and experience. That means it’s a sure thing that all of us, collectively, knows more about previous research in this area, previous ideas, and what has and hasn’t worked. We also have the opportunity to communicate with each other, to make ideas evolve, at least experimentally. That doesn’t remove the need for eventual standardization, but good standards follow practice, not the other way around – something has to work in one place before it can be a shared standard. We also have mechanisms for self-standardization that didn’t exist before. Spoken languages evolve because people collectively work to share common means of communication. You might argue that this leads to a tower of Babel, but then, I’m writing this in English and you’re reading it in the same language and (hopefully) understanding. The same is true of Mandarin, Portuguese, German, Arabic, Hindi, and so on. It’s also true of volunteer adoption on the Internet of HTML, XML, JSON, and RSS.

Music is not the result of notation or standards. It’s the other way around. Musical practice long predated any attempt to write it down. And mathematics and written language each have abilities to describe music and many other media.

To me, two questions remain:
1. What would an implementation of structured messages for pitch and duration look like, perhaps implemented via OSC? What history has been there in this area, and what do you need?
2. How can smarter implementations of a protocol like OSC allow software and hardware to better handle unfamiliar input – as musicians, as they have done since the dawn of time, invent novel physical interfaces?

I look forward to kicking off this discussion and hearing what you think.

Surprise! Plug-in developer Ohm Force, known for their plug-ins (like effects Ohm Boys and Frohmage), today tease an upcoming collaborative host. It looks like the sort of thing Apple could have done, but hasn’t. There’s a GarageBand-style MIDI and audio editing pane, plus semi-modular routing of plug-ins on a pretty, graphical surface that resembles the “cheese grater” perforated aluminum of a Mac tower, and pop-up window palettes that resemble those we’ve seen on the “flattened UI” of the iPad.

The real feature here, though, is collaborative editing in the “cloud”: sessions are uploaded to a server, which in turn keeps track of versioning. (Actually, it’s quite unclear how that works collaboratively – this means you can “undo” from one version to another, but I can’t tell whether collaborators can try different “forks,” or if it’s all one set of linear changes.) The changes are “real-time,” though usually the trick to allowing international collaboration over the Internet is to make things delayed enough that everyone stays in sync.

There’s also an accompanying Web community for connecting with collaborators. Everything else about the product, however – more features, pricing, and specifics of how it all fits together – is as yet unknown. Mac and Windows are both supported, though – something Apple would not have done, most likely.

It’s an interesting idea, one I think we’ll see more frequently as connected applications grow in popularity. Among other options, Ableton had promised something like this with Share and then fell off the radar. Image-Line had a collaborative tool called Collab for its FL Studio, then abandoned it. The most significant competition comes from tools like Indaba. Indaba’s edge: by being powered by Web tech, you can do all your editing right in the browser; serious users can then keep using their host of choice and just bounce out audio. But while Indaba has an offline editor, too, the addition of plug-ins in Ohm Studio is a big change.

I do wonder with all of this, though: are we consigned to collaboration existing only in proprietary, integrated app-website combinations? Isn’t the whole lesson of the Web about open standards and platform-agnostic communication? Having said that, what would a more open tool look like – and what do people really want to do? (For instance, I wonder how hard it’d be to build a system that allowed open chat and transport control, with standards-based versioning and sharing, using the open-source DAW Ardour? See the post I’m … about to write … for the OSC end of this.) On the other hand, is the kind of integration Ohm Studio is offering necessary to make it all work together? (That last question we should be able to answer once this is in our hands and ready to try.)

I don’t wish to pre-judge Ohm Studio – on the contrary, I think this is a provocative product teaser that immediately raises some of these fundamental questions. So bravo, Ohm, for starting that conversation; I can’t wait to see what you’ve cooked up. And anything that gets artists collaborating is potentially a very good thing.

In the meantime, readers, it seems the most important question falls to you. Do you even want to collaborate with other artists? What would an ideal system look like for doing so? What features would you want? How would you want to work? Is real-time important, or do you prefer some time to sit back and think about how elements combine? When you collaborate now, how do you go about it?

Updated: Your closest bet at the moment is NINJAM, which is integrated with Reaper – a host various folks are using already – or on its own. Using compressed audio streams and latency compensation, it allows the exchange of any audio, and it works on Windows and Mac. But it doesn’t exchange MIDI data. (The site refers vaguely to this happening some time in the future, but I’m unsure of their progress.) And it doesn’t have integration with the Web community, though as readers note, you may want to work with people you get to know a bit first, anyway. Thanks to “PooPoo the Korruptah!” for the tip. (Hmm, no way for me to say that and not sound silly.)

More importantly – anyone out there using it? Or is it just easier to send files back and forth?

http://ninjam.com/

The iPad could have its first killer controller app, if you like the looks of midipad. While just a series of screenshots at this point (also promised for iPhone, but not yet available), midipad promises some intriguing features: it’s pretty, it features lovely control layouts and widgets, and it makes use of MIDI networking to make setup and configuration a cinch (especially on the Mac).

• It uses network-MIDI: connect and play. With support for the Zeroconf standard (what Apple calls Bonjour), and MIDI over network communication, the midipad can appear as another network device on the Mac. (It sounds as though you still need software to translate the network MIDI data on Windows, though more on that thought in a separate story.)
• Pre-built layouts and controls: Buttons, trigger pads, sliders, ribbon controllers, rotary knobs, and pre-defined macro “blocks” like transport control combine to form studio, DJ, launching (as in Live), and effects layouts.
• Submixes/multiple setups It’s a little sketchy in the description, but it sounds as though you can also combine different layouts.

No pricing or availability yet, but you can peruse the site:

http://www.midipad.de/

The inevitable comparison will come to the Lemur, the pioneering multi-touch controller from JazzMutant. Those comparisons are fair, indeed. Certainly, the primary appeal I’ve heard from many Lemur users – unique control widgets available on the Lemur platform – are just software. There’s nothing stopping clever developers from making appealing controllers for the iPad (or other Android, Windows, or Linux pads, if any of those prove worthy). And the Lemur doesn’t support this intelligent use of the Mac’s network MIDI support for zero-configuration setup.

Some of the Lemur’s tradeoffs are reproduced here, too, however. Like the Lemur:

• You sacrifice the tactile feedback of real hardware for flexibility.
• It appears you have to create controls in advance, not on the fly.
• The controls still largely emulate the interface paradigm of conventional hardware, rather than imagine new controls for a screen.
• It’s just a controller, separate from a screen, rather than your sound-making software itself having this sort of touch interface.

Those aren’t necessarily criticisms: these kinds of controllers are clearly useful, and it’ll be great to see how this gets used. What I’d say is there’s an opportunity to design touch-based musical tools that aren’t just Lemur-style controllers. (Consider what happens if the sound-making software uses touch in place of the mouse – that could inspire an entirely new entire generation of software, not just one or two useful apps. We’re still just missing the appropriate hardware. More on that soon)

Actually, I only have one potential criticism, which is that while this low-contrast, white-background layout looks refreshingly different and pretty, it’d likely be blinding in a darkened performance environment.

But I’m looking forward to seeing how this works – and I imagine we’ll see a number of other similar apps available even from launch day on the iPad, with more to follow. I’m not buying an iPad, but I will be looking for early adopters who can report back. I am genuinely interested to see how the platform evolves and what developers design for it.

Thanks to everyone who sent this in, and yes, Synthtopia beat me to writing this up.

What about JazzMutant and Stantum?

I think this requires a postlude. JazzMutant is in no danger of going out of business, as far as I know. You’ll see one little line at the bottom of the about JazzMutant page:

“Since mid 2007 JazzMutant has become the Music & Media product division of Stantum Technologies.”

That line means everything. Stantum is a vendor in multi-touch technologies to OEMs, including makers of Windows 7-powered touchpads and laptops:
http://www.stantum.com/en/

One of the things I wrote back in 2004 was that I thought, cool as the Lemur was, eventually we’d see the touch display and the software making music, visuals, and the like merge into one and the same device. (Look at it this way: you don’t buy a separate computer to connect your mouse and keyboard to, do you?) Stantum is the sort of company that could enable just that sort of technology. The fact that they’ve stayed in the Lemur business seems to me to be very intelligent, because it means they continue to interface with musicians and DJs – many of whom have remained loyal to the Lemur. Even if you’re not in the market for a Lemur and get an iPad, Stantum remains one to watch. And if you get neither a Lemur nor an iPad but are interested in where touch is going, Stantum is definitely one to watch.

More on Stantum and the rest of the touch landscape soon; the iPad is important, I think, but partly because there is a bigger picture.

What happens when an interface is no longer locked to the screen? What about making control simply work from your hand, on a different screen, with awareness of the world around it? Simple as the early implementations may be, that’s really the vision behind mobile control applications for music and visuals.

c74 is a lovely iPhone-based app that uses a Max/MSP patch to generate interfaces from a patch that run on your handheld. It isn’t just a control surface, though; access to native APIs on the phone also provide other features.

• GPS for specific location. (How you use that is up to you; I recommend the ability to switch between “West Coast” and “East Coast” beats.)
• Accelerometer data, and specific “shake” gestures.
• Compass orientation.
• Proximity. (That means your proximity to the device, though it’d also be fun with mobile to use Bluetooth to tell when different devices are nearby.)

The external is free. It’s currently Mac OS X-only. (If people respond well, perhaps we can see about a Windows build.) The app itself is paid, but see below — Mac and Max/MSP users, I’ve got some codes to give away.

The app is called c74, but it’s entirely unofficial, the work of Dutch developer Leo van der Veen. (The site even has the cheeky name nr74.org; Cycling ’74, I’d say you have a fan. And, um, I’m glad you’re a small company rather than a litigious big one – that’s better for everyone.)

This isn’t the only app to attempt something like this. The iPhone app Mrmr attempts to produce an entire interface protocol for exchanging templates; I’m collaborating now with the developer of that app on … something. I won’t jinx it by saying more for now; stay tuned. (It’s nothing earth-shaking; I just want to finish it!) The idea of Mrmr is that a performer could beam a control template to a friend, or you could walk up to an art installation and grab the template on your own mobile; eventually, it could even work on different devices (not just Apple ones).

c74 is distinct from that effort in a couple of ways. It’s very specific to the iPhone APIs. It’s really specific to Max, and focuses on generating those templates from a Max patch in a flexible manner. It doesn’t actually even use OpenSoundControl, though the principle is the same – it opens a network socket and communicates wirelessly. (In fact, just as you can represent MIDI messages without using a 5-pin MIDI DIN cable, you can use OSC-formatted messages without needing the whole protocol. And underneath OSC is really just a bunch of standard networking protocols – that’s part of the point. So we almost need another name for general-purpose, open “networking.” Actually, maybe the word “networking” works.)

Right now, there isn’t a whole lot of interoperability between different development environments, different mobile devices (there is a big world beyond iPhone), and different template apps (which are growing in number by the day). But I’m not overly concerned about that. Right now, I think it’s just as well that people try a lot of different experiments.

As that evolves, the next step could really be finding ways of communicating template information in standard ways, based on the real-world experience of how these apps work. TUIO has already caught on as a way of describing multitouch input, as seen first on the ReacTable. What that protocol does is actually fairly cool: it just takes a set of practices used by real artists, and builds the standard protocol around those practices, rather than the other way around. I think it should be possible to continue to expand on these kinds of descriptions.

Want a copy of c74 for your iPhone / iPod touch and your Max 5-running Mac? I have just ten copies to give away. So tell us in comments what Max patch project you’d like to run with this, and convince me that you’ll send us a short video of the results. I’ll take the first ten compelling comments and email you a code. (That means you DO need to leave an email in the comment form; only the site admins and I can see it.)

I’m in touch with Leo, as well, so we’ll certainly keep this discussion going about control in general.

Project site:
http://www.nr74.org/c74.html

Update – important – US ONLY for codes.

It’s drumming, the multi-player game. The Drummer is an open-source application for the Nintendo DS handheld, developed by Andrea Bianchi and Woon Seung Yeo and presented alongside a paper earlier this year at the NIME Conference (The International Conference on New Interfaces for Musical Expression). As with any Nintendo homebrew software, you’ll need a special DS cartridge capable of loading software from flash memory – though if this app were developed more, it could make a terrific DSi app.

The idea is this: while making a handheld game system into an instrument, why not take advantage of its networking features? Grab a friend (or friends) with the Nintendo DS, whip up a drum kit that’s to your liking, then play along.

Oddly, while we live in a networked, Internet age, the client-server model rarely gets applied to music. One of the things I try to explain about the protocol OSC (OpenSoundControl), aside from the fact that it doesn’t have to be about sound, is that it’s really a collection of best practices in open networking communication. It’s not simply about connecting devices to one another in serial fashion, as with MIDI, but forming a network – an idea familiar to anyone who uses instant messaging online. True, latency considerations and other complications can add another dimension of challenge. But I think there’s plenty to explore when it comes to networking devices, and it promises to make computer music a less solitary experience. The Drummer is a good step in that direction.

The Drummer project page + NIME paper
Google Code page

Thanks to Andrea for sending this our way!

You probably think of social networking and messaging as being about text, about saying things like “Wow, this tuna salad sandwich I’m having for lunch is delicious!” But the next Tweet you get on Twitter could be a synthesis preset.

Say what?

Working in Max/MSP, Andrew Spitz has developed a tool called Tweet a Sound. It uses Twitter as a communications platform for “social sound design.” Instead of just saying, “Wow, I be makin’ phat basslines,” you can actually share the sound. Whip up a sound using typical FM synth parameters and Max/MSP’s sound engine, then click “send.” You’ll send a string of numbers to your Twitter account, confusing those friends not in the know. But other users will be able to grab and play with your sound.

Andrew even encourages synthesis n00bs to play without fear – grab those envelopes and mysterious-looking settings and see what comes out. So, I hope you synth geeks do share this with some friends new to synthesis, as I think they’ll have a great time.

Right now, Tweet a Sound is Mac-only; we just need someone to save a Windows standalone version. Someone has asked about a Pd port, but let’s put it this way: this is the tip of a very, very big iceberg of sharing. It’s something worth considering in anything you’re doing, not just with Twitter, but whether you can provide networked capabilities in whatever you’re happening to build.

Ableton, of course, recently added the Share functionality to Live. But with open APIs and basic networking protocols, there’s no reason you can’t explore other features. Why not build a drum machine that lets you collaborate with one of your friends on your IM list, or a sequencer that automatically posts ideas as you revise them? Just doing these things for the sake of it could be a waste of time, but on the other hand, these social features could turn Web 2.0 sites into places that actually inspire you to make and share music rather than distract you with mundane activities.

I love the idea; let us know if you have some fun with it.

Tweet A Sound { sound + software } [Andrew Spitz Blog]

TUIO is a simple but powerful emerging protocol for multitouch control for live music and visuals, as used on the powerful live tangible synth reacTable. Apparently no one told Apple, however. While the App Store rubber-stamps useless toys like fake cigarette lighter flames, they bizarrely rejected a powerful application by a leading digital artist that would enable standardized TUIO control – for free. (More back story below; see an example in action above.)

As a blogger, my reaction is usually to whine and pontificate, for better or worse. The engineering approach would be to find some hack away the problem. That’s what Andrew Turley did with the TUIO protocol. So, Apple won’t allow an app that does the trick? Why not go back to what developers did before the SDK, and just use the iPhone browser?

As Andrew explains it:

After reading the story I started thinking about seeing how far one could push Safari as an application platform, using web apps to get around Apple’s tight control of the app store. Since you would be connecting to another computer anyway to use an OSC application, why not just have the app be a web app running on a web server somewhere on the local network? The web server can then take care of things like sending out OSC messages or playing music or doing whatever it is people want to do.

To that end I created a little system that implements the TUIO protocol. You use an iPhone to run a web app, which in turn talks to the web server, which in turn sends OSC messages.

Details:

touchy feely [Pillowsopher Blog]

Needless to say, there are some downsides: you have to run the Python server, you’re more limited in input and control than you would be in a real app, and you’re stuck inside the Safari browser, which could be a bit inconvenient. So I’m not backing down from my original complaint – I don’t see Apple doing anyone a service by blocking this kind of app, and the only rational explanation seems to be that the folks doing the review process don’t understand what the app is.

Most importantly, I do know that all our griping was sent to Apple, so I’m hopeful the App Store will reconsider the decision once they get it.

But I love solutions, too, and this can be one for many applications.

It also illustrates an important point: the browser on mobile devices (Apple and otherwise) could be a powerful outlet. It doesn’t always make sense to build an entire application; there will be various cases in which a little browser tool will do a job. Need a quick remote control for a live performance / art installation / club lighting rig? You might try the easy solution with the browser first.

Thanks for the great hack, Andy, and I’m curious to hear if anyone uses or extends this.

Previously:

Apple Rejects Free iPhone Tool For Artists Because of “Minimal User Functionality”

Smule, the folks who brought simulated Ocarina to the iPhone, are now thinking multiplayer. Instead of just playing a machine or a few of your friends as in Guitar Hero or Rock Band, Smule’s latest app turns your creations into a reality show with online judging. And the killer app itself? It’s a simulated, touch trombone.

It’s pretty wacky stuff, but Smule’s had some hits on their hands already, so I think the wackiness may be part of their secret. And it comes from a heavy hitter: Dr. Ge Wang, the founder, is also a professor at Stanford’s CCARMA research center, director of the Stanford Laptop Orchestra (SLORK to the East Coast’s PLORK), and creator of the ChucK Programming Language.

New in this version:

• Teaching tools: Floating leaves guide first-time musicians to learn songs in “self tutorial” mode, and a browser-based composition tool helps teach you to compose.
• Global judging: Online audiences can judge songs with emoticons and text and a 1-10 scale. Everything is integrated with the app itself (see image). Tromboning with the Stars, anyone?
• The Power of Silliness: The most important feature, though, may be that everyone sounds a little goofy playing it, which can actually be liberating. As Dr. Ge Wang puts it, “It’s like singing in the shower.” Well, except with judges.

The app is simple, but the concept I think is pretty remarkable. We’ve seen interactive instruments, and we’ve seen music games. By adding the judging element, though, this is a free-form instrument that can also be a game. Now, without getting too ahead of ourselves, you could do the same thing with a Worldwide Online Kazoo Contest. In fact, maybe that’s a great idea. I suppose you could say music itself can be a kind of social game, played out on a stage. But nonetheless, making it an iPhone app can help free people up to get that message.

Oh, yeah, and got any doubts about the business model for open source? ChucK is a completely free, ridiculously powerful programming language for synthesis. It’s infinitely deeper than Leaf Trombone. But that power, packaged for a broad audience, can become a hit business – and likely to be popular well beyond musicians. If that’s possible, I imagine more could be soon. Remember, too, that whatever the Apple fanboys tell you, the iPhone is not a dominant mobile platform – nowhere close. Apple’s proprietary hardware means it isn’t really intended to be.

I hope someone working on platforms like Symbian and Google Android takes note: pack in geeky, nuclear-powered synthesis features, and people will find ways to put them to use in consumer apps that appeal to everyone. Leave them out, and you miss the boat. Or the trombone, anyway.

Direct iTunes App Link [99 cents]

http://smule.com/

Now, some very amusing videos of this thing in action:

The song “Daisy Bell” has a special place in computer history. Max Mathews, who had by the late 50s pioneered digital synthesis using IBM 704 mainframe, arranged the tune in 1961 for vocoder-derived vocal synthesis technology on technology developed by John Larry Kelly, Jr.. Kelly himself is better known for applying number theory to investing in the markets — an unfortunate achievement in the wake of a financial collapse brought down by misuse of mathematical theory.

In 1962, Arthur C. Clarke happened to hear the 704 singing the Mathews/Kelly “Daisy Bell,” and the rest is (fictional) history – the HAL computer in the book and movie sings the song as he is being disconnected, as though the computer had learned this song as a “child.”

Here’s Max himself (namesake for Max, the patching language), overseeing a rendition of his arrangement:

Today, basic vocal synthesis has become part of the fabric of taken-for-granted tech, and the legendary rendition by a singing robotic voice part of our culture. These things are no longer futuristic or strange. Apple this week even launched a music player that announces its own tracks in the form of the new iPod shuffle.

But what happens when those same human beings imitate the computer? That’s the question asked by artists Aaron Koblin and Daniel Massey, crowdsourcing human input by inviting thousands of participants to contribute their voice using custom recording software built in Processing. The basic technique is something Koblin has used before: his Sheep Market massed an Internet labor market, paid two cents on Amazon’s Mechanical Turk, to draw walls full of thousands of sheep. Those sheep proved at once massive in quantity and unique in individual quality, and, if you squinted at them, presented a critique of global labor practice.

Koblin has also done various seminal pieces with the Processing coding language that change our perception of data and technology, like his now oft-cited “Flight Patterns,” tracing the paths of overhead planes.

This time, the computer/human relationship is truly inverted. Each singer participant imitates a sound component from the robot singing. The humans are then combined to synthesize the robot sound instead of the other way around. The result: organic technology combined into a cyborg, online chorus. No one singer knows what it is they’re singing in whole. It’s perhaps the first mass-human synthesis of sound, and the results are truly unusual.

And strange synthesis seems to be what Koblin’s work is fundamentally about. Perhaps it’s not Mathews’ sound experiments, but Kelly’s ideas about quantifying global markets that are most relevant. (For an extra dose of irony, Google HAL – you’ll get stock ticker HAL, for Haliburton, one of the few stocks that has grown in this economy.) In our reality, the University of Illinois didn’t create a super-smart, spaceship-controlling robotic brain – but they did create the Web browser.

And after all, all of us are now living in the aftermath of many crowds of people behaving collectively without genuine larger knowledge of what they were doing. Robots were envisioned at the beginning of the 20th Century as out-of-control automatons, crushing civilization, and were often then appropriated as metaphors for fascist government. Now, the vision can be equally apocalyptic, but the meaning is inverted. It’s human beings acting as automatons – without contact with human scale – that threaten to crush the Earth. And this time, they’re capitalists.

On the other hand, the beauty of art is its ability to mean many things at once. Koblin’s sheep and now his singers never cease to be whimsical. And in their beauty, they suggest that perhaps even massed crowds of Internet-connected people can sing in harmony.

For the future of humanity, I hope so. But then, if we fail, we’ll always have the robots.

“Just what do you think you’re doing, Dave?”

Bicycle Built for 2000: Info