For many, mobile technology and developing for the iPhone and the iPod touch is a fad and a Gold Rush. Good designers, though, take a longer view of how interaction can be expressive. And there are few people with a better sense of the big picture of small devices than Dr. Ge Wang. The co-founder, CTO, and Chief Creative Officer of Smule has a background that goes well beyond the latest Apple platform. Along with Perry Cook at Princeton, Ge Wang is the co-originator of ChucK, a real-time programming language for synthesis so efficient some people use it live onstage. (ChucK, as an open source project, now has a terrific team of people behind it.) ChucK is the sonic engine that powers Smule’s projects. Ge Wang also teaches at Stanford, working with students and fellow researchers to explore new ways of interacting with music technology.

Ge Wang joined me for a lengthy phone conversation recently. He really contextualized why the iPhone is important in the grand scheme of things, but also how the people at Smule and Stanford (and Princeton) can approach technology for musical interaction, focusing on what devices are rather than what they’re not.

(The audio here, believe it or not, is extensively edited – Ge Wang is that easy to talk to. I hope the next time it’s over beers rather than Skype.)

The full interview can be played below, or downloaded directly.

Download MP3 of the interview

Thanks to KORG and the Nano Series for their support of programming on createdigitalmusic.com.

Lastly: a video of the Smule team headquarters and playing around with Leaf Trombone for a Zelda duet!

More information:
The Mule Chronicles [Smule Blog]
http://smule.com/
Center for Computer Research in Music and Acoustics (CCRMA), Stanford University

Previously, for more on Ge Wang and CCRMA:
Make:TV Meets Stanford Musical Inventors, Feedback Piano

Video + Audio Subscriptions, iTunes Podcast

CDM is now launching regular audio content on the artists and inventors we cover as part of our series CDM Sounds. You can subscribe (and review the podcast) via iTunes, where you’ll also find our new video series:

cdm Sounds Podcast [audio]
cdm TV [video]

Or using your software of choice, subscribe directly to RSS. (I like to follow podcasts with Banshee and Winamp this way.)

We’ve fixed some transcoding issues for iPod touch/iPhone on the video podcast. Please do test this and let us know if you have any issues on your software/hardware.

Smule, the folks who brought simulated Ocarina to the iPhone, are now thinking multiplayer. Instead of just playing a machine or a few of your friends as in Guitar Hero or Rock Band, Smule’s latest app turns your creations into a reality show with online judging. And the killer app itself? It’s a simulated, touch trombone.

It’s pretty wacky stuff, but Smule’s had some hits on their hands already, so I think the wackiness may be part of their secret. And it comes from a heavy hitter: Dr. Ge Wang, the founder, is also a professor at Stanford’s CCARMA research center, director of the Stanford Laptop Orchestra (SLORK to the East Coast’s PLORK), and creator of the ChucK Programming Language.

New in this version:

• Teaching tools: Floating leaves guide first-time musicians to learn songs in “self tutorial” mode, and a browser-based composition tool helps teach you to compose.
• Global judging: Online audiences can judge songs with emoticons and text and a 1-10 scale. Everything is integrated with the app itself (see image). Tromboning with the Stars, anyone?
• The Power of Silliness: The most important feature, though, may be that everyone sounds a little goofy playing it, which can actually be liberating. As Dr. Ge Wang puts it, “It’s like singing in the shower.” Well, except with judges.

The app is simple, but the concept I think is pretty remarkable. We’ve seen interactive instruments, and we’ve seen music games. By adding the judging element, though, this is a free-form instrument that can also be a game. Now, without getting too ahead of ourselves, you could do the same thing with a Worldwide Online Kazoo Contest. In fact, maybe that’s a great idea. I suppose you could say music itself can be a kind of social game, played out on a stage. But nonetheless, making it an iPhone app can help free people up to get that message.

Oh, yeah, and got any doubts about the business model for open source? ChucK is a completely free, ridiculously powerful programming language for synthesis. It’s infinitely deeper than Leaf Trombone. But that power, packaged for a broad audience, can become a hit business – and likely to be popular well beyond musicians. If that’s possible, I imagine more could be soon. Remember, too, that whatever the Apple fanboys tell you, the iPhone is not a dominant mobile platform – nowhere close. Apple’s proprietary hardware means it isn’t really intended to be.

I hope someone working on platforms like Symbian and Google Android takes note: pack in geeky, nuclear-powered synthesis features, and people will find ways to put them to use in consumer apps that appeal to everyone. Leave them out, and you miss the boat. Or the trombone, anyway.

Direct iTunes App Link [99 cents]

http://smule.com/

Now, some very amusing videos of this thing in action:

Across series of colored bars, sounds warp and mutate. Vines entangle as organic threads of music. Fingers and objects traverse sonic landscapes in surprising, mysterious ways. Welcome to the worlds of BricK, the musical table interface by Jordan Hochenbaum and Owen Vallis, which, charged with software by Dimitri Diakopoulos, Jim Murphy, and Memo Akten, explores new musical frontiers. The tool uses a combination of open source tools for tracking fingers and objects on a table, then feeds those into sound and music environments.

Just following the landmark, long-awaited release of Processing 1.0, BricK demonstrates the expressive potential of the open-source platform. Processing allows quick and elegant development of stunning visual interfaces, while other tools (ChucK and Reaktor, for instance) serve as sonic engines. Sometimes the sounds themselves are not revolutionary, but by simply replacing the visuals and interaction – just as with changing the look of a score – the music is transformed, too. (At top: experiments with different interfaces for music using the platform they’ve built.)

CDM got to talk to Owen and Jordan about the projects. And now’s a perfect time – the gorgeous Roots is looking for a home, in case we have any curators / galleries / other interested parties in our audience. First, a review of what these platforms are:

Spaces, Multi-Touch Music

Spaces Multi-Touch Music Environment from BricK Table on Vimeo.

Jordan tells CDM about Spaces, their latest creation, which premiered alongside a performance by Daedelus in LA:

Spaces is the latest interactive multi-touch musical application for the Brick Table. Designed as a minimalist interface to free musicians from traditional compositional markers such as frets and keys, the environment enables musicians to compose intuitively through immediate visual and sonic feedback.

In this video, Spaces mediates a spontaneous composition and performance of a slow-moving ambient soundscape.

Spaces was developed by Jordan Hochenbaum, Owen Vallis, Dimitri Diakopoulos, and Jim Murphy.  It was recently used in a performance at the REDCAT lounge at the Walt Disney Theatre, Los Angeles, and further developments are currently underway.

Roots, an Organic Installation

Roots Multi Touch Tangible Installation Teaser from BricK Table on Vimeo.

Roots has been impressive in Web videos, but it’s looking to make the transition to the real world, after a shipping mishap prevented what was supposed to be its premiere showing at New York’s Minitek Festival earlier this fall:

“Roots” is an interactive installation for the Brick Table tangible and multi-touch interface, where multiple people can collaborate in making music in a dynamic & visually responsive environment.  Users use their fingers and tangible objects to create and interact with virtual branch-like vines that move around the screen, allowing users to create either entirely generative, semi-generative, or sudo-composed arrangements and compositions. 

Roots is truly a unique and expressive interactive installation which came together through an internet collaboration between Brick Table’s creators (Jordan Hochenbaum and Owen Vallis), and the super-talented London-based designer/developer Memo Akten.  It was recently selected as a featured Processing Exhibition on Processing.org and we feel it is time to release Roots into the wild…

So! We are calling out to all of you lovely CDM readers out there to get Roots out and into the public.

For more information on how Roots works, please see What is Roots?

Please use the contact on the BricK Table website if you are interested.

Behind the Scenes

CDM: How do the visuals relate to the sound?

BricK: The nature of the vine-like branches in Roots lends itself to creating music with what is — in our opinion — an organic and open feeling. We felt that the music should both sound and feel as if it is coming out of the visuals, and vice-versa, and so we did our best to stay true to this relationship in the overall musical aesthetic of the sounds produced.

The Spaces environment expands on the theme of unconventional visual representations of sound manipulation. Each column is an open space connecting an idea with a musical parameter. Combined with the visual feedback, we decided Spaces would work best with slow-moving ambient soundscapes, although it is certainly possible to experiment with other musical styles..

What sorts of relationships did you experiment with before settling on something you liked?

BricK: With Roots, we first worked with Memo to develop the visual elements before even attempting the musical side of things.  We discussed various approaches to its visual and musical relationships. Did we want it to be completely generative? Did we want a more direct and repeatable relationship between your finger and the resulting sound?  We really liked both ideas, and so we made it all inclusive– making it able to create completely generative, semi-generative, or directly manipulated/composed musical outcomes by the use of finger pressing, sliding, and tangible object interaction.  This really makes Roots unique in comparison to other environments which enable generative musical arrangements.  Each performer can exert as much or as little control over the relationship between physical, visual, and musical interaction as they want at any given moment.

In Spaces, we discussed a few different ideas about the layout and design of the interface. Ultimately, we decided on Spaces being able to control four different instruments, each with four parameters (volume, and three others). We toyed with different methods for visually representing the value of each column without turning them into a traditional slider. We felt the cool-to-hot color morph in each column was fitting: the user has to rely more specifically on the sonic result rather than exact value, veering from more traditional musical interface paradigms.

How did you deal with timing relative to the visuals?

BricK: In Roots, it was necessary to have the generative data play in a relatively synchronized manner to maintain a degree of musicality. As the vines move around the environment, the musical outcomes are quantized to various beats. [Ed.: The quantization all happens in ChucK.] That being said, continuous finger movement scrubs audio in a direct 1:1 relationship that gives the user the feeling of direct manipulation when that is wanted.

Spaces has no generative movement (at the moment) which means timing is always completely synchronous with finger movement, both sonically and visually. We tried to make sure that the way in which the colors morph feel as free and smooth as the slowly evolving musical outcome.

Can you talk a bit about how the sound is generated?

BricK: Roots uses audio buffers as its underlying sound source (although the musical outcome is VERY different than the original material). Each vine gets assigned an audio buffer which is then ’scrubbed’ through as the vine generatively maneuvers around the screen.  The audio and buffer manipulation is done using the ChucK audio programming language. By simply changing its source material, Roots will produce vastly differing musical results.

Spaces generates sounds in a number of different ways, all using Reaktor. Each of the four instruments employs a selection of synthesis methods. Some columns control pitch, other columns control combinations of filters and effects. The clicky percussive sounds are generated from an audio loop which is granulized and re-synthesized with altered delay rate, etc.

What are your future plans for these pieces?

BricK: Roots is ready to go, but in our free-time, Owen and I play with using it as a sequencing device in other ways — using movement and vine-location to pluck notes, control effects and filters, etc.

Exploring Roots along these other avenues will probably create the need for a new GUI interface, which means perhaps Roots will have a new little cousin sometime in the future.  That being said, we are really happy with Roots as is (we reached our specific goals), and we are more interested in giving it the proper debut it deserves, rather than changing the way it works.  We had a great time working with Memo, whose work I actually first came across here on CDM, and would love to work with him again in the future.

Spaces was developed in a very short timeframe for a performance at the REDCAT Lounge at the Walt Disney Concert Hall in downtown Los Angeles, and so we are absolutely looking to expand the possibility of what the Spaces interface is capable of. First, we would like to expand the number of instruments capable of being performed. Secondly, we would also like the interface to be “physics”-enabled, for example, using a flick motion to send a bouncing ball down a column to automate a parameter as the user concentrates on other instruments.

The Software

Just to review, here’s the software powering BricK:

tbeta (“The Beta”): finger tracking. tbeta is an open-source, cross-platform computer vision and multi-touch sensing platform. It’s the successor to the former touchlib, which wasn’t as cross-platform or quite as awesome. More on tbeta on Create Digital Motion.

reacTIVision: fiducial marker tracking for objects. (Fiducial markers are these funny, cellular-looking patterns pictured at right that allow you to track specific objects manipulated on the table. reacTIVision is the open-source library developed by the folks who did reactable. Sounds as though we might get fiducial tracking in the other library, though.)

ChucK: a strongly-timed, quick-to-code sound and synthesis language. It’s elegant enough that it’s used for real-time programming – as in, onstage, in laptop ensembles like PLOrk and (its West Coast descendent we just saw here on CDM) SLOrk.

Native Instruments Reaktor: The modular sequencer, instrument, and effect builder, which we cover regularly on our Kore minisite. It’s the only commercial / non-open-source choice here, though it may actually replace ChucK on Roots in the future.

More Info

Brick website

Created with Admarket’s flickrSLiDR.

The “Arduinome” gifts the Monome with two new possibilities. First, it’s a breakthrough on the availability front: at a time when official Monome kits are backordered, it makes it easier to make your own Monome on a budget. Second, it makes hardware hacking on the Monome far easier, by allowing people to make microcontroller-level modifications on the relatively friendly Arduino platform. (The Arduino was designed not for electrical engineers, but for artists just dipping their toes into electronics, even for the first time.)

Now, if you went out and cloned, say, the latest Roland keyboard, they’d understandably take issue. But part of what tells you this is a different kind of product is that the Monome creators have actually taken an active interest in the Arduinome’s development. Support won’t go through the Monome team, and there are still plenty of reasons to buy the real thing, but true to those open principles, both projects stand to benefit.

I got the scoop on the details of this project, what it’s about, and the gory details of how caffeine can fuel a massive electronics project even with relative newcomers to the craft. Jordan Hochenbaum, a leader of the project, talks to CDM (with some additional comments by his partner in crime, Owen Vallis. (Jordan and Owen, students at CalArts, also won CDM’s Futuristic Design Challenge earlier this year — video of that coming soon, I swear!)

(Skip about two minutes into this video for some side-scrolling LED action.)

Arduinome Nerdscroll Demonstration from BricK Table on Vimeo.

Why clone the Monome? What’s special about this project?
The Monome represents not only a controller interface, but also a new way of thinking about interface design. The very heart of the Monome concept is its minimal, open-ended form. This ideology is reflected in Monome’s decision to make the firmware and software open-source. Coupling these ideas with the strong community development support in the Monome forum, it became clear that the Monome was the perfect interface to try and port to the Arduino microcontroller. What makes this project special is Monome’s willingness to make their controller open-source. No other manufacturer would dream of letting people see how their stuff works, or letting a bunch of curious individuals try to build a clone.

What’s the relationship of Arduinome to Monome? What’s unique about the Arduinome?
We love the Monome, and the concept behind it. While the original Monome is difficult to come by, the Arduinome project was really born out of a desire to learn about physical computing and to better understand interface design. With this knowledge, we hope to be able to better use our Monome clones and develop new ways to push the buttons – er, envelope! We hope to build off the Arduinome in the future and explore new ideas.

What was the response from Brian and Kelly [Brian Crabtree and Kelly Cain, creators of the Monome] to the project?
Brian and Kelly have really provided encouragement towards our project. Actually, Brian was surprised the code wasn’t ported a while ago, especially since they’ve licensed their work for non-commercial use. Brian actually seemed really thrilled to see a bunch of people collaborate, which we think is just great!

What’s done, and what remains to be done? Where do you need contributors?

Completed:

• Finished porting the 40h firmware serial protocol
• Added additional support in Arduinomserial in order to address Arduino serial stability issues.

Still needs work:

• Implementing ADC and Encoder support.
• Testing for additional bugs.
• Porting the 256 serial protocols.
• Adding RGB support

Background – and Behind the Scenes

“Arduinome” is a collaborative project that aims to create a clone of the popular and inspiring Monome USB controller using the Arduino platform. The idea was originally brought up in a “Music Tech Forum” at the California Institute of the Arts. With about two weeks left in the semester, and very little experience, Owen Vallis and I put together a group buy and decided to just go for it. Everyone absolutely loved the original Monome, and we all figured this project would be a great introduction to the crazy world of physical computing.

Owen had already purchased the original Monome button pads, however the rest of us went for the Sparkfun button parts due to their availability and cost. After putting together a parts list, we began researching the 40h protocol and what it would take to port the firmware to the Arduino. The Arduino needed additional multiplexors in order to work with the matrix (specifically the MAX7219 and the 74HC165n/164n); fortunately, we found an amazing Arduino shield already designed by Brad Hill (unsped). After contacting Brad, he was kind enough to share his design. He asked that, in keeping with Monome’s business ideologies, the PCBs be printed locally and environmentally friendly. We were more than happy to do so; Monome’s business ideologies are very admirable, to say the least.

As soon as the parts were in, you could almost always expect to see somebody in the tech shop at CalArts soldering up their LEDs or diodes, etc. It was really amazing. As the semester’s end came near, Owen and I took on the responsibility of finishing the firmware. We started meeting every day or so at Silverlake coffee, feverishly trying out other people’s methods and ideas. (Thanks, Melka, Octinct, and others.) We had come pretty close, but always ended up having difficulty getting the buttons to work properly. After another night of caffeine-fueled failed tries, we came back to my house and decided to work for just a few more hours. At about 2:30am, we still didn’t have it working. We looked at each other, realized we wanted to call it a night, but in the traditional fashion of how Owen and I work, said “f@#!” it! Lets open up the 40h source, and work on porting it over to Arduino, as best as we can.” We were able to keep some things from the code we were working on, and at about 6 a.m. we got it working! With the sun barely rising through the LA smog, we went to get even more coffee and celebrate over breakfast.

Later that day, I made a section on the BrickTable blog detailing a short parts list, a little information, and of course the firmware. We posted the firmware link on the Monome forums. Immediately, there was a ton of interest, and people either seemed to be already in the middle of their own projects, or eager to start a new one. The response was fantastic, and quickly began to reveal the bugs in our early work.

Through the Monome forum, we’ve met and worked with a lot of great people. One person in particular has really pushed the project to where it is today. After two revisions on our website, Ben Southall made some major improvements to the firmware (addressing the Arduino Ports directly), and began working on “ArduinomeSerial”. It’s a MonomeSerial hack, if you will, which is optimized for the Arduinome. Ben realized that the USB serial chip on the Arduino differed from the one on the Monome, requiring changes to the original MonomeSerial in order to keep the serial connection happy. After testing, testing, and more testing, and a premature release, we now have a SourceForge project up with both the firmware and ArduinomeSerial. Anyone interested in delving into the source code, we have an SVN repository for version control, so feel free to dig in!

Guessing the project makers are fans of this site! Thanks!

We are really happy with the speed, response, and stability of both the firmware and ArduinomeSerial. It uses a specifically named protocol “a40h-xxx” vs. “m40h-xxx” like the regular 40h Monomes. However, ArduinomeSerial should be able to work with both regular Monomes as well as Arduinomes – thanks, Ben! I have modified Melka’s instructions on flashing the FTDI serial name on the Arduino to have ArduinomeSerial properly recognize your clone. This is posted on the Brick/Arduinome website (http://bricktable.wordpress.com/), and the process shouldn’t take more than a couple of minutes.
The next steps, of course, are to continue improving speed and reliability, as well as implement the ADC support. After that, there is the possibility of looking into the 256 series protocols as well as RGB support; however, these are both going to require a large rewrite, as well as hardware changes.
Monome is truly an amazing community; I’ve seen it firsthand working on this project. When Owen and I first posted the firmware, people started suggesting ways to improve it. Ben Southall took it to the next level and made crucial changes and additions to the firmware, in addition to creating ArduinomeSerial. We really feel that this kind of collaboration is amazing, and one of the great results of Internet collaboration and making ideas open-source. We can’t thank the Monome community enough for making all this possible, and of course Brian Crabtree, Joe Lake, and the rest of the Monome collective for creating an amazing and inspiring canvas. If you build your own, please send in details, pictures, bug reports, and the like.I know Owen, Ben, and myself would all love to see your work or find out about kinks we need to work out. We have a forum on the SourceForge page as well as a thread on the Monome forum. We believe in the Monome and the community ethic around it, so don’t be strangers!

The Arduinome Site contains a parts list, instructions for building your own Arduinome, and other documentation. We’ll have more on this project very soon!