RFID tags may have negative privacy associations when they’re used without someone’s knowledge. But embed these simple identifiers intentionally, and they can be a cheap, flexible way of tagging the world around you. Add OSC support with a free tool, and you can make anything into a basic music controller. That’s what Martin Kaltenbrunner – best known for his work on the ground-breaking ReacTable music table – has done with his own free software. It’s simple enough that you can easily make use of it, or take it as an opportunity to brush up on OSC and Pd.

This sort of odd, out-of-the-blue example is the perfect illustration of why OSC matters. Quietly, gradually, OSC is describing the world around computers in intelligent ways. In contrast to MIDI, with its resolution limits and arbitrary categories (vibrato rate?), OSC can standardize anything. What previously required advance standardization can now be truly open and even improvisational. The old way of standardizing: go in front of some sort of committee for approval. (RFID tags for music? Not likely.) The new way: go ahead and do the implementation, gather feedback, and if it works, other people will follow your specifications to ensure their stuff works with yours. In this case, Martin plans to add the RFID tagging to his TUIO2 protocol, which made what would have been just a cool one-off project (ReacTable) into a viral phenomenon of work with touch and tangible input. Martin writes:

I just released a new application, which intends to simplify the construction of tangible user interfaces based on RFID readers. Using this tool, the RFID add/remove events can be processed by any OSC
enabled application.

You can download the nfOSC tool from here:
http://code.google.com/p/nfosc/

The demo video shows the nfOSC application used together with the quite affordable touchatag RFID reader. A simple PD example patch receives OSC messages from nfOSC and starts a sample loop, when an RFID tag is detected by the reader device, the loop is stopped when the according tag is removed.

At the moment this tool just defines two simple ADD and REMOVE messages including the RFID tag IDs, but I am planning to integrate the tool into the future TUIO2 toolkit.

On its own, of course, it’s a simple hack, but I can imagine this having powerful implications when used in combination with another control method. And if you like the way the implementation works, you could use this same technique to apply to some other controller.

Software doesn’t have to mean virtualizing everything and letting go of physical objects. On the contrary, it can create all sots of imaginative, new ways of mapping musical ideas to the physical world. And that’s how we wind up with a walnut drum sequencer.

There’s something about virtual drum machines and snacks. We’ve seen bubblegum and Skittles, beer bottle caps, soda bottles, and now walnuts. Don’t stop now: someone has to do Cheetos, even if it means dealing with orange stuff all over your fingers.

That said, it’s not walnuts that make d-touch an important project. Built by Enrico Costanza back in 2003, the project is now available for free download as an open source library, a server (in case you don’t want to get into the C++ code but might want to use this in your own projects), a free, usable drum machine, and a set of documentation that can help you make your own stuff easily. Enrico worked on the original reacTable prototype and has done some really important work in this field. Right now, Enrico and co are looking for feedback, but if you’re ready to just be a tester and play with this – and see what you can do musically – now’s your chance.

d-touch also combines high levels of computer readability for accurate tracking with the ability to make your own tags. Instead of using ugly-looking glyphs, you can make patterns that make sense to human beings as well as computers. Oh, yeah – and mobile fans, this runs at a full 14 fps even on S60 phones.

For more, check out the d-touch site:
http://d-touch.org/ [Register first to make the download available]
and follow them on Twitter:
http://twitter.com/audiodtouch

Thanks to Martin (of reacTable, which is moving toward a commercial product) for sending this our way. Thanks, too, to Ben, who’s working on tangible interfaces with special needs students. I really look forward to hearing how that’s going.

Free software, a webcam, and some stickers printed on an inkjet can turn any object into a real-world controller. That’s what Paul Rose of Institut Fatima and his team did with a soccer ball (translation for the civilized world: football). The software is powered by the same framework used for the reacTable, but in this case there’s no table and no projector: just a ball.

Institut FATIMA uses a Fussball as (des-)controller for triggering drumsamples. The camera detects the symbols on the ball, kicks numbers into the sequencer, the sequencer matches goals. The goal is always music. Software used is reactivision and ableton live. Do it at home.

As it happens, reacTIVision just got a significant update, with more improvements planned. You can read up on the full details on Create Digital Motion:

Free Tangible Tracking: reacTIVision 1.4 Here, TUIO2 Coming Soon

Martin Kaltenbrunner, co-creator of the framework (and the reacTable), has some tips for working with tangible interfaces and music, and where to find more inspiration.

In addition to TUIO, reacTIVision also has an alternative MIDI mode, where you can map the appearance of fiducial symbols to note ON and OFF events, as well as their X,Y and rotation angle to a control channel value. Quite a few people have been using this for the creation of cheap web-cam based MIDI controllers.

http://www.youtube.com/results?search_query=reactivision+midi

Using TUIO, you have more alternatives though, you can currently use Max/MSP, Pure Data, Quartz Composer, Processing, Java, C++, C# and so on to receive the object & finger tracking data. Here are a few cool musical projects, that have been built using reacTIVision:

http://modin.yuri.at/tangibles/?list=7

Patrick H. Lauke (patch pictured, from Flickr) has a video on YouTube that shows some of the basic workflow for combining the free patching environment Pd with TUIO and reacTIVision. He cautions:

this may not be pleasant from a musical point of view, but it only serves as a first test for further experimentation.

Hopefully this gives folks some ammunition if you’re getting involved in the tangible interface hackday! [Project site | on CDMu]

TUIO is a simple but powerful emerging protocol for multitouch control for live music and visuals, as used on the powerful live tangible synth reacTable. Apparently no one told Apple, however. While the App Store rubber-stamps useless toys like fake cigarette lighter flames, they bizarrely rejected a powerful application by a leading digital artist that would enable standardized TUIO control – for free. (More back story below; see an example in action above.)

As a blogger, my reaction is usually to whine and pontificate, for better or worse. The engineering approach would be to find some hack away the problem. That’s what Andrew Turley did with the TUIO protocol. So, Apple won’t allow an app that does the trick? Why not go back to what developers did before the SDK, and just use the iPhone browser?

As Andrew explains it:

After reading the story I started thinking about seeing how far one could push Safari as an application platform, using web apps to get around Apple’s tight control of the app store. Since you would be connecting to another computer anyway to use an OSC application, why not just have the app be a web app running on a web server somewhere on the local network? The web server can then take care of things like sending out OSC messages or playing music or doing whatever it is people want to do.

To that end I created a little system that implements the TUIO protocol. You use an iPhone to run a web app, which in turn talks to the web server, which in turn sends OSC messages.

Details:

touchy feely [Pillowsopher Blog]

Needless to say, there are some downsides: you have to run the Python server, you’re more limited in input and control than you would be in a real app, and you’re stuck inside the Safari browser, which could be a bit inconvenient. So I’m not backing down from my original complaint – I don’t see Apple doing anyone a service by blocking this kind of app, and the only rational explanation seems to be that the folks doing the review process don’t understand what the app is.

Most importantly, I do know that all our griping was sent to Apple, so I’m hopeful the App Store will reconsider the decision once they get it.

But I love solutions, too, and this can be one for many applications.

It also illustrates an important point: the browser on mobile devices (Apple and otherwise) could be a powerful outlet. It doesn’t always make sense to build an entire application; there will be various cases in which a little browser tool will do a job. Need a quick remote control for a live performance / art installation / club lighting rig? You might try the easy solution with the browser first.

Thanks for the great hack, Andy, and I’m curious to hear if anyone uses or extends this.

Previously:

Apple Rejects Free iPhone Tool For Artists Because of “Minimal User Functionality”

Imagine friendly creation of custom synths and sounds by dragging visual nodes. Now imagine you can do that on a mobile device and your computer – and eventually combine the two. That’s the vision of Jasuto, and while it’s not quite there yet, it’s incredibly promising.

The laws of combinatorics predict that, on a regular basis, you’ll see countless soft synths that are slight variations of one another. With the iPhone/iPod touch gold rush in full swing, we’re starting to see the pattern repeat itself, just as it did in Windows and Mac plug-ins. Some are brilliant; others are just the usual variations on a theme.

Of course, even better is the ability to build exactly what you want out of the same buildings blocks. Powerful toolkits like Max/MSP, Pd, Reaktor, SuperCollider, SynthMaker and the like let you do this, but they qualify as the more-sophisticated Erector Set of synthesis. Sometimes you just want some simple, LEGO-style building blocks that cover the basics.

That’s why Jasuto looks so promising. It’s actually two pieces of software – a plug-in for Mac and Windows VST. Combine basic modules, and you get some powerful features, even on the iPhone:

• Multiple synthesis methods: additive, subtractive, FM/PM/AM, hard-sync synthesis
• Basic math functions: add/subtract/delta/constant values
• Filters: LP, HP, BP, and a Moog emulation
• Signal tools: envelopes, dynamics processors, and an envelope follower
• Effects: Delay, reverb, saturation, digital distortion
• iPhone hardware features: Accelerator and mic access (and of course mic access on your computer, too)
• 16-step sequencer with looping, pattern manipulation
• Performance options: a tappable keyboard, glide functions, and so on

That’s just the specs, though. To me, the most interesting thing is the zoomable, nodal design, reminiscent of the reactable. There’s also the ability to record “motion” anywhere in the app, and to modulate everything with everything else. And I especially like the idea that “patches” and “synths” are one and the same: just as on an early modular, creating a new “patch” really is about connecting modules into something unique.

I can’t imagine it being the last computer instrument you need, but if successful, it could well be the last iPhone synth you need. I like the idea of

It’s under “heavy development,” so expect some bugs. The software can be yours for all of US$1.00 on iPhone. The PC/Mac VST looks a little rougher, but it’s available for free download.

Jasuto Site

VST Plug-in

iTunes Link

Thanks to sublamp for turning us on to this via comments.

Ever feel like you’ve found the seam dividing past and future?

The past: restrictive UI frameworks requiring pages and pages of code to produce dated-look 2D displays. Proprietary software with rigid interfaces. Input bottlenecked through the x and y coordinates of a single mouse pointer.

The future: UIs whipped together graphically or with a few lines of code. 3D mixed with 2D. Open-source, friendly frameworks. Creating your own interface or drawing upon a community of creative software makers. Input that uses multitouch for gestures, collaborative input, manipulation of 2D and 3D space, and … well, just a lot more fun.

There’s no need to wait around for the future. Creative software inventors are building it for themselves. Here are two of the most promising multitouch interface projects I’ve seen in my inbox.

In no time at all, you’ll be painting a cow! (Okay, more on that in a moment…)

Make Max Multitouch

Max Multitouch Framework by composer Mathieu Chamagne makes turning your Max patch into a multitouch interface a breeze. When I first reviewed the Lemur, I was frustrated by the hardware-style abstraction between your software and the interface. Why was I having to go through Max patches painstakingly assigning Lemur controls to Max controls – why not just make the Max controls appear on the multitouch screen? Well, that’s exactly what you get with MMF. Using a set of Max abstractions, all you have to do is build your Presentation Mode style UI and add in the MMF ingredients – it automagically becomes touchable on a variety of displays.

It’s not hard to imagine how great this could be for musicians, especially those who have already been building original sonic creations in Max/MSP. Best of all, you don’t need an expensive, non-portable table with a projector inside, either – commodity hardware works just fine right now.

Requirements :

- cycling’74 Max5 (version 5.0.6 required)
- any multitouch interface that sends TUIO messages.

MMF has been tested and works fine with : Stantum SMK-15.4 multitouch screen, HP XT2 tablet pc (windows7 + W2TUIO), Demand Evolution + home made multitouch screen + Reactivision / Tbeta , …

MMF (Max Multitouch Framework) @ Mathieu Chamagne’s Site

Hmmm… apps that send (cough) TUIO, eh? Ah, yes, but that’s why you need companies like Apple to tell you what qualifies as useful in an iPhone app. You see, without Apple’s app review team and their superior wisdom, I might wrongfully assume this sort of app would be something I’d want. Now I know better – thanks! (Sorry, couldn’t resist.)

Back to the on-topic discussion, this does demonstrate a real advantage of Max: it has its UI absolutely nailed, and the open-source alternative Pd is woefully behind. It also demonstrates that the beauty of Presentation Mode is, by abstracting the UI from the underlying guts, you can consider alternative interfaces. We should see that in Max for Live, as well.

Pd is fantastic in other ways, but if there’s anyone out there who fancies writing a lightweight new front-end, it could use it – perhaps in Python. Which brings us to the next item.

pymt demo reel from Thomas Hansen on Vimeo.

PyMT: Juicy Multitouch, Just a Few Lines o’ Code

PyMT is a glimpse of what future development could look like. While Microsoft putzes around with their Windows-only Surface, PyMT makes multitouch platform agnostic, open, and easy. That frees up artists to dream up creative new ways of applying this interface to expressive musical and visual creations (among others). Instead of reinventing the wheel as far as plumbing, you can focus on the reason for using devices in the first place – your art.

PyMT is profoundly portable, using Python and OpenGL to deliver windowing and multimedia features across platforms. Documentation and code are both under heavy development, but there are already some friendly-looking resources. This is almost enough to shake me from my loyalty to Java, though, in fairness, you can do some of the same things with Java and other tools. What’s most important is that there are libraries that are providing standards, like TUIO, and implementations in cross-platform languages that can be easily developed.

There’s good reason to be bullish on the future of this tech. And if you want to see it happen, don’t wait – you can get involved in the project directly.

PyMT at Google Code

PyMT Project Page

Since the dawn of computing, developers have been free to create whatever software they can imagine for computers. Windows, Mac, UNIX, Linux, Atari, Amiga, Apple II, Commodore 64 – it doesn’t matter. Come up with an idea, and short of doing something destructive on the system, you can make it work on a computer. It’s this freedom that has made the computer age possible. Game consoles have been different, a relic of the days when cartridges were physical objects you put in the machine. But mobile devices have generally acted more or less like computer platforms – look at Windows Mobile, Symbian, Linux, Android, Palm OS, Palm’s Web OS, and so on. It wouldn’t be odd to expect the same of the iPhone or iPod touch, which is essentially a Mac running on a low-power platform with a mobile-optimized set of libraries. The iPod doesn’t even connect to a wireless phone network; it uses WiFi just like your computer.

As musicians and artists, this sort of freedom has given us the freedom to make expressive music and art using powerful tools. That same freedom hasn’t applied to comparatively restrictive game platforms, which is why music apps for platforms like PSP and Nintendo DS require hacking hardware and software.

But then there’s the iPhone / iPod touch. Apple claims that they create a superior user experience by controlling quality, and they use that control to pick and choose which applications they think are appropriate for their phone. Never mind that a whole lot of what’s available on the iTunes store is simply worthless crap. And, frankly, that’s okay – users pick and choose the good stuff, and a lot of it’s really great.

But far from simply protecting mobile carriers like AT&T from abusive apps, it’s clear from developer experiences that Apple has extended that supposedly superior judgment to second-guessing developers on design and application purpose.

The latest victim: a fully free wireless multitouch server that would empower iPod touch and iPhone users to control live art and perform, created by one of the world’s leading interactive artists, Memo Akten. It seems Apple’s powers that be rejected the app because they simply don’t understand what the heck it is.

The story so far:

I received the following response from Apple: “Thank you for submitting your application “MSA Remote”. We need clarification in regards to the functionality of the application as we do not have access to the interactive installations that the application controls. If possible, please provide login information for a server application we can use to review your application.”

So I sent them 3 desktop apps: a standard TUIO client, a MSATouch client (multiple devices can control a single client without interference), and an OSC Dumper so they could see all the messages being sent in detail.

Even after receiving that documentation, Apple decided that they knew better than their users and developers. Result: no app.

Apple’s response to this app is “We’ve reviewed your application, MSA Remote, and we have determined that this application contains minimal user functionality and will not be appropriate for the App Store.”

I find it hard to believe that a FREE TUIO SERVER for iPhone and iPod touch is not suitable for the App Store!! Please leave comments below if you think it is suitable and I will get back to them to try and sort it out.

MSA Remote for iPhone

TUIO, for the record, is the open multitouch platform employed in the powerful, expressive reacTable tangible interface that has been used by the likes of (Mac user) Bjork.

Let’s be clear. The point here is not just to rant. Memo is looking for comments from users. I’m hopeful that an upswell of support could show Apple the error of their ways and get them to correct course on this one, and I’ll applaud them if they do that.

But let’s also be clear: restrictive platforms are bad for artists. Apple is setting a dangerous precedent, and I’m frankly tired of the conventional assumption that they’re always right. I think the restrictiveness of the platform – well beyond what is “safe” for users or what might endanger Apple’s relationship with carriers like AT&T – is simply wrong-headed. The reason we love platforms like the Mac is that they have empowered us to express ourselves freely. And having seen the power of the Mac as a platform over the years, I’d be disingenuous not to point out that the iPhone has lost a big part of that soul.

The good news is, complaining and whining and griping can be productive. Tweetie, a powerful Twitter app that was censored because it might expose users to profanity on the open Web service, did make its way to the store after massive public outcry.

So, as I say, I don’t rant just because I like the sound of my own typing. I hope that the rest of you will join in, and it’ll make a difference.

Want to interact with your computer musically using physical objects and other fancy-schmancy, science-fiction-y interfaces? Don’t want to rely on Microsoft or wait until 2019? You’re in luck. It’s like Christmas for DIYers and interactive futurists.

Enter LusidOSC, a set of protocols, libraries, and useful code, and Trackmate, a clever and cheap-to-build system for tangible interfaces. Together, you’ve just got a bunch of tools to help you start playing with blocks — erm, experiencing new spatial interfaces.

• LusidOSC, the library maps “spatial input devices” – really, any tangible devices or sensors in the real world – to applications, like live music or live visuals. It uses the network-savvy OpenSoundControl protocol. The library is available now for Processing, with Java, Python, Flash, and C++ in the works.
• LusidOSC, the apps/code: a set of Processing utilities to get you started includes a basic example app, command launcher, playlist mixer, and a MIDI sequencer. You can build on these for your own Processing apps.
• Your own input device: Tags? Sensors? Markers? Regions? Fingertips? If it’s in physical space, you can map it via LusidOSC. Or, if you want a place to start, try:
• Trackmate: A project of MIT’s Tangible Media Group, Trackmate lets you affix tags to any objects you want and use them as an interface.

LusidOSC is just the protocol; it’s physical applications like Trackmate that get really interesting. In these economically-challenged times, Trackmate gives you tangible interfaces for next to nothing. All you need is a computer (Mac, Windows, or Linux), and a camera (even a webcam will do). Print out randomly-generated tags in the free and open source software, stick them on stuff you want to use, and go to town.

You can make your own Trackmate surfaces out of hardwood or plexiglass for as little as $40 in parts. There’s even a Processing-based simulator app so you can experiment without the physical objects. (Good for troubleshooting on the bus or plane, I imagine.)

Trackmate is just one project, though; LusidOSC promises to support other interface ideas, too.

LusidOSC at Sourceforge, the work of “many research institutions and companies.” (Which, I wonder?)
Trackmate at Sourceforge, a project of the MIT Media Lab Tangible Media Group

Alternatives: TUIO, the protocol used in the Bjork-endorsed Reactable project marches forward, as well. And as both projects are open source, there could be some cross-pollination. I hear a revised TUIO is coming, and in the meantime, LusidOSC acts as a gateway to TUIO.

The idea of the Reactable is to make music tangible, with control of sound mapped to physical objects you move around on a table. But that hasn’t stopped the Psychosynth from creating a virtual version. (Upside: it’s a lot more portable.)

Psychosynth

Watch the video, but they seem to have made the opening minutes as dull as possible to thin out the non-believers. Skip past the generation of the white noise oscillator (wow, white noise!), and somewhere around halfway through, it becomes laugh-out-loud funny, with trance-style vocals about freeing your mind with free software. (Seriously – it’s awesome.)

While it’s in alpha stage, the software is fully free and open source (binaries available for Debian and OpenSUSE) – and that means this could be a good project to snoop around in or code through, those of you who are eager hackers.

At its heart, it’s simply an interactive modular synth inspired by Reactable. Drag objects around and connect them to make sounds and patterns. Underneath is a powerful C++ synth library, a 3D synth, and even a server version – so even if you aren’t sold on this interface, there are pieces here that could be useful. It’s all virtual now, but that’s not to say you couldn’t add input; support for the input library is planned later.

That said, I don’t think they went far enough with the virtual thing. Next, why not simulate virtual players for the virtual Reactable inside the computer. They could even behave like Sims, requiring regular stimulation and bathroom breaks. Eventually, you could unlock Bjork.

Enjoy. If anyone gets this up and running even in alpha state, let us know.

Previously: tables and tangible, like the lovely acoustic sounds of Etiquette and, of course, Spaces / Roots

Across series of colored bars, sounds warp and mutate. Vines entangle as organic threads of music. Fingers and objects traverse sonic landscapes in surprising, mysterious ways. Welcome to the worlds of BricK, the musical table interface by Jordan Hochenbaum and Owen Vallis, which, charged with software by Dimitri Diakopoulos, Jim Murphy, and Memo Akten, explores new musical frontiers. The tool uses a combination of open source tools for tracking fingers and objects on a table, then feeds those into sound and music environments.

Just following the landmark, long-awaited release of Processing 1.0, BricK demonstrates the expressive potential of the open-source platform. Processing allows quick and elegant development of stunning visual interfaces, while other tools (ChucK and Reaktor, for instance) serve as sonic engines. Sometimes the sounds themselves are not revolutionary, but by simply replacing the visuals and interaction – just as with changing the look of a score – the music is transformed, too. (At top: experiments with different interfaces for music using the platform they’ve built.)

CDM got to talk to Owen and Jordan about the projects. And now’s a perfect time – the gorgeous Roots is looking for a home, in case we have any curators / galleries / other interested parties in our audience. First, a review of what these platforms are:

Spaces, Multi-Touch Music

Spaces Multi-Touch Music Environment from BricK Table on Vimeo.

Jordan tells CDM about Spaces, their latest creation, which premiered alongside a performance by Daedelus in LA:

Spaces is the latest interactive multi-touch musical application for the Brick Table. Designed as a minimalist interface to free musicians from traditional compositional markers such as frets and keys, the environment enables musicians to compose intuitively through immediate visual and sonic feedback.

In this video, Spaces mediates a spontaneous composition and performance of a slow-moving ambient soundscape.

Spaces was developed by Jordan Hochenbaum, Owen Vallis, Dimitri Diakopoulos, and Jim Murphy.  It was recently used in a performance at the REDCAT lounge at the Walt Disney Theatre, Los Angeles, and further developments are currently underway.

Roots, an Organic Installation

Roots Multi Touch Tangible Installation Teaser from BricK Table on Vimeo.

Roots has been impressive in Web videos, but it’s looking to make the transition to the real world, after a shipping mishap prevented what was supposed to be its premiere showing at New York’s Minitek Festival earlier this fall:

“Roots” is an interactive installation for the Brick Table tangible and multi-touch interface, where multiple people can collaborate in making music in a dynamic & visually responsive environment.  Users use their fingers and tangible objects to create and interact with virtual branch-like vines that move around the screen, allowing users to create either entirely generative, semi-generative, or sudo-composed arrangements and compositions. 

Roots is truly a unique and expressive interactive installation which came together through an internet collaboration between Brick Table’s creators (Jordan Hochenbaum and Owen Vallis), and the super-talented London-based designer/developer Memo Akten.  It was recently selected as a featured Processing Exhibition on Processing.org and we feel it is time to release Roots into the wild…

So! We are calling out to all of you lovely CDM readers out there to get Roots out and into the public.

For more information on how Roots works, please see What is Roots?

Please use the contact on the BricK Table website if you are interested.

Behind the Scenes

CDM: How do the visuals relate to the sound?

BricK: The nature of the vine-like branches in Roots lends itself to creating music with what is — in our opinion — an organic and open feeling. We felt that the music should both sound and feel as if it is coming out of the visuals, and vice-versa, and so we did our best to stay true to this relationship in the overall musical aesthetic of the sounds produced.

The Spaces environment expands on the theme of unconventional visual representations of sound manipulation. Each column is an open space connecting an idea with a musical parameter. Combined with the visual feedback, we decided Spaces would work best with slow-moving ambient soundscapes, although it is certainly possible to experiment with other musical styles..

What sorts of relationships did you experiment with before settling on something you liked?

BricK: With Roots, we first worked with Memo to develop the visual elements before even attempting the musical side of things.  We discussed various approaches to its visual and musical relationships. Did we want it to be completely generative? Did we want a more direct and repeatable relationship between your finger and the resulting sound?  We really liked both ideas, and so we made it all inclusive– making it able to create completely generative, semi-generative, or directly manipulated/composed musical outcomes by the use of finger pressing, sliding, and tangible object interaction.  This really makes Roots unique in comparison to other environments which enable generative musical arrangements.  Each performer can exert as much or as little control over the relationship between physical, visual, and musical interaction as they want at any given moment.

In Spaces, we discussed a few different ideas about the layout and design of the interface. Ultimately, we decided on Spaces being able to control four different instruments, each with four parameters (volume, and three others). We toyed with different methods for visually representing the value of each column without turning them into a traditional slider. We felt the cool-to-hot color morph in each column was fitting: the user has to rely more specifically on the sonic result rather than exact value, veering from more traditional musical interface paradigms.

How did you deal with timing relative to the visuals?

BricK: In Roots, it was necessary to have the generative data play in a relatively synchronized manner to maintain a degree of musicality. As the vines move around the environment, the musical outcomes are quantized to various beats. [Ed.: The quantization all happens in ChucK.] That being said, continuous finger movement scrubs audio in a direct 1:1 relationship that gives the user the feeling of direct manipulation when that is wanted.

Spaces has no generative movement (at the moment) which means timing is always completely synchronous with finger movement, both sonically and visually. We tried to make sure that the way in which the colors morph feel as free and smooth as the slowly evolving musical outcome.

Can you talk a bit about how the sound is generated?

BricK: Roots uses audio buffers as its underlying sound source (although the musical outcome is VERY different than the original material). Each vine gets assigned an audio buffer which is then ’scrubbed’ through as the vine generatively maneuvers around the screen.  The audio and buffer manipulation is done using the ChucK audio programming language. By simply changing its source material, Roots will produce vastly differing musical results.

Spaces generates sounds in a number of different ways, all using Reaktor. Each of the four instruments employs a selection of synthesis methods. Some columns control pitch, other columns control combinations of filters and effects. The clicky percussive sounds are generated from an audio loop which is granulized and re-synthesized with altered delay rate, etc.

What are your future plans for these pieces?

BricK: Roots is ready to go, but in our free-time, Owen and I play with using it as a sequencing device in other ways — using movement and vine-location to pluck notes, control effects and filters, etc.

Exploring Roots along these other avenues will probably create the need for a new GUI interface, which means perhaps Roots will have a new little cousin sometime in the future.  That being said, we are really happy with Roots as is (we reached our specific goals), and we are more interested in giving it the proper debut it deserves, rather than changing the way it works.  We had a great time working with Memo, whose work I actually first came across here on CDM, and would love to work with him again in the future.

Spaces was developed in a very short timeframe for a performance at the REDCAT Lounge at the Walt Disney Concert Hall in downtown Los Angeles, and so we are absolutely looking to expand the possibility of what the Spaces interface is capable of. First, we would like to expand the number of instruments capable of being performed. Secondly, we would also like the interface to be “physics”-enabled, for example, using a flick motion to send a bouncing ball down a column to automate a parameter as the user concentrates on other instruments.

The Software

Just to review, here’s the software powering BricK:

tbeta (“The Beta”): finger tracking. tbeta is an open-source, cross-platform computer vision and multi-touch sensing platform. It’s the successor to the former touchlib, which wasn’t as cross-platform or quite as awesome. More on tbeta on Create Digital Motion.

reacTIVision: fiducial marker tracking for objects. (Fiducial markers are these funny, cellular-looking patterns pictured at right that allow you to track specific objects manipulated on the table. reacTIVision is the open-source library developed by the folks who did reactable. Sounds as though we might get fiducial tracking in the other library, though.)

ChucK: a strongly-timed, quick-to-code sound and synthesis language. It’s elegant enough that it’s used for real-time programming – as in, onstage, in laptop ensembles like PLOrk and (its West Coast descendent we just saw here on CDM) SLOrk.

Native Instruments Reaktor: The modular sequencer, instrument, and effect builder, which we cover regularly on our Kore minisite. It’s the only commercial / non-open-source choice here, though it may actually replace ChucK on Roots in the future.

More Info

Brick website