Here’s how interfaces normally break down. You’ve got your conventional, tactile interfaces, like a knob. You’ve got your touch interfaces, which lack tactile feedback (you touch them, but they don’t push back). You’ve got your gestural interfaces, which have you waving your hands in the air without touching anything and without any tactile feedback. (They’re generally the most challenging, because your brain has no feedback for what it’s doing.)

Syntact creates an entirely new category. It’s a gestural interface, of the “waving your hands around in the air” sort. But while your hand is in mid-air and isn’t touching anything, it does provide tactile feedback. It pushes back as you move your hand around, giving you interactive feedback. The way it pulls it off: sound. 121 ultrasonic transducers beam sound at a particular point, so that you feel something as you move.

You can see a bit of what this means in the new video, above. I’m hoping to get a hands-on (erm, hands-off) demo soon from the designer. The basic specs:

• Optical analysis of gestures, using a USB camera built into the interface
• MIDI control, for use with any live performance or music making rig (or other media)
• A control panel for selecting different sonic images and adjusting scaling.
• A built-in music solution visualizes sound and makes it easier to map to your own MIDI files.

More information:
http://www.ultrasonic-audio.com/products/syntact.html

Also well worth checking out the directional speaker tech from these Slovenia-based developers – directional sound is another huge area of innovation.
http://www.ultrasonic-audio.com/products/acouspade.html

If you want to try this in person, it’ll be at the Beam Festival in London in late June.

Side note: Yes, I’m looking into that LEAP thing, for more gestures, albeit without tactile feedback. Stay tuned.

Dig into humanity’s past, and alongside the earliest tools, you’ll find some of the earliest instruments. Designing objects for expression seems to be an essential part of civilization.

Martin Kaltenbrunner, a co-designer of the Reactable tangible music interface, is also a professor in Interface Culture at the Linz University of Arts in Austria. There, in the land of Mozart and Haydn, he works with students to explore what interface design is.

So, when I got to spend some time with Martin in New York in September, I was interested in more than just the flashy coolness of the Reactable, the futuristic table-with-blocks interface for music. We got a chance to talk about instrument design generally. The funny thing about the Reactable is that it is closer to the experience of working with a modular synthesizer and oscilloscope than anything else, with the sense of physical connections of sound to object you’d get from classic synths. It is something unique, truly, but that’s its pedigree.

Martin and I got to give a talk together at the Austrian Cultural Forum New York, a terrific hub in which Austrian artists frequently are paired with New York-based folks, all in a lean, tall modern landmark building in Midtown. We also performed together, which for me was a real pleasure; Martin claims not to be a musician as such, but was good fun as an improvisation partner.

The next day, we headed to Manhattan music education center Dubspot, where Martin’s creation quickly attracted crowds of interested students and educators. Dubspot filmed our encounter for the video at top. Amusingly, the prominent synth sounds you hear at the beginning are not the Reactable, but our own MeeBlip open source synth, which I brought along to illustrate conventional tangible instrument design with switches and knobs. (If you’ve been impatiently waiting for news on the MeeBlip, believe me, I’m even more impatient – more announcements on that this week and next, following a production quality issue with a contractor that required us to reboot the run of new instruments.)

Reactable Live @ Dubspot! Interactive Sound Design Workshop Video Recap [Dubspot Blog]

YouTube commenters, that subtle and thoughtful bunch, are complaining that the tangible Reactable will set you back thousands of Euros. But at ACF and Dubspot, I was also equipped with the far more economical and portable alternative: Reactable Mobile runs on both Android and iOS. (I was pleasantly surprised to discover the app runs perfectly on a Galaxy Tab 10.1 from Samsung; I’m still a long, long way from being able to recommend buying an Android tablet, but if you’ve got one, I can certainly recommend this app.) Now, don’t get me wrong: the experience was nowhere near as fun as using the table. On the other hand, you can’t fit the table into a seat-back pocket on easyJet, and the savings in cash is proportional to the sacrifice in experience. What impresses me is that the design of the physical Reactable “flattens” so nicely onto the screen; I think it’s a user experience triumph that you can make that translation. And I was able to load up a few loops of my own music and jam along with the MeeBlip and Martin on the (real) Reactable.

Looking beyond the Reactable, Martin addressed these larger issues of tangible interface design at TEDx Vienna – a fitting locale for talking the history and future of music. His whole presentation, and a sweeping concept map of what he discussed, is available.

Martin Kaltenbrunner: Tangible Music

If you just want to get your Reactable on and can’t afford the table, see Reactable Mobile below. (Seen here on an iPad 2, but I’ve run successfully on the original iPad and the Galaxy Tab – the experience is more or less identical, thanks to portable code.)

And if you can afford the table, you rockstar, uh, can we be your friend?

http://www.reactable.com/products/mobile/
[the awesome table version]
Martin Kaltenbrunner website [with plenty of academic links]

A new design launching this week should appeal to keyboardists who want both more expressive touch control and a keyboard – without sacrificing one or the other.

Yes, yes, multi-touch on tablets does indeed give your fingers access to continuous control for added expression and pitch. But there’s a reason keyboards evolved keys: tangible feedback about where pitches are, and the ability to control dynamics with pressure (itself with additional mechanical tangible feedback) just isn’t matched by touchscreens.

We’ll be looking on an ongoing basis at how you can take the flexibility of those touchscreens and match them with more tangible controls. But here’s one example: the German-engineered Evo keyboard really is a conventional keyboard, with all the advantages therein, but combined with capacitive touch on every single key. In other words, it navigates around the very tradeoffs of which I was recently critical in iPad developments, namely, additional expression coming at the expense of tactile feel. (I got some pretty intense criticism for things I said in that article which remain, to me, fairly obvious: a tablet is not a device you can play with your eyes closed, and – in its present form – no matter how hard you hit it, you can’t control dynamics.)

Of course, this does require buying specialized hardware, and it’s a controller only – unlike that tablet, you’ll still need a sound source and (at least for some tasks) a display, both of which are integrated in the tablet. But it is a compelling alternative that introduces a different set of possibilities for playability.

In fact, it’s also not the first time designers have thought in this direction. All the way back to the Martenot, keyboard designers have looked for ways to bend keys or add additional continuous expression – polyphonic aftertouch being the most common (though still relatively rare) solution. But none of those inventions could build on the accessibility of touch on the keys. I’m curious to see what playing this feels like; fans of getting away from the piano keyboard and all its history entirely will likely (and fairly) scoff, but for those of us who want to merge our piano background, something like this merits consideration.

Here’s how the creator describes it; I hope to catch up with this invention soon in person.

Discover the evo. Worlds first keyboard with touch sensitive keys.

With the help of capacitive touch sensors the evo is able to read your fingers movement on top of a keys surface. Next to pitch and velocity there is now a third layer of polyphonic data input.

Think of polyphonic control of pitch and expression. Think of having a pitch or mod wheel integrated into every single key. Think of never ever leaving a key just to turn or push some knobs or buttons.

So, it’s a combination of classic keyboard key and modern touch technology. The best of both worlds combined in a single keyboard. But best of all! The characteristical function of a key remains unchanged. The evo still features traditional pressure-sensitive keys. But in addition there are now all the advantages from a touch-sensitive input device in every single key.

So. At first this might sound like the evo is an all new instrument. But it’s nothing more than a traditional master-keyboard… With capacitive touch inside every key.

http://www.endeavour.de/discover_the_evo/evo.html

Moving beyond touching a screen as two-dimensional plane, Roger Linn’s concept music controller, the Linnstrument, adds tactile response and expression. Roger calls it “3D Note Expression,” but in lay terms, it means pushing harder on the controller makes it respond differently, as you’d expect from a physical instrument.

Roger this week posts an update on how his development is going and what he imagines – good timing, as this week we also saw another design on the same lines, the Soundplane. The sensing methods are different, enough so that I can easily see room for both, but the upshot is the same. Randy Jones in that story also reflects that, once these things are invented, what will really be essential is musicians to play the things and develop lots of interesting software that can use the controllers. See, previously:

A Glimpse of the Soundplane Controller, Innovative Tactile Multi-Touch, in the Lab; Call to Action

The major change in Roger’s latest update, apart from adjusting the form factor to something longer and more spacious, is the addition of different overlays, including traditional keyboard “manuals,” fret-style grids, and honeycomb-patterns hexagonal grids like the one at top.

It isn’t just a blank slate any more – certainly not in Roger’s mind, at least. Linn, a guitarist by training, has an extensive schema worked out for a grid that would function like the frets on a guitar neck. Movements in any direction can make an adjustment, impacting timbre (perpendicular to the front edge), pitch bends (parallel to that edge), and loudness (pressure).

It’s still all concept, with one working prototype; Roger’s professed style is to work in big-picture concepts. But I’d love to see some of these ideas reach fruition.

I’m still hopeful that a simple, inexpensive, open source option could evolve from this landscape. I hope for that not necessarily even because of a philosophical belief in open source so much as the sense that such an open field could lend itself to experimentation by, say, the people who read this site, not only in creating software but building the instruments. In the case of Roger’s design, unlike the more novel approach used by Randy, patent questions are less of an issue. And a community of people experimenting in such a way could simultaneously yield software that could be compatible with the proprietary and commercial projects.

Lots more ideas and reflections – including detailed notions of how you’d play this thing – on Roger’s site:
Preview: LinnStrument – A New Musical Instrument with 3D Note Expression

And here’s a new video, entitled, cheerily, “Gloomy Sunday” (the music of Rezső Seress):

For review, Roger’s original demo video:

David Merrill, working with Jeevan Kalanithi and (for the audio engine) Josh Kopin, wowed audiences at the TED conference with his Siftables interactive blocks. These strike me as what the Audiocubes have tried, sometimes unsuccessfully, to be — physical objects that react to the proximity of other objects, allowing you to manipulate music and media by moving around tangible blocks. Siftables are gifted with multiple expressive controls (tilt helping them break the plane of the surface), and intelligent screens that make them more adaptable and provide more visual feedback.

The music sequencer is very cool, though I think it’s actually the Scrabble-like game that may be the winner among the demos. But while TED celebrates all things cool and futuristic for their easily-digestible novelty, sometimes I think the most important design achievements are as significant in their shortcomings as their successes. Siftables raises some important questions. Sure, you can now use two hands, as opposed to the single mouse pointer. But do those same tangible blocks actually limit the kinds of interactions you can have, even compared to a traditional UI? Does it sound any different/ And note that — a little bit of tilting aside — the interface is still essentially two-dimensional. I’m personally really stumped by the question of how you can make a successful three-dimensional controller. Yet three dimensions is how all of us interact with space and movement daily. Maybe it’s the fact that we do so much of this, comprehend movement so richly, and take it for granted, that makes mapping those gestures so challenging.

That’s not a criticism of the project – or a claim that I can do any better. On the contrary, I think it’s important to do this sort of work because it can raise those kinds of questions. We’re gifted as a generation to try out and test these ideas with flexibility that was never before possible — and the intelligence built into these objects shows the potential of that power.

More of Siftables after the jump. And it’s well worth checking out David’s other projects, too – when I last ran into him, he was showing off the totable, Linux-powered Audiopint sound-processing box. Oh, yeah — and he’s the face control for guitar guy!

Siftables Music Sequencer from Jeevan Kalanithi on Vimeo.

Siftables project page

More Merrill Goodness

2008 has been an amazing year for music technology. But I can’t bring myself to look back on it on this New Year’s Eve: not when there’s so much to look forward to in 2009. Case in point? An extraordinary, innovative new controller that in a matter of hours was already spreading among connected music technologists around the planet.

At the end of the day, it’s not hard to describe what you might want out of an expressive music controller. Most people would agree on that. The challenge is really an engineering problem. Solve the engineering problem in an artful way, and you can spend the rest of your time just practicing playing your invention. That’s what makes the above video so exciting.

Randall Jones has built a really elegant and wonderful multi-touch hardware controller, as reported by MAKE:blog (and picked up on Hack a Day). With $50 in parts and a lot of clever hardware design and software coding, Jones has built an interface that responds to both touch and pressure and, using some smart sonic mapping, can realistically reproduce instruments like the dumbek and guiro.

Intimate Control for Physical Modeling Synthesis [Project Page / Paper Abstract]

PDF, Randall Jones MSc Research Paper

Who needs a “top 10 technologies of 2008” post for CDM when this particular instrument could pretty easily top the whole list? Let’s just call it done, and uncork the champagne: major congrats, Randy! (This is a master’s thesis!)

Jones’ work does have some precedent, but just to review how much he’s accomplished here: he’s innovated in terms of the sensing, the form factor, the software interpolation, and the way in which the control data is mapped to a synthesis method. (Whew!) That has had a number of specific achievements:

• A clever form factor: The basic design here is elegant and could be adaptable to other form factors. Most importantly, the use of polypropylene and silicone rubber means the interface deforms nicely as you press it, giving you feedback. And that opens lots of other similar design possibilities. Translation: being squishy rocks.
• Multidimensional / force-sensitive input: “Multi-touch” usually only means something that can take multiple touch inputs at one time, as in, from multiple fingers. But as I’ve complained in the past, the problem is that most multi-touch interfaces, like the Lemur and iPhone, don’t respond to the amount of pressure or deform (providing tactile feedback) when you use them. That makes them feel a bit like an ATM screen that happens to take more than one finger at a time. Jones’ prototype responds to how hard you’re pressing or hitting it, and it’s more sensitive – more like a real-world instrument. Jones calls this “multidimensional,” but force-sensitive would be another way to look at it. Translation: it’s more organic than entering your ATM PIN code.
• Audio-rate resolution: By choosing to use audio signal for control rather than … well, anything else, Randy gets an extremely responsive control signal. The controller itself is passive, requiring no power. Everything is done by processing audio created by the sensors. That’s not a new idea, but by returning to it in this context, Randy makes a much more responsive controller than most touch and multi-touch controllers before it, and returns to some of the analog-style control of instruments like the Theremin and acoustic instruments. Translation: it don’t mean a thing if it ain’t got that swing.
• Smart centroid processing: To make touch points accurate, Randy wrote a new object for Jitter that does centroid calculation. Translation: Randy worked through the details.
• Waveguide mesh modeling for deep sonic results: None of this would be meaningful without meaningful sounds coming out of it. Here, Randy builds on previous work in creating a physical model of a drum head (back to the dumbek), but both refines the model and works intelligently through how to match it with the controller. Translation: it makes good noises.

There are some other related touch devices, each with their own strengths and weaknesses, but rather than reinvent his work, I’d suggest reading through Randy’s own research paper. He does a great job talking about what’s great and not-as-great about other research and products, and makes a terrific argument for his design choices. (Basically, see also the Haken Continuum, the Korg WaveDrum, the Tactex MTC Express, the CNMAT multitouch controller, and the Audio-Input Radio Drum.)

Now, if this kind of development had come about a few years ago, the next thing we’d be talking about is how this might be commercialized. Part cost is cheap – that’s good news. I don’t know Randy’s plans for licensing and future development. But we can already compare how this might have gone had this been introduced, say, a few years ago.

The old model:

1. Spend a couple of years more refining the prototype on your own.

2. Show off the work at an academic conference in Italy. People can see it if they’re associated with an academic institution that’s also willing to pick up their airfare and let them leave for a week. (Anyone who’s had to wrestle with their department to get a couple hundred dollars for a thousand dollars in actual expenses knows what I’m talking about here.)

3. Publish in an academic journal a handful of people get, so they can try to figure out what it’s like from diagrams and grainy black-and-white photos (and no sound). Oh yeah – no one can actually comment on the story, either, so advanced researchers lack feedback and newcomers lack context and commentary.

4. Get the prototype ready for NAMM. Blow a bunch of cash on a hotel in Anaheim and a NAMM booth.

5. Work out a complex, expensive production and distribution scheme that in turn jacks up the price. Hope someone finds it and can afford it.

6. To justify the cost, make the product as finished as possible. Sell it as a “Digital Dumbek” to eccentric, loaded rock stars.

Now, I don’t want to sound grumpy or naive. The truth is, some really brilliant instruments have been down this road. But I’m not sure the description above really did a whole lot for their brilliance. I’m not necessarily saying there aren’t benefits to some of the above techniques, but clearly they won’t work for everyone.

The new model:

“Where’s my flying car?” That’s the question that often comes up when technology and a New Year coincide. With music, though, I think what we’ve really been waiting for is a convergence of new technology and new communities. Flip through the Computer Music Tutorial’s section on new instruments, and you’ll see some fantastic, exotic, and often familiar new interfaces. So what has 2009 got that the last few decades didn’t? Think cut-rate, faster tech, and a connected Web community to develop, evangelize, and use new technologies.

1. Get 20,000 Vimeo views the moment you post your first demo. No one has to fly anywhere.

2. Connect with people doing research and experiments worldwide and get instant feedback and ideas and potential collaboration – even if they’re not publishing research or flying to conferences or are even academics.

3. Instead of assuming you’ll build a finished, closed product (hello, “Digital Dumbek”), figure people will want to hack the result. (Randy has already posted parts lists, so it’s possible to build this thing right now if you’re so inclined. And he also says in Vimeo contents he’s thinking about doing a version that transmits OpenSoundControl data, so you could use it to control other instruments, music, or even visuals.)

You’ve already skipped a lot of the steps that were previously necessary just to find interested parties. And that network could in turn be used to figure out how to make the thing around the world at a lower cost and get it directly to people who might want to buy it. It also assumes those people might hack the tool for very different applications, instead of having to target one particular kind of person.

Nothing about this is a panacea for innovation: most of the hard engineering problems remain, and this doesn’t mean you can magically create new products. But there’s no question that even changing the hurdles means there’s new potential. If 2009 is about anything, I hope it’s about people finding new solutions to taking that potential and tapping into it. You can bet it’ll be a major focus of this site.

Congrats, Randy – we’ll be watching.

Happy New Year.

Touch and multi-touch interfaces are getting lots of attention, but they pose one major problem: there’s no tactile feedback. Those supposedly “primitive” buttons and knobs and such start to look a lot better when you realize your fingers are used to touching solid objects. All you get from a touchscreen is the sensation of running your finger against an undifferentiated piece of plastic. That was one of my complaints with the multi-touch music interface, Lemur: it just felt physically wrong.

As more and more interfaces employing touch interfaces, engineers are working on solutions to the problem. ExtremeTech talks about a new deal between mobile phone maker Nokia and feedback gurus Immersion (whom you may know from the gaming market):

Nokia Touchscreen Phones to Add Tactile Feedback

Don’t expect too much here — I think the results will feel more like a vibration when you’re touching a control. Then again, a little goes a long way. The Nintendo Wii very cleverly uses basic vibration to give you a subtle cue as you hit something that can be controller, for instance. The vibration is one-dimensional, but it can be enough to give your brain a connection to what you’re doing. Even Apple’s somewhat flawed Mighty Mouse provides feedback by placing a small speaker under its roller ball, which, whether it’s useful or not, tickles your fingertip so you get the sense of scrolling.

These tools have a long way to go, but they could make touch interfaces more useful for music. Even some basic haptic feedback could make using simple touch interfaces like the Korg KAOSS Pad more fun. It’ll be interesting to watch this stuff evolve — and see if mass-market cell phone technology might trickle up to niche-market music products.