David Merrill, working with Jeevan Kalanithi and (for the audio engine) Josh Kopin, wowed audiences at the TED conference with his Siftables interactive blocks. These strike me as what the Audiocubes have tried, sometimes unsuccessfully, to be — physical objects that react to the proximity of other objects, allowing you to manipulate music and media by moving around tangible blocks. Siftables are gifted with multiple expressive controls (tilt helping them break the plane of the surface), and intelligent screens that make them more adaptable and provide more visual feedback.

The music sequencer is very cool, though I think it’s actually the Scrabble-like game that may be the winner among the demos. But while TED celebrates all things cool and futuristic for their easily-digestible novelty, sometimes I think the most important design achievements are as significant in their shortcomings as their successes. Siftables raises some important questions. Sure, you can now use two hands, as opposed to the single mouse pointer. But do those same tangible blocks actually limit the kinds of interactions you can have, even compared to a traditional UI? Does it sound any different/ And note that — a little bit of tilting aside — the interface is still essentially two-dimensional. I’m personally really stumped by the question of how you can make a successful three-dimensional controller. Yet three dimensions is how all of us interact with space and movement daily. Maybe it’s the fact that we do so much of this, comprehend movement so richly, and take it for granted, that makes mapping those gestures so challenging.

That’s not a criticism of the project – or a claim that I can do any better. On the contrary, I think it’s important to do this sort of work because it can raise those kinds of questions. We’re gifted as a generation to try out and test these ideas with flexibility that was never before possible — and the intelligence built into these objects shows the potential of that power.

More of Siftables after the jump. And it’s well worth checking out David’s other projects, too – when I last ran into him, he was showing off the totable, Linux-powered Audiopint sound-processing box. Oh, yeah — and he’s the face control for guitar guy!

Siftables Music Sequencer from Jeevan Kalanithi on Vimeo.

Siftables project page

More Merrill Goodness

2008 has been an amazing year for music technology. But I can’t bring myself to look back on it on this New Year’s Eve: not when there’s so much to look forward to in 2009. Case in point? An extraordinary, innovative new controller that in a matter of hours was already spreading among connected music technologists around the planet.

At the end of the day, it’s not hard to describe what you might want out of an expressive music controller. Most people would agree on that. The challenge is really an engineering problem. Solve the engineering problem in an artful way, and you can spend the rest of your time just practicing playing your invention. That’s what makes the above video so exciting.

Randall Jones has built a really elegant and wonderful multi-touch hardware controller, as reported by MAKE:blog (and picked up on Hack a Day). With $50 in parts and a lot of clever hardware design and software coding, Jones has built an interface that responds to both touch and pressure and, using some smart sonic mapping, can realistically reproduce instruments like the dumbek and guiro.

Intimate Control for Physical Modeling Synthesis [Project Page / Paper Abstract]

PDF, Randall Jones MSc Research Paper

Who needs a “top 10 technologies of 2008” post for CDM when this particular instrument could pretty easily top the whole list? Let’s just call it done, and uncork the champagne: major congrats, Randy! (This is a master’s thesis!)

Jones’ work does have some precedent, but just to review how much he’s accomplished here: he’s innovated in terms of the sensing, the form factor, the software interpolation, and the way in which the control data is mapped to a synthesis method. (Whew!) That has had a number of specific achievements:

• A clever form factor: The basic design here is elegant and could be adaptable to other form factors. Most importantly, the use of polypropylene and silicone rubber means the interface deforms nicely as you press it, giving you feedback. And that opens lots of other similar design possibilities. Translation: being squishy rocks.
• Multidimensional / force-sensitive input: “Multi-touch” usually only means something that can take multiple touch inputs at one time, as in, from multiple fingers. But as I’ve complained in the past, the problem is that most multi-touch interfaces, like the Lemur and iPhone, don’t respond to the amount of pressure or deform (providing tactile feedback) when you use them. That makes them feel a bit like an ATM screen that happens to take more than one finger at a time. Jones’ prototype responds to how hard you’re pressing or hitting it, and it’s more sensitive – more like a real-world instrument. Jones calls this “multidimensional,” but force-sensitive would be another way to look at it. Translation: it’s more organic than entering your ATM PIN code.
• Audio-rate resolution: By choosing to use audio signal for control rather than … well, anything else, Randy gets an extremely responsive control signal. The controller itself is passive, requiring no power. Everything is done by processing audio created by the sensors. That’s not a new idea, but by returning to it in this context, Randy makes a much more responsive controller than most touch and multi-touch controllers before it, and returns to some of the analog-style control of instruments like the Theremin and acoustic instruments. Translation: it don’t mean a thing if it ain’t got that swing.
• Smart centroid processing: To make touch points accurate, Randy wrote a new object for Jitter that does centroid calculation. Translation: Randy worked through the details.
• Waveguide mesh modeling for deep sonic results: None of this would be meaningful without meaningful sounds coming out of it. Here, Randy builds on previous work in creating a physical model of a drum head (back to the dumbek), but both refines the model and works intelligently through how to match it with the controller. Translation: it makes good noises.

There are some other related touch devices, each with their own strengths and weaknesses, but rather than reinvent his work, I’d suggest reading through Randy’s own research paper. He does a great job talking about what’s great and not-as-great about other research and products, and makes a terrific argument for his design choices. (Basically, see also the Haken Continuum, the Korg WaveDrum, the Tactex MTC Express, the CNMAT multitouch controller, and the Audio-Input Radio Drum.)

Now, if this kind of development had come about a few years ago, the next thing we’d be talking about is how this might be commercialized. Part cost is cheap – that’s good news. I don’t know Randy’s plans for licensing and future development. But we can already compare how this might have gone had this been introduced, say, a few years ago.

The old model:

1. Spend a couple of years more refining the prototype on your own.

2. Show off the work at an academic conference in Italy. People can see it if they’re associated with an academic institution that’s also willing to pick up their airfare and let them leave for a week. (Anyone who’s had to wrestle with their department to get a couple hundred dollars for a thousand dollars in actual expenses knows what I’m talking about here.)

3. Publish in an academic journal a handful of people get, so they can try to figure out what it’s like from diagrams and grainy black-and-white photos (and no sound). Oh yeah – no one can actually comment on the story, either, so advanced researchers lack feedback and newcomers lack context and commentary.

4. Get the prototype ready for NAMM. Blow a bunch of cash on a hotel in Anaheim and a NAMM booth.

5. Work out a complex, expensive production and distribution scheme that in turn jacks up the price. Hope someone finds it and can afford it.

6. To justify the cost, make the product as finished as possible. Sell it as a “Digital Dumbek” to eccentric, loaded rock stars.

Now, I don’t want to sound grumpy or naive. The truth is, some really brilliant instruments have been down this road. But I’m not sure the description above really did a whole lot for their brilliance. I’m not necessarily saying there aren’t benefits to some of the above techniques, but clearly they won’t work for everyone.

The new model:

“Where’s my flying car?” That’s the question that often comes up when technology and a New Year coincide. With music, though, I think what we’ve really been waiting for is a convergence of new technology and new communities. Flip through the Computer Music Tutorial’s section on new instruments, and you’ll see some fantastic, exotic, and often familiar new interfaces. So what has 2009 got that the last few decades didn’t? Think cut-rate, faster tech, and a connected Web community to develop, evangelize, and use new technologies.

1. Get 20,000 Vimeo views the moment you post your first demo. No one has to fly anywhere.

2. Connect with people doing research and experiments worldwide and get instant feedback and ideas and potential collaboration – even if they’re not publishing research or flying to conferences or are even academics.

3. Instead of assuming you’ll build a finished, closed product (hello, “Digital Dumbek”), figure people will want to hack the result. (Randy has already posted parts lists, so it’s possible to build this thing right now if you’re so inclined. And he also says in Vimeo contents he’s thinking about doing a version that transmits OpenSoundControl data, so you could use it to control other instruments, music, or even visuals.)

You’ve already skipped a lot of the steps that were previously necessary just to find interested parties. And that network could in turn be used to figure out how to make the thing around the world at a lower cost and get it directly to people who might want to buy it. It also assumes those people might hack the tool for very different applications, instead of having to target one particular kind of person.

Nothing about this is a panacea for innovation: most of the hard engineering problems remain, and this doesn’t mean you can magically create new products. But there’s no question that even changing the hurdles means there’s new potential. If 2009 is about anything, I hope it’s about people finding new solutions to taking that potential and tapping into it. You can bet it’ll be a major focus of this site.

Congrats, Randy – we’ll be watching.

Happy New Year.

Touch and multi-touch interfaces are getting lots of attention, but they pose one major problem: there’s no tactile feedback. Those supposedly “primitive” buttons and knobs and such start to look a lot better when you realize your fingers are used to touching solid objects. All you get from a touchscreen is the sensation of running your finger against an undifferentiated piece of plastic. That was one of my complaints with the multi-touch music interface, Lemur: it just felt physically wrong.

As more and more interfaces employing touch interfaces, engineers are working on solutions to the problem. ExtremeTech talks about a new deal between mobile phone maker Nokia and feedback gurus Immersion (whom you may know from the gaming market):

Nokia Touchscreen Phones to Add Tactile Feedback

Don’t expect too much here — I think the results will feel more like a vibration when you’re touching a control. Then again, a little goes a long way. The Nintendo Wii very cleverly uses basic vibration to give you a subtle cue as you hit something that can be controller, for instance. The vibration is one-dimensional, but it can be enough to give your brain a connection to what you’re doing. Even Apple’s somewhat flawed Mighty Mouse provides feedback by placing a small speaker under its roller ball, which, whether it’s useful or not, tickles your fingertip so you get the sense of scrolling.

These tools have a long way to go, but they could make touch interfaces more useful for music. Even some basic haptic feedback could make using simple touch interfaces like the Korg KAOSS Pad more fun. It’ll be interesting to watch this stuff evolve — and see if mass-market cell phone technology might trickle up to niche-market music products.