You might imagine sound in space, or dream up gestures that traverse unexplored sonic territory. But actually building it is another matter. Kinect – following a long line of computer vision applications and spatial sensors – lets movement and gestures produce sound. The challenge of such instruments has long been that learning to play them is tough without tactile feedback. Thereminists learn their instrument through a the extremely-precise sensing of their instrument and sonic feedback.

In AHNE (Audio-Haptic Navigation Environment), sonic feedback is essential, but so, too, is feel. Haptic vibration lets you know as you approach sounds — essential, as they’re invisible. The work of Finland-based DJ/VJ Matti Niinimäki, aka MÅNSTERI (“Mons-te-ri”), the project is part of research undertaken at SOPI Research Group at Media Lab Helsinki. Like some sort of sound sorcerer, the user is entirely dependent on movement, feel, and sound as they move unseen sound sources through space. (More technical details below.)

It’s labeled, as always, “proof of concept.” The creator promises more videos to come; we’ll be watching as this evolves, as it looks terribly promising.

Below, “Tension” is a fair bit simpler, in which users walk through a space and control synth parameters. (“You are the knob,” one might say, though I don’t suggest shouting that at someone you don’t know. They could take it the wrong way.)

More descriptions:

AHNE

This is a demonstration video of AHNE – Audio-Haptic Navigation Environment.

It is an audio-haptic user interface that allows the user to locate and manipulate sound objects in 3d space with the help of audio-haptic feedback.

The user is tracked with a Kinect sensor using the OpenNI framework and OSCeleton (github.com/​Sensebloom/​OSCeleton).

The user wears a glove that is embedded with sensors and a small vibration motor for the haptic feedback.

This is just the first proof-of-concept demo. More videos coming soon.

HEI Project 2011
SOPI Research Group
sopi.media.taik.fi/

Aalto University School of Art and Design

AHNE – Sound and Physical Interaction

Tension

A brief video showing Tension. An interactive spatial sound installation for multiple users.

A person enters the space and a generative sound is assigned to that person. The sound pans around in the 6-channel speaker system following the user in the space.

Up to 5 users can use the installation at the same time. Each person modifies the other sounds based on the distance to the other users. The closer you are to other people the more the tension in the sound increases.

Tension – Sound and Physical Interaction

Side note: watching these two videos makes me want to consult with someone on non-verbal expression, posture, and stage presence. That criticism is mounted at myself – I could use it. Perhaps we need an all-physical, unplugged music event for laptopists, controllerists, and electronic musicians. And I can at least say I’ve had some experience in this, working in the dance program at my undergraduate alma mater, Sarah Lawrence. Anyone game? (Sounds like something we could do while CDM is in Berlin in the fall.)

For their part, the Finnish research facility is working with dancers, along with Nokia Research Center. (Sadly, I can’t find documentation.) But I think interesting things happen when us non-dancers learn movement technique, too.

Touch and multi-touch interfaces are getting lots of attention, but they pose one major problem: there’s no tactile feedback. Those supposedly “primitive” buttons and knobs and such start to look a lot better when you realize your fingers are used to touching solid objects. All you get from a touchscreen is the sensation of running your finger against an undifferentiated piece of plastic. That was one of my complaints with the multi-touch music interface, Lemur: it just felt physically wrong.

As more and more interfaces employing touch interfaces, engineers are working on solutions to the problem. ExtremeTech talks about a new deal between mobile phone maker Nokia and feedback gurus Immersion (whom you may know from the gaming market):

Nokia Touchscreen Phones to Add Tactile Feedback

Don’t expect too much here — I think the results will feel more like a vibration when you’re touching a control. Then again, a little goes a long way. The Nintendo Wii very cleverly uses basic vibration to give you a subtle cue as you hit something that can be controller, for instance. The vibration is one-dimensional, but it can be enough to give your brain a connection to what you’re doing. Even Apple’s somewhat flawed Mighty Mouse provides feedback by placing a small speaker under its roller ball, which, whether it’s useful or not, tickles your fingertip so you get the sense of scrolling.

These tools have a long way to go, but they could make touch interfaces more useful for music. Even some basic haptic feedback could make using simple touch interfaces like the Korg KAOSS Pad more fun. It’ll be interesting to watch this stuff evolve — and see if mass-market cell phone technology might trickle up to niche-market music products.