You might imagine sound in space, or dream up gestures that traverse unexplored sonic territory. But actually building it is another matter. Kinect – following a long line of computer vision applications and spatial sensors – lets movement and gestures produce sound. The challenge of such instruments has long been that learning to play them is tough without tactile feedback. Thereminists learn their instrument through a the extremely-precise sensing of their instrument and sonic feedback.

In AHNE (Audio-Haptic Navigation Environment), sonic feedback is essential, but so, too, is feel. Haptic vibration lets you know as you approach sounds — essential, as they’re invisible. The work of Finland-based DJ/VJ Matti Niinimäki, aka MÅNSTERI (“Mons-te-ri”), the project is part of research undertaken at SOPI Research Group at Media Lab Helsinki. Like some sort of sound sorcerer, the user is entirely dependent on movement, feel, and sound as they move unseen sound sources through space. (More technical details below.)

It’s labeled, as always, “proof of concept.” The creator promises more videos to come; we’ll be watching as this evolves, as it looks terribly promising.

Below, “Tension” is a fair bit simpler, in which users walk through a space and control synth parameters. (“You are the knob,” one might say, though I don’t suggest shouting that at someone you don’t know. They could take it the wrong way.)

More descriptions:

AHNE

This is a demonstration video of AHNE – Audio-Haptic Navigation Environment.

It is an audio-haptic user interface that allows the user to locate and manipulate sound objects in 3d space with the help of audio-haptic feedback.

The user is tracked with a Kinect sensor using the OpenNI framework and OSCeleton (github.com/​Sensebloom/​OSCeleton).

The user wears a glove that is embedded with sensors and a small vibration motor for the haptic feedback.

This is just the first proof-of-concept demo. More videos coming soon.

HEI Project 2011
SOPI Research Group
sopi.media.taik.fi/

Aalto University School of Art and Design

AHNE – Sound and Physical Interaction

Tension

A brief video showing Tension. An interactive spatial sound installation for multiple users.

A person enters the space and a generative sound is assigned to that person. The sound pans around in the 6-channel speaker system following the user in the space.

Up to 5 users can use the installation at the same time. Each person modifies the other sounds based on the distance to the other users. The closer you are to other people the more the tension in the sound increases.

Tension – Sound and Physical Interaction

Side note: watching these two videos makes me want to consult with someone on non-verbal expression, posture, and stage presence. That criticism is mounted at myself – I could use it. Perhaps we need an all-physical, unplugged music event for laptopists, controllerists, and electronic musicians. And I can at least say I’ve had some experience in this, working in the dance program at my undergraduate alma mater, Sarah Lawrence. Anyone game? (Sounds like something we could do while CDM is in Berlin in the fall.)

For their part, the Finnish research facility is working with dancers, along with Nokia Research Center. (Sadly, I can’t find documentation.) But I think interesting things happen when us non-dancers learn movement technique, too.

In a whimsical proof of concept, artist and inventor Dan Wilcox harnesses the depth-sensing powers of the Kinect camera to turn a room full of drifting balloons into music. It occurs to me that the basic spatial model can be seen as descended directly from the Theremin – way to go, Leon, still relevant today. The sounds are simple, but it seems something you could continue to develop musically – to say nothing of what it could do for the ball pit at Chuck E. Cheese’s. (Slogan: Where a Kid Can Be a Kid Who Gets Obsessed With Skeeball Prizes / Get Scared Out of a Kid’s Mind By the Other Kids in the Ball Pit. Sorry, it’s an American suburban thing of a certain age, for the more than half of you who have no idea what I’m talking about.)

Full description from Dan:

Experiments in balloon motion and sound using an MS Kinect depth sensing camera.

Created for the Carnegie Mellon 1st & 2nd year MFA Graduate show entitled “Fresh Baked Goods” at Bakery Square, April 2011.

A machine stands in a room surrounded by balloons. Circulating fans blow the balloons over the machine which creates sound based on their movements.

Mode 1: Tones

Balloon height and x/y position control the pitch and panning of a treble and bass voice. The tones can be quantized into a certain key or a glisssando can be employed for a theremin-style effect.

Mode 2: 99 Luftballons

The playback speed of Nena’s 99 Luftballons is controlled by balloon height. The balloons must be kept in the air for the song to play. Feed the machine.

Built using Open Frameworks, ofxKinect, and Open CV for balloon tracking and Pure Data for sound generation/playback.

See danomatika.com/​blog/​balloon-project for more info.

Dan has a master plan with a robotic music-playing suit and other ideas, so I can’t wait to see where this goes.

Nerds: It’s an OSC sequencer. It’s JavaScript-programmable for making your own generative music. It works with hardware and other software. You can use it in real-time.

Everyone: it makes spectacularly strange sounds out of spectacularly beautiful flows of geometries through space.

IanniX, the latest-generation descendant of work done by pioneering experimental composer Iannis Xenakis, has been evolving at rapid pace into what may be the most sophisticated graphical sequencer ever. Xenakis originally had to content himself to drawing elaborate, architectural graphics on paper, then later being one of the first to use a graphical tablet for interactive scores. IanniX, backed by the French Ministry of Culture, is now barely recognizable even from more primitive versions that carried the same name. But the idea is the same: graphical geometries represent events in pitch and time, now sequencing other software (any software that can handle OSC or MIDI) to produce sound.

Free on Mac, Windows, and Linux, and now with growing documentation, IanniX can be seen producing the kinds of warped sounds Xenakis made in his music. But it is one of the first steps toward a graphical sequencer that could be used in all kinds of cases. And it’s free and open source under the GPL v3.

I’ve included some of the recent videos that show off what it can do. I especially like the recursive demo. But since it runs on your OS — well, unless you’re sticking to your beloved Atari ST or BeBox — you can just go grab it yourself.

http://iannix.org/en/index.php

My sense is that IanniX could have implications even beyond this software. Imagine a greater variety of music software that begins to work in spatial and graphical interfaces, not just the traditional piano rolls and linear tape-style arrangement views. And imagine that such tools, using protocols like OSC and MIDI, begin to establish common means of communicating with one another over a network. (OSC and, in particular, MIDI, are in need of some evolution to fully satisfy that. But these kinds of tools might be an ideal way to prod that very evolution.)

Speaking of prodding, thanks to Mark Birchall on Twitter for reminding me to write this up.

Now, if I can just find some hyperspace portal to additional space and time to play with this properly… there must be a productivity jump gate around here somewhere.

Based on work with the Oblong g-speak “spatial operating environment” gestural system – research that inspired the film Minority Report – our friend Trey Harrison has been doing some wonderful work with new Theremin-style interfaces. He writes:

I have been working with Oblong Industries (http://oblong.com) and
took some of my
spare time to combine their technology with my Salvation project
(http://slvtn.com)
and build a theremin-like instrument.

There are three degrees of control:

• Pitch is adjusted by moving hands left and right.
• Volume is adjusted by moving hands up and down.
• Vibrato is adjusted by moving hands foward and backward.

Many players and hands are possible, and the control can be applied to any MIDI instrument.

I like the fact that two hands are only the beginning — invite friends for collaborative sessions and get an octo-armed version! The pitch scaling certainly makes it easier to hit the notes, although it does remove some of the expressive pitch bends of the original Theremin. It’d be nice if an additional gesture (pinching, perhaps?) could allow you to warp between scale degrees.

I love the project; I hope we get to see more.

Chris O’Shea sends along his latest project, a collaboration with sound designer and composer Owen Lloyd called Drop Spin Fade. Part of the Future of Sound tour, Drop Spin Fade allows users to position, sculpt, and play with sound in a 3D environment using gestural control.

Drop Spin Fade

The music/sound environment: Through a series of iterations, Chris and Owen have started simple and built increasingly-sophisticated sonic control using the setup, manipulating granular samples by spinning and bouncing them around the space. It’s not just positioning at work here: you can actually shape the sounds you’re hearing by interacting with the geometric forms in the environment. Eventually, the designers hope to give users more compositional control, making this into a kind of 3D sequencer.

The guts behind the scenes: The work was built to showcase the Illustrious positional sound system, which can use positioning data to create 3D sound environments. For control, the project uses the Gametrak game controller hardware, which you may have seen used in inexpensive golf and other sport games. It happens to be a very nice gestural controller, as well, with extremely low latency when compared to video camera tracking solutions. Visuals and hardware interface are performed in Adobe Director, routing positional control to Illustrious via MIDI and playing a live sound patch built in Max/MSP via OpenSoundControl data. There will be yet another piece as work proceeds on support for the Nintendo Wii controller.

I’m actually quite surprised that more work hasn’t been done with 3D interfaces — though I suppose I shouldn’t be too surprised, as it’s extremely labor intensive! 3D has focused largely on positioning, but with powerful hardware and software capabilities bring 3D to the masses, 3D interfaces are surely next. Iterations and shared research are vitally important to any medium advancing, so I hope we’ll have more projects in this area. (I happen to be working on something different but related in the 3D space in my own research, which I’ll share when I’m ready.)

Previously from Chris O’Shea land: Muon Speakers, with Processing Visuals