Here’s how interfaces normally break down. You’ve got your conventional, tactile interfaces, like a knob. You’ve got your touch interfaces, which lack tactile feedback (you touch them, but they don’t push back). You’ve got your gestural interfaces, which have you waving your hands in the air without touching anything and without any tactile feedback. (They’re generally the most challenging, because your brain has no feedback for what it’s doing.)

Syntact creates an entirely new category. It’s a gestural interface, of the “waving your hands around in the air” sort. But while your hand is in mid-air and isn’t touching anything, it does provide tactile feedback. It pushes back as you move your hand around, giving you interactive feedback. The way it pulls it off: sound. 121 ultrasonic transducers beam sound at a particular point, so that you feel something as you move.

You can see a bit of what this means in the new video, above. I’m hoping to get a hands-on (erm, hands-off) demo soon from the designer. The basic specs:

• Optical analysis of gestures, using a USB camera built into the interface
• MIDI control, for use with any live performance or music making rig (or other media)
• A control panel for selecting different sonic images and adjusting scaling.
• A built-in music solution visualizes sound and makes it easier to map to your own MIDI files.

More information:
http://www.ultrasonic-audio.com/products/syntact.html

Also well worth checking out the directional speaker tech from these Slovenia-based developers – directional sound is another huge area of innovation.
http://www.ultrasonic-audio.com/products/acouspade.html

If you want to try this in person, it’ll be at the Beam Festival in London in late June.

Side note: Yes, I’m looking into that LEAP thing, for more gestures, albeit without tactile feedback. Stay tuned.

What you’re watching in the video above doesn’t involve cameras or motion sensors. It’s the kind of brain-to-machine, body-to-interaction interface most of us associate with science fiction. And while the technology has made the occasional appearance in unusual, niche commercial applications, it’s poised now to blow wide open for music – open as in free and open source.

Erasing the boundary between contracting a muscle in the bio-physical realm and producing electronic sound in the virtual realm is what Xth Sense is all about. Capturing biological data is all the rage these days, seen primarily in commercial form in products for fitness, but a growing trend in how we might make our computers accessories for our bodies as well as our minds. (Or is that the other way around?) This goes one step further: the biological becomes the interface.

Artist and teacher Marco Donnarumma took first prize with this project in the prestigious Guthman Musical Instrument Competition at Georgia Tech in the US. Born in Italy and based in Edinburgh, Scotland, Marco explains to us how the project works and why he took it up. It should whet your appetite as we await an open release for other musicians and tinkerers to try next month. (By the way, if you’re in the New York City area, Marco will be traveling to the US – a perfect chance to collaborate, meet, or set up a performance or workshop; shout if you’re interested.)

Hypo Chrysos live at Trendelenburg AV Festival, Gijon, Spain, December 2011.

CDM: Tell us a bit about yourself. You’re working across disciplines, so how do you describe what you do?

Marco: People would call me a media and sound artist. I would say what I love is performing, but at the same time, I’m really curious about things. So, most of the time I end up coding my software, developing devices and now even designing wearable tech. Since some years now I work only with free and open source tools and this is naturally reflected in what I do and how I do it. (Or at least I hope so!)

I just got back from Atlanta, US, where the Xth Sense (XS) was awarded the first prize in the Margaret Guthman New Musical Instrument, as what they named the “world’s most innovative new musical instrument.” [See announcement from Georgia Tech.]

It’s an encouraging achievement and I’m still buzzing, specially because the other 20 finalists all presented great ideas. Overall, it has been an inspiring event, and I warmly recommend musicians and inventors to participate next year. My final performance:

Make sure to use a proper soundsystem [when watching the videos]; most of the sound spectrum lives between 20-60Hz.

Music for Flesh II live at Georgia Tech Center for Music Technology, Atlanta, USA, February 2012. Photo courtesy the artist.

You’re clenching your muscles, and something is happening – can you tell us how this XS system works?

Marco: My definition of it goes like “a biophysical framework for musical performance and responsive milieux.” In other words, it is a technology that extends some intrinsic sonic capabilities of the human body through a computer system that senses the physical energy released by muscle tissues.

I started developing it in September 2011 at the SLE, the Sound Lab at the Edinburgh University, and got it ready to go in March 2011. It has evolved a lot in many ways ever since.

The Xth Sense wearable biosensors by Chris Scott.

The XS is composed of custom biophysical sensors and a custom software.

At the onset of a muscle contraction, energy is released in the form of acoustic sound. This is to say, similarly to the chord of a violin, each muscle tissue vibrates at specific frequencies and produces a sound (called Mechanomyographic signal, or MMG). It is not audible to human ear, but it is indeed a soundwave that resonates from the body.

The MMG data is quite different from locative data you can gather with accelerometers and the like; whereas the latter reports the consequence of a movement, the former directly represents the energy impulse that causes that movement. If you add to this a high sampling rate (up to 192.000Hz if your sound card supports it) and very low latency (measured at 2.3ms) you can see why the responsiveness of the XS can be highly expressive.

The XS sensors capture the low-frequency acoustic vibrations produced by a performer’s body and send them to the computer as an audio input. The XS software analyzes the MMG in order to extract the characteristics of the movements, such as dynamics of a single gesture, maximum amplitude of a series of gestures in time, etc.

These are fed to some algorithms that produce the control data (12 discrete and continuous variables for each sensor) to drive the sound processing of the original MMG.

Eventually, the system plays back both the raw muscle sounds (slightly transposed to become better audible, say about 50/60Hz) and the processed muscle sounds.

I like to term this model of performance biophysical music, in contrast with biomusic, which is based on the electrical impulses of muscles and brainwaves.

By differently contracting muscles (which has a different meaning than simply “moving”) one can create and sculpt musical material in real-time. One can design a specific gesture that produces a specific sonic result, what I call a sound-gesture. These can be composed in a score, or improvised, or also improvised on a more or less fixed score.

The XS software has also a sensing sequencing time-line: with a little machine learning (just implemented few days ago) the system understands when you’re still or moving, when you’re being fast or slow, and can use this data to change global parameters, functions or to play with the timing of events. For example, the computer can track your behaviour in time and wait for you to stop whatever you’re doing before switching to a different set of funcions.

The XS sensors are wearable devices, so the computer can be forgotten in a corner of the stage; the performer has complete freedom on stage, and the audience is not exposed to the technology, but rather to the expressivity of the performance. What I like most about the XS is that is a flexible and multi-modal instrument. One can use it to:

• capture and playback acoustic sounds of the body,
• control audio and video software on the computer, or
• capture body sounds and control them through the computer simultaneously.

This opens up an interesting perspective on the applications of the XS to musical performance, dance, theatre and interaction design. The XS can also be used only as a gestural controller, although I never use it exclusively this way. We have thousands of controllers out there.

Besides, I wanted the XS to be accessible, usable, hackable and redistributable. Unfortunately, the commercialized product dealing with biosignals are mostly not cheap and — most importantly — closed to the community. See the Emotiv products (US$299 Neuro Headset, not for developers), or the BioFlex (US$392.73). One could argue that the technology is complex, and that’s why those devices are expensive and closed. This could make sense, but who says we can’t produce new technologies that openly offer similar or new capabilities at a much lower cost?

The formal recognition of the XS as an innovative musical instrument and the growing effort of the community in producing DIY EEG, ECG and Biohacking devices are a clear statement in this sense. I find this movement encouraging and possibly indispensable nowadays, as the information technology industry is increasingly deploying biometric data for adverts and security systems. For the geeky ones there are some examples in a recent paper of mine for the 2012 CHI workshop on Liveness.

For those reasons, the XS hardware design has been implemented in the simplest form I could think of; the parts needed to build an XS sensor cost about £5 altogether and the schematics looks purposely dumb. The sensors can be worn on any parts of the body. I worked with dancers who wore them on the neck and legs, a colleague stuck one to his throat to capture the resonances of his voice, I use them on the arms or to capture the pumping of the blood flow and the heart rate.

The XS software is free, based in Pd, aka Pure Data, and comes with a proper, user-friendly Graphical User Interface (GUI) and its own library, which includes over one hundred objects with help files. It is developed on Linux, and it’s Mac OS X compatible; I’m not developing for Windows, but some people got it working there too. A big thumb up goes to our wonderful Pd Community; if I had not been reading and learning through the Pd mailing list for the past 5 years I would have never been able to code this stuff.

The Xth Sense software Graphical User Interface. Built in Pd.

The public release of the project will be in April. The source code, schematics, tutorials, will be freely available online, and there will be DIY kits for the lazier ones. I’m already collecting orders for the first batch of DIY kits, so if anybody is interested please, get in touch:
http://marcodonnarumma.com/contact

I do hope to see the system hacked and extended, especially because the sensors were initially built with the support of the folks at the Dorkbot ALBA/Edinburgh Hacklab. I’m also grateful to the community around me, friends, musicians, artists devs and researchers for contributing to the success of the project by giving feedback, inspiring and sharing (you know who you are!).

Thanks, Marco! We’ll be watching!

More on the Work

http://marcodonnarumma.com/works/xth-sense/
http://marcodonnarumma.com/works/music-for-flesh-ii/
http://res.marcodonnarumma.com/blog/

And the Edinburgh hack lab:
http://edinburghhacklab.com/

Biological Interfaces for Music

There isn’t space here to recount the various efforts to do this; Marco’s design to me is notable mainly in its simplicity and – hopefully, as we’ll see next month – accessibility to other users. I’ve seen a number of brain interfaces just in the past year, but perhaps someone with more experience on the topic would like to share; that could be a topic for another post.

Entirely unrelated to music, but here’s the oddest demo video I’ve seen of human-computer interfacing, which I happened to see today. (Well, unrelated to music until you come up with something this crazy. Go! I want to see your band playing with interactive animal ears.)

Scientific American’s blog tackles the question this week (bonus 80s sci-fi movie reference):
Brain-Machine Interfaces in Fact and Fiction

I’ve used up my Lawnmower Man reference quota for the month, so tune in in April.

FRACT is a curious combination of music studio and puzzle game, merging elements of games like Myst with the sorts of synths and pattern editors you’d expect somewhere like Ableton Live. You have to make sounds and melodies to solve puzzles; by the end of the game, say the creators, you’re even producing original music. The work of a small student team out of Montreal, FRACT looks like it has all the makings of an underground indie hit – at least for music nerds.

As the creators describe it:

FRACT is a first person adventure game for Windows & Mac much in the vein of the Myst titles, but with an electro twist. Gameplay boils down to three core activities: Explore, Rebuild, Create. The player is let loose into an abstract world built on sound and structures inspired by electronic music. It’s left to the player to explore the environment to find clues to resurrect and revive the long-forgotten machinery of this musical world, in order to unlock its inner workings. Drawing inspiration from Myst, Rez and Tron, the game is also influenced by graphic design, data visualization, electronic music and analog culture.

The hub of the game is a virtual studio, collecting patterns and timbres. It’s right now in prototype phase, but it already looks visually stunning, an alien, digital world in which more-conventional step-sequencer views seem to emerge from futuristic landscapes. And you can spot Pd in the background (the free and open source patching tool, Pure Data). Update: the developers confirm that they’re working with the embeddable Pd library, libpd. That enables synths with sounds like phase modulation and classic virtual analog sounds, all modulating and generating sounds in-game.

The developers have also published plenty of sound samples so you can experience the musical side of this. Via SoundCloud:

While never released, one place some similar ideas has shown up is a prototype game inspired by Deadmau5. As in this title, two-dimensional editing screens and synth parameters are mapped to a first-person, three-dimensional environment. However, FRACT appears to take this concept much further, expanding upon the world, building more instruments, and actually turning those interactions into gameplay elements. The video of the Deadmau5 project – apparently done in-house for fun and not endorsed by the mouse-headed artist:

That title was the work of a game house called Floaty Hybrid; music blog Synthtopia got the scoop on this in August:
http://www.floathybrid.com
Mau5Bot Sequencer Lets You Make Music In A 3D World [Synthtopia]

We’ll be watching this one develop, certainly; good luck to the team!
http://fractgame.com/

Electronic music making has had several major epochs. There was the rise of the hardware synth, first with modular patch cords and later streamlined into encapsulated controls, in the form of knobs and switches. There was the digital synth, in code and graphical patches. And there was the two-dimensional user interface.

We may be on the cusp of a new age: the three-dimensional paradigm for music making.

AudioGL, a spectacularly-ambitious project by Toronto-based engineer and musician Jonathan Heppner, is one step closer to reality. Three years in the making, the tool is already surprisingly mature. And a crowd-sourced funding campaign promises to bring beta releases as soon as this summer. In the demo video above, you can see an overview of some of its broad capabilities:

• Synthesis, via modular connections
• Sample loading
• The ability to zoom into more conventional 2D sequences, piano roll views, and envelopes/automation
• Grouping of related nodes
• Patch sharing
• Graphical feedback for envelopes and automation, tracked across z-axis wireframes, like circuitry

All of this is presented in a mind-boggling visual display, resembling nothing more than constellations of stars.

Is it just me, or does this make anyone else want to somehow combine modular synthesis with a space strategy sim like Galactic Civilizations? Then again, that might cause some sort of nerd singularity that would tear apart the fabric of the space-time continuum – or at least ensure we never have any normal human relationships again.

Anyway, the vitals:

• It runs on a lowly Lenovo tablet right now, with integrated graphics.
• The goal is to make it run on your PC by the end of the year. (Mac users hardly need a better reason to dual boot. Why are you booting into Windows? Because I run a single application that makes it the future.)
• MIDI and ReWire are onboard, with OSC and VST coming.
• With crowd funding, you’ll get a Win32/64 release planned by the end of the year, and betas by summer (Windows) or fall/winter (Mac).

I like this quote:

Some things which have influenced the design of AudioGL:
Catia – Dassault Systèmes
AutoCAD – Autodesk
Cubase – Steinberg
Nord Modular – Clavia
The Demoscene

Indeed. And with computer software now reaching a high degree of maturity, such mash-ups could open new worlds.

Learn about the project, and contribute by the 23rd of March via the (excellent) IndieGogo:

http://audiogl.com

Turning music and sound into three-dimensional worlds often yields something that fields like a trip through space. But this feels like a real trip. Through pulsing, glowing starfields, “Versum”‘s audiovisual movements are brain-bendingly transformative. Artist Tarik Barri has created an integrated world of sound and image that makes the interface and the compositional realms seamless. It seems as though this really is a musical universe, through whose harmonies of the spheres you can fly like. Boldly going, indeed.

Ingredients: Max/MSP/Jitter, Processing, Java, SuperCollider, GLSL [the 3D shading language], and … some serious skill and time, I imagine.

The work has been in development for some years (not surprisingly, given the results). But it surfaced again as we brought up the 3Dconnexion SpaceNavigator hardware as a practical controller for 3D. See Create Digital Motion:
Look at Me, I’m Flying: SpaceNavigator Hardware + Blender

Tarik’s work resurfaced after a presentation in the UK. Reader janklug writes:

I’m just back from the M4_u Max/MSP/Jitter conference in Leicester (was great, btw), where Tarik Barri presented his project ‘Versum’, both as an installation and as a performance.
The user (and in case of the performance, Tarik) navigates through this incredible 3D-space-sequencer-universum with the help of a SpaceNavigator; glowing objects floating in this space produce sound, and as you approach them, they even give this nice doppler effect…
It was totally amazing to be able to float between pulsing rhythm-planet-objects and shiny drone-beams; navigation was easy and natural. Tarik uses a combination of Processing and Max/MSP; don’t know which one the SpaceNavigator is connected to.
Having tried this, I immediately ordered one; I think it also could be a great interface for M4L…

More information:
http://tarikbarri.nl/projects/versum
PDF documentation [2009]

Significantly, it’s really the act of flying that controls the music. That remains interactive, but it’s the movement through the three-dimensional space that determines what you hear. As the artist explains:

This virtual world is seen and heard from the viewpoint of a moving virtual camera with virtual microphones attached. This camera, controlled in realtime by means of a joystick (or any other kind of controller) moves through space, similar to how first person shooter games work. Within this space, I place objects that can be both seen and heard, and like in reality, the closer the camera is to them, the louder you hear them. So when the camera moves past several visual objects, you simultaneously hear several sounds fading in and out. Consequently, the way the camera travels past them actually causes melodies and compositional structures to be seen and heard.

The visual position of each object coincides with the panning of its sound: objects to the right of the camera will also be heard on the right, and those behind the camera will be heard from behind in case a surround speaker setup is used. This principle also applies to the Z-axis, meaning that sounds can be heard coming from above and below if the speaker setup supports it.

That’s the essential question, to me, when looking at 3D environments for music. What about the dimensionality will interact with the music? Is it something spatial, or will there be other sorts of interactions? (New Zealander-turned-Berliner Julian Oliver worked extensively with game engines, for instance. One solution for him was modifying the “gun” in those games to be an implement for doing things in the space, turning swords into plowshares after a fact by making the gun produce music rather than kill virtual entities.)

So, now you’ve seen some of the technical demonstration. But Tarik uses his work as an environment in which to make audiovisual performances. Here’s what some actual live playing looks like, in a beautiful, meditative piece called “Eleven”:

In fact, the biggest challenge to me of a piece this awesome is that you want an immersive environment, not just the small, rectangular screens that are often all festivals and venues can afford.

Holodeck, anyone?

More:

Compare the complex model of what a computer can use to control sound and musical pattern in real-time to the visualization. You see knobs, you see faders that resemble mixers, you see grids, you see – bizarrely – representations of old piano rolls. The accumulated ephemera of old hardware, while useful, can be quickly overwhelmed by a complex musical creation, or visually can fail to show the musical ideas that form a larger piece. You can employ notation, derived originally from instructions for plainsong chant and scrawled for individual musicians – and quickly discover how inadequate it is for the language of sound shaping in the computer.

Or, you can enter a wild, three-dimensional world of exploded geometries, navigated with hand gestures.

Welcome to the sci fi-made-real universe of Portland-based Christian Bannister’s subcycle. Combining sophisticated, beautiful visualizations, elegant mode shifts that move from timbre to musical pattern, and two-dimensional and three-dimensional interactions, it’s a complete visualization and interface for live re-composition. A hand gesture can step from one musical section to another, or copy a pattern. Some familiar idioms are here: the grid of notes, a la piano roll, and the light-up array of buttons of the monome. But other ideas are exploded into spatial geometry, so that you can fly through a sound or make a sweeping rectangle or circle represent a filter.

Ingredients, coupling free and open source software with familiar, musician-friendly tools:

• Two projectors
• A monome
• Processing, the elegant and artist-savvy free software for visual code
• Ableton Live and Cycling ’74′s Max for Live, acting as the interactive glue with the sound world
• Drumaxx, Image-Line’s tasty physical-modeled drum synth
• Native Instruments Battery, the sampled drum engine
• Eclipse, the free IDE, for Java coding in this case
• Community Core Vision and reacTIVision (based on our previous info, at least), free and open source community-based projects for making the interfaces you see in movies happen in real life.

Another terrific video, which gets into generating a pattern:

Now, I could say more, but perhaps it’s best to watch the videos. Normally, when you see a demo video with 10 or 11 minutes on the timeline, you might tune out. Here, I predict you’ll be too busy trying to get your jaw off the floor to skip ahead in the timeline.

At the same time, to me this kind of visualization of music opens a very, very wide door to new audiovisual exploration. Christian’s eye-popping work is the result of countless decisions – which visualization to use, which sound to use, which interaction to devise, which combination of interfaces, of instruments – and, most importantly, what kind of music. Any one of those decisions represents a branch that could lead elsewhere. If I’m right – and I dearly hope I am – we’re seeing the first future echoes of a vast, expanding audiovisual universe yet unseen.

Previously:
Subcycle: Multitouch Sound Crunching with Gestures, 3D Waveforms

And lots more info on the blog for the project:
http://www.subcycle.org/

AudioGL, a project teased in videos first in April and then again last week, is a new concept in designing a user interface for real-time music creation. Visuals and sound alike are generative, with the rotating, 3D-wireframe graphics and symbolic icons representing a kind of score for live synthesized music. The tracks in the video may sound like they’ve been pre-synthesized, polished, and sampled from elsewhere, but according to the creator, they’re all produced in the graphical interface you see – what you see is what you hear.

The newest video, released this week, reveals in detail the project’s notions of how to make a 3D, live music interface work. The UI itself is similar to other graphical patching metaphors, but here, like exploding a circuit diagram in space, routings and parameter envelopes are seen and edited in a freely-rotating view in three dimensions rather than on a flat plane.

There’s a reason interfaces like this have been few. Computer displays and pointing methods tend to be heavily biased to two-dimensional use, modeled as flat planes like pieces of paper. Working in two dimension is simply easier; there’s no reason you can’t take another layer of parameters and represent it on a two-dimensional interface. And rotating around in 3D space can make it difficult to keep your bearings.

Those challenges, though, don’t make this less interesting – they make it juicier and more delicious as design problem and stunning, futuristic musical model. Freed in three dimensions, a complex set of envelopes and parameters has room to spread out visually, making a kind of spatial score. This particular project strikes an interesting balance between traditional, iconic UI – operators are represented with graphic symbols – and more free-flowing geometry representing the sequencing and envelopes. To me, the latter is more compelling, but putting the two together may make the program more flexible and familiar to users of other music software.

What could knock you out of your chair, though, is the sheer depth of the software teased in the video. This is no simple tech demo: it’s an attempt to build an entirely new, live-synthesizing music tool from scratch in 3D. It’s like the International Space Station of music software, assembled in some void. I got a couple of tips on this today, and some are even wondering if it’s real.

It appears to be very real. Whether this particular tool is usable or not to me almost isn’t important: a spectacular failure in this arena would even be useful. Anyone waiting for some sort of “singularity” in music tech, I think it’s coming: it’s just going to be a singularity of human software ingenuity, explosive creativity and invention from independent developers. I can’t wait.

Stay tuned to find out more about this particular project.

See also the earlier video (not able to grab the embed code for some reason).

Thanks, Bodo Peeters, among others, for the tip.

A new tool could be for the expressive, not just the lazy. That’s the read of Auto-Tune for guitar, and it makes me excited to see what people will do with it. It could be the advent of the true digital guitar.

Antares teased their efforts to bring Auto-Tune technology to guitars earlier this month, having gotten as far as working proof-of concept. (See Harmony Central’s exclusive video above, and Axetopia, Synthtopia.) I hadn’t worked out anything intelligent to say about it, perhaps because I was cowering in a corner in fear.

As a technologist, I have great respect for what Antares does, and their portfolio goes far beyond just the flagship vocal pitch correction. But suffice to say, Auto-Tune has been used in recording in some pretty unpleasant ways – the fault of the user, not the software, I’d argue. It’s regularly applied in order to suck the life out of great, perfectly-tuned singers, as well as to cover for people who can’t really sing, to the point that producers seem to not understand what the sound of a human voice is in all its complexity. (Case in point: Glee. The talented cast sounds incredible live and onstage, and like they have android stand-ins when they’re on the show. In fact, if you disagree with those uses, please – go use some of Antares’ terrific software for good, not evil, and I’ll write about it.)

Auto-Tune as a name, then, has come to symbolize a revolution, an extraordinary blockbuster of software – and the butt of a joke. So, it’s hard not to see a product called “Auto-Tune for Guitar” and carry some of that bias. Sometimes, as writers we actually need our readers to add some perspective.

Auto-Tune for Guitars could likewise be misused to smooth out some of the guitar’s natural intonation subtleties, though I think the danger is far less so than it is with the voice. But it’s more than that.

Reader Jesse Engel reflects on what it could mean. He notes that the significant advance is building the intelligence into the guitar, not just the computer, and that applications could be varied:

Don’t know if you saw this, but Antares has taken a fresh swipe at HEX guitar, putting a processor in the guitar and using it to do some more modern (Auto-Tune, emulation, etc.) processing. [Ed.: Hex refers to the practice of adding individual pick-ups to each of six strings. -PK]

The hex has been around for a while, but it’s a big deal to use it in this way for guitarists since you don’t need to try to do any polyphonic pitch recognition. Literally direct note access. Also, signals add nonlinearly, so effecting each string individually has a different sound than doing emulation on the mix.

The tech looks like it will help a lot of people fake being better than they are (especially bending to the right note), at the expense of the beautiful imperfections of great playing, but the potential of using hex pickups in these new ways is fun to think about.

The digital guitar has been a vision for a long time, from working out MIDI output to multichannel output. Gibson has been the name behind many of those efforts. Back in January 2004, Wired ran a glowing portrait of Gibson’s efforts in print:

The 100-Megabit Guitar: Gibson’s maverick CEO wants to shove Ethernet up your ax and rock the music world. [Wired 12.1]

It’s worth reading the whole article; the technical limitations of the Gibson system immediately come to light. Suffice to say, that vision never quite came to fruition; Wired even this year claimed that the project had been killed – at least at Gibson. None other than Adrian Freed, OpenSoundControl and alternative instrument design guru at the University of California Berkeley’s CNMAT research center, led the group – he, his colleagues, and his many students go right on innovating with or without Gibson. Updated: I’m not able to find the reference for that story, which I read in print. See comments for commentary by Adrian Freed, who sees otherwise.

At the time, CEO Henry Juszkiewicz, oddly speaking in the third person, pronounced, “Progress will happen. If Henry Juszkiewicz didn’t build a digital guitar, I can assure you the digital guitar would still happen.” That prediction may prove prescient.

The 2007 video below shows the debut of Gibson’s HD.6x-Pro Digital Les Paul – working with individual strings. I also saw a demo with Gibson, Intel, and Cakewalk that used each string in a surround speaker diffusion. It was a psychedelic effect, if not necessarily the most practical demo, but proof that a technology like this could have many uses.

For their part, here’s how Antares describes their technology. Notice that they aren’t only talking intonation, but other applications, as well.

Incorporating our world-renowned Auto-Tune pitch detection and manipulation along with our proprietary modeling technologies, ATG-6 is an entirely DSP-based suite of functions that offer everything you’ve always wanted from a guitar, along with capabilities you never imagined possible. From flawless intonation to astonishing tonal flexibility to alternate tunings that open up entirely new areas of inspiration and creativity, ATG-6 technology seriously expands the flexibility and range of the electric guitar while letting you continue to play your own way.

… Using our new Solid-Tune™ Intonation system, an ATG-6 equipped guitar constantly monitors the precise pitch of each individual string and makes any corrections necessary to ensure that every note of every chord and riff is always in tune, regardless of variables like finger position or pressure or physical limitations of the instrument. As a result, listening to a guitar with Solid-Tune is a revelation, offering a purity of intonation that has simply never before been possible.

Of course, Solid-Tune is smart enough to know when you want to manipulate pitch, so you can play bends and vibrato exactly as you always do. In fact, Solid-Tune Intonation makes it even easier to bend to the right pitch every time.

Antares ATG-6

Updated: Chris Randall chides me on Twitter (and I agree) for not mentioning Roland, specifically — that’s the reference above in Jesse’s from-the-hips comments to “hex” guitar. Roland has built a whole business around products that track notes played on a guitar, adding polyphonic pitch shifters, open tunings, note-by-note replacement, MIDI output, and even DSP effects processing. The difference in the Roland offering is that Roland has done all this work in a separate processing box you connect to their pick-up; Antares appears to be promising something that’s all-in-one in the guitar. And the analysis Antares is doing may well prove more sophisticated than what we’ve seen in the past in terms of distinguishing, say, a bend from different notes. That could open up additional and radically-new expressive possibilities, even if the underlying fundamental concept is more or less the same.

On the other hand, the other difference with the Roland offering relative to both Gibson’s past attempts and Antares’ upcoming ones: Roland successfully shipped and sold theirs. Until Antares does the same, advantage: Roland. We’ll be watching.

If your music and sound collections seem like outwardly-expanding universes, two new tools promise to bring order by representing media as virtual planets and stars. One works on albums and tracks on the iPad; the other uses computer-aided analysis of loops and samples (not just music) on Windows. One will make your eyeballs pop; one might help you manage gigs of samples for a game design project.

Built in the open-source framework Cinder by an all-star team of media artist-designers (Ben Cerveny, Tom Carden, Jesper Sparre Andersen and Robert Hodgin), Planetary should satisfy space nuts and eye candy lovers. The metaphor is pretty direct: artists are stars, albums are planets around the artists, tracks are moons around the planets, and you can filter “constellations” by letter. That means the actual structure is heavily hierarchical, actually, in the tradition of iTunes (and, before it, its predecessor SoundJam). I’m not sure what happens with, say, compilations. But let’s face it: the real draw is that it’s incredibly beautiful to look at. I’d be just as entertained looking at a visualization of my system folder if it looked this pretty.

For now, Planetary is some fascinating eye candy with at least basic playback capabilities, iPad-only. That brings some good news – Airplay wireless works, and since it makes use of standard media code, even features like Last.fm scrobbles function. It also brings some bad — while Apple added support for libraries to third-party apps, Home Sharing isn’t included, so you’re limited to what’s on your iPad. Playlists aren’t supported, either. But hook this up to a projector or large screen TV with some of your favorite music, and I don’t think you’ll be complaining. And as a free tool, it’s incredible.

Planetary is available now; free for the iPad. As seen on creativeapplications.
http://planetary.bloom.io/
iTunes link

Less pretty, but with greater facilities on the utility side, is the Windows-only Soundtorch. (Thanks to Kristian Gohlke for the tip!) Visually, it offers a similar metaphor: media assets live on a continuous plane. Functionally, though, it’s more algorithmic than hierarchic, using something called the Computer Aided Sound Exploration engine (C.A.S.E.). The set of algorithms, which the creators say were based on evaluation of human listening, performs a sophisticated set of extractions of some 600 features from each sound file.

Rather than limit itself to albums and tracks, C.A.S.E. is tuned for audio files and loops. It’s fast enough that it can plow quickly through gigs of material. So, if you’re on Windows and have amassed an enormous collection of loops, samples, field recordings, sound effects, and the like, Soundtorch will use C.A.S.E. to first map all those relationship, then visualize them. You can use the mouse to produce new collections of assets, map relationships visually, export those relationship to XML, copy sounds to the clipboard, export to WAV, or open them in Windows Explorer. That is, all that eye candy is a genuine interface, not a barrier between you and what you might do (as so often happens with these sorts of experimental interfaces).

In fact, you might argue that, despite outward appearances, Soundtorch is entirely different from Planetary, but they share one common conceptual assumption. Related media “orbit” or attract to common materials. The difference is that Soundtorch is relational. In Soundtorch, if you “magnetize” a file, it – and any similar files – become attracted to attractors called “magnets.”

As is appropriate searching for media, the “torchlight” metaphor shines a light through files. Everything under the light plays back simultaneously, so you don’t have to audition sounds one at a time. (That sounds slightly terrifying to me, but I have to spend more time with it in an actual library.)

The creators describe the magic thusly:

Have you ever listened to a sound and felt that there was a similar one somewhere on your hard disk? And the sound you can’t find would just work so much better right now? Well, Soundtorch also remembers all sounds that you ever listened to. Just select any sound on Soundtorch, and let the system suggest the most similar ones from your whole collection.

In other words, SoundTorch is as much about what you can’t see as what you can – the intelligence to determine similarity behind the scenes. Check out the tech talk in the video above for more information on how “aurally and visually-enhanced audio search” could also apply this technology. More research at:
http://www.accessive-tools.com/

Soundtorch 2.0 entered a free public beta last week. It was developed in Microsoft’s C#-based XNA framework.

Grab the download:
http://soundtorch.com

Finally, if you want to hear the “Optimist” track by Zoe Keating without that voiceover and just enjoy Planetary’s gorgeous visuals, here you go:

From innovation in the visual interface to the intelligence underneath that changes how the computer interprets relationships between files, finally, there’s hope. Music and sound might not forever be trapped in views borrowed from spreadsheets, tables modeled on the needs of accountants 30 years ago.

Nearly as long as we’ve had electronics, musical inventors have tried to imagine new electronic instruments. In the crowded world of new instrument design, the Margaret Guthman Musical Instrument Competition has emerged as a key prize for the best work, with creations battling fiercely for attention.

But in the oddball world of sound and music, how do you judge a winner? As a starting point, organizers this year asked the judges what they personally found important. With an expert panel including synth pioneer Tom Oberheim and reacTable creator Sergi Jorda, those answers are themselves revealing.

As for the competitors themselves, even with eclectic entrants, one theme stands out. Human gesture and performance presence is a strong dimension of the winners. And in perhaps the most promising first-prize winner yet, research begins to crack the code of how to make real gestural analysis work, even allowing everyday objects to become musical instruments.

To help us learn more, Competition founder and Georgia Tech Music Technology director Gil Weinberg grants CDM a window into the philosophy of some of these leading technologists, and introduces us to this year’s winners.

The Winners

First Prize: MO, Interlude Consortium. Everyday objects become novel gestural interfaces.

From the project site:

The MO tangible interfaces are a series modules to capture various gestures, from motion to touch. The central module MO contains motion sensors (3D accelerometers and 3axis gyroscopes) and transmits the data wirelessly. Moreover, two accesorries, i.e. other sensors can be added to both side of MO.

Second Prize: MindBox Media Slot Machine, Humatic Berlin. A vintage slot machine is transformed into a compositional interface.

Personnel:

Christian Graupner , Humatic
….media artist, director, composer

Roberto Zappalà
… performer, choreographer

Norbert Schnell, IRCAM — Centre Pompidou
… interactive music & sound design

Nils Peters, Humatic
…system developer and software artist.

Third Prize: Samchillian Tip Tip Tip Cheeepeeeee, Leon Gruenbaum. It began as an ergonomic computer keyboard, but years of layered work on relative pitch makes it an instrument – a bit like a macro keyboard for composition.

Honorable Mention: Hexenkessel, Jacob Sello. A conventional acoustic timpani is both projection surface and multi-touch input.

From the creator’s description on the video:

The Hexenkessel is a modded 22″ timpani using LLP multitouch technology for control of live-electronics & dmx-light. the realisation of the instrument involves a modified led-projector, webcam and IR-Lasers. the programming is done entirely using max/MSP/Jitter + CCV. The instrument-hack is non-destructive and costs less than 300$.The instrument is intended for the use in multimedial stage performances and innovative concepts of new music.

Pioneering Judges Offer Their Philosophies

A musical instrument design may seem like subjectivity atop more subjectivity, a meeting of the aesthetic of the object with personal musical expression. Judges were asked, therefore, to describe the philosophy they brought to the contest. The reason, explains organizer Weinberg: “To steer it away from general statements – this is the better instrument than this – to make it more personal, about the judge’s opinion and artistic manifesto and instrumental manifesto.”

Tom Oberheim, the man who created the first polyphonic synth product, responded:

The first thing that I look for in a new musical instrument is its musicality. This means where appropriate: does is sound good, is it playable, does it add to the music making language. Then I consider if the device has some sort of universality; in other words, can it be used by a variety of musicians from different backgrounds. Finally, I consider the ease with which the device can be learned.

Sergi Jorda, creator of the reacTable tangible interface:

The ultimate goal for any new instrument could arguably be the potential to create a new kind of music. In that sense, baroque music cannot be imagined without the advances of sixteenth- and seventeenth-century luthiers, rock could not exist without the electric guitar, and jazz or hip-hop, without the redefinitions of the saxophone and the turntable. Yet, this extremely ambitious objective is often beyond the reach of its creator (eighty years separate Adolphe Sax from Coleman Hawkins, and no less than thirty go by between Les Paul and Jimi Hendrix). Being a bit more pragmatic, as a performer, my goal when constructing the instruments I will play is clear. I need instruments that are enjoyable to play and that mutually enhance the experience when playing with other musicians. Thereby allowing me to create or co-create music that will surprise me as much as possible, that will keep revealing little hidden secrets at every new performance. Music not necessarily better, nor worse, than a piece that I could compose in a studio, but music, in essence, that could not have been created in any other possible way. As a ‘professional’ luthier, I need to take some additional considerations into account, but the overall goals do not change: my aim is to create instruments which people can enjoy playing; instruments that will be able to enrich and mature the performers’ experiences in any imaginable way; instruments that allow scope for the performer (particularly in the case of a non-expert user) to be proud of the music created. In order to survive in the extremely demanding instrumental ecosystem, any new instrument should clearly excel in something. It should either be able to do one thing that no other instrument could or, at least it should do it better (whatever this can be and whatever “better” may mean). My last advice would be that when envisaging new instruments one should not only concentrate on the instruments’ sonic capabilities, on their algorithmic power or on the amount of sensors used. One should also be especially careful about the instruments’ conceptual capabilities, and consider how new instruments impose or suggest new ways of thinking to the player, as well as new ways of establishing relationships, new ways of interacting, new ways of organizing time and textures; new ways of playing, in short.

Jason Freeman, a composer, technologist, and Assistant Professor at Georgia Tech:

For me, new musical instruments are significant for their potential to transform our experiences with music. They may enable us to create new acoustic or electronic sounds not previously possible. They may encourage us to think about musical content, structure, and hierarchy in unusual ways. They may suggest new methods of musical collaboration, performance, or education. And they may make musical creativity more accessible to everyone. I am interested in instrument makers who have thought deeply about their work from technical, musical, and design perspectives to create musical instruments that transcend novelty to suggest new paradigms for musical creativity.

A Chat with the Organizer

Now in its third year, the Guthman competition has become a coveted award. As a result, says organizer Weinberg, who is director of the hosting Georgia Tech Center for Music Technology, quality and quantity were up in entrants. And, he says, he feels that entrants have transcended some of the typical designs in the field.

“New interfaces for many [means], let’s think about an object that we didn’t use before, and some kind of gesture, stick on some sensors, make some music … But I think the winners of our competition were outside of this realm, really innovative, completely new approaches for playing music,” he says.

On the prize-winning MO tangible interface:

In a section of the performance, they took a ball – a soccer ball – and did some [musical] gestures with it, threw it … moved it … on the hands, on the floor. Each one of these gestures was recorded with the gesture recognition. And then they actually threw the ball to the audience. The audience members started to throw the ball back and forth. If you threw it in a particular way, it made a particular sound — and everything’s wireless, completely — if you threw it back and forth in a different way, it made a different sound. It was really fun; people threw the ball at each other, threw the ball back at the stage. And all made music that was pretty cool to listen to.

Basically, the instrument becomes an intelligent entity. It can sense similar but different gestures and create something smart and relevant musically.

On the slot machine:

The gesture is mostly visual — the intelligence here is of the human performer. He makes his own gestures, accompanied by sounds. And it allows you to manipulate and change [the sound] — and get some surprises, because it is a slot machine, after all.

You can play, explore it. He was able to very expressively pet and touch and click and manipulate the slot machine to create some very nice — not only musical outcomes, but visual outcomes. In some cases, this guy is lying in the sea and making gestures in the sea. Sometimes he’s hanging stuff on the walls, and making sounds with his mouth. Sometimes it’s basic stuff that you can manipulate in real time, with a pretty unique interface — it’s not a monome, it’s a slot machine. It surprises you.

On the Samchillian:

Some instruments – controllers – have this short or sometimes long learning curve, but once you get to a certain point, you know it, and that’s what it can do. And you cannot get better at it. I think the Samchillian is really an instrument with a learning curve that’s very long, and just like other acoustic instruments, violin, piano, there’s a wide range of [technique]. And this guy was really a virtuoso with this instrument. He was able to play chords, all kinds of arpeggiators.

What I liked about it is it’s an instrument more than a controller. There’s always more to learn about how to become better with it. And I think that’s valuable.

Notably, Weinberg has no illusions about the challenge of making new instruments. It’s no accident that the winners were typically the result of years of development and evolution. “I don’t think any of the great instruments were invented in months,” says Weinberg. “It’s a lot of iteration, a lot of building… only a few are good enough to stick.”

And perhaps the great electronic instrument, while getting nearer, hasn’t yet been created. Weinberg says one example of a new instrument design that doesn’t work particularly well is the legendary Theremin – it’s beautiful in the hands of only a couple of artists, but generally a design that stumps musicians and is hard to play.

Looking at the winners this year, though, there are ideas on which new work can be built, not just impressive one-off instruments but real research into handling pitch and gesture. That, at least, should present a bright future. But with the competition heating up, aspiring engineers may want to get started on those designs now.

Thoughts? Questions about the work? Let us know.

More on the MO tangible interfaces from the IRCAM-based Interlude:
MO Interfaces

That work isn’t yet available for download, but an “augmented score viewer” is.