Today, I’m in London doing a hands-on workshop on visual metaphors for music, and covering various topics filed under “synesthesia” at Music Tech Fest. It seems appropriate, with the subject matter on the brain, to revisit the topic of visuals and music in a series of posts.

When you make hardware, with knobs and faders, you’re constrained by physical space – the amount of room on a circuit board, the radius of a knob cap, the size of your fingers. But before you get there, the first step is to sketch an idea. Imagine if you could do that with a marker and paper and ink.

SketchSynth is the latest attempt at a drawable set of controls, letting you turn an illustration on paper into something you can actually use to make music. It’s not the first – this dream of being able to make things come alive with nothing more than a magical pen is an old one – but the execution looks superb. There are two basic approaches to the idea: one is to use some sort of conductive ink to turn the drawing itself into a sensor, and the other is to point a camera at the drawing and calculate where a user makes contact with the drawing. SketchSynth opts for the computer vision approach, by way of OpenFrameworks and the old standby of free and open source vision, OpenCV. (Kyle McDonald’s ofxCV does the heavy lifting.)

Conventional vision fits the task well: faders and knobs respond as expected, even though they’re only ink on paper. While it’s a drawn interface, and could look like anything, the behavioral metaphors all come from hardware: there are sliders, momentary buttons, and pots. Place the tool in “edit” mode, and the computer analyzes what you’re drawing; in “play” mode, the camera tracks your finger as you manipulate what you’ve drawn. The project goes one step further than many that have come before, by overlaying a projection calibrated with your drawing for interactive visual feedback as well as sonic. Sound in this case is provided by Pd, but OpenSoundControl (OSC) messages let you connect to other musical (or visual) tools. See more in the making-of vid, at bottom.

Creator Billy Keyes is working with the right mentor, too, completing this as research for Golan Levin, who has long explored the relationship of drawing and musical gesture. His Sonic Wire Sculptor was a seminal creation in connecting drawings to sound, using a tablet to produce three-dimensional “wire” structures and corresponding sound synthesis. His Messe di Voce performance piece neatly reversed the relationship, using the voice as the input to animate drawings and illustrations. More of Levin’s work at flong.com. See also: composer Xenakis’ UPIC, which lives on today as IanniX, a tool getting a lot of development attention.

Keyes’ project, already getting lots of blog buzz, is notable for its practicality and immediacy; it seems a tool many others might use and build upon rather than a single piece of art.

It’s lovely to see projects, particularly academic projects, come to some form of completion and clarity. Speaking of completion and clarity, at some point a proper survey of drawn musical interfaces seems a must, but that will have to come another day. Where can I get a nice full English breakfast?

I’m late to the party, but hat tip in particular to Creative Applications

Project Page at golancourses.net; Linux source is promised soon

I’ve covered a number of these sorts of projects over recent years.

Paper, Drawing as Musical Controller: A Round-Up (including a number of paper examples)

Drawing Sound: Crazy Touch Interface Sound Experiments with Usine, PC (using only a screen – and thus producing a very different experience of drawing

Imaginary Instruments: Marker and Paper as Controller (this 2009 project is almost a direct analog to SketchSynth, minus the projection)

iPhones, Pencils: Hand-Drawn Music Interactions, Tokyo Subway Mobile Jam (this one isn’t quite the same as the others, using pencil and paper to design an interface for a screen – but it’s nice to recall that you can do that, as well)

Pen and Paper as Graphical, Digital Music Score (from earlier this year, a more open-ended design)

Video: Sonic Wire Sculptor, 2003:

Mobile computing has already had an enormous impact on music making. A modern phone or tablet (and yes, most often, these come from Apple) is capable of out-performing a lot of dedicated hardware and easily runs the synths and workstations that required state-of-the-art desktops just a decade or so ago.

But what if this same computing power – low-energy, low-cost chips – could be in other music gear, too? They could offer significant advantages. Bare boards, while on their own not quite road-ready, can wind up in music-friendly housings. (Think stompboxes – without stomping on your phone, or buying a big, silly dock.) You’ll never have to sign a contract with a phone company to get one, or stop your latest song sketch to take a call. And they could be significantly cheaper: the Raspberry Pi isn’t quite ready for mass consumption yet, but it has already begun shipping at US$25, meaning the entire computer costs what a phone car charger might.

In fact, much as the original personal computing revolution took computing to masses of new audiences, this could extend music computational power worldwide. We’re not just talking strange DIY software, either – these boards run Linux, meaning a lot of off-the-shelf music software will “just work,” including even some fine commercial entries.

If you’re ready to stop dreaming and start making music, now’s a great time. CCRMA at Stanford in the United States and STEIM in Amsterdam, NL have each been working on development. STEIM even has a workshop scheduled for June, taught by Edgar Berdahl (CCRMA) and Florian Goltz (DE):
Satellite CCRMA: Interactive design with open embedded computers

The instructors offer some great inspiration about what this is all about in their description:

These small computers combine the connectivity of a laptop with the computational power of a high-end smartphone; however they are less expensive than either and fit inside a cigar box. We will dedicate much of the workshop to prototyping new functional artworks, for example: musical instruments, effects processors, interactive installation works, and anything else you can imagine that requires high computational power in a small, inexpensive footprint.

In the broader sense this workshop deals with interaction design: What happens when human behaviours meet those of machines?

But even if you’re not able to get to California or Holland, you can give the software a try. The BeagleBoard is now supported by a custom distro; the Raspberry Pi seems a logical next frontier once it starts shipping. With Pd (Pure Data) included, you can even copy-and-paste instruments and effects like synthesizers, step sequencers and drum machines, and granulators built by a broad community – even without necessarily being a master patcher yourself. (And then, when you do want to modify the way it functions or sounds or gets controller, you can.)
https://ccrma.stanford.edu/~eberdahl/Satellite/

It’s not all beginner-friendly yet, but these hacklabs seem the perfect way to begin to move in that direction, as more people test the solutions, gather data on how different patches perform, and make tweaks and write documentation.

Pure Data is a wonder: a free and open source environment for creating your own musical and multimedia creations with graphical programming, from Miller Puckette, the original creator of Max. You can produce everything from interactive sequencers and drum machines to synths to video performance tools by connecting patch cables visually, and you can run on virtually any platform, from BeagleBoards and Rasberry Pi to Mac, Windows, and Linux desktop. Via libpd, you can target other development languages and environments, embed engines in games, or work with Android and iOS.

It’s so important, in fact, that CDM invites Hans-Christoph Steiner, one of the key developers of Pd-extended, to give us a tour of what’s new. (Note: because Pd-extended includes various additional objects or “externals” that Pd Vanilla lacks, you should be careful when building patches for libpd. What I like to do is use Pd-extended as my editing environment, then double-check patches by opening them in Vanilla to make sure I haven’t accidentally used an object that’s not part of the bare-bones version. I can then substitute an object, copy an abstraction, or if necessary build that external.) -Ed.

The Pd-extended 0.43 release has been brewing an extra long time, about 18 months now, mostly because there are lots of big improvements. We wanted to make sure we got it right, so your patches all work, but the improvements all shine, so its taken a while. It’s now solidly beta, so we’re looking for testers. Download a beta build to try here:

http://puredata.info/downloads/pd-extended/releases/0.43.1

First off, the pd-gui side of Pd has been rewritten from scratch. The focus for most of the recent work has been on the editing experience, making your patching experience as productive and flexible as possible. To give some background, Pd has always been made up of two programs: pd is the core engine and pd-gui is the GUI. Since basically all computers now come with multiple CPU cores, this means that pd-gui will usually run on a separate CPU core than pd, so they don’t step on each other’s toes. pd can entirely take over its own core. If you want to make your patch use more CPU cores, then check out the [pd~] object introduced in the last release, but fine-tuned in this one.

There are so many ideas for making a better editing experience in Pd; this release makes big strides to address the editing experience. There are new features like Magic Glass, Autotips, Autopatch and Perf Mode, all available on the Edit menu.

• Magic Glass lets you magically see the messages as they pass through the cords. Just turn it on and hover above a cord, and you’ll see the messages as they go by. You can even look at signal/audio cords.
• Autotips gives you tips about what an object does, what its inlet expects, and what comes out of the outlets.
• Autopatch mode automatically connects objects as you create them.
• Perf Mode, is a mode for performance that makes it harder to accidentally close windows that are part of your performance.

A whole new Pd Window

The Pd Window is also majorly overhauled. First of all, it’s fast. Much much faster than the old one. You can now print thousands of messages per second to the Pd Window and still edit your patch. No more will an accidental dump of info cause the GUI to freeze up (well, okay, maybe if you send 10,000 messages/second, but that is way too many). There are also five levels of printing messages to the Pd Window: fatal, error, normal, debug, all. If you are only interested in fatal errors, switch the Pd Window to 0 – fatal, and you’ll only see the worst problems. You want to see every single message to debug? Switch to 4 – all, and you’ll drink from the firehose.

There is also the new log library, which lets you easily send messages for those different levels. And all messages logged with the objects from the log library are clickable: when you Ctrl-Click or Cmd-click (Mac OS X) on the line in the Pd Window, it’ll pop up the patch where the message came from, and highlight the specific object that printed it. That even works for many messages from other objects, as well.

The Pd Window also includes very basic level meters for monitoring the input and output levels. And for those who want to play with the GUI in realtime, you can type Tcl code in the Tcl entry field, and directly modify and probe the running GUI.

Customize the GUI with Plugins

One thing that you can do now is customize the GUI using GUI plugins. You can change all sorts of colors, some fonts, and many behaviors. Want to create a new object when you triple-click? Try the tripleclick example plugin Want to make the patch cords disappear when you leave Edit Mode? Check out the “only show cords in edit mode” example. Those are the simple ones. There is also Tab Completion, a search engine for the docs, a category browser for the right-click menu, a buttonbar for creating objects, and more.

You can find many GUI plugins in the new section of the downloads page as well as documentation for making your own. (What kind of GUI plugin will you write?)

Write Pd objects in more languages

Traditionally, Pd objects are written in Pd (abstractions), C and some in C++. This new release includes two “loaders”, Lua and Tcl, which allow you to write regular Pd objects in either Lua or Tcl. Pd is not the best for processing strings, so if you need to do that, you can now easily use Lua or Tcl, both very easy scripting languages for working with strings. Lua is often used for OpenGL work, so you can also run Lua objects to work in conjunction with Gem. Also, the Tcl loader lets you write GUI objects in pure Tcl, no C needed.

Multi-processing, Pd-style!

The [pd~] object now works out of box. In case you missed the introduction of the [pd~] object in the last release, we’ll introduce you now. [pd~] is Pd itself incapsulated into an object. You can run any patch inside that instance of Pd, the difference is that the Pd in the [pd~] object runs in a totally separate process. So if your computer has multiple CPU cores, which basically all computers do these days, then the Pd process inside the [pd~] object will run on a separate core. This means you can have a heavy Pd patch spread across multiple cores or CPUs. Or for people who work with video and audio together, you can have one instance for video running at a normal priority, then another instance for audio running at a high priority to make sure there aren’t clicks in the audio caused by heavy video processing.

Autotips, generated from help patches

This release also provides a new “autotips” feature to provide instant information about objects and their inlets and outlets. It is one of the first new developments to showcase all of the meta data that is now included in all of the help patches. (Check out the [pd META] subpatches.) When you hover above an inlet or the object itself in Edit Mode, you’ll see a short text description pop up on the lower left corner. But, of course, using a GUI plugin, you could customize how they are displayed to make it how you want to see it. If you want to add autotips to your object, then just add a [pd META] subpatch to your objects’ help patches, and fill out the description, etc. Voila! They’ll have instant information.

What’s next?

The core pd process still handles a lot of the GUI stuff, but we are working on splitting that out for the 0.44 release. That is a big chunk of work, but it will also bring big gains. In particular, it means that it will be possible for people to write their own GUIs for Pd, covering not just the display of the patch, but also the editing, and everything else. You like OpenFrameworks, Python, iOS, JUCE, Qt, etc.? Write your own pd-gui using the toolkit of your choice. That’s the idea at least. That will take a solid chunk of work, so we are looking for people to join that effort.

Try it yourself:

http://puredata.info/downloads/pd-extended/releases/0.43.1
http://puredata.info/downloads/pd-extended

Where to learn Pd:

Resources to start learning

-Hans-Christoph Steiner for CDM

How do you visualize the invisible? How do expose a process with multiple parameters in a way that’s straightforward and musically intuitive? Can messing about with granular sound feel like touching that sound – something untouchable?

Music’s ephemeral, unseeable quality, and the ways we approach sound in computer music in similarly abstract ways, are part of the pleasure of making noise. But working out how to then design around that can be equally satisfying. That’s why it’s wonderful to see work like the upcoming Borderlands for iPad and desktop. It solves a problem familiar to computer users – designing an interface for a granular playback instrument – but does so in a way that’s uncommonly clear. And with free code and research sharing, it could help inspire other projects, too.

Its creator also reminds, us, though, that the impetus for all of this can be the quest for beautiful sound.

Creator Chris Carlson is publishing source code and a presentation for the NIME [New Interfaces for Musical Expression] conference. But this isn’t just an academic problem or a fun design exercise: he also uses this tool in performance, so the design is informed by those needs. (I’m especially attuned to this particular problem, as I was recently mucking about with a Pd patch of mine that did similar things, working out how to perform with it and what the interface should look like. I know I’m not alone, either.)

The basic function of the app: load up a selection of audio clips, and the software distributes them graphically in the interface. Next:

A “grain cloud” may be added to the screen under the current mouse position with the press of a key. This cloud has an internal timing system that triggers individual grain voices in sequence. The user has control over the number of grain voices in a cloud, the overlap of these grains, the duration, the pitch, the window/envelope, and the extent of random motion in the XY plane. By selecting a cloud and moving it over a rectangle, the sound contained in the rectangle will be sampled at the relative position of each grain voice as it is triggered. By moving the cloud in along the dimension of the rectangle that is orthogonal to the time dimension, the amplitude of the resulting grain bursts changes.

You can see how Chris is imagining this conceptually in a sketch he shares on his site:

An extended demo shows in greater detail how this all works:

Chris is a second-year Master’s student at Stanford University’s Center for Computer Research in Music and Acoustics [CCRMA] in California. The iPad version is coming soon, but you can get started with the Linux and Mac versions right away, and even join a SoundCloud group to share what you’re making. You’ll find all the details, and links to source code, on the CCRMA site. (And if someone feels like building this on Windows, you can save Chris the trouble.)

https://ccrma.stanford.edu/~carlsonc/256a/Borderlands/index.html

I also love this Max Mathews quote Chris shares as inspiration:

Max Mathews, in a lecture delivered at Stanford in the fall of 2010
“Any sound that the human ear can hear can be made by a sequence of digits. And that’s a true theorem. Most of the sounds that you make, shall we say randomly are either uninteresting, or horrible, or downright dangerous to your hearing. There’s an awful lot to be learned on how to make sounds that are beautiful.”

Beyond the technology, beyond this design I admire, anything that sends you on the path to making beautiful sound seems to be a worthy exercise. It’s a challenge you can face every day and never grow tired.

http://modulationindex.com/ [Chris' site, with more information]

Thanks to Ingmar Koch (Dr. Walker) for the tip!

Inside this compact white box lurks a lot of sonic power and technical prowess. Perhaps that explains why the newest version of the open source Shruthi-1 now sports a crazy-badass wolf dog cartoon with glowing eyes.

Since its launch, the Shruthi-1 has gradually evolved new features, with a fairly sophisticated combination of hardware and extensive software. At its core, it’s a “hybrid” synth with digital/virtual analog oscillators and real-analog filter. The digital oscillators allow it to change character, for classic virtual analog subtractive, or wavetable, FM, phase distortion, and vowel synthesis.

The big news with the filter is that the various flavors of filter board are now discontinued. Sadly, the wonderful CEM3379 filter chip is just too rare to have a long-term home in this synth; the Shruthi-1, like other synths (the Dark Energy being a recent example) has hit chip scarcity.

But in its place is something else new and wonderful. The SSM2164 (uh, that doesn’t roll of the tongue, but yes, that filter) combines 15 filter responses with four resonance models, for a total of 60 possible filter sounds. See also the Oberheim Matrix-12 and Xpander for pole-mixing techniques. You also get self-oscillation, and even a Korg DS-inspired diode waveshaper. (I won’t go into any more detail, as maker Mutable describes this in gory precision.)

In a way, the Shruthi-1 – despite its minimal knobs – really hides a semi-modular instrument, one with its own built-in arpeggiator, modulation matrix, duophony, rhythmical oscillator cycling, and lots of other features. If there’s a technical feature possible – just about any feature – the Shruthi-1 does it. Combined with that terrific filter and digital grunge, I think it’s a terrific deal in desktop synths.

In fact, my only real criticism is, it does so much, you’re likely to be stuck paging through menus – or should focus on MIDI programming – because of the minimal controls. I can see why members of the Shruthi-1 community have been building expansion controllers for it to get more hands-on control. But on the other hand, to me, it’s entirely worth the tradeoff going compact – even with a complex menu system. The result is a synth that’s far more affordable and portable. It’s a natural for MIDI users.

The new design is also unquestionably the best-looking Shruthi yet, thanks to translucent white plexiglass and white LED lighting. 130 € buys you the kit (plus another 20 € in parts), but I’d strongly recommend the pre-assembled version if you’re less familiar with bigger builds. There are a lot of parts and two boards, plus a pretty white circuit board that will look grimy if you don’t solder carefully. For experienced builders, it should be a great assembly process into which you’ll want to sink your teeth, wolf-like. But for less-experienced builders – or just people who want to get straight to making sound – I think 349 € is a small price to ask. (A carry bag and European wall wart are included.) Just grab the pre-built version fast; because they’re hand-assembled, they won’t last long.

Full details:
Shruthi-1, 4-Pole Mission edition

Be sure to have a listen to the way the new stuff sounds:

The other important thing to mention about the Shruthi-1 is that it’s a fully open source synth. (An earlier version prohibited commercial use, but it now uses a more permissive license.) The best way to see what lurks inside is to check out GitHub. Apart from being able to modify the Shruthi-1 hardware and software design, there’s a library you can use in your own projects:

https://github.com/pichenettes/shruthi-1

This also means the Shruthi-1 joins our own MeeBlip among open source synths. I’ve been a bit amused at people comparing the two, because what I like about the Shruthi is that it’s basically MeeBlip’s opposite. We kept the design of the MeeBlip as minimal as possible, both with an eye to keeping one-to-one hardware controls and making modification simpler. The Shruthi is lovely because it’s the reverse: it retains a small footprint, but packs lots of sonic options. It’s the maximal alternative.

I’m just happy that the hardware landscape in general offers loads of great choices for people wanting to augment their computer soft synths with hardware. Who says the synthesizer’s best days are in the past?

Who says you can’t make your own consumer electronics? David A. Mellis, a co-creator of Arduino who now is starting a PhD in Leah Buechley’s group, High-Low Tech, at the MIT Media Lab, has shared his Fab Speakers, an open source, portable speaker project:

These portable speakers are made from laser-cut wood, fabric, veneer, and electronics. They are powered by three AAA batteries and compatible with any standard audio jack (e.g. on an iPhone, iPod, or laptop).

Why open source them? Mellis says he designed the speakers to be affordable and easy to assemble, in the hopes that he would “see changes or additions that I didn’t think about and to have those changes shared publicly for others to use or continue to modify.” Speakers are perhaps ideal for this exercise: the housing matters, both aesthetically and functionally, and because a speaker is something relatively straightforward and simple, it’s easy to imagine modifications that retain the basic role of the design.

Big-league design blog Core77 takes note of what sharing this design can mean, as Mellis turns to designer Sarah Pease to imagine an alternative housing:

Here’s a great example of what can happen when experimental research is documented and posted on the web with plenty of explanation and resources. RISD student Sarah Pease, a junior in Furniture Design, took part in an independent study with the High-Low Tech Group at MIT’s Media Lab this past Fall.

Sarah Pease turns to something you probably already have in your house:

Using readily available household items and basic construction methods allow for even further customization and flexibility of the Fab Speakers. Varying jar shapes/sizes can be mixed with alternate feet for different looks.

High-Low Tech Research Group Project’s Jarring Effect

Building speakers was once a common activity, to the point that many, many musicians made their own speakers or amps or simple effect circuits. For all the excitement over DIY these days, a lot of people don’t have this experience – but with Internet documentation, the time is right for more.

Indeed, I’m keen to hear from people who do have experience building speakers: what might improve the sound quality of this design, and looks aside, what would be the best housing shapes and materials?

In the meantime, I’ll have to give this a try:
Fab Speakers [David Mellis @ MIT Media Lab]

http://sarahpease.com/audioJar

More pics:

Via comments, here’s yet another design – Jon Moeller’s adorable “owl” speakers:
http://moeller.io/owl-speakers.html

I have a bunch of jars, so I may need to give the jars a try here.

When we last caught up with the touch-less, gestural music-making of composer Chris Vik, the Australian musician was sharing his own Kinectar software and playing both dubstep and ambient scores for modern dance. Now, Vik is back playing a very substantial physical instrument: Melbourne’s four story-tall, MIDI-retrofitted Town Hall Organ. Here, the Max-powered software takes on some very big sound from some very big pipes.

He writes:

I’ve created my own software Kinectar, which allows the use of the Kinect to control MIDI devices, ie. playing notes through simple gestures and motion. The Melbourne Town Hall Organ got a referb in the late 90s adding the ability of MIDI messages to active the notes… this happened.

Controlling a 4-story pipe organ with the Kinect

Previously: From Beautiful Ambient Modern Dance to Dubstep, Gestures to Music in Kinect (Download the Tool)

Technology may be about the next Big New Thing, but as with music making in general, making music with tech is for many of us a lifetime vocation. So, it’s welcome news to find that time-tested tools, maturing over decades rather than months, are enjoying greater use than ever before. We saw Pure Data (Pd) attracting new interest as the embeddable libpd version allows use in a range of development environments and mobile platforms. Now, it’s about to be Csound’s turn.

Of course, before we get to that, if you’ve got an iPad 2 or “3″ (aka “the new iPad”), you can more or less skip this entire article and start making wild new granular sounds on your tablet. (Sadly, the original iPad is excluded here because it’s a resource-intensive application, though owners of that tablet – and other mobile devices – have plenty more to anticipate in Csound world.)

csGrain is a multi-effects processor that works its sonic-mangling magic live on sound, making use of something called “SyncGrain,” a real-time granular synthesizer. You can record from a mic or import tracks from the iTunes library (including, of course, your own music), or even use an included sample loop. csGrain then processes those sounds via a rich set of sonic tools, either live or to a recording, with sharing via AudioCopy, AudioPaste, email, and Dropbox. You also get setting randomization and a range of live effects, too. If you’re unsatisfied by the “finger against bathroom mirror glass” feeling of the tablet, you can connect an external MIDI input.

And, of course, it sounds amazing:

All of that is nice enough, but the bigger news is how this is all possible – and what is yet in store. Think Csound running everywhere, including learning about the tool and coding with it directly on an iPad.

csGrain uses Csound, the composition and sound design language that traces its roots back to the first-ever digital synthesis languages developed by pioneer Max Mathews. (It’s worth noting that Max’s ideas inspired the work of Miller Puckette on Pd and Max/MSP, too, meaning these are “all in the family,” as it were. Some even argue the model had an indirect influence on modular synths.)

csGrain is the first of a series of apps using Csound, including a massive, promising, everything-you’d-ever-want-with-Csound-on-an-Ipad app, covering documentation and code. This isn’t just an attempt to “cash in” on a geeky sound tool in the midst of the App Gold Rush — far from it. The application is as much a teaching opportunity as product. You’ll be able to use the application and its documentation to learn more about the sound processing technique, and discover the Csound code that makes the app tick. For some, it could be a first introduction to Csound, without having to be enrolled in an academic class. And for developers and sound artists who do want to make their own Csound creations, an upcoming SDK will unlock the power of Csound on other platforms (iOS being just one). That brings the power of “run anywhere” portability to text-based language Csound much as libpd has done for the graphical-patching tool Pd.

This image of a developer build of Csound Touch should be enough to set Csound fans’ hearts racing. (Okay, not a huge segment of the population at large, but I’m fairly certain most of them read this site.)

Developer Dr. Richard Boulanger, who has been a driving force behind almost every element of Csound’s recent development from the stuff under the hood to the documentation and the community, is now taking that sonic energy and applying it here. (And I do mean energy: sonic whiz “Dr. B,” as he’s affectionately called by his students, practically bubbles with enthusiasm and ideas. I’ll have what he’s having.)

He tells us that even in its first day, csGrain has made a big impact – no small task for an App Store inundated with volume and an application most would consider to be pretty niche in appeal.

The launch of csGrain has been pretty exciting. At the App Store, on day one, in the Paid Music Apps Catagory, we reached the rank of #8 (out of 4000+ paid music apps). We had over 957 people view the csGrain video at the Boulanger Labs site, and we were contacted by Richard Devine and Jordan Rudess with praise, congratulations and advice. We got some pretty great reviews at the App Store such as this one…. (I have no idea who this is by the way… which makes it even cooler….

“csGrain is a must-have effects powerhouse for the iPad. The sounds, usability, and musicality of the instrument are at least on par with the Moog apps, but the the potential for extension and customization far surpasses anything else. If you want to blow your mind in two seconds flat with this app, just press and hold any of the parameters (esp the sync grain ones) and a randomization window pops up, change the parameters and get instantly out of this world sounds. I really like the integration with Dropbox and AudioPaste functionality. And while you don’t need it at first, the manual is very well written for when you want to dig deeper. My only complaint is that the app is not yet in Retina-display, but I am sure that will be coming soon. Another nice feature improvement would be to augment the midi capability, which is already awesome, to allow for automatic learning of cc messages based on input like the way Ableton Live works. All in all, this is most powerful effects application on the iPad. Good job!”

I should also direct owners of the first-generation iPad to Richard’s explanation of why you can’t have the app:

It really doesn’t work on iPad1. It’s optimized for iPad2 and the new iPad. In particular, the stereo granular processing is both efficient and amazing, but… it’s also pretty heavy for the iPad1 and we would get some
breakUps in the audio on that platform – so we just made if for the 2 and the new.

But let’s get to what’s coming next. Dr. Boulanger gives us the full scoop. I imagine him sounding like Vince, the Slap Chop guy (and with good reason):

[Last week] was the release of csGrain (one of Boulanger Labs’ focused/targeted “spin-off” apps) and there are others coming over the next few months. But Csound Touch – which is coming in about 1 month – is ALL of Csound on the iPad…

The Csound for iOS API and SDK will be offered in the next day or two with all sorts of working models…
It will blow you away… how “easy” it is to develop audio apps with Csound as the DSP engine.

our .csd files are all offered with the apps.

One will be able to incorporate Csound into their own apps, games, whatever.

csGrain is just ONE huge .csd file – running under the hood… and there is a button there to see the code and it’s in the manual that is included too.

- we are sharing many tricks right there. But there are also tons of tricks shown in all the models that come with the SDK

Csound Touch IS Csound – all of Csound – for the iPad.

It is built upon and synchronized with the latest version of Csound5

It supports the realtime “rendering” of any Csound “.csd” file.

It can render .csd files from within the Csound Touch App or from the Internet.

It includes realTime “Console Output” (for diagnostics and progress monitoring) and supports “OFFline Rendering” for the realization of the most complex and demanding of “orchestras and compositions.” (For instance, if you wanted to create a sound with ten thousand oscillators and five hundred reverbs and two thousands filters it’s not a problem. This is Csound… all of Csound… and with Csound the only limitation is your imagination!)

Csound Touch supports Realtime MIDI control of any Csound-based MIDI instrument.

Csound Touch supports Realtime iPad/GUI Control of Csound Instruments via custom OnScreen Sliders, Knobs, XY controls and a Piano Keyboard.

One can Save to Disk or Render to the speaker or any pro audio interface in RealTime or once can do both Simultaneously! Jam and Capture! Remix and Record.

Csound Touch supports Interapplication File Transfer on the iPad Import/Export via DropBox, Email and AudioCopy.

To get you started with Csound; to inspire your creative spirit; and to support your study and exploration of Computer Music Composition, Software Synthesis, Signal Processing, Algorithmic Composition, Physical Modeling, and so much more…
the Csound Touch App includes:

Chapter 1 of Boulanger’s “foundational text” published by MIT Press – The Csound Book

The “classic” Boulanger Csound “Toots” from the Csound Manual

Boulanger’s “Mastering Csound” Tutorials

and there’s more!

We include the latest Csound FLOSS Manual (and all the Floss Instruments)
We include the Canonical Csound Reference Manual (and all the Manual Instruments)

and even more….

Selections from Boulanger’s Csound Instrument Catalog (30 years of Csound Instruments)
Selected Csound Compositions from the Boulanger Collection and The Csound Mailing List

Selected Algorithmic/Generative Compositions

A diverse and useful assortment of Dr.B’s favorite DSP Instruments
A varied collection of Dr.B’s favorite MIDI Instruments

A collection of OpenSource Audio Samples from the OLPC Sound Sample Archive.

Of course, this isn’t just about iOS: Csound is now in one form another either available or coming soon to Android, Ableton Live (via Max for Live), Max/MSP, standalone desktop applications, and the Mac AudioUnit plug-in format. I agree when Richard calls it “The Csound Renaissance of 2012.”

By the way, I think Richard has given those of us working on Pd, too, a real sense of what we could do with that tool (as well as an excuse to play with Csound anew). He tells us:

PS…. The Pd Rennaissance is also very very wonderful – the new book, the new code, the new possibilities…. all extremely exciting. After reading your blog on bit ago, I ordered the book and am very inspired by this initiative as well. So…. all cool and extremely important.

I couldn’t agree more.

So, once again, the timeline for the Revolution:

Coming soon (in a month)

Csound Touch – all of Csound on the iPad! Run any opcode, run any orchestra, run any composition. Run the entire Csound Book, Csound Catalog, Csound Manual. It all works and it’s all there!

and the a few weeks after that….

More Csound Apps such as:

csSpectral – Realtime vocoding, convolution, and spectral processing.

csFuzz – a rack of guitar effects.

csVoice – a vocal synthesizer, harmonizer, processor.

csGen – algorithmic, probabilistic, and generative composition systems.

csModel – a collection of Physical and Physically Inspired Models.

csClassics – a collection of synths based on the classic techniques – FM/AM/RM/WaveShaping/Granular/Additive/Etc.

The Revolution will not be televised, but it will be at:
http://boulangerlabs.com

You can find csGrain on our exclusive, multi-platform Apps section:
csGrain @ apps.createdigitalmusic.com

An, as always, don’t miss the central repository for all things Csound:
http://www.csounds.com/

We’ll have an interview with Dr. Boulanger later this week, so if you’ve got questions for him, ask away!

Endeavour’s Evo Series One, which we looked at in the fall, does something different with the conventional keyboard: it adds a touch-sensitive surface to the top of the keys, allowing you to run your fingers up and down the keys for added expression. I got a chance to try the Evo today, and I’m impressed. The first feeling is strange: the keys have an action more like an electric keyboard (Rhodes, etc.), and the keys are atypically tall. But as you begin to play, it makes sense: this isn’t a piano for playing Liszt; it’s a unique, hybrid interface. The added length gives you more touch surface to play, and it is possible to get used to a slightly-adjusted playing style without too much effort. In exchange, you get this new dimension of expression – without awkward wiggling motions on the keys or the imprecision of aftertouch.

The Series One has been on sale for a couple of months, but there were a couple of significant revelations today.

First, Endeavour is building their own software. There’s a bridge tool (currently for Mac, with Windows next) that lets you pipe both OSC and MIDI to other programs. That software is free and open source, if you want to do more with it, and you can also get native OSC right out of the keyboard. Endeavour is also building their own, custom synthesizer to take advantage of the added dimension of playability in the input; I saw a Reaktor patch, but it’ll ship as standalone software.

Second, and perhaps most importantly, they’re working on a much more affordable version. The hand-built, unique Series One is a whopping 2700 €, but by the end of the year, we should see something smaller and in a lower price range. (I heard a number I can’t repeat, but that I liked.)

Check out the sound demos at top for a feel of what this can do – and you can see a glimpse of some of the possibilities, as well as the scale of the hardware, in the images below of the device and its editing software.

Previously:
Tactile Touch: Evo Keyboard to Marry Touch Expression, Conventional Keys

Note: my hands aren’t the best way to get scale, as I have relatively small hands. Liszt was always a bit of a stretch.

Updated: Hands-on video from Messe

I shot a quick video for Keyboard Magazine as part of their Musikmesse coverage:

Keyboard Magazine Video

More editing functionality in their videos:
https://vimeo.com/endeavourgmbh/videos

What you’re watching in the video above doesn’t involve cameras or motion sensors. It’s the kind of brain-to-machine, body-to-interaction interface most of us associate with science fiction. And while the technology has made the occasional appearance in unusual, niche commercial applications, it’s poised now to blow wide open for music – open as in free and open source.

Erasing the boundary between contracting a muscle in the bio-physical realm and producing electronic sound in the virtual realm is what Xth Sense is all about. Capturing biological data is all the rage these days, seen primarily in commercial form in products for fitness, but a growing trend in how we might make our computers accessories for our bodies as well as our minds. (Or is that the other way around?) This goes one step further: the biological becomes the interface.

Artist and teacher Marco Donnarumma took first prize with this project in the prestigious Guthman Musical Instrument Competition at Georgia Tech in the US. Born in Italy and based in Edinburgh, Scotland, Marco explains to us how the project works and why he took it up. It should whet your appetite as we await an open release for other musicians and tinkerers to try next month. (By the way, if you’re in the New York City area, Marco will be traveling to the US – a perfect chance to collaborate, meet, or set up a performance or workshop; shout if you’re interested.)

Hypo Chrysos live at Trendelenburg AV Festival, Gijon, Spain, December 2011.

CDM: Tell us a bit about yourself. You’re working across disciplines, so how do you describe what you do?

Marco: People would call me a media and sound artist. I would say what I love is performing, but at the same time, I’m really curious about things. So, most of the time I end up coding my software, developing devices and now even designing wearable tech. Since some years now I work only with free and open source tools and this is naturally reflected in what I do and how I do it. (Or at least I hope so!)

I just got back from Atlanta, US, where the Xth Sense (XS) was awarded the first prize in the Margaret Guthman New Musical Instrument, as what they named the “world’s most innovative new musical instrument.” [See announcement from Georgia Tech.]

It’s an encouraging achievement and I’m still buzzing, specially because the other 20 finalists all presented great ideas. Overall, it has been an inspiring event, and I warmly recommend musicians and inventors to participate next year. My final performance:

Make sure to use a proper soundsystem [when watching the videos]; most of the sound spectrum lives between 20-60Hz.

Music for Flesh II live at Georgia Tech Center for Music Technology, Atlanta, USA, February 2012. Photo courtesy the artist.

You’re clenching your muscles, and something is happening – can you tell us how this XS system works?

Marco: My definition of it goes like “a biophysical framework for musical performance and responsive milieux.” In other words, it is a technology that extends some intrinsic sonic capabilities of the human body through a computer system that senses the physical energy released by muscle tissues.

I started developing it in September 2011 at the SLE, the Sound Lab at the Edinburgh University, and got it ready to go in March 2011. It has evolved a lot in many ways ever since.

The Xth Sense wearable biosensors by Chris Scott.

The XS is composed of custom biophysical sensors and a custom software.

At the onset of a muscle contraction, energy is released in the form of acoustic sound. This is to say, similarly to the chord of a violin, each muscle tissue vibrates at specific frequencies and produces a sound (called Mechanomyographic signal, or MMG). It is not audible to human ear, but it is indeed a soundwave that resonates from the body.

The MMG data is quite different from locative data you can gather with accelerometers and the like; whereas the latter reports the consequence of a movement, the former directly represents the energy impulse that causes that movement. If you add to this a high sampling rate (up to 192.000Hz if your sound card supports it) and very low latency (measured at 2.3ms) you can see why the responsiveness of the XS can be highly expressive.

The XS sensors capture the low-frequency acoustic vibrations produced by a performer’s body and send them to the computer as an audio input. The XS software analyzes the MMG in order to extract the characteristics of the movements, such as dynamics of a single gesture, maximum amplitude of a series of gestures in time, etc.

These are fed to some algorithms that produce the control data (12 discrete and continuous variables for each sensor) to drive the sound processing of the original MMG.

Eventually, the system plays back both the raw muscle sounds (slightly transposed to become better audible, say about 50/60Hz) and the processed muscle sounds.

I like to term this model of performance biophysical music, in contrast with biomusic, which is based on the electrical impulses of muscles and brainwaves.

By differently contracting muscles (which has a different meaning than simply “moving”) one can create and sculpt musical material in real-time. One can design a specific gesture that produces a specific sonic result, what I call a sound-gesture. These can be composed in a score, or improvised, or also improvised on a more or less fixed score.

The XS software has also a sensing sequencing time-line: with a little machine learning (just implemented few days ago) the system understands when you’re still or moving, when you’re being fast or slow, and can use this data to change global parameters, functions or to play with the timing of events. For example, the computer can track your behaviour in time and wait for you to stop whatever you’re doing before switching to a different set of funcions.

The XS sensors are wearable devices, so the computer can be forgotten in a corner of the stage; the performer has complete freedom on stage, and the audience is not exposed to the technology, but rather to the expressivity of the performance. What I like most about the XS is that is a flexible and multi-modal instrument. One can use it to:

• capture and playback acoustic sounds of the body,
• control audio and video software on the computer, or
• capture body sounds and control them through the computer simultaneously.

This opens up an interesting perspective on the applications of the XS to musical performance, dance, theatre and interaction design. The XS can also be used only as a gestural controller, although I never use it exclusively this way. We have thousands of controllers out there.

Besides, I wanted the XS to be accessible, usable, hackable and redistributable. Unfortunately, the commercialized product dealing with biosignals are mostly not cheap and — most importantly — closed to the community. See the Emotiv products (US$299 Neuro Headset, not for developers), or the BioFlex (US$392.73). One could argue that the technology is complex, and that’s why those devices are expensive and closed. This could make sense, but who says we can’t produce new technologies that openly offer similar or new capabilities at a much lower cost?

The formal recognition of the XS as an innovative musical instrument and the growing effort of the community in producing DIY EEG, ECG and Biohacking devices are a clear statement in this sense. I find this movement encouraging and possibly indispensable nowadays, as the information technology industry is increasingly deploying biometric data for adverts and security systems. For the geeky ones there are some examples in a recent paper of mine for the 2012 CHI workshop on Liveness.

For those reasons, the XS hardware design has been implemented in the simplest form I could think of; the parts needed to build an XS sensor cost about £5 altogether and the schematics looks purposely dumb. The sensors can be worn on any parts of the body. I worked with dancers who wore them on the neck and legs, a colleague stuck one to his throat to capture the resonances of his voice, I use them on the arms or to capture the pumping of the blood flow and the heart rate.

The XS software is free, based in Pd, aka Pure Data, and comes with a proper, user-friendly Graphical User Interface (GUI) and its own library, which includes over one hundred objects with help files. It is developed on Linux, and it’s Mac OS X compatible; I’m not developing for Windows, but some people got it working there too. A big thumb up goes to our wonderful Pd Community; if I had not been reading and learning through the Pd mailing list for the past 5 years I would have never been able to code this stuff.

The Xth Sense software Graphical User Interface. Built in Pd.

The public release of the project will be in April. The source code, schematics, tutorials, will be freely available online, and there will be DIY kits for the lazier ones. I’m already collecting orders for the first batch of DIY kits, so if anybody is interested please, get in touch:
http://marcodonnarumma.com/contact

I do hope to see the system hacked and extended, especially because the sensors were initially built with the support of the folks at the Dorkbot ALBA/Edinburgh Hacklab. I’m also grateful to the community around me, friends, musicians, artists devs and researchers for contributing to the success of the project by giving feedback, inspiring and sharing (you know who you are!).

Thanks, Marco! We’ll be watching!

More on the Work

http://marcodonnarumma.com/works/xth-sense/
http://marcodonnarumma.com/works/music-for-flesh-ii/
http://res.marcodonnarumma.com/blog/

And the Edinburgh hack lab:
http://edinburghhacklab.com/

Biological Interfaces for Music

There isn’t space here to recount the various efforts to do this; Marco’s design to me is notable mainly in its simplicity and – hopefully, as we’ll see next month – accessibility to other users. I’ve seen a number of brain interfaces just in the past year, but perhaps someone with more experience on the topic would like to share; that could be a topic for another post.

Entirely unrelated to music, but here’s the oddest demo video I’ve seen of human-computer interfacing, which I happened to see today. (Well, unrelated to music until you come up with something this crazy. Go! I want to see your band playing with interactive animal ears.)

Scientific American’s blog tackles the question this week (bonus 80s sci-fi movie reference):
Brain-Machine Interfaces in Fact and Fiction

I’ve used up my Lawnmower Man reference quota for the month, so tune in in April.