Today, I’m in London doing a hands-on workshop on visual metaphors for music, and covering various topics filed under “synesthesia” at Music Tech Fest. It seems appropriate, with the subject matter on the brain, to revisit the topic of visuals and music in a series of posts.

When you make hardware, with knobs and faders, you’re constrained by physical space – the amount of room on a circuit board, the radius of a knob cap, the size of your fingers. But before you get there, the first step is to sketch an idea. Imagine if you could do that with a marker and paper and ink.

SketchSynth is the latest attempt at a drawable set of controls, letting you turn an illustration on paper into something you can actually use to make music. It’s not the first – this dream of being able to make things come alive with nothing more than a magical pen is an old one – but the execution looks superb. There are two basic approaches to the idea: one is to use some sort of conductive ink to turn the drawing itself into a sensor, and the other is to point a camera at the drawing and calculate where a user makes contact with the drawing. SketchSynth opts for the computer vision approach, by way of OpenFrameworks and the old standby of free and open source vision, OpenCV. (Kyle McDonald’s ofxCV does the heavy lifting.)

Conventional vision fits the task well: faders and knobs respond as expected, even though they’re only ink on paper. While it’s a drawn interface, and could look like anything, the behavioral metaphors all come from hardware: there are sliders, momentary buttons, and pots. Place the tool in “edit” mode, and the computer analyzes what you’re drawing; in “play” mode, the camera tracks your finger as you manipulate what you’ve drawn. The project goes one step further than many that have come before, by overlaying a projection calibrated with your drawing for interactive visual feedback as well as sonic. Sound in this case is provided by Pd, but OpenSoundControl (OSC) messages let you connect to other musical (or visual) tools. See more in the making-of vid, at bottom.

Creator Billy Keyes is working with the right mentor, too, completing this as research for Golan Levin, who has long explored the relationship of drawing and musical gesture. His Sonic Wire Sculptor was a seminal creation in connecting drawings to sound, using a tablet to produce three-dimensional “wire” structures and corresponding sound synthesis. His Messe di Voce performance piece neatly reversed the relationship, using the voice as the input to animate drawings and illustrations. More of Levin’s work at flong.com. See also: composer Xenakis’ UPIC, which lives on today as IanniX, a tool getting a lot of development attention.

Keyes’ project, already getting lots of blog buzz, is notable for its practicality and immediacy; it seems a tool many others might use and build upon rather than a single piece of art.

It’s lovely to see projects, particularly academic projects, come to some form of completion and clarity. Speaking of completion and clarity, at some point a proper survey of drawn musical interfaces seems a must, but that will have to come another day. Where can I get a nice full English breakfast?

I’m late to the party, but hat tip in particular to Creative Applications

Project Page at golancourses.net; Linux source is promised soon

I’ve covered a number of these sorts of projects over recent years.

Paper, Drawing as Musical Controller: A Round-Up (including a number of paper examples)

Drawing Sound: Crazy Touch Interface Sound Experiments with Usine, PC (using only a screen – and thus producing a very different experience of drawing

Imaginary Instruments: Marker and Paper as Controller (this 2009 project is almost a direct analog to SketchSynth, minus the projection)

iPhones, Pencils: Hand-Drawn Music Interactions, Tokyo Subway Mobile Jam (this one isn’t quite the same as the others, using pencil and paper to design an interface for a screen – but it’s nice to recall that you can do that, as well)

Pen and Paper as Graphical, Digital Music Score (from earlier this year, a more open-ended design)

Video: Sonic Wire Sculptor, 2003:

Mobile computing has already had an enormous impact on music making. A modern phone or tablet (and yes, most often, these come from Apple) is capable of out-performing a lot of dedicated hardware and easily runs the synths and workstations that required state-of-the-art desktops just a decade or so ago.

But what if this same computing power – low-energy, low-cost chips – could be in other music gear, too? They could offer significant advantages. Bare boards, while on their own not quite road-ready, can wind up in music-friendly housings. (Think stompboxes – without stomping on your phone, or buying a big, silly dock.) You’ll never have to sign a contract with a phone company to get one, or stop your latest song sketch to take a call. And they could be significantly cheaper: the Raspberry Pi isn’t quite ready for mass consumption yet, but it has already begun shipping at US$25, meaning the entire computer costs what a phone car charger might.

In fact, much as the original personal computing revolution took computing to masses of new audiences, this could extend music computational power worldwide. We’re not just talking strange DIY software, either – these boards run Linux, meaning a lot of off-the-shelf music software will “just work,” including even some fine commercial entries.

If you’re ready to stop dreaming and start making music, now’s a great time. CCRMA at Stanford in the United States and STEIM in Amsterdam, NL have each been working on development. STEIM even has a workshop scheduled for June, taught by Edgar Berdahl (CCRMA) and Florian Goltz (DE):
Satellite CCRMA: Interactive design with open embedded computers

The instructors offer some great inspiration about what this is all about in their description:

These small computers combine the connectivity of a laptop with the computational power of a high-end smartphone; however they are less expensive than either and fit inside a cigar box. We will dedicate much of the workshop to prototyping new functional artworks, for example: musical instruments, effects processors, interactive installation works, and anything else you can imagine that requires high computational power in a small, inexpensive footprint.

In the broader sense this workshop deals with interaction design: What happens when human behaviours meet those of machines?

But even if you’re not able to get to California or Holland, you can give the software a try. The BeagleBoard is now supported by a custom distro; the Raspberry Pi seems a logical next frontier once it starts shipping. With Pd (Pure Data) included, you can even copy-and-paste instruments and effects like synthesizers, step sequencers and drum machines, and granulators built by a broad community – even without necessarily being a master patcher yourself. (And then, when you do want to modify the way it functions or sounds or gets controller, you can.)
https://ccrma.stanford.edu/~eberdahl/Satellite/

It’s not all beginner-friendly yet, but these hacklabs seem the perfect way to begin to move in that direction, as more people test the solutions, gather data on how different patches perform, and make tweaks and write documentation.

Pure Data is a wonder: a free and open source environment for creating your own musical and multimedia creations with graphical programming, from Miller Puckette, the original creator of Max. You can produce everything from interactive sequencers and drum machines to synths to video performance tools by connecting patch cables visually, and you can run on virtually any platform, from BeagleBoards and Rasberry Pi to Mac, Windows, and Linux desktop. Via libpd, you can target other development languages and environments, embed engines in games, or work with Android and iOS.

It’s so important, in fact, that CDM invites Hans-Christoph Steiner, one of the key developers of Pd-extended, to give us a tour of what’s new. (Note: because Pd-extended includes various additional objects or “externals” that Pd Vanilla lacks, you should be careful when building patches for libpd. What I like to do is use Pd-extended as my editing environment, then double-check patches by opening them in Vanilla to make sure I haven’t accidentally used an object that’s not part of the bare-bones version. I can then substitute an object, copy an abstraction, or if necessary build that external.) -Ed.

The Pd-extended 0.43 release has been brewing an extra long time, about 18 months now, mostly because there are lots of big improvements. We wanted to make sure we got it right, so your patches all work, but the improvements all shine, so its taken a while. It’s now solidly beta, so we’re looking for testers. Download a beta build to try here:

http://puredata.info/downloads/pd-extended/releases/0.43.1

First off, the pd-gui side of Pd has been rewritten from scratch. The focus for most of the recent work has been on the editing experience, making your patching experience as productive and flexible as possible. To give some background, Pd has always been made up of two programs: pd is the core engine and pd-gui is the GUI. Since basically all computers now come with multiple CPU cores, this means that pd-gui will usually run on a separate CPU core than pd, so they don’t step on each other’s toes. pd can entirely take over its own core. If you want to make your patch use more CPU cores, then check out the [pd~] object introduced in the last release, but fine-tuned in this one.

There are so many ideas for making a better editing experience in Pd; this release makes big strides to address the editing experience. There are new features like Magic Glass, Autotips, Autopatch and Perf Mode, all available on the Edit menu.

• Magic Glass lets you magically see the messages as they pass through the cords. Just turn it on and hover above a cord, and you’ll see the messages as they go by. You can even look at signal/audio cords.
• Autotips gives you tips about what an object does, what its inlet expects, and what comes out of the outlets.
• Autopatch mode automatically connects objects as you create them.
• Perf Mode, is a mode for performance that makes it harder to accidentally close windows that are part of your performance.

A whole new Pd Window

The Pd Window is also majorly overhauled. First of all, it’s fast. Much much faster than the old one. You can now print thousands of messages per second to the Pd Window and still edit your patch. No more will an accidental dump of info cause the GUI to freeze up (well, okay, maybe if you send 10,000 messages/second, but that is way too many). There are also five levels of printing messages to the Pd Window: fatal, error, normal, debug, all. If you are only interested in fatal errors, switch the Pd Window to 0 – fatal, and you’ll only see the worst problems. You want to see every single message to debug? Switch to 4 – all, and you’ll drink from the firehose.

There is also the new log library, which lets you easily send messages for those different levels. And all messages logged with the objects from the log library are clickable: when you Ctrl-Click or Cmd-click (Mac OS X) on the line in the Pd Window, it’ll pop up the patch where the message came from, and highlight the specific object that printed it. That even works for many messages from other objects, as well.

The Pd Window also includes very basic level meters for monitoring the input and output levels. And for those who want to play with the GUI in realtime, you can type Tcl code in the Tcl entry field, and directly modify and probe the running GUI.

Customize the GUI with Plugins

One thing that you can do now is customize the GUI using GUI plugins. You can change all sorts of colors, some fonts, and many behaviors. Want to create a new object when you triple-click? Try the tripleclick example plugin Want to make the patch cords disappear when you leave Edit Mode? Check out the “only show cords in edit mode” example. Those are the simple ones. There is also Tab Completion, a search engine for the docs, a category browser for the right-click menu, a buttonbar for creating objects, and more.

You can find many GUI plugins in the new section of the downloads page as well as documentation for making your own. (What kind of GUI plugin will you write?)

Write Pd objects in more languages

Traditionally, Pd objects are written in Pd (abstractions), C and some in C++. This new release includes two “loaders”, Lua and Tcl, which allow you to write regular Pd objects in either Lua or Tcl. Pd is not the best for processing strings, so if you need to do that, you can now easily use Lua or Tcl, both very easy scripting languages for working with strings. Lua is often used for OpenGL work, so you can also run Lua objects to work in conjunction with Gem. Also, the Tcl loader lets you write GUI objects in pure Tcl, no C needed.

Multi-processing, Pd-style!

The [pd~] object now works out of box. In case you missed the introduction of the [pd~] object in the last release, we’ll introduce you now. [pd~] is Pd itself incapsulated into an object. You can run any patch inside that instance of Pd, the difference is that the Pd in the [pd~] object runs in a totally separate process. So if your computer has multiple CPU cores, which basically all computers do these days, then the Pd process inside the [pd~] object will run on a separate core. This means you can have a heavy Pd patch spread across multiple cores or CPUs. Or for people who work with video and audio together, you can have one instance for video running at a normal priority, then another instance for audio running at a high priority to make sure there aren’t clicks in the audio caused by heavy video processing.

Autotips, generated from help patches

This release also provides a new “autotips” feature to provide instant information about objects and their inlets and outlets. It is one of the first new developments to showcase all of the meta data that is now included in all of the help patches. (Check out the [pd META] subpatches.) When you hover above an inlet or the object itself in Edit Mode, you’ll see a short text description pop up on the lower left corner. But, of course, using a GUI plugin, you could customize how they are displayed to make it how you want to see it. If you want to add autotips to your object, then just add a [pd META] subpatch to your objects’ help patches, and fill out the description, etc. Voila! They’ll have instant information.

What’s next?

The core pd process still handles a lot of the GUI stuff, but we are working on splitting that out for the 0.44 release. That is a big chunk of work, but it will also bring big gains. In particular, it means that it will be possible for people to write their own GUIs for Pd, covering not just the display of the patch, but also the editing, and everything else. You like OpenFrameworks, Python, iOS, JUCE, Qt, etc.? Write your own pd-gui using the toolkit of your choice. That’s the idea at least. That will take a solid chunk of work, so we are looking for people to join that effort.

Try it yourself:

http://puredata.info/downloads/pd-extended/releases/0.43.1
http://puredata.info/downloads/pd-extended

Where to learn Pd:

Resources to start learning

-Hans-Christoph Steiner for CDM

What you’re watching in the video above doesn’t involve cameras or motion sensors. It’s the kind of brain-to-machine, body-to-interaction interface most of us associate with science fiction. And while the technology has made the occasional appearance in unusual, niche commercial applications, it’s poised now to blow wide open for music – open as in free and open source.

Erasing the boundary between contracting a muscle in the bio-physical realm and producing electronic sound in the virtual realm is what Xth Sense is all about. Capturing biological data is all the rage these days, seen primarily in commercial form in products for fitness, but a growing trend in how we might make our computers accessories for our bodies as well as our minds. (Or is that the other way around?) This goes one step further: the biological becomes the interface.

Artist and teacher Marco Donnarumma took first prize with this project in the prestigious Guthman Musical Instrument Competition at Georgia Tech in the US. Born in Italy and based in Edinburgh, Scotland, Marco explains to us how the project works and why he took it up. It should whet your appetite as we await an open release for other musicians and tinkerers to try next month. (By the way, if you’re in the New York City area, Marco will be traveling to the US – a perfect chance to collaborate, meet, or set up a performance or workshop; shout if you’re interested.)

Hypo Chrysos live at Trendelenburg AV Festival, Gijon, Spain, December 2011.

CDM: Tell us a bit about yourself. You’re working across disciplines, so how do you describe what you do?

Marco: People would call me a media and sound artist. I would say what I love is performing, but at the same time, I’m really curious about things. So, most of the time I end up coding my software, developing devices and now even designing wearable tech. Since some years now I work only with free and open source tools and this is naturally reflected in what I do and how I do it. (Or at least I hope so!)

I just got back from Atlanta, US, where the Xth Sense (XS) was awarded the first prize in the Margaret Guthman New Musical Instrument, as what they named the “world’s most innovative new musical instrument.” [See announcement from Georgia Tech.]

It’s an encouraging achievement and I’m still buzzing, specially because the other 20 finalists all presented great ideas. Overall, it has been an inspiring event, and I warmly recommend musicians and inventors to participate next year. My final performance:

Make sure to use a proper soundsystem [when watching the videos]; most of the sound spectrum lives between 20-60Hz.

Music for Flesh II live at Georgia Tech Center for Music Technology, Atlanta, USA, February 2012. Photo courtesy the artist.

You’re clenching your muscles, and something is happening – can you tell us how this XS system works?

Marco: My definition of it goes like “a biophysical framework for musical performance and responsive milieux.” In other words, it is a technology that extends some intrinsic sonic capabilities of the human body through a computer system that senses the physical energy released by muscle tissues.

I started developing it in September 2011 at the SLE, the Sound Lab at the Edinburgh University, and got it ready to go in March 2011. It has evolved a lot in many ways ever since.

The Xth Sense wearable biosensors by Chris Scott.

The XS is composed of custom biophysical sensors and a custom software.

At the onset of a muscle contraction, energy is released in the form of acoustic sound. This is to say, similarly to the chord of a violin, each muscle tissue vibrates at specific frequencies and produces a sound (called Mechanomyographic signal, or MMG). It is not audible to human ear, but it is indeed a soundwave that resonates from the body.

The MMG data is quite different from locative data you can gather with accelerometers and the like; whereas the latter reports the consequence of a movement, the former directly represents the energy impulse that causes that movement. If you add to this a high sampling rate (up to 192.000Hz if your sound card supports it) and very low latency (measured at 2.3ms) you can see why the responsiveness of the XS can be highly expressive.

The XS sensors capture the low-frequency acoustic vibrations produced by a performer’s body and send them to the computer as an audio input. The XS software analyzes the MMG in order to extract the characteristics of the movements, such as dynamics of a single gesture, maximum amplitude of a series of gestures in time, etc.

These are fed to some algorithms that produce the control data (12 discrete and continuous variables for each sensor) to drive the sound processing of the original MMG.

Eventually, the system plays back both the raw muscle sounds (slightly transposed to become better audible, say about 50/60Hz) and the processed muscle sounds.

I like to term this model of performance biophysical music, in contrast with biomusic, which is based on the electrical impulses of muscles and brainwaves.

By differently contracting muscles (which has a different meaning than simply “moving”) one can create and sculpt musical material in real-time. One can design a specific gesture that produces a specific sonic result, what I call a sound-gesture. These can be composed in a score, or improvised, or also improvised on a more or less fixed score.

The XS software has also a sensing sequencing time-line: with a little machine learning (just implemented few days ago) the system understands when you’re still or moving, when you’re being fast or slow, and can use this data to change global parameters, functions or to play with the timing of events. For example, the computer can track your behaviour in time and wait for you to stop whatever you’re doing before switching to a different set of funcions.

The XS sensors are wearable devices, so the computer can be forgotten in a corner of the stage; the performer has complete freedom on stage, and the audience is not exposed to the technology, but rather to the expressivity of the performance. What I like most about the XS is that is a flexible and multi-modal instrument. One can use it to:

• capture and playback acoustic sounds of the body,
• control audio and video software on the computer, or
• capture body sounds and control them through the computer simultaneously.

This opens up an interesting perspective on the applications of the XS to musical performance, dance, theatre and interaction design. The XS can also be used only as a gestural controller, although I never use it exclusively this way. We have thousands of controllers out there.

Besides, I wanted the XS to be accessible, usable, hackable and redistributable. Unfortunately, the commercialized product dealing with biosignals are mostly not cheap and — most importantly — closed to the community. See the Emotiv products (US$299 Neuro Headset, not for developers), or the BioFlex (US$392.73). One could argue that the technology is complex, and that’s why those devices are expensive and closed. This could make sense, but who says we can’t produce new technologies that openly offer similar or new capabilities at a much lower cost?

The formal recognition of the XS as an innovative musical instrument and the growing effort of the community in producing DIY EEG, ECG and Biohacking devices are a clear statement in this sense. I find this movement encouraging and possibly indispensable nowadays, as the information technology industry is increasingly deploying biometric data for adverts and security systems. For the geeky ones there are some examples in a recent paper of mine for the 2012 CHI workshop on Liveness.

For those reasons, the XS hardware design has been implemented in the simplest form I could think of; the parts needed to build an XS sensor cost about £5 altogether and the schematics looks purposely dumb. The sensors can be worn on any parts of the body. I worked with dancers who wore them on the neck and legs, a colleague stuck one to his throat to capture the resonances of his voice, I use them on the arms or to capture the pumping of the blood flow and the heart rate.

The XS software is free, based in Pd, aka Pure Data, and comes with a proper, user-friendly Graphical User Interface (GUI) and its own library, which includes over one hundred objects with help files. It is developed on Linux, and it’s Mac OS X compatible; I’m not developing for Windows, but some people got it working there too. A big thumb up goes to our wonderful Pd Community; if I had not been reading and learning through the Pd mailing list for the past 5 years I would have never been able to code this stuff.

The Xth Sense software Graphical User Interface. Built in Pd.

The public release of the project will be in April. The source code, schematics, tutorials, will be freely available online, and there will be DIY kits for the lazier ones. I’m already collecting orders for the first batch of DIY kits, so if anybody is interested please, get in touch:
http://marcodonnarumma.com/contact

I do hope to see the system hacked and extended, especially because the sensors were initially built with the support of the folks at the Dorkbot ALBA/Edinburgh Hacklab. I’m also grateful to the community around me, friends, musicians, artists devs and researchers for contributing to the success of the project by giving feedback, inspiring and sharing (you know who you are!).

Thanks, Marco! We’ll be watching!

More on the Work

http://marcodonnarumma.com/works/xth-sense/
http://marcodonnarumma.com/works/music-for-flesh-ii/
http://res.marcodonnarumma.com/blog/

And the Edinburgh hack lab:
http://edinburghhacklab.com/

Biological Interfaces for Music

There isn’t space here to recount the various efforts to do this; Marco’s design to me is notable mainly in its simplicity and – hopefully, as we’ll see next month – accessibility to other users. I’ve seen a number of brain interfaces just in the past year, but perhaps someone with more experience on the topic would like to share; that could be a topic for another post.

Entirely unrelated to music, but here’s the oddest demo video I’ve seen of human-computer interfacing, which I happened to see today. (Well, unrelated to music until you come up with something this crazy. Go! I want to see your band playing with interactive animal ears.)

Scientific American’s blog tackles the question this week (bonus 80s sci-fi movie reference):
Brain-Machine Interfaces in Fact and Fiction

I’ve used up my Lawnmower Man reference quota for the month, so tune in in April.

Apple yesterday described their iPad as “this magical pane of glass that can become anything you want it to be.” So – how about making mobile devices into what you want it to be?

With the help of author Peter Brinkmann and publisher O’Reilly, we’d like to give you a taste of Peter’s new book, Making Musical Apps: Real-time audio synthesis on Android and iOS. Imagining that a lot of you are especially curious about iOS, we’ll include the chapter on how to get started with development. It really gives you a sense of how easy this can be; the challenge is, as it should be, coming up with musical ideas. And Apple did say that they thought that technology was at its best when it was “invisible,” not when it was “inaudible.” So let’s make it make some noise.

(Android developers, libpd actually got its start on Android and runs quite well even on very primitive Android handsets, so consider this a sample; the full book – and the libpd site – include loads of examples on the Android side, too. In fact, because libpd works basically identically on the two platforms, it’s a great choice for making cross-platform development easier.)

In this excerpt, Peter covers:

• How to set up your development environment
• Starting a project with Xcode, and including Pd
• How to make a Pd patch run in your app
• Making the Pd patch and your UI connect with each other (here, from the app’s UI to Pd; the book covers both directions)

In fact, in just a few pages, you’ll have a working guitar tuner for iOS. Have a look:

Read it on CDM’s Scribd page:

Making Musical Apps (Excerpt: How to Build a Music App for iOS)

Direct PDF download link, hosted by CDM (please don’t link to this file directly):
Making Musical Apps (Excerpt) [PDF]

I’ve read an advance copy of the whole book, and my review is simple: if you’re curious about this stuff, get this book. Peter’s style is friendly and precise; no technical detail is left out, and yet those details aren’t overwhelming. The book can be accessible to those new to development, which is essential for a title that’s likely to be read by people who are used to Pd, but dipping their toes into Java and Objective-C for the first time in order to get their patches running on a device.

Ready for the full book?

Get a printed copy on Amazon:

Or read the Kindle edition:

For multi-platform epub, mobi, and PDF formats, head straight to the O’Reilly page:
Making Musical Apps [shop.oreilly.com]

http://libpd.cc/read-the-book/

Pure Data (Pd) is already a free, convenient tool for making synths, effects, and sequencers and other musical generators. But imagine stripping away all the things that tie it to a platform – UI, specific hardware support – so it will run just about anywhere, on anything, in any context.

That’s what libpd, a free, embeddable, open source (BSD) tool for making interactive music, does. Coders can take their favorite language and their favorite platform, and just plug in the power of Pd. They don’t even have to know almost anything about Pd – they can let an intrepid Pd patcher create the interactive sound effects and dynamic music for their game and just drop a patch into their assets.

One of the most powerful applications for this is the ability to add interactive music and sound to mobile apps, on iOS and Android, without writing and testing a bunch of custom DSP code. And that has enabled the use of libpd in apps as successful as Inception: The App. With music by Hans Zimmer and a custom “dream” experience created by RjDj, that app racked up millions of downloads in under a couple of months, and then, far from sitting idle on the app launch screen, went on to clock in over a century of user “dreamtime.”

Okay, so, you’re sold. You want to see what this thing can do, and maybe try it out, and you’re wondering where to start. So, here’s some good news: there’s a new site and a new book to get you going.

The site: libpd.cc

libpd has a new home on the Web, both in the form of a new GitHub repository to organize all the code and docs and samples, and a site that brings together a showcase of what the apps does and points you to where to learn more. The single destination is now hosted here by CDM:

http://libpd.cc

I built that site, so please, if there’s anything you’d like to see or you’ve got your own work created with libpd, let me know about it.

Even just having selected a few key highlights of apps built with libpd, it’s impressive what people are already doing with this tool:

libpd Showcase

The book, and a chat with its author

A new book published by O’Reilly focuses on building mobile apps using libpd, for iOS and Android. (iPhone, iPod touch, Android phones and tablets, and yes, even that “new iPad” introduced yesterday are therefore all fair game.)

You can read a section of the book right here on CDM, for a taste of what’s in store:
How to Make a Music App for iOS, Free, with libpd: Exclusive Book Excerpt

It’s an exceptional, comprehensive look at development using libpd, covering iOS and Android, but also a complete look at the libpd API and how to use it. For Pd patchers just getting started with iOS and Android, it includes all of the basics of how to use libpd in your mobile development environment. For mobile developers new to Pd and patching, it makes clear how you’d communicate with Pd, so you can either dive into Pd yourself or properly interface with patches made by musicians, composers, and sound designers with whom you may be collaborating. It’s an ideal title for anyone interested in taking a game and giving it a more dynamic soundtrack – in sound effects, music or both – or for people building mobile musical instruments and effects, sonic toys, interactive albums, or, really, anything at all that involves sound or music. Since it walks you through the entire development experience, you can sit down with it in the course of a few evenings, and get a complete picture of how to integrate Pd with your development workflow.

Dr. Peter Brinkmann, the principal developer of libpd, is the author of the title. I asked Peter to explain a little bit about the book, who it’s for (hint: you!), and what’s in it (hint: stuff you want to read!) …

CDM: How did this book come about? And the book process really helped drive improvements to libpd, too?

Peter B.: Shawn Wallace, an editor at O’Reilly, contacted me last summer and asked whether I would be interested in writing a short book on libpd. I was interested, and so I talked to my [Google] manager (“No conflict — we all have time-consuming hobbies!”) as well as a couple of colleagues who had written books for O’Reilly. They made a token attempt to dissuade me, but it was clear that they had enjoyed writing their books, and they seemed quite proud of the result, too.

Once I had made up my mind to write a book, the next question was whether to self-publish or go with O’Reilly. Self-publishing is a viable option these days, but then I decided that I really wanted an animal on the cover. Besides, I had never written a book before, and having the support of O’Reilly’s editorial staff made the prospect seem less daunting.

The first draft was done in mid-November, but at that time it was basically science fiction because it presented libpd the way I wanted it to be, not the way it was at the time.

So, after the bulk of the writing was done, libpd needed to be revised so that it would actually be in agreement with the book. In particular, Rich Eakin and I rewrote the iOS components for better performance and usability. That delayed the book by a month or so, which turned out to be a great stroke of luck because that was when I discovered that Xcode 4.2 had changed the entire development model by introducing automatic reference counting, instantly rendering existing
texts obsolete. That included my chapter on iOS, and so I had to sit down and rewrite it.

After that, the rest happened rather quickly — getting reviews, revising the draft, going through the production process. O’Reilly’s toolchain is remarkably efficient, using asciidoc and docbook in a Subversion repository. The editorial staff is great, too. I’m amazed to see how quickly it all came together.

How did you approach writing the book?

For the first draft, I just imagined that I was teaching a class on libpd. When you’re lecturing in front of an audience, you don’t have time to polish every sentence; you just have to talk and maintain some sense of momentum. That approach helps a lot when facing a blank page. After that, it’s many, many rounds of revisions to eliminate weak or redundant sentences.

For the sample code, I picked one project that uses all major components of libpd. That provided a natural progression from idea to completion, while touching on all important points in their proper context. I’m basically providing running commentary on my thought process when making an app, including common mistakes and pitfalls. Like this, readers will know how to recognize and work around most problems.

Another trick is to write more than necessary. The first draft contained a lot of gratuitous editorializing. Those parts were never meant to make it into the finished text, but they were fun to write and they kept me going when I wasn’t quite sure what to write next.

Who it’s for?

The book explains how to patch for libpd, and how to write apps with libpd, with special emphasis on the interface between Pd patches and application code. It’s for mobile developers who want to add real-time audio synthesis to their projects, as well as sound designers who want to deploy their work on mobile devices. It’s light on prerequisites; if you know how to write a basic app for Android or iOS, you’re ready to read the book.

Ed.: I’d add to that, given that there are such great tutorials on app development for Android and iOS – even many of them free, including some very worthwhile documentation from Google and Apple — if you’ve messed with Pd, you should give the book a try. And if you haven’t messed with Pd, this could be a great excuse. This book won’t teach you Pd, but it’ll make very clear how to glue everything together. -PK

Why does a book like this matter? What do you hope will come out of it?

I hope that the book will help popularize real-time procedural audio, in games and other apps. I’m thrilled to see all the projects that use libpd, and I hope that the book will help people create even more awesomeness of this kind. One thing I only fully realized when writing the book is that libpd lets developers use DSP code like a media file: An audio developer creates a Pd patch, and the app developer just drops it into the resources of the app and loads and
triggers it as needed. I guess this was implicit in a blog post I wrote on workflow and prototyping a year ago, but I think the DSP-as-media angle is even more powerful. I hope that the book will bring this out.

The book project has already improved libpd. Whenever I faced the choice between fixing an awkward bit of code or explaining the awkwardness in the book, I chose to fix the code. That took care of all the little things that were sort of bothering me but didn’t seem significant enough to spend time on. It also gave us a deadline for a number of related things that we wanted to do, such as migrating to GitHub and launching the new website, libpd.cc. Ed.: Cough. Yes, glad that gave me that deadline – and thanks to Peter B. for the extra push! -PK

Congrats to Peter on his first animal-on-a-cover! It’s really a great book: you read it, and feel like making more new things, inventing new creations that produce sound and music. And that’s a very good thing.

FRACT is a curious combination of music studio and puzzle game, merging elements of games like Myst with the sorts of synths and pattern editors you’d expect somewhere like Ableton Live. You have to make sounds and melodies to solve puzzles; by the end of the game, say the creators, you’re even producing original music. The work of a small student team out of Montreal, FRACT looks like it has all the makings of an underground indie hit – at least for music nerds.

As the creators describe it:

FRACT is a first person adventure game for Windows & Mac much in the vein of the Myst titles, but with an electro twist. Gameplay boils down to three core activities: Explore, Rebuild, Create. The player is let loose into an abstract world built on sound and structures inspired by electronic music. It’s left to the player to explore the environment to find clues to resurrect and revive the long-forgotten machinery of this musical world, in order to unlock its inner workings. Drawing inspiration from Myst, Rez and Tron, the game is also influenced by graphic design, data visualization, electronic music and analog culture.

The hub of the game is a virtual studio, collecting patterns and timbres. It’s right now in prototype phase, but it already looks visually stunning, an alien, digital world in which more-conventional step-sequencer views seem to emerge from futuristic landscapes. And you can spot Pd in the background (the free and open source patching tool, Pure Data). Update: the developers confirm that they’re working with the embeddable Pd library, libpd. That enables synths with sounds like phase modulation and classic virtual analog sounds, all modulating and generating sounds in-game.

The developers have also published plenty of sound samples so you can experience the musical side of this. Via SoundCloud:

While never released, one place some similar ideas has shown up is a prototype game inspired by Deadmau5. As in this title, two-dimensional editing screens and synth parameters are mapped to a first-person, three-dimensional environment. However, FRACT appears to take this concept much further, expanding upon the world, building more instruments, and actually turning those interactions into gameplay elements. The video of the Deadmau5 project – apparently done in-house for fun and not endorsed by the mouse-headed artist:

That title was the work of a game house called Floaty Hybrid; music blog Synthtopia got the scoop on this in August:
http://www.floathybrid.com
Mau5Bot Sequencer Lets You Make Music In A 3D World [Synthtopia]

We’ll be watching this one develop, certainly; good luck to the team!
http://fractgame.com/

For the last few decades, generally speaking, we’ve had computers, and we’ve had physical, modular, analog gear. Computers are endlessly patchable, but not using physical cords. Modulars use physical cords, but they lack the flexibility (and affordability) of a computer.

Now, US$25 and an Arduino can change that.

rePatcher is a simple, tangible modular interface for computers. It could work with any software, but right out of the gate it already works with two popular (virtual) patching environments, Max/MSP and the free and open source Pure Data (Pd). You use physical patch cords to make connections, and those connections are reflected in the patch you see on the screen. The patch cords are coupled with requisite encoders for dialing in additional parameter changes. (Reason comes up as a possible candidate for additional compatibility, which would, of course, be really sweet.)

rePatcher is built as a shield for Arduino, so you’ll need one of those, but that still keeps the price low enough to say I absolutely have to have one of these right now.

It’s not the first attempt to do something like this, but it might be the most accessible and affordable – and interesting. And while those cute little patch cords are fun, there’s nothing stopping someone from building on this idea and going to bigger cords and something more extensive than this 6×6 matrix.

Best of all: the magic happens entirely over USB, so if you want to make this work with something else – say, your favorite VJ software – you can do so with anything that can communicate over serial.

More information:
http://www.openmusiclabs.com/projects/repatcher/

Continuing our insight into this view into electronic music performance and art through the lens of BodyControlled in Berlin, we’re joined by guest writer Kristin Trethewey. Kristin, a Canadian-born video artist and curator, takes another look at LEAP and BodyControlled, on the eve of its second installment. She gets straight at the question of what “BodyControlled” means, and what it can mean for sonic performance and creation. And I wanted to make sure to subtract myself from this write-up, seeing as I was playing – but see the excellent timelapse of the evening, above. -Ed.

LEAP is one of these spectacular Berlin venues you’ve been hearing so much about. It’s a huge, raw space with a view of Berlin’s landmark TV tower, hosting interesting art events with cheap drinks and the potential for a late-night party. But it’s unique, too, in its focus on electronic arts. And unlike other media arts centers, it’s not filled with computers and half-finished electronic projects. I’ve truly gotten lost trying to find this place (it’s tucked away in a mall), so I would recommend watching the timelapse video LEAP shot that guides you to the entrance before attempting to go there. Tonight is the second edition of BodyControlled, a new bimonthly performance series at the space. This installment, called “matter incompatible,” is held in conjunction with the Transmediale Festival under the satellite program, Vorspiel.

BodyControlled is a series focused on the intersection of performance and electronics. You can expect future programming to focus around ideas of “feedback” and “bio” related electronic performances. In its first installment back in November, a packed LEAP gallery witnessed performances by Robert Henke, Peter Kirn [editor of this site], Stephen Cornford, and Paul Whitty. The event was called “Other Spaces” and took the physical architecture of the gallery as a point of departure. Having the space filled with people made for a secondary concern of space: its use. In a series whose title mentions the body, I witnessed one performance engaging the bodies that were filling the space. Robert Henke’s twelve-hour set activated interactions between the audience, performer, and environment. He moved around, listened and mingled with the audience, even though he had this amazing, souped-up control station complete with ambient lighting.

Other artists put more emphasis on the manipulation and dislocation of space through the use and abuse of electronics. Kirn worked with a custom rig with tablet-controlled original software built in open-source software Pure Data (Pd), controlled by a tablet running Konkreet Performer. Excerpt:

Excerpt – LEAP Gallery Berlin, 26.11 by peterkirn

Whitty and Cornford actively deconstructed electronics in front of the audience:
it pays my way and it corrodes my soul (2011)

Stephen Cornford & Paul Whitty’s performance “it pays my way and it corrodes my soul” seeks out musical material by physically dismembering playback equipment. A reel-to-reel tape recorder is switched on and its mechanism amplified with a variety of microphones while it is taken to pieces. The sounds produced are then fed through an array of pedals: the machine’s belts, gears, switches and casing becoming an instrument subjected to a live audio autopsy

Excerpt:
Excerpt: Stephen Cornford & Paul Whitty, LEAP Berlin, 26 November by cdm

Cornford was also interviewed by LEAP for his installation work, featuring repurposed tape machines:

As João Pais, co-curator of the event with LEAP’s Daniel Franke, puts it:

“BodyControlled means the main direction of the series, to present performance and installation works that have a strong, corporal identity. This can be manifested in many ways, not only implying a “moving performer”. The purpose is to avoid the extreme of abstract performances made by a laptop-er, sitting down as if writing emails. In the first event, this idea was shown by interpreting/filling the space of LEAP through a sound-performance (Kirn, Henke), or an installation (Cornford, Mathy, Oliver).”

See also my write-up for ARTSCARDS from last month:
Other Spaces Generates New Spaces Through Sound at LEAP

The second event, “matter incompatible,” draws reference to the Transmediale theme: In/compatible, acknowledging the less clear, even dark forces at play in the artistic and political climate today. Matter Controlled questions the idea of the object or anti-object within sonification. See CDM’s write-up from yesterday:

Watch Artists Talk About Making Sound From Matter; Thursday Event and Stream in Transmediale Prelude

From the Transmediale podcast, some explanations of the theme of the larger festival:

Jacob Lillemose on the exhibition Dark Drives: Uneasy Energies in Technological Times by transmediale

Kristoffer Gansing elaborates on the festival theme in/compatible, as well as the in/compatible symposium: systems | publics | aesthetics.
Tatiana Bazzichelli is the curator for out new project reSource of transmedial culture and speaks about its concept.
Jacob Lillemose speaks about exhibition Dark Drives: Uneasy Energies in Technological Times which he is curating for transmediale 2012 in/compatible.
Sandra Naumann is the curator for this year’s performance programme The Ghosts in the Maschine, which she explains a bit more in detail.
And Marcel Schwierin tells us about his concept for the video programme he is curating for transmediale 2012 in/compatible.

Performances by Echo Ho, Mario De Vega, Alex Nowitz and Ignaz Schick will investigate this blurry region between the immaterial and material. I am curious to see what objects they will bring to play with. As they potentially seek liberation from the physical objects, by reimagining their sonification, I wonder how they are also reliant and maybe even drawn towards their objectification. Bringing these disparate emotions into play is at the heart of tonights investigation. In today’s climate fractures exist between so many aspects of our lives. These performances seek to bring some of them together, compatible or incompatible as we might discover.

You can watch the proceedings via live Internet stream, for the majority of you not in Berlin for the live show.

www.leapknecht.de

More Photos

About the Author

Kristin Trethewey is a Canadian video artist, cinema performer, and curator. She holds an MFA from Brooklyn College in Performance and Interactive Media. A multi-disciplinary curator and artist for the past ten years, she has recently completed a residency at the Node Center for Curatorial Arts, was co-Director/co-Curator of the INDEX Festival. She currently lives in Berlin.

Truly generative musical scores in games have been few and far between, and “music games” has traditionally meant arcade-style rhythm games in which you repeat phrases or whole songs as accurately as possible. Pugs Luv Beats breaks those molds. Part of a vanguard of new gaming creations that generate dynamic music on the fly, it marries grid-based sequencing and resource-gathering gaming, as music making and gameplay blur together. The interactively-produced music could itself become a new way of delivering a musical signature with sound packs.

And beneath it all lurks a free and open source library, libpd – the embeddable version of tried-and-true free graphical music environment Pure Data. (That library is now on GitHub, and vastly updated, by the way, and we’re expecting a book soon from the library’s principle author Peter Brinkmann.)

Oh, yeah, and don’t forget about some seriously addictive gameplay and adorable pugs. I’m suddenly not concerned about the 15 hours Europe-to-North-America travel I’m doing tomorrow.

Here’s what the gameplay looks like, since it’s much easier to see:

Pugs Luv Beats was just approved on the iTunes App Store for iPhone and iPad.

Co-creator Yann Seznec (The Amazing Rolo) is a terrific musician; I just caught up with him in Edinburgh and Berlin and watched him play a homebrewed pig gut instrument with Matthew Herbert for the performance piece “One Pig,” on tour at Berghain. Working with Pd allowed Yann to focus on those musical impulses and not just engineering, and to let him try things he otherwise would never have imagined on a mobile title. So I asked Yann to walk us through how the project was built. He responded with an exhaustively-detailed examination of the evolution of this title, right down to the Pd patches. (Click through for high-res versions.) If your New Year’s Resolution is doing something with patching, you might want to hang onto these answers. Here’s Yann:

The origins of Pugs Luv Beats date back about two years. After making [musical iPhone game] Mujik, Jon (Jonathan Brodsky, aka jonbro) and I were trying to think of other approaches to music mobile app design, and we started thinking more and more about games. Music games, as a whole, are an oddly passive and traditionalist experience – you play along with a premade track, and you are judged on your accuracy and flair (which is strangely reminiscent of music conservatory mindset…). Obviously there are exceptions (RjDj’s Dimensions, Elektroplankton, etc.), but there you go.

Particularly interesting to me was the idea that game mechanics are often very similar to compositional techniques. So for example, when Sonic runs at a normal speed he collects rings at one rate. However when he powers up and goes super fast, he collects rings at a much higher rate. This could be compared to introducing a melody and then speeding it up – and when there are two players, doing this with two melodies. Instant fugue!

We started looking at how we could make a music game where the music and the game elements were fully intertwined and augmented by each other. So Jon prototyped a space shooter drum machine. It was awesome.

To make a (very very very long and boring) story short, our idea and prototype landed us some funding from Channel 4 and Creative Scotland to work on games that focus on musical creativity and composition.

For various reasons, we decided to put aside the space shooter drum machine for a while, and start from scratch. After going through several full prototyping iterations we eventually settled on a core game mechanic that turned out to be in many ways similar to a Tenori-on [Yamaha grid instrument]/Boiingg-style [monome hardware patch] music generation system – in our final prototype, you controlled a series of little dots that moved around the screen, creating loops. This is super fun from a musical perspective because it’s easy and rewarding within a few seconds, and when you have several loops going it can gain some pretty serious rhythmic and melodic depth.

The key from there for us was turning this into a game. We had been using free Internet graphics packs up until then (we hadn’t hired our artist Sean yet) which featured a ladybug, so we had been referring to the main characters as ‘bugs’. During some discussion one of us accidentally said ‘pugs’, and the game idea was born. We constructed a story about pugs and their love for beets (like the vegetables) which create beats (ha!), and how their love turned into greed and got out of control, destroying their world. The game, therefore, is about helping the pugs rebuild their lost civilization by guiding them to create beats. You grow your galaxy by collecting beats, which you do most efficiently when you dress your pugs up in costumes. What’s not to like?

To get to the part that I imagine CDM readers are most interested in, the app development was done by Jon using openFrameworks, [lightweight language] Lua, our own game engine called Blud, and the audio is all done in Pure Data using libpd (through ofxPd). In hindsight we started using libpd really late in the game, just at the very end of the prototyping stage, which was rather silly. Our adoption of libpd basically made our dev cycle about a million times more efficient. My background is as a musician and sound designer, and I have very little coding knowledge. I do, however, have lots of knowledge of Max/MSP, so picking up Pure Data was pretty easy. This allowed Jon to completely pass off all the audio processing (not to mention aesthetic sound design choices) to me, saving him loads of time, giving me direct control over the sound, and letting me test and prototype different approaches to audio within an environment that I knew would be recreated in the game. Also, as Jon mentioned to me recently, by using PD we are able to take advantage of 20 years of audio DSP research and development. Pretty amazing.

How it all works:

The entire audio engine is contained within this patch. Pardon the messiness.

The simplest part of the patch is the “sounds” section, which is used to playback simple sound effects, for the most part linked with interface actions in the game. I did this by creating a very simple patch which plays a sound when it receives a bang. Which sound it plays is dictated by the argument (in this case, the sound of discovering a new capsule). The process for adding a new sound, then, is as simple as adding the sound file to the /assets/sounds/ folder, and making a new instance of “sounds.pd” and naming it the same as the new sound. Jon, in the project code, created a list called “sounds” which is sent into Pure Data. When that list contains “capsule”, a bang is sent into that subpatch, and the sound is played.

A more complex version of what could be done with this type of data is seen in the voice of Mr Puggles, who helps you learn how to play the game. Mr Puggles pops on and off the screen to guide you through the first few worlds, and when he does he send Pure Data a “puggleShow” and “puggleHide” signal. I wanted to give Puggles a funny synthesizer voice that was different every time – dead simple in PD. To do that, I take the puggleShow bang and use it to trigger five more bangs, spaced out over a second. Each of these bangs triggers a random number which is translated into a MIDI note. This note controls the pitch of two oscillators (a sine and a sawtooth), one of which is slightly modified to make them slightly different pitches. These are played through a short volume envelope and a filter which is also controlled by a random number generator. Result? Hilarious beeping boopy Mr Puggles voice, all coming from one bang.

Every time a player buys or selects a planet, a short list is sent to Pure Data comprised of the planet BPM and a random number seed. The BPM is used to calculate delay times and such, and the random number seed is used to create a sort of musical identity for the planet. This is done by choosing a “beat library” and a musical mode.

The mode is created by building a lookup table that chooses the notes from a chromatic scale that would be used in a particular mode. For example, a major scale (ionian mode) uses notes 1, 3, 5, 6, 8, 10, and 12. Each melodic sound library I used is comprised of a full chromatic octave, and the notes that are played on any given planet are controlled by this table. This ensures not only that all of the different sound libraries being played on a planet will be in the same key, but also that a planet will have a strong melodic identity.

The sound libraries in the game are all controlled by the pugs on the planets. As they run around, each time they land they will trigger a sound. The type of sound is dependent on what terrain they are on – thus, if they run through the snow they play a toy piano, if they run through lava a distorted guitar, etc. There are two states of playing the sound, one if the player deliberately tells the pug to go to that tile, and the second if the pug is traveling over that tile to get somewhere else. It’s super easy to do that kind of thing in Pd; just set up two different ‘play sound’ envelopes, maybe a little extra delay or reverb, and you’re done!

The final piece of the puzzle for making the pugs running around into music is to make each tile be a different note. The terrain of each planet is created by making a sort of height map, where different heights correspond with different terrain types (grass, water, snow, etc). This also means that each tile has a unique number between 0 and 1. When the player buys or selects a planet, a giant random number table is generated in Pure Data which creates a number between 1 and 13 for each possible value between 0 and 1. That value is what is used to pick the note of the mode. This somewhat convoluted approach again lets us make sure that each planet will have a unique, but fully reproducible, musical character.

The actual playing of the sounds is probably the messiest part of the patch structure. Purists look away now.

I wanted to make sure this part of the patch was as flexible as possible, so I ended up using the soundfiler and tabread~ objects, rather than tabplay~, which is great in practice though does look rather uncouth. Additionally, I had some limitations imposed upon the structure of the patch – namely, I had to keep the number of tables down as much as possible, to save on memory. So each sound bank has two voice polyphony – there are many sound banks, and the beats and sound effects aren’t counted in this, so that limitation is not really heard in the final product at all. It did mean I had to work out a decent voice allocation system though!

I think my memory issues were probably my only problem with using PD in this project – though only indirectly. As I mentioned, they were hardly a problem artistically, however it took me a while to get used to the idea that not everything I patched on a computer would work on an iPhone. Similarly, I had to be very careful about things like relative volumes. In a generative music game like Pugs Luv Beats, the player could quite easily send 15 pugs running around making sound, which mounts up pretty quickly. It means that all of the patches and sound need to be designed to withstand lots of triggering without distorting. None of these things are problems, really, all they require is regular testing on devices and simulators – something that every mobile developer is already used to.

That’s the Pure Data audio engine in a nutshell. The end result is a flexible and powerful audio engine that sounds really great and is fully integrated into Pugs Luv Beats. The game is a great combination of music, silliness, and strategy – there’s a bit of something in there for everyone. You can definitely just play with the game to make beats, or you can try and collect all of the costumes, or you can try and make the most efficient planet ever. You can also explore the galaxies being made by your Game Center friends, to hear what they’re up to.

The background story:

And, just for fun, a silly promo featuring real pugs. Anyone traumatized by the sight of Pd patches, these should relax you.

More information at the developer site:
http://luckyframe.co.uk/pugsluvbeats/game.html