The gaming industry has made their bet, and it’s that touchscreens go better with tactile controls. Might digital musicians reach the same conclusion?

A funny thing has happened on the way to the touch era. The vision of a device like the iPad is minimalist to the extreme: an uninterrupted, impossibly-slim metal slate, as impenetrable as some sort of found alien scifi object. The notion is that by reducing physical controls, the software itself comes to the fore. It’s beautiful conceptually … and then you find yourself tapping and stroking a piece of undifferentiated glass. For navigating interfaces – and even, I’d argue, exploring sound design and composition – it works brilliantly. But for live digital performance (what to game lovers is called “gaming”), for anything that wants tactile feedback, it can be imprecise or unsatisfying, or both.

Watching this shake out as a design problem is fascinating, especially coming from the perspective of music. Digital musicians were exploring alternative interfaces since before it was cool. Given the ability to make any sound we can possibly imagine, the question of how you design an interface around sound is compositional, philosophical, essential.

Whatever winds up working in the marketplace, there are some fascinating ideas for combining touch with tactile. Since both are good at certain tasks, why not do both?

We’ve seen several examples among musicians and researchers exploring how to augment the touchscreen with physical input:

Mike Kneupfel’s research at NYU’s ITP program, in the video at top, investigates adding additional inputs. See: Extending the Touchscreen.

We saw that kind of extensibility in an iPad dock concept by Livid Instruments.

While it lacks additional tangible controls, I/O extensibility is featured in a still-as-yet-unreleased “pro” dock by Alesis, and most recently in a DIY dock by circuit bending pioneer Reed Ghazala.

Now, game vendors are moving in the same direction – even with prototypes that look quite a lot like the research project above. (Sometimes, arriving at the obvious conclusion is necessary for a great design.)

Sony’s PlayStation Vita, successor to the PSP mobile game platform, augments touch input with tactile controls in much the same way as Michael Knuepfel’s work does. Notably, it also proposes how these inputs can coexist in a form factor that’s larger than a phone, but smaller than a tablet – scaled roughly to a comfortable holding distance between your two hands. (Microsoft and Apple each unveiled standard split keyboards on Windows 8 and iOS 5, respectively. The era of thumb ergonomics is now fully underway.)

Nintendo’s Wii U controller combines a lot of sensing capabilities into one device. Like Sony’s effort, the centerpiece is the combination of the interactive touch display with analog controls. But true to its Wii heritage, Nintendo is packing other sensing technology, too. While its evolution has been more piecemeal, the same is true of the Xbox 360 in the Kinect era. The Kinect camera is really a bundle of mic and stereoscopic camera sensing with software intelligence for motion analysis and even speech analysis via a variety of methods. While Kinect is touchless, the conventional gamepad still plays a role.

What’s the relevance of all of this evolution to music? Digital music’s demands parallel gaming, requiring precision, accessibility, scalability from beginners to hardcore experts, and real-time interaction. Also, music research has often been at the forefront of experimentation with a variety of means of translating sensory data to expression. And since musical practice itself is roughly as old in human evolution as language, if not older, it’s a key way of glimpsing how ubiquitous interfaces can become meaningful.

Let me put that another way: the stuff game companies are doing now looks a heck of a lot like what computer musicians have been doing for years.

While much of the acclaim for platforms like the iPad has been for their transparency and unadorned interfaces – and while I believe those are valuable concepts – bundles of capabilities for interacting with the world can become powerful. That means efforts like Apple’s addition of USB MIDI connectivity to the iPad, or Google’s nascent work to standardize USB host mode and open hardware development (based on Arduino), take on new meaning. Add to this additional connectivity via Bluetooth and wifi, and it may be that we only really see what these platforms do when, like the PC, they start geting sociable with a range of other gear.

This could also mean that communities like the music community have a chance to prove that the “post-PC era” is a little different than it’s been described in the mainstream press – and maybe a little less a radical departure. The “post PC era,” we’re told, is less about being a hub for a lot of hardware. But as people look for tactile feedback, some of the coolest applications of these platforms may not be in the mainstream use as “consumption” devices, but at the fringe.

I’ve just come from the launch of the Samsung Galaxy Tab 10.1 in New York. You’re not missing much; there were a handful of people snapping up the tablets. (I think the 10.1, and a few other Honeycomb-based tablets, do have a bright future, though their growth may be a bit slow at first as developers get their hands on them and give people a reason to buy them.)

What was most compelling to people at the launch, though, was a planned appearance by pop star Ne-Yo (at least according to some staffers to whom I spoke).

But the connection was, at best, tenuous. It may be when devices like these tablets are made more viable for musicians onstage that that connection starts to make sense. And that may mean that Apple and Google/Android vendors alike need to start to think more aggressively about the larger ecosystem and hardware applications. Remember all those futuristic promises from Apple about hardware accessories? Right now, the most significant hardware is the Square payment add-on, and it uses a hack to make it work through the audio jack. Both Apple and Google can do more work to open up hardware development.

It’s all well and good for the tablet to be a “post PC” device, to be different from PCs, to be better. But they may simultaneously need some of the openness to other gadgets that made the PC age so revolutionary.

Imagine a nightmarish, dark-world, alternative-reality version of Wii Music, one that sends Miyomato-san screaming. That’s what you get from tokoloten, in a very un-Nintendo noise performance, as found on comments. The Wii is just one of his tools:

tokoloten uses a variety of objects such as magnet motors, infrared devices, game controllers… in order to hide his lack of conventional technic. Depending on the venue, the show might be ambient-like, experimental or electronica with weird cinematographic references. But it most often combines all of this.
tokoloten is based in Lausanne, Switzerland.

It’s proof that the controller – any controller – is in the hands of the creator, and what it sounds like is entirely undetermined.

Mapping a hardware input to a sound means making an abstract connection between one physical action and another sonic reaction. What that relationship is is entirely up to you. I was honestly a bit surprised by some of the impassioned critical reactions to yesterday’s brief mention of the use of the Wiimote as a studio recording. Of course, that proves the creed of the blogger – post first, ask questions later, and when in doubt, just post. Amidst some of the frustration, there are some good discussions, though I do dream of an Internet on which we criticize content without name-calling.

But the reality remains: controllers are always abstracted from the sound, by definition, and whether they’re satisfying to you depends on how you’ve mapped them. I don’t know what qualifies as innovative, but then, there have been times when I’ve very much enjoyed turning a knob, so “innovation” isn’t always what matters to me. I tend to fall back on Duke Ellington – “if it sounds good, it is good.” For controllers, that means “if it feels good, it is good.” You’re the one with the controller in your hands.

For an alternative example, musician/artist Kassen has an excellent session on improvising with custom software and game controllers. Below, you can catch some of his talk from Amsterdam’s famed STEIM research center, which has a long history of researching the controller-music connection. After all these years asking that question, what we have is …more questions. But that’s a beautiful thing.

Kassen (DJ, performer, ChucK programmer) from STEIM Amsterdam on Vimeo.

Part of the reason I’ve never liked “controllerism” as a term – sorry, Moldover – is that there is no clear technique, no clear sound, no particular discipline. That is, I understand the case for the term and I’m glad there’s a discussion. But it seems to me that part of why controllerism is interesting is that there is no such thing as controllerism. The beauty of digital music is that you do have wide-open, blank-page possibilities. You can create your own system. It is abstract, simulation, ungrounded in physical reality. But while that is at odds with millenia of acoustic instrument-making, it’s also in tune with centuries of compositional and notational tradition, which are abstract. For the first time, the systems of how we conceive music can themselves become physical.

That to me is an exciting thing.

So, here’s a question — let’s take the example of sensors that handle orientation. How would you want to deal with them in music software, if they could be standardized, if any accelerometer or tilt sensor could announce its orientation? How do you decide which is the x, y, and z axis, for instance? How would you want the data normalized?

Despite its quirks, Windows can be a wildly underrated OS for music. Of course, that has little to do with the way it works out of the box. It’s a matter of tweaking your setup so you reshape it into a finely-tuned musical tool. And I believe in sharing that info, because ultimately you should be able to make music on whichever OS you choose.

Rain Recording, a custom PC vendor that specializes in building systems for music and creative work, asked me to write up some of my favorite tools for just that job. For the first part, I looked at the unpleasant stuff — tools for troubleshooting your system and keeping it operating at maximum efficiency.

Part 2 is more fun — the goodies that actually help your musical workflow. I kept this entirely to utilities for MIDI and control, but thanks to the effort of some passionate musician-programmers, that winds up being an impressive toolkit. Quite a few items are Windows-only. (I do actually intend to cover Mac OS and Linux, too, but Windows stacked up pretty well.)

My picks, all free, donationware (and do donate and support these tools!), or relatively cheap:

1. MIDI-OX: This is usually the first utility I install on any PC — it’s a do-everything MIDI monitor and MIDI-processing utility, for watching messages, troubleshooting, and performing various processing tasks. Donationware.

2. MIDI-Yoke: Unfortunately, Windows doesn’t have built-in inter-app communication between apps using MIDI, but MIDI-Yoke performs the task elegantly. (Note, Processing lovers: it also works with Java, so this can allow you to build wild interfaces for music in Processing that control other apps.) Donationware.

3. Bome MIDI Translator: A fantastic tool for creating custom MIDI mappings, translating MIDI to QWERTY keystrokes (and back again), and building rules for performance. Prices range from free to EUR59 for end-user releases, but this is one spending money on.

4. SendSX from Bome sends System Exclusive data. Free.

5. Bome’s Mouse Keyboard gives you an on-screen, clickable interface for controlling synths — essential for when you’re doing some last-minute synth programming and set editing on the go. (Yes, like if you decide to make a last-second tweak in the hotel room before a gig.) Free.

6. Edrum Monitor This tool is useful enough for drums alone, with powerful features for adapting input from electronic drum kits and drum sensors for better accuracy. But they didn’t stop there: with deep data monitoring tools, visual meters for calibration, and support for keyboard, mouse, and joystick inputs, this is just an insane do-everything tool that deserves its own category. Donationware.

7. Jack for Windows An inter-app or even inter-computer audio server, ported from Linux. Linux does Jack better, but if you can’t bear to part with your Windows software, it’s worth testing this – and hopefully someone can help the talented Jack team support and develop it further on the Windows OS. Free.

8. Wormhole2: Think Portal for your host of choice: insert this VST plug-in, and you can route audio to and from different apps, different PCs, or even between Macs and PCs easily. Finally, you can bridge the platform divide and the Mac can lie down with the Windows PC happily. This began as commercial software from Plasq, but it’s now free and open source.

9. MCmu: Emulate Mackie Control with devices that don’t support it. Make devices that do support Mackie Control better. Get the controller power you need with apps like Ableton Live and SONAR. Brilliant stuff. EUR39.

10. OSCGlue: Broadcast OpenSoundControl messages from within a host, ideal for gluing together music software and live visuals. Free, from the vvvv community.

11. WMIDI: Transmit MIDI from Wacom (or other brand) tablet input, complete with tilt and pressure, to turn your graphics tablet into an expressive musical controller. Free.

12. GlovePIE: Somehow I left this out of my original round-up. Take joysticks, gamepads, mice, keyboards, MIDI input devices, Wiimotes, and other devices, output MIDI, keystrokes, and other forms of control (even OSC). It’s my favorite software for control input/output. Requires some scripting, but there are some good sample scripts; hope to post more soon. Free.

Full details and more commentary (plus some additional picks):

Essential Toolkit for Windows – Part 2: (Mostly) Free Musical Utilities for Power Users [Rain Recording Pro]

Previously:
10 Free Non-Musical Windows Software Every Musician Should Use

Of course, this is just a short list of my personal favorites. Any I left out, Windows users?

Want to hook that joystick / Wii remote / Guitar Hero controller / something odd to your music software? If you’re on Mac, OSCulator is the do-everything solution. It’s pay-what-you-like software ($19 minimum for PayPal), and it just got a big update:

Announcement: OSCulator 2.6 [Unidentified Sound Object, as seen in our sound design round-up]
Download page, with changelog [osculator.net]

There’s a lot new in release 2.6; highlights include:

• Preset management
• Graphical OSC routing editor
• Wii Guitar Hero support (preliminary)
• Hook up more: up to 2 virtual HID joysticks, up to 8 Wiimote (does anyone own that many?)
• Make keyboard shortcuts just by striking the combo

And just to be clear, this app outputs MIDI. That means you can use whatever music software you like — so don’t worry about the OSC business if it’s new to you!

It’s not even really just for OSC, any more — does all kinds of input tasks. Windows and Linux users have plenty to be jealous of in this program. Major kudos to creator Camille Troillard; USO Project points to a terrific SEAMUS newsletter article on the software and its future.

The only sad news: this is the last release that will support Tiger; future versions are Leopard-only. (I’m curious, Camille — why? Lots of us still run Tiger for audio apps. Is this just to streamline testing, or is there really something in Leopard that OSCulator needs?)

You can add this to yesterday’s good news as far as OpenSoundControl — the iPhone/iPod touch app we saw released to the app store in yesterday’s round-up.

Dorkbot Chicago

Any CDM-ers in the Chicagoland area are most warmly invited to this months Dorkbot at Deadtech, 3321 W. Fullerton Ave., on Wednesday at 8pm for food, drink, and brain-swelling information regarding micro-sampling and alternative musical controllers like QWERTY keyboards, game joysticks, and bicycles.

This week’s presenters will be Liz McLean-Knight and Michael Una, contributors to CDM.

See you there!

ByteMe, Perth; CDM Me, Brisbane

Australia is CDM’s second home, land of crazy creative contributors and designers, and birthplace of the CDM logo and graphic identity. And now I get to go there.

First up is an epic visualist festival in Perth, 11/30 – 12/9. (Jaymis and I arrive 12/2.)

ByteMe Festival

Okay, odds are, you aren’t anywhere near Perth, as it’s supposedly the most isolated city on the face of the Earth. But on the off chance that you are in/near Perth, you’ll definitely want to come out for this one. Visualists like Artificial Eyes and Jean Poole, not to mention festival organizers VJZoo, join a convergence of visual artists from game development to experimental film and motion graphics and special effects. I’m on a panel Wednesday night, but mostly Jaymis and I will be hanging around covering the festival and chatting with cool people. And we get to see whether our first in-person meetup creates a geek matter-antimatter temporal singularity.

12/10 – 12/14 we return to Brisbane, and odds are far likelier that you live there. There’s talk of doing some kind of music event in Brisbane. If you’re interested in helping us organize even a casual meet-up, Brisbanites, let me know. -PK

How much do you get out of this "point-one" upgrade to
interactive modular software Plogue Bidule (OS X/Windows)? An insane
amount, including these highlights:

• Use joysticks, mouse, keyboards, and other HID devices for MIDI controls
• New audio looper with more options: graphical loop point editing, adapts automatically to meter and stretches time, and more
• A zillion new modules, including 32-channel audio playing and
  recording, a spectral file looper (!), and lots of new MIDI objects
• VST bridge, bugfixes on OS X

Head to Plogue and check it out.

It's beautiful — even visually. It's elegant. It's simple. It's
stable. It's flexible. It's cheap. In other words, it's not Max/MSP –
and, great as that program is, it's about time there was another tool
on the block. And it hasn't even hit 1.0 yet. Keep trucking, Ploguers.

Cost: US$75
Availability: Now, download only
Compatibility: Win32 / Mac OS X