Composer/sound designer Ben Long has a resume of work on dozens of games. Here on CDM, he shares the topic on which he recently addressed GDC China: mobile. If mobile game audio is going to rise to people’s expectations, it’ll have to get past rushed developers and hardware obstacles, including revisiting the whole mono/stereo debate. Ben lets us know his insider take on that landscape, and shares with us the process for designing sounds for virtual slots. Everyone, drone in C with the slot machines… -Ed.

When I tell people “I make music and sounds for video games,” it usually brings responses ranging from people saying “neat!” to blank stares. This is often followed by the person asking “how did you get into that?”  It’s a long story, but I usually say something along the lines of this: “I have musical ADD and video games was a perfect fit!”

For years, I slept on the floor with some makeshift “studio” crammed in the corner.  Working IT jobs during the day and live gigs at night for 10 years paved the way to working on games. My very first ‘studio’ consisted of the Roland VS-880, a digital 8-track with a 2-inch, non-backlit LCD screen.  Mixing on this was thing like brushing your teeth with a tree branch, but hey, it was digital, baby — and this was back when a CD burner was $600.

I got used to being surrounded by bleeding-edge technology and noticed that the game industry was leading the pack.  It wasn’t long before I had my first gig, creating pirate music with MIDI.

So, having worked in the game industry, where does the future lie, and how can someone be prepared to meet it? The answer is clearly mobile.

Mobile Gaming: Challenges and Opportunities

Mobile is by far the fastest-growing sector of the game industry. Never before in history have so many people been exposed to video games. Mobile developers have the ability to take huge “leaps of faith’ of which other developers would not dream. These creative choices can go horribly wrong, but in the case of games like Angry Birds, they can be hugely successful.

The growth of that market is buoyed by an explosion in hardware. New mobile devices appear almost weekly, each promising heightened user experience. Each one has its own set of limitations and capabilities for audio. It’s no surprise that consumers are expecting better audio in their mobile games. The explosion of music-based games has created an interesting phenomenon: people are listening more.

No longer can mobile game developers rely on the visual canvas alone; there is an empty void around the device that can be occupied with captivating content. If a game is performing well in the marketplace, chances are it sounds great. Making audio for games is difficult, but factor in a tiny, mono speaker with extreme hardware/OS limitations and things get hairy. Luckily, mobile phones have rapidly evolved and the days of developers requesting that I create a 4K MIDI score have disappeared. Nevertheless, creating audio content for today’s mobile gaming devices requires a unique skill set.

Today’s mobile games are typically developed at breakneck speeds. As a result, audio often becomes the last ingredient, as developers scramble just to get their game out. To save time, they sometimes buy music tracks or sound effects from an online store. With the right set of ears, this can work, but audio is usually “thrown in” without much thought or testing. Such practices prevent a good game from climbing the charts. There is a reason why film producers hire a composer, sound designer and audio engineer rather than buying canned content – they want the final product to take on a life of its own.

In Living Mono: Starting with the Hardware

When Apple released the iPhone 3, they heralded the new features – including “enhanced mono”. Soon after release, this descriptive term was deleted from the specs. (Am I the only one that noticed the quick removal of that term from their description?) Technically, the iPhone/iPad speakers are not true stereo, but mono. The silky high frequencies make up for it, though – you can lay the device down and enjoy what you are hearing.

Of course, when using headphones you hear everything across the sonic spectrum. One challenge of creating audio content for mobile is taking into account the possibility of headphone usage. The iPad allows for a bit more mid-range frequency content, so I considered this when working with Backflip Studios on their new game “Backflip Slots.”

Developing Backflip Slots

Typically, a mobile game will have a looping background ambience, background music, dialogue and UI/gameplay sounds. For the menu ambience in Backflip Slots, I visited a casino here in Colorado and brought along my Zoom H4 handheld recorder to capture the casino soundscape. Luckily, security didn’t ask any questions as I entered the building with this strange little device. After finding my way to the rows of slots machines, I sat in the middle and started playing. This location gave me a nicely-balanced ambience, so I placed the recorder on the seat next to me and started experimenting with different configurations. All this was done without headphones, so I had to use my best judgement and just go for it.

The funny thing about casinos is that every slot machine plays their jingles in the key of C. The result is a hypnotizing cacophony that keeps the players hooked. I have actually played live gigs at casinos before and been instructed to keep every song in the key of C. This is surreal, much like playing alongside a choir of robots!

Since “Backflip Slots” was getting a more traditional look, we needed the sound to follow suit. For the reel spin, I went for a looping mechanical sound with a subtle friction texture. To achieve this, I combined the sound of a tractor engine with factory machinery samples from my own sound library. The two sounds were then mixed together and combined with elements from Native Instruments’ Sonic Fiction.

Seamless looping is commonplace in games but often brings technical challenges. That can detract from the creative focus, but it’s a necessary evil, given the technical limitations of the hardware platforms. The reel landing sounds needed more of a chunky ‘click’ feel that would not be too overbearing.

Each bonus spin-character icon has an animated sequence in which they come to life and jump off the screen. These actions received everything from an 8-bit flamethrower to the sound of a samurai sword being unsheathed. One of the keys to creating sonic appeal is subtlety and this can require extensive testing. Game sounds should not grate on the nerves even after being heard hundreds of times. This usually entails experimenting with volume, EQ and pitch shifting in the studio. The end result should always be a pleasant listening experience for the development team and ultimately, the player.

The win sequences were also in sharp contrast to the reel spins and button presses. Win animations explode on the screen with coins raining down on top of an animated logo. Add pulsating lasers and lightning strikes and you now have some serious eye candy! These received a good amount of ‘bling’ on the sonic end and went through extensive revisions before completion. Even when being piled on top of each other, each audio file must be audible and clean. Since the game featured all of their IP (Paper Toss, NinJump, Graffiti Ball, etc), I took the existing sounds from other games and remixed them to work within Slots. In the case of “NinJump”, I blended the hiyaahh with a gong cymbal for the combo win. Some of the combinations are buried deep in the game and will only be heard by the top players.

It’s always a blast working with Backflip on these games so stay tuned for some big suprises in 2011.

About the Author

Ben Long is a composer, sound designer and founder of NoiseBuffet. He recently spoke at GDC China on the subject of “creating audio content for mobile games” Ben’s sonic signature can be heard on over 40 games, including the forthcoming release of LodeRunner Mobile. In addition to games, his music is aired on every major TV network and was recently used in the Stevie Wonder Biography on A&E. Ben is a lifelong gamer and has been featured in three game design textbooks. Learn more about game audio and his new ebook at gameaudio101.com.

I find it quite surprising then, that relatively few of the artists I know use any kind of hearing protection. Stereocilia damage in mammals is permanent, and tinnitus is no fun. Most people know that cheap foam earplugs can help prevent damage, but few seem aware that a slightly larger investment can make gigs considerably more enjoyable.

Etymotic ER-20 are affordable (around US$12), semi-professional, one-size-fits-most earplugs. Their biggest advantage over disposable, foam plugs is their reasonably flat attenuation. Foam and other disposable plugs tend to cut out more high frequencies, causing everything to sound muffled.

When I was learning to VJ, playing 6-hour sets alongside whichever DJs or live acts were in town on the weekend, I always kept a box of foam earplugs in my box-of-adapters-and-miscellaneous-cables. They stopped the tinnitus after a night of throwing photons around the place, but I was loath to use them at bands I’d paid to see. If I was exchanging money for music, I wanted to hear every nuance, even if it made my ears considerably less nuanced the following day.

Fortunately, I was introduced to nice earplugs by a mild-mannered musician who happens to make bionic ears when he’s not hanging upside down playing guitar. The ER-20 plugs aren’t quite as easy on the ear canal as the soft foam-rubber disposable ones, but the slight physical discomfort is definitely worth the increased listening comfort. Not only do they protect your hearing, but for particularly loud PA systems, I find that they allow me to hear the band much more clearly.

The first show I used them at was Mogwai, one of my favorite groups, who I hadn’t been able to see in 7 years. In the middle of the set I tried a couple of songs without the plugs, but found that replacing them allowed me to discern more detail in the wall of sound.

One caveat: I’d be careful using them in a boisterous crowd.  The plastic stems extend a bit beyond the outer ear, and I can imagine a physical blow to the side of the head could lodge them somewhere near your brain stem. To protect yourself from that fate, 20x the investment will give you some custom molded earplugs, and the molding can be used for in the future for an excitingly expensive in-ear monitor system.

For now, I’m happy with the ER-20.

(Available from Amazon.com: US$10)

Yes, for the record, that is a young woman screaming to the sounds of AudioMulch. Believe.

Jordan Harris was able to snag some screenshots of Girl Talk’s rig. There’s not much to tell: a laptop, a mouse, and in a sign of the growing stature of Girl Talk (Gregg Gillis), some very practical plastic wrap to protect the machine. What’s unique about Gregg’s work is that this computer doesn’t clear out the room: it attracts screaming throngs of fans. Especially lady fans, proof that this does not have to be a sport for boys. (As it happens, I find they also like watching American NFL football. Poor girls; everyone is convinced they know what they want but don’t ask.)

Every music tool is supposed to have celebrity users, right? Well, AudioMulch definitely can claim Girl Talk. This $89, currently Windows-only tool (yes, Vista-compatible) has long had an underground following. It’s a real-time modular synthesis, composition, and performance tool, which you might suppose would put it in the same category as the likes of Reaktor and Max/MSP. Unlike those tools, though, its modules are laser-focused on certain sonic capabilities. There are ready-made objects for live performance control, and unique, handy tools for setting up envelopes and sequences. It’s got fantastic pre-built effects like a delay line granulator and live looper. And because AudioMulch is also a VST host, it could be your one and only environment.

AudioMulch is the software equivalent of that deceptively cute little rally car that blows more impressive-looking cars off the road.

Version 2.0 is due early next year with new features and Mac compatibility.

GearWire did a fantastic video tutorial series on AudioMulch last year.

PCs are computers that look ugly and don’t have slick ads with popular songs playing in the background. People believe they’re not used for music, but they are, often by musicians who actually play stuff life (yes, even with a mouse as a controller) rather than playing backing tracks from inside space-alien props.

Plastic wrap, according to Wikipedia,“is a thin plastic film typically used for sealing food items in containers to keep them fresh.” Going by the popular brand name Saran, the 1953 invention was not originally designed to protect computers from beer. But if you play music that people like to dance to and you typically see beer bottles around (note the unprotected shot below), it may be an important music technology accessory. Update: According to an interview, the Saran Wrap is there to protect Gregg’s laptop from .. Gregg? So, either he’s lying about the sweaty hands to sound extra awesome, or he really does have some sweat issues. I can’t say I’ve ever worried about my hands the way I’ve worried about beer. (And I tend not to have those screaming, drunken fans, even.) Hot venues? Hot laptop, powered up to full crunching audio signal? Gregg, if you’re out there, inquiring minds…

Two more photos of AudioMulch after the jump…

Updated: more details from Jordan:

He did have one more spare latop on the table next to him, although it stayed closed the entire show.  No idea if it was the Toshiba in the other pics.  Also, the laptop in pics is a toughbook I believe.  The show was at the abercrombie holiday party at the lifestyles community pavilion here in columbus.  This is a pretty large venue with an indoor capacity of 2200.  The guy tore the place up with just a laptop and sheer force of character.  I believe there are a few videos and some pictures posted here: http://thegrip.wordpress.com/2008/12/15/girl-talk-abercrombie-fitch-2008-holiday-party-at-promowest-pavilion-in-columbus-ohio/#more-3776

There are some good shots there showing just the size of the crowd who came out to see this guy use his laptop on a cheap folding plastic table.  This was a private invite only party, but his show at the Newport Music Hall in January is already sold out. 

Good news: Ableton will not be “evil.” (see photo at right; thanks, Amanda.)

It happens: a hard drive dies, or you lose an entire computer and switch machines. Now, in an ideal world with no copy protection, this wouldn’t be such an issue, but most of us are fairly resigned to some kind of copy protection being a necessary evil. There’s software we rely on that requires some kind of authorization or unlock, if not a hardware dongle. That means you need to get a functioning copy of your software of choice up on your machine – fast.

Ableton Live, for one, has always had a relatively generous demo. It runs unlimited, with only saving and bouncing disabled. I have known Live users to, in a pinch, use that demo to save a live gig – just load your set into a backup machine and play. (For the same reason, I keep a fully bounced version of my sets, in case there’s some problem with third-party instruments.)

Ableton had made some noises over the years about adjusting this policy – which always made me nervous that they’d take away the unlimited demo-only version. (I even got students I taught in a class at Hunter College making use of this.)

Happily, what Ableton did this month was to extend the demo, adding 14 days of saving but keeping the unlimited save-disabled mode. In short, says a representative here in New York:

Get a serial:  saving and exporting enabled for 14 days

After 14 days (or without a serial):  old demo behavior; no saving and exporting, but everything else still works indefinitely.

And naturally, if you’ve been holding back from giving yourself a good Ableton Live test drive to see if it’s for you, you’ve got no excuse now. (Bravo not only to Ableton but the various vendors smart enough to offer a demo. After all, you can try out a guitar at the store before you buy it. It’s an instrument.)

My feature story for Keyboard Magazine on recording live laptop performance is now available online at keyboardmag.com (as well as in the July print issue). When I got the assignment, I think my editor imagined futuristic, sci-fi like network recording, in which audio was streamed entirely virtually from players to a recording server and musicians connected to one another over the ether. Instead, we got just the opposite: quick and dirty solutions for capturing improvisatory computer performance, and intentional efforts to make laptop performances sound more like conventional instrumental ensembles. The case studies:

• The Moscow Laptop Cyber Orchestra hosts laptop jam sessions at the conservatory that bears Leon Theremin’s name. Individual speakers, stereo mic — plus groovy visuals in the background.
• Princeton University’s PLOrk plays with hemispherical speakers so that sound radiates from near the laptop the way it would from a real instrument. Their recording configuration is a little more sophisticated, with not only a stereo pair for the audience but three mics above the stage.
• Share in New York has the toughest challenge of all: a club environment in which anyone can show up with any gear and play. They combine the tried-and-true (old-fashioned analog snakes on the floor) with software tools for standardization (a template in the open source Linux and Mac DAW Ardour).

Check out the full story for details:

Electronica Unplugged

One thing we didn’t broach was what to actually play (these ensembles all experiment with everything from alternative controllers to live coding). But the recording question alone turned out to reveal a lot about laptop performance, and how it’s gradually evolving into just music performance.

Also of interest, Craig Anderton talks about the basics of recording your sets live in Ableton Live. The basic idea: record not only the arrangement, but external audio, as well.

This story also turned out to be an interesting demonstration of what can happen when new online sites (like CDM) interface with a traditional outlet (Keyboard, bringing you music making information since 1976). That’s my ultimate hope: that these outlets will make each other better, and each will expand the knowledge of techniques and what (and who) is out there. Less lofty translation: if Keyboard hadn’t asked me to write this up, I might never have gotten around to it, and conversely, if I didn’t have CDM, I would never have hooked up with folks like the Moscow Laptop Cyber Orchestra.

Speaking of which, let us know how you record your sets and even laptop ensembles, and if I missed anything!

Previously:
Laptop Orchestras Proliferate, from Princeton to Moscow

DIY music can be as much about attitude as specific gear. We had performances Friday and Saturday night during the Maker Faire, and while the performances covered quite a gamut, a common theme was finding new ways of playing old instruments, or to make new instruments out of existing stuff. That’s something not unique to anyone genre — electronic music included — so perhaps that shouldn’t be surprising at all.

Friday night was a Maker Faire “edition” of the regular Robotspeak Sessions electronic music night. The venue is an incredibly cool little electronic music store on lower Haight. Imagine a dream store filled with both vintage gear and the newest stuff, and you’ve got Robotspeak; it’s unreal. Saturday night was the Maker Faire “Chips + Music + Fish after party”, which I planned with the help of Make Magazine’s Paul Spinrad. It turned out to be just as insane as I thought trying to run an event in the middle of Maker Faire, but we had some terrific artists. (And yes, the fish and chips turned out to be the greasiest thing I’ve ever eaten, but tasty!) The venue was a wonderfully quirky place called Edinburgh Castle, and the best part of the evening for me was that we ran into one of the members of a great band called Echodrone that happened to have a projector. He was playing vintage training films on electricity, which we got to watch run behind Barney’s massive home-built Theremin. (See above.)

I don’t believe music should be about gear (surprising as that may be given the site I run), but I do believe you can tell a lot just by looking at the tools musicians choose. Here’s an overview of the artists we encountered.

Barney the Theremin Wizard was a fabulously far-out Theremin player, sculpting ear-splitting soundscapes from home-built Theremins and other gear. (His friend leaned over to me and noted that, even though the sometimes-violent sound worlds sound improvised, each is carefully composed in Barney’s musical mind.) Barney’s rig was probably the closest to the Make ethos, composed almost entirely of home-built gear — a giant Theremin case, plus enough mysterious effects, amps, and speakers to fill my rented Toyota Corolla. Think hard-metal/experimental Theremin rather than classical, but still gorgeous stuff. Check out his MySpace page, with great titles like “When Nerds Collide.”

Pineresin is a new trio performing experimental ambient textures, or, as they described it Friday night, music produced by recording analog sounds (drum kit, etc.) and “f***ing it up.” They’ve got a sharp ear for timbre, and though the group just formed, by Saturday they were already really nailing a unique sound. And anything that can bring in bowed percussion always makes me happy. Mmmm … bowed percussion. (I’ve been messing with modeling that in physical-modeling software, having composed for instrumentalists in the past, but that’s another story …)

Drew shares their rig:

drew – i run a 12 channel mackie onyx mixer with efx loops going out to a moog analog delay pedal, a dl-4 delay, and a total sonic annihilation pedal.

i mix jesse’s four live drum mics, carson’s stereo input, my computer input (G4 laptop running ableton live, and NI B4II), and the delay returns. stereo output goes to the house.

carson – takes his own mic feed from the drums, plays prepared dulcimer, russian folk harp, traveler guitar and uses lafayette echoverb, univox tape echo, electro harmonix big muff, G4 laptop running ableton live for resampling and throughput

jesse – marimba, bowed vibraphone, cup gongs, waterphone. kick and tom, snare, cymbal

Yep, their MySpace page says “ambient / experimental / other.” I’m not a big one for labels, but that actually gets you in the ballpark.

And who says “electronic music” can’t involve marimba?

Chachi Jones is a versatile musician with lots of smart beats at his command, and he’s always capable of delivering a real live, Live set. Ableton Live is a favorite tool for DIYers, with good reason, I think. It can sit in the center of a complex setup of DIY hardware and/or software and synths and keep the performance going. Chachi Jones is a real virtuoso with Live, and true to form did a fully Make-friendly set that brought in all sorts of wild sounds (hello, egg whisk on heat sink!) and simultaneous live visuals. He describes his rig:

TR 606, TB 303, portable turntable — all run directly into my mixer with a little reverb on the sends
Macbook running Ableton Live and Quartz composer (patch was a modified version of a patch called Audioskop, I didn’t make the original, but I forget who did)
Presonus Firebox
Old Mac G4 heatsink with contact mic running into a channel in Ableton Live with Resonator effect which was controlled by a series of dummy clips with follow actions
Circuit-bent Texas Instruments Touch & Tell running into a channel in Live with lots of effects and a series of Beat Repeats used as loopers and auto choppers.

Check out Chachi’s full-blown, serious site for more on his work. I’m really loving the latest CD; been playing it a bit.

I unfortunately only caught the first few minutes of Dedalus’ set, because I had to run to get my car out of garage before it closed at midnight. (Yeah, I’m not so hot at this whole “car” thing, I’m afraid. Maybe I should start traveling with a bike.) His stuff was absolutely brilliant, though, full of colliding rhythms and fantastic sounds, so I’ll have to go catch more. And he’s got this great, Monome-like wooden button box, as photographed by Chachi, which is inscribed “designed by tehn.” Any enclosure with prominent masking tape is a good thing. I expect we’ll see more stuff like this once the Monome kit comes out; more on that soon.

(tehn, FYI, = Brian Crabtree, = designer of the Monome.)

Starpause is a terrific tracker/game musician, powered by a Game Park (Linux-OS) mobile game system running LittleGPTracker, also known as “Piggy Tracker.” He really got the crowd going — in contrast to the usually introverted electronic musician or gaming player, he dropped his normally-quiet facade and danced around with his player, finishing his set on the floor. He said he was disappointed no one danced, but after Maker Faire some of us could barely stand — it was still a hit. Check out his official site for details on his net radio label(s) and plenty of downloadable MP3s.

Okay, I’ll talk my rig, too, since people have been asking. My core, don’t-leave-home-without-it setup has become a MacBook running Ableton Live, a Novation ReMOTE SL keyboard (which is both fun to play as a keyboard and controls Live via Automap), and a Focusrite Saffire audio interface. They’ve just been really solid, so from there I can add other toys. In this case, that involved simultaneously running Resolume for visuals on my Toshiba laptop. But then I can bring in other possibilities, and I’m particularly interested in integrating custom Flash programming with Resolume and Max/MSP stuff with Live. I dialed back on this set, though, just because I felt like having a more quiet evening.

The photo to me is very funny, because I had felt in a couple of recent gigs like I was getting emotionally disconnected from my keyboard, so I put it on my lap, which makes me look like a shy boy hiding in a corner in the photos! I’m still a pianist at heart, so whatever alternative interfaces I may espouse, if I don’t feel that connection as a musician, it doesn’t work for me. But yes, that is me playing the Novation using a MagLite as a drum stick, filming the light using a webcam. Didn’t set out to that, but enjoyed it.

I also got to do some stuff with live camera into Resolume, intentionally abstracting the visuals. Resolume has likewise been truly rock-solid, which means it’s definitely staying in my rig (even if I sometimes swap to Mac and VDMX5). More on that over on Motion soon.

I suppose there wasn’t a “Make” component to my set, per se, but it’s also important to me personally to limit some of the tools at my disposal so I can focus on the performance. It’ll be a completely different set tonight here in New York.

All in all, it was a fantastic weekend. And I’m glad we did the after-hours element, as well; I hope we’ll get to do more Make-themed events in future like this! We won’t be limited to NYC and Bay Area, either; stay tuned.