Video: Multi-Touch in Windows 7

When I reviewed JazzMutant’s Lemur at the end of 2005 (printed in the February 2006 Keyboard Magazine), I wondered if what we were really waiting for wasn’t a computer screen. At the time, I wrote:

There’s no question that multi-touch touchscreens represent the future of computer interfaces, and the Lemur is the biggest leap yet toward that science fiction future. For now, the challenge is that the Lemur’s features lie somewhere between a computer display and music controller, without effectively supplanting either one. The Lemur sacrifices the sensitivity and tactile feedback of physical controls in the name of flexibility, but that payoff is limited by the restrictions of its pre-built interface objects and the difficulty of configuring new layouts and assigning them to software controls.

If the Lemur could be truly fused with the computer display, rather than requiring an entirely independent interface, it would become a must-buy.

JazzMutant Lemur Review

At the same time, I marveled at what multi-touch could mean: interfaces that were as flexible as software, powerful live performance capabilities, and the ability to navigate sound spatialization and timbre in new, freer ways. Rather than a solution in search of a problem (as multi-touch image resizing is, arguably), these were tasks that just weren’t possible via any other interface.

The video above, showing multi-touch integrated with the next version of Windows 7 (expected at the end of next year), demonstrates one thing to me: multi-touch is coming, and it’ll be mainstream. And that’s huge for creative performance.

Microsoft demonstrates Multi-touch at D: All Things Digital Conference [Windows Vista Team Blog]

I make this sort of gesture all day. It works. One place it doesn’t work: when you’re onstage. Photo (CC) hizonic, via Flickr.

When Touch Makes Sense

Ironically, because Microsoft is the first to show off this technology in something resembling a consumer-ready, standard computer, people are lukewarm. (Do you think the reaction would have been this way if it had been Apple showing the same demo?) Now, I’m all for skepticism. It’s nice to see Lifehacker asking its readers whether touch is really necessary. That was the question I asked in regards to the Lemur, as well: touch isn’t the answer to everything. You lose tactical feedback, and a certain amount of accuracy. On the multi-touch iPhone, this is an especially big deal: I can easily out-type any iPhone user on my Blackberry, and multi-touch doesn’t mean a whole lot on a small form factor that can only comfortably accommodate one or two fingers at a time. Lastly, no technology can change the physical size of your finger relative to, say, a stylus.

But when it comes to music performance, I’m convinced multi-touch can be very powerful. Forget Microsoft’s lame piano demo or obligatory but meaningless photo resizing. Onstage, a multi-touch display is ideal. You can make quick gestures, quickly point at stuff without taking your eyes off the screen, and use large-scale interfaces built for performance. Imagine reaching over to quickly swap instruments, or switch between song sets, or make a rapid gesture to adjust the timbre of a sound, or navigate surround sound spatialization. And imagine that you’ll be able to do this without having to content with another piece of gear, as on the Lemur, but on a mainstream laptop, with any software you like.

Beyond Microsoft

What’s ultimately fantastic about the Microsoft announcement is that it should have implications beyond just Windows. Unlike the proprietary, one-device iPhone, having Windows 7 support multi-touch means lots of hardware should follow, with the economy of scale and access that everyone may benefit. Even Microsoft’s commitment to the relatively niche-oriented tablet PC has driven down digitizer prices (a step, not incidentally, toward this announcement). You can buy an affordable tablet PC right now with Linux installed, if you like. While Microsoft has a leg up in the enabling software for multi-touch, I don’t think it’ll be impractical for other frameworks or open-source frameworks to follow. In fact, the real challenge is to think about interface design in a new way. (In an interview with CDM, the developers of the upcoming Circle soft synth specifically mentioned thinking about making touch work in future as a design goal, and they use the cross-platform JUCE framework.)

And while they didn’t make a specific announcement, I would expect Microsoft to be likewise aggressive about promoting multi-touch capabilities in their own application development frameworks. Ultimately, I believe the most interesting multi-touch interfaces will continue to come from individual developers and researchers, not the likes of Microsoft and Apple. That’s been true already, so imagine what will happen when those folks have cheap hardware ready to go and can focus on design. The OLPC project, of course, promised a multi-touch laptop replacement, as well; that’s basically just a mock-up and I’ll believe it when I see it, but someone is going to deliver a multi-touch machine soon. (It’ll be interesting to see if we hear anything from Apple, as well.)

Yep, I Want It

Don’t get me wrong: tangible, hardware controls aren’t going anywhere. On the contrary, I think the experience of using multi-touch displays, which even with haptics are a long way from giving real tactile feedback, reminds us of the range of ways in which software design and hardware interface can fuse. But by going beyond QWERTY and mouse/trackpad, multi-touch displays could make for an exciting future.

And in answer to Mary Jo Foley’s question, do I want multi-touch in a laptop? Not only do I, but stand next to me or any other digital musician struggling with a tiny trackpad onstage, and you’ll see why.

More multi-touch coverage from CDM

  • daryl

    So does multi-touch really only mean dual-touch? I'm curious specifically about Windows 7. Because poly-touch of, oh, say 20 or so, would be great.

  • http://www.createdigitalmusic.com Peter Kirn

    Depends on the implementation. 20 touch points is only useful with more than one user, unless you've got 20 fingers. :) Lemur supports 10. (guess where they got that number?) I don't see more than 2 points in this demo, but it's still early in Win7 development, so tough to say. (It's always amusing to me to see how much single-finger pointing happens, which is funny… then you're basically using your fingers as a mouse.) We'll have to watch and see.

  • daryl

    I've got 10 toes too!

    Actually, I think multi-user could be an exciting aspect of multitouch.

  • http://www.createdigitalmusic.com Peter Kirn

    Absolutely — multi-user is part of the idea! It generally requires a table or at least a larger interface (Surface certainly qualifies). But the basic prerequisite of multi-touch — assuming the possibilities of multiple points of control instead of just one — would hold.

  • Keith

    Nice article Peter :)

    Multi-touch is great.. but has the following challenges for live performance.

    1. You don't know what you're touching unless you look. That is, no haptic feedback. This is very doable but requires low…

    2… latency. Most touch screens have lag due to the processing required to calculate what is being touched and what is not. When timing matters – and it does in much performance – than this lag sux

    Apple solved this to some degree in their Mighty mouse, which uses touch sensors for the Left/Right button sensing and the physical button for actual 'action'. A hybrid technology like this might work well for performance.

    I can see them using this combo in a keyboard which allows for mousing over via brushing your fingers over the keys and normal key input by pressing strongly on each key.This design would negate the need for a trackpad which means they could make a long thin small laption, support multi-touch and be generally ergonomic.

    I've forgotten who makes it, but that control interface which knows if you're touching the knob (but not turning) is a step in the right direction IMO.

  • http://www.createdigitalmusic.com Peter Kirn

    Hey Keith — nice to hear from you.

    1. Lack of tactile feedback — I agree; that's why I argued in the Lemur review that for many tasks, conventional hardware is actually better. But my assumption here is that multi-touch will be useful when you *want* to look — i.e., as an extension of the screen, rather than a controller per se.

    2. Latency — good point, and in fact, lag looks pretty nasty in the Win7 demo (maybe it'll improve). This isn't an insurmountable problem, but it again means this will work better for non timing-critical applications.

    Multi-touch to me is best thought of as a way of making displays more useful, rather than a way of doing everything. Actually, if you think about it, part of the reason we still have the mouse and qwerty keyboard is they excel at a specific task. They really don't do everything.

  • http://www.quoico.com RichardL

    I may be possible to improve haptic feedback with mechanisms like slight vibration or electrical stimulus. I think the Prada phone uses something like this on it's touch screen.

  • ernesto (costa rica)

    it's useful as long as you don't have to keep your hands up to touch the screen, which is really painful for people like me with chronic tendonitis or other kind of similar conditions. i would rather like to see the whole keyboard/trackpad turned into a tactile surface, yet i guess many people would dislike it too.

  • http://www.createdigitalmusic.com Peter Kirn

    @RichardL: definitely, but those are a long, long way from being as useful as tactile feedback from a physical object.

    @ernesto: Good point, and actually you don't need any physical condition for this to get real old real fast. That's something that killed the light pen: holding your hand in an upright position is uncomfortable, as interface innovator Alan Kay points out here:
    http://createdigitalmotion.com/2008/05/27/back-to

  • MonksDream

    Ohhhh!! THAT explains the demise of the light pen! It's obvious now you say it but I've been wondering about it since the . Comfy wins over precision in this case.

  • MonksDream

    (MS – Sorry about that. My tagging skills are rusty! )

    Ohhhh!! THAT explains the demise of the light pen! It's obvious now you say it but I've been wondering about it since the Fairlight article. Comfy wins over precision in this case.

  • http://www.proemland.com proem

    actually the nearness of multi-touch becoming mainstream is the Reason i picked up a tablet PC for live shows

    the Lemur is awesome.

    i want one

    in the worst way

    Royalties alone will not pay for one for me

    the Lemur was what originally powered my love for "alternative" interfaces…

    well that and minority Report

  • Pingback: Mainstream Multi-Touch is Coming, And It’ll Rock for Music, Abelincoln live looper, LittleGPTracker Hits 1.0; Free, GP2x, Linux, Mac, Windows, Does Lots of Stuff, 8bitcollective.com Compilation CD "Hello World!", Kontakt Tutorial Video: Creative Abu

  • http://www.3amnoise.net/runagate/VSTs.html runagate

    Sweet mother of god it's coming!

    I had a touchscreen that broke.

    It wasn't so good.

    But when I bought it for $180 USD I researched a way to get multitouch. There's an Australian bluetooth 1Ghz multi touchscreen/weak PC for $4000… I cannot wait to get my hands on this. My whole live setup is based around getting my hands on the GUIs.

    insertpizhere, in particular, has some amazing touchscreen-ready freeware VSTs for PC.

  • Jay Vaughan

    Well, multi-touch is cool and all, but only because its well packaged.

    You can do cheap, easy, 'touchy feely' haptic computer interfaces with webcams too, but thats not a fun ergonomic .. so when its packed into a nice object that gets warm in your pocket and is often cool to touch, that is the ticket to rock.

    Of course, consumer music making, mainstream music making, has to be reliable too. I dunno how I feel about the feedback of multitouch right now .. but it sure is a hell of a lot more interesting to watch.

  • Pingback: Touch me Baby one more Time at djtechtools.com

  • http://nosuch.com/tjt Tim Thompson

    While having a display under a multitouch interface is really useful, I think it’s causing most people to think of it only as a replacement for mousing/resizing/sliders/controls/etc, overlooking the potential multitouch has to implement an *instrument*. Think how much expressiviness and flexibility you have in your fingers. Think finger-painting, or sculpting. Every finger is at least 3 controllers – x, y, and pressure (or area). You’ve got 30 controllers in two hands. People use fretless string instruments without needing to look at their hands, so I don’t think tactile feedback is an absolute requirement. Responsiveness/latency of a multitouch interface is a big issue in making it an instrument, no doubt. But equally big is just getting people to think of multitouch as more than just a mouse or slider replacement.

  • Steve M

    <q cite="…Unlike the proprietary, one-device iPhone… …While Microsoft has a leg up in the enabling software for multi-touch…">

    Multitouch isn't just on the iPhone, both the Macbook Pro & Air support scrolling and zooming. I'd assume they're going to take this technology further. I'm not convinced MSFT has a leg up here, they've only shown technology demos so far (and most of these based on ideas we've aleady seen), Apple has shipping products…

    Anyway the fact that they are both doing this is good, competition benefits us all. Like you you say most interesting developments will come from the little guys. Saw someone has developed s/w for the iPhone for controlling for Ableton Live.

    Music could be the most ripe area for multitouch development, how many other (fairly) everyday activities use so many 'points of contact'at once?

  • http://rpi.edu/~mcdonk/ Kyle McDonald

    There are some basic solutions to haptic feedback based on tactile illusions (pdf): for example, if you have a solenoid behind the screen you perceive a "tap" from the solenoid similarly to a button depressing. More complex feedback while probably be worn in gloves at first instead of embedded in screens.