The iPhone launch, two short years ago. Photo David Pham.

Apple’s iPhone should be a herald of a new age in interface design. But now, with speculation that Apple and Palm could get into a patent battle, and murky concerns about patents in multi-touch interface design in general, it’s unclear how much intellectual property legal wrangling will have to happen first.

I’m going to resist turning this into a long rant – partly because I think the jury is out on so many issues. It’s never been entirely clear what Apple continues sacred in its intellectual property on the iPhone. It’s even less clear – with similar multi-touch designs spreading back decades and murky law around gestures in general – what their legal standing is. No one knows at this point whether there will actually be a lawsuit between Palm and Apple (or which direction). But one thing I can say with confidence: we need alternatives to Apple. Even if you love your iPhone, I think you’ll agree it’d be tragic if other vendors didn’t push the technology forward. And we need alternatives like Google Android that support real open development, release free and open source code, and provide an option to Apple’s deeply proprietary, restrictive development platform. Innovative music software in particular won’t be able to thrive if alternatives are closed or nonexistent.

Here’s a quick look at where we’ve been, and where things are:

This has been a storm cloud since the beginning. Me, in January 2007, immediately following the keynote: Macworld: Will Apple Keep its iPhone Closed? Multi-Touch Patents? (I wish I had been wrong. No one believed me at the time that these two areas would be big issues.)

The original tech predates the iPhone. Engadget in 2007 on Apple’s multi-touch roots – FingerWorks gave them a patent portfolio and some key technology.

Google may have dropped out of the race. VentureBeat has a source that claims Google voluntarily dropped multi-touch to keep Apple happy. Even if that’s not true, I think potential legal battles with Apple – and the incorrect notion among consumers that this is Apple’s invention – could have a chilling effect. Update: There may indeed be some chilliness in the air, but there’s strong evidence that Google didn’t “cave” to Apple somehow — they just didn’t get around to it. And a multi-touch G1 may not be far off. Just asked the guy who’s already hacked the G1.

Both Apple and Palm are loaded up with patents – and no one knows what will happen. Engadget analyzes the potential for a Palm/Apple legal standoff – but there are two major issues here. One, Palm has a healthy patent portfolio of their own, meaning they could counter-sue. Two, no one knows if anything will come of this – aside from some saber rattling, we’re not even sure there will be a suit.

Just to keep things in perspective, though: I think multi-touch in general is safe. It’s a technology coming to phones, mobile devices, computers, Windows 7, Synaptic trackpads, Linux … the list goes on. To me, the question is whether developers will be free to try ideas without lawyers breathing down their necks, and that’s very much an open question.

And I think the deeper questions about whether open development, as on Android, can be competitive, may prove to be more important in the long run. Apple aside, we need more common-sense, modernized patent law – even if the Android in this case voluntarily dropped a feature, you can see that the issues are linked. And we need to have open development if people are to have freedom to experiment with design. This is about more than Palm and Apple; it’s about how we interact with our tech.

  • Mike

    Ok, so where JazzMutans stands with their Lemurs. which were here looong time before any Iphones. Based on todays claim from Gizmodo, Google with their Android were warned by apple NOT to use the multitouch. Again, I guess Jazzmutans were the first ones.

  • bp

    the threats are not about protecting multi-touch per se. multi-touch just happens to be the major feature that Apple has some patents (valid, enforceable or not) within the mobile computing / phone marketplace. the threat was more a way to cut down a new development platform at launch. the iphone lock-in effect is going to be leveraged around the app store and that is ultimately what apple is looking to protect.

    as for the damping innovation aspect, i am sorry, but that is a red herring. that is *always* a problem and nothing to do with multi-touch. the tech press are mostly a bunch of high-school girls and agitating for a fight between the popular kids. pretty much no IP producing business exist today that doesn't infringe at some point. it is a fact of business life until we get an overhaul of the patent system. from my experience building products in the telecommunication industry (which are far more litigious than the general computing industry), dealing with the lawyers is way down the list of chilling effects of innovation.

  • http://www.createdigitalmusic.com Peter Kirn

    @bp: Well, then that's part of my concern — if Apple isn't going to go after Palm, is this just going to be some good, old-fashioned, Ballmer/Microsoft-style FUD?

    On the other hand, compare: Ballmer and Microsoft went on a concerted campaign to claim Linux and OpenOffice violate their patents. Apple happened to mention it off-hand in a call ** when asked **. So it seems that you're right and this is just the press fanning the flames.

    Nothing wrong with high-school girls, though. (Not to mention, they're consumers of these products…) Let's not offend them by comparing them to the tech blogosphere. ;)

  • rib

    A developer did get multi-touch working on the G1 phone.

    http://lukehutch.wordpress.com/2009/01/25/get-mul

    With Android being open source, if enough people "unofficially" adopt this hack, Apple may have a hard time fighting this.

    As a side note, while I think many see the potential for multi-touch, a lot of people are still in the point and click mindset. A lot of the Lemur demos I've seen have someone using only one finger for the majority of the demo. Of course if multi-touch tech was more widely available, we'd probably see this mindset start to change.

  • Hari Seldon

    I don't understand why you believe that "Multi Touch is safe", I am not very knowledgeable on the subject, so I could be wrong, but apart from some interesting demos from Jeff Han and that big ass table from microsoft (after the iPhone launch), I don't recall ever seeing multi touch before the iPhone. In fact I never heard the term "multi touch", before Steve Jobs' keynote.

  • http://www.createdigitalmusic.com Peter Kirn

    Well, Hari… if it were not from various DIY projects on this site from 2004 on, the Lemur which I reviewed in 2005, FingerWorks' own projects dating to the 90s, basic computer tracking technologies which could trace fingers as early as the 70s, and multitouch as we now know it back to the 80s, you'd be right.

    But prior work doesn't have to be popular prior work. And I think the point is, this stuff is about to be ubiquitous. I don't see Apple fundamentally challenging the basic use of the technology.

  • Hari Seldon

    Peter, do DIY projects really count as prior art?, you mention the 70s and 80s, did these products track two or more fingers at the same time? Look, I understand why you would want to believe that Apple does not own this tech and you could be right, I'm not so sure.

    Surely Apple now owns Fingerworks IP?

    As for the Lemur, the fact that they haven't been sued, does not necessarily mean that they are not infringing IP

  • http://www.createdigitalmusic.com Peter Kirn

    Hari:

    DIY projects count – yes, they do. (Various research projects, as well.)

    Yes, Apple owns FingerWorks IP. I'm sure patent portfolio was part of it here. But there's other work out there, that's the point.

    Did earlier tools track more than one finger at once? Yes, hence multi-touch.

    Lemur came before iPhone, that's my point.

    Look, Apple doesn't own, nor did they invent, multi-touch or gestural control, period, end of story. Now a lot of this isn't legal wrangling at all — it's been marketing. Apple is certainly not the only tech company guilty of presenting stuff as unique to them when it's not, but certainly, there was some smoke and mirrors in that first keynote. They implied to the *audience* that they had invented everything there, when it wasn't true. That's not to say their implementation wasn't unique or that they don't rightfully deserve the rights to some of that, but it's not hard to research multitouch and find out that it's been around for some time.

    There's actually a pretty good overview here:
    http://en.wikipedia.org/wiki/Multi_touch

    Note that not all of these projects use capacitive sensing as the Apple gadgets do. But generally, by multi-touch we mean any device that is capable of tracking more than one touch point, either through some variation on camera tracking, capacitive sensing on the surface, or even physical/mechanical input (each of which has its own advantages).

    If Google really did choose not to use any multi-touch input at all, well, that's just silly. I can see not doing a pinch gesture or something like that, but anything else is just absurd. The hardware capabilities and basic concept of design have been around for some time.

  • J. Phoenix

    I don't see the broad "more than one point simultaneously recognized as input" being challenged by Apple.

    I feel there are too many recognized players in multi-touch interfaces–Jazz Mutant, Jeff Han/Perceptive Pixel, Synaptics touchpads, Microsoft Surface–to take them all on. Its like starting a land war with Asia.

    One thing that Apple can do…and probably will try…is to lock down the rights to specific gestures–the three finger for scrolling down, the diagonal spread for zooming in/out, and so on.

    I can see how they think of gestures proprietary; Palm's handwriting system for stylus recognition is their own after all. But Palm didn't patent "the use of a stylus" and then attempt to sue pencil manufacturers, or anyone writing out the alphabet either.

    I do feel that certain gestures are simply part of human movement…the simplest way for us to do a given task. Like the movement of using chopsticks, turning a page, twisting a knob. I do not think that such things should be patented. I am not sure if they can be.

    But I can see how this could be detrimental to anyone working in multi-touch, as any given "simplest movement solution" could simply end up as something Apple engineers already thought of and added to the patent list, eliminating one more possibility for control.

    As I think the future of our interactions with computers is more towards evolving our interfaces than speeding up processes & memory gains (these are substantial at present) this is disconcerting. Something to pay attention to, definitely.

  • http://www.createdigitalmusic.com Peter Kirn

    @J. Phoenix: Yeah, I tend to agree. What I therefore find curious is why Google would drop multitouch altogether. I'm guessing we may not be anywhere near the whole story via this one leaked source.

    I suppose it's possible Google didn't implement multitouch because they didn't see a reason, once they determined they couldn't use Apple's gestures. Presumably they're not seeing multitouch controllers as a major market, as we are. ;)

    But yeah, no idea. One to watch.

  • cubestar

    We mostly need an alternative to Jazzmutant!

  • http://www.sounddesigntutorials.com SoundDesignTutorials

    I tend to agree that the legal status of this technology is the victim of lots of guesswork and intentional obfuscation. A lawsuit will eventually happen and hopefully blow the doors wide open for this stuff.

  • Pingback: Create Digital Music » Android, Apple, and Multi-Touch, from the Man Who Hacked the G1

  • http://www.createdigitalmusic.com Peter Kirn

    Fred: you're describing the desktop platform. I'm talking about the mobile platform, which while it is actually Mac OS X and even shares similar APIs, in fact has all kinds of restrictions on its developers.

    Oh, and by the way — on the desktop, features like standard inter-app sync, arbitrary inter-app audio routing, and a real-time kernel are possible on Linux but not Windows or Mac because Linux has an open source OS. That's not to say one is better/worse, but there are tradeoffs to having a proprietary OS.

    Also, I think something may be wrong with your CAPS LOCK key.

  • ex fanboy

    whatever, apple lost me as a customer years ago with EXACTLY these kind of stasi tactics.

  • ex fanboy

    additionally:

    patent a gesture?!?

    will i soon be sued because i pressed the c4 key on my oxygen?

    what about double clicks?

    what about moving a mouse or waving my arms to control pitch.

    and people still support that company – it's a riddle to me.

  • AL:

    Patent law seems to be becoming an overly complex and tangled quagmire. I'm amazed at some of the ridiculously vague descriptions of processes that get awarded patents. The legal hurdles and maneuvering alone would seem a disincentive to innovation. Increasingly, without the financial backing to hire a law firm, the small developer has nowhere to turn. Even the cost of defending a challenge to a patent or accusation of infringement, regardless of actual merit, are frightening. A system intended to protect and encourage innovation seems more and more a tool to stifle innovation.

    But that's just my (potentially ill informed) opinion…

  • ex fanboy

    exactly al!

    and the so-called purveyors of innovation, apple, are unfortunately LEADING the charge to block new innovations.

    i understand protecting an assemblage of parts (like a computer or an os etc) but an idea?

  • http://www.createdigitalmusic.com Peter Kirn

    Wait a minute — hang on, folks. I said there was cause for concern, and I stand behind that, but it's not fair to say Apple is blocking new innovation, certainly not yet.

    Apple hasn't taken legal action against Palm. They have actually even refused to talk about Palm directly. What Tim Cook said was that Apple would defend intellectual property — and it was an investor call, so I really wouldn't expect them to say anything else, lest they upset those investors.

    I think if someone clones iPhone features outright, they can expect to hear from Apple, and that shouldn't stifle innovation, either.

    The fundamental problem here, as many are pointing out in this thread, is that patent law is itself murky in its application. It's got huge potential for abuse.

    Now, I do disagree with the way Apple is handling iPhone development. But that's not even worth spending too much energy on — you focus on developing elsewhere if you think there's a better alternative. And I think the press has done a really shabby job on the whole of covering alternatives like Android adequately. So, without sacrificing coverage of some of the Apple-platform apps I think are important, I'll work to make sure we aren't exclusively covering Android.

  • ex fanboy

    fair enough peter! :-)

  • AL:

    Gee, Fred, do you work for Apple?

  • ex fanboy

    "revolution of innovation"?!?

    name ONE thing the iphone inovated.

    really, no kidding. my pocket pc did all of those things 5 years ago.

    including music apps, camera, phone, landscape ebooks etc etc. all without a contract i might add.

    AND it can cut and paste.

  • Polite

    Peter: I agree. These things are concerning. Patents these days seem to be in the habit of stifling innovation and competition in areas we really need them to thrive.

    For examples of this you just have to look at the gaming industry, with people being sued for using vibration in controllers, or for example, someone patenting the idea of loading screen mini games, effectively disallowing them from every existing thereafter. Or even more recently, and rather mind damagingly, Harmonix has recently sued Konami for their use of guitar controllers in their new game, years after Konami lost a lawsuit against Harmonix for copying their music game designs after harmonix effectively ripped off guitar freaks to make guitar hero. Or gibson's suing people for emulating a rock concert experience. ugh.

    It's insane.

    Sorry if that got a little bit off topic, but it saddens me how these things affect the world, and quite often the person filing the patent isn't even the original maker of the idea or concept. Or the concepts themselves are so broad as to be madness.

    ex fanboy – yeah, they go as far as to patent gestures. and anything else that they think is unique to them, or perhaps no one else thought to patent first.

    okay. must. catch breath. stop typing. there. theres a good lad.

  • AL:

    I don't recall anybody making a case that open-source is an entitlement. It's a good idea, it promotes customization and innovation by letting anybody with the coding skills improve/modify the "thing". It makes better business sense on the user end because you don't need to wait sixteen months for the provider to fix problems or suddenly cancel support for the product.

    As for Linux… the business model for Linux was never selling an OS in the traditional sense. The Linux business model sold professional support and add-ons for the OS, not the OS itself. It didn't cripple developers by only releasing half the necessary code libraries, it didn't build in specific hooks to crash competing applications (like MacroShaft… oops, I mean Microsoft), and it didn't require your computer to call the company ever few days to make sure you weren't cheating on a ridiculous constrictive software license that gave the company more power over your computer than you had.

    So the question becomes… why are you so hostile to open-source?

  • ex fanboy

    well fred,

    my statement was about INNOVATION.

    the HARDWARE things you mentioned are NOT innovations, just EXPANSIONS of already existing technology.

    my dell pocket pc has bluetooth, wifi, touchscreen, camera, gps, full internet access, music app that autoconnects to the net if i want to find out what song is playing over the built-in radio or i can hold the ppc up to a radio and use the mic. i can download it as well if i want.

    20,000 apps is a joke compared to what is available for windows mobile; including great sequencers, recording and wav editing apps.

    the dev toolkit is also very easy to use.

    modern ppc's have even better specs of course.

    instead of just being sarcastic, i suggest that you take the time to inform yourself about what has been out there for years already.

    your reaction is sadly typical for someone enslaved by a label. widen your horizons!

    i personally use the tool that works the best for the money. be it from microsoft, linux or apple.

    ps. no, i do not have an accelerometer in my phone, that's 1 point for you ;-)

  • AL:

    I'm trying to figure out what use an accelerometer would be in a phone, other than telling you how fast you've just dropped it on the concrete…

  • ex fanboy

    lol AL

  • ex fanboy

    btw AL,

    your open source comment above is well taken.

    everyone seems to forget that….