mandy

Take a good, long look at your computer screen.

Now imagine you can’t see it.

That’s the reality Mandy Matz is facing. At age 36, she’s losing her vision to glaucoma.

The musician and multimedia artist makes some beautiful, ethereal music, having cut her teeth on Buzz. Listen to the haunting “Alpha Waves,” her first song.

And now, because sight is so deeply connected to the way in which music developers map your brain to software, losing her vision could mean losing her ability to work with digital tools.

It shouldn’t be this way. As Paul Lamere, who is collaborating with Mandy on this project, writes:

Anyone who has been to a Music Hack Day has seen the wide range of non-traditional music interfaces that we create. We’ve made Music gloves, leap-motion-based mixers and orchestras, invisible violins made with iphones, remixing tools that use makey-makey, music controllers made out of kinects, arduinos, webcams and even neckties and coat racks. We build things that make music. With a little guidance we should be able to build things that will help the visually impaired make music too.

If sometimes the projects covered on this site, and in hack days, seem a little strange, remember: these experiments are inquiries into the way you connect thoughts and feelings to software. That means thinking beyond the normal expectations about which senses, which gestures to use.

Now, with that experience, the community that has worked with musical interfaces is uniquely prepared for the possibility that any of us may lose senses or motor skills we take for granted.

The notion Mandy and Paul propose: #hackblindness. Mandy has started a blog, with the hopes of dedicating a track at some future Music Hack Days to the topic. This question comes up on CDM a lot, though, so those hack days could be just the beginning.

Worked yourself on this problem? Have some ideas you’d like to try? Actually, something I really don’t know – are you a blind person who reads CDM? (I definitely hope I haven’t screwed up accessibility settings and the like.) We’d love to hear from you. And check out Mandy and Paul’s work:

Hacking Blindness [Music Machinery, Paul's blog]

Mandy’s new blog, covering her experiences and ideas: Hack Blindness @ Tumblr

  • http://twitter.com/a_w_young a_w_young

    I’ve often feared/fantasized (is there a word for “negative fantasy?”) what would happen if I lost my vision as a musician, where it would leave me. Thanks for an interesting piece.

    • Bret Cain

      It is really something to think about, huh? Again, for me, I have obviously thought more about what it would be like if all of the highs I work with killed my ears, but vision these days is just as important. Wasn’t there a famous old composer…

    • Nuno

      I have an extremely severe form of coloboma which reduces my sight acuity in about 90%. Oddly enough I do not require assistive technologies of any kind, but i do need a certain level of interface customisation most developers seem to ignore. Just recently I’ve purchased Live 9, only to find that, much like the preceding versions, it still does not not allow me to drag6 the panes around. Because the lower third of a computer screen is entirely off my visual field, having the devices chain and tye clip editor stubbornly placed there makes the software a pain to use. I’ve sent a feature request, but doubt they will implement it at all. Low vision folks are often neglected even more than our blind counterparts.

  • Bret Cain

    Pauline Oliveros is doing some interesting things with human-computer interaction for the disabled. Probably an avenue that has been researched by her, but, if not, I pray the connection is made and there is a fruitful result. I worry the most about my ears, but I suppose most of us do. My heart goes out to this poor young gal, I can’t imagine not being able to see let alone the disconnect from digital audio creation this would create. Extending a HUGE virtual hug. So sorry,

  • Bret Cain

    Maybe as an audio community we should come together and have set x-sense deprivation times to better acquaint ourselves with such possibilities and, more importantly, come up with creative new ways to battle such problems.

    • Peter

      I’ve seen Altered States, I know where this is heading… ;)

  • Nettoyeur

    Julien Claassen has been working on Linux audio tools for vision impaired users since 2004 at least: http://juliencoder.de/

  • http://iClif.me/ iClifDotMe

    According to AppAdvice, GarageBand for iOS is accessible in voiceover: “We are elated that GarageBand is now accessible in voiceover. Using multi-touch gestures you can play nearly any instrument and record tracks with the built in 8-track mixer.”

    http://appadvice.com/applists/show/fun-apps-for-blind-and-visually-impaired

  • http://www.facebook.com/jclements77 John Clements

    Great article and a subject that is near and dear to my recent work as well! I have been working on a set of voice-narrated and hotkey/controller mapped user interface abstractions for designing accessible software using Max/MSP since summer 2012, and they should be ready to share hopefully by this summer. This work was done at Berklee College of Music, under the guidance of Dr. Richard Boulanger.

    Here is a short video demo:

    http://vimeo.com/61202849

    Will be following all the work and people mentioned here closely!

    John

    • nick

      This is awesome. Is it possible to actually patch as well? can you connect the output of one object to the input of another using VISIMax? What if there are many objects on the screen?

  • musicmann

    Rob Arbittier from the AudioNowcast podcast has been Stevie Wonders go to guy for over 25 years. He programed Stevie’s first sequencer from a business app in fortran many moons ago and continues to help Stevie with his modifications to existing software. I can’t imagine anyone with more experience than him. Try contacting him through the podcast.

    • Bret Cain

      Can’t believe I didn’t think of Stevie. Good idea, my friend!!!

  • Jake

    As someone whose vision keeps on getting worse and I expect to get glaucoma due to genetic reasons I hope we find a solution. My immediate thought is haptic feedback could be the way forward.

    Anyhow I wish her the best, it’s horrible when your body lets you down, but going further it must be even worse when you’re differently able.

    I hope and trust we find a solution.

    And yeah Stevie has been amazing but we need to move beyond the keyboard for those who would rather use more modern techniques.

    • http://twitter.com/TheAlterEkHo AlterEkho

      I am with her as i’m having my nerves which are failing after my surgery heart operation,
      i hope for her the best.
      Haptic could be a solution but it must be simple and i suppose that an union of different tecniques could make the goal..so voice,touch,haptic,analog,volumetric..i’m thinking about the Leap Motion or Kinect but also the accelerometers in Ios or Wii input devices..
      we as a company in music therapy very often are with blinds makin sessions and i feel their needs very strongly and as an opportunity while science continues the path to find the answer to fix those strong health problems which are affectin the soul,the relationship,the common habits,everything

  • Jonah

    I had an app idea that I think is brilliant and as a bonus would work without sight, but I can’t program.

    Basically, touch screen device split into two areas, each area registers 5 touches, but one side , probably the non dominant hand side adds 5.

    Used for a mixer: place one finger down move it up, moves first channel. On the other side’ place another finger and move either of them up and it moves channel 6. Place two fingers down on the dominant side and it moves channel 2. Place 2 fingers on the dominant hand side and 2 on the other side it moves channel 12. Get it?

    Blind or not, I feel this would allow for more intuitive and accurate control than a “real” mixer.

    Mixer is the easiest most obvious paradigm, and I honestly suggest doing no input feedback mixing if you want to play an instrument, but certainly it could work with applications. You’d get a reasonable range of notes, or you could work with partials in an additive synth.

    Another idea I had is for devices that vibrate based on parameter changes you make. Again this would help the sight impaired, but it’s really a deficiency of electronic, mostly software instruments that they often aren’t giving physical feedback in response, except to change volume. How about pads that feel stiffer the harder you hit them or continue to vibrate…

    I’m a musician aka un employed if anyone wants to hire me…. :)

  • Jonah

    PS These comment boxes get uneditable if you’re comment is too long.

  • Derpinator

    I would imagine audio to midi functions would be quite helpful… i.e. Melody to midi, harmony to midi, beatbox to midi. That and lots of voice commands with feedback.

  • Rui Penha

    I’ve done software for Joseph Malloch’s t-stick – http://www.idmil.org/projects/the_t-stick -, in order to enable its use by four blind musicians in a concert in 2010 – http://josephmalloch.wordpress.com/2010/03/31/viagem-ao-alcance-de-todos/ -. They worked really well for this, as they have a good size, are very easy to move and have indented capacitance sensors that make it very tactile.

  • umma08

    I have a very close friend, who has been completely visually impaired from the age of 4.

    He uses Sonar X10 to produce music, solely because he feels it is the best platform as regards accessibility. His dedication to his art is unbelievable, both instrumentally (he plays piano, guitar, flute and drums) and electronically (he creates electronic variants). He also owns a small project studio – with numerous bits of hardware. He records and mixes bands, and other instrumentalists.

    Every time i see him work – listening to the ‘voice’ of Sonar guiding him through the parameter lists, it breaks my heart. The stop, rewind, play, listen, repeat strategy that we all employ, is augmented with a cold, digital voice, accompanying almost his every procedural move. His biggest gripe with music software is that VST and Audio Units do not carry accessibility functions for use within DAW’s, both through the host, or his screen reading software. That, for a dedicated musician is extremely sad. He is restricted to Sonar’s default plugins.

    His music can be heard at http://screenreader.bandcamp.com.

    I urge you to listen to the album’s title track. It send shivers down my spine still, especially when you appreciate the context with which it was written.

    • Mandy Matz

      Text-to-speech is exactly what I want to stay away from. It seems to cause separation of mind -> computer. The software for sighted users works as a bodily-attached musical instrument. I don’t want to loose this connection.

    • Robin Renwick

      Hi Mandy. Yes, i understand your concerns with text-to-speech, and i applaud your optimism in believing that there are better alternatives. As it stands now though, if you want to use the mainstream DAW’s, it is your only real option. My friend has conceded defeat. A lot of his world is interfaced through text-to-speech; his phone, his email system, his whole computer based life, and he sees his music work as an extension of this. He also realises that music software for the visually impaired is such a small target demographic, that it would take a huge change in focus for large companies to tackle this issue. It is very sad, but very true. If you want, i can put you in touch with my friend. You may both be able to learn from one another. I know that he is a huge inspiration to me, personally, and perhaps others may need that enlightenment as well.

  • Nick Bryan-Kinns

    We’re working on cross-modal interaction (using different combinations of senses), and are actively researching and exploring Accessible Digital Audio Workstations here in London, UK. We’re at the early stages right now (we just ran a Participatory Design Workshop with visually impaired musicians and audio production specilists), and are really interested to hear more stories in the area.

    http://depic.eecs.qmul.ac.uk

  • Mandy Matz

    Thank You, all. My primary goal is to get conversation rolling & the makers making. Let your ears be the guide. I have used music software as a true extension of myself. I don’t want to loose that fluid experience of creating sound. I have posted a few more thoughts on tumblr, most recently about text-to-speech applications. Please give it a read & share your thoughts.

    The response here has caused me to cry those very rare tears of happiness.
    Most Sincere Thanks,
    Mandy

  • mnb

    maybe all those experiments using real-life objects and cameras (building blocks, lego bricks, go- and chess boards… etc.) could be usefull here:
    for example building blocks (with some kind of qr-code on it) for arranging samples. (or lego, if you want quantization ;) ) touching a block could allow to assign parameters (via midi-faders, wii-mote etc.) to the selected block, like sample, start, looppoints, filter etc., while moving it around would shift it on the timeline. (which does not have to be a line, but could also be a circle or spiral around the player/composer…)

  • ianc

    i got a op-1 recently and noticed the underside has braile to indicate the ports on the side. someones thinking! i think a blind person could access a lot of its functionality.

  • Peter

    My vision is fine, but I would be very interested in an entirely-touch based interface, so I could make music while walking around and falling asleep, and just avoid looking at glowing text for pretty much half my time alive on Earth. I know I’ve read about things like gloves that function as a keyboard+mouse, or a touchscreen that is capable of raising and lowering pixels to act as a tactile surface, would be cool to see those developed more. I also suspect we’re not that far away from just plugging something into your brain to act as a visual interface.

    • Peter

      Imagine if max/msp objects, for example, were identified by different types of textures (sticky, slippery, grainy etc) rather than names, you could patch some crazy stuff without looking at it.

  • Bojan Radakovic

    Probably the most difficult thing is not how a blind person can interact, eg tell the computer what to do, it;s the opposite side, computer telling a blind user what is on screen. That is a biggest obstacle in my opinion. I tried to do some work blindfolded, and, you can familiarize yourself with shortcuts, but you often struggle with “where im now and what i am doing” without the visual feedback.

    Sorry for my bad engrish translation my own thoughts.

  • http://www.facebook.com/herrsteiner Malte Steiner

    Already Richard Boulanger is mentioned: lot of blind musicians use CSound for creating their sounds, the script driven interface helps in that case. Other than that I would rely more on haptic instruments for my electronic music like analogue synths with loads of knobs, real modulars…