illucia is a project at the intersection of lots of forms of goodness and imagination. The physical interface – what you see first in the image and video here – is a DIY modular controller, complete with Buchla-style patch cords and arcade buttons and pretty knobs. But while that might lead you to believe we’re in analog synthesis land, the physical controls are just a tangible cockpit for what the artist calls “code-bending.” Connect the USB controller to software, and it modifies audiovisual games in modular, interactive fashion.

Since you probably don’t have this particular physical controller lying about your home, you can also use these games directly with whatever software or hardware interface you like, thanks to OpenSoundControl (OSC) interfacing.

The result: free, Creative-Commons licensed audiovisual “playgrounds.” The video at top is just a teaser, but there’s more to come, says creator Chris Novello. Chris tells CDM:

In a quick sentence: I make games that can play other games, and I’ve designed a physical console to interact with them.

The vimeo video’s accompanying text offers some explanation, but here is some extra info as well:

“Codebending?”

Codebending is the mutant daughter of live coding, circuit bending, and modular synthesizers. It gives software “patch points,” and allows for connections between otherwise-unrelated computer programs. Anything can control anything, and computer software tropes collapse into strange generative art. In terms of implementation, these patch points are really just OSC addresses, which allow for easy transmission between programs. http://www.paperkettle.com/codebending has more general information on codebending.

“Instrument?”

illucia is a physical instrument for making these connections – it gives the software patch points physical counterparts, and is a real life console for routing information between programs. The act of building and breaking connections (of exploring generative emergences via patching) becomes a form of play. In terms of hardware, I use Atmega mircocontrollers to encode the state of the device (connections between jacks, knob positions, and button presses) into a tiny byte array, which then gets sent to my computer over USB. This state is used to update a software model of illucia on the computer, which determines the routing between several games / programs. I use Max/MSP a lot for rapid development of routing utilities.

All of my games are written in Processing. They utilize classic arcade mechanics, but are more like modules in a modular synthesizer. I also add some twists, like:

“Soviet Life Sequencer,” which is a Tetromino step sequencer that is remixable by Conway’s Game of Life. I remember a CDM post about Conway sequencers, as well as one featuring the game Chime, but I think I’m the first person to combine Cellular Automata, falling Tetrominos, and a Step Sequencer : 0

also, “Pile of Secrets,” which is a codebendable text editor. Office software is a surprisingly fun companion to video game mechanics.

I’m in the process of releasing the first wave of info right now, but I’ve got a ton on the way as well!

For more information:
http://vimeo.com/22732435
paperkettle.com/​
twitter.com/​paperkettle
facebook.com/​paperkettle

Chris also says he’s sending over some more images, so I’ll add those to the story.

This should start the conversation, but if you have questions, fire away and I’m sure we can get Chris to answer them!

More images (all photos courtesy Chris Novello; used by permission):

  • http://avanturb.com Primus Luta

    OSC OSC OSC OSC OSC!!!!

  • Peter Kirn

    Careful. If you say its name more than three times, it can manifest.

  • http://avanturb.com Primus Luta

    Here's hoping that comes.

    Seriously though I think the potential of this does more for OSC than what most which have preceded it have been able to.  Mainly because its OSC being utilized for intercommunication not just translation.  It's nice to use OSC for CC messages but then the CC messages are still the object.  Here what we get to see is how OSC allows things which would otherwise seem unrelated to have a rosetta stone.

    Here's hoping people start realizing that the power of OSC expands exponentially when you break it out of current paradigms and realize it can control anything.

  • http://braduro.com James

    playing is exploring is about play. I like the reintegration of visual communication over symbolic language. Also liked how the videographers used the backspace key as a sort of "freudian slip" or free association button. Linear language is not as good at presenting parallel meaning in a traditional sentence structure. With Codebending, it looks as though you get to play with the layers of the message.

  • http://leisuresonic.com/ Christopher Penrose

    This is a fantastic extension of the digital controller paradigm, both in terms of creative potential and as an aesthetic statement.  Such an interface may have a confusing effect however, as people will automatically associate such an interface with analog synthesizers — regardless I really welcome the development.

    I do call shenanigans on the use of the label "code bending", as it really is "abstract parameter remapping" and isn't directly analogous to "circuit bending" as this, whatever we call it, is an intended feature of the interface and its controlling software.

  • Peter Kirn

    Well, circuit bending could be argued to be an intended feature, too – that's certainly the way Reed Ghazala describes it, and he invented the term. I'd bring both back to the idea of on-the-fly, modular remapping, which is what's important. Patching? We don't really have a catch-all term.

  • http://leisuresonic.com/ Christopher Penrose

    Reed had the intention, but Mattel did not.  I would use the term "code bending" to describe the use of software to produce a result different from its design intention.

    For example: you sample a live guitar, force the audio stream to hijack the keyboard input for OpenOffice, hijack the mouse and format the text into multi-columns or some such, and then scan the video framebuffer to read the resultant text from the screen and convert it back to audio.  :)

  • http://leisuresonic.com/ Christopher Penrose

    But you are right, nomenclature shouldn't get in the way of what's important here.  Chris has abstracted physical hardware patching with powerful and inspiring results.

  • http://noisepages.com/members/driftpattern/ driftpattern

    lots of great ideas here. can't wait to check out the processing code for illucia when its released. physical banana patchbay + [matrix~] would be amazing.

  • newmiracle

    "…a result different from its design intention."

    Well, that's actually what I find interesting about this phenomenon. Where as circuit bending has a kind of recycle and reappropriation ethos to it, that won't always apply to software. A toy that gets ciruit bent is like a Word processor being taken over by OSC: yes, the intention is being bent.

    But then, some programs don't really have set intentions, or have very open and broad intentions. The corollary here would be a modular patching enviroment and Max/MSP, puredata, etc.

    And then there are things that will fall in between these two paradigms. Trying to pin a trendy phrase on something can cause eyes to roll, but it really is a unique thing. Physical live coding? Controller patching? Code bending? I can forgive an imprecise etymology since it's such a unique and new things.

  • http://noisepages.com/members/paperkettle/ paper kettle

    Hey all.. Thanks for so many kind words! 

    I've been quietly toiling at this esoteric shiny thing for so long, I just assumed people wouldn't care. The response has been overwhelming, and it is a great feeling. 

    I'm really happy to see you all thinking critically about "codebending." 

    I deliberated for a long time on that one, and considered all the sorts of things @Christopher Penrose brought up – namely, how codebending isn't an immediate analog[rimshot] to circuit bending – simply that probing/subverting existing circuits is different from designing systems that include patch points. Of course, where does this line get drawn / where does the original object begin and end? Is this not an accurate fit for codebending:&nbsp ;http://vimeo.com/13519070 ? Super Mario wasn't designed to support this behavior, but a system that emulates it made it possible (although, even that system needed some bootstrapping)

    I decided I was alright with these issues.

    However, there is something else at play[rimshot]; a different kind of code being bent. It isn't the textual code that the programmer has written, but the code of how we're to relate to (software) systems.  

    Is there no room for joy in the rule structures that constitute the abstraction of "text editor?" 

    An acoustic guitar is a system of limitations through which one can experience a tremendous expressive mutuality. It produces a unique voice!

    Is it absurd to suggest that every structure sort has a voice like this?

    Are game mechanics not similar structures? What is the voice of Space Invaders? I understand that is a ridiculous question at a certain threshold, but… for me, codebending is about exploring the joy in forgetting what I think I know about systems. It is the surprise discovery of strange emergences in things that I've been staring at for decades. Of course, I'm, writing new codes by using patch points.. but for the moment, having them feels like it offers more motion than the alternative.

    In a certain way, if the term can grow to contribute to its own inaccuracy, it will be more and more successful. (That said, I consider this is unlikely)

    @newmiracle yeah, it has a buzzwordness, but it gets a lot done, and hopefully works to draw attention to our relationships with software, "games" as a a medium, and systems in general

    But, anyway, all of that is nonsense. I'm stoked you're finding it interesting, and really really grateful for the positive comments!

  • http://ardour.org/ Paul Davis

    this is really cool, but a part of me (a part that is actually about 18 years in the past at this point) wishes/hopes that the HW architecture here is interrupt driven rather than polled (it sounds polled – am i right?) would it be hard to make it interrupt driven instead?

  • Dan Pat

    Like so many of the cutting-edge articles on this website, I simply don't get it. I don't understand how it works, how a user operates it, what resources it requires, or how its output is useful. But I love to see esoteric subjects covered here. Keep up the good work Chris. I can tell what you're doing is well organized and innovative, I just don't understand the concept yet.

  • Peter Kirn

    @Dan Pat: This was marked as a "teaser," so I think we'll get more concrete evidence of exactly what Chris has in mind!