The Speaking Piano, and Transforming Audio to MIDI
Austrian Composer Peter Ablinger has transformed a child speaking so that it can be played as MIDI events on a mechanically-controlled piano, making the piano a kind of speech speaker. Via Matrixsynth, the readers at Hack a Day get fairly involved with how this may be working.
It seems not quite accurate to describe this as vocoding in the strictest sense, so much as a simple transformation to a (much) lower frequency resolution – that is, the 88 keys of the piano. Ablinger, for his part, describes the events as “pixels.” It’s pretty extraordinary that without a bandpass filter, you get something approximating the noisy sibilance of the speech, but this seems to be the result of having lots of events (that is, lots of resolution in terms of time). Edit: Listening again, the short answer to how you can hear so much of the voice through the piano seems to be, you can’t; the original is almost certainly mixed in. It’s nonetheless an interesting effect, and I’d like to hear the piano on its own. In other words, the basic process is, 1) convert the sound spectrum of the recorded voice to a series of MIDI events, and 2) play back the translated MIDI file. You can see that the MIDI playback is accomplished with Pd (Pure Data) running on a Windows Linux/KDE netbook, though it’s not clear what was used to do the original conversion. (The screen shot with side-by-side audio and MIDI appears as though it may be for demonstration purposes, only.)
Correction: The work is absolutely done in custom software developed by the composer in Pd (Pure Data). It’s an ideal tool for the job, and free and open source. I wouldn’t dare try to replicate the results here, but this is fantastic inspiration for playing with sound in Pd.
One Windows tool that’s capable of the job is TS Audiotomidi, as observed by Hack a Day spacecoyote. Whether or not that’s what’s at work here – and it may well be – that utility is itself interesting. Edit: Yeah, far more likely the whole thing was done in Pd. And Pd should be up to the task.
Of course, this is to say nothing of the lovely work done on the mechanical piano. It’s a beautiful piece. Here’s hoping some government bureaucrats got the message of the declaration. Now, we just need a chorus of something really loud – say a thousand trumpets – shouting out the Universal Declaration of Human Rights.
54 Comments
Leave a CommentDub
Gorgeous! Just gorgious.
October 6, 2009 @ 10:58 am
KULTURTECHNO
Quadraturen…
Als Nachtrag zum Palinsong hier ein just veröffentlichtes YouTube-Filmchen über Peter Ablingers Quadraturen, bei denen ein mechanisch gespieltes Klavier Sprache imitiert.
(via)
……
October 6, 2009 @ 11:34 am
Bjorn Westergard
I would have never guessed that a piano’s frequency specificity would be sufficient for this kind of thing!
It would be cool if he wrote a piece that transitioned gradually from something more recognizably “music” to “speech”.
October 6, 2009 @ 11:48 am
Bjorn Westergard
I also wonder if he deliberately chose a kid with sibilant heavy speech.
October 6, 2009 @ 11:48 am
vcd
Interesting piece, but I’m not so sure you would be able to tell it was based on speech if both the original audio was no played on top of it, or if the transcription was not being shown to read in time. The work
done with the piano is pretty stunning though.
Leave it to Peter to turn something borderline sensational into something completely sensationalized (re: last paragraph).
October 6, 2009 @ 11:56 am
shamburglar
anybody know of any decent audio to midi apps for Mac?
October 6, 2009 @ 11:59 am
Adrian Anders
TS-AudioToMIDI dev should invest the time to make a VST plug-in version of his software. I would be interested in it then.
October 6, 2009 @ 12:46 pm
Das Kraftfuttermischwerk » Das sprechende Piano
[...] Wie genau das funktioniert und warum Peter Ablinger erklärt der Film ausreichend. (Direktlink, via Create Digital Music) [...]
October 6, 2009 @ 12:55 pm
Dano
@shamburglar
Similar thing for Mac:
http://widisoft.com/english/mp3-midi-products.html
October 6, 2009 @ 3:00 pm
Jay Smith
I use WIDI for mac. I made a “player piano” video a while back for the ohm64 using it and show how it is done here http://www.youtube.com/watch?v=KkKESe_QdKE
October 6, 2009 @ 3:01 pm
nick kent
Well he’s encoding in a seemingly similar process vocoder but decoding in a non-traditional way.
If you think he’s mixing the original with the piano then he’s definitely cheating. It doesn’t look realtime to me but perhaps that might be a reason why it would be somewhat excusable to hear the original
If it is not cheating I think is very impressive. A traditional vocoder adjusts each frequency band’s volume continuously. A piano just has a velocity, short attack and long decay that can be dampened, so besides the rich harmonic pitched sound vs bandpass decoding.
October 6, 2009 @ 3:30 pm
kobe
one word: Melodyne.
October 6, 2009 @ 4:53 pm
Ivica Bukvic
Looks to me that the desktop was actually running Linux/KDE with Pd, rather than Windows. Also, the person at the computer looks awful lot like Winfried Ritsch from IEM where they do a lot of work with Pd. So, it seems unlikely that the Windows app in question is being used here and more likely that the whole thing is done in in Pd…
October 6, 2009 @ 9:36 pm
Peter Kirn
@Ivica: You’re completely right. That is indeed very clearly KDE. And there’s a big honking “X” in the other window. ;) So, yes, I agree, and I should get back to learning more Pd signal processing kung fu.
October 6, 2009 @ 10:18 pm
Stij
Wow. I’ve often wondered if something like this was possible, but I’ve never had any idea of how to implement it. If this is legit then it’s very impressive.
October 6, 2009 @ 10:25 pm
jens-oliver
Here’s a piano only video http://vimeo.com/1483630. Not the same text and with additional notes. Very amazing.
October 7, 2009 @ 3:36 am
Stij
Hmm…yeah, it isn’t nearly as intelligible without the original voice mixed in, but you can still hear some of the sibilants.
It also sounds extremely creepy!
October 7, 2009 @ 2:46 pm
GMM
Wow this is amazing. And it is only a piano. Imagine when you have a whole orchestra scored and conducted to reproduce speech, and then further on, a whole orchestra running in realtime as a vocoder!
October 7, 2009 @ 4:12 pm
Peter Kirn
Here we go – here’s the full explanation of how the whole thing works, including a blurry image of the Pd patch.
http://ablinger.mur.at/docu11.html
I must say, I love the idea of pixelation – this is something that, as a general approach, could be attached to a wide variety of work.
Oh, and I actually prefer the more abstract rendition minus the overlaid speech. Who needs intelligibility? It’s gorgeous.
October 7, 2009 @ 4:21 pm
Fishboy
Why are so many commenters focusing on the sibilants? What makes them more interesting than other phonemes/classes of phonemes?
October 7, 2009 @ 4:26 pm
Peter Kirn
I’m not a linguist, but sibilants are essential to understandability, and they’re the thing that would theoretically be hardest to hear on a piano which is least able to produce broad-band noise (versus formants/vowels). If you listen to the piano without the voice, in fact, it’s what seems to be largely missing.
October 7, 2009 @ 4:32 pm
Fishboy
So are you saying you hear vowels in the video without the actual voice layered in? http://vimeo.com/1483630 I couldn’t hear a voice in that one, myself, at least not well enough to make out any words or phonemes – vowels, sibilants, or otherwise. I guess to my ear it sounded vaguely vocal. But anyway, I thought the most interesting would be vowels, especially diphthongs, since the language used is English.
October 7, 2009 @ 4:52 pm
John
I’m not clear, how is this concept of “pixels” really any different than that of wavelets?
As an aside, I’m not wholly convinced that they *are* mixing in the original audio on the feature video. Upon hearing the kid’s plain voice, his formant seems different than what is coming from the piano audio. Is there anything other than subjective listening which would indicate that they are mixing in the original audio? The Vimeo clip Fishboy links to is difficult to compare, simply because of the vastly different acoustics, different piano AFAIK, and it doesn’t seem to have the dampening that the one above does.
Interesting work regardless of this point.
October 7, 2009 @ 6:24 pm
Dub
Also covered by MeFi
October 8, 2009 @ 12:43 am
The Speaking Piano « The Kewl Doodz ‘n’ Chyx
[...] by Martin Poulter on 8 October 2009 An amazing hardware hacking project: a mechanical piano, computer-controlled, becomes a speech [...]
October 8, 2009 @ 8:26 am
vanillechip.de » Speaking Piano
[...] kraftfuttermischwerk & createdigitalmusic] Tweet This!Share this on FacebookPost this to MySpaceShare this on del.icio.usDigg this!Share [...]
October 8, 2009 @ 8:55 am
Audio Damage Tattoo, Synthgeek Kurzweil K2000S samples, Talking Piano, Build a drone synthesizer
[...] # Create Digital Music » The Speaking Piano, and Transforming Audio to MIDI [...]
October 9, 2009 @ 9:50 am
Mechanical piano hacked to talk, says nothing you’d be interested in | Reaction Radio
[...] Read | Permalink | Email this | Comments Go to Source [...]
October 9, 2009 @ 10:10 am
Mechanical piano hacked to talk, says nothing you’d be interested in « Tech Pulse!
[...] Read | Permalink | Email this | Comments Loading… @import url("http://www.google.com/uds/css/gsearch.css"); window._uds_vbw_donotrepair = true; @import url("http://www.google.com/uds/solutions/videobar/gsvideobar.css"); .playerInnerBox_gsvb .player_gsvb { width : 320px; height : 260px; } function LoadVideoBar() { var videoBar; var options = { largeResultSet : !true, horizontal : true, autoExecuteList : { cycleTime : GSvideoBar.CYCLE_TIME_MEDIUM, cycleMode : GSvideoBar.CYCLE_MODE_LINEAR, executeList : ["ytchannel:theworacle","ytchannel:luckymauro","ytchannel:mttdx"] } } videoBar = new GSvideoBar(document.getElementById("videoBar-bar"), GSvideoBar.PLAYER_ROOT_FLOATING, options); } // arrange for this function to be called during body.onload // event processing GSearch.setOnLoadCallback(LoadVideoBar); Filed under: Engadget No Comments Comments (0) Trackbacks (0) ( subscribe to comments on this post ) [...]
October 9, 2009 @ 10:26 am
Mechanical piano hacked to talk, says nothing you’d be interested in
[...] Read | Permalink | Email this | Comments Tagged with: art austria blackmothsuperrainbow engadget internet midi music peter ablinger peterablinger pure data voice [...]
October 9, 2009 @ 10:48 am
Mechanical piano hacked to talk, says nothing you’d be interested in | TechRoo.com| Tech News, Gadget News
[...] Read | Permalink | Email this | Comments [...]
October 9, 2009 @ 11:08 am
Mechanical piano hacked to talk, says nothing you’d be interested in at Technology Nirvana
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email [...]
October 9, 2009 @ 11:09 am
Mechanical piano hacked to talk, says nothing you’d be interested in
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email this | Comments Comments [0]Digg [...]
October 9, 2009 @ 11:10 am
TechTrance.net – Where Technology is a Passion » Blog Archive » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email this | Comments Related ArticlesBookmarksTags [...]
October 9, 2009 @ 11:10 am
:: TecnoloGeek :: » Blog Archive » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email [...]
October 9, 2009 @ 11:12 am
Mechanical piano hacked to talk, says nothing you’d be interested in |
[...] Read | Permalink | Email this | Comments Tags : Art , austria , Black Moth Super Rainbow , BlackMothSuperRainbow , European Environmental Criminal Court , EuropeanEnvironmentalCriminalCourt , hack , midi , mod , Peter Ablinger , PeterAblinger , Piano , pure data , PureData , speech , vocoder , voice No comments for this entry yet… [...]
October 9, 2009 @ 11:12 am
New Mobile Reviews » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email this | Comments Categories: Black Moth Super [...]
October 9, 2009 @ 11:13 am
Mechanical piano hacked to talk, says nothing you’d be interested in – ComputerUser.ca
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email [...]
October 9, 2009 @ 11:15 am
Mechanical piano hacked to talk, says nothing you’d be interested in
[...] Read | Permalink | Email this | Comments Tagged with: 9th-2009 austria hack midi mod networking neutral peter ablinger peterablinger piano Politics Sport [...]
October 9, 2009 @ 11:23 am
Amazing Digital » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email this | Comments Posted on October 9, 2009 at [...]
October 9, 2009 @ 11:30 am
Mechanical piano hacked to talk, says nothing you’d be interested in | Technology South Africa
[...] appeared on Engadget on Fri, 09 Oct 2009 10:07:00 EST. Please see our terms for use of feeds.Read | Permalink | Email [...]
October 9, 2009 @ 11:31 am
Its All About A Digital Technology » Blog Archive » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] Read | Permalink | Email this | Comments [...]
October 9, 2009 @ 11:35 am
Mechanical piano hacked to talk, says nothing you’d be interested in | Tech2Crave
[...] Read | Permalink | Email this | Comments Bookmark This Post Comments (0) [...]
October 9, 2009 @ 11:37 am
Mechanical piano hacked to talk, says nothing you’d be interested in « Mini Apple Store
[...] Read | Permalink | Email this | Comments [...]
October 9, 2009 @ 12:07 pm
piker
so what. he got a computer. good for him.
October 9, 2009 @ 3:39 pm
Mechanical piano hacked to talk, says nothing you’d be interested in | SalDee.com
[...] Read | Permalink | Email this | Comments Related Reading: Moon In My Room Batman Begins [UMD for PSP] iTouchless Stainless-Steel Hands-Free 13-Gallon Infrared Automatic Trash Can Get Smart (Single-Disc Widescreen Edition) Travel Accessories Samsonite UK Grounded Uncle Milton Rainbow In My Room Share this on del.icio.usStumble upon something good? Share it on StumbleUponTweet This!Share this on FacebookPost this to MySpaceShare this on TechnoratiAdd this to Google Bookmarks Permalink|Comments RSS Feed – Trackbacks are closed|post a comment. [...]
October 10, 2009 @ 7:07 am
RSS For Gadgets » Mechanical piano hacked to talk, says nothing you’d be interested in
[...] Read | Permalink | Email this | Comments Rate this topic: (No Ratings Yet) Popularity: 0 You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed. [...]
October 10, 2009 @ 7:41 pm
Fit durch Musik: Das Treppen-Piano soll den Leuten Beine machen | Basic Thinking Blog
[...] ich euch an dieser Stelle nicht vorenthalten möchte. Nachdem ich gestern schon über ein sprechendes Klavier gestolpert bin, ist dieses Treppen-Piano ja mal der [...]
October 16, 2009 @ 10:56 am
IHR #75 (Enhanced) – Podcast Awards, Synth Strings, Music Creator 5, Lily Allen | Inside Home Recording.com
[...] Speaking piano [...]
October 18, 2009 @ 12:58 am
Jhhl
Speaking orchestra? http://www.heraldscotland.com/speakings-a-new-musical-language-1.838418
Harvey has done excellent work for decades.
To blow my own horn: my Amiga program RGS is a real time spectrogram paint program (from 1987 originally), which could send out spectra as MIDI information, therefore being able to make my (microtonally tuned) DX7 to emit intelligible and unintelligible speech. http://www.echo.net/~jhhl/Mp3/RGS/
October 21, 2009 @ 1:20 am
Pianoman
Intelliscore is another program that converts audio to MIDI. It works with the latest versions of Windows, including Vista and Windows7. The website is: http://www.intelliscore.net/
November 18, 2009 @ 3:12 pm
Amazing YouTube Music Video Series #9 -The Talking And Speaking Piano | MusTech.Net: Music Education, Music Technology, & Education!
[...] http://createdigitalmusic.com/2009/10/06/the-speaking-piano-and-transforming-audio-to-midi/ [...]
November 21, 2009 @ 11:06 am
Luke Loeffler :: Recreating the Speaking Piano
[...] was really excited by Peter Ablinger’s Speaking Piano–a system that takes human speech and translates it to a sequence of notes to be played on a piano [...]
January 2, 2010 @ 7:21 pm
Sylvaiw
Why do you say the original voice is mixed with the piano ? Where did you get this information ? I can’t find it.
In my opinion only the piano is heard. and that’s the whole interest of this thing.
January 26, 2010 @ 10:54 am
Leave a comment
RSS feed for comments on this post. TrackBack URI