‘EXPRESS MINE’

Supported by Playtronica, this first of its kind playable AI audiovisual sculpture by Portrait XO is a 3-week snapshot of time through the lens of human-machine collaboration.  Using AI to generate audiovisuals based on the history and inspiration of Zeche Zollverein, this instrumental sculpture is a tribute to the intersection of humans, machines, and animals.  The audience members were prompted to interact by touching pieces of copper wrapped coal to trigger 10 AI audiovisual compositions.  A sculpture that consists of many layers, each audiovisual presents the way machines generate new visuals and sounds from a series of AI models.  The data sets that were fed into each machine learning model were selected to translate the stories of the way canary birds and horses were used as tools for labor during the coal mine industry in Germany.  There were audio recordings of these animals and sounds from the coal mines used for machine learning to generate new sounds.  Using DDSP autoencoder and Timbre Transfer, there are short melodies in the sculpture of a flute and canary birds singing the farewell to underground coal mining song that was sung by coal miners in December 2018 –  'das Steigerlied'.  Additional AI generated audio you hear were created from MelSpecVAE google colab notebook by Moisés Horta Valenzuela.  The visuals are a combination of techniques including the use of VQGAN and StyleGAN models, and Ebosuite for Ableton. 



The pieces of coal in this sculpture are from the local region that were dug up from 1,040 meters below ground. 

Special thanks to Zeche Zollverein for providing the resources, Sasha Pas, Hotgrits + Dent, and the entire open source community. 

---

Welcome to my mini process document.  As I have more time later and remember more details, I will add more to this.  For now, here is a short glimpse on how I used AI for this installation and some AI generated samples using these open source colab notebooks.  Links to samples and my canary bird instrument are all free to use and you’re welcome to download to experiment.  Please feel free to share new interesting findings from your experiments using any of this content.  You don’t have to credit me, but would appreciate it.  And I love hearing what others do with this kind of material. 


To learn about how to make your own DDSP instrument using Google Colab, you can see my process document here.

Download canary bird instrument here
MelSpecVAE coal mine audio samples here
MelSpecVAE canary bird audio samples here

---

My mission during this 3-week residency was to use open source AI tools to translate what I learned about Zeche Zollverein and the coal mine industry in Germany.  As most of my work entails sitting in front of a laptop composing, producing, computing, rendering, and waiting (lots of waiting), I’ve been gravitating towards more physical forms of expression like painting and sculpting.  Thanks to Playtronica, I was able to join the 2 worlds of physical and digital by using my ‘waiting’ time to build this interactive playable sculpture.  Some of the prepwork I did before arriving at Zeche Zollverein involved some material research to figure out what textures I wanted to work with - both conductive and non-conductive supportive material.  I was happy with the result of polyurethane mesh as the main structure that was flexible and thick enough to keep its form.  I used a ton of black cable ties of different weights to tie different joints to hold it all in place.  With the help of Julijonas Urbonas, fishing wire and copper wires were used to suspend the sculpture which helped translate my piece to be symbolic of what is currently happening with cloud storage and computing from what we refer to as the ‘black box’ in machine learning.  



For this process document, I’ll focus mainly on the Google Colab models I used and share its audio outputs.

After learning about the history of how canary birds and horses were used in coal mines, that is when I decided to make this installation a tribute piece to humans, machines, and animals.  Canary birds were used to detect carbon monoxide levels in the coal mines.  They would make abnormal chirp sounds or die if the carbon monoxide levels were too high and that’s how miners knew whether or not to enter.  Horses were transported 1,040 meters below ground and kept in the mines to help transport coal around until they became too weak to continue working in the harsh conditions of 40° C and terrible air quality.  





I decided to create a canary bird instrument and have the sound of canaries sing the melodies of das Steigerlied (German miners folk song) by importing these melodies into the DDSP canary bird instrument model in Timbre Transfer.

I uploaded a 45 minute audio of canary birds into DDSP autoencoder trained the audio for 10 days. This is the longest amount of time I’ve trained audio to turn into an instrument, and the result was interesting but not what I expected.  

On day 10, I stopped training at step: 111158    spectral_loss: 5.90    total_loss: 5.90.  It never got to the recommended 4.5-5 spectral loss range.    



First audio output from Canary Bird instrument translating ‘Abschied von der Steinkohle Bergmänner singen das Steigerlied‘

---

Second audio output after increasing ‘quiet’ parameters (to try and get more melodies after noticing the silence between notes):






You can hear the birds sound pretty monophonic.  I suspect this might be because canary birds don’t sing in such a wide frequency range.  Maybe if you’re interested in experimenting further, you can try CJ’s recommendation:

Since the canary birds ended up sounding so monophonic, I used DDSP’s flute instrument to translate the melodies that gave a better outcome.  In the end, I used both the sound of canary birds and flutes.  

Here’s the result of the same audio imported and translated into the existing flute model provided by Timbre Transfer google colab:

---

Next, I trained the same audio of canary birds into the MelSpecVAE google colab notebook.  Because the way this AI model churns out the sound of ‘latent space’ (the in-between journey of sounds moving from point 1 to another), I was curious to find out if by training the sounds of birds, would there be some strange phase cancellation, and how would they interplay sonically?  I love all this  glitchiness.  Some samples:

---

The end result: people were invited to trigger audiovisuals that were played when 2 coals were touched simultaneously (2 mapped as ground connectors) + 10 individual AI audiovisual pieces.  This installation challenged me to think about interaction design related to space, the relationship between the sculpture and lighting, sound reflections, and communicating ‘how to play’ easy enough for people to understand so the entry point to experience would be easy.
 

<sub>𝙀𝙭𝙥𝙧𝙚𝙨𝙨 𝙈𝙞𝙣𝙚
𝗂𝗌 𝖺𝗇 𝖾𝗑𝗉𝗋𝖾𝗌𝗌𝗂𝗈𝗇 𝗈𝖿 𝗍𝗂𝗆𝖾

𝖶𝗁𝗂𝗅𝖾 𝗆𝗂𝗇𝗂𝗇𝗀 𝖿𝗈𝗋 𝗐𝗁𝖺𝗍 𝖨 𝗉𝗈𝗇𝖽𝖾𝗋 𝗐𝗁𝖺𝗍’𝗌 𝗋𝖾𝖺𝗅𝗅𝗒 𝗆𝗂𝗇𝖾 𝗐𝗁𝗂𝗅𝖾 𝗆𝗂𝗇𝗂𝗇𝗀 𝖿𝗈𝗋 𝗆𝖾𝖺𝗇𝗂𝗇𝗀 𝖻𝖾𝗍𝗐𝖾𝖾𝗇 𝖼𝗈𝖽𝖾𝗌 𝖽𝗎𝗋𝗂𝗇𝗀 𝗍𝗁𝗂𝗌 𝗉𝖾𝖼𝗎𝗅𝗂𝖺𝗋 𝗍𝗂𝗆𝖾 𝗐𝖾 𝗅𝗂𝗏𝖾 𝗂𝗇 𝗐𝗂𝗍𝗁 𝗈𝗏𝖾𝗋𝗐𝗁𝖾𝗅𝗆𝗂𝗇𝗀 𝖺𝗆𝗈𝗎𝗇𝗍𝗌 𝗈𝖿 𝖽𝖺𝗍𝖺, 𝗍𝗁𝗂𝗌 𝗉𝗂𝖾𝖼𝖾 𝖼𝗈𝗇𝖽𝗎𝖼𝗍𝗌 𝗍𝗁𝖾 𝗂𝗇-𝖻𝖾𝗍𝗐𝖾𝖾𝗇 𝗌𝗍𝖺𝗍𝖾𝗌 𝗈𝖿 𝗍𝗁𝗂𝗇𝗄𝗂𝗇𝗀 𝖺𝗇𝖽 𝖻𝖾𝗂𝗇𝗀 𝗂𝗇𝗍𝗈 𝗉𝗁𝗒𝗌𝗂𝖼𝖺𝗅 𝗂𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗂𝗈𝗇.

𝖬𝖺𝗒𝖻𝖾 𝖺𝗌 𝗐𝖾 𝖿𝖾𝖾𝗅 𝗍𝗁𝖾 𝗁𝗂𝗌𝗍𝗈𝗋𝗒 𝗈𝖿 𝖾𝖺𝗋𝗍𝗁 𝖿𝗋𝗈𝗆 𝟣,𝟢𝟦𝟢 𝗆𝖾𝗍𝖾𝗋𝗌 𝖻𝖾𝗅𝗈𝗐 𝗀𝗋𝗈𝗎𝗇𝖽 𝗐𝗁𝗂𝗅𝖾 𝗐𝖾 𝗍𝗈𝗎𝖼𝗁 𝗍𝗁𝖾𝗌𝖾 𝗉𝗂𝖾𝖼𝖾𝗌 𝗈𝖿 𝖼𝗈𝖺𝗅, 𝗐𝖾 𝖼𝖺𝗇 𝗅𝗂𝗌𝗍𝖾𝗇 𝖺𝗇𝖽 𝗀𝖺𝗓𝖾 𝗂𝗇𝗍𝗈 𝗍𝗁𝖾 𝗉𝖺𝗌𝗍, 𝗉𝗋𝖾𝗌𝖾𝗇𝗍, 𝖺𝗇𝖽 𝖿𝗎𝗍𝗎𝗋𝖾 𝗌𝗂𝗆𝗎𝗅𝗍𝖺𝗇𝖾𝗈𝗎𝗌l𝗒. 𝖶𝖾 𝖼𝖺𝗇 𝖺𝗅𝗅𝗈𝗐 𝗈𝗎𝗋 𝖼𝗎𝗋𝗂𝗈𝗌𝗂𝗍𝗂𝖾𝗌 𝗍𝗈 𝗎𝗇𝗍𝖺𝗇𝗀𝗅𝖾 𝖿𝗎𝗋𝗍𝗁𝖾𝗋 𝗐𝗁𝖺𝗍 𝗂𝗍 𝗆𝖾𝖺𝗇𝗌 𝗍𝗈 𝖻𝖾 𝗁𝗎𝗆𝖺𝗇 𝗐𝗈𝗋𝗄𝗂𝗇𝗀 𝗐𝗂𝗍𝗁 𝗆𝖺𝖼𝗁𝗂𝗇𝖾𝗌.

𝖬𝗒 𝗅𝖺𝗍𝖾𝗌𝗍 𝗁𝗎𝗆𝖺𝗇-𝗆𝖺𝖼𝗁𝗂𝗇𝖾 𝖼𝗈𝗅𝗅𝖺𝖻𝗈𝗋𝖺𝗍𝗂𝗈𝗇 𝖻𝖾𝖼𝖺𝗆𝖾 𝖺 𝗉𝗅𝖺𝗒𝖺𝖻𝗅𝖾 sculpture in 3 weeks t𝗋𝖺𝗇𝗌𝗅𝖺𝗍es 𝗍𝗁𝖾 𝖾𝗍𝗁𝖾𝗋𝖾𝖺𝗅 𝗌𝗍𝖺𝗍𝖾 𝗈𝖿 𝗆𝖺𝖼𝗁𝗂𝗇𝖾𝗌 𝗍𝗁𝖺𝗍 𝗅𝖾𝖺𝗋𝗇𝖾𝖽 𝗌𝗈𝗎𝗇𝖽𝗌 𝗈𝖿 𝖼𝖺𝗇𝖺𝗋𝗒 𝖻𝗂𝗋𝖽𝗌 𝖺𝗇𝖽 𝗁𝗈𝗋𝗌𝖾𝗌 𝗍𝗁𝖺𝗍 𝗐𝖾𝗋𝖾 𝗈𝗇𝖼𝖾 𝗎𝗌𝖾𝖽 𝗅𝗂𝗄𝖾 𝗍𝖾𝖼𝗁𝗇𝗈𝗅𝗈𝗀𝗒. F𝖾𝖾𝖽𝗂𝗇𝗀 𝗍𝗁𝖾𝗂𝗋 𝗌𝗈𝗎𝗇𝖽𝗌 𝗂𝗇𝗍𝗈 𝗆𝖺𝖼𝗁𝗂𝗇𝖾 𝗅𝖾𝖺𝗋𝗇𝗂𝗇𝗀 𝗆𝗈𝖽𝖾𝗅𝗌 𝗁𝖾𝗅𝗉𝖾𝖽 𝗆𝖾 𝗍𝗋𝖺𝗇𝗌𝗅𝖺𝗍𝖾 𝗍𝗁𝗂𝗌 𝗂𝗇𝗌𝗂𝗀𝗁𝗍 𝗈𝖿 𝗎𝗌𝗂𝗇𝗀 𝗅𝗂𝗏𝗂𝗇𝗀 𝖼𝗋𝖾𝖺𝗍𝗎𝗋𝖾𝗌 𝖺𝗌 𝗍𝖾𝖼𝗁𝗇𝗈𝗅𝗈𝗀𝗒 𝖻𝗒 𝗁𝖾𝖺𝗋𝗂𝗇𝗀 𝗁𝗈𝗐 𝗍𝗁𝖾𝗒 𝗌𝗈𝗎𝗇𝖽 𝗍𝗁𝗋𝗈𝗎𝗀𝗁 𝗆𝖺𝖼𝗁𝗂𝗇𝖾 𝗅𝖾𝖺𝗋𝗇𝗂𝗇𝗀.


𝖨 𝗊𝗎𝖾𝗌𝗍𝗂𝗈𝗇 𝖼𝗈𝗇𝗌𝗍𝖺𝗇𝗍𝗅𝗒, 𝗁𝗈𝗐 𝖺𝗋𝖾 𝗐𝖾 𝖾𝗏𝗈𝗅𝗏𝗂𝗇𝗀 𝖺𝗇𝖽 𝖽𝖾𝗏𝗈𝗅𝗏𝗂𝗇𝗀?</sub>

RESOURCES:

Freesound: https://freesound.org/
Sounds of Changes: www.soundsofchanges.eu
Orange free sounds: https://orangefreesounds.com/
Kiddopedia Animals: https://www.youtube.com/watch?v=JmDYsWjlMOA
Horse sounds: http://www.theallsounds.com/2018/03/horse-walking-sound-effects-all-sounds.html
Assorted gravel rock and stones: https://commons.wikimedia.org/wiki/File:Assorted_gravel_rock_and_stones.ogg