Apple and Danilin developed the ‘Conductor’ installation for the AI Songcontest international music competition. Appel: “We used a camera to observe people. Initially, we started with hand poses, followed later by key points around the entire body. This then makes it possible to influence the music through physical movement. We used the software programme Ableton to create a piece of music in which we could adjust various parameters. In a number of demos, we looked at what piece of music should be different and what movements people made. If you move your hand up, the pitch also goes up. If you move faster, the pace quickens. If you wave your hands, you generate a wind sound. These are just a few examples of how the music adapts to your movements.” Users will ultimately be able to use their whole body to influence the music.
Meaning thanks to AI
As a participant or user of the installation, you basically ‘work’ within the parameters pre-programmed by Appel and Danilin. There are also some algorithms in there, allowing users to randomly generate new notes with their hands or body. But what exactly makes this installation AI? “The artificial intelligence lies in detecting and interpreting body movements, not so much in creating the music itself,” says Appel. “If you measure a video in which someone is happy or angry, you get purely pixel values. There is not much else you can do with that. AI on the other hand creates meaning by not just seeing an image from the video, but also by interpreting it and attaching a value to it. For example, AI sees a body posture or a happy or angry face and then gives meaning to it by converting it into music to match.”
Anyone can make music
The idea first came – like so many good ideas – in the shower. What is the underlying goal? To give people who do not play an instrument an opportunity to nevertheless express themselves through music. And it works. Appel: “At the Radboud Sounds festival, a visitor came back four times. He was very emotional and grateful for the opportunity to express his feelings in music thanks to this installation. That is the greatest compliment you can get as a creator.”
Music from the mind
Michelle dreams big: “I’d like to find a way to make music with my mind. For a piece of music that is in my mind to be expressed through AI. Here is an example of how that might work. Some of my fellow PhD students are currently working on a way to decode images using AI. Subjects look at faces while their brain data is recorded. AI then knows what they’re looking at. We could do something similar for music; that would be amazing!”
For now, the next step for Appel and Danilin is to continue to work with music and movement. In a new adaptation on their interactive installation, two dancers are influencing the music. The performance can be viewed at the Kunstnacht on 16 September 2023.
This article was inspired by the lustrum festival Radboud Sounds.
Text: Pleuni van Keulen voor Radboud Recharge. Photo: James Coleman via Unsplash
Radboud Recharge: the Sound of Summer, playlist
This summer on Radboud Recharge: the Sound of Summer. Featuring five articles on music, approached scientifically. And with a playlist on Spotify: Radboud Recharge: the Sound of Summer. Michelle Appel en Daniel Danilin added these songs:
* Spacetime traveller, Astrotone (een nummer van Daniel zelf)* Oxygène, C418* Atlantis (I Need you), LTJ Bukem