Concept phase
The brief was a two-week assignment: build an installation at Katoenhuis where the visitor feels like a conductor and follows the intensity of the music. It had to work as a standalone experience for at least one visitor while staying musically in sync with the other installations, with the visuals allowed to differ so people would want to try them all. The stated goal was that the visitor should feel like a conductor the moment they pick up the baton. The main anticipated challenges were timing light and visuals to the audio, signalling to visitors which movements suit which moments, communicating when the music intensifies or calms, and avoiding latency.
Final Product
We presented the church with audio reactive visuals, however we couldnt connect the sensor since our setup was too far from the audio booth.
I also made a randomized number clock but we ended up not using it due to it not really fitting and a group bailing on the idea that connected it.
What I learned from this project
I learned to work in a team, how to read and modify scripts with bad notation and to comprimise when things don't work out.
Process
Days 1–2: The first day went into trying to set up the ZED 2i motion camera, and only on day two did it turn out the wrong cable was being used. In the meantime an Xbox Kinect was set up instead, but its hand tracking proved too buggy to rely on. Once someone pointed out the ZED 2i needs a special cable, it worked, and after debugging some settings in the Unity samples the build could start the next day.
Day 3: With no prior experience in object tracking, the tracking caused too many problems, so the project pivoted to coding visuals driven by sound instead.
Day 4: Teacher-provided code was extended with values controlling shape and color, mapping red to bass, green to peak and blue to 1 so the shader's color shifts with the music; the shader uses multiply nodes to intensify the color. Two snags came up: Unity's camera disables post-processing, so the color emission wasn't visible in the game scene, and flipping the shaders (originally meant to make the plane glow) gave a much better effect. Confidence in the sound-driven visuals stayed limited because testing had to go through a virtual audio cable.
Day 5: Hand tracking still wouldn't work because the MediaPipe solutions were deprecated, so it was handed to a teammate to do in TouchDesigner.