Our integrated audiovisual system employs a co-composed architecture where both the generative soundscape and light installation operate under shared regime logic, progressing through a limited set of pre-programmed behavioral states in parallel with the ‘phenomenological switch’ we mentioned before. While maintaining distinct material identities– the sound material is purely synthetic and the light setup nothing but a grid of 24 white lamps– their generative behaviors remain structurally coupled. The activation of the phenomenological switch prompts coordinated transitions between behavioral regimes for both light and sound, without prescribing specific ones. The secondary binding layer is a meta-parameter we came to call ‘energy’ of the system, which maps real-time measured amount of movement to proportional intensity scaling across both domains – simultaneously affecting light brightness/activation density and sound amplitude/spectral complexity.
Implemented through Max/MSP, the system features:
- A regime engine managing state transitions for both media streams
- Chaotic pattern generators driving the light installation's non-linear on/off sequences
- Synthetic sound synthesis directly informed by concurrent light states
- Spatialization routines that mirror light distribution patterns acoustically
In this experiment, we developed a custom smoothness metric to analyze and interpret movement of a person in real time. This metric was derived from two primary data sources: body landmarks estimated from 2D video capture and accelerometer data collected from a smartphone securely attached to the mover. The smoothness metric allowed us to identify key moments of flow—where high smoothness values indicated that the movement remained within a consistent movement regime—and interruption, where low smoothness values suggested a transition into a new regime. Further on we conceptualised these transitions between flow and interruption as 'phenomenological switches' that served as triggers for modulating a set of behaviors within the framework of the Desire Machine assemblage.
At the center of this setup is an algorithmic agent, which is conceptualised within the logic of the actor-network theory and moderates as the actant behind the 'phenomenological switch', introducing a regime-based logic to the interaction. The algorithmic agent is embedded in a constant feedback loop that links it dynamically to the movement, the sound, and the lights via an OPC (Open Platform Communications) server. Through this protocol, the agent sends and receives data in real time, aggregating movement signals and translating them into a system of thresholds that steer the audiovisual output.3
At the same time this influence is never one-directional: the sound and light, shaped by the algorithm, modulate the movement behavior, which in turn recursively feeds back into the algorithm. This creates a recursive ecology where agency is distributed and always in flux—an entangled field of influence rather than a clear causal chain. By deciding when the movement regime changes, this code actively shapes the evolving action, determining when and how transitions occur between partially preset behaviors of the audio-visual devices. As Müller4 highlights, actor-networks challenge traditional notions of power by foregrounding the relational and processual dimensions of agency, where human and non-human elements contribute to the production of meaning and affect. By foregrounding this mediation, we reconsider the digital component not as a passive conduit, but as an active participant in recomposing the audiovisual landscape.
In moving from a view of the algorithm as a tool to one as an agentive component of the assemblage, we allow for a re-reading of digital technology beyond representation or execution. Its function is not to symbolically interpret the movement, sound and space but to actively co-compose it. Its output is not a mere deterministic function of its input, rather it operates as an autonomous agent in the assemblage. The algorithm, in this context, destabilizes the human/non-human dichotomy and acts as the a-signifying semiotic component that functions without representation but directly modulates flows and operations instead.
3. Artacho, Adrián, and Leonhard Horstmeyer. 2022. SmoothOperator: A Device for Characterizing Smoothness in Body Movement. Proceedings of the 8th International Conference on Movement and Computing (MOCO 2022). University of Music and Performing Arts Vienna.
4. Martin Müller, ‘Assemblages and Actor-Networks: Rethinking Socio-Material Power, Politics and Space’, Geography Compass, 9.1 (2015), 27–41.