Mediation and liveness
Auslander (2008) criticises distinctions between live and mediated performance from only spatial (co-presence and absence) and temporal (simultaneity and anteriority) variables, arguing that the concept of liveness itself is unsettled and subject to historical redefinition in relation to technological development. Hybrid events with live, and otherwise, recorded and technologically mediated elements are now commonplace in both mass-media and popular concert settings and continuously shape our expectations and perceptions of what counts as live performance. One example of such hybrids is the notion of ‘live recording’ often used to represent a recorded concert performance or music produced by musicians performing simultaneously in a recording studio. Liveness may thus be viewed as an affective experience rather than an objective category. Going to the roots of the concepts of ‘technical’ and ‘mediation’, Auslander concludes that ‘there can be no such thing as technologically unmediated performance because performance is itself a technology and the idea of a performance is a mediation that shapes audience identity and perception of an event’ (117). Such critique opens the possibility of considering how various technological mediations shape our perception and understanding of the world (Rosenberger and Verbeek 2015).
As one aspect of mediation, in Dancing Dots, dance movements recorded from a previous live music–dance interaction have been sonified, and these sonification recordings are used with live music and dance, forming a hybrid event of live and pre-recorded performance. The motivation of the piece is to bridge across performance contexts and to expand the social dance — which entails a live interaction — into an immersive visual/sonic performance space. This is attempted through augmented movements aiming to provide the spectator/listener with a heightened impression of the musical qualities embedded in the dance. The paradox the performance is facing is that, despite the pre-recorded character of the sonifications, they are intended to create a heightened experience of the ‘liveness’ of dance and music interaction, in which the hybridity of performance itself challenges the live interaction.
A second aspect of mediation entails the ongoing processes of reproduction, reviving, and reinterpretation that make a music tradition. Rosenberg (2019) examined how ways of transmission affected the stability or variation of aspects of a song performed over time. These aspects — ‘the singer’s imprint’ — were found to be influenced by the learning context, whether it was directly from a performer, a sound recording, or a written music notation. Traditional/folk music performances carry references to previous musicians — and it is common for practitioners to relate their practice to historical sources. In this way, traditional music performance involves a mediation of previous practices and can be seen as interactions between current and historical performances. Fredriksson (2023) studied recent performative uses of archive recordings in contemporary Swedish folk music that reveal such interactions — what Fredriksson calls ‘musical wormholes’ that augment the processes of tradition between past and present performers. In Dancing Dots, the influences of historic performers are not explicitly displayed but are present in the choice of music and dance. Still, an earlier recording provides background for performing other elements, tunes, and improvisations beyond the original recording. This hybridity between past and present, live and pre-recorded, becomes the space for negotiating the liveness of the performance.
The challenge, lying at the centre of the transposition of body-centred to data-centred transmission, is to make digital renderings of dance data accessible and intelligible. This emerging research requires a new analytic approach to dance that is not readily available in existing dance literature.
(Karreman 2017: 83)
Photography of this sort might be better understood as a device for translating the unseen or unseeable into something that looks like a picture of something that we could never see.
(Mitchell, as quoted by Karreman 2017: 101)
In her thesis, ‘The Motion Capture Imaginary: Digital Renderings of Dance Knowledge’, Laura Karreman (2017) discusses motion capture (mocap) as indexical traces of movements. Drawing on semiology, such digital footprints are connected to, without sharing any other similarity with, the object they refer to. Mocap data generally consist of vast amounts of numeric data. However, rendered as a sensory output, such data can ultimately ‘bring out the kinaesthetic properties that are strongly associated with (i.e.) dance’ and, as such, gain an ‘indexical authenticity’ (121). Karreman points out that mocap practices inherited paradigms from their precursors, photography and film, and she discusses problems of optical bias, staging, and dimensionality. Just as photography has offered visual evidence for what lies beyond the capacity of the human eye, mocap promises an unveiling of the unseen by the possibilities of capturing movement and replaying it in multiple spatial angles, dimensions, and scales. Like the development of photography, mocap is conditioned by technical limitations that require careful staging.
Against a presumed indexical authenticity of mocap it stands that, for most renderings, mocap data are processed, smoothened, and manipulated to be then projected on virtual bodies. In the film and media industries, this has affected labour conditions, with the development of new roles, such as performance capture actors that provide performance data for further creative digital post-production. The sharp relation between sign and referent can thus become vague and ‘flickering’ as new hybrid forms of performance develop. Furthermore, doubts can and have been raised about the extent to which data from markers of the surface of dancers’ bodies offer an epistemological entrance to dance knowledge of the ephemeral and embodied experiences of dancing. Karreman describes the uses of motion data in conceptualising and abstracting choreographic ideas and principles, exemplified by Forsythe’s Synchronous Objects (2009), which visualises structures and forms in choreography. Through this and other works, Forsythe established the concept of choreographic objects, intended to communicate the thinking in movements across disciplines. In a related way, the present approach aims to communicate dance by abstracting sounding musical objects through movement sonification.
Mediating dance through sound
Compared with most renderings of mocap data, sonification adds another layer of mediation by converting optically registered movement into sound, which follows a mediation from the visual into the auditory domain. Karreman’s work ignores auditory renderings of motion-captured dance; still, her topics are relevant for the sonification of motion-captured dance. This pertains to sonified movement data as sonic traces of movements. Sonification of recorded markers transfers an optical registration into a sonic modality. The sound design and parameter mapping in such transfer are bound to influence the perceived ‘indexical authenticity’ of these sonic traces. Sonification of movements can take advantage of the fact that human cognition is conditioned by our somatic presence in the world, allowing us to sense the physicality and presence of beings and objects across all sensual modalities. Cross-modal and multi-sensory perceived loops between movement and sound are also prosaically embedded in the practice of folk music and dance: dancing and playing entails seeing and feeling movements through fingers, arms, feet, legs, weight, touch, vibrating strings, bows, etc. Renderings of movement as sounds therefore easily tap into the bodily practices of dancing and playing.
Rosenberg (2021) investigated collective methods for improvisation, creativity, and flow from within a ‘cognitive frame’ of folk singing. This cognitive frame is shaped by the stylistic expressional features of folk singing, contained in the oral/aural character of the tradition: modes of variation, tonality, relations between spoken language, rhythm, and articulation, etc. Similarly, a music and dance style can work as a cognitive frame for an embodied understanding and creative exchange of impulses and responses in movements and sounds. Sonified dance might offer additional affordances for understanding the components of such agreed interpretations by investigating the conditions for experiencing music and dance through the same auditory modality.
A key question for this project is how to use sonification to create a meaningful experience of mocap dance data. Two perspectives can be used to understand the sonification design and interpretation of the sonic results, both related to music and dance practice. The two perspectives are not mutually exclusive and can be used to better understand the variety of possibilities with dance sonification.
The first perspective focuses on how sound renderings become meaningful through an embodied understanding of dance movements. Like Giomi’s (2020) exploration of somatic sonification in dance pedagogy, this perspective relies on the embodied experience of movement, which Dregelid refers to as ‘Gerör’, in connection to qualities in the dance style. As an example, Dregelid stated on dancing to a sonification (see Video 3, 0:05–0:10 and 1:00–1:10) created by white-noise generators mapped to feet data:
With these wind-like sounds, I get a strong inclination for rotation, for the horizontal body movements, and for the shape and speed of how we rotate in the dance. I feel this in contrast to dancing with your live playing when I sense movement in all directions. (Dregelid 2023)
She compares the specificity of the qualities communicated through this sonification to the diversity of movements afforded her by real, live music.
The second perspective focuses on how the sound renderings become meaningful within the musical context of the polska. This perspective relies on knowledge of the music and dance style and can, to some extent, be represented by music theory concepts of rhythmic, metric, and tonal relations. In the Video Abstract, two sonification approaches that were created and motivated by the music are presented. The first approach (see Video 3, 1:30 onwards) uses the maxima of downward velocity, captured from markers on the dancers’ backs, to trigger distinct notes tuned with the music. This sonification was chosen because these trigger points were found to be in time with the beats of the music, and the mapping to shorter sounds aimed to bring out this relation as a music phenomenon. However, in interviews, Dregelid and Berchtold stated that these shorter, distinct sounds felt less suggestive for their dancing as compared to the other sonifications (Berchtold 2022; Dregelid 2022). The second sonification approach (see Video 3, 1:45) uses modulated looped fiddle samples sent through a bandpass filter, narrowly tuned in steps conforming to frequencies of partials of the fiddle tone. This sonification creates a continuous, drone-like sound, with embedded arpeggios of overtones because of the frequency of the bandpass filter following the vertical position of the dancers’ back markers. This sonification brings out the continuous svikt patterns and creates a soundscape fitting the music’s tonality and metre. This design offered multiple interpretations to the dancers without inducing the specificity of a particular movement. Both these sonification approaches provided stylistically functional accompaniments for the fiddler, as illustrated by the fiddle improvisations at the end of the video abstract (Video 3, 2:35).
Sonification of movement can thus serve various objectives depending on purpose, ability, and practice. For a musician attempting to play music to a sonification, there may be a need to extract rhythmic and metric clarity, focusing on movement points that synchronise with beats in the music. For dancers, it may be interesting to experience the speed or arc of a movement, potentially favouring different types of sonification, i.e., continuous sounds. For a student seeking an understanding of the mechanics of the dance, the synchronisation of steps in the rotation can be the goal, while a more music-oriented perspective seeks an understanding of how such synchronisation relates to music metrical structures. Mocap dance data typically contain noise, along with unstructured and redundant parts, which usually require filtering and smoothening depending on the desired outcome. Taking into account these processes, along with aesthetic preferences and modes of mediation, the sonification of movement can assume a variety of sound forms and provide numerous simultaneous outcomes, interpretations, and interaction possibilities, of which this project can only skim the surface.