Sonification - from Motion Capture to Dance-Music

This section provides a detailed account of the process of sonifying the polska dance. The aim is to critically deconstruct this process to offer insights and results, as well as to illuminate performance strategies. The narration begins with the performances in the motion capture (mocap) studio, followed by the selection and processing of data, and the mapping of movement data to sound. Examples of sonifications are examined in detail, focusing on their rhythmic and metric features in relation to the dance movements. Finally, I include recordings of my fiddle playing to demonstrate sonifications that illustrate the musical rhythms emerging from the movement sonifications. The section thus draws a full circle from playing for dancing in the mocap studio to performing with the sonifications of dance movements. 

In the mocap studio

Over the course of four years, 2017–21, I spent repeated sessions in the mocap studio, recording myself playing fiddle for the dancers Ami Dregelid and Andreas Berchtold. At the beginning of this project, these sessions involved more musicians and dancers; however, COVID restrictions limited the recordings to this closer group. I conducted about six sessions recording various dance types — such as polska, springlek, and schottis. Through these experiences, I gradually developed a method for recording in the mocap studio.

Running a mocap session involves many steps and procedures that can be overwhelming: preparing the studio, learning and managing the software, calibrating a mocap space, setting up additional sound and video recording, ensuring markers and Velcro suits are in order, monitoring and running recording software, and cleaning, labelling, and exporting files. By mastering these steps, I gained some competence in performing this myself; however, acting and thinking simultaneously as a musician, researcher, and technician was exhausting. Engaging a technician to manage the recordings during our sessions made a significant difference in my ability to concentrate on the actual performance practice I wished to investigate. An emerging question within our group was how to capture ourselves performing at our best level. We all had experiences while performing that we would describe as becoming more connected, grounded, in time, relaxed, open, able to let go, effortless, and so on. One of us articulated it as remembering ‘how to do it’. We identified this as a shift occurring recurrently during repeated performances in longer sessions when teaching dance in groups. Similarly, we agreed that these experiences were generally accompanied by a sense of intertwinement between music and dance, of affecting and being affected by each other’s performance. We recognised these moments through profound somatic sensations of tension, release, balance, ease and drive.

For such moments to arrive, we needed to become familiar with the conditions of performing in the mocap studio: the room size, the floor, the acoustics, wearing tight mocap suits and Velcro shoe covers and caps, and being monitored surrounded by cameras in a high-technological studio space. We decided to spend at least a whole day in the studio each time to give space for retakes, repetitions, and reflections to allow things to develop. Furthermore, we met online between sessions to review and discuss our last performances. As a method for measuring the quality of our repeated takes during such longer sessions, we collected our individual reactions immediately after each recording. Each participant completed a survey, rating the last performance on a 1–7 Likert scale and scribbling down notes in response to open questions about their performing experience. After filling out forms in silence, we would tell each other our responses and decide whether we would like a new take with the same music/dance or move on to another piece. In these discussions and notes, many practice-specific terms and wordings were used. Many of these were invented by Ami Dregelid (AD) during her decades of pedagogic collaborations with fiddlers like Ellika Frisell, Sven Ahlbäck, and others, including the author.

Some examples:

• Dansdjur (dance-animal, the function of the couple as one unit with a shared balance, AD)
• Stödben (support-leg, the leg that is currently supporting the balance of the couple, AD)
• Luftben (air-leg, the leg that is currently without floor contact, AD)
• Å-så-hej! (onomatopoetic for a phrasing leading to the first beat, AD)
• Sväng-balans (turning-balance, the way to balance the body’s centre of gravity to facilitate turning, as compared to managing the balance, e.g. when standing, walking, or running, AD)
• Upptakt (musical upbeat)
• Svikt (Blom 1981)

In retrospect, I see the procedures of performing at our own pace, allowing retakes, reflecting collectively and individually, acknowledging our experiences and using our own language as necessary for making and inhabiting a space for our performance in the studio. The Likert ratings also turned out to be on the upper half or top end of the scale, from medium to exceptional, which confirmed that overall, we were satisfied with our performances.

The mocap recordings

The recordings were conducted in the PMIL studio at KTH, Stockholm, using an Optitrack optical mocap system with 17 infrared cameras. We were wearing mocap Velcro suits with full-body marker setups and markers attached to the violin and the bow. One such complete setup included a total of 110 markers. The mocap system recorded the three-dimensional position of each marker, resulting in three data variables (XYZ) for each marker at a rate of 120 frames per second. Optical mocap requires that markers are visible from at least three cameras simultaneously. Although the studio space was enough for us to perform in, the dancers were occasionally dancing at the outer limits of the area covered by cameras — the motion capture space — resulting in partial marker drop-outs and data loss. The Optitrack mocap system includes a system for automatically assigning markers to a body model, which becomes rendered as moving avatars in the programme’s graphic interface. However, the closeness of the dancers in sections of the dance, particularly while turning, resulted in sequences where the automatic body tracking failed, as markers were occluded or mixed up by the system — causing the avatars to assume absurd postures with limbs sticking out in impossible positions. The system provides smoothening and gap-filling functions for interpolating occasional marker data drop-outs. Still, this half-automated procedure of data cleaning would have required weeks or months of manual correction time for one performance only. This motivated me to focus on a manageable subset of the data.

Data selection and processing

From all our mocap studio sessions, the data to be used in the Dancing Dots performance was narrowed down to just one of the recordings out of our session: a (highly rated) take of the tune ‘Polska efter Pellar Anna’ from the fiddler Gössa Anders Andersson’s repertoire.

A previous study showed that players were able to align their playing to point-light displays of single markers (Misgeld, Holzapel, and Ahlbäck 2019). In connection with these findings, I extracted a similar set of markers as in the previous study: markers at performers’ feet/ankles and between the dancers’ shoulders. These markers were chosen as consistent substrates of how the dance expresses the metre of the music: feet markers for capturing steps, transfers of weights, and the player’s foot-tapping; back markers for capturing movements at the body’s centre of gravity, including the svikt. Tidying up the labelling of these markers was a manageable task, also since their placement on the bodies of the performers made them less prone to drop-outs. 

When choosing variables for sonification, I primarily scanned for patterns corresponding to the rhythmic/metric structure of the polska. As described above, each marker had three data variables for its coordinates in three-dimensional space. In addition to this position data, I obtained variables for velocity by calculating the first derivative of each dimension. The movement in the vertical dimension (position and velocity variables) were determined by the dancers’ pacing to the music, connecting to the concept of svikt, and relevant to the expression of metre in polska dance. Furthermore, using the Matlab Mocap Toolbox (Burger and Toiviainen 2013), a derivative of the Euclidean norm vector was calculated to measure each marker’s overall movement velocity. This variable represents the speed magnitude of a marker regardless of its spatial direction. This is useful for representing feet and body movements, i.e. during rotation. Therefore, the three movement variables, vertical movement, vertical velocity, and speed, became the focus for mapping with audio in the next step.

Audio mapping and sonic design

The mocap data were sonified by mapping them to audio generators, i.e., digital instruments including noise generators, samplers, and oscillators with gain, filters, and envelope parameters set to be controlled by the data variables. We implemented these mappings in the web-based interface SonifyFOLK, depicted in Sonification examples 1 and 2 (for a further description of the application, see Misgeld, Lindetorp, and Holzapfel 2023). This tool allows users to try out mapping from movement data to sound parameters through a web browser without installing further software. The design of the digital instruments was guided by considering affordances for playing and dancing with sounds fitting within the framework of the folk music/dance style. This was achieved by trying sonifications with playing, getting feedback from the dancers, and conducting workshops with music and dance students (ibid.).

The audio mappings in the SonifyFOLK prototype interface were kept at a minimal complexity, following the idea of providing accessibility to sonification for non-experts. This means that in general, each variable controls a single parameter of one audio generator, for instance, the pitch of an oscillator or the frequency for filtering white noise. The more advanced sound designs involved controlling amplitude envelope parameters for percussive sounds and modulating overtones of a violin sample. For the Dancing Dots performance, these sounds were processed further. Still, the character of transparent movement sonification was maintained, aiming for a playful, energetic, and explorative expression inspired by using a course-grained, straightforward sound synthesis. As the sonifications followed the repetitive dance movements, they provided textures over which the fiddle’s melodic gestures and passages contrasted its warmer acoustic timbre to the synthesised sounds. When sonifications involved pitched sounds, these were selected to support the idiomatic folk music-style fiddle improvisations, for instance, using overtones or drones blending into the tonality of the fiddle music. In the sense of such complementary textures, the sonification sounds and live playing thus reflect the dance-music bi-modality, where each part inhabits its own expressive (sonic or spatial) space but interacts within the shared polska idiom. This relation can crudely be exemplified by how musicians and dancers independently can shift between parts of a melody and sections of walking and turning in the dance.

Sonification examples

The following examples will illustrate some resulting sonifications and show how the choice of marker and movement variable affect the affordances for interpreting sonifications with rhythmic-metric features of the music. In Dancing Dots, these and other sound mappings were used to compose the full sound design of the performance. This space doesn’t allow going into details of all these audio mappings; rather, it attempts to illustrate outcomes of sonifying different dance movements by mapping them to the same parameters of a simple oscillator and a white noise generator.

Sonification example 1: Svikt during promenade and turning 

The vertical positions of markers on the two dancers’ upper backs are mapped to two audio units (Noises 1 and 2), both generating white noise that passes through a bandpass filter. The frequency of each audio unit’s bandpass filter is controlled by the position data of one dancer. The svikt-pattern changing between turning and promenade sections can be heard as alternating between three and two (one long and one short) oscillations per measure, respectively. 

The following Sonification examples 3, 4, and 5 are all sonifications of the dancers’ feet, using four data sources: markers on the left (LF) and right (RF) foot of the left (LD) and right (RD) dancers, from which the variable for overall speed was chosen as the data source controlling the sonification. Each data source is connected to one white noise audio generator with a bandpass filter controlling the output. The parameters affected by the data source include the frequency and bandwidth of the bandpass filter and the output gain of the audio generator. The examples contain repeated sonifications of a subset of the performance data: eighteen measures with the dancers in a turning section. The sounds in the examples are crossfaded between looped combinations of different data source sonifications. This setup made it possible to explore the character of different movement sonifications across the same section of the dance. A sonification of the player’s foot-tapping (marking beats 1 and 3) at the start of each example is added to give an initial cue to the metre.

In example 3, all parameters are mapped so that higher values in the data source (higher speed) correspond with higher frequency, narrower bandwidth, and a higher gain. As a result, the sound becomes louder, higher in pitch, and more whistling when the feet move at a higher speed.

In example 4, all parameters are mapped inverted to (a) — so lower values in the data source (lower speed) correspond with higher frequency, narrower bandwidth, and a higher gain. As a result, the sound becomes louder, higher in pitch, and more whistling when the feet move at a lower speed.

Example 5 combines these sonifications of higher and lower speed for each marker, resulting in one sound combination for each foot.

Each of the examples includes one sound recording of the sonification and one recording of my fiddle playing to the sonification. Here, I use another tune in the tradition of Gössa Anders: ‘Hambraeuspolskan’, which is in a similar style as the tune ‘Polska efter Pellar Anna’ played in the mocap recording session.

The sound waveforms illustrate how the sonification aligns with the polska metre and are presented, from top to bottom, in the order of sounds introduced in each example.