3. Sonification Design
Of the three conventional methods of sonification, Audification, Model-based, and PMSon (parameter mapping sonification), the last is most widely used for small to large datasets with multiple data properties or dimensions. deCampo’s Design Space Map (2007) proves a useful starting point when designing sonifications, however, there appears to be a canonical propensity, in cases where the design space map approach is applied, to view a one-to-one data-to-pitch mapping as the archetype for an effective sonification, i.e. the Auditory Graph (Flowers 2012). It is true that the ear’s ability to resolve frequency deviations – the “just noticeable difference” (JND) – is apt compared to other mapping strategies within certain frequency ranges, however this often results in sonification designs that can be challenging to listen to for extended periods of time (Vickers et al. 2014; Vickers, Hogg and Worrall 2017). Furthermore, if more than one data dimension is sonified, resulting pitch streams must be kept sufficiently separate in frequency so they do not overlap and cause confusion as to which stream is changing. These considerations are ultimately situated within one of two typical “requirements” of sonification listening: “exact value perception” or a high degree of correct value identification, versus the general perception of significant shifts in a continuous dataset. In the case of raising awareness rather than facilitating scientific analyses of data, this challenges perceptual considerations and aesthetic approaches.Conventional Auditory Graphs may seem appropriate for air quality data, given an exploratory or purely scientific context; however, under the aesthetic and functional requirements for public relations outlined above, an alternative interpretation of the Auditory Graph is used, which we describe in the following sections.
3.1. Time Scaling And Parameter Mapping
We recommend that you listen to the sonifications for both Vancouver and Sarnia first (available as part of a focus group playlist) before reading the description of the design details. We contend that this should demonstrate an implementation that does not require detailed understanding or prolonged training on the part of the listener in order for him or her to holistically perceive which city is more polluted.
To begin, there are a few deliberate design ideas built into the sonification. The first is to holistically map more pollutants in the atmosphere to noise, which makes use of its negative connotations – a kind of “anti-aesthetic” – to help communicate the detrimental effects of emissions to the environment. The second is to make it possible for the listener to differentiate and compare the relative levels of pollutants in each city: an aspiration to “hearability” and “intelligibility” (Barrass 2012; Walker and Kramer 2005). The third is to convey this with temporal effectiveness, that is, to enable the listener to differentiate daily/hourly/monthly cycles of air pollution data respective to the time resolution of the sonification. With these goals in mind, the five metrics are sonified in parallel using a simple mapping strategy with a positive polarity (larger data values equal larger acoustic parameter values). Finally, these mappings are designed to have enough aesthetic and functional flexibility to yield five differentiable streams that do not interfere with each other or cause auditory fatigue (Hermann, Hunt and Neuhoff 2011).
The first and most crucial design consideration is time-scaling. For this sonification, each data value, representing an hour of real time, is reduced to 0.2 seconds of sonification time. A 12-hour day is therefore represented in exactly 2.4 seconds, and a year in roughly half an hour. Daily emission patterns in the data are thus easily perceivable at this scale and rest comfortably within the echoic memory range (deCampo 2007). With regard to streaming, several fundamental ideas drawn from psychoacoustics and sound synthesis provide a framework for the final design. The first is Bregman’s seminal stream segregation grouping cue, which states that, “when two concurrent sounds have different fundamental frequencies, the brain can use the fact that the harmonics that comprise each sound will be a whole number multiple of the fundamental” (Carlile 2011). Considering the design criteria for perceivability, the nature of our dataset, and these perceptual attributes, we chose frequency modulation (FM PMSon) as the most suitable mapping strategy based on its ability to efficiently generate rich harmonic spectra in relation to a fundamental. Frequency modulation is a waveshaping synthesis technique that uses one waveform to modulate the frequency of another waveform. Usually both waveforms are sine waves, but other waveforms can also be used. One wave is called the carrier, the other the modulator. When modulation occurs at frequencies above the audio rate, “sinusoidal sidebands are created at frequencies equal to the carrier frequency plus and minus integer multiples of the modulator frequency” (Cook 2011). The index of modulation is a ratio that indicates the amount of deviation from the carrier signal, and this value determines the number of sidebands on either side of the carrier, resulting in a subjective experience of “noise.”
3.2 Creating “Harmonic Identities” Using Stream Segregation
For this FM PMSon, 4 pollutants, CO, O3, SO2, and NO2, were scaled and mapped to the modulation index of an FM synth in SuperCollider. Each pollutant was given its own fundamental frequency and carrier-to-modulator (c/m) ratio in different regions of the auditory spectrum. This mapping offers multiple affordances: first, given different c/m ratios, each pollutant occupies a fixed fundamental frequency in the auditory spectrum which remains unchanged throughout the sonification. This means that once the pollutant positions are known, it becomes easy to identify which one is changing at any given time. Furthermore, because of the different ratios, each pollutant also possesses a distinct array of sidebands that are harmonic multiples of the modulator, creating unique timbral structures, or harmonic identities, for each pollutant. Importantly, this is what allows for the superimposition of streams on top of each other without perceptual and cognitive occlusion (see Fig 1). As the modulation index goes up for each of the pollutants to the point of overlap between the sidebands, the streams remain differentiable, based on the consistent harmonic relationship to an unchanging and unique fundamental. In the same way that one can perceptually “parse” out the sounds of individual instruments playing together in an orchestra, pollutants which are sonified using FM can be aurally segregated (to a degree) based on their harmonic identities. As pollutant levels go up, the sidebands increase, becoming fuller, brighter, and ultimately noisier. The effects of this mapping are evidenced by comparing a relatively less polluted city like Vancouver to an industrial town like Sarnia, or an even larger metropolitan center like Toronto, where one sounds much more distorted and “harsh” than the other. Essentially, we argue here that designing harmonic identities is one way of creating pragmatic and evocative aesthetics in a perceptually-driven sonification model.
3.3. Particulate Matter (PM2.5)
The fifth pollutant, PM2.5, is measured differently than the other chemicals and therefore receives a distinct mapping. Particulate matter is commonly cited as the most dangerous air pollutant among those measured. Rather than consisting of a single chemical, PM2.5is composed of multiple substances, some of which are quite toxic, that penetrate deep into the lungs, inducing cancer and other related diseases. Because of this, PM2.5is mapped to a granular synth whose click rate increases as the particulate matter increases. This is meant to evoke the sonic archetype of a Geiger counter, where the increasing click rate signifies increased urgency/proximity and, in this case, danger to listeners who experience it.
Figure 1: Left: A spectrographic representation of the model’s frequency and time domain with annotated stereo-space mappings (PM2.5 represented in vertical lines). Right: a schematic of FM synthesis in the temporal-dynamics domain.
3.4. The Importance of Redundancy in Communicating a Message
Stream segregation between the four pollutants (CO, O3, NO2, and SO2) is further reinforced by the redundant encodings of the same data in different mapping strategies (Peres and Lane 2005). Data is encoded onto amplitude, so that the sounds of the pollutants become louder and more salient as they increase, as well as onto the presence of noise bands, which timbrally “color” each element in the mix. Each stream is also spatially positioned within the stereo field so that each pollutant sonically occupies a fixed position in space, making its reproduction achievable by a wide array of commercially available (non-specialized) speaker arrangements and listening conditions. In total, each of the four pollutant chemicals possesses four different dimensional attributes. Two of them, spatial position and fundamental frequency, facilitate fast and easy identification of the pollutant. The other two, loudness and number of sidebands, afford the perception of change as semantically/representationally related to the subject matter of air pollution. What is unique and promising in this design is that, here, a redundant one-to-many parameter mapping actually becomes a perceptual strength instead of weakness, owing to FM’s persistent maintenance of harmonic identity, resulting in coherence of the overall listening experience.
Designing for public presentation also means designing with a variety of audio display systems in mind. In many cases, good quality speakers are not readily available, and it might not be possible to place them at a wide enough distance to encompass the entire audience within an immersive “sweet spot.” Under these constraints, sonifications that rely solely on spatial mappings struggle to produce meaning for the audience, provided that they are not already encoded redundantly to other parameters. That is to say, redundant mappings are important not only because their integration with other auditory dimensions emphasize perceptibility, but also because in contexts where one mapping fails, the data remain comprehensible.
