Probably the two most well-known techniques described for displaying information through sound are Auditory Icons (Gaver 1986) and Earcons (Blattner, Sumikawa and Greenberg 1989). The technique known as Auditory Icons was first investigated as a means of extending the use of visual interface icons to the auditory dimension (Gaver 1986). Based on “everyday listening” skills, this approach maps information to recognizable sounds from the real world. By using a recognizable sound, the user can intuitively understand the current action or event suggested by the sound. For example, hitting a tin with a stick is an event that generates a sound. The sound itself conveys information about the material and size of the tin and whether it is full or hollow. The sound also conveys information about the materials involved, the frequency of hitting, and the force of the hitting. This is information we naturally learn to interpret from our everyday experiences.
There are many instances of natural-like sounds used to augment computer interfaces. For example, the SonicFinder integrated running and pouring sounds in the Macintosh interface to represent file manipulations on the desktop (Gaver 1989). The SharedARK application, a virtual physics laboratory for distance education, included sounds such as hums that are mapped to the state of a simulated system (Gaver, Smith and O’Shea 1991). The ARKola bottling system mapped sounds to equipment in a soft drink factory and introduced audio cues for monitoring the bottling process (Gaver, Smith and O’Shea 1991).
The second most common method of designing informational sound is through the use of Earcons (Blattner, Sumikawa and Greenberg 1989). Earcons are abstract, synthetic tones that are structured to create auditory messages. This approach relies on “musical listening” skills as it conveys information using the musical properties of sound, such as rhythm, timbre, and pitch. This can be contrasted with Auditory Icons that use everyday listening skills rather than acquired musical expertise.
Studies on the effectiveness of Earcons for conveying information have been conducted since the 1990s. The effectiveness of Earcons was experimentally tested in the role of providing navigational cues within a structured menu hierarchy (Brewster, Wright and Edwards 1993). The study found that 81.5% of participants successfully identified their position in the hierarchy, indicating that Earcons can be a powerful tool for conveying structured information. The use of Earcons to map common operating-system functions in a graphical interface was also evaluated (Polotti and Lemaitre 2013). In this study it was found that subjects benefited from additional sound feedback when performing key tasks such as cutting and pasting (Polotti and Lemaitre 2013).
Compared with Auditory Icons, Earcons have the advantage of being able to convey complex information about events to the user without any natural associations with a sound source. On the downside, Earcons require prior understanding of the mapping between the sound and the event before the information can be recognized. By contrast, Auditory Icons are considered to be more intuitive, as they capitalize on the existing listening skills of users.
Currently, only a limited amount of work relating Auditory Icons and Earcons to computer games has been published. However, it has been noted that both these approaches can play a role in terms of enhancing control functions for the player by extending the player’s range of perception during the game (Jørgensen 2006). There are also some taxonomies of sound usage described in the context of games (Friberg 2004; Grimshaw and Schott 2008; Stockburger 2003). The use of Earcons and Auditory Icons and their relationship to player performance in Defense of the Ancients 2 (Valve Corporation 2013), a popular multiplayer online battle arena game, have also been analyzed (Ng, Nesbitt and Blackmore 2015). However, the informative use of sound has a much longer history of study in domains outside of games (Kramer 1994), with applications being reported in diverse domains, ranging from file management (Gaver 1989) to hospital operating rooms and vehicle safety systems (Graham 1999; Patterson 1982; Stanton and Edworthy 1999). Making optimal use of these informative approaches to using sound would potentially allow more critical information to be integrated into game interfaces. The intent would be to improve traditional usability criteria such as effectiveness, utility and efficiency (Gaver 1986; Gaver 1989; Nesbitt and Hoskens 2008; Ng and Nesbitt 2013; Smith and Walker 2005) without impacting on the immersive experience that games strive for.
Interestingly, the effects of additional sound information, in addition to existing visual information, are not always beneficial. The auditory Stroop effect (Morgan and Brandt 1989) demonstrates how performance can deteriorate when the visual and auditory information are in conflict. Moreover, given the, ultimately, limited capacity of the brain to process information (Kahneman 1973; Townsend and Eidels 2011), additional sources of information, though relevant to the task at hand, may overload the system and impair performance. Thus, the potential benefit of adding auditory information to visual displays is not clear-cut and requires careful empirical scrutiny. This is precisely the aim of the current study: to evaluate user performance in a multimodal decision-making task.