“It’s probably just me”: The Literacies of Pervasive Sound Narratives


Elizabeth Evans

The development of digital and portable media devices has led to the emergence of new and experimental forms of storytelling. The Memory Dealer (TMD) sits squarely within these developments, epitomising a form of “pervasive” drama that relies on digital technologies to layer a fictional diegesis over real world spaces. Pervasive storytelling is perhaps most popularly seen within the realm of gaming, where puzzles and tasks are hidden within real-world locations or authentic-looking websites (see McGonigal 2003, 2008, 2011; Evans, Martindale and Flintham 2014). Markus Montola (Montola 2005) describes how this involves a process of social, spatial and temporal expansion in which “the game interface is completely ambiguous: Any action could be a game action, and any sensory observation by any participant could be seen as part of the game” (Montola 2005: 1). Pervasive drama is therefore defined through a blurring of the fictional world’s boundaries and a blending of narrative forms and technologies. In the case of TMD, this involves the fictional world of Eve and the XM existing on top of and within real world, non-narrative spaces. Central to TMD’s nature as a pervasive drama is its emphasis on sound as the central storytelling mode. The use of sound, rather than video or text that require a participant’s visual attention, creates the layered form of storytelling at the heart of its “pervasiveness”. This layering, in turn, raises a particular relationship between public space, technology and narrative that participants needed to navigate in order to successfully engage with TMD’s core diegesis. This unconventional, for many participants unique, form of storytelling positions TMD as a fruitful case study for interrogating concepts of media literacy and how participants learn to make sense of new narrative experiences.

 

Recent scholarship on literacy has recognised the increasing importance of non-literary modes of communication. Colin Lankshear and Michele Knobel (Lankshear and Knobel 2011) present a model of literacy that moves beyond written text to encompass any form that represents language, including moving images and sounds:


Literacies can involve any kind of codification system that “captures” language in the sense we have described. Literacy includes “letteracy” (i.e., within the English language, recognition and manipulation of alphabetic symbols), but in our view goes far beyond this, which puts us at odds with scholars who tie literacy to reading and writing alone. In our view, someone who “freezes” language as a digitally encoded passage of speech and uploads it to the internet as a podcast is engaging in literacy. So, equally, is someone who photoshops an image – whether or not it includes a written text component. (Lankshear and Knobel 2011: 40)


Such an approach emerges from the work of Gunther Kress and Theo Van Leeuwen (Kress and Van Leeuwen 2001) who argue the case for understanding “multimodality”, a situation, they argue, “in which common semiotic principles operate in and across different modes, and in which it is therefore quite possible for music to encode action, or images to encode emotion” (Kress and Van Leeuwen 2001: 2). Libo Guo, Siti Azlinda Amasha and Lynde Tan (Guo, Amasha and Tan 2011) echo this approach, arguing for a more complex understanding of where and how meaning is conveyed and understood. They argue that “visual images, gesture, sound, and action have evolved into articulated or partially articulated meaning making systems in the same way that language has” (Guo, Amasha and Tan 2011: 72). In addition to the ability to understand, interpret and engage with media forms, pervasive transmedia dramas such as TMD demonstrate how literacy may also involve the ability to move through different modes of communication simultaneously or in close succession. In pervasive drama the participants shift between being a listener, a reader, a performer, a viewer and a player. Each position requires different skills of understanding and meaning making as well as different technical or logistical competencies. In such a case it is as much about how an individual gains or develops a particular literacy as it is about how they switch between modes of literacy or combine individual modes into a combined, amalgam form of understanding.


Perhaps unsurprisingly, given its alignment with education, much work on literacy focuses on children in order to explore the ways that formative processes of understanding emerge in relation to media texts (see Jewitt 2006). Work that does consider adult media literacy tends to focus around the use of digital technologies or understanding of the economic and political structures behind them (OFCOM 2011), with an emphasis on non-fiction, information-based genres (Livingstone, Couvering and Thumim 2005). The development of new forms of storytelling brought about by mobile and digital technologies offers the chance to interrogate how adults, those who are already highly literate in a range of modes and narrative forms, continue to develop and refine their skills of understanding when faced with different forms of engagement. Whilst media literate adults may instinctively know how to engage with, and interpret, a film, television programme or even videogame, TMD confronted them with a more experimental combination of forms and technologies, a combination that they were unfamiliar with. This unfamiliarity required a certain amount of learning and negotiation on their part in order to ensure successful engagement with the drama. This article will explore how they discussed moments where their ability or inability to navigate TMD’s diegesis became more apparent in order to interrogate how forms of literacy are reworked in new narrative environments and how players articulated, dealt with, and overcame, such moments of illiteracy.


TMD, with its combination of sonic, visual, performative and literary storytelling offers the chance to examine how otherwise “literate” audience members learn new storytelling techniques. During the piece, participants had to not only discover the narrative of Eve, the XM and memory dealing, but also learn how to discover that narrative. Whilst certain forms within TMD, such as video, were familiar to them, others, such as the aural-only opening act or the hidden audio in the hotel room, were less so. Even if participants had experienced pervasive drama or gaming before, the nature of multi-media and multi-modal drama means that each experience can make use of a wide range of narrative modes and technologies. Consequently, the specific nature of TMD, its unique combination of different ways of storytelling, needed to be learnt regardless of any experience with the form more broadly. By considering the observational data and focus group discussions more deeply, it becomes possible to interrogate the different literacies that audience members were expected to develop and draw upon in order to make sense of TMD. In particular, the fact that TMD relies so heavily on aural information, delivered via the app, voiceover and cast members, offers a chance to interrogate how audiences make sense of “sound narratives”, especially when experiencing one for the first time.

“Learning what the rules are”: The Literacies of Pervasive Drama



Colin Lankshear and Michele Knobel (Lankshear and Knobel 2011) define literacies as “socially recognized ways in which people generate, communicate, and negotiate meanings, as members of Discourses, through the medium of encoded texts” (Lankshear and Knobel 2011: 33). In terms of TMD, and what it indicates about the literacies of pervasive dramas, these forms of meaning-making can be divided into two broad categories: narrative and logistical. The former encompasses meanings in relation to the diegesis of TMD, the player’s relationship and interpretation of Eve and the construction of the drama’s political context. The latter encompasses more practical elements of TMD as an experience and can then be divided into several related, but distinct, sub-categories, including technological, geographical and performative. By looking at each, it becomes possible to interrogate how experimental forms such as TMD build on but also challenge existing forms of meaning making and illuminate the ways that media literate adults continue to renegotiate their relationship with narrative. As one participant commented, being a player in TMD involved not only experiencing its narrative form but also “learning what the rules are in a way” (FRA3). The processes by which players learnt these rules points to the multiple, varied forms of literacy that transmedia pervasive drama requires.



Narrative


The first form of literacy discussed by focus group participants related to their ability to understand the narrative of TMD, to follow the key events in the story and to understand who each of the key agents were, including themselves, other performers and the media-only characters of Eve, DC Douglas and DI Stone. Unlike linear narrative forms such as film and television, where audiences are guided through a narrative in a certain order, TMD’s narrative was scattered in pieces throughout different audio segments, physical spaces and moments of interaction with the case. Players therefore had to develop new ways of identifying what counted as “narrative” and how to both comprehend and interpret it. This form of literacy is perhaps the most like traditional understandings of “media literacy”, which relate to the ability to decode messages in multiple forms. Twinned with the ability to merely understand these messages, is the understanding of them as constructed. Colin Lanskhear and Michele Knobel describe media literacy as the


need to learn to “read media or information sources in specialized ways in order to ‘get what is really there’ and/or to avoid being “taken in. This is the idea that there are ways of deciphering media and information more or less wittingly or critically as an “insider or, at least, as an effective receiver or producer within the media spaces in question. To some extent, this implies the ability to identify strategies and techniques being used to produce particular kinds of effects on what we think, believe, or desire. (Lankshear and Knobel 2011: 22)


It would be possible to see TMD as primarily being concerned with this form of literacy. It is, after all, a constructed piece of fiction in which the creator communicates with the viewer via “mediated” means (spoken and written language, images, even the actors), and the participant must decipher the story that is hidden within these encoded messages.


However, the focus groups revealed that narrative literacy related to some elements of TMD as an experience, but not others. Issues of interpretation were not important. The way in which narrative literacy appears within pervasive drama, and in particular how it is distinct from the other forms of literacy also examined here, was articulated by one focus group participant as the difference between understanding and being narratively “lost”: “I think asking if we’re lost is more [appropriate] than understand, because I might have constructed an understanding that’s 6 million miles away from what it was meant to be, but I’m still deluding myself I understand it” (SUB5). This participant not only sets narrative literacy apart from other forms of illiteracy or confusion (“being lost”), which will be discussed below, but also raises the complex relationship between narrative literacy and the notions of encoding and decoding a narrative’s message. In the other forms of literacy discussed here, it became apparent when the participant had insufficient understanding. They became aware if they were logistically or geographically illiterate because they lost contact with production personnel or other players. With narrative illiteracy, the participant implies here that they would be perfectly capable of completing, and enjoying, the drama without knowing if they had “misinterpreted” it or not. As decades of work on television audiences has attested, texts contain multiple potential interpretations. Claims for “misunderstanding” are problematic, with numerous interpretations being valid, even if they did not match the creator’s “intent” (see, for example, Hall 1989; Lewis 1991; Morley 1992; Moores 1993; Ang 1996). What becomes important throughout TMD’s focus groups, then, is the perception of illiteracy, the participants’ articulations of what they perceive as a misalignment between their experiences and a “correct” or “perfect” experience.


This misalignment most readily emerged when participants felt they had missed important narrative information. One participant commented, when discussing the final debate in which groups must decide whether to support Eve or not, that “coming to the end kind of decision, I am not sure I had all the information to contribute something like that, but maybe that was just me” (SAE2). Other participants similarly had difficulty when faced with needing to make a decision on their feelings towards Eve:


I didn’t understand the references to her father. So I sat there trying to work it out and what the significance of that bit was and what I was supposed to take from it … I had to then, oh my, I’m going to have to have a debate about this, and I don’t know what I’m saying … But I felt slightly, like the whole way through I didn’t quite, I was catching up. (FRE2)


These participants’ perceived lack of literacy throughout the drama led them to feel excluded from TMD’s ending, particularly because the ending required them to contribute to a discussion. This exclusion was not necessarily because they could not interpret the situation or Eve’s character. It was more that they felt ill-prepared to take part in the final debate because of a sense they had not been able to “correctly” collect all the narrative clues.


In some discussions this missing of key narrative elements related to the unusual ways in which the narrative was presented. In addition to the soundtrack through their headphones and interactions with actors, TMD’s narrative was “hidden” within objects such newspaper articles or leaflets or, even more obscurely, in the form of audio or audio-visual snippets that were connected to unexpected physical objects, such as the hotel room’s wardrobe and pillows. One participant described how, whilst she liked having to find a narrative that was spread across multiple different spaces, she simultaneously found it challenging:


I liked the idea of piecing the story together using snippets of narrative that were delivered through these installations, but I think perhaps as a goal that wasn’t clear for the audience? … I mean, you were told to go here, like go to the hotel, go to my room in the hotel or go to the surveillance. I don’t know, there was something … it was only after the first two installations that I realised that I needed to collect these bits of story. (FRA1)


The unusual location of narrative information created a barrier to this participant’s ability to collect all the necessary pieces of the narrative. The additional positioning of TMD within a conspiracy-theme genre emphasised the sense that the narrative was a puzzle and that players needed to collect “clues” in order to “solve” it. As a result, moments where they felt, or realised through talking to other players, that they had missed specific sources of narrative information became, for them, indicative of a sense of illiteracy.

Logistical

 

The second group of literacies that emerged in TMD’s focus groups related less to the ability to understand the narrative and more with navigating the spaces, technologies and modes of engagement within the drama. These literacies can therefore be labelled as “logistical”, concerned as they are with the player’s actions and movements throughout TMD’s duration rather than issues of interpretation or understanding. Even more so than the reworking of narrative literacy discussed above, logistical literacies emerge specifically out of the nature of TMD as transmedia and pervasive. Watching television, the cinema or traditional theatre does not require movement through a public space, or shifts between technological platforms. Whilst listening to the radio or an audiodrama on a personal device may involve movement through different, potentially public, spaces (see Bull 2004), it remains a singular form of engagement, and the listener’s attention remains relatively stable. To borrow from sociologist Erving Goffman (Goffman 1974), each of these forms has a single “frame” that organises them and combines technological, spatial and cultural conventions. As a pervasive transmedia drama, TMD builds on more experimental forms, where the players are not only confronted with different modes and spaces of engagement, but also with different roles within that engagement (see Boal 1979; Benford and Giannachi 2011; Evans 2014). The technological, spatial and cultural framing of TMD changes throughout as the participants shift in their role from listener to viewer to reader to performer. Both of these factors contributed to the other three forms of logistical literacy that emerged through TMD: technological (“what do I press?”), geographical (“where do I go?”) and performative (“what do I do?”).


Technological


The emphasis on the smartphone as the key technology for experiencing TMD led to some minor moments of illiteracy and confusion. Colin Lankshear and Michele Knobel identify how “literacy” has come to encompass more technical abilities with the development of digital technologies:


Sometimes this involves “literacy becoming a metaphor for competence, “proficiency or “being functional. Concepts like being computer literate or being technologically literate are sometimes used simply to mean that someone is more or less proficient with a computer or some other device like a video recorder: they can make sense of and use computers, or can program their video player or mobile phone. (Lankshear and Knobel 2011: 21)


The reliance on smartphones throughout TMD for key narrative information meant that players’ level of comfort with using such technology became paramount. This was exacerbated by the way TMD transforms each smartphone. Users with certain types of phones were given iPhones on loan, a technology they may not have been familiar with at all. Even those able to use their own phone had to operate it through a specific app, which replaced the home screen and required unfamiliar interactions. At certain points, for instance, a button labelled “Answer” appeared, cueing the player to press it and hear a recorded message that gave them further instructions.

TMD app showing appearance of “Answer” button

Section of Act One featuring interruption of voiceover with phone call

However, the unfamiliarity of the phone or interface caused some problems:


I have to say that there’s a problem on this handset which we borrowed where I didn’t realise at the time. I thought there was long pauses, but they weren’t pauses. It was the fact that this wasn’t connected properly, and you have to wiggle it to get the sound to come out, so I missed big chunks of things (SAF1)


When we’d gone to the hotel [the app] said “answer it”, but, we’re being shown to rooms so I didn’t answer it … When I answered it at the end, all I heard was a hello, and that was it. There was nothing else. I wasn’t sure if I’d done the wrong thing (FRA4)


The reshaping of otherwise familiar technology within TMD highlighted the problems with disrupting the established literacies that individuals build up with digital technologies. Just as unusual forms of storytelling require participants to rethink their existing narrative literacies, reframing the uses and interfaces of technology requires them to reconsider their assumptions about how to successfully operate digital technologies.

Geographical


Perhaps even more than technological issues, issues around geographical literacy epitomise the consequences of TMD’s reliance on mobile technologies and sound to construct much of the fictional narrative, whilst allowing the natural setting of Bristol to constitute its visuals. In this respect TMD echoes Adriana De Souza e Silva and Larissa Hjorth’s model (De Souza e Silva and Hjorth 2009) of the “phoneur”, which they describe as “Unlike the flâneur that was ordered by the visual, the phoneur is structured by the information city’s ambience, whereby modes such as haptic and aural override the dominance of visual” (De Souza e Silva and Hjorth 2009: 607). This, they go on to argue, “‘forces’ participants to look at the familiar urban spaces with a different set of eyes” (De Souza e Silva and Hjorth 2009: 609). This concept is central to TMD’s form of engagement as, unlike traditional narrative forms, the boundaries between real and fictional space become increasingly blurred. The following section from the opening act exemplifies this blurring, as participants are asked to layer their own memories over the fictional history being created through the narrator:

Section from Act One asking players to reflect on previous visits to Bristol’s Harbourside area

As one focus group participant commented, this layering allowed players to reshape their movement through, and relationship to, the city: “I like just being in a, you know, familiar city, but experience something very different” (FRA1). This restructuring of space by layering a fictional world onto it was pleasurable for players who could successfully navigate their way through it.


Other participants, however, found difficulty in managing the directional instructions within the narrative, causing disruption to their immersion and often additional confusion. One exclaimed how they “went to the wrong hotel!” (FRC5) despite the voiceover directing them to a named hotel and the app providing a map indicating key landmarks and the hotel’s address:

TMD app providing map to hotel

Aural directions to hotel

The unfamiliar structure of key logistical information meant that otherwise clear instructions were missed by some participants. Others similarly discussed moments when they struggled to navigate themselves between TMD’s locations. The car proved to be particularly difficult to find. One participant commented that “I couldn’t find the car to start with. I went too far and came back and then didn’t know how to go back and ask them again, so I was wandering instead” (FRE2). Another had similar problems and described how “We lost the car [laughing]. We came out of the hotel and then met the girl on the bike, and she told us some directions. We went, and we couldn’t find where we were going” (SAI2). It is perhaps unsurprising that the car was a source of confusion for some players as it was the least identifiable space that players were sent to. It could not be named in the way that the hotel or café was, and although it was parked on a relatively quiet road and in-character crew members were nearby to help players, it was easier for it blend into the background. In this instance, the pervasiveness of the drama comes to the fore, raising issues around geographical literacy that are inherent to this form of drama.


These moments of geographical illiteracy were risky for player immersion as logistical issues became more important than engaging with the narrative. One participant commented on how “I got lost coming into the dentist, so I felt very much like myself, rather than Eve” (SAH6). Her geographical illiteracy, the fact that she had to pause to think about where she was and where she was going, made the boundaries of the narrative clearer, forcing her to prioritise thinking about where she needed to be over immersion in the storyworld. For others, however, it was a stronger sense of geographical literacy that became problematic, as that overpowered the construction of TMD’s fictional-physical diegesis:


Because I’m so familiar with the location, when I came up here I was on automatic pilot, and so I was just starting to go on past you, to go home … because that’s my route home, and I had to come back down in order to come into the door. (FRG5)


For each of these players, the pervasive nature of TMD formed a key factor in their ability to navigate, successfully or otherwise, through its core narrative.

Performative


If geographical literacy speaks directly to the pervasive nature of TMD and its relationship to its real world location, performative literacy related more to the position that the player is put in within the narrative. Unlike traditional forms of drama, in which the audience are told the narrative via exposition, pervasive dramas position their audiences or players as agents in constructing their narratives. They become experiential forms of drama (see Evans 2014). The consequence of this is that what the player does becomes central to not only their immersion in the drama, but also their successful progress through it. They are required to “perform” in a number of ways. From the observation data, some players responded to this shift easily, performing assertively and confidently when required to speak out or interact with cast members. For others, though, this performative quality was a source of difficulty:


I’m always very social inhibited, so I found it difficult to interact convincingly when people came up and started talking to me, I just stood there like an idiot for quite a lot of it … I found it hard to play along. (FRE2)


This player felt awkward with taking on a persona and struggled with interacting in a way that she felt was a “convincing” performance. “Performative literacy”, however, does not only equate to performance in the sense that one could consider how actors construct characters. In this context, performance equally refers to a more practical situation in which the player must do or say something in order for the narrative to continue.


In the focus groups, the bridge section (where they met the character of Imogen) and the dentist’s office were identified as sequences that a number of players found difficult in terms of performative literacy. One participant, when discussing needing to find Imogen, said that “just initially, well trying to find someone, some of the people, I found it quite difficult. I wasn’t sure who I could afford to be asking and staying in character” (FRD1). This participant does recall elements of traditional performance, the need to be “in character” and so maintain the integrity of the diegesis. However this moment when he first needed to act out and interact with another person was hard for him, leading to anxiety and uncertainty. Such uncertainty also emerged in conversations about the dentist’s office, as this example demonstrates: “you felt a bit like anxiety in [the dentist’s] wondering about how you were supposed to respond in that situation” (FRC6). This moment was particularly difficult because the narrative called for the player to stand up abruptly and demand to see the dentist. The second person nature of the narrative, however, meant that this instruction was layered within Eve’s memory of her own trip to the dentist:

Aural directions telling participants to stand up and speak in the Dentist’s Office

The exact, performative nature of this moment was partially obscured. One participant recalled that


when I did that, I just sat there and listened to it and didn’t stand up, and I thought, “am I supposed to stand up or not?” I wasn’t sure, if you were supposed - if they were telling you to act it up or if you were supposed to be reliving it and not acting it out, so I hesitated on that bit. (FRE2)


The combination of an unusual form of narrative address and the need to suddenly become a performer highlighted the complex literacies needed for players to fully take on all the roles built into this moment.


The struggles players had over logistical literacy indicate a number of issues inherent in sound-based pervasive dramas that relate to their transmedia nature. Steve Benford and Gabriella Giannachi (Benford and Giannachi 2011) discuss the importance of managing “transitions” in any pervasive drama. They describe transitions as “critical moments in an experience at which users must cross between spaces, rub up against schedules, take on new roles, or engage with interfaces” (Benford and Giannachi 2011: 231). These transitions are moments when the player’s engagement must change either as a result of external factors, such as a shift from public to private space, or as a result of the role they are required to take within the narrative. What is noticeable about the above comments’ relationship to logistical literacy is that each relates to a moment of transition. Those players who got physically lost found difficulty in navigating shifts between spaces whilst simultaneously immersing themselves in the narrative. Those who became unsure of what they should be doing struggled at moments when they were required to shift from being a listener or observer of the drama to being a participant or actor.


Such moments of transition, Benford and Gianacchi argue, threaten the coherence and continuity of the experience (Benford and Giannachi 2011: 231). They also, however, indicate moments where literacy, or illiteracy, becomes more apparent. On one level there are moments where players are required to take on a role, such as performer, that they are perhaps unfamiliar with and which requires a certain level of comfort and competency. On an additional level, however, the shifts within transitional moments require a certain skill in and of themselves. The difficulty that otherwise media literate players had with such moments demonstrates that this skill is different from those required when watching film, television or traditional theatre. These transition moments emerge precisely because mobile technologies allow for players to move between and across spaces and narrative forms. They epitomise the new literacies that such forms require in the way they supplement narrative literacy with forms that are based more around movement and interaction. At the same time, the focus group discussions around these literacies highlight the ways in which sound narratives such as TMD reveal the imbalance within audio-visual literacies and point to the ways that both audiences and creators can work to address them.

Aurality, Self-Blame and Sign Posts: Facilitating Literacy in Pervasive Sound Narratives



The moments of illiteracy discussed above are not automatically negative occurrences. For some participants, moments where they experienced logistical illiteracy actually enhanced their experience of the drama. For some this was by experiencing satisfaction once they had correctly navigated difficult moments. For others, any struggles they had contributed to the “experiential” quality of TMD and facilitated emotional engagement. One participant, for instance, commented on how “that feeling of insecurity, I don’t know what I’m supposed to do next … have I missed something? I think [that] actually helped me get into it in a way” (FRF2). Logistical difficulties helped with narrative immersion, contributing to the drama’s overall tone of paranoia or uncertainty and suggesting the potential to use such moments as a storytelling tool. At the same time, however, a greater understanding of how to manage such moments points to a deeper interrogation of how audiences make sense of stories and continue developing their media literacy.


At a number of points throughout the focus group discussions it became clear that the dominance of sound as a narrative source and the visual bias of media literacy caused particular problems. Greg Goodale (Goodale 2011) has argued that there has been “an ancient history of emphasizing vision rather than hearing in the Western tradition” (Goodale 2011: 3) and that “we have been trained to read visual objects, even technologies, and not sound” (Goodale 2011: 4). Just such a bias often emerged as players realised that they had missed key points in the narrative or had little understanding of Eve as a character:


SAA1: it says early on in the voice in your ear that you have let her down, and that is a very powerful thing to be able to work on us later on, saying you have got to go and get her now …

SAA3: I missed that bit entirely.


Although the first player found this moment particularly effective, the fact that the second missed it highlights Goodale’s argument about differences between the status of sound or image based media. Other examples of mixed responses to TMD’s audio-bias emerged elsewhere:


INT: Did you feel like you knew her through the narrative?

FRJ1: Knew bits about her.

FRJ5: Yes, I mean sort of images would have been nice as well as all the other, you know, means of information with words and things, but more images would have made it more real, I think.

FRJ4: I’ve kind of created it in my own head.


In both of these quotes, it is the audio-only parts of the narrative that are problematic. The moment in the opening act, which outlined the player’s background with Eve, was missed by some focus group participants, despite its length and centrality to this section:

Section of Act One detailing participant’s history with Eve

This Act, which also works to establish Eve’s character traits and relationships to both the player and Doug, forms a key narrative moment. Although the information contained in this moment is supplemented and reinforced elsewhere in the drama via visual means, through the objects in the hotel room or short video clips, such reinforcements rely on the player’s ability to notice and retain the information provided aurally in the opening twenty minutes. The fact that these twenty minutes only consist of sound, required an additional form of aural literacy that some players had not yet developed.


Aurally dominant narratives are certainly nothing new. For decades, television scholars have debated the comparative importance of image and sound to television narrative (see Ellis 1982; Caldwell 1995). Although in many ways reductive of the television viewing experience, which can be highly immersive, John Ellis’ glance theory reflects one mode of viewing in which narrative can be understood in a highly distracted way (Ellis 1982: 128). Christopher Balme similarly argues that theatre has long encouraged audiences to be “willing to adapt to quite heterogeneous ways of seeing and hearing” (Balme 2008: 90). In this respect, then, the multimodal literacies demanded by TMD are not that revolutionary. However, the fact that some participants struggled suggests that there is more going on here and that there are some fundamental differences in the nature of TMD and what it demands of those engaging with it compared to traditional forms of narrative, indicating how media literate adults may not automatically be literate in all forms of mediated narrative.


For some players, the struggle they had with developing a literacy based on listening was compounded by the pervasive nature of the drama. The fact that not only did they have to pay particular attention to sound, something they were not used to, but simultaneously ignore other sounds in the real world around them, led to moments of confusion. The following player, for instance, explicitly identified the experimental and unfamiliar nature of TMD as making it more difficult for him to know what was going on:


It is interesting how little goes in of what you’re hearing. Like, you can be really listening, and actually just because of being in a new situation and kind of being aware of the world around you, I think yes there is a lot of detail that passed me by. (SAE5)


A second player made a similar comment, identifying the public nature of the event as making it particularly difficult for him to follow the narrative via the soundtrack:


The difficulty sometimes was when people were talking to you, some of the actors were talking, and you had the music going on. It wasn’t clear whether there was gonna be some even on the soundtrack triggered, so I had to keep listening to this and listening to them. (FRE1)


These comments echo the concept of the bi-psyche, which has been used to articulate engagement with mobile devices. Originally the result of research for mobile phone company Motorola, the bi-psyche epitomises a sense of dual attention. As Sadie Plant (Plant 2001) argues, “the mobile exposes its users to information which may not be so easily absorbed in a busy, unfamiliar or public context, requiring them to ingest what may be incongruous news.” This, she goes on to argue, means that “the mobile requires its users to manage the intersection of the real present and the conversational present in a manner that is mindful of both and to improvise responses on the move” (Plant 2001: 50). Mobile, pervasive sound narratives such as TMD explicitly play on such dual attention, echoing Jussi Parikka and Jaakko Suominen’s (Parikka and Suominen 2006) application of bi-psyche to mobile gaming: “the user concentrating on Playman Summer Games (2004) while surrounded by urban noise or playing Sumea’s Extreme Air Snowboarding (2003) while navigating through busy urban streets is an exemplary attentive subject, a true bi-psyche” (Parikka and Suominen 2006). Inherent within Parikka and Suominen’s model is a certain kind of skilled attention, the ability to marshal information from multiple competing sources and via multiple senses. This skill, as the TMD participants demonstrate, is not automatic, but must be learnt and developed.


The difficulties of managing a multimodal literacy within a real world space were discussed by a number of focus group participants. Whilst this was compounded by the fact that TMD took place within a busy area of Bristol where the general public and traffic provided multiple distractions, at its core was the aural nature of the narrative. It is only by being aural that such a pervasive narrative is possible. A fully audio-visual experience cannot, for obvious health and safety reasons, take place within public space and real dangers such as traffic, other people or nearby bodies of water. By focusing on sound, it becomes possible to allow players to simultaneously focus visually on the world around them. However, balancing this focus between eyes and ears was not automatic or easy. As one participant commented, the need to balance a narrative that is presented in an unfamiliar way made it harder for her to complete the kind of narrative interpretation that she might otherwise be perfectly capable of: “Which means that as you’re listening, watching those installations, the other part of your brain isn’t necessarily connecting the dots, the storyline” (FRA1). The unusual nature of TMD, for some participants, made it difficult for them to engage in the level of narrative interpretation that they were familiar with from older narrative forms.


There was evidence in the focus groups that this ability to take in information from both visual and aural cues was a literacy that focus group participants felt they should have. A particularly noticeable characteristic of these discussions was that the majority of focus group participants who discussed struggling with TMD’s narrative also felt they were to blame for such moments of illiteracy. Phrases such as “but that’s just me” or “I didn’t do it properly” appeared in many discussions as numerous players sought to justify why they felt they’d failed in their experience of the narrative. Some participants were almost apologetic for not paying “proper” attention, as the following participants exemplifies: “And then I missed the crucial … I was sorting my, I don’t know, I wasn’t listening” (SUF1). Other participants linked this self-blame to more general “failings” in themselves that they had identified previously. One commented on the overall style of TMD’s narrative and how he was “not very good with sparse narratives where you have to fill in a lot anyway” (FRA4). Other participants spoke specifically about the aural nature of TMD but placed the blame on their own sense of hearing or language abilities. When discussing the final installation in the tent, one participants said, for instance, “My hearing’s not good anyway, so I didn’t know if that was just me” (FRF2). Elsewhere, a non-native English speaker positioned her own abilities with spoken English as a source of anxiety for her: “Well for me it is different, because I am not English, so I would worry all the time about if I was going to make a mistake or not” (SUI2). For each of these participants the perception was that they should have been able to easily follow TMD. They felt that their general media literacy should have been translatable even though the experimental form and unusual narrative presentation actually calls upon different skills.


Whilst the majority of participants who struggled with TMD positioned any limitations to their engagement as a result of their own illiteracy, the focus groups also revealed ways in which specific elements of the production aided their development of new literacy skills. Some desired quite straightforward methods in terms of clearer instructions or briefings beforehand. One commented on how “I would have liked a bit more briefing on whether I was supposed to act in this or just kind of follow along” (SUC6). Another promoted the idea of different versions that accounted for different levels of literacy:


I think there ought to be, like, different levels. If you’ve never done it before, you want like a beginners level, like minimum sort of work and … mainly guidance. But that would probably bore people who like to do this sort of thing more often. They might want to get involved a bit more, so like an advanced level where you know there would be more interaction, more involvement and more guidance. (SUH4)


This raises the complex production challenge of managing a range of literacy levels in order to allow players with little experience to learn how to fully engage with the narrative, without alienating more literate players.


Focus groups, however, revealed the potential for using certain production components in a pedagogic way, to guide players more subtly through the narrative. The presence of actors, and the quality of their performances, facilitated a form of guidance that gave unsure players the confidence that they were engaging correctly, while remaining consistent with the fictional diegesis. This was a frequent comment within focus groups, and is epitomised by the following quote:


I thought the characters we interacted with were very good, though, because even if we didn’t really know what we were meant to be saying or doing, it didn’t really matter. It didn’t feel like it mattered that much how we responded. (SAB2)


Another participant made a similar comment: “I think most of it was well-facilitated by actors. They were watching for our appearance and picked us up when we could’ve lost it” (FRE1). Such literacy aids are contained within TMD’s storyworld. The advantage of having live, responsive actors offers the chance to build in a more adaptive level of guidance and so facilitate differing levels of literacy amongst those taking part.


The formal components of TMD’s diegesis emerged as tools for facilitating and managing literacy. Components such as set design, props and costume contributed to players’ sense of belonging and that they were in a logical, coherent and understandable world. This is demonstrated in the following conversation, which discussed both the hotel room and the dentists’ office:


SAJ5: it made it real, because it was so real. There was a room and you were in there and, you know …

SAJ4: I think there’s a real authenticity about it though. So I was sitting in [the] reception, and the receptionist was, she was on the side, and then I looked at her, and she gave me a nod on cue.


The construction of the physical diegetic world helped the players believe, and therefore fully exist, in the narrative of the TMD. This in turn helped them navigate their way through that narrative, without losing focus and immersion. Interestingly, given the problems with the aural focus for core diegetic information discussed above, a number of participants identified the music as a formal component that aided their ability to navigate TMD. One participant commented that the music “reassures you that it’s working, doesn’t it” (FRG3). Another commented that “It mostly gave the whole thing structure. It let you know of- it strung all the bits together, gave it a consistency, like I said” (FRA4). The presence of a soundtrack, with repeated motifs, reassured participants that they were acting “correctly”, that nothing had broken and that they were following the correct, coherent path through the experience.


How these formal components are used in this way is not straightforward. Moments of perceived incongruity negatively affected players’ experience and provided a greater challenge to their successful completion of the narrative. One player, for instance, commented on how “it made no sense that the car would be open and we get into it and look at things” (FRA4). Making access to the car relatively smooth, rather than making them search for a way to get in, could aid some players in ensuring they easily find it but could also be interpreted as problematic. Another participant in the same focus group felt similarly about the memory dealer: “I didn’t believe, like when I went in and I saw the hood, I just thought, ‘I don’t believe this’, like the dealer wearing a hood and looking so obviously like, kind of stereotyped” (FRA1). These players highlight the challenges with using diegetic elements as signposts. The same signpost could be vital to ensuring some players’ successful navigation but simultaneously disruptive to others’. However, the fact that focus group participants called on formal components of TMD – such as the actors, sets, costumes and music – indicate that the reshaping of literacy is not reliant on explicit extra-diegetic material. Instead, the creation of a narrative world that looks and feels authentic, populated by actors who not only perform but are also trained in facilitating the progress of less literate, or confident, players, can contribute greatly to managing any limitations in participants’ literacy.

Conclusion



The experimental nature of TMD brought together multiple narrative forms and correspondingly multiple forms of literacy. On the one hand, issues of understanding and following the core story emerged as spaces where participants were confronted with the limitations of their literacy. The combination of different narrative forms required them to apply the literacies they had already developed, while combining and reshaping them in new ways. On the other hand, new logistical-based literacies emerged, ones that are less associated with traditional media or narrative forms. Each related to the specific nature of TMD as a pervasive transmedia drama. The layering of narrative onto public space via mobile technologies led to a reshaping of the mechanics of locating, understanding and compiling narrative information. Players were required to navigate a fictional world simultaneously with the real world and to take on roles, such as performer or detective, that they had little experience with.


The ways in which players, who are otherwise considered part of the media literate audience, struggled when otherwise familiar components were placed in unfamiliar contexts demonstrates the need to consider media literacy as an on-going process. The increased experimentation with narrative that digital technologies facilitate only exacerbates this. The potential for combining video, audio, text and images in increasingly varied ways and positioning the story outside of the conventional narrative frames of the television set, radio, cinema screen or theatre reveals how audiences are increasingly having to re-learn what narrative is and how it can be experienced. TMD players’ insistence that moments of failure were related to their own shortcomings only further demonstrates their assumptions that by the time they reach adulthood they should be able to understand a story, no matter what form it appears in. However, the experience of TMD players reveals that media literacy is an on-going process, one that individuals must adapt and reshape as the forms of narrative are similarly reshaped. The increased experimentation with mobile media technologies, and the forms of audio, visual and audio-visual narratives that can be presented via them, demonstrates the need to further interrogate the literacy of fictional engagement. In particular, the way in which literate audiences are confronted by new forms of engagement and the need to develop new literacies is rife for further exploration.

References



Ang, Ien (1996). Living Room Wars: Rethinking Media Audiences for a Postmodern World. London: Routledge.


Balme, Christopher (2008). “Surrogate Stages: Theatre, Performance and the Challenge of New Media.” Performance Research 13/2: 80–91.


Benford, Steve and Gabriella Giannachi (2011). Performing Mixed Reality. Cambridge, MA: MIT Press.


Boal, Augustus (1979). Theatre of the Oppressed (trans. Charles McBride). London: Pluto Press.


Bull, Michael (2004). “‘To Each Their Own Bubble’: Mobile Spaces of Sound in the City.” In Nick Couldry and Anna McCarthy (eds.), Mediaspace: Place, Scale and Culture in a Media Age (pp. 275–293). London: Routledge.


Caldwell, John T. (1995). Televisuality: Style, Crisis and Authority in American Television. New Brunswick, NJ: Rutgers University Press.


De Souza e Silva, Adriana and Larissa Hjorth (2009). “Playful Urban Spaces: A Historical Approach to Mobile Games.” Simulation & Games 40/5: 602–625.


Ellis, John (1982). Visible Fictions: Cinema, Television, Video. London: Routledge.


Evans, Elizabeth (2014). “Mobile Engagement: The Technological and Social Dynamics of Transmedia Pervasive Narratives.” In Xiaoge Xu (ed.), Interdisciplinary Mobile Media and Communications: Social, Political and Economic Implications (pp. 125–138). Hershey, PA: IGI Global.


Evans, Elizabeth, Martin Flintham and Sarah Martindale (2014). “The Malthusian Paradox: Performance in an Alternate Reality Game.” Pervasive Ubiquitous Computing 18/7: 1567–1582.


Goffman, Erving (1974). Frame Analysis: An Essay on the Organization of Experience. Cambridge, MA: Harvard University Press.


Goodale, Greg (2011). Sonic Persuasion: Reading Sound in the Recorded Age. Urbana: University of Illinois Press.


Guo, Libo, Siti Azlinda Amasha and Lynde Tan (2011). “Multimodal Literacy in Extended Learning Activities.” In Caroline Ho, Kate Anderson and Alvin Leong (eds.), Transforming Literacies and Language: Multimodality and Literacy in the New Media Age (pp. 70–84). London: Continuum.


Hall, Stuart (1989). “Encoding/Decoding.” In Stuart Hall, Dorothy Hobson, Andrew Lowe and Paul Willis (eds.), Culture, Media, Language: Working Papers in Cultural Studies, 1972–79 (pp. 128–138). London: Hutchinson.


Jewitt, Carey (2006). Technology, Literacy and Learning: A Multimodal Approach. London: Routledge.


Kress, Gunther and Theo van Leeuwen (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Bloomsbury.


Lankshear, Colin and Michele Knobel (2011). New Literacies: Everyday Practices and Social Learning. Third edition. Maidenhead: Open University Press.


Lewis, Justin (1991). The Ideological Octopus: An Exploration of Television and its Audience. London: Routledge.


Livingstone, Sonia, Elizabeth van Couvering and Nancy Thumim (2005). Adult media literacy: a review of the research literature. London: Ofcom.


McGonigal, Jane (2003). “A Real Little Game: The Performance of Belief in Pervasive Play.030303: Collective Play. Research Colloquium, University of California at Berkeley.


McGonigal, Jane (2008). “Why I Love Bees: A Case Study in Collective Intelligence Gaming.” In Katie Salen (ed.), The Ecology of Games: Connecting Youth, Games, and Learning (pp. 199–227). Cambridge, MA: MIT Press.


McGonigal Jane (2011). Reality is Broken. New York: Penguin Press.


Moores, Shaun (1993). Interpreting Audiences: Ethnography of Media Consumption. London: Sage.


Montola, Markus (2005). “Exploring the Edge of the Magic Circle: Defining Pervasive Games.


Morley, David (1992). Television, Audiences and Cultural Studies. London: Routledge.


OFCOM (2001). UK Adults’ Media Literacy.


Parikka, Jussi and Jaakko Suominen (2006). “Victorian Snake? Towards a Cultural history of Mobile Games and the Experience of Movement.Games Studies 6/1.


Plant, Sadie (2001). On the Mobile: The Effects of Mobile Telephones on Social and Individual Life.