Inside the Narrative
VIRTUAL REALITY. Only mentioning the term can create a trembling expectation. A portal into a new and exciting digital world. However, this is no new technology. The 50s are considered the beginning of the development of VR. It is only in recent years that this technology has become available to the public in various forms. Large companies like Sony, Samsung, Google or Facebook, to mention a few, are investing big money into the development of XR technology (extended reality) that we find in various formats like AR mobile apps, 360 videoes on Youtube and the already iconic VR goggles that is being developed as screen technology for both VR, AR, and Mixed Reality. Although VR technology has undergone rapid development, it has not yet become a mainstream media alternative. The use of VR technology and experiences is to a greater extent found in fields such as architecture, product design, medical innovation, and the gaming industry. Despite all the activity around VR, the technology's relationship with storytelling is in a premature phase. How this "new" medium will affect the way we tell and develop stories is not yet clear. To create narrative experiences in VR, we must learn to tell stories differently than we are used to from film and literature. In books, films, comic books, theater, it is often important to capture viewers or readers in such a way that they are in a way locked into the world of fiction. This immersion into the narrative world is staged in the different media forms through numerous measures adapted to each medium. The ability to immerse audiences into a world of fiction is VR technology's most prominent feature. The immersion we are talking about here is not based on our own imagination but on the technology's ability to stimulate the senses in a direct way that automatically incorporates the audience into a potentially narrative universe. One of the major issues faced in relation to storytelling in VR is the relationship with the audience, as they now find themselves on the inside of the story.
Vicky Dobbs Beck executive in charge at ILMxLAB is at a lecture series at Siggraph 2019 stating their goal, in VR, to go from storytelling to the idea of storyliving. She is underlining the importance in VR, that you as the viewer are actually in the world and that you matter. The audience´s presence and actions can move the story forward. This could conceivably mean that you as an observer has a role to play and possibly not just as a fly on the wall? In the same lecture series at Siggraph 2019, Donald Mustard, director of the game Fortnite, at Epic Games, talks about how they in the ongoing production of the game avoid seizing the player's freedom and agency in the Fortnite world. The freedom of the players and their agency is to the core of their thinking. In terms of storytelling in VR, the idea of the player's or audience's possible roles on the inside of the story can be related to the intentions in the production of the game Fortnite and to Dobbs´s emphasis on the viewer's importance in the story within VR. Mustard also describes another key storytelling element that relates to the real-time dimension that is also part of VR as a digital online medium. This is the ability and opportunity of the game creator to provide relevant or surprising feedback to the players along the way. He explains this by the term "The campfire theory" where the storytelling is formed in the meeting between the audience and the narrator, in the same way as sitting around the campfire at night. In Fortnite, this is used in the development of the course of action as a result of players' actions and reactions. This mindset and thinking may perhaps be translated into VR experiences. The idea of feedback in relation to its audience is also found in the theater's feedback loop, describing the same phenomenon, only related to another form of media. Here it is the actors' meeting with their audience that forms the possibility of the story's transformation. When it comes to the audience's possible participation and role in a VR story, the reference to the interactive theater may be relevant.
The research project, Inside the Narrative, derives its name and focus from the narrative point of view of the VR media. The project has its academic position in visual communication at the Faculty of Arts, Music, and Design at the University of Bergen. The project's connection to professional fields such as illustration, animation and graphic design gives us a unique opportunity to help bring forth new perspectives and voices in VR storytelling that may differ from, for example, the engineering field, the computer games industry, journalism, medicine or the film industry. Knowledge from subject areas related to picture books, animation, editorial design, typography, or interaction design are examples of possible inputs into the exploration of VR technology's potential for these fields.
At an eight-day multidisciplinary VR storytelling workshop at Hafjell near Lillehammer in 2019, called Off-Piste Virtual Reality Storytelling Lab, the storytelling itself was put into focus. No Matter Where is the title of an experimental prototype of a VR documentary where you get to meet with various characters and their personal stories in a VR space. The project explores how to obtain, process and present documentary material in the VR format. With this prototype, we wanted to find out what it would be like to be face-to-face with someone who openly shared their life experiences? We meet three different characters within this VR installation. The thematic focus of what is being told is the various individuals' relationship to places and how these have had an impact on memories and their lives.
Wearing VR goggles with headphones in this prototype immerses the viewer into a misty and snowy landscape. The sound of a gentle breeze enhances the experience of the place's presence. The attention of the audience is directed towards a person sitting in the landscape, by means of an enticing sound. As you focus your attention on the person, the person stands up and begins to tell a story. While the person is talking, the audience can move around the space to look at the interview subject from different angles. You can also choose to look away. When you get close to the figure talking, you see in this prototype that the human figure is just a kind of 3-dimensional hollow shell that lacks the backside. This deficiency is related to the volumetric filming technique used to generate the 3-dimensional figure in the VR space. After about 3 minutes, the person ends the monologue and sits down in a waiting position. With the help of another enticing sound, the audience is led on to a new story told by another character in the same space. This time it's the sound of a radio we hear. When you turn around, another character in the landscape is appearing, which in this case is given a kitchen environment that is located in the middle of the misty landscape. As you approach, this person gets up and starts a monologue in the same manner as the first one. In this prototype, each of the stories and sounds that lead the attention is triggered by keystrokes from a technician in the room who observes the progress and course of action of the VR experience. The use of a technician in this project is a possible way to run an event course in a VR experience. The advantage of this solution is that the experience here is personalized to each viewer based on the viewer's body language and attention. This could also be automated to make production less resource-intensive and more easily divisible.
As an approach towards documentary content in this project, we mainly wanted to relate to the real world and photorealism as a starting point for the visual expression. Environmental descriptive elements were therefore gathered with photographic techniques which were later processed into 3D elements and assembled in the game engine called Unity. The snowy and misty landscape backdrop was created by a 360-degree panorama so that we could fill the whole world around the audience. This could just as well have been a 360 video recording if that had been desirable. To create the ground and the place where you stand and where the interview subjects are located, a technique called photogrammetry was used. This technique creates digital 3-dimensional objects by allowing software to analyze depth ratios in the images based on a plethora of photographs taken systematically of a place or object. This way, one can quickly capture and integrate photorealistic objects into a 3D program without necessarily being a 3D specialist.
To give the audience the opportunity to feel that they were in the same room as the interview subjects, we wanted them to be able to move with some freedom in the VR space. Therefore, this prototype does not use a classic 360 video or similar to create a VR environment. With 360 film you get a photographic visual expression, but without being able to move around in a 3-dimensional environment. One is locked to a point in the universe. Instead, the audience is offered an area of 16 m2 where they can move freely and look around both in front and behind the various objects in the space.
To film the interview subjects that we later encounter in the VR experience, we use a technique called volumetric capture. This is done by using a Kinect camera and software called Holo Cap that combines a video recording with 3D data read by an infrared camera. In this way, we could capture a 3-dimensional figure in the space and not only a regular flat 2D video image. It is only from a partially frontal perspective that it works visually from the point of view of the audience. Still, it is not flat as a video recording, so that the figure's body shape becomes visible in space, making it possible to integrate the figure with other 3-dimensional elements in the space.
The sound recordings used in the VR experience were organized digitally in different virtual places in the VR space in the same way as you would distribute 3D objects. In this way, you can organize sound in the game engine Unity (alternatively the game engine Unreal Engine) so that you experience the sound coming from different directions and at different distances.
Through working with the prototype, we engaged in processes and techniques that are largely about creating virtual places with different opportunities for experiences and actions. The audience's role in this particular VR experience was as a kind of mobile observer without so much opportunity to influence the environment. Still, the audience´s meeting with the interviewed subjects and their personal story inside the VR space seemed to be one of apparent emotional impact. The choice of controlling the experience by the use of a technician tells us that VR not only needs to be a lonely digital experience but can also be conceived as a type of performance where the experience is shaped through a feedback loop like in the theater. After working with this VR-prototype, the overall experience is that focusing on the quality of the story itself still remains important also from the inside of the narrative.
