-----------WORK IN PROGRESS------WORK IN PROGRESS------
-----------WORK IN PROGRESS------WORK IN PROGRESS------
SHADOW OF WAR
VR-Ethics and storytelling in the spectrum
of representation, immersion and agency
ARTISTIC RESERACH PROJECT: INSIDE THE NARRATIVE
Authors: Gustav Kvaal, Torkell Bernsen, Simon Gilbertson
Artistic work/ SHADOW OF WAR/ I SKYGGEN AV KRIG (NO)
Gustav Kvaal – Visual storytelling, Illustration, Animation
Torkell Bernsen – Visual storytelling, Animation, Compositing
Odd Torleiv Furnes – Music and VR-Sound
Rafal Hanzl – Unreal Engine facilitator and Realtime Effects
See and experience SHADOW OF WAR at SIDEQUEST:
https://sidequestvr.com/app/34157/shadow-of-war
SoW Trailer on Youtube:
https://www.youtube.com/watch?v=7wl06l_-lWw
ABSTRACT:
The aim of this artistic research project is to create an illustrated VR documentary experience that narrates the story of a time witness from the second world war in Bodø, Norway. The project explores various storytelling approaches and how these affect questions concerning ethics in the encounter between the VR-audience, interviewed subjects and the audiovisual spatial design. Artistic and qualitative research methods have been employed to explore how different visual modes and contexts alter the experience of the narrator and narrative in a media form characterized by its ability to place the viewer in a state of immersion, intimacy, and a sense of presence. This study is situated in an artistic landscapes of visual arts, illustration, media theory and journalism. Concepts such as documentary ethics, representation, media witness, distant others, with the so-called risk of improper distance are relevant for the reflection associated with the project. Based on a comparative study and artistic research methods the project highlights ways of handeling VR-storytelling ethics. The study also propose a view and understanding of ethics towards VR- storytelling that is depending on each experiences placement in a spectrum of different types of VR-experiences, from the totally immersive and interactive to VR-experiences that is more closely related to traditional cinematic screen based experiences. The Shadow of War VR-experience is an example of the latter. With its familiarity to film and traditional illustrative techniques the study is also serving as an educational example of how to introduce new visual storytellers, moving from traditional media into immersive storytelling.
CONTENTS:
ABSTRACT
INTRODUCTION
- Project position and contribution
- Mindset and objectives,
- Envisioning the future of VR
- VR-Feature experince, hybrid format
- Scope and educational approach
- Research question
BACKGROUND
- VR-storytelling (Bosworth & Sarah) (Riggs)
- VR-documentary ( Chris Milk- Empathy machine,
Maud Ceuterick & Chris Ingraham
- VR-experiences & ethics, metaverse VS representation( Murray)
- Postphenomenology
- An early VR-documentary prototype (Off Piste 2019)
METHODS
- Artistic Research methods and storytelling ethics
- comparative study / focus group interview/ user testing
- Building the experience & telling the story
- methodical progress & development - story versus ethics
- educational implementation / VR-stories by students
- theoretical development / Articulating a tool for ethical storytelling
FINDINGS AND DISCUSSION
- methodical findings
- educational findings / experiences
- theoretical findings / VR-ethics, in a spectrum of immersion and agency
About the integrity of the interviewed subject and how the story comes across to the VR-audience
INTRODUCTION:
The study presented in this exposition belongs to the artistic research project, Inside the Narrative. It derives its name and focus from the narrative point of view of the audience engaging in VR-experiences. The research project curiously joins in on the ongoing exploration of storytelling for immersive media and particularly virtual reality as a storytelling platform. What happens to storytelling when the audience is experiencing the story from within, immersed in the storys own world of audiovisual surroundings? Leaving behind the familiar rectangular framing of books and films, into simulated surroundings and interaction, new ways of guiding the audience through a story is needed. As the rules of storytelling are reorganized in a newer media format, attention to challenges concerning ethics comes along as well.
The project Inside the Narrative has its academic position in the subject area of visual communication at the Faculty of Art, Music, and Design at the University of Bergen, Norway. The project's position within artistic fields such as illustration, animation and graphic design gives us a unique opportunity to help bring forth new perspectives on VR storytelling that may differ from insights gathered from, for example the computer games industry, film, journalism, medicine or engineering. Perspectives from the visual communication field put emphasis on communication through visual concepts, expressions, written language in combination with interactivity, moving images and sound in an multimodal fashion. Creating seemless realistic illusions is normally not the main objective when creating visual art, illustration or graphic design. At the same time this study is, maybe paradoxially to the claim of beeing outside the film and video genres occupied by creating a VR-experience that is similar to a videobased 360 VR-documentary. The way it differs from a VR-documentary genre wise is in the hybridity of its means of telling the story, borrowing equally from modes of storytelling also found in edtorial illustration and printed matter. This may describe the Shadow of War VR-experience in this study as an attemt to create a VR-feature documentary that mediawise would fit into an editorial publication.
The mindset and aim of this project is both creating an artistic contemporary contribution by telling a story of a time witness from the second world war through a VR-experience, but also imagining and challenge the future of VR-experiences as a mainstream mediaplatform that may eventually produce a wide range of different types of media productions(REF:::). These productions may, as we imagine, range from a simple image posted in augmented reality to fully interactive billion dollar productions of alternative world VR-simulations, all made acceccible within a type of mataverse platform interface. Like today every mediaproduction is varying in the way it exploits the full potensial of a medium, based on limits of time, money, necessity and other requirements. The Shadow of War production, presented here, would based on its production mode fit into a simpler production format, similar maybe to a potential feature story in an editorial publication, borderlining a listening experience assisted by sherical and spatially designed atmospherical surrounding.
This researchproject scope shaped and structured to establish a base of knowledge concerning VR-experiences that may become applicable and useful for students and practitioners within the industry of design and visual communication as this field is highly influenced and transformed by the expanding field of media technologies. To find a way to bridge a transition from traditional and more established media formats into newer communicationmediums such as virtual reality we suggest the field of storytelling that will not be disappearing because of VR-technology but rather evolve as storytelling becomes part of a new immersive technology (Riggs 2019). Along side exploring the new and different modes of production and storytelling challenging aspects of ethical issues may teach us something about how a VR-experience affects us as users, audience or guests in a virtual space and how the people behind the production may become ethically responsible in their role as VR-storytellers. Practically this projects artistic, technical and technologial may become an educational example of how to produce immersive and spherical visual places for new visual storytellers, moving from traditional media into immersive storytelling.
RESEARCH QUESTION:
How can visual storytellers navigate ethical considerations in non-fiction VR-experiences?
This question is both challenging questions of how to carefully and ethically include partisipants into a documentary film type production, but it is also, at the same time exploring how the storytelling is affected, balanced and maintained. Creating effective and powerful stories will need thoughtful crafting as their origin is based in real people in the real world.
BACKGROUND:
VR-STORYTELLING:
The VR-technology has been around for many decades but it is only in the last decade that theory concering storytelling has started to emerge through publications such as, The end of storytelling (Stephanie Riggs 2019) and Crafing stories for Virtual reality (Bosworth & Sarah 2019). The Inside the Narrative project belongs to this emerging field of research In a recent artistic project called Shadow of War, a non-fiction VR-documentary became an opportunity to investigate the ethics of VR-storytelling. The understanding of ways of telling storytelling in the VR-medium have grown clearer in the last decade. Media theoretical publications offer analytical overviews of the VR storyplex and key aspect of the storytellers palette and possibilities.
VR-DOCUMENTARY:
VR-technology is known for creating a strong sense of presence for the users in virtual reality experiences (…). The feeling of presence also in non-fiction VR-experiences have amongst other aspects prompted the well known notion of VR as an “empathy machine” that say something about how VR as a medium of communication have a potential power that goes beyond former media viewing situations like watching the news on a screen. The feeling of presence seperates this medium from others and challenges debate on what producers of media content for VR-experiences should consider in the production process. This production process opens up for new possibilities of storytelling but also new challenges.
VR-EXPERIENCES & ETHICS, METAVERSE VS REPRESENTATION:
When approaching a discussion of storytelling ethics concerned with the experience of VR-mediaproduction it has been important for us in this study to to make a clear statement about our understanding of what type of human experience a VR-experience is at this point in time. The differrent views about how impactful, convincing or overwhelming a immersive VR-experience is will stear us in different directions on how to understand an ethical discussion. As a way to describe the different perspectives one might say that we are discussing wether or not a VR-experience makes us lose the distinction between what is real and what is virtual. A world of VR-experiences that will almost physically blur the lines of what is real and what is virtual will need very different measures in society in comparison to viewing VR-experiences as mostly a new extension to former audio visual screen technology. Nuancing this picture is important for the further discussion whwere still most of the current VR- experinces is still involving a user concious of the real world (Janet H. Murray 2020). As a way to move from traditional media formats of film and print the project is funded in a viewing of VR-experiences as a new arena for multiple types of experiences, not only the fully immersive, realistic real world simulations. We address the discussion about ethics and VR-technology in storytelling based on VR-experiences as a spectrum of various types of experiences that often might fit into the category of beeing a medium of representation. Still the feeling of presence in every VR experience will have to be taken seriously as a clear distinction from previous screen based mediums, also affecting the issues concerning ethics.
POSTPHENOMENOLOGY AND HCI (Human Computer Interaction):
(Dette perspektivet kan legge grunnlaget og argumentere for bruken av empirisk innhenting av kvalitativt materiale knyttet til VR-opplevelser- fokusgruppene. I tillegg redegjør det for hvordan det er nødvendig å videreuytvikle tankene våre om verden når livene vår eså sterkt er knyttet til teknologi i omgivelser og samfunn, samt egen aktive bruk av teknologi).
Postphenomenology is a philosophy of technology that is concerned
with what technologies do; what active parts they take in mediating the lifeworlds of human beings. As both postphenomenology
and HCI are empirically oriented, there is great potential for synergy.
HCI needs theoretical disciplines to ground their work, and postphenomenology needs
empirical accounts of technology relations. This section provides a brief history of the
field of HCI and accounts for how phenomenological perspectives, and recently also
postphenomenological perspectives, have been increasingly utilised.
(Vindenes 2023)
AN EARLY VR-DOCUMENTARY PROTOTYPE, OFF-PISTE 2019:
At an eight-day multidisciplinary VR storytelling workshop at Hafjell near Lillehammer in 2019, called Off-Piste Virtual Reality Storytelling Lab, the storytelling itself was put into focus. No Matter Where is the title of an experimental prototype of a VR documentary where you get to meet with various characters and their personal stories in a VR space. The project explores how to obtain, process and present documentary material in the VR format. With this prototype, we wanted to find out what it would be like to be face-to-face with someone who openly shared their life experiences? We meet three different characters within this VR installation. The thematic focus of what is being told is the various individuals' relationship to places and how these have had an impact on memories and their lives.
Wearing VR goggles with headphones in this prototype immerses the viewer into a misty and snowy landscape. The sound of a gentle breeze enhances the experience of the place's presence. The attention of the audience is directed towards a person sitting in the landscape, by means of an enticing sound. As you focus your attention on the person, the person stands up and begins to tell a story. While the person is talking, the audience can move around the space to look at the interview subject from different angles. You can also choose to look away. When you get close to the figure talking, you see in this prototype that the human figure is just a kind of 3-dimensional hollow shell that lacks the backside. This deficiency is related to the volumetric filming technique used to generate the 3-dimensional figure in the VR space. After about 3 minutes, the person ends the monologue and sits down in a waiting position. With the help of another enticing sound, the audience is led on to a new story told by another character in the same space. This time it's the sound of a radio we hear. When you turn around, another character in the landscape is appearing, which in this case is given a kitchen environment that is located in the middle of the misty landscape. As you approach, this person gets up and starts a monologue in the same manner as the first one. In this prototype, each of the stories and sounds that lead the attention is triggered by keystrokes from a technician in the room who observes the progress and course of action of the VR experience. The use of a technician in this project is a possible way to run an event course in a VR experience. The advantage of this solution is that the experience here is personalized to each viewer based on the viewer's body language and attention. This could also be automated to make production less resource-intensive and more easily divisible.
As an approach towards documentary content in this project, we mainly wanted to relate to the real world and photorealism as a starting point for the visual expression. Environmental descriptive elements were therefore gathered with photographic techniques which were later processed into 3D elements and assembled in the game engine called Unity. The snowy and misty landscape backdrop was created by a 360-degree panorama so that we could fill the whole world around the audience. This could just as well have been a 360 video recording if that had been desirable. To create the ground and the place where you stand and where the interview subjects are located, a technique called photogrammetry was used. This technique creates digital 3-dimensional objects by allowing software to analyze depth ratios in the images based on a plethora of photographs taken systematically of a place or object. This way, one can quickly capture and integrate photorealistic objects into a 3D program without necessarily being a 3D specialist.
To give the audience the opportunity to feel that they were in the same room as the interview subjects, we wanted them to be able to move with some freedom in the VR space. Therefore, this prototype does not use a classic 360 video or similar to create a VR environment. With 360 film you get a photographic visual expression, but without being able to move around in a 3-dimensional environment. One is locked to a point in the universe. Instead, the audience is offered an area of 16 m2 where they can move freely and look around both in front and behind the various objects in the space.
To film the interview subjects that we later encounter in the VR experience, we use a technique called volumetric capture. This is done by using a Kinect camera and software called Holo Cap that combines a video recording with 3D data read by an infrared camera. In this way, we could capture a 3-dimensional figure in the space and not only a regular flat 2D video image. It is only from a partially frontal perspective that it works visually from the point of view of the audience. Still, it is not flat as a video recording, so that the figure's body shape becomes visible in space, making it possible to integrate the figure with other 3-dimensional elements in the space.
The sound recordings used in the VR experience were organized digitally in different virtual places in the VR space in the same way as you would distribute 3D objects. In this way, you can organize sound in the game engine Unity (alternatively the game engine Unreal Engine) so that you experience the sound coming from different directions and at different distances.
Through working with the prototype, we engaged in processes and techniques that are largely about creating virtual places with different opportunities for experiences and actions. The audience's role in this particular VR experience was as a kind of mobile observer without so much opportunity to influence the environment. Still, the audience´s meeting with the interviewed subjects and their personal story inside the VR space seemed to be one of apparent emotional impact. The choice of controlling the experience by the use of a technician tells us that VR not only needs to be a lonely digital experience but can also be conceived as a type of performance where the experience is shaped through a feedback loop like in the theater. After working with this VR-prototype, the overall experience is that focusing on the quality of the story itself still remains important also from the inside of the narrative.
METHODS:
OVERVIEW:
When discussing ethics concerning production of VR-experinces it might be relevant to distinguish between the various types of experinces and and understand what factors that decide wether you are working on a production that is exposed to a higher degree of ethical challenges or not.
