Volumetric filmmaking, new mediums and formats for digital audiovisual storytelling

1. Introduction

Artistic practice has always been a spearhead in the experimentation with the emerging technologies of its time. With the emergence of photography and later the cinematographer, several authors pioneered the use of these means for cultural expression. If there is a difference in the audiovisual medium, the seventh art and the predecessor media, is the subversion of the order of necessity. In this sense, if we look at painting and go back to the moment of art-magic, the painters of the caves of Altamira found first the need for expression before the natural reality that escaped from their control, and starting from it they sought the procedure, pigments, soil and elements in order to conduct their paintings. In cinematic and audiovisual practice, we see an inversion of these aspects, being created first the technique and subsequently identified the need for expression through it. We could understand this process as a mediation between the need for expression and technology, between cultural production and the means put at its service.

At the end of the twentieth century, with the appearance of the first camcorders, in addition to the satellite broadcast systems, there were multiple artists and creators who began to introduce these new procedures into their artistic practice. The discourses of the so-called video art began to break into the contemporary art scene. During the last decade, and after overcoming aesthetic questions regarding the objectification of the cultural product and the classic concept of work of art, digital media, and between them virtual and augmented reality, break into the field of audiovisual productions. In the face of these technological devices, contemporary creators begin to take the pulse of their languages, heirs of audiovisual artistic practice, who combine in their own procedures aspects of cinema, mass culture, video game, 3D and video art, performing arts and installation, exploring new methods to create more immersive audience experiences, capable of introducing the observer as part and co-creator of the storytelling.

In this regard, Francisco Javier Gómez Tarín introduces the discussion about the fragmentation of contemporary audiovisual discourse in his text La quiebra de los paradigmas audiovisuales: hibridación Vs canon (2010):

The truth is that we face a fragmented world whose essence itself is hybridization (say that , more than ever , not as an issue that now appears and did not have an existence before), and a simple look is enough to detect that this – hybridization – takes place in various areas that affect the whole of the audiovisual and that we organize here according to its levels of abstraction:

1. Technologies;
2. Format and media, both for creation and consumption;
3. Significations, as a result of the use of expressive and narrative resources; and, finally,
4. Discourses, understood as dialectical, and even interactive final processes, between emisors and recipients (Gómez Tarín, 2010: 26).

We see, therefore, in this first classification, the elements that come into play in the development of audiovisual productions and their hybrid mediation between the different technologies, languages, creation and consumption strategies, and the processes of communication between author and audience, where the participation of the viewer is paramount, and this varies depending on those interfaces applied in the construction of the discourse.

1.1. Art-technology mediation as an activist strategy

First video artists, in which we could see the origin of the medial cultural practices between art and technology, in the late 60s and 70s, began an audiovisual practice that in most cases connected with political activism and criticism of mass media as social builders. Burris in this regard states:

Whereas previous new media had emerged gradually with one or two styles or aesthetic approaches, the most distinctive aspect of video’s formation was the immediate and simultaneous emergence of multiple genres: activist, documentary, synthesized and image-processed, abstract or abstractive, performance, conceptual, ecological, diaristic, agit-prop, dance, music, bio-feedback and other forms made their appearance in the years 1968-1972. […] There was an atmosphere of mutual support and a sense of a shared and privileged destiny investing video with powerful aspirations to be what no other medium had been, nor had been asked to be: at one and the same time a medium through which to view the world, a means to test the limits of the world, a political tool, a communications tool, and a responsive art medium (Burris, 1996).

Some of today’s cinematic practices with VR and AR technologies are framed in this activist line, beyond the first productions that were solely searching for the “wow” effect of these new modes of creation. Taking advantage of the characteristics of these media, the most devoted creators try to promulgate discourses with the intention inherited from those video artists committed with their time. Audiovisual image is a reflection of the author’s thinking, and in turn, is able to build the audience’s ways of thinking, ultimately generating the paradigm of contemporary thought. According to Enrique Álvarez Asiáin, in his text about the work of Gilles Deleuze: “The cinematic image is, therefore, a critique of representation and, as such, is capable of producing a ‘new image of thought’ that shows philosophy a way out of dogmatic imaging” (2007). Rosler, at the same time, and in relation to the first video artists, refers to the criticism included, both systemic and utopian, with a clear intention of transformation of the system inherited from the avant-garde, because we must not leave aside that every artistic practice has had in its intrinsic nature a modeling function of the societies of its time:

Not only a systemic but also utopian critique was implicit, the effort was not to enter the system but to transform every aspect of it and -- inherited from the revolutionary avant-garde project -- to redefine the system outwards, uniting art with social life and making viewer and producer have exchange-exchanged functions (Rosler, 1990: 31).

In addition, it already intuits the need to involve the audience in the processes of creation and production, in the same discourse itself, anticipating the current issues regarding the phenomenon of prosumerism. Therefore, the breeding ground is raised for contemporary creators to explore the possibilities of VR and AR to generate activist audiovisual discourses, capable to move the awareness of the viewer, which transcends from observer to activist subject within the work itself. One of the characteristics of these media, which is currently being tested for its impact capacity, is its use as a tool for generating empathy across its immersive power, through which it engages the audience as part of the discourse and enhances its subsequent involvement in the problems addressed in these audiovisual works, whose narratives transcend the rectangular screen to generate simulated experiences.

1.2. VR and AR as generators of empathy

From its technological appeal to new generations, virtual and augmented reality films, as a result of the fusion of traditional media with technological advances, function as prosocial tools and catalysts of social activism, since their conception as machines of empathy, according to visual artist Chris Milk: “virtual reality connects humans with other humans in a profound way that I have never seen before in any other form of media, and can change people’s perception of each other” (Milk, 2015).

Researchers on the field of VR have demonstrated that it could work as an empathy machine. There are some research works as 1000 Cutjourney, developed by Jeremy Bailenson and Professor Courtney D. Cogburn. A VR perspective-taking experience where the participant is exposed as a black male in different virtual situations of racism during his lifetime. Also Building long-term empathy: A large-scale comparison of traditional and virtual reality perspective-taking (Herrera, Bailenson, Weisz, Ogle & Zaki, 2018), demonstrates that VR perspective-taking tasks are more effective in terms of developing empathy on the audience in a medium and long term, in this case regarding homeless situation. In this research, they worked with the VR experience Becoming Homeless: A Human Experience (2018), where the users get involved in the situation of an individual that loses his job and all his properties, so the audience experience how it feels to be a homeless. These works tackle social issues regarding minorities through the use of “being in another’s shoes” strategy:

[...] extensive research has shown that when we’re asked to take on the perspective of the person we’re judging, in a virtual setting, we tend to give them the same benefit of doubt we’d give ourselves (Seward Jr., 2017)

Another research work about the use of VR to generate embodied experiences to catalyst helping conducts in the user, The effect of embodied experiences on self-other merging, attitude, and helping behaviour (Ahn et al., 2013), concludes:

With EE, the user is able to vividly, accurately, and realistically experience the sensations of another person and feel as if they have merged with that person. This sense of self-other merging in the virtual environment transfers to the physical world and translates in actual helping behaviour, even when the other person is a complete stranger.

Moreover, the article Walk A Mile in Digital Shoes: The Impact of Embodied Perspective-Taking on The Reduction of Negative Stereotyping Immersive in Virtual Environments (Yee & Bailenson, 2006), sets the hypothesis of virtual environments to provide a direct way of taking the perspective of another, thus reducing the negative stereotypes, in this case about the elderly, demonstrating that with the use of VR, negative stereotypes were reduced when the individuals were put in the skin of old people.

One of the first VR films that puts the audience in the position of experiencing the intimate interior worlds of others in the transgender, vulnerable, diversity context is Queerskins: a love story (Illya Szilak and Cyril Tsiboulski, 2018). An haptic cinematic VR experience that aims to put the viewer in the skin of an LGTBIQ person, in the material reality that they experience every day, as a means for changing the users own mind, accepting and supporting others within or outside their community.

Another relevant work in the context of film productions with VR is Carne y Arena (González Iñárritu, 2017). This work also explores the presence factor, the personification of the experience in the spectator, who gets in the first person at the situation of an illegal migrant who tries to cross the Sonora desert to reach the United States. Beyond its prosocial perspective, this interactive VR film is a pioneer in the exploration of new audiovisual languages for immersive environments:

[...] The fact of placing the viewer in the place of the actor, within the recreation of the events, and the possibilities to interact, to direct their attention autonomously and to interpret the piece from the place of the camera, generates new ways of understanding and creating audiovisual diegesis (Martínez-Cano, 2018).

Placing the audience in a virtual context where experiencing a specific situation makes the viewer empathize, in a direct way, with their double in virtual fiction, thus promoting empathetic behaviors towards the topics covered. Undoubtedly, this characteristic, together with the possibility of offering the viewer a layer of interaction within the diegesis of the work, are the two pillars that promote its purpose as a tool of social activism. However, it is not only from the cinematic field from which these artistic practices are enacted, but there are, from the hybridization of technologies and media, several fronts from which these discourses are broadcast, whether from the field of film practice, artistic practice, performing arts or video games.

1.3. The impossibility of classifying VR and AR audiovisual practice

Given the complexity of areas from which the audiovisual practice is being developed, it is difficult to establish an appropriate classification to the level of expansion of these cultural productions and their different creation frameworks. The classifications and curatorship models, used in the context of contemporary art, are not applicable to identify artists who currently work with the moving image and its multiple technological mediations, since these are distributed from museums and theaters, galleries, film festivals and performing arts, to other areas designed as non-high culture, such as social media, mainstream cinema, video on demand platforms or video games.

Film festivals, as film promotion entities, include new sections over the past few decades that accommodate productions that are difficult to catalog within conventional categories. The Tribecca Immersive, Sundance with its New Frontier section, going through the VR AT EFM Berlinale, Cannes XR Development Showcase, VR FEST Las Vegas or the Tampere Film Festival Open Eyes, are some examples of the incorporation of these recent sections, indicating the advancement of VR and AR productions in international circuits, as well as their impact. Currently, many of these events are included as “a complementary activity, in official section but out of competition”. The festivals specialized in this field are not yet comparable in number to the traditional ones, “but many support this format in competitive section” (Jurado-Martín, 2020).

It is for this reason that the media ecosystem, in which we find ourselves, requires a new paradigm when it comes to structuring it, starting from the multidisciplinary perspective of its cultural productions. In addition, it must be a model that does not separate the different emerging media categories, but one that unites them in the same space of creation, analysis and study, from which it is possible to enhance their evolution, while identifying what are the challenges that the authors face in this new scenario. As Benjamin Ogrodnik points out, in his review of the book-catalogue of the exhibition Dreamlands: Immersive Cinema and Art, 1905-2016, held at the Whitney Museum of American Art:

In place of the filmmaker-poet or artist-activist, the protagonist of Iles’s show is the cyborgian artist. Iles adopts Donna Haraway’s term in her catalog essay, using it to designate a protean figure who manipulates the interpenetration of technology and nature, capitalism and revolutionary politics. In Dreamlands, cyborgian artwork exists in multiple forms and formats, unconcerned with barriers of high and low culture or creative medium (Ogrodnik, 2017: 110).

2. Methodology

This work is based on a three-phase methodology. A first bibliographic review, from which the previous considerations and state of the art of the first film productions with VR and AR are established. Subsequently, we delve into traditional film production techniques and their adaptation to emerging media, concentrating our vision on the technical development of volumetric cinema and its application to the construction of virtual and augmented audiovisual diegesis. Finally, an analysis tool has been developed, which we will apply for the studio of the augmented interactive documentary film Terminal 3 (Malik, 2018). An audiovisual work that exploits the potential of this medium as a prosocial tool.

This model consists mainly of three blocks containing the variables that we consider relevant for this type of audiovisual production. The first is the concept of presence, in which we will assess whether the presence of the work under study can be considered as a social, environmental or personal, according to the classification established by Heeter (1992: 263-264). In this same block, the rules of interaction, whether or not navigation exists, as well as the type of movements within the virtual environment, as considered by Slater and Wilbur in relation to an effective presence achievement (1997) will also be evaluated. In the second, we will address the issues related to audiovisual realization, from those aspects related to the camera, point of view, movements, direction of attention, sound production, lighting, assembly and editing, creation of identity in the user, etc. The third block will address issues relating to its narrative structure.

3. VR and AR filmmaking

The conventions of audiovisual and cinematographic language, developed during the twentieth century, are established as the basis on which the strategies of building narratives with VR and AR begin to be developed. To build new cultural products with these technologies, it is necessary to deepen experimentation with these means that are in their early stage of evolution. In line with the concept of Bazin’s “total cinema” (1966), Bucher points out that the kinematic VR, which will combine film techniques with the immersive principles of these media, will result in a new, yet unclassified experience that will increase the meanings in the viewer (2017:6).

As for research on VR and AR filmmaking, it focuses on several basic concepts. The suspension of disbelief (Bates, 1991), the technical parallelism between film productions as we know them and the new productions, deepening the engagement of the audience through the user’s point of view (Cho et al., 2016), the difficulty of directing the audience’s gaze, key factor for narrative understanding (Syrett, Calvi and Van Gisbergen, 2016), along with Green and Brock’s literary theory of transportation (2000: 701), translated into the concept of presence (Mateer, 2017), which takes over the filmic concept of “suspension of disbelief”, coined by the poet Samuel Taylor Coleridge in the early 19th century. As for the adaptation of traditional moving image language resources in the context of VR and AR, continuity was initially presented as a differentiating factor, because the first 360 productions did not feature any editions. In fact, raccord turns out to be a common factor, from the perspective of internal or external montage, since post-production for VR and AR has already been used (Ijäs, 2016). This issue becomes more complex, as one of its differential additions is user interaction, resulting in multiple and branched narrative structures. It is therefore necessary to check the extent to which traditional filmmaking can be adapted to the VR and AR media.

3.1 Adaptation of conventional techniques from cinematic direction to VR and AR

The adaptation of traditional procedures of audiovisual production to the field of VR and AR is in an initial state. While differentiating between the two media, as the conditions of VR differ in certain aspects from those of the AR. First, pre-production phases are common and adaptable, even if narrative structures do not abandon linearity, ramifications and connections between events are generated, in both cases due to user interaction in the diegetic course. Therefore, the script for VR and AR results in a more complex narrative architecture. The direction of the gaze is another differentiating aspect that presents a challenge, when the limits of the “diegetic window” disappear, In addition, while in virtual and augmented environments, the user can turn their attention freely, in the AR the virtual elements are overprinted with the real ones, so the direction of the gaze and the camera position require a scenic design in the real space, on which the overprinted virtual elements are configured. However, for VR, the entire environment is built in 3D, and the user can be in a diaphanous space, while in the virtual location.

As for the point of view (Genette, quoted by Jost and Gaudreault,1995), we look at a new way of telling stories, because with VR and AR we must leave the diegetic course in the hands of the audience. Another key issue is to move the viewer away from the technical equipment (Bordwell, Staiger and Thompson, 1985), a complicated issue because the modes of audiovisual consumption with VR and AR depend on specific HMD viewers, and controls as gamepads to be able to interact with the work, but we can keep any technical element out of image, also a complex issue, especially when we talk about VR, because while shooting a 360 image we have to hide all the human and technical equipment in front of a system that records everything around it.

Another important aspect is the position of the camera and its movements. In VR and AR, framing disappears, and the camera moves as the user explores the space, although it is usually positioned statically, sitting or standing. There is also the movement of the virtual 360 image itself, as in the case of the archetypal VR roller coaster experience. According to Zelcer: “Virtual reality images are then a sequence made available on which the vision is determined, as opposed to the cinema, by the user” (Zelcer, 2017:337). This affirmation also serves AR, although in its case, the user visualizes virtual elements coexisting in the actual space, generating a duality that connects in a different way than in VR. This concept of “dual ubiquity” (De La Peña, 2015) works in both cases as a new strategy of audiovisual activism, placing ourselves inside the events, on the scene, within the diegesis of fiction, being aware that we are not there, but at the same time feeling the experience as if we were really part of it, giving the creators access to the generation of empathetic links with the audience in relation to the events to which they approach, as interactive voyeurs in a simulation, as opposed to the representation of the traditional moving image. We could emphasize that both AR and VR place the viewer in a simulated context, although in the virtual environment through a three-dimensional reconstruction of the space-time where the work takes place, unlike the augmented audiovisual, which requires a physical stage montage in which the viewer is consciously located, where the overprinted virtual elements are synchronized. In both cases the user enters in an immersive fictional space.

Therefore, filmmaking for virtual and augmented reality is mainly about positioning the camera, in addition to the preparation of the elements of the virtual or physical environment in the case of AR, and in case it is going to move, it must be found the precise mode to justify such movement, since usually the viewer occupies the view of the camera as a subjective point of view. To this respect, Márquez points out: “If for some reason VR systems are characterized it is by its first-person vision, with the cinematic effect of the subjective camera in motion” (2015: 206, quoted in Zelcer,2017: 335). Kelly (2016), on the contrary, argues that it is not a subjective point of view, but conceptualizes it as “you person view”. However, Mariano Zelcer states that if there is no intentionality of the gaze, you cannot point out the camera position in virtual reality as a first-person point of view:

[...] there is no “subjective camera” in virtual reality as seen in cinema, since there is no definition of a directionality of the gaze on the image, but only a position. This particularity of VR images, plus those already indicated concerning the change that occurs in them with the movement of the spectator subject, make the subjective camera not in these images a marked enunciation effect but, on the contrary, one of the principles constituting its transparency: things are simply there, the events occur and we observe them (Zelcer, 2017: 336).

Hence, Jessica Brillhart, principal filmmaker for VR at Google, warns that what is really important in terms of the point of view, and in particular in interactive productions, is to give the viewer an identity, and connects it to the concept of presence and energy. The latter related to editing, when it addresses the traditional resources of cinematic language and its application to audiovisual creation for VR, using as an example the visual contact that we could have in a virtual reality film with one of the actors, identifying it as a possible analogy to the traditional close-up. According to Brillhart: “Energy, or the emotional journey of the audience through the experience, and perception, how the viewer is experiencing the world, should drive the technical decision making in VR storytelling” (Brillhart, cited in Bucher, 2017: 16). We could point to a loss of frame scale, which somehow still exist in the context of VR, since as Brillhart states, even if it is not a fitted image, the energy that gives off while looking at a certain place of virtual fiction can be associated with a particular type of shot.

3.1.1 Editing in VR and AR

Current procedures and software allow the editing of the image record for virtual reality, making it possible to guide the viewer’s gaze through the directions and movements of the camera, as lateral or perpendicular pannings, maintaining the position of the camera. In addition, you can construct an alternation of shots through fades, which tend to keep the effect more immersive, but also by cutting. However, these procedures are far from being defined at present:

In terms of the statement, VR experiences do not finish setting up a discourse as we knew it in the cinematic language, which involved a certain operational syntagmatic contiguity that included the selection of shots or montage; instead, there is a certain temporal space setting over which the selection and combination of shots (or rather, the shapes acquired by a large sequence shot) will be defined each time (Zelcer, 2017: 338).

In addition, as regards the question of cutting editing, Zelcer adds that this method draws us from the sensation of being immersed in the fictional space, using it as a differentiating element between virtual reality film and experience, because in the construction of experiences it is necessary to maintain the immersion of the viewer. Brillhart, in relation to editing warns that direct cut edition does not work the same in the conventional medium as in virtual reality, and connects it to the concept of energy:

You edit things for focus. You build in systems to translate feeling and vibe and energy to someone who’s not a part of that world. What’s interesting is that energy is very palpable in VR. You can basically have a person in a space and can make them feel very awkward very fast. The same kind of quick cuts and the disorientation doesn’t really work the same as it does in conventional cinema. Again, it’s that whole unpacking idea where energy means something different in VR. Energy means the way that people relate to you. Energy is the chaos of the space (Brillhart, cited in Bucher, 2017: 14).

The montage of stories in virtual reality is therefore also a fundamental piece, and decisions regarding it have to be contingent on the systems of interaction and the model of narrative structure. You cannot establish at the present time any final defined standard on audiovisual editing for interactive formats, it is necessary to continue with the experimentation, to observe through proof of concept the different systems that can be obtained, in order to translate the feeling and energy of the author’s intentionality and his account to someone who is not part of that diegetic universe. What is constant is that continuity, in its multiple forms, is a key element in formulating an immersive audiovisual discourse.

However, the assembly for VR and AR is contingent on the user’s interactive processes over the plot development, usually produced with video game engines capable of managing the future of diegetic events through the inputs of the audience. In this sense, the editing and post-production for AR requires a particular approach to the elements that are overprinted to the real-world image of the viewer, maintaining a branched structure. The camera does not move, as it presents a static point of view so that it fits in a synchronized manner in the spatial context in which the magnified image is displayed. The issues of editing for AR seem to be simpler, but these become more complex when we consider the main difference between the two media. Obviously, there will be no cut-off assembly, the fade in and fade out will be used, whether or not the objects enter the real scene where we locate the viewer, but the post-production will be much more intense, due to the necessary synchronization of its components, as well as the key nodes in the interactive development of the storytelling.

3.1.2 Volumetric cinema

Audiovisual productions for virtual and augmented reality, as mentioned before, began to develop from 3D graphics, a synthetic image that approximated the medium to the video game. Later, with the development of software for stiching, a technique consisting of ‘sewing’ the images recorded by a variable set of cameras located in a spherical structure, the image record appears 360 degrees, photographic and moving. Through UDK and Unity video game engines, the merging of both methods was made possible, placing virtual items over the 360 image. Currently, with the development of software, and using a synchronized Kinect sensor and DSLR camera, you could get a volumetric recording of the shot, which can later be composed with virtual elements and 360 video, or overprinted in a scenic space, resulting in a new audiovisual production system for VR and AR. This method is used in the virtual short film Queerskins: a Love Story (Szilak and Tsiboulsk, 2018), and in the augmented interactive documentary Terminal 3 (Malik, 2018).

The creation of volumetric films is an expanding movement around immersive audiovisual content, characterized by interactive experiences created with scanned images predominantly in 3D, through motion sensors. Techniques such as volumetric video and photogrammetry are used in conjunction with video game engines to enable viewer ubiquity in rebuilt realistic environments, through real image recording in volumetric video format, combined with 3D synthetic elements, or the ubiquity of holograms in fictional audience-dwelling physical space, where interaction interferes in the course of events. A hybrid between video games and cinema, the making of volumetric films is inspired by related creative disciplines such as documentary, immersive theater, cinematic virtual reality or interactive installations, generating a new system of audiovisual expression.

From a technical perspective, the final image for VR is a three-layer composition:

1. Video recording 360º of outdoor environments.
2. 3D reconstruction of sets and action space elements.
3. Volumetric video recording with 4K cameras, Kinect sensors and Depthkit software for recording actor performance in chroma key sets.

However, for AR it would consist only of the volumetric video record of the third point, as this record would be overprinted in the viewer’s vision through the AR display, as an hologram over the actual space surrounding the user. In this sense, the second layer would consist of that actual space, which the director must design to support the audiovisual narrative and reflection he wants to propose.

4. Analysis of Terminal 3 (Malik, 2018)

The work under study has been defined as an augmented interactive documentary. Produced by 1RIC studio, it addresses the conflict generated by U.S. immigration policies based on certain nationalities, race and religious beliefs, when entering the American country. In particular, the viewer is invited to access an installation, which recreates the spaces of an airport and the rooms where some immigrants are taken, with the aim to get in the shoes of a border patrol agent interrogating an hologram of a so-called Muslim, to decide whether or not can enter the country. It was presented at the Tribecca Film Festival 2018, where a customs-controlled facility was created, in which the viewer immerses himself in the holographic work through Microsoft’s Hololens goggles.

Terminal 3 is based on the idea of cultural augmentation, which allows to place and individual in front of a person’s hologram in a space that would never inhabit. According to its author: “It challenges people to share presence with ideologies and humans that otherwise would not cross paths with” (Malik, 2018). Based on his own experience, the author poses a narrative that questions prejudices to different ideologies, in order for the viewer to finish with a moving new vision. Cultural identities, which transfer and cross borders, confronting various contradictions, this is the breeding ground of this augmented film, halfway between documentary and fiction, as the six holograms that the audience can access to interrogate are actual people, who expose situations lived in their own skin.

When accessing the fictional space, we are given an Hololens and offered a seat in the room, in front of a stool in which is located the volumetric record of the individual to which we have to interview. When the interrogation ends, we are directed to another room, where we are supposed to attend another interrogation, but first we must communicate to the voiceover that asks us rigorously whether or not that person can enter the country. The final turn, is that once we have made and communicated our decision, as we move to the other room, we meet the real person we have just interviewed in the hologram, as a final catharsis the work unites the virtual and the actual to touch the audience.

These sorts of questions to choose from are structured into a branched and individual narrative for each of the holograms that includes this augmented footage. Unlike VR productions, we are not placed in the shoes of the main character, but we are urged to a position of power, in a dynamic that takes a turn in its end, transferring that initial power of our character, to the real person once we meet her or him. At the same time, the plot is not exposed in a direct way, but the viewer must perform the exercise of building the identity of the person represented in the holographic image. The viewer must build the identity of the person he is questioning. It’s not a matter of the character we are dealing with, but about the person depicted and personal conflict we face while we are immersed in the play. Human relations are questioned, as are the dehumanization mechanisms of the system.

4.1 Achieving the presence effect

The feeling of presence in the viewer is reached in the first place by the recreation of the space where the action is located, as a scenario where the user is immersed in. The camera position comes into play when the footage filmed as 3D holograms is accessed through the AR display. In this sense we not only attend the creation of the presence effect on the part of the viewer, but also of the presence effect that reaches the volumetric record of the moving image that is shown. On the one hand, taking the concepts of Heeter (1992, 263-264), we identify a strategy of environmental presence, from the point of view that the medium itself offers oral interaction with the volumetric image, contributing to the suspension of disbelief while fomenting the feeling that the cinematic element knows of our presence and reacts to our stimuli, which attests to the existence of social presence. Personal presence is also identified, as the user is able to visualize his own hands in the context of the synchronization of the actual installation space and the diegetic virtual elements.

Likewise, this augmented film incorporates into its diegesis the three rules of Slater and Wilbur (1997), with the aim of achieving the presence effect in the viewer and maintaining it throughout the audiovisual discourse. In this sense, a voice interaction system intervenes with simple holographic recording, composed of two questions to choice one, offered at certain points of the narrative, to which a simple navigation of the user in the environment is added, through the image overprinted on the physical environment shown in the HMD and the implicit movement of the user’s gaze towards the reference, since its location is defined in a position, sitting in the fictional space recreated through the actual scenography of the work.

4.2 Volumetric audiovisual direction

First, collecting the concepts set out by Brillhart, in which the creation of the identity of the viewer in the context of the immersive work is paramount, Terminal 3 establishes this criterion and executes it effectively, however the identity of the viewer in the work is not the main aspect of the discourse, since it holds a secondary role, although it is created in a consistent way, giving the user a power within a dynamic that takes a turn in his second act. The importance in this sense lies in the creation of the identity of the hologram that is shown through the volumetric motion picture, main character of the plot. We are sitting in a room to interrogate the character shown to us, so the camera-viewer follows the justified trajectory in the diegetic context, without the existence of camera maneuvers beyond the head moves that the user can perform to direct his or her gaze. In this way the user’s attention is directed towards the discursive element of the work. Navigation around the volumetric image is not possible, since this recording of the holograms has been made from a specific point of view. This aspect is not relevant, because the position of the viewer is delimited from where is sitting in the recreated space, so the camera is located on the basis of this precept, to show a perfect sync of the hologram projected.

The treatment of the sound in this work is fundamental, part of the spatial system that incorporates the Hololens, and that is determined by the interaction of the user depending on the questions that are chosen to ask the character shown, so it can be defined as interactive spatial sound. In this sense, there is a design of soundscapes and wide sound records, which depending on what the hologram answers according to the question that is asked, it takes the audience to those facts or situations that are described in its response, being able to transport us to the Sahara desert, listen to the arrival of a car in the distance and the footsteps after someone gets off it, among others. Through this deep sound treatment, focused on the different situations that can be reflected in the work, the immersive effect of audiovisual storytelling is enhanced.

Regarding the issues of mountage, there is no need for a joint between different types of shot, obviously because the volumetric record that builds the hologram is maintained in a still sequence shot, according to the point of view where the viewer is going to be located, because this volumetric image does not complete the 360 degrees. Post-production issues are more focused on the construction of audiovisual work in Unity video game engine, the creation of the ramifications of the narrative structure according to the user’s interaction, also created and generated in this video game development software, and the introduction in the virtual environment of the sound productions that must accompany every moment of the interrogation. A complex process that includes not only issues of audiovisual image, but also of computer programming, virtual development, design of video game mechanics, spatial sound design and ultimately the implementation of other disciplines to the creation of the final AR interactive film.

4.3 Narrative structure

One of the most important aspects when generating a deep immersion of the spectator in the work is the interaction system. Usually and in the context of VR works, interaction devices such as gamepads in the hands of the user result in a kind of external elements that take us away from the immersive purpose of the work by its very nature. However, Terminal 3 offers a voice interaction system, giving it a more actual immersive effect. Yet these questions are pre-directed, as the user is offered two options at a time, from which to choose one. These options are displayed through the HDM display, and activated at the time the user asks one of these two questions to the hologram. Answers are also pre-recorded.

The narrative, therefore, flows depending on our decisions regarding the questions that we choose to ask, receiving information dependent on these decisions and therefore, generating an individualized experience within the finite limits of the branching storytelling. The holograms initial representation is abstract, a glitch image, which gradually over the course of the interrogation acquires a real appearance, as we know more about the person we interrogate. In this sense, the criterion of representation and its evolution are also part of the development of the narrative structure.

Each hologram presents a story built on a complex branched narrative structure, in which the user navigates freely, conditioned on the decision between two possible question choices. In this sense we can choose the most human and intimate question, or the most institutional and dehumanized one, giving the viewer the possibility to exercise power over the volumetric record. The structure is divided into two acts, a first in which we interrogate the holographic image and interact with it, and a second in which after having made the decision to let that individual into the country or not, we find the real person in the next cabin, which generates a connection between the virtual and the real, leading to a switch of power in the dynamics of relationship between the viewer and the hologram.

5. Discussion and conclusions

We are faced with the first augmented interactive film, a walk between the documentary, since the record caters to actual characters and stories, yet fictional, since the discourse is co-created from the user’s interaction, in a kind of illusion of choice power as to what questions she/he asks to the hologram. All this attests to the evolution of contemporary storytelling, through the experimentation and production of works capable of creating interactive audiovisual diegetic systems, resulting in a mixture of disciplines with an eagerness to transcend a message to the audience, transferring a deep reflection on conflicts of human nature. To this end, the use of the subjective point of view, which assumes the use of axes of action, is applied as it is not a complete three-dimensional record, indicating its connection with traditional film practice. Add to this issue, there is a type of internal montage marked by user interaction during the course of the storytelling. This implies an emotional factor due to the juxtaposition of the virtual and the actual presence in the configuration of the work in two acts.

From the prosocial perspective, the narrative does not evolve without the participation of the viewer, and in this process that incorporates the action of the receiver in a context of virtual simulation generated with AR, unlike the strategies used in VR, it does not try to put you in the situation of the other, but puts you in front of the other that coexists in your same space-time, being yourself, becoming a trigger, and being responsible for the fate of the person in the hologram. In this regard, the strategy seeks to change the way we observe our reality, through a dynamic of power. It is in this approach where its prosocial function lies. The audience’s reaction reflects the impact that AR can achieve on reality, and reflects the ability of the augmented media to build the presence and to process a change in the way the viewer perceives and beholds reality.

The term empathy machine seems to be lately overused and worn, since in a sense it promulgates a kind of moral superiority of the audience that attends a virtual audiovisual piece. Just by placing them as observers of an extreme situation, remote to their usual location witnessing a social drama, we cannot think that this approach is able to change the perspective of the spectator. Connecting with this idea, the author justifies the use of AR and putting the user in a position of secondary actor within the work, establishing a distance with the stereotyped strategies of virtual audiovisual media, that are proclaimed as the new machines of empathy.

AR audiovisual language is in development, yet only by experimentation and the creation of more AR films it will become an standardized media. To this end, the work object of study contributes directly. Terminal 3 is one of the contemporary exponents of AR volumetric filmmaking, configured with the aim to transform the audience perception of reality. It is proven that storytelling is expanding through the combination of several artistic practices, putting interaction as the major feature for the AR movies to come.

6. Bibliographic references

Ahn, Sun Joo, Le, Amanda Minh Tran & Bailenson, Jeremy. 2013. “The effect of embodied experiences on self-other merging, attitude, and helping behavior” in Media Psychology, nº 16(1): 7-38.

Asiáin, Enrique Álvarez. 2007. “La imagen del pensamiento en Gilles Deleuze; Tensiones entre cine y filosofía” in Observaciones filosóficas, (5), 3.

Bates, Joseph. 1992. “Virtual reality, art, and entertainment” in Presence: Teleoperators & Virtual Environments, nº 1(1): 133-138.

Bazin, André. 1966. ¿Qué es el cine? Madrid: Rialp.

Bordwell, David, Staiger, Janet & Thompson, Kristin. 1985. The classical Hollywood cinema: Film style & mode of production to 1960. Columbia University Press.

Bucher, John. 2017. Storytelling for virtual reality: methods and principles for crafting immersive narratives. Routledge.

Burris, Jon. 1996. “Did the portapak cause video art? Notes on the formation of a new médium” in Millennium Film Journal nº 29: 2-28.

Cho, Jaehee, Lee, Tsung-Han, Ogden, Joel, Stewart, Amy, Tsai, Tsung-Yu, Chen, Junwen & Vituccio, Ralph. 2016, July.

“Imago: presence and emotion in virtual reality” in ACM SIGGRAPH 2016 VR Village (p. 6). ACM.

Cogburn, Courtnye D., Bailenson, Jeremy, Ogle, Elise, Asher, Tobin, & Nichols, Teff. 2018, August. “1000 Cut journey” In ACM SIGGRAPH 2018 Virtual, Augmented, and Mixed Reality (p. 1). ACM.

De La Peña, Nonny. 2015. The future of news. Virtual reality. Video (May 2015). Retrieved February, 19, 2020, from: https://bit.ly/2UWzICB

Gaudreault, André & Jost, François. 1995. El relato cinematográfico. Buenos Aires: Paidós.

Gómez Tarín, Francisco Javier. 2010. “La quiebra de los paradigmas audiovisuales: Hibridación vs. Canon” in Revista Latina de Comunicación Social, 23-42.

Green, Melany C. & Brock, Timothy C. 2000. “The role of transportation in the persuasiveness of

public narratives” in Journal of personality and social psychology, 79(5): 701.

Heeter, Carrie. 1992. “Being there: The subjective experience of presence” in Presence:

Teleoperators & Virtual Environments, 1(2): 262-271.

Herrera, Fernanda, Bailenson, Jeremy, Weisz, Erika, Ogle, Elise & Zaki, Jamil. 2018. “Building long-term empathy: A large-scale comparison of traditional and virtual reality perspective-taking” in PloS one, 13(10), e0204494.

Ijäs, Nina. 2016. Transitioning Between Worlds Editing and Pre-production in Cinematic Virtual Reality.

Jurado-Martín, Montserrat. 2020. “Festivales de cine especializados en realidad virtual en España” in Sphera Pública, 20(1).

Kelly, Kevin 2016. “Hyper Vision” in Wired, April (digital version). Retrieved November 05, 2019, from: https://bit.ly/2x8y0oH

Malik, Asad. J. 2018. “Making of TERMINAL 3”. Video (April, 2018). Retrieved February 19, 2020, from: https://bit.ly/34qSSUn

Martínez-Cano, Francisco-Julián. 2018. “Impresiones sobre Carne y Arena: práctica cinematográfica y realidad virtual” in Miguel Hernández Communication Journal, nº 9: 161-190.

Mateer, John. 2017. “Directing for Cinematic Virtual Reality: how the traditional film director’s craft applies to immersive environments and notions of presence” in Journal of Media Practice, nº 18(1): 14-25.

Milk, Chris. 2015. “How virtual reality can create the ultimate empathy machine” in TED 2015.

Retrieved November 22, 2019, from: https://bit.ly/2JVzZ2a

Ogrodnik, Benjamin. 2017. “Dreamlands: Immersive Cinema and Art, 1905–2016: Whitney Museum of American Art, October 26, 2016 – February 5, 2017” in Contemporaneity: Historical Presence in Visual Culture, nº 6: 109-112.

Rosler, Martha. 1990. “Video: Shedding the utopian momento” in Illuminating Video: An Essential Guide to Video Art, 31-50.

Seward Jr., Kenneth. 2017. “How virtual reality can be used to fight racism and prejudice in society”. Retrieved January 10, 2020, from: https://bit.ly/2XBGsrp

Slater, Mel & Wilbur, Sylvia. 1997. “A framework for immersive virtual environments (FIVE):

Speculations on the role of presence in virtual environments” in Presence: Teleoperators & Virtual Environments, nº 6(6): 603-616.

Syrett, Hannah, Calvi, Licia & van Gisbergen, Marnix. 2016, June. “The oculus rift film experience: a case study on understanding films in a head mounted display” in International Conference on Intelligent Technologies for Interactive Entertainment (pp. 197-208). Springer, Cham.

Yee, Nick & Bailenson, Jeremy. 2006. “Walk a mile in digital shoes: The impact of embodied perspective-taking on the reduction of negative stereotyping in immersive virtual environments” in Proceedings of PRESENCE, 24, 26.

Zelcer, Mariano. 2018. “Pantalla, imágenes y cámara en la realidad virtual: una aproximación” in Imagofagia, nº 18: 322-343.