Methodologies and Procedures for the Audiovisual Creation and Direction of Virtual Reality Short Films with Volumetric Capture. An Innovation Project for Higher Education

Introduction

The advent of Virtual Reality (VR) technology has brought about a revolutionary shift in the field of audiovisual creation in recent times. With its immersive experiences that meld fiction and reality, this innovative medium has exceeded the limitations of traditional storytelling. In this context, making and directing virtual reality short films has become a cutting-edge and captivating artistic medium.

VR short films use the power of immersive technology to immerse audiences in carefully constructed virtual environments, allowing them to actively participate in the story that is being told around them. VR short films, as opposed to regular movies, encourage viewers to become more than just passive viewers and to get intimately involved in the narrative. When users put on virtual reality headsets, they are submerged in a 360-degree world in which every look, touch, and movement affects the experience, adding several affordances to the language of immersive narratives.

Visionary filmmakers and artists with a special combination of technical know-how, artistic vision, and narrative inventiveness are in charge of VR short film productions. These pioneers of virtual reality storytelling skillfully negotiate the challenges presented by immersive technology, fusing cutting-edge approaches like spatial audio, interactive components, and dynamic storytelling routes to create engrossing stories that arouse the senses and stimulate the mind. Storytellers are able to push the limits of traditional filmmaking and discover new creative avenues through the VR lens, creating remarkable experiences that spectators remember long after the headset is taken off. VR short films provide an unmatched platform for artistic expression and discovery, ranging from thought-provoking dramas to pulse-pounding thrillers and awe-inspiring adventures.

The field of audiovisual creation and directing of virtual reality short films is at the forefront of innovation in this ever-changing and dynamic landscape, redefining the possibilities of cinematic immersion and influencing the direction of storytelling in the future. Entering the limitless world of virtual reality looks to be an exciting voyage of wonder and discovery as technology develops and artistic boundaries expand.

Brief Historical Approach to VR Filmmaking

From the 1990s, when the concept of VR first gained popularity and traction, to the early 2010s, when significant advances in VR technology were introduced, the production of immersive storytelling witnessed a non-stop evolution, setting VR technology as a viable medium for crafting stories. Early VR short films were experimental in nature, and they still are, trying to showcase the capabilities of the medium rather than focusing on narratives. One of the first examples is Strangers with Patrick Watson (Lajeunesse and Raphaël, 2013), consisting of an immersive music experience that took the audience on stage with Patrick Watson. It demonstrated the novelty of the medium; at this stage, early VR short films featured simplistic narratives and experimental visuals.

During the mid-2010s, VR films started to explore narrative-driven experiences, such as Henry (Oculus Story Studio, 2015), a charming animated short film that took home the Emmy for Best Original Interactive Program and starred Henry the hedgehog. In 2015, the premiere of Clouds over Sidra (2015), one of the most acclaimed 360 VR films, generated much debate regarding the potential of VR as an “empathy machine.” During the late 2010s, there was a ground-breaking impulse for VR animation films, following Henry, as in the cases of Allumette (Penrose Studios, 2016) and Pearl (Osborne, 2016), the first immersive VR film nominated for an Academy Award in 2017. Another stunning VR short film production from this period is Notes on Blindness (Colinart and La Burthe, 2016). Halfway between animation and experimental computer-generated cinema, it provides a visceral and empathetic exploration of blindness and perception. Dear Angelica (Unseld, 2017) is another VR animation that showcases the emotional depth and storytelling potential of VR, immersing viewers in a poignant tale of memory and loss. Filmmakers started to experiment with interactive elements, allowing the viewer to take part in the actions depicted and, in some cases, influencing the narrative through their actions, as in Queerskins: A Love Story (Szylak and Tsiboulski, 2018), where the audience has the chance to interact with several 3D virtual elements that connect with the storyline. In the same year, Wolves in the Walls (Fable Studio, 2018) was released, an interactive VR experience based on a Neil Gaiman book where viewers interact with characters and shape the outcome of the story. In the second chapter, Whispers in the Night (Fable Studio, 2019), they started combining VR with AI to produce a stronger sense of immersion through the NLP capabilities of the main character (Martínez-Cano, 2024), Lucy, with whom the user can talk while Lucy remembers conversations, which makes for a more intimate and personal narrative experience. The third and last chapter of this film series, They Are Everywhere (Fable Studio, 2019), continues the exploration of using AI to strengthen the narrative affordances of VR storytelling. In 2020, Atlas V released Gloomy Eyes (Maldonado and Tereso, 2020), a stunningly illustrated virtual reality experience that uses hand-tracking and spatial audio to tell an engrossing tale in a post-apocalyptic setting. During this period, filmmakers experimented with interactive elements, branching storylines, and multiple endings, providing the audience with a sense of agency within the virtual world. 2019 was also the year of Traveling While Black (Ross Williams), a fully immersive documentary that puts the viewers in the shoes of black travelers throughout history as it investigates the history of African American travel in the United States. This masterpiece of immersive 360-degree documentaries for VR explores the capabilities of editing from an outstanding creative perspective, producing stylistic resources never seen before regarding video overlay, camera movements in the VR environment, and POV.

After the period of experimentation with user interaction run during the late 2010s, VR short films continued to evolve in the 2020s, leveraging the advancements in immersive technologies such as spatial audio, haptic feedback, and photorealistic graphics and volumetrics, which allowed the filmmakers to shoot 3D videos of real actors and implement them in the VR setting. In the early 2020s, The Line (Laganaro, 2020), an embodied VR narrative that utilized cutting-edge technology to create a visually stunning and emotionally evocative narrative experience, won the category “Best Virtual Reality Experience” at the 76th Venice Film Festival. Creative experimentation and technological advancements run together, while authors have maintained in recent years a strong commitment to continue the development of immersive VR audiovisual experiences. Currently, we are witnessing mainstream recognition and expansion with major film festivals such as Sundance, La Biennale di Venezia, and Tribeca featuring dedicated VR storytelling showcases. As a result, the themes and genres explored by VR filmmakers are diversifying, ranging from drama and horror to documentary and experimental art.

Project Proposal and Methodology

Based on the results and approaches of VR filmmakers and their works, it is identified that there is a need to start implementing these cinematographic practices in higher audiovisual communication and film studies so that students and new generations pick up the baton and participate in the evolution of immersive audiovisual narrative experiences and the development of these media languages.

According to Social Learning Theory (Bandura, 1977), the audiovisual as a cultural product brings together a variety of disciplines, such as literature, the performing arts, music, technological progress, design, and the plastic arts, and in the case of immersive virtual reality productions, interactive design, video game strategies, and their languages. The audience is encouraged to engage in varied behaviors by seeing and imitating models. Their production can affect the audience as well as serve as a tool for increasing awareness in the context of higher education. In this regard, we think that integrating students in the creation of immersive prosocial virtual reality audiovisual content can incorporate instruction in social inclusion, equality, and justice values in addition to technical training regarding the creation and use of the technologies for these new immersive media and professional working procedures.

In light of these concepts, it is imperative that students studying the relevant subjects and degrees be properly instructed in the analysis of these productions, as well as in recognizing the potential prosocial effects of the narratives developed for this purpose, as well as the discursive strategies and technical methodologies employed in the development and production of immersive virtual reality productions. In this way, students are inspired to use their professional practice as a means of bringing about change, in addition to gaining and expanding their understanding of these subjects. In this regard, certain professionals in the cultural and media industries—such as Maccoby & Wilson (1957), Wilson & O’Leary (1980), Shapiro & Rucker (2004), and Cohen (2017)—serve as references for prosocial models and professional connections. Through their work, particularly when utilizing emerging technologies, these professionals act as change agents for the betterment of society.

This project aims to involve students in the production of immersive VR audiovisual narratives for pro-social purposes. The aim is to instruct students in the technical process of developing these productions and, at the same time, to promote social values and pro-social behavior. In this regard, an innovative educational project was designed in order to set the first step in connecting students with VR filmmaking practices. Titled Production of Immersive Prosocial Virtual Reality Audiovisual Narratives (PIEU_B/2023_69), it was implemented as a short training activity during the course 2023-2024 at the Audiovisual Communication Undergraduate Degree of Universidad Miguel Hernández, in which 15 students participated. This educational workshop included five in-class sessions combining case studies and creative production techniques such as 360 video, 3D computer graphics, and volumetrics.

This project proposes an active learning approach for students to deepen their knowledge and development of technical skills in immersive audiovisual production of virtual reality through 360 video, volumetric capture, editing, and post-production of 360 video and volumetric video, video game engines, and interactive design, as well as to promote a pro-social impact on the students and on the subsequent audience of these contents.

In order to accomplish the goal, the following assignments are suggested:

• To analyze and determine the tactics, techniques, and creative resources used in the configuration of a number of immersive virtual reality audiovisual experiences.
• Create a proposal for the creation of a fiction-based virtual reality short film with pro-social goals.
• Conduct a series of meetings with focus groups to discuss the project’s various technical and development procedures.
• Use teamwork techniques to help students develop the multidisciplinary and collaborative mindset needed to address today’s challenges.

The benefits of the project lie in the technical training of the students, who will be able to develop audiovisual productions for virtual reality, complementing their professional skills in this field. At the same time, the students will receive training in values through pro-social models and strategies for the creation of audiovisual content capable of having a positive impact on the audience.

Five working sessions were planned as follows, in a preproduction, production, and postproduction scheme for the two final sessions:

• Session 1: Exploring Technical Resources and Hybrid Narratives in VR Short Films: After watching a series of VR short films, a focus group was carried out to identify the technical and creative resources used in the cases of study.
• Session 2: Hands-On Learning 360 Video Recording and Volumetric Capture Techniques: The students learn and experiment with 360-degree video and volumetrics, as well as the technical means for preparing and editing the clips.
• Session 3: Integrating Volumetric Clips and 3D Elements in VR Environments Using Unity: Introduction to VR project settings to create immersive experiences within the Unity video game engine. The students learn how to prepare and implement the contents created in Unity and combine 360 video, volumetric video, and 3D graphics as three layers of virtual imaging composition. Technical procedures to prepare the clips and sound files of 360 videos, volumetrics, and 3D elements inside the Unity project.
• Session 4: Pre-Production and Creative Approaches to VR Filmmaking: The students worked in the preproduction of a VR short film, reviewed the script, and prepared the mise-en-scène.
• Session 5: Recording and Preparing Volumetric Clips for VR Short Film: Shooting the Main Scene of the VR Short Film Proposed Editing of all the different registry layers (360 video, volumetrics, and 3D graphics) to finally produce a VR immersive experience.

Results

During the first session, VR short films were shown (Figure 1), followed by a group session to identify with the students the technical resources used in the different case study productions. Queerskins: A Love Story (Szilak & Tsiboulski, 2018), This is Not a Ceremony (Van Loon, 2022), Traveling While Black (Ross Williams, 2019), Notes on Blindness (Colinart & La Burthe, 2019), Allumette (Penrose Studios, 2016), They Key (Tricart, 2019), and MLK: Now is the Time (Tricart, 2023) were shown, trying to address all types of productions and emphasizing hybrid narratives for VR that use different types of image registration. All these VR experiences are included in the VR for Good catalog, a Meta platform that promotes immersive storytelling focused on social impact and fully utilizes the special potential of virtual reality to produce narratives centered on individuals that encourage empowerment and empathy. Following the focus group during which the technical and language aspects of the VR fiction films viewed were analyzed, we delved into 360 video, volumetric capture, and 3D computer graphics as three of the most commonly used image registration resources and their combined possibilities for the creation of hybrid VR narratives. We delved into the prosocial aims of these productions and the types of narrative strategies they use to promote them. We also reviewed the technical devices that we were going to use in the following sessions and introduced the students to their technical specifications. For watching the VR contents, we used five Meta Quest 2 VR HDMs and one OCULUS Rift S HDM. We also delved into the differences between 3DoF (3 degrees of freedom) and 6DoF (6 degrees of freedom) VR experiences.

In the second session, we began to experiment with the technical aspects of 360 video recording (Figure 2). In this session, we recorded different exteriors with an Insta360 ONE X2 camera and then worked on the editing and exporting modes of the recorded clips using Insta360 Studio software. We then set up the set for volumetric capture using a green chroma background, two omnidirectional microphones on either side of the set with a Zoom H4 recorder, and a Microsoft Azure Kinect sensor connected to the Depthkit volumetric capture recording software. A basic lighting scheme was proposed with two fill lights and two background lights. After configuring the set and establishing the recording settings of the Depthkit volumetric video capture and editing software, we proceeded to record different volumetric video recordings and performances of the students in front of the Azure Kinect sensor, as could be seen in figure 2. After the recordings, the students learned how to edit and export the volumetric clips. Volumetrics were captured from a frontal position with only one sensor (180 degrees), trying different high levels of the sensor to test different ways of mimicking the POV of the audience.

During the third session, we started with the introduction of the Unity game engine and its configuration for the creation of VR experiences. Once the VR project was set up in Unity, the students learned how to configure and install the Depthkit packages for Unity in order to incorporate volumetric clips in the virtual environment. Once all the initial procedures were completed, we started the experimentation with the different logs obtained. First, the import and implementation of 360 video as the outer layer for the creation of the proposed environment were performed. The next step was the import of volumetric video clips and their implementation within the 360 video. Finally, we continued exploring the possible incorporation of graphics and 3D elements created with Unity or imported from three-dimensional modeling software such as Autodesk Maya.

We worked on pre-production for the main scene of The Stigma Machine project’s second episode during the fourth session. The mise-en-scène was taken into consideration, and the script was revised. Different creative approaches were put forth to create a virtual reality short film that constantly appeals to the audience without allowing for direct interaction or agency. This system of subtle interaction was established with the goal of creating an immersive experience where the viewer takes on the role of the main character and places himself in the mise-en-scène. The approach was inspired by other VR short films that use the perspective-taking task as a strategy to instill prosocial values, but in this case from the perspective of a witness. This question poses the initial challenge for the creation of the script and the staging of the action to be filmed.

The scene was recorded with a professional actress and actor in the fifth and final session (Figure 3). The set had the same lighting and audio recording equipment as the second session, along with a Chroma background. After the session, the audio and volumetric clips were ready, and we started creating the 3D scene that will house the volumetric recordings for the upcoming virtual reality short film, which will hopefully last no more than two minutes. Finally, we worked on the editing of all the different registry layers (360 video, volumetrics, and 3D graphics).

Conclusion and Final Reflection

The innovation project for higher education focused on teaching audiovisual direction and production through techniques such as 360 video, volumetric video, and 3D graphics. The results detail the production setup, technical workflow, and procedure for developing low-budget VR short films. Compared to 2D productions, the high production effort for VR films is significantly more complex due to the integration of volumetric aspects into story concepts and the combination of various VR imaging types. Challenges include the perception of performers being “in the same room as the audience, spatial affordances for interaction” without direct agency, and giving sincere, emotionally charged feedback (Bitter, Senk, & Spierling, 2023).

The “Production of Immersive Prosocial Virtual Reality Audiovisual Narratives” initiative has provided university students, particularly in audiovisual communication studies, film direction, and production, with a transformative educational opportunity. Over five intensive sessions, students collaborated with their professor to master VR short film creation, leveraging volumetric capture, 360 video, and 3D graphics. This hands-on approach equipped students with practical skills and a deeper understanding of the narrative and technical complexities of VR filmmaking, fostering empathy, understanding, and social responsibility.

A notable aspect of the project was its emphasis on prosocial elements, using the educational process to produce immersive experiences that promote empathy and social responsibility. Students were inspired to integrate prosocial ideas into their work, creating VR content that encourages viewers to adopt different perspectives. This dual focus on technical proficiency and ethical storytelling not only enriched the students’ educational experience but also highlighted VR’s potential for prosocial impact.

The project’s success underscores the importance of integrating innovative techniques into curricula to prepare future filmmakers for the evolving immersive media landscape. It demonstrates how immersive technology can foster empathy and prosocial behavior in both creators and audiences. This comprehensive approach ensures that graduates emerge as skilled professionals and responsible storytellers, committed to making a positive societal impact. The initiative instills a sense of purpose and accountability in students, paving the way for VR filmmaking to become a catalyst for meaningful discourse and societal development.

The value of multidisciplinary cooperation between filmmakers, visual artists, and virtual reality experts was imparted to the students as they learned how to produce VR short films using volumetric capture. By working together, we can push the frontiers of immersive storytelling and produce more complex, emotionally charged material. Together, these varied professionals can solve technical issues and advance the industry, opening the door for more advancements in virtual reality filmmaking. Future educational projects can benefit from the project’s emphasis on technical proficiency, ethical narrative, and prosocial impact, which prepares emerging creators to explore and develop the potential of immersive media.

Funding

This project was funded by the University Educational Innovation Programme PIEU-UMH 2023 of the Vice-Rectorate for Studies of the Miguel Hernández University of Elche.

Este proyecto ha sido financiado por el Programa de Innovación Educativa Universitaria PIEU-UMH 2023 del Vicerrectorado de Estudios de la Universidad Miguel Hernández de Elche

References

Bandura, A. (1977). Social learning theory (Vol. 1). Prentice Hall: Englewood cliffs.

Bitter, J. L., Senk, G., & Spierling, U. (2023, March). Effects of volumetric video capture on interactive storytelling in immersive augmented reality: a short film experiment. In Proceedings of the 2023 7th International Conference on Virtual and Augmented Reality Simulations (pp. 45-51).

Cohen, J. (2017). School climate, social emotional learning, and other prosocial “camps”: Similarities and a difference. Teachers College Record, 1-8.

Maccoby, E. E., & Wilson, W. C. (1957). Identification and observational learning from films. The Journal of abnormal and social psychology, 55(1), 76.

Martínez-Cano, F. J. (2024). A Look at the Crossroads of Artificial Intelligence, Films, Video Games, and Virtual Reality in R. V. Benítez Rojas and F. J. Martínez-Cano (Eds.), Revolutionizing Communication. The Role of Artificial Intelligence. CRC Press, Routledge - Taylor & Francis Group.

Shapiro, J., & Rucker, L. (2004). The Don Quixote Effect: Why Going to the Movies Can Help Develop Empathy and Altruism in Medical Students and Residents. Families, Systems, & Health, 22(4), 445.

Wilson, G. T., & O’leary, K. D. (1980). Principles of behavior therapy. Prentice-Hall.

Filmography

Colinart, A., & La Burthe, A. (2016). Notes on Blindness [Virtual reality short film]. Arte.

Fable Studio. (2018). Wolves in the Walls [Virtual reality short film].

Fable Studio. (2019a). Whispers in the Night [Virtual reality short film].

Fable Studio. (2019b). They are Everywhere [Virtual reality short film].

Laganaro, R. (2020). The Line [Virtual reality short film]. Arvore Immersive Experiences.

Lajeunesse, F., & Raphaël, P. (2013). Strangers with Patrick Watson [Virtual reality short film]. Felix & Paul Studios.

Maldonado, F., & Tereso, J. (2020). Gloomy Eyes [Virtual reality short film]. 3DAR, Atlas V.

Milk, C. & Arora, G. (2015). Clouds over Sidra [360 video film]. N SDG Action Campaign,UNICEF Jordan, Vrse.

Oculus Story Studio. (2015). Henry [Virtual reality short film].

Osborne, P. (2016). Pearl [Virtual reality short film]. Google Spotlight Stories.

Penrose Studios. (2016). Allumette [Virtual reality short film].

Ross Williams, R. (2019). Traveling While Black [Virtual reality short film]. Felix & Paul Studios.

Spielberg, S. (Producer), & Tricart, C. (Director). (2023). MLK: Now is the Time [Virtual reality short film]. Flight School & TIME Studios.

Szylak, I., & Tsiboulski, J. (2018). Queerskins: A Love Story [Virtual reality short film]. Cloudred.

Tricart, C. (Director). (2019). The Key [Virtual reality short film]. Lucid Dreams & Oculus VR for Good.

Unseld, S. (2017). Dear Angelica [Virtual reality short film]. Oculus Story Studio.

Van Loon, C. (2022). This Is Not a Ceremony [Virtual reality short film]. National Film Board of Canada.