Simulate This! Virtual Production and Cinematic Education with cineDESK

1 Introduction

Virtual Production (VP) marks the gradual culmination of the profound digital revolution that has swept cinema since the advent of computer-generated visual effects in the 1980s (Prince 1996, 27). VP allows film scenes to be shot in virtual environments without the need for elaborate post-production processes. Visual effects have thus moved into the realm of live production and can be captured directly by the camera when displayed on a LED wall in a VP studio. This takes hybrid image production, which Lev Manovich has described as a constituent element of Digital Cinema (Manovich 2001, 302), and in which live-action footage is only one of several image sources, to a new real-time level. British production designer Alex McDowell compares this apparent revolution to the transition to talkies in the 1930s (Halligan 2023, 138). The fact that hardly any distinction is made between virtual and real image elements can also be understood as a defining component of the term Post-Cinema (cf. Denson and Leyda 2016, 6; Chateu and Moure 2020, 14).

In addition to shooting virtual content on a real set, processes such as location scouting in virtual reality environments as well as the previsualization of film scenes with hybrid image components are also referred to and summarized as VP (Ilmaranta 2020, 326).

A core component of VP are game engines. Initially conceived as collections of pre-built program code modules to produce behaviors within a game environment, these tools have grown so powerful and reliable in rendering complex graphics in real time that they have found their way into entertainment content creation. The two most regularly used game engines are Unreal Engine ¹ and Unity ² (Frank 2022, 43).

Today, the best-known VP procedure is to combine a physical foreground set that live actors can interact with (e.g. sitting on chairs, touching objects etc.) with a virtual background set that appears on a semi-curved array of high-quality LED monitors, the so called LED volume (Frank 2022, 87). The camera on set is tracked and thus synchronized with the virtual camera within the 3D space provided by a game engine. Using the chosen lens parameters and the movement of the physical camera as inputs, the game engine renders the digital assets in real time and displays them in the LED volume. Unlike traditional 2D background replacement techniques, the physical tracking of the on-set camera allows for a correct rendering of parallax movement within the background image. Thus, digital elements become a seamlessly integrated component of the physical studio space.

A recent comparative study shows that audiences tend not to notice any difference between identical films that were shot on a real location and in a VP environment (Iseli et al. 2022). VP also marks a shift from linear to non-linear production workflows as it blurs the boundaries of the traditional sequence of preproduction, production and postproduction (Ilmaranta 2020, 321)

Here, we explore an aspect of VP known as real-time previsualization in the context of film school curricula. We present a novel method of simulation training with cineDESK ³, a previsualization tool that has been developed at the Zurich University of the Arts.

2 Features of cineDESK

cineDESK facilitates the simulation of film scenes prior to shooting. Film crews can collaboratively explore how space, light, props and acting are translated into cinematic sequences.

In film education, cineDESK allows us to teach the visual craft of directing a film without the cognitive overload of a real film shoot. Due to its intuitively structured user interface, no in-depth knowledge of 3D software is required to operate cineDESK.

cineDESK uses a large touchscreen as its user interface. The screen is laid out like a table around which the film crew members are gathered. It displays the top view of a virtual location that the filmmakers have either previously scanned or generated using 3D software. With their fingertips, they can move and animate their virtual characters as well as the virtual cameras. A separate monitor displays the output of the active virtual camera (Image 1).

Origin

cineDESK is the result of a collaboration between the Zurich University of the Arts (ZHdK), Mirko Lempert (Stockholm University of the Arts) and Simon Andersson (KTH Royal Institute of Technology) initiated in 2018.

Lempert and Andersson developed the prototype of a collaborative previsualization system, the so-called Previs Table (Larrson 2016) based on the Unity game engine. Using a motion capture system and a ceiling mounted projector, the system visualized the top view of a film location modeled in 3D on a white table. The virtual characters as well as the virtual cameras were represented in the real word by small objects on the table equipped with tracking markers. A second screen showed the real-time rendered image of the virtual camera, depending on its position on the table.

In 2020, researchers and developers Valentin Huber and Norbert Kottmann from ZHdK initiated the cineDESK project. Inspired by the idea of the Previs Table as a collaborative previsualization tool, they developed new software from scratch.

Overview

cineDESK is based on the Unreal Engine, a game engine with a very large feature set for cinematic applications. As a simplified and streamlined user interface for the Unreal Editor, cineDESK causes any action on its touchscreen to trigger an equivalent action in the Unreal Editor and vice versa.

The cineDESK application⁴ is designed as a plug-in and can be added to any existing Unreal project. The development of cineDESK is ongoing. The source code, the plug-in package and a detailed documentation are available for collaborators on GitHub⁵.

Assets

The standard ingredients of a cineDESK project are the 3D assets of a location, props, characters, lights and cameras. These assets must be prepared before the start of a previsualization session and require knowledge of both 3D content creation and the Unreal Engine.

• Location & Props: 3D models of a location and props can be created from scratch using 3D software, or generated by scanning physical locations and objects with photogrammetry⁶ and LiDAR⁷-based techniques. Alternatively, 3D models from asset libraries or marketplaces can be used.
• Characters: Digital characters can be modelled and rigged (prepared for animation) from scratch. This process is very time-consuming. Alternatively, existing characters from online marketplaces can be used or created using the cloud-based Unreal MetaHuman Creator ⁸.
• Cameras: Virtual cameras are part of the Unreal Engine toolset. They simulate real world cameras and lens parameters such as sensor size, focal length and aperture.
• Lights: Lights are part of the Unreal Engine toolset. There are several types of lights with different characteristics: directional, spot, point and rectangular.

These assets are combined into a level that provides the environment for a previsualization session.

Workflow

After the preparation of a level, scene work with cineDESK takes place in three different modes: scene blocking, staging and editing. A timeline allows you to go back and forth in the recorded scene and thus review and refine your work in an iterative process.

• Blocking: The virtual actors are animated using standard animation loops like walking, running, sitting down or standing up. Once the animation is defined, character movement can be recorded by touchscreen input. Advanced features of cineDESK allow blocking with real actors, thus allowing for more idiosyncratic and sophisticated character animation. These advanced features require motion capture as well as face capture systems. Animation and dialogue can either be recorded in advance or performed and recorded directly on the cineDESK (Image 2).
• Staging: Here, the virtual camera is animated and recorded, while the characters’ movements are played back. Camera parameters (angle, framing, focal length, aperture, focus etc.) are defined (Image 3). Instead of controlling the virtual camera on the touchscreen, it can be operated more intuitively with the help of a position-tracked tablet, as if it were a physical, hand-held device (Image 4).
• Editing: A basic editing feature allows users to create a linear cut of their staging. Users can thus review their shot breakdown and the timing of the scene instantly.

Testing

The development of cineDESK benefitted from continuous testing in practical scenarios within the BA and MA film program at ZHdK, where cineDESK workshops are now an integral part of the curriculum. Workshop formats include staging and lighting courses as well as students using cineDESK to previsualize their film projects.

3 Teaching Visual Language

Helping young directors and cinematographers to develop an idiosyncratic visual style is a core part of film school curricula. Traditionally, cinematic visual language is taught in two modalities:

• Setting #1: A rule-based approach in a classroom setting, using whiteboards, diagrams, slides and examples from existing films.
• Setting #2: A trial-and-error-based approach, where students shoot their own films.

Setting #1 teaches visual language as abstract principles, requiring students to memorize rules (Image 5). But when these rules are later applied in setting #2, students are so focused on coping with the logistical and psychological complexities of a film shoot that a conscious and reflected exploration of visual principles becomes all but impossible. Thus, the classroom reflection on their finished movies often becomes more of a debriefing about interpersonal and logistical issues than a discussion about the first principles of visual storytelling.

With cineDESK, we have developed a workshop framework to overcome these limitations. Rather than shooting a film, we simulate a film shoot. And rather than following the traditional linear (and irreversible) sequence of production, we create a cyclical pipeline where shot design, execution and editing are iteratively repeated. Thus, we provide a non-linear, fast and reversible learning environment. Furthermore, cineDESK allows us to address specific components of cinematic production in isolation, thus reducing the complexity of the learning environment and providing focused guidance for specific tasks.

Workshop design

For our simulation workshops, we present a pre-recorded, motion-captured acting scene (Image 6) to our students in an abstract empty space (Image 7). The students work in teams of two as directors and cinematographers. Each team chooses a location for the scene from a pre-defined list and configures the appearance of the actors to their own liking using the MetaHuman Creator tool.

Once all elements are in place, the students analyze the dramaturgy of the given scene on paper. Next, they create a shotlist based on their analysis and artistic intention. This shotlist is then transferred to cineDESK. Now, the virtual shooting process begins. Shot timing, lens choices, framing and movement are continuously discussed with the tutors, allowing for detailed and timely feedback, to a degree that would not be possible on a real film set.

Once all shots are recorded, they are immediately cut and screened. The group analyzes this draft cut, zooms in on the problems and reshoots. This process is repeated until a satisfactory result is reached. At the end of the workshop, all versions of the scene are screened and discussed (Image 8).

Lessons learned

After each workshop, debriefings with both students and teaching staff are conducted. The lessons learned are:

• Isolation: On a real film production, the creative sub-tasks of casting, designing, framing, staging, lighting and editing are inseparably intertwined. With cineDESK, we can isolate an individual sub-task for a more thorough analysis.
• Preparation: In order to prevent the students from being overwhelmed by the sheer abundance of creative possibilities cineDESK provides, a highly structured conceptual preparation phase in traditional «pen and paper» mode is necessary.
• Learning Curve: Depending on their technical qualifications, some students have a slightly steeper learning curve to master cineDESK’s user interface. Workshop schedules have to be adjusted accordingly.
• Interaction Quality: On a real film shoot, critical interventions from the teaching staff are often detrimental to the bond between the director and their crew because the authority of the director on set is tarnished. With cineDESK, no crew is present, so the input of and discussions with the teachers appear less threatening and more constructive to the students.
• Turnover: For a real film shoot exercise, the process from script to finished movie scene usually takes between two and three weeks. With cineDESK, the same work takes just three days. This rapid prototyping approach is ideally suited to teaching visual principles.
• Comparable Results: Because each student team worked with the exact same actor’s motion and voice data, we can compare the creative results and provide a thorough analysis of the artistic choices made and their dramaturgical consequences. The resulting discussions and exchanges between students and staff in class are often flagged by the students to be one of the most unique and enlightening features of the workshop design.

4 Simulation Training and its Effects

To our knowledge, this simulation approach to teaching visual language is a novel method for film school curricula. However, tech-based simulation trainings have been widely known for more than a century in other fields. Flight simulators and the corresponding training appeared as early as 1910 and became more sophisticated and standardized during the 1930s (Ören et al. 2023, 123). Simulation training also has a long tradition in space technology, health care and emergency management. As mirrored in publications, simulation training has recently grown more important in science, technology, engineering and mathematics (STEM), as well as in practical professions and management training (Grossman et al. 2009, 2056); (Belland et al. 2017, 310); (Chernikova et al. 2020, 500).

The professional training described in these studies mostly aims at gaining or enhancing motor skills, handling specific equipment, decision-making, improving communication, as well as operational skills (Ören et al. 2023, 122). In the context of primary and secondary education, a meta-study shows that computer-based simulation training can support students in acquiring basic skills in the STEM domain (D’Angelo et al. 2014, 39). Important factors are not only the use of simulation training itself, but also additional support and coaching for the trainees. A comprehensive meta-study by Chernikova et al. (2020, 499) shows a large overall positive effect of simulation in higher education and professional training. The findings specifically demonstrate that simulations are among the most effective means to facilitate the learning of complex skills across domains.

Simulation teaching environments are often accompanied with scaffolding methods that can facilitate simulation-based learning during different phases of the development of knowledge and skills. Scaffolding in teaching refers to a variety of instructional techniques used to support students’ learning and help them achieve their goals. It involves providing temporary guidance, assistance, or structure to help learners develop new skills or understanding, which they can then apply independently.

Some key features of scaffolding in teaching include:

• Breaking down complex tasks: Teachers break down a complex task into smaller, manageable steps, allowing students to focus on one aspect at a time (Belland 2017, 33).
• Modeling: Teachers demonstrate the process, skill, or concept they want students to learn, providing a clear example for students to follow (Belland 2017, 109).
• Asking guiding questions: Teachers ask open-ended questions that encourage students to think critically and reflect on their learning.
• Providing support and feedback: Teachers regularly monitor students’ progress and provide timely feedback, encouragement, and assistance as needed. (Belland 2017, 22).
• Gradual release of responsibility: As students gain competence and confidence, teachers gradually reduce their support, allowing students to take on more responsibility for their own learning. (Belland 2017, 21).
• Adjusting support based on individual needs: Teachers tailor their scaffolding strategies to meet the diverse needs of their students, providing more support for those who need it and less for those who are already proficient. (Belland 2017, 5-6).

Overall, scaffolding in teaching helps students build a strong foundation for learning, promotes independent problem-solving, and supports the development of higher-order thinking skills.

cineDESK simulations in context

In simulation trainings with the cineDESK, the primary goal is always to find unique artistic solutions. In contrast to most other practical fields, there is no direct link to correct or incorrect outcomes or behaviors and no specific bodily or motor skills that need to be acquired on an expert level (except in the field of camera operating). Nevertheless, many positive effects of simulation trainings also apply to filmmaking:

• Problem solving: Chernikova et al. (2020, 501) point out that – in order to be able to make professional decisions and implement solutions – critical thinking and problem solving are among the most relevant skills that students should acquire in addition to domain-specific knowledge and skills.
• Coping with complexity: Grossman et al. (2009, 23) demonstrate how approximations of practice in which complexity is significantly reduced help learners to avoid confusion and inefficiency.
• Gaining confidence: Alinier and Oriot (2022, 4), Husebø et al. (2018, 5), Cant and Cooper (2010, 3) as well as Lateef (2010, 352) all describe the effect that simulation training leads to a significant increase in the trainees’ confidence.
• Teamworking: Ören et al. (2023, 135) as well as Husebø et al. (2018, 5) and Lateef (2010, 351) point out that communication and collaboration are the key aspects of teamwork when multiple professionals need to solve problems, make decisions and take action. However, in their meta-analysis, Chernikova et al. (2020, 523) found only a medium positive effect of simulations on teamworking skills. They conclude that this might be due to the high complexity of live simulations which were taken into account.

All four listed skills are highly relevant for filmmaking students when confronted with the multi-layered task of directing and finding a genuinely individual approach to visual language. Thus, the cineDESK simulation workshop formats encourage development of these skills and allow students to continuously gain more confidence in their own artistic expression.

5 Results and Strategic Objectives

In October 2022, a hands-on demonstration of the cineDESK was presented at the CILECT conference in San Sebastian (Hendry and Kottmann, 2022) to gauge interest from other film schools for simulation-based training. Critical issues with traditional visual language teaching methodologies were confirmed by the peer group and a shared understanding of the potential of VP for cinematic teaching emerged.

In February 2023, a three-day introductory cineDESK workshop for international film lecturers and film school administrators was conducted in Zurich. From this workshop, the following observations and ideas emerged:

• Haptic Controllers: Instead of using the touchscreen to input camera movements, many teachers used the tablet to replicate physical camera movements in space (cf. Image 4). This embodied practice is much more lifelike and should be further extended by using physical focus pull devices and maybe even dolly tracks with a dummy camera. Thus, the system could also be used for graduate-level camera workshops.
• Physical constraints: Currently, the virtual camera can move through walls and has no physical dimensions that constrain its movement. Here, a more realistic physical simulation could further the real-world component of our teaching. Adding grip equipment like dolly tracks or cranes inside the cineDESK could extend the simulation as well.
• Multi-camera work: Our colleagues from Hong Kong mentioned that they always work with multi-camera setups for feature film productions. The related issue of invisible camera placement could be tackled on the cineDESK by making the camera bodies visible.

After the workshop, the cineDESK software was made available to selected partners, and a network of film schools is currently being established to work with and further develop the cineDESK. The goal is to create a shared library of assets (e.g. motion-captured scenes, scanned locations) and further evolve the software collaboratively in an open-source format. Thus, we can use cineDESK as a platform to foster intercultural exchange between different national film traditions and create shared telepresence workshop formats across geographical boundaries.

6 Conclusion

In today’s film industry, VP is a complex and particularly dynamic sector, incorporating not only the on-camera recording of virtual content in real environments, but also the previsualization of entire feature film productions. In this production context as well as in the field of education, cineDESK allows us to simplify the complex and often overwhelming task of finding a unique visual language by means of simulation. Specific creative questions can be addressed in a targeted manner and in multiple iterations. These reproducible and perfectly comparable simulations allow artistic decision-making processes to be more consciously perceived and reflected.

Considering international studies about the effects of simulation training in various professional fields, overarching effects that apply equally to the field of cinematic education become apparent – notably, an increased ability to solve problems and to cope with complexity. Furthermore, the effect of elevated self-confidence provided by simulation training is crucial for young filmmakers, as is the increased competence in teamwork.

Given these results, it is essential to give students hands-on experience with VP-related tools. cineDESK provides both an opportunity for the introduction of VP techniques and a platform to develop VP-based teaching formats that go beyond the traditional film school curriculum.

CRediT author statement ⁹

• Manuel Flurin Hendry: Methodology, Investigation, Writing - Original Draft, Writing - Review & Editing, Visualization.
• Norbert Kottmann: Conceptualization, Investigation, Software, Writing - Original Draft, Writing - Review & Editing, Visualization.
• Valentin Karl Huber: Conceptualization, Investigation, Software.
• Michael Schaerer: Methodology, Investigation.
• Christian Iseli: Writing - Original Draft, Writing - Review & Editing, Supervision, Project administration, Funding acquisition.

Acknowledgements

The authors would like to thank Chantal Haunreiter, Deputy Head of the film program at ZHdK, and our film students and workshop participants for their contributions to cineDESK. The February 2023 cineDESK workshop was conducted at ZHdK’s Immersive Arts Space with support from GEECT and contributions by Stella Speziali, Oliver Sahli, Corinne Soland, Kristina Jungic and Christopher Salter.

Final notes

¹Unreal Engine is a real-time 3D creation tool for games, photoreal visuals used in film and immersive experiences. It is developed by Epic Games (https://www.unrealengine.com).

²Unity is a real-time development platform for creating 2D and 3D multi-platform games and interactive experiences (https://www.unity.com).

³cineDESK is a previsualization software developed by the Zurich University of the Arts (https://www.cinedesk.ch).

⁴cineDESK must run in «Play in Editor» mode and cannot be packaged to a standalone executable application.

⁵GitHub is an online version control service. It is commonly used to host open-source software development projects. The cineDESK source code and documentation is available for collaborators (https://github.com/immersive-arts/cineDESK).

⁶Photogrammetry is a technique to generate 3D models with texture by taking a series of photos of an object, structure or landscape from different angles. These are then processed in using software.

⁷LiDAR is an acronym for Light Detection and Ranging and stands for a laser scanning process.

⁸MetaHuman is a product developed by Unreal Engine that allows users to create high-fidelity digital humans either by modifying predefined templates or creating own characters using photogrammetry. They come with a full facial and body rig (skeleton) and are ready to animate in the Unreal Engine (https://metahuman.unrealengine.com).

⁹(Allen et. al. 2019)

Bibliography

Alinier, Guillaume, and Denis Oriot. 2022. “Simulation-based education: deceiving learners with good intent.” Advances in Simulation 7, no. 1 (December): 8.

Allen, Liz, Alison O’Connell, and Veronique Kiermer. “How Can We Ensure Visibility and Diversity in Research Contributions? How the Contributor Role Taxonomy (CRediT) Is Helping the Shift from Authorship to Contributorship.” Learned Publishing 32, no. 1 (January 2019): 71–74.

Belland, Brian R. Instructional Scaffolding in STEM Education: Strategies and Efficacy Evidence. Springer Nature, 2017. https://doi.org/10.1007/978-3-319-02565-0.

Belland, Brian R., Andrew E. Walker, Nam Ju Kim, and Mason Lefler. 2017. “Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education: A Meta-Analysis.” Publisher: American Educational Research Association, Review of Educational Research 87, no. 2 (April 1, 2017): 309–344.

Beller, Hans. 2009. Handbuch der Filmmontage: Praxis und Prinzipien des Filmschnitts. 5., gegenüber der 4. unveränderte Auflage. Praxis Film 36. Konstanz: UVK Verlagsgesellschaft.

Cant, Robyn P., and Simon J. Cooper. 2010. “Simulation-based learning in nurse education: systematic review.” _Eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-648.2009.05240.x, Journal of Advanced Nursing 66 (1): 3–15.

Chateu, Dominique, and José Moure. Post-Cinema. Cinema in the Post-art Era. Amsterdam University Press, 2020.

Chernikova, Olga, Nicole Heitzmann, Matthias Stadler, Doris Holzberger, Tina Seidel, and Frank Fischer. 2020. “Simulation-Based Learning in Higher Education: A Meta-Analysis.” Review of Educational Research 90, no. 4 (August): 499–541.

D’Angelo, Cynthia, Daisy Rutstein, Christopher Harris, R Bernard, E Borokhovski, and G Haertel. 2014. Simulations for STEM learning: Systematic review and meta-analysis. SRI International.

Frank, Laura. 2022. Real-Time Video Content for Virtual Production & Live Entertainment: A Learning Roadmap for an Evolving Practice. 1st ed. New York: Focal Press, December 2, 2022.

Grossman, Pam, Christa Compton, Danielle Igra, Matthew Ronfeldt, Emily Shahan, and Peter W. Williamson. 2009. “Teaching Practice: A Cross-Professional Perspective.” Teachers College Record: The Voice of Scholarship in Education 111, no. 9 (September): 2055–2100.

Halligan, Fionnuala. FilmCraft: Production Design. Focal Press, 2013.

Hendry, Manuel Flurin, and Kottmann, Norbert. 2022. “cineDESK: A Simulation Enviroment for Virtual Filmmaking”. Presented at the CILECT Congress 2022, San Sebastián (Spain), October 11. 2022 doi:10.5281/zenodo.7258008.

Husebø, Sissel Eikeland, Minna Silvennoinen, Eerika Rosqvist, and Italo Masiello. 2018. “Status of Nordic research on simulation-based learning in healthcare: an integrative review.” Advances in Simulation 3, no. 1 (December): 12.

Ilmaranta, Katriina. 2020. “Cinematic Space in Virtual Production.” In Augmented Reality, Virtual Reality, and Computer Graphics, ed. by Lucio Tommaso De Paolis and Patrick Bourdot, 321–332. Lecture Notes in Computer Science. Cham: Springer International Publishing.

Iseli, Christian, Thomas Gerber, Valentin Huber, Michael Schaerer, and Miriam Laura Loertscher. 2022. “Virtually Real: Aesthetics and Perception of Virtual Spaces in Film.” Interactive Film and Media Journal 2, no. 1 (January 30, 2022): 92–105.

Larsson, Peter. 2016. “Forskarna som siktar mot Hollywood.” KTH, https://www.kth.se/om/nyheter/centrala-nyheter/forskarna-som-siktar-mot-hollywood-1.624753. Last access on 08/05/2023.

Lateef, Fatimah. 2010. “Simulation-based learning: Just like the real thing.” Journal of Emergencies, Trauma, and Shock 3 (4): 348.

Manovich, Lev. Language of New Media. MIT Press, 2001.

Ören, Tuncer, Umut Durak, Ernest H. Page, Andreas Tolk, and Saikou Y. Diallo. 2023. “Simulation as Experience to Enhance Three Types of Skills.” In Body of Knowledge for Modeling and Simulation: A Handbook by the Society for Modeling and Simulation International, ed. by Tuncer Ören, Bernard P. Zeigler, and Andreas Tolk, 121–139. Simulation Foundations, Methods and Applications. Cham: Springer International Publishing.

Prince, Stephen. “True Lies: Perceptual Realism, Digital Images, and Film Theory.” Film Quarterly 49, no. 3 (April 1, 1996): 27–37.