Abstract

The traditional media landscape is in the middle of a monumental shift: the new prosumer profile, the need for faster and more efficient communication, and the search for more user-driven and accessible multimedia experiences. New technologies (and more specifically, immersive environments) can provide great opportunities in the entertainment sector, and also in communication, learning, arts and culture. These technologies are gaining popularity due to the COVID-19 crisis as they enable interactive, hyper-personalised and engaging experiences anytime and anywhere. The EU-funded projects TRACTION (870610) and MEDIAVERSE (957252) are embracing new technologies in order to establish an effective participatory production workflow and are exploring novel audio-visual art representation formats. TRACTION will provide a bridge between opera professionals and specific communities at risk of exclusion based on trials, understood as experimental attempts at fostering an effective community dialogue between diverse individuals at risk of exclusion in three different situations, across three countries: Ireland, Portugal and Spain. MEDIAVERSE will enable the creation of a decentralised network of intelligent, automated, and accessible services, tools, and authoring platforms for digital asset management; legal and monetisable discovery and distribution of verified content, and barrier-free usage and integration in target media and platforms.

Keywords: Culture, New technologies, Immersive environments, Media accessibility, Co-creation

Introduction

The media landscape across Europe has experienced a drastic change in 2020 as a result of the COVID-19 crisis. The pandemic has meant a devaluation of the physical format and the focus has shifted towards the highly competitive area of the digital world. Most of the large-scale events such as award ceremonies, conventions, conferences or trade shows have been cancelled, and face-to-face meetings have become a thing of the past. This situation has also brought many unexpected changes to the work environment, with the majority of communication shifting toward the virtual realm.

During this time of social distancing, technology has become the tool that allows interaction between friends, relatives and colleagues. In addition, both companies and public organisations are trying to reinvent their approach in terms of delivering services and connecting with their customers. According to a survey conducted by Kaur and Sahdev (2020, 400), 74% of respondents accepted the fact that their life has become more dependent on digital and social media after the outbreak of COVID-19. People confined to their flats and houses have been forced to adopt new behaviours. Such a change has created renewed opportunities in digital technologies, e.g., distance education, e-gaming, internet streaming, cashless payment, e-commerce and home delivery (Brem, Viardot and Nylund 2020).

While companies are exploring new ways to engage with audiences, 360º videos (aka VR360 videos) have become a simple, cheap and effective way to provide Virtual Reality (VR) experiences (Montagud et al. 2020). The potential of VR360 videos has led to the development of a wide variety of players for all platforms and devices (Papachristos et al. 2017), like computers, smartphones and Head Mounted Displays (HMDs). The HMD is a type of display device or monitor worn over the head and allows the total immersion of the user in whatever experience the display is intended for. The 360º environment accessed when wearing the HMD may be an animation (such as a video game or an animated movie) or live-action (such as a movie or a documentary). Depending on the type of media, the content will either be installed on a PC, on the HMD itself or stored on the cloud.

VR can provide great opportunities, not only in the entertainment sector, but also in communication, learning, arts and culture. In the case of the latter, some of the most recent innovations are virtual events and meet-ups and virtual tours. During a time when nearly every live conference and event has been cancelled, some have shifted to a virtual environment. Virtual events and meet-ups are stepping beyond webinars on the flat screen, taking viewers to a 3D world where interactive avatars provide quite an accurate feeling of presence (Jauhiainen 2020). In the case of tourism, the COVID-19 crisis could be an opportunity to reform this sector to be more sustainable (Ioannides and Gyimóthy 2020), more dependent on the use of technology (e.g., mobility tracking technologies, VR applications, virtual tours and digital identity controls in airports among others), through the development of innovative and digitalised tourism experiences (UNWTO 2020). Hence, telepresence has also been adopted as a safer solution to travel. The main example are virtual tours, which offer the possibility to visit different places without having to physically travel there (El-Said and Aziz 2021).

This paper is structured as follows. The first section offers a general overview of immersive content, focusing on AR, MR, VR and 360º video. The second section deals with the relationship between culture and new technologies (immersive environments) and presents TRACTION and MEDIAVERSE, two H2020 European projects aiming to establish an effective participatory production workflow and exploring novel audio-visual art representation formats. Finally, the conclusion section discusses existing limitations and possible solutions to be addressed during the following years until both projects finish.

Is Virtual the new Reality? AR, MR, VR and 360 videos

Extended reality (XR) is generally used as an umbrella term referring to the hardware, software, methods, and experience that make mixed reality (MR), augmented reality (AR) and VR a reality. The main goal of any immersive content is to make people believe that they are “physically there.” This feeling is called “presence”, and refers to tricking at least three of the five senses of your brain into believing that you’ve been taken to another place. The more immersed a user is in an environment, the higher the level of presence they feel. Still, immersive videos are limited in nature as they can make people motion or simulator sick, be unrealistic, and can more generally be a suboptimal experience that turns people off VR as a medium (Smith 2015).

AR is an experience that uses a camera to change or enhance something in the user’s real world. The properties of AR are that it “combines real and virtual objects in a real environment; runs interactively, and in real time; and registers (aligns) real and virtual objects with each other” (Azuma et al. 2001, 34). This experience can be app- or web-based, though app-based experiences are more common today. Although the term includes the word “reality”, the experience does not need to look realistic.

According to Milgram and Kishino (1994, 1321), MR visual displays “involve the merging of real and virtual worlds somewhere along the ‘virtuality continuum’ which connects completely real environments to completely virtual ones”. This technology allows the viewer to see the real world, where virtual objects are incorporated seamlessly into that environment. These virtual objects are anchored to the real world and thus, the objects change shape according to the user’s perspective. For example, objects will get bigger as a user approaches them. This differs from AR where virtual objects do not change shape as a user moves.

VR is an experience made to be significantly more immersive than standard video assets. VR renders a world in which, essentially, the consumer operates as a natural extension of the creator’s environment, moving beyond 360º video by enabling the viewer to explore and/or manipulate a malleable space. In 360º video, the consumer is a passenger in the storyteller’s world; in VR, the consumer takes the wheel. The storyteller directs the viewer’s gaze through this situational content by using elemental cues, such as light, sound and stage movement. The traditional notion of the fourth wall has been eliminated.

360º videos are video recordings where a view in every direction is recorded at the same time, shot using an omnidirectional camera or a collection of cameras. During playback, the viewer has control of the viewing like a panorama. VR360 videos can allow for “highly immersive experiences that activate a sense of presence that engages the user and allows them to focus on the video’s content by making the user feel as if he or she is physically a part of the environment” (Rupp et al. 2016).

Not only technology is embracing the virtual world, in fact, about 75% of the Forbes World’s Most Valuable Brands have created some form of VR or AR experience for customers. According to research by the International Data Corporation (IDC), worldwide spending on AR and VR is forecast to accelerate out of the pandemic, growing from just over $12.0 billion in 2020 to $72.8 billion in 2024. This study focuses on face-to-face interactions in and out of the physical workplace and highlights the adoption of AR/VR to create personalised and immersive customer experiences. In the case of 360º video, the trend of watching 360º content within web browsers, tablets or mobiles has also increased. Even though PCs, tablets and mobiles are still currently the main devices for watching 360º video, the usage of VR Headsets for watching 360º is starting to grow as well. Three main factors could explain why the 360º video market is thus far considered developed: 360 video capture devices are more sophisticated and affordable, the increasing number of web and mobile players (YouTube, Facebook, Twitter and the use of mobile phones as HMD), and the slightly decreasing price of HDMs.

VR360 video is now being used by all kinds of people and organisations to share immersive stories and extreme experiences in stunning locations. Traditional paper format publications have now become immersive experiences such as the Immersive Anatomy Atlas (Gloy et al. 2021) for students to explore human anatomical structures interactively through virtual simulated dissection. After studying 32 participants the results showed an improvement of learning efficiency and memory retention concluding that active learning seems to be an effective alternative to classical teaching. Other industrial sectors from estate agents, airlines and the hospitality industry are embracing VR360 video to show off their goods (Rizzo et al. 2008; Fida et al. 2018; Parham et al. 2019); while broadcasters (Zheng 2020), educators (Molly et al. 2021) and social media platforms (Ruofei, Li and Varshney 2019) are also experimenting with the format.

Regarding the entertainment industry, there are some indicators which show that VR is also penetrating in fiction genres. In 2017, the immersive short film Pearl was nominated for the Oscars, in the category for Best Animated Short Film. Coco, the animated film by Pixar, also developed and brought a VR and interactive experience to market. However, most fictional immersive experiences are computer-generated and not images recorded with 360º cameras. According to the BBC, “truly interactive VR video is in its infancy and can be expensive to create, but total or partial animation or CGI can be used very effectively and efficiently, while other production techniques may yet emerge or become more accessible over time” (Conroy 2017). Hence, immersive technologies still need to be developed to create successful fictional films or movies. A reason could be the lack of knowledge in delivering well-written immersive stories. So far, storytelling in immersive environments has taken stock of regular 2D productions, without taking into account that the change of medium also affects the plot. Since time is no longer linear, immersive environments give viewers a leading role that is lacking in 2D productions. It is in this context of development that the integration of accessible services in the production line such as subtiling or audiodescription should be researched, always adopting a user-centred approach. It is crucial to hear the users’ voices to understand their needs before this new technology is fully implemented.

Technology meets culture

The rapid diffusion of the new communication technology in the globalised context of the West, linked to the crisis derived from the COVID-19 has provoked a drastic change in the way we create and consume culture. How to think about the future of the European culture and how to consciously intervene in the current situation are main issues for philosophers, sociologists, and psychologists in all countries. The COVID-19 pandemic has been a brutal reminder of the ongoing exclusion of persons with disabilities from society and their amplified and disproportionate exposure to risk during crisis situations. But it has also provided opportunities for public institutions and governments to learn, giving clear focus to their obligations to remove these inequalities.

A good example is that of opera, which uses all the visual and performing arts to create extraordinary experiences of passion, ideas and sensibility. It is rightly recognized as a great achievement of European culture and yet a form that has inspired social and artistic change is now often seen as the staid preserve of the elite. Within this context, the three-year research project TRACTION funded by the European Commission through the H2020 programme, aims to reveal opera as a path for social and cultural inclusion, and a factor in positive social change. It does not aim to make opera palatable to those who don’t attend but aims to find new ways in which marginalized groups (people living in poverty, migrants, young offenders, people with disabilities and others) can co-create opera performances with professional artists to tell stories that matter to them, and reconnect the form with its potential for social change.

By combining best practice in participatory art with digital technology’s innovations of language, form and process; TRACTION explores new approaches to co-creation, and innovates in three fields: 1) Opera creation and production; 2) Immersive and interactive digital media; and 3) Social integration and community development.

Experimental trials in inner-city Barcelona (LICEU), a youth prison in Portugal (SAMP) and three rural and peripheral communities in Ireland (INO) are testing new ideas in partnership with the other members of the TRACTION consortium: Vicomtech, CWI, Virtual Reality Ireland, Dublin City University, Universitat Autònoma de Barcelona and François Matarasso. Of the three trials, the one performed by INO in Ireland involves XR. INO will test the potential of VR technologies in the creation and presentation of opera. Ireland only has full-scale opera venues in Dublin, Cork and Wexford (and Belfast in the North). Although INO’s predecessor, Opera Theatre Company, performed in more than 100 places on the island, it often presented reduced renditions. Radio, TV and the internet give some access to opera, but also offer partial experiences. Now, VR technology has the potential to offer immersive experiences which, while not the same as live performance, still have the emotional power that makes opera what it is. The INO trial is testing whether VR could be a way to create intense opera experiences in a contemporary idiom, freed from the technical requirements and costs of traditional opera production.

The INO trial is artistically innovative, since there have been few attempts to create opera in virtual reality. There will be no physical, live performance. The work will not be a recording of a traditional opera: it will exist only in digital form. The extent of its technological innovation will depend on the choices made as the trial develops. The use of VR also has constraints: e.g., the length of time people may be willing to use a VR headset, the technology’s inaccessibility to people with sensory disabilities, and the challenge of replicating the communal experience that is, for many, central to the joy of opera.

This trial will involve co-creation with three very different communities: in outer Dublin, in a medium-sized town in the midlands and in an Irish-speaking island. Here, social exclusion can be a matter of geography, language or culture, and may be hidden. It takes the form of restricted capabilities, so the opera will draw on stories shared by members of each community to create work that resonates with the diversity of contemporary life in Ireland. The project has the overall title ‘Out of the Ordinary’, which expresses both the extraordinary experiences offered by virtual reality headsets and the intention to create a workout of everyday life experiences and stories. It also recognises how the COVID-19 situation has changed ordinary life in Ireland.

As with TRACTION, MEDIAVERSE is a three-year H2020 funded project. It aspires to support new media forms of expression, confronting and blending traditional broadcast with the new media landscape. Towards a fair, user-driven, sustainable and technologically advanced media ecosystem, MEDIAVERSE has eight specific objectives, most of them linked to immersive environments. One of them is to introduce novel tools and methods for the next generation media analysis and annotation. It includes the development of cost-effective solutions for the analysis and understanding of media assets, such as 3D objects and 360º video with advanced precision and realism.

To provide innovative XR authoring tools to produce immersive media experiences, MEDIAVERSE proposes an integration of innovative tools that abstract all the coding and programming skills needed for authoring immersive media experiences. It will allow content creators to experiment with interactive 360º storytelling, as well as native 3D and hybrid XR productions (VR, AR, and MR) in an easy and fast way, diminishing technical entry barriers. The authoring environment includes asset provision, behavioural templates for application logic, and deployment procedures to allow platform-specific experience development for non-expert users.

Regarding large scale pilots and experiments, MEDIAVERSE aims at carrying out three use cases: 1) a citizen journalism pilot, with the objective of showcasing the project in its ability to connect cross-media silos and prosumers, providing all the necessary tools for accelerating publishing to a wide audience using the social media analytics engine; b) a co-creation pilot with the objective of exploring MEDIAVERSE co-creation media enrichment and the immersive authoring possibilities with non-professional users who will deploy and validate the accessibility tools, social collaboration spaces and immersive storytelling authoring tools; and c) STARTS pilot, with the objective of assessing the value and use of the project for artistic communities, inspiring new scientific and technical approaches for next generation media.

The rise of the digital era has brought novel immersive, accessible, and personalised user experiences to media, thereby disrupting traditional media. MEDIAVERSE proposes new media formats and creation mechanisms in order to address all three directions, emphasizing on the fact that the new era for the media landscape demands improved speed to market it at lower costs. Moreover, this project is committed to facilitating the production of accessible media content and interfaces for everybody, following the latest EU regulations.

Both TRACTION and MEDIAVERSE aim at including all members of society, especially vulnerable groups through the co-creation of cultural activities. Given the fact that vulnerable groups may have issues accessing media content, both projects will take care of media accessibility as an activity from the onset of the projects.

Conclusions

The current situation has already pushed the world out of its comfort zone, but this is the perfect opportunity to experiment and enter the immersive dimension. The state of XR technology today is advanced enough to enable easy, fast and affordable access for wide audiences. The strategies for surviving the ‘new normal’ imposed by COVID-19 include a successful adoption of advanced technologies (including XR), and reinventing our approach to work, learning, culture, social interaction and individual lifestyle.

The article has presented two European projects that have two things in common: on the one hand, they are coexisting and adapting to the current Covid-19 situations; and on the other hand, both use immersive environments as a tool to bring culture closer to the population. One of the main risks for both projects is the resistance to or dislike of VR, as it is a new and unknown experience for many. The key to solving this issue is to ensure a supportive and attentive approach to immersive environments, making sure that all the participants have the same access to XR.

More than ever, technology is allowing the empowerment of end-users, both as consumers and prosumers. Still, technology is designed with accessibility as an afterthought: away from the user-centric design. As for every media content, accessible experiences need to be accessible. One of the main purposes of both TRACTION and MEDIAVERSE is to bring accessibility to all immersive content generated within the projects. User interaction in today’s Information Society plays a key role in the social integration and democratic participation of all its citizens. Only by promoting the accessibility concepts will it be possible to achieve the full democratic participation of all people with equal access to culture.

Acknowledgments

TRACTION and MEDIAVERSE have received funding from The European Union’s Horizon 2020 research and innovation programme under grant agreement No.870610 and No.957252 respectively. The author is a member of the TransMedia Catalonia research group funded by Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, under the SGR funding scheme (ref. code 2017SGR113). This article reflects only the author’s views and the funding institutions hold no responsibility for any use that may be made of the information it contains.

Bibliography

Azuma, Ronald, Yohan Baillot, Reinhold Behringer, Steven K. Feiner, Simon Julier, and Blair MacIntyre. 2001. “Recent advances in augmented reality” in IEEE Computer Graphics and Applications nº 21(6): 34–47.

Brem, Alexander, Eric Viardot, and Petra Nylund. 2020. “Implications of the coronavirus (COVID-19) outbreak for innovation: Which technologies will improve our lives?” in Technological Forecasting and Social Change nº 163. https://doi.org/10.1016/j.techfore.2020.120451.

Conroy, Andy. 2017. The BBC and Virtual Reality, BBC Research & Development. http://www.bbc.co.uk/rd/blog/2016-06-the-bbc-and-virtual-reality. Accessed 20 March 2021.

El-Said, Osman, and Heba Aziz. 2021. “Virtual Tours a Means to an End: An Analysis of Virtual Tours’ Role in Tourism Recovery Post COVID-19” in Journal of Travel Research. https://doi.org/10.1177/0047287521997567.

Fida, Benish, Fabrizio Cutolo, Gregorio di Franco, Mauro Ferrari, and Vicenzo Ferrari. 2018. “Augmented reality in open surgery” in Updates in Surgery, no. 70: 389–400.

Gloy, Kilian, Paul Weyhe, Eric Nerenz, Maximilian Kaluschke, Verena Uslar, Gabriel Zachmann, and Dirk Weyhe. 2021. “Immersive Anatomy Atlas: Learning Factual Medical Knowledge in a Virtual Reality Environment” in American Association for Anatomy. https://doi.org/10.1002/ase.2095

Holly, Michael, Johanna Pirker, Sebastian Resch, Sandra Brettschuh and Christian Guetl. 2021. “Designing VR Experiences -Expectations for Teaching and Learning in VR” in Educational Technology & Society nº 24: 107-119.

Ioannides, Dimitri, and Szilvia Gyimóthy. 2020. “The COVID-19 Crisis as an Opportunity for Escaping the Unsustainable Global Tourism Path” in Tourism Geographies nº 22 (3): 624–32.

Jauhiainen, Jussi. 2020. “Virtual 3D platforms in entrepreneurship and innovation events during the COVID-19 pandemic” in The case of SHIFT in Finland in October 2020 on the VirBELA platform.

Kaur, Navleen, and Supriya Sahdev. 2020. “Fighting COVID-19 with technology and innovation, evolving and advancing with technological possibilities” in International journal of advanced research in engineering and technology, nº 11: 395-405. https://doi.org/10.34218/IJARET.11.7.2020.039.

Milgram, Paul, and Fumio Kishino. 1994. “A taxonomy of mixed reality visual displays” in IEICE Transactions on Information Systems, no. 77: 1321–1329.

Montagud, Mario, Pilar Orero, and Sergi Fernández. 2020. Immersive media and accessibility: Hand in hand to the future, ITU Journal. https://www.itu.int/pub/S-JOURNAL-ICTS.V3I1-2020-2. Accessed 11 February 2021.

Papachristos, Nikiforos M., Ioannis Vrellis, and Tassos A. Mikropoulos. 2017. “A Comparison between Oculus Rift and a Low-Cost Smartphone VR Headset: Immersive User Experience and Learning” in 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT), 477-481. http://doi.org/10.1109/ICALT.2017.145.

Parham Groesbeck, Eric G. Bing, Anthony Cuevas, Boris Fisher, Jonathan Skinner, Richard Sullivan, and Mulindi H. Mwanahamuntu. 2019. “Creating a low-cost virtual reality surgical simulation to increase surgical oncology capacity and capability” in Ecancermedicalscience, nº 13.

Rizzo, Albert, Maria Schultheis, Kimberly A. Kerns, and Catherine Mateer. 2004. “Analysis of assets for virtual reality applications in neuropsychology” in Neuropsychological Rehabilitation, nº 14:1-2: 207–239.

Ruofei Du, David Li, and Amitabh Varshney. 2019. “Geollery: A Mixed Reality Social Media Platform” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19). Association for Computing Machinery, nº 685: 1-13. https://doi.org/10.1145/3290605.3300915

Rupp, Michael, James Kozachuk, Jessica R. Michaelis, Katy Odette, Janan Smither, and Daniel S. McConnell. 2016. “The effects of immersiveness and future vr expectations on subjec-tive-experiences during an educational 360 video” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting nº 60(1): 2108–2112. https://doi.org/10.1177/1541931213601477.

Smith, Will. 2015. Stop calling google cardboard’s 360-degree video ‘vr’. Wired. https://www.wired.com/2015/11/360-video-isnt-virtual-reality/

UNWTO (United Nations World Tourism Organization). 2020. “UNWTO World Tourism Barometer May 2020. Special Focus on the Impact of COVID-19.” https://webunwto.s3.eu-west-1.amazonaws.com/s3fs-public/2020-05/Barometer%20-%20May%202020%20-%20Short.pdf Accessed March 21, 2021.

Zheng, Liping. 2020. “Discussion of Design and Application of Live Broadcasting System Based on 5G + VR Technology” in 2020 International Conference on Computer Engineering and Application (ICCEA) 757-760. https://doi.org/10.1109/ICCEA50009.2020.00165.