ASOMROF: A Transmedia Project of Sci-Fi Content Inspied by the Local Ethnic History for Technology Art Works

The origin of a technology art project

In 2021, faculties from the department of design and the department of graphic arts and communications in National Taiwan Normal University, as well as the college of performing art in the National Taiwan University of Arts, applied to the project solicitation of “technology arts” from National Science Council of Taiwan.

The term “technology arts” generally refers to the use of technology as a medium or tool in creating works of art. This can encompass a wide range of techniques, from digital art and computer-generated graphics to interactive installations and immersive experiences (Atkins 2013).

One notable reference for technology art is the movement of “new media art,” which emerged in the late 20th century and encompasses a diverse range of works that use technology as a primary element. This can include interactive installations, video art, and other forms of digital media (Paul 2015; Manovich 2001; Lister et al. 2009).

Another reference for technology art is the field of “creative coding,” which involves using programming languages to create interactive and generative art. This field has gained popularity in recent years as more artists explore the potential of code as a medium for creative expression (Reas 2010; Maeda 2004).

The term “digital art” shares the similar definition with technology art. It describes artwork created with the aid of software, computers, or other electronics. Digital art includes everything created or produced using digital media, including animations, images, drawings, videos, paintings, and more (Colson 2007). It emerged in 1960s, when pioneers such as John Whitney began to create art works using early computing technology. Issues of digital art history have arisen and been discussed since the rapid evolution of computational tools such as AI (Baca, Helmreich, and Gill 2019; Drucker 2013).

Overall, technology art is a rapidly evolving field that reflects the ongoing evolution of technology and its impact on society and culture. As such, it continues to push boundaries and challenge traditional definitions of art and creativity.

Due to a potential collaboration between the project’s host and the Hakka Communication Foundation, the topic was initially chosen to be related to Hakka culture. One of the technical forms that this project also aims to explore is the remote collaboration between virtual characters and physical dancers. With the fast data transmission of fifth-generation networks (5G), there have been recent performances that utilize this form by projecting virtual performance images and interacting with real dancers, providing audiences with new sensory experiences (Meador et al. 2004; Lisowski et al. 2023).

The concept of this project is to use traditional Hakka opera stages as performance venues as shown in image 1.

These Hakka opera stages were traditionally used to perform what is known as Hakka opera during important traditional festivals such as the Lunar New Year to repay the gratitude to the deities. These operas tell traditional stories of loyalty and filial piety that have been passed down from China, as well as some entertaining dramas about love and relationships (Hsieh 2017).

A brief history of early Hakka immigrants to Taiwan as the inspiration for the sci-fi content

Hakka, which means “guest people” in the original language, is an ethnic group that has been migrating from central China since the end of the Han dynasty around 200 A.D.(Lo 1992). For over a millennium, numerous waves of migration have occurred, leading to the settlement of people in southern China. Based on current literature, it is possible that the migration of Hakka people, as they are currently known, to southern China may have occurred later than that of Cantonese and Hokkien (Fujianese) people (Chiu 2001). As latecomers, the Hakka people were at a disadvantage in terms of resources and had to cultivate limited agricultural land near mountainous areas, leading to a relatively difficult living environment.

The Hakka people migrated to Taiwan from China espacially Guangdong (Canton) province during the 17th and 18th centuries. The reasons for their migration are complex, but it is believed that many Hakka people were pushed out of their homelands due to wars, famine, and natural disasters. Others migrated to escape oppression and persecution, particularly during the Qing dynasty (Campbell 2020; Riess 2019). Due to the fact that Hakka people are a relatively late-arriving ethnic group, they are referred to as “guests” (in Chinese: 客人) regardless of whether they live in Guangdong, Fujian, or Taiwan. Conversely, the true meaning of a guest is reversly placed in text (in Chinese: 人客) and pronounce (Chiu 2001).

The Hakka people’s migration to Taiwan was not a single event but occurred over several centuries. Many Hakka people settled in the central and southern regions of Taiwan, where they established communities and contributed to the island’s early development (Yin 2003). The Hakka people’s settlement patterns in Taiwan were shaped by a variety of factors, including topography, climate, and social and political conditions. Many Hakka people settled in hilly areas where they could cultivate crops and raise livestock. They also established communities near rivers and streams, where they could access water for irrigation and transportation. Hakka communities in Taiwan were typically organized around clans and extended families. They often built fortified villages, known as tulou, which were designed to protect against attacks from bandits and other enemies. Tulou were typically multi-story structures with thick walls and narrow windows, and they were built to accommodate several families.

Despite their contributions to Taiwan’s development, the Hakka faced many challenges and conflicts in their early years on the island. Due to the relatively late arrival of Hakka people in Taiwan compared to Hokkien (Fujian) immigrants, as well as their smaller population, they tended to settle in more remote and less fertile agricultural areas near mountainous regions (Chiu 2001). However, some scholars argue that the Hakka people did not arrive in Taiwan later than the Hokkien (Fujianese), and that their relocation to mountainous areas was driven by the economic advantages of cultivating high-value crops such as tea, as well as the benefits of proximity to water sources for agriculture (Yin 2003). Regardless, due to the competition over resources, there have been numerous instances of armed conflict between Hakka people and Hokkien immigrants throughout the history of Taiwan. Furthermore, conflicts have arisen between Hakka settlers and indigenous communities due to overlapping territorial claims, as Hakka settlements are predominantly located in close proximity to mountainous regions (Huang 1989).

The history of competition and conflict between Hakka, Hokkien, and indigenous people inspired the story content for this project. This may also jog the collective memory of other nations with similar histories of new land exploration. Imagining the future, if humans were to migrate to other planets due to the Earth becoming uninhabitable, a similar situation may occur. That is to say, migrants would encounter earlier migrants and residents who have received assistance from extraterrestrial beings. The three parties would compete and conflict with each other due to resources and interests.

The story content for the three-acts dancing performance

The performance was scheduled to last about ten minutes. The performance’s plot was divided into three acts.

The first act is named “warship”. It descrbe that the first group of humans to leave Earth traveled on a giant spaceship named HKA05, and after a long journey in cryosleep, they arrived at the planet. This planet has only one large continent called Asomrof, and the rest is ocean. Most of the flat and fertile land has already been occupied by the earlier alien immigrants, the “Raves.” The humans could only move to the edge of the enormous Hopoloon mountain in the continent to cultivate the land. The alien climate and diseases, unlike those on Earth, made farming very difficult, and complaints were widespread. Captain Johnny Lo then prayed to the gods to ease the people’s hearts, and during the ceremony, a deity appeared, claiming to be the “Great King of Wei Hui Hong Ying Feng” from the Three Mountains Kingdom, and promised stable weather and the end of the epidemic in the following year.

The second act is named “fighting”. The human settlement is located on the side of a mountain, where the planet’s indigenous people, the “Ayaris,” live. They coexist with plants and animals, have strong physiques and agile movements, and sometimes can transform into beasts. Due to their dislike of the invading human settlers who frequently intruded and caused conflicts, they launched a large-scale attack on the human base during a full moon night. During the adventure to escape with her parents, a human girl named Lea got lost and entered the Raves’ territory. After attempting to communicate, she discovered that they also came from the Earth and were a later race of humans after the great changes on the Earth. They seemed small and weak, unable to speak, but could communicate with each other through their minds. Three or more of them could cooperate and display stronger abilities than humans. Although they left the Earth more than a hundred years after humans did, they arrived at the planet earlier due to their more advanced technology.

The third act is named “reconcilation.” After collective communication, the Rave people agreed to Lea’s request and dispatched mechanical forces to rescue the humans. The human-rave joint army fought fiercely against the man-beast legion of Ayaris on the south side of the Mountain in Hopoloon, but they were at a disadvantage. Captain Jolean once again prayed to the gods and the Wei Hui Hong Ying Feng responded by appearing in a huge image. The Ayaris people also sacrificed and summoned the God of War, “Tatawuli”, led by their leader. The two deities clashed and revealed their strange forms, which were two irregular luminous bodies. The two luminous bodies in the air cause strong winds and rainstorms, as well as earthquakes and tremors. All the Rave people sat down quietly, gathering their consciousness and communicating with the three luminous bodies, understanding that they were extraterrestrial beings existing as spiritual entities and had been on the planet for more than 10,000 years, absorbing the energy of worship to become incarnations of various ethnic deities to assist in cultivation and living. Witnessing the current cruel war, they were sad and wanted to leave. After learning the information from the Rave people, Lea conveyed it to humans and Ayaris people, and in order to keep the gods, both tribes prayed for the war to reconcile and end peacefully.

It is noted that the human group represents Hakka immigrants, while the rave people represent the Hokkien people for their early arrival on the new planet. Ayaris, the reverse spelling of Siraya, is the symbol of indigenous communities. The Siraya people are an indigenous tribe in Taiwan and belong to the larger Austronesian ethnic group. They are classified as a plains tribe because they reside in the southern plains, and are distinguished from the mountain tribes. The Siraya people were the first indigenous group that European explorers encountered when they arrived in Taiwan, and in history, they were also known as “Formosan” people (Steere 2002). The name of the continent “ASOMROF” is the revrse spelling of Formosa.

Technical aspects of the collective performance with virtual characters and physical dancers

The performance of remote collaboration between virtual characters and physical performers took place on August 29th, 2022, at Digiblock in Taipei. This location is an industry innovation base established by collaboration between the central government and Taipei City government, focusing on digital development and key topics such as cross-domain new intelligent Internet of Things (‘Exoplanet Exploration: Planets Beyond Our Solar System’). The performance is a collaboration of the project with Digiblock to test 5G network connection technology, which was also a technical support project for this base. The image below shows a glimpse of the performace.

As shown in the figure, the physical dancer performs directly facing the audience on stage, while another dancer wears motion capture clothing to perform in another space. The image presented to the audience is the virtual character image in the projection. This creates opportunities for both physical dancers and virtual characters to interact with the audience.

The dance performance is imagistic, with three scenes of worship, conflict, and reconciliation presented through music and visuals, without a clear narrative. The character design is also relatively simplified, with three characters designed by project members: Earth immigrants, alien natives, and the god. The style of the character presentation uses traditional Chinese paper cutting or collage art as the material for virtual characters, as shown in the figure below.

The scene was designed to complement the characters’ paper-cutting style. Flat 3D objects such as mountains and clouds were created as the background using the software Maya and Blender. A spaceship was created using 3D software at the start of the performance to represent Earth immigrants arriving on an alien planet. The design of the spaceship was inspired by traditional Hakka elements such as earthen buildings, paper umbrellas, and terraced fields, as seen in the image below.

The characters were also created using Maya, with textures applied and skeletons rigging, and finally integrated with the 3D scene using the Unity game engine.

Regarding the motion capture aspect, we used the Xsens system (‘Make the most of every move’ 2023) because it is based on miniature inertial motion sensors (gyroscopic sensors) to track motions and a wireless transmission system, which has fewer restrictions on the space required. Another popular optical motion capture system necessitates optical tracking equipment and a dedicated setup space, making it less suitable for this project. The entire workflow is shown as the following figure.

The biggest challenge in the entire virtual-physical integrated dance performance is still the issue of network connectivity. Despite the technical support from collaborating parties, the 5G network connection speed can still cause a time lag between the performances of virtual and physical dancers. If considering the traditional Hakka outdoor performance in the project, the problem of longer distance network connection and time delay may be even more difficult to solve.

Another issue is the positioning of the virtual dancers presented on the projection through the transferring motion capture data, which requires careful coordination with the physical dancers. This aspect requires repeated practice and rehearsal, as well as on-site adjustments during the performance.

The evolving of the content for the augmented reality exhibition

Aside from the remote collaboration performance between virtual characters and physical performers, an augmented reality (AR) exhibition based on the same content is also in the works. This augmented reality performance will be displayed at Taiwan’s Kaohsiung Museum of Fine Arts, along with other teams who received National Science Council funding for Technology Art. This exhibition is also planned to be shown in Ars Electronica Festival 2023, Linz, Austria, as well as the Immersive Tech Week 2023, Rotterdam, Netherland.

Augmented Reality refers to a technology that superimposes computer-generated objects, images, or information onto the real world. AR enhances the perception of the physical environment by overlaying it with digital elements, creating an interactive and immersive experience (Azuma 1997; Milgram 1994; Billinghurst and Kato 1999).

This AR exhibition is expected to use the ARCore Geospatial API developed by Google to place virtual 3D models onto real outdoor buildings. Based on the aforementioned science fiction content, the performance has been designed, and the audience can watch it through a mobile app at the scheduled time and designated area. This provides an experience of a large-scale performance in which the real and virtual worlds are integrated, which is difficult to achieve in real-world situations. An image of the test in the mobile app is shown below.

The white area that encircled the building is the set mask that will prevent the virtual 3D character models from appearing. When more advanced display tools, such as glasses, become more widely available, the exhibition can be further developed in the concept of mixed reality (XR), which is an umbrella term that includes all forms of virtual reality (VR), augmented reality (AR), and everything in between. It includes technologies that connect the physical and digital worlds, allowing users to interact with digital content in real time while still in the physical world (Rokhsaritalemi, Sadeghi-Niaraki, and Choi 2020).

The development of novel and manga from the content based on the scientific knowledge

As previously stated, the dancing performance was imagistic and symbolic, with no clear narrative. The plot of the three acts was also relatively simple. This provides a starting point for developing a more complete story in other media. The text format of a novel, as well as the combination of graphic and text as a comic book, are relatively simple ways to get started.

To create a novel with existing sci-fi content, more detailed world settings and characters are required. The search for the planet where the story would take place was carried out to make the scenario more realistic. NASA (National Aeronautics and Space Administration) funded the open exoplanet archive on the internet to investigate life beyond Earth (‘Exoplanet Exploration: Planets Beyond Our Solar System’). It is also an important database for estimating the suitability of humans to immigrate and live based on current scientific truth. The Earth Similarity Index (ESI) is a scale used to compare other planets to Earth physically. The scale runs from 0 (no resemblance to Earth) to 1. (Earth-like). Planets with ESIs between 0.8 and 1.0 are more likely to be Earth-like.

Habitable zone is a term defined as “the distance from a star at which liquid water could exist on orbiting planets’ surfaces” (‘Exoplanet Exploration: Planets Beyond Our Solar System’). Several star systems, including the TRAPPIST-1 system and the TOI 700 system, have been discovered to have planets in their habitable zones. Another factor to consider is the distance of the star system from the Earth, as it is always preferable for immigrants to travel less time in the future.

Teegarden’s Star b (also known as Teegarden b) is then chosen as the planet where the human immigrants are targeted in the story. It is an exoplanet discovered orbiting within the habitable zone of Teegarden’s Star, located approximately 12 light years from Earth (‘Teegarden’s star b’). It is currently one of the closest potentially habitable exoplanets, as well as one of the exoplanets discovered with the highest ESI. Teegarden b has a mass of 1.05 Earths and takes 4.9 days to complete one orbit around its host star, a M-type red dwarf star. The majority of red dwarfs, such as Kepler-438b and Proxima Centauri, emit powerful flares that can deplete the atmospheres of their planets, rendering them uninhabitable. Teegarden b’s host star is relatively inactive and quiet, which makes it possibly more inhabitable (‘Teegarden’s Star b – an Exoplanet’).

Since it is so close to its host star, it is possibly tidal locking, also known as synchronous rotation, which means that one planet takes the same amount of time to rotate around its own axis as it does to revolve around its host. For example, the same side of the Moon always faces the Earth. Because of the tidal locking, the planet’s twilight stripe is the only region suitable for life, and this is where the story takes place. The story scenario is built around these scientifically knowledge settings about the exoplanet Teegarden b, as well as inspiration from the history of Hakka immigration to Taiwan. The novelization of the sci-fi story, as well as the manga, are ongoing. Image 5 depicts a page from the manga. Following the current transmedia trend in popular entertainment industries such as the United States and Japan (Jenkins 2010; Schiller 2018; Ryan 2015), this story could eventually be adapted into a video format such as a TV series, movie, or even a game. The transmedia journey of ASOMROF is still ongoing.

Bibliography

Atkins, Robert. 2013. ArtSpeak: A Guide to Contemporary Ideas, Movements, and Buzzwords, 1945 to the Present. USA: Abbeville Press.

Azuma, Ronald T. 1997. ‘A Survey of Augmented Reality’, Presence: Teleoperators and Virtual Environments, 6: 355-85.

Baca, Murtha, Anne Helmreich, and & Melissa Gill. 2019. ‘Digital Art History’, Visual Resources, 35: 1-5.

Billinghurst, Mark, and Hirokazu Kato. 1999. ‘Collaborative augmented reality’, Communications of the ACM, 45: 64-70.

Campbell, William. 2020. Formosa Under the Dutch: Described from Contemporary Records with Explanatory Notes and a Bibliography of the Island. UK: Routledge.

Chiu, Yen-Kuei (邱貴彥)

Wu Chung-Chieh (吳中杰). 2001. The Map of Taiwan Hakka. Taiwan: Owl Publishing House.

Colson, Richard. 2007. The Fundamentals of Digital Art. USA: Fairchild Books.

Drucker, Johanna. 2013. ‘Is There a “Digital” Art History?’, Visual Resources, 29: 5-13.

‘Exoplanet Exploration: Planets Beyond Our Solar System’. NASA, Accessed 8 April 2023. https://exoplanets.nasa.gov/.

Hsieh, Yi-Wen (謝宜文). 2017. A Study on the Application of Hakka Eight Sounds in the Meinong Area’s Hakka Ancestral Deity Festival (客家還神祭典與客家八音運用之研究). Taiwan: Mourning Star (晨星).

Huang, Jung-Lo (黃榮洛). 1989. The Threnody of Crossing Taiwan: A History of Exploration and Struggle in Taiwan (渡台悲歌 : 臺灣的開拓與抗爭史話). Taiwan: Tai Yuan Publishing.

Jenkins, Henry. 2010. ‘Transmedia Storytelling and Entertainment: An annotated syllabus’, Continuum, 24: 943-58.

Lisowski, Dan, Kevin Ponto, Shuxing Fan, Caleb Probst, and Bryce Sprecher. 2023. ‘Augmented Reality into Live Theatrical Performance.’ In Andrew Yeh Ching Nee and Soh Khim Ong (Eds.), Springer Handbook of Augmented Reality. Springer: Swizerland.

Lister, Martin, Jon Dovey, Seth Giddings, Iain Grant, and Kieran Kelly. 2009. New Media: A Critical Introduction. USA: Routledge.

Lo, Hsiang Lin (羅香林). 1992. An Introduction to the Study of the Hakkas in Its Ethnic, Historical, and Cultural Aspects (客家研究導論). Taiwan: SMC Publishing.

Maeda, John. 2004. Creative Code: Aesthetics + Computation. UK: Thames and Hudson.

‘Make the most of every move’. 2023. Movella, Accessed 5 April 2023. https://www.movella.com/products/motion-capture.

Manovich, Lev. 2001. The Language of New Media. USA: MIT Press.

Meador, W. Scott, Timothy J. Rogers, Kevin O’Neal, Eric Kurt, and Carol Cunningham. 2004. ‘Mixing Dance Realities: Collaborative Development of Live-Motion Capture In a Performing Arts Environment’, ACM Computers in Entertainment, 2: 1-15.

Milgram, Paul. 1994. ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Transactions on Information Systems, E77-D: 1321-29.

Paul, Christiane. 2015. Digital Art (World of Art). UK: Thames & Hudson.

Reas, Casey. 2010. Form+Code in Design, Art, and Architecture. USA: Princeton Architectural Press.

Riess, Ludwig. 2019. Geschichte der Insel Formosa (福爾摩沙島史). Taiwan: National Museum of Taiwan History.

Rokhsaritalemi, Somaiieh, Abolghasem Sadeghi-Niaraki, and Soo-Mi Choi. 2020. ‘A Review on Mixed Reality: Current Trends, Challenges and Prospects’, Applied Sciences, 10: 636.

Ryan, Marie-Laure. 2015. ‘Transmedia Storytelling: Industry Buzzword or New Narrative Experience?’, Storyworlds: A Journal of Narrative Studies, 7: 1-19.

Schiller, Melanie. 2018. ‘Transmedia Storytelling: New Practices and Audiences.’ In Ian Christie and Annie van den Oever (Eds.), Stories. (pp.97-107) Amsterdam University Press: Netherland.

Steere, Joseph Beal. 2002. Formosa and its inhabitants. Taiwan: Academia Sinica.

‘Teegarden’s star b’. NASA, Accessed 9 April 2023. https://exoplanets.nasa.gov/exoplanet-catalog/7423/teegardens-star-b/.

‘Teegarden’s Star b – an Exoplanet’. Assignment Point, Accessed 9 April 2023. https://assignmentpoint.com/teegardens-star-b-an-exoplanet/.

Yin, Chang-Yi (尹章義). 2003. A Research on Hakka History in Taiwan (臺灣客家史研究). Taiwan: Taipei Hakka Affairs Commission.