Aural and Visual Imagery in the Multimedia Music Performance

1 Introduction

As a professional piano performer, I have interpreted many works of composers from different periods and countries. Amongst them, Claude Debussy is one of the Western composers who has specifically interested me and whose work I appreciate and follow.

I have been deeply attracted to the flowing pictures and looming oriental style colours evoked by his music, in particular, the 24 Piano Preludes. This is because each of prelude has a short but delicate form, are at the same tempo, encompass the free flow of phrases and rhythm in their musical texture, and plasticity of the musical narratives that reflect the natural images.

By coincidence, I came across the album of 24 Preludes by the Chinese composer Shengjing Liao. This work was created following the twenty-four seasonal and amorous scenes of the Chinese solar calendar. Liao, concurrently, carried out in-depth research and organization of the Chinese traditional pentatonic scale system and incorporated twenty-four different tones in his preludes. I have discovered some similarities in the preludes of these two composers, which piqued my interest and enthusiasm to further study and research the subject. One of the core reasons for my curiosity and enthusiasm is how to connect the imagery to my recreation of these preludes. Both Debussy and Liao created imagery titles for their preludes, portraying their insights into musical colors, nuances of the inner layers, as well as the frameworks of the harmonic, rhythmic and melodic textures. My undertaking will be in exploring if these musical components and sensibilities could, in some way, be re-created by a performer, using other forms of artistic expression, especially related to imagery and pictures.

2 Multimedia music performance

In order to investigate and understand the elements involved in the study into performance, it’s necessary to confront the emergent challenge, which is how complementary points of juxtaposition of knowledge and interdependence could exist in a performance. The artistic research will be accomplished by joining tempo together with the quality of the artistic performances, it could be a single artistic experience or interdisciplinary artistic experiences. The artists/researchers could perform, reform and transform what they conceive and transmit this “action knowledge” to the audience. This kind of performance not only require an artistic action, but also an artistic knowledge that will create new interpretations. Therefore, the most important step during the research process is how could achieve favorable aural and visual quality in the interpretation of the musical imagery. if this recreation of visual images combined with musical expressiveness by piano performance could achieve mutual complementation between auditory and visual sensations and reach the audience’s resonance and empathy during the performance.

3 Theoretical Evidence

When the images appear as a combined form of musical performance, there is a connection of synesthesia between them, having a similar effect on the narrative, emotion, structure and among other aspects. Sometimes they complement each other’s feelings and sensations like an organic mechanism. According to Nicholas Cook: “both music and pictures can be understood in terms of distributional analysis, and the relationship between them can be understood as an interplay of structurally congruent media: this more or less amounts to saying that the pictures can be analyzed musically” (Cook, 1998, p. 159). The author also recognized that there is a narrow relationship between sound and image, which affects each as a whole with regards to conformance, complementation and contest (Ibid.). In other words, Lipscomb and Tolchinsky come to confirm the idea of Cook, referring that “the connotative qualities of the music complement the denotative qualities of the words and pictures” (Lipscomb & Tolchinsky, 2005, p.391).

In the field of cinematographic research, Michel Chion demonstrated his “Added Value” theory and affirmed that the influence between sound and image and recognized these two mediums exist in an entirety by their individual structure and form, with their own characters and meaning. Despite this, in the audio-visual contract, at the same time, the sound and the image develop their own narrative, and there is something new created by the combination of the two (Chion, 1994). He also emphasized that “the visual and auditory perceptions are of much more disparate natures than one might think. The reason we are only dimly aware of this is that these two perceptions mutually influence each other in the audiovisual contract, leading each other their respective properties by contamination and projection” (Chion, 1994, p.9). This view is similar to Cook’s conception of “Complementation” and “Contest”.

In the research related with the multimedia music performances, Meghan Stevens studied the music and image in the concert environments. The author explored the characteristics of the musical image in terms of congruency, narrative, meaning, emotion and structure. She mentioned the theory of “Added Value” in her book: “Chion’s theory of Added Value states that one sensory perception influences another and transforms it, resulting in us not seeing the same thing when we are listening, and not hearing the same thing when we are seeing. In other words, if we look at something while listening to music, whatever we see may influence and may change our interpretation of what we see and we may see something different. The two senses can combine to create a different experience to what we would have seen or heard had each sense been experienced without the other” (Stevens, 2009, p.10).

However, in this research, the focus is on the revelation of the auditory and visual imagery from the perspective of the pianist. The congruence of synesthesia is from the same subject, while the musical sense is transformed into short videos, through the recreation of the pianist, and while this recreation of visual images is expressed musically by the execution of the piano. In such a way, the multimedia performance could achieve a mutual complementation, namely in emotions, colors, harmonies, rhythms, textures and structures.

In the diagram below, it is possible to verify the bi-connection between performer and listener, through the synesthesia of the auditory and visual senses. The performer prepared an interpretive elasticity on the piano, in order to add timbre colors, emotions, rhythmic and harmonic movements in visual images. At the same time, visual images created by performer also outline the musical figures, adding colors, lights, shapes, surrounds and other elements.

Related to the understandings and feelings of audience, the expected effect is that each listener (spectator) can experience the music and video created by the performer during a specific time/space; and each individual can create their own meaning, a meaning that represents their own emotional or internal experience during the multimedia performance.

4 Procedure of creating the multimedia music

During this investigation, 12 preludes were chosen to be created in short videos. In the expectation of summarizing and highlighting the essential characteristics of the study, only 4 copies are exposed in this article, namely La Fille aux Cheveux de Lin, The Beginning of Autumn – Reverie of The Chinese Qixi Festival (立秋-七夕遐思), Rainwater – The sound of rain dripping down in Spring (雨水-春雨淅沥) and Des pas sur la neige.

4.1 La Fille aux Cheveux de Lin

The Prelude La Fille aux Cheveux de Lin is inspired by the Poem of the same name by the French poet Charles-Marie René Leconte de Lisle.

The imagery in the poem depicts a girl with flaxen hair, in a flower-filled meadow, sunlight, and larks, images that symbolically reveal the poet’s inner emotions. In the Prelude, these images and emotions are aurally represented by melody, harmony, and particular timbres, that create the metaphoric tone of the poem.

The first step in creating the visual imagery was to identify the thematic material that I required for the video and to integrate them with the pentatonic melody of the music theme, in the opening section. In the visual theme, the girl with the flaxen hair was the obvious choice, given the wholeness exposition and recapitulation of the musical theme. The colours of the natural light, the flowering meadow, the Lark, and the clouds also became essential elements for the layout of the visual imagery. For the video production, I also needed to build sensory bridges between the music and the visual elements, such as mediating pictorial colour with musical colour, image structure with musical structure, visual harmony with aural harmony, and video speed with musical rhythm and meter.

In terms of colour, it is musically determined by tonality, which is distributed in the entire composition in: G-flat (bars 1-3), E-flat (bars 4-7), G-flat (bars 8-13), C-flat (bars 14-16), G-flat (bars 17-18), E-flat (bars 19-23), and G-flat (bars 24-39). There is a relatively smooth change of the tone and, simultaneously, the pentatonic scale harmony is blended with the diatonic scale harmony. Therefore, the melodic colour of the entire piece is hazy, warm, and corresponding to the images.

Regarding the music and image structure, the Prelude can be divided into three parts, each of which has the melodic theme symbolising the flaxen-haired young girl. This theme is produced on the pentatonic scale, which provides the music with a pure and peaceful atmosphere. Therefore, when constructing the images, the primary pictures of the three appearances of the theme are basically the same apart from being suitably rendered with different lights and textures. For example, in order to highlight the hazy and quiet emotion of the music, a new image is added to the theme image to blur it. In the editing software, the first step is selecting the main screen, then choosing a soft light within the image blending mode and adjusting the aspect ratio to the images.

As shown in the image, there are the theme.

Concerning the visual harmony and its correspondence to the music harmony, we find that the triads of diatonic scales are often used, with the seventh chords and pentachords, to obtain the colourful harmony texture in the music. Therefore, to show the different senses of harmony, the visual combination is a necessary component to reflect this atmosphere. Two ways were chosen to express the sense of harmony, one was by using images with a strong contrast of light and shadow, and the other was overlapping the images with different levels of sharpness. As shown in the pictures below, the 9th and the 24th bar harmonies of the Prelude are represented respectively.

The speed of the video playback was an important component. It had to match the metric and rhythmic aspects of the Prelude. There are many expressive terms for recording the movements of the prelude, which, in addition to expressing the basic mood of the music, can also show suggestions for the tempos of the performance. Three rather obvious changes in tempo are verified in the music. The first tempo change is found at bar 19, where the expressive term states Un peu anime. Thus, the jump in colour assists the tempo advance, and at the same time adjusts the speed of the picture transition to the corresponding position. In terms of colour, I chose the visual colour switch from purple to light green to bright green, shown in the pictures.

The second tempo change is found in bars 23-24, marked by the term Cédéz, indicating the impeded tempo here and then moving on to the next bar. At the beginning of bar 24, where it is marked by the term sans lourdeur, which is interpreted as meaning carefree without heaviness. In this section, which is all about parallel chord movements, the pianist needs to play a little more gradually combined with the slowing down of the video. Furthermore, this parallel movement uses pentatonic chords, it is therefore important to maintain an ethereal feeling with the timbre colour. The duration of Murmuré et en retenant peu à peu, at the beginning of bar 32, is used to show a quiet and slow transition to the end of the piece, so the imagery in the picture depicts a slow and blurred process from complexity to simplicity, reducing the number and speed of the interchanging pictures. Simultaneously, the combination of the content and speed of the pictures is achieved by the effect of murmur and quietness.

4.2 The Beginning of Autumn – Reverie of The Chinese Qixi Festival (立秋-七夕遐思)

In the Chinese solar calendar, Liqiu (立秋), indicates the very beginning of the Autumn season. The title of this piece does not only describe the Autumn scenery but adds the Chinese mythology of The Cowherd and Weaver Girl love story (牛郎织女). Therefore, this Prelude is inspired by a fairy tale essence. At the end of the piece, the composer also wrote a narrative: “On the night of Qixi Festival, one was resting outdoors and looked up to see stars twinkling in the galaxy. Did the Cowherd and the Weaver Girl meet each other on this day? A reverie arose…”. Like Debussy’s Prelude La Fille aux Cheveux de Lin, in this Prelude, I also chose the visual elements to build the basic forms, such as maple leaves, a starry sky, and larks.

The whole piece is structured in a Ternary Form. According to this structure, I also prepared the video to consist of three parts, which correspond to the musical form.

Part I (bars 1-9) is also the introduction, written with two phrases, each of which consists of a melodic motif, a cadential-like phrase, and a chordal harmony. Therefore, the different musical elements were framed in the images of the video by different figures and in different rhymes. In the score, the thematic motif is in the F sharp Jiao mode and the F sharp Gong mode as shown in the example:

The acoustic effect of this combination of two modes creates a bleak, mournful emotional atmosphere. In the imagery, it was necessary to avoid bright colours and to focus on the sudden cold, and clear sense of the atmosphere. Maple leaves were chosen for the visual material.

When the melodic motif is connected to the cadential-like phrase, the flowing movement pushes the musical thought to an improvised rhythm. Therefore, the visual imagery was selected to get material, which creates a sensation of movement, captured by the visual effects of slowly falling maple leaves, landing, and floating on the water.

In order to visually highlight the harmonies in the music, the base materials are overlaid. Technically speaking, the contents of the two images were copied separately. The video was then played in reverse, overlapping the original visual image, which added a sense of harmony to it without destroying the theme. As shown in the pictures:

Part II (bars 10-58) consists of three sections A, B, and A1. Section A (10-25) is a narrative of the melodic theme, in F sharp Jiao Gong mode. There are three lines in the score, in which the first line of the soprano showing the rhythmic gestures like ; the second line of the soprano showing the main melodic progression of the theme, like ; the third line in the bass showing the harmonic progression like , and keeping up with the theme.

In order to correspond to this overlaying structure in the video framework, a background of a starry sky was chosen. The faint and twinkling stars juxtapose the maple leaves. The maple leaves, the Milky Way Galaxy, and the waning moon (which symbolises the sadness and the inability to reunite) highlight these three lines of the score. Multiple thematic elements were superimposed to vitalise the visual content, and editing software was used to process the harmony of the picture and the light, ensuring the projection into the main, and secondary material.

Section B (bars 26-37) differed from the previous one, reverting to a three-part Canon polyphonic writing. In the overall sound of this section, there is a higher degree of fluidity and liveliness. On the visual side, the colouration of the picture becomes more complex. Continuing from the previous thematic materials, a new element is added: The Lark. It is added together with a colour adjustment, as shown in the picture.

Section A1 (bars 38-58) reproduces the return of the texture, material and tonality of section A. The climax of the piece is reached with an extension of the pillar chord, in bar 54. Therefore, the rising moon is used as the base image, while the falling leaves are added in. This image has an echo effect to correspond to the recapitulation of the music, placed in bar 55. As shown in the picture:

Part III (bars 59-75) is considered as a Coda of the whole piece. The material of the introduction part returns to here, to create an echoing effect. Thus, the visual images are mostly the same for the most part. There is a dynamic change in the last 9 bars of the music, a melodic line arising from pp to fff. In my opinion, the music expresses emotions of nostalgia and strong attraction, therefore, in the video, the image of a shadow of lovers holding hands is placed in the starry sky background. This expresses both a sense of longing and a deep regret from the darkness of the background, thus corresponding to what the composer wrote in the narrative.

4.3 Rainwater – The sound of rain dripping down in Spring (雨水-春雨淅沥)

In the Chinese solar calendar, Rainwater represents the period of wet weather at the beginning of Spring. It is said that the rains of spring are as precious as oil. The composer expresses the joy of the farmers at the end of the piece: “As the proverb goes, spring rain is as precious as oil, although the noise of the raindrops is hardly annoying, the peasants hear in it the message of a good harvest.” Thus, the elements of the visual thematic material for this music can be identified as the Spring rains.

The piece is structured in Ternary Form, which can be divided into three mains parts with a corresponding instruction and coda.

The rhythm of the introduction (bars 1-7) ranges from sparse to dense, with a raindrop-like motif in the soprano voice. This motif is created by 2nd minor jumped to octave bellow, always falling in the third beat of each bar. The lower voice keeps a steady harmony, but with a chromatic downward movement in the base notes.

The harmonic progression and melodic texture in this part are simple, it therefore cannot be too visually complex. The dynamic visual of a drop of water is used to imitate the feeling of falling rain, seen in the picture below. It was a crucial element to ensure that each moment of the water drop was adjusted in the video to correspond to the last beat of each bar in the music. To do this it was necessary to select Speed Changes in Curves in the editing software. Then Custom Speeds was selected to record the moment in the editing software. When the last beat of each bar occurs according to the time played. In this way the music is synchronised with the video.

In part A (bars 8-23), the remaining bars consist of a harmonic and melodic texture with similar material. It is all melodic development by continuous triplets and fills each bar with a sustained bass, as shown in the example is the subject material of Part A. The continuous triplet runs can describe the feeling of falling rain. Thus, the picture of rain dripping is mainly shown in the image.

Between the sentence divisions of each of the 4 bars, it is necessary to switch and adjust the image, for example, to fill the look successively with the elements of Spring, such as green leaves and sunshine.

In Part B (bars 23-39), the melodic line is concentrated to the left hand, with the right hand being a continuation of semiquavers, imitating the falling of the raindrops.

The phrase division is based on eight measures, with two phrases in total, so the images are changed twice. In the image frame, the image is zoomed into rain falling onto the shoots of new newly sprouting crop field. The colours are more vivid embracing rain and sun, which portrays the emotion of joy.

In Part A1 (bars 39-57) there is a recapitulation of the A. The overall texture and harmony are the same as in part A, but with a 2-bar extension (bars 56 to 57). The harmonic function changes in IV-V-I, thus giving a calm and quiet sound effect. In connection with this effect, the visual image portrays the impression that the weather is gradually clearing up, with the rain starting to fall lighter. Consequently, the picture focuses only on the plants in the sunlight and the lighter rainfall.

In the Coda (bars 57-61), both the melody and the texture are derived from the introduction, therefore, the same material as the introduction for the visuals is chosen, forming an echo from the beginning to the ending.

4.4 Des pas sur la neige

The title of the Prelude Des pas sur la neige gives a very clear visual image such as snow and footprints.

The whole piece is structured in a Binary form, consisting of the introduction and Part A, bridge phrase and Part A1, and the Coda. The form is relatively free between all the parts.

Beginning with the 4-bar, the piece’s introduction is in the key of D minor, and the motif that appears in the first bar is a fixed figure throughout the piece, as shown in the below example.

The dynamic of the first bar is marked pp, they enter very slowly, extending by one beat each time they appear. This motif creates a very clear sensation of the footprints in the snow-filled landscape, which portrays feelings of cold and an emotional loneliness.

In Part A (bars 5-15), the melody of the first phrase is built on D Dorian mode, which creates an atmosphere of regret. The melody of the first phrase (bars 5-7) is not developed, and here the lower line of the musical texture begins to be thickened by the chords with extended positions, which fills a harmonic tension to the music, combined with the intermediate line of the footprint motif and with the upper melodic line.

The second phrase is written entirely in a highly chromatic form. It contrasts sharply with the first phrase. The texture is increased to four voices, with the melodic voices placed in the bass range below the fixed figure. In addition to the fixed figure voice, the other three voices are in different rhythms.

In this part, since the musical lines are overlaid, multi-layered images are required as well. However, as the overall atmosphere of the piece is cold and lonely with a snowy landscape as the main scene, a video overlay would have destroyed the overall visual cleanliness. Therefore, multiple levels of images are found in the same scene. As shown in the picture below, from top to bottom, the sun, the clouds, the wind-blown snow are all visible. These four images are clean and simple in terms of colour, and all make logical sense in the same image at the same time.

In the Connecting phrase (bars 16-19), the material is derived from the introduction, but with some minor extensions in contrast. The bass counterpoint line is added in bar 16, and the soprano melody is added in bar 17. The soprano melody is expanded between bars 18-19, for a more intense mood. As the material in this phrase is an expansion from the introduction, it is important that the visual material be similar to the introduction.

In Part A1 (bars 20-31), the first phase (bars 20-25) expands on part A, building on the F Phrygian mode. The second phase (in bars 29-31) is introduced by the chromatic passage, with the parallel triad replacing the fixed figure. This part is a variation of the first part, therefore visually similar, with only subtle changes in bars 29-31. The images used are only snow and wind blowing to convey the high and low voices in the music, as well as a foreshadowing before the arrival of the Coda.

In the Coda (bars 32-36), the footprint motif is moved to the soprano and by the parallel octaves and finished on the D minor chord. In this section, the visual image of the stopped clock is added in and layered over the snow-covered landscape. This effect comes to reveal the psychological meaning of the music, which expresses a sense of breaking reality, bleakness and cold, while responding more to the title.

Conclusion

Based on the analysing and performing experiences of the Piano Preludes written by C. Debussy and S. Liao, and in the recreation of the music in video clips, this research proved that when an optimized connection and interpretation between music and image is activated and merged, new sensations and imaginary reactions can arise and bring new meanings to the multimedia music performance. This study also demonstrated that from the pianist’s point of view and using synaesthesia to interchange the aural and visual senses, a mutual complementation of various audio-visual components is developed, such as narrative, texture, colours, rhythms, harmonies, and so on.

The study was able to obtain a practical result of music performance and video creation, achieving a bidirectional connection in the performative action, placed in a particular space and time of communication. In a broader view, the aspects of interdisciplinary artistic fusion in performance were highlighted in this study, which can be useful to explore artistic research in the empirical studies that cross different cultures contexts and conducted to further test and refine these results.

Bibliography

Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press.

Lipscomb, Scott D., and Tolchinsky, David E. 2005. The Role of Music Communication in Cinema. In Musical Communication, edited by Dorothy Miell. Oxford: Oxford University Press.

Chion, Michel. 1994. Sound on Screen. New York: Columbia University Press.

Stevens, Meghan. 2009. Music and Image in Concert. Sydney: Music and Media Press.