Hearing color, seeing sound: The relation between sound and visual, and music and art

Bernarda Nibera Conič

SYNOPSIS

The article explores the parallels between sound and image, the differences between visual and auditory perception, the correspondence between art and music theory concepts, and the connections between visual and auditory in art. It compares the elements of art theory and the components of music theory. It defines the importance of a modern interactive audiovisual installation and illustrates this with a concrete example.

KEYWORDS:

visual and auditory perception, visual and sound, art theory, music theory, audiovisual installation, synesthesia, color, and music

MENTORS:

Dr. Petra Černe Oven, Sašo Sedlaček

INTRODUCTION

The concept that colors could be heard and sound could be seen seems impossible at first glance. But at the same time, the question arises. Why have we been connecting visually and auditory into unities for many years, and why so many scientists have tried to solve this riddle. I do not think great scientists like Aristotle, Newton, Schellen, and others would waste time with the “impossible” theory.

Exploring the relationship between image and sound is a topic that encompasses the field of visual and auditory, compares the areas with each other, and seeks interrelationships. This kind of topic opens up many questions for us, but does it also give us clear and accurate answers? Audiovisual correlations have been researched by many scientists, mathematicians, physicists, philosophers, artists, musicians, doctors, and psychologists.

Many studies about audiovisual connection have been done in history, and the fact is that these studies differ from each other. Many have been looking for ways to translate sound into visual and vice versa, but the topic still lacks complete and fixed results. The translation is possible through various approaches, and most such translations in history have been done through laws of physics and psychological aspects.

The problem with audiovisual correspondence is the discrepancy in the results. The results of translations vary from scientist to physicist and from psychologist to artist. Physicists are trying to translate musical tone into colored light through electromagnetic waves and frequency. Psychologists translate sound into color through experience, feelings, and emotions. Furthermore, there are synesthetes who simultaneously experience a feeling of a specific color while listening to music, even though the eyes have not been exposed to visible sensory stimuli. This phenomenon is called synesthesia and is a rare neurological phenomenon that causes the brain to mix information about sensory perceptions and automatically connect them. There are several types of synesthesia, but here we focus on people who perceive colors while listening to sound and vice versa.

The synesthete was also the famous composer Alexander Scriabin, who claimed to see colors while playing music. He patented his system of translating musical tones into colors and a light piano, which emits the appropriate color light when a particular piano key is pressed. Such an instrument is called the Color organ in the audiovisual world. Alexander Wallace Rimington first defined the term Color organ, and quite a few such devices or instruments have been made throughout history.

It is essential to know the difference between the term Color organ and music visualization. The Color organ creates colors according to musical elements (tone, octave, chord…), and music visualization creates an image according to sound by distinguishing sound levels in different frequency ranges. The color organ was mainly a translation of tones into color through physics or psychology, while the visualization of music is a mechanical or technological translation.

I focus mainly on the connection between musical tones and color in my research. However, it is not negligible to mention the correspondence between the terms used in art theory and music theory. If we place these theories side by side, we find that they borrow terms from each other; color, tone, rhythm, harmony, and composition are used in music and art theory.

There are many relationships between visual and sound in different areas. Correlations exist at different levels and are difficult to determine. Correlations between color brightness and sound volume, analogies between color saturation and color of voice, and relationships between color volume and sound duration are known. Audiovisual translation is extensive, difficult to understand, and somewhat mysterious. There are still no clear answers today, or scientists disagree because the results differ. The fact is that the visual and auditory areas are connected and that the correspondences between visual and auditory exist. However, much more needs to be explored in this area for clear answers.
The connection between the visual and the audio is a complex area. I wonder if a color has its voice and if it is possible that we can assign a color to a sound in the form of a tone. In making such a determination, we must begin to investigate the very origin of the visual and the auditory. In order to get the meaning and significance of visual and auditory connection, human is crucial here.

Human perceives light, color, noise, and sounds through the senses … The perception of sound and colors is a highly complex theory.

Graphicist and pedagogue Črtomir Frelih, musician Marjan Kneževič, painter Peter Ciuha, and musician and philosopher Božidar Svetek worked on the theory of the connection between sound and visual on Slovenian soil. Around the world, such a topic has been researched by Aristotle, Newton, Goethe, Helmholtz, Scriabin, Ostwald, Munsell, Kandinsky, Gombrich and others.

VISIBLE AND HEARING PERCEPTION

Knowledge and understanding of the senses and the nervous system are essential in order to understand the dependencies of art on the physical, physiological, and psychological laws.

Milan Butina cites the senses as an indispensable intermediate link between the work of art and the viewer. The senses are divided into categories of physical stimuli. There are five basic senses, including the sense of sight and the sense of hearing. Hearing is classified as a mechanical stimulus. The sense of sight is responsible for electromagnetic stimuli. The senses cooperate and support each other. They act kinesthetically. Sight and hearing inform a person about space.

Butina also mentions other senses involved in the perception of visual images. Sense of sight, sense of one’s own body, sense of balance, and sense of touch are involved in artistic creations without the presence of sound.

When sound is present, the sense of hearing is responsible for its perception. Hearing allows us to articulate, design, and convey messages, speak and create music. hearing transmits sounds and noises to us in the environment, so auditory messages are indirectly related to the human spatial world. Butina says that speech and music expression are much more free and independent of the physical world than visual art. However, the artistic senses bring us the most useful information about the outside world. (Thigh , ­2-6)

Descartes believed that nerves are tiny tubes through which a particular liquid substance, spiritus, spreads. The spiritus is supposed to open and close the “door” in the brain and direct it to various organs. If we replace nerve fiber tubes and doors with brain centers in Descartes’ model, the display is surprisingly similar to recent research. Descartes lived in the 17th century. (Pečjak, 27-33) 

The task of the senses is to understand and recognize what is happening around us and to respond appropriately to the situation. The ears or hearing aid must properly arrange and understand all the received audio signals at the moment. By analyzing higher harmonic sounds, the ear can determine how many independent sound sources it can hear. Each newly detected source means new useful or dangerous information. The harmonic relations of sound do not disturb the sense of hearing, and incorrect relations between frequencies require special attention. Hearing is a sense that never rests while the eye has eyelids. (Ciuha, 3)

However, artistic is not identical to the sensual, visual, and tactile. Visually is everything that comes from the environment into our nervous system through the sensory pathway of vision. The artist’s challenge is how to transfer what the nervous system has sensibly arranged to the canvas and physically embody it. This is what separates the art from the visual. Therefore, art does not have to be a copy or imitation of the visual but an interpretation. (Thigh, 24)

Sight and hearing are essential parts of the body’s senses. Although the color and sound in the frequency bands are different, they are identical in physical attributes because they can be explained by wavelength or vibration. However, studies on the mutual conversion between sound and color image have not yet been so actively done. The sense of hearing and sight have always existed in humans. As we have said, the sound propagates mechanical vibrations through any material medium. However, the frequency of the vibrations is what we feel like the tone of the sound.

On the other hand, light propagates oscillations of electric and magnetic fields. It does not need a material substance in which to expand. The frequency of oscillations of visible light is what we perceive as the color of light. Sound waves detected by the human ear fluctuate between 20 Hz and 20 kHz, while electromagnetic waves detected by the human eye fluctuate between 390 THz and 750 THz. Based on the similarity in physical frequency information between light (or color) and sound, it is possible to mathematically map the sound band’s frequency to the visible range. (CAIVANO, 153 –168)

SYNESTHESIA

Synesthesia = fusion of perceptual disturbances of different sensory areas, simultaneous experience[1]

Synesthesia is a neurological phenomenon, a condition or phenomenon in humans, that causes the brain to mix information about sensory stimuli and connect them. Synesthesia causes a person to automatically experience another sensory stimulus when one sensory stimulus is detected and mixes them. Synesthete hears sounds while tasting, tastes flavors while reading words and numbers, or sees colors while listening to a sound. The cause of this condition comes from the brain, which nervous system is connected differently.[2]

Synesthesia is derived from the Greek words Syn + Aisthesis and means the experience of transverse representations. Irritation of one sense simultaneously causes the perception of another sense. Synesthetic experiences multiple sensory perceptions simultaneously, even though only one sensory organ has been stimulated. Of the various types of synesthesia, color hearing (synopsis) and sound vision (phonopsia) have been the most researched.

Russian painter Vasily Kandinsky lived and worked with synesthesia syndrome. His synesthesia allowed him to hear sounds while seeing colors and to see colors while listening to sounds. There are estimated to be only 2,000 synesthetics worldwide, many more women than men. As the first abstract artist in 1911, Kandinsky removed all recognizable and figural elements from the painting canvas and tried to stimulate the eyes and the ears with colors. While listening to music, he painted many abstract works and thus took advantage of his synaesthetic perceptions.[3]

Scriabin was a synesthete who saw colors while playing music. He compared the sounds with the color spectrum, and after that, he arranged the tonal modes in the fifth circle. Scriabin’s color-tone system is not supposed to be limited to his double synaesthetic sensations, but is based deep in cosmological thinking. The basis of Scriabin’s tonal-color system is his subjective aesthetic experience, which no longer belongs to communicated behavior from a more general cultural tradition. In this system, the interrelationships are determined by individual tones with a harmonic flow.

Scriabin’s psychological system of tonal-color relations : (Motte Haber, 22–25)

Scriabin’s work Prometheus contains a part called Luce in addition to the instrumental ensemble. This part is a set of colored lights. At the time of composing, the composer designed an instrument called the Light Piano, and some musicologists also call it the Color Organ. This musical-light instrument had a piano keyboard, and each key was a switch for a specific color light. When the piano key was touched, a specific colored light was lit synchronously, and it had to shine in harmony with the corresponding note template. (Stanic, 2-5)

JL Hoffmann was the first to describe so-called color hearing. Behind him are well-known synesthetics: Goethe, A. de Musset, Baudelaire, Grillparzer, Morike, Heine, Th. Gautier, JK Huysmans, Rimbaud, Maupassant, L. Ganghofer and Kandinsky. Even some great musicians have associated some tones and melodies with color feelings. Among them are the famous Bach, who associated some tones with yellow and orange, Beethoven with black, Haydn with blue, Mozart with light blue, Schubert with purple, Liszt with purple, Wagner with juicy green, and Chopin with light green. Rimsky-Korsakov connected C, D, A, F, and F major with white, yellow, pink, green, and gray-green. Scriabin associated these same Duras with red, yellow, green, pink, and light blue. Listening to the sound of the flute, Kandinsky felt light blue, dark blue at the forehead, green at the violin, red at the fanfares, cinnabar at the drums, orange at the bells, and purple at the sound of the horn. Wellek also found more characteristic, definite, and complex associations in specific individuals. (Adlešič, 155 –164)

COLOR AND SOUND

Sound is energy, and energy has colors. If a person can listen to music, he can hear or see its colors. In 1997, Marjan Kneževič and Franci Rihter conceived an entirely new music theory closely related to colors. They developed a whole new musical scale and a new color-based notation system. “I listen to music with my eyes closed. Many years ago, I found that music has color. But it took me ten, fifteen years to put this down on paper. Each tone has its value, depth, volume, and color. I proceeded from this in formulating an entirely new music theory, such as is not known in the world, “said Marjan Kneževič in 1997. The musical notation conceived by Kneževič and Rihter has eight lines, and each line has its color.

Kneževič emphasizes: “Everyone can see music if they have the desire. It is only necessary to change the mindset because thought is also energy. If you think positively and want to accept, you will see. But if you have obstacles in you, you will never see.” Each tone has its color and character. The tone Do is green, the tone Re is red, the tone Mi is green, Fa is indigo blue, So is red, La is yellow, and Si is red.

Seven years later, Kneževič presents a new music theory and explains: “Such music, which carries the energy of sound and light, creates a special information structure or form with its movement. I have developed and am still developing the ability to see music in order to contribute to a deeper understanding and perception of the musical message.” (Kokot, 10)

There are many connections between the perception of colors and tones. However, we feel the colors entirely differently than the tones and sounds. The similar physiological and physical formation of tones and colors cause us to perceive different wavelengths or frequencies in the field of hearing different heights and in the field of vision differently colored. Sounds and colors are shown to people differently because pure tones are rare in nature, just like pure and single-color colors. (Adlešič, 155 –164)

ART THEORY AND MUSIC THEORY COMPARISON

In the professional article Music as a Source of Artistic Creation (the relationship between musical and visual language), Črtomir Frelih explains the connection between music and fine arts. He says that we find many expressions from the world of sounds and colors in an everyday speech, which we borrow from each other to describe the neighboring language area’s contents, conditions, and characteristics in a more professional way. Expressions that speak of the connection between sound and art: screaming colors, color harmony, harmonious colors, silent colors, light tones, composition, voice color …

Frelih hypothesizes that musical and visual language are two different languages with specific laws. He says direct translation is not productive. The translation is possible based on analogies and equality of human response to external stimuli (sharpness/softness, aggression/harmony, cold/heat…) (Frelih, 10 –16)

If we briefly restore the essence of art theory, we can list the essential art elements. Butina calls them the basic categories of artistic thinking: shape, point, line, value, and color. From the point of view of art theory, these concepts are primarily expressive and formative means. For these artistic means of expression to work, art practice must unite all these components into a whole, which we call composition. (Thigh, 28)

Visual composition means composition and structure when we talk about the completed work. In artistic composition, we talk about the interaction of artistic elements and variables and syntax of its parts. Within the composition, we compare the present artistic elements. The variables determine the connections between them and the laws of syntax. In the syntax itself, we use organizational principles, which we call the principles of art order or compositional elements. The principles of art order are proportion, balance, rhythm, contrast, harmony, domination, and unity. (Šuštaršič, Butina, Gleria, Skubin, Zornik, 204)

The musical components are made of basic musical materials. These are the types of tones, sounds, and noises. The first musical component is a melody that forms a sequence of tones of different pitches. The second component is harmony, representing the simultaneous ratio of several voices. Rhythm is an ingredient formed by different tonal durations. The meter represents the ratio of accents, the tempo determines the execution speed, and the intensity of execution determines the dynamics. We also have the sound’s color determined by choice of instrument or human voice. Musical element tone also has its characteristics: height, strength, color, and duration. The number of oscillations per second determines the pitch, and the regularity is called frequency. The hertz unit is used to measure the number of oscillations per second. Lower tones have fewer oscillations, while higher ones have higher ones. The tone’s strength, power, or volume is measured with the unit phon. A larger amplitude of oscillation also means a stronger tone. (Mihelčič, 5-24)

ART ELEMENT COLOR

Color is an outline art element that is divided into primary and secondary colors. We are talking about color substances here, so we are talking about subtractive color mixing.

In colorimetry, we divide colors into three dimensions:

The color tone tells us what color it is. Color brightness depends on the power of light energy. However, the saturation of the color hue depends on the purity of the light wave. Colored lights are the purest, and among the colored substances, the purest are most similar to colored lights.

Colors are mixed in two ways. Colored lights are mixing additively. The amount of color light adds up, so mixtures of these colors are always brighter than the original color lights. Color substances are mixing subtractively. Each color substance absorbs some colored light, so the mixtures are darker than the base colors. (Thigh, 30)

MUSICAL ELEMENT TONE

We distinguish between relative and absolute pitch. The relative pitch is used to consciously determine the composition’s rhythmic, melodic, and harmonic course. In relative intonation, we use unique syllables that arise in the tonal system from the first level of tonality. The music theory deals only with absolute intonation or absolute naming of tones, which have a specific pitch and a constant relationship with the chamber tone. The constant naming of notes justifies the absolute pitch of the tones. We know the primary tones: C, D, E, F, G, A, H, (c). The theoretical basis of the tonal system is the basic type of tones. Tone names have white keys for key instruments. The piano has 52 white keys, which means 52 basic tones. Each base tone, however, can be raised or lowered. New tones are raised or lowered base tones. After seven basic tones (C, D, E, F, G, A, H – c) the eighth appears as a repetition of the first. As we now know, the octave is the distance between the first and eighth tones. The basic tone range is repeated seven to eight times in the musically usable range, meaning that the usable tone range has just over seven octaves of range.

RHYTHM IN MUSIC

The primary dimension of music is time. Music is measured by the time we listen to in space. The time course of musical events is determined by specific musical components: rhythm, meter, and tempo. Rhythm is the successive repetition of at least two contrasting sound elements. It accompanies us in everyday life. It is a natural phenomenon. Rhythmic movement can be observed in the heart’s beating, the exchange of day and night, the exchange of seasons, and others. (Mihelčič, 5-24)

Rhythm is the speed of performing tones that connect into rhythmic groups. There are many rhythms, as each group of long and short tones can have its rhythm. It is essential that we hear the rhythm as an exchange between long and short tones.

RHYTHM IN ART

In a work of art or composition, rhythm means the movement of repeated, changed, and intensified elements. There can be elements in a composition that are repeated or alternated in different relationships. They can be the same, similar or different. In fine arts, rhythm is tied to space or plot. Within the composition, it is expressed by straight and curved lines, in the direction of their course and in repetitions and escalations of contrasting units between colors, surfaces, lights, forms, spaces, between concave and convex volumes, etc. (Šuštaršič, Butina, Gleria, Skubin, Zornik, 304)

HARMONY IN MUSIC

Harmony is a Greek word and means consonance. In music, harmony means connecting tones into chords and connecting chords to each other. All sounds, not just tones, tend to connect. They sound together in different ways. When the choir sings, a harmony of voices are created, while the guitar strings are strummed, a harmony is created between the strings, which produces a harmony of tones. When at least three different tones sound together, a chord is formed.

HARMONY IN ART

Harmony in the art world means the coexistence of art units in a composition. It represents the compliance, coherence, and harmony of the composition parts. Harmony calms and harmonizes the contradictions between artistic elements. The concept of harmony is associated with the syntax of similar elements of art. The similarity lies between two extremes, equality, and diversity. In addition to monotony and contrast, harmony represents the third concept of the principles of the artistic order. It lies between two extremes, monotony and contrast, and moderately combines the characteristics of both. The characteristic of harmony is that it eases the contrast but does not erase it. (Šuštaršič, Butina, Gleria, Skubin, Zornik, 314)

CORRESPONDENCES

If we place art theory alongside music theory, the theories use identical terms but have different meanings. The concepts are the same in word and not in meaning. Both theories use the word color; in art, it is the color of light or substance, and in music, it is the color of voice or tone. They also share the word tone. The tone in music is the sound caused by the instrument, while in art theory, it is a light or color tone. Here we must not forget the term composition, which means the arrangement or placement of visual elements in a piece of artwork. In music, it is a musical composition, a fully arranged musical notation of the composition.

I see exciting correspondences between the principles of art theory and musical components. I notice that both theories use the term rhythm. Rhythm in music is based on the relationship of tones in terms of duration and emphasis. Rhythm in the art world represents the movement of repeated, changed, and intensified elements within a composition or a completed whole.

Furthermore, there is the concept of harmony. In visual language, harmony is the principle of the artistic order, which means the coexistence of artistic elements in the composition. For musicians, harmony is the connection of tones into chords and consonance.

CONNECTING VISUAL AND AUDITORY IN ART

In the art world, sound and image are connected in different fields. These areas are difficult to name, as some are predominantly commercial, or the line between artistic and commercial is difficult to draw. For example, it would be difficult to classify a video or film in the field of audiovisual art, as this type of medium is mostly for financially based purposes based on market needs. The video and the film contain a visual display and a musical or sound background, but this does not mean they can be described as audiovisual art. An art video with music or sound inserts falls into this area more than a music video.

Music video represents a range from entirely abstract videos created with an emphasis on color and movement to videos that tell a story. These music videos build on narration and mainly act as short feature films. (Vernaillis, 7-9)

The connection between visual and musical language is found in the moving image in the film. Film or video is a medium that combines the temporal dimension of music with the spatial dimension of visual art. Sound support for moving images is intended for experiential and mood effects. The film stimulates several senses simultaneously and enhances the experience analogously. In most cases, it is the medium in which the image is in the foreground and complemented by sound effects, while in the music video, the sound is the main one and is enriched by visual displays. In the film, the sound is made subordinate to the image, while in videos, the image is created subordinately and depending on the music. (Frelih, 10 – 16)

We can undoubtedly classify these fields into the audiovisual art genre: art film or video, animated film, performance, audiovisual installations … Installations are often associated with interactive art and thus get a new art form, which can be called an interactive audiovisual installation.

INTERACTIVE AUDIOVISUAL INSTALLATION

The term audiovisual (AV) means the simultaneous use of audio and visual components such as video presentations, films, television programs, live theater performances … 

Computer-aided audiovisual equipment is widely used in education; many schools and universities install projection equipment and use interactive whiteboard technology. Another audiovisual term is the visual representation of sound (visual music).[6]

Audiovisual art explores kinetic abstract art and music or sound combinations in relation to others. These include visual music, abstract film, audiovisual performances, and installations.[7]

Installation is an artistic genre of three-dimensional works placed in space and designed to transform the perception of space. This term is generally used for interiors, while outdoor interventions are often referred to as public art or intervention art; however, the boundaries between these conditions overlap.

An interactive installation often involves an audience acting on a work of art or responding to user activity. There are several types of interactive installations created by artists. These are web installations, gallery installations, digital installations, electronic devices, mobile devices, etc. Interactive installations appeared in the late 1980s and became a genre in the 1990s when artists became particularly interested in engaging audiences with an art object and activating and revealing the meaning of the installation.[8]

So if we summarize and connect these ideas, an interactive audiovisual installation is a three-dimensional art object in space that contains both visual and sound components. The audience or viewers participate, or the installation responds to the movement or interaction of the audience.

Some of these installations, which are more related to musical elements than sound, are also called an instrument or color organ (Color organ).

EXAMPLE OF A MODERN INTERACTIVE AUDIOVISUAL INSTALLATION: SPECTROZATOR

Spectrozator is an interactive audiovisual installation created by interdisciplinary artist Bernarda Nibera Conič at the Academy of Fine Arts and Design under the mentorship of prof. Sašo Sedlaček and prof. dr. Petra Černe Oven. Spectrozator is a project that explores the relationship between visual and auditory, draws parallels between sound and image, seeks the differences between visual and auditory perception, and discovers the possibilities of translating a musical tone into a color hue.

Spectrozator is an interactive audiovisual installation whose surface varies according to the spectral color unit determined by a specific sound. The sound is triggered through a synthesizer, whose task is to control the installation. The synthesizer acts as a tool to trigger sound and light simultaneously. Color light responds to sound recording and sound frequencies coming from the environment. When the light and sound condition is enabled, the Spectrozator can operate. Without light, sound and human presence, the installation does not exist.