Synesthesia Art and Media

Heis, Synesthesia, Art and Media 1

Directed Research Report on

Synesthesia, Art and Media

Kevin Heis June 13, 2008

Background Synesthesia (sin-əs-THEE-sha) is a neurological condition in which one sensory function can simultaneously and inadvertently trigger other sensory functions. A synesthete, or a person who has synesthetic experiences on a regular basis, can experience taste from sounds, color from words and letters, and physical feeling from visual stimuli, stimuli , among other possible combinations. This report on my research will discuss some background on synesthesia before focusing on the artistic implications of synesthesia, and will wi ll end on my own composition, Chroma Improv, that was inspired by synest syne sthetic hetic works.

Despite historic skepticism, synesthesia is a real, diagnosable condition in one in about two thousand adults.1 It is much more common in adult females than adult males, by about a six to one ratio. 2 Many neurologists who have studied synesthesia believe that all, or many, children are born synesthetic and gradually lose it as they get older.3 Formal research studies on synesthesia began with Richard Cytowic in the 1980s, and modern research using brain imaging technology gives little doubt that the condition is very real for synesthetes.4 1

Oliver, Sacks. Musicophilia: Tales Tales of Music and the Brain. B rain. Knopf, 2007.

2

Oliver, Sacks. Musicophilia: Tales Tales of Music and the Brain. B rain. Knopf, 2007. B. B. “When Brains Wring Colors from Words.” Science News, 2002. Vol. 161, No. 12. pp. 189.

3


4

Van Campen, Cretien. “Artistic and Psychological Experiments with Synesthesia.” Leonardo, 1999. Vol. Vol. 32, No. 1. pp 9-14. Oliver, Sacks. Musicophilia: Tales Tales of Music and the Brain. B rain. Knopf, 2007.


Cytowic’s five criteria for diagnosing synesthesia have remained as the standard. The five criteria are: 1) it must be involuntary, 2) the synesthete realizes that the experiences are not happening in reality, reality, 3) the experiences must be long-term, consistent, and simple in correlation, 4) the synesthete must be able to remember and recall the experiences, and 5) the experiences must be emotional.5 This differentiates from a multisensory experience in a few ways: a multisensory experience can be voluntary, it can be substantiated in reality, it can happen as many times as the experience occurs, and the experience does not need to be eithe r memorable or emotional. Perhaps one in two thousand sounds like a very rare condition not worthy of research, but in reality, even if a person is not a diagnosable synesthete, it does not invalidate the possibility of synesthetic experiences. Some studies suggest that about one in twenty-three people have had memorable synesthetic experiences. 6 And, a 2006 study reported in Discover magazine found that given a choice between animations made by synesthetes and non-synesthetes, most people found the animations created by synesthetes were better synchronized. This means that even regular people can link different senses unknowingly, a view also held by Cytowic.7

5

Cytowic, Richard E. The Man Who Tasted Shapes. New York: G. P. Putnam's Sons, 1993. Cytowic, Richard E. Synesthesia: A Union of the Senses. New York: Springer-Verlag, 1989.

6


7

Garfield, Kathryn. “Are We All Synesthetes?” Discover, Dec. 15, 2006.


Perhaps then, synesthetic experiences can be induced or learned by nonsynesthetes. Researchers are looking into several avenues for inducing or learning synesthesia. Examples of this are through odor-taste synesthesia, which seems to be the most natural form of it,8 drugs like cannabis and opiates, and in particular LSD,9 certain forms of mediation,10 and more extreme methods including sensory deprivation, where having one sense shut off incurs synesthetic experiences, epilepsy, and electrical stimulation of the brain.11 All of these forms, to varying degrees, have been found to induce synesthetic exp e xperiences eriences in non-synesthetic people. Also, researchers have found a person’s quality of vision can be dramatically altered in quality by what they hear, as well as the interpretation of visual events and motion especially in terms of time. 12 There are three traditional, and four more modern, theories about understanding how synesthesia works. The three traditional theories are miswiring, leakage, and wrong feedback; the more modern view vie ws are via the limbic system, neonatal pruning, which means everyone has it and loses it as we get older, brain plasticity, which uses the argument that the fact the brain can replicate lost functions may explain synesthesia, and disinhibition.13 Richard Cytowic, one of the

8

Calvert, Gemma, Charles Spence, and Barry E. Stein, eds. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004. Chapter 5.

9

Van Ca Campen, Cretien. The Hidden Sense: Synesth Syne sthesia esia in Art and Science. Cambridge: MIT Press, 2008. Chapter 7

10

Van Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008. Chapter 7

11

Cytowic, Richard E. The Man Who Tasted Shapes. New York: York: G. P. Putnam's Sons, 1993. Chapter 16

12

Calvert, Gemma, Charles Spence, and Barry E. Stein, eds. The Handbook of Multisensory Processes. Cambridge:

MIT Press, 2004. Chapter 2. 13

Van Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008. Chapters

8 and 9


leading researchers on synesthesia, says synesthetic experiences look similar physically to migraines, in which sensory distortion causes disconnect between different brain functions. 14 Cytowic believes more so in the more modern theories of synesthesia, particularly in neonatal pruning, writing that the condition exists in other primates and calling synesthetes simply ‘more aware’ than the general population.15 The focus of my research is not on the neurological or psychological research of synesthesia. Rather, it is about the artistic and media implications of synesthesia. It is also about the artists, both synesthetes and non-synesthetes, who have created works that explore the synesthetic phenomenon.

Common Ground It is important to recognize when discussing synesthetic artists is that all synesthetes have unique correlations between the senses, no two synesthetes are the same.16 When discussing specific artists later on in this report, I will go into detail on their own connections, but first I would like to discuss the similarities, to establish common ground between synesthetic artists. Influences on synesthetes are both external and internal. And with imagination and creativity being linked together, and synesthetes being seen as imaginative by the general population’s standards, synesthetes are viewed as

14

Cytowic, Richard E. The Man Who Tasted Shapes. New York: York: G. P. Putnam's Sons, 1993. Chapter 20.

15

Cytowic, Richard E. The Man Who W ho Tasted Shapes. New York: G. P. Putnam's Sons, 1993. Chapter 20.

16

Oliver, Sacks. Musicophilia: Tale s of of Music and the Brain. Knopf, 2007. pp. 172. Van Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008. Chapters 8

and 9 Steen, Carol. “Visions Shared: A Firsthand Firsthand Look into Synesthesia and Art.” Leonardo, 2001. Vol. 34, No. 3. pp 203-208.


creative people.17 Many synesthetes are closeted, or were at some time, and find hostility when describing their experiences, which also shows up in synesthetic work.18 Internal influences are perhaps even more significant however. Research has found that, for most synesthetes, color is what is triggered by other senses, making it the dominate synesthetic response. Taste and sound as a response are also not uncommon.19 Research has also found that when synesthetes experience conflict between their own internal correspondences and what they literally experiencing, it causes distress in the synesthete.20 Unpleasant color combinations can trigger negative synesthetic experiences.21 Extremity in general has an interesting effect on synesthetes. Lar Large literal changes can cause even greater changes in perception, leading to the idea that synesthesia is effected by context. 22 Loud or extreme sensory information can also blur for synesthetes whether o r not the synesthetic experience is real.23

17

Cytowic, Richard E. Synesthesia: A Union of the the Senses. New York: Springer-Verlag, 1989. Chapter 8

18

Steen, Carol. “Visions Shared: A Firsthand Firsthand Look into Synesthesia and Art.” Leonardo, 2001. Vol. Vol. 34, No. 3. pp

203-208. 19


8 and 9 20

Haack, Paul A. and Rudolf E. Radocy. “A Case Study of a Chromesthetic.” Journal of Research in Music

Education, 1981. Vol. 29, No. 2. pp. 85-90. 21


203-208. 203-208. 22

Hubbard, Timothy L. “Synesthesia-like Mappings of Lightness, Pitch, and Melodic Interval.” The American

Journal of Psychology, 1996. Vol. 109, No. 2. pp. 219-238. 23

Steen, Carol. “Visions “Visio ns Shared: A Firsthand Look into Synesthesia and Art.” Leonardo, 2001. Vol. 34, No. 3. pp

203-208.


There are some interesting observations made about synesthetic work, of course these are all generalizations. Synesthetes tend to prefer soft to well defined shapes,24 and prefer to keep their work simple as opposed to complex, using black or white backgrounds, keeping their source material restricted to a smaller amount, and the end result is typically more asymmetrical. 25 Finally, linguistics may influence synesthetic perception as well, for example the phrase ‘red hot.’26 Again, all synesthetes are unique in their correspondences between senses. However, However, there are some things that are common in association. Lighter colors are associated with higher pitch, darker colors are associated with lower pitch. 27 The vowels ‘u’ and ‘o’ are darker, ‘a’ in the middle, and ‘e’ and ‘i’ are typically lighter in color.28 Another way of associating pitch with color: colors tend to st art art at red on the note ‘C’ and move through the col c olor or wheel until green at the note ‘G’, and are shades of pink and purple thereafter. 29 Matching qualities are bright, smooth, hard, cold, and and sharp; another set is dull, rough, soft, blunt, warm, and heavy. heavy.30

24


203-208. 25

Hubbard, Timothy L. “Synesthesia-like Mappings of Lightness, Pitch, and Melodic Interval.” The American


Hubbard, Timothy L. “Synesthesia-like Mappings of Lightness, Pitch, and Melodic Interval.” The A merican merican


Hubbard, Timothy L. “Synesthesia-like Mappings of Lightness, Pitch, and Melodic Melodic Interval.” The American



8 and 9 29

Brougher, Kerry, Jeremy Strick, Ari Wiseman, and Judith Zilczer, eds. Visual Music. Los Angeles: Thames &

Hudson, 2005. pp 213 30


8 and 9


Perhaps a connection between synesthesia and the general population is the use of ratios in media and art. From Cytowic’s earlier book: design principles, like the divine proportion and thirds, as well as musical temperaments, were not forced but

developed

intuitively.

Earlier

societies

gave

shapes

specific

color

correspondences, like yellow triangles, blue circles, or red squares, and these combinations are also seen in renaissance artwork. Ratios and numbers are associated with particular shapes, like multiples of three for triangles and the pattern 2, 4, 16... for squares. These ratios across perceptual boundaries have ha ve actually been found to be similar in correspondence for synesthetes. 31 With that, let’s move on to synesthetic artists and the artwork that has been make in the name of synesthesia.

Artistic Synesthetes and Synesthetic Art Prometheus: Poem of Fire (1910), a symphonic poem by Alexander

Scriabin, is one of the most famous synesthetic works. Prometheus is about twenty minutes in length, and is the story of an early man stealing fire from the gods. It makes use of a color organ which outputs both music and colored lig l ight ht in specified shapes.32 The instrument did not exist at the time the work was composed. 33 Scriabin articulated in the score how the colors and shapes corresponded with the music. The colors and shapes correspond with musical keys and not by note,

31

Cytowic, Richard E. Synesthesia: A Union of the Senses. New York: Springer-Verlag, 1989. Chapter 8

32

Van Campen, Cretien. “Artistic and Psychological Experiments with Synesthesia.” Leonardo, 1999. Vol. 32, No.

1. pp 9-14. Berman, Greta. “Synesthesia and the Arts.” Leonardo, 1999. Vol. Vol. 32, No. 1. pp. 15-22. 33



moving around the circle of fifths, 34 leading some to call Scriabin not a true synesthete.35 Regardless of that, the mapping Scriabin developed is then used to tell the story of Prometheus, with red signifying fire, blue signifying the gods, and so forth. Perhaps the most significant output of the symphonic poem is the mystic chord, sometimes called the Prometheus chord, that Scriabin used as a foundational structure of Prometheus and at pivotal moments directly composed four times in the work.36 The six note chord is based on a pattern of fourths. While dissonant, it sounds familiar and comforting. Die glückliche Hand , an opera by Arnold Schoenberg was composed

1910-1913, and like Prometheus approximately twenty minutes in length. It is the story of love lost, and infuses themes of artistic criticism and repetition of mistakes. Schoenberg uses colors of lights, costumes, and set pieces to synchronize with the onstage personalities and the plot, using conventions as opposed to internalized synesthesia. The greatest example of synesthesia in Die gluckliche Hand is known as the color crescendo, appearing in the third of five scenes. This

dramatic short sequence rapidly integrates the precedences in color-sound correspondence earlier in the opera, and also synchronizes with the rapid changes

34

A concept in tonal music theory, using an intervallic relationship to map musical keys.

35

Van Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008. Cook, Nicholas. Nicholas. Analysing Analysing Musical Multimedia. Oxford: Clarendon Press, 1998 Galeyev, B. M. and and I. L. Vanechkina. “Was Scriabin a Synesthete?” Leonardo, 2001. Vol. 34, No. 4. pp 357-361. 36

Brougher, Kerry, Jeremy Strick, Ari Wiseman, an d Judith Judith Zilczer, eds. Visual Music. Los Angeles: Thames &

Hudson, 2005.


in emotion of the main character. It serves as a powerful transition in the outlook of the main character in the story.37 Schoenberg was a member of the artistic group Der Blaue Reiter (The Blue Rider), which had at least one truly synesthetic member. Wassily Kandinsky saw deep rooted connections between all the senses and was inspired a variety of sensory information, which he translated into paint. His most famous synesthetic work is call Der gelbe Klang or The Yellow Sound (1912), a theater work involving several other types of artists, where Kandins Kandi nsky ky correlates watercolor painting, lighting, movement on film, dance, and music. musi c.38 Chron Chronochromie, ochromie, Des Canyons aux Etoiles and Couleurs de la Cité Céleste

are examples of orchestral music by the sound-color synesthete Oliver Messiaen (composed 1929 to 1974) that had synesthetic influences. The works are not mixed media, they are only music. Messiaen’s sound-color synesthesia is not as well documented as other composers, but certainly made references to them throughout his works.39 Messiaen was largely a twelve tone composer, and so analyzing his synesthetic experiences from it is complex.40 His music has been found to link chord spacing and absolute pitch with color. 41 The very form of the three works

37

All information on this opera comes from these two sources: Cook, Nicholas. Analysing Musical Multimedia. Oxford: Clarendon Press, 1998. pp41

Crawford, John C. “Die gluckliche Hand”: Shoenberg’s “Gesmatkunstwerk”. The Musical Quarterly, Vol. 60 No. 4., Oct. 1974. pp. 583-601. 38 Van 39

Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008.

Bernard, Jonathan W. W. “Messiaen’s Synaesthesia: The Correspondence between Color and Sound Structure in His

Music.” Music Perception, 1986. Vol. 4, N o. 1. pp. 41-68. 40


41

Cook, Nicholas. Analysing Musical Multimedia. Oxford: Clarendon Press, 1998.


listed above is based on color. 42 Messiaen did not use lighting or other techniques to demonstrate his perception of the generated colors, but they are still useful to look at as they influence his music. While Messiaen’s sound-color correspondences have been found to not be one hundred percent consistent, 43 they are still a valuable tool for analyzing his music. Michael Torke is a living synesthetic composer who primarily composes orchestral works. He primarily associates colors with key signatures and notes in the context of their key signature. 44 His most famous syne synesthetic sthetic composition is Ecstatic Orange, and his other synesthesia inspired works include Green, Purple, Ash, and Bright Blue Music. All of these works have dance choreographed to them

as well.45 Torke interprets these works as dreams in particular colors, exploring the possibilities within each of the colors.46 Another more current example of synesthesia in artwork is the video game Rez. In Rez Rez, as the player interacts with the very abstract and colorful game

environment by shooting enemies and objects, different sounds are generated along with various colors and shapes. The designers and artists behind Rez cite Kandinsky’s Kandinsky’s paintings as inspiration inspiration for creating the game.47

43

Bernard, Jonathan W. W. “Messiaen’s Synaesthesia: The Correspondence between Color and Sound Structure in His

Music.” Music Perception, 1986. Vol. 4, No. 1. pp. 41-68. 44

Oliver, Sacks. Musicophilia: Tales Tales of Music and the Brain. Knopf, 2007.

45

Berman, Greta. “Synesthesia and the Arts.” Leonardo, 1999. Vol. Vol. 32, No. 1. pp. 15-22.

46

Berman, Greta. “Synesthesia and the Arts.” Leonardo, 1999. Vol. Vol. 32, No. 1. pp. 15-22.

Oliver, Sacks. Musicophilia: Tales Tales of Music and the Brain. B rain. Knopf, 2007. 47

Hawkins, Matthew. “Go To To Synesthesia... Jake Kazdal’s Journey Though The Heart of Rez.” Gamasutra, May 6,

2005.


There are many other examples of synesthesia in art and media, including works by electroacoustic composer Edgard Varese, painters Kurt Schwitters, Piet Mondrian, and David Hockney, and the films Fantasia and Dog Star Man. There is also an entire movement in video art called Visual Music, which I cannot even begin to list the number of works created as part of it.

Creating Synesthetic Experiences All of this leads to the question: how would one create a synesthetic piece of art or media? The most obvious is to use the diagnosis of synesthesia as a guide and from what I have written about influences and similarities: •

keep keepin ing g it it simp simple le in corr correl elat atio ion, n,

•

keep keepin ing g it cons consis iste tent nt in in corr correl elat atio ion, n,

•

emph emphas asiz izin ing g colo colors rs and and word words, s,

•

comb combin inin ing g mult multip iple le sen sense ses, s,

•

limi limiti ting ng sour source ce mate materi rial al,,

•

blur blurri ring ng extr extrem emit itie ies, s,

•

and usin using g ratios ratios to to create create the the corre correlat lation ionss betwee between n senses senses

would all make sense. But perhaps there is more we can do, more we can use from synesthesia in art and in media. The academic journal Leonardo had a series of articles about synesthesia and art in the late eighties and early nineties. One article described the requirements requiremen ts of the ultimate synesthetic medium:

it does a wide variety of

mediums, it is spatial oriented, it has an organizational system to keep everything


connected, and it takes full advantage of ability to “capture, generate, process and spatially manipulate” the source material.48 The book Analysing Musical Multimedia is a treasure of knowledge about mixed media aesthetics, and particularly strong at connecting them to synesthesia. The basic question to ask, according to the book, is not to ask what a medium has or is, but what it does to the other mediums involved. This implies making connections between mediums, and to hide the synchronization between mediums, using things like movement, color, shape, and emotion. One medium can significantly alter the time perception of others; particularly in the case of sound, which tends to be more abstract, on visual, which tends to be more literal. Metaphors are one of the best models for multimedia, using compliments instead of contrast, limiting the use of contrast for emphasis. 49 Simply connecting mediums is not synesthetic in and of itself, but the idea that multisensory aesthetics are rooted in synesthesia is a powerful one.

Chroma Improv My composition, Chroma Improv, while I do not consider it a truly synesthetic work, it is inspired by synesthesia and previous work done by synesthetic artists. My focus of the piece is on the relationship between color and sound, and sound in the form of musical sonorities. The piece, as indicated in the title, is an improvisational piece of approximately three to ten minutes. This

48

Mallary, Mallary, Robert. “Spatial-Synesthetic “Spatial-Synesthetic Art through 3-D P ro jection: The Requirements of a Computer-Based Supermedium.” Leonardo, 1990. Vol. Vol. 23, No. 1. pp. 3-16.

49

Cook, Nicholas. Analysing Musical Multimedia. Oxford: Clarendon Press, 1998.


improvisation functions similar to the way a true synesthete would perceive color and sound fusion: on the fly. I do not consider this to be a synesthetic work because the multi-sensory experience is forced onto the audience rather than being internal. The video part is an attempt to create what a synesthete might perceive visually when hearing the sonorities generated by the computer, an aid for non-synesthetes to understand the concept. The setup of the work involves an electric bass player, standing in front of a projected screen, and a camera positioned to view the the front of the bass player. The score is set up in a six by six grid of twelve preselected pitches, with register preselected, and the bass player improvises by moving across the grid in various directions with no predetermined rhythm, volume, articulation, or tempo. As the bass player plays specific notes, the computer categorizes the pitch though a pitch recognition algorithm as one of twelve pitches. The computer then, based on which pitch is currently being played, outputs data for both sound and video. For the sound part, the computer transposes the incoming signal three to six discreet times. It then pulses out the transposed signals individually at a rate determined by which pitch is being performed by the bass player, and each transposed pitch has its own unique pulse rate. This is in addition to outputting the dry bass signal for reference. Perhaps a future version of this work will also use sound filtering to change the timbre of the sound output on the basis of the incoming pitch as well. For the video part, red, green, and blue values on a scale from zero to two hundred fifty-five have been preselected for each one of the twelve possible


pitches, and on the basis of which pitch is currently being played the computer filters out the color of the video signal from the camera on the bass player and outputs the video signal onto the projector. While this seemingly simple concept that synchronizes sonority, pulse rates, and video color may seem too basic to be an interesting composition, it is true to the concept of the work. The end result is more than the sum of its parts, which only aids to prove that synesthesia is something we all experience at varying degrees.

The future, conclusions Synesthesia will grow as an area of interest as our world accelerates towards combining senses for entertainment, for education, and for information. Things like motion sensors, virtual reality, and feedback systems are all on the rise in many different mediums, like films, games, and interactive works. It is possible that because of this, the number of cases of diagnosable synesthesia will increase. The more we understand this unique combination of the senses, the better we can craft mixed media messages and artwork to generate the maximum response in the audience.


Sources B. B. “When Brains Wring Colors from Words.” Science News, 2002. Vol. 161, No. 12. pp. 189. Berman, Greta. “Synesthesia and the Arts.” Leonardo, 1999. Vol. 32, No. 1. pp. 15-22. Bernard, Jonathan W. “Messiaen’s Synaesthesia: The Correspondence between Color and Sound Structure in His Music.” Music Perception, 1986. Vol. 4, No. 1. pp. 41-68. Brougher, Kerry, Jeremy Strick, Ari Wiseman, and Judith Zilczer, eds. Visual Music. Los Angeles: Thames & Hudson, 2005. Calvert, Gemma, Charles Spence, and Barry E. Stein, eds. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004. Cook, Nicholas. Analysing Musical Multimedia. Oxford: Clarendon Press, 1998. Crawford, John C. “Die gluckliche Hand”: Shoenberg’s “Gesmatkunstwerk” . The Musical Quarterly, Vol. 60 No. 4., Oct. 1974. pp. 583-601. Cytowic, Richard E. Synesthesia: A Union of the Senses. New York: SpringerVerlag, 1989. Cytowic, Richard E. The Man Who Tasted Shapes. New York: G. P. Putnam's Sons, 1993. Galeyev, B. M. and I. L. Vanechkina. “Was Scriabin a Synesthete?” Leonardo, 2001. Vol. 34, No. 4. pp 357-361. Garfield, Kathryn. “Are We All Synesthetes?” Discover, Dec. 15, 2006. Haack, Paul A. and Rudolf E. Radocy. “A Case Study of a Chromesthetic.” Journal of Research in Music Education, 1981. Vol. 29, No. 2. pp. 85-90.


Hawkins, Matthew. “Go To Synesthesia... Jake Kazdal’s Journey Though The Heart of Rez.” Gamasutra, May 6, 2005. Hubbard, Timothy L. “Synesthesia-like Mappings of Lightness, Pitch, and Melodic Interval.” The American Journal of Psychology, 1996. Vol. 109, No. 2. pp. 219-238. Mallary,

Robert.

“Spatial-Synesthetic

Art

through

3-D

Projection:

The

Requirements of a Computer-Based Supermedium.” Leonardo, 1990. Vol. 23, No. 1. pp. 3-16. Oliver, Sacks. Musicophilia: Tales of Music and the Brain. Knopf, 2007. Steen, Carol. “Visions Shared: A Firsthand Look into Synesthesia and Art.” Leonardo, 2001. Vol. 34, No. 3. pp 203-208. Van Campen, Cretien. “Artistic and Psychological Experiments with Synesthesia.” Leonardo, 1999. Vol. 32, No. 1. pp 9-14. Van Campen, Cretien. The Hidden Sense: Synesthesia in Art and Science. Cambridge: MIT Press, 2008. Youngblood, Gene. Expanded Cinema. New York: E. P. Dutton, 1970.

Synesthesia Art and Media

Recommend Documents