Jessica Smith

Professor Gendle

Behavioral Neuroscience

April 28, 2018

A Study of Semantics in Synesthesia: How Synesthesia Works Across Languages

Synesthesia is the experience of having the activation of one sense automatically and

uncontrollably activate another sense (Neckar & Bob, 2014). So, for some people numbers have

specific colors or words have certain tastes (Neckar & Bob, 2014). Specifically, grapheme-color

synesthesia occurs when a person experiences colors for certain letters or numbers, and

lexical-color synesthesia occurs when a person experiences colors for certain words (Neckar &

Bob, 2014). A lot of studies, including those by Laura J. Blazej and Ariel M. Cohen-Goldberg

(2016), have shown that for lexical-color synesthesia the color given to the word is most often

based on the first letter of the word, and not on the meaning of the word. Similarly for

grapheme-color synesthesia, the color has been shown to relate to the shape or form of the

letters, and similarly shaped letters have been shown to have similar colors (Brang, Rouw,

Ramachandran, Coulson, 2011). However, there are synesthetes who experience colors based

on the meaning of the word for words very closely related to colors, and for words having to do

with sequences (months, days, years), as shown by Yokoyama et al. (2014). The goal of this

experiment is to look further at the semantic basis for synesthesia. This experiment will look at if

colors remain the same across languages by examining the consistency of colors seen by

synesthetes for the same words in both native and foreign languages.

Synesthesia is an uncommon sensory experience affecting approximately 1 in 1150

females and 1 in 7150 males (Rich, Bradshaw, and Mattingley, 2005). According to a large
 Smith 2

survey conducted by Rich and colleagues (2005), the most common forms of synesthesia

involve colors relating to specific letters, numbers and/or words, which stay relatively consistent

throughout life.

There are many theories concerning the neurological basis of synesthesia. V. S.

Ramachandran (2012), one of the leading synesthesia researchers, proposes that it has to do

with the modularity of the brain. Basically, because seperate parts of the brain process very

specific aspects of stimuli, such as shape or color, the combining of sensory experiences seen

in synesthesia could be due to excess connections between these different modularities

(Ramachandran, 2012). For example, Ramachandran (2012) argues that grapheme-color

synesthesia could be caused by excess connections between the systems responsible for

processing colors and numbers, which are located very closely within the fusiform gyrus. This

could also be the case for chromesthesia, or experiencing colors when hearing certain sounds,

because the mechanisms are also closely located (Ramachandran, 2012).

In further support of the idea of cross-wiring, or excess connections between different

modularities, Tilot and associates (2018) found a strong relationship between genes related to

axogenesis and synesthesia. These genes are associated with the formation of new axons and

are expressed during early childhood when synesthetic associations are formed, and they

strongly support the idea that there are excess axonal connections between different

modularities in those who experience synesthesia (Tilot et. al, 2018). Tilot’s study supports a

genetic component, and somewhat supports Ramachandran’s popular theory that synesthesia

could be caused by less neural pruning between different sensory areas during development

(2012).

Studies have also shown that there are different levels of synesthesia related to the type

of processing involved and the sensory experience ( Neckar & Bob, 2014). By looking at brain
 Smith 3

imaging, findings suggest two separate groups depending on if the synesthetic experience is

expressed in abnormal activation of the visual areas, or abnormal activation of the parietal lobes

(Neckar & Bob, 2014). These separate groups also relate to two different levels of synesthesia,

“ ‘lower’ (referring to lower perceptual processes) and ‘higher’ (referring to higher cognitive

processes), in which the different forms of synesthesia represent different stages of brain

processing” (Neckar & Bob, 2014). Neckar’s findings suggest that some synesthetic

experiences could be determined by lower processing, such as just the shape of the letters, but

others could be determined by higher level processing, such as the meaning of the words.

There have been many studies looking at if the meaning behind the word, letter, or

number effects the color that synesthetes experience. For the most part, there has been little

support for semantic influences in the coloring of numbers or words ( Rich et. al, 2005, Blazej &

Cohen-Goldberg, 2016). Most synesthetes who experience colors for whole words experience

color for individual letters as well, and they usually base colors for whole words on the colors

experienced from the first letter of the words (Rich et. al 2005). Additionally, even nonwords can

be colored based off of their first letter (Blazej & Cohen-Goldberg, 2016). This can support the

idea that the colors are rule-based depending on the letters seen each time and are not stored

and retrieved for each word (Blazej & Cohen-Goldberg, 2016). Studies have even shown that

similar shaped letters produce similar colors, further supporting the importance of visual

processing in the synesthetic experience (Brang et. al, 2011). It is clear to see from the results

of these past studies that most synesthetic experiences are produced by bottom-up processes,

because the colors are related to the shape and form of individual letters. Reading, however, is

a highly top-down process, which could explain why semantics usually don’t play a role in

synesthesia. Although, reading is processed in three distinct levels, the first of which is

pre-lexical and is just the process of perceiving the forms of individual letters ( Blazej &
 Smith 4

Cohen-Goldberg, 2016). The way the reading process works, combined with the fact that most

words are colored based on the forms of the letters, could support the idea that synesthesia

works early in the reading process.

Conversely, there are some studies that do support a role of semantics in color

formation. For example, Dixon and colleagues (2014) showed that an ambiguous grapheme

could produce different colors when presented as a number or letter, showing the importance of

context, and therefore semantics, on the color seen. Also, it is common for sequence words,

such as months, to be given colors that contradict the color that would typically be dictated by

the letters within the word (Rich et. al, 2005). Similarly, Hung and associates’ (2014) experiment

with Chinese characters showed that semantics play a role in color formation. Chinese

characters are usually made of two parts, one for the meaning of the word and one for the

pronunciation. Hung found that both parts play a role in determining the synesthetic color

experienced (2014). Finally, there have been multiple brain-imaging experiments that show that

different areas of the brain light up for different levels of synesthesia, either for semantic-based

experiences or for visual ones (Yokoyama et al., 2014). The results of these brain imaging

studies could show an interaction of latter levels of the reading process and synesthesia.

Overall, there has been no consistent support of the effects of semantics on synesthesia, but

there are many semantically based experiences that need to be explained.

Similarly, there have been studies that show languages do play a role in synesthetic

experiences. People who speak multiple languages have been reported as having synesthetic

experiences across languages ( Rich et. al, 2005). While this could support a semantic effect

considering the people studied speak both languages, the degree of fluency did not affect the

synesthetic experience. This could suggest more of an impact from visual processing because

the synesthetes did not need to fully understand all of the words they experienced colors for (
 Smith 5

Rich et. al, 2005). Moreover, people have reported experiencing synesthesia for languages they

do not speak at all (Rich et. al, 2005), which highly suggests a more visually based system.

The point of this study is to further look at synesthesia across languages to see if there is

more of a semantic influence than currently theorized, as suggested by Ramachandran in his

book “The Tell-Tale Brain” (2012). This study will look at grapheme-color and lexical-color

synesthesia in order to see if the colors experienced are based on the meaning of the letters

and words. Typically these types of synesthesia are measured using the pop-out effect and test

- retest (Ramachandran 2012). The pop-out effect is used to prove that the person’s

synesthesia is a real sensory experience, by showing that the person can identify a

letter/number faster than average because of its distinct color (Ramachandran 2012).

Consistency and validity are tested by test-retest to see if the colors remain the same across

extended periods of time (Ramachandran 2012). If they remain consistent that lends credibility

to the person’s experience. This experiment will utilize both tests to validate people’s

experiences.

Because this experiment is looking at synesthesia across languages, the sample will

include people who speak English only, as well as people who speak English and either

Chinese or Spanish. For the English only group we will try to get those who have very minimal

exposure to the other languages, especially Spanish because it is such a common language

spoken in the United States. The sample will include synesthetes, and an equal number of non

synesthetes as a control. There will also be an equal amount of people within the different

language groups (ie. the same number of English only speakers as people who can speak

English and Chinese).This will most likely be a very small sample considering synesthesia is

experienced by a very small portion of the population, but the sample should be as

representative as possible.
 Smith 6

The first step of this experiment is to prove that people’s synesthetic experiences are

real perceptual occurrences and are reasonably consistent over time. This will be done by

having the synesthetes self-report their experiences for a list of 100 random English words. The

subjects will report the color seen for each word by using a computer program to choose the

exact hex color that matches their experience, as done by many synesthesia experiments

(Blazej & Cohen-Goldberg, 2005). The colors reported will then be evaluated to determine the

rules each person’s perceptual system seems to use to determine the colors experienced by

certain words. The words will be grouped based on if their colors are determined by the first

letter, another letter in the word, the meaning of the word, or for anything else - as based off of

Blazej and Cohen-Goldberg’s categorization (2005). These categorizations will help to

determine each individual’s coloring rules, and will be helpful in comparing those rules across

languages. The expectation is to have most words’ colors based off of the first letter. The

consistency of the subject’s experience will be measured by having the subjects report colors

for the same list two weeks later. Any subject whose answers are not significantly consistent (at

least an 85% consistency between sessions) will be removed from the study.

The second part of the study will actually look at synesthetic experiences across

languages. The groups will be divided based on language abilities and synesthetic experiences.

So, for the Chinese group there will be both synesthetes and non synesthetes who speak only

English or English and Chinese. And, for the Spanish group there will be both synesthetes and

non synesthetes who speak only English or English and Spanish. Each person will be given a

list of 200 words, 100 of which will be in English and the other 100 will be either in Spanish for

the Spanish group or Chinese for the Chinese group. The words will be the same in both

languages, so there will be 100 words in English and the same 100 words presented in Chinese

or Spanish. The words will be the same to determine if the colors match across languages. The
 Smith 7

words will be presented in a random order regardless of language, and the same list will be

presented to each subject within the language group regardless of competency with the second

language. The subjects will again report the color seen for each word by using a computer

program to choose the exact hex color that matches their experience. The same test will be

administered one month later to look at consistency over time. The results of the second part of

this study will be used to compare consistency of experience over time and across languages,

as well as consistency of the rules used to determine colors across languages.

There are many different aspects to compare when looking at the expected results for

this study. The first aspect to compare is the expected results for non synesthetes. For non

synesthetes who speak only English, the expectation would be to see no words having the

same color in both languages. Not only would the English-only non synesthetes be arbitrarily

assigning colors to the words, but they would also lack any semantic basis to help to pick the

color for the non-English words. The assumption is that non synesthetes mainly make their

guesses on word-color associations based on either semantics or early learning associations (

Rich et. al, 2005). So, for languages they do not know, non synesthetes would be completely

guessing. However, for the bilingual non synesthetes the expectation would be to see the same

colors for the same words across languages, because of the assumption that they are guessing

based on semantics and early learning associations. If these expected similarities are not seen

it could show that the non synesthetes are not reporting colors based on the assumed reasons,

but instead either assigning colors completely randomly or basing their guesses on other

principles.

The expected results for synesthetes varies based on language abilities. Based on past

research it is assumed that there is little semantic influence, but how that is shown in the results

depends on the languages being compared. For example, because Chinese and English share
 Smith 8

no similar characters whatsoever, consistent coloring for the same words across languages for

bilingual speakers highly suggests a semantic influence. If a word has the same color for both

languages it clearly suggests that the meaning of the word is having some effect. Besides, none

of the individual characters could match between languages at all- eliminating any effect that

having the same letters in both versions of the word could have on the word’s coloring. If most

words do not match in color across languages, then that would suggest a highly visual-based

system. It would imply that the colors are determined by the individual characters themselves,

and probably the specific forms and shapes of those characters. The study will also look into if

any words have similar colors based off of similarly shaped Chinese and English characters. If

so, that would further support the significance of a visual based color determination system.

Similarly for bilingual English-Spanish synesthetes, coloring consistency across

languages supports semantic influences. If colors are similar for the same word in both

languages that shows that the meaning of the word must come into play more so than just the

individual letters in the word. Again, if the results were to show that most words are not the

same color for both languages, that supports the idea that synesthesia is more visually based.

Specifically, there would be strong proof for a visual-based system if similarly colored words had

the same first letter or many of the same letters in both English and Spanish but had different

meanings.

The results for synesthetes who only speak English support different theories than the

results of the synesthetes who speak two languages. For those who only speak English in the

Chinese group, the expectation would be to see no consistency in coloring for the same words

in both languages. This expectation exists because there are no shared characters between the

two languages and because subjects lack the knowledge to have the meaning of the words

influence their colors. The individual forms of the characters in each word would be the only
 Smith 9

influences on the color determination. If there is any consistency of word colors across

languages seen in the results, the only guess for why would be that similarly shaped Chinese

and English characters could produce the same or similar colors. But the likelihood of those

characters being used in both the English and Chinese versions of the same word are very low.

Basically, the only way words would have the same colors in both languages would be because

of chance. There is the possibility that the monolingual synesthetes could not experience any

colors for the Chinese words, which could have two explanations: either the words and

individual characters lack semantic ties completely so the subjects lack a basis for coloring the

words, or the English-only speakers have had such minimal exposure to those types of

characters that the words are not processed in the same area of the brain as their native

language and therefore they do not have the excess connections to produce a complimentary

color. In general, it would be expected to have no words be reported as the same color in both

English and Chinese by synesthetes who only speak English.

Similarly, the expectation would be to see minimal similarities in colors for words across

languages for English only speakers in the Spanish group. But, there would most likely be more

similarly colored words than in the Chinese group, because English and Spanish share the

same characters. If there are many similar characters in a word in English and in Spanish, then

the word could have similar colors. Because the English-only speakers lack any semantic

knowledge of the Spanish words, any similarities between words would suggest a highly visual

system based on the individual letters. Especially if there are similarities for words that start with

the same letters in both languages, and even more so if those words have completely different

meanings.

It is important to also look at if the individual subjects’ rules for color determination

remain the same across languages. It will presumably be the case that for most people the color
 Smith 10

they see is determined by the first letter of the word, and if not the color is most likely tied to

other letters within the word or the semantic meaning of the word. Coloring rules are expected

to remain the same across languages, especially for the Spanish group because the characters

in both languages are the same. If the coloring rules seem to change across languages that

would suggest that a different mechanism is influencing how the words are colored, which would

most likely be semantics or early learning associations. For the English-only speakers in the

Chinese group, this expectation does not apply because their rules cannot carry over as easily

because they have such a limited knowledge of the Chinese language and characters.

Finally, if the same words have the same colors for both synesthetes and non

synesthetes that would support a greater influence of semantics and/or early learning

associations on the coloring of the words. Again, because the assumption is that non

synesthetes are assigning colors based on common word-color associations and the meanings

of the words, if their colors matched those of synesthetes it would show that semantics are

influencing the synesthetes’ color determinations in similar ways. Any consistency in colors for

words across languages for those who only speak English do not suggest any semantic

influence, because those colorings would have no semantic basis regardless of synesthetic

experience. The results are expected to show few words that are reported to have the same

colors by those who do and do not experience synesthesia.

Overall, there is no expectation to see results supporting the influence of semantics on

the colors experienced, regardless of language(s) spoken. However, if seen, similarities in word

colors across languages could add new support to the small role semantics might play.

There are some possible problems with the validity of the results of this study. Mainly,

that synesthesia is a highly subjective, personal experience that can sometimes vary over time.

This means that while self-reports of colors seen are the best way of trying to understand what
 Smith 11

people experience, there is no way to be sure of their accuracy. Self-reports in general are not

the most reliable way to gather data, because people can manipulate or change their answers,

sometimes unknowingly. And, there is rarely consistency between individuals’ experiences, so

the results are extremely difficult to generalize. Also, there could be other mediating factors in

determining the colors seen for certain words. There are other aspects of words besides

individual letter form and semantics that could affect the colors seen, such as the sounds of the

words. In a future study it would be interesting to look at if, for both Chinese and Spanish

speaking bilinguals, there are consistent colors across languages for similar sounding words. If

so, the results could show an underlying auditory mechanism that affects color determination. It

is specifically important to look at this in bilinguals because they know how the words are

pronounced correctly, and are not just guessing based off of the letters/characters they can

recognize. If the results were to support an auditory mechanism, they could open up a whole

new door in understanding the integration of visual and auditory processing.

Finally, this study purposefully did not use words that are typically colored based on

semantics (sequence words) in order to be able to look deeper at the semantic coding for

everyday words. However, this same study could be done again using sequence words and

other commonly semantically colored words to see if the effect of semantics is still significant

across languages.

Overall the results of this study could help in solidifying our understanding of the

mechanisms underlying what colors synesthetes see. The expectation is to see a minimal role

of semantics. But there are many semantically influenced synesthetic experiences that have

been reported and have yet to be explained. So, if the results do not follow expectations, they

could help in understanding those experiences.

 Smith 12

In conclusion, studying synesthesia helps us to understand more than just the process of

synesthesia itself, because it also sheds light on how different mechanisms within the brain are

connected overall. Clearly, the fact that some words’ colors are determined semantically and

some are not shows that the visual processing of words and their subsequent semantic

meanings are processed separately. By understanding at what point between these systems

synesthesia occurs, scientists could figure out how and where these systems intertwine.

Currently scientists know that the visual process starts by breaking everything down into

individual properties, such as color or sharp edges, and eventually puts all of that information

back together again so we can understand the world around us. If scientist can understand at

what point in that process semantic meaning comes into play we could understand so much

more about how our visual system truly works, especially in terms of reading. Similarly,

shedding light on the role of semantics in synesthesia could provide insight into higher versus

lower level processing in the visual system and the brain overall. By understanding how that

system affects synesthesia we could understand more about how it affects everything we

perceive. Overall, after looking at past studies, it would be surprising to find much support for a

semantic basis for most synesthetic experiences, but it does occur for some words, so it would

be very interesting to see if it occurs across languages. The results of this study could show

some interesting effects of semantics on synesthesia, and could help explain semantic

processing in the long run.

 Smith 13

Works Cited

Blazej, L. J., & Cohen-Goldberg, A. M. (2016). Multicolored words: Uncovering the relationship
between reading mechanisms and synesthesia. ​Cortex,​ ​75,​ 160-179.
doi:10.1016/j.cortex.2015.11.017

Brang, D., Rouw, R., Ramachandran, V.S., & Coulson, S. (2011). Similarly shaped letters evoke
similar colors in grapheme–color synesthesia. ​Neuropsychologia,49(​ 5), 1355-1358.
doi:10.1016/j.neuropsychologia.2011.01.002

Brang, D., & Ramachandran, V. S. (2011). Survival of the Synesthesia Gene: Why Do People
Hear Colors and Taste Words? ​PLoS Biology,​ ​9​(11). doi:10.1371/journal.pbio.1001205

Dixon, M., Smilek, D., Duffy, P., Zanna, M., & Merikle, P. (2006). The Role of Meaning in
Grapheme-Colour Synaesthesia. ​Cortex,​ ​42(​ 2), 243-252. doi:10.1016/s0010-9452(08)70349-6

Hung, W., Simner, J., Shillcock, R., & Eagleman, D. M. (2014). Synaesthesia in Chinese
characters: The role of radical function and position. ​Consciousness and Cognition,​ ​24​, 38-48.
doi:10.1016/j.concog.2013.12.009

Neckar, M., & Bob, P. (2014). Neuroscience of synesthesia and cross-modal associations.
Reviews in the Neurosciences, 25​(6), 833-840.
doi:http://dx.doi.org/10.1515/revneuro-2014-0033

Ramachandran, V. S. (2012). ​The tell-tale brain: A neuroscientist's quest for what makes us
human​. New York: W.W. Norton.

Rich, A., Bradshaw, J., & Mattingley, J. (2005). A systematic, large-scale study of synaesthesia:
Implications for the role of early experience in lexical-colour associations. ​Cognition,​ ​98(​ 1),
53-84. doi:10.1016/j.cognition.2004.11.003

Tilot, A. K., Kucera, K. S., Vino, A., Asher, J. E., Baron-Cohen, S., & Fisher, S. E. (2018). Rare
variants in axonogenesis genes connect three families with sound–color synesthesia.
Proceedings of the National Academy of Sciences,115(​ 12), 3168-3173.
doi:10.1073/pnas.1715492115

Yokoyama, T., Noguchi, Y., Koga, H., Tachibana, R., Saiki, J., Kakigi, R., & Kita, S. (2014).
Multiple neural mechanisms for coloring words in synesthesia. ​NeuroImage,​ ​94,​ 360-371.
doi:10.1016/j.neuroimage.2014.01.039