HANDBOOK OF VISUAL COMMUNICATION THEORY, METHODS, AND MEDIA Edited by Ken Smith University of Wyoming Sandra Moriarty University of Colorado Gretchen Barbatsis Michigan State University Keith Kenney University of South Carolina [EA LAWRENCE ERLBAUM ASSOCIATES, PUBLISHERS. 2005 Mahwah, New Jersey LondonCHAPTER 6 Representation Theory Kerrn KENNEY University of South Carolina REPRESENTATION: WHAT DO WE “SEE” WHEN WELOOK AT A “PICTURE”? Behaviors atempt to observe and measure the real world directly, Phenomenologists are ‘aclusively interested in a person's introspective experience. Semioticians and rhetoricians ‘ny to understand the linkages between our internal world and the external world, and that linkage is necessary, they believe, because the external world is always mediated by our senses and our mind, Whereas rhetoricians have investigated how humans create and manipulate symbols in order to persuade other humans, semioticians have been nor interested in how humans (and other animals) interpret al kinds of signs, including. symbols, that were created by other people, as well as natural signs that may have resulted fiom plants, animals, or itorganic matter. Both rhetoricians and semioticians, therefore, are concerned with how signs “mediate” between the external world and our internal “world,” orhow a sign “stands for” or “takes the place of” something from the real world inthe mind ofa person. What these scholars are concerned with is called representation. This chapter explains the strengths and weaknesses of four types of theories of how pictures represent ‘The concept of representation has been thought of as a relationship with two, three, and four parts, The two-part model is associated with Saussure (among many others), who defined the linguistic sign as a “two-sided psychological entity” consisting of a sign hide and its meaning, He used the word signifier for the sign vehicle (the antecedent capeience, or the word, or expression, or speech sound) and the word signified for the meaning of the sign (the consequent experience, or the thing, or the content, or the response in hearer) ‘The three-part model is associated with Peirce (among many others), who defined representation as a relationship among sign, object, and interpretant. For Peitce, semiosis 99100 KENNET ‘occurs when an existing sign is connected with the object signified to produce meaning in the mind of the interpreter. ‘The four-part model is associated with Mitchell (1990). The additional dimension is the maker of the representation. Mitchell envisions representation as a quadrilateral with two diagonal axes, one connecting the representational object to that which it represents (like Saussure’s dyadic model), and the other connecting the maker of the representation to the viewer: The lines connecting the signifier and object are called the axis ofrepresenta- tion, The lines connecting the maker and the viewer are called the axis of communication. Peirce’s triadic model omits the fourth dimension (maker) because itallows the possibility of natural signs, which do not have makers who intend to communicate. One advar- tage of Mitchell's model, therefore, is that it emphasizes communication, appropriate for the study of pictures, which presumably were created with the intention to communi cate, or to express the creator's feeling, or to elicit an (intended) emotional response in viewers. ‘A crucial consideration for the analysis of representation is the relationship between the sign and the object. Semioticians distinguish three types of relationship: an iconic relationship that stresses resemblance, a symbolic relationship that is primarily arbitrary, and an indexical relationship that is based on cause and effect, ora relation such as physical proximity or connectedness. Most representations use more than one type of sign-object relationship. ‘The fourtypes oftheory ofpictorial representation are directly connected o these three types of relationship between signs and their objects. Causal relation theories (including transparency theory and recognition theory) emphasize indexical andiconicrelations. Re- semblance theories (including nonperceptual and perceptual) emphasize iconic relations. ‘Convention theories emphasize symbolicrelations, Mental construction theories (inclu. ingiillusion, make-believe, and “seeing.in”) emphasize iconic and symbolic relations. ‘This chapter also relates the four types of theory to Mitchells (1990) model of representation. In order to begin to understand theories of pictorial representation, please think about ‘what happens when YOU look at a photograph of a dancer stretching. Do you see areal dancer stretching? Are you reminded of a dancer stretching? Does the picture look like a dancer stretching? Or do you have an experience that i just like the experience of looking st aa dancer stretching? Do you read the language of the photograph and realize it concernsa dancer stretching? Do you have the illusion you are in the presence of'a dancer stretching? Do you make believe you see a dancer stretching? Do you see a dancer stretching in the picture and also see the surface of the picture? The sometimes subtle differences between these questions (and their answers) reflect key ideas in the various theories. CAUSAL RELATION THEORIES ‘Transparency Theory: The Picture Is Made Automatically Kendall Walton (1984) made the provocative statement that photographs (or films or videos) are transparent, and “we see the world through them” (p. 251). He considered ‘camera pictures to be “aids to vision” just as we look “through’” (transparent) eyeglasses, telescopes, and microscopes in order to see things we would not be able to see with just6. REPRESENTATION THEORY 101 cour eyes. We supposedly can see the real dancer stretching when we see a photograph of her stretching even though the dancer is not in our presence, Moreover, Walton claimed that camera pictures allow us to see into the past. He wrote that we “see, quite literally, cour dead relatives themselves when we look at photographs of them” (1984, p. 252). ‘This power of photography, wrote Walton, comes from its “mechanical” or “auto- matic” origins (see also Arnheim, 1974; Bazin, 1967; Brubaker, 1993; Cavell, 1971). A photograph of a couple having a picnic on a grassy hill, for example, was mechanically ‘caused by light reflecting off of the man and woman, bottles of soda and wine, blan- ket, plastic plates, and so on and onto 2 piece of light-sensitive film. ‘The image also is mechanical because if someone placed a camera with the same film, lens, position, and so on as with the first photo, and if the picnickers and background elements and light never changed, then the same results would automatically occur. It seems, there fore, that we see things firsthand through the transparent photograph. “Handmade” pictures (i.e, drawings and paintings) would be considered secondhand information be- cause apictore-maker’s judgment intervenes between us and the picnickers, and because cye-hand coordination cannot make exact copies as a mechanical camera can. ‘Transparency theory states that a camera picture represents a phenomenon because it was automatically and mechanically caused by the existence of the phenomenon and because the picture looks like exactly like the phenomenon. In terms of Goodman's four-dimensional model of representation, it exclusively focuses on the vertical axis run- ning from the object to its sign (1976). Transparency theory is silent about the com. ‘munication axis running from maker to viewer because it ignores the viewer and the maker, ‘Transparency theory ignores the viewer because it assumes that viewers are so naive aso think they are actually seeing the objects themselves when looking at a camera picture, The theory ignores the fact that viewers interpret pictures. Transparency theory also pretends that a picture of a migrant worker stands for just that individual, when, in fact it may stand for a whole group of migrant workers ‘Transparency theory ignores the maker and his or her ability to control the taking, developing, and printing of film. When Walton tries to rebut arguments about the control ofpicture-makers (and, therefore, the lack of mechanical automaticity), he argued that just because we see things through a photographer's eyes doesn’t mean that we don't really see them, “I do see those things myself,” he wrote (1984, p. 262). In other words, the viewer has control over seeing, not the maker. This is an important philosophical point because whoever controls the “seeing” decides where attention and mind will be fixed. The controller intends the meaning, photographers control the seeing, ass generally believed, then we look at the results oftheirintentions. We can, of course, misinterpret their intentions, and we can see more orless than was intended. For example, we can draw on our background knowledge (or lack of knowledge) of dancing in general, or of this particular dancer, to “see” something, ‘more (or less) than wes intended, But the photographer is in control photography isan “aid to vision,” as Walton wrote, then we shouldbe able to control the seeing, just as if we used a telescope to see the dancer stretching; we could control the telescope and change what we see depending on our interest. We cannot, however, control the seeing with a camera picture, so Walton cannot see things himself,102 KENNEY In fact, the only time that we as viewers can control imagery is with virtual-reality equipment. If we had access to real-time, photographic quality, three-dimensional ims agery of the dancer stretching, and ifthe dancer took 10 seconds to hold a stretch and we saw her stretching for 10 seconds, and if we could control our angle and distance of view. point, then we might agree the medium is “transparent” and that we can see “through” the medium to the dancer, and there was no “intent” by the maker, and, essentially the ‘maker could be eliminated from the representation process (see Carroll, 1995; Curti, 1991; Martin, 1986; Warburton, 1988 for criticisms of the transparency theory), If we reject the transparency theory and acknowledge that photographers communi: cate their intentions, then camera pictures seem kind of ike paintings, yet because there 4s some causal connection between the film image and the subject, camera pictures also seem “special.” Maynard (1989) used the concepts of depiction and detection to explain the special nature of camera pictures. To depictis to represent, which means “to stand for” ‘orto communicate by signs or symbols. To detect isto discover something with certain: Like paintings, camera pictures depict because they represent things that may be real or fiction. Unlike paintings, camera pictures detect because they are the results (or traces) of ‘photochemical actions, Camera pictures used as detection have a similar automatic quality as sunburns (another example of trace of photochemical action). Camera pictures aso seem to give evidence of the fact of photochemical effects just like sunburns. A “pure” photographie depiction would represent something, but not be a trace of that thing. A photograph of a fairy, an angel, or the Loch Ness monster would be a pure depiction because it would represent a fictional phenomenon and yet the fictional phe. ‘nomenon would not have caused the photochemical action, For example, a photographic depiction of “blueberry ice cream” may actually be a photograph of mashed potatoes with blue food coloring, A “pure” photographie detection would ascertain the presence of something bat it ‘would not represent, or be about, anything. Examples of photographs used purely for detection would include traces of light from stars and traces of radiation from X-rays Another example of a nondepicting photograph would be any image that resulted from accidentally pressing the shutter. From viewing the resulting image we could discover ‘what had been in front of the camera, but that Scene would not represent anything ‘When we say: “This is a photograph of a dancer stretching,” we generally mean itis both a trace of the dancer and a depiction of the dancer. If everything that was depicted was also detected, then we would consider it to be a documentary photograph, with an aura of realism and objectivity. Pure documentary photographs and films, however, are impossible to make, FI wanted to shoota pure documentary, [would have to exercise litle ‘orno control over the essential shots, The more I aimed my camera at subjects I believed would interest an audience, and the more I followed a “style,” or concerned myself with aesthetics, including decisions about exposure, sharpness, composition, camera angle, and the like, and the more I tried to shoot pictures to fit a storyline, the less purely documentary the result. An ideal documentary would seem to be surveillance camera footage; each shot is a trace of ts subject and is only connected to other subject traces. Not only is it difficult to make a pure documentary, itis also difficult to view it. We understand photographs and films by fitting them into our prior knowledge of genres, schemas, and other preexisting categories. Viewing “traces” will inevitably be affected by6, REPRESENTATION THEORY 103 our own expectations and background experiences. The moment shown in the picture will evoke many ideas that are not shown, It may seem “easy” to “perceive” the “traces” of photographs and film, but our minds do not stop at that point; we inevitably make comections, and those connections lead to narrative content, Aphotographer who tried to take “pure” documentary photographs placed a camera con a tripod and aimed it outside the passenger window of his car. He used the camera's, automatic features and he mechanically connected the camera to his odometer so the shutter automatically would be pressed each mile, After driving across the United States, he had 3,000 or so photographs that were traces lacking depictive qualities. He was dissatistied with the results, however, because he had driven on interstate highways and ‘many photos were too redundant, so he repeated the trip and this time took smaller roads, Apparently, however, he was still worried whether these “traces” would be sufficiently ineresting. During a conference presentation, he admitted “cheating” by connecting a cable the camera so he could manually override the automatic shutter release if he saw something particularly interesting right before or after the one-mile interval, His need to rake the images appealing eventually overcame his need to follow his own procedure of climinating che photographer (the “maker” in Mitchell's model). Even if he had followed procedure, however, he still would have selected a route across county and so would have plyed a role in the image-making. In conclusion, transparency theory is important because many viewers, and some picture-makers, continue to implicitly, if not explicitly, believe chat camera pictures are aidsto vision that allow us to see the world directly. People enjoy looking at photographs ‘ofloved ones; they use photographs as evidence in courts of law, and news organizations use photographs to report on people and events. From a theoretical perspective, the idea that camera pictures have a (limited) indexical relationship to their objects and a (limited) iconic relationship is valid, but not new. The idea that camera pictures do not (yet) offer viewers control over what they see is more interesting because that fundamental idea may change with the development of virtual-reality equipment. ‘The concepts of depiction and detection are useful for clarifying the nature of documentary pictures, Overall, however, the insufficient roles given to makers and viewers hurt this theory of pictorial representatic Recognition Theory: The Picture Reminds You of the Object or Person ‘Whatifinstead of claiming that we see the dancer through the photograph, we claim that wwe recognize the dancer in the picture because we can recognize Sandra face to face. In other words, the recognition skills we bring to pictures depend on and extend the recog nition skills we use in ordinary perception. This theory states that a picture represents a phenomenon because the picture looks exactly like the phenomenon. It emphasizes the iconic relationship between sign and object and deemphasizes any symbolic relationship. In terms of Goodman's four-dimensional model of representation, recognition theory focuses both on the vertical axis running from the object to its sign and on the viewer (1976), It resembles Peirce’s triadic model of the sign. Akey concept of recognition theory is aspects. Wilkerson (1991) suggested that aspect setng has five main features. One, seeing aspects involves noticing resemblances. We104 KENNEY see a dancer-aspect in the picture of Sandra by consciously focusing attention on 2 resemblance between the picture and a dancer. Two, seeing aspects is an imaginative activity. We visualize the scene of a dancer stretching even if the dancer is not present Three, seeing aspects is subject to the will, We can try co see an aspect, and others can {guide our efforts by drawing our attention to crucial parts of the picture; whereas with ordinary seeing we do not make an attempt to see what isin front of us. Four, secing an aspect is detached from belief. We may see a swan-aspect in a cloud (or picture), yet we ‘won't believe that the cloud (or picture) is literally a swan. Five, seeing an aspect often involves definite experiences, and one of them, the experience of suddenly noticing an aspect, ora change of aspect, is akin to experiencing a sudden dawning of understanding We look at the picture and think, “Aha, that must be Sandra at 16 because she has the same nose and chin as her mother.” Recognition theory deemphasizes the symbolic relationship between sign and object because it states that picturing is generative, which means we do not need to learn the meaning of unfamiliar pictures in the way we must learn the meaning of unfamiliar words. We can “generate” understanding of almost any picture of a familiar object ata sglance provided we have mastered the pictorial system. When we encounter new pictorial systems, orexample, cubism, we must firstlearn the system, and then we can again recog- nize any familiar objects, Pictorial competence, therefore, is system-relative, and pictorial recognition abilities can vary in speed and accuracy with the system of representation. Conventionalists, who emphasize the symbolic relationship between sign and object, believe that first we learn the conventional codes, grammar, language, vocabulary, ot other system based on social agreement, and then we can “read” pictures. Recognition theories, on the other hand, take almost the opposite approach. They believe we can see pictures and identify objects in pictures, and thereby learn about the world and real life. Ina sense, the picture books of our youth become the socially agreed on conventions for understanding the world and rea life Pictorial recognition theory differs from simplistic understandings of iconic relations, generally thought of as resemblance between sign and object, because recognition is dy namic. Whereas simple recognition ability would require an overwhelming similarity between incoming information and information stored from the past, dynamic recog- nition ability can link currently perceived objects (or pictures) with objects perceived in the past despite what may amount to radical changes in appearance. Face-recognition is especially dynamic. A face may be contorted by laughter or grief and other expressions, ‘yet we can recognize it. To say that recognition is dynamic isto say that features can be recognized under different aspects (Lopes, 1996). Wittgenstein (1953) considered whether noticing an aspect is seeing or interpreting. ‘When we change from seeing something one way to seeing it another way and thereby notice an aspect of it, do we really see differently, or do we merely interpret what wwe see differently? Does our visual experience change, or does our visual experience remain constant and the interpretation that we place upon what we see alter? The answer probably is “both.” The concept of seeing an aspect lies between the concept of seeing color or shape and the concept of interpreting: it resembles both of these concepts, but in different respects (Budd, 1987). This point is important for pictorial representation because if noticing an aspect was only seeing, then transparency theory6, REPRESENTATION THEORY 105 would be supported, and if noticing an aspect was only interpreting, then convent theory would be supported, Instead, noticingan aspectisbetween seeingand interpreting, and recognition theory is between transparency and convention theory. In conclusion, recognition theory is related to transparency theory because pictures and objects have the same properties. Even if the picture is a cubist painting, we could follow a circuitous route from the features of the painting to the features of the dancer stretching. Recognition theory also recalls Walton’s idea that photographs are an aid to vision, About pictorial recognition, Lopes writes: “Pictures are visual prostheses; they ex tendthe informational system by gathering, storing, and transmitting visual information about their subjects in ways that depend upon and also augment our ability to identify things by their appearance” (1996, p. 144). Instead of being limited to photographs, however, recognition theory can explain any type of picture. In addition, it recognizes the necessary role of the viewer to interpret the picture, and the sophisticated concept of aspects explains how that interpretation occurs. The theory, however, allows too much latitude to the spectator’s imagination and does not explain how picture-makers limit our roving imaginations and constrain what we see as the representative features of the picture (Peetz, 2001). Another problem is that the theory cannot explain how pictures can misrepresent or be inaccurate (Rollins, 1999). RESEMBLANCE THEORIES ‘There are two major types of resemblance theories. Both are based on the idea that pictures resemble their objects, which means that the iconic nature of a sign-object relationship is emphasized, In terms of Goodman's four-dimensional model of represen: tation, resemblance theories exclusively focus upon the vertical axis running from the ‘object to its sign (1976), They are silent about the communication axis running from maker to viewer because they ignore the viewer and the maker. Resemblance theories apply to pigeons as well as humans, so no higher-level thinking skills are required. Nonperceptual Resemblance Theory: The Picture Looks Just like My Aunt Hilda ‘A nonperceptual theory states that pictures represent by virtue of similarity with their subjects. This theory has intuitive appeal because when we look at pictures, we seem to see similarities between the picture and what it represents Goodman (1976) criticized this theory, writing that while a picture may resemble its, subject in some respects, in many respects it does not. A two-dimensional, 4-inch-by- 6inch photograph of a dancer, for example, does not look like a 110-pound woman in ‘most respects. To understand what someone means when she or he says this picture looks lite that subject, we must know which kinds of similarity she or he is referring to, and knowing which kinds are pertinent would depend on the context of the picture and on custom (Black, 1972). Hyman (2000) overcame this criticism by explaining that we are not comparing a flat object to a subject; instead we are comparing the marks of a picture to the matks of a105 KENNEY subject. He wrote, there isa “strict and invariable relationship between the lines, shapes, and colours on a picture's surface and its internal subject” that can be defined without referring to the picture's psychological effect on a spectator (2000, p. 24). Despite its appeal, nonperceptual resemblance theory has several problems. Fist resemblance is a symmetrical relationship. If the color picrure resembles the dancer stretching, then the dancer stretching must also resemble the color picture. Represents tion, however, is not symmetrical; we believe that the photo may represent the dancer, but the dancer cannot represent the photo (but see Files, 1996). Therefore, a resemblance relationship does not distinguish a representation from what it represents. Second, ¢ picture resembles a copy of itself more than it resembles the dancer, buta picture cannot represent a picture; nor can it represent itself, Resemblance, therefore, is not sufficient for pictorial representation because resemblance exists without representation. Thitd js the chicken-egg problem. Do we notice picture-subject similarities and then know what is represented, or do we first know what is represented and later notice similarities? Perception-based Resemblance Theory: Pictures of Predators Can Be Recognized by Pigeons or People ‘The second type of resemblance theory, like recognition theory, is based on perception (see Budd, 1993; Hopkins, 1994; Manns, 1971; Neander, 1987; Novitz, 1975). A painting resembles its subject if the painting has the same power to affect our organs of sight as its subject. According to resemblance theory, at an early stage of vision, at least, We just receive visual information and what we see is not affected by our beliefs, values, or background knowledge (Rollins, 1999), ‘This early visual information is segregated from higher cognitive processes. It also is separated from information from other senses (hearing, smelling, etc.). The process of acquiring this early visual information is the same whether we view pictures or real life; both lead to the same perceptual experience (Gilman, 1992). Ifthis theory was correct, then we could initially view pictures with an “innocent eye.” It also would mean that someone from 2 “primitive” culture, who had never seen a photograph before, but who was used to seeing elephants, should be able to recognize a photo of an elephant, Moreover, animals should be able to recognize piccures ‘of predators. In fact Danto (1993) wrote about pigeons that were able to distinguish ‘among pictures on the basis of their subject matter even though the pictures’ settings, lighting, and viewpoints varied. Danto speculated that pigeons may even be able to appreciate stylistic differences in photographs as they recognize and classify pictures. I resemblance theory istrue, theninstead ofnoticingaspects, we recognize objets (Charlton, 2000). Resemblance theory also may explain the appeal of Andreas Gursky’s large pho- tographs. Up to 16 feet across and full of detail, they seem to offer viewers an opportunity to recognize more objects per image than most other pictures. One picture, for example, shows eight crowded aisles of a grocery store and another shows tens of thousands of people at a rock concert. Another example of pictures that seem real because they look just like their subjects is IMAX films, which are shot with larger film stock so more detail can be captured. The films then are projected onto huge screens, so we cannot see the edges of the screen, and we fee] immersed in the subject.6, REPRESENTATION THEORY 107 Perceptual resemblance theory, however, has problems. First, pereeptual experience isnot solely based on basic visual information. If it was, then we could not distinguish a picture of a dancer from the real dancer. Because we can (easily) distinguish, some higher order processing, including culturally inculeated beliefs, must occur. Second, if pictures and realty affect our visual sense organs the same way, then the feelings We have in looking at pictures should be the same as those we have in viewing real scenes, bucthey are not, We might expect that viewing reality would arouse stronger emotions. Chariton, however, points out that pictures move us profoundly and that they have certain advantages over looking at things. He wrote: “The feelings we have in looking a pictues are likely to be much stronger than those we have in looking at real people, realanimals, and even real inanimate objects that are useful or dangerous” (2000, p. 478). ‘Three, resemblance theory could not allow for different marks, from different pictorial traditions, depicting the same things. Four, it could not allow for similar marks, with different provenances, differing in meaning. Resemblance theory goes still further. Peacocke (1987) suggested that resemblance is not based on objective similarities between a picture and subject, but on subjective similarities between our experience of pictures and our experience of the subject, He dis- tinguished “sensational” and “representational” properties of experience. When we draw picture ofthe dancer's outstretched foot, we inevitably draw the foot the way we “know” a foot is shaped and colored, or its representational content. Depending on the angle We actually viewed her outstretched foot, however, the shape may be distorted, andthe color ay be affected by nearby objects, so we see its sensational content. Forany subject, there is a myriad of ways in which it can be seen, and to each of these ‘ways there corresponds a different possible visual fied, defined as an imaginary plane interposed between the scene and viewer. A picture's sensational content looks like its subject with respecttothe properties ofthe spectator’s visual field, nottosome “objective” viewpoint. Because of the concept of visual fields, many pictures may resemble the same subject, yet those pictures may differ radically. Peacocke’s additional concepts offer the advantage of including the viewer in the representational process, at least to some extent. One problem with Peacocke’s concept ‘ofthe visual fields that it s closely tied to Alberti’s use ofa fixed frame in order to drawa scene with “normal” perspective. Resemblance theories based on the idea of a visual field, therefore, will “work’ for photographs and other realistic pictures, but the theories don't ‘work with cubist or splitstyle pictures or pictures with reversed perspective or pictures that distort their subjects to ridicule them. (For other criticisms, see Kemp, 1990.) Some philosophers see @ connection between recognition and resemblance theories. ‘They believe that we notice a resemblance between a picture and its subject before we recognize the picture. Schier(1986) rejected thisidea because itpresupposes ahomuncular theory of mind, In other words, some subcognitive part ofthe person would then have to ‘note” resemblances, Sartwell (1991), however, countered that there is no need for a recourse to homunculi, Similar properties, he wrote, are simply the immediate causal aniecedent of recognition. Inconclusion, pictures resemble their subjects better than words, but pictures don’t necessarily resemble their subjects in all ways, and tryingto identify which waysis difficult. Itreally depends on the picture, Sometimes shapes may be important, but not always.