Now you see it, now you don't: mediating the mapping between language and the visual world

Gerry T. M. Altmann, Yuki Kamide

Research output: Chapter in Book/Report/Conference proceedingChapter (peer-reviewed)peer-review

136 Citations (Scopus)


The goal of much psycholinguistic research is to understand the processes by which linguistic input is mapped onto a hearer's mental representation of his or her world. Within the context of a sentence such as "The mouse chased the cat into the basket," we can ask questions such as "At what stage is the cat interpreted as the thing being chased?" "At what stage do we determine which cat?" and, more generally, "How, and when, do we map the components of a sentence onto components of the world?" Historically, there has been somewhat of a shift in respect of the answer to these questions. In the 1960s, "click-detection" studies suggested the possibility that interpretation may be bounded by clausal structure, with clause boundaries being the site of significant processing effort; in the 1980s, researchers proposed that interpretation does not "lag" behind syntactic parsing, but takes place incrementally, as each word is encountered; and more recently, it has been demonstrated that interpretation can on occasion be driven by expectations made on the basis of linguistic input that precedes the actual linguistic items that confirm those expectations. In this chapter, we review a number of these latter studies. Specifically, we explore the timing of interpretive processes in relation to the mapping of language onto a concurrent visual world. We shall review first a series of studies that explore the mapping of I/I/ sentences onto the visual world and that show that the processor is able to anticipate what is likely to be referred to next. We then review a series of studies that show how, in fact, the mapping is not onto the visual world but rather onto a mental world; this is shown by exploring patterns of eye movements when the visual world is absent at the time of the linguistic input, and, in other studies, when the visual world remains constant, but certain "facts" about that world are changed in the linguistic context that precedes the target sentence . Finally, we review a study that delves a little deeper into the relationship between linguistic structure and the structure of events that are portrayed in the visual world. We shall conclude in the final section that the interpretation of a sentence situated in a visual world may be as much to do with non-linguistic, primarily visually driven, processes as with linguistic processes.
Original languageEnglish
Title of host publicationThe interface of language, vision, and action: eye movements and the visual world
EditorsJohn M. Henderson, Fernanda Ferreira
PublisherPsychology Press
Number of pages35
ISBN (Print)1841690899
Publication statusPublished - 2004


Dive into the research topics of 'Now you see it, now you don't: mediating the mapping between language and the visual world'. Together they form a unique fingerprint.

Cite this