The link between language and vision

For a translator to turn one language (say, English) into another (say, Greek), she has to be able to understand both languages and what common meanings they point to, because English is not very similar to Greek.

It turns out that a similar task must be carried out within our own minds when translating visual information into linguistic information — a task Alon Hafri, an assistant professor in the Department of Linguistics and Cognitive Science at the University of Delaware and director of the Perception and Language (PAL) Lab, is trying to better understand. His research is on the connection between language and visual perception, or, in other words, between what we say and what we see.

Understanding these two activities — seeing and speaking — may at first seem trivial, Hafri said. Indeed, we're so good at these things that we do them without thinking. But seeing and speaking are some of the most complex processes that the human mind carries out. Hafri’s goal is to understand how they work and how those two systems share information.

Hafri gives a seemingly simple example: a cat on a mat. If we saw a cat on a mat, we'd have no problem describing the scene. Or if someone said, "Look, a cat on a mat," we'd easily be able to identify it out in the world. Yet at first glance, an image of a cat on a mat and the sentence “a cat on a mat” have nothing in common: the image has colors, edges, shapes and locations, while the sentence has sounds, words, phrases and such.

“If you think about the problem that the mind has to solve when you're going from a visual scene to a sentence that you produce about it, there's nothing in common really between words in a certain order and a visual image,” Hafri said. “The kind of information that they start with is very d​ifferent. Something I'm really interested in is how visual information gets translated — sort of like an English-to-Greek translator would do — into a format that language can use, such that we can talk about what we see. And vice versa, so that we can recognize the things people are talking about in the world. The kind of questions I ask are, well, how does that happen? What is the translation process?”​

Watch a video on symmetrical meaning​

In a recent project, Hafri asked research participants whether the notion of symmetry exists not only in vision (such as in a butterfly’s wings), but also in language. He uses tasks that force people to make judgments of how a linguistic and a visual stimulus relate, presenting images of shapes that are symmetric and non-symmetric and asking people to choose from a set of words the one that best describes them. Hafri gives the words “marry” and “adopt” as examples. "Marry" has notions of symmetry as a part of its meaning (if Bill marries John, then John marries Bill), while "adopt" does not (if Bill adopts John, John doesn't necessarily adopt Bill).

Despite the fact that most participants said they felt they were picking images and shapes at random, Hafri found that people were quite sensitive to the connection between visual symmetry and language for symmetry, such that they associated symmetrical images with words that have symmetrical meaning. (You can try out the tasks for yourself.)

“There seems to be some deep way in which symmetry in vision and symmetry and language are connected,” Hafri said. “I think it's because there's some internal language or internal code — you can think of it almost like your mental computer code — that has this property of symmetry that both language and vision link to. So when you see a visual image or an event that's symmetrical, it somehow gives rise to this abstract mental representation of symmetry — this internal language of symmetry.”

Hafri’s research will help cognitive scientists to better understand the internal code — or internal language — of the human mind.

“Language can be studied on its own, but at some level, it has to connect to other areas of the mind,” Hafri said. “For [vision and language] to connect, it has to happen through some internal code, because a visual image and words and sentences are just so different, in so many ways. For that reason, if you can find the points at which they make contact, you can get a sense of what that internal language or that internal code is like in a way that you couldn't otherwise.”

​ Article by Amy Wolf, illustration and video by Jeffrey C. Chase
published  January 19, 2023​