Home > News > The link between language and vision

More News

 
From homelessness to a new career path

From homelessness to a new career path

McNair Scholar dropped out of UD but returned strong and is planning a medical career
 
GPPC Undergraduate Philosophy Conference

GPPC Undergraduate Philosophy Conference

On April 26-27, the Department of Philosophy hosts the 34th annual Greater Philadelphia Philosophy Consortium Undergraduate Conference.
 
CONNECT
EMailTwitterFacebookYouTubeMake a Gift

The link between language and vision

Image Picker for Section 0
Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

UD Prof. Alon Hafri studies the connection between language and visual perception

graphic of eyes and mouths

​​

​Seeing and speaking are two of the most complex processes happening in the human mind. Through his research, UD’s Alon Hafri seeks to understand how they work and how those two systems share information.​

For a translator to turn one language (say, English) into another (say, Greek), she has to be able to understand both languages and what common meanings they point to, because English is not very similar to Greek.

It turns out that a similar task must be carried out within our own minds when translating visual information into linguistic information — a task Alon Hafri, an assistant professor in the Department of Linguistics and Cognitive Science at the University of Delaware and director of the Perception and Language (PAL) Lab, is trying to better understand. His research is on the connection between language and visual perception, or, in other words, between what we say and what we see.

Understanding these two activities — seeing and speaking — may at first seem trivial, Hafri said. Indeed, we're so good at these things that we do them without thinking. But seeing and speaking are some of the most complex processes that the human mind carries out. Hafri’s goal is to understand how they work and how those two systems share information.

Hafri gives a seemingly simple example: a cat on a mat. If we saw a cat on a mat, we'd have no problem describing the scene. Or if someone said, "Look, a cat on a mat," we'd easily be able to identify it out in the world. Yet at first glance, an image of a cat on a mat and the sentence “a cat on a mat” have nothing in common: the image has colors, edges, shapes and locations, while the sentence has sounds, words, phrases and such.

“If you think about the problem that the mind has to solve when you're going from a visual scene to a sentence that you produce about it, there's nothing in common really between words in a certain order and a visual image,” Hafri said. “The kind of information that they start with is very d​ifferent. Something I'm really interested in is how visual information gets translated — sort of like an English-to-Greek translator would do — into a format that language can use, such that we can talk about what we see. And vice versa, so that we can recognize the things people are talking about in the world. The kind of questions I ask are, well, how does that happen? What is the translation process?”​

Watch a video on symmetrical meaning​

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

In a recent project, Hafri asked research participants whether the notion of symmetry exists not only in vision (such as in a butterfly’s wings), but also in language. He uses tasks that force people to make judgments of how a linguistic and a visual stimulus relate, presenting images of shapes that are symmetric and non-symmetric and asking people to choose from a set of words the one that best describes them. Hafri gives the words “marry” and “adopt” as examples. "Marry" has notions of symmetry as a part of its meaning (if Bill marries John, then John marries Bill), while "adopt" does not (if Bill adopts John, John doesn't necessarily adopt Bill).

Despite the fact that most participants said they felt they were picking images and shapes at random, Hafri found that people were quite sensitive to the connection between visual symmetry and language for symmetry, such that they associated symmetrical images with words that have symmetrical meaning. (You can try out the tasks for yourself.)

“There seems to be some deep way in which symmetry in vision and symmetry and language are connected,” Hafri said. “I think it's because there's some internal language or internal code — you can think of it almost like your mental computer code — that has this property of symmetry that both language and vision link to. So when you see a visual image or an event that's symmetrical, it somehow gives rise to this abstract mental representation of symmetry — this internal language of symmetry.”

Hafri’s research will help cognitive scientists to better understand the internal code — or internal language — of the human mind.

“Language can be studied on its own, but at some level, it has to connect to other areas of the mind,” Hafri said. “For [vision and language] to connect, it has to happen through some internal code, because a visual image and words and sentences are just so different, in so many ways. For that reason, if you can find the points at which they make contact, you can get a sense of what that internal language or that internal code is like in a way that you couldn't otherwise.”

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

​ Article by Amy Wolf, illustration and video by Jeffrey C. Chase
published  January 19, 2023​

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Image Rendition

Change the way the image is cropped for this page layout.

Media Size

Cycle through size options for this image or video.

Original
50%
66%
100%
Fixed Portrait 1
Fixed Portrait 2
Cancel
Media Right/Left-Align

Align the media panel to the right/left in this section.

Insert Image

Open the image pane in this body section. Click in the image pane to select an image from the image library.

Insert Video

Open the video pane in this body section. Click in the video pane to embed a video. Click ? for step-by-step instructions.

Remove Image

Remove the image from the media panel. This does not delete the image from the library.

Remove Video

Remove the video from the media panel.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Media Right/Left-Align

Align the media panel to the right/left in this section.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Media Right/Left-Align

Align the media panel to the right/left in this section.

Move Up

Move this whole section up, swapping places with the section above it.

Move Down

Move this whole section down, swapping places with the section below it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Media Right/Left-Align

Align the media panel to the right/left in this section.

Move Up

Move this whole section up, swapping places with the section above it.

Code Cleaner

Check for and fix problems in the body text. Text pasted in from other sources may contain malformed HTML which the code cleaner will remove.

Accordion is OFF

Accordion feature turned off, click to turn on.

Accordion is ON

Accordion featurd turned on, click to turn off.

Media Right/Left-Align

Align the media panel to the right/left in this section.

News Story Supporting Images and Text
Used in the Home Page News Listing and for the News Rollup Page
UD Prof. Alon Hafri studies the connection between language and visual perception
Linguistics and Cognitive Science
 
1/19/2023
No
Page Settings and MetaData:
(Not Shown on the Page)
Page Settings
link between language and vision
No
 
 
MetaData for Search Engine Optimization
The link between language and vision