General Introduction (31/01/2024)
Introduction of basic concepts
- “My girlfriend is pregnant” this sentence has one unique meaning: “My female partner with
whom I am not (yet) married is expecting a child”
- However, the same sentence can have different extra connotations (based on context, intonation
and nonverbal cues)
1) [Yessss] My girlfriend is pregnant (joy)
2) [Oh no!] My girlfriend is pregnant (fear)
3) [What?] My girlfriend is pregnant (surprise)
4) [Arghh] My girlfriend is pregnant (anger)
- In other words: a sentence can be used in different ways so that extra meanings are generated
* Can be derived: from context (situational, discourse), from the way a sentence is uttered
For example someone’s facial expression makes you interpret it like another connotation
(global)
- Non-verbal communication (NVC): the ensemble of features that do not determine what you say,
but how you say it
* Variation in Voice
* Variation in Body Language
1) Variation in Voice (auditory)
- Features that you can “hear” (encoded in the speech signal itself it isn’t body language, because
it's audible and not visible)
* Intonation (speech melody): the rise and fall of pitch in speech. It adds emotional nuance
and can indicate whether a statement is a question, statement, or exclamation
* Loudness, energy (adaption to the situation): the volume and energy level of speech, which
can convey enthusiasm, anger, or other emotions. Adjusting loudness helps in adapting
communication to the context
* Tempo (Are you talking to a child of a more advanced speaker? You adapt it (fast/slow)
to the person you talking to): the speed at which someone speaks. It can convey excitement,
urgency, or calmness and is often adjusted based on the listener’s characteristics or the
context
* Rhythm (typical for a language you can’t hear what language they speak but you can
recognize that it can’t be some specific languages by hearing the rhythm): the pattern of
stressed and unstressed syllables in speech. While it’s typical for a language, variations may
give clues about a speaker’s background or emotional state
* Voice quality: the way a person’s voice sounds, including factors like pitch, tone and
modulation. It can convey emotions or qualities such as confidence, aggression, or warmth
* Pauses (filled like uhmmm or unfilled): the strategic use of pauses in speech. They can
convey hesitation, emphasis (klemtoon/nadruk), or allow for listener processing
2) Variation in Body Language (visual)
- Features that you can “see” (not always encoded in the speech signal, non-auditory)
* Facial expressions: movements and configurations of the face that convey emotions and
attitudes. They are crucial for understanding the emotional context of communication
* Gaze patterns (what/who you’re looking at the fact that you look at one person means
that you are speaking to that person): where and how a person looks. They indicate attention,
interest, or may convey messages about dominance, submission, or attraction
* Hand gestures: movements of the hands that complement or emphasize verbal
communication. Hand gestures (and their frequency) vary between individuals and cultures
and can enhance the clarity and impact of a message
* Pointing: directing attention or indicating something without verbalizing/naming it. Pointing
is a nonverbal way of conveying information or expressing a specific idea
* Posture: the position and orientation of the body. Posture can indicate confidence,
openness, shyness, or even aggression. This is processed similarly to facial expressions
, * Distance (is it normal to touch/hug people or not?): proxemics, the use of personal space.
Cultural norms influence the acceptable distance between individuals during communication,
and it can convey intimacy, formality, or discomfort
Voice and Body Language
- There has especially been an interest in use of facial expressions; often without studying auditory
prosodic features (here we focus on the combined use of those 2 sets of features)
- What can be signalled in one modality (voice or body language) is also often cued in the other
modality, but there may be strength differences (caused by individual differences, cultural norms,
personal communication styles, or the specific context of the communication)
* The fact that signals in one modality are often mirrored in the other, indicates a connection
between the two
How important are voice and body language?
- Quintilianus (first century PC): institution oratoria
* Rethorics as the art to persuade an audience (e.g. stylistic tricks, ordering of information….)
- Rethorics was mainly dealing with oral language (e.g. Cicero)
- One important aspect of rethorics is called “pronunciatio”, which refers to various forms of
nonverbal communication: intonation, but also body language, facial expressions, gestures, etc
- Ideally, those (nonverbal) features, including voice and body language, should match the content of
the spoken utterances
* e.g. happy message with a happy voice and facial expression
- Presidental debates: current-day politicians are very much aware of the potential
importance/impact of nonverbal communication this awareness emphasizes the importance of
effectively using voice and body language to convey messages and influence audiences
Importance of nonverbal features
- It is a popular statement to say that nonverbal features account for more than 90% of the
communication
- This can probably be traced back to early experimental work of Mehrabian and colleagues on the
recognition of emotions in stimuli with conflicting cues but it is not all clear whether it generalizes
to all aspects of non-verbal communication (people can exhibit contradictory nonverbal cues: people
can smile during a funeral, but they aren’t happy)
Study of nonverbal features
- A relatively new field communication beyond verbal language
- Interesting paradox: strong intuition that nonverbal features matter a lot but we are only beginning
to learn how important they really are (and discover the extent of their influence)
- The study of nonverbal features has long been hampered by a lack of tools to record, measure or
analyse specific features
Multimodality
- Our perception of reality/the world is multimodal (in general): our perceptual system
integrates/fuses information coming from different sensory modalities (vision, hearing, touch, taste…)
Modalities in communication: verbal language, non-verbal language, visual elements, auditory
elements, etc
- Imagine a dinner, served in:
* A nice restaurant with candle light and romantic music
* A dirty stable with cobwebs and the noise of pneumatic boring machine
Your perception of the food changes, you pasta tastes differently while it’s the same pasta
Visual expressions
- Multimodal communication has been the most natural form of human communication for ages
- It is a normal situation that a speaker observes his/her addressee, and that the addressee sees the
speaker; spoken communication without visual contact is a relatively new phenomenon and still
occurs relatively infrequently
- Starting assumption: human beings do not only interact through auditory information, but also
through visual information (in addition to other sensory information)
,Relation between faces and speech
- The relation between faces and speech is multifaceted, involving the integration of visual and
auditory information to shape our perception of spoken language
- Faces are often observed in combination with speech (talking heads visual and auditory
components are coordinated)
- When people speak, we often see them speaking (facial expressions and lip movements)
- The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect: general term for the perceptual effect when an auditory and visual signal are
offered simultaneously in different locations, we build a perceptual construct suggesting that the
source of the audio is spatially related to the visual signal
* Brain: links the sound source to the visual signal
* Strong effect which humans can hardly suppress (suggests some form of “recalibration”
brein past zich aan om het verschil in visuele en auditieve locatie te overbruggen)
McGurk effect: combined a movie clip (vision only) of someone uttering /ga/ with the speech
sound of someone uttering /ba/ you perceive the sound as in between, /da/
* Discovered by accident when McGurk and his research assistant John MacDonald were
studying how children in different age categories perceive speech (are children more
interested in face or voice of their mom?)
In addition
- Faces have an impact on how we perceive the speech of others
* Cocktail party phenomenon (you can focus on one person and your conversation with that
one person in a noisy/crowded environment) we have the ability to focus on one person’s
speech in a noisy or crowded environment
* Lipreading (visual cues from the lips contribute to speech perception in challenging auditory
conditions)
- Compensatory effects: when there is noise on the auditory or visual channel the other channel
compensate or the missing or unclear information
- Distribution of vocal and visual cues in human interactions (e.g. feedback mechanisms all of the
cues provide immediate feedback to the speaker, indicating comprehension, agreement, confusion,
emotional state, etc in the listener)
* People use both verbal and nonverbal signals to convey and receive information during
social exchange
Development of communicative skills
- As children grow older, they improve their skills to communicate with other people (lexicon,
grammar, pronunciation, etc)
- They also become better in using and interpreting nonverbal features (voice and body language)
defined as the set of features that do not so much determine what people say, but rather how they
say it
- Adults are very skilled users of nonverbal features, so how does nonverbal features develop in a
growing child?
Roots
- The acquisition of nonverbal features starts very early
* Intonation patterns, rhythm and features of the voice acquired while in mother’s womb
(they hear people outside the womb speak)
* Young infants can imitate facial gestures, like tongue protrusion and mouth opening
* Infants quickly learn to integrate information coming from different modalities
- Importance of biological and physiological factors, such as:
* Preference for low-ending (lower pitch or frequency) contours air pressure, lung energy
decreases, naturally (so… intonation and rhythm are influenced by the innate biological
predispositions of infants)
* Facial expressions similar across the globe (genetically determined, innate)
How about older children?
, - But as a child grows older, nonverbal features become more functional in nature (children learn to
associate specific nonverbal forms with particular communicative or social functions)
- A child learns to associate specific forms with specific communicative or social functions, and to
manipulate forms (nodding, higher intonation at the end of a sentence for a question)
- This is related to the fact that the child becomes more socially aware
* Increasingly varied environment (family, school, society)
* Becomes more aware of another person’s perspective (Piaget)
Younger child: Still very egocentric, Older child: More other-directed (socially aware)
Social awareness
- Nonverbal features may reveal differences in social awareness between younger children, older
children and adults (working hypothesis)
How do we acquire nonverbal features?
- Nonverbal features are important and multifunctional nonverbal features encompass a range of
cues such as facial expressions, body language, gestures, and tone of voice, all of which play a crucial
role in communication. Understanding how individuals acquire these nonverbal features involves
considering the interplay between nature (biological factors) and nurture (environmental influences)
- How do people acquire the use and function of such features?
* By nature? (nonverbal features are pre-wired, genetically programmed)
* By nurture? (nonverbal features learned by exposure and experience)
Work into cues to basic emotions suggests a strong genetic, biological basis
* Baby: crying when sad
* Dog: happy when tail moves
As individuals grow and their social awareness expands, nonverbal features evolve to play
a more diverse role in communication (beyond the simple expression of basic emotions)
Blind people do it: genetic basis for expressions (more similar between family members), blind
people also produce expressions towards each other even tho they lack visual exposure to facial
expression, this suggests that there is indeed a genetic foundation for the use of nonverbal features
So….
- There is probably a genetic basis for the use of auditory and facial expressions, especially related to
the expression of basic emotions
- But these expressions become more relevant for a wider range of communicative functions
(nonverbal communication evolves as individuals grow and become more social awareness as they
navigate various social contexts and relationships, they learn to adapt and employ nonverbal cues for
purposes beyond basic survival instincts: expressing emotions, conveying intentions, establishing
rapport, negotiating social hierarchies, and facilitating cooperation, among others)
So, there is probably a genetic basis for nonverbal communication but they adapt later in live and
serve a wider range of communicative functions (due to the growth of social awareness)
* There is an interplay between genetic factors and environmental influences
Example of a study (surprise experiment)
- Most of the presented claims are based on empirical work ecologically valid (representative)
- Many studies use a game-based approach (people are unaware of the real purpose of the study;
they do not consciously reflect on their non-verbal features, more natural and spontaneous behavior)
Example: study on the expression and perception of surprise
- How do speakers behave nonverbally in expected (normal) versus unexpected (surprise) contexts?
* How are surprise events cued auditorily and visually?
* Can we distinguish positive from negative surprise events?
* Are there degree differences in the production of surprise?
* Is surprise signalled differently in social vs non-social contexts?
