In this document, you will find my notes from all lectures. All material covered in the lectures can be found in this document. It is written in English.
In this document you will find my notes of all of the lectures. All material covered in the lectures can be found in this document. It was writ...
Inhoud
College 1: Introduction...........................................................................................................................2
College 2: What to measure?.................................................................................................................8
College 3: Accents and prominence......................................................................................................11
College 4: Manipulation of speech.......................................................................................................18
College 5: Chuncking............................................................................................................................22
College 6: Synthesis..............................................................................................................................30
College 7: Confidence marking.............................................................................................................35
College 8: Error handling and feedback................................................................................................45
College 9: Emotion and surprise...........................................................................................................53
College 10: Stance and Irony................................................................................................................64
College 11: Mimicry..............................................................................................................................70
College 12: Metaphors..........................................................................................................................79
,College 1: Introduction
Today’s programme
Introduction of basic concepts
Regarding voice and body language
o How important are voice and body language?
o How do voice and body language interact?
o How do voice and body language develop with age?
o How universal are aspects of voice and body language?
Example of a study
Introduction of basic concepts
The meaning of a sentence
The same sentence can have different extra connotations
A sentence can be used in different ways so that extra meanings are generated (not
expressed through words or syntax)
The extra information can be derived from context or from the way a sentence is
uttered. E.g. ‘my girlfriend is pregnant’
Non-verbal communication
Informal definition: “The ensemble of features that do not determine what you say,
but how you say it”
Two forms:
o Variation in Voice
o Variation in Body Language
Voice
Features that you can “hear” (encoded in the speech signal itself)
o Intonation (speech melody)
o Loudness, energy (laud and fast talking)
o Tempo (talking fast or slow)
o Rhythm (different with different languages, like Italian have a different rhythm
than Dutch)
o Voice quality (warm voice, sharp voice)
o Pauses (‘uuuhm’ then you fill the pause but you don’t really say anything)
Body language
Features that you can “see” (not always encoded in the speech signal)
o Facial expressions
o Gaze patterns (waar je heen kijkt)
o Hand gestures (different among cultures)
o Pointing (to an object, who don’t see it, don’t know it)
o Posture (shy, aggressive)
o Distance (also culturally dependent)
Voice and body language
There has especially been an interest in use of facial expressions; often without
studying auditory prosodic features
, In this series of lectures, focus on the combined use of those 2 sets of features
What can be signalled in one modality is also often cued in the other modality, but
there may be strength differences
How important are voice and body language?
Long time ago
Quintilianus (first century PC): institution oratoria
Rethorics as the art to persuade an audience (e.g. stylistic tricks, ordering of
information)
Rethorics
Rethorics was mainly dealing with oral language
One more important aspect of rethorics is called “pronunciation”, which refer to
various forms of nonverbal communication: intonation, but also body language, facial
expressions, gestures
In a museum: it is painting from a long time ago, but you still recognize the facial
expressions
Ideally, those features should match the content of the spoken utterances (e.g. happy
message with a happy voice and face)
Presidential debates
Current-day politicians are very much aware of the potential importance of
nonverbal communication
Importance of nonverbal features
It is a popular statement to say that nonverbal features account for more than 90%
of the communication.
This can probably be traced back to early experimental work of Mehrabian and
colleagues on the recognition of emotions in stimuli with conflicting cues…
But it is not at all clear whether it generalizes to all aspects of non-verbal
communication
Study of nonverbal features
A relatively new field
Interesting paradox
o Strong intuition that nonverbal features matter a lot
o But we are only beginning to learn how important they really are
The study of nonverbal features has long been hampered by a lack of tools to record,
measure or analyze specific features
How do voice and body language interact?
Multimodality
Our perception of reality is multimodal; our perceptual system integrates/fuses
information coming from different sensory modalities (vision, hearing, touch, taste)
Imagine a dinner, served in:
o A nice restaurant with candle light and romantic music
o A dirty stable with cobwebs and the noise of a ps.
, o The same pasta will taste different
Visual information is very important, specifically signals coming from a speaker’s face
Visual expressions
Multimodal communication has been the most natural form of human
communication for ages
It is a normal situation that a speaker observes his/her addressee, and that the
addressee sees the speaker; spoken communication without contact is a relatively
new phenomenon and still occurs relatively infrequently
o Zoals bellen?
Starting assumption: human beings do not only interact through auditory
information, but also through visual information (in addition to other sensory
information)
Relation between faces and speech
Faces are often observed in combination with speech (talking hands)
When people speak, we often see them speaking
The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect
You connect what is moving with the sound
Buikspreken effect
General term for the following perceptual effect: when an auditory and visual signal
are offered simultaneously in different locations, we build a perceptual construct
suggesting that the source of the audio is spatially related to the visual signal
Strong effect which humans can hardly suppress
Suggests some form of recalibration
McGurk effect
Ogen dicht: je hoort ba ba
Ogen open: da da
Open-dicht-open: je hoort verschillende dingen
McGurk effect
Discovered by accident when Harry McGurk (psychologist of the University of Surrey)
and his research assistant John MacDonald were studying how children in different
age categories perceive speech
Combined with a movie clip (vision only) of someone uttering ‘ga’ with the speech
sound of someone uttering ‘ba’
Perceived as ‘da’
In addition
Faces have an impact on how we perceive the speech of others
o Cocktail party phenomenon
Je kunt je focussen op 1 persoon ook al is het geluid van iedereen even
hard
o Lipreading
Compensatory effects: when there is noise on the auditory or visual channel
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller anoukdegroot1. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.99. You're not tied to anything after your purchase.
