Samenvatting
Summary & Extensive Lecture Notes of Multimodal Communication
- Instelling
- Tilburg University (UVT)
The summary and extensive lecture notes help you prepare for each lecture and study for the final exam
[Meer zien]Voorbeeld 3 van de 25 pagina's
Voorbeeld 3 van de 25 pagina's
In winkelwagengesponsord bericht van onze partner
Verkoper
Volgen
Voorbeeld van de inhoud
Multimodal CommunicationSummary slides
Lecture 1
The Multimodal Parallel Architecture
Modality
Humans are equipped with three modalities:
- Graphic
- Vocal
- Bodily
Each allows us to:
- Express thoughts
- Communicate thoughts
Grammar
Grammar is a system to combine smaller elements into larger sequences. There grammars
range from simple (“one-unit”) to complex (“recursive”).
- Visual grammar (I-P-R)
- Verbal grammar (word tree)
- Bodily grammar (utterance)
Grammar is a system to combine smaller elements into larger sequences. In language (our
most familiar grammar): The smaller units are the words of the language. The larger
sequence is ‘an utterance’, or ‘a sentence’
Many but not all expressions ‘use’ grammar: NO: ‘Coffee?’ or ‘You... coffee?’
Some expressions even don’t ‘have’ grammar: KA... ZAM!!
As a rule of thumb:
A grammar in a sequence is ‘present’ when the sequence offers options for ordering the
smaller units, that you feel are ‘not right’.
Meaning
,Conceptual structure ‘is’ meaning. The thoughts we express and communicate by using
modalities and grammars.
As far as meaning is concerned (and it is concerned VERY far!!) modalities do not matter that
much.
We say: ‘meaning is modality-agnostic.’
We mean: ‘each modality is capable of ‘waking up’ a thought in our minds – such as “tree”’.
But as we will see some modalities are better to evoke some kinds of thoughts, yet less to
evoke other kinds of thoughts. That is the whole point of humans having a multimodal
communicative faculty!
Use of MPA
- Monomodal expressions
o Single words OR gestures OR icons OR emoji WITH no grammar.
o Complex OR figurative images WITH no grammar, link directly to the meaning.
o Abstract art (‘play’ WITH modality, no grammar, no meaning).
- Sequential images (strips)
o Modality, grammar, and meaning.
- Spoken (written) sentences
o Modality, grammar, and meaning.
- Musical phrases
o Musical grammar is complex, and hierarchical, and uses levels like metrical
grouping, pitch, etc.
- Dance
o ‘Dance grammar’ uses sequences of movements, constrained by certain rules.
Semantic weight
- Balanced when: text and image support each other.
- Imbalanced when: text or image can be left out.
modality grammar meaning
Spoken language Vocal Yes Yes
Visual language Graphic Yes Yes
Sign language Bodily Yes Yes
Words Vocal No Yes
Single images Graphic No Yes
Gestures Bodily No Yes
Music Sound Yes Yes
Dance Bodily Yes No
Abstract art Graphic/vocal/sound Yes No
Silence No No
Film/video Light Expressions like these certainly must be
Photography Light perceived, so they do use modalities.
Sampling Auditory But they ‘record’ natural modality’s
expressions. So, they have no meaning or
grammar on their own. Instead, they ‘lend’ the
, grammar and meaning from the modalities
they record. We call those ‘tool-mediated’
modalities.
Why use multimodal?
- To communicate the contents of this traffic sign, visuals (, and) work much more
efficiently!
- For other meanings, verbal messages have better credentials.
- Multimodality allows you to ‘optimize’ your communication!
Reference types
- Iconic (X)
- Indexical ()
- Symbolical (word)
Lecture 2
Modality
Descriptions in dictionaries Examples Definitions
1. State of being, existence, - Pictorial signs A socially and culturally shaped
quale, quality, etc. - Written signs resource for making meaning (Gunther
2. Expression of how - Layout Kress, 2010).
something might be, or should - Spoken signs
be, is allowed to be, etc. - Gestures (...) presentation media or modes
3. Way of doing something - Sounds (Bateman, Wildfeuer & Hiippala, 2017)
4. Channel to transmit signs - Music
5. Mode of transport - Color “A sign system interpretable because
6. Path of communication - Smells of specific perception processes.”
between humans or humans - Tastes (Forceville, 2009)
and machines - Touches
7. Code - Moving
8. Medium - Film
Etc. - Photos
- Image
Etc.
A definition of modality should specify:
1. Stimulus type (the physical stuff that is ‘out there’)
2. Sensory process (how we perceive the signal)
3. Production process (how we produce the signal ourselves)
4. Mental organization (how modality-properties are mentally represented)
(This is what Cohn & Schilperoord Ch. 2 is (mostly) about)
A definition of multimodality should specify:
5. The way modalities interact through spatial and/or temporal organization.
6. The way modalities interface
(This is what Cohn 2013 is (partly) about)
Voordelen van het kopen van samenvattingen bij Stuvia op een rij:
Verzekerd van kwaliteit door reviews
Stuvia-klanten hebben meer dan 700.000 samenvattingen beoordeeld. Zo weet je zeker dat je de beste documenten koopt!
Snel en makkelijk kopen
Je betaalt supersnel en eenmalig met iDeal, creditcard of Stuvia-tegoed voor de samenvatting. Zonder lidmaatschap.
Focus op de essentie
Samenvattingen worden geschreven voor en door anderen. Daarom zijn de samenvattingen altijd betrouwbaar en actueel. Zo kom je snel tot de kern!
Veelgestelde vragen
Wat krijg ik als ik dit document koop?
Je krijgt een PDF, die direct beschikbaar is na je aankoop. Het gekochte document is altijd, overal en oneindig toegankelijk via je profiel.
Tevredenheidsgarantie: hoe werkt dat?
Onze tevredenheidsgarantie zorgt ervoor dat je altijd een studiedocument vindt dat goed bij je past. Je vult een formulier in en onze klantenservice regelt de rest.
Van wie koop ik deze samenvatting?
Stuvia is een marktplaats, je koop dit document dus niet van ons, maar van verkoper jacobienbeekmans. Stuvia faciliteert de betaling aan de verkoper.
Zit ik meteen vast aan een abonnement?
Nee, je koopt alleen deze samenvatting voor €6,99. Je zit daarna nergens aan vast.
Is Stuvia te vertrouwen?
4,6 sterren op Google & Trustpilot (+1000 reviews)
Afgelopen 30 dagen zijn er 75057 samenvattingen verkocht
Opgericht in 2010, al 14 jaar dé plek om samenvattingen te kopen