Multimodal Communication
Summary slides
Lecture 1
The Multimodal Parallel Architecture
Modality
Humans are equipped with three modalities:
- Graphic
- Vocal
- Bodily
Each allows us to:
- Express thoughts
- Communicate thoughts
Grammar
Grammar is a system to combine smaller elements into larger sequences. There grammars
range from simple (“one-unit”) to complex (“recursive”).
- Visual grammar (I-P-R)
- Verbal grammar (word tree)
- Bodily grammar (utterance)
Grammar is a system to combine smaller elements into larger sequences. In language (our
most familiar grammar): The smaller units are the words of the language. The larger
sequence is ‘an utterance’, or ‘a sentence’
Many but not all expressions ‘use’ grammar: NO: ‘Coffee?’ or ‘You... coffee?’
Some expressions even don’t ‘have’ grammar: KA... ZAM!!
As a rule of thumb:
A grammar in a sequence is ‘present’ when the sequence offers options for ordering the
smaller units, that you feel are ‘not right’.
Meaning
,Conceptual structure ‘is’ meaning. The thoughts we express and communicate by using
modalities and grammars.
As far as meaning is concerned (and it is concerned VERY far!!) modalities do not matter that
much.
We say: ‘meaning is modality-agnostic.’
We mean: ‘each modality is capable of ‘waking up’ a thought in our minds – such as “tree”’.
But as we will see some modalities are better to evoke some kinds of thoughts, yet less to
evoke other kinds of thoughts. That is the whole point of humans having a multimodal
communicative faculty!
Use of MPA
- Monomodal expressions
o Single words OR gestures OR icons OR emoji WITH no grammar.
o Complex OR figurative images WITH no grammar, link directly to the meaning.
o Abstract art (‘play’ WITH modality, no grammar, no meaning).
- Sequential images (strips)
o Modality, grammar, and meaning.
- Spoken (written) sentences
o Modality, grammar, and meaning.
- Musical phrases
o Musical grammar is complex, and hierarchical, and uses levels like metrical
grouping, pitch, etc.
- Dance
o ‘Dance grammar’ uses sequences of movements, constrained by certain rules.
Semantic weight
- Balanced when: text and image support each other.
- Imbalanced when: text or image can be left out.
modality grammar meaning
Spoken language Vocal Yes Yes
Visual language Graphic Yes Yes
Sign language Bodily Yes Yes
Words Vocal No Yes
Single images Graphic No Yes
Gestures Bodily No Yes
Music Sound Yes Yes
Dance Bodily Yes No
Abstract art Graphic/vocal/sound Yes No
Silence No No
Film/video Light Expressions like these certainly must be
Photography Light perceived, so they do use modalities.
Sampling Auditory But they ‘record’ natural modality’s
expressions. So, they have no meaning or
grammar on their own. Instead, they ‘lend’ the
, grammar and meaning from the modalities
they record. We call those ‘tool-mediated’
modalities.
Why use multimodal?
- To communicate the contents of this traffic sign, visuals (, and) work much more
efficiently!
- For other meanings, verbal messages have better credentials.
- Multimodality allows you to ‘optimize’ your communication!
Reference types
- Iconic (X)
- Indexical ()
- Symbolical (word)
Lecture 2
Modality
Descriptions in dictionaries Examples Definitions
1. State of being, existence, - Pictorial signs A socially and culturally shaped
quale, quality, etc. - Written signs resource for making meaning (Gunther
2. Expression of how - Layout Kress, 2010).
something might be, or should - Spoken signs
be, is allowed to be, etc. - Gestures (...) presentation media or modes
3. Way of doing something - Sounds (Bateman, Wildfeuer & Hiippala, 2017)
4. Channel to transmit signs - Music
5. Mode of transport - Color “A sign system interpretable because
6. Path of communication - Smells of specific perception processes.”
between humans or humans - Tastes (Forceville, 2009)
and machines - Touches
7. Code - Moving
8. Medium - Film
Etc. - Photos
- Image
Etc.
A definition of modality should specify:
1. Stimulus type (the physical stuff that is ‘out there’)
2. Sensory process (how we perceive the signal)
3. Production process (how we produce the signal ourselves)
4. Mental organization (how modality-properties are mentally represented)
(This is what Cohn & Schilperoord Ch. 2 is (mostly) about)
A definition of multimodality should specify:
5. The way modalities interact through spatial and/or temporal organization.
6. The way modalities interface
(This is what Cohn 2013 is (partly) about)
Summary slides
Lecture 1
The Multimodal Parallel Architecture
Modality
Humans are equipped with three modalities:
- Graphic
- Vocal
- Bodily
Each allows us to:
- Express thoughts
- Communicate thoughts
Grammar
Grammar is a system to combine smaller elements into larger sequences. There grammars
range from simple (“one-unit”) to complex (“recursive”).
- Visual grammar (I-P-R)
- Verbal grammar (word tree)
- Bodily grammar (utterance)
Grammar is a system to combine smaller elements into larger sequences. In language (our
most familiar grammar): The smaller units are the words of the language. The larger
sequence is ‘an utterance’, or ‘a sentence’
Many but not all expressions ‘use’ grammar: NO: ‘Coffee?’ or ‘You... coffee?’
Some expressions even don’t ‘have’ grammar: KA... ZAM!!
As a rule of thumb:
A grammar in a sequence is ‘present’ when the sequence offers options for ordering the
smaller units, that you feel are ‘not right’.
Meaning
,Conceptual structure ‘is’ meaning. The thoughts we express and communicate by using
modalities and grammars.
As far as meaning is concerned (and it is concerned VERY far!!) modalities do not matter that
much.
We say: ‘meaning is modality-agnostic.’
We mean: ‘each modality is capable of ‘waking up’ a thought in our minds – such as “tree”’.
But as we will see some modalities are better to evoke some kinds of thoughts, yet less to
evoke other kinds of thoughts. That is the whole point of humans having a multimodal
communicative faculty!
Use of MPA
- Monomodal expressions
o Single words OR gestures OR icons OR emoji WITH no grammar.
o Complex OR figurative images WITH no grammar, link directly to the meaning.
o Abstract art (‘play’ WITH modality, no grammar, no meaning).
- Sequential images (strips)
o Modality, grammar, and meaning.
- Spoken (written) sentences
o Modality, grammar, and meaning.
- Musical phrases
o Musical grammar is complex, and hierarchical, and uses levels like metrical
grouping, pitch, etc.
- Dance
o ‘Dance grammar’ uses sequences of movements, constrained by certain rules.
Semantic weight
- Balanced when: text and image support each other.
- Imbalanced when: text or image can be left out.
modality grammar meaning
Spoken language Vocal Yes Yes
Visual language Graphic Yes Yes
Sign language Bodily Yes Yes
Words Vocal No Yes
Single images Graphic No Yes
Gestures Bodily No Yes
Music Sound Yes Yes
Dance Bodily Yes No
Abstract art Graphic/vocal/sound Yes No
Silence No No
Film/video Light Expressions like these certainly must be
Photography Light perceived, so they do use modalities.
Sampling Auditory But they ‘record’ natural modality’s
expressions. So, they have no meaning or
grammar on their own. Instead, they ‘lend’ the
, grammar and meaning from the modalities
they record. We call those ‘tool-mediated’
modalities.
Why use multimodal?
- To communicate the contents of this traffic sign, visuals (, and) work much more
efficiently!
- For other meanings, verbal messages have better credentials.
- Multimodality allows you to ‘optimize’ your communication!
Reference types
- Iconic (X)
- Indexical ()
- Symbolical (word)
Lecture 2
Modality
Descriptions in dictionaries Examples Definitions
1. State of being, existence, - Pictorial signs A socially and culturally shaped
quale, quality, etc. - Written signs resource for making meaning (Gunther
2. Expression of how - Layout Kress, 2010).
something might be, or should - Spoken signs
be, is allowed to be, etc. - Gestures (...) presentation media or modes
3. Way of doing something - Sounds (Bateman, Wildfeuer & Hiippala, 2017)
4. Channel to transmit signs - Music
5. Mode of transport - Color “A sign system interpretable because
6. Path of communication - Smells of specific perception processes.”
between humans or humans - Tastes (Forceville, 2009)
and machines - Touches
7. Code - Moving
8. Medium - Film
Etc. - Photos
- Image
Etc.
A definition of modality should specify:
1. Stimulus type (the physical stuff that is ‘out there’)
2. Sensory process (how we perceive the signal)
3. Production process (how we produce the signal ourselves)
4. Mental organization (how modality-properties are mentally represented)
(This is what Cohn & Schilperoord Ch. 2 is (mostly) about)
A definition of multimodality should specify:
5. The way modalities interact through spatial and/or temporal organization.
6. The way modalities interface
(This is what Cohn 2013 is (partly) about)
