Summary
Summary & Extensive Lecture Notes of Multimodal Communication
- Course
- Institution
The summary and extensive lecture notes help you prepare for each lecture and study for the final exam
[Show more]
Seller
Follow
jacobienbeekmans
Content preview
Multimodal CommunicationSummary slides
Lecture 1
The Multimodal Parallel Architecture
Modality
Humans are equipped with three modalities:
- Graphic
- Vocal
- Bodily
Each allows us to:
- Express thoughts
- Communicate thoughts
Grammar
Grammar is a system to combine smaller elements into larger sequences. There grammars
range from simple (“one-unit”) to complex (“recursive”).
- Visual grammar (I-P-R)
- Verbal grammar (word tree)
- Bodily grammar (utterance)
Grammar is a system to combine smaller elements into larger sequences. In language (our
most familiar grammar): The smaller units are the words of the language. The larger
sequence is ‘an utterance’, or ‘a sentence’
Many but not all expressions ‘use’ grammar: NO: ‘Coffee?’ or ‘You... coffee?’
Some expressions even don’t ‘have’ grammar: KA... ZAM!!
As a rule of thumb:
A grammar in a sequence is ‘present’ when the sequence offers options for ordering the
smaller units, that you feel are ‘not right’.
Meaning
,Conceptual structure ‘is’ meaning. The thoughts we express and communicate by using
modalities and grammars.
As far as meaning is concerned (and it is concerned VERY far!!) modalities do not matter that
much.
We say: ‘meaning is modality-agnostic.’
We mean: ‘each modality is capable of ‘waking up’ a thought in our minds – such as “tree”’.
But as we will see some modalities are better to evoke some kinds of thoughts, yet less to
evoke other kinds of thoughts. That is the whole point of humans having a multimodal
communicative faculty!
Use of MPA
- Monomodal expressions
o Single words OR gestures OR icons OR emoji WITH no grammar.
o Complex OR figurative images WITH no grammar, link directly to the meaning.
o Abstract art (‘play’ WITH modality, no grammar, no meaning).
- Sequential images (strips)
o Modality, grammar, and meaning.
- Spoken (written) sentences
o Modality, grammar, and meaning.
- Musical phrases
o Musical grammar is complex, and hierarchical, and uses levels like metrical
grouping, pitch, etc.
- Dance
o ‘Dance grammar’ uses sequences of movements, constrained by certain rules.
Semantic weight
- Balanced when: text and image support each other.
- Imbalanced when: text or image can be left out.
modality grammar meaning
Spoken language Vocal Yes Yes
Visual language Graphic Yes Yes
Sign language Bodily Yes Yes
Words Vocal No Yes
Single images Graphic No Yes
Gestures Bodily No Yes
Music Sound Yes Yes
Dance Bodily Yes No
Abstract art Graphic/vocal/sound Yes No
Silence No No
Film/video Light Expressions like these certainly must be
Photography Light perceived, so they do use modalities.
Sampling Auditory But they ‘record’ natural modality’s
expressions. So, they have no meaning or
grammar on their own. Instead, they ‘lend’ the
, grammar and meaning from the modalities
they record. We call those ‘tool-mediated’
modalities.
Why use multimodal?
- To communicate the contents of this traffic sign, visuals (, and) work much more
efficiently!
- For other meanings, verbal messages have better credentials.
- Multimodality allows you to ‘optimize’ your communication!
Reference types
- Iconic (X)
- Indexical ()
- Symbolical (word)
Lecture 2
Modality
Descriptions in dictionaries Examples Definitions
1. State of being, existence, - Pictorial signs A socially and culturally shaped
quale, quality, etc. - Written signs resource for making meaning (Gunther
2. Expression of how - Layout Kress, 2010).
something might be, or should - Spoken signs
be, is allowed to be, etc. - Gestures (...) presentation media or modes
3. Way of doing something - Sounds (Bateman, Wildfeuer & Hiippala, 2017)
4. Channel to transmit signs - Music
5. Mode of transport - Color “A sign system interpretable because
6. Path of communication - Smells of specific perception processes.”
between humans or humans - Tastes (Forceville, 2009)
and machines - Touches
7. Code - Moving
8. Medium - Film
Etc. - Photos
- Image
Etc.
A definition of modality should specify:
1. Stimulus type (the physical stuff that is ‘out there’)
2. Sensory process (how we perceive the signal)
3. Production process (how we produce the signal ourselves)
4. Mental organization (how modality-properties are mentally represented)
(This is what Cohn & Schilperoord Ch. 2 is (mostly) about)
A definition of multimodality should specify:
5. The way modalities interact through spatial and/or temporal organization.
6. The way modalities interface
(This is what Cohn 2013 is (partly) about)
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller jacobienbeekmans. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.60. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
75057 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now