Multimodal Communication lecture notes (880419-M-6)
15 views 1 purchase
Course
Multimodal Communication (880419M6)
Institution
Tilburg University (UVT)
Notes from the Multimodal Communication course in the Communication and Information Sciences course at Tilburg University. Supplemented with notes from the mandatory scientific articles. The document is written in English.
, Grammar interactions ....................................................................................................................... 15
How do grammars interact? .......................................................................................................... 15
How are the grammars distributed relative to each other? .......................................................... 15
Case study: substitutive allocation ............................................................................................ 16
Lecture 4 (13/11/2023) ......................................................................................................................... 16
Conceptual semantics ....................................................................................................................... 17
Multimodal interactions .................................................................................................................... 17
Single multimodal interactions ...................................................................................................... 17
Conceptual Domain Model for single images............................................................................ 17
Types of auxiliary domains ........................................................................................................ 18
Overview.................................................................................................................................... 19
Sequential multimodal interactions .............................................................................................. 19
Conceptual Domain Model for sequential images .................................................................... 20
Cues for changes in domain constructions ................................................................................ 20
Overview.................................................................................................................................... 21
Summary ........................................................................................................................................... 21
Lecture 5 (20/11/2023) ......................................................................................................................... 21
The Multimodal Parallel Architecture ............................................................................................... 21
Modality ........................................................................................................................................ 22
Grammar........................................................................................................................................ 22
Meaning......................................................................................................................................... 22
Examples............................................................................................................................................ 23
Unimodal expressions ................................................................................................................... 23
Multimodal expressions ................................................................................................................ 23
Silence ............................................................................................................................................... 24
Lecture 6 (27/11/2023) ......................................................................................................................... 25
Case studies ....................................................................................................................................... 25
2-Image Expressions .......................................................................................................................... 25
Generalized emergent state for 2-I expressions ............................................................................ 25
Before-After messages as 2-I expressions ..................................................................................... 25
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 26
How BA-expressions make meaning.............................................................................................. 26
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 27
The generalized BA-construction ................................................................................................... 27
2
, The full-formed F-function for BA-expressions ............................................................................. 28
The nature of change..................................................................................................................... 29
Continuity and activity constraints ................................................................................................ 30
Distinct entities .......................................................................................................................... 30
Identical entities ........................................................................................................................ 30
Examples of violations of the continuity constraint .................................................................. 30
Examples of trouble with activity .............................................................................................. 31
Examples of trouble with inherited values ................................................................................ 32
Playing with modality .................................................................................................................... 32
Not all 2-I expressions are BA-expressions .................................................................................... 32
BA-features ................................................................................................................................ 33
Summary ....................................................................................................................................... 33
Internet memes as multimodal expressions ..................................................................................... 34
Comparison of the model .............................................................................................................. 34
Meme evolution ............................................................................................................................ 35
Lecture 7 (4/12/23) ............................................................................................................................... 36
Why do we have the MPA? ................................................................................................................ 36
Types of reference ......................................................................................................................... 36
How can we talk about and point at what we see? ...................................................................... 37
Spatial structure ........................................................................................................................ 37
Structures and modalities ............................................................................................................. 38
Cross-modal interactions ................................................................................................................... 38
Graphicw........................................................................................................................................ 38
Synesthesia .................................................................................................................................... 38
3
, Lecture 1 (23/10/2023)
Multimodality is what happens when an expression uses more than one modality to communicate a
message. The context of the message doesn't matter that much. Examples of multimodal messages are
posters, memes, advertisements, comic strips, diagrams, infographics, traffic signs, speech-gesture
alignment, text-emoji, personal documents, maps and rebuses.
Multimodal messages can be characterized in terms of:
• Modalities used: for example, verbal speech and bodily movements or written speech and
visual images.
• Ordering:
o Ordering in time: they happen simultaneously and are durative (they disappear).
o Ordering in space: they are non-durative (permanent).
• Senses/perceptions: for example, do you need to use your eyes only or your eyes and ears.
Example
1. Does this message use more than 1 modality?
Yes, it uses visuals (drawings) and written
speech (VROOOAAOOOOOAW, Good Year and
Jean Gration).
2. What does this expression mean? Sometimes
the meaning is a simple description of what is
seen on the image. So, what scene do we see?
a. The scene is that of a racetrack.
b. Two men are looking at some
document.
c. Race cars driving by.
d. One of the cars is sponsored by ‘good
year’.
e. They make lots of noise.
3. What part of this meaning is carried by what modality?
a. Visual: (a) the scene is that of a racetrack; (b) Two men are looking at some document;
and (c) Race cars driving by.
b. Verbal: (d) One of the cars is sponsored by ‘good year’; and (e) They make lots of noise.
4. Is the job done jointly, or mostly by one modality? Most meaning is carried by the visuals. The
written parts add to the whole, but not essentially.
5. Look closely at these 3 written parts:
a. “Good Year”: are part of what is shown here. If you were there, you would see them.
b. “VROOOAAOOOOAW”: is a written impression of what can be heard here. If you were
there, you wouldn’t see it.
c. “Jean Graton: is superimposed on the image. It’s part of neither the scene, nor the
action.
The Multimodal Parallel Architecture
The multimodal parallel architecture characterizes communicative forms as interactions of one or
several modalities, grammar types and conceptual structure. Multimodality follows from combinations
of different emergent forms, resulting in multiple activation patterns across the parallel architecture.
4
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller joymennings1. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.68. You're not tied to anything after your purchase.
