Aantekeningen van het vak Multimodal Communication van de opleiding Communicatie- en Informatiewetenschappen aan Tilburg University. Aangevuld met aantekeningen van de verplichte wetenschappelijke artikels. Het document is in het Engels geschreven.
, Grammar interactions ....................................................................................................................... 15
How do grammars interact? .......................................................................................................... 15
How are the grammars distributed relative to each other? .......................................................... 15
Case study: substitutive allocation ............................................................................................ 16
Lecture 4 (13/11/2023) ......................................................................................................................... 16
Conceptual semantics ....................................................................................................................... 17
Multimodal interactions .................................................................................................................... 17
Single multimodal interactions ...................................................................................................... 17
Conceptual Domain Model for single images............................................................................ 17
Types of auxiliary domains ........................................................................................................ 18
Overview.................................................................................................................................... 19
Sequential multimodal interactions .............................................................................................. 19
Conceptual Domain Model for sequential images .................................................................... 20
Cues for changes in domain constructions ................................................................................ 20
Overview.................................................................................................................................... 21
Summary ........................................................................................................................................... 21
Lecture 5 (20/11/2023) ......................................................................................................................... 21
The Multimodal Parallel Architecture ............................................................................................... 21
Modality ........................................................................................................................................ 22
Grammar........................................................................................................................................ 22
Meaning......................................................................................................................................... 22
Examples............................................................................................................................................ 23
Unimodal expressions ................................................................................................................... 23
Multimodal expressions ................................................................................................................ 23
Silence ............................................................................................................................................... 24
Lecture 6 (27/11/2023) ......................................................................................................................... 25
Case studies ....................................................................................................................................... 25
2-Image Expressions .......................................................................................................................... 25
Generalized emergent state for 2-I expressions ............................................................................ 25
Before-After messages as 2-I expressions ..................................................................................... 25
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 26
How BA-expressions make meaning.............................................................................................. 26
Example 1 .................................................................................................................................. 26
Example 2 .................................................................................................................................. 27
The generalized BA-construction ................................................................................................... 27
2
, The full-formed F-function for BA-expressions ............................................................................. 28
The nature of change..................................................................................................................... 29
Continuity and activity constraints ................................................................................................ 30
Distinct entities .......................................................................................................................... 30
Identical entities ........................................................................................................................ 30
Examples of violations of the continuity constraint .................................................................. 30
Examples of trouble with activity .............................................................................................. 31
Examples of trouble with inherited values ................................................................................ 32
Playing with modality .................................................................................................................... 32
Not all 2-I expressions are BA-expressions .................................................................................... 32
BA-features ................................................................................................................................ 33
Summary ....................................................................................................................................... 33
Internet memes as multimodal expressions ..................................................................................... 34
Comparison of the model .............................................................................................................. 34
Meme evolution ............................................................................................................................ 35
Lecture 7 (4/12/23) ............................................................................................................................... 36
Why do we have the MPA? ................................................................................................................ 36
Types of reference ......................................................................................................................... 36
How can we talk about and point at what we see? ...................................................................... 37
Spatial structure ........................................................................................................................ 37
Structures and modalities ............................................................................................................. 38
Cross-modal interactions ................................................................................................................... 38
Graphicw........................................................................................................................................ 38
Synesthesia .................................................................................................................................... 38
3
, Lecture 1 (23/10/2023)
Multimodality is what happens when an expression uses more than one modality to communicate a
message. The context of the message doesn't matter that much. Examples of multimodal messages are
posters, memes, advertisements, comic strips, diagrams, infographics, traffic signs, speech-gesture
alignment, text-emoji, personal documents, maps and rebuses.
Multimodal messages can be characterized in terms of:
• Modalities used: for example, verbal speech and bodily movements or written speech and
visual images.
• Ordering:
o Ordering in time: they happen simultaneously and are durative (they disappear).
o Ordering in space: they are non-durative (permanent).
• Senses/perceptions: for example, do you need to use your eyes only or your eyes and ears.
Example
1. Does this message use more than 1 modality?
Yes, it uses visuals (drawings) and written
speech (VROOOAAOOOOOAW, Good Year and
Jean Gration).
2. What does this expression mean? Sometimes
the meaning is a simple description of what is
seen on the image. So, what scene do we see?
a. The scene is that of a racetrack.
b. Two men are looking at some
document.
c. Race cars driving by.
d. One of the cars is sponsored by ‘good
year’.
e. They make lots of noise.
3. What part of this meaning is carried by what modality?
a. Visual: (a) the scene is that of a racetrack; (b) Two men are looking at some document;
and (c) Race cars driving by.
b. Verbal: (d) One of the cars is sponsored by ‘good year’; and (e) They make lots of noise.
4. Is the job done jointly, or mostly by one modality? Most meaning is carried by the visuals. The
written parts add to the whole, but not essentially.
5. Look closely at these 3 written parts:
a. “Good Year”: are part of what is shown here. If you were there, you would see them.
b. “VROOOAAOOOOAW”: is a written impression of what can be heard here. If you were
there, you wouldn’t see it.
c. “Jean Graton: is superimposed on the image. It’s part of neither the scene, nor the
action.
The Multimodal Parallel Architecture
The multimodal parallel architecture characterizes communicative forms as interactions of one or
several modalities, grammar types and conceptual structure. Multimodality follows from combinations
of different emergent forms, resulting in multiple activation patterns across the parallel architecture.
4
Voordelen van het kopen van samenvattingen bij Stuvia op een rij:
Verzekerd van kwaliteit door reviews
Stuvia-klanten hebben meer dan 700.000 samenvattingen beoordeeld. Zo weet je zeker dat je de beste documenten koopt!
Snel en makkelijk kopen
Je betaalt supersnel en eenmalig met iDeal, creditcard of Stuvia-tegoed voor de samenvatting. Zonder lidmaatschap.
Focus op de essentie
Samenvattingen worden geschreven voor en door anderen. Daarom zijn de samenvattingen altijd betrouwbaar en actueel. Zo kom je snel tot de kern!
Veelgestelde vragen
Wat krijg ik als ik dit document koop?
Je krijgt een PDF, die direct beschikbaar is na je aankoop. Het gekochte document is altijd, overal en oneindig toegankelijk via je profiel.
Tevredenheidsgarantie: hoe werkt dat?
Onze tevredenheidsgarantie zorgt ervoor dat je altijd een studiedocument vindt dat goed bij je past. Je vult een formulier in en onze klantenservice regelt de rest.
Van wie koop ik deze samenvatting?
Stuvia is een marktplaats, je koop dit document dus niet van ons, maar van verkoper joymennings1. Stuvia faciliteert de betaling aan de verkoper.
Zit ik meteen vast aan een abonnement?
Nee, je koopt alleen deze samenvatting voor €7,16. Je zit daarna nergens aan vast.
