Bateman, John A./Wildfeuer, Janina/Hiippala, Tuomo (2017): Multi modality. Foundati ons,
Research and Analysis. A Problem-Oriented Introducti on. Berlin, New York: de Gruyter.
1. Introduction: the challenge of multimodality
“Multimodality is a way of characterizing communicative situations (considered very broadly) which
rely upon combinations of different ‘forms’ of communication to be effective.”
Focusing on particular areas has been immensely important for gaining deeper knowledge of the
individual forms addressed. However, in many disciplines the awareness is growing that it is not
sufficient to focus on individual ‘forms of expression’ within a communicative situation as if these
were occurring alone* multimodality.
* E.g., film studies move to consider TV and other audiovisual media.
The mechanisms of combination involved when distinct forms of expression appear together often
remain obscure move beyond a superficial recognition that there appears to be ‘something’ being
combined, and ask what it means to ‘combine’ forms of expression at a more fundamental level
what is the ‘multi’ in ‘multimodality’?
Add understandings of ways of dealing with the particular challenges and questions that
combining diverse forms of meaning-making raises.
1.1 First steps … a multimodal turn?
All ‘multimodal’ artefacts or performances pose significant and interesting challenges for systematic
investigation – it is not obvious which methods, disciplines or frameworks can help. Combinations of
diverse forms is now considered the norm in many of the current media, due to
changing media landscape in our daily life and professional environments;
the increasing number of products and artefacts resulting from this ever-growing landscape;
and our habits of using these as well.
It is often the ‘co-contextualization’ of forms that provides the key to their successful and effective
use.
Awareness of the role of multimodality for everyday communication practices and within the
broader media landscape as a whole is by no means limited to academic discussions a
‘multimodal turn’, the awareness of multimodality working its way not only into academic
discussions (the structure and application of forms) but also into our everyday ways of thinking and
acting.
At the broadest level, there is the growing orientation to multimodality as a cultural phenomenon
the emergence ‘super-media’, capable of simulation or copying other media* = media convergence:
there are no longer separate media, but media that are capable of doing the jobs of many.
* E.g., an iPad showing a newspaper or film.
The need for the skill of being able to engage productively and critically with combinations of quite
different forms of expression is recognized at all levels. For example, many national school curricula
now include the requirement that combinations of forms (e.g., text, film and images) be addressed,
without saying much about how this can be done.
The multimodal turn is then the willingness and perceived need to examine combinations of
expressive resources explicitly and systematically. This is fast working its way through all
disciplines and practices.
1
,1.2 The journey ahead
A metaphor that is commonly employed in work on multimodality is that of meaning multiplication.
An emphasis on the fact that multimodality is more than just putting two (or more) modes next to
each other more seems to emerge from the combination than just the simple sum of the parts.
Then the question is what the nature of this ‘more’ might be? multiplication of possibilities /
multiplying meanings.
The multiplication metaphor has been a useful frame for pushing conceptualizations of multimodal
situations further, but it has some drawbacks. Most problematic is the following: in order to engage
in multiplication, things need to multiply, as ‘modes’ - consequently, the vast majority of existing
discussions simply list examples of the modes they consider and proceed to the next step. However,
this does not offer much guidance for how to take the investigation further causing challenges.
1. Various forms will generally have very different properties from each other – using forms
together then requires that ways are found of making sense of heterogeneous ways of
making meaning.
2. There is a presupposition that things to multiply can be found in the first place – if
multimodality is seen as a kind of multiplication of meaning, then it suggests that the
meanings are somehow fixed and only need to be combined.
The use of different modes together can have an influence on what each is contributing modes
need to be interpreted with respect to one another (not on their own) and cannot be considered
independently, making interdependence a difficult issue.
The fundamental property of multimodal ‘meaning making’ makes the task of identifying semiotic
modes considerably harder. Almost anything that one considers as potentially contributing to a
meaning-making situation may come to be treated as a ‘mode’.
The current lack of more inclusive accounts of the nature and operation of multimodal
communicative activities dramatically compromises our abilities to deal with situations where
multimodality is in play, despite the fact that these situations are already frequent and, in many
contexts, even constitute the norm.
2. Recognizing multimodality: origins and inspirations
Achieving a principled stance with respect to the broad phenomena of multimodality is challenging in
its own right. Here, a beginning is made to lay the foundation stones for any approach to
multimodality that subsequently is tried to apply or build, regardless of existing disciplinary
orientations. Clarity must be achieved about what constitutes problems and challenges of
multimodality: how can such problems and challenges arise and what do they involve?
2.1 The ‘problem space’ of multimodality as such
In current research of multimodality, researchers largely select methods on the basis of personal
preferences and circumstances of training. What is needed for a field of multimodality to emerge
is therefore rather different: characterize the problems and then select approach or method.
A foundation will be built on the ‘problem space’ of multimodality, with the starting points of
materiality, language, semiotics and society. Working foundationally is intended to help you pursue
multimodal analysis regardless of disciplinary orientation and what the particular problem addressed
is.
2
, The field of semiotics have attempted to answer the foundational question “how can it be that
something comes to mean something for someone?”
2.2 Materiality and the senses: sound
As with all the areas that will be addressed, there are a lot of disciplines that take this particular area
as their point of departure. Here, the levels of descriptive abstraction are involved: dealing with
something close to materiality, to actual physical events, or with objects/arts of interpretation, based
on evidence or regularities but going beyond the merely physical?
2.2.1 Physical acoustics
Sound is the result of a wave travelling through a medium, with the usual properties frequency,
amplitude, and wavelength. It is a longitudinal or compression wave: the dimension of the direction
of displacement of the material carrying the wave is the same as the direction the wave is travelling
in.
In contrast to the bare physics of sound, how it is perceived is more complex. Sound perception
comes with indications of direction, hardness (amplitude and frequency), sources, and indications of
the properties of the space we are in (e.g., echoing in a large space).
These different meanings of sound are why talking of ‘modes’ in terms of ‘sensory channels’
can be quite misleading.
In terms of sound in language, it is even more complexed, since indications of emotional state,
physical state (age, health, tiredness) and much more come along.
2.2.2 Music, sound design and more
The perception of music has specific properties.
It maintains the individual contributions of its contributing frequencies; it can profit from
combining notes, instruments and voices to form polyphonic wholes where individual
contributions can still be heard and related to one another.
Capability both of remembering sequences of sounds arranges musically and of predicting
outcomes of harmonic progressions.
Combining, for example, melody and lyrics (connection of music with the visual) builds bridges not
only between the musical form and representational contributions of the language involved but also
between other common facets of the two forms of expression, including rhythm, amplitude, and
many other properties carried by sound as a physical medium.
These connections of music with the visual raise issues for multimodality due to the reoccurring
question of how distinct forms of expression may best be combined for common purposes.
Another dimension of multimodality in sound is due to the physicality of the medium. Sounds can
have deep effects on emotion, mood and feeling, which is employed in the field of sound design.
So, some described above, there are several types of multimodality around music.
Melody and lyrics.
Music with visual.
Physicality of the medium.
Musical compositions in a visual form (e.g., sheet music).
3
