ARM samenvatting artikelen
Artikel 1 week 1 – observational methods
Goal: Define what distinguishes observational from other methods and to delineate particular issues
users of observational methods must resolve.
Observational methodology: a particular systematic approach to the business of quantifying
behaviour. The aim is to define beforehand various forms of behaviour – forms that are embodied in
predefined behavioural codes – and then ask observers to record whenever behaviour is
corresponding to any of the predefined codes occurs.
Observer accuracy is a major concern, and typically considerable effort is spent training observers so
that they will produce essentially similar records.
There is nothing inherently nonexperimental or experimental about observational methods.
Narrative reports subjective lack of replicability. Narrative reports often serve users of
observational methodology during the pilot phase of a new investigation.
Rating scales more advantages compared to observational research; e.g. less time consuming.
Ratings are especially useful when investigators want to describe individual differences in
behavioural style, but observational methods are clearly better suited when one wants to understand
process or the mechanisms of social patterns.
Coding scheme is the basic measuring instrument of observational research. The coding scheme is a
hypothesis of sorts, a lens with which an investigator has chosen to view the world. Difficult process,
so theoretical underpinnings and assumptions of the coding scheme should be questioned.
There are a number of distinctions that we think almost anyone could find useful while
engaged in the often arduous and time-consuming task of coding scheme development:
- Momentary versus Duration events
o Momentary = frequency behaviours (burps, yelps)
o Duration = duration behaviours (REM-sleep, hunting etc.)
- The case of behavioural states
o Behavioural state is a particular or specific kind of duration event, and can
prove extremely useful in observational research for 3 reasons
1. It’s an integrative concept that protects investigators from
becoming lost in a morass of detail
2. Behavioural state may qualify or modify the meaning of other
acts
3. A set of behavioural state codes is likely to be mutually
exclusive and exhaustive.
- Mutually exclusive and exhaustive codes
o Exclusive: in each case only one code from the scheme can be associated with a
particular event
o Exhaustive: for each event some code in the scheme applies
- Physically versus socially based codes
o Physically: classify behaviour with clear and well-understood roots in the
organism’s physiology (often more time consuming to learn and apply)
o Socially: deal with behaviour whose very classification depends more on ideas
in the minds of the researchers than on mechanisms in the body more
disagreement
, -
Authors think that it’s more important that codes fit the research question at hand than that they be
“objective”.
Method: the logic of the way a coding scheme is applied for the passing stream of behaviour,
extracting from it numbers that can be used for subsequent quantitative analysis.
Two distinctions in developing appropriate recording methods:
- What is coded? Events versus intervals:
o Coding of events if often more complex detect that a codable event has
occurred + code that event.
- Continuous versus intermittent recording:
o Continuous: if successive events or intervals are coded
o Intermittent: if there are gaps between the events or intervals coded
Four strategies for recording observational data (table article):
1a. Coding events without any time information. Simply note each time a codable event occurs.
Momentary or mutually exclusive and exhaustive behavioural states
1b. Coding events; onset, offset, or pattern change times recorded
Expensive and difficult to use electronic instrumentation
Advantage: completeness of the record and the operations for analysis that completeness
allows for
A variety of standard descriptive statistics for the various events can be derived, and also
quite different kinds of events, like infant’s affective expressions and engagement states can
be related to each other.
1c. Timing pattern changes
Observers can note whenever a codable event begins or they can note the status of codes in
each set whenever there is any change in the overall pattern
Practical only when there are relatively few sets of codes
2. Coding intervals
More frequently used, especially in older literature
Observation during period of time. Period is divided into a number of relatively brief
intervals. Observers categorize each successive interval or else note which codable events, if
any, occurred during each successive interval.
Simplicity and low cost
Neither frequency nor duration information may be rendered very accurately length of
interval plays a role
Observers find it often easier to categorize a clearly bounded interval than to determine
exactly when in the stream of behaviour a particular event began and ended.
3. Cross-classifying events
Least used
Intermittent
Like tallying momentary events in that observers are instructed to note whenever a
particular kind of event occurs
Usually only one kind of event is defined, and when it occurs, observers classify it on a
number of dimensions
Advantages: techniques for analysing cross-classified data have received a good deal of
attention and are relatively well worked out, recording such data is extremely simple and
straightforward, the very structuredness of the coding scheme should require a certain
amount of conceptual analysis and forethought, clear and simple descriptive statistics
typically result
, Disadvantage: tightly focused and highly structured and requires observers to impose that
structure on the passing stream of behaviour. Moreover, the amount of descriptive
information that can be extracted from this method is somewhat limited.
4. Time sampling
Method of repeated short samples
A number of relatively brief, non-successive time intervals are categorized and the
percentage of time intervals assigned a particular code is used to estimate the proportion of
time an individual devotes to that kind of activity
Advantage: simplicity of the recording instruments
Disadvantage: limited kinds of analysis that are appropriate for the data collected
One final point: occasionally investigators claim to have used a time sampling methodology
when in fact successive intervals were coded. In such cases, it is difficult to see what was
“sampled” and so a phrase like interval coding would seem more appropriate.
Recording devices
- An advantage is the feasibility of pencil and paper recording, except for coding events along
with their occurrence times
- More complex means can be used for coding events without time information
- Electronic devices: typically have a keyboard and an internal clock, are portable, and work as
follows; whenever a codable event occurs, the observer depresses the appropriate keys.
This is stored in the device’s memory.
- Advantage of video tapes possibility to stop, rewind and replay what’s then important
is that the videotapes are “time stamped” in some way.
- Note of caution: sophisticated recording systems appear very attractive, because they hold
out the promise that, once mastered, savings in time and labour should result. Once
mastered because almost inevitably such sophisticated equipment requires that users
pay an “entry cost”. This includes not just the initial cost of the equipment, but the time it
takes to learn how to use it and keep it working properly. Such costs may be justified only if
the coding task is quite sizable.
Assessing reliability: can and do observers agree?
- Investigator bias
o Solution: to keep observers naïve as to the hypotheses under investigation
investigators can almost never serve as observers.
- 3 reasons that investigators would want to determine how well observers agree:
o 1. We need to assure ourselves, as investigators, that observers are coding
events in accord with our definitions
Compare their coding of events with one of our own use of
videotapes
o 2. We need to provide observers with feedback to aid in their training
o 3. We need to assure others that our observers are accurate and our
procedures replicable
- Cohen’s kappa
- Correlation coefficient (less reliable)
- Two other reasons:
o Calibration between observers (calibrate observers against each other, or
better yet, against a standard protocol)
o Drift within observers (over time)
, Percentage agreement
- Probably most common used agreement statistic
- But also probably the least adequate don’t correct for the fact that given a particular
coding scheme and a particular recording strategy, some agreement would occur just by
chance alone
- Moreover, the amount of chance agreement varies depending on the number of codes in
the coding schemes and how often they occur similar values of derived from different
studies could represent different levels of agreement no comparability
Cohen’s Kappa
- Major advantages are that it correct for chance and that it can be tested for significance
- In addition, the table of agreements and disagreements used to compute kappa provides
useful feedback for observers
How Cohen’s kappa can be applied to the various recording strategies mentioned earlier:
- Time sampling and interval coding
o In both cases observers are not responsible for determining the boundaries of
the “unit” coded
o Application of Cohen’s kappa is straightforward
o If more than one mutually exclusive and exhaustive coding scheme is used to
code intervals, then a separate kappa would be computed for each scheme
- Cross-classifying events
o Two kinds of agreement need to be considered
Agreement as to whether an event occurred
Percentage of agreement used
Agreement as to how events are classified
Cohen’s kappa used first, agreement matrices are
established for each coding scheme used. Then for each
event cross-classified, agreements and disagreements
are tallied for each of the dimensions defined and
kappa’s computed
- Timing and coding events
o Patterns change between those two, so there are different recommendations
o In such cases, it’s not at all obvious how even a percentage agreement would
be determined
o Recommendation: treat the data as though it all had resulted from interval
coding then it’s easy to compute kappa.
o Advantage: reflects with some fidelity just what observers using a continuous
recording strategy are expected to do, and that is to remain continuously alert,
making decision on a moment-by-moment basis.
o Using a percentage agreement, coders would get “credit” just when both noted
an affective expression at the same time. They won’t get credit for all the times
they decided not to code an affective expression.
o With kappa, they get credit for agreeing on both the presence or the absence of
an affective expression.
- Coding events without timing
o Most problematic for determining observer agreement.
o When the material presented to coders is already ‘unitized’ there is no
problem.
