Preparation for Tutorial Meeting 1
Pilot Study on Effects on Short -Term Exercise Program
Pilot Study: used to test research methods that maybe improved in a comprehensive study
Goal: Therapeutic effects of a short-term Tai Chi exercise program for the elderly were evaluated in a pretest-
posttest quasi-experimental design. This pilot study evaluated changes in flexibility, balance, sway, pain, and mood
after a short slow-motion exercise.
Program description: The program consisted of a series of movements involving turning, shifting weight, bending,
and arm movements in combination with diaphragmatic breathing with slow movements.
*research questions:
1. Is there a difference in the degree of flexibility in the extremities of elderly persons after participating in a Tai Chi
program?
2. Is there a difference in balance and sway in elderly persons after participating in a Tai Chi program?
3. Is there a difference in the amount of pain experienced, with active and passive range of motion of specific
extremities of elderly persons, after participating in a Tai Chi program?
4. Is there a difference in the mood of elderly persons after participating in a Tai Chi program?
=> NOTE: all poorly phrased as yes/no questions better: “What is the effect of the Tai Chi program on…?”
Method:
-> design: pretest-posttest quasi-experiment
-> measured effects: improved balance, sway, range of motion, decreased perceived pain, and lessened trait anxiety.
-> Participants: 11 elderly females => convenience sample bias?
-> Instruments: standard goniometry, the Multiple Affect Adjective Check List, stopwatch measures of single-leg
stance and a tandem walk(sway), and visual analogue measurement of pain
-> procedure:
1) participants were screened to determine their physical acceptability for the study
2) exclusion criteria (e.g. attending at least 90% of all classes) were applied to exclude subjects
3) informed consent verbally and in written form was gained
4) pre-test measures on the included variables (see above)
5) classes took place for 8 weeks
6) identical post-test measures (as to the pre-test measures) were conducted
7) basic demographics, such as gender, age and race, were recorded -> possible confounders?!
-> data-analysis: pre-test-post-test comparisons were made on all measures on both group and individual level
=> t-tests were conducted limitation: small sample size did not allow for further analysis & results are preliminary
Results:
1) Findings included significant improvement (p= .05) in trait anxiety and pain perception.
-> NOTE: “The difference was significant, t(10) = 2.19, p= .053.” questionable if p < 0.05 was set
2) non-significant improvements in flexibility, balance and sway on group level
-> HOWEVER: some individuals had significant improvements on an individual level (regarding all 3 aspects)
3) Demographics played no significant role in moderating/mediating the effect.
Conclusion: Improvements in mood, flexibility, and balance may have a profound effect on the incidence of falls,
injuries, resulting disability, and overall quality of life. However, future studies can improve certain aspects to
increase the validity and reliability of the study.
,Pilot Study: used to test research methods that maybe improved in a comprehensive study
*additional insights from the pilot study, which should be improved in a comprehensive study (IMPORTANT):
-> elderly participants needed frequent reminders & encouragement to attend the beginning classes, until the group
developed some cohesiveness and skills
-> several participants demonstrated marked achievement in balance & sway during the last classes but did not
demonstrate this same level of achievement during post-testing
-> testing elderly participants requires time for slow movement between testing areas, privacy for each test, and
comfortable seating
-> testing appointments prior to 10 am were not selected by any of the participants, and food snacks after testing
were well received
=> OVERALL: Tai Chi program = safe, viable method of exercise for the elderly leading to some health benefits
yet, recommendations for future studies are provided: e.g. larger sample size & post-test right after last class
, Preparation for Tutorial 2
Goodwin Chapter 5 – Introduction to Experimental Research
1) Essential Features of Experimental Research & Other Basics
-experimental study: experimenter manipulates 1/more variables (independent variables/predictors) and holds all
other variables constant (control of confounders) so that he can observe changes in the dependent variable
(criterion)
-early concepts of inductive logic by Mill -> psychological research uses 2 methods:
1) method of agreement -> if X then Y => accomplished by an experimental group
2) method of disagreement -> if not X then not Y => accomplished by a control group
IMPORTANTLY: both criteria are never fully met in modern research joint method should be research standard
1. FEATURE OF EXPERIMENTAL METHOD: ESTABLISHING INDEPENDENT VARIABLES
-manipulated independent variables vs subject variables (not directly manipulated -> e.g. age)
-> both are studied to see whether changes in them can elicit changes in the response variable
=> always include 2/more levels/conditions
-different kinds of independent manipulated variables:
1) task variables -> task that participants have to perform varies in complexity, duration, etc.
2) situational variables -> features in the environment that participants might encounter
=> example: a person may be surrounded by no, 3, or 7 bystanders in a helping situation
3) instructional variables -> instructions regarding how participants have to perform a particular task vary
-laboratory studies vs field experiments (outside the laboratory)
-control groups are only included when a baseline-level is compared against a treatment group
-> they are not needed, for example, when genders are compared (male vs female group)
2. FEATURE OF EXPERIMENTAL METHOD: CONTROLLING EXTRANEOUS VARIABLES
-extraneous variables = variables, which are not supposed to misleadingly affect the dependent variable
-> confounder = uncontrolled variable, which could affect the results => make it impossible to tell whether a result is
due to the experimental condition or the confounder or a combination of both
3. FEATURE OF EXPERIMENTAL METHOD: MEASURING DEPENDENT VARIABLES
-poorly chosen dependent variables can cause problems:
1) ceiling effect -> very high average scores for both groups makes spotting a difference tough -> measure to easy
2) floor effect -> very low average scores for both groups makes spotting a difference tough -> measure to hard
=> pilot studies can help avoiding these problems
MANIPULATED VS SUBJECT VARIABLES
-subject variables = not directly manipulated -> e.g. sex, age, race… => participant characteristics
NOTE: trait characteristics can be manipulated in advance (independent variable) or just taken as they are
(subject variable) example: create anxiety artificially in 1 group while not doing so in another VS anxiety trait
levels separating subjects into 2 groups
NOTE: subject variables are still referred to as independent variables by most researchers
-> experiment is then often called quasi-experiment OR natural groups study
DRAWING CONCLUSIONS WHEN USING SUBJECT VARIABLES
-causal conclusion can only be drawn with manipulated variables – fulfil 3 criteria of causality (covariation, temporal
order of X and Y, absence of confounders (mostly)) -> HOWEVER: when using a subject variable, the 2 groups could
differ in other characteristics (e.g. age, confidence) => THEREFORE: absence of confounders is less controllable
,because there is no random allocation of participants into 2 groups
CONSEQUENCE: we can only say that 2 groups differed on the measure of the dependent variable
2) Validity of Experimental Research
STATISTICAL CONCLUSION VALIDITY
-statistical conclusion validity = concerns the extent to which the researcher uses statistics properly & draws the
appropriate conclusions from the statistical analysis => can be violated in multiple ways:
1) researchers might do the wrong analysis or violate some of the assumptions required for performing a particular
analysis -> example: computing alpha coefficient without the test being parallel
2) researchers might selectively report some analyses that came out as predicted but might not report others
3) low reliability of dependent measures results in a failure to find true effects (type 2 errors)
CONSTRUCT VALIDITY
-construct validity = refers to the adequacy of the operational definitions for both the independent & the dependent
variables used in the study
EXTERNAL VALIDITY
-external validity = degree to which research findings generalize beyond the specific context of the experiment being
conducted -> results can be generalized to…
1) other populations -> a lot of research has been conducted on college students (also goes for males as opposed to
females) + care should be taken when generalizing across cultures (especially with collectivistic & individualistic)
2) other environments -> from lab to the field?! => some researchers call for more ecological validity = research with
relevance for the everyday activities of people
3) other times -> can be difficult due to societal changes over time
NOTE: external validity is nice to have but an experiment is not invalidated by not being generalizable to other
populations, environments & times => it still can give important insights into human behaviour
INTERNAL VALIDITY
-internal validity = degree to which an experiment is methodologically sound and confound-free
-> degree to which researcher feels confident that the results, as measured by the dependent variable, are directly
associated with the independent variable and are not the result of some other, uncontrolled factor (confounder)
3) Threats to Internal Validity
-any extraneous factor, which can’t be properly controlled is a threat BUT some problems require special attention
-> these problems become particularly important when no control groups are included
PROBLEMS ARISING FROM STUDIES EXTENDING OVER TIME
-typical research design:
-> problem of multiple confounders arises
whenever the control group is absent:
1) pre-post difference could be caused by history -> societal/environmental changes in between
2) they could be caused by personal maturation -> fluctuations in trait level within individuals
3) they could be caused by regression to the mean -> fact that if score #1 from a subject is an extreme score, then
score #2 from the same person will be closer to whatever the mean for the larger set of scores is
4) testing effects (e.g. practice effects & fatigue effects) & instrumentation effects (measure changes from 1 testing
occasion to the other)
,PARTICIPANT PROBLEMS
-subjects selection effects may cause inequivalent groups -> acts as a potential confounder
=> they can also interact with other threats to internal validity (e.g. history) history may affect one group but not
the other group history x selection confounder
-attrition (subject mortality) -> not all participants that started the study may end it for different reasons
=> becomes problematic if particular types of people (e.g. males) are more likely to drop out than others (e.g.
females) results in another form of the subject selection problem
Goodwin Chapter 6 – Methodological Control in Experimental Research
-between-subjects design -> problem: equivalence of groups is hard to establish
-> within-subjects design -> problem: sequence effects (e.g. fatigue & practice effects)
1) Between-Subjects Design
-between-subject designs are sometimes required (e.g. with subject variables) -> you can’t change someone’s age
=> another case: experience from 1 experimental condition makes participation in the other condition impossible
uninformed participants are needed in every condition of some studies
-advantage of between-subjects design: every subject is uninformed & has no prior experience with any other
related conditions (no sequence effects)
-disadvantage of between-subjects design: large sample is required + inequivalent groups may confound results
OPTIONS OF CREATING EQUIVALENT GROUPS
1. simple random assignment -> method for randomly placing participants, once selected for a study, into the
different study conditions => implication: every person volunteering for the study has an equal chance of being
placed in any of the groups being formed
*goal of random assignment: eliminating individual difference factors (confounders) that could influence the study
by spreading them evenly throughout the different groups => IMPORTANT: random assignment is not always
successful at creating equivalent groups, BUT chance of doing so increases with larger sample sizes
**guaranteeing equal numbers of participants in every condition -> using a block random assignment
=> each block contains all of the conditions of the study in randomized orders ensures that each condition will be
occupied by a participant, before any conditions are repeated
2. matched pairs -> participants are grouped together on some subject variable (matching variable -> e.g. trait
anxiety) & then distributed randomly to the different groups in the experiment => helpful when sample is small and
random assignment is not particularly likely to produce equivalent groups
*2 requirements for justified conduction of matching:
1) you must have good reason to believe the matching variable will have a predictable effect on the study’s outcome
2) there must be a reasonable way of measuring or identifying participants on the matching variable
**problem of matching: how many matching variables do you want to include?
-> as this is very difficult to determine & multiple variables could act as confounders, researchers usually recruit large
samples & hope for random assignment to create equivalent groups
2) Within-Subjects Design
-also referred to as repeated-measures design
-practical advantage: less participants are required
-> THEREFORE: within-subjects design particularly of interest when population is small (e.g. studying astronauts)
-another advantage: groups are always equivalent as they are the same
, -problem: once a participant has completed the first condition of a study, the experience or altered circumstances
could influence performance in later conditions of the study (order/sequence effects) -> 2 types:
1) progressive effects -> it is assumed that performance changes steadily (progressively) from trial to trial
=> examples: practice & fatigue effects
2) carryover effects -> some sequences might produce effects different from those of other sequences
=> example: easy condition first may motivate the participant to do well in the tough condition afterwards, BUT
tough condition first may discourage the participant even to try in the easy condition afterwards
motivational influence depending on the order of conditions varies
whenever carryover effects are suspected to manipulate the results, researchers choose between-subjects design
OPTIONS OF CONTROLLING ORDER EFFECTS
-common strategy: counterbalancing -> works better for progressive effects than for carryover effects
=> difference options arise depending on whether participants are tested once or multiple times per condition
TESTING ONCE PER CONDITION
1. complete counterbalancing -> every possible sequence will be used at least once (X! = number of possible
sequences, X = number of conditions) => participants are randomly assigned to 1 particular sequence until all
sequences are occupied once then: maybe second round takes place (and so on…)
problem: a lot of participants are required when conditions exceed 4 or 5 (6! = 720)
2. partial counterbalancing -> using a subset of the total number of possible sequences
=> accomplished by sampling the required number of sequences equal to the number of participants
commonly done when few participants are available OR many conditions exist
*advanced method compared to partial counterbalancing
-> Latin Square approach => assures that:
1) every study condition occurs equally often in every sequential position
2) every condition precedes & follows every other condition exactly once
=> each condition occurs once in each row and once in each column
NOTE: number of conditions = number of rows (6 CONDITIONS ON THE LEFT)
requirement: number of participants must be a multiple of the conditions/rows, so that they can evenly be
randomly assigned to 1 sequence (row) of the Latin Square (18 participants would be ok in this case)
TESTING MULTIPLE TIMES PER CONDITION
1. reverse counterbalancing -> experimenter presents the conditions in one order, and then presents them again in
the reverse order -> exemplary for 4 conditions: A–B–C–D, then D–C–B–A => you can repeat the sequence if subjects
are meant to complete each task more than twice
half of the participants are assigned randomly to ABCDDCBA & the other half is assigned to DCBAABCD
2. block randomization -> randomizing the order of all conditions making up a block => exemplary: ACDB CDBA
=> eliminates the possibility of the subjects predicting what’s coming next
3) Methodological Control in Developmental Research
-developmental research uses longitudinal & cross-sectional designs to investigate age as independent variable
-longitudinal studies -> within-subjects approach => e.g. comparing language performance of the same children
when they are 3y, 4y & 5y old problems: 1) attrition rates & 2) time-consuming
ADDITIONALLY: they require constant updates on informed consent, as opinions could change over time
-cross-sectional studies -> between-subjects approach => e.g. comparing language performance of 3y, 4y & 5y olds
advantage over longitudinal studies: less time-consuming
problem: potential cohort effects -> cohorts (people born at the same time) may differ from each other
=> political attitude from people born in the 40s differs from the one born in the 60s
