lOMoARcPSD|945481
Chapter 1
You begin with an observation that you want to
understand, and this observation could be
anecdotal or could be
based on some data. From your initial
observation you generate explanations, or
theories, of those observations, from which you
can make predictions (hypotheses). Here’s
where the data come into the process because
to test your predictions you need data. First you
collect some relevant data (and to do that you need to identify things that can be
measured) and then you analyse those data. The analysis of the data may support your
theory or give you cause to modify the theory. As such, the processes of data collection
and analysis and generating theories are intrinsically linked: theories lead to data
collection/analysis and data collection/analysis informs theories! This chapter explains
this research process in more detail.
Data collection : what to measure
To test hypotheses we need to measure variables. Variables are just things that can
change (or vary); they might vary between people (e.g. IQ, behaviour) or locations (e.g.
unemployment) or even time (e.g. mood, profit, number of cancerous cells). Most
hypotheses can be expressed in terms of two variables: a proposed cause and a proposed
outcome. A variable that we think is a cause is known as an independent variable
(because its value does not depend on any other variables). A variable that we think is an
effect is called a dependent variable because the value of this variable depends on the
cause (independent variable).
As we have seen in the examples so far, variables can take on many different forms and
levels of sophistication. The relationship between what is being measured and the
numbers that represent what is being measured is known as the level of measurement.
Variables can be split into categorical and continuous, and within these types there are
different levels of measurement:
Categorical (entities are divided into distinct categories):
Binary variable: There are only two categories (e.g. dead or alive).
Nominal variable: There are more than two categories (e.g. whether someone is an
omnivore, vegetarian, vegan, or fruitarian).
Ordinal variable: The same as a nominal variable but the categories have a logical
order (e.g. whether people got a fail, a pass, a merit or a distinction in their exam).
Continuous (entities get a distinct score):
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
Interval variable: Equal intervals on the variable represent equal differences in the
property being measured (e.g. the difference between 6 and 8 is equivalent to the
difference between 13 and 15).
Ratio variable: The same as an interval variable, but the ratios of scores on the scale
must also make sense (e.g. a score of 16 on an anxiety scale means that the person is, in
reality, twice as anxious as someone scoring 8).
We have seen that to test hypotheses we need to measure variables. Obviously, it’s also
important that we measure these variables accurately. Ideally we want our measure to be
calibrated such that values have the same meaning over time and across situations.
Weight is one example: we would expect to weigh the same amount regardless of who
weight us, or where we take the measurement but in other cases we are forced to use
indirect measures such as self-report, questionnaires and computerized tasks . There will
often be a discrepancy between the numbers we use to represent the thing we’re
measuring and the actual value of the thing we’re measuring (i.e. the value we would get
if we could measure it directly). This discrepancy is known as measurement error.
One way to try to ensure that measurement error is kept to a minimum is to determine
properties of the measure that give us confidence that it is doing its job properly. The first
property is validity, which is whether an instrument actually measures what it sets out to
measure. The second is reliability, which is whether an instrument can be interpreted
consistently across different situations. Criterion validity is whether the instrument is
measuring what it claims to measure (does your lecturers’ helpfulness rating scale
actually measure lecturers’ helpfulness?). In an ideal world, you could assess this by
relating scores on your measure to real-world observations. With self-report
measures/questionnaires we can also assess the degree to which individual items
represent the construct being measured, and cover the full range of the construct
(content validity). Validity is a necessary but not sufficient condition of a measure. A
second consideration is reliability, which is the ability of the measure to produce the
same results under the same conditions. To be valid the instrument must first be reliable.
The easiest way to assess reliability is to test the same group of people twice: a reliable
instrument will produce similar scores at both points in time (test–retest reliability).
Sometimes, however, you will want to measure something that does vary over time (e.g.
moods, blood-sugar levels, productivity). Statistical methods can also be used to
determine reliability.
Data collection : How to measure
If we simplify things quite a lot then there are two ways to test a hypothesis: either by
observing what naturally happens, or by manipulating some aspect of the environment
and observing the effect it has on the variable that interests us. The main distinction
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
between what we could call correlational or cross-sectional research (where we
observe what naturally goes on in the world without directly interfering with it) and
experimental research (where we manipulate one variable to see its effect on another)
is that experimentation involves the direct manipulation of variables Most scientific
questions imply a causal link between variables; we have seen already that dependent
and independent variables are named such that a causal connection is implied (the
dependent variable depends on the independent variable)
|
When we collect data in an experiment, we can choose between two methods of data
collection. The first is to manipulate the independent variable using different participants.
This method is the one described above, in which different groups of people take part in
each experimental condition (a between-groups, between-subjects or independent
design). The second method is to manipulate the independent variable using the same
participants. Simplistically, this method means that we give a group of students positive
reinforcement for a few weeks and test their statistical abilities and then begin to give
this same group negative reinforcement for a few weeks before testing them again, and
then finally giving them no reinforcement and testing them for a third time (a within-
subject or repeatedmeasures design). As you will discover, the way in which the data
are collected determines the type of test that is used to analyse the data.
So, chimps who score highly in condition 1 will also score highly in condition 2, and those
who have low scores for condition 1 will have low scores in condition 2. However,
performance won’t be identical, there will be small differences in performance created by
unknown factors. This variation in performance is known as unsystematic variation. If
we introduce an experimental manipulation (i.e. provide bananas as feedback in one of
the training sessions), then we do something different to participants in condition 1 to
whatwe do to them in condition 2. So, the only difference between conditions 1 and 2 is
the manipulation that the experimenter has made (in this case that the chimps get
bananas as a positive reward in one condition but not in the other). Therefore, any
differences between the means of the two conditions is probably due to the experimental
manipulation. So, if the chimps perform better in one training phase than the other then
this has to be due to the fact that bananas were used to provide feedback in one training
phase but not the other. Differences in performance created by a specific experimental
manipulation are known as systematic variation. Imagine again that we didn’t have an
experimental manipulation. If we did nothing to the groups, then we would still find some
variation in behaviour between the groups because they contain different chimps who will
vary in their ability, motivation, IQ and other factors. In short, the type of factors that
were held constant in the repeated-measures design are free to vary in the independent-
measures design. So, the unsystematic variation will be bigger than for a repeated-
measures design. As before, if we introduce a manipulation (i.e. bananas) then we will
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
see additional variation created by this manipulation. As such, in both the repeated-
measures design and the independent-measures design there are always two sources of
variation:
MM Systematic variation: This variation is due to the experimenter doing something to
all of the participants in one condition but not in the other condition.
MM Unsystematic variation: This variation results from random factors that exist
between the experimental conditions (such as natural differences in ability, the time of
day, etc.)
The role of statistics is to discover how much variation there is in performance, and then
to work out how much of this is systematic and how much is unsystematic. In a repeated-
measures design, differences between two conditions can be caused by only two things:
(1) the manipulation that was carried out on the participants, or (2) any other factor that
might affect the way in which a person performs from one time to the next. The latter
factor is likely to be fairly minor compared to the influence of the experimental
manipulation. In an independent design, differences between the two conditions can also
be caused by one of two things: (1) the manipulation that was carried out on the
participants, or (2) differences between the characteristics of the people allocated to
each of the groups. The latter factor in this instance is likely to create considerable
random variation both within
each condition and between them. Therefore, the effect of our experimental manipulation
is likely to be more apparent in a repeated-measures design than in a between-group
design because in the former unsystematic variation can be caused only by differences in
the way in which someone behaves at different times.
In both repeated-measures and independent-measures designs it is important to try to
keep the unsystematic variation to a minimum. By keeping the unsystematic variation as
small as possible we get a more sensitive measure of the experimental manipulation.
Generally, scientists use the randomization of participants to treatment conditions to
achieve this goal. Many statistical tests work by identifying the systematic and
unsystematic sources of variation and then comparing them. This comparison allows us
to see whether the experiment has generated considerably more variation than we would
have got had we just tested participants without the experimental manipulation.
Randomization is important because it eliminates most other sources of systematic
variation, which allows us to be sure that any systematic variation between experimental
conditions is due to the manipulation of the independent variable. We can use
randomization in two different ways depending on whether we have an independent- or
repeated-measures design.
Let’s look at a repeated-measures design first. When the same people participate in more
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
than one experimental condition they are naive during the first experimental condition
but they come to the second experimental condition with prior experience of what is
expected of them. At the very least they will be familiar with the dependent measure
(e.g. the task they’re performing). The two most important sources of systematic
variation in this type of design are: MM Practice effects: Participants may perform
differently in the second condition because of familiarity with the experimental situation
and/or the measures being used. MM Boredom effects: Participants may perform
differently in the second condition because they are tired or bored from having
completed the first condition. Although these effects are impossible to eliminate
completely, we can ensure that they produce no systematic variation between our
conditions by counterbalancing the order in which a person participates in a condition.
We can use randomization to determine in which order the conditions are completed. The
best way to reduce this eventuality is to randomly allocate participants to conditions.
Analaysing data
Once you’ve collected some data a very useful thing to do is to plot a graph of how many
times each score occurs. This is known as a frequency distribution, or histogram,
which is a graph plotting values of observations on the horizontal axis, with a bar showing
how many times each value occurred in the data set. Frequency distributions can be very
useful for assessing properties of the distribution of scores. In an ideal world our data
would be distributed symmetrically around the centre of all scores. As such, if we drew a
vertical line through the centre of the distribution then it should look the same on both
sides. This is known as a normal distribution and is characterized by the bell-shaped
curve with which you might already be familiar. This shape basically implies that the
majority of scores lie around the centre of the distribution (so the largest bars on the
histogram are all around the central value. There are two main ways in which a
distribution can deviate from normal:
(1) lack of symmetry (called skew) and (2) pointyness (called kurtosis). Skewed
distributions are not symmetrical and instead the most frequent scores (the tall bars on
the graph) are clustered at one end of the scale. So, the typical pattern is a cluster of
frequent scores at one end of the scale and the frequency of scores tailing off
towards the other end of the scale. A skewed distribution can be either positively skewed
(the frequent scores are clustered at the lower end and the tail points towards the higher
or more positive scores) or negatively skewed (the frequent scores are clustered at the
higher end and the tail points towards the lower or more negative scores). Distributions
also vary in their kurtosis. Kurtosis, despite sounding like some kind of exotic disease,
refers to the degree to which scores cluster at the ends of the distribution (known as the
tails) and how pointy a distribution is. A distribution with positive kurtosis has many
scores in the tails (a so-called heavy-tailed distribution) and is pointy. This is known as a
leptokurtic distribution. In contrast, a distribution with negative kurtosis is relatively thin
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
in the tails (has light tails) and tends to be flatter than normal. This distribution is called
platykurtic. Ideally, we want our data to be normally distributed (i.e. not too skewed,
and not too many or too few scores at the extremes!).
We can also calculate where the centre of a frequency distribution lies (known as the
central tendency). There are three measures commonly used: the mean, the mode and
the median. The mode is simply the score that occurs most frequently in the data set.
This is easy to spot in a frequency distribution because it will be the tallest bar! To
calculate the mode, simply place the data in ascending order (to make life easier), count
how many times each score occurs, and the score that occurs the most is the mode! One
problem with the mode is that it can often take on several values. Another way to
quantify the centre of a distribution is to look for the middle score
when scores are ranked in order of magnitude. This is called the
median. when we have an even number of scores there won’t be a
middle value. The mean is the measure of central tendency that you are most likely to
have heard of because it is simply the average score and the media are full of average
score. To calculate the mean we simply add up all of the scores and then divide by the
total number of scores we have.. It can also be interesting to try to quantify the spread,
or dispersion, of scores in the data. The easiest way to look at dispersion is to take the
largest score and subtract from it the smallest score. This is known as the range of
scores. One convention is to cut off the top and bottom 25% of scores and calculate the
range of the middle 50% of scores – known as the interquartile range. One rule of
thumb is that the median is not included in the two halves when they are split (this is
convenient if you have an odd number of values), but you can include it (although which
half you put it in is another question). Therefore, if we have any data that are shaped like
a normal distribution, then if the mean and standard deviation are 0 and 1 respectively
we can use the tables of probabilities for the normal distribution to see how likely it is
that a particular score will occur in the data. The obvious problem is that not all of the
data we collect will have a mean of 0 and standard deviation of 1. Luckily any data set
can be converted into a data set that has a mean of 0 and a standard deviation of 1. First,
to centre the data around zero, we take each score and subtract from it the mean of all.
Then, we divide the resulting score by the standard deviation to ensure the data have a
standard deviation of 1. The resulting scores are known as z-scores and in equation
form, the conversion that I’ve just described is.
The normal distribution and z-scores allow us to go a first step beyond our data in that
from a set of scores we can calculate the probability that a particular score will occur. So,
we can see whether scores of a certain
size are likely or unlikely to occur in a distribution of a particular kind. The first important
value of z is 1.96 because this cuts off the top 2.5% of the distribution, and its
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
counterpart at the opposite end (–1.96) cuts off the bottom 2.5% of the distribution.
Having looked at your data the next step is to fit a statistical model to the data. Scientific
statements, as we have seen, can be split into testable hypotheses. The hypothesis or
prediction that comes from your theory is usually saying that an effect will be present.
This hypothesis is called the alternative hypothesis and is denoted by H1. (It is sometimes
also called the experimental hypothesis but because this term relates to a specific type of
methodology it’s probably best to use ‘alternative hypothesis’.) There is another type of
hypothesis, though, and this is called the null hypothesis and is denoted by H0. The
reason that we need the null hypothesis is because we cannot prove the experimental
hypothesis using statistics, but we can reject the null hypothesis. If our data give us
confidence to reject the null hypothesis then this provides support for our experimental
hypothesis. However, be aware that even if we can reject the null hypothesis, this doesn’t
prove the experimental hypothesis – it merely supports it. Finally, hypotheses can also
be directional or non-directional. A directional hypothesis states that an effect will occur,
but it also states the direction of the effect
Chapter 2
Whatever the phenomenon we desire to explain, we collect data from the real world to
test our hypotheses about the phenomenon. Testing these hypotheses involves building
statistical models of the phenomenon of interest. As researchers, we are interested in
finding results that apply to an entire population of people or things. Scientists rarely, if
ever, have access to every member of a population. Therefore, we collect data from a
small subset of the population (known as a sample) and use these data to infer things
about the population as a whole. The bigger the sample, the more likely it is to reflect the
whole population. If we take several random samples from the population, each of these
samples will give us slightly different results. However, on average,
large samples should be fairly similar. One of the simplest models used in statistics is the
mean, which we encountered already. the mean is a model created to summarize our
data. With any statistical model we have to assess the fit. The easiest way to do this is to
look at the difference between the data we observed and the model fitted. The diagram
also has a series of vertical lines that connect each observed value to the mean value.
These lines represent the deviance between the observed data and our model and can
be thought of as the error in the model. We can calculate the magnitude of these
deviances by simply subtracting the mean value (x) from each of the observed values
(xi). Now, how can we use these deviances to estimate the accuracy of the model? One
possibility is to add up the deviances (this would give us an estimate of the total error).
The sum of squared errors (SS) is a good measure of the accuracy of our model.
However, it is fairly obvious that the sum of squared errors is dependent upon the
amount of data that has been collected – the more data points, the higher the SS. To
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
overcome this problem vwe calculate the average error by dividing the SS by the number
of observations (N) If
we are interested only in the average error for the sample, then we can divide by N
alone.. However, we are generally interested in using the error in the sample to estimate
the error in the population and so we divide the SS by the
number of observations minus 1 (the reason why is
explained in Jane Superbrain Box 2.2 below). This measure is
known as the variance and is a measure that we will come
across a great deal.
Degrees of freedom (df) is a very difficult concept to explain. The degrees of freedom is
one less than the number of players. In statistical terms the degrees of freedom relate to
the number of observations that are free to vary. if we hold one parameter constant then
the degrees of freedom must be one less than the sample size. This fact explains why
when we use a sample to estimate the standard deviation of a population, we have to
divide the sums of squares by N −1 rather than N alone. The variance is, therefore, the
average error between the mean and the observations made (and so is a measure of how
well the model fits the actual data). There is one problem with the variance as a measure:
it gives us a measure in units squared (because we squared each
error in the calculation). For this reason, we often take the square root of the variance
(which ensures that the measure of average error is in the same units as the original
measure). This measure is known as the standard deviation and is simply the square root
of the variance. The sum of squares, variance and standard deviation are all, therefore,
measures of the ‘fit’. Small standard deviations (relative to the value of the mean itself)
indicate that data points are close to the mean.
We’ve seen that the standard deviation tells us something about how well the mean
represents the sample data, but I mentioned earlier on that usually we collect data from
samples because we don’t have access to the entire population. If you take several
samples from a population, then these samples will differ slightly; therefore, it’s also
important to know how well a particular sample represents the population. This is where
we use the standard error. Many students get confused about the difference between
the standard deviation and the standard error (usually because the difference is never
explained clearly). sampling variation: that is, samples will vary because they contain
different members of the population; a sample that by chance includes some very good
lecturers will have a higher average than a sample that, by chance, includes some awful
lecturers. The end result is a nice symmetrical distribution known as a sampling
distribution. A sampling distribution is simply the frequency distribution of sample
means from the same population. The sampling distribution tells us about the behaviour
of samples from the population, and you’ll notice that it is centred at the same value as
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
, lOMoARcPSD|945481
the mean of the population. This means that if we took the average of all sample means
we’d get the value of the population mean. If you were to calculate the standard
deviation between sample means then this too would give you a measure of how much
variability there was between the means of different samples. The standard deviation of
sample means is known as the standard error of the mean (SE). Therefore, the
standard error could be calculated by taking the difference between each sample mean
and the overall mean, squaring these differences, adding them up, and then dividing by
the number of samples. Finally, the square root of this value would need to be taken to
get the standard deviation of sample means, the standard error. Of course, in reality we
cannot collect hundreds of samples and so we rely on approximations of the standard
error. Luckily for us some exceptionally clever statisticians have demonstrated that as
samples get large (usually defined as greater than 30), the sampling distribution has a
normal distribution with a mean equal to the population mean.
The basic idea behind confidence intervals is to construct a range of values within which
we think the population value falls. Up until now I’ve avoided the issue of how we might
calculate the intervals. The crucial thing with confidence intervals is to construct them in
such a way that they tell us something useful. Therefore, we calculate them so that they
have certain properties: in particular they tell us the likelihood that they contain the true
value of the thing we’re trying to estimate (in this case, the mean). Typically we look at
95% confidence intervals, and sometimes 99% confidence intervals, but they all have a
similar interpretation they are limits constructed such a certain percentage of the time
(be that 95% or 99%) the true value of the population mean will fall within these limits. To
calculate the confidence interval, we need to know the limits within which 95% of means
will fall. How do we calculate these limits? Remember back in section 1.7.4 that I said that
1.96 was an important value of z (a score from a normal distribution with a mean of 0 and
standard deviation of 1) because 95% of z-scores fall between –1.96 and 1.96. This
means that if our sample means were normally distributed with a mean of 0 and a
standard error of 1, then the limits of our confidence interval would be –1.96 and +1.96.
Luckily we know from the central limit theorem that in large samples (above about 30)
the
sampling distribution will be normally distributed.
In Chapter 1 we saw that research was a five-stage process:
1 Generate a research question through an initial observation (hopefully backed up by
some data).
2 Generate a theory to explain your initial observation.
3 Generate hypotheses: break your theory down into a set of testable predictions.
4 Collect data to test the theory: decide on what variables you need to measure to test
your predictions and how best to measure or manipulate those variables.
Gedownload door Sjaak Banaan (rdpnnbrg@live.nl)
