Summary The role of statistics in behavioral research
Sander Los
Chapter 1 why statistics?
1.2 the problem
Tess and Manuel want to know if there is a difference in mean response time between Dutch and
Belgians. There are three scenarios, each with a different hypothesis and outcome.
Scenario 1
First, they have to take a random sample from the Dutch and Belgian population. Every Dutch has an
equal chance of being in the Dutch sample and every Belgian has an equal chance of being in the
Belgian sample. To measure the response time, they let each participant press a button as soon as a
white square appears on a black computer screen. How large should the sample be? Tess suggests
two Dutch and two Belgians, so they finish quickly. Manuel says that wouldn’t be big enough to
generalize to the population. Imagine two Dutch have a mean of 280ms and two Belgians have a
mean of 300ms. When you add one more in both groups, it can change the means drastically. For
example, if the third Dutch has a score of 340 and the Belgian 240, the mean for the Dutch would be
300 and for the Belgians 280. It would be the exact opposite as before.
Scenario 2
Imagine drawing 10 participants instead, from both populations, and find the same means (300 for
Dutch, 280 for Belgians). In this case, when adding an extra participant, the mean wont change as
much and the Dutch will still be faster. However, when there is a very fast Belgian (200ms) and a very
slow Dutch (500ms), the mean will again be the opposite. But, this does mean you need two
extremes to get a reversed outcome.
Scenario 3a
Imagine taking a 100 people sample, per group, and there is, again, a difference from 20ms. Even
with 6 very slow additional Dutch and 6 very fast Belgians, you’d not get a reverse outcome.
Scenario 3b
Alternatively, suppose we have a much larger mean difference between the groups. For example, the
dutch measure 280ms and the Belgians 480ms: an effect of 200ms. Suppose all individual
observations lie between 200 and 800 ms. This seems to be a reasonably widespread, because the
fastest participant responds no less than four times as fast as the slowest participant. In that case, to
cancel out the 200 ms mean difference, the scores of the additional Dutchman and Belgian need to
be very extreme.
Scenario 3c
Another thought: what if there is an effect of 20ms, but now the population is limited to Dutch and
Belgian top athletes. Then, a small variation in the individual scores is very plausible. This small
spread of scores in the sample strongly suggests that the spread within the population is also small.
The conclusion of all scenarios is that new observations can always undo an observed difference
between means, but some observations are very unlikely compared to observations made earlier. If
there are extreme scores, you’d rather doubt their reliability instead of the reliability of the mean
effect without those scores. However, “we can never be 100% certain” if we don’t test the entire
population.
1.3 some conclusions
,You can never have complete certainty about population values based on sample data. It may be that
when your results show a difference between two mean values, there is no difference in the
population, or even a difference in opposite directions. Conversely, if your results do not show a
difference between two values, there may still be a difference in population. However, if we adopt
this argument, conducting research would be pointless. Scientist are usually interested in statements
that apply to the entire population.
Fortunately, if we would actually find the results outlined in scenario 3a-3c, this would strongly
suggest that there is a difference in mean response time between Dutch and Belgians in the
population. Let’s start with the hypothesis that there is no difference between certain populations:
the null hypothesis. Suppose the null hypothesis is true, how likely are the findings of my sample?
Finding a difference between the sample would be unlikely, especially if samples are big. So, if we
ding a big difference between the samples, it is reasonable to conclude that the null hypothesis is
incorrect. We then reject the null hypothesis and accept the alternative hypothesis, which states
that there is a difference between the populations. Adopting a alternative hypothesis involves the
risk that we make a mistake: our findings may be unlikely under the null hypothesis but not
impossible. How do we decide whether our findings are unlikely under the null hypothesis or not?
We would like to calculate a probability. Specifically, we would like to calculate the probability of our
findings given that the null hypothesis is true. If the probability is small, then it makes sense to reject
the null hypothesis and adopt the alternative hypothesis. More about this are elaborated in later
chapters.
1.4 the probability of data under the null hypothesis
In the hypothetical study of Tess and Manuel, three factors turned out to be crucial: sample size, size
of the effect, and size of the spread of individual data. Sample size: the larger the sample size, the
smaller the probability of a given effect under the null hypothesis. For a given mean effect in our
sample, we feel more justified to reject the null hypothesis and accept the alternative hypothesis if
the sample size is larger. Effect size: the greater the mean effect, the less likely it is under the null
hypothesis. The justification of this belief is that, all else being equal, a big mean effect is less likely
under the null hypothesis than a smaller mean. In the case of a big mean effect, it makes sense to
reject the null hypothesis and accept the alternative hypothesis. Spread of the observations: the
smaller the spread, the less likely it is to find a given mean difference under the null hypothesis. One
way to see this is to think of spread as a measure of noise. In the case of a large spread, the data are
noisy, reducing our confidence in the mean of these data. As the spread of the individual data is
smaller, a given mean effect is less likely under the null hypothesis, so we are more inclined to reject
the null hypothesis and adopt the alternative hypothesis.
In practice, we should consider the three factors simultaneously. Looking back at the scenarios
above, Tess seemed to assume that the ratio between the effect size and spread of observations is
important: a large effect with a large spread is about as reliable as a small effect with a small spread.
This idea is correct.
1.5 more about populations
In statistics, the term population does not necessarily refer to a group of individuals, but to a total set
of ‘units’ that one wants to make statements about. So in an experiment on whether a certain
therapy might work on neurotic patients, there is a population of neurotic patients undergoing
therapy and a population of neurotic patients not undergoing therapy.
How and under what circumstances can an investigator try to control the effect size and spread of his
sample data? As regards the effect sizer, a researcher has a high level of control in the case of
experimental research. The researcher manipulates the independent variable, which means that she
, controls the levels of the independent variable. By varying the independent variable at more extreme
levels, the researcher may reasonably expect to observe a stronger effect on the dependent variable.
For example, the researcher will find a larger effect when he measures after 20 therapy sessions than
after 5 sessions (if the therapy is effective). In addition, a researcher has some degree of control over
the spread that he will observe in his research. For example, a researcher can choose to only draw
individuals from the population of Belgian and Dutch top athletes instead of the entire population.
This results in less spread in the individual data then when the sample consist of a wide collection of
elderly, toddlers, top athletes, young students and visually impaired people. However, limitation of
the sample also has direct consequences for the generalization of the results: a possible difference in
response time between Belgian and Dutch top athletes cannot be generalized to the entire
population of Belgian and Dutch.
