SUMMARY: INTRODUCTION TO THE NEW STATISTICS
CHAPTER 1: ASKING AND ANSWERING RESEARCH QUESTIONS
Inferential statistics = the art of using information from a sample to estimate what might be true about the
world as a whole
Meta-analysis = combining results from multiple studies
Open science = an evolving set of practices intended to reduce bias by increasing the openness of research
and thus ensuring that research results are accurate and worthy of our trust
CONFIDENCE INTERVAL (CI)
= a range of values calculated from our data that, most likely, includes the true value of what we're
estimating about the population
• We are 95% confident the interval [51,55] includes the true level of support in the population
o Point estimate is 53, confidence interval is [51,55], margin of error is 2%
• 95% CIs are by far the most common
• Point estimate = the best single value the data can give us for what we're estimating about the
population
• We can refer to a CI as interval estimate = an interval containing the most plausible values for the
population value
• What does a short(er) CI mean?
o Sample estimate is most likely very close to the true value
o The margin of error is smaller
o The precision is greater
• The curve illustrates how plausibility and likelihood varies across and beyond the interval
o Values near the center of a CI → most plausible
o Plausibility decreases toward the ends of the CI and beyond
Margin of error (MoE) = the likely greatest error in the point estimate
• Our measure of the precision of estimation
• Small MoE → a short CI and a precise estimate
• The point estimate is unlikely to be more than the MoE away from the true value
• Half the length of the CI
Sampling variability = the variability in results caused by using different samples
• Variation from poll to poll, when they are all carried out at the same time and in the same way, but
using different samples
• Larger sample → result is closer to the true value in the population → CI is shorter → CI is more
precise
• A sample four times as large gives a CI about half as long
Uncertainty
• Beyond sampling variability, there are other sorts of uncertainty:
o Do we have the full story or were these results selected in some way that might give a
misleading message?
, ▪ Selected results: maybe there were other results that the researcher didn't report → the
results are selected to reflect some preference or bias
o Do we have full details about how the data were collected?
▪ How was the sample of people obtained? Was it representative of the whole population?
What questions were asked? How many of the people in the sample answered the
questions? Etc.
The replicability crisis and open science
• Replicability crisis = the alarming discovery that a number of widely known and accepted research
findings cannot be replicated
o Replication = a repeat of an original study, similar to the original but with a new sample
• Requirements for open science
o Avoid misleading selection of what's reported
o Report research in full detail
• Open science addresses the crisis by aiming to reduce the chance that incorrect research results are
obtained and reported. Why 'open'? Because full information about every stage of research should be
openly available.
META-ANALYSIS
= if we have results from two or more similar studies, we can use meta-analysis to combine the results
• Usually, this gives a shorter CI, because adding information from additional studies reduces
uncertainty about where the population average lies
• Forest plot = displays individual study results → shows point and interval estimates for individual
studies and displays the meta-analysis result as a diamond
• Meta-analytic thinking = thinking that prompts us to watch out for any opportunity for meta-analysis to
help. It prompts us to think of any study as one contribution, to be combined where possible with
earlier studies, replications and other related studies yet to come.
Step-by-step plan for estimation
1. State the research question
− Express it as 'how much' or 'to what extent'
2. Identify the measure that's most appropriate for answering that question
3. Design a study that uses that measure and gives us good point and interval estimates to answer our
question
4. After running the study, examine the data, calculate the point and interval estimates, and make a figure
5. Interpret these, using judgment in the research context
6. Report the study, making sure to state there was no selective reporting or just some of the results, and
giving full details of every aspect of the study
7. Adopt a meta-analytical thinking throughout. Seek other similar studies and, if appropriate, conduct a
meta-analysis. Consider conducting replication.
2
,CHAPTER 2: RESEARCH FUNDAMENTALS
Inference from sample to population
• Statistical inference = a set of techniques for drawing conclusions from a sample to the population the
sample came from
o Estimation = our favored approach to statistical inference
• Mean = another name for average
• For the sample we use Roman letters and refer to sample statistics
• For the population we use Greek letters and refer to population parameters
• Population = a usually very large set of people about which we are interested in drawing conclusions
• Sample = a set of people selected from a population
• Descriptive statistic = a summary number that tells us about a set of data
• Such as the sample mean
• Inferential statistic = calculated from sample data and tells us about the underlying population
• Such as CI
Random sampling
• A sample needs to be representative of the population
• How to do that? Seek a random sample! This requires that:
1. Every member of the population has an equal probability of being chosen
2. All members of the sample are chosen independently. The choice of one sample member
has no influence on whether another is chosen.
• Random sample = a sample that is chosen from the population such that all population members have
the same probability of beings chosen and all sample members are chosen independently
• Convenience sample = a practically achievable sample from the population
o In practice it is rarely possible to meet the requirements of random sampling, so you are likely to
have to use a convenience sample. If you need to, you should:
▪ Come as close as possible to meeting the requirements for random sampling
▪ Make the assumption in order to calculate CI's, and then making a judgment as to
whether it's reasonable to draw conclusions about a particular population
▪ Statistical model = a set of assumptions
o Random sampling is one assumption in the statistical model we use for calculation of inferential
statistics
3
, Making comparisons
• To make a comparison, a study needs at least two variables
1. Independent variable (IV)
= the variable whose values are chosen or manipulated by the researcher
• Can take two values / two levels
o Levels = different values taken by the independent variable
▪ Are also called conditions or treatments
▪ Control condition = the condition that provides a baseline or starting point for a
comparison
• Control group = a group that experiences the control condition
2. Dependent variable (DV)
= the variable that's measured in the study and provides the data to be analyzed
Experimental and non-experimental research
• Non-experimental research = uses pre-existing groups, not formed by random assignment or
manipulation of the IV
o Is easy and convenient, but the trouble is that groups almost certainly differ in more ways than
independent variable
o Therefore we can't conclude that any difference we observe was caused by the independent
variable. It could have been caused by any of the possible differences between the two groups.
• Confound = an unwanted difference between groups, which is likely to limit the
conclusions we can draw from a study
• Experimental research = uses random assignment of participants to groups or conditions, to the
different levels of the IV that is being manipulated
o Random assignment tends to even out all the other differences → helps avoid confounds → can
justify a causal conclusion
Measurement
• Imagine we want to measure the length of a table. We could say that length is our construct of interest
and that we operationalize the length of the table as the reading of the tape measure.
o Construct of interest = the underlying characteristic we wish to study
o Operationalize = a measure operationalizes a construct if it provides a practical way of
measuring that construct
Reliability
= repeatability or consistency
• If you measure again, are you likely to get the same or a very similar value?
o Example: a tape measure is very reliable
• Test-retest reliability = the extent that a measure gives the same result when repeated in the same
situation
o The simplest way to measure reliability
o Test is taken from the same person, under conditions as close as possible to the same
Validity
= the extent a measure actually measures what we want it to measure
• We can be confident that a tape measure does measure length, and not density or temperature. But
does our questionnaire tap anxiety, rather than arousal or introversion?
• Ways to asses validity
1. Compare or correlate your scores with other scores
2. Investigate how well scores predict a relevant outcome
4
