Statistics & Methodology
MSc Data Science & Society
Tilburg University
1
,Lecture 1. Statistical Inference, Modeling, and Prediction
Introduction to Statistical Inference
Motivating Example
Imagine you are working for a F1 team. Your job is to use data from past seasons to optimize the
baseline setup of your team’s car (e.g., tires, brakes,…). Suppose you have two candidate setups (A &
B) that you want to compare. For each setup, you have 100 past lap times. How do you distill those
200 lap times into a succinct decision between the two setups?
Suppose I tell you that the mean lap time for Setup A is 118 seconds and the mean lap time for Setup
B is 110 seconds. Let’s assume for now that this is the only information that you have available at the
moment. Can you confidently recommend Setup B? What caveats might you consider? Controlling for
factors such as driver experience and weather conditions is a good way of thinking but there is
one more fundamental issue that we need to consider before making a decision: variability.
First scenario: suppose I tell you that the standard deviation for the times under Setup A is 7 seconds
and the standard deviation for the times under Setup B is 5 seconds. How would you incorporate this
new information into your decision? Second scenario: suppose that the standard deviation of times
under Setup A is 35 seconds and the standard deviation under Setup B is 25 seconds. How should you
adjust your appraisal of the setups’ relative benefits? You are much more confident recommending
the setup B under the first scenario, because we measured the average lap times with a much
greater precision. An average lap time of 110 seconds with only 5 seconds standard deviation is
quite a precise measure. One with 25 seconds is not particularly precise. We might not be able
to differentiate between the two setups because the means might be different, but the individual
scores overlap quite a bit. This is why we do not only have to consider the mean of the two
distributions of lap times, but also the variability (e.g., the standard deviation).
Statistical Reasoning
The preceding example calls for statistical reasoning.
- The foundation of all good statistical analyses is a deliberate, careful, and thorough
consideration of uncertainty.
- In the previous example, the mean lap time for Setup A is clearly longer than the mean
lap time for Setup B. If the times are highly variable, with respect to the size of the mean
difference, we may not care much about the mean difference.
- The purpose of statistics is to systematize the way that we account for uncertainty when
making data-based decisions.
Statistics for Data Science
Data scientists must scrutinize large numbers of data and extract useful knowledge.
- Data contain raw information.
- To convert this information into actionable knowledge, data scientists apply various data
analytic techniques.
- When presenting the results of such analyses, data scientists must be careful not to over-
state their findings. Too much confidence in an uncertain finding could lead your
employer to waste large amounts of resources chasing data anomalies.
- Statistics offer us a way to protect ourselves from ourselves (e.g., from conformation
biases).
2
,Recap Probability Distributions
Probability distributions quantify how likely it is
to observe each possible value of some
probabilistic entity (e.g., height). Probability
distributions are re-scaled frequency
distributions. We can build up the intuition of a
probability density by beginning with a
histogram, and finding the frequency of, e.g.
‘1.70m’ is.
With an infinite number of bins, a histogram smooths into a continuous
curve. In a loose sense, each point on the curve gives the probability of
observing the corresponding X value in any given sample. E.g.: for a
value of x=2, we say that the probability is something around 0.06.
However, we only talk about areas under the curve; not about a single
point.
The area under the curve must integrate to 1.0, because the total
probability of an event happening is under the entire curve. If you
consider all the possible outcomes, then one of the outcomes will
happen, and therefore the curve must integrate to one.
Reasoning with Distributions
This figure represents respectively
the first and second scenario of the
motivating example. In scenario 1,
the setups clearly show two
separate distributions; whereas in
the second 2, it becomes much
more difficult to distinguish
whether the observations
originally come from Setup A or
Setup B.
Statistical Testing
In practice, we may want to distill the information in the preceding plots into a simple statistic
so we can make a judgment. One way to distill this information and control for uncertainty when
generating knowledge is through statistical testing. When we conduct statistical tests, we weight
the estimated effect by the precision of the estimate. A common type of statistical test, the Wald
Test, follows this pattern:
𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 − 𝑁𝑢𝑙𝑙 𝐻𝑦𝑝𝑜𝑡ℎ𝑒𝑠𝑖𝑧𝑒𝑑 𝑉𝑎𝑙𝑢𝑒
𝑇=
𝑉𝑎𝑟𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦
If we want to test the null hypothesis of a zero mean difference, applying Wald test logic to
control for the uncertainty in our estimate, results in the familiar t-test:
(𝑋;< − 𝑋;= ) − 0
𝑡=
C (𝑛BF BF
@𝑆<B= E + 𝑛= )
3
, where:
𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒 = 𝑋;< − 𝑋;=
and:
C (𝑛BF BF
𝑉𝑎𝑟𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = H𝑆<B= E + 𝑛= )
(𝑛< − 1)𝑆<C + (𝑛= − 1)𝑆=C 1 1
= I L + M
𝑛< + 𝑛= − 2 𝑛< 𝑛=
Applying the preceding formula to the first scenario of our example problem produces:
(118 − 110) − 0 8
𝑡= ≈ ≈ 9.30
(100 − 1)7 C + (100 − 1)5C 1 1 0.86
H Q100 + 100R
100 + 100 − 2
And to the second scenario (where the effect is the same, but the variability changes):
(118 − 110) − 0 8
𝑡= ≈ ≈ 1.86
(100 − 1)𝟑𝟓C + (100 − 1)𝟐𝟓C 1 1 4.30
H Q100 + 100R
100 + 100 − 2
Generally, the larger the test statistic, the better.
Statistical Testing in R
4
