Detailed Statistical Learning notes from a First Class student. A lot of solved examples and added captions with verbal explanations from lecturer for extra understanding.
Chapter 1 – Inference Using the Multivariate Normal
Distribution
LECTURE 1
Multivariate Normal Distribution
- Multivariate analyses are often carried out with the aim to explore the data,
rather than to test hypotheses.
- Nevertheless, there are cases when the complexities of the data require us to
make use of parametric distributions to proceed with our analysis.
- In these cases, the multivariate normal distribution plays an important role in
multivariate statistics.
Definition
A vector of
means. So - We say that 𝑋 = (𝑋1 , … , 𝑋𝑝 )𝑇 follows a p-variate normal distribution with
each variable
- Mean 𝜇 ∈ ℝ𝑝 and
of 𝑋 has its
own mean - Covariance matrix ∑ , a (p x p)-dimensional symmetric, positive definite matrix.
-
- If its p.d.f 𝑓(𝑋) is such that for any 𝑥 = (𝑥1 , … , 𝑥𝑝 )𝑇 , In the diagonals of the
covariance matrix will be
1 1 the variance of each
𝑓𝑋 (𝑥) = 𝑒𝑥𝑝{− (𝑥 − 𝜇)𝑇 ∑−1 (𝑥 − 𝜇)} variable and on each other
(2𝜋)𝑝/2 |∑|1/2 2
entry will be the
covariance of each pair
- We write 𝑋 ~ 𝑀𝑁(𝜇, ∑)
with each other.
The
determinant of
variance-
covariance
matrix.
Covariance and correlation matrix
- In general, 𝜎𝑖𝑗 is the (i,j)-th element of ∑ with 𝑉𝑎𝑟(𝑋𝑖 ) = 𝜎𝑖𝑖 = 𝜎𝑖 2 .
- The correlation between the i-th and j-th variable is
𝜎𝑖𝑗
𝜌𝑖𝑗 =
√𝜎𝑖 2 √𝜎𝑗 2
- Which gives 𝜎𝑖𝑗 = 𝜌𝑖𝑗 𝜎𝑖 𝜎𝑗
- The correlation matrix is:
1 𝜌12 ⋯ 𝜌1𝑝
𝜌21 1 ⋯ 𝜌2𝑝
𝑃=
⋮ ⋮ ⋱ ⋮
(𝜌𝑝1 𝜌𝑝2 ⋯ 1 )
, STATISTICAL LEARNING
Example: Bivariate Normal Random Vector (p=2)
- In this case, we have
𝜎2 𝜌𝜎1 𝜎2
𝑋 = (𝑋1 , 𝑋2 )𝑇 , 𝜇 = (𝜇1 , 𝜇2 )𝑇 and ∑ = ( 1 )
𝜌𝜎1 𝜎2 𝜎2 2
- Where ρ is the correlation coefficient. Clearly,
1 𝜎1 2 𝜌𝜎1 𝜎2
|∑| = 𝜎1 2 𝜎2 2 (1 − 𝜌2 ) ; ∑−1 = 2 2 ( )
𝜎1 𝜎2 (1 − 𝜌2 ) 𝜌𝜎1 𝜎2 𝜎2 2
Example: Bivariate Normal Random Vector with 𝝁𝟏 = 𝝁𝟐 and 𝝈𝟏 = 𝝈𝟐 = 𝟏
2d contour
plot of the
3d plot.
Each of the
marginals is
a standard
normal.
Where this
axis
represents
the value of
one of the
random
variables.
And so does
the other. Easy to plot for bivariate normal. As soon as we go to more than two dimension it becomes
more difficult to visualise. Clearly in one dimension we just plotted the bell-shaped curve
and now it 2-dimensions, it looks like a bump. In all cases we have mean to zero and
variance to 1 and all that changes between the three is the correlation.
How each marginal is correlated with each other depends on 𝜌.
The y axis just indicates the pdf evaluated at the two values of these two variables.
Positive correlation means when one variable is positive it is likely that the second variable
is also positive. Higher density areas on the contour is where both are positive, or both are
negative.
And vice versa for a negative correlation
, STATISTICAL LEARNING
Properties of 𝑿 ~ 𝑴𝑵(𝝁, ∑)
1. 𝑋 has the same distribution as 𝜇 + Σ1/2 (𝑌1 , … , 𝑌𝑝 )𝑇 , where 𝑌1 , … , 𝑌𝑝 are
independent N(0,1).
2. 𝐸[𝑋] = 𝜇 and 𝐶𝑜𝑣(𝑋) = Σ
1
3. Σ −2 (𝑥 − 𝜇) ~ 𝑀𝑁(0, 𝐼𝑝 ) ; each element is independent N(0,1) .
1
4. Sample mean: 𝑋̅ = 𝑛 ∑𝑛𝑖=1 𝑋𝑖 ; 𝑋̅~ 𝑀𝑁 (𝜇, 𝑛−1 Σ).
5. 𝑋̅ is the MLE for 𝜇 and
∑𝑛 ̅ ̅ 𝑇
𝑖=1(𝑋𝑖 −𝑋 )(𝑋𝑖 −𝑋 )
𝑆= 𝑛
is the MLE for Σ
Importance of Multivariate Normal Distribution
- Multivariate central limit theorem: If 𝑋1 , … , 𝑋𝑛 are i.i.d random vectors, with
𝐸[𝑋𝑖 ] = 𝜇 and 𝐶𝑜𝑣(𝑋𝑖 ) = Σ then,
𝑛
1 1
∑ 𝑋𝑖 → 𝑀𝑁 (𝜇, Σ) 𝑎𝑠 𝑛 → ∞
𝑛 𝑛
𝑖=1
- It is mathematically convenient: the p.d.f is completely determined by 𝐸[𝑋] = 𝜇
1
and 𝐶𝑜𝑣(𝑋) = 𝛴 . The number of parameters is 2 𝑝(𝑝 + 3).
First row would be
covariance between the
first variable of 𝑥1 with
each variable of 𝑥2 .
Repeat for k times for
the k variables of 𝑥1 .
This will give us Σ12
𝑥1 of dimension k. quadrant.
𝑥2 of dimension p-k.
So, if we want the sub vector 𝑥1 of size k. It would be represented as below:
We won’t
prove this
expression
but will be
using at some
point in this
module.
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller ardit026. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $9.79. You're not tied to anything after your purchase.
