Lecture notes

Extensive Statistical Learning Notes

0 purchase

Module
Statistical Learning

Institution
The University Of Kent (UKC)

Detailed Statistical Learning notes from a First Class student. A lot of solved examples and added captions with verbal explanations from lecturer for extra understanding.

[Show more]

Preview 4 out of 113 pages

View example

Uploaded on August 8, 2021
Number of pages 113
Written in 2021/2022
Type Lecture notes
Professor(s) Alexa
Contains All classes

£7.49

Add to cart

Add to wishlist

100% satisfaction guarantee
Immediately available after payment
Both online and in PDF
No strings attached

STATISTICAL LEARNING

Contents
Chapter 1 – Inference Using the Multivariate Normal Distribution...................................................................................... 2
LECTURE 1.................................................................................................................................................................... 2
LECTURE 2.................................................................................................................................................................... 9
LECTURE 3...................................................................................................................................................................13
Chapter 2 – Principal Components Analysis.......................................................................................................................16
LECTURE 4...................................................................................................................................................................16
LECTURE 5...................................................................................................................................................................23
LECTURE 6 - PRACTICAL 1 ............................................................................................................................................31
Chapter 3 – Factor Analysis ..............................................................................................................................................32
LECTURE 7...................................................................................................................................................................32
LECTURE 8...................................................................................................................................................................39
LECTURE 9 - PRACTICAL 2 ............................................................................................................................................46
Chapter 4 – Canonical Correlation Analysis .......................................................................................................................47
LECTURE 10.................................................................................................................................................................47
LECTURE 11.................................................................................................................................................................52
LECTURE 12 - PRACTICAL 3...........................................................................................................................................53
Chapter 5 – Discriminant Analysis .....................................................................................................................................54
LECTURE 13.................................................................................................................................................................54
LECTURE 14.................................................................................................................................................................65
LECTURE 15 – PRACTICAL 4 ..........................................................................................................................................66
Chapter 6 – Cluster Analysis .............................................................................................................................................67
LECTURE 16.................................................................................................................................................................67
LECTURE 17.................................................................................................................................................................71
LECTURE 18 – PRACTICAL 1 ..........................................................................................................................................72
LECTURE 19.................................................................................................................................................................75
LECTURE 20.................................................................................................................................................................78
LECTURE 21 – PRACTICAL 2 ..........................................................................................................................................80
Chapter 7 – Graphical Models ..........................................................................................................................................82
LECTURE 22.................................................................................................................................................................82
LECTURE 23.................................................................................................................................................................86
LECTURE 24 – PRACTICAL 3 ..........................................................................................................................................89
LECTURE 25.................................................................................................................................................................91
LECTURE 26.................................................................................................................................................................96
LECTURE 27 – PRACTICAL 4 ..........................................................................................................................................98
Chapter 8 – Mixture Models ........................................................................................................................................... 101
LECTURE 28............................................................................................................................................................... 101
LECTURE 29............................................................................................................................................................... 112
LECTURE 30 – PRACTICAL 5 ........................................................................................................................................ 113

, STATISTICAL LEARNING

Chapter 1 – Inference Using the Multivariate Normal
Distribution
LECTURE 1

Multivariate Normal Distribution

- Multivariate analyses are often carried out with the aim to explore the data,
rather than to test hypotheses.
- Nevertheless, there are cases when the complexities of the data require us to
make use of parametric distributions to proceed with our analysis.
- In these cases, the multivariate normal distribution plays an important role in
multivariate statistics.

Definition
A vector of
means. So - We say that 𝑋 = (𝑋1 , … , 𝑋𝑝 )𝑇 follows a p-variate normal distribution with
each variable
- Mean 𝜇 ∈ ℝ𝑝 and
of 𝑋 has its
own mean - Covariance matrix ∑ , a (p x p)-dimensional symmetric, positive definite matrix.
-
- If its p.d.f 𝑓(𝑋) is such that for any 𝑥 = (𝑥1 , … , 𝑥𝑝 )𝑇 , In the diagonals of the
covariance matrix will be
1 1 the variance of each
𝑓𝑋 (𝑥) = 𝑒𝑥𝑝{− (𝑥 − 𝜇)𝑇 ∑−1 (𝑥 − 𝜇)} variable and on each other
(2𝜋)𝑝/2 |∑|1/2 2
entry will be the
covariance of each pair
- We write 𝑋 ~ 𝑀𝑁(𝜇, ∑)
with each other.
The
determinant of
variance-
covariance
matrix.

Covariance and correlation matrix

- In general, 𝜎𝑖𝑗 is the (i,j)-th element of ∑ with 𝑉𝑎𝑟(𝑋𝑖 ) = 𝜎𝑖𝑖 = 𝜎𝑖 2 .
- The correlation between the i-th and j-th variable is
𝜎𝑖𝑗
𝜌𝑖𝑗 =
√𝜎𝑖 2 √𝜎𝑗 2
- Which gives 𝜎𝑖𝑗 = 𝜌𝑖𝑗 𝜎𝑖 𝜎𝑗
- The correlation matrix is:
1 𝜌12 ⋯ 𝜌1𝑝
𝜌21 1 ⋯ 𝜌2𝑝
𝑃=
⋮ ⋮ ⋱ ⋮
(𝜌𝑝1 𝜌𝑝2 ⋯ 1 )

, STATISTICAL LEARNING

Example: Bivariate Normal Random Vector (p=2)

- In this case, we have
𝜎2 𝜌𝜎1 𝜎2
𝑋 = (𝑋1 , 𝑋2 )𝑇 , 𝜇 = (𝜇1 , 𝜇2 )𝑇 and ∑ = ( 1 )
𝜌𝜎1 𝜎2 𝜎2 2
- Where ρ is the correlation coefficient. Clearly,
1 𝜎1 2 𝜌𝜎1 𝜎2
|∑| = 𝜎1 2 𝜎2 2 (1 − 𝜌2 ) ; ∑−1 = 2 2 ( )
𝜎1 𝜎2 (1 − 𝜌2 ) 𝜌𝜎1 𝜎2 𝜎2 2

Example: Bivariate Normal Random Vector

- The density is:
1 1 𝑄(𝑥)
𝑓𝑋 (𝑥) = exp {− }
(2𝜋)𝜎1 𝜎2 √1 − 𝜌2 2 (1 − 𝜌2 )
- Where,
1 𝑇 𝜎2 2 −𝜌𝜎1 𝜎2
𝑄(𝑥) = ( 𝑥 − 𝜇) ( ) (𝑥 − 𝜇)
2
𝜎1 𝜎2 2 −𝜌𝜎1 𝜎2 𝜎1 2
𝑥1 − 𝜇1 2 𝑥1 − 𝜇1 𝑥2 − 𝜇2 𝑥2 − 𝜇2 2
=( ) − 2𝑝 ( )( )+( )
𝜎1 𝜎1 𝜎2 𝜎2

Example: Bivariate Normal Random Vector with 𝝁𝟏 = 𝝁𝟐 and 𝝈𝟏 = 𝝈𝟐 = 𝟏
2d contour
plot of the
3d plot.

Each of the
marginals is
a standard
normal.
Where this
axis
represents
the value of
one of the
random
variables.
And so does
the other. Easy to plot for bivariate normal. As soon as we go to more than two dimension it becomes
more difficult to visualise. Clearly in one dimension we just plotted the bell-shaped curve
and now it 2-dimensions, it looks like a bump. In all cases we have mean to zero and
variance to 1 and all that changes between the three is the correlation.

How each marginal is correlated with each other depends on 𝜌.

The y axis just indicates the pdf evaluated at the two values of these two variables.

Positive correlation means when one variable is positive it is likely that the second variable
is also positive. Higher density areas on the contour is where both are positive, or both are
negative.
And vice versa for a negative correlation

, STATISTICAL LEARNING

Properties of 𝑿 ~ 𝑴𝑵(𝝁, ∑)

1. 𝑋 has the same distribution as 𝜇 + Σ1/2 (𝑌1 , … , 𝑌𝑝 )𝑇 , where 𝑌1 , … , 𝑌𝑝 are
independent N(0,1).
2. 𝐸[𝑋] = 𝜇 and 𝐶𝑜𝑣(𝑋) = Σ
1
3. Σ −2 (𝑥 − 𝜇) ~ 𝑀𝑁(0, 𝐼𝑝 ) ; each element is independent N(0,1) .
1
4. Sample mean: 𝑋̅ = 𝑛 ∑𝑛𝑖=1 𝑋𝑖 ; 𝑋̅~ 𝑀𝑁 (𝜇, 𝑛−1 Σ).
5. 𝑋̅ is the MLE for 𝜇 and
∑𝑛 ̅ ̅ 𝑇
𝑖=1(𝑋𝑖 −𝑋 )(𝑋𝑖 −𝑋 )
𝑆= 𝑛
is the MLE for Σ

Importance of Multivariate Normal Distribution

- Multivariate central limit theorem: If 𝑋1 , … , 𝑋𝑛 are i.i.d random vectors, with
𝐸[𝑋𝑖 ] = 𝜇 and 𝐶𝑜𝑣(𝑋𝑖 ) = Σ then,
𝑛
1 1
∑ 𝑋𝑖 → 𝑀𝑁 (𝜇, Σ) 𝑎𝑠 𝑛 → ∞
𝑛 𝑛
𝑖=1

- It is mathematically convenient: the p.d.f is completely determined by 𝐸[𝑋] = 𝜇
1
and 𝐶𝑜𝑣(𝑋) = 𝛴 . The number of parameters is 2 𝑝(𝑝 + 3).

First row would be
covariance between the
first variable of 𝑥1 with
each variable of 𝑥2 .
Repeat for k times for
the k variables of 𝑥1 .
This will give us Σ12
𝑥1 of dimension k. quadrant.
𝑥2 of dimension p-k.

So, if we want the sub vector 𝑥1 of size k. It would be represented as below:

We won’t
prove this
expression
but will be
using at some
point in this
module.

The benefits of buying summaries with Stuvia:

Guaranteed quality through customer reviews

Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.

Quick and easy check-out

You can quickly pay through credit card for the summaries. There is no membership needed.

Focus on what matters

Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!

Frequently asked questions

What do I get when I buy this document?

You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.

Satisfaction guarantee: how does it work?

Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.

Who am I buying these notes from?

Stuvia is a marketplace, so you are not buying this document from us, but from seller ardit026. Stuvia facilitates payment to the seller.

Will I be stuck with a subscription?

No, you only buy these notes for £7.49. You're not tied to anything after your purchase.

Can Stuvia be trusted?

4.6 stars on Google & Trustpilot (+1000 reviews)

69411 documents were sold in the last 30 days

Founded in 2010, the go-to place to buy revision notes and other study material for 15 years now

Start selling