Regression Analysis
After this lecture, you understand
- What regression analysis is
- How regression analysis works
- How regression analysis results are interpreted
Regression analysis: is a dependence technique, it examines the relationship between
dependent (outcome) and independent (predictor) variables. It is about understanding how
one variable aCects or is associated with another.
- It is one of the most frequently used data analysis methods
- Regression primarily looks at the linear relationship between a single dependent
variable and one or more independent variables, which are both usually numeric.
The potential applications for regression analysis are:
• Cause analysis: regression can analyze how x aCects y
• Prediction: regression can predict outcomes of x based on predictors like y and z
• Data types: Regression can be used for both time series data (data collected over
time) and cross-sectional data (data collected at a single point in time)
Simple linear regression model:
𝑦 = 𝛽! + 𝛽"!
This deterministic model assumes perfect predictions, which is often unrealistic in real-
world data. Therefor the probabilistic model includes an error term (𝜖) to account for
random variations or unobserved factors:
𝑦 = 𝛽! + 𝛽"! + 𝜖
Regression coeCicients are needed to be estimated when the population values of the
intercept and the slopes are unknown. Which can happen if there only is a sample of the
populations, so the parameters have to be estimated using the sample data. The
coeCicients are estimated using ordinary least squares (OLS).
, OLS is a method used to find the best-fitting regression line by minimizing the sum of
squared residuals.
The residual is the diCerence between the actual value (𝑦# ) and the predicted value (𝑦'# ) for
each datapoint: 𝑒# = 𝑦# − 𝑦'# .
The goal of OLS is to minimize the total squared residuals. The formula for that is:
$
2! + 𝛽3" 𝑥# ))&
Minimize 0(𝑦# − (𝛽
#%"
Squaring residuals ensures both positive and negative deviations are treated equally. It
penalizes large deviations more heavily, leading to a line that is as close as possible to the
datapoints.
Two key concepts in regression analysis are predicted values and residuals.
• Predicted values: The outputs of the regression line for the given values of the
independent variable x.
• Residuals: Is the diCerence between the actual value and the predicted value.
Residuals help to evaluate the fit of the regression model
o A positive residual means the model underestimated the actual value.
o A negative residual means the model overestimated the actual value.
o If residuals are small, the model closely approximates the actual data.
o If residuals are large, it indicates poor predictive performance.
Residuals are useful to estimate the variance of the error term and to judge the fit of our
model. Estimating the variance of the error term helps us understand how much variability
in y is not explained by the regression model.
∑$ ) "#
"%& (
To estimate the variance in the error term: 𝜎' & = $*&
∑$ ) " *,-) )#
"%&(, ∑$
"%& ( ) "#
To judge the fit of the model: 𝑅& = ∑$ ) " *,-)
= 1 − ∑$ (,)" *,-) )#
"%&(, "%&
𝑅& tells us the proportion of variation in y that can be explained by x. The closer R2 is to 1,
the better the fit.
After this lecture, you understand
- What regression analysis is
- How regression analysis works
- How regression analysis results are interpreted
Regression analysis: is a dependence technique, it examines the relationship between
dependent (outcome) and independent (predictor) variables. It is about understanding how
one variable aCects or is associated with another.
- It is one of the most frequently used data analysis methods
- Regression primarily looks at the linear relationship between a single dependent
variable and one or more independent variables, which are both usually numeric.
The potential applications for regression analysis are:
• Cause analysis: regression can analyze how x aCects y
• Prediction: regression can predict outcomes of x based on predictors like y and z
• Data types: Regression can be used for both time series data (data collected over
time) and cross-sectional data (data collected at a single point in time)
Simple linear regression model:
𝑦 = 𝛽! + 𝛽"!
This deterministic model assumes perfect predictions, which is often unrealistic in real-
world data. Therefor the probabilistic model includes an error term (𝜖) to account for
random variations or unobserved factors:
𝑦 = 𝛽! + 𝛽"! + 𝜖
Regression coeCicients are needed to be estimated when the population values of the
intercept and the slopes are unknown. Which can happen if there only is a sample of the
populations, so the parameters have to be estimated using the sample data. The
coeCicients are estimated using ordinary least squares (OLS).
, OLS is a method used to find the best-fitting regression line by minimizing the sum of
squared residuals.
The residual is the diCerence between the actual value (𝑦# ) and the predicted value (𝑦'# ) for
each datapoint: 𝑒# = 𝑦# − 𝑦'# .
The goal of OLS is to minimize the total squared residuals. The formula for that is:
$
2! + 𝛽3" 𝑥# ))&
Minimize 0(𝑦# − (𝛽
#%"
Squaring residuals ensures both positive and negative deviations are treated equally. It
penalizes large deviations more heavily, leading to a line that is as close as possible to the
datapoints.
Two key concepts in regression analysis are predicted values and residuals.
• Predicted values: The outputs of the regression line for the given values of the
independent variable x.
• Residuals: Is the diCerence between the actual value and the predicted value.
Residuals help to evaluate the fit of the regression model
o A positive residual means the model underestimated the actual value.
o A negative residual means the model overestimated the actual value.
o If residuals are small, the model closely approximates the actual data.
o If residuals are large, it indicates poor predictive performance.
Residuals are useful to estimate the variance of the error term and to judge the fit of our
model. Estimating the variance of the error term helps us understand how much variability
in y is not explained by the regression model.
∑$ ) "#
"%& (
To estimate the variance in the error term: 𝜎' & = $*&
∑$ ) " *,-) )#
"%&(, ∑$
"%& ( ) "#
To judge the fit of the model: 𝑅& = ∑$ ) " *,-)
= 1 − ∑$ (,)" *,-) )#
"%&(, "%&
𝑅& tells us the proportion of variation in y that can be explained by x. The closer R2 is to 1,
the better the fit.
