This is a summary of all the study material for the course 'Data Analytics for engineers' (2IAB0) given at the TU/e. All the needed information for the exam is summarized in this document, nothing is left out and hence this summary is a bit longer than others.
Indeed, I passed with an 8. So guys, its definitely worth buying this notes!!!
Seller
Follow
LukevDongen
Reviews received
Content preview
Summary – Data Analytics 2021
EDA – week 1
Data types
❖ numerical data - data that has intrinsic numerical value
o continuous data – data that can attain any value on a given measurement
scale (has no gaps)
▪ interval data (continuous data for which only differences have
meaning, has no fixed “zero point”,, e.g. temperature in Celsius (0
degrees is not a fixed minimum), pH, clock time, IQ scores, birth year,
longitude)
▪ ratio data (continuous data for which ratios make sense, has fixed
“zero point”, so ratios also do make sense) Movies: budget. Other
examples: temperature in Kelvin, distance, time duration.
o discrete data – data that can only attain certain values (e.g., integers)
❖ categorical data – data that has no intrinsic numerical value
o nominal: two or more outcomes that have no natural order, e.g. movie genre
(adventure, action, Western,…), hair colour (blond, brown, grey, ….)
o ordinal: two or more outcomes that have a natural order, e.g., movie ratings
(bad, neutral, good)
❖ Dichotomous data – binary data with ‘success’ or ‘failure’ (1 or 0, respectively)
Tables
Reference table: store “all” data in a table so that it can be looked up easily
Demonstration table: table to illustrate a point (so present just enough data)
Elementary Statistical Plots
Dot plots Cumulative histogram
Shows counts or percentages of the
- Good for showing actual values and
current bin together with the counts
structure of numerical variables
or percentages of all bins to the left
- Not suitable for large data sets
of that bin.
- The jitter option
cumulative histograms are useful to
illustrate thresholds (problem:
choosing fixed bin size)
Histogram: distribution of numerical data
The range of data values is split in bins (intervals of values)
- you can choose the number of bins, or
- choose the bin width you would like to have
The histogram show the number of observations in the data set for every bin
(there are versions that show percentages).
Histogram are sensitive to bin width
- bin width too small → too wiggly
- bin width too large → too few details
Rule of thumb for choosing a sensible nr of bins: ≈√𝑛
Bar charts and histogram
A histogram should not be confused with a bar chart .
Bar charts are for categorical data, histograms for numerical data.
Scatter plot
Scatter plot allow to investigate relations between quantities.
,Summary Statistics (summary statistics are convenient ways to summarize data in a numerical form)
❖ level: location summary statistics → what are “typical” values
❖ spread: scale summary statistics → how much do values vary?
❖ relation: association summary statistics → how do values of different quantities vary
simultaneously
Location summary statistics (location statistics are good to describe “typical” values)
(it is not sufficient to only use location summary statistics)
1 𝑛
- mean (average): ∑ 𝑥, denoted by 𝑥̅
𝑛 𝑖=1 𝑖
- median:
o odd # of observations: middle value when ordered from small to large
o even # of observations: or average of two middle values when order from
small to large
- mode: most frequently occurring value, may be non-unique
Mean is sensitive to “outliers”, the median is not → Mean can be misleading / difficult to
interpret for non-symmetric distributions. Median is robust, less sensitive to outliers.
- Quartiles
Re-order the data from small to large:
• 1st quartile = cut-off point for 25% of the data
• 2nd quartile = cut-off point for 50% of the data (= median)
• 3rd quartile = cut-off point for 75% of the data
- Percentiles
• Pth percentile – a cut-off point for P% of data
• We define the 0th percentile to be the minimal element of the dataset and the
100th percentile to be the maximal element of it.
• For a percentile P we compute its location in a data
𝑃
set of n observations: 𝐿𝑃 = 1 + 100 (𝑛 − 1)
50
(example: 𝑛 = 12, 𝐿50 = 1 + (12 − 1) = 6.5)
100
• Computing Pth percentile value by linear
interpolation : let L and h be the observations at the
position 𝐋𝐏 and 𝐋𝐏 in the ordered data set.
• Pth percentile value = 𝒍 + ( 𝐋𝐏 − 𝐋𝐏 ) (𝐡 − 𝐥)
Scale statistics (scale summary statistics are good to describe spread or fluctuations)
- range = max – min
- interquartile range (IQR) = 3rd quartile - 1st quartile
1
- sample variance (= 𝒔𝟐 ) = 𝑛−1 ∑𝑛𝑖=1(𝑥𝑖 − 𝑥̅ )2 (often denoted by 𝜎 2 or 𝑆 2 )
1
- sample standard deviation s =√𝑛−1 ∑𝑛𝑖=1(𝑥𝑖 − 𝑥̅ )2
- median absolute deviation (MAD): median of the range
absolute deviation from the median (first find the median, then calculate the absolute
deviations from that median, then calculate the median of these new values)
The sample variance is more convenient mathematically.
The range, sample variance and sample standard deviation are sensitive to “outliers”, IQR and MAD are not.
The sample standard deviation can be used as a general unit to describe variability.
, Standardization (z-score normalization) (use the standard deviation as general unit to measure distance)
The z-score transforms data in their original units into universal statistical unit of standard
𝑥−𝑥̅
deviation from the mean 𝒙′ = 𝑠
The mean value of the z-scores of data set is 0 and the standard deviation is 1.
Negative z-score the value is below the mean,
Positive z-score the value is above mean.
Rule of thumb: observations with a z-score larger than 2.5 are considered to be extreme
(“outliers”).
Association statistics (try to capture in a number how strong the relation between two quantities is)
Indicates whether an association is:
- a positive association (e.g., higher budget → higher profit)
- a negative association (e.g., higher strength of material → less impact when force applied)
Sample correlation
1
- Sample covariance 𝑠𝑥𝑦 = ∑𝑛 (𝑥 − 𝑥̅ ) (𝑦𝑖 − 𝑦̅)
𝑛−1 𝑖=1 𝑖
In order to be useful, the sample covariance must be scaled :
𝑠
- Sample correlation 𝑟𝑥𝑦 = 𝑥𝑦 (𝑠𝑥 =std. dev of x). We have −1 ≤ 𝑟𝑥𝑦 ≤ 1
𝑠𝑥 𝑠𝑦
“No” relation: 𝑟𝑥𝑦 close to 0
“Perfect” relation: 𝑟𝑥𝑦 close to -1 (negative correlation) or +1 (positive correlation).
Warning: correlation only measures strength linear relations (“straight lines”)
Advanced Statistical Plots
Typical distribution shapes
Improved histograms: kernel density plots
• good tool to explore distribution shape
• kernel density plots overcome the drawbacks of histograms because they do not
have fixed bins
• Choose a bandwidth to be taken around each data point (the choice can be
delegated to software)
• Generate a kernel with the chosen bandwidth for every data point
• Count the data points weighted by the kernel.
• Bandwidth choice is important!!! (similarly to the choice of the bin
size for histograms)
• There is no direct interpretation of the scale of the y-axis!
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller LukevDongen. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $3.77. You're not tied to anything after your purchase.
