Summary
Advanced Data Analysis Theory Course Summary
- Course
- Institution
summary based on powerpoints and own notes
[Show more]
Seller
Followlauraheyndrickx
Reviews received
Content preview
ContentsWhat is data...........................................................................................................................................3
Data mining............................................................................................................................................4
Tasks...................................................................................................................................................5
Common data processing steps..............................................................................................................7
Processing steps for specific data types...............................................................................................10
Meta-genomics.....................................................................................................................................12
Introduction......................................................................................................................................12
Binning..............................................................................................................................................13
Downstream analysis........................................................................................................................13
Introduction..........................................................................................................................................14
Clustering..........................................................................................................................................14
Similarity...........................................................................................................................................14
Dendrograms....................................................................................................................................14
Hierarchical clustering..........................................................................................................................15
Single linkage, complete linkage, group average, centroid, Ward’s method....................................16
Partitional clustering............................................................................................................................17
k-means............................................................................................................................................18
Multivariate data..................................................................................................................................20
Data transformation.........................................................................................................................20
Normalization...............................................................................................................................20
Comparison between variables............................................................................................................21
Data projection.................................................................................................................................21
Working mechanism of PCA.................................................................................................................22
PCA loadings.....................................................................................................................................24
PCA scores........................................................................................................................................24
Scree plot..........................................................................................................................................25
Example case........................................................................................................................................25
possum dataset................................................................................................................................25
Nutrition data...................................................................................................................................27
t-SNE.....................................................................................................................................................30
Linear classifier.....................................................................................................................................32
Multiclass classification problem......................................................................................................33
Confusion matrix..............................................................................................................................36
Nearest neighbor classifier...................................................................................................................37
1
,Linear regression..................................................................................................................................38
Simple linear regression...................................................................................................................38
Multiple linear regression.................................................................................................................38
Non-linear regression...........................................................................................................................40
Logistic regression............................................................................................................................40
Cox regression..................................................................................................................................41
K-fold cross validation..........................................................................................................................41
Feature selection..............................................................................................................................43
Regularized regression..................................................................................................................43
Decision tree and random forests........................................................................................................46
Random forest..................................................................................................................................49
Neural networks...................................................................................................................................51
deep learning....................................................................................................................................53
Examples..............................................................................................................................................53
2
,Introduction
What is data
There were hypes in AI before, but recent hype is very important
o Something is happening that will transform the way of working
Data = collection of objects (items that you study) and their attributes
o Attribute = variable = feature = property or characteristic of an object
Eg: eye color, temperature…
A collection of attributes describes an object
Attribute values are numbers or symbols assigned to an attribute
Same attribute can be mapped to different attribute values
(eg: height can be measured in feet or meters)
Different attributes can be mapped to the same set of values
o However properties of attribute values can still be different
Different types possible
Nominal (like zip-codes, eye color…(=)), ordinal (rankings(<>)),
interval (calender dates, temperatures in Celsius(+-)) and ratios (*/)
Discrete (countable infinite set of values) of continuous (real
numbers -> only finite number of digits possible)
Big data = data for which conventional computer-techniques are not sufficient anymore due
to size, complexity…
o Even though computers get more powerful, conventional ways of building a program
to solve the problem will not be sufficient enough
A lot of things that worked for small data, did not work anymore on larger
datasets
o Characterized by the four V’s
Volume: drastically increasing volume of data
Eg: costs of reading out a full human genome has decreased
drastically when new sequence-techniques and computer
programmes were developed
o Now 1000 dollar for whole genome instead of 100.000.000
dollar -> within reach for medical applications
o Also important is that patents have expired -> prediction is
that reading of genome will even get cheaper
Velocity: the way all data is received
While you are waiting on previous data, more data is coming
towards you -> instant solutions needed
o Smartphones and -watches are measuring all the time
Not to bad to wait longer for certain information
o In hospital environment we need our data immediately
sometimes the data gets in so fast, that you don’t have enough
people to analyse this data = data management gap
need for new, effective, high-tech data transfer approach
o amount of data that has to be transferred is so much, that it
is more efficient to put it on harddrives and bring it by bike
to other places all over the world
3
, amount of wifi needed could be very high so that in
certain places in the world it isn’t a possibility
Variety: heterogeneity of data
No single type of data, but very diverse and a lot of background
information is needed to understand the data
o A lot of data is unstructured (not organized in matrices)
-> context information is very important to understand the
amount of data
Veracity: how trustable is all the data?
Because of large scale data and AI a new data intensive research paradigm has formed
Different dataset types
o Record = data consisting of collection of records, each which consists of a fixed set of
attributes
If data objects have the same fixed set of numeric attributes, then the data
can be thought of as points in a multi-dimensional space
Every dimension represents a distinct attribute
Represented by an m by n matrix
o M (rows) are objects and n (columns) are attributes
o Number of dimensions is dependent on the number of
attributes
Each document becomes a ‘term’ vector
Each term is a component (attribute) of the vector
o The value of each component is the number of times the
corresponding term occurs in the document
-> classification of documents is possible
Transaction data as a special type of record data
Each record (transaction) involves a set of items
o Eg: supermarket basket: each record is consisting of a set of
items (0-many), but also every record has a different amount
and combination of products
o Graph =way to represent complex information in a network
o Ordered = data that follows a specific order, where the order is essential for the
information
Eg: DNA sequence: Not only the amount of letters is important, but also the
places where the letters will appear
Data mining1
Non-trivial extraction of implicit, previously unknown and potentially useful information from
a dataset
o Automatic or semi-automatic means could discover meaningful patterns in large
quantities of data -> converting extracted information into useful knowledge
The data is already there, but an explicit answer to a question isn’t
o Eg: which names are more prevalent in certain locations, personalized book
recommendations, prioritize genes based on text co-occurrences
1
Data mining is the process of sorting through large data sets to identify patterns and relationships that can
help solve business problems through data analysis
4
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller lauraheyndrickx. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $16.73. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
56326 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now