Summary
JADS Master - Data Engineering Summary
- Course
- Institution
Summary for the Data Engineering course of the Master Data Science and Entrepreneurship.
[Show more]
Seller
Follow
tomdewildt
Reviews received
Content preview
1. IntroductionData Engineer
Develops the architecture that helps analyse and process data in the way the organization
needs it.
Data Science Lifecycle
Big Data
Term for a collection of datasets so large and complex that it becomes difficult to process using
traditional data processing applications.
Structured Data Semi-Structured Data Unstructured Data
RDMS XML, RDF, JSON, etc. Video, Images, Text, etc.
V3 Model
● Volume: enterprises are always growing in terms of data
● Velocity: sometimes 2 minutes is too late, data must be used as streams in order to
maximize its value
● Variety: structured and unstructured data, new insights are found when analyzing these
data types together
Data Pipeline
Aggregates, organizes and moves data for storage, insights and analysis.
1
, ML Ops
● Software engineering approach: enables team to efficiently produce high quality software
● Cross-functional team: experts with different skill sets and workflows (DE, DS, ML, Dev,
Ops)
● Producing software based on code, data and models: all artifacts of ML software
production process require different tools and workflows → must be versioned and
managed accordingly
● Small and safe increments: the release of software artifacts is divided into small
increments → allows visibility and control around the levels of variance of its outcomes
● Reproducible and reliable software release: model outputs are non-deterministic →
process of releasing ML software is reliable and reproducible → leverage automation as
much as possible
● Software release at any time: ML software needs to be delivered into production at any
time → when to release is a business decision rather than a technical decision
● Short adaptation cycles: short cycles mean development cycles are in the order of days
or hours
2. Cloud Computing & Virtualization
Cloud
A type of distributed system consisting of interconnected and virtualized computers dynamically
provisioned and presented as one (or more) unified computing resource(s) based on
service-level agreements established through negotiation between service providers and
consumers.
● Cloud contains “your” data
● Cloud computes “your” data
● Cloud hosts “your” data intensive applications
▶ Central ideas:
● Utility computing over data
● SOA (Service Oriented Architectures)
● SLA (Service Level Agreements)
2
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller tomdewildt. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $5.89. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
67474 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now