Summary
Data Science and Society Midterm Summary
- Course
- Institution
These notes are a summary of all the lectures, literature and assignments needed for the UU Data Science and Society midterm exam.
[Show more]
Seller
Followsamoyediran4
Reviews received
Content preview
Data Science & SocietyNotes of the Lectures, Assignments, and Literature
Not representative of all the matter
I left out material I considered obviou s knowledge
Some parts are copied from other sum maries
Skipped some lecture material since it ’s already covered in Literature
Content
Week 1 – Stair Reynolds – IS ......................................................................................................... 2
Lecture 1 – Catching up with SQL .................................................................................................. 2
Assignment 1: Bash Fundamentals ............................................................................................... 3
Lecture 2 – Applied Data Science for Student Empowerment ..................................................... 4
Lecture 3 – The Knowledge Discovery Process for Societal Impact ........................................... 5
Week 2 – Chapman – CRISP-DM 1.0 ............................................................................................. 6
Week 2 – Davenport – Data Scientist: Sexiest Job of 21st Century .............................................. 9
Week 2 – Chang Grade – NIST Big Data Interoperability Framework ........................................ 10
Week 2 – Spruit, Lytras – Applied Data Science.......................................................................... 14
Week 2 – Braschler – ADS............................................................................................................ 16
Assignment 2 – Methods & Statistics in R .................................................................................. 19
Lecture 4 – Hadoop & MapReduce .............................................................................................. 20
Lecture 5 – Methodology, Statistics and Pitfalls ......................................................................... 23
Week 3 – Lazer – Google Flu: Big Data Traps ............................................................................. 25
Week 3 – Broniatowski, Lazer – Twitter: Big Data Opportunities ............................................... 27
Week 4 – Dean, Ghemawat – MapReduce .................................................................................. 28
Week 4 – Chambers, Zaharia – Spark Guide [Chapters 1-3] ...................................................... 31
Week 4 – Ambrose – Big Data in historical perspective ............................................................. 35
Assignment 3 – MapReduce in Hadoop & Spark ........................................................................ 42
Lecture 6 - NoSQL, Spark & Big Data ............................................................................................ 43
Lecture 7 – Statistics 2 ................................................................................................................. 48
1
,Week 1 – Stair Reynolds – IS
Quite basic concepts of Information Systems. Recommend skimming through the pages and
read the bold definitions and meaning.
Lecture 1 – Catching up with SQL
• Some definitions
o Create: Creation of database objects
o Alter: Modify the structure and/or the characteristics of database objects
o Drop: Deletion of database objects
o Truncate: Deletion of data in tables without altering the structure
• The core parts of SQL:
o Data definition language (DDL): Used to define database structures
▪ CREATE TABLE, DROP TABLE
o Data manipulation language (DML): define, update and request data (queries)
▪ INSERT (add row), UPDATE (modify values in existing row/collection of
rows), DELETE (delete a row/collection of rows), SELECT (select rows),
DISTINCT (addition to select, to prevent duplicate rows are shown),
WHERE (addition for criteria records should meet), AND/OR/NOT
(addition for multiple matching criteria), BETWEEN (self-explanatory),
ORDER BY (to sort results), GROUP BY (for subtotals), HAVING (to limit
how much data is shown)
▪ Built-in SQL functions
• COUNT: the number of rows that match the criteria
• MIN: minimal value in certain column
• MAX: maximum value in certain column
• SUM: of values in certain column
• AVG: average of values in certain column
▪ A query retrieves data from one or more tables and creates a new
(temporary) table
▪ Subqueries are queries that are used as input for another query
2
,Assignment 1: Bash Fundamentals
Command Action
cat Displays output of a file
cd Change directory
cd .. Go up one directory
chmod Change permission to read, write and execute (000 = none, 777 = all)
cp Copy file to given directory
echo Returns given value (functionality of the echo commands can vary) (‘-e’ is used
if you use escapes in your string, such as ‘\n’)
grep Filters a given input
mkdir Create a directory
mv Move file to another directory
ls Returns names of files and directories in your current directory (use ‘-l’ for
additional information)
paste Merge two output streams by column (use ‘>’ to save it in a new file)
pwd Return full path of current directory
rm Remove a file (use ‘rm -r’ for an entire directory including all content)
sort Sort an input, ‘-r’ to reverse and ‘-n’ for numeric
| Use output of command before ‘|’ as input for the command after ‘|’
> Get the output of the command and write it to a file
>> Get the output of a command and add it to a file
* Placeholder for every character or sequence
~ Home directory
# Indicated comment (everything on the same line after ‘#’ is ignored)
\n \t Character for new line and a tab
Bash scripts
You can create a bash-script to bundle a number of commands. This is just a text-file with a
.sh extension. You can execute the script by typing the path to the script in the command line.
3
, Lecture 2 – Applied Data Science for Student Empowerment
Applied Data Science is where Analytical Applications are combined with Data Science
Data Science:
- Theoretical
- Algorithms
Applied Data Science:
- Solution-oriented
- Meta-Algorithmic Models
Citizen Data Science:
- Applied
- Automated Software Tools
Self-Service Capability: “To empower non-
data scientist with automated software
tools and meta-algorithmic models to self-
service their own data analyses on their
own data sources in a reliable, usable, and
transparent manner.
4
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller samoyediran4. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.49. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
50843 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now