1.introduction To Bigdata Chap1

DCCS208(02) Korea University 2019 Fall
Introduction
to Big Data
Chapter 1 & 2 (Week 1)
Course overview & introduction
Asst. Prof. Minseok Seo
mins@korea.ac.kr
Course Overview
Introduction to Big Data 01
Contents
1. Course Overview
 Brief introduction of professor & course
 Object & Aim of the course
 Assignments & Quiz
 Evaluation
2. Introduction to Big Data

 Definition of Big Data
 Key techniques in Data Science
 Core technology of Informatics
Course Overview
Course information
Introduction to Big Data, DCCS208(02), Fall 2019.
 Lecture time: Wed. (6,7) and Thu. (6)
 Location: Wed. (7-310) and Thu. (7-315)
 Completion division: Major elective subject
 Level: Junior / Senior
copyrightⓒ 2018 All rights reserved by Korea University 4 / 20

Course Overview
Definition of Big Data (Cont.)
VS.
Which is bigger, elephant or rat?

Course Overview
 What is Data?
Attributes (Dimension; Features; Variables)

Objects (Samples, Individuals)
ID Height Weight Age

Student 1 189 cm 81 kg 24
… … … …
Student N 162 cm 71 kg 21

Course Overview
 In a narrow sense, Big Data means only sample size.
 In a broad sense, Big Data represents both sample size and dimensionality.

Course Overview
 3V’s (Volume, Velocity, and Variety)

Course Overview
 5V’s (Volume, Velocity, Variety, Veracity, and Value)
 Volume: Data size

 Velocity: Data production speed
 Variety: Data oriented from various things
 Veracity: Data accuracy (Trustworthy)
 Value: Data value
Value*

Course Overview
Relationship between Big-data & Data Science
 The amount of data and information is not directly correlated with

knowledge generation.
 But the demand for data scientists will be growing.

Course Overview
Job market of Big data
Furht B., Villanustre F. (2016) Introduction to Big Data. In: Big Data Technologies and Applications. Springer, Cham
It is the time to prepare for an academic course to cultivate data analysts
commensurate with demand.

Course Overview
Object & Aim of the course
 Students who have taken this course expect to be able to learn:
Concept of
Big Data
Computational
Basic Skill in
approaches for
Data Science
Big Data
Introduction to
Big Data
Statistical
R
approaches for
programming
Big Data
Visualization
for Big Data

Course Overview
Course schedule (Before Mid-term exam)
Week Period Study Contents
1 09.02 - 09.08 Introduction to Big Data & Data Science
Overall workflow, Computer Software issues, and applications in the

2 09.09 - 09.15 Big Data era
3 09.16 - 09.22 Introduction to R programming
4 09.23 - 09.29 Descriptive & Fundamental Statistics
5 09.30 - 10.06 Understanding Data Structures (Types of random variable)
6 10.07 - 10.13 Data Visualization
7 10.14 - 10.20 Preprocessing of Big Data (Quality Control and Prescreening)
8 10.21 - 10.27 Mid-term Exam

Course Overview
Course schedule (After Mid-term exam)
Week Period Study Contents
9 10.28 - 11.03 Parallel and Distributed Processing for Big Data
10 11.04 - 11.10 Statistical Estimation & Modeling
11 11.11 - 11.17 Computational approach for statistical modeling with robustness
12 11.18 - 11.24 Clustering analysis (Unsupervised learning methods)
13 11.25 - 12.01 Classification analysis (Supervised learning methods)
14 11.02 - 12.08 Algorithms of Dimensionality Reduction for Big Data
15 12.09 - 12.15 Trends in various academic & industrial fields for application of Big Data
16 12.16 - 12.22 Final Exam

Course Overview
Two types of lectures per week
Wed. day Thu. Day

2hrs 1hr
Lecture for Theory Hands-on lecture
The methodology learned in theory class will be exercised in the computer lab. on Thursday.
 There are two representative computer language for Big data analysis, R and
Python.
 R will be used in this class.
 It is not required any prior knowledge of the R language because I plan to provide
example code for student's practice.
https://cran.r-project.org/

Course Overview
Exam, Quiz, and Homework
Midterm and Final exams

 There will be two exams.
 I will ask you to understand the basic computational/statistical algorithm.
Quiz
 There will be two simple quizzes in class to check the student's learning
progress of the course (before and after midterm respectively).
Homework
 There will be 4 times assignments.
 This will be a report on the theory and practice of data analysis learned in
class.

Course Overview
Evaluation plan
Midterm Final Quiz Assignment Attendance
10%
30%
20%
10%
30%
 Absolute grading system

Score ≥ 95, you will get A+
Score ≥ 90, you will get A
Score ≥ 85, you will get B+
and...

Course Overview
Textbook
 No Textbook
 This course will be proceed based on the presentation slide
 I will upload presentation slide in Blackboard & my homepage

Homepage: https://scholar.harvard.edu/msseo
Teaching >> Introduction to Big Data >> Related Materials
 Reference 1 (Kor. Version)

R for Practical Data Analysis
(online textbook and free)
http://r4pda.co.kr/pdf/r4pda_2014_03_02.pdf
 Reference 2 (Eng. Version)
Introduction to Data Science by Rafael A. Irizarry, 2019.
https://rafalab.github.io/dsbook/
 Reference 3 (Eng. Version)
R for Data Science by Garrett Grolemund.
https://r4ds.had.co.nz/

Course Overview
Contact information
 Prof. Minseok Seo

Location: 7-203
Tel: 044-860-1379
Email: mins@korea.ac.kr
 TA. Heechan Chae

Location: 7-328
Email: chay219@korea.ac.kr
 If you have any questions about the course please email me and I will reply as
soon as I see it.
 If you need to meet in person, please make an appointment by email first.
 I will be available at Mon: 12:00 - 17:00 | Wed: 10:00 - 13:00 | Thu: 10:00 - 13:00.

End of
Orientation
Contents
1. Course Overview
 Brief introduction of professor & course
 Object & Aim of the course
 Assignments & Quiz
 Evaluation
2. Introduction to Big Data

 Concept of Big Data
 Key techniques in Data Science for Big data
Characteristics of Big Data
Remind concept of Big Data
 5V’s (Volume, Velocity, Variety, Veracity, and Value)
 Volume: Data size

 Velocity: Data production speed
 Variety: Data oriented from various things
 Veracity: Data accuracy (Trustworthy)
 Value: Data value
Value*

Petabyte era
1 PB = 1000000000000000B = 1015bytes = 1000terabytes
1000 PB = 1 exabyte (EB)
 transferred about 197 PB of data thorough its network each data (2018)
 processed about 24 petabytes daily (2009)
In fact, we can say that we have already entered the exabyte

era.

How do you recognize if it's big data or not?
Computer Scientist
My computer is low on memory for

handling this data!!
That is Big Data
No!!!! This data is over 2TB. Where do I

store it?????
That is Big Data
In short, if you’re having trouble with data processing on your computer (멘붕에
빠지면), it will be due to the Big Data.

How do you recognize if it's big data or not?
Statistician
When does this calculation end? I was

only waiting for 10 years ...
Dimensionality is too high!!!! I can’t build

statistical model using this data!!!
That is Big Data
In short, if you’re having trouble with data analysis on your computer (멘붕에 빠지
면), it will be due to the Big Data.

Core technologies of Big Data era
IT technologies to resolve issue derived from the Big data
Software Hardware
Prescreening techniques
Data Visualization
Feature selection
Parallel processing
Clouding computing
Distributed processing
Difficulties arise in both hardware and software.
But students can approach software difficulties.

Computational language for Big Data
R and Python
Wed. day Thu. Day

2hrs 1hr
Lecture for Theory Hands-on lecture
 There are two representative computer language for Big data analysis, R and
Python.
 R programming language (free and relatively easy) for hands-on lecture.
 Let’s connect R homepage
https://cran.r-project.org/

Install R
(Step 1) Download the R installer

Install R
(Step 2) Download the RStudio
 Download Rstudio from https://www.rstudio.com/products/rstudio/download/

Install R
(Step 3) Install R and Rstudio

What is R
 R is an interpreted computer language.
 It is possible to interface procedures written in C, C+, and etc., languages for

efficiency.
 System commands can be called from within R
 R is used for data manipulation, statistics, and graphics.

R, S, and S-plus (History of R)
 S: an interactive environment for data analysis developed at Bell Laboratories since
1976
1988 - S2: RA Becker, JM Chambers, A Wilks
1992 - S3: JM Chambers, TJ Hastie
1998 - S4: JM Chambers
 Exclusively licensed by AT&T/Lucent to Insightful Corporation, Seattle WA. Product

name: “S-plus”.
Implementation languages C, Fortran.
 R: initially written by Ross Ihaka and Robert Gentleman at Dep. of Statistics of U of

Auckland, New Zealand during 1990s.
 Since 1997: international “R-core” team of ca. 15 people with access to common
CVS archive.

What R does and does not
 Possible
(1) data handling and storage: numeric, textual
(2) matrix algebra
(3) has tables and regular expressions
(4) high-level data analytic and statistical functions
(5) OOP (classes)
(6) Graphic
(7) Programming language: loops, branching, subroutines, and etc.,
 Impossible
(1) R is not a database, but connects to DBMSs
(2) R has no GUI, but connect to Java, TclTk
(3) R is fundamentally very slow, but allows to call own C/C++ code
(4) R is no spreadsheet view of data, but connects to Excel/MsOffice
(5) R is no professional & commercial support
 But all R users in the world are developers (Power of Collective intelligence; 집단지성).
 If you make a meaningful package at any time, you can publish it within 1 second.
 Therefore, applying latest algorithms are faster than any programming language.

Install R
(Step 3) Install R and Rstudio

End of Slide

1.introduction To Bigdata Chap1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1.introduction To Bigdata Chap1

Uploaded by

Copyright:

Available Formats

DCCS208(02) Korea University 2019 Fall

2. Introduction to Big Data

Introduction to Big Data, DCCS208(02), Fall 2019.

 Lecture time: Wed. (6,7) and Thu. (6)

 Location: Wed. (7-310) and Thu. (7-315)

 Completion division: Major elective subject

 Level: Junior / Senior

copyrightⓒ 2018 All rights reserved by Korea University 4 / 20

Which is bigger, elephant or rat?

copyrightⓒ 2018 All rights reserved by Korea University 5 / 20

Attributes (Dimension; Features; Variables)

ID Height Weight Age

copyrightⓒ 2018 All rights reserved by Korea University 6 / 20

 In a narrow sense, Big Data means only sample size.

copyrightⓒ 2018 All rights reserved by Korea University 7 / 20

 3V’s (Volume, Velocity, and Variety)

copyrightⓒ 2018 All rights reserved by Korea University 8 / 20

 5V’s (Volume, Velocity, Variety, Veracity, and Value)

 Volume: Data size

copyrightⓒ 2018 All rights reserved by Korea University 9 / 20

 The amount of data and information is not directly correlated with

 But the demand for data scientists will be growing.

copyrightⓒ 2018 All rights reserved by Korea University 10 / 20

copyrightⓒ 2018 All rights reserved by Korea University 11 / 20

 Students who have taken this course expect to be able to learn:

copyrightⓒ 2018 All rights reserved by Korea University 12 / 20

Week Period Study Contents

1 09.02 - 09.08 Introduction to Big Data & Data Science

Overall workflow, Computer Software issues, and applications in the

3 09.16 - 09.22 Introduction to R programming

4 09.23 - 09.29 Descriptive & Fundamental Statistics

5 09.30 - 10.06 Understanding Data Structures (Types of random variable)

6 10.07 - 10.13 Data Visualization

7 10.14 - 10.20 Preprocessing of Big Data (Quality Control and Prescreening)

8 10.21 - 10.27 Mid-term Exam

copyrightⓒ 2018 All rights reserved by Korea University 13 / 20

Week Period Study Contents

9 10.28 - 11.03 Parallel and Distributed Processing for Big Data

10 11.04 - 11.10 Statistical Estimation & Modeling

11 11.11 - 11.17 Computational approach for statistical modeling with robustness

12 11.18 - 11.24 Clustering analysis (Unsupervised learning methods)

13 11.25 - 12.01 Classification analysis (Supervised learning methods)

14 11.02 - 12.08 Algorithms of Dimensionality Reduction for Big Data

16 12.16 - 12.22 Final Exam

copyrightⓒ 2018 All rights reserved by Korea University 14 / 20

Wed. day Thu. Day

 R will be used in this class.

copyrightⓒ 2018 All rights reserved by Korea University 15 / 20

Midterm and Final exams

 I will ask you to understand the basic computational/statistical algorithm.

copyrightⓒ 2018 All rights reserved by Korea University 16 / 20

Midterm Final Quiz Assignment Attendance

 Absolute grading system

copyrightⓒ 2018 All rights reserved by Korea University 17 / 20

 This course will be proceed based on the presentation slide

 I will upload presentation slide in Blackboard & my homepage

 Reference 1 (Kor. Version)

copyrightⓒ 2018 All rights reserved by Korea University 18 / 20

 Prof. Minseok Seo

 TA. Heechan Chae

 If you need to meet in person, please make an appointment by email first.

copyrightⓒ 2018 All rights reserved by Korea University 19 / 20