Big data and effective

distributed computing
W h a t is Bigdata?

3 V Definition

Volume Velocity Variety

How much data generated in 1-Minite
➢4,166,667 post on facebook

➢3,47,222 tweets on twitter

➢1,736,111 post on instagram

➢300hr of video on youtube

➢18,327 cast the vote on REDDIT

W h a t a r e T y p e s o f D a t a?
S t r u c t u r e d Data:

➢ There are 3 AspectsDefined:

➢ Name
➢ Type
➢ Length
First_name Hire_date Salary
Gopichand 36/02/2013 $6000
Rahul 28/01/2013 10000

➢ Example: Hire_date date

➢ Example: Salary number(6,0)
➢ e.g. Oracle, SQL Server, DB2, Sybase, Teradata, MySQL (All RDBMS’s)
S e m i - S t r u c t u r e d Data:
➢Is this valid in Excel?
First_name Hire_date Salary
Gopichand 36/02/2013 $6000
Rahul 28/01/2013 10000

➢Is this valid in XML?

<row>
<name>Gopichand</name>
<hire_date>36/02/2013</ire_date> Example: Excel, XML,JSON, csv
<salary>$6000</salary>
</row>
<row>
<name>Rahul</name>
<hire_date>28/01/2013</hire_date>
<salary>10000</salary>
</row>
U n s t r u c t u r e d Data

➢ Free Flowing Text(Plain Text)

➢ Tweets on Twitter
➢ Messages on Whatsapp
➢ Emails on gmail, ymail etc.
➢ Word doc, pdfs, Comments on Facebook Posts
➢ Videos on Youtube
➢ Log files from Webservers, log files from app
servers, audit logs etc.
Can this be Processed by the Legacy
F r a m e w o r k s Av a i l a b l e ?
➢ Like ETLtools (ab Initio, Informatica,
Datastage etc.)
➢ Like RDBMS’s (Oracle, SQL Server, DB2,
Sybase, Teradata, MySQL)
➢ Mainframes(IBM mainframe, AS400 etc.)

Problem is: Client Server Architecture

How do Legacy Frameworks Work?

1. Client writes a select query

Select * 2. Sends to server

From emp
Where salary > 5000; Server
4. Sends the result to
client

3. Server processes data

W h a t is Scaling Problem?

Vertical Scaling Horizontal Scaling

 W h a t H a p p e ns to these Databases w h e n
Data increases?

➢Select Queries would run very slow….This

increases time taken for reports to be
generated
➢Buffers/Cache Overflow can lead to a server
crash
Bigdata i n Telecom
Data:
➢ Incoming Calls Incoming Calls
➢ Outgoing Calls
Pimple
➢ Data Usage
Saudagar
➢ SMS
➢ VAS
➢ Call Detail Records
Deccan Kharadi
Think About!
➢ Total Activity
➢ Millions of Users
➢ Huge number of CDR Logs Generated
➢ Million Logs per Tower
➢ So many Towers in a City Hinjawadi Nal Stop
➢ Multiple Towers in the Country
Bigdata
Bigdata in Banking

Cheque Loans: Swipes: Social Media Spending

Net Banking ➢ Home Loans ➢ Credit Card Patterns
➢ Car Loans ➢ Debit Cards
➢ POS
➢ Gold Loans

Millions of Users…
➢ Investment Banking ➢ Mutual Funds
Millions of Logs…
➢ Retail Banking ➢ SIP
Millions of Rows…
➢ Corporate Banking ➢ EMIs
Gigabytes and Terabytes of Data
Bigdata in E-commerce

➢ Advertising and SEM(Search Engine Marketing)

Millions of Customers…
Millions of Clicks…

Clicks x Cost Per Click (CPC)

1 Click Rs. 50-500/-
Crores of Money

Click Based Logs

Millions and Millions of
Clickstream Logs…
Bigdata in Healthcare

Billing Medicines

Insurance Test
➢ Operator Information
Claims Reports
➢ Doctor/Nurse Information
➢ Medical Records
➢ Drug Information Room Old
Booking
➢ Emergency Work Log Reports
➢ In-Patient Record
➢ Out-Patient Record New
Hospital
Staff Reports
Every Hour…
Every Day…
Every Branch…
Every Hospital…

Gigabytes and Terabytes of Data

Android Apps

➢ When you install an Android

App, you authorize them to
read your messages, mails,
phonebook, and access your
media etc.
Te m p l e R u n a n d A d v e r t i s e m e n t

Advertisements shown here

 H o w d o we solve t h i s p r o b l e m ?
Distributed Processing

Select Host
Terminal Terminal Terminal Terminal
Query Computer

Host Terminal Terminal Terminal Terminal

Computer

Code Host
Written Terminal Terminal Terminal Terminal
Computer
at one
place

Unlimited Scaling!!
Hadoop
History of Hodoop

2003 :- Google launch one file system for storing data. GFS (Google
File System)

2004 :- Google launch one software framework for processing data

called MapReduce.

2004:- Google publish one white paper on GFS & MapReduce

Yahoo work on white paper, publish by google and come out on

conclusion i.e. Hadoop
Who is the inventor of hadoop

Doug Cutting is inventor of hadoop

Doug Cutting small kids playing with toy elephant and the toy name is hadoop, so
this name was given to technology called as “hadoop