Big Data

Extremely large data sets

Dilip Kumar
Assistant Professor
Dilip Kumar Computer Science & Engineering
1
+919454018844
 Syllabus
Unit-1

Dilip Kumar
2
+919454018844
 Evolution of Technology

Dilip Kumar
3
+919454018844
 IOT Devices

Dilip Kumar
4
+919454018844
 Social Media

Dilip Kumar
5
+919454018844
 What is Big Data?

Dilip Kumar
6
+919454018844
 What is Big Data?
• Big data is term used to describe data that is too large and complex to store in traditional
databases or process it efficiently.
• Big Data is difficult to store, collect, maintain, analyze, and visualize.
• Big Data is a collection of data that is huge in volume, yet growing exponentially with time.

Dilip Kumar
7
+919454018844
 What is Big Data?
• It became difficult to manage and process the data using traditional data processing tool(s).
• Big data is the term for a collection of data sets which are so large and complex that it
becomes difficult to process using on-hand database management tools or traditional data
processing application.

Dilip Kumar
8
+919454018844
 Data
• Data is numbers, alphabets or special symbols.
• The quantities, characters, or symbols on which operations are performed by a computer,
which may be stored and transmitted in the form of electrical signals and recorded on
magnetic, optical, or mechanical recording media.
• Data growth has seen exponential acceleration since the advent of the computer and Internet.
• In fact, the computer and Internet duo has imparted the digital form to data.
Information: is a processed data.

Dilip Kumar
9
+919454018844
 Digital Data
• Digital data is information stored on a computer system as a series of 0’s and 1’s in a binary
language.

Example: Whenever we send an email, read a social media post, or take pictures with our digital
camera, we are working with digital data.

Dilip Kumar
10
+919454018844
 Types of Digital Data
• Digital data can be classified into three forms:
1. Unstructured
2. Semi-structured
3. Structured

Dilip Kumar
11
+919454018844
 Types of Digital Data
• Digital data can be classified into three forms:
1. Unstructured
2. Semi-structured
3. Structured

Digital Data

Unstructured Semi-structured Structured

Dilip Kumar
12
+919454018844
 1. Un-structured Data
• Any data with unknown form or the structure is classified as unstructured data.
• The data which does not confirm to a data model or is not in a form that can be used easily by
a computer program is categorized as unstructured data.
• About 80—90% data of an organization is in this format.
Example: Images, videos, letters, researches, white papers, the body of an email, Memos, chat
rooms, PowerPoint presentations etc.

Dilip Kumar
13
+919454018844
 1. Un-structured Data
• Unstructured data is the kind of data that doesn’t adhere to any definite schema or set of rules.
Its arrangement is unplanned and haphazard.
• Photos, videos, text documents, and log files can be generally considered unstructured data.
• Even though the metadata accompanying an image or a video may be semi-structured, the
actual data being dealt with is unstructured.
• Additionally, Unstructured data is also known as “dark data” because it cannot be analyzed
without the proper software tools.

Dilip Kumar
14
+919454018844
 1. Un-structured Data
• Unstructured data is information that either does not organize in a pre-defined manner or not
have a pre-defined data model.
• Unstructured information is a set of text-heavy but may contain data such as numbers, dates,
and facts as well.
• Videos, audio, and binary data files might not have a specific structure. They’re assigned to
as unstructured data.

Dilip Kumar
15
+919454018844
 2. Structured Data
• Structured data is generally tabular data that is represented by columns and rows in a
database. Databases that hold tables in this form are called relational databases.
• Any data that can be stored, accessed and processed in the form of fixed format is termed as a
'structured data'.
• Structured data is easy to store, manage, and analyze. All of the data follows the same format.
• Relationships exist between entities of data, such as classes and their objects.
Example: Data stored in databases.

Dilip Kumar
16
+919454018844
 2. Structured Data
• Structured data can be defined as the data that resides in a fixed field within a record.
• It is type of data most familiar to our everyday lives. for ex: birthday, address.
• A certain schema binds it, so all the data has the same set of properties.
• Structured data is also called relational data.
• It is split into multiple tables to enhance the integrity of the data by creating a single record to
depict an entity. Relationships are enforced by the application of table constraints.
• The business value of structured data lies within how well an organization can utilize its
existing systems and processes for analysis purposes.
• A Structured Query Language (SQL) programming language used for structured data.

Dilip Kumar
17
+919454018844
 2. Structured Data
• Examples of structured data include numbers, dates, strings, etc. The business data of an e-
commerce website can be considered to be structured data.

Name Class Section Roll No Grade

Ajay 11 A 1 A

Vijay 11 A 2 B

Ramesh 11 A 3 A

Dilip Kumar
18
+919454018844
 3. Semi-structured Data
• Semi-structured data is information that doesn’t consist of Structured data (relational
database) but still has some structure to it.
• The data is not in the relational format and is not neatly organized into rows and columns like
that in a spreadsheet.
• However, there are some features like key-value pairs that help in discerning the different
entities from each other.

• Semi-structured data consist of documents held in JavaScript Object Notation (JSON)

format.

• It also includes key-value stores and graph databases.

Dilip Kumar
19
+919454018844
 3. Semi-structured Data
• Semi-structured data is information that does not reside in a relational database or any other
data table.
• Semi-structured data is not bound by any rigid schema for data storage and handling.

Dilip Kumar
20
+919454018844
 3. Semi-structured Data
• Since semi-structured data doesn’t need a structured query language, it is commonly
called NoSQL data.
• The data which does not confirm to a data model but has some structure is categorized as semi-
structured data.
• However, it is not in a form that can be used easily by a computer program.

Dilip Kumar
21
+919454018844
 3. Semi-structured Data
• Semi-structured content is often used to store metadata about a business process but it can also
include files containing machine instructions for computer programs.
• This type of information typically comes from external sources such as social media platforms
or other web-based data feeds.
Example: Emails, XML, markup languages like HTML, etc. Metadata for this data is available
but is not sufficient.

Dilip Kumar
22
+919454018844
 Structure Vs Unstructured

Dilip Kumar
23
+919454018844
Structured Vs Semi-Structured Vs Unstructured
Differences between Structured, Semi-structured and Unstructured data
Properties Structured data Semi-structured data Unstructured data

Technology It is based on Relational It is based on It is based on character

database table XML/RDF(Resource and binary data
Description
Framework).

Transaction Matured transaction and Transaction is adapted No transaction

management various concurrency from DBMS not management and no
techniques matured concurrency

Version management Versioning over tuples, Versioning over tuples Versioned as a whole
row, tables or graph is possible

Flexibility It is schema dependent It is more flexible than It is more flexible and

and less flexible structured data but less there is absence of
flexible than schema
unstructured data
Dilip Kumar
24
+919454018844
Structured Vs Semi-Structured Vs Unstructured
Differences between Structured, Semi-structured and Unstructured data
Properties Structured data Semi-structured data Unstructured data

Scalability It is very difficult to It’s scaling is simpler It is more scalable.

scale DB schema than structured data

Robustness Very robust New technology, not —

very spread

Query performance Structured query allow Queries over Only textual queries are
complex joining anonymous nodes are possible
possible

Dilip Kumar
25
+919454018844
 Characteristics of Big-Data
Also known as 5Vs of Big Data
Volume: Means large amount of data.
Velocity: Means the rate at which data is generated.
Variety: Means the type of data.
Value: Raw data are processed and we get meaningful data.
Veracity: Consistency of Data generates

Dilip Kumar
26
+919454018844
 Characteristics of Big-Data
1. Volume:
• Incredible amount of data.
• The amount of data which generated every seconds.
• Volume is a factor which define data is big data or not.
• Big data technologies can handle large amounts of data.
• Big Data is a vast “volumes” of data generated from many sources daily, such as business
processes, machines, social media platforms, networks, human interactions, and so on.
Example: From cell phones, social media, online transaction etc.
Facebook generates approximately a billion messages, 4.5 billion times the “Like” button is
recorded,
Dilip Kumarand more than 350 million new posts are uploaded each day.
27
+919454018844
 Characteristics of Big-Data
2. Velocity:
• Speed at which data is
• Generated
• Collected
• Analyzed

• Velocity refers to the speed or rate by which data is generated in real-time.

• Velocity plays an important role compared to others.
• It contains the linking of incoming data sets speeds, rate of change, and activity bursts.
• The primary aspect of Big Data is to provide demanding data rapidly.
Example of data that is generated with high velocity - Twitter messages or Facebook posts.
Dilip Kumar
28
+919454018844
 Characteristics of Big-Data
3. Variety:
• Big Data can be structured, unstructured, and semi-structured that
are being collected from different sources.
• 80% unstructured data.
• Data were only collected from databases and sheets in the past, But these days the data will
come in an array of forms ie.- PDFs, Emails, audios, Social Media posts, photos, videos, etc.
• Types of data are:
1. Structured
2. Semi-Structured
3. Un-Structured
Dilip Kumar
29
+919454018844
 Characteristics of Big-Data
4. Value:
• Raw data are processed and we get meaningful data.
• Value is an essential characteristic of big data.
• It is not the data that we process or store, it is valuable and reliable data that we store, process
and analyse.

Dilip Kumar
30
+919454018844
 Characteristics of Big-Data
5. Veracity
• Trust worthiness of data.
• Veracity refers to the quality of the data that is being analysed.
• It is the process of being able to handle and manage data efficiently.
Example: Facebook posts with hashtags.

Dilip Kumar
31
+919454018844
 Characteristics of Big-Data
9 Vs of Big Data
1. Volume
2. Velocity
3. Variety
4. Value
5. Veracity
6. Validity
7. Variability
8. Volatility
9. Visualization

Dilip Kumar
32
+919454018844
 Sources of Big-Data
• E- Commerce Website

• Social Media

• IOT devices

• Banks

• Stock Market

Dilip Kumar
33
+919454018844
 Uses of Big-Data
• Prediction systems

• Recommendation Engine

• Fraud Detection

• Sentimental Analysis Etc.

Dilip Kumar
34
+919454018844
 Application or Uses of Big-Data
• In today’s world big data have several applications, some of them are listed below:
1. Tracking Customer Spending Habit, Shopping Behaviour
2. Recommendation
3. Smart Traffic System
4. Secure Air Traffic System
5. Auto Driving Car
6. Virtual Personal Assistant Tool
7. IOT
8. Education Sector Energy Sector

Dilip9.Kumar
Media and Entertainment Sector
35
+919454018844
 Application or Uses of Big-Data
Tracking Customer Spending Habit, Shopping Behaviour:
In big retails stores, the management team has to keep data of customer’s spending habits,
shopping behaviour, most liked product, which product is being searched/sold most, based on that
data, the production/collection rate of that product gets fixed.

Dilip Kumar
36
+919454018844
 Application or Uses of Big-Data
Recommendation:
By tracking customer spending habits, shopping behaviour, big retail stores provide
recommendations to the customers.

Dilip Kumar
37
+919454018844
 Application or Uses of Big-Data
Smart Traffic System:
Data about the condition of the traffic of different roads, collected through cameras, GPS devices
placed in the vehicle.
All such data are analysed and jam-free or less jam way, less time taking ways are recommended.
One more profit is fuel consumption can be reduced.

Dilip Kumar
38
+919454018844
 Application or Uses of Big-Data
Secure Air Traffic System:
At various places of flight, sensors are present.
These sensors capture data like the speed of flight, moisture, temperature, and other
environmental conditions.

Dilip Kumar
39
+919454018844
 Application or Uses of Big-Data
Secure Air Traffic System:
Based on such data analysis, an environmental parameter within flight is set up and varied.
By analysing flight’s machine-generated data, it can be estimated how long the machine can
operate flawlessly and when it can be replaced/repaired.

Dilip Kumar
40
+919454018844
 Application or Uses of Big-Data
Auto Driving Car:
In the various spots of the car camera, a sensor is placed that gathers data like the size of the
surrounding car, obstacle, distance from those, etc.
These data are being analysed, then various calculations are carried out.
These calculations help to take action automatically.

Dilip Kumar
41
+919454018844
 Application or Uses of Big-Data
Virtual Personal Assistant Tool:
Big data analysis helps virtual personal assistant tools like Siri, Cortana and Google Assistant to
provide the answer to the various questions asked by users.
This tool tracks the location of the user, their local time, season, other data related to questions
asked, etc.
Analysing all such data provides an answer.

Dilip Kumar
42
+919454018844
 Application or Uses of Big-Data
Virtual Personal Assistant Tool :
Example: Suppose one user asks “Do I need to take Umbrella?”The tool collects data like
location of the user, season and weather condition at that location, then analyses these data to
conclude if there is a chance of raining, then provides the answer.

Dilip Kumar
43
+919454018844
 Application or Uses of Big-Data
IOT :
Manufacturing companies install IOT sensors into machines to collect operational data.
Analysing such data, it can be predicted how long a machine will work without any problem
when it requires repair.
Thus, the cost to replace the whole machine can be saved.

Dilip Kumar
44
+919454018844
 Application or Uses of Big-Data
Education Sector Energy Sector:
Online educational courses conducting organization utilize big data to search candidates
interested in that course.
If someone searches for a YouTube tutorial video on a subject, then an online or offline course
provider organization on that subject sends an ad online to that person about their course.

Dilip Kumar
45
+919454018844
 Application or Uses of Big-Data
Media and Entertainment Sector:
Media and entertainment service providing company like Netflix, Amazon Prime, Spotify do
analysis on data collected from their users.
Data like what type of video, music users are watching, listening to most, how long users are
spending on site, etc. are collected and analysed to set the next business strategy.

Dilip Kumar
46
+919454018844
Big Data importance and applications
• Cost Saving
• Time Reductions
• Understand the Market condition
• Social Media Listening
• Provide better customer service
• Solve advertisers problems
• Create personalized marketing , can increase revenue and profits
• Fast decision making
• Predicting future needs

Dilip Kumar
47
+919454018844
Big Data importance and applications
Big Data Importance:
Big Data importance doesn't revolve around the amount of data a company has but lies in the fact
that how the company utilizes the gathered data.
Every company uses its collected data in its own way. More effectively the company uses its data,
more rapidly it grows.
By analysing the big data pools effectively the companies can get answers to:
Cost Savings:
• Some tools of Big Data like Hadoop can bring cost advantages to business when large amounts of data are to
be stored.
• These tools help in identifying more efficient ways of doing business.
Dilip Kumar
48
+919454018844
Big Data importance and applications
Big Data Importance:
Time Reductions:
• The high speed of tools like Hadoop and in-memory analytics can easily identify new sources of data which
helps businesses analysing data immediately.
• This helps us to make quick decisions based on the learnings.
Understand the market conditions:
• by analysing big data we can get a better understanding of current market conditions.
for example: By analysing customers’ purchasing behaviours, a company can find out the products that are sold
the most and produce products according to this trend. By this, it can get ahead of its competitors.

Dilip Kumar
49
+919454018844
Big Data importance and applications
Big Data Importance:
Control online reputation:
• Big data tools can do sentiment analysis.
• Therefore, you can get feedback about who is saying what about your company.
• If you want to monitor and improve the online presence of your business, then big data tools can help in all
this.

Dilip Kumar
50
+919454018844
Big Data importance and applications
Big Data Importance:
Using Big Data Analytics to Boost Customer Acquisition(purchase) and Retention:
• The customer is the most important asset any business depends on.
• No single business can claim success without first having to establish a solid customer base.
• If a business is slow to learn what customers are looking for, then it is very likely to deliver poor quality
products.
• The use of big data allows businesses to observe various customer-related patterns and trends.
Using Big Data Analytics to Solve Advertisers Problem and Offer Marketing

Insights:
• Big data analytics can help change all business operations.
• Like the ability to match customer expectations, changing company’s product line, etc.
Dilip• Kumar
And ensuring that the marketing campaigns are powerful.
51
+919454018844
 Challenges of Big-Data Analytics
• Complex to Store and Manage
• Complex to Analysis
• Integrating data from a variety of sources
• Low Quality and Inaccurate Data
• Hardware failure
• Searching
• Sharing
• Transfer
• Presentation or visualization

Dilip Kumar
52
+919454018844
 Challenges of Conventional Systems
• Big data is the storage and analysis of large data sets.
• They are so large that it is not possible to work on them with traditional analytical tools.
• These are complex data sets that can be both structured or unstructured.
• One of the major challenges of conventional systems was the uncertainty of the Data
Management Landscape.
• Big data is continuously expanding, there are new companies and technologies that are being
developed every day.

Dilip Kumar
53
+919454018844
 Challenges of Conventional Systems
• A big challenge for companies is to find out which technology works bests for them without
the introduction of new risks and problems.
• These days, organizations are realising the value they get out of big data analytics and hence
they are deploying big data tools and processes to bring more efficiency in their work
environment.

Dilip Kumar
54
+919454018844
 Big Data Architecture
• Big data architecture is designed to handle the ingestion, processing, and analysis of data that
is too large or complex for traditional database systems.

Dilip Kumar
55
+919454018844
 Big Data Architecture
The big data architectures include the following components:
Data sources: All big data solutions start with one or more data sources.
Example
• Application data stores, such as relational databases.
• Static files produced by applications, such as web server log files.
• Real-time data sources, such as IoT devices.
Data storage: Data for batch processing operations is stored in a distributed file store that can
hold high volumes of large files in various formats (also called data lake).
Example: Azure Data Lake Store or blob containers in Azure Storage.

Dilip Kumar
56
+919454018844
 Big Data Architecture
Batch processing: Since the data sets are so large, therefore a big data solution must process data
files using long-running batch jobs to filter, aggregate, and prepare the data for analysis.

Real-time message ingestion: If a solution includes real-time sources, the architecture must
include a way to capture and store real-time messages for stream processing.

Stream processing: After capturing real-time messages, the solution must process them by
filtering, aggregating, and preparing the data for analysis. The processed stream data is then
written to an output sink. We can use open-source Apache streaming technologies like Storm and
Spark Streaming for this.
Dilip Kumar
57
+919454018844
 Big Data Architecture
Analytical data store: Many big data solutions prepare data for analysis and then serve the
processed data in a structured format that can be queried using analytical tools. Example: Azure
Synapse Analytics provides a managed service for large-scale, cloud-based data warehousing.

Analysis and reporting: The goal of biggest data solutions is to provide insights into the data
through analysis and reporting. To empower users to analyse the data, the architecture may
include a data modelling layer. Analysis and reporting can also take the form of interactive data
exploration by data scientists or data analysts.

Dilip Kumar
58
+919454018844
 Big Data Architecture
Orchestration: Most big data solutions consist of repeated data processing operations, that
transform source data, move data between multiple sources and sinks, load the processed data
into an analytical data store, or push the results straight to a report. To automate these workflows,
we can use an orchestration technology such as Azure Data Factory.

Dilip Kumar
59
+919454018844
 Big Data Components

1. Ingestion:
• The ingestion layer is the very first step of pulling in raw data.
• It comes from internal sources, relational databases, non-relational databases, social media,
emails, phone calls etc.
• There are two kinds of ingestions:
Batch, in which large groups of data are gathered and delivered together.
Streaming, which is a continuous flow of data. This is necessary for real-time data analytics.
Dilip Kumar
60
+919454018844
 Big Data Components
2. Storage:
• Storage is where the converted data is stored in a data lake or warehouse and eventually
processed.
• The data lake/warehouse is the most essential component of a big data ecosystem.
• It needs to contain only thorough, relevant data to make insights as valuable as possible.
• It must be efficient with as little redundancy as possible to allow for quicker processing.

Dilip Kumar
61
+919454018844
 Big Data Components
3. Analysis:
• In the analysis layer, data gets passed through several tools, shaping it into actionable insights.
• There are four types of analytics on big data:
Diagnostic: Explains why a problem is happening.
Descriptive: Describes the current state of a business through historical data.
Predictive: Projects future results based on historical data.
Prescriptive: Takes predictive analytics a step further by projecting best future efforts.

Dilip Kumar
62
+919454018844
 Big Data Components
4. Consumption:
• The final big data component is presenting the information in a format digestible to the end-
user.
• This can be in the forms of tables, advanced visualizations and even single numbers if
requested.
• The most important thing in this layer is making sure the intent and meaning of the output is
understandable.

Dilip Kumar
63
+919454018844
 Drivers for Big Data
Big Data has quickly risen to become one of the most desired topics in the industry.
The main business drivers for such rising demand for Big Data Analytics are:
1. The digitization of society
2. The drop in technology costs
3. Connectivity through cloud computing
4. Increased knowledge about data science
5. Social media applications
6. The rise of Internet-of-Things(IOT)
Example: A number of companies that have Big Data at the core of their strategy like:
Apple, Amazon, Facebook and Netflix have become very successful at the beginning of the 21st
Dilip Kumar
century.
+919454018844 64
 Features of Big-Data
• It should support variety of data format
• It should provide data analysis and reporting tools
• It should provide real-time data analysis software
• It should have tools for searching the data through large data set
• It should have capability for rapid development

Dilip Kumar
65
+919454018844
 Big Data features
Big Data features –security, compliance, auditing and protection
BIG DATA SECURITY :
• Big data security is the collective term for all the measures and tools used to guard both the
data and analytics processes from attacks, theft, or other malicious activities that could harm
or negatively affect them.
• For companies that operate on the cloud, big data security challenges are multi-faceted.

• When customers give their personal information to companies, they trust them with personal
data which can be used against them if it falls into the wrong hands.

Dilip Kumar
66
+919454018844
 Big Data features
Big Data features –security, compliance, auditing and protection
BIG DATA COMPLIANCE :
• Data compliance is the practice of ensuring that sensitive data is organized and managed in
such a way as to enable organizations to meet enterprise business rules along with legal and
governmental regulations.
• Organizations that don’t implement these regulations can be fined up to tens of millions of
dollars and even receive a 20-year penalty.

Dilip Kumar
67
+919454018844
 Big Data features
Big Data features –security, compliance, auditing and protection
BIG DATA AUDITING :
• Auditors can use big data to expand the scope of their projects and draw comparisons over
larger populations of data.
• Big data also helps financial auditors to streamline the reporting process and detect fraud.

• These professionals can identify business risks in time and conduct more relevant and accurate
audits.

Dilip Kumar
68
+919454018844
 Big Data features
Big Data features –security, compliance, auditing and protection
BIG DATA PROTECTION :

• Big data security is the collective term for all the measures and tools used to guard both the data and
analytics processes from attacks, theft, or other malicious activities that could harm or negatively
affect them. That’s why data privacy is there to protect those customers but also companies and their
employees from security breaches.

• When customers give their personal information to companies, they trust them with personal data
which can be used against them if it falls into the wrong hands.

• Data protection is also important as organizations that don’t implement these regulations can be
Dilip Kumar
fined up to
+919454018844 tens of millions of dollars and even receive
69 a 20-year penalty.
 Thankyou

Dilip Kumar
70
+919454018844