Module 3

BIG DATA AND ANALYTICS
Subject Code : 18CS72 CIE Marks : 40
Lecture Hours : 50 SEE Marks : 60
Credits : 04
Arun Kumar P
Dept. of ISE, JNNCE, Shivamogga
10 Hours
NoSQL Big Data Management, MongoDB and Cassandra
Arun Kumar P
Introduction
• Big Data uses Distributed Systems

• Distributed system consists of multiple data nodes at clusters
• Tasks execute in a parallel with data nodes in clusters
• Computing nodes communicate with the application through network
Arun Kumar P
Introduction
Following are features of distributed computing architecture

1. Increased realiability and fault tollerance
2. Flexibility: Easy to install, implement and deug new services in
distributed environment
3. Sharding: storing the different parts of data onto different sets of
data nodes, clusters or servers
4. Speed:Computing power increases as shards run parallely on individual
data nodes in cluster independently
Arun Kumar P
Introduction
Following are features of distributed computing architecture

5. Scalability: Both horizontal and vertical sacaling possible
6. Resources Sharing: Memory, Machines and N/w, reduces the cost
7. Open systems: Service accessible to all nodes
8. Performance: Collection of processors increases performance
Arun Kumar P
Introduction
Demerits
1. Issues in troubleshooting in a larger networking infrastructure
2. Additional software requirements
3. Security risks for data and resources
Arun Kumar P
Introduction
Following are key terms used in Database Systems

1. Class: Template of a program codes that is extendable
Objects, Initial values for members (states), member functions
2.Object: Instance of class
3.Tupple: Ordered set of data, contributes a record
4. Transaction: execution of instructions in two interrelated entities (Query)
5. DB transactional model: model for transactions, following ACID properties
or BASE properties
Arun Kumar P
Introduction

6. My SQL: widely used open-source database excels as a
content management server
7.Oracle: Widely used object-relational DBMS, written in C++
8. DB2: Database server from IBM, has support to BigData analytics
9. Sybase: Database server based on relational model for business on UNIX
Sybase was the first enterprise level DBMS in Linux.
Arun Kumar P
Introduction

10. MS SQL : Microsoft developed RDBMS for enterprise level databases
that supports SQL and NoSQL
11. PostgreSQL: Enterprise level, object-relational DBMS.
PostgreSQL uses procedural language like Perl and
Python, in addition to SQL.
Arun Kumar P
NoSQL Data Store
• SQL is a programming language based on relational algebra

• It is a declarative language and it defines the data schema
• SQL creates databases and RDBMSs
• RDBMSs uses tabular data store with relational algebra
• Relations are set of tuples
• Tuples are named attributes
• Tuple identifies by uniquely by candidate keys
• Transactions are exhibit ACID properties
Arun Kumar P
NoSQL Data Store
ACID properties in SQL

1. Automicity: All operations in transaction must complete
If interrupted, must be undone/roll back
2. Consistency: Transaction must maintain integrity constraint
Follow consistency principle
3.Isolation: Each transaction must be separated
4. Durability: Transaction must persist once completed.
Arun Kumar P
NoSQL Data Store
Triggers, Views and Schedules in SQL Databases
• Trigger is a special stored procedure
• Executes when specific action/s occurs within DB, like change in table data
or actions like UPDATE, INSERT and DELETE
Arun Kumar P
NoSQL Data Store
• Views refers to logical construct, used in query statements or
• A view is a virtual or logical table that allows to view or manipulate
parts of the tables.
• A view saves a division of complex query instructions and
reduces query complexity
CREATE VIEW view_name AS
SELECT column_name(s)
FROM table_name
WHERE condition;
Arun Kumar P
NoSQL Data Store
• Chronological sequence of instructions which executes concurrently
• When a transaction in schedule, then all instructions of the transaction
are included in the schedule
• Enables execution of multiple transactions in allotted time intervals
Arun Kumar P
NoSQL Data Store
Join in SQL Databases
• SQL DBs facilitate combining rows from two or more tables
based on related columns
• By using JOIN function during database transaction
SELECT KitKatSales
FROM TransactionTb1 INNER JOIN ACVMSalesTb1
ON TransactionsTab1. KitKatSales = ACVMSalesTb1. KitKatSales ;
• There are scalability and distributed design issues
Arun Kumar P
NoSQL Data Store
SQL compliant format means
• Database tables constructed using SQL and they enable processing
queries written using SQL
NoSQL term coveys two different meanings
1. Does not follow SQL compliant formats
2. Not Only SQL use SQL compliant formats with variety of other
querying and access methods
Arun Kumar P
NoSQL Data Store
New category of Data store is NoSQL, offers many features
• Schema flexibility
• Simple relationships
• Dynamic schemas
• Auto sharding
• Replication
• Integrated caching
• Horizontal scalability of shards
• Distributable tuples
• Semi-structures data and flexibility in approach
Arun Kumar P
NoSQL Data Store
Issues
• Lack of standardization in approaches
• Processing difficulties for complex queries
• Depends on eventually consistent results in place of consistency in all
states
Arun Kumar P
NoSQL Data Store
Big-Data NoSQL
• NoSQL DB does not require specialized storage and HW for processing
• Storage can be on cloud
• NoSQL records are in non-relational data store systems
• They use flexible data models
• The records use multiple schemas
• NoSQL data store are considered as semi-structured data
Arun Kumar P
NoSQL Data Store
NoSQL Data Store characteristics
1. NoSQL is a class of non-relational data storage system with flexible
data model
Example: Key-value pairs
Name-value pairs
Column family big data store
Tabular data store
Cassandra (used in Facebook/Apache)
HBase
Arun Kumar P
NoSQL Data Store
Example: Hash table [Dynamo (Amazon S3)]
Unordered keys using JSON (CouchDB)
JSON (PNUTS)
JSON (MongoDB)
Graph Store
Object Store
Ordered keys
Arun Kumar P
NoSQL Data Store
2. NoSQL does not has fixed schema, such as table
Do not use concept of joins
Data written at one node can be replicated at multiple nodes
Arun Kumar P
NoSQL Data Store
Features in NoSQL Transactions
1. Relax one or more ACID properties
2. Characterize by two out of three properties of CAP theorem
Two are at least present for the Application/Service/Process
3. Can be characterized by BASE properties
Consistency: All copies have the same values
Arun Kumar P
NoSQL Data Store
CAP theorem
Consistency: All copies have the same values or
All nodes observe same data at same time
Therefore, operations in one partition of the DB should reflects in other
related partitions in case of distributed DB.
Arun Kumar P
NoSQL Data Store
CAP theorem
Availability: At least one copy is available in case a partition become
inactive or fails
Replication ensures availability
Network failure leads to unavailability
Arun Kumar P
NoSQL Data Store
CAP theorem
Partitions: Division of large database into different databases without
affecting operations on them by adapting specific procedures
Arun Kumar P
NoSQL Data Store
Brewer’s Consistency Availability and Partition Tolerance
CAP theorem demonstrates that any distributed system cannot guarantee
C, A and P together
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Big-Data NoSQL Solutions
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Schema-less Models
- Schema refers to designing of a structure for datasets

- Data structures for storing into databases
- NoSQL data not necessarily have a fixed table schema
- NoSQL data model offers rlaxation in one or more ACID properties
Arun Kumar P
NoSQL Data Store
Characteristics of schema-less Models
Arun Kumar P
NoSQL Data Store
Increasing flexibility for Data Manipulation: BASE properties
• Basic Availability: Ensures by distribution of shards across many data

nodes with high degree of replication
• Soft state: Ensures processing even in the presence of inconsistencies
but achieving consistency eventually
• Eventually consistency: Consistency requirement inNoSQL databases
meeting at some point of time in future
BASE model is not necessarily appropriate in all cases but it is flexible
Arun Kumar P
NoSQL Data Store Example: Student Database in NoSQL
Arun Kumar P
NoSQL Data Store
NoSQL Data Architecture Patterns
• Key-Value Store: High performance, Scalable and flexible
Data retrieval is fast in key-value store
- A Simple string called, key maps to large data string or BLOB
- Key-value store accesses use a primary key for accessing the values
Arun Kumar P
NoSQL Data Store
Advantages of Key-Value store
1. Any data type in the value field. Store Information BLOB of data
like text, hypertext, images, audio and video
2. Query just request the values and returns the values as single item,
values can be of any data type.
3. Key-value store is eventually consistent
4. May be hierarchical or may be ordered key-value store
5.Returned values can be used to convert into lists, table columns, data-
frame fields
Arun Kumar P
NoSQL Data Store
Advantages of Key-Value store
6.Scalable, Reliable, portable and low operational cost
7. Key can be synthetic or auto generated. Also flexible and in many format
i) Artificially generated strings created from hash of a value
ii) Logical path names to images or files
iii) REST web service calls (request response cycles)
iv) SQL queries
Arun Kumar P
NoSQL Data Store
Limitations
i) No indexes are maintained on values, thus a subset of values is not
searchable
ii) Key-value store does not provide traditional DB capabilities, such as
atomicity, consistency
iii) Maintaining unique values as keys may become more difficult when
the volume of data increases
iv) Queries cannot be performed on individual values. No clause like
where in a relational DB usable that filters a result set.
Arun Kumar P
NoSQL Data Store
Key-Value store provides client to read and write values using key as
follows
i) Get(key) returns value associated with key
ii) Put(key,value) associates value with key and updates a value if key
exists
iii) Multi-get(key1,key2,…,keyN) returns the list of values associated with
list of keys
iv) Delete(key) removes key and its value from data store
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Typical uses of key-value store are

1. Image store
2. Document or file store
3. Lookup table
4. Query cache
Arun Kumar P
NoSQL Data Store
Key-Value: Riak
• Open source Erlang language data store
• It is a key-value data store system
• Data auto-distributes and replicates in Riak
Amazon’s DynamoDB, Redis, Memcached and its flavors,
Berkely DB, upscaledb, Project Voldemort and Couchbase.
Arun Kumar P
NoSQL Data Store
Document Store: Characteristics
• High performance
• Flexible
• Scalability varies, depends on stored contents
• Complexity is low compared to tabular, object and graph data stores
Arun Kumar P
NoSQL Data Store
Document Store: Features
1. Stores unstructured data
2. Storage has similarity with object store
3. Data stores in nested hierarchies
Ex: JSON, XML DOM, BLOB, Document tree
4. Querying is easy
Ex: Section number, sub section number, figure caption, table headings
are used to retrieve document partitions
Arun Kumar P
NoSQL Data Store
Document Store: Features

5. No object relational mapping enables easy search by following paths
from root of document tree
6. Transactions on the document data store exhibit ACID properties
Arun Kumar P
NoSQL Data Store
Uses of Document Store

1. Office documents
2. Inventory store
3. Forms data
4. Document exchange
5. Document search
Arun Kumar P
NoSQL Data Store
Demerits of Document Store

- Incompatible with SQL
- Complex for implementation
Tools are
- CouchDB
- MongoDB
- Terrastore
- OrientDB
- RavenDB
Arun Kumar P
NoSQL Data Store
Document JSON Format in CouchDB Database

- Open source
- Provides mapping functions during querying, combining and filtering of
information
- Deploys JSON Data store model for documents. Each document
maintains separate data and metadata
- CouchDB is multi-master application. Write does not require field locking
during concurrency control.
Arun Kumar P
NoSQL Data Store
Document JSON Format in CouchDB Database

- COuchDB query language is Javascript.
- CouchDB access the documents using HTTP API
- Data replication results fault tolerance and reliability
Arun Kumar P
NoSQL Data Store
XML Document Architecture Pattern and Discovering Hierarchical Structure
Arun Kumar P
NoSQL Data Store
Document Architecture Pattern and Discovering Hierarchical Structure
- Xpath and Xquery query languages for finding and extracting elements
and attributes of documents
- Xpath treats XML document as a tree of nodes
- Xpath queries are expressed in the form of Xpath expressions
Arun Kumar P
NoSQL Data Store
Document Architecture Pattern and Discovering Hierarchical Structure
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Arun Kumar P
NoSQL Data Store
Tabular Data Store

- Uses rows and columns
- Row head field used as key to access multiple values from successive
columns in that row
- OLTP is fast on in-memory row-form data
- Relational DB store is in-memory row-form data
- Key in the first column of row is at memory address
- Values in successive columns at successive addresses, makes OLTP
easier
Arun Kumar P
NoSQL Data Store
Tabular Data Store

- All fields of row are accessed at a time together
- Makes data searching and accessing faster during transaction
processing
Arun Kumar P
NoSQL Data Store
Tabular Data Store

- In-memory column-based data has the keys (row-head keys) in first
column of each row at successive memory addresses
- The next column of each row after key has the values at successive
memory addresses, so on upto N columns
- Column based makes OLAP easier
- All fields of column access together
Arun Kumar P
NoSQL Data Store
Tabular Data Store : Column-Family Data Store

- Columnar Data Store
- Column families
- Column-family data store
- Sparse column fields
- Grouping column families
- Grouping into rows
Arun Kumar P
NoSQL Data Store
Tabular Data Store : Characteristics of Column-Family Data Store

- Scalability: Interface to field is simple(Row ID and Column name)
- Partitionability: row group, column group
- Availability: Cost of replication is lower
- Tree-like columnar structure: Columns, Columns-family,
Column-family groups
- Adding New Data easy:
Arun Kumar P
NoSQL Data Store
Tabular Data Store : Characteristics of Column-Family Data Store
- Querying all field values: in column, column-family,
column-family group
- Replication of column: default replication factor=3
- No optimization for join: Similar to sparse matrix
Uses
i) Web crawling ii) Large sparsely populated tables
iii) System that has high variance
Example :Googles BigTable
Arun Kumar P
NoSQL Data Store
Record Columnar (RC) File Format

Optimized Row Columnar (ORC) File format: row grouped data-stripes
Arun Kumar P
NoSQL Data Store
Parquet File Formats: nested hierarchical columnar storage concept.

Nesting sequence: Table, row group, column chunk and chunk page.
Arun Kumar P
NoSQL Data Store
Object Data Store :

An Object data store is a repository, which stores
1. Objects (Files, images, documents, folders and business reports
2. System metadata filename, creation_date, last_modified, language
used (C, C++, C# etc), access permissions, supported query language
3. Custom metadata subject, sharing permissions, catogery
Arun Kumar P
NoSQL Data Store
Object Data Store consists 11 functions supporting API for

1. Scalability
2. Indexing
3. Large collections
4. Querying language, processing and optimization
5. Transactions
6. Data replication for high availability, distribution model and integration
7. Schema evolution
Arun Kumar P
NoSQL Data Store
Object Data Store consists 11 functions supporting API for

8. Persistency
9. Persistent object life cycle
10. Adding modules
11. Locking and caching stratergy
Amazon S3 and MS Azure BLOB support object store
Arun Kumar P
NoSQL Data Store
Object Relational Mapping
Arun Kumar P
NoSQL Data Store
Graph Database
- Series of interconnected nodes
G = (E,V)
- Edges encode relationship between nodes
- Complexity is high
- Performance is variable with scalability
- Enables fast network search
- Linked data sets, like social media data
Arun Kumar P
NoSQL Data Store
Graph Database
- Querying for data uses Graph traversal along the paths
- Traversal may be single step, path expression or recursion
Characteristics
1. Use special query language (RDF uses SPARQL)
2. Can have hyper edge.
3. Relationship between node is consistent in Graph store
4. Have poor scalability
Arun Kumar P
NoSQL Data Store
Graph Database
Uses
- Link analysis
- Friend of friend queries
- Rules and inference
- Rule induction
- Pattern matching
Ex: Neo4j, AllegroGraph, HyperGraph, Infinte Graph, Titan and FlockDB
Arun Kumar P
NoSQL Data Store
Variations of NoSQL Architectural patterns

- Selected pattern may need variation due to business requirement
- Business requirements are ease of using pattern and long term
competitive advantage
- Kelly-McCreary, co founder of ‘NoSQL Now’ suggested that when
selecting a NoSQL pattern
- The pattern may need change and require variation to another patterns
Arun Kumar P
NoSQL Data Store
Variations of NoSQL Architectural patterns

Some reasons are
1. Focus changing from performance to scalability
2. Changing from modifiability to agility
3. Affording capacity, availability support, ability for searching and
monitoring the actions
Arun Kumar P
NoSQL Data Store
Steps for selecting pattern
1. Select an architecture
2. Perform difficulty analysis for each six patterns
Difficulties may be low, medium or high in following processes
i) Ingestion ii)Validation of structure and fields
iii) Updating process using batch or record by record approach
iv) Searching process is using full text or by changing sorting order
v) Export results in HTML, XML or JSON
3. Estimate total efforts for each pattern required
Arun Kumar P
NoSQL to Manage Big Data
NoSQL solutions for Big Data and Characteristics
1. High and easy scalability horizontally
2. Support to replication
3. Distributable
4. Usage less expensive servers
5. Usage of open source tools
6. Support schema less model
7. Support integrated caching
8. Flexible
Arun Kumar P
Types of Big Data problems
Problems arise due to limitations of NoSQL
1. Solutions must drop support for Database join
2. Open source tools may not have standards
- No stored procedure in MongoDB
- GUI mode tools not available in market
- Some scarify ACID compliancy
Arun Kumar P
Arun Kumar P
Shared Nothing Architecture
- Big Data store consists SN architecture
- Node does not share data with other node
- So easily shards can be created
- Partition processes different queries on data of different users at each
node independently
- A coordination protocol controls the processing at all SN nodes.
- An SN architecture optimizes massive parallel data processing
Arun Kumar P
Features
- Independence: No memory sharing, possesses computational
self-sufficiency
- Self healing: Link failure causes creation of another link
- Each node functioning as shard: Each node stores a partition
- No network contention
Arun Kumar P
Choosing Distribution model: 1.Single Server Model
- Simplest, suits well for Graph DB
- Also used by key-value, column-family, BigTable
- An application executes data sequentially in Single server
Arun Kumar P
Choosing Distribution model: 2.Sharding very large Database
- Application process runs on multiple shards in parallel
- Performance improved
- In case a link failure, migrate the shard DB to
another node
Arun Kumar P
Choosing Distribution model: 3.Master Slave Distribution Model
Arun Kumar P
Choosing Distribution model: 4.Peer-to-peer (Cassandra)
Arun Kumar P
- Maste-Slave replication provides greater scalability for read

operations
- Replication provides resilience during read but not at write
operation
- Peer-to-peer replication provides resilience for both read and write
Arun Kumar P
Ways of handling Big Data Problems
1. Evenly didtribute data on

cluster using hash rings
2. Use replication to horizontally
distribute client read-request
3. Moving queries to data, not data
to queries
4. Queries distribution to multiple
nodes
Arun Kumar P
MongoDB DataBase
- Opensource DBMS, used to create and manage DBs
- Manages collection and document data store
- Functions: Viewing, Querying, Changing (updating, inserting,
appending or deleting), Visualizing, Running the transactions
Arun Kumar P
MongoDB DataBase
1. Open source
2. Non-relational
3. NoSQL
4. Distributed
5. Document based (JSON like documents)
6. Cross platform
Arun Kumar P
MongoDB DataBase
7. Scalable
8. Flexible Data Model
9. Indexed
10. Multi-master
11. Fault tolerant
Arun Kumar P
MongoDB
Typical applications
• Content management and delivery system
• Mobile Applications
• User Data management
• Gaming
• E-Commerce
• Analytics
• Archiving and logging
Arun Kumar P
MongoDB
Features
1. Physical container for collections: Each DB get its own
set of files on the file system. Number of DBs can run on
single MongoDB Server.
DB server in MongoDB is mongod and client is mongo
2. Collection: Stores number of MongoDB documents

Documents of collection are schema-less
Arun Kumar P
MongoDB
Features
3. Document Model: Document is unit of data in MongoDB
Uses JSON to store data in documents
JSON data basically has key-value pairs
Have dynamic schema
4. Document Data Store: One collection holds different documents

No. of feilds, content and size of document
differ from one document to another
Arun Kumar P
MongoDB
Features
5. Storing of data is flexible: fields can vary from doc to doc
Data structure can be changed over time
6. Storing of documents on disk is in BSON serialization format.

BSON is binary representation of JSON documents
Arun Kumar P
MongoDB
Arun Kumar P
MongoDB
Features
7. Querying, Indexing and real time aggregation :
allows accessing and analyzing data efficiently
8.Deep Query-ability: Supports dynamic queries on documents using

document-based query language
9. No complex joins
10. Distributed DB leads high availability, provides horizontal scalability
Arun Kumar P
MongoDB
Features
11.Indexes on any field in collection of documents:
support queries and operations
12. Atomic operations on single document
13.Fast in place updates: does not allocate new memory and write full
new copy of the object when update required
14. No configurable cache: Uses all free available memory
15. Conversion/mapping: application objects to data store objects not
needed
Arun Kumar P
MongoDB
Arun Kumar P
MongoDB
Replication in MongoDB
- MongoDB replicates with help of replica set
- Replica set is group of mongod processes that store same dataset
- Replica set has has minimum 3 nodes
- Any one out of them called primary, remaining are secondary
- Primary node receives write operations
- Data replicates from primary to secondary
Arun Kumar P
MongoDB
Arun Kumar P
MongoDB
Arun Kumar P
MongoDB
Auto Sharding in MongoDB

- Distribute data across multiple machines
- Sharding provides additional write capability by distributing write load
over number of mongod instances
- Each shard is an independent DB
- Collections of DB forms single logical DB
- 1TB Data-> 20 shards, each 50GB
Arun Kumar P
MongoDB
Data types which MongoDB documents supports
Arun Kumar P
MongoDB Data types which MongoDB documents supports
Arun Kumar P
MongoDB Data types which MongoDB documents supports
Arun Kumar P
MongoDB Features of MongoDB wrt RDBMS
Arun Kumar P
MongoDB MongoDB Query Commands
Arun Kumar P
MongoDB MongoDB Query Commands
Arun Kumar P
MongoDB: Sample usage of Commands
To create Database
use DATABASE >use tata
To check your currently selected database
db >db
To check your databases list
show dbs >show dbs
To drop a database
db.dropDatabase()
>use tata
>db.dropDatabase()
Arun Kumar P
To create a collection
>db.COLLECTION_NAME.insert(document)
db.tata.insert({
"type": "car",
"model": "2018",
“name": “Nexon",
})
Arun Kumar P
To add array in collection db.tata.insert( [ {

"type": "car",
"model": "2018",
“name": “Nexon",
},
"type": "car",
"model": "2020",
“name": “Safari,
}
"type": "car",
"model": "2021",
“name": “Punch",
}
] )
Arun Kumar P
To view all collection document
db.COLLECTION_NAME.find()
>db.tata.find()
To update a document
db.COLLECTION_NAME.update(SELECTION_CRITERIA, UPDATED_DATA)
db.tata.update({‘model':2020'}, {$set:{‘name':Harrier'}},{multi:true})
To delete a document
db.COLLECTION_NAME.remove(DELLETION_CRITTERIA)
db.tata.remove((“model”:2020))
Arun Kumar P
CassandraDB
- Developed by Facebook and released by Apache

- Later, IBM released enhancement of Cassandra, as open source version
- Includes IBM Data Engine which process NoSQL data store
- Cassandra column family database
- Has distributed design of Dynamo, written in Java
- Facebook, IBM, Twitter, Cisco, Rackspace, eBay, Netflix adopted
Arun Kumar P
CassandraDB: Characteristics
- Open source
- Scalable
- Non-relational
- NoSQL
- Distributed
- Column based
- Decentralized
- Fault tolerant
- Tuneable Consistency
Arun Kumar P
CassandraDB: Features
- Maximizes the number of writes – writes are not very costly

- Maximizes data duplication
- Does not support joins, group by, OR clause and aggregations
- Uses classes consisting of ordered keys and semi-structured data
storage systems
- No Master node
- Peer to peer distribution
Arun Kumar P
- Stores and hadles massive data of structured, semi-structured and

unstructured format
- It contains set of prgrams for create and manage the Databases
- Provides functions for Querying, viewing, changing, visualize and

transactions performing
Arun Kumar P
CassandraDB: Data Replication
- Stores data on multiple nodes, so no single point of failure

- Data replication uses a replication stratergy
- Replication factor determines no. of replicas placed on different nodes
- Cassandra returns most recent value of the client
- If any node responds stale value, Cassandra detects and performs
a read repair to update stale values
Arun Kumar P
CassandraDB: Components
Arun Kumar P
CassandraDB: Components
Arun Kumar P
Scalability:
- Provides linear scalability which increases the throughput
- Decreases response time on increase in number of nodes at cluster
Transaction Support:
- Supports ACID properties
Arun Kumar P
Replication Option: Two strategy’s

1. Simple strategy
Simply specify replication factor for cluster
2. Network Topology strategy

Setting replication factor for each data center independently
Arun Kumar P
CassandraDB: Data Types
Arun Kumar P
Arun Kumar P
Arun Kumar P
CassandraDB: Data Model

- Based on Google’s BigTable
- Each value maps with two strings (row key, column key) and timestamp
- DB can considered as sparse distributed, multi-dimensional sorted map
- Google file system splits table into multiple tablets along a row
- Each tablet called META1 tablet
- Each META1 tablet maximum size is 200MB
- META0 is master-server
- META1 tablet retrieved by META0 server through querying
Arun Kumar P
CassandraDB: Data Model
Cassanra Data Model consists 4 main components

1. Cluster: made up of multiple nodes and keyspaces
2. Keyspace: Namespace to to group multiple column families, one per partition
3. Column: Consists column name, value and timestamp
4. Column-family: Multiple columns with row key reference

Arun Kumar P
CassandraDB: DDL Commands
Arun Kumar P
Arun Kumar P
Arun Kumar P
CassandraDB: CONSISTENCY command

- Consistency command shows current consistency level
CONSISTENCY <LEVEL> Sets new level consistency

Valid consistency levels are
ALL ANY ONE
TWO THREE QUORUM
LOCAL_ONE LOCAL_QUORUM EACH_QUORUM
SERIAL LOCAL_SERIAL
Arun Kumar P
ALL : Highly consistent. A write must be written to commitlog and memtable

on all replica nodes in the cluster
ANY : A write must be written to commitlog and memtable of at least one node
ONE : A write must be written to commitlog and memtable of at least one

replica node
Arun Kumar P

TWO, THREE : (Same as one, but) A write must be written to commitlog
& memtable of at least two or three replica nodes respectively
QUORUM : A write must be written to commitlog and memtable on quorum of

replica nodes in all data centers
LOCAL_ONE : A write must be written for at least one replica node in the local
data center
Arun Kumar P

LOCAL_QUORUM: A write must be written to commitlog and memtable
on quorum of replica nodes in the same data center
EACH_QUORUM: A write must be written to commitlog and memtable

on quorum of replica nodes in all data centers
SERIAL : Linearizable consistency to prevent unconditional update
LOCAL_SERIAL : Same as serial but restricted to local data center

Arun Kumar P
CassandraDB: Keyspaces
- An object that contains all column families as bundle
- Generally there is one keyspace per application
- Keyspace in Cassandra is a Namespace that defines data replication on

nodes
- A cluster contains one keyspace per node

Arun Kumar P
CREATE KEYSPACE <keyspace name>
WITH replication = { ‘class’ : ‘<Strategy name>’,

‘replication_factor’ : ‘<No. of replicas>’ }
AND durable_writes = ‘<TRUE/FALSE>’;
Arun Kumar P
ALTER KEYSPACE <Keyspace Name>
WITH replication = { ‘class’ : ‘<Strategy name>’,

‘replication_factor’ : ‘<No. of replicas>’ };
DESCRIBE KEYSPACE : Displays existing keyspaces

DROP KEYSPACE : Drops a keyspace
Use KEYSPACE : Connects the client session with keyspace
Arun Kumar P
CassandraDB: CQL (Cassandra Query Language
Arun Kumar P
Arun Kumar P
Arun Kumar P
Arun Kumar P

Module 3

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Module 3

Uploaded by

Copyright:

Available Formats

BIG DATA AND ANALYTICS

Subject Code : 18CS72 CIE Marks : 40

Lecture Hours : 50 SEE Marks : 60

NoSQL Big Data Management, MongoDB and Cassandra

• Big Data uses Distributed Systems

Following are features of distributed computing architecture

Following are features of distributed computing architecture

Following are key terms used in Database Systems

Following are key terms used in Database Systems

Following are key terms used in Database Systems

• SQL is a programming language based on relational algebra

ACID properties in SQL

- Schema refers to designing of a structure for datasets

• Basic Availability: Ensures by distribution of shards across many data

Typical uses of key-value store are

Document Store: Features

Uses of Document Store

Demerits of Document Store

Document JSON Format in CouchDB Database

Document JSON Format in CouchDB Database

Tabular Data Store

Tabular Data Store

Tabular Data Store

Tabular Data Store : Column-Family Data Store

Tabular Data Store : Characteristics of Column-Family Data Store

Record Columnar (RC) File Format

Parquet File Formats: nested hierarchical columnar storage concept.

Object Data Store :

Object Data Store consists 11 functions supporting API for

Object Data Store consists 11 functions supporting API for

Amazon S3 and MS Azure BLOB support object store

Object Relational Mapping

Variations of NoSQL Architectural patterns

Variations of NoSQL Architectural patterns

- Maste-Slave replication provides greater scalability for read

1. Evenly didtribute data on

- Opensource DBMS, used to create and manage DBs

- Manages collection and document data store

- Functions: Viewing, Querying, Changing (updating, inserting,

appending or deleting), Visualizing, Running the transactions

5. Document based (JSON like documents)

8. Flexible Data Model

11. Fault tolerant

2. Collection: Stores number of MongoDB documents

4. Document Data Store: One collection holds different documents

6. Storing of documents on disk is in BSON serialization format.

8.Deep Query-ability: Supports dynamic queries on documents using

Auto Sharding in MongoDB

Data types which MongoDB documents supports

To add array in collection db.tata.insert( [ {

- Developed by Facebook and released by Apache

- Maximizes the number of writes – writes are not very costly

- Stores and hadles massive data of structured, semi-structured and

- It contains set of prgrams for create and manage the Databases

- Provides functions for Querying, viewing, changing, visualize and

CassandraDB: Data Replication

- Stores data on multiple nodes, so no single point of failure

Replication Option: Two strategy’s

2. Network Topology strategy

CassandraDB: Data Model

CassandraDB: Data Model

Cassanra Data Model consists 4 main components