Q. 1. State and explain Machine learning
2]What is deep learning and describe 4
algorithm in Apache spark ?
Ans:- Apache Spark MLlib is a machine
learning library that provides a wide range of
algorithms for classification, regression,
clustering, collaborative filtering,
dimensionality reduction, and feature
extraction. MLlib is built on top of Spark's
distributed computing engine, which makes it
scalable to large datasets.
MLlib algorithms are implemented in two
different APIs:
a.)RDD-based API: This API is built on top of
Spark's Resilient Distributed Datasets (RDDs).
It is the original MLlib API and is still
supported, but it is in maintenance mode.
b.)DataFrame-based API: i.)This API is built on
top of Spark's DataFrames. It is the newer and
preferred MLlib API, and it is the primary API
for machine learning in Spark.
ii.)The DataFrame-based API provides a
higher-level abstraction for machine learning,
making it easier to build and deploy machine
learning pipelines. It also provides a number
of features that are not available in the RDD-
based API, such as support for model
persistence and distributed hyperparameter
tuning.
popular machine learning algorithms in
Apache Spark:
a.)Classification: Logistic regression, naive
Bayes, decision trees, random forests,
gradient-boosted trees
b.)Regression: Generalized linear regression,
survival regression, decision trees, random
forests, gradient-boosted trees
c.)Clustering: K-means, Gaussian mixtures,
latent Dirichlet allocation (LDA)
d.)Collaborative filtering: Alternating least
squares (ALS)
e.)Dimensionality reduction: Principal
component analysis (PCA), singular value
decomposition (SVD)
f.)Feature extraction: Word2Vec, TF-IDF, one-
hot encoding
How to use machine learning algorithms
in Apache Spark:-To use a machine learning
algorithm in Apache Spark, you first need to
load the data into a Spark DataFrame. Once
the data is loaded, you can use the MLlib API
to create and train a model. Once the model
is trained, you can use it to make predictions
on new data.
Q.
applications of deep learning and
challenges?
Ans:- Deep learning is a subset of machine learning
that has gained significant popularity due to its ability
to learn and make predictions from data with minimal
feature engineering. It is based on artificial neural
networks, which are inspired by the structure and
function of the human brain. Deep learning has found
numerous applications across various domains, and I'll
explain some of them in detail:
1.)Computer Vision:-
a.)Image Classification: Deep learning models
like Convolutional Neural Networks (CNNs)
can classify images into different categories.
This is widely used in applications such as
facial recognition, object detection, and
identifying diseases in medical images.
2.)Natural Language Processing (NLP):-
a.)Language Translation: Recurrent Neural
Networks (RNNs) and Transformer models like
BERT have revolutionized machine
translation, enabling systems like Google
Translate to provide more accurate
translations.
3. *Speech Recognition*:
- Deep learning is used in automatic speech
recognition systems, converting spoken
language into text. Applications include voice
assistants, transcription services, and
accessibility tools for people with disabilities.
4. *Reinforcement Learning*:
- Deep reinforcement learning combines
deep neural networks with reinforcement
learning algorithms to enable machines to
learn how to make decisions by interacting
with an environment. This is used in
autonomous robotics, game playing (e.g.,
AlphaGo), and optimization problems.
1
Q.3]what is graph processing and describe
about -prequel, giraph, Apache graph x ?
Ans:- Graph processing is a type of data
processing that deals with analyzing and
manipulating data structured as a graph. In a
graph, data is represented as a collection of
nodes (vertices) and edges that connect these
nodes. Graphs are used to model various
types of relationships and networks, such as
social networks, transportation networks,
biological networks, and more.
Pregel:
a.)Pregel is a graph processing framework
developed by Google and inspired by Google's
web crawling algorithms.
b.)It provides a programming model that is
centered around the concept of vertices and
their associated states.
c.)Pregel is designed to handle large graphs
and distributed computing environments
efficiently.
d.)It allows users to define custom vertex and
message handlers to perform computations
on the graph.
Giraph:
a.)Apache Giraph is an open-source graph
processing framework based on Pregel,
designed to work with the Hadoop
ecosystem.
b.)It extends the Pregel model and is
integrated with Apache Hadoop's HDFS
(Hadoop Distributed File System) for data
storage.
c.)Giraph is well-suited for processing large-
scale graphs in parallel across a cluster of
machines.
c.)It is often used for tasks like graph traversal,
community detection, and graph analytics.
Apache Gelly (part of Apache Flink):
a.)Apache Gelly is a graph processing library
that is part of the Apache Flink project, which
is a stream processing framework.
b.)Gelly leverages Flink's capabilities for
distributed data processing to handle graph
data.
c.)It provides a high-level API for graph
operations, making it easier for developers to
work with graph data in a Flink-based
environment.
d.)Gelly supports common graph algorithms
and can be used for various graph analytics
tasks.
2
Q.4] Describe key features of Mango DB ?
Ans:- 1. Document Oriented:-MongoDB
stores all the data in the form of documents
instead of tables like in RDBMS. In these
documents, the data is stored in key-value
pairs instead of rows and columns which
make the data much more flexible in
comparison to RDBMS where each document
contains a unique ID.
2. Schema-less Database:-Schema-less
database is a great feature provided by
MongoDB which means one collection can
hold different types of documents in it. In the
MongoDB database, a single collection can
hold multiple documents and these
documents may consist of the different
numbers of fields, content, and size. It is not
required that a document is similar to another
document as it is in the case of relational
databases. Due to this amazing feature,
MongoDB provides great flexibility to
databases.
3. Scalability:-MongoDB provides horizontal
scalability with the help of a mechanism
known as sharding. Sharding refers to the
process of distributing data on multiple
servers, here a large amount of data is
partitioned into data chunks using the shard
key, and these data chunks are evenly
distributed across shards that reside across
many physical servers. It can also add new
machines to a running database.
4. Indexing:-MongoDB database indexes
every field in the documents with primary
and secondary indices which makes it easier
and takes less time to get or search data from
the pool of the data. If the data is not
indexed, then the database searches each
document with the specified query which
takes lots of time and is not so efficient.
5. Aggregation:-MongoDB also allows to
perform operations on the grouped data and
get a single result or computed result. It
provides three different aggregations, namely,
aggregation pipeline, map-reduce function,
and single-purpose aggregation methods.
6. High Performance
Due to its features like scalability, indexing,
replication, etc., the performance of
MongoDB becomes very high as also data
persistence, as compared to any other
databases.
Q.5]How mango DB provide DB
replication?
Ans:- a.)Replication exists primarily to offer
data redundancy and high availability. We
maintain the durability of data by keeping
multiple copies or replicas of that data on
physically isolated servers. That’s replication:
the process of creating redundant data to
streamline and safeguard data availability and
durability. b.)Replication allows you to
increase data availability by creating multiple
copies of your data across servers. This is
especially useful if a server crashes or if you
experience service interruptions or hardware
failure. c.)If your data only resides in a single
database, any of these events would make
accessing the data impossible. But thanks to
replication, your applications can stay online
in case of database server failure, while also
providing disaster recovery and backup
options. d.)With MongoDB, replication is
achieved through a replica set. Writer
operations are sent to the primary server
(node), which applies the operations across
secondary servers, replicating the data.
e.)If the primary server fails (through a crash
or system failure), one of the secondary
servers takes over and becomes the new
primary node via election. If that server
comes back online, it becomes a secondary
once it fully recovers, aiding the new primary
node.
Q.6] Explain horizontal scaling and
vertical scaling in mango DB?
Ans:- Scaling alters the size of a system. In
the scaling process, we either compress or
expand the system to meet the expected
needs. The scaling operation can be achieved
by adding resources to meet the smaller
expectation in the current system, by adding
a new system to the existing one, or both:_
Vertical Scaling: When new resources are
added to the existing system to meet the
expectation, it is known as vertical scaling.
Consider a rack of servers and resources that
comprises the existing system. (as shown in
the figure). Now when the existing system
fails to meet the expected needs, and the
expected needs can be met by just adding
resources, this is considered vertical scaling.
Vertical scaling is based on the idea of
adding more power(CPU, RAM) to existing
systems, basically adding more resources.
Vertical scaling is not only easy but also
cheaper than Horizontal Scaling. It also
requires less time to be fixed. a.)It expands
the size of the existing system vertically.
b.) It is harder to upgrade and may involve
downtime. c.)It is easy to implement.
d.) It is cheaper as we need to just add new
resources.
Horizontal Scaling: When new server racks
are added to the existing system to meet the
higher expectation, it is known as horizontal
scaling. Consider a rack of servers and
resources that comprises the existing
system. (as shown in the figure). Now when
the existing system fails to meet the
expected needs, and the expected needs
cannot be met by just adding resources, we
need to add completely new servers. This is
considered horizontal scaling. Horizontal
scaling is based on the idea of adding more
machines to our pool of resources.
Horizontal scaling is difficult and also costlier
than Vertical Scaling. It also requires more
time to be fixed.
a.)It expands the size of the existing system
horizontally. b.) It is easier to upgrade. c.) It
is difficult to implement d.) It is costlier, as
new server racks comprise a lot of resources
3
Q. 1. State and explain Machine learning
algorithm in Apache spark ?
Ans:- Apache Spark MLlib is a machine
learning library that provides a wide range of
algorithms for classification, regression,
clustering, collaborative filtering,
dimensionality reduction, and feature
extraction. MLlib is built on top of Spark's
distributed computing engine, which makes it
scalable to large datasets.
MLlib algorithms are implemented in two
different APIs:
a.)RDD-based API: This API is built on top of
Spark's Resilient Distributed Datasets (RDDs).
It is the original MLlib API and is still
supported, but it is in maintenance mode.
b.)DataFrame-based API: i.)This API is built on
top of Spark's DataFrames. It is the newer and
preferred MLlib API, and it is the primary API
for machine learning in Spark.
ii.)The DataFrame-based API provides a
higher-level abstraction for machine learning,
making it easier to build and deploy machine
learning pipelines. It also provides a number
of features that are not available in the RDD-
based API, such as support for model
persistence and distributed hyperparameter
tuning.
popular machine learning algorithms in
Apache Spark:
a.)Classification: Logistic regression, naive
Bayes, decision trees, random forests,
gradient-boosted trees
b.)Regression: Generalized linear regression,
survival regression, decision trees, random
forests, gradient-boosted trees
c.)Clustering: K-means, Gaussian mixtures,
latent Dirichlet allocation (LDA)
d.)Collaborative filtering: Alternating least
squares (ALS)
e.)Dimensionality reduction: Principal
component analysis (PCA), singular value
decomposition (SVD)
f.)Feature extraction: Word2Vec, TF-IDF, one-
hot encoding
How to use machine learning algorithms
in Apache Spark:-To use a machine learning
algorithm in Apache Spark, you first need to
load the data into a Spark DataFrame. Once
the data is loaded, you can use the MLlib API
to create and train a model. Once the model
is trained, you can use it to make predictions
on new data.
Q.2]What is deep learning and describe 4
applications of deep learning and
challenges?
Ans:- Deep learning is a subset of machine learning
that has gained significant popularity due to its ability
to learn and make predictions from data with minimal
feature engineering. It is based on artificial neural
networks, which are inspired by the structure and
function of the human brain. Deep learning has found
numerous applications across various domains, and I'll
explain some of them in detail:
1.)Computer Vision:-
a.)Image Classification: Deep learning models
like Convolutional Neural Networks (CNNs)
can classify images into different categories.
This is widely used in applications such as
facial recognition, object detection, and
identifying diseases in medical images.
2.)Natural Language Processing (NLP):-
a.)Language Translation: Recurrent Neural
Networks (RNNs) and Transformer models like
BERT have revolutionized machine
translation, enabling systems like Google
Translate to provide more accurate
translations.
3. *Speech Recognition*:
- Deep learning is used in automatic speech
recognition systems, converting spoken
language into text. Applications include voice
assistants, transcription services, and
accessibility tools for people with disabilities.
4. *Reinforcement Learning*:
- Deep reinforcement learning combines
deep neural networks with reinforcement
learning algorithms to enable machines to
learn how to make decisions by interacting
with an environment. This is used in
autonomous robotics, game playing (e.g.,
AlphaGo), and optimization problems.
1
Q.3]what is graph processing and describe
about -prequel, giraph, Apache graph x ?
Ans:- Graph processing is a type of data
processing that deals with analyzing and
manipulating data structured as a graph. In a
graph, data is represented as a collection of
nodes (vertices) and edges that connect these
nodes. Graphs are used to model various
types of relationships and networks, such as
social networks, transportation networks,
biological networks, and more.
Pregel:
a.)Pregel is a graph processing framework
developed by Google and inspired by Google's
web crawling algorithms.
b.)It provides a programming model that is
centered around the concept of vertices and
their associated states.
c.)Pregel is designed to handle large graphs
and distributed computing environments
efficiently.
d.)It allows users to define custom vertex and
message handlers to perform computations
on the graph.
Giraph:
a.)Apache Giraph is an open-source graph
processing framework based on Pregel,
designed to work with the Hadoop
ecosystem.
b.)It extends the Pregel model and is
integrated with Apache Hadoop's HDFS
(Hadoop Distributed File System) for data
storage.
c.)Giraph is well-suited for processing large-
scale graphs in parallel across a cluster of
machines.
c.)It is often used for tasks like graph traversal,
community detection, and graph analytics.
Apache Gelly (part of Apache Flink):
a.)Apache Gelly is a graph processing library
that is part of the Apache Flink project, which
is a stream processing framework.
b.)Gelly leverages Flink's capabilities for
distributed data processing to handle graph
data.
c.)It provides a high-level API for graph
operations, making it easier for developers to
work with graph data in a Flink-based
environment.
d.)Gelly supports common graph algorithms
and can be used for various graph analytics
tasks.
2
Q.4] Describe key features of Mango DB ?
Ans:- 1. Document Oriented:-MongoDB
stores all the data in the form of documents
instead of tables like in RDBMS. In these
documents, the data is stored in key-value
pairs instead of rows and columns which
make the data much more flexible in
comparison to RDBMS where each document
contains a unique ID.
2. Schema-less Database:-Schema-less
database is a great feature provided by
MongoDB which means one collection can
hold different types of documents in it. In the
MongoDB database, a single collection can
hold multiple documents and these
documents may consist of the different
numbers of fields, content, and size. It is not
required that a document is similar to another
document as it is in the case of relational
databases. Due to this amazing feature,
MongoDB provides great flexibility to
databases.
3. Scalability:-MongoDB provides horizontal
scalability with the help of a mechanism
known as sharding. Sharding refers to the
process of distributing data on multiple
servers, here a large amount of data is
partitioned into data chunks using the shard
key, and these data chunks are evenly
distributed across shards that reside across
many physical servers. It can also add new
machines to a running database.
4. Indexing:-MongoDB database indexes
every field in the documents with primary
and secondary indices which makes it easier
and takes less time to get or search data from
the pool of the data. If the data is not
indexed, then the database searches each
document with the specified query which
takes lots of time and is not so efficient.
5. Aggregation:-MongoDB also allows to
perform operations on the grouped data and
get a single result or computed result. It
provides three different aggregations, namely,
aggregation pipeline, map-reduce function,
and single-purpose aggregation methods.
6. High Performance
Due to its features like scalability, indexing,
replication, etc., the performance of
MongoDB becomes very high as also data
persistence, as compared to any other
databases.
Q.5]How mango DB provide DB
replication?
Ans:- a.)Replication exists primarily to offer
data redundancy and high availability. We
maintain the durability of data by keeping
multiple copies or replicas of that data on
physically isolated servers. That’s replication:
the process of creating redundant data to
streamline and safeguard data availability and
durability. b.)Replication allows you to
increase data availability by creating multiple
copies of your data across servers. This is
especially useful if a server crashes or if you
experience service interruptions or hardware
failure. c.)If your data only resides in a single
database, any of these events would make
accessing the data impossible. But thanks to
replication, your applications can stay online
in case of database server failure, while also
providing disaster recovery and backup
options. d.)With MongoDB, replication is
achieved through a replica set. Writer
operations are sent to the primary server
(node), which applies the operations across
secondary servers, replicating the data.
e.)If the primary server fails (through a crash
or system failure), one of the secondary
servers takes over and becomes the new
primary node via election. If that server
comes back online, it becomes a secondary
once it fully recovers, aiding the new primary
node.
Q.6] Explain horizontal scaling and
vertical scaling in mango DB?
Ans:- Scaling alters the size of a system. In
the scaling process, we either compress or
expand the system to meet the expected
needs. The scaling operation can be achieved
by adding resources to meet the smaller
expectation in the current system, by adding
a new system to the existing one, or both:_
Vertical Scaling: When new resources are
added to the existing system to meet the
expectation, it is known as vertical scaling.
Consider a rack of servers and resources that
comprises the existing system. (as shown in
the figure). Now when the existing system
fails to meet the expected needs, and the
expected needs can be met by just adding
resources, this is considered vertical scaling.
Vertical scaling is based on the idea of
adding more power(CPU, RAM) to existing
systems, basically adding more resources.
Vertical scaling is not only easy but also
cheaper than Horizontal Scaling. It also
requires less time to be fixed. a.)It expands
the size of the existing system vertically.
b.) It is harder to upgrade and may involve
downtime. c.)It is easy to implement.
d.) It is cheaper as we need to just add new
resources.
Horizontal Scaling: When new server racks
are added to the existing system to meet the
higher expectation, it is known as horizontal
scaling. Consider a rack of servers and
resources that comprises the existing
system. (as shown in the figure). Now when
the existing system fails to meet the
expected needs, and the expected needs
cannot be met by just adding resources, we
need to add completely new servers. This is
considered horizontal scaling. Horizontal
scaling is based on the idea of adding more
machines to our pool of resources.
Horizontal scaling is difficult and also costlier
than Vertical Scaling. It also requires more
time to be fixed.
a.)It expands the size of the existing system
horizontally. b.) It is easier to upgrade. c.) It
is difficult to implement d.) It is costlier, as
new server racks comprise a lot of resources