1) What is EDA? Explain any two types of 4) What is kurtosis? Explain its types. 7) What is Data wrangling?
7) What is Data wrangling? Explain with any one
visualization.
> EDA stands for Exploratory Data Analysis. It is a
process of investigating and analyzing a dataset to
discover patterns, trends, and relationships. EDA is an
essential step in any data science project, as it helps to
ensure that the data is well-understood and that any
subsequent analysis is appropriate and meaningful.
Data visualization is a key component of EDA. It
involves creating visual representations of the data,
such as charts, graphs, and maps. This makes it easier
to identify patterns and trends, and to spot anomalies
or outliers. Two common types of data visualization
used in EDA are: Histograms: Histograms show the
distribution of a quantitative variable. They are useful
for identifying outliers, skewness, and other patterns
in the data. Scatter plots: Scatter plots show the
relationship between two quantitative variables. They
can be used to identify correlations, trends, and
patterns.
2) What is data normalization? Illustrate any one
type of data normalization technique with an
example.
> Data normalization is a process of organizing data in
a database to minimize redundancy and improve data
integrity. It involves dividing the data into smaller,
more manageable tables that are linked together using
relationships. There are several different types of data
normalization, each with its own set of rules. The most
common types of data normalization are: First normal
form (1NF): Each attribute in a table must be atomic,
meaning that it cannot be divided into smaller parts.
Additionally, each record in a table must be unique.
Second normal form (2NF): All non-key attributes in
a table must be fully dependent on the primary key.
This means that they cannot be dependent on only a
part of the primary key. Third normal form (3NF):
All non-key attributes in a table must be directly
dependent on the primary key. This means that they
cannot be transitively dependent on the primary key
through another non-key attribute. Here is an
example of a data normalization technique, using
the first normal form (1NF):
StudentID Name Address Courses
1 John Doe 123 Main Math,
Street English
2 John Doe 456 Elm Science,
Street History
Normalized table:
Student ID Name Address
1 John Doe 123 Main
Street
2 John Doe 456 Elm
Street
Course ID Student ID Course Name
1 1 Math
2 1 English
3 2 Science
4 2 History
The original table is not in 1NF because the Courses
attribute contains multiple values for each record. This
is called a repeating group. To normalize the table, we
split the Courses attribute into a separate table,
Courses, and link the two tables using the Student ID
attribute. Now, each record in both tables is unique
and each attribute is atomic. The database is also
easier to maintain and update, as we can now add or
remove courses without having to edit the student
records. Data normalization is a complex topic, but it is
an important skill for any data scientist or database
administrator to have. By normalizing your data, you
can improve the quality and integrity of your database,
and make it easier to use and maintain.
3) What is data? Explain types of data.
> Data is any collection of information that has been
organized so that it can be processed and analyzed.
Data can be quantitative or qualitative, structured or
unstructured, and big or small. Quantitative data is
numerical data that can be measured and analyzed
using mathematical methods. Examples of quantitative
data include customer ages, product sales, and
employee salaries. Qualitative data is non-numerical
data that represents descriptions, characteristics, or
opinions. Examples of qualitative data include
customer reviews, social media posts, and product
descriptions. Structured data is data that is organized
in a predefined format, such as a database or
spreadsheet. This makes it easy to store, query, and
analyze the data. Examples of structured data include
customer contact information, product catalogs, and
financial transactions. Unstructured data is data that
does not have a predefined format. This type of data
can be more difficult to store, query, and analyze, but
it can also contain valuable insights. Examples of
unstructured data include images, videos, and text
documents. Big data is a term used to describe
datasets that are too large and complex to be
processed using traditional data processing methods.
Big data datasets can contain billions or even trillions
of data points. Examples of big data include data from
social media, sensors, and financial markets.
Kurtosis is a statistical measure of the shape of a
probability distribution. It describes how peaked or flat
the distribution is, and how thick or thin the tails are.
There are three main types of kurtosis: Mesokurtic: A
mesokurtic distribution has a normal kurtosis, meaning
that it has a moderate peak and tails that are neither
too thick nor too thin. Leptokurtic: A leptokurtic
distribution has a positive kurtosis, meaning that it has
a sharp peak and heavy tails. This type of distribution
is often associated with outliers. Platykurtic: A
platykurtic distribution has a negative kurtosis,
meaning that it has a flat peak and light tails. This type
of distribution is often associated with data that is
tightly clustered around the mean. Kurtosis can be
used to identify anomalies in data, to assess the risk of
an investment, and to understand the behaviour of
complex systems. For example, a leptokurtic
distribution may indicate that an investment is more
likely to experience extreme returns, either positive or
negative.
5) What is Box plot? Decribe the process to
identify an outlier with Box plot.
> A box plot, also known as a box-and-whisker plot, is
a graphical method for displaying the distribution of
data. It is a useful tool for identifying outliers and
understanding the spread of the data. A box plot is
constructed by first calculating the following statistics:
Median: The middle value in the sorted dataset
First quartile (Q1): The median of the lower half of
the dataset Third quartile (Q3): The median of the
upper half of the dataset Interquartile range (IQR):
The difference between the third and first quartiles
(IQR = Q3 - Q1) The box plot is then constructed as
follows: A box is drawn from Q1 to Q3. A line is drawn
through the box at the median. Whiskers are drawn
from the box to the minimum and maximum values in
the dataset, or to 1.5 IQRs from the box, whichever is
smaller. Any data points that fall outside of the
whiskers are considered to be outliers. Outliers are
plotted as circles outside of the whiskers. Box plots are
a simple but effective way to identify outliers and
understand the distribution of data. They are often
used in data science and other fields to explore and
analyze data. Here are the steps to identify an outlier
with a box plot: Calculate the median, first quartile,
third quartile, and interquartile range of the data.
Construct the box plot by drawing a box from the first
quartile to the third quartile, a line through the box at
the median, and whiskers from the box to the minimum
and maximum values in the dataset, or to 1.5 IQRs
from the box, whichever is smaller. Identify any data
points that fall outside of the whiskers. These data
points are considered to be outliers.
6) Explain the terms data, information and
knowledge.
> Data is raw, unprocessed facts. It can be anything
from numbers and letters to images and videos. Data is
often collected and stored in databases or
spreadsheets. Information is data that has been
processed and organized so that it has meaning. It is
the result of giving context and interpretation to data.
For example, the number "100" is data, but the
statement "The average temperature today was 100
degrees Fahrenheit" is information. Knowledge is the
understanding of how to use information to solve
problems or make decisions. It is the ability to apply
information to real-world situations. For example,
knowing that the average temperature today was 100
degrees Fahrenheit might help you decide whether or
not to go for a walk. Here is a simple analogy to help
understand the difference between data, information,
and knowledge: Data is like a pile of bricks.
Information is like a blueprint for a house. Knowledge
is like the ability to build a house using the blueprint.
Data is the foundation of information and knowledge.
Without data, there would be nothing to process or
understand. Information is the bridge between data
and knowledge. It provides the context and
interpretation that is needed to turn data into
knowledge. Knowledge is the ultimate goal of data
science. It is the ability to use data to solve problems
and make better decisions.
10) Explain MapReduce Architecture?
MapReduce and HDFS are the two major components
of Hadoop which makes it so powerful and efficient to
use. MapReduce is a programming model used for
efficient processing in parallel over large data-sets in a
distributed manner. The data is first split and then
combined to produce the final result. The libraries for
MapReduce is written in so many programming
languages with various different-different
optimizations. The purpose of MapReduce in Hadoop is
to Map each of the jobs and then it will reduce it to
equivalent tasks for providing less overhead over the
cluster network and to reduce the processing power.
The MapReduce task is mainly divided into two phases
Map Phase and Reduce Phase Components of
MapReduce Architecture: Client: The MapReduce
client is the one who brings the Job to the MapReduce
for processing. There can be multiple clients available
that continuously send jobs for processing to the
Hadoop MapReduce Manager. Job: The MapReduce
Job is the actual work that the client wanted to do
which is comprised of so many smaller tasks that the
client wants to process or execute. Hadoop
MapReduce Master: It divides the particular job into
subsequent job-parts. Job-Parts: The task or sub-jobs
that are obtained after dividing the main job. The
result of all the job-parts combined to produce the final
output. Input Data: The data set that is fed to the
MapReduce for processing. Output Data: The final
result is obtained after the processing.
package.
> Data wrangling is the process of cleaning,
transforming, and manipulating data to prepare it for
analysis. It is a crucial step in any data science project,
as it ensures that the data is of high quality and in a
format that can be easily analyzed. Data wrangling can
be a complex and time-consuming process, but it is
essential for producing accurate and meaningful
results. The specific steps involved in data wrangling
will vary depending on the nature of the data and the
desired outcome. However, some common tasks
include: Identifying and removing errors: This may
involve correcting typos, removing duplicate records,
and handling missing values. Transforming the data:
This may involve changing the format of the data,
converting units of measurement, and creating new
variables. Cleaning the data: This may involve
removing outliers, identifying and correcting
inconsistencies, and normalizing the data. There are a
number of software packages that can be used for data
wrangling. One popular package is Pandas, a Python
library that provides high-performance, easy-to-use
data structures and data analysis tools.
import pandas as pd
# Create a DataFrame
df = pd.DataFrame({'name': ['Alice', 'Bob',
'Carol'], 'age': [25, 30, 35]})
# Identify and remove duplicate records
df = df.drop_duplicates()
# Transform the data by changing the format of
the 'age' column
df['age'] = df['age'].astype('int32')
# Clean the data by removing outliers
df = df.loc[df['age'] < 40]
# Normalize the data by scaling the 'age' column
df['age'] = df['age'] / df['age'].max()
# Print the DataFrame
print(df)
output
name age
0 Alice 0.625
1 Bob 0.75
2 Carol 0.875
8) Explain data, information and knowledge
triangle.
> The data, information, and knowledge (DIKW)
triangle is a model that illustrates the relationship
between data, information, and knowledge in the
context of data science. The triangle is often depicted
as a pyramid, with data at the bottom and knowledge
at the top. Data is the raw, unprocessed facts that are
collected. It can be in any format, such as numbers,
letters, images, or videos. Information is data that has
been processed and organized so that it has meaning.
It is the result of giving context and interpretation to
data. Knowledge is the understanding of how to use
information to solve problems or make decisions. It is
the ability to apply information to real-world situations.
The DIKW triangle shows that data is the foundation
for information and knowledge. Without data, there
would be nothing to process or understand.
Information is the bridge between data and
knowledge. It provides the context and interpretation
that is needed to turn data into knowledge. Knowledge
is the ultimate goal of data science. It is the ability to
use data to solve problems and make better decisions.
The DIKW triangle is a useful model for understanding
the different stages of data science projects. Data
scientists typically start by collecting and cleaning
data. Once the data is clean, they can begin to process
and analyze it to extract information. The information
can then be used to generate knowledge, which can be
used to solve problems or make decisions.
9) Explain the process of Web crawling
> Web crawling, also known as web spidering or web
indexing, is the process of automatically exploring the
World Wide Web and downloading its pages to create a
copy of the web. Web crawlers, or spiders, are
programs that systematically browse the web, starting
from a list of known URLs, and following hyperlinks to
discover new pages. Web crawling is an essential
process in data science, as it allows researchers to
collect large datasets from the web. These datasets
can be used for a variety of purposes, such as:
Search engine optimization (SEO): Web crawlers
are used by search engines to index the web and
provide relevant results to users. Market research:
Web crawling can be used to collect data on consumer
behavior, product sentiment, and market trends.
Natural language processing (NLP): Web crawling
can be used to collect large corpora of text data, which
can be used to train NLP models. Machine learning:
Web crawling can be used to collect data to train
machine learning models for a variety of tasks, such as
image classification, spam filtering, and fraud
detection.
10) What is NoSQL? Briefly explain its types.
> NoSQL is an umbrella term for non-relational
database management systems (DBMS). NoSQL
databases are designed to handle large volumes of
data that are difficult to store and manage in relational
databases. NoSQL databases are typically classified
into four types: Document stores: Document stores
store data in JSON, XML, or other document-oriented
formats. Document stores are flexible and scalable,
and they are often used to store web application data.
Key-value stores: Key-value stores store data as key-
value pairs. Key-value stores are simple and efficient,
and they are often used for caching and session
management. Column-family stores: Column-family
stores store data in columns, which makes them well-
suited for analytical workloads. Column-family stores
are scalable and fault-tolerant, and they are often used
for big data applications. Graph databases: Graph
databases store data as nodes and edges, which makes
them well-suited for representing relationships
between data points. Graph databases are scalable and
efficient, and they are often used for social network
analysis and fraud detection. MongoDB is a document
store database that is popular among web developers
and data scientists. MongoDB is easy to use and scale,
and it offers a variety of features that make it well-
suited for storing and managing complex data.
11) What is collection in MongoDB? Give an
example to create collection MongoDB.
A collection in MongoDB is a database object that
stores documents. A document is a JSON-like object
that can contain any type of data, such as strings,
numbers, arrays, and embedded documents.
Collections are similar to tables in relational
databases, but they are more flexible and scalable.
Collections can be dynamically created and expanded,
and they can store any type of data, regardless of its
structure. To create a collection in MongoDB, you can
use the db.createCollection() command. For
example, to create a collection called users, you would
use the following command:
db.createCollection("users")
You can also create a collection by inserting a
document into a non-existent collection. MongoDB will
automatically create the collection if it does not
already exist. For example, to create a collection called
products and insert a document into it, you would use
the following command:
db.products.insertOne({
name: "Product 1",
price: 100,
quantity: 10
}) Once you have created a collection, you can start
adding more documents to it. You can also query and
update the documents in a collection using the
MongoDB query language.
12) Write note on XPath.
> XPath (XML Path Language) is a language for
navigating around XML documents. It uses a path-like
syntax to identify nodes in an XML document. XPath
can be used to select nodes, extract data from nodes,
and modify nodes. XPath is used in a variety of
applications, including: XSLT (Extensible Stylesheet
Language Transformations): XSLT is a language for
transforming XML documents into other formats, such
as HTML or PDF. XPath is used in XSLT to select the
nodes in the input XML document that are to be
transformed. XQuery (XML Query Language):
XQuery is a language for querying XML documents.
XPath is used in XQuery to select the nodes in the XML
document that are to be returned by the query.
XML Schema: XML Schema is a language for defining
the structure and content of XML documents. XPath is
used in XML Schema to specify the constraints on the
nodes in an XML document. XPath Syntax: XPath
expressions use a path-like syntax to identify nodes in
an XML document. The path starts at the root element
of the document and then follows a sequence of steps
to the desired node. Each step in the path consists of a
node type, a predicate, and a list of axis specifiers. The
node type specifies the type of node to select, such as
element, attribute, or text node. The predicate is an
optional expression that can be used to filter the nodes
that are selected. The axis specifiers specify the
relationship between the current node and the node to
be selected. XPath Example: The following XPath
expression selects all of the element nodes with the
name product that are descendants of the element
node with the name category: /category//product
13) Write a note on HBase.
> HBase is a distributed, column-oriented NoSQL
database that runs on top of Hadoop. It is designed for
storing and querying large amounts of data, such as
log files, web pages, and social media data. HBase is
modeled after Google's Bigtable database, and it
provides similar features, such as scalability, fault
tolerance, and real-time access to data. HBase stores
data in tables, which are similar to tables in relational
databases. However, HBase tables are column-
oriented, meaning that data is stored in columns
rather than rows. This makes HBase well-suited for
querying data by column, which is a common operation
in big data applications. HBase also provides a number
of features that make it well-suited for real-time
applications. For example, HBase can be configured to
flush data to disk asynchronously, which means that
data can be written to HBase and made available for
querying immediately. Additionally, HBase supports
atomic read and write operations, which means that
multiple applications can read and write to the same
data without causing conflicts. HBase is used by a
variety of companies to store and query large amounts
of data. For example, Facebook uses HBase to store
and query user data, Twitter uses HBase to store and
query tweets, and Yahoo uses HBase to store and
query search logs.
14) Discuss the 5 V's of Data.
> The 5 V's of Data are a framework for understanding
the characteristics of big data. They are:
Volume: The amount of data that is being generated
and stored. Velocity: The speed at which data is being
generated and processed. Variety: The different types
of data that are being generated and stored. Veracity:
The accuracy and reliability of the data. Value: The
usefulness of the data for analysis and decision-
making. Volume: The volume of data is growing
rapidly. In 2023, it is estimated that the world will
generate 97 zettabytes of data. This is an enormous
amount of data, and it is difficult to store and process
using traditional methods. Velocity: The velocity of
data is also increasing. Data is now being generated in
real time from a variety of sources, such as social
media, sensors, and financial transactions. This real-
time data can be used to power applications such as
fraud detection and stock trading. Variety: The variety
of data is also increasing. Traditional data sources,
such as relational databases, typically store structured
data. However, new data sources, such as social media
and sensors, are generating unstructured data.
Unstructured data is more difficult to store and
process than structured data, but it can contain
valuable insights. Veracity: The veracity of data is
important for ensuring that the results of analysis are
accurate and reliable. Data can be inaccurate or
incomplete for a variety of reasons, such as human
error, sensor errors, and data corruption. It is
important to clean and validate data before using it for
analysis. Value: The value of data is the most
important V of all. Data is only valuable if it can be
used to generate insights and make better decisions.
Data scientists use a variety of techniques to extract
value from data, such as machine learning, statistical
analysis, and data visualization. The 5 V's of Data are a
useful framework for understanding the challenges
and opportunities of big data. By understanding the
different characteristics of big data, organizations can
develop the right tools and processes to manage and
analyze their data effectively.
15) How to create indexes in MongoDB? Give
example.
> To create an index in MongoDB, you can use the
createIndex() method. This method takes a document
as its argument, which specifies the fields to be
indexed and the sort order for each field. For example,
to create an ascending index on the name field of the
users collection, you would use the following
command: db.users.createIndex({ name: 1 });
To create a compound index on the name and age
fields, you would use the following command:
db.users.createIndex({ name: 1, age: 1 });
You can also create a unique index, which prevents
duplicate values from being stored in the indexed field.
To create a unique index, you would set the unique
option to true. For example, to create a unique index
on the email field of the users collection, you would
use the following command: db.users.createIndex({
email: 1 }, { unique: true });
const MongoClient =
require('mongodb').MongoClient;
const client = new
MongoClient('mongodb://localhost:27017');
client.connect(function(err, db) {
if (err) throw err;
const usersCollection = db.collection('users');
usersCollection.createIndex({ name: 1 }, {
unique: true }, function(err, indexName) {
if (err) throw err;
console.log('Index created successfully: ' +
indexName);
client.close();
});
});
16) What is MongoDB? State its features.
> MongoDB is a NoSQL document database that
stores data in flexible, JSON-like documents. It is a
popular choice for modern applications because it is
scalable, performant, and easy to use.
Features of MongoDB: Document model: MongoDB
stores data in documents, which are self-contained
units of data that can be easily queried and updated.
This makes MongoDB a good choice for applications
that need to store complex data structures. Schema-
less: MongoDB does not require a predefined schema,
which gives developers more flexibility in how they
store and manage their data. Horizontal scaling:
MongoDB can be scaled horizontally by adding more
servers to a cluster. This makes it well-suited for
applications that need to handle large amounts of data
and traffic. Replication: MongoDB supports
replication, which allows you to create multiple copies
of your data for high availability and disaster recovery.
Performance: MongoDB is a very performant
database, especially for read-heavy applications.
23) Write a short note on Smoothing by mean
technique. in data science
> Smoothing by mean technique is a data smoothing
technique that uses the mean of the neighboring data
points to smooth the data. This technique is often used
to reduce noise and improve the readability of data. To
smooth data using the mean technique, first calculate
the mean of the neighboring data points. Then, replace
the original data point with the mean. This process is
repeated for all of the data points in the dataset. The
number of neighboring data points to use depends on
the desired level of smoothing. A higher number of
neighboring data points will result in smoother data,
but it will also lose more detail. Smoothing by mean
technique is a simple and effective data smoothing
technique that can be used to improve the readability
and accuracy of data. It is often used in a variety of
data science applications, such as time series analysis,
forecasting, and machine learning.
17) How to create, use, show and delete
databases in MongoDB? Give example.
> Create a database in MongoDB
To create a database in MongoDB, you can use the use
command. This command will create the database if it
does not already exist, or switch to the database if it
does exist. For example, to create a database named
my_database, you would use the following command:
use my_database MongoDB will prompt you with a
message that it has switched to the new database.
Use a database in MongoDB: Once you have created
a database, you can use the use command to switch to
it. For example, to switch to the my_database
database, you would use the following command:
use my_database You can also use the db object to
reference the current database. For example, to get a
list of all the collections in the current database, you
would use the following code:
db.listCollections().toArray(function(err,
collections) {
console.log(collections);
}); Show databases in MongoDB To show a list of all
the databases in MongoDB, you can use the show dbs
command. This command will print a list of all the
database names to the console. For example, to show a
list of all the databases, you would use the following
command: show dbs Delete a database in MongoDB
To delete a database in MongoDB, you can use the
db.dropDatabase() method. This method will delete
the database and all of its collections. For example, to
delete the my_database database, you would use the
following command: db.dropDatabase() MongoDB
will prompt you to confirm that you want to delete the
database.
18) Explain in detail homogeneous distributed
database and heterogeneous distributed
database.
> A homogeneous distributed database is a distributed
database system in which all nodes are identical. This
means that all nodes run the same database
management system (DBMS) software and store the
same database schema. Homogeneous distributed
databases are relatively easy to manage and maintain,
as there is no need to worry about compatibility issues
between different DBMS software or database
schemas. Examples of homogeneous distributed
database systems include: MySQL Cluster. PostgreSQL
Cluster. Oracle Real Application Clusters (RAC).
A heterogeneous distributed database is a distributed
database system in which different nodes can run
different DBMS software and store different database
schemas. This makes heterogeneous distributed
databases more flexible and versatile than
homogeneous distributed databases, as they can be
used to store and manage a wider variety of data.
However, heterogeneous distributed databases can be
more complex to manage and maintain, as there is a
need to ensure compatibility between different DBMS
software and database schemas. Examples of
heterogeneous distributed database systems include:
Apache Hadoop HBase. Apache Hive. Apache Spark
SQL.
19) Write a short note on Hadoop Architecture.
State its advantages.
> Hadoop is a distributed computing framework that
provides a way to process large datasets in a scalable
and fault-tolerant manner. Hadoop is based on the
MapReduce programming model, which breaks down a
large processing job into smaller tasks that can be
executed in parallel on multiple nodes. The Hadoop
architecture consists of two main components:
Hadoop Distributed File System (HDFS): HDFS is
a distributed file system that stores data across
multiple nodes in the cluster. HDFS is highly scalable
and fault-tolerant, as it replicates data across multiple
nodes to ensure that data is not lost if a node fails.
YARN: YARN is a resource management framework
that allocates resources to Hadoop applications. YARN
is responsible for scheduling and monitoring Hadoop
jobs, and it ensures that jobs are completed efficiently.
Advantages of Hadoop Architecture: Scalability:
Hadoop is highly scalable, as it can scale horizontally
by adding more nodes to the cluster. This makes
Hadoop well-suited for processing large datasets.
Fault tolerance: Hadoop is fault-tolerant, as it
replicates data across multiple nodes in the cluster.
This means that Hadoop applications can continue to
process data even if a node fails. Cost-effectiveness:
Hadoop is a cost-effective solution for processing large
datasets, as it can run on commodity hardware.
Flexibility: Hadoop is a flexible platform that can be
used to process a variety of data types, including
structured, unstructured, and semi-structured data.
20) What is JSON? How to read JSON file in R
with an example?
> JSON (JavaScript Object Notation) is a lightweight
data-interchange format that is easy for humans to
read and write. It is based on a subset of JavaScript
syntax, but it is language-independent and can be used
with any programming language. JSON is used to store
and exchange data between different systems and
applications. It is a popular format for web APIs, and it
is also used in many other applications, such as
databases, configuration files, and log files. To read a
JSON file in R, you can use the jsonlite package. The
jsonlite package provides a simple and efficient way
to read and write JSON data in R. Here is an example
of how to read a JSON file in R:
Code snippet # Load the `jsonlite` package.
library(jsonlite) # Read the JSON file.
json_data <- fromJSON("example.json")
# Print the JSON data.
print(json_data) Use code with caution. Learn more
The fromJSON() function takes the path to the JSON
file as input and returns the JSON data as a list in R.
30) Write a short note on Ensemble Methods.
> Ensemble methods are a type of machine learning
algorithm that combines multiple models to produce a
more accurate and robust prediction. Ensemble
methods work by training multiple models on the same
data and then averaging their predictions. This
averaging process helps to reduce the variance of the
model and improve its overall performance. There are
a number of different ensemble methods, including:
Bagging: Bagging works by creating multiple subsets
of the training data and training a model on each
subset. The predictions of the models are then
averaged to produce the final prediction. Boosting:
Boosting works by training a sequence of models,
where each model is trained on the residuals of the
previous model. The predictions of the models are then
weighted and averaged to produce the final prediction.
Stacking: Stacking works by training a meta-model on
the predictions of multiple base models. The meta-
model is then used to make the final prediction.
Ensemble methods can be used for both classification
and regression tasks. They are often used in
applications where high accuracy is required, such as
fraud detection and medical diagnosis. Advantages of
ensemble methods: Ensemble methods are more
accurate than individual models. Ensemble methods
are more robust to noise and outliers in the data.
Ensemble methods can be used to improve the
performance of any type of machine learning model.
Disadvantages of ensemble methods: Ensemble
methods can be computationally expensive to train.
Ensemble methods can be difficult to interpret.
31) Explain Forecasting. List the steps in
forecasting.
> Forecasting is the process of predicting future
events or trends based on historical data and current
conditions. It is used in a wide variety of fields,
including business, economics, finance, and
meteorology. Steps in forecasting: Identify the goal
of the forecast. What are you trying to predict? What
is the time horizon of the forecast? Gather historical
data. This data should be relevant to the goal of the
forecast and should cover a sufficiently long period of
time. Clean and prepare the data. This may involve
removing outliers, filling in missing values, and
transforming the data into a format that is suitable for
forecasting. Choose a forecasting method. There
are a variety of forecasting methods available, each
with its own strengths and weaknesses. The best
method to use will depend on the nature of the data
and the goal of the forecast. Build the forecasting
model. This involves fitting the forecasting method to
the historical data. Evaluate the forecasting model.
This involves testing the model on a held-out dataset to
assess its accuracy. Make the forecast. Once the
forecasting model has been evaluated and found to be
accurate, it can be used to make predictions for the
future. Monitor the forecast and update it as
needed. Forecasts should be monitored regularly and
updated as new data becomes available or as
conditions change. Popular forecasting methods:
Time series analysis: This method uses historical
data to identify patterns and trends. Causal
forecasting: This method uses relationships between
variables to make predictions. Judgmental
forecasting: This method uses human judgment and
expertise to make predictions.
32) What is AIC, BIC? State their mathematical
formula.
> Akaike information criterion (AIC) and Bayesian
information criterion (BIC) are two statistical model
selection criteria that are used to compare different
models of the same data. They both penalize models
with more parameters, but the BIC penalizes more
strongly. AIC formula: AIC = 2k - 2 * ln(L)
where: k is the number of parameters in the model
L is the likelihood of the model
BIC formula: BIC = -2 * ln(L) + k * ln(n)
where: k is the number of parameters in the model
L is the likelihood of the model n is the number of data
points in the training data. Both AIC and BIC can be
used for both classification and regression tasks. They
are often used in conjunction with other model
selection criteria, such as cross-validation.
Advantages of AIC and BIC: AIC and BIC are
relatively easy to calculate and interpret. AIC and BIC
are consistent model selection criteria, which means
that they will select the true model with probability 1
in the limit as the sample size increases.
Disadvantages of AIC and BIC: AIC and BIC can be
sensitive to the choice of the prior distribution.
AIC and BIC can be computationally expensive to
calculate for large datasets.
33) Give the formula for information gain &
entropy.
> Information gain is a measure of how much
information about the target variable is gained by
knowing the value of a predictor variable. It is
calculated as the difference between the entropy of the
target variable before and after partitioning the data
based on the predictor variable. The formula for
information gain is as follows:
IG(T | X) = H(T) - H(T | X)
where: IG(T | X) is the information gain of the target
variable T given the predictor variable X. H(T) is the
entropy of the target variable T before partitioning the
data based on the predictor variable X. H(T | X) is the
entropy of the target variable T after partitioning the
data based on the predictor variable X
Entropy: Entropy is a measure of the uncertainty in a
distribution. It is calculated as the sum of the
probabilities of each possible outcome, weighted by
the logarithm of those probabilities. The formula for
entropy is as follows: H(T) = -sum(p(t) * log2(p(t)))
where: H(T) is the entropy of the target variable T
p(t) is the probability of the target variable T taking on
the value t
26) What is Decision tree? What are its
advantages?
A decision tree is a machine learning algorithm that
uses a tree-like structure to make predictions. Each
node in the tree represents a feature of the data, and
each branch represents a possible decision. The tree is
constructed by starting at the root node and then
recursively splitting the data into smaller subsets
based on the values of the features. The process stops
when all of the data points in a subset have the same
label. Once the tree is constructed, it can be used to
make predictions on new data points by starting at the
root node and following the branches down the tree
based on the values of the features in the new data
point. The prediction is made at the leaf node where
the tree terminates. Decision trees have a number
of advantages, including: Interpretability: Decision
trees are very interpretable, which means that it is
easy to understand how the tree makes predictions.
This is because the tree can be visualized as a
flowchart. Robustness: Decision trees are robust to
noise and outliers in the data. This is because the tree
is constructed by splitting the data into smaller
subsets, which helps to reduce the impact of noise and
outliers. Efficiency: Decision trees are efficient to
train and predict with. This is because the tree can be
constructed recursively, and the predictions can be
made by following a simple path down the tree.
27) Explain Multiple Linear Regression.
> Multiple linear regression (MLR) is a statistical
technique that uses two or more independent variables
to predict the outcome of a dependent variable. It is an
extension of ordinary least-squares (OLS) regression,
which uses only one independent variable. MLR is
used in a wide variety of fields, including economics,
finance, marketing, and healthcare. It can be used to
answer a wide range of questions, such as: How does
income affect spending? How does advertising affect
sales? How do patient characteristics affect health
outcomes? MLR works by fitting a linear equation to
the data. The equation has the following form:
y = b0 + b1 * x1 + b2 * x2 + ... + bk * xk
where: y is the dependent variable
x1, x2, ..., xk are the independent variables
b0, b1, b2, ..., bk are the coefficients of the regression
model. The coefficients of the regression model are
estimated using a variety of methods, such as ordinary
least squares (OLS) or maximum likelihood estimation
(MLE). Once the coefficients of the regression model
have been estimated, the model can be used to predict
the value of the dependent variable for a new set of
independent variables. Advantages of multiple
linear regression: MLR is a powerful tool for
understanding the relationships between variables.
MLR can be used to make predictions about the
dependent variable. MLR is relatively easy to interpret.
Disadvantages of multiple linear regression: MLR
can be sensitive to outliers and non-linear
relationships in the data. MLR can be computationally
expensive to fit for large datasets.
28) Write note on Bias/Variance Tradeoff?
> The bias-variance tradeoff is a fundamental concept
in machine learning. It describes the relationship
between two sources of error in a predictive model:
bias and variance. Bias is the error due to the
simplifying assumptions made by the model to make
the target function more straightforward to
approximate. A model with high bias will underfit the
training data and will not be able to generalize well to
new data. Variance is the error due to the model's
sensitivity to small fluctuations in the training data. A
model with high variance will overfit the training data
and will not be able to generalize well to new data. The
bias-variance tradeoff states that it is impossible to
build a model with both zero bias and zero variance.
Any model will have some degree of bias and variance,
and the goal is to find a balance between the two that
minimizes the overall error of the model. How to
reduce bias and variance? There are a number of
techniques that can be used to reduce bias and
variance in machine learning models.
To reduce bias: Use a more complex model. However,
this can lead to overfitting. Use regularization
techniques to penalize complex models.
Use more training data. To reduce variance: Use a
simpler model. However, this can lead to underfitting.
Use bagging or boosting techniques. Use more
training data.
29) Explain hierarchical clustering.
> Hierarchical clustering is a type of unsupervised
machine learning algorithm that groups similar data
points together. It does this by creating a hierarchy of
clusters, with each cluster containing more similar
data points than the clusters above it. Hierarchical
clustering can be performed using two different
approaches: Agglomerative clustering: This
approach starts by treating each data point as its own
cluster. It then iteratively merges the most similar
clusters until a desired number of clusters is reached.
Divisive clustering: This approach starts with all of
the data points in a single cluster. It then iteratively
splits the cluster into smaller clusters until a desired
number of clusters is reached. Hierarchical clustering
is often visualized using a dendrogram, which is a tree-
like structure that shows the relationships between the
clusters. Advantages of hierarchical clustering:
Hierarchical clustering is a versatile algorithm that
can be used to cluster data of any type. Hierarchical
clustering is able to identify clusters of different
shapes and sizes. Hierarchical clustering can be used
to explore the underlying structure of a dataset.
Disadvantages of hierarchical clustering:
Hierarchical clustering can be computationally
expensive for large datasets. Hierarchical clustering
can be sensitive to the distance metric that is used.
Hierarchical clustering does not produce a unique set
of clusters.
21) Write a short note on AWS.
> Amazon Web Services (AWS) is a cloud computing
platform that offers a broad set of global compute,
storage, database, analytics, application, and
deployment services that help organizations move
faster, lower IT costs, and scale applications. AWS
offers over 200 fully featured services from data
centers globally. Millions of customers — including the
fastest-growing startups, largest enterprises, and
leading government agencies — rely on AWS to power
their most critical applications, all with high
availability, scalability, and reliability. AWS is the
world's leading cloud platform, and it is used by a wide
variety of companies, from small businesses to large
enterprises. AWS offers a wide range of services,
including computing, storage, networking, databases,
analytics, machine learning, and artificial intelligence.
Benefits of using AWS: Scalability: AWS is highly
scalable, and it can be used to scale applications up or
down as needed. This makes AWS well-suited for
applications with variable workloads. Reliability: AWS
is a reliable platform, and it offers high availability for
applications. AWS also offers a variety of features to
help improve the reliability of applications, such as
disaster recovery and load balancing. Security: AWS
is a secure platform, and it offers a variety of security
features to help protect applications and data. AWS
also regularly audits its security practices and
infrastructure. Cost-effectiveness: AWS is a cost-
effective platform, and it offers a variety of pricing
options to help customers save money. AWS also offers
a variety of discounts for customers who commit to
using AWS services for a long period of time.
22) Explain any 3 ways to do web scraping.
> There are many different ways to do web scraping,
but three common methods are: 1. Using a web
browser extension: There are a number of web
browser extensions available that can be used to
scrape data from websites. These extensions typically
allow you to select the elements on the page that you
want to scrape, and then export the data to a file or
database. Some popular web browser extensions for
web scraping include: Scraper (Chrome) Octoparse
(Chrome, Firefox) Web Scraper (Chrome, Firefox)
2. Using a web scraping library: There are a
number of web scraping libraries available for
different programming languages. These libraries
provide a variety of features for scraping data from
websites, such as parsing HTML and CSS, handling
dynamic content, and exporting data to different
formats. Some popular web scraping libraries include:
Beautiful Soup (Python) Scrapy (Python) PyQuery
(Python) Cheerio (JavaScript). Puppeteer (JavaScript)
3. Using a web scraping API: There are a number of
web scraping APIs available that can be used to scrape
data from websites. These APIs typically provide a
simple and easy-to-use interface for scraping data, and
they can be used from any programming language.
Some popular web scraping APIs include: ScraperAPI
ProxyCrawl. BrightData. Apify.
24) What is K-NN? Explain with the help of an
example.
> K-nearest neighbors (K-NN) is a supervised machine
learning algorithm that can be used for both
classification and regression tasks. It works by finding
the K most similar instances in the training data to a
new instance and then using the labels of those K
instances to predict the label of the new instance.
Example: Suppose we have a dataset of images of cats
and dogs, and we want to train a K-NN classifier to
classify new images as either cat or dog. We would
first train the classifier on a set of labeled images (i.e.,
images that have already been labeled as either cat or
dog). To classify a new image, the classifier would first
find the K most similar images in the training data to
the new image. The similarity between two images can
be measured using a variety of metrics, such as the
distance between the two images in pixel space. Once
the classifier has found the K most similar images in
the training data, it would then predict the label of the
new image based on the labels of those K images. For
example, if the majority of the K most similar images
are cats, then the classifier would predict that the new
image is also a cat.
25) Explain Bayesian Information Criterion.
> The Bayesian Information Criterion (BIC), also
known as the Schwarz criterion, is a statistical model
selection criterion that is used to compare different
models of data. The BIC is based on the likelihood
function of the model, and it penalizes models with
more parameters. This is because models with more
parameters are more likely to overfit the training data.
The BIC is calculated as follows:
BIC = -2 * ln(L) + k * ln(n)
where:
L is the likelihood of the model
k is the number of parameters in the model
n is the number of data points in the training data
The BIC is a relative measure, which means that it can
only be used to compare different models of the same
data. The model with the lowest BIC is the model that
is most likely to generalize well to new data. The BIC is
often used in conjunction with other model selection
criteria, such as the Akaike Information Criterion
(AIC). The AIC is similar to the BIC, but it does not
penalize models with more parameters as strongly.
Advantages of the BIC: The BIC is a consistent
model selection criterion, which means that it will
select the true model with probability 1 in the limit as
the sample size increases. Disadvantages of the BIC:
The BIC can be sensitive to the choice of the prior
distribution. The BIC can be computationally expensive
to calculate for large datasets.