CS3362 - Data Science Laboratory - Manual - Final-1

CS3362 DATA SCIENCE LABORATORY LTPC
0 04 2
COURSE OBJECTIVES:
 To understand the python libraries for data science
 To understand the basic Statistical and Probability measures for data science.
 To learn descriptive analytics on the benchmark data sets.
 To apply correlation and regression analytics on standard data sets.
 To present and interpret data using visualization packages in Python.
LIST OF EXPERIMENTS:
1. Download, install and explore the features of NumPy, SciPy, Jupyter, Statsmodels
and Pandas packages.
2. Working with Numpy arrays
3. Working with Pandas data frames
4. Reading data from text files, Excel and the web and exploring various commands
for doing descriptive analytics on the Iris data set.
5. Use the diabetes data set from UCI and Pima Indians Diabetes data set for
performing the following:
a. Univariate analysis: Frequency, Mean, Median, Mode, Variance, Standard
Deviation, Skewness and Kurtosis.
b. Bivariate analysis: Linear and logistic regression modeling
c. Multiple Regression analysis
d. Also compare the results of the above analysis for the two data sets.
6. Apply and explore various plotting functions on UCI data sets.
a. Normal curves
b. Density and contour plots
c. Correlation and scatter plots
d. Histograms
e. Three dimensional plotting
7. Visualizing Geographic Data with Basemap
CS3362 Data Science Laboratory 1

List of Equipments: (30 Students per Batch)
Tools: Python, Numpy, Scipy, Matplotlib, Pandas, statmodels, seaborn, plotly, bokeh
Note: Example data sets like: UCI, Iris, Pima Indians Diabetes etc.
TOTAL: 60 PERIODS
COURSE OUTCOMES:
At the end of this course, the students will be able to:
CO1: Make use of the python libraries for data science.
CO2: Make use of the basic Statistical and Probability measures for data science.
CO3: Perform descriptive analytics on the benchmark data sets.
CO4: Perform correlation and regression analytics on standard data sets.
CO5: Present and interpret data using visualization packages in Python.

1(a). Download and install the different packages like NumPy, SciPy,
Jupyter, Statsmodels and Pandas
AIM:
To learn how to download and install the different packages of NumPy, SciPy, Jupyter,
Statsmodels and Pandas.
ALGORITHM:
1. Download Python and Jupyter.
2. Install Python and Jupyter.
3. Install the pack like NumPy, SciPy Satsmodels and Pandas.
4. Verify the proper execution of Python and Jupyter.
Python Installation
 Open the python official web site. (https://www.python.org/)
 Downloads ==> Windows ==> Select Recent Release. (Requires Windows 10 or
above versions)
 Install "python-3.10.6-amd64.exe"
Jupyter Installation
 Open command prompt and enter the following to check whether the pyton was
installed properly or not, “python –version”.
 If installation is proper it returns the version of python
 Enter the following to check whether the pyton package manager was installed
properly or not, “pip –version”
 If installation is proper it returns the version of python package manager
 Enter the following command “pip install jupyterlab”.
 Enter the following command “pip install jupyter notebook”.
 Copy the above command result from path to upgrade command and paste it and
execute for upgrade process.
 Create a folder and name the folder accordingly.
 Open command prompt and enter in to that folder. Enter the following code
“jupyter notebook” and then give enter.
 Now new jupyter notebook will be opened for our use.
pip Installation
Installation of NumPy
 pip install numpy
Installation of SciPy
 pip install scipy
Installation of Statsmodels

 pip install statsmodels
Installation of Pandas
 pip install pandas
Sample Output
RESULT:
NumPy, SciPy, Jupyter, Statsmodels and Pandas packages were installed properly and
the execution also verified.

1(b). Explore the features of NumPy
AIM:
To learn the different features provided by NumPy package.
ALGORITHM:
1. Install the NumPy package
2. Study all the features of NumPy package.
NumPy
 NumPy is a Python library used for working with arrays.
 It also has functions for working in domain of linear algebra, fourier transform,
and matrices.
Features
These are the important features of NumPy
1. Array 2. Random 3. Universal Functions
1. Arrays
1.1 Array Slicing
 Slicing in python means taking elements from one given index to another
given index.
 We pass slice instead of index like this: [start:end].
 We can also define the step, like this: [start:end:step].
 If we don't pass start its considered 0
 If we don't pass end its considered length of array in that dimension
 If we don't pass step its considered 1
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6, 7])
print(arr[1:5:2])
1.2 Array Shape & Reshaping

1.2.1 Array Shape
NumPy arrays have an attribute called shape that returns a tuple with each
index having the number of corresponding elements.
import numpy as np
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(arr.shape)
1.2.2 Array Reshaping

 Reshaping means changing the shape of an array.
 The shape of an array is the number of elements in each dimension.

By reshaping we can add or remove dimensions or change number of
elements in each dimension.
 Convert the following 1-D array with 12 elements into a 3-D array.
The outermost dimension will have 2 arrays that contains 3 arrays, each with 2
elements:
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
newarr = arr.reshape(2, 3, 2)
print(newarr)
2. Random
Random Permutations
A permutation refers to an arrangement of elements. e.g. [3, 2, 1] is a permutation of
[1, 2, 3] and vice-versa.
The NumPy Random module provides two methods for this: shuffle() and
permutation().
from numpy import random
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
random.shuffle(arr)
print(arr)
2.1 Seaborn
Seaborn is a library that uses Matplotlib underneath to plot graphs. It will be used to
visualize random distributions.
import matplotlib.pyplot as plt
import seaborn as sns
sns.distplot([0, 1, 2, 3, 4, 5])
plt.show()
2.2 Normal (Gaussian) Distribution

It is also called the Gaussian Distribution after the German mathematician Carl
Friedrich Gauss. It fits the probability distribution of many events, eg. IQ Scores, Heartbeat
etc.
It uses the random.normal() method to get a Normal Data Distribution.
It has three parameters:
loc - (Mean) where the peak of the bell exists.
scale - (Standard Deviation) how flat the graph distribution should be.
size - The shape of the returned array.
Generate a random normal distribution of size 2x3 with mean at 1 and standard
deviation of 2:
x = random.normal(loc=1, scale=2, size=(2, 3))
print(x)

2.3 Binomial Distribution
Binomial Distribution is a Discrete Distribution.
It describes the outcome of binary scenarios, e.g. toss of a coin, it will either be head
or tails.
n - number of trials.
p - probability of occurence of each trial (e.g. for toss of a coin 0.5 each).
Given 10 trials for coin toss generate 10 data points:
x = random.binomial(n=10, p=0.5, size=10)
print(x)
2.4 Poisson Distribution

It estimates how many times an event can happen in a specified time. e.g. If someone
eats twice a day what is probability he will eat thrice?
It has two parameters:
lam - rate or known number of occurences e.g. 2 for above problem.
Generate a random 1x10 distribution for occurence 2:
x = random.poisson(lam=2, size=10)
print(x)
2.5 Uniform Distribution

Used to describe probability where every event has equal chances of occuring. E.g.
Generation of random numbers.
a - lower bound - default 0 .0.
b - upper bound - default 1.0.
Create a 2x3 uniform distribution sample:
x = random.uniform(size=(2, 3))
print(x)
2.6 Logistic Distribution

Logistic Distribution is used to describe growth.
Used extensively in machine learning in logistic regression, neural networks etc.
loc - mean, where the peak is. Default 0.
scale - standard deviation, the flatness of distribution. Default 1.

Draw 2x3 samples from a logistic distribution with mean at 1 and stddev 2.0:
x = random.logistic(loc=1, scale=2, size=(2, 3))
print(x)
2.7 Multinomial Distribution

Multinomial distribution is a generalization of binomial distribution.
It describes outcomes of multi-nomial scenarios unlike binomial where scenarios must
be only one of two. e.g. Blood type of a population, dice roll outcome.
n - number of possible outcomes (e.g. 6 for dice roll).
pvals - list of probabilties of outcomes (e.g. [1/6, 1/6, 1/6, 1/6, 1/6, 1/6] for dice roll).
Draw out a sample for dice roll:
x = random.multinomial(n=6, pvals=[1/6, 1/6, 1/6, 1/6, 1/6, 1/6])
print(x)
2.8 Exponential Distribution

Exponential distribution is used for describing time till next event e.g. failure/success
etc.
scale - inverse of rate ( see lam in poisson distribution ) defaults to 1.0.
Draw out a sample for exponential distribution with 2.0 scale with 2x3 size:
x = random.exponential(scale=2, size=(2, 3))
print(x)
2.9 Chi Square Distribution

Chi Square distribution is used as a basis to verify the hypothesis.
df - (degree of freedom).
Draw out a sample for chi squared distribution with degree of freedom 2 with size 2x3:
x = random.chisquare(df=2, size=(2, 3))
print(x)
2.10 Rayleigh Distribution

Rayleigh distribution is used in signal processing.
scale - (standard deviation) decides how flat the distribution will be default 1.0).

Draw out a sample for rayleigh distribution with scale of 2 with size 2x3:
x = random.rayleigh(scale=2, size=(2, 3))
print(x)
2.11 Pareto Distribution

A distribution following Pareto's law i.e. 80-20 distribution (20% factors cause
80% outcome).
It has two parameter:
a - shape parameter.
Draw out a sample for pareto distribution with shape of 2 with size 2x3:
x = random.pareto(a=2, size=(2, 3))
print(x)
2.12 Zipf Distribution

Zipf distritutions are used to sample data based on zipf's law.
Zipf's Law: In a collection the nth common term is 1/n times of the most common
term. E.g. 5th common word in english has occurs nearly 1/5th times as of the most
used word.
a - distribution parameter.
Draw out a sample for zipf distribution with distribution parameter 2 with size 2x3:
x = random.zipf(a=2, size=(2, 3))
print(x)
3. Universal Functions
Create Your Own ufunc (Universal)
To create you own ufunc, you have to define a function, like you do with normal
functions in Python, then you add it to your NumPy ufunc library with the frompyfunc()
method.
The frompyfunc() method takes the following arguments:
function - the name of the function.
inputs - the number of input arguments (arrays).
outputs - the number of output arrays.
Create your own ufunc for addition:
import numpy as np
def myadd(x, y):
return x+y
myadd = np.frompyfunc(myadd, 2, 1)
print(myadd([1, 2, 3, 4], [5, 6, 7, 8]))

3.1 Simple Arithmetic
You could use arithmetic operators + - * / directly between NumPy arrays, but this
section discusses an extension of the same where we have functions that can take any array-
like objects e.g. lists, tuples etc. and perform arithmetic conditionally.
Addition
Add the values in arr1 to the values in arr2:
import numpy as np
arr1 = np.array([10, 11, 12, 13, 14, 15])
arr2 = np.array([20, 21, 22, 23, 24, 25])
newarr = np.add(arr1, arr2)
print(newarr)
Subtraction
Subtract the values in arr2 from the values in arr1:
import numpy as np
arr1 = np.array([10, 20, 30, 40, 50, 60])
arr2 = np.array([20, 21, 22, 23, 24, 25])
newarr = np.subtract(arr1, arr2)
print(newarr)
Multiplication
Multiply the values in arr1 with the values in arr2:
import numpy as np
arr1 = np.array([10, 20, 30, 40, 50, 60])
arr2 = np.array([20, 21, 22, 23, 24, 25])
newarr = np.multiply(arr1, arr2)
print(newarr)
Division
Divide the values in arr1 with the values in arr2:
import numpy as np
arr1 = np.array([10, 20, 30, 40, 50, 60])
arr2 = np.array([3, 5, 10, 8, 2, 33])
newarr = np.divide(arr1, arr2)
print(newarr)
Power
Raise the valules in arr1 to the power of values in arr2:
import numpy as np
arr1 = np.array([10, 20, 30, 40, 50, 60])
arr2 = np.array([3, 5, 6, 8, 2, 33])
newarr = np.power(arr1, arr2)
print(newarr)
Remainder
Return the remainders:
import numpy as np
arr1 = np.array([10, 20, 30, 40, 50, 60])

arr2 = np.array([3, 7, 9, 8, 2, 33])
newarr = np.mod(arr1, arr2)
print(newarr)
Absolute Values
Return the quotient and mod:
import numpy as np
arr = np.array([-1, -2, 1, 2, 3, -4])
newarr = np.absolute(arr)
print(newarr)
3.2 Rounding Decimals

There are primarily five ways of rounding off decimals in NumPy:
 truncation  floor
 rounding  ceil
3.2.1 Truncation
Remove the decimals, and return the float number closest to zero. Use the trunc() and
fix() functions.
Truncate elements of following array:
import numpy as np
arr = np.trunc([-3.1666, 3.6667])
print(arr)
3.2.2 Rounding
The around() function increments preceding digit or decimal by 1 if >=5 else do
nothing.
Round off 3.1666 to 2 decimal places:
import numpy as np
arr = np.around(3.1666, 2)
print(arr)
3.2.3 Floor
The floor() function rounds off decimal to nearest lower integer.
Floor the elements of following array:
import numpy as np
arr = np.floor([-3.1666, 3.6667])
print(arr)
3.2.4 Ceil
The ceil() function rounds off decimal to nearest upper integer.
Ceil the elements of following array:
import numpy as np
arr = np.ceil([-3.1666, 3.6667])
print(arr)

3.3 Logs
NumPy provides functions to perform log at the base 2, e and 10.
We will also explore how we can take log for any base by creating a custom ufunc.
All of the log functions will place -inf or inf in the elements if the log can not be
computed.
Find log at base 10 of all elements of following array:
import numpy as np
arr = np.arange(1, 10)
print(np.log10(arr))
3.4 Summations
Addition is done between two arguments whereas summation happens over n
elements
Add the values in arr1 to the values in arr2:
import numpy as np
arr1 = np.array([1, 2, 3])
arr2 = np.array([1, 2, 3])
newarr = np.add(arr1, arr2)
print(newarr)
3.5 Products
To find the product of the elements in an array, use the prod() function.
Find the product of the elements of this array:
import numpy as np
arr = np.array([1, 2, 3, 4])
x = np.prod(arr)
print(x)
3.6 Differences
A discrete difference means subtracting two successive elements.
To find the discrete difference, use the diff() function.
Compute discrete difference of the following array:
import numpy as np
arr = np.array([10, 15, 25, 5])
newarr = np.diff(arr)
print(newarr)
3.7 LCM (Lowest Common Multiple

The Lowest Common Multiple is the least number that is common multiple of both of
the numbers.
import numpy as np
num1 = 4
num2 = 6
x = np.lcm(num1, num2)

print(x)
3.8 GCD (Greatest Common Denominator)

The GCD (Greatest Common Denominator), also known as HCF (Highest Common
Factor) is the biggest number that is a common factor of both of the numbers.
Find the HCF of the following two numbers:
import numpy as np
num1 = 6
num2 = 9
x = np.gcd(num1, num2)
print(x)
3.9 Trigonometric Functions

NumPy provides the ufuncs sin(), cos() and tan() that take values in radians and
produce the corresponding sin, cos and tan values.
Find sine value of PI/2:
import numpy as np
x = np.sin(np.pi/2)
print(x)
Find sine values for all of the values in arr:

import numpy as np
arr = np.array([np.pi/2, np.pi/3, np.pi/4, np.pi/5])
x = np.sin(arr)
print(x)
3.10 Hyperbolic Functions

NumPy provides the ufuncs sinh(), cosh() and tanh() that take values in radians and
produce the corresponding sinh, cosh and tanh values..
Find sinh value of PI/2:
import numpy as np
x = np.sinh(np.pi/2)
print(x)
Find cosh values for all of the values in arr:

import numpy as np
arr = np.array([np.pi/2, np.pi/3, np.pi/4, np.pi/5])
x = np.cosh(arr)
print(x)
3.11 Set Operations

A set in mathematics is a collection of unique elements.

3.11.1 Create Sets in NumPy
We can use NumPy's unique() method to find unique elements from any array. E.g.
create a set array, but remember that the set arrays should only be 1-D arrays.
Convert following array with repeated elements to a set:
import numpy as np
arr = np.array([1, 1, 1, 2, 3, 4, 5, 5, 6, 7])
x = np.unique(arr)
print(x)
3.11.2 Finding Union
To find the unique values of two arrays, use the union1d() method.
Find union of the following two set arrays:
import numpy as np
arr1 = np.array([1, 2, 3, 4])
arr2 = np.array([3, 4, 5, 6])
newarr = np.union1d(arr1, arr2)
print(newarr)
3.11.3 Finding Intersection

To find only the values that are present in both arrays, use the intersect1d() method.
Find intersection of the following two set arrays:
import numpy as np
arr1 = np.array([1, 2, 3, 4])
arr2 = np.array([3, 4, 5, 6])
newarr = np.intersect1d(arr1, arr2, assume_unique=True)
print(newarr)
Sample Output:

RESULT
Thus the feature study of NumPy was completed successfully.

1(c). Explore the features of SciPy
AIM:
To learn the different features provided by SciPy package.
ALGORITHM:
1. Install the SciPy package
2. Study all the features of SciPy package.
SciPy
SciPy stands for Scientific Python, SciPy is a scientific computation library that uses
NumPy underneath.
Features
These are the important features of SciPy
1. Constants 2. Sparse Data 3. Graphs
4. Spatial Data 5. Matlab Arrays 6. Interpolation
1. Constants in SciPy
As SciPy is more focused on scientific implementations, it provides many built-in
scientific constants.
These constants can be helpful when you are working with Data Science.
1.1 Constants in SciPy
Metric
Return the specified unit in meter
ex: print(constants.milli)
Binary
Return the specified unit in bytes
ex: print(constants.kibi)
Mass
Return the specified unit in kg
ex: print(constants.stone)
Angle
Return the specified unit in radians
ex: print(constants.degree)
Time
Return the specified unit in seconds
ex: print(constants.year)
Length
Return the specified unit in meters
ex: print(constants.mile)
Pressure
Return the specified unit in pascals
ex: print(constants.bar)

Area
Return the specified unit in square meters
ex: print(constants.hectare)
Volume
Return the specified unit in cubic meters
ex: print(constants.litre)
Speed
Return the specified unit in meters per second
ex: print(constants.kmh)
Temperature
Return the specified unit in Kelvin
ex: print(constants.zero_Celsius)
Energy
Return the specified unit in joules
ex: print(constants.calorie)
Power
Return the specified unit in watts
ex: print(constants.hp)
Force
Return the specified unit in newton
ex: print(constants.pound_force)
2. Sparse Data
Sparse data is data that has mostly unused elements (elements that don't carry any
information).
It can be an array like this one:
[1, 0, 2, 0, 0, 3, 0, 0, 0, 0, 0, 0]
Sparse Data: is a data set where most of the item values are zero.
Dense Array: is the opposite of a sparse array: most of the values are not zero.
2.1 CSR(Compressed Sparse Row) Matrix

We can create CSR matrix by passing an arrray into function
scipy.sparse.csr_matrix().
Create a CSR matrix from an array:
import numpy as np
from scipy.sparse import csr_matrix
arr = np.array([0, 0, 0, 0, 0, 1, 1, 0, 2])
print(csr_matrix(arr))
3. Graphs
Graphs are an essential data structure.
SciPy provides us with the module scipy.sparse.csgraph for working with such data
structures.

Adjacency Matrix
Adjacency matrix is a nxn matrix where n is the number of elements in a graph.
The values represents the connection between the elements.
3.1 Dijkstra
Use the dijkstra method to find the shortest path in a graph from one element to
another.
It takes following arguments:
return_predecessors: boolean (True to return whole path of traversal otherwise False).
indices: index of the element to return all paths from that element only.
limit: max weight of path.
Find the shortest path from element 1 to 2:
import numpy as np
from scipy.sparse.csgraph import dijkstra
arr = np.array([
[0, 1, 2],
[1, 0, 0],
[2, 0, 0]
])
newarr = csr_matrix(arr)
print(dijkstra(newarr, return_predecessors=True, indices=0))
3.2 Depth First Order

The depth_first_order() method returns a depth first traversal from a node.
This function takes following arguments:
the graph.
the starting element to traverse graph from.
Traverse the graph depth first for given adjacency matrix:

import numpy as np
from scipy.sparse.csgraph import depth_first_order
arr = np.array([
[0, 1, 0, 1],
[1, 1, 1, 1],
[2, 1, 1, 0],
[0, 1, 0, 1]
])
print(depth_first_order(newarr, 1))

3.3 Breadth First Order
The breadth_first_order() method returns a breadth first traversal from a node.
This function takes following arguments:
the graph.
the starting element to traverse graph from.
Traverse the graph breadth first for given adjacency matrix:
import numpy as np
from scipy.sparse.csgraph import breadth_first_order
arr = np.array([
[0, 1, 0, 1],
[1, 1, 1, 1],
[2, 1, 1, 0],
[0, 1, 0, 1]
])
print(breadth_first_order(newarr, 1))
4. Spatial Data
Spatial data refers to data that is represented in a geometric space.
E.g. points on a coordinate system.
We deal with spatial data problems on many tasks.
E.g. finding if a point is inside a boundary or not.
4.1 Triangulation
A Triangulation of a polygon is to divide the polygon into multiple triangles with
which we can compute an area of the polygon.
A Triangulation with points means creating surface composed triangles in which all of
the given points are on at least one vertex of any triangle in the surface.
One method to generate these triangulations through points is the Delaunay()
Triangulation.
Example:
Create a triangulation from following points:
import numpy as np
from scipy.spatial import Delaunay
points = np.array([
[2, 4],
[3, 4],
[3, 0],
[2, 2],
[4, 1]
])
simplices = Delaunay(points).simplices

plt.triplot(points[:, 0], points[:, 1], simplices)
plt.scatter(points[:, 0], points[:, 1], color='r')
plt.show()
4.2 Convex Hull

A convex hull is the smallest polygon that covers all of the given points.
Use the ConvexHull() method to create a Convex Hull.
Example
Create a convex hull for following points:
import numpy as np
from scipy.spatial import ConvexHull
points = np.array([
[2, 4],
[3, 4],
[3, 0],
[2, 2],
[4, 1],
[1, 2],
[5, 0],
[3, 1],
[1, 2],
[0, 2] ])
hull = ConvexHull(points)
hull_points = hull.simplices
plt.scatter(points[:,0], points[:,1])
for simplex in hull_points:
plt.plot(points[simplex,0], points[simplex,1], 'k-')
plt.show()
4.3 KDTrees
KDTrees are a datastructure optimized for nearest neighbor queries.
E.g. in a set of points using KDTrees we can efficiently ask which points are nearest
to a certain given point.
The KDTree() method returns a KDTree object.
The query() method returns the distance to the nearest neighbor and the location of the
neighbors.
Example
Find the nearest neighbor to point (1,1):
from scipy.spatial import KDTree
points = [(1, -1), (2, 3), (-2, 3), (2, -3)]
kdtree = KDTree(points)
res = kdtree.query((1, 1))
print(res)

4.4 Distance Matrix
There are many Distance Metrics used to find various types of distances between two
points in data science, Euclidean distsance, cosine distsance etc.
The distance between two vectors may not only be the length of straight line between
them, it can also be the angle between them from origin, or number of unit steps required etc.
Many of the Machine Learning algorithm's performance depends greatly on distance
metrices. E.g. "K Nearest Neighbors", or "K Means" etc.
Let us look at some of the Distance Metrices:
4.4.1 Euclidean Distance

Find the euclidean distance between given points A and B.
Example
Find the euclidean distance between given points.
from scipy.spatial.distance import euclidean
p1 = (1, 0)
p2 = (10, 2)
res = euclidean(p1, p2)
print(res)
4.4.2 Cosine Distance

Is the value of cosine angle between the two points A and B.
Example
Find the cosine distsance between given points:
from scipy.spatial.distance import cosine
p1 = (1, 0)
p2 = (10, 2)
res = cosine(p1, p2)
print(res)
Hamming Distance
Is the proportion of bits where two bits are difference.
It's a way to measure distance for binary sequences.
Example
Find the hamming distance between given points:
from scipy.spatial.distance import hamming
p1 = (True, False, True)
p2 = (False, True, True)
res = hamming(p1, p2)
print(res)
5. Matlab Arrays
We know that NumPy provides us with methods to persist the data in readable
formats for Python. But SciPy provides us with interoperability with Matlab as well.

Working With Matlab Arrays
We know that NumPy provides us with methods to persist the data in readable
formats for Python. But SciPy provides us with interoperability with Matlab as well.
Exporting Data in Matlab Format
The savemat() function allows us to export data in Matlab format.
The method takes the following parameters:
filename - the file name for saving data.
mdict - a dictionary containing the data.
do_compression - a boolean value that specifies whether to compress the
result or not. Default False.
Example
Export the following array as variable name "vec" to a mat file:
from scipy import io
import numpy as np
arr = np.arange(10)
io.savemat('arr.mat', {"vec": arr})
Import Data from Matlab Format

The loadmat() function allows us to import data from a Matlab file.
The function takes one required parameter:
filename - the file name of the saved data.
It will return a structured array whose keys are the variable names, and the
corresponding values are the variable values.
Example
Import the array from following mat file.:
from scipy import io
import numpy as np
arr = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9,])
# Export:
io.savemat('arr.mat', {"vec": arr})
# Import:
mydata = io.loadmat('arr.mat')
print(mydata)
6. Interpolation
Interpolation is a method for generating points between given points.
For example: for points 1 and 2, we may interpolate and find points 1.33 and 1.66.
Interpolation has many usage, in Machine Learning we often deal with missing data in
a dataset, interpolation is often used to substitute those values.
This method of filling values is called imputation.
Apart from imputation, interpolation is often used where we need to smooth the
discrete points in a dataset.

6.1 1D Interpolation
The function interp1d() is used to interpolate a distribution with 1 variable.
It takes x and y points and returns a callable function that can be called with new x
and returns corresponding y.
Example
For given xs and ys interpolate values from 2.1, 2.2... to 2.9:
from scipy.interpolate import interp1d
import numpy as np
xs = np.arange(10)
ys = 2*xs + 1
interp_func = interp1d(xs, ys)
newarr = interp_func(np.arange(2.1, 3, 0.1))
print(newarr)
6.2 Spline Interpolation

In 1D interpolation the points are fitted for a single curve whereas in Spline
interpolation the points are fitted against a piecewise function defined with polynomials
called splines.
The UnivariateSpline() function takes xs and ys and produce a callable funciton that
can be called with new xs.
Example
Find univariate spline interpolation for 2.1, 2.2... 2.9 for the following non linear points:
from scipy.interpolate import UnivariateSpline
import numpy as np
xs = np.arange(10)
ys = xs**2 + np.sin(xs) + 1
interp_func = UnivariateSpline(xs, ys)
newarr = interp_func(np.arange(2.1, 3, 0.1))
print(newarr)
Sample Output

RESULT
Thus the feature study of SciPy was completed successfully.

1(d). Explore the features of Pandas
AIM:
To learn the different features provided by Pandas package.
ALGORITHM:
1. Install the Pandas package
2. Study all the features of Pandas package.
Pandas
 Pandas is a Python library used for working with data sets.
 It has functions for analyzing, cleaning, exploring, and manipulating data.
 Pandas allows us to analyze big data and make conclusions based on statistical
theories.
 Pandas can clean messy data sets, and make them readable and relevant.
Features
These are the important features of Pandas.
1. Series 2. DataFrames 3. Read CSV
4. Read JSON 5. Viewing the Data 6. Data Cleaning
7. Plotting
1. Series
 A Pandas Series is like a column in a table.
 It is a one-dimensional array holding data of any type.
 Create a simple Pandas Series from a list:
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a)
print(myvar)
1.1 Create Labels

With the index argument, you can name your own labels.
Example
Create you own labels:
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a, index = ["x", "y", "z"])
print(myvar)
1.2 Key/Value Objects as Series

You can also use a key/value object, like a dictionary, when creating a Series.

Example
Create a simple Pandas Series from a dictionary:
import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = pd.Series(calories)
print(myvar)
2. DataFrames
A Pandas DataFrame is a 2 dimensional data structure, like a 2 dimensional array, or a
table with rows and columns.
Example
Create a simple Pandas DataFrame:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
#load data into a DataFrame object:
df = pd.DataFrame(data)
print(df)
3. Read CSV
A simple way to store big data sets is to use CSV files (comma separated files). CSV
files contains plain text and is a well know format that can be read by everyone
including Pandas.
Example
To print maximum rows in a CSV file
import pandas as pd
pd.options.display.max_rows = 9999
df = pd.read_csv('data.csv')
print(df)
4. Read JSON
 Big data sets are often stored, or extracted as JSON.
 JSON is plain text, but has the format of an object, and is well known in the world
of programming, including Pandas.
Load the JSON file into a DataFrame:
import pandas as pd
df = pd.read_json('data.json')
print(df.to_string())

5. Viewing the Data
One of the most used method for getting a quick overview of the DataFrame, is the
head() method. The head() method returns the headers and a specified number of rows,
starting from the top.
5.1 Info About the Data

The DataFrames object has a method called info(), that gives you more information
about the data set.
Example
Print information about the data:
import pandas as pd
print(df.info())
6. Data Cleaning
Data cleaning means fixing bad data in your data set.
Bad data could be:
 Empty cells
 Data in wrong format
 Wrong data
 Duplicates
6.1 Empty Cells

6.1.1 Remove Rows
One way to deal with empty cells is to remove rows that contain empty cells.
This is usually OK, since data sets can be very big, and removing a few rows will not
have a big impact on the result.
Example
Return a new Data Frame with no empty cells:
import pandas as pd
new_df = df.dropna()
print(new_df.to_string())
inplace() method
It remove all rows with NULL values:
import pandas as pd
df.dropna(inplace = True)
6.1.2 Replace Empty Values

Another way of dealing with empty cells is to insert a new value instead.

Example
Replace NULL values with the number 130:
import pandas as pd
df.fillna(130, inplace = True)
6.1.3 Replace Using Mean, Median, or Mode

A common way to replace empty cells, is to calculate the mean, median or mode
value of the column.
Pandas uses the mean() median() and mode() methods to calculate the respective
values for a specified column:
mean()
import pandas as pd
x = df["Calories"].mean()
df["Calories"].fillna(x, inplace = True)
median()
import pandas as pd
x = df["Calories"].median()
mode()
import pandas as pd
x = df["Calories"].mode()[0]
6.2 Data of Wrong Format

Cells with data of wrong format can make it difficult, or even impossible, to analyze
data.
To fix it, you have two options: remove the rows, or convert all cells in the columns
into the same format.
Example
import pandas as pd
df['Date'] = pd.to_datetime(df['Date'])
6.2.1 Removing Rows

Remove rows with a NULL value in the "Date" column:
import pandas as pd
df['Date'] = pd.to_datetime(df['Date'])

df.dropna(subset=['Date'], inplace = True)
6.3 Fixing Wrong Data

6.3.1 Wrong Data
"Wrong data" does not have to be "empty cells" or "wrong format", it can just be
wrong, like if someone registered "199" instead of "1.99".
Sometimes you can spot wrong data by looking at the data set, because you have an
expectation of what it should be.
6.3.2 Replacing Values

One way to fix wrong values is to replace them with something else.
Example
Set "Duration" = 45 in row 7:
import pandas as pd
df.loc[7,'Duration'] = 45
6.3.3 Removing Rows

Another way of handling wrong data is to remove the rows that contains wrong data.
Example
Delete rows where "Duration" is higher than 120:
import pandas as pd
for x in df.index:
if df.loc[x, "Duration"] > 120:
df.drop(x, inplace = True)
6.4 Removing Duplicates

6.4.1 Discovering Duplicates
Duplicate rows are rows that have been registered more than one time.
duplicated() method
import pandas as pd
print(df.duplicated())
6.4.2 Removing Duplicates

To remove duplicates, use the drop_duplicates() method.
import pandas as pd
df.drop_duplicates(inplace = True)

7. Plotting
We can use Pyplot, a submodule of the Matplotlib library to visualize the diagram on
the screen.
Pandas uses the plot() method to create diagrams.
7.1 Scatter Plot

Specify that you want a scatter plot with the kind argument:
kind = 'scatter'
Example
import sys
import matplotlib
matplotlib.use('Agg')
import pandas as pd
df.plot(kind = 'scatter', x = 'Duration', y = 'Maxpulse')
plt.show()
plt.savefig(sys.stdout.buffer)
sys.stdout.flush()
7.2 Histogram
Use the kind argument to specify that you want a histogram:
kind = 'hist'
Example
import sys
import matplotlib
matplotlib.use('Agg')
import pandas as pd
df["Duration"].plot(kind = 'hist')
plt.show()
plt.savefig(sys.stdout.buffer)
sys.stdout.flush()

Sample Output

RESULT
Thus the feature study of Pandas was completed successfully.

1(e). Explore the features of statsmodels
AIM:
To learn the different features provided by statsmodels package.
ALGORITHM:
3. Install the statsmodels package
4. Study all the features of statsmodels package.
Statsmodels
statsmodels is a Python module that provides classes and functions for the estimation
of many different statistical models, as well as for conducting statistical tests, and statistical
data exploration.
Features
These are the important features of statsmodels
1. Linear regression models
2. Survival analysis
1. Linear regression models

Linear regression analysis is a statistical technique for predicting the value of one
variable(dependent variable) based on the value of another(independent variable).
In simple linear regression, there’s one independent variable used to predict a single
dependent variable. In the case of multilinear regression, there’s more than one independent
variable.
The independent variable is the one you’re using to forecast the value of the other
variable. The statsmodels.regression.linear_model.OLS method is used to perform linear
regression. Linear equations are of the form:
Y=mX+C (m=slope; c=constant)
Syntax:
statsmodels.regression.linear_model.OLS(endog, exog=None, missing=’none’,
hasconst=None, **kwargs)
Parameters:
 endog: array like object.
 exog: array like object.
 missing: str. None, decrease, and raise are the available alternatives. If the value is
‘none,’ no nan testing is performed. Any observations with nans are dropped if ‘drop’
is selected. An error is raised if ‘raise’ is used. ‘none’ is the default.
 hasconst: None or Bool. Indicates whether a user-supplied constant is included in the
RHS. If True, k constant is set to 1 and all outcome statistics are calculated as if a
constant is present. If False, k constant is set to 0 and no constant is verified.

 **kwargs: When using the formula interface, additional arguments are utilised to set
model characteristics.
Step 1: Import packages.

Importing the required packages is the first step of modeling. The pandas, NumPy,
and stats model packages are imported.
import numpy as np
import pandas as pd
import statsmodels.api as sm
Step 2: Loading data
To access the CSV file click here. The CSV file is read using pandas.read_csv()
method. The head or the first five rows of the dataset is returned by using the head() method.
Head size and Brain weight are the columns.
df = pd.read_csv('headbrain1.csv')
df.head()
Visualizing the data:
By using the matplotlib and seaborn packages, we visualize the data. sns.regplot()
function helps us create a regression plot.
# import packages
import pandas as pd
df = pd.read_csv('headbrain1.csv')
sns.regplot('Head Size(cm^3)', 'Brain Weight(grams)', data=df)
plt.show()
Step 3: Setting a hypothesis.
Null hypothesis (H0): There is no relationship between head size and brain weight.
Alternative hypothesis (Ha): There is a relationship between head size and brain
weight.
Step 4: Fitting the model
statsmodels.regression.linear_model.OLS() method is used to get ordinary least
squares, and fit() method is used to fit the data in it.
The ols method takes in the data and performs linear regression. we provide the
dependent and independent columns in this format :
inpendent_columns ~ dependent_column:
left side of the ~ operator contains the independent variables and right side of the
operator contains the name of the dependent variable or the predicted column.
df.columns = ['Head_size', 'Brain_weight']
model = sm.ols(formula='Head_size ~ Brain_weight', data=df).fit()
Step 5: Summary of the model.
All the summary statistics of the linear regression model are returned by the
model.summary() method. The p-value and many other values/statistics are known by this
method. Predictions about the data are found by the model.summary() method.
print(model.summary())

2. Survival analysis
The statsmodels.api.SurvfuncRight class can be used to estimate survival functions
using data that may be censored to the right. SurvfuncRight implements several inference
methods, including confidence intervals for survival quantiles, pointwise simultaneous
confidence intervals for survival functions, and plotting methods. The duration.survdiff
function provides a test procedure for comparing survival distributions.
Here we are creating a SurvfuncRight object using the data from the Moore study
available from the R dataset repository. Adjust the survival distribution for 'low' fcategory
subjects only.
Example:
# Importing libraries
import statsmodels.api as sm
X = sm.datasets.get_rdataset("Moore", "carData").data
# Filtering data of low fcategory
X = X[X['fcategory'] == "low"]
# Creating SurvfuncRight model
model = sm.SurvfuncRight(X["conformity"], X["fscore"])
# Model Summary
model.summary()
Sample Output
Linear regression models

Survival analysis
RESULT
Thus the few important features of study statsmodels was completed successfully.

2. Working with Numpy arrays
AIM:
To work with different features provided by Numpy arrays.
ALGORITHM:
1. Install the numpy package
2. Work with all the features of numpy array.
Arrays
1. Creating Arrays
 0-D Arrays
Each value in an array is a 0-D array.
import numpy as np
arr = np.array(42)
print(arr)
 1-D Arrays
An array that has 0-D arrays as its elements is called 1-D array.
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
print(arr)
 2-D Arrays
An array that has 1-D arrays as its elements is called a 2-D array.
import numpy as np
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)
 3-D arrays
An array that has 2-D arrays (matrices) as its elements is called 3-D array.
import numpy as np
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(arr)
Example:
import numpy as np
a = np.array(42)
b = np.array([1, 2, 3, 4, 5])
c = np.array([[1, 2, 3], [4, 5, 6]])
d = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])
print(a.ndim)
print(b.ndim)
print(c.ndim)
print(d.ndim)

2. Access Array Elements
Access 2-D Arrays
To access elements from 2-D arrays we can use comma separated integers
representing the dimension and the index of the element.
import numpy as np
arr = np.array([[1,2,3,4,5], [6,7,8,9,10]])
print('2nd element on 1st row: ', arr[0, 1])
Access 3-D Arrays
To access elements from 3-D arrays we can use comma separated integers
representing the dimensions and the index of the element.
import numpy as np
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
print(arr[0, 1, 2])
3. Array Slicing
 Slicing in python means taking elements from one given index to another given
index.
 We pass slice instead of index like this: [start:end].
 We can also define the step, like this: [start:end:step].
 If we don't pass start its considered 0
 If we don't pass end its considered length of array in that dimension
 If we don't pass step its considered 1
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6, 7])
print(arr[1:5:2])
4. Data Types
NumPy has some extra data types, and refer to data types with one character, like i for
integers, u for unsigned integers etc.
Below is a list of all data types in NumPy and the characters used to represent them.
i - integer M - datetime
b - boolean O - object
u - unsigned integer S - string
f - float U - unicode string
c - complex float V - fixed chunk of memory
m - timedelta for other type (void)
Example:
import numpy as np
arr = np.array([1, 2, 3, 4], dtype='S')
print(arr)
print(arr.dtype)

5. Copy & View
5.1 Copy:
Make a copy
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
x = arr.copy()
arr[0] = 42
print(arr)
print(x)
5.2 View:
Make a view
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
x = arr.view()
arr[0] = 42
print(arr)
print(x)
6. Array Shape & Reshaping

6.1 Array Shape
NumPy arrays have an attribute called shape that returns a tuple with each index
having the number of corresponding elements.
import numpy as np
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(arr.shape)
6.2 Array Reshaping

 Reshaping means changing the shape of an array.
 The shape of an array is the number of elements in each dimension.
 By reshaping we can add or remove dimensions or change number of elements in
each dimension.
 Convert the following 1-D array with 12 elements into a 3-D array.
 The outermost dimension will have 2 arrays that contains 3 arrays, each with 2
elements:
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
newarr = arr.reshape(2, 3, 2)
print(newarr)
7. Array Iterating
 Iterating means going through elements one by one.

 As we deal with multi-dimensional arrays in numpy, we can do this using basic
for loop of python.
import numpy as np
arr = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
for x in arr:
print(x)
8. Joining Array
Joining means putting contents of two or more arrays in a single array.
import numpy as np
arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])
arr = np.concatenate((arr1, arr2))
print(arr)
9. Splitting Array
Splitting is reverse operation of Joining.
Joining merges multiple arrays into one and Splitting breaks one array into multiple.
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 6])
newarr = np.array_split(arr, 3)
print(newarr)
10. Searching Arrays

You can search an array for a certain value, and return the indexes that get a match.
To search an array, use the where() method.
import numpy as np
arr = np.array([1, 2, 3, 4, 5, 4, 4])
x = np.where(arr == 4)
print(x)
11. Sorting means putting elements in an ordered sequence.

 Ordered sequence is any sequence that has an order corresponding to elements,
like numeric or alphabetical, ascending or descending.
 The NumPy ndarray object has a function called sort(), that will sort a specified
array.
import numpy as np
arr = np.array([3, 2, 0, 1])
print(np.sort(arr))
12. Filtering Arrays

Getting some elements out of an existing array and creating a new array out of them is
called filtering. In NumPy, you filter an array using a boolean index list.

If the value at an index is True that element is contained in the filtered array, if the
value at that index is False that element is excluded from the filtered array.
import numpy as np
arr = np.array([41, 42, 43, 44])
x = [True, False, True, False]
newarr = arr[x]
print(newarr)
Sample Output:

RESULT
Thus the important features of numpy array was completed successfully.

3. Working with DataFrame
AIM:
To work with dataframe provided by pandas.
ALGORITHM:
1. Install the pandas package
2. Work with all the features of dataframe.
1. DataFrame
A Pandas DataFrame is a 2 dimensional data structure, like a 2 dimensional array, or a
table with rows and columns.
Example
Create a simple Pandas DataFrame:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
print(df)
2. Locate Row
As you can see from the result above, the DataFrame is like a table with rows and
columns.
Pandas use the loc attribute to return one or more specified row(s)
Example
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
print(df.loc[0])
3. Named Indexes
With the index argument, you can name your own indexes.
Example
Add a list of names to give each row a name:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]

}
df = pd.DataFrame(data, index = ["day1", "day2", "day3"])
print(df)
4. Locate Named Indexes

Use the named index in the loc attribute to return the specified row(s).
Example
Return "day2":
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
df = pd.DataFrame(data, index = ["day1", "day2", "day3"])
print(df.loc["day2"])
5. Load Files Into a DataFrame

If your data sets are stored in a file, Pandas can load them into a DataFrame.
Example
Load a comma separated file (CSV file) into a DataFrame:
import pandas as pd
print(df)
Sample Output:

RESULT
Thus the dataframe features of pandas was completed successfully.

4. Reading data from iris data set and doing descriptive analytics on the Iris data set
AIM:
To read data from files and exploring various commands for doing descriptive
analytics on the Iris data set.
ALGORITHM:
1. Download “Iris.csv” file from GitHub.com
2. Load the “Iris.csv” into google colab.
3. Perform descriptive analysis on the Iris file.
Importing Iris.csv
 Login to google colab by using gmail.
 Login to google drive and create a folder with required name.
 Move the Iris file from system to google drive.
 Click on the “file” icon and click on “Mount Device”.
 Code will appeared on a typing area, execute the same code.
 It requires authentication verification, complete the authentication.
 After successful verification it shows the message “Mounted at /content/drive”
 Find the Iris.csv file and copy the path for future references.
About Iris Database

Iris Dataset is considered as the Hello World for data science. It contains five columns
namely – Petal Length, Petal Width, Sepal Length, Sepal Width, and Species Type. Iris is a
flowering plant, the researchers have measured various features of the different iris flowers
and recorded them digitally.
You can download the Iris.csv file from the above link. Now we will use the Pandas
library to load this CSV file, and we will convert it into the dataframe. read_csv() method is
used to read CSV files.
Example:
import pandas as pd
# Reading the CSV file
df = pd.read_csv("/content/drive/MyDrive/Data_Science/iris.csv")
# Printing top 5 rows
df.head()
Getting Information about the Dataset

We will use the shape parameter to get the shape of the dataset.
df.shape -> returns no of rows and columns
df.info() -> returns column data types.

Checking Missing Values
We will check if our data contains any missing values or not. Missing values can
occur when no information is provided for one or more items or for a whole unit. We will use
the isnull() method.
Example:
df.isnull().sum()
Checking Duplicates
Let’s see if our dataset contains any duplicates or not. Pandas drop_duplicates()
method helps in removing duplicates from the data frame.
Example:
data = df.drop_duplicates(subset ="variety",)
data
Data Visualization
Visualizing the target column
Our target column will be the Species column because at the end we will need the
result according to the species only. Let’s see a countplot for species.
Example:
# importing packages
sns.countplot(x='Species', data=df,)
plt.show()
Relation between variables

We will see the relationship between the sepal length and sepal width and also
between petal length and petal width.
Example 1: Comparing Sepal Length and Sepal Width
sns.scatterplot(x='petal.length', y='petal.width',hue='variety', data=df, )
# Placing Legend outside the Figure
plt.legend(bbox_to_anchor=(1, 1), loc=2)
plt.show()
Example 2: Comparing Petal Length and Petal Width

sns.scatterplot(x='petal.length', y='petal.width', hue='variety', data=df, )

# Placing Legend outside the Figure
plt.legend(bbox_to_anchor=(1, 1), loc=2)
plt.show()
Handling Correlation
Pandas dataframe.corr() is used to find the pairwise correlation of all columns in the
dataframe. Any NA values are automatically excluded. For any non-numeric data type
columns in the dataframe it is ignored.
Example:
data.corr(method='pearson')
Sample Output

RESULT
Iris.csv file was loaded into google colab and descriptive analytics was made on the
Iris data set successfully.

5(a). Perform Univariate analysis on the diabetes data set
AIM:
Use the diabetes data set from UCI and Pima Indians Diabetes data set for Univariate
analysis.
ALGORITHM:
1. Download diabetes data set from UCI and Pima Indians Diabetes data set.
2. Load the above data files into google colab.
3. Perform analysis like Frequency, Mean, Median, Mode, Variance, Standard
Deviation, Skewness and Kurtosis.
Univariate analysis
 The term univariate analysis refers to the analysis of one variable.
 There are three common ways to perform univariate analysis on one variable:
Summary statistics – Measures the center and spread of values.
1. Central tendency — mean, median, mode
2. Dispersion — variance, standard deviation, range, interquartile
range (IQR)
3. Skewness — symmetry of data along with mean value
4. Kurtosis — peakedness of data at mean value
5. Frequency table – Describes how often different values occur.
File Importing:
# Reading the UCI file
import pandas as pd
df = pd.read_csv("/content/drive/MyDrive/Data_Science/UCI_diabetes.csv")
df.head()
# Reading the Pima file
import pandas as pd
df = pd.read_csv("/content/drive/MyDrive/Data_Science/Pima_diabetes.csv")
df.head()
1. Central Tendency
We can use the following syntax to calculate various summary statistics like Mean,
Median and Mode.
1.1 Mean:
It is average value of given numeric values
 Mean of UCI data
import pandas as pd

df = pd.read_csv("/content/drive/MyDrive/Data_Science/UCI_diabetes.cs
v")
# Mean of UCI data
df.mean(axis=0)
 Mean of Pima data
import pandas as pd
df =
pd.read_csv("/content/drive/MyDrive/Data_Science/Pima_diabetes.csv")
# Mean of Pima data
df.mean(axis=0)
1.2 Median:
It is middle most value of given values
 Median of UCI data
import pandas as pd
df =
pd.read_csv("/content/drive/MyDrive/Data_Science/UCI_diabetes.csv")
# Median of UCI data
df.median(axis=0)
 Median of Pima data

import pandas as pd
df =
# Median of Pima data
df.median(axis=0)
1.3 Mode:
It is the most frequently occurring value of given numeric variables
 Mode of UCI data
import pandas as pd
df =
pd.read_csv("/content/drive/MyDrive/Data_Science/UCI_diabetes.csv")
# Median of UCI data
df.mode(axis=0)
 Mode of Pima data

import pandas as pd

df =
# Mean of Pima data
df.mode(axis=0)
2. Dispersion
2.1 Variance
The range is the difference between the maximum and minimum values of a data set.
Example
import pandas as pd
df =
# variance of the BMI column
df.loc[:,"BMI"].var()
2.2 Standard deviation

Standard deviation is a measure of how spread out the numbers are. A large standard
deviation indicates that the data is spread out, - a small standard deviation indicates that the
data is clustered closely around the mean.
Example
import pandas as pd
df =
# Standard deviation of the BMI column
df.loc[:,"BMI"].std()
2.3 Range
Range is the simplest of the measurements but is very limited in its use, we calculate
the range by taking the largest value of the dataset and subtract the smallest value from it, in
other words, it is the difference of the maximum and minimum values of a dataset.
Example
df=pd.read_csv("/content/drive/MyDrive/Data_Science/Pima_diabetes.cs
v")
print("Range is:",df.BloodPressure.max()-df.BloodPressure.min())
2.4 Interquartile range

The interquartile range, often denoted “IQR”, is a way to measure the spread of the
middle 50% of a dataset. It is calculated as the difference between the first quartile* (the 25th
percentile) and the third quartile (the 75th percentile) of a dataset.
Example
# Importing important libraries

import numpy as np
import pandas as pd
plt.style.use('seaborn')
data =
pd.read_csv('/content/drive/MyDrive/Data_Science/Pima_diabetes.csv')
# Removing the outliers
def removeOutliers(data, col):
Q3 = np.quantile(data[col], 0.75)
Q1 = np.quantile(data[col], 0.25)
IQR = Q3 - Q1
print("IQR value for column %s is: %s" % (col, IQR))

global outlier_free_list
global filtered_data
lower_range = Q1 - 1.5 * IQR

upper_range = Q3 + 1.5 * IQR
outlier_free_list = [x for x in data[col] if (
(x > lower_range) & (x < upper_range))]
filtered_data = data.loc[data[col].isin(outlier_free_list)]
for i in data.columns:
if i == data.columns[0]:
removeOutliers(data, i)
else:
removeOutliers(filtered_data, i)
# Assigning filtered data back to our original variable

data = filtered_data
print("Shape of data after outlier removal is: ", data.shape)
3. Skewness
 Skewness essentially measures the symmetry of the distribution.
Example
# importing pandas as pd
import pandas as pd
# Creating the dataframe
df =
# skip the na values
# find skewness in each row
df.skew(axis = 0, skipna = True)

4. kurtosis
kurtosis determines the heaviness of the distribution tails.
Example
import pandas as pd
df =
df['BloodPressure'].kurtosis()
5. Frequency
Frequency is a count of the number of occurrences a particular value occurs or
appears in our data. A frequency table displays a set of values along with the frequency with
which they appear. They allow us to better understand which data values are common and
which are uncommon.
Example
# import packages
import pandas as pd
import numpy as np
# reading csv file
data =
# one way frequency table for the species column.
freq_table = pd.crosstab(data['Age'], 'BMI')
# frequency table in proportion of species
freq_table= freq_table/len(data)
freq_table
Sample Output

RESULT
Thus the Univariate analysis on the Diabetes data of UCI and Pima was performed
successfully.

5(b). Perform Bivariate analysis on the diabetes data set.
AIM:
To use the UCI and Pima Indians Diabetes data set for Bivariate analysis.
ALGORITHM:
3. Perform various methods of bivariate.
Bivariate analysis
The term bivariate analysis refers to the analysis of two variables. The purpose of
bivariate analysis is to understand the relationship between two variables
There are three common ways to perform bivariate analysis:
1. Scatterplots
2. Correlation Coefficients
3. Simple Linear Regression
1. Scatterplots
A scatterplot is a type of data display that shows the relationship between two
numerical variables
Example
import numpy as np
import pandas as pd
# import packages
data =
# Diabetes Outcome
g1 = data.loc[data.Outcome==1,:]
# Pregnancies, Glucose and Diabetes relation
g1.plot.scatter('Pregnancies', 'Glucose');
2. Correlation Coefficients
The correlation coefficient is a statistical measure of the strength of the relationship
between the relative movements of two variables. The values range between -1.0 and 1.0.
Correlation of -1.0 shows a perfect negative correlation, while a correlation of 1.0 shows a
perfect positive correlation. A correlation of 0.0 shows no linear relationship between the
movement of the two variables.
Example
# Import those libraries
import pandas as pd
from scipy.stats import pearsonr

# Import your data into Python
df =
# Convert dataframe into series
list1 = df['BloodPressure']
list2 = df['SkinThickness']
# Apply the pearsonr()
corr, _ = pearsonr(list1, list2)
print('Pearsons correlation: %.3f' % corr)
3. Simple Linear Regression

Simple linear regression is a statistical method that we can use to find a relationship
between two variables and make predictions. The two variables used are typically denoted as
y and x. The independent variable, or the variable used to predict the dependent variable is
denoted as x. The dependent variable, or the outcome/output, is denoted as y.
A simple linear regression model will produce a line of best fit, or the regression line.
You may have heard about drawing the line of best fit through a scatter plot of data.
Example
import numpy as np
import pandas as pd
# Importing the dataset
dataset =
X = dataset.iloc[:, :-1].values #get a copy of dataset exclude last column
y = dataset.iloc[:, 1].values #get array of dataset in column 1st
# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1/3,
random_state=0)
# Fitting Simple Linear Regression to the Training set
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

Sample Output
RESULT:
Thus the Bivariate analysis on the diabetes data set was executed successfully.

5(c). Perform Multiple Regression Analysis on the diabetes data set
AIM:
To use UCI and Pima Indians Diabetes data set for Multiple Regression Analysis.
ALGORITHM:
3. Perform multiple regression analysis on data sets.
Multiple Regression Analysis

Multiple regression is like linear regression, but with more than one independent
value, meaning that we try to predict a value based on two or more variables.
Example
# Pima_diabetes
import pandas
from sklearn import linear_model
df =
pandas.read_csv("/content/drive/MyDrive/Data_Science/Pima_diabetes.cs
v")
X = df['Pregnancies ', 'Glucose ']
y = df['BloodPressure ']
regr = linear_model.LinearRegression()
regr.fit(X, y)
#predict the Blood Pressure based on Pregnancies and Glucose level:
predictedBP = regr.predict([[4, 120]])
print(predictedBP)
# UCI-Diabetes
import pandas
from sklearn import linear_model
df =
pandas.read_csv("("/content/drive/MyDrive/Data_Science/UCI_diabetes.
csv")
X = df[['Time', 'Code']]
y = df['Value']
regr = linear_model.LinearRegression()
regr.fit(X, y)
#predict the Diabetes based on Time and Code:
predictedBP = regr.predict([[13:23, 46]])
print(predictedBP)

Sample Output
RESULT
Thus the Multiple Regression analysis on the Diabetes data of UCI and Pima was
performed successfully.

6(a). Apply and explore Normal curves & Histograms plotting functions on UCI-Iris
data sets
AIM:
To apply and explore Normal curves & Histograms plotting functions on UCI-Iris
data sets.
ALGORITHM:
1. Download Iris data set from UCI.
2. Load the above Iris data files into google colab.
3. Plot the normal curve and Histograms for Iris data set.
Normal Curves
It is a probability function used in statistics that tells about how the data values are
distributed. It is the most important probability distribution function used in statistics because
of its advantages in real case scenarios.
Example
import numpy as np
from scipy.stats import norm
import statistics
# import dataset
df = pd.read_csv("/content/drive/MyDrive/Data_Science/iris.csv")
# Plot between -10 and 10 with .001 steps.
x_axis = np.arange(-20, 20, 0.01)
# Calculating mean and standard deviation
mean = df["sepal.length"].mean()
sd = df.loc[:,"sepal.width"].std()
plt.plot(x_axis, norm.pdf(x_axis, mean, sd))
plt.show()
Histograms plotting functions

A histogram is basically used to represent data provided in a form of some groups.It is
accurate method for the graphical representation of numerical data distribution.It is a type of
bar plot where X-axis represents the bin ranges while Y-axis gives information about
frequency.
Example
import pandas as pd
import numpy as np
df = pd.read_csv('/content/drive/MyDrive/Data_Science/iris.csv ')
data = df[' sepal.length']
bins = np.arange(min(data), max(data) + 1, 1)
plt.hist(data, bins = bins, density = True)

plt.ylabel('sepal.width')
plt.xlabel( petal.length')
plt.show()
Sample Output
RESULT
Thus the UCI data set was plotted using Normal Curve and Histogram plotting was
executed successfully.

6(b). Density and contour plotting functions on UCI-Iris data sets.
AIM:
To apply and explore Density & Contour plotting functions on UCI-Iris data sets.
ALGORITHM:
3. Plot the density and contour plotting for Iris data sets.
Density Plotting
Density Plot is a type of data visualization tool. It is a variation of the histogram that
uses ‘kernel smoothing’ while plotting the values. It is a continuous and smooth version of a
histogram inferred from a data.
Density plots uses Kernel Density Estimation (so they are also known as Kernel
density estimation plots or KDE) which is a probability density function. The region of plot
with a higher peak is the region with maximum data points residing between those values.
Example - Density plot of several variables

# libraries & dataset
# set a grey background (use sns.set_theme() if seaborn version 0.11.0 or
above)
sns.set(style="darkgrid")
df = sns.load_dataset('iris')
# plotting both distibutions on the same figure
fig = sns.kdeplot(df['sepal_width'], shade=True, color="r")
fig = sns.kdeplot(df['sepal_length'], shade=True, color="b")
plt.show()
Contour plotting
Contour plots also called level plots are a tool for doing multivariate analysis and
visualizing 3-D plots in 2-D space. If we consider X and Y as our variables we want to plot
then the response Z will be plotted as slices on the X-Y plane due to which contours are
sometimes referred as Z-slices or iso-response.
Contour plots are widely used to visualize density, altitudes or heights of the
mountain as well as in the meteorological department.
Example
import pandas as pd
import matplotlib as mpl

px_orbital = pd.read_csv("/content/drive/MyDrive/Data_Science/iris.csv
")
x = px_orbital.iloc[0, 1:]
y = px_orbital.iloc[1:, 0]
px_values = px_orbital.iloc[1:, 1:]
mpl.rcParams['font.size'] = 14
mpl.rcParams['legend.fontsize'] = 'large'
mpl.rcParams['figure.titlesize'] = 'medium'
fig, ax = plt.subplots()
ticks = np.linspace(pmin, pmax, 6)
CS = ax.contourf(x, y, px_values, cmap="RdBu", levels=levels)
ax.set_aspect('equal')
ax.set_xlabel('x')
ax.set_ylabel('y')
fig.colorbar(CS, format="%.3f", ticks=ticks)
Sample Output
RESULT
Thus the UCI data set was plotted using Density & Contour plotting was executed
successfully.

6(c). Correlation and scatter plotting functions on UCI data sets.
AIM:
To apply and correlation & Scatter plotting functions on UCI-Iris data sets.
ALGORITHM:
3. Plot the correlation and scatter plotting for Iris data sets.
Correlation Matrix Plotting

Correlation gives an indication of how related the changes are between two variables.
If two variables change in the same direction they are positively correlated. If the change in
opposite directions together (one goes up, one goes down), then they are negatively
correlated.
You can calculate the correlation between each pair of attributes. This is called a
correlation matrix. You can then plot the correlation matrix and get an idea of which
variables have a high correlation with each other.
This is useful to know, because some machine learning algorithms like linear and
logistic regression can have poor performance if there are highly correlated input variables in
your data.
Example
# Correction Matrix Plot
import pandas
import numpy
url =
"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-
indians-diabetes.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pandas.read_csv(url, names=names)
correlations = data.corr()
# plot correlation matrix
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(correlations, vmin=-1, vmax=1)
fig.colorbar(cax)
ticks = numpy.arange(0,9,1)
ax.set_xticks(ticks)
ax.set_yticks(ticks)
ax.set_xticklabels(names)
ax.set_yticklabels(names)
plt.show()

Scatter Plotting
A scatterplot shows the relationship between two variables as dots in two dimensions,
one axis for each attribute. You can create a scatterplot for each pair of attributes in your
data. Drawing all these scatterplots together is called a scatterplot matrix.
Scatter plots are useful for spotting structured relationships between variables, like
whether you could summarize the relationship between two variables with a line. Attributes
with structured relationships may also be correlated and good candidates for removal from
your dataset.
Example
# Scatterplot Matrix
import pandas
from pandas.plotting import scatter_matrix
url =
"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-
indians-diabetes.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pandas.read_csv(url, names=names)
scatter_matrix(data)
plt.show()
Sample Output

RESULT
Thus the UCI data set was plotted using Correlation and scatter plotting was executed
successfully.

7. Visualizing Geographic Data with Basemap
AIM:
To visualizing the Geographic Data with Basemap using Zomato geographic data.
ALGORITHM:
1. Study the basics of Basemap.
2. Use Zomato data to plot city names and restaurants details.
Basemap Introduction
Basemap is a toolkit under the Python visualization library Matplotlib. Its main
function is to draw 2D maps, which are important for visualizing spatial data. basemap itself
does not do any plotting, but provides the ability to transform coordinates into one of 25
different map projections.
Zomato data Visualization
import numpy as np
import pandas as pd
import os
from glob import glob as gb
#list all the directories

dirs=os.listdir("C:/Users/IT LAB-I\Desktop/Data_Science/zomato_data/")
dirs
len(dirs)
#storing all the files from every directory

li=[]
for dir1 in dirs:
files=os.listdir(r"C:/Users/IT LAB-I\Desktop/Data_Science/zomato_data/"+dir1)
#reading each file from list of files from previous step and creating pandas data fame
for file in files:
df_file=pd.read_csv("C:/Users/IT LAB-
I\Desktop/Data_Science/zomato_data/"+dir1+"/"+file,quotechar='"',delimiter="|")
#appending the dataframe into a list
li.append(df_file.values)
len(li)

#numpys vstack method to append all the datafames to stack the sequence of input
vertically to make a single array
df_np=np.vstack(li)
#no of rows is represents the total no restaurants ,now of coloumns(12) is columns for
the dataframe
df_np.shape
#creating final dataframe from the numpy array

df_final=pd.DataFrame(df_np)
#adding the header columns

df_final=pd.DataFrame(df_final.values, columns
=["NAME","PRICE","CUSINE_CATEGORY","CITY","REGION","URL","PAGE
NO","CUSINE TYPE","TIMING","RATING_TYPE","RATING","VOTES"])
#displaying the dataframe

df_final
#header column "PAGE NO" is not required ,i used it while scraping the data from
zomato to do some sort of validation,lets remove the column
df_final.drop(columns=["PAGE NO"],axis=1,inplace=True)
#display the dataframe again

df_final
#lets count how many unique cities are there

df_final["CITY"].unique()
# import json and requests library to use googl apis to get the longitude ant latituide
values
import requests
import json
#creating a separate array with all city names as elements of array
city_name=df_final["CITY"].unique()
li1=[]
#googlemap api calling url
geo_s ='https://maps.googleapis.com/maps/api/geocode/json'
#iterating through a for loop for each city names
for i in range(len(city_name)):
#i have used my own google map api, please use ypur own api
param = {'address': city_name[i], 'key': 'AIzaSyD-kYTK-
8FQGueJqA2028t2YHbUX96V0vk'}
response = requests.get(geo_s, params=param)

response=response.text
data=json.loads(response)
#setting up the variable with corresponding city longitude and latitude
lat=data["results"][0]["geometry"]["location"]["lat"]
lng=data["results"][0]["geometry"]["location"]["lng"]
#creating a new data frame with city , latitude and longitude as columns
df2=pd.DataFrame([[city_name[i],lat,lng]])
li1.append(df2.values)
#numpys vstack method to append all the datafames to stack the sequence of input
vertically to make a single array
df_np=np.vstack(li1)
#creating a second dataframe with city name, latitude and longitude

df_sec=pd.DataFrame(df_np,columns=["CITY","lat","lng"])
#display the second dataframe contents

df_sec
#merge this data frame to the existing df_final data frame using merge and join features
from pandas,and creating a new data frame
df_final2=df_final.merge(df_sec,on="CITY",how="left")
#display the contents , it will have longitude and latitude now

df_final2
#creating pandas series to hold the citynames and corresponding count of restuarnats in
ascending order
li2=df_final["CITY"].value_counts().sort_values(ascending=True)
li2
#creating a empty dictionary

dc={}
#setting dictionary values as city name , count of restuarnat and key will city names as
well
for i,j in li2.items():
x=i + "," +str(j)
dc.update({i:[i,j]})
#displaying the dictionary

dc

#creating another data frame from the above dictionary
df_map=pd.DataFrame.from_dict(dc,orient="index",columns=["CITY","COUNT"])
#displaying the data frame

df_map
#merging this data frame with df_sec data frame(which we created using city
names,longitude and latitude)
df_map_final=df_map.merge(df_sec,on="CITY",how="left")
#displaying the new data frame this frame will be used for map ploting
df_map_final
#importing the libraries for map ploting

from matplotlib import cm
from matplotlib.dates import date2num
from mpl_toolkits.basemap import Basemap
# for date and time processing
import datetime
#lets take one data frame for top 20 cities with most retaurants counts
df_plot_top=df_map_final.tail(20)
#displaying the data frame

df_plot_top
#lets plot this inside the map corresponding to the cities exact co-ordinates which we
received from google api
#plt.subplots(figsize=(20,50))
plt.figure(figsize=(50,60))
map=Basemap(width=120000,height=900000,projection="lcc",resolution="l",llcrnrlon
=67,llcrnrlat=5,urcrnrlon=99,urcrnrlat=37,lat_0=28,lon_0=77)
map.drawcountries()
map.drawmapboundary(color='#f2f2f2')
map.drawcoastlines()
lg=np.array(df_plot_top["lng"])
lat=np.array(df_plot_top["lat"])
pt=np.array(df_plot_top["COUNT"])
city_name=np.array(df_plot_top["CITY"])
x,y=map(lg,lat)
#using lambda function to create different sizes of marker as per thecount
p_s=df_plot_top["COUNT"].apply(lambda x: int(x)/2)
#plt.scatter takes logitude ,latitude, marker size,shape,and color as parameter in the
below , in this plot marker color is always blue.

plt.scatter(x,y,s=p_s,marker="o",c='BLUE')
plt.title("TOP 20 INDIAN CITIES RESTAURANT COUNTS PLOT AS PER
ZOMATO",fontsize=30,color='RED')
#lets plot this inside the map corresponding to the cities exact co-ordinates which we
received from google api ,here marker color will be different as per marker size
map.drawcountries()
x,y=map(lg,lat)
below , in this plot marker color is different.
plt.scatter(x,y,s=p_s,marker="o",c=p_s)
#lets plot with the city names inside the map corresponding to the cities exact co-
ordinates which we received from google api ,here marker color will be different as per
marker size
map.drawcountries()
x,y=map(lg,lat)

for a,b ,c,d in zip(x,y,city_name,pt):
#plt.text takes x position , y position ,text ,font size and color as arguments
plt.text(a,b,c,fontsize=30,color="r")
#lets plot with the city names and restaurants count inside the map corresponding to the
cities exact co-ordinates which we received from google api ,here marker color will be
different as per marker size
map.drawcountries()
x,y=map(lg,lat)
for a,b ,c,d in zip(x,y,city_name,pt):
#plt.text takes x position , y position ,text(city name) ,font size and color as arguments
plt.text(a,b,c,fontsize=30,color="r")
#plt.text takes x position , y position ,text(restaurant counts) ,font size and color as
arguments, like above . but only i have changed the x and y position to make it more
clean and easier to read
plt.text(a+60000,b+30000,d,fontsize=30)

Sample Output

RESULT
Thus the visualization of Zomato geographic data was visualized using Basemap.

CS3362 - Data Science Laboratory - Manual - Final-1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CS3362 - Data Science Laboratory - Manual - Final-1

Uploaded by

Copyright:

Available Formats

CS3362 DATA SCIENCE LABORATORY LTPC

CS3362 Data Science Laboratory 1

CS3362 Data Science Laboratory 2

CS3362 Data Science Laboratory 3

CS3362 Data Science Laboratory 4

1.2 Array Shape & Reshaping

1.2.2 Array Reshaping

CS3362 Data Science Laboratory 5

2.2 Normal (Gaussian) Distribution

CS3362 Data Science Laboratory 6

2.4 Poisson Distribution

2.5 Uniform Distribution

2.6 Logistic Distribution

CS3362 Data Science Laboratory 7

2.7 Multinomial Distribution

2.8 Exponential Distribution

2.9 Chi Square Distribution

2.10 Rayleigh Distribution

CS3362 Data Science Laboratory 8

2.11 Pareto Distribution

2.12 Zipf Distribution

CS3362 Data Science Laboratory 9

CS3362 Data Science Laboratory 10

3.2 Rounding Decimals

CS3362 Data Science Laboratory 11

3.7 LCM (Lowest Common Multiple

CS3362 Data Science Laboratory 12

3.8 GCD (Greatest Common Denominator)

3.9 Trigonometric Functions

Find sine values for all of the values in arr:

3.10 Hyperbolic Functions

Find cosh values for all of the values in arr:

3.11 Set Operations

CS3362 Data Science Laboratory 13

3.11.3 Finding Intersection

CS3362 Data Science Laboratory 14

CS3362 Data Science Laboratory 15

CS3362 Data Science Laboratory 16

2.1 CSR(Compressed Sparse Row) Matrix

CS3362 Data Science Laboratory 17

3.2 Depth First Order

Traverse the graph depth first for given adjacency matrix:

CS3362 Data Science Laboratory 18

CS3362 Data Science Laboratory 19

4.2 Convex Hull

CS3362 Data Science Laboratory 20

4.4.1 Euclidean Distance

4.4.2 Cosine Distance

CS3362 Data Science Laboratory 21

Import Data from Matlab Format

CS3362 Data Science Laboratory 22

6.2 Spline Interpolation

CS3362 Data Science Laboratory 23

CS3362 Data Science Laboratory 24

1.1 Create Labels

1.2 Key/Value Objects as Series

CS3362 Data Science Laboratory 25

CS3362 Data Science Laboratory 26

5.1 Info About the Data

6.1 Empty Cells

6.1.2 Replace Empty Values

CS3362 Data Science Laboratory 27

6.1.3 Replace Using Mean, Median, or Mode