Python-II (Part a)

Python For Data Scientists
Department of Financial Mathematics

University of Karachi
Engineer Syed Umaid Ahmed
BE (EE), ME (Mechatronics)
NED University of Engineering & Technology
Telephone
Smart Phone
Diary
Advanced Era Address Book
Designed by Syed Umaid Ahmed 2

Cloud Storage
Floppy Disk
Hard Disk
Smart Cars with Sensors

Internet of Things (IoT)
 We produce quintillion bytes of data each day1810
 Various device are connected through Internet and they communicate with each other
 Millions of devices around us like refrigerators, cars, machines and dish washers are producing da
21
 This data is measured in Zettabytes 10
 One Zettabyte is equal to trillion Gigabytes (GB)

Why we need Data Science ?
 All the traditional data processing methods fail to analyze these big amount of data
 We can’t handle these trillion Megabytes of data
 That’s where the data science comes in !

Social Media Platforms
Facebook
Twitter
 Every minute 1.7 million pictures are uploaded on Instagram Instagram

 On Twitter 347,222 tweets are published in a minute

4,166,667 likes and 200,000 photos every minute LinkedIn
Other Sources of Data Generation

Walmart Used Case ?
 Walmart is retailer with 20,000 stores in 28 countries
 They used data to get useful insights about customer shopping patterns
 They used and analyzed data from their customers

Halloween and Sales of Cookies
 During Halloween, a specific cookie was popular around all Walmart stores
 All of stores were selling them very well but two stores were not selling them at all
 The situation was investigated and it was found that there was simple stocking oversight
 Cookies weren’t put on shelves for sale
Hurricane and Strawberry Pop Tarts

 They found the relation between Hurricane and Pop Tarts using Data Mining
 It’s a random practice of people
 So they started increasing the sales of Pop Tarts

Social Media and their Trending Product
 They found out their trending product through social media platforms
 They are quite smart in analyzing the data through person likes or dislikes
 They saw that Facebook users were crazy about cake pops so they introduce it immediately
on their stores
 The success behind this that they don’t see the huge data as burden but they are using it for bene
 Data worth's a lot !
 Theyprocess data, analyze it and get useful insights from it
 They find hidden patterns after spending lots of time on data
Designed by Syed Umaid Ahmed

10
What is Data Science ?
“Torture the data and it will confess to anything”
- Ronal Coase, Economics, Nobel Prize
Data Science is the process of extracting knowledge and insights from data by using scientific methods
Programming +Statistics + Business
Netflix is the using the process of data science for exploring user interests
Facebook and other social media platforms are using same exploration techniques
How the data scientists get useful insights from the data ?

 Data Scientist must have good skill in Mathematics, Statistics and Probability
 They should be good in terms of Technology
 Business problems can be solved by Data Scientists
Skill Set Required for Data Science
 Machine Learning
 Statistics
 Big Data Processing (Unstructured)
 Programming Language
 Data Visualization
 Data Extraction and Processing
 Probability
 Data Wrangling (Cleaning) and Exploration

Data Life Cycle
Business Requirements 1 Data Exploration 4
 Identify the Problem  Understand the patterns in your data
 Identify Central Objectives  Retrieve useful insights
 Identify Variable that needs to be predicted
Data Acquisition (Mining) 2 Data Modeling 5

 What I need for the project ?  First slitting data into test and train dataset
 What are the sources of data ?  Build a model buy using training dataset
 How can I obtain the data ?  Evaluate model by testing dataset
 What is the most efficient way to store and access all of it ?
Data Processing 3 Deployment 6

 Transform data into desired format  Check deployment and efficiency for dependency issue
 Data Cleaning  Deploy model in test environment
o Missing values  Monitor the performance
o Corrupted data
o Remove unnecessary data
o Time consuming
Statistics and Probability
 Mathematics is embedded in every part of nature e.g. in the petals of a flower
Agenda: What is data ?
Categories of data
What is Statistics ?
Basic Terminologies in Statistics
Sampling Techniques
Types of Statistics
Descriptive Statistics
Probability
Inferential Statistics

Data refers to facts and statistics collected together for reference or analysis

Qualitative data can’t be measured easily but can be observed subjectively
Data with no inherent order or ranking such as gender or race, such data is nominal data
What is your gender ?
What is your hair color ?
What do you live ?
Data with ordered series, such data is ordinal data (SEQUENCE)
ID:0 Good
ID:1 Bad
ID:2 Average

Quantitative data can deal with things that be measured objectively
Discrete Data:
 Discrete data known as categorical data, it can hold finite number of possible values (CAN BE COUNTED)
 No of students in class, Eggs in a Basket, Likes on Facebook Post, Pages in a Book
Continuous Data:
• Continuous data that can hold infinite number of possible values (IN A CERTAIN RANGE)
• Windspeed, Water Temperature, Voltages of Electricity
• The amount of time to complete a project
• The speed of cars
• Time to wake up

What is Statistics ?
It is an area of applied mathematics which is concerned with the data collection
analysis, interpretation and presentation

The company has created a medicine to cure cancer. How would you test it’s effectiveness through
statistics ?
 You and friend have a bit that Shahid Afridi will make 24 runs in next over
 Sales data have come in the company. The manager ask you to make a report
That on which the company is lacking ? Where should we invest.
What should you look for ?

Basic Terminologies in Statistics
Population:
A collection or set of individuals or objects or events whose properties are to be analyzed
Sample:
Sample of the population, A well chosen sample will contain most of the information about
a particular population parameter

Sampling Techniques
Eating habits of 42 millions teens in United States and their health, There count is growing daily
How we collect their data ?
 We have to each and every house
 Will we knock every home door that please tell us about your son ?
 What are the things he eats daily ?
Therefore Sampling is a shortcut to study the entire population
You are going to take a sample, apply your statistics to the particular sample
This sample will help to represent the characteristics of entire population

Random:
Each individual has chance of being selected

Systematic:
Every nth record is chosen from the
population as part of sampling
Stratified:
A stratum is a set of population that shares at least

one common characteristic, incase
its gender male or female
Random sampling is used to select a sufficient

number of subjects from each stratum

Types of Statistics
a. Descriptive Statistics
• This statistics use data to provide descriptions of the population, either through numerical calculations
or graphs or tabled
• It is basically focused on main characteristics of data. It provides graphical summary of the data
Case:
Suppose you want to gift T-shirts to all the students of your class.
The steps you will take:
Take the size of all students
By applying descriptive statistics, you will find how many maximum, minimum & average shirt size

b. Inferential Statistics
• This statistics make inferences and predictions about a population based on the sample of data taken
• It generalizes a large dataset and applies probability to draw a conclusion
Larg
e
Mediu
m
Sma
ll
You’ve grouped the people in large medium and small
Take some set of people from class
Inferential statistics allows us to infer data parameters based on statistical model using a sample data

Descriptive statistics is a method used to describe and understand the features of a specific dataset by giving
Short summaries about the samples and measures of data
There are two important measures of Descriptive Statistics:
 Measures of Central Tendency

 Measures of Variability (spread)
Measures of Central
Tendency
Mean Median Mode

Descriptive statistics is a method used to describe and understand the features of a specific dataset by giving
Short summaries about the samples and measures of data
There are two important measures of Descriptive Statistics:
 Measures of Central Tendency

 Measures of Variability (spread)
Measures of spread
Standard
Range Inter Quartile Range Variance
Deviation

Mean: It is the average of all values in a sample

Median: It is measure of central value of the sample set is called median

Mode: It is value most recurrent in the sample set is called Mode
Mode: The most repeated value is called Mode
22,3,4,34,4,8,8,9,5,7,12,45,89,4,4,9,0,12 MODE

In the last lecture we’ve studied how the data is spread around the central,
Which is the measure ofCentral Tendency: Mean, Median & Mode
Now the practical performance of the measures of Central Tendency
1. Generate Random Data

2. Generate Mean, Median and Mode of the Data

Example # 2

2. Extract Sample from the Data and Find mean to generalize Information

Cheat Sheet of Numpy Usage
import numpy as np # Import Library
np.random.randint() # Generate a Random Number
np.random.randint(10,20,200) # Generate Random Numbers (Start, End, Dataset)
Stores 50 random numbers between 12-47 in a variable
a = np.random.randint(12,47,50)
np.mean(a) # Calculate the mean of the data
np.median(a) # Calculates the median of the data
from statistics import mode # For mode first import it from statistics
mode(a) # Calculates the mode of the data
sample_1 = np.random.choice(a, 10)

Measures of Spread:
Measures of spread
Inter Quartile Standard

Range Range Variance
Deviation

Examples of Range (Highest – Lowest)
We have find the range easily but we haven’t understand that we are going to do basically ?
What we are going to predict and interpret ?

Quartile: It tells about the spread of a dataset by breaking data into quarters.

What is Inter Quartile Range ?
It tells about the measure of variability, based on dividing the dataset into quartiles
‫ﺗﻐﯿﺮ‬
Quartile divides a rank-ordered dataset into four equal parts which is Q1, Q2, Q3, rest
The Inter Quartile Range (IQR) is equals to
Q3 – Q1

Variance
Populatio
Sample
n
Variance
Variance
Population Variance : It is the average of squared deviations
For Understanding
of Sample & Population
Move toSlide # 19

Sample Variance : It is the average of squared differences from the mean

Standard Deviation: It is the measure of thedispersion of a set of data from its mean
Check the examples provided below for clearing all the concepts

Preparing Your Understanding For all Above
Formulae

Example # 2

Standard Deviation: It is the measure of thedispersion of a set of data from its mean
The greater the value of the standard deviation, the further the data tend to be
dispersed from the mean

Some Mind Blowing Facts
0
American Men : Mean of Heights
-0.15%
0.15%
5’ 1’’ 5’ 4’’ 5’ 7’’ 5’ 10’’ 6’ 1’’ 6’ 4’’ 6’ 7’’

What interesting it suggests ?

Entropy (A Brief Explanation)
Entropy comes from Physics, so we can start it from the states of matter
Entropy is freedom to move, Ice (very low), Liquid (Less), Gas (Greater)
Pick a ball from Container 1, 2 & 3
Combinations
Direct Relation Between

Entropy and Knowledge
ENTROPY : Low Medium High

High Knowledge Medium Knowledge Less Knowledge
Formulae Entropy and Information Gain
Information Gain = Entropy (Parent) – Weighted Average x Entropy (Children)

Information Gain (A Brief Explanation)

Forecast whether game is to be played or not ?

Step # 1 Check the Target variable Yes = 9, No = 5
Total = 14
Step # 2 Calculate Entropy of Target (Parent)
Entropy (Parent) =
Entropy (Parent) = 0.940

Step # 3 For Making Decision Tree, Choose Root Node,
Brake all columns into pieces
strong weak

Step # 4
Check the Information Gain of Each Column, The Highest Information Gain will be the Top Node
strong weak

Highest Information Gain

Decision Tree is Here !!!

Assignment Question:

Confusion Matrix and its KPI’s
(for Classification Models)

1. Generate Raw Data with Numpy

2. Three values from sklearn.metrics(Accuracy, Confusion Matrix, Precision, Recall, Report)

Confusion Matrix Example Case

What is Probability ?
Probability is the measure of how likely an event will occur
Desired Outcomes
Total Outcomes
If there are six faces of a dice, What is probability of getting a number 3 ?

1/6

Relationship
P = Model (Causes) to Predict Events
S = Past Events To Causes
Statistics
Basic Terminologies in Probability
Random Experiment: (Result can’t be predicted but know ALL possibilities)
An experiment or a process for which a result cannot be predicted with certainty
e.g. Rolling a dice is a random experiment, Cards
Sample Space: (If we put all & all results of Random Experiment in “Set”, it’s called sample space)
The entire possible set of outcomes for a Random experiment

e.g. The 1,2,3,4,5,6 are there, you need to find probability of 2
Event: (The one element who come in sample space)

One or more outcomes of an experiment, It is a subset of sample space
e.g. my event is probability of getting a 2

Disjoint Event:
1. A man can be dead or alive
2. A student can be passed or fail
3. A single card drawn can be King or Queen
Non-Disjoint Event:
1. A student can be obtain 100 marks in accounting and 100 marks in Python
2. An outcome of a ball can be no ball and sixer both as well

Probability Distribution Terminologies
1. Probability Density Function
2. Normal Distribution
3. Central Limit Theory s
?
o u
ti n
o n
C
1. Probability Density Function
The equation describing a continuous probability distribution is called a Probability Density Function
PDF gives the probability of a variable lies between Range A and B
Trying to find probability of continuous random variable lies in range

Bell Curve
Properties
1 Graph of a PDF will be continuous
2 Area bounded by the curve of density function and the x-axis is

equals one
i.e., Basically it’s the area below the curve
a b
3 Probability that a random variable assumes a value between a and b
Is equal to the area under the PDF bounded by a and b
2. Normal Distribution (Gaussian Distribution)
The graph of the normal distribution depends upon two factors:
a. Mean: It determines the “center” of the graph
b. Standard Deviation: It determines the “height” of the graph

3. Central Limit Theory
If we had a large population and we divide it many samples, the mean of all samples will almost be equal to populati
Different Types of Probability
1. Marginal Probability
It is the probability of the occurrence of a single event
Take an example of selecting a Heart Card from the Deck of 52 Cards
13/52

2. Joint Probability
It is the measure of two events occurring at the same time
Probability of a card is Ace and Red. It is joint Probability means Intersection Property
This Photo by Unknown

Author is licensed under CC
2/52
BY-SA

3. Conditional Probability

3. Conditional Probability
Probability of an event or outcome based on the occurrence of a previous event or outcome
Conditional Probability of an Event B is the probability that the event will occur given that Event A has already occurre
If A and B are dependent events than the If A and B are independent events than the
expression for conditional probability is: expression for conditional probability is:
P(B|A) = P(A and B) / P(A) P(B|A) = P(B)

Edureka’s Used Case
1. Analyze the data
2. It is formatted and structured already
3. Draw your conclusions

Question:1 Find the Probability that a candidate has gone Edureka’s Training ?
MARGINAL PROBABILITY 45/105 = 0.42

Question:2 Find the Probability that a candidate has gone Edureka’s Training + Good Package
JOINT PROBABILITY 30/105 = 0.28

5 30 Designed by Syed Umaid Ahmed 95
Question:3 Find the Probability that a candidate has good package and not undergone Edureka’s Trainin
CONDITIONAL PROBABILITY 5/60 = 0.08

3

Baye’s Theorem
• It is used as the building block in Naïve Baye’s Machine Learning Technique
• It is easy, elegant and old Theorem with powerful implementation

Baye’s Theorem
• It is used in Gmail spam folder
• Most of you have noticed it in your email addresses. This is done through ML by Naïve Bayes

Take Example of Applied AI from Wiki
A factory produces an item using three machines—A, B, and C—which account for 20%, 30%, and 50%
of its output, respectively.
Items produced by machine A, 5% are defective; similarly, 3% of machine B’s items and 1% of
machine C's are defective.
If a randomly selected item is defective, what is the probability it was produced by machine C?
P(A1) = 0.2 P(A2) = 0.3 P(A3) = 0.5
Defective and Produced from Machine A : Condition Found, Probability is Conditional
P(B|A1) = 0.05 P(B|A2) = 0.03 P(B|A3) = 0.01

STEP#1
Probability of Total Defective ?
P(B|A1)*P(A1) + P(B|A2)*P(A2) + P(B|A3)*P(A3)
(0.05)*(0.2) + (0.03)*(0.3) + (0.01)*(0.5)
Answer of only defective Total = 0.024

STEP#2 (Important) Baye’s Theorem
Probability that the defected part comes from Machine C ? (First See the Formula in Black Backgroun
P(A3 | B) = P (B |A3) P(A3)

Defect
P(B)
= 0.01 x 0.50
0.024
Probability of Coming from C ?
= 5/24

Inferential Statistics
• This statistics make inferences and predictions about a population based on the sample of data taken
• It generalizes a large dataset and applies probability to draw a conclusion
• This is done by different methods. i.e, point estimation, and interval estimate

Confidence Interval

Hypothesis Testing
Statisticians use hypothesis testing to formally check whether the hypothesis is accepted or rejected
It is performed in the following ways:
1. State the Hypothesis: This involves stating Null Hypothesis (Ho) and Alternative hypothesis (H1)
Innocent Until we don’t have any Information, all innocent

(Null Hypothesis) Ho
Guilty After the Information

(Alternative Hypothesis) H1

Formulate an Analysis Plan: This state involves construction of analysis plan
Analyze Sample Data: Calculation and Interpretation of test statistics as described in plan
Interpret Results: Involves the application of decision rule described in analysis plan
Used Case
Lena Catherine Furqan Asad
They all bunks a class, So as a punishment they have to clean the classroom every
week.
Furqan decided that we put he name of everyone in a chit and select one chit everyday.
One who got a name will clean the room. Three days passed, there is no name of
Furqan
Three days passed, Everyone name came out except Furqan.
Assume that the event is Free of Bias. What is Probability that John name is not coming out !
What is Probability that he is not cheating ?
P (Furqan not picked for a day) = ¾
P (Furqan not picked for three days) = ¾ x ¾ x ¾ x ¾ = 0.42 (Approx.)
P (Furqan not picked for 12 days) = ¾ x ¾ x ¾ x ¾ x ¾…12 times = 0.032 < 0.05
Probability of Furqan Cheating is Fairly High, Statisticians define a threshold value

Considering the situation:
Probability of Event < 0.05 (5%) means the Event is Biased

Machine Learning
It provides the machine, the ability to learn automatically
and improve the experience without the need of being
explicitly programmed.
• Extract Useful Insights
• Make Decisions
• Handle Big Data and Extract Patterns
i.e.
Netflix Recommendations, Facebook Tagging &
Gmail Spam Filtering

Machine Learning Terms and Definitions
 Algorithm: A set of rules and statistical techniques used to learn patterns from the data
 Model: A model is trained by using a Machine Learning Algorithm
 Predictor Variable: It is the feature of the data used to predict the output
 Response Variable (Target): The variable that needs to be predicted
 Training Data: The Machine Learning model is built using Training Data, It is splitted into two parts, (8
 Testing Data: The Machine Learning Model is always evaluated using Testing Data. It is small (20%)

By 2020 onwards, 1.7MB of data is generated by every person on Earth
Make method that makes sense of data using Machine Learning
Why Machine Learning is Important ?
Lot of Data Generation
Improve Decision Making
Forecast sales, Downfall in Stock Market, Anamolies
Uncover Data Insights

Machine Learning Process

Weather Forecasting using Machine Learning
Step 1 Define the objective of the Problem
To predict the possibility of rain by studying the weather conditions
1. What we are trying to predict ?
2. What are the target features ?
3. What is the input data ?
4. What kind of problem are we facing ? Binary Classification or Clustering ?
Step 2 Data Gathering and Collection

1. Which type of data is required to predict ?
2. Data such as Humidity, Temperature and Pressure
3. Download the dataset or used Web Scrapping

Step 3 Preparing Data
1. Data is never in required format. You need to clean it
2. Fill in the missing values, redundant variables, remove unnecessary data, duplicate values and so on.
3. Transform into required format
Step 4 Exploratory Data Analysis
1. Become a Detective and dive deep into the data
2. Understanding the trends and patterns in the data
3. Co-relation between the range of variables
i.e. If Temperature is low, it will be raining

Step 5 Building a Machine Learning Model
1. Machine Learning model is built by using training data
2. The model is the Machine Learning algorithm that predicts the output by using data fed to it
3. The Algorithm may be decision trees, Linear Regression, etc
Step 6 Model Evaluation and Optimization
1. It is evaluated by using testing dataset
2. The accuracy of the model is calculated
3. Further modification is done in the model using techniques like Parameter Tuning, Cross Validations

Step 7 Predictions
1. The final outcome is predicted after performing parameter tuning and improving the accuracy of model
2. It is used to predict he final outcome.
3. It may be a categorical variable or Continous quantity

Link Two Videos of Murtaza Robotics and AI:
https://www.youtube.com/watch?v=kmSab3AuLdY&list=RDCMUCYUjYU5FveRAscQ8V21w81A&start_radio=1&t=3
https://www.youtube.com/watch?v=xs99sViYwJA&list=PLMoSUbG1Q_r-rw4MC03RUrzQzm0NuBCRj&index=6
https://www.youtube.com/watch?v=VjHBotwi5a8&list=RDCMUCYUjYU5FveRAscQ8V21w81A&index=3
Corona Predictor:
https://www.youtube.com/watch?v=6CZiz-FLZF0&list=PLMoSUbG1Q_r-rw4MC03RUrzQzm0NuBCRj&index=8


Python-II (Part a)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Python-II (Part a)

Uploaded by

Copyright:

Available Formats

Python For Data Scientists

Department of Financial Mathematics

Advanced Era Address Book

Designed by Syed Umaid Ahmed 2

Smart Cars with Sensors

 One Zettabyte is equal to trillion Gigabytes (GB)

Designed by Syed Umaid Ahmed 4

 We can’t handle these trillion Megabytes of data

 That’s where the data science comes in !

Designed by Syed Umaid Ahmed 5

 Every minute 1.7 million pictures are uploaded on Instagram Instagram

Designed by Syed Umaid Ahmed 7

 They used and analyzed data from their customers

Designed by Syed Umaid Ahmed 8

 Cookies weren’t put on shelves for sale

Hurricane and Strawberry Pop Tarts

 It’s a random practice of people

 So they started increasing the sales of Pop Tarts

Designed by Syed Umaid Ahmed 9

 Data worth's a lot !

 Theyprocess data, analyze it and get useful insights from it

 They ﬁnd hidden patterns after spending lots of time on data

Designed by Syed Umaid Ahmed

Programming +Statistics + Business

Designed by Syed Umaid Ahmed 11

 They should be good in terms of Technology

 Business problems can be solved by Data Scientists

Skill Set Required for Data Science

Designed by Syed Umaid Ahmed 12

Data Acquisition (Mining) 2 Data Modeling 5

Data Processing 3 Deployment 6

Agenda: What is data ?

Basic Terminologies in Statistics

Designed by Syed Umaid Ahmed 14

Designed by Syed Umaid Ahmed 15

What is your gender ?

What is your hair color ?

What do you live ?

Data with ordered series, such data is ordinal data (SEQUENCE)

Designed by Syed Umaid Ahmed 16

 No of students in class, Eggs in a Basket, Likes on Facebook Post, Pages in a Book

• Windspeed, Water Temperature, Voltages of Electricity

• The amount of time to complete a project

• The speed of cars

Designed by Syed Umaid Ahmed 17

That on which the company is lacking ? Where should we invest.

What should you look for ?

Designed by Syed Umaid Ahmed 18

A collection or set of individuals or objects or events whose properties are to be analyzed

a particular population parameter

Designed by Syed Umaid Ahmed 19

How we collect their data ?

 We have to each and every house

 What are the things he eats daily ?

Therefore Sampling is a shortcut to study the entire population

This sample will help to represent the characteristics of entire population

Designed by Syed Umaid Ahmed 20

Each individual has chance of being selected

Designed by Syed Umaid Ahmed 21

Every nth record is chosen from the

population as part of sampling

A stratum is a set of population that shares at least

Random sampling is used to select a suﬃcient

Designed by Syed Umaid Ahmed 22