Operators and operands Methods of strings Functions Replace Missing w/ Value *( ) empty bracket leaves original price = stocks.
nctions Replace Missing w/ Value *( ) empty bracket leaves original
line_upper = line.upper() # to upper case
line_lower = line.lower() # to lower case
line_cap = line.capitalize()# first letter
to upper case
line_swap = line.swapcase() # Swap upper
and lower case
line_title = line.title() # Capitalize
the 1st letter of each word
The replace() method  for string
string_uk = string_us.replace('modeling',
'modelling')
List methods  empty_list = []
.append()# item is added to the end of list
.extend([,])# add multiple items to the
Notes: syntax for variable names:
- Only one word list
.insert(2, '')# Insert item at index 2
- Only consist of letters, numbers, and underscores
.remove('')# only removes the first
- Cannot begin with a number
occurrence of specific item
- Avoid contradictions with Python keywords or other
.pop(2)# Remove and return the item at index
variable/function names
2. If the position index is not specified,
Comparison operators
then the pop() method removes the last item
of the list.
output = [n for n in range(1,101) if n%7 ==
0 and n%5 > 0]
print(greetings[::-1]), result: dlroW olleH
Built-in data structures
Membership operators
Tuples
Item = ‘’, ‘’# Comma-separated items, tuples Avg: data.mean( ) #only numerical, incl.
can go without parentheses unlike lists
Item = ('', ‘’) # Comma-separated items
within parentheses
Logical operators feel_empty = () #len() = 0
tuple_one = 'here', # The comma creates a
tuple type object, comma is neccessary
s2 = range(3) # A tuple with three items
mixed = ['Jack', 32.5, (1, 2)] # Comma-
separated items within parentheses
 type(mixed)  <class ‘list’>
 type(mixed[2])  <class ‘tuple’>
item_one = ('there') # Only the parentheses
do not create a tuple  String
range() function  if only 1 argument: stops before that argu
x, y, z = 'abc' # Unpack a string of
three characters
cage=’bad’ trav=’good’ 
cage,trav=trav,cage #swapping variables
for outcome, prob in zip(outcomes, probs)
#zip function
break vs continue
- Break the loop by break Dictionaries  {'key': 'value'}
for name in stocks.keys() OR stocks:
- Skip the subsequent code by continue
a_string = 'abcdef' print(name,stocks[name])
new_string = '' for price in stocks.values():
for letter in a_string: print(price) #print value only
new_string = new_string + letter for item in stocks.items():
if letter == 'c': print(item)#iterate both keys and values values)
break in parallel
print(new_string) a /n b Create a new dictionary all_dao that
print(new_string) include all DAO courses.
abc all_dao = {}
if letter == 'c': #skip this for item in courses:
character to the next iteration if 'DAO' in item:
directly all_dao[item] = courses[item]
continue #skip the subsequen OR for item, value in courses.items():
code, if no code is under continue if 'DAO' in item:
then no code will be skipped  abdef all_dao[item] = value
*The output of the input() function is always a str type *data type of a list inside dictionaries  Series
object Gives a list of ALL possible name combinations
price = stocks.values #convert into array  shows
result = print('What?') unchanged value without index and col name
print(result) # The returned value of the - gdp.subset.fillna(‘ABC’) fill all NaN items w/ ‘ABC’ pd.Series(x).var() #sample variance
print function is None - output=gdp.subset.fillna(0, inplace=True) np.array(x).var() #population variance
print(type(result))# The data type of None original data changed. inplace=True is similar to dropna( ) np.array(x).var() * (n/(n-1)) #= sample variance
Specifying row indexes: set_index(‘colname’) np.array(x).std() * (n/(n-1))**0.5 #sample std
is NoneType
The row indexes can be converted into a new column, while the - Transforming the variances to standard deviations
Notes: In Python programming, all positional arguments must default integer row indexes are recovered, by import numpy as np
be specified before keyword arguments, otherwise there will be the reset_index() method. var = [3.5, 6.2, 7.3, 8.5]
an error message. output.reset_index(drop=True) #drop the original std = np.array(var) ** 0.5
pd.DataFrame .columns & .index index std = list(std)
.astype()  used in pandas objects/ .dtypes: data type of each col Operations on series Discrete random variables
pd.Series  series.values (values of data elements)  Data as series, cannot apply string operations → need convert via from scipy.stats import <>
series.index: RangeIndex(start=0, stop=6, step=1)
.str. first! with .str. , function will be applied to all rows
- The indexer iloc[] is for integer-position based indexes, 1) Vectorized len: eg. is_long =
the stop index is exclusive of the selection.
condo[‘name’].str.len( ) > 20
- The indexer loc[] is for label based indexes, the stop index is - Output as Boolean  condo.loc[is_long] shows all rows w/
inclusive of the selection. names > 20 len
data_frame.iloc[row:row, col:col] 2) Vectorized count: *good for counting no. of words
- pandas.DataFrame objects are also mutable eg. project.str.count('A')
e.g. data_frame.loc[2:3, 'educ'] = 9.0 3) Vectorized Indexing eg. if name starts w D
- The loc[] indexer can even by used to create new columns. lower_names = condo[‘name’].str.lower()
e.g. data_frame.loc[:, 'new col'] = 'value' *convert lowercase for ‘D’
Boolean indexing is typically used for filtering or segmenting d_start = lower_names.str[0]== ‘d’ 
data according to given conditions. condo.loc[d_start]
e.g. is_female = data_frame['gender'] != 'F'
# True if the record is a male
4) Data Type Conversion (convert all elements in series to a P(X ≥ 6)=1−P (X ≤5)
certain type)
Bitwise operators Eg. levels from str to int (’06 to 10’) to 6th floor to 10th floor
"and"  & / "or"  | / "not"  ~ #1 Create New Column #2 Index & Slice with .str. #3 astype(int) 50 multiple-choice questions, given the only correct answer
e.g. cond1 = data_frame['gender']=='F' - eg. condo[‘level_from’] =
among four choices. If John is certain about n = 28 questions and
cond2 = data_frame['married'] condo[‘level’].str[:2].astype(int)
is randomly guessing the remaining questions. Probability that
is_wife = cond1 & cond2 condo[‘level_to] = condo[‘level’].str[-
John can answer at least m=32 questions correctly?
data_frame.loc[is_wife, 'remarks'] = 'Wife' 2:].astype(int) n = 28
new column named "husband" where each element equals to 1 if stocks.pct_change() method to calculate the rate of m = 32
this individual is a male and is married, otherwise the element is change between rows p_right = 1 / 4
0 Numpy p_wrong = 1 - p_right
wage_data.loc[:, 'husband'] = 1 1D: similar to list (1 row)  tuple | 2D: (row, col) (rows are p_answer = binom.cdf(50 - m, 50 - n,
wage_data.loc[(wage_data['female']==1)| vertical down to show no. of rows) p_wrong)
(wage_data['married']==0), 'husband'] = 0
print(p_answer)
Continuous random variables
Boolean (proportion of T)
Median: data.median( ) | SD: data.std( ) |
Var: data.var( )
data.max( ) | data.min( )
data[‘gender’].value_counts( )# Actual count
of each category
data[‘gender’].value_counts(normalize=True)
#Proportion of each category
data.corr()
Create NumPy arrays  numpy.array()
data.cov() e.g. Probability that the amount of the soft drink is between 11.92
e.g. locate specific value ounces to 12.12 ounces
corr_table.loc['educ'].loc['expr']
from scipy.stats import norm
corr_table.loc['educ'].iloc[2] prob = norm.cdf(12.12, 12, 0.05) -
summary = data.describe( ) # will show norm.cdf(11.92, 12, 0.05)
count, mean, std, min, - Inverse of cdf
quartiles(25%,50%,75%),max. median will be
shown as 50% quartile
def sum_sq(x):
Function arange(startdefault=0, stop, step)
output = sum([item**2 for item in x])
.reshape((rows, cols))
x = [1, 1, 1]
np.sin() / np.cos()
y = [2, 2, 2]
print(sum_sq(y))  result: None due to no Functions and array methods
print(np.log(3)) # Natural logarithm of 3
Return in the function
print(np.exp(np.arange(1, 3, 0.5)))
Handling Missing Data (NaN: Not a Number for missing
# Natural exponentials of 1, 1.5, 2, 2.5
print(np.square(np.arange(3)))
Detecting Missing Data: eg. gdp.loc[:,
‘1960’,’1961’].isnull() # Squares of 0, 1, 2
print(np.power(2, np.arange(3)))
isnull( ): Returns True if NaN, False if not missing
# 2 to the power of 0, 1, 2
notnull( ): Returns True if not missing, False if NaN
-Selects only True (not null): eg. gdp1960=
gdp.loc[gdp[‘1960’].notnull()
Dropping Missing *( ) empty bracket default leaves original data
unchanged
- gdpsubset.dropna() returns new data frame w/o missing
value row
- output= gdpsubset.dropna(inplace=True) original
data frame is overwritten, but the dropna() method returns

Poisson
Expected Values and Variances
Var = std ** 2
Let X ∧Y be two random variables ,∧a , b∧c
are three constants , thenwe have
What is the probability that after one trading day, i.e. at time 𝑡=2,
Var (c )=0 your investment is worth less than $990?
2
Var (aX+ c)=a Var ( X )
Var (aX+ bY +c)=a2 Var ( X )+b2 Var (Y )+ 2 abCov(X , Y )
¿ a special case that X∧Y are uncorrelated , that after five trading day, i.e. at time
What is the probability
𝑡=6, your investment is worth less than $990?
the covariance between X ∧Y is zero ,
so the last equation can be written as
2 2 What is the expected value & variance of the investment after
Var ( aX +bY + c ) =a Var ( X )+ b Varfive( Ytrading
) day, i.e. at time 𝑡=6?
If you buy $1000 worth of this stock at time 𝑡=1, what is the
probability that after one trading day, i.e. at time 𝑡=2, your
investment is worth less than $990?
P(Q2 ≤ 990)=P(r 2 ≤ log (990/1000))=P (r 2 ≤ log(0.99))
The log return
If you buy $1000 worth of this stock at time 𝑡=1, what is the
first = names['first']
last = names['last']
all_names = []
for a in first:
for b in last:
all_names.append(a+' '+b)
sample = np.random.normal(mean, std,
size=1000) # 1000 records following a normal distribution
sample = np.random.poisson(mean&variance,
size=1000) # Poisson distribution
rolls = np.random.choice(outcomes, p=probs,
size=(smp_size, n))# discrete random variables
Simulations with continuous random variables
default setting of the var() (or std()) method of NumPy arrays,
the method calculates the population variance (or population
standard deviation). For the sample variance (or sample standard
deviation)
 Specify the keyword argument ddof=1. Here, the argument
ddof indicates the delta degrees of freedom.
 Convert the NumPy array into a series, as shown by the code
below. The var() (or std()) method calculates the sample variance
(or sample standard deviation) by default. E.g.
pd.Series(q6).var()
Simulations with discrete random variables
Alternative
nothing (print(output)  None)
Monte Carlo simulations for decision-making
Alternative
Sampling Distributions
Estimate the mean value via testing a sample that contains a
small fraction of the overall population.
Ensure that each record is selected independently, the keyword
argument replace is specified to be True, meaning that the
samples are selected with replacement.
sample = bulb['Lifespan'].sample(25,
replace=True)
sample.mean()
 In order to illustrate the sampling distribution of the sample
mean, we repeat the sampling experiments for 1000 times
The Central Limit Theorem (CLT): For a relatively large
sample size, the random variable is approximately normally
distributed, regardless of the distribution of the population. The
approximation becomes better with increased sample size.
Confidence intervals
mean_pop = population.mean()
When the population standard deviation 𝜎 is known (z-value)
sigma = population.values.std()
repeat the sampling experiment by 1000 times using the for-loop
Among the 1000 experiments, the chance that the population
mean value 𝜇 falls between the confidence intervals is around
1−𝛼=95%.
When the population standard deviation 𝜎 is unknown (t-value)
Null hypothesis assumes the population proportion 𝑝=𝑝0, we
choose the z-test model (standard normal distribution)
phat=0.5 ¿ phat that maximises the sd
Sample Proportion = p-hat = m/n
The sample proportion is centered at the population proportion p
The variance of the sample proportion is decreased as n increases
The shape of the sampling distribution approaches a normal
distribution as n increases
lower = -1.5
p_in = 0.90 #random variable has a probability of 0.9 to be
population proportion, 𝜇, is less than a given constant value 𝜇0,
𝐻𝑎: 𝜇 < 𝜇0  𝜇 is smaller than 𝜇0, or in other words, the test
checks if 𝜇 is on the left-hand-side of 𝜇0, so it is called a left-
tailed test.
population proportion, 𝜇, is greater than a given constant value
𝜇0, 𝐻𝑎: 𝜇 > 𝜇0  𝜇 is greater than 𝜇0, or in other words, the test
checks if 𝜇 is on the right-hand-side of 𝜇0, so it is called a right-
tailed test.
If 𝑃-value is larger than 𝛼, there is insufficient evidence to
reject the null hypothesis under the given significance level. In
other words, we do not have sufficient evidence to support the
conclusion that the alternative hypothesis is true
If 𝑃-value is lower than 𝛼, we reject the null hypothesis in
favor of the alternative hypothesis, which implies that the
alternative hypothesis is true
1. Hypotheses  prove that the population mean is larger than a
given value, so it is a right tailed test
Null hypothesis: 𝐻0: 𝜇 ≤ 𝜇0 = 1340
Alternative hypothesis: 𝐻𝑎: 𝜇 > 𝜇0 = 1340
2. Sampling distribution
3. Significance from 𝑃-value
two-tailed test
probs = np.array([[0.08, 0.13, 0.09, 0.06,
0.03], [0.03, 0.08, 0.08, 0.09, 0.07],[0.01,
0.03, 0.06, 0.08, 0.08]])
Y = np.arange(5) #Number of purchases
mean = (probs.sum(axis=0)*Y).sum()
var = (probs.sum(axis=0)*(Y -
mean)**2).sum()
Prob of at least 3 purchases  prob_purc[-
2:].sum()
probability that after five trading day, i.e. at time 𝑡=6, your Cut-off value
investment is worth less than $990? cut_value = norm.ppf(1-alpha) * se
cut_value = - norm.ppf(alpha) * se
Distribution of 𝑋
within the interval.
From scipy.stats import norm
p_temp = 1 - p_in - norm.cdf(lower)
upper = norm.ppf(1 - p_temp)
Hypothesis testing
decide whether a population mean, 𝜇, is different from a given
constant value 𝜇0, 𝐻𝑎: 𝜇 ≠ 𝜇0 𝜇 is either larger or smaller than
the constant 𝜇0, so such a test is called a two-tailed test, where 𝜇
could vary from 𝜇0 in two directions.