Statistics for Data Science - 2

Formula file

Discrete random variables:

Distribution PMF (fX (k)) CDF (FX (x)) E[X] Var(X)


 0 x<0
1
,x=k

 k−a+1

k ≤x<k+1
Uniform(A) n
n a+b n2 −1
n=b−a+1
A = {a, a + 1, . . . , b} k = a, a + 1, . . . , b − 1, b 2 12
k = a, a + 1, . . . , b



x≥n

1

0 x<0
( 
p x=1
Bernoulli(p) 1−p 0≤x<1 p p(1 − p)
1−p x=0 
1 x≥1



 0 x<0
k


n
Ci pi (1 − p)n−i

P
n
Ck pk (1 − p)n−k , k ≤x<k+1
Binomial(n, p) i=0 np np(1 − p)
k = 0, 1, . . . , n
k = 0, 1, . . . , n





1 x≥n


0
 x<0
1 1−p
(1 − p)k−1 p, k
Geometric(p) 1 − (1 − p) k ≤ x < k + 1
k = 1, . . . , ∞  p p2
k = 1, . . . , ∞


 0 x<0
e−λ λk


 k λi
Poisson(λ) , e−λ
P
k ≤x<k+1 λ λ
k!
k = 0, 1, . . . , ∞ 
 i=0 i!

 k = 0, 1, . . . , ∞
 Continuous random variables:

Distribution PDF (fX (k)) CDF (FX (x)) E[X] Var(X)


 0 x≤a
(b − a)2

x − a
1 a+b
Uniform[a, b] ,a≤x≤b a<x<b
b−a 
 b−a 2 12
1 x≥b

(
0 x≤0 1 1
Exp(λ) λe−λx , x > 0
1 − e−λx x > 0 λ λ2
−(x − µ)2
 
1
√ exp ,
Normal(µ, σ 2 ) σ 2π 2σ 2 No closed form µ σ2
−∞ < x < ∞
β α α−1 −βx α α
Gamma(α, β) x e ,x>0
Γ(α) β β2
Γ(α + β) α−1
x (1 − x)β−1 α αβ
Beta(α, β) Γ(α)Γ(β)
α+β (α + β)2 (α + β + 1)
0<x<1

1. Markov’s inequality: Let X be a discrete random variable taking non-negative values

with a finite mean µ. Then,
µ
P (X ≥ c) ≤
c
2. Chebyshev’s inequality: Let X be a discrete random variable with a finite mean µ
and a finite variance σ 2 . Then,
1
P (| X − µ |≥ kσ) ≤
k2

3. Weak Law of Large numbers: Let X1 , X2 , . . . , Xn ∼ iid X with E[X] = µ, Var(X) =

σ2.
X 1 + X2 + . . . + X n
Define sample mean X = . Then,
n
σ2
P (|X − µ| > δ) ≤
nδ 2

4. Using CLT to approximate probability: Let X1 , X2 , . . . , Xn ∼ iid X with E[X] =

µ, Var(X) = σ 2 .
Define Y = X1 + X2 + . . . + Xn . Then,
Y − nµ
√ ≈ Normal(0, 1).
nσ

Page 2
 5. Bias of an estimator: Bias(θ̂, θ) = E[θ̂] − θ.
1P n
6. Method of moments: Sample moments, Mk (X1 , X2 , . . . , Xn ) = Xk
n i=1 i
Procedure: For one parameter θ

• Sample moment: m1
• Distribution moment: E(X) = f (θ)
• Solve for θ from f (θ) = m1 in terms of m1 .
• θ̂ : replace m1 by M1 in the above solution.

7. Likelihood of i.i.d. samples: Likelihood of a sampling x1 , x2 , . . . , xn , denoted

n
Y
L(x1 , . . . , xn ) = fX (xi ; θ1 , θ2 , . . .)
i=1

8. Maximum likelihood (ML) estimation:

n
Y
θ1∗ , θ2∗ , . . . = arg max
∗ ∗
fX (xi ; θ1 , θ2 , . . .)
θ1 ,θ2 ,...
i=1

9. Bayesian estimation: Let X1 , . . . , Xn ∼ i.i.d.X, parameter Θ.

Prior distribution of Θ : Θ ∼ fΘ (θ).
Samples, S : (X1 = x1 , . . . , Xn = xn )
Posterior: Θ | (X1 = x1 , . . . , Xn = xn )
Bayes’ rule: Posterior ∝ Prior × Likelihood
Posterior density ∝ fΘ (θ) × P (X1 = x1 , . . . , Xn = xn | Θ = θ)

10. Normal samples with unknown mean and known variance:

X1 , . . . , Xn ∼ i.i.d. Normal(M, σ 2 ).
Prior M ∼ Normal(µ0 , σ02 ).
nσ02 σ2
  
Posterior mean: µ̂ = X + µ0
nσ02 + σ 2 nσ02 + σ 2

Page 3
11. Hypothesis Testing

• Test for mean

Case (1): When population variance σ 2 is known (z-test)

Test H0 HA Test statistic Rejection region

T =X
right-tailed µ = µ0 µ > µ0 X − µ0 X>c
Z= σ√
/ n
T =X
left-tailed µ = µ0 µ < µ0 X − µ0 X<c
Z= σ√
/ n
T =X
two-tailed µ = µ0 µ ̸= µ0 X − µ0 |X − µ0 | > c
Z= σ√
/ n

Case (2): When population variance σ 2 is unknown (t-test)

Test H0 HA Test statistic Rejection region

T =X
right-tailed µ = µ0 µ > µ0 X − µ0 X>c
tn−1 = S/√n

T =X
left-tailed µ = µ0 µ < µ0 X − µ0 X<c
tn−1 = S/√n

T =X
two-tailed µ = µ0 µ ̸= µ0 X − µ0 |X − µ0 | > c
tn−1 = S/√n

Page 4
 • χ2 -test for variance:

Test H0 HA Test statistic Rejection region

(n − 1)S 2
right-tailed σ = σ0 σ > σ0 T = ∼ χ2n−1 S 2 > c2
σ02

(n − 1)S 2
left-tailed σ = σ0 σ < σ0 T = 2
∼ χ2n−1 S 2 < c2
σ0
α
(n − 1)S 2 S 2 > c2 where = P (S 2 > c2 ) or
two-tailed σ = σ0 σ ̸= σ0 T = ∼ χ2n−1 2
σ02 α
S 2 < c2 where = P (S 2 < c2 )
2

• Two samples z-test for means:

Test H0 HA Test statistic Rejection region

T =X −Y
σ12 σ22
 
right-tailed µ1 = µ2 µ1 > µ2 X −Y >c
X − Y ∼ Normal 0, + if H0 is true
n1 n2
T =Y −X
σ22 σ12
 
left-tailed µ1 = µ2 µ1 < µ2 Y −X >c
Y − X ∼ Normal 0, + if H0 is true
n2 n1
T =X −Y
σ12 σ22
 
two-tailed µ1 = µ2 µ1 ̸= µ2 |X − Y | > c
X − Y ∼ Normal 0, + if H0 is true
n1 n2

• Two samples F -test for variances

Test H0 HA Test statistic Rejection region

S12 S12
one-tailed σ1 = σ2 σ1 > σ2 T = ∼ F(n1 −1,n2 −1) >1+c
S22 S22

S12 S12
one-tailed σ1 = σ2 σ1 < σ2 T = ∼ F(n1 −1,n2 −1) <1−c
S22 S22
S12 α
> 1 + cR where = P (T > 1 + cR ) or
S12 S22 2
two-tailed σ1 = σ2 σ1 ̸= σ2 T = ∼ F(n1 −1,n2 −1)
S22 S12 α
< 1 − cL where = P (T < 1 − cL )
S22 2

Page 5