Review of Statistics

Topics
Descriptive Statistics
Mean, Variance

Probability
Union event, joint event

Random Variables
Discrete and Continuous
Distributions, Moments

Two Random Variables

Covariance and correlation

Central Limit Theorem

Hypothesis testing
z-test, p-value

Simple Linear Regression

Statistical Methods

Statistical
Methods

Descriptive
Statistics

Inferential
Statistics

Descriptive Statistics
Involves
Collecting Data
Presenting Data
Characterizing Data

Purpose
Describe Data

90
80
70
60
50
40
30
20
10
0

East
West
North

1st 2nd 3rd 4th

Qtr Qtr Qtr Qtr

Inferential Statistics
Involves
Estimation
Hypothesis
Testing

Purpose
Make Decisions
About Population
Characteristics

Population?

Descriptive Statistics

Mean

Measure of central tendency

Acts as Balance Point
Affected by extreme values (outliers)
Formula:
n

X =

Xi
i =1

X1 + X2 +
n

+ Xn

Median

Measure of central tendency

Middle value in ordered sequence

If odd n, Middle Value of Sequence
If even n, Average of 2 Middle Values

Value that splits the distribution into two

halves

Not Affected by Extreme Values

Median (Example)
Raw Data: 17 16 21 18 13 16 12 11
Ordered: 11 12 13 16 16 17 18 21
Position: 1 2 3 4 5 6 7 8

Median =

16 + 16
2

= 16

Mode
Measure of Central Tendency
Value That Occurs Most Often
Not Affected by Extreme Values
There May Be Several Modes
Raw Data:
17
16
Ordered:
11
12

21

18

13

16

12

11

13

16

16

17

18

21

Sample Variance
n

S =

(Xi X)
i =1

n 1
2

n - 1 in denominator!
(Use n if population
variance)

(X1 X) + (X2 X) +
n 1

+ (X n X)

Sample Standard Deviation

S=

S
n

(Xi X )
i =1

n 1
2

(X1 X ) + (X 2 X ) +
n 1

+ (Xn X )

Probability

Event, Sample Space

Event:
Sample space:

one possible outcome

collection of all the possible events

S ={

Probability of an outcome: proportion of times that the
outcome occurs in the long run
The complement of event A: includes all the events that
are not part of the event A: Symbol A
Event A { }
Complement of A A {

Properties of an Event
1. Mutually Exclusive
Two outcomes that cannot occur
at the same time

2. Collectively Exhaustive
One outcome in sample space
must occur

Experiment: Observe
gender of one person

Joint Events
Joint event: Event that has two or more characteristics
means intersection of event (set) A and event (set) B
Example:
A and B, (AB): Female, Under age 20

Compound Events
Union of event A and event B ( A B ): Total area of the
two circles
A B contains all the outcomes which are part of event
(set) A, part of event (set) B or part of both A and B
means union of event A and event B

Compound Probability Addition Rule

Used to Get Compound Probabilities for Unions of Events
P(A OR B)

= P(A B)
= P(A) + P(B) - P(A B)

For Mutually Exclusive Events:

P(A OR B)
= P(A B) = P(A) + P(B)
Mutually Exclusive Events

A
B

Random variables
Random variable
numerical summary of a random outcome
a function that assigns a numerical value to each simple
event in a sample space
Discrete or continuous random variables
Discrete: only a discrete set of possible values
=> summarized by probability distribution:
list of all possible values of the variables and the
probability that each value will occur.
Continuous: continuum of possible values
=> summarized by the probability density function
(pdf)

Discrete Probability Distribution

1.

List of pairs [ Xi, P(Xi) ]

Xi = Value of Random Variable (Outcome)
P(Xi) = Probability Associated with Value

2.

Mutually exclusive (no overlap)

3.

Collectively exhaustive (nothing left out)

4.

0 P(Xi) 1

5.

P(Xi) = 1

Joint Probability Using Contingency

Table
Conditional
probability:

Event
Event

B1

B2

Total

A1

P(A1 B1) P(A1 B2) P(A1)

A2

P(A2 B1) P(A2 B2) P(A2)

Total

P(B1)

P(B2)

Joint Probability

Marginal Probability

Joint distribution:

Marginal distributions:
Conditional distribution:

P( A B1 )
P ( A1 B1 )
P ( B1 )
P ( A2 B1 )
P( B1 )

Contingency Table Example

Joint Event: Draw 1 Card. Note Kind, Color
Color
Type
Ace

Red

Black

Total

2/52

2/52

4/52

Non-Ace 24/52

24/52 48/52

26/52

26/52 52/52

Total
P(Red)

P(Ace)

P(Ace AND Red)

Moments Discrete Case

Moment: Summary of a certain aspect of a
distribution
Mean, Expected Value
Mean of Probability Distribution
Weighted Average of All Possible Values
= E(X) = Xi P(Xi)
Variance
Weighted Average Squared Deviation about
Mean
2 = E[ (Xi )2 ] = (Xi )2 P(Xi)

Statistical Independence
When the outcome of one event (B) does not affect the
probability of occurrence of another event (A), the events A
and B are said to be statistically independent.
Example: Toss a coin twice => no causality
Condition for independence:
Two events A and B are statistically independent if and
only if (iff)
P(A | B) = P(A)

Bayes Theorem and Multiplication Rule

Bayes Theorem
P(A | B) =

P(A B)
P(B)

The difficult part is P(A B)

Use above equation to derive P(A B)
P(A and B) = P(A B)
= P(A)P(B | A)
= P(B)P(A | B)
For independent events:
P(A and B) = P(A B) = P(A)P(B)

Covariance
Measures the joint variability of two random variables
N

= (X )(Y )P(X Y )
i
X
i
Y
i, i
XY
i=1

Can take any value in the real numbers

Depends on units of measurement (e.g., dollars, cents,
billions of dollars)
Example:
positive covariance = y and x are positively related;
when y is above its mean, x tends to be above its mean;
when y is below its mean, x tends to be below its mean.

Correlation
Standardized covariance, takes values in [-1, 1]
Does not depend on unit of measurement
Correlation coefficient () formula:

cov( XY ) XY
=
=
X Y
X Y
Covariance and correlation measure only linear dependence!
Example: Cov(X,Y)=0
Does not necessarily imply that y and x are independent.
They may be non-linearly related.
But if X and Y are jointly normally distributed, then they
are independent.

Sum of Two Random Variables

Expected Value of the Sum of Two Random
Variables

E(X + Y) = E(X) + E(Y)

Variance of the Sum of Two Random Variables
2
Var (X + Y) = X+Y
= 2X + 2 Y + 2XY

Continuous Probability Distributions Normal Distribution

Bell-Shaped, symmetrical
Mean, median, mode are equal

f(X)

Infinite range
68% of the data are within 1 standard
deviation of the mean

95% of the data are within 2 standard

deviations of the mean
In early 1800's, German
mathematician and physicist Karl
Gauss used it to
analyze astronomical
data, therefore known
as Gaussian distribution.

Mean, Median,
Mode

Normal Distribution
Probability Density Function
1 X
2

f (X )= 1 e
2
f(X)

=
=
=
=
=

frequency of random variable X

3.14159; e = 2.71828
population standard deviation
value of random variable (- < X < )
population mean

Effect of Varying Parameters ( & )

f(X)
B
A

C
X

Normal Distribution Probability

Probability is the
area under the
curve!

P (c X d ) =

f ( x)dx ?
c

f(X)

Infinite Number of Normal Distribution

Tables
Normal distributions differ by
mean & standard deviation.

Each distribution would

require its own table.

f(X)

X
Thats an infinite number!

Standardize the Normal Distribution

Z=

Normal
Distribution

Standardized
Normal Distribution

z = 1

Z = 0
One table!

Standardizing Example
6
.
2
5

Z=
=
= 0.12

10
Normal
Distribution

= 10

= 5 6.2 X

Standardized
Normal Distribution

Z = 1

Z= 0 .12

Moments: Mean, Variance

(Continuous Case)
Mean, Expected Value
Mean of probability distribution
Weighted average
of
all
possible
values

= E(X) = X f(X) dX
-

Variance
Weighted average squared
deviation about mean
2 = E[ (X )2 ] = (X- )2 f(X) dX
-

Moments: Skewness, Kurtosis

E(X )
Skewness:
S=
Measures asymmetry in distribution
3
The larger the absolute size of the skewness, the more
asymmetric is distribution.
A large positive value indicates a long right tail, and a large
negative value indicates a long left tail. A zero value
indicates symmetry around the mean.
3

E(X )

K=
Kurtosis:
4
Measures thickness of tails of a distribution
A kurtosis above three indicates fat tails or leptokurtosis,
relative to the normal, i.e. extreme events are more likely to
occur.

Central Limit Theorem: Basic Idea

As sample size
gets
large
(n 30) ...

sample mean
will have a normal
distribution.

Important Continuous Distributions

All derived from normal
distribution
2

distribution: arises
from squared normal
random variables,
t distribution: arises
from ratios of normal
2

and
variables
F distribution: arises
2

from ratios of
variables.

distribution

t distribution (red),
normal distribution (blue)

F distribution

Fundamentals of Hypothesis
Testing

Identifying Hypotheses
1.
2.
3.

Question, e.g. test that the population mean is equal to 3

State the question statistically (H0: = 3)
State its opposite statistically (H1: 3)
Hypotheses are mutually exclusive & exhaustive
Sometimes it is easier to form the alternative
hypothesis first.
4. Choose level of significance
Typical values are 0.01, 0.05, 0.10
Rejection region of sampling distribution: the
unlikely values of sample statistic if null hypothesis is
true

Identifying Hypotheses: Examples

1. Is the population average amount of TV viewing
12 hours?
= 12
12
H0: = 12
H1: 12
2. Is the population average amount of TV viewing
different from 12 hours?
12
= 12
H0: = 12
H1: 12

Hypothesis Testing: Basic Idea

Sampling Distribution
It is unlikely
that we would
get a sample
mean of this
value ...

... Therefore, we
reject the null
hypothesis that
= 50.
... if in fact this were
the population mean.
20

= 50

H0

sample mean

Example: Z-test statistic ( known)

1. Convert Sample Statistic (e.g., X ) to Standardized
Z Variable

Z=

X x
x

X
=

n
2. Compare to Critical Z Values
If Z-test statistic falls in critical region, reject H0;
Otherwise do not reject H0

p-value
Probability of obtaining a test statistic more
extreme ( or ) than actual sample value
given H0 is true
Smallest value of for which H0 can be
rejected
Used to make rejection decision
If p value , do not reject H0
If p value < , reject H0

One-Tailed Test: Rejection Region

H0: 0 H1: < 0

H0: 0 H1: > 0

Reject H0

Reject H0

Must be significantly
below .

Here: Small values dont

contradict H0.

One-Tailed Z Test: Finding Critical Z

Values
What Is Z Given = 0.025?
.500
- .025
.475

Z = 1

/2 = .025

Standardized Normal
Probability Table (Portion)

.05

.06

.07

1.6 .4505 .4515 .4525

1.7 .4599 .4608 .4616

1.9 .4744 .4750 .4756

0 1.96 Z

1.8 .4678 .4686 .4693

Two-Tailed Test: Rejection Regions

H0: = 0 H1: 0

Sampling Distribution

Level of Confidence

Rejection
Region

Rejection
Region
1-

1/2

1/2

Nonrejection
Region

Critical
Value

H0
Sample Statistic
Value Critical
Value

t-test, F-test
Test statistic may not be normally distributed
=> z-test not applicable
Examples: Variance unknown, but estimated.
Hypothesis that the slope of a regression line differs
significantly from zero.
=> t-test
Hypothesis that the standard deviations of two normally
distributed populations are equal.
=> F-test

Jarque-Bera test
Assesses whether a given sample of data is normally
distributed.
Aggregates information in the data about both skewness
and kurtosis. Test of the hypothesis that S = 0 and K = 3,
based on S and K .
2
T
1

2
Test statistic:
JB = S + K 3

(
4

(here T is the number of observations)

Under the null hypothesis of independent normallydistributed observations, the Jarque-Bera
statistic is
2
distributed in large samples as a random variable with 2
degrees of freedom.

Simple Linear Regression

Simple Linear Regression Model

y-intercept

random iid
error t (0, 2 )

slope

yt = 0 + 1 xt + t
dependent
(response)
variable

independent
(explanatory)
variable

Linear Regression Assumptions

1. x is exogenously determined
2. t are iid(0,2)

(iid = independently and identically distributed)

Zero mean
Independence of errors (no autocorrelation)
Constant variance (homoscedasticity)
More things to think about:
Normality of t (if not satisfied, inference procedures only
asymptotically valid)
Model specification (e.g. linearity, 1 constant over time?)

Simple Linear Regression Model

y

yt = 0 + 1 xt + t

observed
value

t = disturbance

E y x* = 0 + 1 x*

x
observed value

-- Sample Linear Regression Model

y

y i = b0 + b1x i + ei
ei = Random
Error

y i = b0 + b1x i

Unsampled
Observation

x
Observed Value

Ordinary Least Squares

T

OLS minimizes sum of squared residuals ( yt yt )

T

x
min
y

t
0
1 t

0 , 1

t =1

t =1

= et 2
t =1

predicted
value

yt = 0 + 1 xt + t

e4

e2

e1

e3

fitted value
(in-sample forecast)
yt = 0 + 1 xt

On Thursday: Evaluating the Model

1. Examine variation measures
coefficient of determination (goodness of fit)
standard error of the estimate

2. Analyze residuals e
serial correlation

3. Test coefficients
for significance

yt = 0 + 1 xt

Random Error Variation

1.Variation of Actual Y from Predicted Y
2. Measured by Standard Error of Estimate
Sample Standard Deviation of e
Denoted SYX

3. Affects Several Factors

Parameter Significance
Prediction Accuracy

Measures of Variation in Regression

1.Total Sum of Squares (SST)
Measures variation of observed Yi around the
mean,Y

2.Explained Variation (SSR)

Variation due to relationship between
X&Y

3.Unexplained Variation (SSE)

Variation due to other factors

Variation Measures

Yi
Total Sum
of Squares
(Yi - Y)2

Unexplained Sum of
Squares (Yi - Y^i)2

Yi = b0 + b1X i
Explained Sum of
Squares (Y^i - Y)2

Xi

Coefficient of Determination
Proportion of Variation Explained by
Relationship Between X & Y
0 r2 1
2

r =

Explained Variation
Total Variation
n

i =1

i =1

SSR
SST

b0 Yi + b1 X iYi n (Y)
n

Yi
i =1

n (Y)

Coefficients of Determination (r2) and

Correlation (r)
Y r2 = 1, r = +1

Y r2 = 1, r = -1
^=b +b X
Y
i

^=b +b X
Y
i
0
1 i

X
Yr2 = .8, r = +0.9

X
Y

^=b +b X
Y
i
0
1 i
X

1 i

r2 = 0, r = 0
^=b +b X
Y
i
0
1 i
X

Standard Error of Estimate

n

SYX =

SYX =

Y
i =1

(
Y

Y
)
i i
i =1

n2
n

i =1

i =1

b0 Yi b1 X i Yi
n2

Residual Analysis
1.Graphical Analysis of Residuals
Plot residuals vs. Xi values
Residuals mean errors
Difference between actual Yi & predicted Yi

2.Purposes
Examine functional form (linear vs. non-Linear
Model)
Evaluate violations of assumptions

Test of Slope Coefficient for Significance

1.Tests If There Is a Linear Relationship
Between X & Y
2.Hypotheses
H0: 1 = 0 (No Linear Relationship)
H1: 1 0 (Linear Relationship)

3.Test Statistic

b1
1
t
=
n2 S
b
1

S
YX
where S =
b
n 2
2
1
X

n
(
X
)
i
i =1