Estimation of Parameters

Parameters: In statistics parameters are values that define PDF or PMF completely.
For example, in X ~ N (, 2), we can specify a complete PDF if we know and 2 .
So in the normal distribution and 2 are parameters..
Distribution

Parameters

X ~ Binomial(n, p)

X ~ Poisson ()

X ~ Exponential ()

X ~ N (, 2)

, 2

X ~ Uniform (a, b)

a, b

As long as we know the parameters, we can see how a random variable behaves
specifically, and if we want to, we can calculate specific probability.
Parameters also specify characteristics of the population: If 1 is greater than 2 in
normal distribution, we know that group 1 has bigger mean than group 2 has.
In most cases, we do not know exact values of parameters. Just think about the
population of all the highschool students. We cannot calculate their mean of heights.
So we estimate them.
The question is how to estimate parameters and, if there are different ways to
estimate, how we judge which one is better than the others.
Point Estimation and Interval Estimation
A point estimation estimates parameter by a single value while an interval
estimation estimates an interval that may contains the true value of the parameter.
ex)

= 170
165 < < 175

point estimation
interval estimation

Criteria of good estimator

An estimator is a function of sample values to estimate the parameter.
For example a sample mean is a function (linear) of sample values.
(1) Unbiaed Estimator
(2) Consistent Estimator
(3) Efficient Estimator
ex 1) Suppose we use the sample mean as an estimator of the
parameter . Then, we know that the is a linear function of the sample values
because = (x1 + x2 + .. + xn ) / n.

ex 1) Suppose we use the sample mean as an estimator of the

parameter . Then, we know that the is a linear function of the sample values
because = (x1 + x2 + .. + xn ) / n.
ex 2) The sample variance s2 , an estimator of the population variance 2
function (not a liniar function though) of samples.

is a

s2 =

Unbiaed Estimator: If the expectation value of an estimator becomes the

parameter, i.e. E( An estimator) = parameter, then this estimator is an unbiased
estimator of the parameter.
/
1/
ex) E( ) = E
(x1 + x2 + .. + xn ) =
is an unbiased estimator of .

/) + E(x ) + . . + E(x )) =
(E(x1/
1
2
n

/
1/
(n * ) = . Therefore

Consistent Estimator: An consistent estimator approaches to the true value of the parameter
probabilistically when the sample size increases.
ex) is a consistent estimator to the parameter . E( ) = , Var() = 2 / n.
So, as n is getting bigger, the variance of becaomes smaller. This means that is
approaching to the true value of in a stocatic sense.

is a consistent estimator : Its variance is getting smaller as n (the size of sample) increases

Efficient Estimator : If an estimator with smaller variance is relative efficient estimator.

from n = 20 is relative efficient estimator to from n= 10 because
Variance of is 2/n
2 / 20 is smaller than 2 /10.

Note: = 1/2 (x1 + x2) is the least variance linear estimator of .

Prrof: Let c = w1x1 + w2x2 is another linear estimator for
We can show that E(c) = . Variance of c is (w12 + w22 )2
This variance minimizes whan w1 = w2 = 1/2
Why the formula for the sample mean is

not

Therefore

is an unbiased estimator of 2.

If we use, instead

, we underestimate 2 .

Summary for the Point Estimation

(1) The sample mean is an unbaised estimator of
(2) is a consistent estimator of
(3) The bigger sample size of n is, the more efficient estimator becomes.

is an unbiased estimator of the population variance 2.

Interval Estimation: Cofidence Interval Estimation of .
We know that if X ~ N(, 2), then

distributes Z (0, 1).

Z: Standard normal distribution

So, P( -a < Z =

<a)=p

If we arrange the inequality in terms of ,

This is the interval that will contins true with the confidence (probability)
of p * 100 % .
Let suppose p = 0.95. If we find the value for the p = 0.95,
1.959964

= NORM.INV(0.95 + (1- 0.95)/2, 0, 1)

So,

.. (1)

From this, we can define 's 95 % confidence interval as:

For the 90% confidence interval, a = 1.64, and for 99 %, a is 2.54.

Therefore, 's confidence interval are:
90%

1.281552

95%

1.959964

99%

2.575829

So far we have assumed that the population variance 2 is known.

If it is no known, we have to use t distribution.
If we replace with s (= sample standard deviation)
distributes t with degree of freedom n - 1.

Using the t distribution, we can calculate the confidence interval of as:

p * 100 % confidence interval

For example with the df = 10,

=T.INV(0.9+(1-0.9)/2, 10)

90%

t=

1.812461

95%

t=

2.228139

99%

t=

3.169273

=T.INV(0.9+(1-0.9)/2, 10)

Error and Sample Size

Error:

From (1),

If we arrange the inequality in terms of

If we define the
upper limit of the error.

- ,

as error of estimation,

is the

So, when we set the upper limit as a given value, say b,

b=

, and n = ((1.96 / b) 2

For example, when = 10, to be 95% confident that the estimation error should be
less than 5, the sample size should be at least 16.
n=

15.3664

=((1.96*10)/ 5)^2


&

) = . Therefore

e of the parameter

ns that is

e of sample) increases

ent estimator.

1.644854

1.959964

Exercise
1. Prove = 1/n (x1 +.. + xn) is the most efficient estimator among all other
estimators with linear function of n samples.
Hint: Let c = w1 x1 + w2 x2 + . + wnxn, where w1 + w2 + . + wn = 1
Calculate the variance of c. Minimize the variance subject to w 1 + w2 + . + wn = 1

. + wn = 1

((
)

/
1/

((
)

()