Nonparametric Inference: Module 21

What we provide in this module

• Different inferential set up

• Functionals

• Plug-in estimators

• Examples

1
Co-Ordinator: Dr. Rahul Bhattacharya, Department of Statistics, Calcutta University

1
1 A short recap
So far we have learnt that data can be non normal in real practice. Traditional parametric
methods then fail to describe the features of the data set. If forcefully the normality assump-
tions are made, the results can lead to severe error, especially in small sample cases. Then
an alternative inferential method is adopted, which does not require any particular distri-
butional assumption unlike parametric methods. These methods are called Non Parametric
methods.

2 The set up
Here the data at hand is x = (x1 , x2 , ..., xn ), where xi ∈ (−∞, ∞) is the observed value of a
random variable Xi , i = 1, 2, .., n. F (x) = P (X1 ≤ x1 , ..., Xn ≤ xn ) is the distribution func-
tion(DF) from which x is generated. The components of X are not in general independent,
but we assume independence. i.e, F (x) = ni=1 Fi (xi ) for marginal DFs Fi , i = 1, ..n. The
Q

pair (X, F ) generates the induced probability space (X , B, P ), where X is the sample space,
B is the σ field generated by the all possible subsets of X , andP is the induced probability
measure. Suppose p(x) is the generalised density of X. In any situation, p(x) is either par-
tially or completely unknown. In statistical inference, we try to give some idea about p(x)
or some suitable function or functional of it.

3 Different inferential set up

Depending on the knowledge of p(x)(i.e. whether completely or partly unknown), inferential
set ups are either-i. Parametric,ii. Nonparametric and iii. Semiparametric

2
4 Classification
Suppose p(x) is partially known. We can write p(x) = p(x, θ), θ ∈ Ω, where p is functionally
known. θ (numerical or abstract valued) is an unknown quantity indexing p, often called
a labelling parameter. Ω is the space of all possible values of θ. It is called the parameter
space.

Example of parametric set up

1 If θ is real or vector valued, the set up is parametric. For example, suppose Xi are
iid observations from a Bernoulli distribution with mean π ∈ (0, 1). Then θ = π is
unknown and real valued. p(x, θ) = ni=1 θxi (1 − θ)1−xi is a known functional form
Q

and θ ∈ (0, 1) real valued. Thus the set up is parametric.

2 Suppose Xi are iid observations from a N (µ, σ 2 ) distribution, µ ∈ R and σ > 0. Then
1 2
θ = (µ, σ) is unknown and vector valued. Here p(x, θ) = ni=1 σ√12π e− 2σ2 (xi −µ) is a
Q

known functional form and θ ∈ (−∞, ∞)X(0, ∞) ⊂ R2 . Thus the set up is parametric.

Example of semi parametric set up

1 If θ is partly real or vector valued and partly abstract valued, the set up is semi-
parametric. For example, suppose Xi are iid observations from a density f (x) =
2
1 √1
2 2π
exp(− (x−µ)
2
) + 12 g(x) where g is unknown but known to be continuous. Then
p(x, θ) = ni=1 f (xi ) is a partly known functional form and θ
Q
= (µ, g) is partly ab-
stract valued and partly real valued. Thus the set up is semiparametric.

2 Consider the regression set up with one non stochastic covariate. Suppose Xi are inde-
pendent observations from the continuous density fi , i = 1, 2, .., n with E(Xi ) = α+βzi
and V ar(Xi ) = σ 2 . zi are known quantities, i = 1, 2, .., n. Then θ = (α, β, σ, f1 , .., fn )
is partly abstract valued and partly real valued and hence the set up is semiparametric.

3
Example of non parametric set up

1 If θ is completely abstract valued, the set up is nonparametric.

Suppose Xi , i = 1, 2, .., n are iid observations from an unknown DF F (x), where The
functional form of F is unknown but known to be continuous. Then θ = F is unknown
and abstract valued. Ω = is the class of all absolutely continuous DFs. Thus the set
up is nonparametric.

2 Consider two independent samples {Xi , i = 1, .., m} and {Yj , j = 1, .., n}, where Xi
are iid observations from an unknown density f (x) Yj are iid observations from an
unknown density g(x), where f and g are both unknown but known to be continuous.
Then θ = (f, g) is unknown and abstract valued. Thus the set up is nonparametric.

5 Concept of functional
In a nonparametric set up, the basic assumption is the continuity of the underlying dis-
tribution F. The unknown quantity of interest (parametric function in parametric set up)
is defined in terms of a functional. If F is the class of all absolutely continuous distribu-
tions, then a functional θF = θ(F ) is a real valued function defined for every F ∈ F Then
θF : F → R is a mapping from the abstract space to real line. In nonparametric infer-
ence, the objective is to learn the value of θF = θ(F ) for a known functional θ based on iid
observations from an unknown DF F.

Example of functionals

Suppose Xi , i = 1, 2, .., n are iid observations from an unknown DF F (x), where F is unknown
but known to be continuous. Few possible functionals are:
1. θF = EF I(X ≤ a), for some known a, where F is the class of all absolutely continuous
DFs.
2. θF = EF (X), where F = {F : EF (|X|) < ∞}

4
3. θF = EF (X 2 ), where F = {F : EF (X 2 ) < ∞}

Linear functional

Each of the functionals can be expressed as EF ψ(X), for some ψ.

For examples 2 and 3, ψ(X) = X and X 2 .
For example 1, ψ(X) = I(X ≤ a), where I( ) is the indicator function. Then θ(αF1 +
(1 − α)F2 ) = αθF1 + (1 − α)θF2 for any 0 < α < 1. The above functionals are called linear
functionals.

Nonlinear functionals

Suppose Xi , i = 1, 2, .., n are iid observations from an unknown DF F (x), where F is unknown
but known to be continuous. Then the following are also valid functionals:
1. θF = {EF (X)}2 , where F = {F : EF (|X|) < ∞}
2. θF = VarF (X), where, F = {F : EF (X 2 ) < ∞}
The above functionals are nonlinear functionals as θ(αF1 (1 − α)F2 ) 6= αθF1 + (1 − α)θF2 for
0 < α < 1.

Further examples

Suppose Xi , i = 1, 2, .., n are iid observations from an unknown DF F (x), where F is un-
known but known to be continuous. Then the following are also nonlinear functionals:
1. θF = F −1 ( 21 ), the median , where, F is the class of all absolutely continuous DFs.
F −1 ( 43 )−F −1 ( 41 )
2. θF = 2
, the quartile deviation, where, F is the class of all absolutely continu-
ous DFs.

A critical example

Suppose F is class of all absolutely continuous DF with finite expectation. Then the question
is what will be the type of the kernel θF = {EF (X)}2 ?

5
Take Fi ∈ F, i = 1, 2 Then for any α ∈ (0, 1),
EαF1 +(1−α)F2 (X) = αEF1 (X) + (1 − α)EF2 (X).
But {EαF1 +(1−α)F2 (X)}2 6= α{EF1 (X)}2 + (1 − α){EF2 (X)}2 .
Therefore, θF is not a linear functional.

Plug in estimator: Analogue to statistic

Suppose Xi , i = 1, 2, .., n are iid observations from an unknown DF F (x), where F is unknown
but known to be continuous. Fn is the empirical DF based on the data. θ(Fn ) is the plug-in
estimator corresponding to the functional θ(F)
1. EFn ψ(X) is the plug-in estimator corresponding to the functional EF ψ(X).
2. Fn−1 ( 21 ), the sample median is the plug-in estimator corresponding to F −1 ( 12 ).

A misleading example

Suppose F is class of all absolutely continuous DF with symmetry at the origin. Then what
will be the type of the functional θF = P (X > 0)?
At a first look it appears as a linear functional. But symmetry at the origin gives θF = 21 .
Therefore, θF is not a functional.