Exponential Family

Definition (canonical exponential family):

The exponential family has the density in the canonical form,

. f (x1 , x 2 ,K, x p ;1 , 2 ,K, s ) = f ( x, ) = exp j T j ( x ) A( ) h(x )

j =1

Example 1:

Let X ~ N , 2 . Then,

x x 2
( x )2
f x; , =
exp
= exp 2 2
2
2

2
2
2

2 log 2 2
2 +
2
2

where

2 1
2
1 = 2 ,T1 (x) = x,2 = 2 ,T2 (x) = x , A( ) = 2 + log(2 2 ) .
2

2
2
Important Result 1:

E Tj (X ) =

A( )
j

and

cov T j ( X ), Tk ( X ) =

2 A( )
.
j k

Example 2:
Let X ~ Binomial (n, p ) . Then,

n
n
n x
f (x; p) = p x (1 p) = exp[x log( p) + (n x) log(1 p)]
x
x

n
p
+ n log(1 p)
= exp x log
1 p

x
1

where

n
p
,T1 ( x) = x, h(x) =
1 p
x

1 = log

p
1

1 + exp(1 ) =
Qexp(1 ) =

1 p
1 p
A(1 ) = n log(1 p) = n log(1 + exp(1 ))

log[1 + exp(1 )] = log(1 p)

Thus,

p
A(1 ) n exp(1 )
1 p
E[T1 ( X )] = E( X ) =
=
= n
= np,
1
1
1 + exp(1 )
1 p
2 A(1 ) n exp(1 ) n exp(1 ) exp(1 )
Var[T1 ( X )] = Cov[T1 ( X ), T1 ( X )] =
=

1 + exp(1 )
12
(1 + exp(1 ))2 .
= np np2 = np(1 p)

Important Result 2:
The moment generating function for T ( X ) = [T1 ( X ) T2 ( X ) L Ts ( X )] is

M T (t ) = M T (t1 , t 2 ,K, t s ) = E{exp[t1T1 ( X ) + t 2T2 ( X ) + L + t sTs ( X )]}

= exp[A( + t ) A( )]

and the cumulant generating function is

T (t ) = log[M T (t )] = A( + t ) A( ) .
Example 3:
Let X ~ Poisson ( ) . Then,

f (x; ) =

x exp( )
x!

= exp[x log( ) ]

1
x!

where

1 = log( ), T1 (x) = x, A(1 ) = = exp(1 ) .

2

The moment generating function of T1 ( X ) = X is

MT (t ) = exp[A(1 + t ) A(1 )] = exp[exp(1 + t ) exp(1 )]

= exp[exp(1 )(exp(t )) 1] = exp[(exp(t ) 1)]

and the cumulant generating function is

T (t ) = (exp(t ) 1) .
Note that

T' (0) = [ exp(t )]t =0 = = E( X )

and

T' (0) = [ exp(t )]t =0 = = Var( X ) .

Important Result 3:
T ( X ) = [T1 ( X ) T2 ( X ) L Ts ( X )] is distributed according to an exponential

family with density

f (t1 , t 2 ,K, t s ;1 , 2 ,K, s ) = f (t , ) = exp j t j A( ) h (t ) .

j =1

Important Result 4:
T ( X ) = [T1 ( X ) T2 ( X ) L Ts ( X )] is sufficient and complete.