Sample space axioms:

1. empty set : P() = 0

2. Nonnegative: event A, P(A) 0
3. Countable additivity: Pairwise disjoint An, have
P(

n=1
) =
=1

P(A
n
).
4. P()=1
Remarks:
1. () = = , similarly, () = =
2. Countable Additivity only talks about countable union and summation
3.
P( A
n n
= lim
k
P( A
n
k
n=1
)
and
P( A
n n
= lim
k
P( A
n
)
k
n=1
Bayes Theorem
P(A
n
|B) =
P(A
n
B)
P(B)
=
P(B|A
n
)P(A
n
)
P(B)
=
P(B|A
n
)P(A
n
)

k=1
N
P(B|A
k
)P(A
k
)

When Ak is a partition of
Borel Set:
() is the smallest -algebra that contains all intervals of the forms:
(, ), (, ], [, ), [, ]; ,
Independence: and
1. For arbitrary functions h(), ():
[h()()] = [h()][()].
2.

(, ) =

()

().
3.

(, ) =

()

().
4. Char Func:

(
1
,
2
) =

(
1
)

(
2
).

Correlation Coeff:
=
cov(,)
var()var()

Cauchy-Schwarz INEQ:
|, |
2
, , , . h h = .

Functional Independent:

1
,
2
, ,

are independent:

1
=
1
(
1
), ,

), are independent

1
=
1
(
1
,
2
),
2
=
2
(
2
,
3
), Independent

PGF: (Dsc) MGF: (Cts)

() = [

] =
=0

P
X
().

() [

] =

()

.
CHAR:

() [

]
Expanding of Exponential component

=
=0

!

3 Inequalities:
Markov: X NN, i.e. P(X<0) = 0,
> 0 P(X > )
[X]

, , P(X >
r
)
[X
r
]

Cheybyshev: ( By define Y = |X [X]|
2
)
P(|X [X]| )
Var(X)

2

Chernoff:
P(X )

X
(s), 0, . . P(X )
argmin
s0
{

X
(s)}

MMSE Estimation: observe X, estimate Y
Estimator
MMSE tr ([(

)(

])
Linear

= =

1
tr(

)
Affine

= +;
(

= (

) +

)
=

1
; =

tr(

)
UnCnstrned

= [|] [
2
] [[|]
2
]

Gaussian r. vec.
=

() =

, h

() = exp {()
1
2

2
} , can tell the ,
2
from left

() = exp {

+
1
2

}, similarly, ,
2

Combination of ind. Gaussian R. Vec., notice each item in

1
is 1/

2
.
= [
1
,
2
, ,

() =

=1

=
1
(2)

=1

exp {
1
2
( [])

1
( [])}
Conditional distribution, Gaussian
Gaussian r. vec. [

, = ~([| = ],
|
)
where
|

,
[| = ] = (

) +

, solves

When C_X|Y invertible

|
(|) =
1
(2)

det
|
exp {
1
2
( ())

|
1
( ())},
where () [| = ] = (

) +

.
We can see x^ = E[X|Y=y] = A(y-my)+mx, affine to Y, C_X|Y doesn't depend
on Y's value
# Synthesis #
r. vec. X is std zero-mean unit-var Gaussian. To generate Y with mean "m",
Cov "C_Y":
= +, where ,

.
moreover,

= =

, =
1/2
, =
1/2
+
= [
1
, ,

], where

are eigenvectors of

Whitening: de-correlating
Given with

and cov() =

, where

=
1/2

), we can see W is std Gaussian.

Remarks on covariance matrix:

(, )
2

(, )

(, ), h h h = , by CS INEQ
Jacobian det
h() =
[

h
1
()

1

h
1
()
(

)

h

()

1

h

()
(

) ]

X, Y be r. vecs, with func = (), h =

1
, = h(),
then the density func:

() =

(h())

| det h()|
e.g. = +, = ,

(, ) = (, ) = ( + , ) (, ) = h(, ) = (h
1
:
+
2
, h
2
:

2
).
h(, ) = [
1
2
,
1
2
;
1
2
,
1
2
]

(, ) =

(h(, )) | det h()| =

1
2

(
+
2
,

2
)

Random Process

(, ) [

] = h()

(, )

(, ) = [

] = h()h()

( , )
= h() h()

( + )

() =

= (

) = ||
2

()

Let f(x, ) be a function s.t. f(x,) exists, and is continuous. Then,

( (, )
()
()
) =

(, )
()
()
+ ((), )

() ((), )()

Functions of r.v.:
Consider = (), To find

(), observe:

() P(() ) = P(

),

{ : () )
. .

() = P(

) =

()

Also, when

() is continuous, = (), g(u) is differentiable, i.e. exist

() =
()

, and, on countable many x,

1
({}) = { : () = }:

() =

)
|

)|
.
Functional Expectation: E[] = ()

()
+

= (()]

Let X(i) denotes the ith smallest element in sequence X1, , Xn;
Then (
(1)
,
()
) = (
()
) (
(1)
,
()
)

Conditional Prob.
P( | )
P(, )
P()
, DRV

|
(|)

(,)

()
, ( | = )
|
(|)

, CRV
The Law of total Prob.
P( ) =

P( | = )P

(), DRV
( ) = ( | = )

()

, CRV
The Substitution Law
= (, ), P((|) = | = ) = P((, ) = | = )
((, ) | = ) =

(, )
|
(|)

= ((, ) | = )
Chain Rule of cond. Prob.
P
|
(, |) = P
|
(|, )P
|
(|)
Conditional Expectation
[|] =

|
(|), which is function of r.v. X
FOR CRV, [|] =
|
(|)

1. [[|]] = [], "Smoothing prop"
2. [|] = h(), = h()
3. [[|, ]|] = [|], "Tower porp"
D
i
s
t
r
i

P
M
F

o
f

P
D
F

E

V
a
r

P
/
M

G
F

U
n
i
(
a
,

b
)
;

D
s
c

P
x
(
i
)
=
1
/
n


(
a
+
b
)
/
2

(
n
^
2
-
1
)
/
1
2

+
1
)

(
1

)


U
n
i
(
a
,
b
)
;

C
t
s

1
/
(
b
-
a
)

(
a
+
b
)
/
2

(
b
-
a
)
^
2
/
1
2

)


P
o
i
s
(

)


P
x
(
k
)
=

k
e

k
!

e
x
p
{

1
)
}



B
o
u
n

(
p
)

P
x
(
X
=
1
)
=
p
;


P
x
(
X
=
0
)
=
1

p



p


p
(
1

p
)

G
e
o

(
p
)

P
x
(
n
)
=
(
1

p
)
n

1
p



1
/
p


(
1

p
)
/
p
^
2

(
1

B
i
n
(
n
,
p
)


P
x
(
k
)
=
(

)
p
k
(
1

p
)
n

k


n
p



n
p
(
1

p
)

(
1

G
a
u
(

)
=
1

)
2
2

e
x
p
{

+
12

2
}


E
x
p
(

)
=

1
/

1
/

^
2

(
1