Signal Processing 68 (1998) 113—118

Fast communication

Optimization of QAM signal constellation in

the presence of Rayleigh fading
Jong-Ki Han*, Hyung-Myung Kim
Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 373-1 Kusong-dong Yusong-gu Taejon
305-701, South Korea
Received 30 March 1998

Abstract
This paper considers a communication system consisting of a vector quantizer (VQ) whose selected codevectors are
mapped into points in the modulation signal space. The modulation signal is transmitted over Rayleigh flat fading
channel. We optimize the modulation signal constellation to minimize the overall distortion of the system for the fixed
VQ codebook. The optimization is obtained by partitioning the modulation signal space. Simulation results show that
the signal constellation based on the proposed optimization scheme outperforms the conventional ones. ( 1998
Elsevier Science B.V. All rights reserved.

Keywords: VQ; QAM; Rayleigh flat fading channel

1. Introduction is the quadrature amplitude modulation (QAM)

Many applications of VQ can be found in the
several signal processing areas. As in most data
processing systems, it is of great importance that
a VQ system is robust against noise. We consider the
system of transmitting information from a vector
source over noisy channels without employing the
channel coding technique. The digitally modulated
VQ codevector is assumed to be transmitted over
Rayleigh flat fading channel. Furthermore, it is
assumed that the modulation format of the system
* Corresponding author. Tel.: #82 42 869 3440; fax: #82
42 869 3410; e-mail: hjk@rabbit.kaist.ac.kr.
which is widely used for the high-speed data trans-
mission [1,4,6,7].
In order to optimize the VQ communication
system, the design of modulation signal set has
been considered in [1,4,6,7]. In [1], Foschini con-
sidered equally probable signal points and an equal
cost for error events, and optimized the signal con-
stellation under the criterion of minimum symbol
error probability. Sundberg et al. derived the
weighted QAM constellations for 8-bit PCM in
[6]. In [1,6], the modulation signals were modified
to reduce the symbol error probability without
considering codevectors. In [4], Liu et al. optimized
the modulation signals for the fixed codebook.

0165-1684/98/$19.00 ( 1998 Elsevier Science B.V. All rights reserved.

PII: S 0 1 6 5 - 1 6 8 4 ( 9 8 ) 0 0 0 8 2 - 6
114 J.-K. Han, H.-M. Kim / Signal Processing 68 (1998) 113–118
Thus, the relative effect of each codevector to the
overall distortion was considered. While the sys-
tems described in [1,4] had been optimized for
additive white Gaussian noise (AWGN) channel,
Webb [7] proposed Star QAM constellation for
Rayleigh fading channels to reduce the symbol
error probability without considering the code-
vectors.
The objective of this paper is to design a modula-
tion signal set of QAM system over Rayleigh flat
fading channels, where the signals are optimized for
the fixed codebook. The optimization is achieved
by partitioning the modulation signal space. The
paper is organized as follows. In Section 2, a gen-
eral communication system model is described and
the design problem considered in this paper is for-
mulated. In Section 3, the optimization method of
QAM signal constellation is proposed. The simula-
tion results are presented in Section 4. Section 5
contains conclusions.
Fig. 1. A communication system consisting of VQ encoder over Rayleigh fading channel.
Fig. 2. Square QAM constellation.
2. System model
The communication system model considered in
this paper is described in Fig. 1. A k-dimensional
data vector ½ is quantized by the VQ encoder into
a codeword X chosen from a set of k-dimensional
i
vectors, C"MX ,2, X N, where N denotes
0 N~1
the size of the VQ codebook. The binary index-
vector I(X ) of the codevector X is directly mapped
i i
into the modulation signal S . The set S"
i
MS , S ,2, S N forms the QAM signal constella-
0 1 N~1
tion given in Fig. 2. Each signal S is represented
i
as a complex number. We assume that
!¸)ReMS N, ImMS N)¸, where ¸ denotes the
i i
limit of the signal space, so as to maintain a limited
average power of signal set S.
As shown in Fig. 3, S is transformed to SI due to
i i
the Rayleigh fading, and then the Gaussian noise
displace the faded signal SI into the received signal
i
Z . The signal Z arrived at the receiver can be
r r
Fig. 3. The effect of Rayleigh fading and AWGN.
 J.-K. Han, H.-M. Kim / Signal Processing 68 (1998) 113—118 115

represented as 3. Optimization of QAM signal constellation

(O-QAM)
Z "FS #G"SI #G, (1)
r i i
where F denotes the amplitude attenuation factor In order to minimize D in Eq. (5) for the fixed C,
due to the Rayleigh fading factor whose probability it is necessary to describe the transition probabili-
ties P(S DS ) of Eq. (5), by considering the relation-
density function and the second moment are given j i
ship among the transmitted signal S , the faded
by, respectively, i
signal SI , and the received signal Z shown in Fig. 3.
i r

A B
f f2 Let SI (x) denote a faded signal with the amplitude x.
P ( f )" exp ! , 0(f(R, (2) i
F p2 2p2 Then SI (x) is represented as SI (x)"x ) d , where
F F i i i
d denotes a two-dimensional unit vector in the
and E[ f 2]"2p2. G denotes the AWGN of the i
F direction of S . It is known [8] that the amplitude of
channel. Hard decision detector ¶ maps the re- i
the faded signal SI has the following probability
ceived signal Z into the modulation signal S if i
r j density function:
Z belongs to a region R among the disjoint parti-
r j
tion X"MR , R ,2, R N of the two-dimen-
A B
0 1 N~1 x x2
sional received signal space R2, i.e., P I (x)"P(SI (x)DS )" exp ! ,
Si i i p2 2p2
i i
¶(Z )"S , if Z 3R , j3M0, 1,2, N!1N. (3)
r j r j 0(x(R, (6)
We will describe the optimal partition of R2 in the
next section. where 2p2 denotes the second moment E[x2] of x.
i
The performance of the above system is mea- Then p2"ES E2p2 since x"S ) f from Eq. (1).
i i F i
sured in terms of the mean squared error (MSE) Hence, the transition probability P(S DS ) can be
between the transmitted vector X and its recon- j i
i described as
structed vector X . Let the transition probability
j
P(X DX ) denote the probability that the codevector
P
j i =
X is decoded when X is transmitted, 0)i, P(S DS )" P I (x)P(S DSI (x)) dx, (7)
j i j i Si j i
j)N!1. Then, the distortion due to the channel 0
noise is given by where P(S DSI (x)) denotes the probability that
j i
N~1 N~1 a faded signal SI (x) is decoded to S by the detector
D" + + P(X DX )d(X , X ), (4) i j
j i i j and is given by
i/0 j/0

P
where d(X , X ) is the squared Euclidean distance
i j P(S DSI (x))" P(Z DSI (x)) dZ . (8)
between two vectors X and X , i.e. EX !X E2. j i r i r
i j i j R j
Since I(X )s are assigned to S s, the transition
i i
probability is P(X DX )"P(I(X )DI(X ))"P(S DS ). Let N be the variance of the white Gaussian noise.
j i j i j i o
Therefore, Eq. (4) can be rewritten as Then the transition probability that a faded signal
SI (x) becomes the received signal Z due to AWGN
N~1 N~1 i r
D" + + P(S DS )d(X , X ). (5) can be represented as [5]
j i i j
i/0 j/0

C D
1 d(SI (x),Z )
For a given VQ codebook C, D becomes a function P(Z DSI (x))" exp ! i r , (9)
of S only. Hence, the channel distortion D can be r i J2pN 2N
o o
minimized by the optimal construction of modula-
tion signal constellation. We will show that the where d(SI (x),Z ) is the squared distance between
i r
equivalence of obtaining the optimal partition X of a faded signal SI (x) and a received vector Z , i.e.,
i r
the received signal space and constructing the opti- ESI (x)!Z E2"x2#EZ E2!2xEZ Ecos h , where
i r r r i
mal signal constellation in the next section. h denotes the angle between S and Z .
i i r
116 J.-K. Han, H.-M. Kim / Signal Processing 68 (1998) 113–118

Substituting Eqs. (6)—(8) into Eq. (5) gives where

P A B
N~1 N~1 =x
P
x2 =
D" + + d(X ,X ) exp ! C(x)" e~ttx~1 dt, x'0, (16)
i j p2 2p2
i/0 j/0 0 i i 0
]
P
Rj
P(Z DSI (x)) dZ dx.
r i r
(10)
D (x)"ex2@4
~2 S AS
p
2
2
p C A BDB
e~x2@2!x 1!U
x
J2
,

(17)
When a particular Z is in a region R *, i.e.
r j

P
Z 3R *, the amount of distortion introduced by
r j 2 x 2
Z becomes U(x)" e~t dt. (18)
r Jp 0
N~1
D(Z )" + d(X ,X *) Let
r i j
i/0
Mp2#N N EZ Ecos h
P A B
=x x2 b" i o and c"! r i. (19)
] exp ! P(Z DSI (x)) dx (11) 2N p2 N
p2 2p2 r i o i o
0 i i
N~1 Thus by applying Eq. (15) to Eq. (13) along with
" + d(X , X *)¼(Z , S ), (12) (19) and C(2)"1 [2] (p. 938), Eq. (13) becomes
i j r i
i/0

A B
where 1 EZ E2 N p2
¼(Z , S )" exp ! r o i
r i p2J2pN 2N (p2#N )
exp(!EZ E2/2N ) i o o i o
¼(Z , S )" r o
r i
A B
p2J2pN EZ E2p2cos2h
i o ]exp r i i

P A B
= !(N #p2)x2#2EZ Ep2cos h x 4N (p2#N )
] x exp o i r i i dx. o i o
2N p2
A B
0 o i EZ Ecos h p
]D ! r i i . (20)
(13) ~2 JN (p2#N )
o i o
Thus, for each Z , a region number j* should be And U(x) can be represented as
r
chosen to minimize D(Z ). It becomes clear that for
r
the fixed C, the problem of minimizing the channel U(x)"2[Q(0)!Q(DxD)], (21)
distortion D is identical to that of partitioning
R2 into the disjoint regions MR NN~1 so as to where Q(x)"(1/J2p):= e~t2@2 dt. Hence,
i i/0 x
minimize D. The optimum partition X"MR ,
A B
0 DxD
R ,2, R N is such that D (x)"e~x2@4!xex2@4J2pQ , (22)
1 N~1 ~2 J2

G
N~1
R " Z : + d(X , X )¼(Z , S )
j r i j r i and it can be evaluated from the tabulated Q(x)
i/0 function. Therefore, we can compute ¼(Z , S )
r i
H
N~1 simply with Eqs. (20) and (22), instead of (13).
) + d(X , X )¼(Z , S ), ∀l, ∀Z . (14)
i l r i r It is easy to see that, for the fixed X"
i/0
MR , R ,2, R N, the new optimum signal set
The integral part of Eq. (13) can be simplified using 0 1 N~1
S"MS* , S* ,2, S* N must satisfy
the following relationship [2] (pp. 337, 930, 933, 0 1 N~1
1067):
S*"min E[d(Z ,y)D»"S ], j3M0,1,2,N!1N,

P
= j r j
x exp(!bx2!cx) dx y|R2
0 (23)

AB A B
c2 c where » denotes a variable at the output of the
"M2bN~1C(2)exp D , b'0, (15)
8b ~2 J2b detector and E[ ) ] is the mathematical expectation.
 J.-K. Han, H.-M. Kim / Signal Processing 68 (1998) 113—118 117

A successive application of Eqs. (14) and (23)

results in a sequence of signal set S and the parti-
tion X pairs for which the corresponding channel
distortion D form a nonincreasing sequence of non-
negative numbers. We will refer the final S—X pair
as the optimal QAM (O-QAM) signal constella-
tion.

4. Simulation results

In this section, numerical results on the perfor-

mance of the O-QAM system are given and com-
pared with those of other QAM systems. In Fig. 4. The performance comparison (at p "1.0) of various
F
designing the VQ codebook, 19 different 512]512 QAM systems.
real images are used as the training data for LBG
algorithm [3]. The codevector dimension and the
number of codevectors are set to k"16 and
N"64, respectively. In the simulation, Z s are gen-
r
erated by the uniform sampling of signal space R2,
where !¸)ReMZ N, ImMZ N)¸. By making the
r r
number of sampled Z s very large ('100 000 N),
r
we let the interval between the Z s become small
r
and effectively cover the continuous signal space.
A Lena image is used in the performance evalu-
ation. The simulation results are presented in terms
of PSNR and are measured at different values of
the CSNR (Channel Signal to Noise Ratio) and p .
F
CSNR is defined as

1 +N~1ES E2 Fig. 5. The performance comparison (at CSNR"23.2 dB) of

CSNR¢10 log N i/0 i . (24)
10 N
o
In Figs. 4 and 5 , the performance of the system
consisting of O-QAM signal is compared with
those of other systems. The O-QAM system is
obtained by applying the proposed algorithm at
p "5.0 and CSNR"23.2 dB. Foschini’s constel-
F
lation [1] had been obtained by minimizing symbol
error probability for AWGN channel without con-
sidering codebook. The constellation proposed by
Liu [4] had been obtained by minimizing the
averaged distortion due to AWGN for the fixed
codebook. Star QAM signals were proposed for
Rayleigh fading channels [7] (Chapter 11).
The simulations of Fig. 4 have been performed at
several CSNRs for the fixed p "1.0. The perfor-
F i
mances of Foschini’s, Liu’s and Square QAM
various QAM systems.
systems are very poor in the entire range of CSNR
since those systems are not optimized for Rayleigh
fading channels. Thus, they are very sensitive to the
Rayleigh fading.
The O-QAM system outperforms Star QAM
system. The reason is that O-QAM system is opti-
mized for the given codebook while Star QAM
system was proposed for equally probable signal
points. In Fig. 5, the performances of the systems
are measured at different p s for CSNR"23.2 dB.
F
It can be observed that the best performance of all
systems are achieved at p "1.0. This is due to the
F
fact that when S is a transmitted signal, the most
i
likely value of the faded signal SI is p [8], i.e. S p .
i i F
In other words, when p "1.0, the most likely
F
118 J.-K. Han, H.-M. Kim / Signal Processing 68 (1998) 113–118
received signal is equal to the transmitted signal.
From these results, it is obvious that the O-QAM is
very effective in reducing the effect of Rayleigh
fading and AWGN of the channel.
5. Concluding remarks
We have optimized the QAM signal constella-
tion whose signals are transmitted over Rayleigh
fading channels. The optimization is obtained by
partitioning the signal space for the fixed code-
book. In simulations, it is shown that the proposed
system provides better performance than the con-
ventional systems for noisy fading channels.
References
[1] G.J. Foschini, R.D. Gitlin, S.B. Weinstein, Optimization of
two-dimensional signal constellations in the presence of
Gaussian noise, IEEE Trans. Commun. 22 (January 1974)
28—38.
[2] I.S. Gradshteyn, I.M. Ryzhik, Table of Integrals, Series, and
Products, Academic Press, New York, 1980.
[3] Y. Linde, A. Buzo, R.M. Gray, An algorithm for vector
quantizer design, IEEE Trans. Commun. 28 (January 1980)
84—95.
[4] F.-H. Liu, P. Ho, V. Cuperman, Joint source and channel
coding using a non-linear receiver, ICC 93, 1993, pp.
1502—1507.
[5] H. Stark, F.B. Tuteur, J.B. Anderson, Modern Electrical
Communications, Prentice-Hall, Englewood Cliffs, NJ,
1988.
[6] C.E.W. Sundberg, W.C. Wong, R. Steele, Logarithmic
PCM weighted QAM transmission over Gaussian and
Rayleigh fading channels, IEE Proc. 134 (October 1987)
557—570.
[7] W.T. Webb, L. Hanzo, Modern Quadrature Amplitude
Modulation, IEEE Press and Pentech Press, 1994.
[8] M.D. Yacoub, Foundations of Mobile Radio Engineering,
CRC Press, 1993.