Received August 10, 2021, accepted August 25, 2021, date of publication August 30, 2021, date of current

version September 8, 2021.

Digital Object Identifier 10.1109/ACCESS.2021.3108625

Channel Estimation and Hybrid Precoding for

Millimeter Wave Communications: A Deep
Learning-Based Approach
QIUJIN LU , TIAN LIN , (Graduate Student Member, IEEE), AND YU ZHU , (Member, IEEE)
State Key Laboratory of ASIC and System, School of Information Science and Technology, Fudan University, Shanghai 200433, China
Corresponding author: Yu Zhu (zhuyu@fudan.edu.cn)
This work was supported by the National Natural Science Foundation of China under Grant 61771147.

ABSTRACT Hybrid analog and digital beamforming (HBF) has been regarded as a key technology for future
millimeter wave (mmWave) communication systems due to its ability to obtain a good trade-off between
achievable beamforming gain and hardware cost. In this paper, we investigate the channel estimation and
hybrid precoding for mmWave MIMO systems with deep learning. We adopt the hierarchical codebook
based algorithm for channel estimation as it requires limited number of pilot transmissions, and enhance its
performance by proposing a new codebook design algorithm based on manifold optimization (MO). With
the estimated channel state information (CSI) as the input, we develop a robust HBF network (HBF-Net)
by applying convolutional layers and attention mechanism, which can be trained to generate a robust HBF
matrix targeting at spectral efficiency maximization with imperfect CSI. To further improve the performance,
we propose a joint channel estimation and HBF optimization network (CE-HBF-Net). Considering that the
adaptively selected HBF vectors in the hierarchical codebook based channel estimation are different for
different channel realizations, we skillfully propose an index assign-and-input method to efficiently feed such
information to the CE-HBF-Net to reduce the network input dimensions and make the network trainable.
Furthermore, we propose a signal self-attention mechanism to enable the CE-HBF-Net to intelligently assign
larger weight coefficients to those signals that contribute more to channel estimation. Simulation results
show that the well-designed HBF-Net and CE-HBF-Net outperform the conventional HBF algorithms with
imperfect channel and exhibit robustness to mismatches between offline training and online deployment
stages.

INDEX TERMS Millimeter wave (mmWave), channel estimation, hierarchical codebook, manifold opti-
mization (MO), hybrid beamforming (HBF), deep learning (DL), attention mechanism.

I. INTRODUCTION A. RELATED WORKS AND MOTIVATIONS

Hybrid analog and digital beamforming (HBF) is a promising
technology to balance the beamforming gain, the hard-
ware cost, and the power consumption for future mas-
sive multiple-input and multiple-output (MIMO) millimeter
wave (mmWave) communication systems. By separating the
whole beamformer into a low-dimensional baseband digital
one and a high-dimensional analog one implemented with
phase shifters, the HBF architecture can significantly reduce
the number of radio frequency (RF) chains but still guarantee
a sufficient beamforming gain [1]–[4].
The associate editor coordinating the review of this manuscript and
approving it for publication was Cunhua Pan . deep learning (DL) have received much attention and shown
To deal with the complicated HBF optimization problem
with non-convex constraints, several algorithms have been
proposed recently. In [1], the authors devoted to selecting
analog beamformers from a pre-defined codebook via the
orthogonal matching pursuit (OMP) method. Subsequently,
the authors in [2]–[4] proposed some iterative algorithms
to further improve performance. Although these HBF algo-
rithms can deal with the multivariate optimization problem
with the non-convex constant modulus constraint in the ana-
log beamforming matrix, they either require some approxi-
mations to simplify the original objective function, or require
a lot of time-consuming serial iterations to obtain a solu-
tion. Recently, intelligent communications with the power of

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
120924 VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications

many benefits in tackling intractable conventional physical

level problems [5]–[11]. Some existing works applied DL to
the beamforming optimization problem [12]–[18]. In [12],
the authors proposed a neural network (NN) to select the
analog beamformer from a pre-defined codebook for max-
imizing the spectral efficiency. However, the restriction of
solution space might result in inevitable performance loss.
In [14], the authors designed an NN which can directly out-
put the optimized analog beamformer satisfying the constant
modulus constraint. The authors in [15] proposed a deep
FIGURE 1. Block diagrams of HBF designs with imperfect channel
NN (DNN) framework to construct an auto-precoder. The estimation. (a) Conventional channel estimation and HBF design;
authors in [17] unfolded the gradient ascent beamforming (b) Proposed MO codebook for HC-CE enhancement and robust DL-based
HBF design for spectral efficiency improvement; (c) Proposed DL-based
algorithm with a residual neural network. However, most of joint channel estimation and HBF optimization.
the above works assumed that perfect channel state informa-
tion (CSI) is available to form the beamformers, which may there are mainly three problems in the conventional design
not be satisfied in practical systems. that can be improved.
As perfect CSI is not available in practical systems, chan-
1) The design of the hierarchical codebook is critical to
nel estimation is usually carried out before HBF optimization.
channel estimation. The conventional HBF hierarchical
Many channel estimation algorithms with the HBF architec-
codebook requires a number of RF chains to achieve
ture have been proposed [19]–[30], including these [19]–[22]
the desired beam pattern. It is necessary to design a
applying the DL method. The authors in [19] regarded the
hierarchical codebook with good beam response and
channel as a 2D image and used image processing technol-
fewer RF chains.
ogy and a convolutional neural network (CNN) to achieve
2) Most of the existing works on HBF optimization
super-resolution image restoration and channel reconstruc-
assumed that perfect CSI is available to form the beam-
tion. The authors in [20] used residual denoising techniques
formers, which cannot be satisfied in practical systems.
to estimate noise and used an CNN to estimate the channel.
It is necessary to investigate the robust HBF design
The authors in [21] proposed to use NNs to learn the basic
considering the effect of channel estimation errors.
characteristics of the channel from low-rank measurements
As DL has been widely used in communication systems
and map the characteristics to a channel matrix. The authors
to achieve good performance [12]–[18], it motivates us
in [22] proposed a DL-based channel estimation algorithm in
to design robust HBF with DL.
a time-varying Rayleigh fading channel to dynamically track
3) In the conventional design, channel estimation and
the CSI. The authors in [23]–[25] converted channel estima-
HBF optimization are usually carried out in series with
tion into a sparse signal recovery problem. In [23], the authors
different goals. For example, people usually take the
achieved sparse signal recovery and finally achieved channel
objective of mean square error (MSE) minimization
estimation with the OMP algorithm. In [24], the authors
for channel estimation and that of spectral efficiency
proposed a structured random sensing matrix with robustness
maximization for HBF optimization. Thus, a chan-
in a low signal to noise ratio (SNR) range. The authors
nel estimate with a less MSE may not always result
in [25] proposed a compressive channel estimation frame-
in an HBF solution with a higher spectral efficiency.
work based on multiple measurement vectors and reduced the
Therefore, it is meaningful to jointly optimize channel
computational cost. The hierarchical codebook based channel
estimation and HBF with only one objective. There
estimation (HC-CE) algorithm [26]–[28] is another attractive
have been some works that jointly optimize multiple
method as it utilizes the sparse prior knowledge of mmWave
communication modules and achieve competitive per-
channels and requires a small number of pilot transmissions.
formance [31]–[33]. These works motivate us to apply
The authors in [26] designed a hierarchical codebook and
DL to jointly optimize channel estimation and HBF.
proposed an adaptive channel estimation algorithm with the
pre-designed codebook in both the single-path case and the
multi-path case. The authors in [27] compared the exhaustive B. CONTRIBUTIONS AND PAPER ORGANIZATION
search and hierarchical search from different aspects. In [28], To deal with the above problems in the conventional design,
the authors designed a hierarchical codebook by exploiting we have made the following improvements, which are also
beam widening with the sub-array technique. highlighted with red color in Fig. 1(b) and (c). Our contribu-
Since the HC-CE algorithm has been well studied, we take tions can be summarized as follows:
it as a practical channel estimation algorithm to obtain a 1) Manifold Optimization (MO) Based Codebook: Differ-
channel estimate. It normally includes three basic steps: code- ent from the conventional hierarchical codebook design
book design, hierarchical search, and channel reconstruction. approach [26], where the HBF codebook is optimized
Fig. 1(a) shows a typical block diagram of the conventional with the objective of approaching a digital beamform-
HBF design with imperfect channel estimation. However, ing DBF) codebook, in our approach, we optimize

VOLUME 9, 2021 120925

 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 2. Diagram of the downlink of a typical MIMO system with hybrid precoding.
each HBF codeword by directly targeting the expected
beam response and applying MO [2] to deal with the
constant modulus constraint in the analog beamformer.
The proposed MO codebook is shown to achieve better
beam response and better performance via simulation.
2) Robust HBF Design Based on DL: We propose
a DL-based robust HBF optimization network,
i.e., HBF-Net. As shown in Fig. 1(b), the input of the
HBF-Net is the estimated channel, and the output is
the optimized hybrid precoder. The HBF-Net adopts
convolutional layers and a depth attention mechanism
to efficiently exploit the characteristics of the imperfect
CSI. By setting the loss function of the HBF-Net to
the opposite value of the average spectral efficiency
over all training samples with perfect CSI during the
training stage, it can be trained to approach the ideal
spectral efficiency (the one without estimation errors)
as much as possible and thus exhibits robustness to
channel estimation errors.
3) Joint Channel Estimation and HBF Network: We fur-
ther propose a novel NN for joint channel estima-
tion and HBF optimization, which is referred to as
CE-HBF-Net. As shown in Fig. 1(c), the CE-HBF-Net
does not take the estimated channel as its input like
that in Fig. 1(b), but takes the intermediate data in
the hierarchical search process. Thus, the CE-HBF-Net
has the ability of joint channel estimation and HBF
optimization, and is expected to exhibit stronger robust-
ness and achieve higher spectral efficiency than the
HBF-Net. As the intermediate data must include not
only the information of which HBF codewords are
selected in the hierarchical search process but also the
resulting signals after HBF, in order to ensure that the
CE-HBF-Net is trainable, we skillful propose an index
assign-and-input (IAI) method to compress the dimen-
sions of the intermediate data. In addition to applying
convolutional layers and the depth attention mechanism
like that in the HBF-Net, we specially adopt a signal
self-attention mechanism in the design of the CE-HBF-
Net structure for better performance.
The rest of the paper is organized as follows. We intro-
duce the system model and the HBF problem formulation in
Section II. In Section III, we briefly review the procedure
120926
of the conventional HC-CE algorithm, and then optimize
the hierarchical codebook with MO. The basic ideas and
implementation details of the HBF-Net and the CE-HBF-
Net are presented in Section IV. We demonstrate various
simulation results in Section V. Finally, we conclude the
paper in Section VI.
We use the following notation throughout this paper: A
is a matrix, a is a vector and a is a scaler. IP is a P × P
identity matrix. kak denotes the norm of a. |A| denotes the
determinant of A and kAkF denotes its Frobenius norm.
A∗ , AH , A−1 are the complex conjugate, complex conjugate
transpose, and inverse of A, respectively. A† denotes the
pseudo-inverse of A, i.e., (A)† = (AH A)−1 AH . Finally,
if J (A) represents a real function of a complex matrix A, then
∇J (A) denotes the conjugate gradient of J (A) with respect
to A, i.e., ∇J (A) = ∂J∂A(A)
∗ .
II. SYSTEM MODEL
Consider the downlink of a typical narrowband MIMO
mmWave system shown in Fig. 2, where a base station (BS)
equipped with Nt antennas and NRF RF chains transmits Ns
data streams (Nt ≥ NRF ≥ Ns ) to a user equipped with Nr
antennas. For the ease of presentation, in this paper, we focus
on the design of hybrid precoder and assume that the user
uses a fully DBF architecture with Nr = Ns . The extension
to the design of hybrid combiner is similar to that of hybrid
precoder. At the BS, denote the Ns × 1 original symbol vector
by s with normalized average symbol energy, i.e., E{ssH } =
INs . After hybrid precoding with an NRF × Ns DBF matrix VB
and an Nt × NRF analog one VRF implemented using phase
shifters, from the equivalent baseband representation point of
view, the precoded signal can be represented as x = VRF VB s.
Assume that x is transmitted through an mmWave MIMO
channel H, and the received signal vector at the user is pro-
cessed with an Nr × Ns fully digital combiner WB . The com-
bined signal can be represented as y = WH H
B Hx+WB n, where
n denotes the additive noise vector satisfying the complex
circularly symmetric Gaussian distribution with zero
mean
and covariance matrix σ 2 INr , i.e., n ∼ CN 0, σ 2 INr . In this
paper, a widely-used mmWave channel model in [3], [26] is
applied, which is
r L
Nt Nr X
H= α` ar (θ` ) aH
t (φ` ) , (1)
L
`=1
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 3. Hierarchical codebook F5 with L = 3, K = 2, and N = 96.
where α` denotes the complex gain of the `-th path, for
` = 1, . . . , L, and at (φ` ) and ar (θ` ) denote the antenna array
response vectors at the BS and the user, respectively, with φ`
and θ` denoting the angle of departure (AoD) and the angle
of arrival (AoA) associated with the `-th path, respectively.
Similar to [1], [3], we take the spectral efficiency as the
performance metric. Under the assumption that the optimal
fully digital receiver is adopted at the user side with Nr = Ns ,
the objective can be expressed as
HVRF VB VH H
B VRF H
H
R = log INs + . (2)
σ2
The hybrid precoding optimization problem can be formu-
lated as follows
max R
VRF ,VB
s.t. [VRF ]ij = 1, ∀i, j
kVRF VB kF ≤ 1,
where the first constant modulus constraint comes from the
implementation of phase shifters for analog beamforming,
and the second constraint denotes the total transmit power
constraint with a normalized power.
III. CHANNEL ESTIMATION FOR mmWave CHANNELS
Channel estimation for the mmWave system with the HBF
structure is different from the conventional one with the fully
DBF structure mainly due to the fact that one should estimate
the high-dimensional air-interface MIMO channel based on
the low-dimensional observation signal space. In this section,
we first breifly review the conventional HC-CE algorithm,
and then further propose an enhanced codebook with MO for
better channel estimation.
A. HIERARCHICAL CODEBOOK BASED CHANNEL
ESTIMATION ALGORITHM
The basic idea of the HC-CE algorithm is to design a hierar-
chical codebook with the beamwidth becoming narrower for
higher beam-vector levels. Using the hierarchical codebook,
the HC-CE algorithm can gradually narrow the promising
angle range and finally get the estimated channel with a small
number of pilot transmissions.
VOLUME 9, 2021
Assume that the number of multiple channel paths, L,
is known at the BS.1 We denote the whole codebook by FS ,
where S is the number of the levels. Assume that each AoD
range in a certain level in the codebook is divided further
into K sub-ranges in the next level. At the first level, the full
AoD range is divided equally into KL exclusive sub-ranges.
Then, the total number of sub-ranges in the last level (i.e.,
the S-th level) is N = LK S , which determins the resolu-
tion of the codebook. Fig. 3 shows a BS codebook F5 for
a uniform linear antenna array (ULA) with the full AoDs
range [0, π], where f(s,n) denotes n-th codeword of the s-th
level. To estimate an L-path channel, the HC-CE algorithm
needs L outer iterations. In each one, an estimated AoD/AoA
is obtained after S inner iterations going through the BS
hierarchical codebook from the top level to the bottom level.
After computing each path gain by the least square algorithm,
the estimated channel can be reconstructed with (1). For more
details of the hierarchical search process, please refer to [26].
(3)
B. HIERARCHICAL CODEBOOK ENHANCEMENT WITH MO
The accuracy of channel estimation highly depends on the
hierarchical codebook. Ideally, choosing a codeword means
testing the corresponding angle range. However, the HBF
codebook suffers from beam power leakage and fluctuation
due to hardware limitations, which leads to beam alignment
deviation, and the received power is not only determined
by the path gain in the test angle range. In the two-step
HBF codebook design algorithm in [26], [34], for each HBF
vector in the codebook, an DBF vector satisfying the expected
beam response requirement is first designed, and then a cor-
responding HBF vector is generated to approach the DBF
vector. However, this algorithm requires a number of RF
chains to guarantee the DBF vector approximation accuracy.
In this subsection, we propose a new HBF codebook design
algorithm, where every HBF vector is optimized with fewer
RF chains by directly targeting the expected beam response
objective with MO.
Let  denote the cosine value of an AoD. We design an
S-level hierarchical codebook by dividing the  range instead
of AoD in [26]. Take Fig. 3 as an example, the full AoDs range
1 In practical, L can be estimated by classical direction-of-arrival (DOA)
estimation methods, e.g., multiple signal classification (MUSIC) and estima-
tion of signal parameters via rotational invariant techniques (ESPRIT).
120927
 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications

Algorithm 1: MO Based HBF Algorithm Although the problem in (8) is still difficult to solve due
Input: DBS , g to the non-convex constant modulus constraint, we can use
Output: FRF , fBB MO [2] to obtain a suboptimal solution. This is because the
Initialization FRF,0 randomly and set i = 0; constant modulus constraint defines a Riemannian manifold
Compute fBB according to (7); and MO uses the Riemannian gradient descent algorithm to
Repeat ensure that the searched point always satisfies the constant
1. Compute ∇J (FRF ) according to (11); modulus constraint. The Riemannian gradient is defined as
2. Use MO method to compute FRF,i+1 ; the projection of the Euclidean conjugate gradient onto the
3. i ← i + 1; tangent space of a point on the Riemannian manifold. The
4. Update fBB according to (7); challenge of applying MO is to derive the Euclidean conju-
Until a stopping criterion triggers gate gradient. According to [4], the differential of J (FRF ) can
be expressed as
 
d (J (FRF )) = tr ∇J (FRF ) d(FH RF ) , (9)
is [0, π], i.e.,  ∈ [−1, 1]. Each codeword design goal can
be expressed as
( where d (·) denotes the differential operation. According to
1 if φ̄u ∈ (s,n) the differential rule of complex matrix and the property of
f(s,n) aBS φ̄u =
H


(4) tr (AB) = tr (BA), we have
0 if φ̄u ∈
/ (s,n) ,
n
where aBS φ̄u is the antenna array response vector asso- d (J (FRF )) = tr DBS DH H H
BS FRF (fRF DBS DBS FRF )
−1

ciated with φ̄u , s = 1, · · · , S and n = 1, · · · , LK s . As a × fH

H H H
RF − INt DBS gg DBS FRF (fRF
result, the codeword f(s,n) is designed to achieve a constant  o
projection on the array response vectors within the expected × DBS DH −1
BS FRF ) d FRF
H
. (10)
directions and zero projection on the other directions. By uni-
formly quantizing range (s,n) and ignoring the quantization By comparing (10) with (9), it can be found that ∇J (FRF )
error, (4) can be rewritten as is given by

DHBS FRF,(s,n) fBB,(s,n) = g(s,n) , ∇J (FRF ) = DBS DH

(5) H H −1
BS FRF (fRF DBS DBS FRF )
H H H


aBS φ̄1 , . . . , aBS φ̄u , . . . , aBS φ̄N




where DBS = × fRF − INt DBS gg DBS FRF
RF DBS DBS FRF ) .
−1
defines an NBS ×N BS AoD dictionary matrix with N ≥ NBS , × (fH H
(11)
φ̄u = −1 + 2 · (u − 1)/N for u = 1, · · · , N , and g(s,n) is an
N × 1 vector with ones in locations u if φ̄u ∈ (s,n) and zeros With the derived Euclidean conjugate gradient, we can
if φ̄u ∈
/ (s,n) . For example, in Fig. 3, g(1,1) is an 96 × 1 vector compute the Riemannian gradient, obtain a point in the tan-
where the first 32 elements are ones, and the other elements gent space along the Riemannian gradient, and finally retract
are zeros. the searched points into the manifold. The MO process is
Since for any s and n, the codeword design method is summarized in the Algorithm 1. Note that as the transmitter
the same, for simplicity, we omit the subscript (s, n) in the has a total power constraint, the final HBF codeword is given
following derivation. The optimization problem becomes by
2
min DH
BS FRF fBB − g f=
FRF fBB
. (12)
FRF ,fBB
kFRF fBB k
s.t. [FRF ]ij = 1, ∀i, j
kVRF VB kF ≤ 1. (6) After calculating all codewords, an MO based HBF code-
book is generated. Although MO requires multiple iterations
To solve this problem, we first ignore the total power to obtain a local optimal solution, the HBF codebook is
constraint temporarily and obtain the optimal DBF vector generated offline and does not increase time overhead.
given a fixed analog beamforming vector FRF with the least It is worth noting that compared with the conventional
square algorithm, which is given by two-step HBF codebook design algorithm in [26], the pro-
† posed MO based algorithm directly optimizes each HBF
fBB = (DH
BS FRF ) g. (7) codeword aiming at the expected beam response objective
By substituting (7) back into (6) and defining J (FRF ) , and thus leads to a more accurate HBF codebook and the
2 resulting AoD estimates. It is also worth noting that although
DH H †
BS FRF (DBS FRF ) g − g , the problem in (6) becomes the above MO based HBF algorithm is proposed in the
min J (FRF ) case of fully connected HBF architecture, it can be readily
FRF extended to other cases such as the partially connected HBF
s.t. [FRF ]ij = 1, ∀i, j. (8) architecture.

120928 VOLUME 9, 2021

Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
IV. HBF OPTIMIZATION WITH DL
While the HBF optimization is a complicated non-convex
problem and it is unlikely to find a closed-form optimal solu-
tion [1], DL has been regarded as an efficient design approach
and is expected to handle this intractable problem [35]. In this
section, we first propose a robust HBF optimization network,
i.e., HBF-Net, with the input of imperfect CSI. We also
propose a joint channel estimation and HBF optimization
network, i.e., CE-HBF-Net, for better performance.
A. DESIGN OF HBF-Net
There have been some iterative algorithms in the literature to
deal with the intractable constant modulus constraint in the
HBF optimization [1]–[3]. However, these algorithms either
require approximations to simplify the original objective
function or require a lot of time-consuming serial iterations
to obtain a solution. Besides, most of these works are based
on the knowledge of perfect CSI and thus suffer from certain
performance loss due to channel estimation errors. As the
DL-based scheme has exhibited its ability in extracting the
complicated characteristics of the wireless channels [6], [7],
in this subsection, we propose to apply DL to solve (3) with
imperfect CSI.
1) THE INPUT AND OUTPUT SETTING
Generally speaking, in almost all the conventional HBF opti-
mization algorithms [1]–[4], the HBF optimization can be
essentially represented as a function of perfect CSI and SNR,
VRF , VB = f (H, γ ).
This inspires us to apply DL to solve the same problem,
as the NN has already demonstrated its ability to fit complex
functions with a large number of neurons and nonlinear acti-
vation functions [6], [35].
Since the perfect CSI, H, is normally not available in prac-
tical systems, the estimated CSI, Ĥ, is set as the input of the
HBF-Net. By separating the real and imaginary parts of the
estimated CSI, the input Ĥ is represented by an Nr × Nt × 2
real-valued tensor. To achieve the generalization for various
SNRs, we feed the noise power into the HBF-Net as the
transmit power is normalized.
In order to ensure that the analog precoder VRF meets the
constant modulus constraint and the HBF precoder meets the
total power constraint, the HBF-Net first outputs the phase
information of each element, i.e., Nt NRF × 1 real-valued
vector (the output of a fully connected layer). The vector is
reshaped to an Nt × NRF matrix (denoted by 8), and then
finally reconstructs the analog precoder with Euler’s formula.
The desired analog beamformer VRF is given by
VRF = exp(j · 8) = cos(8) + j · sin(8), (14)
√ operations, we obtain the final depth attention values. The
where j = −1. We denote the transformation from 8 to
VRF as ‘‘Lambda1’’. As to the digital precoder VB , we con-
vert the output from a 2Ns NRF × 1 real-valued vector (the
output of a fully connected layer) to an Ns × NRF complex-
valued matrix (denoted by V0B ), and then perform power
VOLUME 9, 2021
normalization (denoted by ‘‘Lambda2’’). The desired digital
beamformer is given by
V0B
VB = . (15)
kVRF V0B kF
In summary, the HBF-Net can be expressed as a nonlinear
function g(·) of the estimated CSI and the noise power. That
is,
VRF,k , VB,k = g(Ĥk , σk2 ), (16)
where Ĥk and σk2 denote the estimated CSI and the noise
power associated with the k-th sample. Note that the proposed
HBF-Net can optimize the digital matrix and analog matrix
simultaneously instead of alternating optimization like that
in [2]–[4].
2) STRUCTURE OF HBF-Net
The overall structure of the HBF-Net is shown in Fig. 4(a).
It adopts two convolutional (Conv) blocks for feature extrac-
tion. To enhance the generalization ability for different SNRs,
the output tensor of the second convolutional block is flat-
tened to a one-dimensional vector and concatenated with the
noise power σ 2 before entering the subsequent two fully con-
nected layers with Nt NRF and 2Ns NRF neurons, respectively.
The detailed structure of the convolutional block is shown
in Fig. 4(b), which includes a convolutional layer, a depth
attention module, and a batch normalization layer.
• Convolutional Layer: According to (1), the sparsity of
(13) the channel and the correlation among its elements moti-
vate us to apply the convolutional layer, which has been
widely used in computer vision to enhance the opera-
tional efficiency of NNs.
• Depth Attention Module: Depth attention mechanism
can enhance the feature extraction ability of the con-
volutional layer by assigning different weights to depth
dimensions of its outputs, as those corresponding fea-
tures have different contributions to the NN target [36].
• Batch Normalization (BN) Layer: It can make the out-
put distribution more uniform and prevent gradient
dispersion.
In the first convolutional block, the convolutional layer uses
4Nt different 3 × 2 filters with the well-known ReLu (i.e.,
f (x) = max(0, x)) as activation function. In the realization of
the depth attention, both global average pooling and global
maximum pooling are used to obtain the depth descriptor.
Two dense layers are used to extract the non-linear depen-
dency between depth dimensions. Note that to reduce the
trainable parameters, dense layers after pooling operation
share parameters. After adding the outputs of two excitation
re-weighting operation Fscale (·) outputs the weighted spatial
feature map. The implementation of the second convolutional
block is similar to the first one, except that the convolutional
layer uses 2Nt different 3 × 1 filters. The output shapes of
the main layers/blocks of the HBF-Net are listed in Tab. 1.
120929
 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 4. Network structure of the proposed HBF-Net. (a) The overall network structure; (b) The detailed structure of the convolutional (Conv)
block in (a).
TABLE 1. Implementation details of the HBF-Net.
B. DESIGN OF CE-HBF-Net
Although the conventional HC-CE algorithm has the advan-
tage of fewer pilot transmissions compared with the exhaus-
tive beam search algorithm, it has several limitations that can
be considered for improvement. Firstly, in each angle range
refining iteration, a ‘‘hard’’ decision is made about whether
the AoD belongs to a particular range or not. However,
the ‘‘soft’’ information of how likely the AoD belongs to a
range is not sufficiently utilized. Furthermore, the channel
estimation accuracy is limited by the resolution of the last
level of the codebook. Secondly, in the multipath scenario,
the AoDs/AoAs of the multiple paths are not jointly estimated
but estimated one by one. The multipath interference may
greatly affect the estimation accuracy. Finally, the HC-CE
algorithm estimates the path gain at the last level in each
outer iteration. Actually, the upper-level searches are also
useful for the path gain estimation. According to the above
discussion, the HC-CE algorithm can be further enhanced.
As the estimation error in the conventional algorithm comes
from multiple aspects and is challenging to model, we resort
to designing an NN to improve the performance.
Intuitively, we can design two networks to implement
channel estimation and HBF optimization, respectively. How-
ever, such a two-step optimization is suboptimal and the joint
channel estimation and HBF optimization is more attractive
due to the following reasons:
• Implicit Channel Estimation Considering HBF Opti-
mization: Conventional channel estimation and HBF
design are usually optimized separately, which indicates
that the HBF design does not affect the channel estima-
tion. Joint optimization allows the channel estimation
to take the HBF design into account and obtain an esti-
mated channel more beneficial to the HBF design.
120930
• More Effective Information Extraction for the Hierar-
chical Search: In the conventional HC-CE algorithm,
the function of the upper-level search is limited to nar-
rowing the beam alignment ranges of the lower-level,
and the estimation does not fully utilize the information
such as the power of the resulting signal after HBF
(representing how likely an AoD (or AoA) belongs to
a range). joint optimization is expected to extract more
information from all the received signals and obtain an
HBF design with a higher spectral efficiency.
• Joint Optimization With the Same Goal: The conven-
tional design approach for channel estimation and HBF
optimization in mmWave MIMO communication sys-
tems is usually split into two independent blocks with
different optimization objects (e.g. MSE minimization
for channel estimation and spectral effecient maximiza-
tion for HBF optimization). As the goals of these two
blocks do not always entirely match, the joint optimiza-
tion with a unified goal is supposed to achieve higher
spectral efficiency than the conventional separate design
approach.
Although the basic idea and motivation of the joint opti-
mization are clear, it is difficult to sovle the problem with the
conventional model-based method. Fortunately, DL provides
an attractive approach. In this paper, we propose the CE-HBF-
Net to implement channel estimation and HBF optimization
jointly.
1) THE INPUT AND OUTPUT SETTING
One key issue in the design of the CE-HBF-Net is how to
set its input, which includes: What should be input to the
CE-HBF-Net in order to achieve better channel estimation
and HBF optimization? How to input efficiently?
a: RECEIVED SIGNALS
One necessary input is the collection of the received signals.
By feeding the combined signals in each outer and inner
iteration into the CE-HBF-Net, the CE-HBF-Net is expected
to fully utilize the historical ‘‘soft’’ information and learn
intelligently how to compensate for the codebook quantiza-
tion error and how to reduce the multipath interference in
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 5. Network structure of the proposed CE-HBF-Net. (a) The overall network structure; (b) The detailed
structure of the convolutional (Conv) block in (a).
the offline training stage. As the backpropagation algorithm
of the NN only supports real number operations, we feed
the real and imaginary parts of the digital signals into the
CE-HBF-Net. Such processing does not lose any information
because the real and imaginary parts can completely char-
acterize the amplitude and phase of the combined signals.
We denote the collection of the real and imaginary parts of
the received digital signals as R. Denoting the number of pilot
transmissions as Np , then the dimension of R is 2Np .
b: PRECODING AND COMBINING VECTORS
Different from the existing works on the DL-based chan-
nel estimation [18], [37], where the precoders and combin-
ers are the same for different channel realizations, in the
CE-HBF-Net, the precoders and combiners are adaptively
selected from the codebook. These selected vectors are
closely related to the AoDs/AoAs of each realized channel
and thus are different for different channel realizations. It is
necessary to tell the CE-HBF-Net the precoding and combin-
ing vectors associated with R.
However, if we directly input the selected vectors to the
CE-HBF-Net, the input dimension would be extremely high
and even prohibitive. For example, assuming that NBS = 64,
and a total of 30 HBF complex vectors are selected for signal
precoding in the whole estimation process, a long vector with
64 × 30 × 2 = 3840 real numbers is then needed to represent
all the selected precoding vectors. The input dimension for
representing the precoding vectors would come up to 3840.
As a result, the CE-HBF-Net would be very complex and hard
to converge.
To reduce the input dimensions of the selected vectors,
we need to find some smart method. For the BS side, note that
VOLUME 9, 2021
no matter how different the channel is for each realization,
the whole hierarchical codebook is the same. Thus, an effi-
cient method is to assign an index (i.e., a unique number)
to each vector in the hierarchical codebook and input only
the indices of the selected vectors to the CE-HBF-Net. It is
expected that with enough training samples, the CE-HBF-Net
can intelligently learn the relationship between the combined
signal, the index, and the selected vector. We refer to this
index assign-and-input method as IAI.
One way to assign indices can be found by noting that each
selected HBF vector has a dedicated position in the codebook,
which is determined by the row number (in which level) and
the column number (which one in this level) of the codebook,
as shown by the subscript (s, n) of each HBF vector f(s,n)
in Fig. 3. Thus, the pair of numbers (s, n) can be taken as
the HBF index. Actually, the row number s is not necessary
to be input to the CE-HBF-Net since the HBF vectors are
always selected from the top to the bottom level. The first
level of the codebook will always be selected, so there is no
need to assign indices, as in [18], [37]. Moreover, if an AoD
range in the s-th level is selected as one candidate range where
the path is probably in (for example, assuming it is [ π6 , 2π 6 ]
in the 1st level in Fig. 3), then the K adjacent HBF vectors
related to this range in the (s + 1)-th level will be all selected
and tested for determining a finer sub-range (f(2,3) , f(2,4) in
the 2nd level in Fig. 3). The column number of the selected
HBF vectors in the upper level can fully represent the K
selected HBF vectors in the next level. As a result, it is only
necessary to assign indices to the HBF vectors that win in
the angle test of each level, and the indices of 1/K vectors
need to be input to the CE-HBF-Net. It is expected that the
CE-HBF-Net can intelligently learn this pattern. In the above
120931
 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
TABLE 2. Implementation details of the CE-HBF-Net.
example with NBS = 64 and a total of 30 selected HBF
vectors, if assuming K = 2, the input dimension can be
reduced from 64 × 30 × 2 = 3840 to 30
K = 15 by using
the IAI method. The user codebook has only one level, and
the estimated AoA is obtained in the first inner iteration. For
an L-path channel, the number of indices is L. We denote
the collection of the indices of the selected precoding and
combining vectors as Isel , and the corresponding dimension
is Nsel .
c: SNR
The SNR is an optional input, and it would be a good indicator
for the CE-HBF-Net. Since the transmitter has a total power
constraint, we feed noise power σ 2 to the CE-HBF-Net if it
is available.
The output setting is as the same as the HBF-Net. In sum-
mary, the CE-HBF-Net can be expressed as a nonlinear
function h(·) of the received signal collection R, the index
collection Isel and noise power σ 2 . That is,
VRF,k , VB,k = h(Rk , Isel,k , σk2 ), (17)
where Rk , Isel,k and σk2 denote the collection of the real
and imaginary part of the recieved signals, the collection of
indices of the selected precoding and combining vectors and
the noise power associated with the k-th sample, respectively.
Similar to the HBF-Net, the proposed HBF-Net also can
optimize the digital matrix and analog matrix simultaneously
instead of alternating optimization like that in [2]–[4].
2) STRUCTURE OF CE-HBF-Net
In the HC-CE algorithm, as the BS sends a number of pilots
using different HBF codewords from the top to the bottom
level of the hierarchical codebook, the contribution of differ-
ent received pilot signals to channel estimation in the hierar-
chical search is different. For example, since there is no prior
knowledge of the actual channel, doing a hierarchical search
in a large range is necessary. Naturally, those searches aligned
to the AoDs and AoAs contribute more to channel estimation.
Theoretically, the received signal power represents the possi-
bility that the corresponding target range contains scattering
paths, and also represents the attention value of the signal.
However, the beam response of each codeword is imper-
fect with power leakage and fluctuating beam gain, and the
received signal does not correspond precisely to the desired
120932
target range. As it is challenging to eliminate such errors,
we propose a signal attention mechanism to compensate for
the beam pattern imperfections and enable the NN to learn
how to assign signal attention values accordingly.
The block diagram of the proposed signal attention mech-
anism is shown in Fig. 5(a). First, concatenate the received
signal collection R, the index collection Isel and noise power
σ 2 together. Then, input them to the proposed signal attention
vector generator and obtain the learned signal attention vec-
tor. In this paper, we use one dense layer to generate the signal
attention vector. The output of the dense layer is determined
by its learnable parameters, which are updated to reduce the
loss of CE-HBF-Net. As a result, the attention vector gener-
ator tends to learn an attention vector conducive to network
convergence. Finally, the received signals weighted with the
signal attention vector.
After processing by the signal attention mechanism,
the weighted received signals are connected with the indices
collection Isel and the noise power σ 2 , and are fed into the
CE-HBF-Net. The second dense layer of the CE-HBF-Net
can be regarded as the first step of implicit channel esti-
mation. As the number of real dimensions of the mmWave
channel matrix is 2Nr Nt , the number of neurons in the fully
connected layer is set to 6Nr Nt to ensure sufficient feature
extraction and allow some redundant information. The output
of the fully connected layer is reshaped to Nr × Nt × 6, where
the size of the space feature map is Nr × Nt and the number
of depth dimensions is 6. The subsequent feature extraction
is performed by two convolutional blocks.
The structure of the convolutional block in CE-HBF-Net is
as shown in Fig. 5(b). Compared with the convolution block
in the HBF-Net, since the CE-HBF-Net allows redundancy,
the average pooling layer is added to reduce some redun-
dancy. In the first convolutional block, the convolutional layer
applies 8Nt different 3 × 2 filters with the ReLu as activation
function. The pooling window size is 2 × 1, and the step size
is 2. The implementation of the second convolutional block
is similar to the first, except that the convolutional layer uses
6Nt different 3 × 1 filters. The output shapes of the main
layers/blocks of the CE-HBF-Net are listed in Tab. 2.
C. UNSUPERVISED LEARNING STRATEGY BASED NEURAL
NETWORK TRAINING
1) UNSUPERVISED LEARNING AND LOSS FUNCTION DESIGN
Most of the conventional intelligent communication designs
use the supervised learning strategy. For example, in [6],
[7], [38], the labels are set to the transmitted bits or perfect
channel estimates, and the loss function is set to the MSE
between the output signals of the NN and the labels. However,
it is difficult to find a proper label for the design of VRF and
VB . If we take an optimized HBF beamformer based on a
conventional algorithm as the label and adopt the MSE as the
loss function, then the NN can only be trained to approach
the conventional algorithm but cannot outperform it. Perhaps
taking the fully digital beamformer as the label is another
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 6. Offline training and online deployment strategy. (a) HBF-Net;
(b) CE-HBF-Net.
choice, but such a matrix approximation method also results
in some performance loss. Notice that the ultimate objective
of HBF optimization is to maximize the spectral efficiency in
(3). Here, we use an unsupervised learning strategy without
the label, and propose to optimize VB and VB directly aiming
at the HBF optimization objective. A loss function directly
related to the objective of spectral efficiency in (3) is adopted,
which is given by
N
1 X Hk VRF,k VB,k VH H H
B,k VRF,k Hk
L=− log INs + , (18)
N
k=1
σk2
where N denotes the total number of training samples, and
VRF,k and VB,k represents the output of the HBF-Net and
CE-HBF-Net in the k-th training. It can be clearly seen
from (18) that the physical meaning of the definition of
the loss function exactly corresponds to the opposite value
of the average spectral efficiency over all training samples.
The less the loss value, the higher the average spectral effi-
ciency. As DL is essentially a gradient descend method [35],
the loss value is guaranteed to converge to a local minimum
point with training iterations and leads to the increase of
the average spectral efficiency. Besides, with the particular
loss function associated with the perfect CSI, the HBF-Net
and the CE-HBF-Net can be trained to approach the ideal
spectral efficiency (the one without estimation errors) as
much as possible. In particular, since the optimization goal
of the CE-HBF-Net is to maximize the spectral efficiency,
the implicit channel estimation tends to be jointly opti-
mized targeting at higher spectral efficiency, and thus the
CE-HBF-Net is expected to achieve better performace than
the HBF-Net due to such joint optimization.
2) OFFLINE TRAINING AND ONLINE DEPLOYMENT
For the sake of avoiding a time-consuming online training
process, similar to [6], [39], both the HBF-Net and the
CE-HBF-Net are trained offline with simulated samples as
shown in Fig. 6. Note that based on the channel model in (1),
enough samples can be generated by simulation during the
offline training stage. There is a trade-off between the num-
ber of training samples and the performance of achievable
VOLUME 9, 2021
spectral efficiency. However, it is worth noting that as the
networks are trained offline, the computational complexity of
the training stage is a less concern. This is because the size of
the training set only affects the offline time overhead, which
is not strictly limited [14]. In this paper, we use 1.8 × 106
training samples in the offline training stage to ensure that
both the HBF-Net and CE-HBF-Net fully converge, and set
the sizes of the verification and test data sets to 2 × 105 and
104 , respectively. To achieve the generalization for various
SNRs, σk2 is generated randomly between −10dB and 10dB.
The Adam optimizer is applied to train the HBF-Net and
the CE-HBF-Net via the stochastic gradient descent method.
We use the learning rate decay strategy. If the loss does not
decrease after training for 20 epochs, the learning rate is
multiplied by 0.2, where the initial and minimum values of
the learning rate are set to 10−3 and 5 × 10−5 , respectively.2
As shown in Fig. 6, the trained HBF-Net can replace the
conventional HBF designer, and the trained CE-HBF-Net can
replace the conventional channel estimation algorithm and
the conventional HBF designer. Most conventional HBF algo-
rithms require a number of serial iterations to achieve good
performance [2]–[4]. In contrast, the proposed DL-based
schemes with a limited number of matrix multiplications
can operate fast due to the acceleration of parallel com-
puting and is, therefore, more applicable for high-speed
communications.
D. COMPLEXITY ANALYSIS
In this subsection, we analyze the computational complexity
of the proposed HBF-Net and CE-HBF-Net in terms of the
number of real multiplications. The number of real multipli-
cations for a dense layer is NI NO , where NI and NO denote the
input and output dimensions, respectively. The number of real
multiplications for a convolutional layer is CI CO kw kh WH ,
where CI and CO denote the input and output depth dimen-
sions, respectively, kw × kh is the size of convolution kernel,
and W × H is the output spatial dimension [40]. By substitut-
ing the specific hyper-parameter settings, we obtain the com-
putational complexity as shown in Tab. 3 and Tab. 4, where
Ns , Nt and NRF are the number of data streams, the number
of BS antennas and the number of RF chains, respectively,
and Np and Nsel are the number of pilot transmissions and
the dimension of the selected indices, respectively. According
to [4], for the conventional model-based HBF algorithms
proposed in [3] and [4], the asymptotic computational com-
plexity in terms of the number of complex multiplications is
O(Nt3 ) and Nlp (4Nt2 NRF + 13Nt NRF 2 + 3N N 3
t RF + 4NRF +
3
4O(NRF )), respectively, where Nlp denotes the number of
total iterations.
Numerically, for Ns = NRF = Nr = 2, Nt =
64, Nsel = 48 and Np = 180, the proposed HBF-Net
and CE-HBF-Net, and the conventional HBF algorithms
2 All source codes and trained models are provided openly in
https://github.com/LQJecho/Channel-Estimation-and-Hybrid-Precoding-
for-Millimeter-Wave-Systems-Based-on-Deep-Learning
120933
 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
TABLE 3. Computational complexity of the proposed HBF-Net.
TABLE 4. Computational complexity of the proposed CE-HBF-Net.
proposed in [3] and [4] require around 7.46×106 , 2.08 × 107 ,
1.18 × 107 , and 6.64 × 106 real multiplications, respec-
tively. It can be seen that the proposed HBF-Net has com-
petitive computational complexity when compared with the
conventional model-based HBF algorithms. As the task of
the CE-HBF-Net is more complicated, including implicit
channel estimation and HBF optimization, its complexity
is the highest. However, different from the conventional
algorithms, the matrix multiplication of the HBF-Net and
the CE-HBF-Net can be efficiently accelerated by parallel
computation via graphics processing unit [40]. Under the
hardware configuration of Intel(R) Core(TM) i9-9900K CPU
and GeForce GTX 1080 Ti, the conventional HBF algorithms
proposed in [4] and [3] (implemented in MATLAB) take
1.14 × 10−2 s and 9.10 × 10−3 s for a hybrid precoding opti-
mization, while the HBF-Net and the CE-HBF-Net (imple-
mented in Python) only need 4.6 × 10−5 s and 5.8 × 10−5 s.
The proposed DL-based NNs can better meet the demand
of real-time communication under the existing hardware
configuration.
V. SIMULATION RESULTS
Throughout the simulation, a half-wave spaced ULA is
deployed at the BS and the user with Nt = 64 and Nr =
2. At the BS, the number of RF chains and that of data
streams are set to NRF = Ns = 2. The mmWave channel
model in (1) is used with exactly the same parameters as
those in [3], [26], where the path number L is set to 3, α`
satisfies independently and circularly symmetric Gaussian
distribution with zero mean and unit variance, and both φ`
and θ` satisfy independently uniform distribution in [0, π].
Suppose that the estimation of the path number is correct,
i.e., Lest = L.
The BS adopts an HBF codebook with S = 5 levels,
and each AoD range in a level is divided further into two
sub-ranges in the next level, i.e., K = 2. The resolu-
tion N of the BS codebook is 96. The number of outer
iterations is equal to the number of paths, i.e., L = 3.
According to [26], the HC-CE algorithm requires a total of
120934
Np = (KL × KL + KL × (S − 1)) × L = 180 pilots,
the BS selects M = KL × S × L = 90 HBF vectors for each
channel realization, and the user selects KL × L = 18 DBF
vctors for each channel realization. From the discussion in
Section IV-B1, the input dimension of the CE-HBF-Net is 409
(dim(R) = 2Np = 360, dim(Isel ) = Nsel = M K + L = 45 +
3 = 48 by using the IAI method proposed in Section IV-B,
and dim(σ 2 ) = 1).
Our simulation results consist of four parts. The first part
is to show the performance improvement of the proposed
MO codebook. The second part is to compare the DL-based
HBF NNs with the conventional algorithms. The third part
is to verify the effectiveness of our network structure, where
the advantages of signal attention mechanism, deep attention
mechanism, convolution block, and CE-HBF-Net input set-
tings are demonstrated via simulations. The fourth part is to
demonstrate the robustness of the DL-based HBF NNs to
the mismatches between the offline training stage and the
online deployment stage. For example, channel path mis-
match, channel model mismatch, and non-ideal device char-
acteristics such as random errors of phase shifters.
A. CODEBOOK ENHANCEMENT
Fig. 7(a) shows the normalized MSE (NMSE) of imper-
fect channel estimation as a function of the pilot to noise
ratio (PNR) for three different codebooks with 5 levels in
the hierarchical search of the HC-CE algorithm. Fig. 7(b)
further shows the resulting spectral efficiency of the HBF
optimization algorithm in [3] by directly taking the imperfect
estimated channel from that in Fig. 7(a), when the SNR
is fixed at 0dB. As shown in these two figures, compared
with the two conventional codebooks in [26] and [28], our
proposed MO codebook can achieve more accurate channel
estimation and higher spectral efficiency in the entire PNR
range.
Next, to investigate how pilot overhead affects channel
estimation accuracy and spectral efficiency, Fig. 8 demon-
strates the NMSE and achievable spectral efficiency with
different numbers of codebook levels when PNR = 0dB and
SNR = 0dB. As shown in this figure, the performance of
all the three codebooks is improved with the increase of the
number of codebook levels, and the proposed MO codebook
always results in higher spectral efficiency and more accurate
channel estimation than the conventional codebooks. A close
observation to the performace curves of the proposed MO
codebook reveals that there is little performance increase
when the number of levels is larger than 5. Considering the
trade-off between performance and pilot overhead, a good
choice to the number of the MO codebook levels is 5,
i.e., S = 5, which is used in the following simulations.
B. DIFFERENT HBF ALGORITHMS
In this part, we compare the spectral efficiency of different
HBF algorithms to show the superiority of our proposed two
DL-based NNs. Fig. 9 shows the spectral efficiency versus
the SNR for different HBF algorithms when the PNR is set
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications

FIGURE 7. Performance comparison between the conventional codebooks and the proposed MO codebook. (a) Channel
estimation error in terms of NMSE v.s. PNR; (b) Spectral efficiency v.s. PNR.

FIGURE 8. The influence of the number of BS codebook levels on channel estimation. (a) NMSE v.s. the number of BS codebook
levels; (b) Spectral efficiency v.s. the number of BS codebook levels.

to 0dB. As shown in the figure, the proposed HBF-Net and
CE-HBF-Net can achieve higher spectral efficiency than the
two conventional HBF algorithms with imperfect channel
estimation, where the CE-HBF-Net achieves the better per-
formance as it can extract information conducive to HBF
design more flexibly and implement joint channel estimation
and HBF optimization.
Fig. 10 shows the spectral efficiency versus the PNR for
different HBF algorithms when the SNR is fixed at 0dB.
As shown in this figure, all the HBF algorithms achieve
higher spectral efficiency with more accurate channel esti-
mation (due to the increase of PNR). The proposed HBF-Net
and CE-HBF-Net both outperform the two conventional HBF
algorithms over the entire PNR range.
Fig. 11 shows the spectral efficiency versus the number
of BS codebook levels S for different HBF algorithms when
SNR = 0dB and PNR = 0dB. As shown in Fig. 11, no matter
what value S takes, both the HBF-Net and the CE-HBF-Net
perform better than the two conventional algorithms. Note
that the CE-HBF-Net under different values of S has different
input dimensions because of its joint channel estimation and
HBF optimization design approach. Thus, the CE-HBF-Net
under different S needs to be retrained. However, the perfor-
mance gain of CE-HBF-Net is more significant, especially
with a smaller S, which indicates that the CE-HBF-Net has
stronger robustness to channel estimation errors than other
algorithms. On the other hand, as the input of HBF-Net is the
estimated channel, no matter what value S takes, the input
dimension of HBF-Net remains the same. It means that we
only need to train the HBF-Net once since a well-trained
NN model usually has good generalization ability. In this
simulation, the HBF-Net was trained under S = 5 but test for
different values of S. As shown in Fig. 11, the HBF-Net can
achieve higher spectral efficiency than the two conventional
algorithms under different S without retraining. Therefore,
the proposed HBF-Net and CE-HBF-Net provide two choices
to practical systems depending on the target performance
requirement and the computational complexity budget.
C. NETWORK STRUCTURE DESIGN
The performance improvement shown in Section V-B not
only comes from the basic idea of joint channel estima-
tion and robust HBF optimization, but also from the effi-
cient design in the network structures in Fig. 4 and Fig. 5.
In this part, we compare the performance of HBF-Net and
CE-HBF-Net with different network structures to demon-
strate the effectiveness of our well-designed structures.
Fig. 12(a) shows the performance of the CE-HBF-Net with
or without the signal attention mechanism, where the system
setup and the codebook are the same as those in Fig. 9 and the

VOLUME 9, 2021 120935

 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 9. Spectral efficiency v.s. SNR for different HBF algorithms.
FIGURE 10. Spectral efficiency v.s. PNR for different HBF algorithms.
PNR is set to 0dB. As shown in Fig. 12(a), the CE-HBF-Net
with the signal attention mechanism (the curve labeled with
‘‘CE-HBF-Net’’) can obtain higher spectral efficiency than
that without the signal attention mechanism (the curve labeled
with ‘‘CE-HBF-Net w/o SA’’).
Fig. 12(b) shows the performance of the HBF-Net and the
CE-HBF-Net with or without the depth attention mechanism.
It can be seen that the introduction of the depth attention
mechanism brings a spectral efficiency improvement of about
0.5bits/s/Hz to both the HBF-Net and the CE-HBF-Net.
Fig. 12(c) compares the performance of the proposed
HBF-Net and CE-HBF-Net with that of their variants by
replacing the network structure with the fully connected
structure (labeled as ‘‘HBF-Net DNN’’ and ‘‘CE-HBF-Net
DNN’’, respectively) while keeping the total number of train-
able parameters approximately unchanged. It can be seen
from this figure that, compared to the simple DNN structure,
there is about a performance gain of 0.7bits/s/Hz in the
spectral efficiency at high SNRs for both the DL-based HBF
NNs due to the specific considerations of using the convo-
lutional layer and the depth attention mechanism for both
the HBF-Net and the CE-HBF-Net, and the signal attention
mechanism for the CE-HBF-Net.
To demonstrate the novelty of the CE-HBF-Net input set-
ting with the IAI method, we also simulate two variants
of the CE-HBF-Net with different input settings: (a) Input
only σ 2 and R (labeled as ‘‘CE-HBF-Net-V1’’). (b) Input
only σ 2 , R and Isel in the last inner iteration of every
120936
FIGURE 11. Spectral efficiency v.s. the number of BS codebook levels for
different HBF algorithms.
outer iteration (labeled as ‘‘CE-HBF-Net-V2’’). Both two
CE-HBF-Net variants have the same network structure as the
CE-HBF-Net except for input settings. Comparing the curves
of the CE-HBF-Net and the CE-HBF-Net-V1 in Fig. 12(d),
we can see the performance gap with or without inputting
the information of the precoding and combining vectors,
and the efficiency of the proposed IAI method. Furthermore,
as shown in this figure, the CE-HBF-Net outperforms the
CE-HBF-Net-V2 since it utilizes the ‘‘soft’’ information of
upper-level pilot transmissions, i.e., how likely the AoD
belongs to a sub-range.
D. ROBUSTNESS TO MISMATCHES BETWEEN OFFLINE
TRAINING AND ONLINE DEPLOYMENT
For the sake of avoiding the time-consuming training process
online, both the HBF-Net and the CE-HBF-Net are trained
offline with simulated samples. However, there are inevitably
mismatches between actual scenarios and training scenarios.
Therefore, the robustness to these mismatches is another
critical metric for our proposed two DL-based HBF NNs.
In the previous simulation, it was assumed that the esti-
mation of the path number is correct. However, in practical
applications, the estimation of the path number may not be
accurate. Therefore, the robustness to this mismatch should
be evaluated. We assume that the channel in the offline
training stage and the actual one in the online test follow
the same model, but with different numbers of containing
paths, denoted by Lsim and L, respectively. Fig. 13 shows the
spectral efficiency as a function of SNR for different HBF
algorithms with a mismatch between Lsim and L. It is assumed
that during the offline training stage, both the HBF-Net and
the CE-HBF-Net are trained with Lsim = 3, and the estima-
tion of the path number is always correct in the offline training
stage. However, Considering the sparsity of the mmWave
channels, L of the actual channel during the online test stage
is set to L = 1 or L = 2. As shown in Fig. 13, compared
with the two conventional HBF algorithms, the proposed two
NNs are more robust to the mismatch, and the more serious
the mismatch, the stronger the robustness. For example, for
a target spectral efficiency of 5bits/s/Hz, the SNR gains of
the HBF-Net and CE-HBF-Net over the conventional HBF
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications

FIGURE 12. Spectral efficiency of the DL-based HBF NNs with different network structures. (a) Impact of signal attention (SA)
mechanism on the CE-HBF-Net; (b) Impact of depth attention (DA) mechanism on the HBF-Net and the CE-HBF-Net;
(c) Spectral efficiency v.s. SNR for the HBF-Net and CE-HBF-Net with the proposed structures and the fully connected
networks; (d) Spectral efficiency v.s. SNR for the CE-HBF-Net with different input settings.

FIGURE 13. Spectral efficiency v.s. SNR for different HBF algorithms with the mismatch in the number of paths. (a) Lsim = 3, L = 1;
(b) Lsim = 3, L = 2.

algorithm in [3] are 1.4dB and 2.1dB, respectively, for L = 2,
and are increased to 2.6dB and 3.4dB, respectively, for L = 1.
In the previous simulation, it was assumed that all the
paths are non-line-of-sight (NLoS) and the complex gain
of each path satisfies independently and circularly symmet-
ric Gaussian distribution with zero mean and unit variance.
However, the actual wireless channel may be line-of-sight
(LoS). Therefore, it is necessary to consider the robustness
of the DL-based HBF NNs to the mismatch in the channel
model. Fig. 14 shows the spectral efficiency of different HBF
algorithms under the LoS channel model. For the proposed
HBF-Net and CE-HBF-Net, they were trained in the NLoS
channel model but tested directly in the LoS model. The
NLoS channel model is the same as that in the previous sim-
ulations. For the LoS model, it is assumed that there are still
three paths, where the gain of the LoS path is fixed at 2, and
the complex gain of the other two paths satisfy independently
and circularly symmetric Gaussian distribution with zero
mean and 0.5 variance. As shown in the figure, the HBF-Net
and the CE-HBF-Net can achieve higher spectral efficiency
than conventional algorithms without retraining, which indi-
cates that the DL-based NNs can learn the characteristics of

VOLUME 9, 2021 120937

 Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
FIGURE 14. Spectral efficiency v.s. SNR for different HBF algorithms
tested in the LoS channel model.
FIGURE 15. Spectral efficiency v.s. SNR for different HBF algorithms with
imperfect phase shifters.
the mmWave propagation channels and have robustness to the
channel model mismatch.
Finally, we consider the effect of imperfect phase shifters
in practical systems. As the two NNs are trained with
the assumption of perfect phase shifters, there is mismatch
between the perfect phase shifters in the offline training stage
and the imperfect ones in practical implementation in the
online deployment. Unlike the quantization error, random
phase and gain errors are neither known nor reciprocal [13].
Therefore, it is meaningful to explore the robustness of the
DL-based HBF NNs to imperfect phase shifters with random
phase and gain errors. According to [13], an imperfect
√ phase
shifter can be modeled as γ ej(φ−δ) , where j = −1, φ is the
ideal phase shift, δ denotes the phase error satisfying a Gaus-
 and variance σδ , i.e., δ ∼
sian distribution with zero 2 munication systems,’’ IEEE J. Sel. Areas Commun., vol. 37, no. 11,
 mean
N 0, σδ2 , and γ ∼ N 1, σγ2 denotes the non-ideal gain of


the phase shifter. According to [41] and [42], a typical σδ is
set to 0.1rad and a typical σγ is set to 0.2.
Fig. 15 shows the spectral efficiency of different HBF
algorithms with imperfect phase shifters, where the dotted
lines represent the performance with perfect phase shifters
and the solid ones represent the performance with imperfect
phase shifters, respectively. As shown in this figure, the spec-
tral efficiency loss of the proposed HBF-Net, CE-HBF-Net,
and the HBF algorithm [4] is less than 0.4bits/s/Hz. How-
ever, the spectral efficiency loss of the HBF algorithm [3] is
120938
2.1bits/s/Hz. These results show that the proposed HBF-Net
and CE-HBF-Net have good robustness to imperfect phase
shifters.
VI. CONCLUSION
We have enhanced the HBF optimization with imperfect CSI
from both the improvement of channel estimation and the
HBF optimization approach. In terms of codebook design,
we have applied MO to solve the HBF vector optimization
problem directly targeting the beam response requirement.
In terms of the HBF optimization, we proposed the HBF-Net
to solve the HBF optimization problem with DL and make
the HBF robust to channel estimation errors. We further
proposed the CE-HBF-Net to achieve the joint channel esti-
mation and HBF optimization. In particular, with the pro-
posed IAI method, the information of the selected codewords
in the hierarchical search process is efficiently input to the
CE-HBF-Net with much lower input dimensions. Simulation
results have shown that with a well-designed loss function
and the introduction of the attention mechanism, the HBF-
Net and CE-HBF-Net can well handle the challenges of hard-
ware constraints and channel estimation errors, and achieve
significant performance improvement over the conventional
algorithms.
REFERENCES
[1] O. E. Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and R. W. Heath, Jr.,
‘‘Spatially sparse precoding in millimeter wave MIMO systems,’’ IEEE
Trans. Wireless Commun., vol. 13, no. 3, pp. 1499–1513, Mar. 2014.
[2] X. Yu, J.-C. Shen, J. Zhang, and K. B. Letaief, ‘‘Alternating minimization
algorithms for hybrid precoding in millimeter wave MIMO systems,’’
IEEE J. Sel. Topics Signal Process., vol. 10, no. 3, pp. 485–500, Apr. 2016.
[3] F. Sohrabi and W. Yu, ‘‘Hybrid digital and analog beamforming design for
large-scale antenna arrays,’’ IEEE J. Sel. Topics Signal Process., vol. 10,
no. 3, pp. 501–513, Apr. 2016.
[4] T. Lin, J. Cong, Y. Zhu, J. Zhang, and K. Ben Letaief, ‘‘Hybrid beamform-
ing for millimeter wave systems using the MMSE criterion,’’ IEEE Trans.
Commun., vol. 67, no. 5, pp. 3693–3708, May 2019.
[5] Y. Zhou, T. Lin, and Y. Zhu, ‘‘Automatic modulation classification in
time-varying channels based on deep learning,’’ IEEE Access, vol. 8,
pp. 197508–197522, Oct. 2020.
[6] H. Ye, G. Y. Li, and B.-H. Juang, ‘‘Power of deep learning for channel esti-
mation and signal detection in OFDM systems,’’ IEEE Wireless Commun.
Lett., vol. 7, no. 1, pp. 114–117, Feb. 2018.
[7] H. He, C.-K. Wen, S. Jin, and G. Y. Li, ‘‘Deep learning-based channel esti-
mation for beamspace mmWave massive MIMO systems,’’ IEEE Wireless
Commun. Lett., vol. 7, no. 5, pp. 852–855, Oct. 2018.
[8] F. A. Aoudia and J. Hoydis, ‘‘Model-free training of end-to-end com-
pp. 2503–2516, Nov. 2019.
[9] A. M. Elbir, ‘‘DeepMUSIC: Multiple signal classification via deep learn-
ing,’’ IEEE Sensors Lett., vol. 4, no. 4, pp. 1–4, Apr. 2020.
[10] Y. Yang, F. Gao, X. Ma, and S. Zhang, ‘‘Deep learning-based channel
estimation for doubly selective fading channels,’’ IEEE Access, vol. 7,
pp. 36579–36589, Mar. 2019.
[11] E. Balevi, A. Doshi, and J. G. Andrews, ‘‘Massive MIMO channel esti-
mation with an untrained deep neural network,’’ IEEE Trans. Wireless
Commun., vol. 19, no. 3, pp. 2079–2090, Jan. 2020.
[12] A. Alkhateeb, S. Alex, P. Varkey, Y. Li, Q. Qu, and D. Tujkovic, ‘‘Deep
learning coordinated beamforming for highly-mobile millimeter wave sys-
tems,’’ IEEE Access, vol. 6, pp. 37328–37348, Jun. 2018.
[13] W. Wang, H. Yin, X. Chen, and W. Wang, ‘‘Robust and low-overhead
hybrid beamforming design with imperfect phase shifters in multi-user mil-
limeter wave systems,’’ IEEE Access, vol. 8, pp. 74002–74014, Apr. 2020.
VOLUME 9, 2021
Q. Lu et al.: Channel Estimation and Hybrid Precoding for mmWave Communications
[14] T. Lin and Y. Zhu, ‘‘Beamforming design for large-scale antenna arrays
using deep learning,’’ IEEE Wireless Commun. Lett., vol. 9, no. 1,
pp. 103–107, Jan. 2020.
[15] H. Huang, Y. Song, J. Yang, G. Gui, and F. Adachi, ‘‘Deep-learning-based
millimeter-wave massive MIMO for hybrid precoding,’’ IEEE Trans. Veh.
Technol., vol. 68, no. 3, pp. 3027–3032, Mar. 2019.
[16] W. Xia, G. Zheng, Y. Zhu, J. Zhang, J. Wang, and A. P. Petropulu, ‘‘A deep
learning framework for optimization of MISO downlink beamforming,’’
IEEE Trans. Commun., vol. 68, no. 3, pp. 1866–1880, Mar. 2020.
[17] C.-H. Lin, Y.-T. Lee, W.-H. Chung, S.-C. Lin, and T.-S. Lee, ‘‘Unsuper-
vised ResNet-inspired beamforming design using deep unfolding tech-
nique,’’ in Proc. IEEE Global Commun. Conf. (GLOBECOM), Dec. 2020,
pp. 1–7.
[18] M. Wenyan, Q. Chenhao, Z. Zhang, and J. Cheng, ‘‘Sparse channel estima-
tion and hybrid precoding using deep learning for millimeter wave massive
MIMO,’’ IEEE Trans. Commun., vol. 68, no. 5, pp. 2838–2849, Feb. 2020.
[19] S. Ramjee, S. Ju, D. Yang, X. Liu, A. El Gamal, and Y. C. Eldar,
‘‘Fast deep learning for automatic modulation classification,’’ 2019,
arXiv:1901.05850. [Online]. Available: http://arxiv.org/abs/1901.05850
[20] Y. Jin, J. Zhang, S. Jin, and B. Ai, ‘‘Channel estimation for cell-free
mmWave massive MIMO through deep learning,’’ IEEE Trans. Veh. Tech-
nol., vol. 68, no. 10, pp. 10325–10329, Nov. 2019.
[21] N. Song, C. Ye, X. Hu, and T. Yang, ‘‘Deep learning based low-rank chan-
nel recovery for hybrid beamforming in millimeter-wave massive MIMO,’’
in Proc. IEEE Wireless Commun. Netw. Conf. (WCNC), May 2020, pp. 1–6.
[22] Q. Bai, J. Wang, Y. Zhang, and J. Song, ‘‘Deep learning-based channel
estimation algorithm over time selective fading channels,’’ IEEE Trans.
Cognit. Commun. Netw., vol. 6, no. 1, pp. 125–134, Mar. 2020.
[23] J. Lee, G.-T. Gil, and Y. H. Lee, ‘‘Channel estimation via orthogonal
matching pursuit for hybrid MIMO systems in millimeter wave communi-
cations,’’ IEEE Trans. Commun., vol. 64, no. 6, pp. 2370–2386, Jun. 2016.
[24] C.-R. Tsai and A.-Y. Wu, ‘‘Structured random compressed channel sensing
for millimeter-wave large-scale antenna systems,’’ IEEE Trans. Signal
Process., vol. 66, no. 19, pp. 5096–5110, Oct. 2018.
[25] C.-R. Tsai, Y.-H. Liu, and A.-Y. Wu, ‘‘Efficient compressive channel
estimation for millimeter-wave large-scale antenna systems,’’ IEEE Trans.
Signal Process., vol. 66, no. 9, pp. 2414–2428, May 2018.
[26] A. Alkhateeb, O. El Ayach, G. Leus, and R. W. Heath, Jr., ‘‘Channel
estimation and hybrid precoding for millimeter wave cellular systems,’’
IEEE J. Sel. Topics Signal Process., vol. 8, no. 5, pp. 831–846, Oct. 2014.
[27] C. Liu, M. Li, S. V. Hanly, I. B. Collings, and P. Whiting, ‘‘Millimeter wave
beam alignment: Large deviations analysis and design insights,’’ IEEE
J. Sel. Areas Commun., vol. 35, no. 7, pp. 1619–1631, Jul. 2017.
[28] Z. Xiao, H. Dong, L. Bai, P. Xia, and X.-G. Xia, ‘‘Enhanced channel
estimation and codebook design for millimeter-wave communication,’’
IEEE Trans. Veh. Technol., vol. 67, no. 10, pp. 9393–9405, Oct. 2018.
[29] C. Chen, Y. Dong, X. Cheng, and L. Yang, ‘‘Low-resolution PSs based
hybrid precoding for multiuser communication systems,’’ IEEE Trans. Veh.
Technol., vol. 67, no. 7, pp. 6037–6047, Jul. 2018.
[30] K. Chen and C. Qi, ‘‘Beam training based on dynamic hierarchical code-
book for millimeter wave massive MIMO,’’ IEEE Commun. Lett., vol. 23,
no. 1, pp. 132–135, Jan. 2019.
[31] T. O’Shea and J. Hoydis, ‘‘An introduction to deep learning for the physical
layer,’’ IEEE Trans. Cogn. Commun. Netw., vol. 3, no. 4, pp. 563–575,
Dec. 2017.
[32] T. J. O’Shea, T. Erpek, and T. Charles Clancy, ‘‘Deep learning based
MIMO communications,’’ 2017, arXiv:1707.07980. [Online]. Available:
http://arxiv.org/abs/1707.07980
[33] D. Wu, M. Nekovee, and Y. Wang, ‘‘Deep learning-based autoencoder for
m-User wireless interference channel physical layer design,’’ IEEE Access,
vol. 8, pp. 174679–174691, Sep. 2020.
[34] S. Noh, M. D. Zoltowski, and D. J. Love, ‘‘Multi-resolution code-
book and adaptive beamforming sequence design for millimeter wave
beam alignment,’’ IEEE Trans. Wireless Commun., vol. 16, no. 9,
pp. 5689–5701, Sep. 2017.
[35] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge,
MA, USA: MIT Press, 2016.
[36] X. Cheng, X. Li, J. Yang, and Y. Tai, ‘‘SESR: Single image super resolution
with recursive squeeze and excitation networks,’’ in Proc. 24th Int. Conf.
Pattern Recognit. (ICPR), Aug. 2018, pp. 147–152.
VOLUME 9, 2021
[37] D. Hu, Y. Zhang, L. He, and J. Wu, ‘‘Low-complexity deep-learning-based
DOA estimation for hybrid massive MIMO systems with uniform circular
arrays,’’ IEEE Wireless Commun. Lett., vol. 9, no. 1, pp. 83–86, Jan. 2020.
[38] C.-K. Wen, W.-T. Shih, and S. Jin, ‘‘Deep learning for massive MIMO
CSI feedback,’’ IEEE Wireless Commun. Lett., vol. 7, no. 5, pp. 748–751,
Oct. 2018.
[39] S. Dörner, S. Cammerer, J. Hoydis, and S. ten Brink, ‘‘Deep learning based
communication over the air,’’ IEEE J. Sel. Topics Signal Process., vol. 12,
no. 1, pp. 132–143, Feb. 2018.
[40] K. He and J. Sun, ‘‘Convolutional neural networks at constrained time
cost,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., May 2015,
pp. 5353–5360.
[41] R. Garg and A. S. Natarajan, ‘‘A 28-GHz low-power phased-array receiver
front-end with 360◦ RTPS phase shift range,’’ IEEE Trans. Microw. Theory
Techn., vol. 65, no. 11, pp. 4703–4714, Nov. 2017.
[42] C. W. Byeon and C. S. Park, ‘‘A low-loss compact 60-GHz phase shifter
in 65-nm CMOS,’’ IEEE Microw. Wireless Compon. Lett., vol. 27, no. 7,
pp. 663–665, Jul. 2017.
QIUJIN LU received the B.Eng. degree (Hons.)
in communication science and engineering from
Fudan University, in 2018, where she is cur-
rently pursuing the M.S. degree in communica-
tion science and engineering. Her current research
interests include channel estimation, hybrid beam-
forming design, deep learning, and millimeter
wave signal processing.
TIAN LIN (Graduate Student Member, IEEE)
received the B.Eng. degree (Hons.) in commu-
nication science and engineering from Fudan
University, in 2017, where he is currently pursu-
ing the Ph.D. degree. His current research inter-
ests include hybrid beamforming for massive
MIMO systems, millimeter wave signal process-
ing, passive beamforming and channel estima-
tion for intelligent reflecting surface-aided MIMO
systems, and deep learning for physical layer
communication.
YU ZHU (Member, IEEE) received the B.Eng.
degree (Hons.) in electronics engineering and the
M.Eng. degree (Hons.) in communication and
information engineering from the University of
Science and Technology of China, in 1999 and
2002, respectively, and the Ph.D. degree from the
Department of Electronic and Computer Engineer-
ing, The Hong Kong University of Science and
Technology, in 2007.
Since 2008, he has been with Fudan University,
where he is currently a Professor with the School of Information Science and
Technology. His current research interests include broadband wireless sys-
tems and networks and signal processing for communications. He received
Shanghai Pujiang Scholar Award, in 2008, and Fudan Zhuoxue Award,
in 2012. He served as the Physical Track Co-Chair for the IEEE WCNC
2013, the Co-Chair for the Wireless Communications Systems Symposium
of the IEEE ICCC 2014 and IEEE ICCC 2019, and the Workshop Co-Chair
for the Asia–Pacific Conference on Communications, in 2017. He served as
an Editor for the IEEE Wireless Communications Letters and the Journal of
Communications and Information Networks.
120939