IEEE SIGNAL PROCESSING LETTERS, VOL.
27, 2020
Accurate and Reliable Facial Expression Recognition
Using Advanced Softmax Loss With Fixed Weights
Ping Jiang , Gang Liu , Quan Wang , and Jiang Wu
Abstract—An important challenge for facial expression recogni-
tion (FER) is that real-world training data are usually imbalanced.
Although many deep learning approaches have been proposed to
enhance the discriminative power of deep expression features and
enable a good predictive effect, few works have focused on the mul-
ticlass imbalance problem. When supervised by softmax loss (SL),
which is widely used in FER, the classifier is often biased against
minority categories (i.e., smaller interclass angular distances). In
this letter, we present advanced softmax loss (ASL) to mitigate the
bias induced by data imbalance and hence increase accuracy and
reliability. The proposed ASL essentially magnifies the interclass
diversity in the angular space to enhance discriminative power
in every category. The proposed loss can easily be implemented
in any deep network. Extensive experiments on the FER2013 and
real-world affective faces (RAF) databases demonstrate that ASL is
significantly more accurate and reliable than many state-of-the-art
approaches and that it can easily be plugged into other methods
and improves their performance.
Index Terms—Deep learning, convolutional neural networks,
softmax loss, multiclass imbalance, facial expression recognition.
I. INTRODUCTION
ACIAL expression recognition (FER) is a popular topic in
F computer vision and machine learning because of its vast
array of potential applications in human-computer interfaces [1],
[2]. In recent years, the focus of FER has transitioned from
controlled laboratory environments to real-world conditions due
to the success of deep learning techniques [3]–[7]. FER in
realistic settings is still a challenging problem, since real-world
data contain additional factors that are unrelated to expressions,
such as head pose, illumination, and gender.
In the FER literature, the most widely used loss function is
softmax loss (SL) [8]–[12]. Under the supervision of SL, deeply
Manuscript received January 2, 2020; revised April 4, 2020; accepted April
11, 2020. Date of publication April 22, 2020; date of current version May
21, 2020. This work was supported in part by the National Natural Science
Foundation of China (NSFC) under Grant 61702395, Grant 61972302, and Grant
61711530248, in part by the Key Program of Shaanxi Technology Committee
of China under Grant 2019NY-182 and Grant 2020NY-167, and in part by the
High-Level Culture Program of Yulin College under Grant 207010074. The
associate editor coordinating the review of this manuscript and approving it for
publication was Prof. Mylene Q. Farias. (Corresponding author: Gang Liu.)
Ping Jiang is with the School of Computer Science and Technology, Xidian
University, Xi’an 710071, China, and also with the School of Information
Engineering, Yulin University, Yulin 719000, China (e-mail: jiangping@yulinu.
edu.cn).
Gang Liu and Quan Wang are with the School of Computer Science and Tech-
nology, Xidian University, Xi’an 710071, China (e-mail: gliu_xd@163.com;
qwang@xidian.eud.cn).
Jiang Wu is with the School of Information Engineering, Yulin University,
Yulin 719000, China (e-mail: wujiang@yulinu.eud.cn).
Digital Object Identifier 10.1109/LSP.2020.2989670
1070-9908 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:25:00 UTC from IEEE Xplore. Restrictions apply.
725
learned features are likely to be separable in the angular space but
not sufficiently discriminative. Hence, many deep embedding
approaches have been presented to enhance the discriminative
power of deep features by reducing intraclass variation and
enhancing interclass differences. Wen et al. [13], [14] proposed
classical center loss (CL) to force deep features of the same class
to their centers. Li et al. [15], [16] reduced intraclass variation
by minimizing the distance between a sample and the center
of its neighbors. Based on the CL and the locality-preserving
loss, Luo et al. [17] designed local subclass loss to constrain
intraclass variation. On the basis of SL and CL, Cai et al. [18]
added island loss (IL) to simultaneously increase the interclass
angular distance. The exclusive regularization in [19] is actually
a special case of IL. However, almost no methods have addressed
the problem that real-world expression datasets are usually
imbalanced [20].
Under the supervision of SL, the angular space of minority
classes is often compressed and thus leads to poor discrimination
results. That is, minority categories usually have less interclass
diversity. Numerous equidistant prototype embeddings [21]–
[23] have been used to constrain the target vectors to be equidis-
tant in Euclidean space, but could not guarantee that they were
uniformly distributed in the angular domain. Therefore, this
work proposes advanced softmax loss (ASL) to mitigate the
bias against minority categories by using fixed (unlearnable)
weights in which all weight vectors are uniformly distributed in
the angular space (where a weight vector corresponds to a class).
Our major contributions are as follows:
- An analysis of interclass diversity illustrates that it is ideal
for all classes to be uniformly distributed in the angular
domain.
- A novel method ensures that the deeply learned expression
features are uniformly distributed and hence enhances the
discriminative ability for minority classes.
- Almost all works have reported the best scores, but these
results may be incorrect because the recognition results
vary across implementations (e.g., in our experiments, the
difference between the best score and the worst is more than
2% for some methods). Thus, we present a new performance
metric (reliability) to measure the variation in classification
scores and employ a fairer metric (the mean score) to
evaluate the conventional recognition accuracy.
- Exhaustive experiments on the FER2013 dataset [24] and
real-world affective faces (RAF) database [15] demonstrate
that the proposed method is more accurate and reliable
than many state-of-the-art approaches and that it can easily
 726 IEEE SIGNAL PROCESSING LETTERS, VOL. 27, 2020

Algorithm 1: Computing Weights for ASL.

Input: Rate α
Output: Weights wk (k ∈ N )
1: Randomly initialize wk with a normal distribution
1
2: while ( max {|wTk wl − |} < 10−4 ) do
k,l∈N 1−n
k=l
3: Compute the gradients Δwk as in Eq. (5)
4: Update the weights:
5: wk ← wk − αΔwk
6: end while
Fig. 1. Distributions of 2D deep features trained on a part of the MNIST 7: Normalize wk to wk  = 1
dataset under the supervision of SL and ASL.

be incorporated into other methods (such as loss func- Algorithm 2: Training Strategy for ASL.
tions [25]–[28] that are extensions of SL). Input: Training data {Ii , yi }m
i=1 , learning rate η, number of
iterations T
II. PROPOSED METHOD Output: Network layer parameters θ
1: Randomly initialize network layer parameters θ 1 ,
A. Softmax Loss
generate ASL parameters wk according to Algorithm 1
SL is widely used in FER works and is defined as follows: 2: for t = 1 to T do
3: Compute the loss LtS as in Eq. (1):
1 
T
ewyi xi
m
∂Lt
LS = − log  wT
, (1) 4: Compute the back-propagation error: ∂xSt
m i=1 k xi
k∈N e
i
5: Update the network layer parameters:
 t
∂LS ∂xit
where N denotes the set of sample labels and (·)T denotes 6: θ t+1 = θ t − η m i=1 ∂xti ∂θ t
the transpose operator. xi ∈ Rd is the deep feature of the ith 7: end for
sample, and yi ∈ N is the corresponding class label. wk ∈ Rd
denotes the kth column (vector) of the weight parameters W =
[w1 , w2 , . . . , wn ] ∈ Rd×n in the SL. n and m are the numbers Based on the concept of the polytope [31], [32], we know that
of classes and training samples, respectively. ŵTk ŵl = 1/(1 − n) is a solution of Eq. (3), i.e., these points are
It is well known that the deeply learned features supervised by the vertexes of a regular (n − 1)-simplex (note that d ≥ n − 1
SL exhibit a ‘radial’ distribution and are likely to be separable in must hold, where d is the feature dimension). That is, to enhance
the angular domain. For example, Fig. 1 shows the 2D features the discriminative power between different classes, it is ideal that
trained on the training set of MNIST [29] (there are only 1,000 the weight vectors wk are uniformly distributed in the angular
samples for digits 2 and 5, respectively) using LeNet++ networks space (i.e., every Akl equals 1/(1 − n) radians). Hence, the bias
as in [13], [14], [30]. However, SL usually compresses the due to the imbalanced training data is mitigated by forcing wk
angular space of minority classes (see digits 2 and 5 in Fig. 1(a)) to be uniformly distributed in the angular domain.
and thus leads to poor generalization to the test samples.
In well-trained convolutional neural networks (CNNs), the B. Advanced Softmax Loss
weight vector wk can be regarded as the cluster center of all xi As mentioned above, if every Akl is equal to arccos(1/(1 −
with yi = k [19]. To augment the discriminative power of the n)) (where arccos denotes the inverse of the cosine function),
CNN, different classes should be as widely separated as possible; this is an optimal scenario for enhancing discriminative power;
therefore, we examine the loss function below, which is similar thus, we aim to ensure that the learned features are uniformly
to IL [18]. distributed in the angular domain.
  wT wl  Note that ŵTk ŵl = 1/(1 − n) is one solution of Eq. (3) but not
Linter = k
= ŵTk ŵl , (2)
wk  wl  the only one. For example, the vectors (1;0;0), (0;1;0), (-1;0;0),
k∈N l∈N k∈N l∈N
l=k l=k and (0;-1;0) satisfy Eq. (3) but are not the desired solution. Thus,
we cannot achieve the ideal case by optimizing the loss Linter
where ŵk and ŵl are normalized weight vectors and  · 
or by solving Eq. (3). This is why IL is not very efficient (see
denotes the Euclidean norm. ŵk T ŵl is the cosine value of Akl ,
Tables I-II).
which denotes the vectorial angle between wk and wl , and the
For convenience, we suppose that wk is a unit vector (i.e.,
loss Linter is the sum of the cosine values of all pairwise angles.
wk  = 1), and then we have cos(Akl ) = wTk wl . Since Akl ∈
Therefore, Linter magnifies the interclass angular distances.
[0, π], our final goal Akl = 1/(1 − n) is the unique solution of
Linter takes its minimal value −n only when the following
the equation J = 0, where J is defined as follows:
equation holds:
 1    1
2
ŵk = − ŵl . (3) J = wk wl −
T
. (4)
2n(n − 1) 1−n
l∈N ,l=k k∈N l∈N ,l=k

Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:25:00 UTC from IEEE Xplore. Restrictions apply.
 JIANG et al.: ACCURATE AND RELIABLE FACIAL EXPRESSION RECOGNITION USING ADVANCED SOFTMAX LOSS WITH FIXED WEIGHTS
TABLE I
PERFORMANCE COMPARISON ON THE FER2013 DATABASE. THE BEST
RESULTS ARE MARKED IN BOLDFACE
TABLE II
PERFORMANCE COMPARISON ON THE RAF DATASET. THE BEST RESULTS ARE
MARKED IN BOLDFACE
The gradient of J with respect to wk is:
   are trained for 100 epochs, and the learning rate is initialized to
1 1
Δwk = wTk wl − wl .
n(n − 1) 1−n
l∈N ,l=k
The desired wk is independent of the deep features xi ; there-
fore, the ideal wk is not necessarily updated in a mini-batch as
in conventional SL. In the proposed ASL, the weight vector wk
is first randomly initialized with a normal distribution and then
optimized by using the gradient descent method. The details for
computing wk are shown in Algorithm 1, and the rate α is set
as 0.1 in this letter.
Algorithm 2 illustrates the training process when the deep
networks are under the supervision of ASL. Compared with
conventional SL, the fundamental difference is that ASL sets
the weights wk according to Algorithm 1 when the networks
are initialized, and they are unchanged in the training phase.
Fig. 1(b) shows an example of ASL with the same settings
as in Section II-A, where one can clearly see that all classes
are nearly uniformly distributed in the angular space. The ASL
enlarges the interclass angular distance of minority categories
(digits 2 and 5) and hence improves the recognition performance.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:25:00 UTC from IEEE Xplore. Restrictions apply.
727
III. EXPERIMENTS
A. Implementation Details
Our experiments are implemented by using Python with the
PyTorch framework on a workstation computer with the follow-
ing specifications: Intel Xeon Silver 4110 2.10 GHz CPU, 32 GB
RAM and an NVIDIA Titan RTX 24 G GPU.
1) Tested Databases: A series of experiments were con-
ducted on two benchmark expression databases: the FER2013
database [24] and the RAF dataset [15]. The FER2013 database
contains 28,709, 3,589 and 3,589 images for training, private
testing and public testing, respectively, and these training images
include 3,995 angry expressions, 436 disgusted expressions,
4,097 fearful expressions, 7,215 happy expressions, 4,830 sad
expressions, 3,171 surprised expressions, and 4,965 neutral ex-
pressions. The RAF dataset provides 12,271 images for training
and 3,068 images for testing, where the training set consists of
705 angry images, 717 disgusted images, 281 fearful images,
4,772 happy images, 1,982 sad images, 1,290 surprised images,
and 2,524 neutral images.
2) Image Preprocessing: A 44 × 44 image is randomly
cropped from the input 48 × 48 grayscale image and then nor-
malized by dividing all pixels by 255. The cropped image is
randomly horizontally flipped with a 50% probability to aug-
ment the data, and it is ultimately expanded to 3 × 44 × 44 by
being copied.
3) Training & Testing: For a fair comparison, this letter em-
ploys the same fundamental network architecture over a series of
experiments; the architecture is an extension of ResNet18 [33].
The output of the final dropout layer is the deeply learned
expression feature xi .
All models are trained in an end-to-end fashion by using the
stochastic gradient descent (SGD) method with a mini-batch
of 64 samples and a momentum coefficient of 0.9. All weights
are initialized by using a variance scaling initializer (He initial-
izer) [34], and the L2 weight decay is 5 × 10−4 . All of the models
0.01 and decays by a γ of 0.9 every 5 epochs after 40 epochs.
(5)
This letter utilizes the ten-crop method to recognize the testing
images. A testing image is cropped to ten 44 × 44 images, and
then it is classified into the class with the highest average score
with respect to these ten cropped images.
B. Performance Evaluation
There are two popular criteria for evaluating the performance
of classifiers: the overall accuracy rate (OA) and the average
class accuracy (ACA), which are defined as follows:

zc
OA =  c∈N × 100%, (6)
n
c∈N c
1  zc
ACA = × 100%, (7)
|N | nc
c∈N
where zc represents the number of samples that belong to the cth
class and are correctly classified, nc is the number of samples in
the cth class, and |N | denotes the number of categories. OA is the
most widely used criterion, and ACA can index the precision of
 728 IEEE SIGNAL PROCESSING LETTERS, VOL. 27, 2020

each category. For a good classifier, both OA and ACA should TABLE III
THE RESULTS OF JOINT SUPERVISION FOR THE FER2013 DATASET. THE BEST
be as large as possible. RESULTS ARE MARKED IN BOLDFACE
In our experiments, every model was repeatedly tested N
(N = 10) times, and there are thus N results. Therefore, we
measure the recognition performance through the following two
statistical values of each evaluation criterion EC (i.e., OA or
ACA):

1 
N
M ean = ECi , (8)
N i=1


1  N
ST D =  (ECi − M ean)2 , (9)
N i=1

where ECi represents the ith value of EC. M ean indicates

the accuracy; i.e., the classifier is more accurate if M ean is
greater. The standard deviation (ST D) indicates the reliability TABLE IV
(variation) of the recognition results; i.e., the recognizer is more THE RESULTS OF JOINT SUPERVISION FOR THE RAF DATABASE. THE BEST
reliable when ST D is smaller. RESULTS ARE MARKED IN BOLDFACE

C. Experimental Results and Analysis

1) Performance Comparison: In this experiment, we exam-
ine the performance of our ASL function. The basic networks
were supervised by different loss functions (i.e., SL, IL [18],
CL [14], L2SL [25], and ASL), and the results with respect to
the FER2013 and RAF databases are shown in Tables I and II.
On the FER2013 dataset, it can clearly be observed that
our ASL outperforms all of the other methods with respect
to accuracy (i.e., greatest M ean) and reliability (i.e., smallest
ST D) for both ACA and OA.
On the RAF database, ASL achieves the best OA and the
greater ACA (see M ean in Table II). Although IL has the highest
reliability, its recognition accuracy is definitively the lowest.
This experiment illustrates that the proposed ASL is more
accurate and reliable than many state-of-the-art methods.
2) Joint Supervision: As is the case for conventional SL, the
proposed ASL can be jointly applied with other loss functions.
In this experiment, we tested two joint signals (i.e., ASL+CL
and ASL+L2SL) on the two databases, and the results are
shown in Tables III and IV. One can clearly see that the joint
signals ASL+CL and ASL+L2SL are superior to CL and L2SL,
respectively, in every case. Although ASL+CL and ASL+L2SL
produce a trivial enhancement of accuracy, their outperformance
with respect to reliability is remarkable.
This experiment demonstrates that our proposed ASL can
cooperate with many state-of-the-art methods and improve their
performance.
3) Discussion: The proposed method has the following ad-
vantages:
First, the deeply learned expression features supervised by
our ASL have greater and more balanced (unbiased) interclass
angular distances, and hence, it performs better on both majority
classes and minority categories. Consequently, the proposed
ASL is superior to many state-of-the-art approaches.
Second, since the proposed method constrains only the weight
vectors, it can be easily plugged into other methods and improves
their capability.
Finally, the proposed ASL is easy to implement and has
lower computational complexity because it is very simple. For
example, the time to train one epoch on the FER2013 database
for ASL, SL, IL, CL, and L2SL is 40.02 s, 40.75 s, 49.22 s,
41.73 s, and 43.83 s, respectively. Although the ASL needs
approximately 30 s to initialize network parameters, it has lower
computational complexity overall.
IV. CONCLUSION
In facial expression recognition tasks, the expression data are
usually imbalanced; however, few FER approaches effectively
cope with this problem. Hence, this letter presents a novel loss
function, called advanced softmax loss, to mitigate the bias
induced by imbalanced training expression data. The proposed
loss ensures that every class has an equivalent classification do-
main and ability by using fixed (unlearnable) weight parameters
that have the same magnitude and are uniformly distributed in
the angular space. The experimental results on two benchmark
real-world expression databases demonstrate that our proposed
ASL is more accurate and reliable than many state-of-the-art
FER works. The proposed loss can be individually employed as
the supervision signal and can also be used to jointly supervise
deep networks with other loss functions.
The proposed ASL can be used to many classification prob-
lems in theory; however, it cannot efficiently tackle with a large
number of categories (e.g., face recognition) because the number
of pairwise angles dramatically increases with the growth of
classes. Therefore, how to address this problem is our future
work.

Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:25:00 UTC from IEEE Xplore. Restrictions apply.
 JIANG et al.: ACCURATE AND RELIABLE FACIAL EXPRESSION RECOGNITION USING ADVANCED SOFTMAX LOSS WITH FIXED WEIGHTS
REFERENCES
[1] R. K. Gupta and S. D. Senturia, “Real time face detection and facial
expression recognition: Development and applications to human computer
interaction,” in Proc. Conf. Comput. Vis. Pattern Recognit. Workshop,
2003.
[2] J. Deng, C. Pang, Z. Zhang, Z. Pang, H. Yang, and G. Yang, “CGAN based
facial expression recognition for human-robot interaction,” IEEE Access,
vol. 7, pp. 2169–3536, 2019.
[3] G. E. Hinton, A Practical Guide to Training Restricted Boltzmann Ma-
chines. Berlin, Germany: Springer, 2012, pp. 599–619.
[4] A. Majumder, L. Behera, S. Member, and V. K. Subramanian, “Automatic
facial expression recognition system using deep network-based data fu-
sion,” IEEE Trans. Cybern., vol. 48, no. 1, pp. 103–114, Jan. 2018.
[5] Y.-H. Lai and S.-H. Lai, “Emotion-preserving representation learning via
generative adversarial network for multi-view facial expression recogni-
tion,” in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit. Workshops,
2018, pp. 263–270.
[6] H. Yang, Z. Zhang, and L. Yin, “Identity-adaptive facial expression
recognition through expression regeneration using conditional generative
adversarial networks,” in Proc. IEEE Int. Conf. Autom. Face Gesture
Recognit. Workshops, 2018, pp. 294–301.
[7] F. Zhang, T. Zhang, Q. Mao, and C. Xu, “Joint pose and expression
modeling for facial expression recognition,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit., 2018, pp. 3359–3368.
[8] S. Li and W. Deng, “Deep facial expression recognition: A survey,”
Oct. 2018, arXiv:1804.08348v2.
[9] B. Fasel, “Robust face analysis using convolutional neural networks,” Ob-
ject Recognit. Supported User Interact. Service Robots, vol. 2, Aug. 2002,
pp. 40–43.
[10] B. Sun, L. Li, G. Zhou, and J. He, “Facial expression recognition in the
wild based on multimodal texture features,” J. Elect. Imag., vol. 25, no. 6,
2016, Art. no. 061407.
[11] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies
for accurate object detection and semantic segmentation,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580–587.
[12] K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties
for large-scale facial expression recognition,” 2020, arXiv:2002.10392v2.
[13] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning
approach for deep face recognition,” in Proc. Eur. Conf. Comput. Vis.,
2016, pp. 499–515.
[14] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A comprehensive study on center
loss for deep face recognition,” Int. J. Comput. Vision, vol. 127, pp. 668–
683, Jun. 2019.
[15] S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality-
preserving learning for expression recognition in the wild,” in Proc. Conf.
Comput. Vision Pattern Recognit., 2017, pp. 2584–2593.
[16] S. Li and W. Deng, “Reliable crowdsourcing and deep locality-preserving
learning for unconstrained facial expression recognition,” IEEE Trans.
Image Process., vol. 28, no. 1, pp. 356–370, Jan. 2019.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:25:00 UTC from IEEE Xplore. Restrictions apply.
729
[17] Z. Luo, J. Hu, and W. Deng, “Local subclass constraint for facial expression
recognition in the wild,” in Proc. 24th Int. Conf. Pattern Recognit., 2018,
pp. 3132–3137.
[18] J. Cai, Z. Meng, A. S. Khan, Z. Li, J. O. Reilly, and Y. Tong, “Island loss
for learning discriminative features in facial expression recognition,” in
Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2018, pp. 302–309.
[19] K. Zhao, J. Xu, and M.-M. Cheng, “RegularFace: Deep face recognition
via exclusive regularization,” in Proc. IEEE Conf. Comput. Vision Pattern
Recognit., 2019, pp. 1136–1144.
[20] S. Li and W. Deng, “A deeper look at facial expression dataset bias,”
Apr. 2019, arXiv:1904.11150v1.
[21] W. Deng, Y. Liu, J. Hu, and J. Guo, “The small sample size problem
of ICA: A comparative study and analysis,” Pattern Recognit., vol. 45,
pp. 4438–4450, 2012.
[22] W. Deng, J. Hu, X. Zhou, and J. Guo, “Equidistant prototypes em-
bedding for single sample based face recognition with generic learning
and incremental learning,” Pattern Recognit., vol. 47, pp. 3738–3749,
2014.
[23] M. Hayat, S. Khan, W. Zamir, J. Shen, and L. Shao, “Max-margin class im-
balanced learning with Gaussian affinity,” Jan. 2019, arXiv:1901.07711v1.
[24] J. Ian et al., “Challenges in representation learning: A report on three
machine learning contests,” Neural Netw., vol. 64, pp. 59–63, Apr. 2015.
[25] F. Wang, X. Xiang, J. Cheng, and A. L. Yuille, “Normface: L2 hypersphere
embedding for face verification,” in Proc. ACM Int. Conf. MultiMed., 2017,
pp. 1041–1049.
[26] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hy-
persphere embedding for face recognition,” in Proc. IEEE Conf. Comput.
Vision Pattern Recognit., 2017, pp. 6738–6746.
[27] H. Wang et al., “CosFace: Large margin cosine loss for deep face recog-
nition,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2018,
pp. 5265–5274.
[28] J. deng, J. Guo, and N. Xue, “ArcFace: Additive angular margin loss
for deep face recognition,” in Proc. IEEE Conf. Comput. Vision Pattern
Recognit., 2019, pp. 4685–4694.
[29] Y. Lecun and C. Cortes, “The MNIST database of handwritten digits,”
Oct. 2019.
[30] R. Gao, F. Yang, W. Yang, and Q. Liao, “Margin loss: Making faces
more separable,” IEEE Signal Process. Lett., vol. 25, no. 2, pp. 308–312,
Feb. 2018.
[31] W. Rudin, Principles of Mathematical Analysis, vol. 3. New York, NY,
USA: McGraw-Hill, 1964, ch. 10.
[32] M. A. EI-Gebeily and Y. A. Fiagbedzi, “On certain properties of the regular
n-simplex,” Int. J. Math. Educ. Sci. Technol., vol. 35, no. 4, pp. 617–629,
2004.
[33] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2016,
pp. 770–778.
[34] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification,” in Proc.
IEEE Int. Conf. Comput. Vision, 2015, pp. 1026–1034.