2018 IEEE Asia Pacific Conference on Circuits and Systems
A High-Speed and Low-Complexity Architecture
for Softmax Function in Deep Learning
Meiqi Wang, Siyuan Lu, Danyang Zhu,
Jun Lin, Member, IEEE, and Zhongfeng Wang, Fellow, IEEE
School of Electronic Science and Engineering, Nanjing University, Nanjing, China
Email: mqwang99@gmail.com, sylu@smail.nju.edu.cn, zhudanyang10@foxmail.com, {jlin, zfwang}@nju.edu.cn
Abstract—Recently, significant improvement has been achieved
for hardware architecture design of deep neural networks
(DNNs). However, the hardware implementation of one widely
used softmax function in DNNs has not been much investigated,
which involves expensive division and exponentiation units. This
paper performs an efficient hardware implementation of softmax
function. Mathematical transformations and linear fitting are
used to simplify this function. Multiple algorithmic strength
reduction strategies and fast addition methods are employed to
optimize the architecture. By using these techniques, complicated
logic units like multipliers are eliminated and the memory
consumption is largely reduced while the accuracy loss is negli-
gible. The proposed design is coded using hardware description
language (HDL) and synthesized under the TSMC 28-nm CMOS
technology. Synthesis results show that the architecture achieves a
throughput of 6.976 G/s for 8-bit input data. The power efficiency
of 463.04 Gb/(mm2 · mW) is achieved and it costs only 0.015mm2
area resources. To the best of our knowledge, this is the first
work on efficient hardware implementation for softmax in open
literature.
Index Terms—softmax, hardware implementation, efficient
hardware architecture, deep neural network, VLSI
I. I NTRODUCTION
Deep neural network (DNN) has achieved great success
in big data area since its resurgence in 2006 [1]. Efficient
hardware architectures for DNNs is one of the hot topics in
both academic and industrial societies [2]- [6]. The softmax
layer is widely used in different DNNs [7]- [9]. However,
most previous works focus on matrix multiplications opti-
mization in DNNs and efficient hardware implementation for
softmax function has not been much investigated. Efficient
hardware architectures of softmax layer are desired for high
speed deep learning systems. The exponentiation and division
calculations in softmax function are quite expensive, especially
in embedded systems. A hardware architecture for softmax
has been proposed in [10]. However, no detailed discussion
about implementation of exponential and the logarithmic units,
which have very high complexity, was presented in [10]. In
this paper, we perform an efficient hardware implementation
of softmax function. Our main contributions are as follows:
• Mathematical transformations are used to transform the
exponential and logarithmic (exp-log) functions into exp-
log operations with limited input range, as well as some
other simple operations including constant multiplication,
shift and addition. Linear fitting is used to calculate
978-1-5386-8240-1/18/$31.00 ©2018 IEEE
the exp-log function within a specific range. After the
optimization, the main calculations in the exponential
unit require only a shift unit, two adders and a constant
multiplier. Calculations in the logarithmic unit require
only a shift unit, a leading one detector (LOD) unit and
three adders.
• The constant multiplier is optimized using the algorithmic
strength reduction strategy and it is replaced by multiple
adders. Adders are further optimized using carry-save
method [11]- [12].
The rest of this paper is organized as follows. Section II
gives a brief review of softmax and the domain transformation
of this function. Section III and Section IV propose designs
for the exponential unit and the logarithmic unit, respectively.
The proposed architecture of the softmax function is evaluated
in Section V. Section VI concludes this paper.
II. BACKGROUND
The softmax function can be expressed as follows:
exi
f (xi ) = PN (i = 1, 2, ..., N ), (1)
j=1 exj
where x1 , x2 , ..., xN are the input values of the softmax layer
and the output value f (xi ) represents the probability that the
sample belongs to the i-th category.
Mathematical transformation of softmax involving expo-
nential and logarithmic operations is adopted to perform the
hardware implementation, which is shown in (2).
exi
f (xi ) = exp(ln( PN ))
xj
j=1 e
XN (2)
= exp(xi − ln( xj
e ))(i = 1, 2, ..., N ).
j=1
III. T HE A RCHITECTURE OF THE E XPONENTIAL U NIT
The overall architecture for the softmax function are shown
in Fig .1. It consists of multiple exponential units(EXPU),
one adder tree, one natural logarithmic unit(LNU) and some
other simple units. The most expensive calculations are in the
EXPU and the LNU. In Section III and Section IV, efficient
architectures for the two units are introduced.
223

Fig. 1. The overall architecture for the softmax function.
A. Mathematical Transformation of Exponential Calculation
Exponential operations are composed of a number of mul-
tiplications. Multiple multiplications will consume plenty of
resources and cause long datapath delays in hardware imple-
mentation. Also, if the range of input cannot be limited to a
specific range, the value of the exponent will be difficult to
calculate. Considering these difficulties, mathematical trans-
formation is applied to the exponential function.
First, we use the base number 2 to take place of the base
number e and the exponential function will be changed to:
eyi = 2yi ·log2 e
Then the value yi · log2 e is split into an integer number ui
and a decimal number vi . Note that ui + vi = yi · log2 e, and
ui = byi · log2 ec. In the hardware implementation, the value
yi · log2 e is a fixed point number, which makes it easy to split
the integer and the decimal. The calculation can be simplified
as follows:
2vi << ui

yi ui +vi ui vi ui > 0
e =2 =2 ·2 = vi
2 >> (−ui ) ui ≤ 0
(4)
Through this transformation, we replace the original oper-
ation ”f (yi ) = eyi , −∞ < yi < +∞” with the operation
”f (vi ) = 2vi , 0 < vi ≤ 1” and a simple shift operation. The
operation ”f (vi ) = 2vi , 0 < vi ≤ 1” can be implemented with
a lookup table or a linear fitting function, which can greatly
reduce the hardware complexity.
B. Proposed Design for the EXPU
The hardware architecture of the EXPU is shown in Fig.
2. It consists of two adders, a constant multiplier, and a shift
operation. This hardware architecture is designed to support
both exponential operations in (2), and the two operations are
slightly different. In the second exponential calculation, lnF
(the output of the LNU in Fig. 1 ) needs to be subtracted from
the input value. Moreover, the results of the first exponential
calculation are accumulated and stored, and those of the
second one are taken as the final outputs.
Fig. 2. The proposed design for the EXP Unit.
The range of vi in f (vi ) = 2vi is limited to (0, 1], so the
function f (vi ) = 2vi can be approximated as functions f (x) =
x + d1 or f (x) = x + d2 within this specific range. The
two bias values d1 and d2 correspond to the first and second
exponential operations, respectively. In this way, no multiplier
or lookup table is needed in these approximated functions and
the calculation is simplified with negligible accuracy loss.
C. Improved Architecture for the Constant multiplier
The constant multiplication related to log2 e appears many
times in the proposed architecture, and it is the critical
path of the system before optimization. In this subsection,
optimizations for this calculation at both algorithmic level and
architectural level are developed.
First, we use algorithm strength reduction strategy to reduce
the total number of operations in the constant multiplication.
The approximation value of log2 e is 1.0111 (binary), and it
(3) can be expressed as: 1.0111 = 1+0.1−0.0001, called ”ADD-
ADD-SUB” operation. In this way, we can simplify the three
additions and three shifts in the constant multiplier into one
addition, one subtraction and two shifts.
In hardware implementation, we first transform the ”ADD-
ADD-SUB” operation into ”ALL-ADD” operation, then we
adopt the carry-save technique to simplify these additions. The
overall architecture of the constant multiplier is shown in Fig.
3.
Supposing there is E=A*1.0111, in our implementation, A is
an unsigned number, so we have: B = A >> 1, D = A >> 4
and E = A + B − D. The subtraction of D is equivalent to
adding its two’s complement. Defining that F =∼ D, we have:
E = A + B + F + 1.
In our architecture, both A and E have Q bits. The critical
path of the traditional method is the operation time of 3Q FAs
(full adders). If we use carry-save method, the critical path can
be reduced to the operation time of Q + 1 FAs. Also, the total
number of FAs can be reduced from 3Q to 2Q.
Compared with the results without optimization, we find
that not only the critical path is reduced, but also the area
and power consumption are significantly reduced due to the
simplification. Carry-select method [13]- [14] can be used to
224

Fig. 3. The architecture for the constant multiplier.
replace the Q-bit carry-ripple adder in our design and reduce
the critical path further. Considering the balance of area and
speed, it is not adopted.
IV. T HE A RCHITECTURE OF THE NATURAL L OGARITHMIC
U NIT
PN xj
The LNU has a positive input F = j=1 e , and the
output is lnF . By limiting the input range of the logarithmic
function, the calculation in LNU can be simplified.
A. Mathematical Transformation of Logarithmic Calculation
We define k ∈ [1, 2) and w as an integer. The real number
pair (k, w) satisfies the following conditions:
∀F > 0, ∃!(k, w) : F = 2w · k.
A leading one detector (LOD) is required to compute k and
w in the proposed architecture. In the LNU, the input of the
LOD is F , and the output is the highest bit “1” position of F .
It will be easy to calculate the value of k and w, if the decimal
point and the highest one’s position is given. An example of
the relationships between F and LOD’s output (k and w) is
shown in TABLE I, assuming that the width of F is 8, and
the decimal point is between the fifth bit and the sixth bit.
TABLE I
A N E XAMPLE OF T HE R ELATIONSHIP B ETWEEN F A ND (k,w)
F (Binary) Highest 1 in F k(Binary)
11011.111 0 1.1011111
01010.101 1 1.0101010
00001.101 4 1.1010000
00000.011 6 1.1000000
lnF = ln2 · log2 F = ln2(w + log2 k)).
According to (5) and (6), the range of the input of loga-
rithmic operation is limited within [1, 2). To compute log2 k,
a linear fitting is used:
log2 k ≈ k − 1, k ∈ [1, 2). (7)
At last, the calculation of lnF can be expressed as:
lnF = ln2(k − 1 + w). (8)
B. Proposed Design for the LNU
According to (8), if we figure out k and w, the value of
lnF will be easy to get. Notice that the value of k is limited
within [1,2), so the calculation of (k − 1 + w) dose not require
any adder. The fractional part of the value of (k − 1 + w) is
equal to that of k, and w is the integer part of the value of
(k − 1 + w).
The hardware architecture for the LNU is described in Fig.
4. Note that the output of the LOD is w, and k = F >> w.
The binary number 0.1011 is adopted as the approximation of
number ln2. The constant multiplier related to 0.1011 involves
”ADD-ADD-ADD” operations. It is implemented with the
same carry-save method as in the EXPU.
Fig. 4. The proposed design for the LNU.
V. E XPERIMENTAL R ESULTS
(5) A. Simulation Results and Discussion on Accuracy
We choose four sets of random values with different ranges,
represented as rand0.1, rand1, rand5 and rand10 as the input
data for the experiment. Each set has 4096 numbers within
the corresponding range. For instance, numbers in set rand10
are random values lie in the range [-10, +10]. In order to
meet the accuracy needs of different application scenarios, our
architecture is designed to support input and output of both
8bit and 16bit. We take 128bit of input data at a time, and use
16 EXPUs to process the data concurrently. These units can
be reconfigured to support sixteen 8bit inputs or eight 16bit
inputs.
xi − xmax , (i = 1, 2, ..., N ) are used as the input of the
w hardware architecture, which brings two advantages. On the
4 one hand, all input data are non-positive, ensuring that the
3 results of the exponential operation are not greater than 1
0 and thus avoiding numerical overflow. On the other hand, the
-2 calculations related to input value don’t involve any sign bit,
which saves the storage space and simplifies the calculation.
In the experiment, we preprocess the input data with
(6) software to transform the floating point numbers into fixed
point numbers. Also, the software gives the value Nin which
indicates where the decimal point of these fixed point numbers
is. The hardware architecture uses the input data together with
this value Nin to get the output of softmax function. Similarly,
225
the hardware part gives the number Nout to help the software
calculate the final results. With this method, the calculation
accuracy can be greatly improved with a small increase in
computational complexity and without affecting the speed.
The calculation accuracy under a certain error threshold
8.00e − 5 are shown in Fig .5. The accuracy is defined as
follows:
ncorrect results
accuracy = , (9)
ninput numbers
where ncorrect results is the number of correct results and
ninput numbers is the total number of input data. Here cor-
rectness is determined by comparing the results calculated
by the architecture proposed using fixed point numbers with
those calculated by C code using floating point values. If the
difference between the corresponding two results are less than
the error threshold, the result is defined as a correct result. As
we can see, our architecture achieves very high accuracy at a
low error threshold.
Fig. 5. The calculation accuracy under the error threshold 8.00e − 5 for four
sets of data.
B. Synthesis Results
The architecture is evaluated on two platforms, Xilinx Zynq-
7000 ZC706 development board and the TSMC 28-nm CMOS
technology. The synthesis results are presented in Table II. The
throughput (T ) in the table is defined as the number of outputs
per second:
128bit · fclk
T = . (9)
data width
The efficiency(E) is defined as:
T network,” in System-On-Chip Conference, 2017, pp. 323326
E= . (9)
area · power
As is shown in the table, the data processing speed is
sufficient for real-time applications.
VI. C ONCLUSION
In this paper, an efficient hardware implementation of
softmax function is presented for deep neural networks.
Algorithm-hardware co-optimizations, including multiple al-
gorithm strength reduction methods, linear fitting and compo-
nent reuse are performed to reduce hardware complexity and
226
TABLE II
KEY FEATURES OF THE PROPOSED HARDWARE
ARCHITECTURE FOR SOFTMAX
Characteristic FPGA TSMC 28nm
Clock Frequency 436MHz1 2.8GHz
Area 2564LUTs, 0.015mm2
2794Registers
Power 0.751W 51.60mW
Throughput 6.976 G/s(8bit) 44.8G/s(8bit)
3.488 G/s(16bit) 22.4G/s(16bit)
Efficiency 57.88G/(mm2 · mW)(8bit)
- 28.94G/(mm2 · mW)(16bit)
1 This frequency is a representation of how fast the architecure can
run. The frequency in practical applications will be limited by the
speed of data transmission and the clock speed of the development
board.
datapath delay. In addition, regular multipliers and lookup ta-
bles are eliminated in this design. With all these optimizations,
the architecture proposed achieves a very high efficiency of
463.04Gb/(mm2 · mW). So it demonstrates great potential for
many real-time applications on embedded systems.
R EFERENCES
[1] G. E. Hinton, S. Osindero, and Y. Teh, “A fast learning algorithm for
deep belief nets,” Neural Computation, vol. 18, pp. 1527-1554, 2006.
[2] D. Kim, J. Ahn, and S. Yoo, “A novel zero weight/activation-aware
hardware architecture of convolutional neural network,” in Design,
Automation & Test in Europe Conference & Exhibition, 2017, pp.
14661471.
[3] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and B. Dally,
“Deep compression and eie: Efficient inference engine on compressed
deep neural network,” in Hot Chips 28 Symposium, 2017, pp. 16.
[4] S. Han, J. Kang, H. Mao, Y. Hu, X. Li, Y. Li, D. Xie, H. Luo, S. Yao,
and Y. Wang, “Ese: Efficient speech recognition engine with sparse lstm
on fpga,” 2017.
[5] Z. Wang, J. Lin, and Z. Wang, “Accelerating recurrent neural networks:
A memory-efficient approach,” IEEE Transactions on Very Large Scale
Integration Systems, vol. PP, no. 99, pp. 113, 2017.
[6] S. Wang, Z. Li, C. Ding, B. Yuan, Q. Qiu, Y. Wang, and Y. Liang,
“Clstm: Enabling efficient lstm using structured compression techniques
on fpgas,” in Proceedings of the 2018 ACM/SIGDA International
Symposium on Field-Programmable Gate Arrays. ACM, 2018, pp. 1120.
[7] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning
applied to document recognition, Proceedings of the IEEE, Nov. 1998.
[8] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with
recurrent neural networks,” in International Conference on Machine
Learning, 2014, pp. 17641772.
[9] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-
to-end attention-based large vocabulary speech recognition,” in Acous-
tics, Speech and Signal Processing (ICASSP), 2016 IEEE International
Conference on. IEEE, 2016, pp. 49454949
[10] B. Yuan, “Efficient hardware architecture of softmax layer in deep neural
[11] T. G. Noll, “Carry-save architectures for high-speed digital signal
processing,” Journal of Vlsi Signal Processing Systems for Signal Image
& Video Technology, vol. 3, no. 1-2, pp. 121140, 1991.
[12] C. Nagendra, M. J. Irwin, and R. M. Owens, “Area-time-power tradeoffs
in parallel adders,” IEEE Transactions on Circuits & Systems II Analog
& Digital Signal Processing, vol. 43, no. 10, pp. 689702, 2002.
[13] O. J. Bedrij, “Carry-select adder,” Electronic Computers Ire Transactions
on, vol. EC-11, no. 3, pp. 340346, 1962.
[14] B. Ramkumar and H. M. Kittur, “Low-power and area-efficient carry se-
lect adder,” IEEE Transactions on Very Large Scale Integration Systems,
vol. 20, no. 2, pp. 371375, 2012.