2017 International Conference on Tools with Artificial Intelligence
Enhancing AlexNet for Arabic Handwritten words
Recognition Using Incremental Dropout
Rolla Almodfer1 Shengwu Xiong1, 2 Mohammed Mudhsh1 and Pengfei Duan1, 2*
1
School of Computer Science and Technology, Wuhan University of Technology, Wuhan
430070, China.
2
Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology, Wuhan 430070, China
duanpf@whut.edu.cn
Abstract— Currently, the growth of mobile technologies, lead
to a necessity to develop handwritten recognition applications.
While the recognition of handwritten Latin and Chinese has been
extensively investigated using various techniques, so little works
have been done on Arabic handwritten recognition, and none of
the existing techniques is accurate enough for practical
application. Over the past few years, deeper convolutional neural
networks (CNNs) have widely been employed for improving
handwritten recognition performance. In this paper, we enhance
the popular AlexNet for Arabic Handwritten Words Recognition
(HWR). By adopting a dropout regularization, we prevented our
system against overfitting problem and reduced the error
recognition rate. We also investigated ReLU and tanH activation
functions performance in the fully connected layers. Through
several settings of experiments using the benchmarking IFN/ENIT
Database, we achieved a new state-of-the-art classification
accuracy of 92.13% and 92.55%. Lastly, we compared our best
result to those of previous state-of-the-art.
Keywords—Arabic handwritten; AlexNet; Overfitting;
Dropout
I. INTRODUCTION AND STATE OF THE ART
Automatic handwriting words recognition is among the most
important axes in the NLP (Natural language processing).
Applications like postal address and zip code recognition,
passport validation, check processing demonstrated the need for
recognition of handwritten characters. During the last few
decades, numerous research results have been reported on
handwriting recognition. Although there are promising results
for recognition Latin, Chinese and Japanese script, accuracies on
recognition handwriting Arabic scripts fall behind. This is due
to unlimited variation in human handwriting, the large variety of
Arabic character shapes, the presence of ligature between
characters and overlapping of the components. The different
approaches of handwritten word recognition (HWR) fall into
either the on-line or off-line category. In on-line HWR, the
computer recognizes the words as they are written. Off-line
recognition is performed after the writing is completed. We here
focus on Off-line HWR, which has traditionally been tackled by
following two main approaches: (i) Analytic approach and (ii)
holistic approach. The analytic approach [1], [2] a word is
decomposed into a set of smaller components (e.g., characters,
graphemes, allographs) and then features are extracted for each
component. Finally, the word is transformed into sequential
2375-0197/17/31.00 ©2017 IEEE
DOI 10.1109/ICTAI.2017.00106
feature vectors suited for training and recognition. Large
varieties of techniques/classifiers have been employed for the
analytic approach. El Hajj et al. in [3] proposed an offline
Arabic handwritten recognition system based on the
combination of three HMM classifiers. The baseline-
independent and baseline-dependent features were extracted
using a sliding window in three directions for the purpose of
writing inclination, overlapping ascenders and descenders and
shifted position of some diacritical marks. Then, an HMM
classifier was put forward for in each direction to classify the
extracted features. The results showed that this combination
gave better results than a single HMM. The obtained results
given by the proposed model was higher than 90% accuracy.
Jayech et al. in [4] developed a dynamic hierarchical Bayesian
network for Arabic handwriting recognition. After the
preprocessing step, an explicit segmentation based on the
smoothed vertical histogram projection. Then, a set of statistical
features was extracted for each character using the invariant
moment, such as Hu and Zernike. After that, the dynamic
hierarchical Bayesian network used to recognize the Arabic
words. Based on the nature of Arabic writing, Parvez et al. [5]
proposed an off-line Arabic handwritten text recognition
system using structural techniques. A text line segmented into
words and sub/words and dots were extracted leaving the Parts
of Arabic words (PAW). The Arabic characters were modeled
by fuzzy polygons. Fuzzy polygons matching algorithm were
later applied to recognize the Arabic word. Khémiri et al. [6]
proposed a system based on Probabilistic Graphical Models
(PGM). The system is divided into three stages, preprocessing,
feature extraction and word classification. Preprocessing
includes baseline estimation. Structural features (ascendants,
descendants, loops and diacritic points) and statistical features
at the pixel level (pixel density distributions and local pixel
configurations) are then extracted from word images. Words are
recognized using a variety of PGMs.
Although being successful, the performance of such
approaches has always been substantially dependent on the
selection of right representing features, which is a difficult task
for cursive writing. In a holistic approach, the entire word is
recognized without prior decomposition of the word [7], [8]. In
this case, the feature vectors are extracted from the word as a
whole. Convolutional Neural Networks (CNNs) are the current
state-of-the-art model architecture for handwritten recognition
tasks. CNNs apply a sequence of filters to the raw image data to
663
extract and learn higher-level features, which the model can then
use for recognition.
Recently, Deep Neural Networks (DNN) has acquired a
reputation for solving many computer vision problems, and its
application to the field of HWR has been shown to provide
significantly better results than traditional methods [9]. In this
vein, the first reported successful use of DNN for Arabic
handwritten word recognition was multidimensional recurrent
neural networks [10]. The authors obtained 91.4% accuracy on
IFN/ENIT database. Later, Elleuch et al. [11] presented a
Convolutional Deep Belief Network (CDBN) to recognize
Arabic words. The authors obtained 83.7 Accuracy rate on
IFN/ENIT Database. In spite of the previous works, using Deep
architectures on Arabic handwriting recognition are relatively
scarce compared to other languages.
By introducing ImageNet [12] large-scale visual recognition
challenge (ILSVRC), many new state-of-the-art network
architectures are proposed to improve the accuracy of image
classification such as AlexNet [13], Google Net [14] and
VGGNet [15]. However, training these networks needs large
amounts of data and high computational resources. Currently, a
delicate method called transfer learning has presented to deal
with training problems. Transfer learning means fine-tuning
CNN models (supervised) pre-trained from one dataset to
another dataset. For example, a fully trained net on ImageNet
database can also be used without the output layer to extract
features from samples of any other dataset. Nevertheless,
directly adapting any of fully trained net for the HWR task
cannot yield a good performance as the handwritten has specific
attribute different from those of ImageNet database. Training
from scratch is required. Training a convolutional net on a small
dataset greatly affects the convolutional net’s ability to
generalize, often result in overfitting. The error on the training
set is driven to a very small value, but the network error becomes
large when new data is presented. The network has memorized
the training samples, but it has not learned to generalize to new
ones.
During these years, many solutions to the overfitting
problem have been presented; one of them is called dropout.
Dropout has overwhelmed the others regularization methods
due to its simplicity and its good empirical results. Dropout
was proposed by Hinton et al. [16] as a form of regularization
for fully connected neural network layers. Each element of a
layer’s output is kept with probability p, otherwise, is set to 0
with probability (1 − p) [17]. Experimental results show that
Dropout improves the network’s generalization ability, giving
improved test performance. However, not all Dropout methods
will improve performance and should not be used “blindly.”
In this paper, we introduce state of the art Arabic HWR
model based on deep learning. For this purpose, we trained
AlexNet from scratch on IFN/ENIT Database. Dropout was
adopted and applied in different positions using different values
to enhance the performance of AlexNet to reach a good
accuracy. We found that using incremental dropout values, the
classification accuracy further improved consistently and
significantly. We also investigate the effect of ReLU and tanh
activation functions in the fully connected layers performance.
After several experiments and parameters adjustments, we
664
achieve a state of the art recognition accuracies equal to 92.13%
and 92.55%.
The rest of the paper is organized as follows: Section 2
start with Arabic handwritten characteristics and challenges; in
section 3 we give an overview of Deep Neural Networks, the
details of AlexNet and the proposed enhancing methods.
Section 4 is dedicated to the conducted experiments and their
results. Finally, we draw our conclusion in section 5.
II. ARABIC HANDWRITTEN CHARACTERISTICS AND
CHALLENGES
Arabic script is cursive by nature; which makes its recognition
more challenging than that of Latin, Chinese and Japanese.
Arabic is composed of 28 main characters and written from right
to left in both printed and handwritten forms. Each character has
two or four different shapes depending on its position in the
word, which will increase the number of classes to be recognized
from 28 to 84. The number and position of dots dominate
characters that have similar shapes [18], [19]. Fifteen characters
have dots with the character and 52 basic character shapes
without dots. Some challenging structural characteristics of the
characters in the database are described below:
1. Arabic word consists of one or more connected
components (sub-words), and each one contains one or
more characters that can be overlapped with other
characters or diacritics. Moreover, multiple characters
can be combined vertically to form a ligature (Fig. 1a).
2. Some words have touching or broken characters (Fig.
1b).
3. Every writer has an individual writing style (Fig. 1c).
4. Some Arabic characters have diacritic marks (a
diacritic may be placed above or below the body of the
character). These diacritics can be overlapped (Fig.
1d).
(a) Ligature and
overlapping
(b) Touching
and broken
characters
(c) Different writers
different style
(d) Confusion in
assigning
dots/diacritics
Fig. 1 Complexities in Arabic handwriting recognition
III. METHOD
In this section, we briefly summarize a standard DNN,
standard AlexNet. We also give an overview of Overfitting
problem. We then describe our proposed Enhancing models for
Arabic HWR.
A. Deep Neural Networks
A DNN is one of the most advanced machine learning
techniques, it consists of a succession of convolutional and
max-pooling layers, and each layer receives connections from
its preceding layer. The most popular image classification
structure of DNN is constructed by three primary processing
layers: Convolutional Layer, Pooling Layer and Fully
Connected Layer (or classification layer). DNN units are
described below:
Convolutional Layer:
Let ∈ ℝ × represents the ith map in the lth layer, jth
kernel filter in the lth layer connected to the ith map in the (l-1)th
layer denoted ∈ ℝ × and index maps set Mj ={i|ith in the
(l-1) Layer map connected to jth map in the lth layer}. The
th
convolution operation can be given by equation (1).
= (∑ ∗ + ) (1)
∈
Where f (.) is non-linearity activation function and is bias.
Max-Pooling layer
The max-pooling layer abstracts the input feature into a
lower dimensional feature. It has been shown that max-pooling
can lead to faster convergence select superior invariant features,
and improve generalization. Pooling equation can be described
in equation (2).
−1
= ( ) (2)
Where Max (.) is Max-sampling function to compute the max
value of each n×n region in −1 map.
Classification layer
The fully connected layer is used at the end of the
network. After multiple convolutional and max-pooling layers,
a shallow Multi-Layer Perceptron (MLP) is used to complete
the DNN. The output maps of the last convolution layer are
either down-sampled to 1 pixel per map, or a fully connected
layer combines the outputs of the topmost convolutional layer
into a 1D feature vector. The last layer is always a fully
connected layer with one output unit per class in the
classification task. This layer may have a non-linear activation
function or a softmax activation to output probabilities of class
predictions.
B. Alexnet Architecture
AlexNet (proposed by Krizhevsky et al. [13]) is one of the
deep convolution nets designed to deal with complex scene
classification task on Imagenet database. The task is to classify
the given input into one of the 1000 classes. The arrangement
and configuration of all the layers of AlexNet are depicted in fig.
2. AlexNet includes five convolutional layers, three max-
pooling layers, two Local Response Normalization (LRN)
layers, three fully connected layers, and a linear layer with
665
softmax activation in the output. The fully connected layers are
simply implementing the dot product between the input and
weight vectors, where each neuron in layer l is connected to all
outputs of neurons in layer l − 1. Moreover, it uses the dropout
regularization method to reduce overfitting in the fully
connected layers and applies Rectified Linear Units (ReLUs) for
the activation of those and the convolutional layers function to
speed up the learning process.
C1
M1
C2 M2 FC1 FC2
M3 FC3
5 C3 C4 C4
5
5
5 3 3 3
3 3 3
256 384 384 256
96 1000
Fig. 2 AlexNet architucure 4096 4096
C. Overfitting
Training deep neural network is particularly dependent
on the availability of large quantities of training data, which are
important to learn a non-linear function from input to output
that generalizes well and yields high classification accuracy on
unseen data. Not having enough quality data will generate
overfitting, which means that the network is highly biased to
the data it has seen in the training set and, therefore will not be
able to generalize the learned model to any other samples[20].
An elegant solution to this problem is a dropout. Dropout
simply refers to “dropping out units”; units representing both
hidden and visible in the deep network. Dropped out neurons
do not contribute to the forward pass and backpropagation.
Dropping a unit out means temporarily removing it from the
network with all its incoming and outgoing connections.
Dropout regularization reduces complex co-adaptations of
neurons as a neuron cannot rely on the existence of other
neurons. Therefore, the neurons forced to learn the robust
features that are effective in conjunction with the other neurons
[21]. Dropout is reported to achieve success on several
benchmark databases [22].At test time, all the neurons are used
but multiply their outputs by the dropout value, which is a
reasonable approximation to taking the geometric mean of the
predictive distributions produced by the exponentially many
dropout networks [23].
D. Enhancing methods
Dropout is a widely used strategy to increase
generalization performance, which can be implemented by
randomly dropping units for each layer with a given probability.
Each element of a layer’s output is kept with probability p,
otherwise, is set to 0 with probability (1 − p). Experiments show
that Dropout improves the network’s generalization ability,
giving improved test performance. However, not all Dropout
methods will improve performance and should not be used
“blindly.” In the proposed enhancing method, we trained
AlexNet from scratch and called it Net1, then Net2, Net3 and
Net4 were build using the baseline Net1. Each of the four
networks tests the input image and produces the same number
of classes. All the networks share the same configuration of the
number of layers, the number of filters and the dropout after
fully connected layers. The four networks can answer the
following two questions: where to add the dropout and to use
which value? Which activation function can improve the
performance? ReLU or tanH? The networks architectures are
depicted in Fig. 3 and the description of each network are as
follow:
Net1 (Fully connected dropout): this net represents standard
AlexNet. In Net1 dropout value set to 0.5. The dropout applied
immediately after fully connected layers.
Net2 (Convolution dropout): The dropout value set to 0.5
after convolution layer 1 and convolution layer 2.
Net3 (Max-pooling dropout): The dropout value set to 0.5
after max-pool layer 1, max-pool layer 2 and max-pool layer 3.
Net4 (Convolution and max-pooling dropout): In this net, the
dropout probability is increased with respect to the depth of the
network. The dropout rate increased by 0.1 after each
convolution layer. The dropout rate increased by 0.2 after each
max-pool layer. The purpose of doing so is that the bottom layers
of a deep network are usually harder to train compared with the
top layers [24].
IV. EXPERIMENTAL RESULTS
To examine the performance of the proposed models, we
have conducted many experiments. The network's performance
has been measured in term of Classification Accuracy (CA).
The objective of the experiment part was twofold. First, it was
a comparison between the state-of-the-art. Second, we wanted
to check whether dropout was better after convolution or after
max pooling layer and in which values.
Net 1:
D D
C1 M1 C2
C M2 C
C3 C4 C5 M3 Fc1 0.5 Fc2
F 0
0.5
D D D D
Net 2: C1 0.5 M1 C2 0.5 M2 C3 C4 C5 M3 Fc1 0.5 Fc2 00.5
D D D D
Net 3: C C3 C4 C5 M3 0.5 Fc1
C1 M1 0.5 C2 M2 0.5 C F 0.5 Fc2 0.5
D D D D D D D D
Net 4: C1 0.0 M1 0.1 C
C2 0.1 M2 0.3 C3 0.2 C
C C4 0.3 C5 0.4 M
0
Convolution layer Max pool layer
Net name Net1 Net2
Dropout AlexNet 0.5 After C1 and C2
Fig. 3 The proposed architectures
A. Training Method:
Suppose we have T categories and the training data for the
each category are denoted as (xi, yi), where i = {1,..., N }, with xi
∈ ℝ and yi ∈ ℝ Being the feature vector and label apart.
The objective of training is to iteratively minimize the following
cross-entropy loss function:
⎡ ⎤
1
( )=− ⎢ 1{ = } = ⎥
⎢ ⎥
⎣ ⎦
where θ is model parameters, ∑ is a factor of
normalization, 1(.) is an indicating function.
The loss function of J(T) can be minimized by using one of
optimization algorithms such as Stochastic Gradient Descent
SGD, adam or adadelta, during the training process.
To put our results in comparison with another state of the art
methods, we have used the offline IFN/ENIT benchmark
database. IFN/ENIT database contains 32492 binary images of
Arabic handwritten words written by more than 400 writers.
19724 words for training and 12768 for testing. The words
represent 937 Tunisian town/village names. Before training, we
shuffle our training data; we normalize the offline words image
to a size of 227 x 227.
The experiments conducted on an open source library called
keras [25]. As mentioned before, AlexNet composed of 5
convolution layers and three max-pooling layers. A max pooling
layer follows the layer 1, layer 2 and layer 5. The receptive field
of each convolutional layer is 3×3. Three Fully-Connected (FC)
layers follow the fifth convolutional layer: the first two has 512
channels each. Since the original AlexNet was trained on 1000
classes, its last fully connected layer produces 1000 outputs. We
replace this layer with a new fully-connected layer that has as
many outputs as the number of classes (937 for the IFN/ENIT
D
D D
M3 0.5 F
Fc1 0.5 Fc2 0.5
Dropout layer Fully connected layer
Net3 Net4
0.5 after M1, M2 and M3 Incremental dropout
666
database). The final layer is the soft-max layer. As we used the
latest version of Keras, we replace the Local Normalization
Layer with Batch Normalization layer. This layer is useful and
versatile. The training is carried out by using Stochastic Gradient
Descent SGD which is a highly common optimization algorithm
utilized in deep networks to update the weights. The batch size
was set to 64 and the momentum to 0.9. The training was
regularized with the weight of five · 10−4 and dropout
regularization for the first two fully connected layers (dropout
ratio set to 0.5). The learning rate was set to 10−2.The type of
non-linearity used for the convolution layers is Rectified Linear
Unit (ReLU). AlexNet was trained for 200 epochs. The whole
training procedure for a single network took at most 3 hours on
a desktop PC with an Intel i7 3770 processor, a NVidia GTX780
graphics card and 16 gigabytes of on-board RAM.
B. The impact of tanH and ReLU activation functions
ReLU (as shown in Fig.4) is an effective activation function
for use in neural network. ReLU function is given by:
f(x) = max(0,x)
Unlike fully connected layer, convolution layer extracts
feature. Feature extraction requires sparsity in the input feature
maps, and it should set to 0 as many features as possible. Unlike
ReLU, the sparsity does not come into effect with other
activation functions as they can generate small values instead
of zeros. The sparsity in the features helps in speeding up the
computation process by removing the undesired features. In
AlexNet, the fully connected layers used tanH (shown in Fig.4)
as an activation function. tanH function is given by:
f(x) = tanh(x)
The focus of fully connected layers is to generate new
features rather than extracting features as convolution layers do.
Moreover, as fully connected layers are close to the output
layer, so it is less affected by the vanishing gradient problem.
ReLU TanH
10 1.5
8 1
6 0.5
4 0
2 -0.5
0 -1
-2 -1.5
-6 -2 0 2 4 6 -6 -4 -2 0 2
Fig. 4 ReLu and tanH activation functions
In our experiments, we first test tanH as activation function
in the fully connected layers. Net1 the standard AlexNet
achieves classification accuracy equal to 83.3%., the training
reached 100% after 26 epochs. While when replace the
activation function in the fully connected layers to ReLU, the
Net1 performed better and the accuracy grows to 85.53%. The
training reached 100% after 86 epochs. Net2 achieved equal to
85.96% when using tanH and 85.32% using ReLU. Net2
performed the worst among the four nets. Net3 achieved
accuracy equal to 89.55%, but when replacing its activation
function with ReLU, the accuracy grows to 90.43%. The best
performer between our nets is Net4. This net achieved the
highest classification accuracies 92.13% using ReLU and
92.55% using tanH activation function. These two results have
outperformed the state of the art results. The results depicted in
Table 1 in term of Classification accuracy (CA). The overall
performances of the four networks using ReLU are depicted in
Fig. 5 (a), and using tanH are depicted in Fig. 5 (b).
C. The impact of dropout
Dropout consists of setting to zero the output of each hidden
neuron with probability p. If the neurons in CNN are dropped
out, they do not contribute to the forward pass and do not
participate in backpropagation. In this paragraph, we provide
additional insight into the performance of the proposed method.
In Fig. 5(a), (b), we show the accuracy performance during
training and testing of the model.
The blue and the orange ascending curves correspond to the
classification accuracy values for the training and the testing. As
shown in Fig. 5, Net1 (dropout in fully connected layers), the
training accuracy is increased very fast, and after 26 epochs, it
reaches 100%. However, the test accuracy of Net1 is not
promising due to the overfitting on training data (see Table 1).
Net2 performance is the poorer comparative to other networks.
This is maybe due to higher drop probability (dropout equal to
0.5) is applied to the first and the second convolution layers.
With the help of dropout, Net3 performs better than Net1, and
Net2. Dropout after each max-pool layer significantly improves
the accuracy to 89.55% and 90.43%. Net4 outperforms Net1,
Net2 and Net3 achieving the higher accuracies.
The very high accuracies 92.13% and 92.55% prove the
effectiveness of incremental dropout in improving the
generalization performance of the deep neural network.
However, when dropout is applied to every convolution and
max-pool layers in deep CNN, training process can be slow
since activation signals are dropped exponentially as dropout is
repeatedly applied [19]. Note that in the testing process, dropout
is no longer used for all nets. The results of using dropout in the
four networks are depicted in table 1.
TABLE 1. THE PERFORMANCE OF THE ALL EXPERIMENTAL NETS
Net CA% + ReLU CA% + TanH
Net1 85.53% 83.3%
4 6
Net2 85.32% 85.96%
Net3 90.43% 89.55%
Net4 92.13% 92.55%
667
 (a) Net 1 Net 2 Net 3 Net 4

Train Test Train Test Train Test Train Test

100% 100% 100% 100%

Accuracy

Accuracy

Accuracy

Accuracy
80% 80% 80% 80%
60% 60% 60% 60%
40% 40% 40% 40%
20% 20% 20% 20%
0% 0% 0% 0%

Epoch Epoch Epoch Epoch

(b) Net 1 Net 2 Net 3 Net 4

Train Test Train Test Train Test Train Test

100% 100% 100% 100%

Accuracy

Accuracy

Accuracy

Accuracy
80% 80% 80% 80%
60% 60% 60% 60%
40% 40% 40% 40%
20% 20% 20% 20%
0% 0% 0% 0%

Epoch Epoch Epoch Epoch

Fig. 5 The networks performance for 200 epochs (a)using ReLU activation function (b) using TanH activation function

D. Comparison with the state of the art
To show the performance of the proposed enhanced
AlexNet, we compare the performances of different methods
using the IFN/ENIT database. The results are given in
Table 2. Elleuch et al [11] achieved 83.7 % recognition
accuracy (16.3% error rate). Graves et al [10] achieved 91.4 %
recognition accuracy (8.6% error rate). The result of Graves was
the highest in literature. It can be seen that Net4 achieved the
higher accuracies on IFN/ENIT database. AlexNet with
Incremental Dropout + ReLU achieved 92.13% classification
accuracy (7.87% error rate). And AlexNet with Incremental
Dropout + TanH outperformed the state-of-the-art with 92.55%
recognition accuracy (error rate of 7.45%).
TABLE 2. COMPARISON WITH THE STATE- OF- THE- ART
Author Model
Net4:AlexNet with
Present work
Incremental Dropout+TanH
Net4:AlexNet with
Present work
Incremental Dropout+ReLU
MDLSM
Graves et al[10]
CDBN
Elleuch et al[11]
V. CONCLUSION
In this paper, we enhance the famous network AlexNet for the
purpose of Arabic HWR, which demonstrated the efficiency,
applied on IFN/ENIT databases. In order to protect the network
against overfitting, dropout is applied in different positions in
Net2, Net3, and Net4. Dropout regularization helps to reach a
good accuracy. We show incremental improvements of the
word recognition comparable to approaches used Deep Belief
Network (DBN) or Recurrent Neural Network (RNN). We have
achieved promising performance which is superior to the state-
of-the-art systems. The classification accuracy was 92.87%
using ReLU activation function and 92.55% using the
activation function. These two results are the new state-of-the-
art records using the deep convolutional neural network to our
best knowledge. Using incremental dropout, the classification
accuracy further improved consistently and significantly. As
future works, we plan to discover the performance of other deep
networks like VGGNet, GoogLeNet and ResNet on IFN/ENIT
database.
ACKNOWLEDGMENT
CA%
This research was supported in part by Science &
Technology Pillar Program of Hubei Province under Grant
92.55% (#2014BAA146), Nature Science Foundation of Hubei Province
under Grant (#2015CFA059), Science and Technology Open
92.13% Cooperation Pro-gram of Henan Province under Grant
(#152106000048).
91.4%
REFERENCES
83.7%
[1] Kim K K, Jin H K, Yun K C, et al. Legal Amount Recognition Based on
the Segmentation Hypotheses for Bank Check Processing[C]//
International Conference on Document Analysis and Recognition, 2001.
Proceedings. IEEE Xplore,2001:964-967.
[2] Vinciarelli A. A survey on off-line Cursive Word Recognition [J]. Pattern
Recognition, 2002,35(7):1433-1446.
[3] Alhajj M R, Likformansulem L, Mokbel C. Combining slanted-frame
classifiers for improved HMM-based Arabic handwriting recognition.[J].
IEEE Transactions on Pattern Analysis & Machine Intelligence, 2009,
31(7):1165.

668
[4] Jayech K, Trimech N, Mahjoub M A, et al. Dynamic hierarchical
Bayesian network for Arabic handwritten word recognition[C]// Fourth
International Conference on Information and Communication
Technology and Accessibility. IEEE, 2014:1-6.
[5] Parvez M T, Mahmoud S A. Arabic handwriting recognition using
structural and syntactic pattern attributes[J]. Pattern Recognition, 2013,
46(1):141-154.
[6] Khemiri A, Kacem A, Belaid A. Towards Arabic Handwritten Word
Recognition via Probabilistic Graphical Models[C]// International
Conference on Frontiers in Handwriting Recognition. IEEE, 2014:678-
683.
[7] Madhvanath S, Govindaraju V. The role of holistic paradigms in
handwritten word recognition[J]. IEEE Transactions on Pattern Analysis
& Machine Intelligence, 2001, 23(2):149-164.
[8] Ruiz-Pinales J, Jaime-Rivas R, Castro-Bleda M J. Holistic cursive word
recognition based on perceptual features[J]. Pattern Recognition Letters,
2007, 28(13):1600-1609.
[9] Wu C, Fan W, He Y, et al. Handwritten Character Recognition by
Alternately Trained Relaxation Convolutional Neural Network[C]//
International Conference on Frontiers in Handwriting Recognition. IEEE,
2014:291-296.
[10] Graves A. Offline Arabic Handwriting Recognition with
Multidimensional Recurrent Neural Networks[J]. Advances in Neural
Information Processing Systems, 2012:545-552.
[11] Elleuch M, Tagougui N, Kherallah M. Deep Learning for Feature
Extraction of Arabic Handwritten Script[M]// Computer Analysis of
Images and Patterns. Springer International Publishing, 2015:371-382.
[12] Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical
image database[C]// Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on. IEEE, 2009:248-255.
[13] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep
convolutional neural networks[C]// International Conference on Neural
669
Information Processing Systems. Curran Associates Inc. 2012:1097-
1105.
[14] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]//
Computer Vision and Pattern Recognition. IEEE, 2015:1-9.
[15] Simonyan K, Zisserman A. Very Deep Convolutional Networks for
Large-Scale Image Recognition [J]. Computer Science, 2014.
[16] Hinton G E, Srivastava N, Krizhevsky A, et al. Improving neural networks
by preventing co-adaptation of feature detectors[J]. Computer Science,
2012, 3(4):págs. 212-223.
[17] Wan L, Zeiler M, Zhang S, et al. Regularization of neural networks using
dropconnect[C]// International Conference on Machine Learning.
2013:1058-1066.
[18] Aburas A A, Gumah M E. Arabic handwriting recognition: Challenges
and solutions[C]// International Symposium on Information Technology.
IEEE, 2008:1-6.
[19] Srihari S N, Ball G. An Assessment of Arabic Handwriting Recognition
Technology [M]// Guide to OCR for Arabic Scripts. Springer London,
2012:3-34.
[20] Lemley, J.; Bazrafkan, S.; Corcoran, P., Smart Augmentation-Learning
an Optimal Data Augmentation Strategy. IEEE Access 2017.
[21] Krizhevsky, A.; Sutskever, I.; Hinton, G. E. In Imagenet classification
with deep convolutional neural networks, Advances in neural information
processing systems, 2012; pp 1097-1105.
[22] Fraser-Thomas, J.; Côté, J.; Deakin, J., Understanding dropout and
prolonged engagement in adolescent competitive sport. Psychology of
sport and exercise 2008, 9 (5), 645-662.
[23] Wu, H.; Gu, X., Towards dropout training for convolutional neural
networks. Neural Networks 2015, 71, 1-10.
[24] Zhang X Y, Bengio Y, Liu C L. Online and Offline Handwritten Chinese
Character Recognition: A Comprehensive Study and New Benchmark[J].
Pattern Recognition, 2016, 61(61):348-360.
[25] Chollet F. , “Keras,” https://github.com/fchollet/keras, 2015.