Combination of Hough Transform and Neural
Network on recognizing mathematical symbols
Aouadi Nabil
Latice Laboratory
Tunis, Tunisia
nabil.aouadi@utic.rnu.tn
Abstract—Offline printed mathematical symbol recognition is
a particularly difficult task. Recognizing mathematical symbols is
one stage within the overall system for recognition of mathemati-
cal documents. We describe many experiments using MultiLayer
Perceptron (MLP), Hough Transform (HT), k Nearest Neighbors
2021 8th International Conference on ICT & Accessibility (ICTA) | 978-1-6654-6641-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICTA54582.2021.9809779
(kNN) and structural Freeman chain code, to enhance symbol
recognition of printed mathematics. First, we investigate the use
of a MLP based method. Second, we compare the performance
of a proposal neural structural method, named HT-MLPs, on
symbols that initial MLP usually confuses. The inclusion of HT
in MLP reduces symbol confusion rate by 21% and improves
recognition rates from 72% to 93%. To improve the efficiency of
the proposed method, we compare it to KNN then to Freeman
code based methods, commonly used in pattern recognition.
While analyzing results, we show that HT-MLP always gives
a lower mean confusion and rejection and higher success rates
than the others solutions.
Index Terms—Perceptron Multilayer Network, Hough Trans-
form, Mathematical Symbol Specialized Neural Network
I. I NTRODUCTION
The big challenges of Human Computer Interaction research
is the full inclusion of people with special needs into the
digital world. We specially thinks about blind people. Many
studies such as printed and handwritten mathematical symbol
recognition aim at addressing their needs [1]. Significant
research effort has been reported for handwritten math formula
or symbol recognition. It’s an attractive field of pattern recog-
nition leading to practical applications [2] and [3]. According
to [4], Symbol recognition process has some limitations that do
not allow achieving high accuracy. Three types of limitations
can be considered :
• Symbols with different meanings and similar shapes such
as: 9q gq bh GC KR DO UV QO B8 Z2 S5 I1 Uv.
• Upper and lower case symbols with same shapes such as:
nN fF wW zZ xX yY cC uU sS kK mM pP vV.
• interpreting the mathematical expression
This paper addresses the problem of recognizing math-
ematic notation which poses challenges from its two-
dimensional pattern, the rich set of used symbols, the range of
similar looking symbols whose bold, calligraphic, and italic
varieties must be recognized distinctly, the imbalance and
paucity of an available training data, and the impossibility
of final verification through spell check. Many researchers
have worked on Handwritten Mathematical Expression (HME)
recognition for many years such as Andersons effort in the
978-1-6654-6641-7/21/$31.00 ©2021 IEEE
Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.
60s [5] [6]. In this work, the general objective, covered is to
contribute to an automatic system which is required to rec-
ognize mathematical formulas, automatically extracted from
image of scientific documents [7]. Notice that mathematical
formula recognition is mainly composed of three major steps:
formula segmentation, symbol extraction and recognition and
finally formula structural analysis. The first step segments
the input image into a set of symbols. The second and third
steps analyze the spatial arrangement of this set of symbols to
recover the information content of the mathematical formula.
Though it is known that the segmentation phase is generally
easier than the latter one, the effort of reducing recognition
errors is still important because not only wrong symbol recog-
nition results often give bad effect to the structural analysis
phase, but also it is useful to reduce search space of the latter
phase. Although the proposed system achieved good results
for symbol recognition before this investigation, its failure to
distinguish certain common symbols would be bothersome to
any serious user. So we aim 1) to expand the symbol set,
used for the training and test steps, in order to recognize other
symbols, Greek letters and new operators and connectors, and
2) to improve the accuracy of single symbol recognition. To
this end, we implemented some distinct recognition algorithms
and performed experiments to determine which algorithms
were best suited for recognition of mathematical symbols.
The paper content will be as follows. First, we examine the
existing mathematics notation recognition literature. Then, we
study the ability of a MLP, as neural classifier, to distinguish
mathematical symbols. Next, we show how the classifier may
be integrated with the Hough Transform (HT ) to improve
its classification ability. Afterwards, we analyze and compare
our experimental results with those obtained by the KN N
method and by the Freeman code. We close the paper with
some concluding remarks and future works.
II. P REVIOUS RELATED RESEARCH WORK
In [8], we report that the failure of conventional
OCR systems to treat mathematics symbols has several
consequences:
- Readers of mathematical documents cannot automatically
search for earlier occurrences of a variable or operator, in
tracing the notation and definitions used by a journal article.
- The appearance of mathematics on the same line as text
often confounds OCR treatment of surrounding words.
- Equations can only be represented as graphics by semantic
transformation systems, such as those converting digital
documents into braille for accessibility by blind readers.
Note that several research centers are interested in the
recognition of mathematical formulas and symbols (printed
or handwritten) as shown in table 1.
In [9], algorithms based on nearest neighbor scheme for
offline character recognition, can get high accuracy but is
slow. Authors proposed a classifier based on nearest neighbor
classification in a space of 27 features. For each symbol, its
bounding box is divided a 5 by 5 rectangular grid and the
percentage of black pixels in each cell will be extracted as a
feature. The other two features are aspect ratio and the absolute
height in pixels of the bounding box. Euclidean distance
is calculated as the dissimilarity between two symbols. For
Alvaro et al in [10], each bounding box was normalized to a
fixed size and the Euclidean distance between two images is
calculated based on the difference of each pixel.This method
obtained a 94.24% symbol recognition rate in InftyCDB-1
database.
In last decades, many papers have been made which high-
lights the importance of this research in pattern recognition
and analysis of documents. Hidden Markov Models (HMMs)
have been already used by [11] and [12] for symbol recog-
nition. They had 82 symbols which were written 50 times.
A combination of HMMs and Artificial Neuronal Networks
(ANNs) was then proposed by [13]. In another method,
proposed by [14], an improved version of Kohn-Munkres
matching-algorithm is used with a 94 symbol set. Later on,
the authors, in [15] proposed a recognition based method using
Support Vector Machine (SVM) with a 43 mathematic symbol
set. Methods based on combining classifiers for mathematic
symbol recognition was tested by [16]. They used feature
template matching together with HMMs in a 198 symbol set.
Notice that majority of symbol recognition research in mathe-
matics notation focuses on handwritten input produced online
via data tablet. Generally, in pattern recognition in general,
there are two major approaches: segmentation-based methods
(structural approach) and free segmentation methods (statis-
tical approach). In segmentation-based methods, the symbol
is split into segments to be recognized, using segmentation
algorithm. In contrast, segmentation-free methods use features
of the whole symbol image. Current trends are no longer
using a single method. The use of mixed methods tends to
generalize. In [4], two stages needed to the process of symbol
recognition:symbol segmentation and accurate classification
for over 300 classes. Many multidimensional mathematical
symbols need both horizontal and vertical projection to be
segmented. In the next section, we will describe the use
of different methods for symbol recognition (neuron based
method: MLP, neural-structural method: HT-MLPs, statistical
method: KNN and structural method based on freeman code)
and present the results of our experiments along with analysis.
Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.
III. P ROPOSAL METHODS FOR SYMBOL RECOGNITION
A basic method in symbol recognition is to record several
samples for each symbol of interest, then to compare input
symbols against those stored models. There are currently 24
symbol classes in our database, with 140 samples for each.
The two third of the database are used for the training and the
rest for the test. These symbols are extracted, from different
scientific journal, articles and books with various sizes and
fonts then scanned using a resolution of 300 dpi. We believe
that there are two mainly factors which influence on the
training: sample quality and value diversity.
A. MLP, neural classifier
This section will be about the use of a MLP to classify
mathematical symbols. Recall that MLP is a trainable algo-
rithm that can learn to solve complex problems from training
data that consists of a set of pairs of inputs and desired outputs.
It is composed of interconnected processing elements, called
neurons, that work together to produce an output. It is a
modification of the standard linear perception in that it uses
three or more layers of neurons with nonlinear activation func-
tions, and is more powerful than the perceptron in that it can
distinguish data that is not linearly separable. In the proposed
MLP (see figure 1), the input and model symbols both consist
of bitmaps of size 20*15 pixels. It is a three layer network. The
input layer comprises, according various experiments: 1) 80 or
150 neurons which respectively correspond to the black pixel
densities of each 4 or 2 pixels portion of symbol bitmap, 2) 300
neurons when considering each pixel from the bitmap symbol
and 3) 7 neurons if the seven Hu moments are used as input
to the MLP. The output layer consists of 24 neurons referring
to the different classes of mathematical symbols, associated
with their activation degrees.
After several tests, it was shown that the best MLP design
is a threelayer network with 15 neurons in the unique hidden
layer and using 0.3 as training factor. To measure the MLP
performance, we compute the recognition (R), the rejection
(J) and the confusion (C) rates. Figure 2 gives the obtained
average rates in function of the MLP input layer content.
We conclude that 1) the best results are obtained with a
4pixel portion black pixel densities used as input layer and
2) the HU moments seem to be not suitable to characterize
mathematical symbols because they are insensitive to rotation
so many confusion cases are noted (e.g. ⊂ versus ∩ and ∪,
→ versus ←, ( versus ), [ versus ] ).
Analyzing the obtained results with 4pixels portion densities
as input layer, and observing the event of confusion, we find
certain distinct symbols, such as , ∈, (, ̸∈, ← are in close
resemblance. They have roughly similar pixel densities. We
remark that the structure of symbol should play a decisive
role to remove such ambiguities. By counting the number of
segments in a symbol, for example, we can distinguish many
variants of symbols (see Figure 3).
To this end, we had resort to HT , to detect lines especially
as most symbols are composed of segments. We demonstrate,
as it will be shown in the next subsection, that it is possible
to improve the M LP results by the combination of HT as
frontal part. We call this hybrid neural structural method:
HT − M LP s.
B. HT-MLPs, neural structural classifier
It is about to extract segments from symbol bitmap using
HT (after a skeletonization step using a thinning algorithm)
then according to their number (0, 1, 2 segments or more),
we call the appropriate M LP as shown in Figure 8. This
implies that initial MLP will be split into three specialized
M LP s: MLP0, MLP1 and MLP2+ , each one will serve to
learn and recognize symbols respectively composed of 0, 1, 2
or more segments. Here are the best obtained results for the
three MLPs:
- MLP0: Input layer (80 neurons), 1 hidden layer (4 neurons),
Number of iterations:14.
- MLP1: Input layer: 80 neurons Hidden layer : 1 layer
(3 neurons) Number of iterations:100 - MLP2+: Input layer:
80 neurons Hidden layer : 1 layer (12 neurons) Number of
iterations:1100
Figure 7 summarizes the obtained results with the proposal
neural structural method: HT-MLPs. We can observe a distinc-
tive improvement compared to initial MLP since the symbol
misrecognition [11] rate is reduced by 21% and the recognition
rate is increased from 72% to 93%. However, there exists yet
some confusion between similar symbols, ∏ composed∩ of the
same number ∑ of segments (e.g. versus (, versus , [versus
], ] versus ). We considered some of the misrecognitions to
be too difficult for any classifier to resolve on the basis of only
the number of segments. Further structural primitives should
be taken into account to avoid such confusions.
C. KNN classifier using Hu moments
It is reported that KNN algorithm is amongst the simplest of
all machine learning algorithms. We applied here for symbol
recognition. Thus, a symbol is classified by a majority vote
of its neighbors, with the symbol being assigned to the class
most common amongst its k nearest neighbors. K is a positive
integer, typically small. If k = 1, then the symbol is simply
assigned to the class of its nearest neighbor. The neighbors are
taken from a set of symbols for which the correct classification
is known. This can be thought of as the training set for the
algorithm, though no explicit training step is required. In order
to identify neighbors, the symbols are here represented by
Hu moments in a multidimensional feature space. We used
the Euclidean distance to compare between input and model
symbols Hu moment values. After several tests by varying the
value of k from 1 to 4 considering Hu moments, we find that
the best recognition rate is obtained with k equals to 4 and
the first three moments of Hu.
D. Structural Freeman chain code
Another method, known as contour, is the structural Free-
man chain code of the symbols border. To detect symbol
border, we tested different filters such as filters of Robert
and Sobel. We find that Sobel filter is simplest and gives
Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.
good results. Then, we used Freeman code to describe the
contours of symbols. Recall that this code moves along a
digital curve or a sequence of border pixels based on eight
connectivities. Each movement is classified as a direction
and encoded with a numbering scheme i—i=0,1,2,7 denoting
an angle of 45i counterclockwise from the positive xaxis.
In this method chain code stores the absolute position of
the first pixel and the relative positions of successive pixels
along the symbols border. Thus, input and model symbols are
converted to sequences of digits, called chain codes. Digit
sequences are compared using histogram of slopes along
contour and histogram matching techniques. In fact, we used
various metric of distances (e.g. Euclidian, Manhattan, and
Canberra distances), and similarities (histogram intersection)
to respectively compare between two histograms (stored as
vectors). In case of a distance, we usually look for the
nearest elements (to minimize distance). In the case of a
similarity, we look for the most similar elements, (to maximize
similarity index). This method achieved a recognition rate near
to 46.67%.
IV. C ONCLUSION
In this paper, we mainly described a successful classifi-
cation method based on three specialized MLPs, utilizing
HT. We evaluated effects of this hybridization, comparing
its recognition rates with those of initial MLP and other
methods based on KNN and Freeman code. Results of this
hybridization are very satisfactory, given the small number
of samples. It achieved 93% of single symbol recognition.
These results show that: 1) giving slightly higher weight to the
structural information: segment number, and 2) specializing
the MLP, produces better results. But, the problem of symbol
recognition is still opened especially as the ultimate objective
is to recognize any mathematical symbol whatever isolated or
in the formula context, handwritten or printed. It is important
to note that the rates of recognition, confusion and rejection are
not the only criteria to evaluate this work since the formula
context should play a decisive role to pronounce about the
real identity of the symbol. To improve this work, we plan
further research in particular, the following: 1) to elaborate
tests of efficiency and performance of the proposal neural
structural method on a wider set of mathematical symbols.
Until now, we formed a reasonable size set. Classical meth-
ods have limitations, but the development of robust methods
depends on the database availability of large size to test the
performance, robustness, reliability of the proposed methods
and conduct meaningful statistical tests to compare against
each other, 2) to distinguish the symbols that confuse yet
the HTMLPs classifier considering additional features such as
segment orientation, other structural primitives like presence
of loops, branchpoints [11], endpoints that could distinguish
many variants of symbols and 3) to combine the results of
several recognizers which should give more accurate results
than using only one.
 R EFERENCES
[1] Bianchini, Claudia and Borgia, Fabrizio and De Marsico, Maria, ”A
concrete example of inclusive design: deaf-oriented accessibility”, arXiv
preprint arXiv:1911.13207, 2019.
[2] Ayeb, Kawther Khazri and Meguebli, Yosra and Echi, Afef Kacem,
”Deep Learning Architecture for Off-Line Recognition of Handwritten
Math Symbols”, Mediterranean Conference on Pattern Recognition and
Artificial Intelligence,2020.
[3] Ayeb, Kawther Khazri and Echi, Afef Kacem and Belaı̈d, Abdel,
”Arabic/Latin and handwritten/machine-printed formula classification
and recognition”, 1st International Workshop on Arabic Script Analysis
and Recognition (ASAR),2017.
[4] Nazemi, Azadeh and Tavakolian, Niloofar and Fitzpatrick, Donal and
Suen, Ching Y and others, ”Offline handwritten mathematical symbol
recognition utilising deep learning”,arXiv preprint arXiv:1910.07395,
2019.
[5] R. H. Anderson, ”Syntax-directed Recognition of Hand-printed Twodi-
mensional Mathematics”, in Symposium on Interactive Systems for
Experimental Applied Mathematics: Proceedings of the Association for
Computing Machinery Inc. Symposium, 1967, pp. 436459.
[6] D. BLOSTEIN and A. N. N. GRBAVEC, ”RECOGNITION OF MATH-
EMATICAL NOTATION”,in Handbook of Character Recognition and
Fig. 1. The CROHME competition for mathematic formula and symbol
Document Image Analysis, pp. 557582.
recognition.
[7] A. Kacem, A. Belad and M. Benahmed, ”Automatic extraction of
printed mathematical formulas using fuzzy logic and propagation of
context”, in Jour. of IJDAR, volume 4, Number 2, pp. 97108, December
2001.
[8] , Malon, Christopher and Uchida, Seiichi and Suzuki, Masakazu, ”Math-
ematical symbol recognition with support vector machines”, Pattern
Recognition Letters, volume 29 , number 9 , pages 1326–1332 ,2008
[9] Yang, Michael and Fateman, Richard, ”Extracting mathematical expres-
sions from postscript documents”,Proceedings of the 2004 international
symposium on Symbolic and algebraic computation, pages 305–311,
2004.
[10] Alvaro, Francisco and Zanibbi, Richard, ”A shape-based layout descrip-
tor for classifying spatial relationships in handwritten math”, Proceed-
ings of the 2013 ACM symposium on Document engineering, pages
123–126, 2013.
[11] M. Koschinski, H.J. Winkler and M. ang, ”Segmentation and recognition
of symbols within handwritten mathematical expressions”, in Proc. of
CASSP, vol. 4, Detroit, MI, pp. 24392442, 1995.
[12] H.J. Winkler,H. Fahner and M. lang, ”A softdecision approach for
structural analysis of handwritten mathematical expressions”, in Proc.
of ICASSP, vol. 4, Detroit, MI, pp. 24592462, 1995.
[13] A. Kosmala, G. Rigoll, S. Lavirotte and L. Pottier, ”Online handwritten
formula recognition using hidden Markov models and context dependent
graph grammars”, in Proc. of ICDAR, Bangalore, Karnataka, India,
pp.107110, 1999.
[14] Z. Xuejun, L. Xinyu, P. Boachang and Y. tang, ”Online recognition
handwritten mathematical symbols”, in Proc. of ICDAR, Ulm, Germany,
pp. 645648, 1997.
[15] E. Topia and R. Rojas, ”Recognition of online handwritten mathematical
formulas in the Echalk system”, in Proc. of ICDAR, Edinburgh, U.K.,
pp. 980984, 2003.
[16] U. Garain, B. Chaudhuri, and A. Ray Chaudhuri, ”Identification of
embedded mathematical expressions in scanned documents”, in Proc.
of ICPR, Cambridge, UK, pp. 138149, 2004.
Fig. 2. MLP design with 4 pixel portion black pixel densities as input.

Fig. 3. MLP rates.

Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.
 Fig. 4. symbol confusion cases.

Fig. 7. MLP2+ rates.

Fig. 5. MLP0 rates.

Fig. 8. Average Rates of HT-MLPs.

Fig. 6. MLP1 rates.

Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.
 Combination of Hough Transform and Neural
Network on recognizing mathematical symbols
Aouadi Nabil
Latice Laboratory
Tunis, Tunisia
nabil.aouadi@utic.rnu.tn

Fig. 9. Proposed System for Mathematical Symbol Recognition

Combination of Hough Transform and Neural

Network on recognizing mathematical symbols
Aouadi Nabil
Latice Laboratory
Tunis, Tunisia
nabil.aouadi@utic.rnu.tn

Authorized licensed use limited to: Zhejiang University. Downloaded on August 18,2023 at 22:46:06 UTC from IEEE Xplore. Restrictions apply.