To Improve the Performance of Handwritten digit Recognition using Support Vector Machine

Anshuman Sharma (anshuman515@gmail.com)

Abstract: Handwritten Numeral recognition plays a vital role in postal automation services especially in countries li e !ndia where multiple languages and scripts are used "iscrete Hidden #ar ov #odel (H##) and hy$rid o% Neural Networ (NN) and H## are popular methods in handwritten word recognition system. &he hy$rid system gives $etter recognition result due to $etter discrimination capa$ility o% the NN. A ma'or pro$lem in handwriting recognition is the huge varia$ility and distortions o% patterns. (lastic models $ased on local o$servations and dynamic programming such H## are not e%%icient to a$sor$ this varia$ility. )ut their vision is local. )ut they cannot %ace to length varia$ility and they are very sensitive to distortions. &hen the S*# is used to estimate glo$al correlations and classi%y the pattern. Support *ector #achine (S*#) is an alternative to NN. !n Handwritten recognition+ S*# gives a $etter recognition result. &he aim o% this paper is to develop an approach which improve the e%%iciency o% handwritten recognition using arti%icial neural networ Keyword Handwriting recognition+ Support *ector #achine+ Neural Networ !" Introduction Handwritten ,ecognition re%ers to the process o% translating images o% hand-written+ typewritten+ or printed digits into a %ormat understood $y user %or the purpose o% editing+ inde.ing/searching+ and a reduction in storage si0e. Handwritten recognition system is having its own importance and it is adopta$le in various %ields such as online handwriting recognition on computer ta$lets+ recogni0e 0ip codes on mail %or postal mail sorting+ processing $an chec amounts+ numeric entries in %orms %illed up $y hand and so on. &here are two distinct handwriting recognition domains1 online and o%%line+ which are di%%erentiated $y the nature o% their input signals. !n o%%line system+ static representation o% a digiti0ed document is used in applications such as che2ue+ %orm+ mail or document processing. 3n the other hand+ online handwriting recognition (3H,) systems rely on in%ormation ac2uired during the production o% the handwriting. &hey re2uire speci%ic e2uipment that allows the capture o% the tra'ectory o% the writing tool. #o$ile communication systems such as 4ersonal "igital Assistant (4"A)+ electronic pad and smart-phone have online handwriting recognition inter%ace integrated in them. &here%ore+ it is important to %urther improve on the recognition per%ormances %or these applications while trying to constrain space %or parameter storage and improving processing speed. 5igure 1 shows an online handwritten 6ord recognition system. #any current systems use "iscrete Hidden #ar ov #odel $ased recogni0er or a hy$rid o% Neural Networ (NN) and Hidden #ar ov #odel (H##) %or the recognition

3nline in%ormation captured $y the input device %irst needs to go through some %iltration+ preprocessing and normali0ation processes. A%ter

normali0ation+ the writing is usually segmented into $asic units (normally character or part o% character) and each segment is classi%ied and la$eled. 7sing H## search algorithm in the conte.t o% a language model+ the most li ely word path is then returned to the user as the intended string 819. Segmentation process can $e per%ormed in various ways. However+ o$servation pro$a$ility %or each segment is normally o$tained $y using a neural networ (NN) and a Hidden #ar ov #odel (H##) estimates the pro$a$ilities o% transitions within a resulting word path. &his research aims to investigate the usage o% support vector machines (S*#) in place o% NN in a hy$rid S*#/H## recognition system. &he main o$'ective is to %urther improve the recognition rate8:+;9 $y using support vector machine (S*#) at the segment classi%ication level. &his is motivated $y success%ul earlier wor $y <anapathira'u in a hy$rid S*#/H## speech recognition (S,) system and the wor $y )ahlmann 8=9 in 3H,. <anapathira'u o$tained $etter recognition rate >ompared to hy$rid NN/H## S, system. !n this wor + S*# is %irst developed and used to tra 0in an 3H, system using character data$ases. S*# with pro$a$ilistic output are then developed %or use in the hy$rid system. (ventually+ the S*# will $e integrated with the H## module %or word recognition. 4reliminary results o% using S*# %or character recognition are given and compared with results using NN reported $y 4oisson 8?9. &he %ollowing data$ases were used: !,3N355+ 7N!4(N and the mi.ture !,3N355-7N!4(N data$ases. #" $%isting Techni&ues #"! Modified discrimination function 'M()*+ ,-assifier <. S. @ehal and Nivedan )hatt 81A9 designed a recognition system %or handwritten "evangari Numeral using #odi%ied discrimination %unction (#B"5) classi%ier. A recognition rate and a con%usion rate were o$tained as =?C and D.5C respectively. #"# .eura- .etwor/ on )evenagari .umera-s ,. )a'a'+ @. "ey+ S. >haudhari 8119 used neural networ $ased classi%ication scheme. Numerals were represented $y %eature vectors o% three types. &hree di%%erent neural classi%iers had $een used %or classi%ication o% these numerals. 5inally+ the outputs o% the three classi%iers were com$ined using a connectionist scheme. A E-layer #@4 was used %or implementing the classi%ier %or segment-$ased %eatures. &heir wor produced recognition rate o% =?.:=C. #"0 1aussian )istribution *unction

,. F. ,amte e et.al applied classi%iers on GAAA numerals images o$tained %rom di%%erent individuals o% di%%erent pro%essions. &he results o% 4>A+ correlation coe%%icient and pertur$ed moments are an e.perimental success as compared to #!s. &his research produced ?G.G=C recognition rate $y considering ;; %eature dimensions. #"2 *u33y c-assifier on Hindi .umera-s #. Hanmandlu+ A.*. Nath+ A.>. #ishra and *.H. #adasu used %u00y mem$ership %unction %or recognition o% Handwritten Hindi Numerals and produce ?:C recognition rate. &o recogni0e the un nown numeral set+ an e.ponential variant o% %u00y mem$ership %unction was selected and it was constructed using the normali0ed vector distance. #"4 Mu-ti-ayer Perceptron 7''wal )hattacharya+ ). ). >haudhuri 8119 used a distinct #@4 classi%ier. &hey wor ed on "evanagari+ )engali and (nglish handwritten numerals. A $ac propagation ()4) algorithm was used %or training the #@4 classi%iers. !t provided ??.G;C and ??.ADC recognition accuracies on the original training and test sets o% "evanagari numeral data$ase+ respectively. #"5 (uadratic c-assifier for )evanagari .umera-s 7. 4al+ &. 6a a$ayashi+ N. Sharma and 5. Himura 81D9 developed a modi%ied 2uadratic classi%ier %or recognition o% o%%line handwritten numerals o% si. popular !ndian scripts1 vi0. &hey had used :D dimensional %eatures %or high-speed recognition. A %ive-%old cross validation techni2ue has $een used %or result computation and o$tained ??.5:C accuracy %rom "evnagari scripts+ respectively. 0" Proposed Approach 0"! Support Vector Machine 'SVM+ S*# in its $asic %orm implement two class classi%ications. !t has $een used in recent years as an alternative to popular methods such as neural networ . &he advantage o% S*#+ is that it ta es into account $oth e.perimental data and structural $ehavior %or $etter generali0ation capa$ility $ased on the principle o% structural ris minimi0ation (S,#). !ts %ormulation appro.imates S,# principle $y ma.imi0ing the margin o% class separation+ the reason %or it to $e nown also as large margin classi%ier. &he $asic S*# %ormulation is %or linearly separa$le datasets. !t can $e used %or nonlinear datasets $y indirectly mapping the nonlinear inputs into to linear %eature space where the ma.imum #argin decision %unction is appro.imated. &he mapping is done $y using a ernel %unction. #ulti class classi%ication can $e per%ormed $y modi%ying the G class

scheme. &he o$'ective o% recognition is to interpret a se2uence o% numerals ta en %rom the test set. &he architecture o% proposed system is given in %ig. E.&he S*# ($inary classi%ier) is applied to multi class numeral recognition pro$lem $y using one-versus-rest type method. &he S*# is trained with the training samples using linear ernel. >lassi%ier per%orms its %unction in two phases1 &raining and &esting. 8G?9 A%ter preprocessing and 5eature (.traction process+ &raining is per%ormed $y considering the %eature vectors which are stored in the %orm o% matrices. ,esult o% training is used %or testing the numerals. Algorithm %or &raining is given in algorithm. 0"# Statistica- 6earning Theory Support *ector #achines have $een developed $y *apni in the %ramewor o% Statistical @earning &heory 81E9. !n statistical learning theory (S@&)+ the pro$lem o% classi%ication in supervised learning is %ormulated as %ollows: 6e are given a set o% l training data and its class+ {(x1,y1)...(xl,yl)} in Rn R sampled according to un nown 'oint pro$a$ility distri$ution P(x,y) characteri0ing how the classes are spread in Rn R. &o measure the per%ormance o% the classi%ier+ a loss %unction L(y+%(%)) is de%ined as %ollows:

that re2uires designing and training potentially large num$er o% individual classi%iers. 7sing S@&+ we do not need to do that. <enerali0ation error can $e directly minimi0ed $y minimi0ing an upper $ound o% the ris %unctional R(f). &he $ound given $elow holds %or any distri$ution 4(.+y) with pro$a$ility o% at least 1- :

where the parameter h denotes the so called *> (*apni ->hervonen is) dimension. is the con%idence term de%ined $y *apni 81A9 as :

(,# is not su%%icient to %ind good classi%ier $ecause even with small Remp(f), when h is large compared to l+ will $e large+ so R(f) will also $e large+ ie: not optimal. 6e actually need to minimi0e Remp(f)and at the same time+ a process which is called structural ris #inimi0ation (S,#). )y S,#+ we do not need test set %or model selection anymore. &a ing di%%erent sets o% classi%iers F1, F2 with nown h1, h2 we can select f %rom one o% the set with minimal Remp(f)+ compute and choose a classi%ier with minimal R(f).No more evaluation on test set needed+ at least in theory. However+ we still have to train potentially very large num$er o% individual classi%iers. &o avoid this+ we want to ma e h tuna$le (ie: to cascade a potential classi%ier Fi with *> dimension I h and choose an optimal f %rom an optimal Fi in a single optimi0ation step. &his is done in large margin classi%ication. 0"0 SVM formu-ations S*# is reali0ed %rom the a$ove S@& %ramewor . &he simplest %ormulation o% S*# is linear+ where the decision hyper plane lies in the space o% the input data %.

L(y+%(%)) is 0ero i% f classi%ies . correctly+ one otherwise. 3n average+ how f per%orms can $e descri$ed $y the ,is %unctional: (,# principle states that given the training set and a set o% possi$le classi%iers in the hypothesis space F+ we Should choose f F that minimi0es Remp(f). However+ which generali0es well to unseen data due to over %itting phenomena. Remp(f) is a poor+ over-optimistic appro.imation o% R(f)+ the true ris . Neural networ classi%ier relies on (,# principle. &he normal practice to get a more realistic estimate o% generali0ation error+ as in neural networ is to divide the availa$le data into training and test set. &raining set is used to %ind a >lassi%ier with minimal empirical error (optimi0e the weight o% an #@4 neural networ s) while the test set is used to %ind the generali0ation error (error rate on the &est set). !% we have di%%erent sets o% classi%ier hypothesis space F1, F2 e.g. #@4 neural networ s with di%%erent topologies+ we can select a classi%ier %rom each hypothesis space (each topology) with minimal Remp(f) and choose the %inal classi%ier with minimal generali0ation error. However+ to do

character. )ased on normali0ed central moments+ a set o% seven moment invariants is derived. 5urther+ the resultant image was thinned and seven moments were e.tracted. &hus we had 1D %eatures (; original and ; thinned)+ which are applied as %eatures %or recognition using <aussian "istri$ution 5unction. &o increase the success rate+ the new %eatures need to $e e.tracted $y applying A%%ine !nvariant #oment method. 2"# Affine Moment Invariants &he A%%ine #oment !nvariants were derived $y means o% the theory o% alge$raic invariants. 5ull derivation and comprehensive discussion on the properties o% invariants can $e %ound. 5our %eatures can $e computed %or character recognition. &hus overall 1= %eatures have $een used %or Support *ector #achine. 4" $%periment 4"! )ata Set )escription !n this paper+ the 7>! #achine learning data set are used. &he 7>! #achine @earning ,epository is a collection o% data$ases+ domain theories+ and data generators that are used $y the machine learning community %or the empirical analysis o% machine learning algorithms. 3ne o% the availa$le datasets is the 3ptical ,ecognition o% the Handwritten "igits "ata Set. &he dataset o% handwritten assamese characters $y collecting samples %rom D5 writers is created. (ach writer contri$uted 5G $asic characters+ 1A numerals and 1G1 assamese con'unct consonants. &he total num$er o% entries corresponding to each writer is 1=E (I 5G characters J 1A numerals J 1G1 con'unct consonants). &he total num$er o% samples in the dataset is =GE5 ( I D5 K 1=E ). &he handwriting samples were collected on an i$all =A:A7 e.ternal digiti0ing ta$let connected to a laptop using its cordless digital stylus pen. &he distri$ution o% the dataset consists o% D5 %olders. &his %ile contains in%ormation a$out the character id (!")+ character name (@a$el) and actual shape o% the character (>har). !n the raw 3ptdigits data+ digits are represented as EG.EG matrices. &hey are also availa$le in a preprocessed %orm in which digits have $een divided into non-overlapping $loc s o% D.D and the num$er o% on pi.els have $een counted in each $loc . &his generated =.= input matrices where each element is an integer in the range A.1:.

!n this case the hypothesis space is a su$set o% all hyper planes o% the %orm: f(x) = wx +b. S*# %inds an optimal hyper plane as the solution to the learning 4ro$lem which is geometrically the %urthest %rom $oth classes since that will generali0e $est %or %uture unseen data. &here are two ways o% %inding the optimal decision hyper plane. &he %irst is $y %inding a plane that $isects the two closest points o% the two conve. hulls de%ined $y the set o% points o% each class+ as shown in %igure G. &he second is $y ma.imi0ing the margin $etween two supporting planes as shown in %igure E. )oth methods will produce the same optimal decision plane and the same set o% points that support the solution (the closest points on the two conve. hulls in %igure G or the points on the two parallel supporting planes in %igure E). &hese are called the support vectors. 2" *eature $%traction 2"! Moment Invariants &he moment invariants (#!s) 819 are used to evaluate seven distri$uted parameters o% a numeral image. !n any character ,ecognition system+ the characters are processed to e.tract %eatures that uni2uely represent properties o% the

analysis is plotted in a pie graph %or easy visual inspection. 3nce the analysis is complete+ we can test its classi%ication a$ility in the testing data set. &he green rows have $een correctly identi%ied $y the discriminant space (uclidean distance classi%ier. 6e can see that it correctly identi%ies ?=C o% the testing data. &he testing and training data set are dis'oint and independent.

*ig2 7 Sample digits e.tracted %rom the raw 3ptdigits dataset. 4"# )ata Preprocessing 5or the e.periments using S*#+ e.ample isolated characters are preprocessed and ; local %eatures %or each point o% the spatially resample online signal were e.tracted. 5or each e.ample character there are E5A %eature values as input to the S*#. 6e use S*# with ,)5 ernel+ since ,)5 ernel has $een shown to generally give $etter recognition result 8;9. <rid search was done to %ind the $est values %or the > and gamma (representing in the original ,)5 ernel %ormulation) %or the %inal S*# models and $y that+ > I = and gamma I were chosen.

4"2 $%perimenta- Resu-ts 4"2"! Test application Analysis &he test application accompanying the source code can per%orm the recognition o% handwritten digits. &o do so+ open the application (pre%era$ly outside *isual Studio+ %or $etter per%ormance). >lic on the menu 5ile and select 3pen. &his will load some entries %rom the 3ptdigits dataset into the application.

Fig.# $sing the "efa%lt &al%es in the application 4"4 Resu-ts A%ter the analysis has $een completed and validated+ we can use it to classi%y the new digits drawn directly in the application. &he $ars on the right show the relative response %or each o% the discriminant %unctions. (ach class has a discriminant %unction that outputs a closeness measure %or the input point. &he classi%ication is $ased on which %unction produces the ma.imum output.

Fig.5 !pt"igits "ata loa"e" into the application &o per%orm the analysis+ clic the ,un Analysis $utton. 4lease $e aware that it may ta e some time. A%ter the analysis is complete+ the other ta$s in the sample application will $e populated with the analysisL in%ormation. &he level o% importance o% each %actor %ound during the discriminant Fig # 'e can see the analysis also pe(fo()s (athe( well on co)pletely new an" p(e&io%sly %nseen "ata. (.periments were per%ormed on di%%erent samples having mi.ed scripting languages on numerals using single hidden layer.

Tab-e !: "etail ,ecognition per%ormance o% S*# on 7>! datasets

Tab-e # "etail ,ecognition per%ormance o% S*# and H## on 7>! datasets

Tab-e 0 ,ecognition ,ate o% (ach Numeral in "A&AS(&. !t is o$served that recognition rate using S*# is higher than Hidden #ar ov #odel. However+ %ree parameter storage %or S*# model is signi%icantly higher. &he memory space re2uired %or S*# will $e the num$er o% support vectors multiply $y the num$er o% %eature values (in this case E5A). &his is signi%icantly large compared to H## which only need to store the weight. H## needs less space due to the weight-sharing scheme. However+ in S*#+ space saving can $e achieved $y storing only the original online signals and the penup/ pen-down status in a compact manner. "uring recognition+ the model will $e e.panded dynamically as re2uired. &a$le E shows the comparison o% recognition rates $etween H## and S*# using all three data$ases. S*# clearly outper%orms in all three isolated character cases. &he result %or the isolated character cases a$ove indicates that the recognition rate %or the hy$rid word recogni0er could $e improved $y using S*# instead o% H##. &hus+ we are currently implementing word recogni0er using $oth H## and S*# and comparing their per%ormance. *ig 8 <raph ,epresentation $etween H## and S*# 5" ,onc-usion

Handwriting recognition is a challenging %ield in the machine learning and this wor identi%ies Support *ector #achines as a potential solution. &he num$er o% support vectors can $e reduced $y selecting $etter > and gamma parameter values through a %iner grid search and $y reduced set selection 6or on integrating the S*# character recognition %ramewor into the H## $ased word recognition %ramewor is on the way. !n the hy$rid system+ word preprocessing and normali0ation needs to $e done $e%ore S*# is then used %or character hypothesis recognition and word li elihood computation using H##. !t is envisaged that+ due to S*#Ms $etter discrimination capa$ility+ word recognition rate will $e $etter than in a H## hy$rid system. R$*R$.,$S 7 819 Sandip Haur+ N ,ecognition o% Handwritten "evanagri Script using 5eature )ased on Oerni e #oments and Ooning and Neural Networ >lassi%ierP+ A #. &ech. &hesis ,eport+ 4an'a$i 7niversity+ 4atiala+ GAAD+ pp. 8G9 <aurav Fain+ Fason Ho+ NHandwritten "igits ,ecognitionP+ #ultimedia Systems+ 4ro'ect ,eport+ 7niversity o% &oronto+ Novem$er G1+ GAA=+ pp. 1-E. 8E9 Scott ". >onnell+ ,.#.H. Sinha+ Ani1 H. Fain N,ecognition o% 7nconstrained 3n-@ine "evanagari >haractersP+ GAAA+ !(((. 8D9 A.H. Fain+ ,o$ert 4.6."uin+ Fianchang #ao+ N Statistical 4attern ,ecognition: A ,eviewP+ !((( &rans. 4A#!+ *ol.GG+ No. 1+ GAAA. 859 Anu' Sharma+ N3nline Handwritten <urmu hi >haracter ,ecognitionP+ A 4h. ". &hesis report+ School o% 8:9 Shu$hangi ".>.+ 4.S.Hiremath+ N#ulti->lass S*# >lassi%ier %or (nglish Handwritten "igit ,ecognition using #anual >lass SegmentationP+ 4roc. !ntMl >on%. on Advances in >omputing. >ommunication and >ontrol (!>A>EMA?) GAA?+ pp. E5E-E5:. 8:9 Sa$ri A. #ahmoud and Sameh #. Awaida+ N,ecognition 3% 3%%-@ine Handwritten Ara$ic (!ndian) Numerals 7sing #ulti-Scale 5eatures And Support *ector #achines *s. Hidden #ar ov #odelsP &he Ara$ian Fournal 5or

Science And (ngineering+ *olume ED+ Num$er G$+ 3cto$er + GAA?+4p. DEA-DDD. 8;9 A.)or'i+ and #. Hamidi+ NSupport *ector #achine %or 4ersian 5ont ,ecognitionP+ !nternational Fournal o% !ntelligent Systems and &echnologies+ Summer GAA;+ pp. 1=D-1=; 8=9 >. *asantha @a shmi+ ,itu Fain+ >. 4atvardhan+ NHandwritten "evanagari Numerals ,ecognition 6ith Higher AccuracyP+ 4roc. o% !((( !nt. >on%. on >omputational !ntelligence and #ultimidia Application+ GAA;+ pp G55-G5?. 8?9 7.)hattacharya+ ).).>haudhari+ NHandwritten Numeral "ata$ases o% !ndian Scripts and #ultistage ,ecognition o% #i.ed NumeralsP+ !((( &rans. on 4A#!+ *ol.E1+ No.E+ GAA?+ pp.DDD-D5;. 81A9 7. 4al+ &. 6a a$ayashi+ N. Sharma and 5. Himura+ NHandwritten Numeral ,ecognition o% Si. 4opular !ndian ScriptsP+ 4roc. ?th !>"A,+ >uriti$a+ )ra0il+ *ol.G (GAA;)+;D?-;5E. 8119 >hristopher #. )ishop+ N4attern ,ecognition and #achine @earningP+ Springer 4u$lication+ Singapore+ GAA:+ 4p. 1-E+ EA=-EGA.