Int. J., Vol. x, No.
x, 200x
Structure Evolution of Hidden
Markov Models for Audiovisual Arabic
Speech Recognition
Abstract. In this paper, we present an Audio-
Visual Automatic Speech Recognition system
(AVASR) combining the acoustic and the visual
data. The algorithm proposed here, for modeling
the multimodal data, is a Hidden Markov Model
(HMM) hybridized with the Genetic Algorithm
(GA) to determine its optimal structure. This
algorithm is combined with the Baum-Welch
algorithm which allows an effective re-estimation
of the probabilities of the HMM. Our experiments
have shown the improvement in the performance
of the most promising audiovisual system, based
on a merger the combination of GA/HMM model.
Keywords: automatic speech recognition; HMM;
hidden Markov model; GA; genetic algorithm;
hybridization; audio-visual fusion; computer
vision.
1 Introduction
Speech is one of the most natural way by which people
communicate. The Automatic Speech Recognition (ASR)
aims to transform the acoustic signal into a sequence of
words, which ideally corresponds to a sentence pronounced
by a speaker. Recognition systems that use as input only the
acoustic signal have reached their limitations especially in
noisy environmental situations. In this case, the use of
additional information in conjunction with those extracted
from the acoustic signal is a new methodology that
improves the performance and robustness of ASR.
Numerous works on speech perception have shown the
importance of the visual information like body and facial
expressions, or lips and tongue movements in the
recognition process among humans (Potamianos et al.,
2003; Iwano et al., 2007; Pao et al., 2009). The use of data of
the lips shape and movement of the speaker seems to be a
promising issue for speech recognition because the visual
signal is both correlated to the acoustic speech signal and
contains complementary information to it.
The most models of visual speech perception focused on
sensory interaction of type fusion or integration. At this
level, the remains posed question is: where and how this
fusion of acoustic and visual modalities happens with
humans. Classically, we distinguish two types of fusion in
AVASR systems: the fusion of parameters and the fusion of
scores.
In order to improve the performance of ASR systems,
researchers have explored other paradigms like neural and
Bayesian networks, discriminative training techniques, state
duration modeling and the use of support vector machines
with HMM (Zhi-yi et al., 2006).
GA is a stochastic search method that can perform
global search within the defined searching space and
Copyright © 200x Inderscience Enterprises Ltd.
1
hopefully obtain the global optimal solution (Man et al.,
1996). In addition, GA has the potential to generate the
optimization of both feature subsets and HMM parameters
at the same time (Xueying et al., 2007; Pérez et al., 2007;
Goh et al., 2010).
Therefore, in this work we propose an alternative
solution to find the optimal structure and the candidate
structure of HMM using GA. It is based on evolution of a
population of individuals, which encode potential solutions
to the problem and transverse the fitness landscape by
means of genetic operators that are supposed to bias their
evolution towards better solutions. Experimental results
showed that our GA for HMM training can obtain more
optimized HMM than the Baum-Welch algorithm (which is
an Expectation Maximization (EM) algorithm).
This paper is organized as follows: Section 2, we briefly
present a review of some research literature related to our
research interest. In section 3, we will deal with the
background knowledge to understand our proposed
approach, and we will explain all the method used in this
work. The performance of all the system is evaluated and
discussed in section 4, and the conclusions drawn in the
final section.
2 Related works
Audio-visual speech recognition has emerged in recent
years as an active field, gathering researchers in computer
vision, signal and speech processing, and pattern
recognition (Potamianos et al., 2003). Humans use visual
information subconsciously in order to understand speech,
especially in noisy conditions, but also when the audio is
clean. The movement of the speaker’s lips offers hints about
the place of articulation, which are automatically integrated
by the brain. The McGurk effect (McGurk, 1976) proves
this by showing that, when presented with inconsistent
audio and visual stimuli, the brain perceives a different
sound from the one that was spoken.
The first foray into visual ASR and AVASR occurred in
1984 with Petajan’s Doctoral Thesis (Petajan, 1984). His
system extracted the height, width, perimeter, and area of a
speaker’s mouth and used these features as inputs to a
speech recognition system for visual ASR.
In addition, Arabic speech recognition faces many
challenges. For example, Arabic has short vowels which are
usually ignored in text. Therefore, more confusion will be
added to the ASR decoder. Additionally, Arabic has many
dialects where words are pronounced differently. The work
in (Elmahdy and Gruhn, 2009) summarized the main
problems in Arabic speech recognition, which include
Arabic phonetics, grapheme-to-phoneme relation, and
morphological complexity.
In contrast, automatic lipreading has been minimally
investigated, with the exception of a few pioneering efforts
(Potamianos et al., 2003; Iwano et al., 2007).For example
too, the work in Zhao et al. (2007) proposed local
spatiotemporal descriptors to represent and recognize
spoken isolated phrases based solely on visual input. An
automatic visual feature extraction approach is discussed by
(Tsang-Long Pao et al., 2009) to extract the visual features
of the lips that can be used in the AVASR system. These
features are important to the recognition system, especially
in noisy condition. They present recognition performance
using various visual features to explore their impact on the
recognition accuracy.
Nock et al. (2003) tested in three different approaches:
the mutual information, assuming either joint Gaussian
distributions or discrete distributions, and a specific measure
adequate to the speaker case, which was based on the
modeling of audio-visual features using HMM. This work
represents the first exhaustive evaluation of the performance
of several audio and video representations and fusion
methods in this domain.
The representations that they tested were the Discrete
Cosine Transform (DCT), the pixel’s intensity and pixel’s
intensity changes for the video signal and the Mel-
Frequency Cepstral Coefficients (MFCC) for the audio
signal. The analyzed sequences belong to the CUAVE
audio-visual database in (Patterson et al., 2002) and consist
on two persons speaking in turns. The authors concluded
that the pixel’s intensity changes together with the Gaussian
MI were the more suitable combination in the speaker
localization task. Reikeras et al. (2010) proposed a basic
AVASR system that uses MFCCs as acoustic features,
Active Appearance Model (AAM) parameters as visual
features, and Dynamic Bayesian Networks (DBNs) as
probabilistic models of audio-visual speech.
Galatas et al. (2012) have built on their recent work, an
incorporated facial depth data of a speaker captured by the
Microsoft Kinect device, as a third data stream in an audio-
visual automatic speech recognizer. The obtained results in
this work have shown the improvement of system
performance due to the depth modality, as well as
considerable accuracy increase, when using both the visual
and depth modalities over audio only speech recognition.
Lip motion and audio analysis provide a large amount of
information that can be integrated to produce more robust
audiovisual Voice Activity Detection (VAD) schemes, as
discussed in the recent work of (Minotto et al., 2013).
HMM is considered as a basic component in speech
recognition systems (Rabiner and Juang, 1993). The
estimation of model parameters affects the performance of
the recognition process. It is performed so that the
recognition error is minimized.
HMM parameters are determined during iterative process
called "training process". One of the conventional methods
that are applied in setting HMM model parameters values is
Baum-Welch algorithm. One drawback of this method is
that it converges generally to a local optimum. Several
stochastic and no stochastic versions of this algorithm were
proposed, but none of them have resulted in satisfactory
version, which allows the exploration of large research
space. GA computes simultaneously several solutions
instead of a single one, exactly and in the same way as in
the EM algorithm. At each step of the GA, combining
individuals of current population, applying genetic
operators, selection, crossover and mutation creates a new
population. The proposed models as well as the different
parts of the system are described in the following
paragraphs.
3 Proposed system overview
Human speech perception is bimodal in nature: Humans
combine audio and visual information in deciding what has
been spoken, especially in noisy environment. The visual
modality benefit to speech intelligibility in noise has been
quantified as far back as in (Potamianos et al., 2003; Iwano
et al., 2007).
AVASR introduces new and challenging tasks compared
to traditional, audio-only ASR (Potamianos et al., 2003). It
consists of three modules, which are: the acoustic
recognition module, the visual recognition module and the
fusion module.
The acoustic recognition module uses a stochastic
approach based on HMM that are optimized by a GA. The
generic process of this module is based on three stages
which are the parametrization of the acoustic signal, in our
case using the RelAtive SpecTral Analysis-Perceptual
Linear Predictive method (RASTA-PLP), the learning of
models based on genetic research of optimal parameters of
the best model among a heterogeneous population of HMM
(containing different architectures) and an optimization of
the model by a gradient algorithm (Baum-Welch) and
decoding by the Viterbi algorithm. The visual recognition
module uses the same stochastic approach; it differs only by
the parameterization phase which it based on the DCT as
shown in Figure 1.
Figure 1 Architecture of the proposed AVSR system
This section briefly introduces the main components of
our proposed system, as well as the methods intended for
their implementation.
3.1 The used audiovisual databases
In this paper two databases are used to test our proposed
Audio-Visual Automatic Speech Recognition System:
3.1.1 CUAVE database
In order to test our system we used the Clemson University
Audio-Visual Experiments (CUAVE) database (Patterson et
al. 2002). It has been created by the Digital Speech and
Audio Processing Group at Clemson University. They
distribute it on one DVD royalty-free for research purposes.
It is an audio-visual database which contains over 7,000
utterances of both connected and isolated digits recorded by
36 individual speakers (17 female and 19 male) and 20 pairs
of speakers. It is aimed at testing multi-speaker solutions. In
addition to that, it includes both still and moving speakers in
order to be robust to speaker movement.
The database contains around 3 hours of speech
recorded through a Mini DV camera. Video was then
compressed into MPEG-2 files (stereo audio at a 44 kHz
sampling rate, 16-bit). It also includes audio files checked
for synchronization (mono rate of 16 kHz, 16-bit) and
annotation files. The figure 2 shows some snapshot from the
CUAVE database.
Figure 2 Snapshot from CUAVE database
Various works have used this database in order to
evaluate their proposed systems, including, but not limited
to, in Patterson et al. (2002) the authors seek to introduce a
challenging audio-visual database (CUAVE database) that
is flexible and fairly comprehensive. Also, an image
processing based contour technique, an image-transform
method, and a deformable template scheme are used in this
comparison to obtain visual features. This work also
presents methods and results in an attempt to make these
techniques more robust to speaker movement, too an initial
baseline speaker-independent results are included using all
speakers.
The work in (Reikeras et al., 2010), propose a basic
AVASR system that uses DBNs for modeling the audio-
visual speech and the CUAVE database in order to increase
performance, in particular in noisy acoustic environment.
Therefore, we have chosen it to test and to compare the
performance of our system with this works in order to
stimulate research and fuel advances in speech processing.
3.1.2 Audiovisual Arabic database
In this first work, we have also used our own database of
Audio-Visual ARaBic speech (AVARB) database. This
multi-speaker database was recorded in a real environment
(classroom very noisy). It contains pronunciations of Arabic
words isolated, taken at a sampling frequency of 16 KHz.
Video data were captured with a 690 ×340 pixel
resolution at 30 frames/s frame rate with variations of pose
(profile view and front view) and saved in AVI format. The
database is collected from 18 speakers (2 female and 16
male), these speakers are from different regional dialects,
and each speaker pronounces each word 9 times (Arabic
numerals from zero to nine) with different modes of
pronunciation (normal, slow and fast).The distance between
the camera and the speaker as well the luminance are
adjustable to add more diversification in the audio and the
visual streams during the learning, the average distance is to
16.5 cm. In our basic corpus which contains only isolated
words, the size of each record is 2 seconds which is enough
time to utter a word slowly in Arabic. Figure 3 shows
examples of frames from some individuals in our AVARB
data corpus.
Figure 3 Some examples of frames of our AVARB database
3.2 The acoustic feature extraction
Recognizing the speech requires to extract a set of
acoustic parameters from the recorded signal. This
parameterization must be performed on successive frames
of speech of short duration (typically 10 to 30 ms) over
which the signal can generally be considered as quasi-
stationary. To improve the analysis and to limit the edge
effects the frames are weighted by a temporal window and
the extremities flattened to minimize discontinuities in the
signal that could introduce artifacts in the spectra (Harris,
1978). Finally, successive time windows overlap in part.
In this work, the feature extraction step is accomplished
using the RASTA-PLP technique which is an improvement
of the traditional Perceptual Linear Prediction (PLP) method
(Hermansky et al., 1991). It consists in a special filtering of
the different frequency channels of a PLP analyzer. The
previous filtering is done to make speech analysis less
sensitive to the slowly changing or steady-state factors in
speech. For the RASTA-PLP features, an additional filtering
is applied after decomposition of the spectrum into critical
bands. This RASTA filter removes the low modulation
frequencies which are supposed to stem from channel
effects rather than from speech characteristics. The process
of computing RASTA-PLP coefficients is described by the
Figure 4.

Figure 4 Steps of Rasta-PLP analysis

(a) (b) (c)

A typical human face follows a set of anthropometric

3.3 The visual feature extraction
It is well known that the zone of the body the most
responsible for sense is undoubtedly the face. The mouth
produces speech, eye position provides information on the
object or the observed area, and expression wrinkles are the
mirrors of our emotions. Briefly, the face is the center of the
human communications.
In speech recognition process, the focuses will be on the
lips because they constitute a large part of our research for
AVASR. As in the face detection step, Viola and Jones
(2001) face detector algorithm is used. We have chosen this
method based on Haar-like features, because it joins the
performance and the simplicity. Indeed, it guarantees an
acceptable detection rate with a relatively low error rate.
The face detector is then applied over the detected skin
regions to detect the target face, and a rectangular region is
constructed with the same width as the detected face with a
height of four times longer than the length of the face of the
target model as shown in Figure 5.
The Viola-Jones face detector uses Haar-like features
(Papageorgiou et al., 1998), which are reminiscent of Haar
basis functions, to train the stage classifier for the cascaded
classifier. The Haar-like features are predefined and
computed directly on the integral image of the gray image.
standards which have been utilized to narrow the search of a
particular facial feature to smaller regions of the face
(Khandait et al., 2009). The possibility of the use of distinct
hue of the lips taking into account the reflected light is
implemented. This point is fetched by a defined hue value.
In contrast to other methods (Potamianos et al., 2004), this
method is not light independent, thus intensity and direction
of the light can influence results.
In our work we use the following generic steps for the
facial feature detection and extraction from the localized
face image as shown in Figure 6:
 For color image, convert it into gray scale image and
adjust the intensity of both the type of images.
 Find the gradient of the region of interest (ROI) in
detected image using Sobel/Prewitt edge detection
operator and then take lower part of face and project it
vertically to get mouth localization.
 Draw rectangular box on each of the detected feature
components.
Figure 6 Examples of detected mouth region from: (a) CUAVE
database (b) AVARB database
(a)

Figure 5 An example of face detection: (a) original image; (b)

skin detection with noise suppression and (c) face
detection result

(b)
 Once the ROI is isolated, it is recommended to extract
useful information using a minimum number of attributes to
avoid statistical modeling difficulties due to high dimension
of attribute space.
To characterize video signals, in this work, we use the
coefficients of the upper-left corner of the DCT of the
resulting image of the ROI with respect to the following
relation (Gupta and Garg, 2012):

[ ] [ ]
M N
1 ( 2 x +1 ) uπ ( 2 y +1 ) vπ
F (u , v )= α ( u ) α (v ) ∑ ∑ f ( x , y ) cos cos
√ MN x=1 y=1 2M 2N

[ ] [ ]
M N
1 ( 2 x +1 ) uπ ( 2 y +1 ) vπ
F (u , v )= α ( u ) α (v ) ∑ ∑ f ( x , y ) cos cos
√ MN x=1 y=1 2M 2N
(b)

[ ] [ ]
M N
1 ( 2 x +1 ) uπ ( 2 y +1 ) vπ
F (u , v )= α ( u ) α ( v ) ∑ ∑ f ( x , y ) cos cos The indices of the most energetic coefficients are
√ MN x=1 y=1 2M obtained 2onN a training set in a way that the visual DCT

[ ] [ ]
M N
1 ( 2 x +1 ) uπ ( 2 y +1are
coefficients ) vπsubsequently extracted by fitting the DCT to
F (u , v )= α ( u ) α ( v ) ∑ ∑ f ( x , y ) cos cos
each frame in the video as it shown in Figure 7. Only the
√ MN x=1 y=1 2M 2N
corresponding coefficients are retained (in our case, we kept
(1) the 100 most energetic coefficients).
Where M and N are the dimensions of the image, u is the
horizontal spatial frequency, v is the vertical spatial 3.4 The hybrid GA/HMM modeling
frequency, f(x, y) is the pixel value at coordinates (x, y),
F(u,v) is the DCT coefficient of size M×N at coordinates (u, The HMM is an ubiquitous tool, for representing probability
v) and α() is defined as follows: distributions over sequences of observations, which has
emerged as the predominant technology in speech

{
1 recognition in recent years. Indeed, it has been proved that
, w=1
α ( w )= √2 (2) HMM are more adapted to the speech recognition domain
and images modeling than the traditional HMM (Fujiwara et
1 , otherwise ;
al., 2008).
By definition, HMM is a stochastic process determined
by two interrelated mechanisms: Markov chain which
Figure 7 Selection process of the DCT coefficients with a determines the state at time t, St=st, and a state-dependent
sample from: (a) CUAVE database (b) AVARB process which generates the observation Ot = ot depending
database
on the current state st.
HMM is usually represented by three sets of probability
distributions and their definitions are given as follows:
- Π: initial state probabilities,
π i=P ( q 1=i ) ,1 ≤i ≤ N (3)
Where N is the number of states in the model.
- A: the transitional probability matrix, A={ai,j},
a i , j =P ( qt +1= j/ qt =i ) ,1 ≤ i , j ≤ N (4)
with qt the current state and qt+1 the next state.
- B: the emission probabilities, B ={ bi(ot) }, where bi(ot) is
the probability of observation ot generated from state i
(a)
(
b i=P o t=
j
qt )
=i , 1 ≤ j≤ M , 1 ≤i ≤ N (5)

Where M is the number of observation symbols finding by

K-means algorithm (see (Kanungo et al., 2002) for more
details).
There are three fundamental problems related to the
HMM; the evaluation problem, the problem of
determination of the path of states, the problem of learning
(supervised or unsupervised). Some other problems related
to HMM are: the over flow of representation of numbers in
machine, insufficient data for learning, the update of
models when processes vary in time, the choice of HMM
architecture the best adapted to data, the choice of a good
initial estimate of the probabilities of the HMM. The
Forward algorithm allows the calculation of the likelihood
of the observations, the Viterbi algorithm finds the optimal
path which is the most likely to follow and the Baum-
Welch algorithm performs the supervised learning (re-
estimate the parameters of HMM).These algorithms are
detailed in (Rabiner and Juang, 1993).
The EM methods like the Baum-Welch algorithm are
known to converge to an optimum of the solution space, but
nothing proves that this optimum is the global one. In fact,
these algorithms tend to converge to the local optimum
which is the closer to the starting values of the optimization
procedure. The traditional remedy to this problem consists
in running several times the EM algorithm, using different
sets of random starting values, and keeping the best
solution. More advanced versions of the EM algorithm
(ECM, SEM ...) can also be used, but they do not constitute
a definitive answer to the local optima problem (Sun and
Yu, 2007). In this article, we presented a hybrid GA/HMM
algorithm to address this problem.
Another reason to present this hybrid algorithm due to
slow convergence and high computational power needed by
classical GA, especially when the generated chromosomes
cannot satisfy the conditions of the HMM parameters. For
this reason, not included in the offspring and they are
replaced by new chromosomes.
The principle of the proposed GA/HMM algorithm is as
follows: a GA manipulates a population of HMM consists
of individuals whose architecture is not unique. The
algorithm will be looking to obtain optimal HMMs, i.e. the
HMM with the highest probability of generating a given
observation O. This probability can be rapidly calculated by
the Baum-Welch algorithm. Local optimization is
performed by the Baum-Welch algorithm in conjunction
with genetic operators specialized for HMMs (Oudelha and
Ainon, 2010; Xueying et al., 2007; Goh et al., 2010).
Among these operators, for example, we find the operator of
crossover which integrates the stochasticity constraints that
apply on the HMM. There is also a normalization operator
in order to restore the same constraint after the mutation
phase. We give below a frame of GA/HMM algorithm.
Using a marker named "parent" simply to treat only
individuals required during optimization and evaluation.
1- Initialization: Create a population of size S, randomly,
the most natural encoding is to fabricate the
chromosome by reorganizing all the coefficients of the
HMM. The simplest way is to juxtapose all rows of all
matrices. We so obtain a coding in real numbers while
respecting the constraints related to the HMM. The
representation of a population is given in Figure 8.
Figure 8 Chromosome representation method in the GA/HMM
training
π1 … πN a1,1 … a1,N a2,1 … aN,N
b1,1 … b1,M b2,1 … bN,M
2- Optimization: Apply to each HMM of the unmarked
population "parent" the Baum-Welch algorithm from
the observation O.
3- Evaluation: The quality of an individual (also called
fitness) describes the adequacy of this one with its
environment. More precisely the individual will have a
note all the more high as it will be a good solution to the
problem. In the problem of optimizing an HMM, it is
desired to quantify the ability of an HMM to learning an
observation. In our GA, the fitness values are the results
of the objective function. The likelihood P(o j|λi) is an
appropriate criterion used in the objective function to
determine the quality of the chromosomes.
The probability P(o|λ) is calculated by maximum
likelihood method or Baum-Welch algorithm. They
maximize the probability that the given HMM λi
generated the training utterances oj:
Pn
f ( λi )= N
(6)
∑ Pi
i=1
M
∑ log ( P ( oi∨λ i ) ) (7)
P n=
i=1
M
It is proven that Baum-Welch algorithm leads to a local
maximum of function f(λi). However, it is possible that
other better maxima of f(λi)exist for given training set. In
this paper we tried to overcome this problem by using
genetic algorithms for maximization of f(λi) (Goh et al.,
2010).
4- Selection: Among all the individuals of the population,
select a number S'<S, which will be used as parents to
regenerate the S-S’ other individuals not selected. The
selection is done according to the best calculated scores
in step 3. Each selected individual is marked "parent."
5- Crossover/recombination: For each unmarked
individual "parent" randomly select two individuals
from the population of those marked "parent" and the
cross. The crossover is in a crossover point, and is
realized between two rows of the matrices of the
HMM, which allows to obtain in return two children. It
retains only one of both children, at random.
6- Mutation/ normalization: On each unmarked individual
"parent" we apply the mutation operator. This one
consists in modifying a small random quantity each
coefficient of the matrices of the HMM. Each
coefficient is modified according to the value of the
mutation probability. After treating an individual, we
apply on him an operator of normalization, to ensure
that this individual still answers the constraints of the
HMM. We should verify that the matrices of the HMM
 are stochastic. This operator is applied after the 1
C
operation of mutation because subsequent operations R s= ∑
C−1 i=1
( max log P(o∨λ j )+log P(o∨λ i))
imperatively work on HMM.
7- Evaluation of the stop condition: If the maximum (9)
number of iterations is not reached, then return to
step2, otherwise go to step 8. Where C is the number of classes being considered to
8- Finally return the best HMM among the current measure the reliability of each modality, and sϵ { A ,V }.
population. After that, we can calculate the integration weight of
audio reliability measure A by:
Note that this algorithm has been adapted to
optimization of vectors of observations. The re-estimation RA
made so that the HMM has a maximal probability to
w A= (10)
RA+ RV
generate the set of vectors.
Where R A and RV are the reliability measure of the
3.5 The audio-visual fusion outputs ofthe acoustic and visual GA/HMM respectively,
and the weighting factor of visual modality can be found by
In Decision fusion (or late integration), the visual and
relation:
auditory information are processed separately and are each
wA + wV = 1 for 0 < wA, wV <1 (11)
transmitted to a classifier as in (Rogozan et al., 1997).The
results at the output of each of the two recognition processes
are fused in an integration module which provides the final
result. 4 Results and discussion
In this work, match score level is used to combine the
audio and visual identification outputs. Several decision In this paper we performed the proposed AVASR system on
fusion strategies have been tested (product, sum, minimum, the CUAVE database and our Arabic audiovisual database.
maximum, vote ...) and all show a significant improvement In order to achieve this recognition system we have used⅔
of the data for the learning stage and the remaining ⅓ to test
in results compared to the consideration of a single
the effectiveness of our system.
modality, which conduct us to focus in this work on the use The proposed AVASR system using RASTA-PLP as
of the model of separate fusion, i.e. the fusion scores from audio features extraction, DCT coefficients as visual
each recognizer GA/HMM. Their sets of log-likelihood can features extraction, and a GA/HMM for audio-visual speech
modeling, was implemented as described in the previous
be combined using weights that reflect the reliability of each
particular flow, the combined scores then take the following sections on Matlab simulation software. As a result, and by
form (Islam and Rahman, 2010): integrating the first and second derivative of the parameters,
we obtain matrix of 27 parameters for the audio stream after
the segmentation of the sampled speech data into 0.025
AV A V seconds frames with an overlap of 0.010 seconds. The
log P(o ∨λ)=w A log P( o ∨λ A )+w V log P(o ∨λ V ) GA/HMM recognizers were built using the Hidden Markov
(8) Model Toolkit (HTK) as in (Young et al. 2005).
In Table 1 and Table 2, we have presented various kinds
Where λ Aand λ V are the acoustic and the visual of instance with different GA control parameters that have
GA/HMMs respectively and log P(o ∨λ A ) and
A been solved with our algorithm to evaluate performance of
the proposed system. We ran each instance 15 times with a
log P(o V ∨ λV )are there log-likelihood. The reliability of different number of clusters, crossover probability values
each modality can be calculated by the most appropriate and between 0.5-0.9, and mutation probability with the value
best in performance (Islam and Rahman, 2010), the average 0.01, and obtained the maximum P(o|λ) values after 50
difference between the maximum log-likelihood and the generation.
other ones, can be found as,

Table 1 GA parameters training HMM for audio-only: (a) AVARB database (b) CUAVE database
Number Pc Pm Average Number Pc Pm Average
of P(o|λ) of P(o|λ)
clusters clusters
3 0.5 0.01 -2.3630 3 0.5 0.01 -3.7416
5 0.6 0.01 -1.5838 5 0.6 0.01 -3.2604
7 0.7 0.01 -1.1396 7 0.7 0.01 -3.4235
9 0.8 0.01 -3.3185 9 0.8 0.01 -3.9134
12 0.9 0.01 -4.0122 12 0.9 0.01 -4.3637
 (a) (b)
Table 2 GA parameters training HMM for video-only: (a) AVARB database (b) CUAVE database
Number Pc Pm Average Number Pc Pm Average
of P(o|λ) of P(o|λ)
clusters clusters
3 0.5 0.01 -7.7629 3 0.5 0.01 -5.1860
4 0.5 0.01 -7.0046 5 0.6 0.01 -5.2987
7 0.8 0.01 -7.1555 7 0.7 0.01 -5.4743
9 0.8 0.01 -7.6595 9 0.8 0.01 -5.8747
12 0.9 0.01 -7.8234 12 0.9 0.01 -6.0890

(a)
We observe that the results are varied according to the
parameters training of the GA, also to the number of
clusters obtained by the vector quantization phase, e.g.
with a 7 clusters, Pc=0.7 and Pm= 0.01, for the Arabic
audio database and 5 clusters, Pc=0.6 and Pm= 0.01 for the
visual Arabic database are superior to all the other
approaches in all cases. Therefore, we use theme in our
(b)
GA/HMM. The same thing for the CUAVE audio
database with 4 clusters, Pc=0.6 and Pm= 0.01, and for the
visual CUAVE database the best performance is obtained
with 3 clusters, Pc=0.5 and Pm= 0.01.
Figures 9 and 10 give the rate of recognition with
respect to the number of clusters used in the experiment.

Figure 9 Comparison of recognition rates of audio-only, video-only and audio-visual ASR for the CUAVE database by using: (a) an
HMM-based and (b) GA/HMM

(a) (b)

Based on Figure 9, we can see that the recognition rates
obtained with our GA/HMM are better in most cases
compared to those obtained with the standard HMM. The
above figures also indicate that the AVASR system
outperform significantly overall by achieving the highest
recognition rates.
For the CUAVE database the results show that the
average rate of recognition achieved a best rate with
86.8% using standard HMM recognizer with 5 clusters for
the clustering phase, and 98.1% using GA/HMM
recognizer with 3 clusters.

Figure 10 Comparison of recognition rates of audio-only, video-only and audio-visual ASR for AVARB database by using: (a)an HMM-
based and (b)GA/HMM
 (a)
In Figure 10, we have noted almost the same previous
observations with our AVARB database, i.e. we found the
best average rate of recognition equal to 93.7% and 97.6%
by the use of standard HMM and hybrid GA/HMM
recognizers respectively, and with 7 clusters for both.
More generally we found a percentage increase varying
from almost 5% to 28% of our test results, but this rise in
the recognition rates given is not fixed, along with the
increase in the size of the population. It could be that
given rates worse or the same standard HMM system
before optimizations. This is due to the characteristic of
the GA method that is random and this system using the
standard general replacement process.
5 Conclusions
In this primary work, we presented a new AVASR system,
which uses DCT coefficients as visual features extraction,
RASTA-PLP as acoustic features extraction. To overcome
the drawbacks of the traditional training of HMM using
Baum-Welch algorithm which are mainly the converge to
a local optimum which is the closer to the starting values
of the optimization procedure and the estimation of the
recognizer parameters that requires a more careful
examination, we have chosen to use an hybrid GA/HMM
algorithm for modeling the audio-visual speech.
Two databases were used to experiment our AVASR
system: The CUAVE database and our AVARB database.
From the several test results, we conclude that the
system modeled by HMM and trained by our GA/HMM
training have higher rates of recognition than the HMM
trained by the Baum-Welch algorithm.
For future work, we are planning to cover more issues
about improving the performance of the proposed system.
Finally, we also intend to test our system with other
alternatives methods recognition and compare them with
our proposed GA/HMM algorithm.
References
(b)
Potamianos, G., Neti, C., Gravier, G., Garg, A. and Senior A.W.
(2003) ‘Recent advances in the automatic recognition of
audiovisual speech’, Proceedings of the IEEE, Vol. 91, No.
9, pp.1306–1326.
Man, K.F., Tang, K.S. and Kwong, S. (1996) ‘Genetic
Algorithms Concepts and Applications’, IEEE Transactions
on Industrial Electronics, Vol. 43, No. 5, pp.519–532.
McGurk, H. and MacDonald, J. (1976) ‘Hearing lips and seeing
voices’, Nature 264 (5588), pp.746–748.
Petajan, E.D. (1984) ‘Automatic lipreading to enhance speech
recognition’,Proceedings of the IEEE Communication
Society Global Telecommunications Conference, Atlanta,
Georgia.
Elmahdy, M. and Gruhn, R. (2009) ‘Modern standard Arabic
based multilingual approach for dialectal Arabic speech
recognition’, Eighth international symposium on natural
language, pp.169–174.
Iwano, K., Yoshinaga, T., Tamura, S. and Furui, S. (2007)
‘Audio-Visual Speech Recognition Using Lip Information
Extracted from Side-Face Images’, EURASIP Journal on
Audio, Speech, and Music Processing, Vol. 2007, No. 1,
pp.4–4.
Zhi-yi, Q. Yu, L. Li-hong, Z. and Ming-xin, S. (2006) ‘Hybrid
SVM/HMM architectures for speech recognition’,
Proceedings of the First International Conference on
Innovative Computing, Information and Control , Vol. 2,
pp.100–104.
Zhao, G., Pietikäinen, M. and Hadid, A. (2007) ‘Local
Spatiotemporal Descriptors for Visual Recognition of
Spoken Phrases’, Proceedings of the international workshop
on Human-centered multimedia, pp.57–66.
Pao, T.L., Liao, W.Y., Wu, T.N. and Lin, C.Y.
(2009)‘Automatic Visual Feature Extraction for Mandarin
Audio-Visual Speech Recognition’, Proceedings of the
IEEE International Conference on Systems, Man and
Cybernetics, San Antonio, TX, USA, pp.2936–2940.
Nock, H.J., Iyengar, G. and Neti, C. (2003) ‘Speaker localisation
using audio-visual synchrony: An empirical study’, In
Proceedings of International Conference on Image and video
retrieval (CIVR), Vol. 2728, pp.488–499.
Patterson, E.K., Gurbuz, S., Tufekci, Z. and Gowdy, J.N. (2002)
‘Moving-talker speaker-independent feature study and
baseline results using the CUAVE multimodal speech
corpus’, EURASIP Journal on Applied Signal Processing,
Vol. 11, pp.1189–1201.
Reikeras, H., Herbst, B.M., du Preez, J. and Engelbrecht, H.
(2010) ‘Audio-Visual Automatic Speech Recognition using
 Dynamic Bayesian Networks’, Proceedings of the Twenty- International Symposium in Information Technology (ITSim),
First Annual Symposium of the Pattern Recognition Vol. 2, pp.542–545.
Association of South Africa, pp.233–238. Kanungo, T., Netanyahu, N.S. and Wu, A.Y. (2002) ‘An
Galatas, G., Potamianos, G. and Makedon, F. (2012) ‘Audio- Efficient k-Means Clustering Algorithm: Analysis and
visual speech recognition using depth information from the Implementation’, IEEE Transactions on Pattern Analysis
Kinect in noisy video conditions’, Proceedings of the 5th and Machine Intelligence, Vol. 24, No. 7, pp.881–892.
International Conference on PErvasive Technologies Related Rogozan,A., Deléglise, P. and Alissali, M. (1997) ‘Adaptive
to Assistive Environments (PETRA’12), Crete, Greece, determination of audio and visual weights for automatic
Article No. 2. speech recognition’, ESCA Workshop on Audio-Visual
Minotto, V.P., Lopes, C.B.O., Scharcanski, J., Jung, C.R. and Speech Processing (AVSP'97), pp.61–64.
Lee, B. (2013) ‘Audiovisual Voice Activity Detection Based Islam, R. and Rahman, F. (2010) ‘Likelihood Ratio Based Score
on Microphone Arrays and Color Information’, IEEE Fusion for Audio-Visual Speaker Identification in
Journal of Selected Topics in Signal Processing, Vol. 7, No. Challenging Environment’, International Journal of
1, pp.147–156.
Computer Applications, Vol. 6, No. 7, pp. 6–11.
Rabiner, L. and Juang, B. (1993) ‘Fundamentals of Speech
Recognition’, Prentice-Hall, Englewood Cliffs, NJ. Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu,
Xue-ying, Z., Yiping, W. and Zhefeng, Z. (2007) ‘A Hybrid X.A., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev,
Speech Recognition Training Method for HMM Based on V. and Woodland P. (2005) ‘the HTK Book’ (for HTK
Genetic Algorithm and Baum Welch Algorithm’, IEEE 2nd Version 3.4).
International conference on Innovative Computing,
Information and Control (ICICIC’07), pp.572.
Pérez, Ó, Piccardi, M. and García, J. (2007) ‘Comparison
between genetic algorithms and the Baum-Welch algorithm
in learning HMMs for human activity classification’,
Proceeding ofEvoWorkshops’7, pp.399–406.
Goh, J., Tang, L. and Al turk, L. (2010) ‘Evolving the Structure
of Hidden Markov Models for Micro aneurysms Detection’,
UK Workshop on Computational Intelligence (UKCI), pp.1–
6.
Viola, P.A. and Jones, M.J, (2001) ‘Rapid Object Detection
using a Boosted Cascade of Simple Features’, Proceedings of
the 2001 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR’01),Vol. 1,pp.511–
518.
Papageorgiou, C.P., Oren, M. and Poggio, T. (1998) ‘A General
Framework for Object Detection’, Sixth International
Conference on Computer Vision (ICCV'98), pp.555–562.
Khandait, S.P., Khandait, P.D. and Thool, Dr.R.C. (2009) ‘An
Efficient Approach to Facial Feature Detection for
Expression Recognition’, International Journal of Recent
Trends in Engineering, Vol. 2, No. 1, pp.179–182.
Potamianos, G., Neti, C., Luttin, J. and Matthews I. (2004)
‘Audio-visual automatic speech recognition: an overview, in
issues in audio-visual speech processing’,Issues in Visual
and Audio-Visual Speech Processing,G. Bailly, E. Vatikiotis-
Bateson, and P. Perrier, eds., MIT Press,Chapter 10.
Harris, F.J. (1978) ‘On the use of windows for harmonic analysis
with the discrete Fourier transform’. Proceedings of the
IEEE, Vol. 66, pp.51–83.
Hermansky, H., Morgan, N., Bayya, A. and Kohn P. (1992)
‘RASTA-PLP Speech Analysis’, IEEE International
conference on Acoustics, speech and signal processing, Vol.
1, pp.121–124.
Gupta, M. and Garg, Dr.A.K. (2012) ‘Analysis of image
compression algorithm Using DCT’, International Journal of
Engineering Research and Applications (IJERA), Vol. 2, No.
1, pp.515–521.
Fujiwara, Y., Sakurai, Y. and Yamamuro, M. (2008) ‘SPIRAL:
efficient and exact model identification for hidden Markov
models’, In Proceeding of the 14th ACM SIGKDD
international conference on Knowledge discovery and data
mining KDD, pp.247–255.
Sun, K. and Yu, J. (2007) ‘Video Affective Content Recognition
Based on Genetic Algorithm Combined HMM’, Proceedings
of the 6th international conference on Entertainment
Computing, pp. 249 – 254.
Oudelha, M. and Ainon, R.N. (2010) ‘HMM parameters
estimation using hybrid Baum-Welch genetic algorithm’,