2013 IEEE 9th International Conference on Mobile Ad-hoc and Sensor Networks
Tongue Rehabilitation Training Method of Hearing-
impaired Child Based on Visualization Model
Shi Lijuan1.2 An Zhiyong1 Zhao Jian 3
1 College of Opto-electronic Engineering
Changchun University of Science and Technology
Changchun, Jinlin ,China
2 College of Electronic Information and Engineering
Changchun University
Changchun, Jilin,China
e-mail: shilijuan.ccu@gmail.com
Abstract—launched from the establishment of 3D tongue model
in this thesis, the tongue rehabilitation training method of
hearing-impaired child combined by 3D modeling technology and
sensor technology is applied and the method can reveal the
tongue motor process of average person when pronouncing,
guide the hearing-impaired child to do tongue pronunciation
training, overcome the problem of non-visibility of tongue
training process in speech training of hearing-impaired child,
evaluate rehabilitation training degree and cover the shortage of
no feedback diagnosis in the rehabilitation training of hearing-
impaired so as to enhance the efficiency of hearing-impaired
child pronunciation training and accuracy of pronunciation
training.
Keywords—Hearing-impaired Child; Tongue Training;3D
Modeling Technology; Visualization Model
I. INTRODUCTION
Due to hearing disorder, the language competence of
hearing-impaired child lags far behind the sane child of the
same age and the words of hearing-impaired child can not be
understood by average people or are difficult to understand and
thus language rehabilitation training should be conducted on
hearing-impaired child, it has been proved that most of the
hearing-impaired children can grasp verbal language through
wearing appropriate assisted listening devices and scientific
rehabilitation training. The language rehabilitation training of
hearing-impaired child includes Ԙ respiratory training; ԙ
Tongue training; Ԛ lip training; ԛ nasal training; Ԝ throat
training;ԝphoneme training;Ԟspelling training;ԟfour tones
training[1] and tongue is both an important part of vocal organ
and the most flexible part in oral organ and also it plays an
978-0-7695-5159-3/13 $31.00 © 2013 IEEE
DOI 10.1109/MSN.2013.89
Wang Lirong2,* Du Qinsheng3 Ma Lidan4
3 College of Computer Science and Technology
Changchun University
Changchun, Jilin,China
4 Patent Searching and Consulting Center
State Intellectual Property Office of People’s Republic
China
Beijing, China
Corresponding author: wanglirong.ccu@gmail.com (Wang
Lirong)
important role in pronunciation. Due to hearing loss, obstructed
language development, small tongue movement extent, ossified
lingualis, inflexible rotation, incapability to keep up with
language beat and pronounce or inaccurate pronunciation,
tongue training is an important link of language rehabilitation
training and the basis for learning manner of articulation and
articulation skill and developing favorable pronunciation habits
so as to lay good foundation for speech training of hearing-
impaired child[2,3]. The key aspect of tongue training is
tongue movement and the traditional tongue training method is
to organize the hearing-impaired children to do tongue exercise,
such as stretching tongue, resisting cheek and overturning
tongue so that the children can improve tongue stretching,
flexibility, limpness and validity. However, such training mode
is boring and makes the children boring and thus they lose
patience for training, therefore, interestingness should be added
in training process and visual stimulation should be given to
children through role[4,5,6], dialogue and plot so that the
children can speak and play while working and the process
should be simple and amusing and inconvenience and baldness
should be avoided. In addition, the traditional training method
has no feedback and thus the rehabilitation training degree and
effect can not be grasped[7]. Hence, a kind of tongue training
method of hearing disorder children is studied based on
visualization model in this thesis[8], 3D modeling technology
and sensor technology are combined in this method and
authentic motion model of tongue in the oral cavity is
established to simulate the motion state of tongue and its fitting
degree with the upper jaw when average person pronounces so
as to guide the hearing-impaired child to exercise
pronunciation and overcome the non-visibility problem of
tongue training process in speech training of hearing-impaired
child; besides, the system can evaluate rehabilitation training
degree and cover the shortage of no feedback diagnosis in
rehabilitation training of hearing-impaired child so as to
enhance the efficiency of pronunciation training of hearing-
impaired child and accuracy of pronunciation training.
481
 II. OVERALL DESIGN SCHEME parameters of geometrical model and texture map of
front and back side of the tongue so as to conduct
In the study, 3D modeling technology and sensor texture matching.
technology are combined to establish the standard database
based on visual feedback model and suitable for language In figure, “l”is the length, “w” width, “h” is the
training of hearing-impaired child; besides, sensor and audio thickness, “muz”is the On the longitudinal muscle,
technology are combined to analyze and guide the tongue “mdz”is the Under longitudinal muscle, “mh”is the
movement of pronunciation process of hearing-impaired child Transverse muscle of tongue ,“mc”is the Vertical
based on the establishment of standard database so as to help muscle and “mk”is the chin Genioglossus.
the hearing-impaired child to pronounce correctly during the x Determine feature points
language competence recovery process. The integrated system
design is shown in figure 1. In order to increase flexibility of tongue movement,
part of the points are added on the basis of tongue
feature points defined in MPEG-4 Standard and the
number of the points is determined according to the
Audio Audio
information information
accuracy of actual demand[10].
Hearing- collection process Data x Transform 3D standard grid through tongue feature
impaired Fusion point set and establish 3D grid model of tongue[11].
Child process
Sensor
Tongueǃjaw
information x Acquire the tongue model with third dimension
sensor through texture mapping.
analysis
Tongue Model Generation Scheme is shown in figure 3.
Evaluation results
Standard database
output

Determine the Feature point A three-

geometric model selection and dimensional
Figure 1 integrated system design
of human tongue calibration grid

A. Tongue Model Establishment

The purpose of tongue training is to enhance motor
sensibility of intrinsic lingual muscles and enhance flexibility
of tongue, such as tongue stretching, shrinking, resisting,
rolling and flipping; a tongue model with high naturalness Get tongue Texture Model generates
texture Mapping tongue
should be established based on flexibility of tongue movement
to simulate tongue movement.
x Establishing 3D corpus linguae model[9] Figure 3 Tongue Model Generation Scheme

B. Tongue Pronunciation Information Collection

Figure 2 Tongue geometry Model
Determine the geometrical model of the main body
of tongue, confirm the upward and downward parts of
tongue and lateral view according to the features of 3D
model to obtain the length, width, thickness and other
Palate sensor technology is applied for tongue
pronunciation information collection with 126 gilded palate
sensor evenly distributed in network structure; when tongue
contacts with these electrodes, the dedicated collecting device
can collect the contact condition between tongue and palate
prostheses (the data can reveal the position of tongue in the
palate when speaking), extract 3D dynamic parameter
information of tongue and palate when pronouncing[12,13],
verify the parameter information according to the checking
message of tongue and palate of the corresponding model
extracted in advance and then acquire the 3D parameters
information. Deposit the 3D parameters into 3D visual
dynamic speech database. Extraction of 3D Tongue
Information is shown in figure 4.

482
 x if unqualified, encourage the hearing-impaired child
and replay the correct 3D tongue model pronunciation
process, infer the mistakes during pronunciation
process through video analysis and the system return
to the last training link or choose to log out.
During the whole learning process, the system can
establish trainee database, record the learning situation of
trainee, and summarize and appraise the pronunciation
process of the trainee and identify the language training
level of the trainee through evaluation.
The system flow is shown in figure 5.

Into the system

Select tongue
pronunciation training
sessions

Showing tongue to
pronounce process model

Figure 4 Extraction of 3D Tongue Information

Hearing-impaired
children pronunciation
III. SYSTEMATIC DESIGN AND FLOW exercises

Divide the hearing-impaired child tongue training into

several levels, such as vowel training, consonant training and Reward
tone training. The hearing-impaired child can choose a training Record user The sensor and the audio
information analysis
And record user
informati information
link according to the need, the system inputs the necessary
phoneme unit in the corresponding language training level and
takes parameters from the parameter library to drive the tongue
model, play the tongue model pronunciation,process and
Quality or
pronunciation of the information on the screen and the qualified
pronunciation process is completed through driving 3D tongue Judgment to assess
model by parameters.
the hearing-impaired child do pronunciation exercises
according to the pronunciation process displayed on the screen, Unqualified
the system collects the audio information and sensor
information of the practice progress of hearing-impaired child
Encourage and Quit
and analyzes through sensor technology and speech recognition Quit prompt an error
technology and then evaluates the pronunciation of hearing-
impaired child through judgment module: excellent, qualified,
unqualified:
Figure 5 System Flow Diagram
x if qualified, then encourage the student to continue
exercising and strive for excellent standard; then, the
hearing-impaired child can choose to continue IV. CONCLUSION
exercising or return to the training content just now; if
do not want to continue studying the content, go to the
Tongue is an important part of vocal organ and plays an
next training link.
important role in pronunciation; in this thesis, 3D visual
x if excellent, reward the hearing-impaired child, such as Tongue model and the tongue rehabilitation training method of
a video or cartoon and encourage the hearing-impaired hearing-impaired child based on this model are established; in
child to continue exercising; if continue learning, go to the study, vivid D model is applied to guide the hearing-
the next training link and the system return to the last impaired child to do tongue pronunciation training so that the
training link or choose to log out. hearing-impaired child can understand the action sequence of

483
tongue pronunciation from all directions and let the hearing-
impaired child to do pronunciation exercise from the basic
morphemes; the interest of hearing-impaired child can be
mobilized through incentives, which can shorten the time for
pronunciation training of hearing-impaired child and enhance
the efficiency of pronunciation training. At the same time,
sensor and audio feedback methods are applied in the thesis to
evaluate the pronunciation training of hearing-impaired child
and record the hearing-impaired child training process and
training effect according to the characteristics of hearing-
impaired children. The research is suitable for both
rehabilitation institution and household use to enhance
rehabilitation effect and reduce the cost of rehabilitation
training; besides, full play should be given to the educational
potential of parents, which has become an effective means for
enhancing the training amount and quality of hearing-impaired
child.
ACKNOWLEDGMENT
This work was supported in part by China Postdoctoral Scie
nce Foundation under Grant No. 2013M541298; The Fund of
Ministry of Education of China No.Z2012062; Jilin Province S
cience and Technology Development Plan under Grant No. 201
101088,20130522162JH; JiLin Province Education Developme
nt Plan under Grant Nos. 2013278, 2013285,2011223.
REFERENCES
[1] http://www.chinadeaf.org/.
[2] Yang lina,Investigation and Analysis on the Role of Language
Environment in DeafChildren’s Phone Gaining, Master's Thesis Ji lin
University,pp:3-5.2011.
484
[3] Ding Yingchun, A Contrast Study on the Chinese Intonation Acoustic
Characteristics between Healthy and Hearing-impaired Children, ,pp:15-
18,Master's Thesis, East China Normal University.2011.
[4] Ji Jingyuan,Huang Zhaoming,Development of multimedia technology in
aural rehabilitation and deaf education,pp:44-46,China Scientific Journal
of Hearing and Speech Rehabilitation.no.1,2005.
[5] Zhao Jian, Wang Lirong, Zhang Chao,Shi Lijuan,Yin Jia: Pronouncing
Rehabilitation of Hearing-impaired Children based on Chinese 3D
Visual-speech Database [C]. In Proceeding of the The 5th International
Conference on Frontier of Computer Science and Technology
(FCST2010).Changchun: 2010, pp:625-630.
[6] Yin Baocai,Wangkai,Wang Lichun,A Synthesis Method of Three-
dimensional Facial Animation,JOURNAL OF BEIJING UNIVERSITY
OF TECHNOLOGY.Vol.37 No.2,pp:266-271, Feb.2011.
[7] Zhao Jian , Wang Lirong, Zhang Chao,Shi Lijuan,Yin Jia: Pronunciation
of Rehabilitation Methods Based on 3D-Talking Head [C]. In
Proceeding of the International Conference on Modeling and
Computation in Engineering (CMCE 2010).Changchun: 2010, pp:17-20.
[8] Li Gang, Wang Mengjun, Lin Ling,Mandarin Chinese Visua-l speech
Database for Speech-impaired People, Chinese Journal ofBiomedical
Engineering, Vol.26 No.3, pp:355-360,June 2007.
[9] Chen Zhixiang.Researeh of Virtual TOngue MOVementS and MOUth
ExPressions Animation University of Seienee and Teehnology of China
A dissertation for doctor’s degree pp:16-17.
[10] With Multiple Elements Blending Based on MPEG-4.
[11] Chen Zhixiang, Cheng Yiming, Zeng Dan, et al. 3D muscle-controlled
tongue and mouth model. Journal of the Graduate School of the Chinese
Academy of Sciences, 2008, 25( 3) : 372~ 378.
[12] Jian Zhao, Wang Lirong, Zhang Chao,Shi Lijuan,Yin Jia: Pronunciation
of Rehabilitation Methods Based on 3D-Talking Head [C]. In
Proceeding of the International Conference on Modeling and
Computation in Engineering (CMCE 2010).Changchun: 2010, pp:17-20.
[13] Shi Lijuan , An Zhiyong,Zhao Jian㧘Wang Lirong ,Du Qinsheng: The
3D Face Feature Extraction and Driving Method on Pronunciation
Rehabilitation for Impaired Hearing [C]. In Proceeding of the 14th IEEE
Int. Conf. on Computational Science and Engineering, (CSE 2011). ,
pp:547-550,Dalian,China: 2011.
.