Department of Robotics and Mechatronics Engineering

RME 410
Project / Dissertation
Deep Model Based Autism Spectrum Disorder (ASD)
Detection through Activity Recognition
Presented by- Supervised by-
Omar-Ibne-Shahid Dr. Sejuti Rahman
Exam Roll: 413 Assistant Professor,
Registration ID: 2015-716-909 Department of Robotics
and Mechatronics Engineering,
University of Dhaka

5 January, 2020
 2
Outline:
 Introduction
 Objectives
 Related Works
 Dataset Description
 Methodology
 Comparative Results
 Conclusions
 3
Introduction(1/8)

Autism, also called autism

spectrum disorder (ASD), is a What is
complicated condition that Autism
includes problems with Spectrum
communication and
behavior. It can involve a Disorder
wide range of symptoms and (ASD)
skills.

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 4
Introduction(2/8)
Surveillance Birth Combined Prevalence This Is About
Year Year (per 1000 children) 1 in X • ASD is reported to occur
Children... in all racial, ethnic, and
2000 1992 6.7 (4.5-9.9) 1 in 150 socioeconomic groups
2002 1994 6.6 (3.3-10.6) 1 in 150
• ASD is about 4 times State of
more common among
2004 1996 8.0 (4.6-9.8) 1 in 125
boys than among girls
Autism
2006 1998 9.0 (4.2-12.1) 1 in 110 • Studies in Asia, Europe, in global
2008 2000 11.3 (4.8-21.2) 1 in 88 and North America have
2010 2002 14.7 (5.7-21.9) 1 in 68 identified individuals scenario
2012 2004 14.5 (8.2-24.6) 1 in 69 with ASD with an
average prevalence of
2014 2006 16.8 (13.1-29.3) 1 in 59
between 1% and 2%

Ref.1 “Centers for disease control and prevention.” https://www.cdc.gov/ncbddd/ autism/data.html, 2014.

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 5
Introduction(3/8)
Rough assumption (by IPNA, BSMMU):
• In Bangladesh, nearly 10.5 lakhs
individuals may have autism
• Prevalence on autism in Bangladesh is Situation
8.5/ 1000 population
• Out of every 94 boys, one is in our
affected by Autism. For girls, it is one country
in every 150
• 30/ 1000 in Dhaka city and 7/ 10,000 in
rural area

Ref.2 F. Rahman, S. Akhter, A. Biswas, and A.S. Abdullah, “Study of prevalence of autism in
Bangladesh,” 2016.
Related Dataset Comparative
Introduction Objectives Methodology Conclusion
Works Description Results
 6
Introduction(4/8)
MRI Image Analysis

ASD
Detection
Techniques

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 7
Introduction(5/8)
Screening by experts

1. Developmental Screening
Developmental screening is a short test to
tell if children are learning basic skills at ASD
proper time Detection
2. Comprehensive Diagnostic Evaluation Techniques
Analyze child’s behavior, expressions,
interaction and relation with others

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 8
Introduction(6/8)

• Assist victims in their

daily activities
• Give proper attention Significance
• Aware of their special of ASD
needs
Detection
• Never pressurize

• Time to time screening

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 9
Introduction(7/8)
 There is no recognized standard
automated method

 Screening depends on expert physician

Limitations of
 Can’t be detected by parents or non-
expert people Conventional
Detection
 Taking MRI of child is very tough
System
 Level detection differs from person to
person

 Whole process is manual

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 10
Introduction(8/8)

Future
Prospects

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 11
Objectives(1/3)

Automated
System

ASD vs. TD
Related Dataset Comparative
Introduction Objectives Methodology Conclusion
Works Description Results
 12
Objectives(2/3)

Autistic child perform

same action with Action Style
different style in Detection
response with
environmental difference

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 13
Objectives(3/3)

Applying
Deep
Model

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 14
Related Works(1/13)

What is
Human activity recognition is
an ability to interpret human Human
body gesture or motion via Activity
sensors and determine human
activity or action Recognition
(HAR)

Ref.3 M. J. Roshtkhari and M. D. Levine, “Human activity recognition in videos using a single example,”
Image and Vision Computing, vol. 31, no. 11, pp. 864–876, 2013

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 15
Related Works(2/13)

Classification
of HAR

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 16
Related Works(3/13)
Group Activities
Daily activities Brushing teeth, cleaning floor,
sleeping, cooking, having lunch,
drinking water, vacuuming
Transportation Riding a bus, driving car, cycling.
 Surveillance System
Ambulation Walking, running, sitting, standing Application
still, lying, climbing stairs
 Health Care
Fitness/exercises Rowing, lifting weights, spinning,
Domain
Nordic walking, doing push ups  Human-Computer Interaction of HAR
Military Crawling, kneeling, situation
assessment, opening a door
Nursing activities Patient sitting, measure blood
pressure, ECG, Attach bust bandage

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 17
Related Works(4/13)

ASD child may show difference by ASD

doing some unusual action, different
types of facial expression, exceptional
Detection
gait patterns, abnormal gaze patterns. and
HAR

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 18
Related Works(5/13)
Repetitive Behaviors

 Rocking back and forth in the chair

 Hands stemming Identifying

 Toe walking characteristics
 Jumping to classify
 Running in circles ASD

Ref.4 O. Rihawi, D. Merad, and J.-l. Damoiseaux, “3D-AD: 3D-autism dataset for repetitive behaviors with
Kinect sensor,” in 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance
(AVSS), pp. 1–6, IEEE, 2017

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 19
Related Works(6/13)
Abnormal Gaze Pattern

ASD child exhibit attentional Identifying

biases in social situations, also
known as visual social attention, characteristics
that differ significantly from typical to classify
developed (TD) individuals
ASD

Ref.5 Q. Guillon, N. Hadjikhani, S. Baduel, and B. Rog´e, “Visual social attention in autism spectrum disorder:
Insights from eye tracking studies,” Neuroscience & Bio-behavioral Reviews, vol. 42, pp. 279–297, 2014

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 20
Related Works(7/13)
Unusual Gait Pattern

A wide range of abnormal gait

patterns in various aspects of gait
Identifying
parameters such as basic gait characteristics
measurements, kinematic joint
angles, and kinetic joint moments to classify
during walking in ASD
individuals with ASD

Ref.6 D. Kindregan, L. Gallagher, and J. Gormley, “Gait deviations in children with autism spectrum disorders:
a review,” Autism research and treatment, 2015

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 21
Related Works(8/13)

 Unavailability of Datasets
 Dataset Diversity
 Data Annotation
Challenges
 Experimental Setup
in this
 Capturing Long Context domain
 Privacy Issues

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 22
Related Works(9/13)
Video Dataset Zunino et al.
It presents the first video dataset
composed by simple gestures (bottle
grasping) performed by ASD and TD
children
Available
Video
Datasets

Ref.7 A. Zunino, P. Morerio, A. Cavallo, C. Ansuini, J. Podda, F. Battaglia, E. Veneselli, C. Becchio, and V. Murino, “Video
gesture analysis for autism spectrum disorder detection,” in 2018 24th International Conference on Pattern Recognition
(ICPR), pp. 3421–3426, IEEE, 2018

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 23
Related Works(10/13)
3D CNN

Popular
3D convolutions applies a 3 Models
dimensional filter to the dataset
and the filter moves 3-direction to Classify
(x, y, z) to calculate features Video
Gestures

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 24
Related Works(11/13)
CNN LSTM Output

Popular
Models
VGG-16 to Classify
Video
Gestures

Ref.8 K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image
recognition,” arXiv preprint arXiv:1409.1556, 2014
Related Dataset Comparative
Introduction Objectives Methodology Conclusion
Works Description Results
 25
Related Works(12/13)
CNN LSTM Output

Popular
Models
ResNet-50 to Classify
Video
Gestures

Ref.9 K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), 2016

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 26
Related Works(13/13)
CNN LSTM Output

Popular
Models
Inception v3 to Classify
Video
Gestures

Ref.10 C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for
computer vision,” IEEE conference on computer vision and pattern recognition, pp. 2818–2826, 2016.

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 27
Dataset Description(1/4)
fMRI (functional Magnetic Resonance Imaging)

Foundation

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 28
Dataset Description(2/4)
Same Activity with Different Intention

Bottle grasping
Activity
1. grasp-to-place
2. grasp-to-pour
3. grasp-to-pass to place
4. grasp-to-place to pour

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 29
Dataset Description(3/4)

Experimental
Setup

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 30
Dataset Description(4/4)

ASD child (Annotated by DSM-5) : 20 ( 18 males, 2 females)

TD child (Annotated by IQ score) : 20 ( 16 males, 4 females) Subject

Description

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 31
Methodology(1/10)
• Extracting frames with 30 fps

• Apply Gaussian filter to reduce details of visual appearance

Data
Preparation

Original Filtered

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 32
Methodology(2/10)
• Classified dataset-

4 Action class-
1. grasp-to-pour
2. grasp-to-place Data
3. grasp-to-pass to pour
4. grasp-to-pass to place
Preparation
2 class-
1. ASD
2. TD

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 33
Methodology(3/10)
Input

VGG-16/
ResNet-50/
Inception v3
Approaches
FC Layer

LSTM

3D CNN
Output
CNN + LSTM
Related Dataset Comparative
Introduction Objectives Methodology Conclusion
Works Description Results
 34
Methodology(4/10)

3D
CNN

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 35
Methodology(5/10)
Simple 3D CNN

• Number of epoch: 10
• Batch size: 56 3D
• Loss function:
Categorical Cross-
CNN
entropy
• Optimizer: Adadelta
• Learning rate decay:
0.95

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 36
Methodology(6/10)
Modified 3D CNN

3D
CNN

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 37
Methodology(7/10)

Transfer
Learning

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 38
Methodology(8/10)

Transfer
Learning

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 39
Methodology(9/10)

Transfer
Learning

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 40
Methodology(10/10)
Hyper Parameters:
Number of epoch: 12
Batch size: 56 Transfer
Loss function: Categorical Cross-entropy Learning
Optimizer: Adam
Learning rate decay: 0.95

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 41
Comparative Results(1/7)
Accuracy
Model Train Test
(%) (%)
Accuracy

Simple
3D CNN
43.34 37.18
4 Action
Modified 63.67 58.39 Class
3D CNN

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 42
Comparative Results(2/7)
Accuracy
Model Train Test
(%) (%)
Accuracy

VGG-16 70.31 67 4 Action

Resnet-50 73.10 67.23
Inception- 75.18 69.57
Class
v3

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 43
Comparative Results(3/7)

Previous
Work

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 44
Comparative Results(4/7)

Previous
Work

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 45
Comparative Results(5/7)
Accuracy
Model Train Test
(%) (%)
Accuracy

Simple 49 44 2 Class
3D CNN
Modified 68 60.35
(ASD & TD)
3D CNN

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 46
Comparative Results(6/7)

Accuracy
Model Train Test
(%) (%)
Accuracy

VGG-16 72.23 69 2 Class

Resnet-50 76.38 71 (ASD & TD)
Inception- 78.62 73
v3

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 47
Comparative Results(7/7)
Best Approach:
Feature Extraction using Inception v3 + Single LSTM network

Classification using Train Test

Decision
Inception v3 Accuracy Accuracy
4 Action Class 75.18 69.57
2 class (ASD & TD) 78.62 73

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 48
Conclusion(1/3)

 Our work mainly focuses on detecting ASD and TD child

 Our previous work classify style of the activities of ASD Contributions

and TD differently
 Our work analyses various methods of classification and
their comparative results

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 49
Conclusion(2/3)

 Lower Accuracy
 Can’t Predict Autism Level Limitations
 Applicable for Experimental Constrained Dataset

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 50
Conclusion(3/3)

 Enlarging Dataset
 Autism Level Annotation using DSM-5 Screening Future
 Train Model from Scratch Scopes
 Proper Tuning of Parameters

Related Dataset Comparative

Introduction Objectives Methodology Conclusion
Works Description Results
 51
References:
1 “Centers for disease control and prevention.” https://www.cdc.gov/ncbddd/ autism/data.html, 2014
2 F. Rahman, S. Akhter, A. Biswas, and A.S. Abdullah, “Study of prevalence of autism in Bangladesh,” 2016
3 M. J. Roshtkhari and M. D. Levine, “Human activity recognition in videos using a single example,” Image and Vision Computing, vol.
31, no. 11, pp. 864–876, 2013
4 O. Rihawi, D. Merad, and J.-l. Damoiseaux, “3D-AD: 3D-autism dataset for repetitive behaviors with Kinect sensor,” in 2017 14th
IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6, IEEE, 2017
5 Q. Guillon, N. Hadjikhani, S. Baduel, and B. Rog´e, “Visual social attention in autism spectrum disorder: Insights from eye tracking
studies,” Neuroscience & Bio-behavioral Reviews, vol. 42, pp. 279–297, 2014
6 D. Kindregan, L. Gallagher, and J. Gormley, “Gait deviations in children with autism spectrum disorders: a review,” Autism research
and treatment, 2015
7 A. Zunino, P. Morerio, A. Cavallo, C. Ansuini, J. Podda, F. Battaglia, E. Veneselli, C. Becchio, and V. Murino, “Video gesture analysis for autism spectrum
disorder detection,” in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3421–3426, IEEE, 2018
8 K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556,
2014
9 K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016
10 C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,”IEEE conference on
computer vision and pattern recognition, pp. 2818–2826, 2016.
THANK YOU