Advanced Machine Learning

Course Introduction
Amit Sethi
Electrical Engineering, IIT Bombay
 Why an advanced ML course

• Basic ML inadequate for many applications

• Advanced ML usually gives better solutions

• True innovations are in advanced ML

 Learning outcomes for the course

• Characterize data input and desired output

• Identify problems and design ML solutions

• Read and critically compare research papers

• Code complex learning frameworks

 How we will accomplish our goals

• Lectures for frameworks

• Coding exercises for implementation

• Reading assignments for concepts and

mathematics
 Topics to be covered
• Revision of basic ML • Learning with less data
– Transfer
• NN architectures – semi-supervised
• Optimization – weakly supervised
techniques • Generative adversarial
networks
• Loss function
engineering • Robust NNs
• Training very deep NNs • Generative models
• Attention in NNs • Probabilistic models
• Insights into NNs • Training PGMs
 Evaluation structure
• Assignment 1 – 15% • 90%  AA
• 80%  AB
• Assignment 2 – 15% • 70%  BB
• 60%  BC
• Exam 1 – 20% • 50%  CC
• 40%  CD
• Exam 2 – 30% • 30%  DD

• Project – 20% • 40%  Audit pass

 Where would basic ML struggle?

Large number of dimensions on a grid

• What features
should we extract
from an image?
• Should pixels be
features?
• Is Euclidean
distance a good
metric?

Image source: Wikipedia

 Where would basic ML struggle?

Learning from similar domains

• If we have lots of
labeled digits
• But only a few
labeled examples
of an ancient
script
• Can we transfer
learning?

Image source: Wikipedia

 Where would basic ML struggle?

Scaling feature extraction with input

• More samples

• Higher
dimensional
samples

Image source: Wikipedia

 Where would basic ML struggle?

Depth vs. width

y1 y2 … yn

… … …

h11 h12 … h1n1

x1 x2 … xd
 How to optimize complex models

Image source: Unknown

 How to regularize models

Image source: Wikipedia

 Where would basic ML struggle?

Data dimensions form a graph

• What if variables are
related as graphs?

• What if different
variables are missing
in different samples?
 Where would basic ML struggle?

How to use unlabeled data

 Where would basic ML struggle?
What if labels are given to bags of
samples
 Where would basic ML struggle?

How to generate more samples

Discriminative Generative
Questions and suggestions
 Next lecture
• Revision of supervised machine learning

• Revision of neural networks