EE 8520 Advanced Topics in Signal Processing

Spring Semester 2019

Analytical Foundations of Deep Learning

Overview: Deep learning has achieved a meteoric rise in popularity in recent years, due in large
part to the increasing availability of data and corresponding programming/computational resources,
its widespread successes in numerous disciplines, and a wealth of (mostly positive) media coverage.
Even so, leading experts in this emerging and exciting discipline often extoll its virtues with a word
of caution that, paraphrasing, not much is known about the theoretical aspects of deep learning.
While this is perhaps true in a purely holistic sense, several classical works and more numerous
contemporary works have examined, rigorously, various analytical aspects of this general approach.
The primary objective of this course is to examine these results via a “survey-type” overview, weav-
ing established insights from the literature into a comprehensive narrative. Our specific aim will
be to examine the performance of deep learning and deep neural networks systematically, using the
rigorous perspective(s) of signal processing, statistical learning theory, and approximation theory.
As a secondary objective, we will identify several of the powerful analytical and probabilistic tools
that are used here (and elsewhere) when establishing theoretical results for general learning-type
problems.

Prerequisites: Knowledge of basic probability theory and probabilistic models (e.g., Gaussian,
Bernoulli, Binomial, etc.), linear algebra, and vector space concepts (e.g., norms and inner prod-
ucts) will be assumed. More advanced concepts will be discussed on an “as needed” basis, when
they arise in the context of the topics being covered.

Lecture: MW 11:15am–12:30pm, 102 Mechanical Engineering

Instructor: Associate Professor Jarvis Haupt (ECE Department), Email: jdhaupt@umn.edu

Course Resources: There will be no required textbook. The majority of the course content will
be derived from research papers (classic and contemporary), to be announced; links will be made
available on the course website. Some additional resources that will be utilized for portions of the
course content include:

1. “Deep Learning” by I. Goodfellow, Y. Bengio, & A. Courville

(available at https://www.deeplearningbook.org)

2. “The Nature of Statistical Learning Theory” (2nd Ed.) by V. N. Vapnik

3. “The Elements of Statistical Learning” (2nd Ed.) by T. Hastie, R. Tibshirani, & J. Friedman

4. “Probability in Banach Spaces” by M. Ledoux & M. Talagrand

Deliverables: There will be no midterm or final exams. Instead, grades will be assigned based on
performance on homework sets throughout the semester, a final project, and course participation.

1
Deliverables (details): It is anticipated that 6-10 homework sets will be assigned. In total, these
will include a mix of theoretical exercises (e.g., based on recent lecture material), applied exercises
(e.g., small-scale programming efforts), and some additional exercises that are more at the “com-
prehension” level. This latter thrust is practical, but somewhat experimental (for the instructor!).
Their aim is to give students an opportunity, and feedback, on how to read papers systematically
and, in a sense, hierarchically, in order to achieve various comprehension goals (e.g., identification
of the problem being examined, the main result(s), relevant literature, key proof insights, etc.).

The final project will entail a survey/overview based effort, or examination of an original research
direction, on topic(s) to be approved by the course instructor. Projects will be performed by small
groups of 2-3 students nominally, though some individual projects may be approved at the discre-
tion of the instructor, and are to culminate in a written project report. Additionally, depending on
the enrollment totals, there may be in-class project presentations (e.g., if the enrollment is modest
enough to make this feasible within a reasonable amount of time).

The final deliverable will be course participation. Invariably, this will entail scribing efforts for
lecture notes. Specifically, each student will be expected to scribe one of the lectures prepared by
the instructor to a set of LaTeX notes; a template will be provided. Additionally, in-class project
presentations (if performed) will comprise a component of the participation grade.

Homework Grading and Timeliness Policy: Homework assignments will nominally be equally
weighted when calculating final grades. Late homework submissions will be accepted, but their
graded scores will be weighted by a factor of 1/2. In other words, late submissions can achieve at
most half of the points of a corresponding on-time submission.

You are encouraged to work with others when solving the homework problems. However, solu-
tions submitted for grading should be formulated by each student independently!

Grading: Final grades will be based on the following breakdown:

Participation – 10-20%
Homework Sets – 50%
Final Project – 30-40%

The variation on the participation and final project components are contingent on the steady-
state enrollment, as alluded above.

Computer Usage: While the applied aspects of deep learning are not among the primary objec-
tives of the course, students will be expected to perform (nominally small-scale) computer exercises
to become familiar with the computational and practical aspects of this technology. This will entail
utilization of existing deep learning toolboxes such as PyTorch (https://pytorch.org) or Ten-
sorFlow (https://www.tensorflow.org), both of which are freely available for Windows, Mac,
and Linux operating systems. To this end, students should also be (or, will become!) familiar with
the Python programming language (environments for which are, again, freely available for a wide
variety of operating systems).

No dedicated GPU resources will be made available to the class. Thus, GPU-based, large-scale
computational efforts will NOT be necessary for successful completion of the course!

2
Other Important Information:
Student Academic Integrity and Scholastic Dishonesty:
Academic integrity is essential to a positive teaching and learning environment. All students en-
rolled in University courses are expected to complete coursework responsibilities with fairness and
honesty. Failure to do so by seeking unfair advantage over others or misrepresenting someone else’s
work as your own, can result in disciplinary action. The University Student Conduct Code defines
scholastic dishonesty as follows:

Scholastic Dishonesty: Scholastic dishonesty means plagiarizing; cheating on assignments or exam-

inations; engaging in unauthorized collaboration on academic work; taking, acquiring, or using test
materials without faculty permission; submitting false or incomplete records of academic achieve-
ment; acting alone or in cooperation with another to falsify records or to obtain dishonestly grades,
honors, awards, or professional endorsement; altering, forging, or misusing a University academic
record; or fabricating or falsifying data, research procedures, or data analysis.

A student responsible for scholastic dishonesty can be assigned a penalty up to and including an
“F” or“N” for the course. For additional information, refer to the student conduct code available
here: http://regents.umn.edu/sites/default/files/policies/Student_Conduct_Code.pdf

Incompletes:
A grade of I for Incomplete is given at the discretion of the course instructor when, due to
extraordinary circumstances, the student who has successfully completed a substantial portion of
the courses work with a passing grade was prevented from completing the work of the course on
time. Students must fill out an Incomplete Grade Agreement form found in Keller 3-166. The
maximum time to remove and replace an incomplete grade is one year.

Mental Health:
As a student you may experience a range of issues that can cause barriers to learning, such as
strained relationships, increased anxiety, alcohol/drug problems, feeling down, difficulty concen-
trating and/or lack of motivation. These mental health concerns or stressful events may lead
to diminished academic performance or reduce a students ability to participate in daily activ-
ities. University of Minnesota services are available to assist you with addressing these and
other concerns you may be experiencing. You can learn more about the broad range of con-
fidential mental health services available on campus via the Student Mental Health Website at
http://www.mentalhealth.umn.edu.

Classroom Disruptions:
Please respect the rights of the other students in class to hear the lecture and to participate in
classroom discussions. Likewise, activities that are not conducive to your own learning experience
should be kept to a minimum. Please be courteous by silencing your cell phones during class.