Machine Learning

Organisation and Introduction, AIN-B, SS2024

Markus Mayer
markus.mayer@th-deg.de
 Contents

Organisation

Revision: AI?

Topics and goals of the lecture

M.Mayer, ML, March 18, 2024 2/34

 Exam viewing PM and Statistics

▶ General exam related questions?

▶ Exam viewing: Tuesday, 26.3 and Wednesday, 27.3, afternoon
▶ Send me a mail including:
▶ Name
▶ Exam
▶ Immatriculation number, seat number
▶ If you are blocked in a certain timeframe on the respective dates
▶ Send a mail even if you already sent one!
▶ I’ll schedule meetings. There will be ∼ 15min for everyone.
▶ You’ll receive an answer mail with your scheduled appointment.
▶ Deadline for registration is 2024-03-24, 20.00 o’clock. I will
not accept late registrations.

M.Mayer, ML, March 18, 2024 3/34

 Exam viewing
A cite from Prof. Ionescu, which I found so good I have to copy it:

Please make sure you have the correct expectations for this procedure!
You may ask me to review the grade for a specific questions, if you
suspect that an error has been made. I will then do so and either confirm
the grade or change it. Do not expect me to offer a full solution to the
exam questions, together with lengthy explanations of why each point
was awarded or withdrawn: that is not what the exam review is about.

M.Mayer, ML, March 18, 2024 4/34

 Course organization

▶ Total: 5 ECTS = 150h. 15 x 4h presence = 60h. 90h self-study.

▶ No lecture on: 17.4 (even if its marked in Thabella),
1.5 (public holiday)
▶ No exam, but project work (see next slides).
▶ This lecture: Introduction. Last lecture(s): Project presentations.
In addition, we might need more time for project work → ≤14
lectures!
▶ Collaborative effort:
▶ (Almost) no programming by me in the lectures.
▶ You’ll teach the programming work yourself!
▶ However: We discuss your exercise results on a voluntary base.
▶ Beside my slides: A lot of other (possible) material.
▶ More later...

M.Mayer, ML, March 18, 2024 5/34

 Project work

Plan:
▶ Details anounced and start after easter (3.4) .
▶ Presentations on 15/16.07.2024 (Monday/Tuesday).
▶ Mandatory Check-Ins during the semester.
▶ We have reserved an additional 2h in our schedule for the
project work. They will be used after the projects have started.
We might have to use lecture time for project work, too. This
will be announced.
▶ 60 points to gather, grading scheme will be announced.

M.Mayer, ML, March 18, 2024 6/34

 Programming tasks

▶ In the first 23 of the semester, there will be worksheets with tasks

▶ Programming language: Python Well, actually I don’t care. You may also use R or Matlab.
We’ll discuss tasks in Python.

▶ Procedure:
▶ For each sheet, there will be a iLearn task.
▶ There are name suggestions for the solution.
▶ On a voluntary base, you can upload solution(s) on iLearn and
therefore allow me to discuss them in the lecture.
▶ The solutions do not have to be correct (or even perfect).
▶ Things important: Use only modules included in pip or
conda. The solution should be commented and follow the
default Python style guide.
▶ Include all the people that worked together in the header
comment!

M.Mayer, ML, March 18, 2024 7/34

 Programming tasks (2)

▶ You may do the tasks in groups - do not include people in the

header that did not participate in the specific task(s).
▶ People that hand in solutions will get up to 6 extra points (up to
2 grade steps) for the project work. (You can of course reach a 1.0 without
handing in tasks. The task points are bonus points).
▶ Regularity is more important than mass (i.e. I value handing 1
task in for each sheet more than handing in all tasks for the first
two sheets).
▶ I’ll always look at the solutions on tuesday afternoon and
prepare the next lecture.

M.Mayer, ML, March 18, 2024 8/34

 Textbooks, course material

The course material is loosely based on the following sources:

Figure: Sources for this course

▶ An Introduction to Statistical Learning, James, Witten, Hastie,

Tishirani, Taylor, 2nd Edition, 2021 and Python edition, 2023
▶ MIT Open learning library: Introduction to machine learning

M.Mayer, ML, March 18, 2024 9/34

 An Introduction to Statistical Learning (ITSL)

Figure: Book cover

▶ Available as a PDF online: An Introduction to Statistical Learning

▶ My number 1 textbook and learning resource for machine
learning beginners. The majority of this course is based on this.
▶ There is a “hardcore” version: The Elements of Statistical
Learning, Hastie, Tibshirani and Friedman, 2009
▶ 612 pages, Examples are in R or Python (depending on the edition).
M.Mayer, ML, March 18, 2024 10/34
 Introduction to Machine Learning (MIT-ITML)

Figure: MIT Open Learning

▶ Available only online: MIT Open learning library: Introduction

to machine learning
▶ My number 1 online course for machine learning beginners.
▶ Readings can be downloaded without registration
▶ After registration and enrolling, you can do the exercises and
track progress. (Hint: It is worth doing so for practice)
▶ Examples in Python.

M .M ayer, M L, M arch 18, 2024 11/34

 Remarks on sources
▶ Both are remarkably good, but also more detailed then we
are able to do in our course If you work closely through the MIT course or ITSL, your are well
prepared for the project work. But you will for sure have spend way more time compared to attending the lectures

and collaborating with your colleagues on ML programming techniques!

▶ Both are copyrighted, but free - I will refer to them instead of

pasting the content in iLearn.
▶ Both have completely valid, but completely different
orderings of topics and explanation approaches (and
naming schemes).
I will do another different (also valid) approach in ordering and
selecting topics. The field of ML is vast!
▶ The slides will be a only a guideline! The references to the ITSL,
MIT course, Widipedia and work on the whiteboard will complete the
picture.
M.Mayer, ML, March 18, 2024 12/34
 Contents

Organisation

Revision: AI?

Topics and goals of the lecture

M.Mayer, ML, March 18, 2024 13/34

 A collection of AI keywords

▶ Note down > 3 keywords that come to your mind when you think
of “AI”!
▶ Write them down anywhere on the blackboard (or whiteboard)
In case there is no black/whiteboard: Tell them to me and I’ll write them down.

▶ Keywords can be from the preconditions, themes, topics, questions

etc. of/towards AI. Everything related that comes to your mind.

M.Mayer, ML, March 18, 2024 14/34

 A collection of AI keywords (result)

▶ Can we find a "sorting" within the notes?

▶ Do you miss something?

M.Mayer, ML, March 18, 2024 15/34

 A collection of AI keywords (discussion)

▶ What is part of this lecture?

▶ What will be taught in other lectures?
▶ Which topics do interest you most?

M.Mayer, ML, March 18, 2024 16/34

 A collection of AI keywords (discussion)

▶ What is part of this lecture?

▶ What will be taught in other lectures?
▶ Which topics do interest you most?

▶ Can you define "Artificial Intelligence"?

M.Mayer, ML, March 18, 2024 16/34

 Definitions of AI

▶ There is no fixed definition of what "artificial intelligence" is.

▶ In fact, even two professors at the DIT may give you different
definitions - each valid in its own sense.
▶ We first look at 2 definitions:
▶ Mine
▶ From the EU.
▶ We then span a map of the various fields contained in "AI"
▶ And then discuss what we’ll do in this lecture...

M.Mayer, ML, March 18, 2024 17/34

 Prof. Mayers view

▶ I never studied "AI" intentionally.

▶ We had an "AI" lab in Erlangen (where I studied), but I did not
take any courses there.
▶ What I did study and made my PHD thesis in: Pattern
recognition
▶ Wikipedia definition: "Pattern recognition" is the automated
recognition of patterns and regularities in data.
▶ My definition: "Pattern recognition" is the research field that
deals with creating meaning from data (images, signals, text, ...).
It mainly uses methods from Machine Learning, but also
heuristics (task specific algorithms) and optimization.
▶ My PHD thesis: "Automated Glaucoma Detection with OCT"
▶ In 2019 I suddenly realized: "I’m an AI expert!" (My knowledge and skills did
not change in a very short amount of time. The scientific and common view on what AI currently is did!)

M.Mayer, ML, March 18, 2024 18/34

 From a presentation on AI (Prof. Mayer, ≤ 2021)

The following is directly taken over from the original presentation. In the meantime, I discovered
great shortcomings (do you see them, too?). My new version is on the upcoming slides.
▶ AI research definition is vague: “Cutting edge”, of what machine
learning (combined with other research field) can do.
▶ Ever changing!
▶ Challenges, that no scientist would label “AI” anymore (machine
learning is used, but the tasks are considered to be solved):
▶ Hand written letter recognition
▶ Human speech to text
▶ Challenges, that are currently in AI research focus:
▶ Natural language interaction (Machine learning + Data
representation)
▶ Self driving cars
▶ Game AI (Chess, Go. . . )
M.Mayer, ML, March 18, 2024 19/34
 Prof. Mayers view

▶ AI is a research field, that tries to find algorithms/methods

that can be applied to and solve a lot of different tasks.
▶ Most of the tasks that AI research tackles have in common that
they are/were performed by humans.
▶ The algorithms are made applicable to specific tasks by deriving
their parameters from either human knowledge or existing data.
▶ The focus and methods of AI are ever changing.
▶ Algorithms once researched in AI may be used in other
disciplines without calling them AI anymore.

M.Mayer, ML, March 18, 2024 20/34

 Discussion: Prof. Mayers definition/view
What are your thoughts about this definition?

M.Mayer, ML, March 18, 2024 21/34

 Discussion: Prof. Mayers definition/view
What are your thoughts about this definition?
▶ AI is defined as a research field! Perhaps we can agree on that?
▶ I tried to avoid both the words "artificial" and "intelligence" in
my definition of the research field.
▶ I’m an engineer - that results in a practically focused view on
tasks/problems.
▶ This is more a view then a formal definition.
▶ Question: Do you think that we’ll find a more compact and
resilient definition when > 50 experts from companies (a.o. IBM,
SAP, Airbus, Bayer, Google...) and institutions (a.o. Fraunhofer,
University of Oxford, TU Delft...) sit together to formulate an "AI"
description?

M.Mayer, ML, March 18, 2024 21/34

 A current definition of AI
In April 2019, the High-Level Expert Group on Artificial Intelligence of
the EU published the following document:

A definition of Artificial Intelligence: main capabilities and scientific

disciplines

We could do an (perhaps endless) discussion of the pros and cons of

the EU definition... However, we leave that out and focus on one
graph that what shown in the document.

M.Mayer, ML, March 18, 2024 22/34

 Subfields of AI (as of the EU definition)

Figure: A simplified overview of AI’s sub-disciplines and their relationship. Both machine
learning and reasoning include many other techniques, and robotics includes techniques that
are outside AI. The whole of AI falls within the computer science discipline. (From the EU AI
definition)
M.Mayer, ML, March 18, 2024 23/34
Subfields of AI (more general))

Artificial Intelligence

Foundations of AI Reasoning Machine Learning Close application

fields
Knowledge Representation Supervised learning
Philosophy Natural
Formal rules, informa-
Logic Language
Regression
tion theory, mind and Fuzzy Logic, Prolog, Answer Processing
knowledge, language Linear Regression,
set programming...
theory... Deep Learning…
Computer Vision
Rules
Math Expert systems Classification
kNN, Bayesian Classifiers, Sup- Robotics
Logic, computability
theory, statistics, ... Structured Objects port Vector Machines, Decision
Trees, Deep Learning…
Ontologies, Frames...

Computer
Science Search
Feature engineering
Complexity theory, da- State-Action Systems...
ta structures, opti- Image and signal processing,
mization, quantum descriptive statistics...
computing...
Optimization
Unsupervised learning
Graph optimization
Clustering...
Scheduling, Segmentation...

Function optimization Reinforcement learning

Planning, Registration, Seg- State-Action systems, Bayesian
mentation, …. networks, Deep Learning...

Figure: A detailed caption is on the next slide.

M.Mayer, ML, March 18, 2024 24/34
Subfields of AI (more general): Notes)
Artificial Intelligence

Foundations of AI Reasoning Machine Learning Close application

fields
Knowledge Representation Supervised learning
Philosophy Natural
Formal rules, informa-
Logic Language
Regression
tion theory, mind and Fuzzy Logic, Prolog, Answer Processing
knowledge, language Linear Regression,
set programming...
theory... Deep Learning…
Computer Vision
Rules
Math Expert systems Classification
kNN, Bayesian Classifiers, Sup- Robotics
Logic, computability
theory, statistics, ... Structured Objects port Vector Machines, Decision
Trees, Deep Learning…
Ontologies, Frames...

Computer
Science Search
Feature engineering
Complexity theory, da- State-Action Systems...
ta structures, opti- Image and signal processing,
mization, quantum descriptive statistics...
computing...
Optimization
Unsupervised learning
Graph optimization
Clustering...
Scheduling, Segmentation...

Function optimization Reinforcement learning

Planning, Registration, Seg- State-Action systems, Bayesian
mentation, …. networks, Deep Learning...

Figure: A more general overview of AI’s subfields.

▶ Braindump from the lecturer.

▶ Size relations have no meaning.
▶ This is definitely not complete. Some things were left out deliberately.
▶ The italics are again subfields or specific methods. Again: No
completeness (this is just to show that the field tree spreads branches
way further)
▶ Notice: Some methods (e.g. Deep Learning) appear multiple times.
▶ Subfields may have a lot of overlap (e.g. Optimization and Machine
Learning)
M.Mayer, ML, March 18, 2024 25/34
Subfields of AI: Relevance

Figure: Knowledge representation vs. Machine Learning

▶ There is a great common sense in the AI community that these are the
two main fields of AI.
▶ But was has (currently) more relevance?
M.Mayer, ML, March 18, 2024 26/34
Subfields of AI: Relevance (2)

Figure: Knowledge representation vs. Machine Learning: In your studies...

▶ The basics of reasoning and knowledge representation are covered in

your studies.
▶ Most lectures are related to Machine Learning.
M.Mayer, ML, March 18, 2024 27/34
Subfields of AI: Relevance (3)

Figure: Knowledge representation vs. Machine Learning: New developments...

▶ Almost every current development in the AI field uses Machine

Learning...

M.Mayer, ML, March 18, 2024 28/34

 Contents

Organisation

Revision: AI?

Topics and goals of the lecture

M.Mayer, ML, March 18, 2024 29/34

 Course topics
(Provisional!) plan:
▶ What is machine learning?
▶ One classification example from beginning to end
▶ The machine learning experiment
▶ Error measures, training and test data
▶ The kNN classifier

▶ The Bayes classifier (LDA, naïve Bayes, QDA)

▶ Linear regression
▶ Feature engineering (lifting, selection), outliers, cross validation,
resampling, optimization (well - a short introduction)
▶ Unsupervised learning (clustering)
▶ Support vector machine
▶ Decision Trees
▶ Machine learning evaluation pitfalls
M.Mayer, ML, March 18, 2024 30/34
 NO goals of the course
NO goals of this lecture:
▶ You can implement all of the algorithms.
Instead: You can implement the most basic algorithms (LDA, naïve
Bayes, linear regression, k-means, k-nearest neighbor, decision trees,
next-best feature) and have a general idea how the more complex ones
could be implemented.
▶ You understand all the mathematical linkages in between algorithms
and algorithms and cost functions and can proof them.
Instead: You know some of the linkages (I’ll omit a lot), without proof.
But you know the sources where to find all the answers.

M.Mayer, ML, March 18, 2024 31/34

 Goals of the course
The goals of this lecture:
▶ You have a thorough understanding of what “Machine
learning” can be and the variety of possible tasks and data.
▶ You know basic clustering, regression and classification
methods and can apply them.
▶ You understand the mathematical difference between
classification algorithms and the impact on the expected
results.
▶ You know how to construct a basic Machine learning
evaluation and the pitfalls of this task.

M.Mayer, ML, March 18, 2024 32/34

Goals of the course (2)

Figure: Scikit Learn ML map (from https://scikit-learn.org)

M.Mayer, ML, March 18, 2024 33/34

 Goals of the course (3)

▶ A student pointed this (very nice) Map of ML algorithms from

the popular Skikit learn package to me.
▶ Have we succeeded if you learn this by heart?

M.Mayer, ML, March 18, 2024 34/34

 Goals of the course (3)

▶ A student pointed this (very nice) Map of ML algorithms from

the popular Skikit learn package to me.
▶ Have we succeeded if you learn this by heart?
▶ NO! Our goal is that you can not only answer which method to
apply, but mainly WHY this is a valid choice. In the map, the
“why” is completely missing. We want to be (hopefully) thinking
AI engineers!
So we don’t do the complete map (as said). But the things we do, we try to reason.

M.Mayer, ML, March 18, 2024 34/34