Course 1: How Google Does ML Module 4: Inclusive ML Lesson Title: Introduction Format: Talking Head

Course 1: How Google does ML
Module 4: Inclusive ML
Lesson Title: Introduction
Format: Talking head
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
Talent
Inclusive Machine Learning

How Google does ML
Machine Learning on Google Cloud Platform
© 2017 Google Inc. All rights reserved. Google

and the Google logo are trademarks of Google Inc.
All other company and product names may be
trademarks of the respective companies with
which they are associated.
You will learn how to:
● Identify the origins of bias in ML

● Evaluate ML models with biases
● Make models inclusive
Python
Intro to What it means How to make ML
How Google does ML notebooks in
Specialization to be AI-first inclusive
the Cloud
Machine learning and human bias
Agenda
Evaluating metrics with inclusion for your ML system
Equality of opportunity
How to find errors in your dataset using Facets
Module 4: How to make ML Inclusive
Lesson Title: Machine Learning and Human Bias
Format: Screencast
Human biases lead to biases in machine learning models
Module 4: How to Make ML Inclusive
Lesson Title: Evaluation Metrics Across

Subgroups
Format: Screencast
Agenda
Evaluate your model over subgroups also
The confusion matrix leads to evaluation metric insights
Model Predictions
Positive Negative
True Positives (TP)
Label says something exists.
✅
Model predicts it.
The confusion matrix leads to Model
says Yes
evaluation metric insights
True Positives (TP)
Label says something exists
Model predicts it exists
Model Predictions
Positive Negative
True Positives (TP) False Negatives (FN)
Something exists.
✅ Something exists
Type II Error
Model predicts it. Model doesn’t predict it
says No
False Negatives (FN)

Type II Error
Label says something exists
Model predicts it doesn’t exist
Model Predictions
Positive Negative
Label = something exists.
✅ Label = something
Type II Error
exists
True Negatives (TN)

Something doesn’t exist
Model doesn’t predict it
says No
True Negative (TP)
Label says something doesn’t exist
Model predicts it doesn’t exist
Model Predictions
Positive Negative
Something exists.
✅ Something exists.
Type II Error
False Positives (FP) True Negatives (TN)

Type I Error ✅
Something doesn’t exist
Something doesn’t exist Model doesn’t predict it
Model predicts it
says Yes
False Positives (FN)

Type I Error
Model predicts it exists
False positives and false negatives errors occur when
predictions and labels disagree
Model Predictions
Positive Negative
True Positives (TP) False Negatives (FN) Model
says No
✅ Type II Error

Type I Error
Model
says Yes
Module 4: Inclusive ML
Lesson Title: Statistical Measurements
Format: Screencast
Evaluation metrics can help highlight areas where machine
learning could be more inclusive
Model Predictions
Positive Negative
True Positives (TP) False Negatives (FN) Model
Type II Error says No
Label says something exists. Label says something exists
The model predicts it. Model doesn’t predict it

Type I Error
Label says something doesn’t exist Label says something doesn’t exist
Model predicts it Model doesn’t predict it
Model
says Yes
False negative rate is the fraction of true faces that are not
detected by the ML system
Model Predictions
Positive Negative
Type II Error
False Negative False Negatives

=
Rate False Negatives + True Positives
False negative rate is the fraction of true faces that are not
detected by the ML system
Model
Model says No
says Yes

Type II Error
False Negatives
False Negative Rate =
False Negatives + True Positives
False positive rate is the fraction of the faces that the ML
model detects that are not really faces
Model Predictions
Positive Negative
True Positives (TP)
Label says something exists. False False Positives
The model predicts it.
Positive = False Positives + True
Rate Negatives
False Positives (FP)
Type I Error
Model predicts it
False positive rate is the fraction of the faces that the ML
model detects that are not really faces
Model
says No
Model
says Yes
True Positives (TP) False Positives (FP)

Label says something exists. Type I Error
The model predicts it. Label says something doesn’t exist
Model predicts it
False Positives
False Positive Rate =
False Positives + True Negatives
Sometimes false positives are better than false negatives
Privacy in images
False positive False negative
Sometimes false negatives are better than false positives
False Negative: E-mail that is SPAM is not caught, so you see it in your inbox.
False Positive: E-mail flagged as SPAM is removed from your inbox.
Find the threshold that brings the precision or recall to
acceptable values
Criterion value
TN
TP
FN FP
Check the precision/recall you obtain with that threshold in
each of your subgroups
TN
TN TP TP
TN TP
FN FP FN FP
FN FP
Sub-group 1 Sub-group 2 Sub-group 3
False Positives False Negatives
Privileged & Confidential

Module 4: Human-centered ML
Lesson Title: Equality of Opportunity
Format: Screencast
Agenda
The Equality of Opportunity approach strives to give individuals
an equal chance of desired outcome
Qualify for classification

Equal chance
of selection
Desired outcome
A toy classifier to predict who will pay back their loan involves
two populations that might overlap
Picking a credit score threshold involves a tradeoff
Threshold too high

Threshold too low
The impact of a threshold on credit score is evaluated based
on its impact on customers and on loan repayment
Simulating the impact of a threshold on profit
Classification and Discrimination must obey the Equality
of Opportunity Principle
A=0
Y=1
A=1
Y=0
Lesson Title: Simulating Decisions
Format: Screencast
Simulating decisions with no constraints can lead to unequal
distribution
A successful loan makes $300

An unsuccessful loan costs $700
Credit scores are between 0 - 100
Simulating decisions with no constraints can lead to unequal
distribution
Threshold
● Credit Score of 50 for Blue Group
● Credit Score of 50 for Orange Group
Simulating decisions for max profit result in unequal standards
Threshold Total profit: 32400

● Credit Score of 61 for Blue Group
● Credit Score of 50 for Orange Group
Simulating decisions with group unaware holds everyone to
the same standard, which can be unfair to some groups
Simulating decisions equal opportunity results in an identical
true positive rate for all groups
Lesson Title: Finding Errors in Your Dataset

Using Facets
Format: Screencast
Agenda
https://research.googleblog.com/2017/07/facets-open-source-visualization-tool.html
Facets gives users a quick understanding of the distribution of
values across features of their datasets
In Facets features are sorted by non-uniformity, with the
feature with the most non-uniform distribution at the top
Facets features are sorted by distribution distance
Facets Dive provides an easy-to-customize, intuitive interface
Color the data points by one feature, then facet in another
Explore CIFAR-10 for Errors using Facets Dive
Exploring a dataset for Errors
cloud.google.com
Images by Connie Zhou

Course 1: How Google Does ML Module 4: Inclusive ML Lesson Title: Introduction Format: Talking Head

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Course 1: How Google Does ML Module 4: Inclusive ML Lesson Title: Introduction Format: Talking Head

Uploaded by

Copyright:

Available Formats

Course 1: How Google does ML

Lesson Title: Introduction

Format: Talking head

Inclusive Machine Learning

Machine Learning on Google Cloud Platform

© 2017 Google Inc. All rights reserved. Google

● Identify the origins of bias in ML

How to find errors in your dataset using Facets

Module 4: How to make ML Inclusive

Lesson Title: Machine Learning and Human Bias

Module 4: How to Make ML Inclusive

Lesson Title: Evaluation Metrics Across

How to find errors in your dataset using Facets

True Positives (TP)

Label says something exists

Model predicts it exists

Model predicts it. Model doesn’t predict it

False Negatives (FN)

Model predicts it doesn’t exist

True Negatives (TN)

True Negative (TP)

Label says something doesn’t exist

Model predicts it doesn’t exist

Model predicts it. Model doesn’t predict it

False Positives (FP) True Negatives (TN)

False Positives (FN)

Model predicts it exists

False Positives (FP) True Negatives (TN)

Lesson Title: Statistical Measurements

False Positives (FP) True Negatives (TN)

False Negative False Negatives

True Positives (TP) False Negatives (FN)

True Positives (TP) False Positives (FP)

False positive False negative

False Positive: E-mail flagged as SPAM is removed from your inbox.

Privileged & Confidential

Lesson Title: Equality of Opportunity

How to find errors in your dataset using Facets

Qualify for classification

Threshold too high

Lesson Title: Simulating Decisions

A successful loan makes $300

Threshold Total profit: 32400

Lesson Title: Finding Errors in Your Dataset

How to find errors in your dataset using Facets

You might also like