Autoencoders (Slide-1)

Topics to be covered (Slide-2)

Introduction
Autoencoder
Components of Autoencoder
Properties of Autoencoder
Architecture
Topics to be covered (Slide-3)
Hyperparameters
Types of Autoencoder
Visualization
Challenges
Regularization
Conclusion
Introduction (Slide-4)
Autoencoders are specific type of feed forward neural
networks where the input is the same as the output.
They compress the input into a lower-dimensional code
and then reconstruct the output from this representation.
The code is a compact “summary” or “compression” of
the input, also called the latent-space representation.
 Autoencoder (Slide-5)

Components of Autoencoder (Slide-6)

An autoencoder consists of 3 components:
 Encoder
 Code
 Decoder
Encoder compresses the input and produces the code.
Decoder then reconstructs the input only using this
code.
 Properties of Autoencoder (Slide-7)
 Data-specific
 Lossy
 Unsupervised

Properties of Autoencoder (Slide-8)

 Data-specific: Autoencoders are only able to
meaningfully compress data similar to what they
have been trained on.
 Lossy: The output of the autoencoder will not be
exactly the same as the input, it will be a close but
degraded representation.
 Unsupervised: Autoencoders are considered an
unsupervised learning technique since they don’t
need explicit labels to train on. But to be more
precise they are self-supervised because they
generate their own labels from the training data.

Architecture (Slide-9)
 Architecture (Slide-10)
 Both the encoder and decoder are fully- connected
feed forward neural networks.
 The number of nodes in the code layer (code size)
is a hyper parameter that we set before training the
autoencoder.

Hyperparameters (Slide-11)
There are 4 hyperparameters that we need to set before
training an autoencoder:
 Code size: number of nodes in the middle layer.
Smaller size results in more compression.
 Number of layers: the autoencoder can be as deep
as we like. In the figure above we have 2 layers in
both the encoder and decoder, without considering
the input and output.

Hyperparameters (Slide-12)
 Number of nodes per layer: The layers are
stacked one after another. The number of nodes per
layer decreases with each subsequent layer of the
encoder, and increases back in the decoder.
 Loss function: We either use mean squared error
(mse) or binary crossentropy. If the input values
 are in the range [0, 1] then we typically use
crossentropy, otherwise we use the mean squared
error.

Types of Autoencoder (Slide-13)

The Four Types of Autoencoders are:-
 Vanilla autoencoder
 Multilayer autoencoder
 Convolutional autoencoder
 Regularized autoencoder

Vanilla Autoencoder (Slide-14)

 The autoencoder is three layers of a neural net with
one hidden layer. The input and output are the
same, and we learn how to reconstruct the input.
 Here, we see that we have an undercomplete
autoencoder as the hidden layer dimension
(64) is smaller than the input (784). This constraint will
impose our neural net to learn a compressed
representation of data.
 Multilayer Autoencoder (Slide-15)
 If one hidden layer is not enough, we can obviously
extend the autoencoder to more hidden layers.
 Now our implementation uses 3 hidden layers
instead of just one. Any of the hidden layers can be
picked as the feature representation but we will
make the network symmetrical and use the middle-
most layer.
Convolution Autoencoder (Slide-16)
 We may also ask ourselves: can autoencoders be
used with Convolutions instead of Fully-connected
layers
 The answer is yes and the principle is the same, but
using images (3D vectors) instead of flattened 1D
vectors. The input image is down sampled to give a
latent representation of smaller dimensions and
force the autoencoder to learn a compressed
version of the images.

Regularized Autoencoder (Slide-17)

There are other ways we can constraint the
reconstruction of an autoencoder than to impose a
hidden layer of smaller dimension than the input.
Rather than limiting the model capacity by keeping the
encoder and decoder shallow and the code size small,
regularized autoencoders use a loss function that
encourages the model to have other properties besides
the ability to copy its input to its output.

Visualization (Slide-18)

Challenges (Slide-19)
 They are indeed pretty similar, but not exactly the
same. We can notice it more clearly in the last digit
“4”. Since this was a simple task our autoencoder
performed pretty well.
 We have total control over the architecture of the
autoencoder. We can make it very powerful by
increasing the number of layers, nodes per layer
and most importantly the code size. Increasing
these hyperparameters will let the autoencoder to
learn more complex codings.
 Challenges (Slide-20)
But we should be careful to notmake it too powerful.
 Over-fitting
The autoencoder will reconstruct the training data
perfectly, but it will be over fitting without being able
to generalize to new instances, which is not what we
want.
Challenges (Slide-21)
Undercomplete
The autoencoder is said to be undercomplete. It won’t
be able to directly copy its inputs to the output, and will
be forced to learn intelligent features. then an
undercomplete autoencoder won’t be able to recover it
perfectly.

Regularization (Slide-22)
We would like to learn meaningful features without
altering the code’s dimensions (Overcomplete or
undercomplete).
We usually find two types of regularized autoencoder:
 Sparse Autoencoder
 Denoising Autoencoder
 Sparse Autoencoder (Slide-23)
Sparse autoencoders are typically used to learn features
for another task such as classification. An autoencoder
that has been regularized to be sparse must respond to
unique statistical features of the dataset it has been
trained on, rather than simply acting as an identity
function. In this way, training to perform the copying
task with a sparsity penalty can yield a model that has
learned useful features as a byproduct.

Denoising Autoencoder (Slide-24)

This way the autoencoder can’t simply copy the input to

its output because the input also contains random noise.
We are asking it to subtract the noise and produce the
underlying meaningful data.
 Visualization (Slide-25)

Conclusion (Slide-26)
Autoencoders are a very useful dimensionality
reduction technique. They are very popular as a
teaching material in introductory deep learning courses,
most likely due to their simplicity.