Neural Networks

2022-2023
LAB 2
Agenda
1. GUI using Tkinter
2. NumPy and Matplotlib review
3. Introduction to Neural Networks
4. Activation Functions

a. Linear activation functions.

b. Non-Linear activation functions.

Matplotlib & Tkinter - Installation
• Matplotlib is a Python 2D plotting library
• Tkinter is a standard GUI library for Python. Python when combined with
Tkinter provides a fast and easy way to create GUI applications.

• Activate your environment : conda activate enviroment_name

• Install Matplotlib library : conda install matplotlib

• Install Tkinter library : conda install -c anaconda tk

Matplotlib
Matplotlib
Tkinter
Tkinter
Numpy
• Np.empty : create an empty array
• Array.fill : set all values in an array

• np.zeros : create an array with zero values

• np.ones : create an array with one values
Numpy
• Array Slicing
>>> a[0,3:5]
array([3, 4])

>>> a[4:,4:]
array([[44, 45],
[54, 55]])

>>> a[:,2]
array([2,12,22,32,42,52])
STRIDES ARE ALSO POSSIBLE

>>> a[2::2,::2]
array([[20, 22, 24],
[40, 42, 44]])
Numpy
• Array.size : get the number of elements in the array.
• Array.ndim: get the dimensions of the array
Numpy
• In NumPy dimensions are called AXES. The number of axes is RANK.
Numpy
• Reshaping array : You can change the dimension of a NumPy keeping
the same size using the reshape method
Numpy
• Create an array with random values: you can initialize an array with
random values from 0 to 1 using rand function

np.random.uniform(low=0.5, high=2, size=(3,3))

Numpy
• Np.dot(a.b): Dot product of two arrays
Numpy
• Broadcasting: When there are multiple inputs, then they all must be
“broadcastable” to the same shape.
Numpy
• Broadcasting: When there are multiple inputs, then they all must be
“broadcastable” to the same shape.

mismatch!
 Neural Networks
• Neural Networks consist of the following components

• An input layer, x

• An arbitrary amount of hidden layers

• An output layer, ŷ

• A set of weights and biases between each layer,

 W and b

• A choice of activation function for each hidden

and output layer, σ the output of the hidden
layer is= σ(W1.T*x+b)
Neural Networks
 Neural Networks
• In the beginning of each task, we should define the following:

1. NN Architecture (i.e. Single Layer or Multi-Layer)

2. Learning Algorithm

3. Activation Function

4. Target (i.e. Classification, Data Reduction, or Clustering)

 Activation Function
• Activation functions are important for an Artificial Neural Network to learn and make sense
of complicated data as it provides non-linear complex functional mappings between the inputs
and response variable. They introduce non-linear properties to our Network.

• Without an activation function our Neural network would not be able to learn and model
complex data such as images, videos , audio , speech etc.

• If we do not apply an Activation function, then the output signal would be a simple linear
function. And it doesn’t matter how many hidden layers we attach in the neural net; all layers
will behave the same way because the composition of two linear functions is a linear function
itself.

• This means that a Neural Network without an activation function would simply be a Linear
regression Model, which has limited power.
Activation Function
Activation Function - Types

1 2

Linear Non-linear
Activation Activation
Function Functions
 Activation Function (Linear Activation Function)
• Equation : Linear function has the equation similar to as of a straight line i.e.
f(x) = x
• Range : -inf to +inf
 Activation Function (Non-Linear Activation Function)
1. Step function
2. Signum function
3. Sigmoid function
4. Hyperbolic Tangent Sigmoid function
5. ReLU (Rectified Linear Unit)
6. Leaky ReLU
7. Softmax
 Activation Function (Non-Linear Activation Function)
1. Step function (Binary step)
• It is also called threshold function.
• Desired output values are binary (0 or 1).
 Activation Function (Non-Linear Activation Function)
2. Signum function
• Is symmetric threshold function.
• Desired output values are -1 or 0 or 1.
 Activation Function (Non-Linear Activation Function)
3. Sigmoid function
• It is also called logistic sigmoid function.
• Range : 0 to 1
 Activation Function (Non-Linear Activation Function)
4. Tanh function
• It’s mathematically shifted version of the sigmoid function. Both are similar
and can be derived from each other.
• Range : -1 to 1
 Activation Function (Non-Linear Activation Function)
5. ReLU function
• One of the most used activation functions in the world right now. Since it is
used in almost all the convolutional neural networks or deep learning.
• Range : 0 to infinity
 Activation Function (Non-Linear Activation Function)
6. Leaky ReLU function
• It is superior to the ReLU activation function. It has all of the features of
ReLU and will never suffer from the Dying ReLU problem. Usually, the
value of a is 0.01 or so.
• Range : - infinity to infinity
 Activation Function (Non-Linear Activation Function)
7. Softmax function
• The output is a vector of values that follows a probability distribution whose
total sums up to 1.
• Range : 0 to 1
 Activation Function
Activation Output Range Advantages Disadvantages When to Use
Function
Sigmoid [0 to 1] • Predicts Probability • Negative inputs are mapped very close to Preferred for use
• Clear predictions zero in the output layer
• Gradient Vanishing*
• Slow because it uses exponential
operations
Tanh [-1 to 1] • Converges faster than • Gradient Vanishing Preferred for use
sigmoid* • Slow because it uses exponential in the hidden
• Negative input will be operations layers
mapped to negative
ReLU [ 0 to • Faster than sigmoid or • Dying ReLU* Usually used in
infinity] tanh MLP and Deep
• Less gradient vanishing Learning
Leaky [ -infinity to • Solves the dying ReLU • Adds learning complexity (“a”) Usually used in
ReLU infinity] problem • Sometimes hard-zero of ReLU is more deep learning
suitable for the data
 Activation Function
• The choice of activation function in the hidden layer will control how well the network
model learns the training dataset. The choice of activation function in the output layer
will define the type of predictions the model can make.

• It depends on the problem type and the value range of the expected outcome.

• In a single layer neural network, a linear, step or signum function is usually used.

• In a multilayer network, depending on the depth, tanh or ReLu for hidden layers and
sigmoid or SoftMax for output layers

• In deep learning, ReLu or Leaky ReLu for hidden layers, Softmax or sigmoid for output
layer
Tasks & Project Team Registration
Teams Registration Form will be published on Thursday 20/10/2022.
The form will be closed on Tuesday 25/10/2022 11:59 PM.
Each team should consist of (3-5) members.
Team members should all be from the same department.
These teams will be the same for the tasks and the project.
Thank you