An Introduction to the Neural Network

Artificial neural networks can be considered as function approximation

algorithms.
In a supervised learning setting, when presented with many input
observations representing the problem of interest, together with their
corresponding target outputs, the artificial neural network will seek to
approximate the mapping that exists between the two.
A neural network is a computational model that is inspired by the
structure of the human brain.
– Page 65, Deep Learning, 2019.

The human brain consists of a massive network of interconnected

neurons (around one hundred billion of them), with each comprising a
cell body, a set of fibres called dendrites, and an axon:

The dendrites act as the input channels to a neuron, whereas the axon
acts as the output channel. Therefore, a neuron would receive input
signals through its dendrites, which in turn would be connected to the
(output) axons of other neighbouring neurons. In this manner, a
sufficiently strong electrical pulse (also called an action potential) can
be transmitted along the axon of one neuron, to all the other neurons
that are connected to it. This permits signals to be propagated along
the structure of the human brain

So, a neuron acts as an all-or-none switch, that takes in a set of inputs

and either outputs an action potential or no output.
– Page 66, Deep Learning, 2019.
An artificial neural network is analogous to the structure of the human
brain, because (1) it is similarly composed of a large number of
interconnected neurons that, (2) seek to propagate information across
the network by, (3) receiving sets of stimuli from neighbouring neurons
and mapping these to outputs, to be fed to the next layer of neurons.
The structure of an artificial neural network is typically organised into
layers of neurons (recall the depiction of a tree diagram). For example,
the following diagram illustrates a fully-connected neural network,
where all the neurons in one layer are connected to all the neurons in
the next layer:

The inputs are presented on the left hand side of the network, and the
information propagates (or flows) rightward towards the outputs at
the opposite end. Since the information is, hereby, propagating in
the forward direction through the network, then we would also refer
to such a network as a feedforward neural network.
The layers of neurons in between the input and output layers are
called hidden layers, because they are not directly accessible.
Each connection (represented by an arrow in the diagram) between
two neurons is attributed a weight, which acts on the data flowing
through the network, as we will see shortly.

The Mathematics of a Neuron

More specifically, let’s say that a particular artificial neuron (or
a perceptron, as Frank Rosenblatt had initially named it)
receives n inputs, [x1, …, xn], where each connection is attributed a
corresponding weight, [w1, …, wn].
The first operation that is carried out multiplies the input values by
their corresponding weight, and adds a bias term, b, to their sum,
producing an output, z:
z = ((x1 × w1) + (x2 × w2) + … + (xn × wn)) + b
We can, alternatively, represent this operation in a more compact form
as follows:

This weighted sum calculation that we have performed so far is a

linear operation. If every neuron had to implement this particular
calculation alone, then the neural network would be restricted to
learning only linear input-output mappings

However, many of the relationships in the world that we might want to

model are nonlinear, and if we attempt to model these relationships
using a linear model, then the model will be very inaccurate.
– Page 77, Deep Learning, 2019.
Hence, a second operation is performed by each neuron that
transforms the weighted sum by the application of a nonlinear
activation function, a(.):
We can represent the operations performed by each neuron even more
compactly, if we had to integrate the bias term into the sum as another
weight, w0 (notice that the sum now starts from 0):

The operations performed by each neuron can be illustrated as follows:

Therefore, each neuron can be considered to implement a nonlinear

function that maps a set of inputs to an output activation.