APM120, Homework #10

Applied Linear Algebra and Big Data

Last updated: Saturday 20th April, 2019, 16:47
Assigned April 16, Due April 23, 10:00am, via canvas, as a single pdf file
Supervised learning: classification II, Neural networks

Show all steps in all calculations explicitly. Attach all code used for this HW, well docu-
mented, and all relevant plots and Matlab/ python output. Please note, though, that a code
printout is not a substitute for complete solutions. Your solution should stand alone without
the Matlab/ python code or output. See python equivalent of Matlab instructions at end of
this HW1 . It is fine to use Matlab/ python when not required otherwise, but make sure you
can do all calculations using no more than a hand-held calculator as well.

1. Neural networks: consider the following neural network: The hidden layer(s) are
composed of “tansig” neurons and the output layer is linear,
w2=[ 2.67047, -3.42171;
-0.423653, -3.951;
-2.17264, 1.3573];
b2=[ 0.332287;
10.1217;
0.29723];
w3=[ 1.78197, 0.0390082, 0.565396;
0.562076, 0.469971, -0.499013];
b3=[ -2.28352;
0.728266];
w4=[ -1.05368, -1.57008];
b4=[ 1.25094];
The two inputs can be any number in the range of (1,2). (A) draw the network and
write all of its equations explicitly. (B) Calculate the outputs for 5 possible inputs.
Show all intermediate neuron values for the example input [1.5;1.7]. (C) verify that
given the input x1 , x2 the network returns y = x2 /x1 . Don’t you think it’s amazing
that this actually works? :-) Hint: in Matlab, you can define activation functions using
“anonymous functions”,2
sigmoid = @(x) 1.0./(1.0+exp(-x));
softmax = @(x) exp(x)/sum(exp(x(:)));
tansig = @(x) 2./(1+exp(-2.0*x))-1;
and then used as,
z=[1 2 3]';
sigmoid(z)
% or
softmax(z)

1
2. Neural networks, back-propagation: consider the following training data point,
X= [2 3]'; y= 1;
for a neural network with all-sigmoid nodes, and with the following initial weights,
w2=[ 0.2, -0.2;
0.4, 0.2;
0.1, -0.1];
b2=[ 0.4;
0.3;
0.4];
w3=[ 0.1, 0.2, 0.2];
b3=[ 0.1];
and assume a quadratic cost function C = 12 (y − aL )2 . (A) Write the explicit equations
for the network, and draw its detailed architecture. Use the feedforward network to
calculate the output and the cost for the training data point and label, show the values
of all intermediate neurons. (B) Write the explicit backpropagation algorithm for this
network, and use it to calculate the gradient with respect to all weights wl and biases
bl . (C) Calculate the gradient with respect to w2(1, 2) using a direct perturbation
approach with  = 0.002 and compare to the appropriate gradient element calculated
using backpropagation. (D) Use a learning rate of η = 1 to adjust the weights, and
perform a steepest decent iteration for the data point provided. Calculate the cost for
the revised weights. Is it lower? Discuss.
3. Neural networks using Matlab/python routines: consider the following training
data for a 2d classification/ regression problem,
X=[5,7,2, 6,1,10,4,0,1, 0,0, 5, 2,7,8,4,2,1,2,8,9,6,5,6,7,8,7,4,4,7;
2,9,7,10,1, 5,1,3,4,10,2,10,10,7,6,3,1,4,8,7,4,3,2,9,3,6,2,1,4,7];
V=[4,3,3, 3,1, 2,1,1,1, 3,1, 3, 3,3,4,4,1,1,3,3,2,4,4,3,4,4,2,1,4,3];
(A) Use the built-in Matlab Neural Network toolbox (or python’s Keras package plus
Google’s tensorflow backend3 ) to construct a feed-forward classification neural net-
work and calculate and contour the labels calculated for the finer grid specified by
x1=[0:0.1:10]; x2=[0:0.1:10]; (B) Use the same training data to construct a feed-
forward regression neural network and calculate and contour the regression values
calculated for the same x,y values. Note: In both cases you need to choose appropri-
ate architectures that are able to classify the given data. Hint: you may start with the
following two examples on the course web page:
neural networks2 simple 2d classification example.m/py
neural networks3 simple 2d regression example.m/py
Try different architectures (number of hidden layers, number of neurons), until you find
something that works for classifying the training data and for which the classification
of the grid of points seems reasonable. This requires some experimentation, although
the programming effort is minimal, given the above two codes. Discuss your results. . .

2
 * [What’s the point of ***optional extra credit challenge problems: apart from
the fun of doing them, they may bring the total HW score to up to 110% therefore counting
against problems you may have missed in this or other HW assignments...]

Python equivalents
1 These python commands assume you have first given the following commands: import numpy as np;
from numpy import linalg; import scipy as scipy; import matplotlib.pyplot as plt; import
matplotlib;
Input matrix A, column vector b and row vector c into python in the form
A=np.array([[a11,a12,a13],[a21,a22,a23],[a31,a32,a33]];
b=np.array([[b1],[b2],[b3]]); c=np.array([[c1,c2,c3]]); or convert Matlab arrays given in HW
directly to python arrays using, e.g., A=np.array(np.matrixlib.defmatrix.matrix('[1 2 3; 4 5 6]'));
2 def sigmoid(x):
sig=1.0/(1.0+np.exp(-x));
return sig
def softmax(x):
soft=np.exp(x)/np.sum(np.exp(x[:]));
return soft
def tansig(x):
tsig=2/(1+np.exp(-2.0*x))-1;
return tsig
and then use as,
z=np.array([[1, 2, 3]]); z=z.T;
sigmoid(z)
or
softmax(z)
3 see installation instructions