Machine Learning for Engineer

ML Competition 2

Handwritten Digit Recognition from the MNIST Database using Shallow Neural Networks
Follow the instructions below, complete the following tables and upload as a Word document and your modified
MATLAB script traintestMLPonMNIST.mlx. You may share and discuss approaches to the competition
with others but do not share code. When you have arrived at your final neural model for each table below, rerun
your code 5 times and calculate an average and standard deviation to give results in Av % ± SD %. This is to take
into account sampling variations.

GROUP NUMBER/LETTER:
GROUP NAMES CONTRIBUTING (including UoL username):

1.
2.
3.
4.
5.
6.

Results

GROUP: Table 1 Best MLP NN model using nprtool applied to the test data provided.

Classification Most common No of hidden units No of training

Accuracy on test misclassified Digit in epochs to
data % the test data convergence
Best MLP NN % True class:
model +- % Predicted class:
How many mistakes?:
Training Performance Plot

Confusion matrix

[20 Marks for correct values (group)]

GROUP: Table 2 Best MLP NN model using your modified traintestMLPonMNIST.mlx without any extra
features added to the network applied to the test data provided.

Classification Most common No of hidden units No of training

Accuracy on test misclassified Digit in epochs to
data % the test data convergence
Best MLP NN % True class:
model +- % Predicted class:
How many mistakes?:
Training function: Performance function:
Training Performance Plot

Confusion matrix

[20 Marks for good performance (group)]

GROUP: Table 3 Best MLP NN model using traintestMLPonMNIST.mlx without any extra features: 5
challenging mistake examples in test image classification from the test image dataset.

Test Image index Digit Image Plot Actual Digit Network

for Digit (position Class (0-9) predicted
in 10,000 image class (0-9)
series)

[20 Marks for challenging-for-humans digits found with corresponding network responses (group)]

INDIVIDUAL&GROUP: Table 4 Best MLP NN model using your modified traintestMLPonMNIST.mlx with
your extra 3 features for each group member added to the network applied to the test data provided.

Classification Most common No of hidden units No of training

Accuracy on test misclassified Digit in epochs to
data % the test data convergence
Best MLP NN % True class: Keep the same as
model +- % Predicted class: table 2
How many mistakes?:
Group Feature Equation for your feature:
member 1 code MATLAB code for your feature:
1 and Justify why you believe this is a useful feature:
NAME: details:

Average connection weight for this feature after training:

Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:

Average connection weight for this feature after training:

Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:

Average connection weight for this feature after training:

Comment on the performance of your features:

Group Feature Equation for your feature:

member 1 code MATLAB code for your feature:
2 and Justify why you believe this is a useful feature:
NAME: details:

Average connection weight for this feature after training:

Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:

Average connection weight for this feature after training:

Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Comment on the performance of your features:

Group Feature Equation for your feature:

member 1 code MATLAB code for your feature:
3 and Justify why you believe this is a useful feature:
NAME: details:
Average connection weight for this feature after training:
Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Comment on the performance of your features:

Group Feature Equation for your feature:

member 1 code MATLAB code for your feature:
4 and Justify why you believe this is a useful feature:
NAME: details:
Average connection weight for this feature after training:
Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Comment on the performance of your features:

Group Feature Equation for your feature:

member 1 code MATLAB code for your feature:
5 and Justify why you believe this is a useful feature:
NAME: details:
Average connection weight for this feature after training:
Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Comment on the performance of your features:

Group Feature Equation for your feature:

member 1 code MATLAB code for your feature:
6 and Justify why you believe this is a useful feature:
NAME: details:
Average connection weight for this feature after training:
Feature Equation for your feature:
2 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Feature Equation for your feature:
3 code MATLAB code for your feature:
and Justify why you believe this is a useful feature:
details:
Average connection weight for this feature after training:
Comment on the performance of your features:

Network training combining ALL group member’s features together:

Training function: keep the same as table 2 Performance function: keep the same as table 2
Training Performance Plot
 Confusion matrix

Where any of the misclassified images found in Table 3 now corrected? If so which?

[15 Marks good features engineered (individual), 15 Marks feature originality (individual), 10 Marks good
classifier performance and performance ranking (group)]

Table entries may be verified against the data and code uploaded during marking.

Background

Data-set summary

The MNIST database (see details here including state-of-the-art classification performance table for this dataset
together with the machine learning method applied) is a collection of 60,000 28x28 greyscale pixel images of
real-world handwritten Arabic numerals (0-9) that has become a classic database to test various algorithms in
machine learning and AI. Grey scale pixel values are normalised [0-1]. This is a supervised classification problem
to correctly identify the 0-9 digits (10 classes) for all of the test images (evaluated by % classification
performance). The complete dataset is made available in four files of training images and class labels and 10,000
test images to evaluate classification performance.

train-images-idx3-ubyte: training set images

train-labels-idx1-ubyte: training set labels
t10k-images-idx3-ubyte: test set images
t10k-labels-idx1-ubyte: test set labels

The raw data is encoded in a custom ubyte format and so included with the data is a MATLAB Live Script
preparedata.mlx to read the data for you. Below is a MATLAB plot of the first 100 training images as
generated by preparedata.mlx.

What makes this database particularly challenging from a machine learning perspective is you can see from
Figure 1 that many of the numerals are unpredictably skewed according to the preferences and writing style of
the human generating them (e.g. compare all the 1 digit orientations in the figure) and other idiosyncrasies. 250
different writers were used in the construction of the training database provided and the researchers were
careful to ensure that the writers of the separate test images also provided did not create any of the training
images. This makes the problem particularly well suited to a neural network approach to automatically extract
features from the pixel values to deal with the variation across the data-set. Whilst deep learning networks with
large number of hidden layers can perform exceptionally well on this data-set (close to and even beyond human
recognition abilities) shallow feedforward neural networks such as the multi-layer perceptron we study in this
course can also perform surprisingly well under the right conditions.

Figure 1: The first 100 training images of the handwritten numeral MNIST database together with their class
labels (0-9). This dataset even challenges human abilities at recognition e.g. look at the first 9 in the sixth row
could easily be misunderstood as a 4. There are many other challenging recognition images in the 60,000
examples such that a trained human observer would not reach 100% accuracy on a blind trail.

Useful Code Provided

A MATLAB live script file preparedata.mlx included does the necessary pre-processing of the data-files for
you.

It includes two functions loadMNISTLabels and loadMNISTImages to read the custom ubyte file
format of the raw data files that should not need to change. From the raw data files included
preparedata.mlx creates the following workspace variables to classify: images (training image array of
double precision floating point values normalised [0,1] 784 pixels x 60,0000 images), labels (training class
labels vector of double precision 60,000 x 1 corresponding to each image), tstimages (test image array of
double precision floating point values normalised [0,1] 784 pixels x 10,0000 images) and tstlabels (training
class labels vector of double precision 60,000 x 1 corresponding to each image). Once these workspace variables
are generated you will no longer need to work with the raw data files.

A loop then reshapes each image vector of 784 double values into a 28x28 pixel image 2d-array and plots the
first 100 images in greyscale together with the corresponding class labels (generating the Figure 1 image) for
visualization.

A neural network for the MNIST classification problem requires 10 output nodes, one for each digit (0-9) that
require target training values for each image used to train the network. Target values should be a binary vector
containing a single ‘1’ value corresponding to the class label for the corresponding image. For instance image 1 in
the training image series has a class label of ‘5’ and so the target binary vector applied to the 10 output nodes
should be [0 0 0 0 1 0 0 0 0 0]. paredata.mlx generates these binary vectors for each image in the training
and test set and saves them into workspace variable labels and testlabels ready for applying directly to
the neural network during the training and subsequent test stages. No further processing of the target values
should be necessary.

Although there are different ways to implement shallow neural networks such as the multi-layer perceptron in
MATLAB patternnet(hiddenSizes,trainFcn,performFcn)is recommended as a straightforward
approach (help on patternnet is here). You can also use the interactive nprtool GUI based upon
patternnet to train the neural network, but it does not provide sufficient flexibility of different training and
performance functions so we will use the coded approach here. The beginnings of a second Live Script to create,
train and test the network traintestMLPonMNIST.mlx is included with the data and code package that
you can modify and complete.

patternnet training splits the input data-set into training, validation and testing partitions (from the original
training data applied). Validation is used periodically during the training phase to check for overfitting of the
model to the training data so that training can automatically complete once a convergence criteria is met.
traintestMLPonMNIST.mlx sets these partitions for you to use 80% of the images data for training and
20% validation (testing is performed separately in the steps below). None of this training data is to be used for
testing as we have separate tstimages and tstlabels to test with as created by different writers which
makes the problem more challenging, as specified in the script:

net.divideParam.trainRatio = 80/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 0/100;
Theory

This competition builds on the methods you looked at in the Machine Learning with MATLAB course using
shallow feedforward neural networks for supervised learning and classification. You will be implementing a
feedforward single hidden layer multilayer perceptron shown below.

Figure 2: A single hidden layer multilayer perceptron for applying to the MNIST dataset with normalised input
pixel values x1, x2, … xn (n=28 is fixed in this example for the number of image pixels) with all-to-all weight
connections to m hidden layer neurons a1(2), a2(2), …. am(2) with all-to-all weight connections to the output layer
a1(3), a2(3), … ak(3) (k=10 is fixed for the number of digits in this example). The ideal network performance after
training is for the output node corresponding to the input image having a value of 1.0 and all other output nodes
0.0, e.g. image with class label 5 is applied and [a1(3), a1(3), … ak(3)] = [0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0].

Procedure

Please follow these steps. Make sure you have completed the Machine Learning with MATLAB course before
starting these steps – in particular the feedforward classification neural networks section of the course. You have
4 options to run MATLAB: on a campus machine, download your own home use license and toolboxes (use latest
release), MATLAB online through a web browser or Remote Desktop.

Training neural networks can be computationally intensive due to the number of training epochs required to
reach convergence. You might want to explore different MATLAB options to obtain the fastest model training for
your experiments.

Always use images for model training and tstimages for model testing to avoid model overfitting to the
data. Note that a neural network is generally initialized with random connections weights and the partitioning of
the data into train and validation sets also randomises each time you run traintestMLPonMNIST.mlx so
you may find performance variations over repeat training trials.

1. Download the code and data package zip file from the Blackboard site Assessment section and extract
to a separate folder. Open MATLAB and make sure that the “current folder” is set to the extracted
location of preparedata.mlx. Open and Run the preparedata.mlx live script in MATLAB to
generate the training and test variables described above. Make sure you run all sections of the .mlx file
and check that all of the variables listed above are created in the Workspace correctly.
2. You can use nprtool from the command line to launch the interactive GUI to explore neural network
training and performance. Select “Predictors” to images and “Responses” to labels and you can
explore different numbers of hidden units and training/test/validation partitions using this GUI as well
as generate confusion matrices, performance and training state plots. After training you can separately
test on the tstimages and tstlabels data using the same GUI. However this does not give
sufficient flexibility to fully optimize the neural network as you cannot alter the training or performance
functions. Set the “Validation data” to 20% and “Test data” to 5%. Try training different numbers of
hidden units and complete Table 1 above for your best classification performance separately tested on
tstimages and tstlabels data using nprtool.
3. To gain full control of patternnet training and test options we take a coded approach using the Live
Script traintestMLPonMNIST.mlx. Modify and add to the script to specify the training function,
performance function, plot functions and add the necessary code elements to train the network on
images and labels. Do not add additional layers to your network, it must remain a single hidden
layer multi-layer perceptron as shown in Figure 2. The train() function you can find in the Live Script
should bring up the Neural Network Training GUI once you have populated the code that allows you to
explore the various plots.
4. Now test the trained network on tstimages and tstlabels by populating the test() and
perform() functions to obtain a percentage performance on the test data. The Live Script calculates
the % error rate for you. Repeat steps 3 and 4 using a variety of hidden unit numbers and training and
performance functions to obtain the lowest error. Complete Table 2. Is the classification performance
better than that found in step 2?
5. Add some code to traintestMLPonMNIST.mlx to plot out some examples of misclassified test
image digits [hint get the code to compare variables tind and yind and you are welcome to adapt
the code in preparedata.mlx to plot examples]. Try to find some test image examples that
humans could find a challenge to classify. Complete Table 3 with the most challenging 5 examples you
find. Check to see what the network predicted for these examples and add the details to the table.
6. Finally as an additional challenge add some code to traintestMLPonMNIST.mlx to calculate 3
additional features for each of the images and tstimages. These features values should be
 appended as the last three rows of both images and tstimages, which can then be used to retrain
the network using the same steps as above. You might get some inspiration for your features from the
research papers listed on the MNIST page here detailing research papers that have already tried a wide
variety of feature extraction techniques to obtain meaningful features. Each group member should
identify and implement 3 unique features to be connected to additional input neurons. By examining
the network connections after training is complete calculate the average connection weight to each
separate input feature neuron which indicates how useful the learning algorithm found each feature for
digit discrimination. Check whether any of the images you found in Table 3 have now been correctly
classified as a result of adding your 3 extra features. Complete Table 4.

Check that you have completed all tables, save this Word file and upload this together with the final version of
your modified traintestMLPonMNIST.mlx file. The competition group winner will have the best verified
accuracy performance in the tables as indicated above.

TCP