04b_asl_predictions about:srcdoc

Deploying Your Model

Now that we have a well trained model, it's time to use it. In this exercise, we'll expose new images to our
model and detect the correct letters of the sign language alphabet. Let's get started!

Objectives

Load an already-trained model from disk

Reformat images for a model trained on images of a different format
Perform inference with new images, never seen by the trained model and evaluate its performance

Loading the Model

Now that we're in a new notebook, let's load the saved model that we trained. Our save from the previous
exercise created a folder called "asl_model". We can load the model by selecting the same folder.

In [1]: from tensorflow import keras

model = keras.models.load_model('asl_model')

If you'd like to make sure everything looks intact, you can see the summary of the model again.

1 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [2]: model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 28, 28, 75) 750
_________________________________________________________________
batch_normalization (BatchNo (None, 28, 28, 75) 300
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 75) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 14, 14, 50) 33800
_________________________________________________________________
dropout (Dropout) (None, 14, 14, 50) 0
_________________________________________________________________
batch_normalization_1 (Batch (None, 14, 14, 50) 200
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 50) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 7, 7, 25) 11275
_________________________________________________________________
batch_normalization_2 (Batch (None, 7, 7, 25) 100
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 4, 4, 25) 0
_________________________________________________________________
flatten (Flatten) (None, 400) 0
_________________________________________________________________
dense (Dense) (None, 512) 205312
_________________________________________________________________
dropout_1 (Dropout) (None, 512) 0
_________________________________________________________________
dense_1 (Dense) (None, 24) 12312
=================================================================
Total params: 264,049
Trainable params: 263,749
Non-trainable params: 300
_________________________________________________________________

Preparing an Image for the Model

It's now time to use the model to make predictions on new images that it's never seen before. This is also
called inference. We've given you a set of images in the data/asl_images folder. Try opening it using the left
navigation and explore the images.

You'll notice that the images we have are much higher resolution than the images in our dataset. They are
also in color. Remember that our images in the dataset were 28x28 pixels and grayscale. It's important to
keep in mind that whenever you make predictions with a model, the input must match the shape of the data
that the model was trained on. For this model, the training dataset was of the shape: (27455, 28, 28, 1). This
corresponded to 27455 images of 28 by 28 pixels each with one color channel (grayscale).

2 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

Showing the Images

When we use our model to make predictions on new images, it will be useful to show the image as well. We
can use the matplotlib library to do this.

In [3]: import matplotlib.pyplot as plt

import matplotlib.image as mpimg

def show_image(image_path):
image = mpimg.imread(image_path)
plt.imshow(image, cmap='gray')

In [4]: show_image('data/asl_images/b.png')

Scaling the Images

The images in our dataset were 28x28 pixels and grayscale. We need to make sure to pass the same size
and grayscale images into our method for prediction. There are a few ways to edit images with Python, but
Keras has a built-in utility that works well.

In [5]: from tensorflow.keras.preprocessing import image as image_utils

def load_and_scale_image(image_path):
image = image_utils.load_img(image_path, color_mode="grayscale", ta
rget_size=(28,28))
return image

3 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [6]: image = load_and_scale_image('data/asl_images/b.png')

plt.imshow(image, cmap='gray')

Out[6]: <matplotlib.image.AxesImage at 0x7f8f6818dcc0>

Preparing the Image for Prediction

Now that we have a 28x28 pixel grayscale image, we're close to being ready to pass it into our model for
prediction. First we need to reshape our image to match the shape of the dataset the model was trained on.
Before we can reshape, we need to convert our image into a more rudimentary format. We'll do this with a
keras utility called image_to_array.

In [7]: image = image_utils.img_to_array(image)

Now we can reshape our image to get it ready for prediction.

In [8]: # This reshape corresponds to 1 image of 28x28 pixels with one color ch
annel
image = image.reshape(1,28,28,1)

Finally, we should remember to normalize our data (making all values between 0-1), as we did with our
training dataset:

In [9]: image = image / 255

Making Predictions
Okay, now we're ready to predict! This is done by passing our pre-processed image into the model's predict
method.

4 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [10]: prediction = model.predict(image)

print(prediction)

[[5.2839108e-35 1.0000000e+00 4.0649995e-35 1.1931706e-35 1.5361722e-

24
1.6491860e-16 2.2272560e-29 1.7765694e-30 9.5728675e-25 0.0000000e+
00
8.2485774e-26 1.6495214e-36 2.4929905e-37 0.0000000e+00 8.7300889e-
36
0.0000000e+00 5.6686917e-38 0.0000000e+00 0.0000000e+00 2.8069085e-
26
0.0000000e+00 5.0752984e-31 3.4684984e-21 2.0412960e-30]]

Understanding the Prediction

The predictions are in the format of a 24 length array. Though it looks a bit different, this is the same format as
our "binarized" categorical arrays from y_train and y_test. Each element of the array is a probability between
0 and 1, representing the confidence for each category. Let's make it a little more readable. We can start by
finding which element of the array represents the highest probability. This can be done easily with the numpy
library and the argmax (https://numpy.org/doc/stable/reference/generated/numpy.argmax.html) function.

In [11]: import numpy as np

np.argmax(prediction)

Out[11]: 1

Each element of the prediction array represents a possible letter in the sign language alphabet. Remember
that j and z are not options because they involve moving the hand, and we're only dealing with still photos.
Let's create a mapping between the index of the predictions array, and the corresponding letter.

5 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [12]: # Alphabet does not contain j or z because they require movement

alphabet = "abcdefghiklmnopqrstuvwxy"
dictionary = {}
for i in range(24):
dictionary[i] = alphabet[i]
dictionary

Out[12]: {0: 'a',

1: 'b',
2: 'c',
3: 'd',
4: 'e',
5: 'f',
6: 'g',
7: 'h',
8: 'i',
9: 'k',
10: 'l',
11: 'm',
12: 'n',
13: 'o',
14: 'p',
15: 'q',
16: 'r',
17: 's',
18: 't',
19: 'u',
20: 'v',
21: 'w',
22: 'x',
23: 'y'}

We can now pass in our prediction index to find the corresponding letter.

In [13]: dictionary[np.argmax(prediction)]

Out[13]: 'b'

Exercise: Put it all Together

Let's put everything in a function so that we can make predictions just from the image file. Implement it in the
function below using the functions and steps above. If you need help, you can reveal the solution by clicking
the three dots below.

6 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [14]: def predict_letter(file_path):

# Show image
FIXME
# Load and scale image
image = FIXME
# Convert to array
image = FIXME
# Reshape image
image = FIXME
# Normalize image
image = FIXME
# Make prediction
prediction = FIXME
# Convert prediction to letter
predicted_letter = FIXME
# Return prediction
return predicted_letter

Solution

Click on the '...' below to view the solution.

In [16]: def predict_letter(file_path):

show_image(file_path)
image = load_and_scale_image(file_path)
image = image_utils.img_to_array(image)
image = image.reshape(1,28,28,1)
image = image/255
prediction = model.predict(image)
# convert prediction to letter
predicted_letter = dictionary[np.argmax(prediction)]
return predicted_letter

7 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

In [17]: predict_letter("data/asl_images/b.png")

Out[17]: 'b'

Let's also use the function with the 'a' letter in the asl_images datset:

In [18]: predict_letter("data/asl_images/a.png")

Out[18]: 'a'

Summary

8 of 9 12/07/2021, 05:00 pm
04b_asl_predictions about:srcdoc

Great work on these exercises! You've gone through the full process of training a highly accurate model from
scratch, and then using the model to make new and valuable predictions. If you have some time, we
encourage you to take pictures with your webcam, upload them by dropping them into the data/asl_images
folder, and test out the model on them. For Mac you can use Photo Booth. For windows you can select the
Camera app from your start screen. We hope you try it. It's a good opportunity to learn some sign language!
For instance, try out the letters of your name.

We can imagine how this model could be used in an application to teach someone sign language, or even
help someone who cannot speak interact with a computer. If you're comfortable with web development,
models can even be used in the browser with a library called TensorFlow.js (https://www.tensorflow.org/js).

Clear the Memory

Before moving on, please execute the following cell to clear up the GPU memory.

In [19]: import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

Out[19]: {'status': 'ok', 'restart': True}

Next

We hope you've enjoyed these exercises. In the next sections we will learn how to take advantage of deep
learning when we don't have a robust dataset available. See you there! To learn more about inference on the
edge, check out this nice paper (https://research.fb.com/wp-content/uploads/2018/12/Machine-Learning-at-
Facebook-Understanding-Inference-at-the-Edge.pdf) on the topic.

Now that we're familiar building your own models and have some understanding of how they work, we will
turn our attention to the very powerful technique of using pre-trained models to expedite your work.

9 of 9 12/07/2021, 05:00 pm