4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

How to Pass Image Datasets to CNN Models

using Image Data Generations
Md Shahbaz Alam · Follow
4 min read · Nov 2, 2023

Listen Share More

“Data will talk to you if you’re willing to listen.” — Jim Bergeson

C onvolutional Neural Networks (CNNs) have revolutionized the field of

computer vision, enabling machines to understand and process visual data.
To train a CNN effectively, it’s crucial to prepare and feed high-quality image
datasets. In this blog post, we’ll explore how to use the ImageDataGenerator class to
efficiently pass image datasets to CNN models.

Understanding Image Data Generation

ImageDataGenerator is a powerful tool provided by the Keras library that allows for
on-the-fly data augmentation during the training of CNN models. Data
augmentation involves applying various transformations to the training images,
such as rotation, zooming, flipping, and more. This helps the model generalize
better to unseen data and reduces the risk of overfitting.

ImageDataGenerator methods
(1) Flow_from_directory:
The flow_from_directory() method allows you to read the images directly from the
directory and augment them while the neural network model is learning on the
training data.

The method expects that images belonging to different classes are present in
different folders but are inside the same parent folder.

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 1/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

➊ The root directory contains at least two folders one for train and one for the test.

➋ The train folder should contain n sub-directories

Open in app each containing images of
respective classes.
99+
Search
➌ The test folder should contain a single folder, which stores all test images.

Steps for Image generation using flow from directory

Step 1: Import the Libraries.

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator

Step 2: Set Up Your Directory Structure.

Organize your image dataset into directories. You should have separate folders for
training, validation, and test sets. Each of these folders should contain subfolders
for each class.

Step 3: Create Image Data Generators.

Next, you’ll create instances of ImageDataGenerator for the training, validation, and
test sets. You can specify the augmentation techniques you want to apply here.

train_datagen = ImageDataGenerator(
rescale=1./255,
https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 2/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)

val_datagen = ImageDataGenerator(rescale=1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

In the above code, we’re applying various transformations like rotation, shifting,
shearing, zooming, and horizontal flipping to the training data. For validation and
test data, we only rescale the images.

Step 4: Flow Data From Directories.

Now, you’ll use the flow_from_directory method to load the images from the
directories and apply the transformations specified by the data generators.

train_generator = train_datagen.flow_from_directory(
'dataset/train',
target_size=(150, 150),
batch_size=32,
class_mode='binary'
)

val_generator = val_datagen.flow_from_directory(
'dataset/validation',
target_size=(150, 150),
batch_size=32,
class_mode='binary'
)

test_generator = test_datagen.flow_from_directory(
'dataset/test',
target_size=(150, 150),
batch_size=32,
class_mode='binary'
)

Ensure that target_size matches the input size expected by your CNN model.

Step 5: Train Your CNN Model.

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 3/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

With the data generators in place, you can now use them to train your CNN model.

model.fit(
train_generator,
epochs=10,
validation_data=val_generator,
verbose=1
)

Step 6: Evaluate Your Model.

Finally, you can evaluate your model on the test set.

test_loss, test_accuracy = model.evaluate(test_generator)

print(f'Test accuracy: {test_accuracy}')

By following these steps, you’ve successfully passed image datasets to a CNN model
using image data generation using flow from directory. This process not only helps
in efficient data handling but also enhances the model’s performance through data
augmentation.
(2) Flow_from_dataframe:
The flow_from_dataframe() method takes the Pandas DataFrame and the path to a
directory and generates batches of augmented/normalized data.The downloaded
dataset contains .csv file. we will use this .csv file with flow_from_dataframe()
function.

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 4/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Steps for Image generation using flow from dataframe.

Step 1: Import the Libraries

import pandas as pd
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import img_to_array, load_img

Step 2: Load and Process the DataFrame

Assuming your DataFrame has two columns: ‘filename’ (containing the file paths of
the images) and ‘class’ (containing the corresponding class labels), you can load and
process

df = pd.read_csv('image_data.csv') # Load your DataFrame

# Process file paths and labels

df['filename'] = 'path_to_images/' + df['filename'] # Adjust the path to your
df['class'] = df['class'].astype(str) # Make sure 'class' is in string format

Step 3: Create an Image Data Generator.

Create an instance of ImageDataGenerator with the desired data augmentation
settings:

datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)

Step 4: Flow Data from DataFrame

Use the flow_from_dataframe method to load images from the DataFrame:

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 5/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

generator = datagen.flow_from_dataframe(
df,
x_col='filename',
y_col='class',
target_size=(150, 150), # Adjust to match your model's input size
batch_size=32,
class_mode='categorical', # Change to 'binary' if you have binary classes
shuffle=True
)

Step 5: Train Your CNN Model

Now, you can use the generator to train your CNN model:

model.fit(
generator,
epochs=10,
verbose=1
)

Step 6: Evaluate Your Model

Finally, evaluate your model

test_loss, test_accuracy = model.evaluate(generator)

print(f'Test accuracy: {test_accuracy}')

By following these steps, you’ve successfully passed image datasets to a CNN model
using image data generation flow from dataframe. This process not only helps in
efficient data handling but also enhances the model’s performance through data
augmentation.

Data Science Deep Learning Artificial Intelligence Machine Learning Cnn

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 6/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Follow

Written by Md Shahbaz Alam

26 Followers

😄
Data Scientist, ML/DL enthusiast, gamer. I share meaningful stories, hoping they bring value to you. If they
do, follow for more. 50 claps are free!

More from Md Shahbaz Alam

Md Shahbaz Alam

Demystifying Model Architectures in TensorFlow: A Comprehensive

Guide
“Artificial intelligence is the new electricity.” — Andrew Ng

4 min read · Nov 1, 2023

128

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 7/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Md Shahbaz Alam

Transfer Learning in Deep Learning: Leveraging Pre-trained Models for

Faster and Better Training
“Transfer Learning will be the next driver of Machine Learning success. — Andrew Ng”

5 min read · Nov 2, 2023

100

Md Shahbaz Alam

Exploring Cultural Shifts in a Globalized World.

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 8/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

In this interconnected world, understanding and respecting diverse cultures is not just an
option, it’s a necessity

3 min read · Nov 1, 2023

105 1

Md Shahbaz Alam

Sarcastic Comments Detection-REDDIT(An End-to-End Case study)

“Information is the oil of the 21st century, and analytics is the combustion engine.” — By Peter
Sondergaard

11 min read · Sep 8, 2021

297 2

See all from Md Shahbaz Alam

Recommended from Medium

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 9/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Sarka Pribylova

Image classification using CNN

Convolutional Neural Network (CNN) is a well established data architecture. It is a supervised
machine learning methodology used mainly in…

5 min read · Feb 25, 2024

Luís Fernando Torres in LatinXinAI

Convolutional Neural Network From Scratch

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 10/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

The most effective way of working with image data

21 min read · Oct 16, 2023

911 7

Lists

Predictive Modeling w/ Python

20 stories · 1060 saves

Natural Language Processing

1349 stories · 828 saves

Practical Guides to Machine Learning

10 stories · 1265 saves

data science and AI

40 stories · 122 saves

Vipul Sarode

U-net Unleashed: A step-by-step guide on implementing and training

your own segmentation model in…
In the last part, we saw how to build the U-Net from scratch. In this part, we will use the U-Net
and perform segmentations on real-world…

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 11/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

5 min read · Jan 20, 2024

186 1

Rodrigo Silva in Towards Data Science

Exploring Feature Extraction with CNNs

Using a Convolutional Neural Network to check specialization in feature extraction

8 min read · Nov 25, 2023

368 1

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 12/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Juan C Olamendy

Back to Basics: Feature Extraction with CNN

If you’ve ever wondered how computers can see and understand the world through images,
you’re in for a treat!

4 min read · Oct 19, 2023

Maahi Patel

The Complete Guide to Image Preprocessing Techniques in Python

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 13/14
4/3/24, 11:19 PM How to Pass Image Datasets to CNN Models using Image Data Generations | by Md Shahbaz Alam | Medium

Have you ever struggled with poor quality images in your machine learning or computer vision
projects? Images are the lifeblood of many Al…

11 min read · Oct 23, 2023

143 3

See more recommendations

https://msalamiitd.medium.com/how-to-pass-image-datasets-to-cnn-models-using-image-data-generations-b2d9497c7a35 14/14