Project Work:

Final-ISA (Review 4)
Project Title : Automatic Image Colourization using Generative Models

Project Guide : Prof. Suresh Jamadagni

Project Team : Mudit Jha - 01FB16ECS214
Saahil Jain - 01FB16ECS321
Sayantan Nandy - 01FB16ECS345
 Problem Statement

Automatic Image Colourization using Generative Models

The application of deep learning to this problem domain is a recent advent.
Until recently the established solution was to use CNN based models,
generally with mean-square loss functions.

We aim to apply generative models such as GANs to this particular problem

in order to improve upon the results obtained from CNNs.
 User Profile

● The problem of image colourization has applications ranging from

restoration of old images to developing of video colourization techniques
to colour old black-and white films/recordings.

● The advantages of having colours in these domains is that they are more
visually appealing and realistic than grayscale.
 Literature Survey

1. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-
to-image translation with conditional adversarial networks. 2016
2. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation. In
International Conference on Medical
Image Computing and Computer-Assisted Intervention. Springer, 2015.
3. http://irvlab.cs.umn.edu/projects/adversarial-image-colorization
Given resource provides an overview of some different GAN
architectures which have been or can be applied to this task.
 Literature Survey (contd.)

3. Sergey Ioffe and Christian Szegedy. Batch normalization:

Accelerating deep network training by reducing internal covariate
shift. In International Conference on
Machine Learning, 2015.
4. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung,
Alec Radford,
and Xi Chen. Improved techniques for training gans..
5. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks
for semantic segmentation.
 Proposed Solution

Our proposed solution is to try the two primary kinds of generative

models - autoencoders and GANs. The similarity between these
two is that they learn some latent distribution of the input which is
not something done by models like Convnets.
 Why is this solution better?

Generative models like GANs and Auto Encoders are becoming

more common for image-to-image translation tasks.

Although GANs are more difficult to train, the

generated images are much better in terms of colour
brightness and sharpness, whereas images generated
by ConvNet have patches of dimmer colour.
 Technologies and Methodologies
OpenCV: Specifically the cv2 Python package

Pandas,numpy: For manipulating the data arrays

Tensorflow: The core ML library

Keras: The higher level API used. It is now present as a part of

Tensorflow
 Dependencies and Risks

Moreover, the nature of the task requires access to high-end GPUs as

are used in similar work done.
The software required does not pose an issue as it is all open source.
The tricky nature of training GANs – as referenced in most of the
papers we read through – is likely to be a hindrance.
Based on the nature of the output obtained, we may have to opt with a
different end model(CNN for example).
System Architecture: Autoencoder
System Architecture: Autoencoder
System Architecture: GAN
System Architecture: GAN
 UI/Use Case

The use case of our project is:

● Restoration of old images in museums.

● Image colorization in black-white comics.

 Modules

● GAN based coloring Module

● Auto Encoder based coloring Module

● Color Image to Gray Scale Conversion Module

 Design Approach

Two models are used for colorization of gray-scale images. One of

them is AutoEncoder Based. The other model is GAN based.
 Design Constraints,Assumptions and Dependencies

Since it involves the training of two networks at the same time,

GANs are a very hard model to design and requires great expertise
for implementation.
 Test Strategy

We have divided the training and testing data in 5:1

ratio
 Implementation Details

Two Models were implemented

1. Generative Adversarial Network

2. Auto Encoder Decoder Model

Project Completion
 Project Results

For the autoencoder :

Total of 30 epochs with training loss of 0.1196
and validation loss of 0.2496

For GAN :
Total 200 epochs
Discriminator loss : 0.693
Generator loss : 0.693
Project Results
Project Results
 Project Results

Grey Scale for Gans

 Project Results

Original Image
 Project Results

Gan Output
 Planned Effort vs Actual Effort

Review 1 : Looked at existing solution i.e cnn model and

proposed a generative model approach

Review 2 : Implemented an simpler version of the final GAN

model

Review 3 : Implemented an autoencoder model and tested on

images

Review 4 : Implemented an GAN model and tested on images

 Lessons Learned

We learned a lot about generative models in

general and image processing techniques.
Thank You