G H Raisoni College of Engineering and Management, Pune

(An Autonomous institute affiliated by SPPU)

B.TECH FINAL YEAR PROJECT

SYNOPSIS

Guided by: Prof. Pramod Dhamdhere

Group ID: Group 5

Submitted by: Eklavya Kirote A21

Shashank Salgarkar A59
Shubham Chole A65
Sneha Gaware A66
Staff Use only:
Selected / Rejected Name of the Guide Name of the Project coordinator
Prof. Pramod Dhamdhere
1. Project Topic / Title:
Exploring A.I creativity from sketch to A.I generative Image using NCA

Finalized (Guide will write):

2. Details of Group Member:

Roll Div
Mobile
. Class isio Name of Student Email Id
Number
No. n
A21 B.Tech – A Eklavya kirote 9067632702 Eklavya.kirote.ai@ghrcem.raisoni.n
A et
A59 B.Tech – A Shashank Salgarkar 83799 50418 Shashank.salgarkar.ai@ghrcem.rais
A oni.net
A65 B.Tech – A Shubham Chole 93078 89189 Shubham.chole.ai@ghrcem.raisoni.
A net
A66 B.Tech - A Sneha Gaware 93075 22730 Sneha.gaware.ai@ghrcem.raisoni.n
A et

3. Project Overview / Background:

Introduction:
The emergence of AI has revolutionized various industries, including art and design. One fascinating
aspect of this intersection is the ability to transform human creativity into computational processes. Our
project focuses on leveraging AI techniques, particularly Neural Cellular Automata, to bridge the gap
between hand-drawn sketches and generative AI images.

Need of Work / Reason for Selection of this Project:

In recent years, the intersection of artificial intelligence (AI) and visual arts has witnessed remarkable
advancements. This project aims to contribute to this domain by developing an innovative AI program
capable of converting hand-drawn sketches into generative AI images. The proposed solution utilizes
Neural Cellular Automata (NCA), a promising framework within the field of AI, for its ability to
simulate complex patterns and behaviors. This report presents the design, implementation, and
evaluation of the developed AI system, discussing its architecture, functionalities, and future prospects.

Problem Statement:

The conventional process of transforming hand-drawn sketches into digital artwork is often time-
consuming and requires advanced skills in digital image editing software. Additionally, achieving
realistic and aesthetically pleasing results may be challenging, particularly for individuals with limited
artistic expertise. As such, there is a pressing need for an automated solution that streamlines this
process while maintaining the creative intent of the original sketches. Furthermore, the field of artificial
intelligence (AI) has witnessed remarkable advancements in recent years, particularly in the domain of
image generation. Neural Cellular Automata (NCA) have emerged as a promising framework for
simulating complex patterns and behaviors, offering a novel approach to generative art and image
synthesis. Hence, the problem statement revolves around the development of an AI program capable of
converting hand-drawn sketches into generative AI images using Neural Cellular Automata. This
program should address the following key challenges: Accuracy and Fidelity: The AI program must
accurately interpret hand-drawn sketches and generate AI images that closely resemble the original
artwork, preserving key elements such as lines, shapes, and textures. Automation and Efficiency: The
solution should automate the process of converting hand-drawn sketches to AI images, reducing the
manual effort required by users. It should also ensure efficiency in processing time and computational
resources.

Objective:

Develop an AI Program Infrastructure Create a robust software infrastructure capable of handling input
hand-drawn sketches and generating corresponding generative AI images using Neural Cellular
Automata (NCA) algorithms. Data Preprocessing Implement preprocessing techniques to clean and
normalize input sketch data, ensuring consistency and compatibility with the AI model. Model Training
Train the Neural Cellular Automata model using a diverse dataset of hand-drawn sketches and
corresponding AI images, optimizing model parameters for accurate and efficient image generation.
Integration of Neural Cellular Automata Integrate the trained NCA model into the AI program,
enabling real-time generation of AI images from input sketches while leveraging the capabilities of
Neural Cellular Automata for pattern synthesis and generative. User Interface Design an intuitive and
user-friendly interface for the AI program, providing seamless interaction for users to input hand-drawn
sketches, customizes parameters, and visualizes generated AI images. Accuracy and Fidelity:** Ensure
that the AI program accurately interprets input sketches and produces generative AI images that closely
resemble the original artwork, preserving key features, shapes, and textures. Automation and Efficiency
Automate the conversion process to minimize manual intervention, optimizing computational
efficiency and processing time for generating AI images from input sketches. Creative Enhancement
Features Implement creative enhancement features within the AI program, allowing users to explore
different artistic styles, effects, and variations in the generated AI images. Testing and Evaluation
Conduct thorough testing and evaluation of the developed AI program, assessing its accuracy,
efficiency, user-friendliness, and creative enhancement capabilities through quantitative metrics and
user feedback. Documentation and Dissemination Prepare comprehensive documentation detailing the
development process, implementation details, and usage instructions for the AI program. Disseminate
the findings through academic publications, presentations, and online platforms to contribute to the
broader research community. By accomplishing these objectives, the AI program aims to provide a
powerful and accessible tool for artists, designers, and enthusiasts to transform hand-drawn sketches
into captivating generative AI images, unlocking new avenues for creative expression and exploration
in the realm of digital art and design.

Benefits to the Surrounding/Society:

The development and deployment of an AI program capable of converting hand-drawn sketches into
generative AI images using Neural Cellular Automata (NCA) offer numerous benefits to both the
surrounding community and society at large Accessibility to Digital Art Creation: By automating the
process of converting hand-drawn sketches into digital artwork, the AI program lowers the barrier to
entry for individuals interested in digital art creation. It enables artists of all skill levels, including
beginners and those with limited access to traditional art supplies, to explore their creativity in the
digital domain. Empowerment of Artists and Designers: The AI program empowers artists and
designers by providing them with a versatile tool for generating AI images from their hand-drawn
sketches. It encourages experimentation with different artistic styles, effects, and variations, fostering
innovation and pushing the boundaries of traditional art forms. Enhancement of Digital Artistic
Expression: Through the integration of Neural Cellular Automata, the AI program offers unique
opportunities for enhancing digital artistic expression. Artists can leverage the capabilities of NCA
algorithms to create complex patterns, textures, and visual effects that would be challenging to achieve
manually, leading to the creation of visually stunning and captivating artworks. Promotion of Creativity
and Innovation: By democratizing access to advanced image generation techniques, the AI program
promotes creativity and innovation within the artistic community. It encourages artists to explore new
artistic concepts, experiment with unconventional approaches, and push the limits of their creative
imagination, ultimately driving forward the evolution of digital art and design. Educational Benefits:
The AI program can serve as a valuable educational tool for students and educators in art and design-
related fields. It provides a hands-on platform for learning about AI algorithms, image processing
techniques, and computational creativity, facilitating interdisciplinary exploration and collaboration
between art and technology disciplines. Cultural Preservation and Heritage Conservation: The AI
program can contribute to the preservation and conservation of cultural heritage by digitizing and
enhancing hand-drawn sketches depicting cultural artifacts, landmarks, and traditions. It enables artists
and cultural organizations to digitize and restore historical artworks, ensuring their longevity and
accessibility for future generations. Commercial Opportunities: The AI program opens up commercial
opportunities for artists, designers, and creative professionals by enabling them to produce high-quality
digital artworks efficiently. It facilitates the creation of custom artworks for various purposes, including
advertising, branding, merchandising, and digital entertainment, thereby expanding economic
opportunities within the creative industries. Social Impact: By fostering a vibrant and inclusive artistic
community, the AI program contributes to social cohesion and cultural enrichment.

4. Methodology- Proposed :

Data Collection and Preprocessing Gather a diverse dataset of hand-drawn sketches and
corresponding AI images. Preprocess the data to standardize formats, normalize pixel values, and
enhance image quality if necessary. Model Architecture Selection Choose a suitable architecture for
the Neural Cellular Automata model. Consider factors such as network depth, convolutional layers,
and recurrent connections to capture spatial and temporal dependencies within the input sketches.
Model Training Train the Neural Cellular Automata model using the preprocessed dataset. Utilize
optimization techniques such as stochastic gradient descent or Adam to minimize the loss function
and improve model convergence. Hyper parameter Tuning Fine-tune model hyper parameters,
including learning rate, batch size, and regularization techniques, to optimize performance and
generalization ability. Integration with Sketch Input Develop a mechanism to accept hand-drawn
sketches as input to the trained Neural Cellular Automata model. Convert input sketches into a format
compatible with the model architecture. Image Generation Implement the image generation process
within the Neural Cellular Automata framework. Iteratively apply the model to evolve the input
sketch into a generative AI image, capturing complex patterns and structures. Quality Assessment
Evaluate the quality of generated AI images using metrics such as structural similarity index (SSIM),
peak signal-to-noise ratio (PSNR), and perceptual metrics. Compare generated images with ground
truth AI images to assess fidelity and realism. Optimization and Deployment Optimize the
performance of the Neural Cellular Automata model for efficiency and scalability. Deploy the trained
model within the AI program infrastructure for real-time or batch processing of hand-drawn sketches
to generative AI images. User Interface Integration Integrate the Neural Cellular Automata-based
image generation module into the user interface of the AI program. Provide intuitive controls for users
to input sketches, adjust parameters, and visualize generated AI images. Testing and Validation
Conduct comprehensive testing and validation of the AI program using diverse datasets and user
feedback. Validate the accuracy, efficiency, and user experience of the system to ensure its
effectiveness in converting hand-drawn sketches to high-quality generative AI images.

Project Flowchart:
Project Workflow
 Advantages:
1. Streamlined Workflow: - project offers a streamlined workflow for artists and designers to
convert their hand-drawn sketches into digital artwork seamlessly. This eliminates the need for
manual digitization processes and accelerates the transition from traditional to digital art creation.

2. Accessibility: - By providing an automated solution for converting hand-drawn sketches to

generative AI images, your project enhances accessibility to digital art creation tools. Artists of all
skill levels, including beginners and those with limited digital art experience, can easily explore
digital art techniques using your AI program.

3. Creative Exploration: - AI program encourages creative exploration by allowing artists to

experiment with different artistic styles, effects, and variations in the generated AI images. This
fosters creativity and innovation, empowering artists to express their unique vision and artistic
identity through digital artwork.

4. Efficiency: - The use of Neural Cellular Automata enables efficient and parallel processing of input
sketches, leading to faster generation of AI images. This improves workflow efficiency and
productivity for artists, enabling them to focus more on the creative process rather than technical
intricacies.

5. Customization: - AI program offers customization options, allowing users to adjust parameters and
settings to achieve desired artistic effects. This level of customization enables artists to tailor the
output to their specific preferences and artistic goals, enhancing creative control and expression.

6. Artistic Preservation: - By digitizing hand-drawn sketches and transforming them into generative
AI images, your project contributes to the preservation and conservation of artistic heritage. It ensures
the longevity and accessibility of traditional artwork in the digital age, preserving cultural artifacts for
future generations.

8. Innovation: - project represents an innovative approach to digital art creation, leveraging cutting-
edge AI technology to push the boundaries of creative expression. By embracing novel methodologies
such as Neural Cellular Automata, your AI program paves the way for new possibilities in digital art
and design.

Limitations:

1. 1. Complexity of Hand-drawn Sketch Interpretation: - The accuracy of the AI program in

interpreting hand-drawn sketches heavily depends on the complexity and quality of the input. It may
struggle with highly detailed or abstract sketches, leading to inaccuracies or misinterpretations in the
generated AI images.

2. 2. Limited Artistic Diversity: - The AI program may have limitations in capturing diverse artistic
styles and nuances present in hand-drawn sketches. It might prioritize certain patterns or features over
others, resulting in a lack of diversity in the generated AI images.

3. Over fitting to Training Data: - If the Neural Cellular Automata model is trained on a limited
dataset of hand-drawn sketches, it may suffer from over fitting, where it memorizes specific patterns
in the training data without generalizing well to new sketches. This can lead to poor performance and
limited creativity in the generated AI images.
Applications:

1. Digital Art Creation: - The primary application of the project is enabling artists and designers
to create digital artwork from hand-drawn sketches efficiently. This includes illustrations,
concept art, character designs, and other forms of visual expression.

2. Educational Tool: - The project can serve as an educational tool in academic settings,
allowing students to learn about AI-driven image generation techniques and explore the
intersection of art and technology. It provides hands-on experience with cutting-edge AI
algorithms in a creative context.

3. Cultural Preservation: - Digitizing hand-drawn sketches and preserving them as generative

AI images contributes to the preservation and conservation of cultural heritage. Traditional
artwork can be archived, restored, and shared digitally, ensuring its longevity and accessibility
for future generations.

4. Creative Exploration: The AI program encourages creative exploration by enabling artists to

experiment with different artistic styles, effects, and variations in their artwork. It fosters
innovation and encourages artists to push the boundaries of their creativity.
5. Project requirements:

1. Functionality: The AI program should accurately convert hand-drawn sketches into

generative AI images using Neural Cellular Automata. It should provide options for users to
adjust parameters and customize the output according to their preferences.

2. User Interface: - Design a user-friendly interface for seamless interaction with the AI
program. Include features for uploading hand-drawn sketches, adjusting settings, and
visualizing the generated AI images.

3. Performance: - Ensure efficient processing and generation of AI images, minimizing

computational resources and processing time. Strive for high-quality output that closely
resembles the original sketches, with attention to detail and fidelity.

4. Scalability: - Design the AI program to handle varying input sizes and complexities,
accommodating different artistic styles and levels of detail. Ensure scalability to support a
large user base and accommodate future enhancements and updates.

5. Compatibility: - Ensure compatibility with multiple platforms and devices, including desktop
computers, laptops, tablets, and mobile phones. Optimize the user interface for different screen
sizes and resolutions to provide a consistent experience across devices.

6. Documentation: - Prepare comprehensive documentation detailing the project's objectives,

methodology, implementation details, and usage instructions. Include user guides, technical
specifications, and troubleshooting tips to assist users and developers.
6. References:

1. Title: Sketch Generation with RNN-based Variational Autoencoders

Authors: Yiwen Guo, Jianping Shi, Eric P. Xing
Link: https://ieeexplore.ieee.org/document/8516988/

2. Title: DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and
Caricature Modeling
Authors: Yi Yuan, Tong Sun, Chao Xu, Jianrui Cai, Xiangyang Ji
Link: IEEE Xplore

3. Title: Sketch-based 3D Shape Retrieval Using Convolutional Neural Networks

Authors: Haolin Chen, Guoxian Dai, Chenyang Zhu, Junwei Han, Yongtao Wang
Link: https://ieeexplore.ieee.org/document/9190963/

4. Title: Sketch-Based Image Retrieval via Siamese Convolutional Neural Network

Authors: Yonggang Qi, Yi-Zhe Song, Honggang Zhang, Jun Liu
Link: https://ieeexplore.ieee.org/document/7532801/

5. Title: Sketch Generation Using Recurrent Variational Autoencoders with Spatial Attention
Authors: Haoyu Wu, Li Tan, Lei Zhu, Qiu Chen
Link: https://ieeexplore.ieee.org/document/9163112/

6. Title: Sketch Recognition with Multi-Scale Convolutional Neural Networks

Authors: Huiqi Li, Hongbin Zhang, Wenjuan Gong, Xiangyang Ji
Link: IEEE Xplore

7. Title: Sketch-Based Image Retrieval via Deep Learning: Challenges and Solutions
Authors: Xiaoqiang Li, Shijian Lu, Minsi Wang, Hongfu Liu
Link: IEEE Xplore