Tensorflow Lite

Object Detection With TF Lite Model Maker & Android Application

Proposal - Google Summer Of Code 2024
Personal Details:
Name:
Sasidhara Kashyap Chaturvedula
IRC Nick Name:
Sasidhara Kashyap Chaturvedula
Telephone:
+91 8500419303
Email:
sasidhara.kashyap@gmail.com
GitHub:
https://github.com/sasidhara-kashyap0903
Country of Residence:
India
Timezone:
IST (India)
Language:
English
Project Proposal:
Project Title:
Object Detection With TF Lite Model Maker & Android Application
Abstract:
This project proposes leveraging the strengths of two emerging libraries, Tensorflow Lite Task
Library and Model Maker, to enhance the existing Tensorflow Lite object detection example.
The Task Library facilitates efficient data processing and optimized inference, while Model
Maker simplifies custom model training through transfer learning, thus reducing data

1
requirements and training time. This approach not only enables efficient object detection on
resource-constrained platforms but also fosters deeper understanding of model behavior and
image/video content through localized object identification. Consequently, contributing to GSoC
2024 will involve integrating object detection into the Tensorflow Lite Task Library and training
custom models for this purpose.
Background:
I am a motivated Computer Science student in my final year at GRIET College, affiliated
with JNTUH University. Eager to expand my horizons beyond the classroom, I embarked
on my first open-source journey during the Google Cloud Program, contributing to the
Dialogflow Chatbot Project. My passions lie in the realms of AI, Machine Learning, Deep
Learning, Android development, and Cloud Computing. Driven by this deep interest, I am
thrilled by the prospect of joining the Tensorflow Mentor Org and actively participating in
its vibrant community. My technical skillset spans diverse areas, honed through valuable
industry internships. I recently gained practical experience in Salesforce administration
and development through Smart Internz's Virtual Internship Program. Google Developers
and Future Ready Talent Intern programs further nurtured my Android development
expertise, culminating in a successful Virtual Internship.

I have written Four Medium Blogs on Guides to get selected for GSoC
2024:

Blog Links:

1) GSoC 2024 Beginner Guide PART I

2) GSoC 2024 Beginner Guide PART II

3) GSoC 2024 Intermediate Guide PART I

4) GSoC 2024 Intermediate Guide PART II

Task:

Project Reference Links:

● Detecting Object Using TF Lite Model Maker
● Tensorflow Lite
● TensorFlow Lite Model Maker
● Download Notebook

2
 ● Run in Google Colab
● GitHub Link
● GitHub Link 2
● Object Detection using TFLite Model Maker (Python)
● Object Detection using Android Application Model (Kotlin)
Step 1:
Choose an object detection model architecture:
This tutorial uses the EfficientDet-Lite0 model. EfficientDet-Lite[0-4] is a family of mobile/IoT-
friendly object detection models derived from the EfficientDet architecture.

Here is the performance of each EfficientDet-Lite model compared to each other.

Average
Model architecture Size(MB)* Latency(ms)**
Precision***
EfficientDet-Lite0 4.4 37 25.69%

EfficientDet-Lite1 5.8 49 30.55%

EfficientDet-Lite2 7.2 69 33.97%

EfficientDet-Lite3 11.4 116 37.70%

EfficientDet-Lite4 19.9 260 41.96%

* Size of the integer quantized models.

** Latency measured on Pixel 4 using 4 threads on CPU.
*** Average Precision is the mAP (mean Average Precision) on the COCO 2017 validation
dataset.
Step 2:
Load the dataset:

3
The Model Maker will take input data in the CSV format. Use the
object_detector.DataLoader.from_csv method to load the dataset and split them into the
training, validation, and test images.

● Training images: These images are used to train the object detection model to recognize
salad ingredients.
● Validation images: These are images that the model didn't see during the training process.
You'll use them to decide when you should stop the training, to avoid overfitting.
● Test images: These images are used to evaluate the final model performance.

You can load the CSV file directly from Google Cloud Storage, but you don't need to keep your
images on Google Cloud to use Model Maker. You can specify a local CSV file on your
computer, and Model Maker will work just fine.

Step 3:

Train the TensorFlow model with the training data:

● The EfficientDet-Lite0 model uses epochs = 50 by default, which means it will go
through the training dataset 50 times. You can look at the validation accuracy during
training and stop early to avoid overfitting.
● Set batch_size = 8 here so you will see that it takes 21 steps to go through the 175 images
in the training dataset.
● Set train_whole_model=True to fine-tune the whole model instead of just training the
head layer to improve accuracy. The trade-off is that it may take longer to train the
model.

Step 4:
Evaluate the model with the test data:
After training the object detection model using the images in the training dataset, use the
remaining 25 images in the test dataset to evaluate how the model performs against new data it
has never seen before.

As the default batch size is 64, it will take 1 step to go through the 25 images in the test dataset.

The evaluation metrics are the same as COCO.

Step 5:

Export as a TensorFlow Lite model:

4
Export the trained object detection model to the TensorFlow Lite format by specifying which
folder you want to export the quantized model to. The default post-training quantization
technique is full integer quantization.

Step 6:
Evaluate the TensorFlow Lite model:
Several factors can affect the model accuracy when exporting to TFLite:

● Quantization helps shrink the model size by 4 times at the expense of some accuracy
drop.
● The original TensorFlow model uses per-class non-max suppression (NMS) for post-
processing, while the TFLite model uses global NMS which is much faster but less
accurate. Keras outputs a maximum of 100 detections while Tflite outputs a maximum of
25 detections.

Therefore you'll have to evaluate the exported TFLite model and compare its accuracy with the
original TensorFlow model.

This model can be integrated into an Android or an iOS app using the ObjectDetector API of
the TensorFlow Lite Task Library.

See the TFLite Object Detection sample app for more details on how the model is used in a
working app.

Full Description:

Object Detection is a fascinating field of computer vision that involves identifying and
locating objects of interest within images or videos. Imagine you're looking at a photo of a
bustling city street. Your eyes effortlessly spot cars, pedestrians, buildings, and even traffic
lights. Object detection aims to replicate this human ability with the power of computers.

5
The flow diagram you sent me shows the process of creating and using a training
set for object detection. Here's a step-by-step explanation:

● Dataset Collection: In this step, you collect the data that you will use to
● Splitting dataset: The labeled dataset is then split into two parts: a training set
and a testing set. The training set is used to train the object detection model,
while the testing set is used to evaluate the performance train your object
detection model. This data can come from a variety of sources, such as images,
videos, or point clouds.
● Dataset Labeling: Once you have collected your data, you need to label it.
This means that you need to identify the objects that are present in each image
or video, and then draw bounding boxes around them of the model.
● Evaluation: This step is used to evaluate the performance of the object
detection model on the testing set. The model is evaluated by measuring its
accuracy, precision, and recall.
● Object Detection: This block represents the object detection model itself. The
model is trained on the training set and then used to detect objects in new
images or videos.
● Network Training: This block represents the process of training the object
detection model. The model is trained using a gradient descent algorithm that
minimizes the loss function.
● Hyperparameters tuning: This block represents the process of tuning the
hyperparameters of the object detection model. Hyperparameters are
parameters that control the learning process of the model.
● Transfer Learning: This block represents the process of using a pre-trained
object detection model as a starting point for training a new model. This can be
a helpful way to improve the accuracy of the model without having to collect
and label a large amount of new data.

6
Pipeline for developing the TensorFlow Lite Object Detection Model:

Screenshot:

7
Timeline:

● Before Community Bonding Period (May 1st, 2024):

a) Familiarizing myself with the codebase.

b) Continuing to contribute to Object Detection using Tensorflow Lite Model Maker (Python) and
Android-based App model (Kotlin).

c) Adding documentation for Tensorflow Lite Model Maker (Python) and Android-based App
model (Kotlin).

d) Enhance my understanding more of Tensorflow Lite Task Library.

● Community Bonding Period (May 1st, 2024 - May 26th, 2024):

a) Discuss with mentors to work out more details about this plan.

b) Getting familiar with the structure and codebase of Tensorflow Lite Model Maker (Python) and
Android-based App model (Kotlin).

c) Finishing Tensorflow Lite Model Maker (Python) and Android-based App model (Kotlin) with
documentation.

d) Enhance my understanding more of Tensorflow Lite Task Library.

8
● Coding Period (May 27th, 2024):

I will push daily commits and send weekly pull requests during the coding period. I
will test thoroughly on Tensorflow Lite Model Maker (Python) and Android-based
App model (Kotlin) and make sure the outcome is as it was planned. I will share
screenshots and GIFs to show the actual working of the app.

● Mid Evaluation (July 8th, 2024 - July 22nd, 2024)-I:

a) Study the design pattern of Object Detection using TF Lite Model Maker in
Python.

b) Implement the TF Task Library of Object Detection using TF Lite Model

Maker in Python.

c) Assimilate the changes and improvements suggested by mentors.

d) Build and Test (Object Detection using TF Lite Model Maker in Python).

e) Clean up the code and write documentation.

f) Discuss with mentors about my progress and schedule.

g) Preparing codebase for mid-evaluation.

h) Fixing Bugs and Testing.

● Mid Evaluation (July 22nd, 2024 - August 19th, 2024)-II:

a) Study the design pattern of Object Detection using an Android Example in Kotlin.

b) Implement the TF Task Library of Object Detection using Android Example in Kotlin.

c) Assimilate the changes and improvements suggested by mentors.

d) Build and Test (Object Detection using Android Example in Kotlin).

e) Clean up the code and write documentation.

9
 f) Discuss with mentors about my progress and schedule.

g) Preparing codebase for mid-evaluation.

h) Fixing Bugs and App Testing.

● Final Evaluation (August 19th, 2024 - September 23rd, 2024)-I:

a) Completing the documentation of Object Detection using TF Lite Model Maker in Python.

b) Reviewing the code, and documentation and submitting for final evaluation.

● Final Evaluation (September 23rd, 2024 -November 11th, 2024)-II:

a) Completing the documentation of Object Detection using TF Lite Model Maker in Python &
Android Example in Kotlin.

b) Reviewing the entire code, entire documentation and submitting for final evaluation.

Previous Contributions To Open Source & Projects Worked On:

● Mini Project
● Salesforce Guided Project
● Movie Ticket Booking Chatbot
● Pizza Chatbot
● Android Application Project

Prior Experience With Machine Learning:

I have been doing Machine Learning when I was just in my first year of graduation. At that
time, I started learning about mathematical intuition behind Linear Regression, K Nearest
Neighbor, Logistic Regression, etc. After learning and practicing algorithms, I got hands-on
with SKLearn.

The previous year, I started deep learning. First I started learning the mathematical part of the
activation functions, forward propagation, backward propagation, etc. At first, I started with

10
Artificial Neural Networks, and then I moved to Computer Vision. My main interest was in
computer vision more than in other fields.

I have also been working on Machine Learning Projects with the AAC Team in my college
where I do tasks related to Computer Vision, Model Optimisation, and deploying models in
edge devices or servers.

Courses I Have Taken:

● Google Cloud Digital Leader Training

● Google Professional Workspace Administrator
● Salesforce Developer Virtual Internship Certificate
● Salesforce Admin Virtual Internship Certificate
● Google Android Developer Internship Certificate

Most Exciting Tensorflow Product:

The TensorFlow product I like the most is TensorFlow Lite. The support of edge
devices makes this product amazing because, in 2021, everyone has a mobile phone
that can act as an edge device. TF-lite files are so lightweight that they can be found
in edge devices like smartphones.TensorFlow Lite is a set of tools that enables on-
device machine learning by helping developers run their models on mobile,
embedded, and edge devices.

About Open Source Technology:

Open Source Technologies lets people collaborate. Every developer learns how to write code
in real-world projects. The proper cycle i.e from creating the issue then getting assigned and
then solving the bug and making a pull request is a journey itself. It lets people add value to
the company and even developers get the taste of writing clean and good code. It also gives
motivation and happiness to fellow developers when pull requests get merged. The concept

11
of collaborative learning leads the community to grow beautifully. These are my thoughts on
open-source technology.

Since my childhood, I have been quite fascinated by programming and to what extent was a
few lines of code capable of. From predicting your music taste to deciding your feed
recommendations, back when I didn't even know the proper terms, to me they were just
algorithms. Just a few years ago, the field was still in its infancy. But today, it has grown into
something massive. You can find machine learning everywhere. In social media, music
streaming services, autonomous vehicles… even in your cell phone camera! Even though this
is exciting, many of us are still confused with all the terminologies and the new terms that we
hear in everyday life. So you can say that I have been excited about machine learning before
even developers coined the term "Machine Learning"!

Open Source Software is software with source code that anyone can inspect, modify, and
enhance.

"Source Code" is the part of the software that most computer users don't ever see; it's the code
computer programmers can manipulate to change how a piece of software—a "program" or
"application"—works. Programmers who have access to a computer program's source code can
improve that program by adding features to it or fixing parts that don't always work correctly.

Area Of Focus:

My major in my college is Computer Science and Engineering (CSE). With computer

programming from primary school, I found it interesting and meaningful for my life. I found
a purpose for myself. I started learning a lot of new things at an early phase. The most I
found were Machine Learning, Python, Deep Learning, and Computer Vision. So yes I
started learning in-depth. Its application has many scopes, almost everywhere, from
automobiles to smartphones. I would love to grow here and do a lot of research on this.

Social Media Links:

● LinkedIn

12
 ● Twitter
● Medium
● GitHub

Why Tensorflow?

My strong passion for TensorFlow and desire to contribute to its development make Tensorflow
Mentor Org an ideal avenue for my participation in Google Summer of Code (GSoC). Their
diverse project portfolio ensures a high probability of finding an endeavor that aligns with my
skill set and interests.

Beyond project variety, Tensorflow Mentor Org prioritizes excellent mentorship, a key factor in
GSoC success. I stand to benefit from the guidance of experienced mentors who can not only
steer me through my selected project but also equip me with new skills.

The thriving and expansive Tensorflow community presents a valuable resource for GSoC
participants like myself. This network facilitates knowledge sharing, allowing me to connect
with other developers, seek clarification on any doubts, and receive constructive feedback on my
work.

The project spectrum offered by Tensorflow Mentor Org caters to various skill levels, ranging
from beginner-friendly options to more complex challenges. This flexibility allows me to choose
a project that perfectly matches my current skillset and aligns with my aspirations.

Furthermore, GSoC presents an opportunity to pursue specific professional goals I hold, such as
acquiring deep learning expertise, honing my open-source development skills, and bolstering my
professional portfolio. Tensorflow Mentor Org's supportive environment and comprehensive
resources offer the ideal platform for achieving these aspirations.

Why Me?

I approach the 12-week program with unwavering dedication, acknowledging the potential for
exceeding those requirements through additional learning, planning, and communication.

13
Maintaining active engagement with my mentor and the community is paramount for project
success, while adhering to established milestones fosters consistent progress and bolsters my
support network.

Beyond technical proficiency, I prioritize striving for excellence in code quality, documentation,
and overall contributions that meaningfully benefit the TensorFlow projects. I actively seek new
learning opportunities to deepen my understanding of TensorFlow within the project context and
readily embrace constructive criticism and feedback from both my mentor and the community as
invaluable tools for continuous improvement.

Engaging with the TensorFlow community is integral to my experience. I will actively

participate by answering questions, helping others learn, and documenting my journey to inspire
future GSoC participants. Additionally, I aim to foster strong relationships with mentors and
fellow contributors to explore collaborative opportunities beyond the program's duration.

My commitment to TensorFlow extends beyond GSoC. I am passionate about contributing

further and maintaining ongoing engagement with the community. I believe in sharing my GSoC
experience to encourage others to participate in open-source projects and utilizing the acquired
skills and knowledge to broaden my technical expertise and enhance my career prospects.

Acknowledgment:

We extend our deepest gratitude to the dedicated TensorFlow GSoC mentors and the inspiring
community of past GSoC participants. Your unwavering commitment, expert guidance, and
invaluable contributions are the cornerstones of TensorFlow's future and the driving force behind
nurturing the next generation of open-source champions.

The TensorFlow GSoC program flourishes due to the exceptional mentorship provided by
experienced developers and researchers. They generously share their knowledge, time, and
encouragement, empowering students to embark on a transformative journey into the world of
TensorFlow and open-source development. The accomplishments of past GSoCers illuminate the
path forward. Their groundbreaking projects, insightful documentation, and active community

14
engagement set a high standard and inspire us all to push the boundaries of what's possible with
TensorFlow.

15