VISION AI
Abstract— Vision AI is the automated
extraction of information from images.
Information can mean anything from 3D
models, camera position, object detection and
recognition to grouping and searching image
content. In this book we take a wide definition of
computer vision and include things like image
warping, de-noising and augmented reality.
Sometimes vision AI tries to mimic human
vision, sometimes uses a data and statistical
approach, sometimes geometry is the key to
solving problems.
1. INTRODUCTION
As humans, we are capable of understanding and
describing a scene encapsulated in an image. This
involves much more than detecting four people in
the foreground, one street, and several cars as in the
image below.
Fig. 1
Aside from that basic information, we are able to
understand that the people in the foreground are
walking, that one of them is barefoot — a curious
thing — and we even know who they are. We can
reasonably infer that they are not in danger of being
hit by a car and that the white Volkswagen is
poorly parked. A human would also have no
problem describing the clothes they are wearing
and, in addition to indicating the color, guessing at
the material and texture of each outfit.
These are also the skills a vision AI system needs.
In a few words, the main problem solved by
computer vision can be summarized as follows:
Given a two-dimensional image, a vision AI
system must recognize the present objects and their
characteristics such as shapes, textures, colors,
sizes, spatial arrangement, among other things, to
provide a description as complete as possible of the
image.
2. THE VISION AI
Vision AI is an interdisciplinary scientific field that
deals with how computers can gain high-level
understanding from digital images or videos. From
the perspective of engineering, it seeks to
understand and automate tasks that the human
visual system can do.
Vision AI tasks include methods for
acquiring, processing, analyzing and understanding
digital images, and extraction of high-
dimensional data from the real world in order to
produce numerical or symbolic information, e.g. in
the forms of decisions.  Understanding in this
context means the transformation of visual images
(the input of the retina) into descriptions of the
world that make sense to thought processes and can
elicit appropriate action. This image understanding
can be seen as the disentangling of symbolic
information from image data using models
constructed with the aid of geometry, physics,
statistics, and learning theory.
The scientific discipline of vision AI is concerned
with the theory behind artificial systems that
extract information from images. The image data
can take many forms, such as video sequences,
views from multiple cameras, multi-dimensional
data from a 3D scanner, or medical scanning
device. The technological discipline of vision AI
seeks to apply its theories and models to the
construction of computer vision systems.
3. WORKING
When Machine see the image of a person, we
apply neurons to the input and the model extract
the features of that person’s face by applying
some operations and as soon as that features is
extracted, we apply convolutional neural network
to make the model learn that face.
 4. FEATURES OF VISION AI

Vision AI is being used today in a wide variety of

real-world applications, which include:
4.1 Optical character recognition (OCR):
reading handwritten postal codes on letters(or
numbers) and automatic number plate recognition
(ANPR);

Fig. 2

Machine reads the image in a matrix format. Like it

reads image in a 3D matrix or say RGB format.
The features of the image is nothing but pixel
values of the matrix. We convert that matrix into
grayscale format or say 3D to 1D to extract the
features in short time, so training will be done in
less amount of time. As soon as it done, machine
sends the model into dense neural network and let
it finish training the features for that face. And at
last machine learns the features of that face and
hence model trained.
Fig. 3
4.2 Machine inspection:
rapid parts inspection for quality assurance using
stereo vision with specialized illumination to
measure tolerances on aircraft wings or auto body
parts or looking for defects in steel castings using
X-ray vision;

Fig. 4
4.3 Retail:
object recognition for automated checkout lanes
Fig. 3

Simply we can state, vision AI mimics natural

processes: retrieves visual information, handles it,
and interprets it. And state-of-the-art algorithms,
so-called neural nets used for vision AI tasks,
replicate natural neural networks.
Its all about the data in vision AI. If the data is not
proper, there will be no accuracy . The proper the
data, the better will be result.
 Fig. 5
4.4 3D model building (photogrammetry):
fully automated construction of 3D models from
aerial photographs used in systems such as Bing
Maps;
4.5 Medical imaging:
registering pre-operative and intra-operative
imagery (Figure 1.4d) or performing long-term
studies of people’s brain morphology as they age;

Fig. 8

5 TECHNIQUES OF VISION AI

5.1 IMAGE CLASSIFICATION

Fig. 6
4.6 Automotive safety:
detecting unexpected obstacles such as pedestrians
on the street, under conditions where active vision
techniques such as radar or lidar do not work well.
Autonomous driving also included.
Given a set of images that are all labeled with a
single category, we’re asked to predict these
categories for a novel set of test images and
measure the accuracy of the predictions. There are a
variety of challenges associated with this task,
including viewpoint variation, scale variation, intra-
class variation, image deformation, image
occlusion, illumination conditions, and background
clutter.
The most popular architecture used for image
classification is Convolutional Neural Networks
(CNNs). A typical use case for CNNs is where you
feed the network images and the network classifies
the data. CNNs tend to start with an input “scanner”
which isn’t intended to parse all the training data at
once. For example, to input an image of 100 x 100
pixels, you wouldn’t want a layer with 10,000
nodes.

Fig. 7
4.7 Match move:
merging computer-generated imagery (CGI) with
live action footage by tracking feature points in the
source video to estimate the 3D camera motion and
shape of the environment.
Fig. 9
5.2 OBJECT DETECTION Fig. 11
Object detection allows us to identify and locate
Object Tracking methods can be divided into 2
objects in an image or video. With this kind of
categories according to the observation model:
identification and localization, object detection can
generative method and discriminative method. The
be used to count objects in a scene and determine
generative method uses the generative model to
and track their precise locations, all while
describe the apparent characteristics and minimizes
accurately labeling them. Object detection is
the reconstruction error to search the object, such as
commonly confused with image recognition. So, it
PCA.
is necessary to distinguish between them. Image
recognition assigns an label to an image. Object
detection detects an object and draw a line around 5.4 SEMANTIC SEGMENTATION
it.

Central to Vision AI is the process of segmentation,

which divides whole images into pixel groupings
which can then be labelled and classified.

Particularly, Semantic Segmentation tries to
semantically understand the role of each pixel in the
image (e.g. is it a car, a motorbike, or some other
type of class?). For example, in the picture above,
apart from recognizing the person, the road, the
cars, the trees, etc., we also have to delineate the
boundaries of each object. Therefore, unlike
Fig. 10 classification, we need dense pixel-wise predictions
from our models.

5.3 OBJECT TRACKING

Object Tracking refers to the process of following a

specific object of interest, or multiple objects, in a
given scene. It traditionally has applications in
video and real-world interactions where
observations are made following an initial object
detection. Now, it’s crucial to autonomous driving
systems such as self-driving vehicles from
companies like Uber and Tesla.

Fig. 12

5.5 INSTANCE SEGMENTATION

Beyond Semantic Segmentation, Instance

Segmentation segments different instances of
classes, such as labelling 5 cars with 5 different
colors. In classification, there’s generally an image
with a single object as the focus and the task is to
say what that image is. But in order to segment 1990s. Multiplex recording devices were
instances, we need to carry out far more complex introduced, together with cover video surveillance
tasks. We see complicated sights with multiple for ATM machines.
overlapping objects and different backgrounds, and
we not only classify these different objects but also
identify their boundaries, differences, and relations
to one another!

Fig. 13

Fig. 13
6 HISTORY
2000s. Two researchers at MIT introduced the first
face detection framework (Viola-Jones) that works
in real-time. Google started testing robot cars on
1970s. When Vision AI first started out in the
roads. Google released Goggles, an image
early 1970s, it was viewed as the visual perception
recognition app for searches based on pictures
component of an ambitious agenda to mimic
taken by mobile devices. To help tag photos,
human intelligence and to endow robots with
Facebook began using facial recognition. Facial
intelligent behavior. At the time, it was believed by
recognition was used to help confirm the identity of
some of the early pioneers of artificial intelligence
Osama bin Laden after he is killed in a US raid.
and robotics (at places such as MIT, Stanford, and
Google Brain’s neural network recognized pictures
CMU) that solving the “visual input” problem
of cats using a deep learning algorithm. Google
would be an easy step along the path to solving
launched open-source Machine learning-system
more difficult problems such as higher-level
TensorFlow
reasoning and planning. According to one well-
known story, in 1966, Marvin Minsky at MIT
asked his undergraduate student Gerald Jay
Sussman to “spend the summer linking a camera to
7 DISTINGUISHING VISION AI
a computer and getting the computer to describe
what it saw” (Boden 2006, p. 781).5 We now know FROM RELATED FIELDS
that the problem is slightly more difficult than
that.6 What distinguished vision AI from the
already existing field of digital image processing
(Rosenfeld and Pfaltz 1966; Rosenfeld and Kak
It is important to understand that vision AI
1976) was a desire to recover the three-dimensional
accomplishes much more than other fields such as
structure of the world from images and to use this
image processing or machine vision, with which it
as a stepping stone towards full scene
shares several characteristics. Let’s have a look at
understanding.
the differences between the fields.
1980s. In the 1980s, a lot of attention was focused
on more sophisticated mathematical techniques for Image processing
performing quantitative image and scene analysis.
Image processing is focused on processing raw
images to apply some kind of transformation.
Usually, the goal is to improve images or prepare
them as an input for a specific task, while in vision
AI the goal is to describe and explain images
For instance, noise reduction, contrast, or rotation
operations, typical components of image
processing, can be performed at pixel level and do
not need a complex grasp of the image that allows
for some understanding of what is happening in it.
Machine vision
This is a particular case where vision AI is used to
perform some actions, typically in production or
manufacturing lines. In the chemical industry,
machine vision systems can help with the
manufacturing of products by checking the
containers in the line (are they clean, empty, and
free of damage?) or by checking that the final
product is properly sealed.
Vision AI
Vision AI can solve more complex problems
such as facial recognition (used, for example, by
Snapchat to apply filters), detailed image analysis
that allows for visual searches like the ones Google
Images performs, or biometric identification.
8. CHALLENGES OF VISION AI
Vision AI is difficult to implement because:
8.1 CLOUD IS NOT ENOUGH FOR MOST
USECASES
Artificial Intelligence is present in many areas of
our lives, providing visible improvements to the
way we discover information, communicate or
move from point A to point B. AI adoption is
rapidly increasing not only in consumer areas such
as digital assistants and self-driving vehicles, but
across all industries, disrupting whole business
models and creating new opportunities to generate
new sources of customer value.
Focusing on Visual AI, the number of use-cases for
applying AI that performs at human level or better
is increasing exponentially, given the fast-paced
advances in Machine Learning.
Visual AI encompasses techniques used in the
image processing industry to solve a wide range of
previously intractable problems by using Vision AI
and Deep Learning.
However, high innovation potential does not come
without challenges.
AI inference requires a considerable amount of
processing power, especially for real-time data-
intensive applications. AI solutions can be
deployed in cloud environments (Amazon
AWS, Google GCP, Microsoft Azure) in order to
take advantage of simplified management and
scalable computing assets.
Nevertheless, in most circumstances Cloud is not
the adequate environment for deploying Artificial
Intelligence.
 What if your solution needs to run real-
time and requires fast response times?
 How to operate a system that is mission
critical and running off-grid?
 How to handle the high operating costs of
analyzing massive data in the cloud?
 What about data privacy if sending and
storing video material in the cloud?
Therefore, vision AI solutions will need to be
deployed on edge endpoints for most use cases.
This allows to process the data where it is captured
while only the results (light data) are sent back to
the cloud for further analysis.
8.2 MOVING TO THE EDGE REQUIRES
HARDWARE KNOWLEDGE
For most use-cases, deploying AI solutions on
Edge devices is the only reasonable way to solve
the challenge. A fine example is a farming
analytics system. The system has to capture and do
inference for 30 images per second per camera. For
an average setup of 100 cameras, we get a volume
of 259.2 million images per day. This is highly
inefficient to process in the cloud.
The best option for this use-case is to run inference
in real-time at the Edge. Analyse the data where it
is being generated! And only communicate key
data points to the cloud backend for data
aggregation and further analysis. Edge computing
is considered to be a current key trend in IT
industry.
Considering the rapid growth of AI inference
capabilities in Edge hardware platforms (Intel
NUC, Intel NCS, Nvidia Jetson, ARM Ethos),
transferring the processing requirements from
Cloud to Edge becomes a very attractive option for
a wide range of businesses.
8.3 COMPLEXITY LETS MOST
ORGANIZATIONS FAIL AT SCALING
VISUAL AI
Even with the promise of great hardware support
for Edge deployments, developing a Visual AI
solution remains a complex process.
In a traditional approach, several of the following
building blocks may be necessary for developing
your solution at scale:
 Collecting input data specific to the
problem
 Expertise with Deep Learning tools
like Tensorflow, PyTorch, Keras, Caffe,  easily, no matter the device type and
MXnet for training and evaluating Deep
Learning models
 Selecting the appropriate hardware (e.g.
Intel, NVIDIA, ARM) and software
platforms (e.g. Linux, Windows, Docker,
Kubernetes) and optimizing DL models
for the deployment environment
 Managing deployments to thousands of
distributed edge devices
 Managing updates, data analysis and real-
time insights
 Knowledge about data privacy and
security best practices
There is a high level of development risk
associated with this approach. Especially when
considering development time, required domain
experts and difficulties in developing a scalable
infrastructure.
Fortunately, there’s a finer path to
make your vision a reality, one that allows you to
reduce your development costs and time to market
by an order of magnitude.
9. SOLUTIONS
Vision AI problems can easily be solved using
Machine Learning algorithms. Since Machine
Learning has expanded computers’ ability to
understand images and extract different
information from visual data. Below listed Visual
Programming can also play a vital role in solving
vision AI challenges.
Visual Programming: 
Use a visual approach to build complex Visual AI
solutions on the fly. The visual programming
approach can reduce development time by over
90%. It does not only greatly reduce the effort to
write code from scratch, but gives the visibility of
how the vision application works.
9.1 Device Management: 
Add and manage thousands of edge devices
architecture (amd64, aarch64, …). Create a
device image and flash it to your device to
make it appear in your workspace. Check
device health metrics, online or deployment
statuses without writing a single line of code.
9.2 Deployment Management: 
Use an integrated device management tool to
enroll and manage endpoint devices. Deploy
AI applications to numerous edge devices at
the click of a button. While you can focus on
the algorithm development, deployment,
 versioning and device management is taken their patients, monitor the evolution of diseases,
care of for you. and prescribe the right treatments. The technology
9.3 Modular Approach:  not only helps medical professionals to save time
on routine tasks and give more time to the patients.
Benefit from many pre-existing software
modules to build your own use case. Viso
Suite provides the most common deep learning
models for Object Detection, Image
Classification, Object Segmentation or
Keypoint Detection off-the-shelf. Select the
suitable model and create your application
with thousands of ready-to-choose logic
modules.
9.4 Flexibility where needed:

Add your own algorithms and code where

needed for your custom vision AI solution.
Only build code where it does not exist yet and
to market 10x faster and with a very limited
risk. Fig. 14

10.2 SECURITY

Using biometric analysis such as retinal and

10. APPLICATIONS fingerprint scanning to uniquely identify
individuals for security purposes. As cameras and
sensors proliferate, vision AI provides insights into
Vision AI technology of today is powered by deep public spaces and workplaces, airports and
learning algorithms that use a special kind of neural industrial sites, so that security and safety officers
networks, called convolutional neural network can make sense of a flood of images and data from
(CNN), to make sense of images. These neural both remote and high-traffic areas.
networks are trained using thousands of sample
images which helps the algorithm understand and
break down everything that’s contained in an
image. These neural networks scan images pixel by
pixel, to identify patterns and “memorize” them. It
also memorizes the ideal output that it should
provide for each input image (in case of supervised
learning) or classifies components of images by
scanning characteristics such as contours and
colors. This memory is then used by the systems as
the reference while scanning more images. And
with every iteration, the AI system becomes better Fig. 14
at providing the right output. Following are a few
areas in which vision AI technology is being used
or tested:
10.3 MANUFACTURING
10.1 HEALTH CARE
Inspecting manufacturing processes and finished
Diagnosing diseases by analyzing images obtained products fo non-conformance and defects. It helps
from CT scans and other medical imaging manufacturing process in several domains such as
processes. Vision AI is today assisting an time-efficiency, accuracy, repeatability, reduced
increasing number of doctors to better diagnose costs, post-pandemic value.

Fig. 15
10.4 TRANSPORTATION
Guiding autonomous vehicles by identifying
obstacles, people, and road signs along the way.
One of the best example is, Self-driving vehicles,
in order to navigate safely through the streets, must
be able to identify the obstacles, sign-posts, other
vehicles, and any person that may come in the way.
Fig. 16
11. FUTURE
With further research on and refinement of the
technology, the future of vision AI will see it
perform a broader range of functions. Not only will
vision AI technologies be easier to train but also be
able to discern more from images than they do
now. This can also be used in conjunction with
other technologies or other subsets of AI to build
more potent applications. For instance, image
captioning applications can be combined with
natural language generation (NLG) to interpret the
objects in the surroundings for visually challenged
people. Vision AI will also play a vital role in the
development of artificial general intelligence (AGI)
and artificial super intelligence (ASI) by giving them
the ability. to process information as well as or
even better than the human visual system
12. CONCLUSION
Based on the above discussion, it is clear
that vision AI technology will continue to be a very
useful tool in tackling a wide variety of challenges
in meat classification and quality prediction. Good
predictions and classification Considering the
capabilities of present-day vision AI, it might be
hard to believe that there are more benefits and
applications of the technology that remain
unexplored. The future of vision AI will pave the
way for artificial intelligence systems that are as
human as us. However, before doing so, there are
a few challenges that must be overcome, the
biggest of them being the demystification of the
black box of AI. That’s because just like other deep
learning applications, vision AI, while being
functionally effective, is undecipherable when it
comes to its inner workings.
9. REFERENCES
[1] Computer Vision: A Modern Approach.
Book by David Forsyth and Jean Ponce
[2] https://insights.daffodilsw.com/blog/vision-
ai-what-is-it-and-why-does-it-matter
[3] https://www.usmsystems.com
[4] Programming Computer Vision With
Python. Book by Jan Erik Solem.
[5]https://www.forbes.com/sites/cognitiveworl
d/2019/06/26/the-present-and-future-of-
computer-vision/?sh=70b13835517d
 [6]https://www.sciencedirect.com/topics/food- [8]https://tryolabs.com/resources/introductory-
science/computer-vision-technology guide-computer-vision

[7]https://viso.ai/deep-learning/why-computer-
vision-is-difficult/