UNIT IV SEGMENTATION BY Lecture 8Hrs

CLUSTERING

What is Segmentation, Human Vision: Grouping and Gestalt,

Applications: Shot Boundary Detection and Background Subtraction.
Image Segmentation by Clustering Pixels, Segmentation by
Graph-Theoretic Clustering. The Hough Transform, Fitting Lines,
Fitting Curves

1.1 What is Segmentation?

Segmentation in computer vision refers to the process of dividing an image

into meaningful and semantically coherent regions or segments. The goal
of image segmentation is to assign a label or category to each pixel in the
image, grouping together pixels that belong to the same object or region
while separating those belonging to different objects or regions.

Image segmentation plays a crucial role in various computer vision tasks,

including object recognition, object tracking, scene understanding, and
image editing. By segmenting an image, we can extract important
information about the objects present in the scene, their boundaries, and
their spatial relationships.

There are different types of image segmentation techniques, including:

1. **Thresholding**: This technique involves setting a threshold value and

classifying each pixel as foreground or background based on its intensity or
color.

2. **Edge-based segmentation**: Here, edges or boundaries in the image

are detected using techniques like gradient operators, edge detectors (e.g.,
Canny edge detection), or contour detection algorithms (e.g., active
contours or level sets).

3. **Region-based segmentation**: This approach groups pixels into

regions based on similarity measures such as color, texture, or other image
properties. Techniques like region growing, region splitting and merging, or
graph cuts are commonly used.
4. **Semantic segmentation**: This form of segmentation aims to assign a
semantic label to each pixel in the image, classifying it into specific object
classes or categories (e.g., person, car, tree). Deep learning-based
methods, such as convolutional neural networks (CNNs) and their variants
(e.g., U-Net, FCN), have achieved significant advancements in semantic
segmentation tasks.

5. **Instance segmentation**: In this technique, each individual instance of

an object is segmented and differentiated from others. It not only provides
segmentation masks for each object but also distinguishes between
separate instances of the same object class (e.g. , identifying and
segmenting multiple cars in an image).

Segmentation algorithms vary in complexity and performance, depending

on the specific task and image characteristics. Deep learning techniques,
especially convolutional neural networks, have revolutionized image
segmentation by achieving state-of-the-art results on various benchmark
datasets and real-world applications.

1.2 Human Vision

Human vision plays a significant role in computer vision, which is the field of
study focused on enabling computers to understand and interpret visual
information. Computer vision aims to replicate or augment human vision
capabilities using algorithms, machine learning, and image processing
techniques.
Here are a few ways in which human vision influences computer vision:

1. Image Acquisition: Computer vision systems typically rely on image

acquisition devices such as cameras to capture visual data. The design
and specifications of these devices often take inspiration from human
vision, including factors like resolution, field of view, and color sensitivity.

2. Image Preprocessing: Computer vision algorithms often apply

preprocessing techniques to enhance and prepare images for further
analysis. These techniques are
designed to mimic the early stages of human vision, including tasks like
noise reduction, contrast enhancement, and image normalization.

3. Feature Extraction: Just as humans identify objects based on certain

features, such as shapes, colors, and textures, computer vision algorithms
extract relevant features from images to recognize and classify objects.
These features may be derived from edges, corners, textures, or other
visual characteristics, similar to how humans identify objects based on their
visual attributes.

4. Object Detection and Recognition: Human vision allows us to effortlessly

detect and recognize various objects in our environment. Similarly,
computer vision aims to detect and recognize objects within images or
video streams. Techniques like object detection, object tracking, and object
recognition are employed to identify and categorize objects, following
similar principles to those used by humans.

5. Scene Understanding: Human vision allows us to understand and

interpret complex visual scenes, including spatial relationships, context,
and semantics. Computer vision strives to achieve similar scene
understanding capabilities. For instance, algorithms are developed to infer
scene depth, estimate object poses, and recognize activities or events
occurring in a visual scene.

6. Human-Computer Interaction: Computer vision also contributes to

human-computer interaction by enabling computers to interpret and
respond to human gestures, facial expressions, and other visual cues. This
involves analyzing visual data to detect and track human body parts or
facial features, which facilitates natural and intuitive interaction between
humans and machines.

It's important to note that while computer vision aims to emulate aspects of
human vision, it also encompasses unique techniques and approaches that
are tailored to solving specific problems and challenges associated with
visual data analysis.

1.3 Grouping and Gestalt

Grouping and Gestalt principles are concepts derived from cognitive
psychology that have been applied to computer vision to aid in object
recognition and scene understanding. These principles help to explain how
humans perceive and organize visual information, and they have proven to
be useful in developing algorithms for computer vision tasks.

Grouping refers to the process of organizing visual elements into coherent

objects or groups based on their visual characteristics. This concept helps
to explain how individual elements are perceived as belonging together,
forming meaningful structures. In computer vision, grouping is often
employed in tasks such as object detection, segmentation, and tracking.

There are several principles of grouping that have been identified:

1. Proximity: Elements that are close to each other in space are more likely
to be perceived as belonging together. This principle is commonly used in
clustering algorithms to group pixels or regions based on their spatial
proximity.

2. Similarity: Elements that are similar in terms of color, texture, shape, or

other visual attributes are grouped together. This principle can be utilized to
segment objects in an image based on their visual similarity.

3. Continuity: Elements that form smooth, continuous curves or lines are

perceived as belonging together. This principle is often used in edge
detection algorithms to trace contours and boundaries of objects.

4. Closure: When presented with incomplete or fragmented visual

information, humans tend to mentally complete the missing parts to form
complete objects. Closure is often used in image completion or inpainting
tasks, where missing regions are inferred based on the surrounding
context.
5. Symmetry: Symmetrical elements are grouped together based on their
mirror-image properties. This principle is utilized in tasks such as shape
recognition or symmetry detection.

Gestalt principles complement grouping principles by focusing on the

holistic perception of visual information. These principles include:

1. Figure-Ground: The visual field is perceived as consisting of a

foreground (figure) and background (ground). This principle helps in
segmenting objects from the background in computer vision tasks like
object recognition or scene understanding.

2. Similarity: Similar elements are perceived as belonging to the same

group or category. This principle aids in clustering or categorizing visual
elements based on their similarities.
3. Common Fate: Elements that move together or share a common motion
are perceived as belonging together. This principle is utilized in
motion-based object tracking or activity recognition.

By incorporating these grouping and Gestalt principles into computer vision

algorithms, researchers aim to mimic human perception and enhance the
ability of machines to recognize objects, understand scenes, and organize
visual information in a more human-like manner.

1.4 Applications: Shot Boundary Detection and Background Subtraction.

Shot Boundary Detection and Background Subtraction are two important

applications in the field of computer vision. They are widely used in video
processing and analysis for various purposes, such as video editing,
surveillance systems, and object tracking.

1. Shot Boundary Detection:

Shot Boundary Detection aims to identify the boundaries between different
shots or scenes in a video. A shot refers to a continuous sequence of
frames captured without any interruption. Shot boundary detection is crucial
for video analysis, editing, and indexing. It allows for automatic
segmentation of a video into shots, which can be used for various
purposes, such as content-based video retrieval, video summarization, and
scene understanding.

The most common types of shot boundaries are:

- Cut: A direct transition from one shot to another without any noticeable
effect or transition.

- Fade: A gradual change in the intensity of the image, usually

accompanied by a change in scene content.

- Dissolve: A gradual transition where one shot fades out while another
shot fades in simultaneously.

- Wipe: A transition where one shot replaces another by moving across the
frame.

Shot boundary detection algorithms typically analyze frame-to-frame

differences or features to detect these transitions. Techniques such as pixel
intensity comparison, color histogram comparison, motion analysis, and
optical flow analysis can be employed to identify shot boundaries
accurately.

2. Background Subtraction:

Background Subtraction is a technique used to extract foreground objects

or regions of interest from a video sequence by separating them from the
background. It is commonly used in surveillance systems, video analytics,
and motion detection applications.

The basic idea behind background subtraction is to model the static

background of a video and identify the differences between the current
frame and the background model.
Any significant variation is considered as foreground, representing moving
objects or events in the scene.

There are several approaches to background subtraction, including:

- Frame differencing: This method computes the absolute difference

between consecutive frames to detect changes.

- Statistical modeling: Techniques like Gaussian Mixture Models (GMM) or

Kernel Density Estimation (KDE) can be used to model the background and
identify deviations.

- Learning-based methods: Machine learning algorithms, such as neural

networks, can be trained to learn the background model and identify
foreground objects.

Background subtraction algorithms can handle challenges like illumination

changes, shadows, and dynamic backgrounds, but they may also produce
false positives or miss certain objects in complex scenarios.

Both shot boundary detection and background subtraction are fundamental

tasks in video analysis and have numerous practical applications in various
domains, including video editing, surveillance, video indexing, and
content-based retrieval.

1.5 Image Segmentation by Clustering Pixels

Image segmentation is the process of partitioning an image into multiple

regions or segments to extract meaningful information from the image. One
approach to image segmentation is clustering pixels based on their
similarity.

In computer vision, clustering algorithms such as k-means, mean shift, or

hierarchical clustering can be used for pixel clustering and image
segmentation. Here's a general overview of how pixel clustering works for
image segmentation:
1. Preprocessing: The first step is to preprocess the image to enhance
features or reduce noise, if necessary. Common preprocessing techniques
include resizing, smoothing, and color space conversion.

2. Feature extraction: Next, relevant features are extracted from each pixel
or a set of pixels in the image. These features can include color values,
texture, gradient, or any other attributes that help distinguish different
regions in the image.

3. Clustering: Once the features are extracted, clustering algorithms are

applied to group similar pixels together. The choice of clustering algorithm
depends on the specific requirements of the segmentation task.

- K-means clustering: This algorithm aims to partition the pixels into k

clusters, where k is predefined. It iteratively assigns each pixel to the
cluster with the nearest mean value and recalculates the mean after each
assignment.

- Mean shift clustering: This algorithm does not require the number of
clusters to be predefined. It starts by randomly selecting initial seed points
and iteratively shifts each seed point towards the high-density region of
pixels until convergence. Pixels that converge to the same mode are
assigned to the same cluster.

- Hierarchical clustering: This algorithm builds a hierarchy of clusters by

either merging or splitting existing clusters based on a similarity measure. It
can result in a tree-like structure known as a dendrogram, which can be cut
at different levels to obtain the desired number of segments.

4. Post-processing: After clustering, post-processing techniques can be

applied to refine the segmentation results. Common post-processing steps
include morphological operations (e.g., erosion and dilation) to remove
noise or fill gaps between segments.

5. Visualization or further analysis: Finally, the segmented regions can be

visualized using different colors or labels to distinguish them. These
segments can be further
analyzed or used for various computer vision tasks like object detection,
tracking, or image understanding.

It's important to note that pixel clustering for image segmentation is just one
approach among many, and its effectiveness depends on the specific
characteristics of the images and the clustering algorithm used. Other
techniques, such as graph-based segmentation, watershed segmentation,
or deep learning-based methods, can also be employed for image
segmentation, depending on the complexity and requirements of the task.

1.6 Segmentation by Graph-Theoretic Clustering

Segmentation by graph-theoretic clustering is a method used in computer

vision to partition an image or a scene into meaningful regions or objects. It
is based on the principles of graph theory, which models the image as a
graph, where the nodes represent pixels or image elements, and the edges
represent the relationships or similarities between them.

The process of segmentation using graph-theoretic clustering typically

involves the following steps:

1. Graph Construction: A graph is constructed based on the image data.

Each pixel or image element is represented as a node in the graph, and the
edges between nodes are defined based on the similarity or proximity
between the corresponding image elements. Common approaches include
connecting neighboring pixels or using a similarity measure such as color,
texture, or intensity.

2. Similarity Measurement: The similarity between image elements is

computed based on certain criteria, depending on the specific application.
For example, if the goal is to segment an image based on color, the
similarity measure could be the Euclidean distance between color values.
3. Graph Partitioning: Graph partitioning algorithms are applied to divide
the graph into clusters or segments. The goal is to group similar image
elements together while
keeping dissimilar elements separated. Popular graph partitioning
techniques include spectral clustering, normalized cuts, and
minimum-cut/max-flow algorithms.

4. Region Formation: Once the graph is partitioned, the resulting clusters or

segments are merged to form coherent regions or objects. This step may
involve further processing, such as region growing or region merging, to
refine the segmentation results.

5. Post-processing: Finally, post-processing steps can be performed to

enhance the segmentation results. This may include applying smoothing
filters, removing small or noisy regions, or incorporating prior knowledge or
constraints to improve the segmentation quality.

Graph-theoretic clustering-based segmentation methods have been

successfully applied in various computer vision tasks, such as image
segmentation, object detection, and image retrieval. They offer the
advantage of capturing both local and global image relationships, allowing
for more accurate and robust segmentation results compared to traditional
pixel-based approaches. However, the choice of graph construction,
similarity measure, and partitioning algorithm can significantly affect the
segmentation performance, and careful parameter tuning is often required
to obtain optimal results for specific applications.

1.7 The Hough Transform

The Hough Transform is a popular technique in computer vision used for

detecting geometric shapes, particularly lines and circles, in digital images.
It was first introduced by Paul Hough in 1962 and has since been widely
adopted in various computer vision applications.

The primary goal of the Hough Transform is to identify specific shapes that
can be represented mathematically in an image, even when they are
distorted, incomplete, or
affected by noise. It works by converting the image space into a parameter
space, where each point in the parameter space corresponds to a possible
instance of the desired shape in the image.

Let's focus on the Hough Transform for line detection, as it is the most
commonly used variant. The steps involved in the Hough Transform for
lines are as follows:

1. Edge Detection: The first step is typically to perform an edge detection

algorithm, such as the Canny edge detector, to extract the edges from the
input image. This step helps to reduce the complexity of the subsequent
processing by focusing only on the regions with significant intensity
changes.

2. Parameter Space Initialization: A parameter space is created to

represent the possible lines in the image. The parameter space is typically
a two-dimensional array, where each element corresponds to a possible
line in the image. The dimensions of the parameter space depend on the
desired precision of the line representation.

3. Voting: For each edge point in the image, a voting process takes place in
the parameter space. For a given edge point (x, y), the corresponding line
parameters (θ, ρ) are calculated and incremented in the parameter space.
 The parameters (θ, ρ) represent the angle (θ) of the line with respect to a
reference axis and the perpendicular distance (ρ) from the origin to the line.

4. Accumulation: After the voting process, the parameter space contains

accumulated values that indicate which lines are well-supported by edge
points. The peaks in the parameter space correspond to the lines that have
the most significant support from edge points.

5. Thresholding and Line Extraction: A thresholding step is applied to the

parameter space to identify the peaks above a certain threshold value. The
corresponding lines are
then extracted from the parameter space based on the identified peaks.
Each peak represents a line in the image, characterized by the
corresponding (θ, ρ) parameters.

6. Line Reconstruction: The final step involves converting the line

parameters (θ, ρ) back into the image space. This process allows the
identified lines to be visualized or further processed.

The Hough Transform for circle detection follows a similar principle but
operates in a three-dimensional parameter space, representing the center
coordinates (x, y) and radius (r) of the circles.

Overall, the Hough Transform is a powerful technique for detecting

geometric shapes in images, offering robustness to noise, occlusion, and
shape variations. However, it can be computationally expensive, especially
for large images, and parameter tuning is crucial for achieving good results.
Various extensions and optimizations have been proposed over the years
to enhance its efficiency and accuracy, making it a fundamental tool in
computer vision.

1.8 Fitting Lines

Fitting lines in computer vision is a common task that involves estimating
the parameters of a line that best describes a set of observed points in an
image or a scene. Line fitting is useful in various applications, such as edge
detection, shape recognition, and line-based object tracking.

There are several approaches to fitting lines in computer vision, and I'll
explain two commonly used methods:

1. Least Squares Line Fitting:

The least squares method is a popular technique for line fitting. Given a
set of points, the goal is to find a line that minimizes the sum of squared
distances between the observed points and the line. This approach
assumes that the points have Gaussian noise.
The line equation in the form of y = mx + b represents a line in 2D, where
m is the slope and b is the y-intercept. To fit a line using the least squares
method, you can follow these steps:

- Calculate the mean of the x-coordinates and the mean of the

y-coordinates of the observed points.

- Calculate the sums of squared differences between each point's

x-coordinate and the mean x-coordinate, and between each point's
y-coordinate and the mean y-coordinate.

- Calculate the covariance between the x and y coordinates.

- Compute the slope, m, using the formula: m = covariance /

variance of x. - Calculate the y-intercept, b, using the formula: b

= mean y - (m * mean x).

 Once you have the parameters m and b, you can draw the fitted line
using the equation y = mx + b.

2. Random Sample Consensus (RANSAC):

RANSAC is an iterative method that is robust to outliers. It is useful when

there may be noise or incorrect data points in the observed set. RANSAC
works by randomly selecting a minimal subset of points and fitting a line to
that subset. Then, it determines how well the remaining points in the set fit
the estimated line. This process is repeated for a predefined number of
iterations, and the line with the best consensus is selected as the final fit.

Here are the steps for line fitting using RANSAC:

- Randomly select a minimal subset of points (e.g., two points) from the

observed set. - Fit a line to the selected points using a method like least

squares.

- Compute the distance between the fitted line and each remaining

point in the set. - Determine the inliers (points close to the line) based

on a distance threshold.

- Repeat the process for a defined number of iterations, selecting the line
with the largest number of inliers.

- Optionally, refine the line fit using all the inliers.

Both the least squares method and RANSAC have their advantages and
disadvantages. Least squares is simple and efficient but sensitive to
outliers. RANSAC, on the other hand, is more robust to outliers but can be
computationally expensive. The choice of method depends on the specific
requirements of your application and the characteristics of the data you are
working with.

1.9 Fitting Curves

Fitting curves in computer vision is a common task used in various

applications, such as object detection, image segmentation, and shape
analysis. The goal is to find a mathematical representation of curves that
accurately captures their shape and characteristics within an image.

There are several approaches to curve fitting in computer vision, and the
choice depends on the specific problem and data at hand. Here are a few
commonly used techniques:

1. Polynomial fitting: Polynomial curves are often used to approximate

simple curves. The degree of the polynomial determines the complexity of
the curve that can be represented. Polynomial fitting involves finding the
coefficients of the polynomial equation that best fits the given data points.
Methods like least squares regression can be used to minimize the error
between the fitted curve and the data.
2. B-spline fitting: B-splines are piecewise-defined curves composed of
polynomial segments. They provide more flexibility in capturing complex
curve shapes by controlling the degree and the number of control points.
B-spline fitting involves adjusting the control points to minimize the
difference between the curve and the data points. Algorithms like the
least-squares B-spline approximation can be used for this purpose.

3. Bezier curve fitting: Bezier curves are widely used for representing
smooth curves in computer graphics. They are defined by a set of control
points that influence the shape of the curve. Bezier curve fitting involves
adjusting the control points to minimize the distance between the curve and
the data points. Techniques like the least-squares fitting or the de Casteljau
algorithm can be employed.

4. Active contours (Snakes): Active contours, also known as snakes, are

deformable models that can be used to fit curves to object boundaries.
They iteratively evolve and deform based on internal forces, such as
smoothness and curvature, and external forces derived from image
features, such as edges or intensity gradients. Active contour models are
particularly useful for segmenting objects with irregular shapes.

5. Deep learning approaches: With the advent of deep learning,

convolutional neural networks (CNNs) have been used for curve fitting
tasks. For example, architectures like U-Net or Mask R-CNN can be utilized
for curve segmentation and fitting. These methods leverage large amounts
of annotated data to learn complex mappings between image inputs and
corresponding curve representations.

It's worth mentioning that the choice of curve fitting technique depends on
the complexity of the curves, the amount of noise in the data, and the
specific requirements of the computer vision task. Experimentation and
evaluation of different methods are often necessary to determine the most
suitable approach.