L28: Video Segmentation

Versha - 2021UCD2156
What do you mean by Video Segmentation?

It is the process of dividing a video into meaningful regions. These regions can be
based on various characteristics like:

● Object boundaries
● Motion
● Color
● Texture
● Other visual features
 Types of Video Segmentation

Video Object Segmentation (VOS) Video Semantic Segmentation (VSS)

a. Automatic (Unsupervised)
a. (Instance-Agnostic) VSS
b. Semi-automatic (Semi-supervised)
b. Video Instance Segmentation
c. Interactive
c. Video Panoptic Segmentation
d. Language-guided
Video Object Segmentation
It focuses on tracking objects within
a video and is used in applications
such as surveillance and
autonomous vehicles.

Methodology:
● Object initialization - identifying
the object in the first frame of
the video
● Object tracking - tracking its
movement throughout the rest
of the video
Approaches
1. Unsupervised VOS
- aims to segment objects in a video without using any labeled data.
- e.g. Focus on Foreground Network (F2Net).

1. Semi-supervised VOS
- use a small amount of labeled data to guide the segmentation process
and unsupervised methods to refine the segmentation results.
- useful in cases where obtaining labeled data is difficult or expensive.
- additionally, the unsupervised methods used in semi-supervised video
object segmentation can help to improve the robustness and
generalization of the segmentation results.
- e.g. Sparse Spatiotemporal Transformers (SST).
3. Interactive VOS
- User can specify the initial location of an object in the first frame of the
video or draw a bounding box around the object.
4. Language-guided VOS
- uses natural language input to guide the segmentation and tracking of
objects within a video.
Video Semantic Segmentation

It focuses on understanding the

overall scene and its contents and
is used in applications such as
augmented reality and video
summarization.

Methodology
● Feature extraction using CNN
● Features are used to classify
each pixel using FCN
Approaches
1. (Instance-Agnostic) Video Semantic Segmentation
- It is a method to identify and segment objects in a video sequence without
considering the individual instances of the objects.
- It is in contrast to instance-aware semantic segmentation, which tracks and
segments individual instances of objects within a video, making it less
computationally demanding.
1. Video Instance Segmentation
- It identifies and segments individual instances of objects within a video sequence.
- It is in contrast to the instance-agnostic semantic segmentation, which only
identifies and segments objects within a video without considering individual
instances.
3. Video Panoptic Segmentation
- identifies and segments both objects and their parts in a video sequence
in a single step. This approach combines the strengths of both instance-
agnostic semantic segmentation and video instance segmentation.
Challenges and Limitations of Video Segmentation

1. Variability in video content and quality.

2. Lack of temporal consistency.
3. Occlusions.
4. Complexity of visual scenes.
5. Lack of training data.
6. Computational complexity.
7. Evaluation and benchmarking: Evaluating the performance of video
segmentation approaches can be difficult due to the lack of standardized
benchmarks and evaluation metrics.
Applications of Video Segmentation

1. Special effects in movies: Isolate a character for green screen effects.

2. Self-driving cars: Identify pedestrians and lanes for safe navigation.
3. Video editing: Automatically separate objects for easier manipulation.
Shot Boundary Detection (SBD)
It is a sub-task of video segmentation that focuses on identifying transitions
between different shots in a video.
Breakdown:
1. Shots: A shot is a continuous sequence of frames captured by a single
camera operation. It's like holding the camera still and recording something
that happens.
2. Transitions: When the camera work changes, that's a shot boundary. This can
be:
a. Abrupt: A sudden cut from one scene to another.
b. Gradual: A slow transition, like a fade-out, dissolve, or wipe.
Why find shot boundaries?
1. Video Summarization: Identify key segments of a video, like different scenes
in a movie.
2. Video Editing: Automatically detect places to insert cuts or transitions.
3. Video Indexing: Create a searchable index of the video content, allowing you
to jump to specific shots.
Pixel Based Approaches
Pixel-based approaches are the foundation of video segmentation and shot
boundary detection. They work by analyzing individual pixels in each video frame.
The Power of a Pixel:
● A video frame is like a digital patchwork, made up of tiny squares called
pixels. Each pixel holds information about color, brightness, and sometimes
even depth.
● Pixel-based approaches treat each frame as a giant grid and analyze the
properties of each pixel.
1. Thresholding: Like a Brightness
Checkpoint
● Set a specific brightness value
(threshold).
● Pixels brighter than the threshold
are one object, darker ones are
another.
● A big difference in average
brightness between consecutive
frames (like day to night) might
indicate a cut.
2. Color Segmentation: Grouping by Color Similarities
● Group pixels with similar color properties.
● Useful for separating objects with distinct colors (red car vs. blue sky).
● Significant changes in color distribution (histograms) between frames could
suggest a shot change.
3. Region Growing: Expanding Based on Similarities
● We start with a single pixel and expand outward, including neighboring
pixels with similar features (color, brightness).
● Analyze how these regions change between frames to identify potential
shot boundaries.

4. Edge
● Identify sharp changes in intensity
between pixels (edges).
● Significant variations in edge patterns
between frames could suggest a shot
change.
Strengths and Limitations

● Sensitive to noise: Variations in lighting or camera movement can affect pixel

values, leading to segmentation errors. Imagine a red car with some shadows
- thresholding might break it into separate parts.
● Limited for complex scenes: Pixel-based approaches might struggle with
objects with similar colors or overlapping features. They don't consider the
bigger picture or object shapes.
● Computational Cost: While generally faster than deep learning methods,
some pixel-based techniques, especially region growing, can be
computationally expensive for high-resolution videos.