Segmentation

and Tracking
Dr Mahesh Kolekar
IIT Patna
 Issues

• The position and orientation of a camera influence

the degree of occlusion.
• If the camera’s optical axis is horizontal, then
occlusion will occur in the segment parallel to the
horizontal axis of the image plane.
• Occluded blobs will be indistinguishable due to
overlapping in depth.

The camera need to be installed relatively

high and tilted down for inspection.
 Rules for Segmentation
R1) The number of blobs in the segmentation
results should be the same as the number of
persons.

R2) In the histogram-based approach, the areas of

the segmented blobs should be reasonably large.

R3) In the ellipse based approach, the shape of

each blob after segmentation should be larger
than a certain threshold.
 Two approaches

1) Histogram-based approach, which depends

on the peak and valley features in the
vertical projection histogram

2) Ellipse-based approach, which uses an

ellipse to cut off merging persons from the
top to the bottom
 Human counting
circum rectangle
of occluded blob
in frame K+1

When blobs merge,

circum rectangles
we can compute the before occlusion
in frame K
number of persons
according to the
individual blobs in the
previous frame…
contours of two
separate persons in
frame number k
Histogram-based segmentation

Occluded blob Segmentation Result

 Histogram-based segmentation
Horizontal Proj.

DFT

Select lowest 20 frequency

IDFT
T
Smooth curve

Occluded blob Segmentation Result

 Steps in histogram based segmentation

• Project vertically onto a horizontal line, the blobs

change into the shapes shown in the previous slide.
• Compute DFT and select the 20 lowest frequency
components and discard the others.
• Compute IDFT. This process smoothes the original
curve and benefits the following segmentation.
• On the smooth curve, the significant peaks are
usually located at the horizontal position of the head
• Significant valleys are usually located at the
effective cut points for segmenting the occlusion.
Failure of Histogram based approach

Small blobs
(Not following R2)

• When people are occluded at different

depth.
• When detected blobs are of small size
 Demerits of Histogram method

The blob cannot be

segmented when the
smoothed histogram
curve does not bear
two peaks

Solution: Integration of histogram based

approach with the ellipse-based approach.
 Ellipse Based Approach

• Every person is assigned an ellipse, the long axis is

aligned vertically to fit the normal walking shape.
• For simplicity, size of the ellipse is the same, but it
can be adjusted experimentally according to depth
information.
• Given the number of persons (n) contained in the
occlusion blob, we can split the vertically merging
blob into n parts.
 Steps of Ellipse Based Approach

• Step-1: The person whose head is at the top of the

occlusion blob is cut off using the ellipse by matching
the top points of the blob and the ellipse.
• Step-2: The small isolated blob fragments that result
from the cut are treated as noise and filtered out.
• Step-3: The remaining n−1 persons are split
sequentially thereafter.
 Ellipse Based Approach

segmentation by
failure of the Histogram of the using an ellipse-
histogram-based occluded blob
approach based algorithm
 Demerits of Ellipse based approach
• Location of the topmost point of the ellipse is
subject to the topmost pixel of the blob or
remaining blob
• Lack of a filtering process makes it sensitive
to noise.
Flow Chart for
Segmentation
Tracking
 Image 1 Image 2 Image 3 Image 4

• Can we estimate the position of the object?

• Can we estimate its velocity?
• Can we predict future positions?
 The Tracking Problem
• Given Sequence of Images
• Find center of moving object
• Camera might be moving or stationary

• We assume: We can find object in

individual images.
• The Problem: Track across multiple
images is a fundamental problem in
computer vision.
 Hybrid Tracking Method
1.Hybrid Tracking Method
– Distance Tracking
– Color Tracking
2.Particle Filter-based Tracking Method
3.Local Binary Pattern-based Tracking
Method
– Multiple Target Tracking
– Kalman Filter
– LBP Histogram Distance
– Blob Classification
 1.1 Distance Tracking
1. To choose candidate point, scan the line that is a few pixels
down from the top one.
2. Assign index number to each blob.
3. Blobs are matched by the matching of the semi-topmost
points.
4. Prediction Model: The distances between the points in the
previous point set and those of the current point set can be
investigated to determine the index of the current points.

5. Each un-indexed point in the set of the current frame is then

matched to the closest indexed point in the predicted set.
 1.1 Distance Tracking: Prediction Model

Prediction 2xN matrix 2xN matrix

of P(n+1) Row: 2D position Row: 2D position of
of point in the point in the (n-1)th
nth frame Frame

N: total number of persons

 Limitation In Distance Tracking

• Targets purposely change direction to

fool the system
• More sensitive to occlusion
 1.2 Color Tracking
1. Convert the blob from RGB to HSV space.
2. Scan each pixel in the blob for its hue value,
and plot hue histogram.
3. Normalize the hue distribution by dividing total
number of pixels in the blob.
4. Compute DFT.
5. Select lowest 20 low frequency components.
6. Compute IDFT to transform the histogram
back into a time (hue) domain.
1.2 Color Tracking (color spectrum)
Person-C Person-A

Person-B

Color Spectrum
Hue-Histogram of three person
of one person matching
 More about color tracking

If the number of persons is small, just two

features the mean and the variance are
sufficient.
 Special Cases
• During video surveillance, people occasionally
move out of the camera’s FOV and re-enter it,
which results in blobs being assigned another
index number.
• Solution: Color tracking
• Instead of assigning a new index, the color
tracking algorithm searches in its “memory”
for a similar color distribution.
 Results

Seriously
occluded
situation
Thank You