Tracking Players at the Edge

Aiden Jeffrey – Codethink

April 2021
Summary
● Modern approaches to:
○ Detection
○ Tracking
● System overview:
○ Multi-camera Tracking
○ Camera Space Player Tracker
● Demo
Object Detection
Traditional computer vision approaches:

https://en.wikipedia.org/wiki/Scale-invariant_feature_transform
● Lots of research and history
● Difficult to configure and not generally robust
● Algorithms designed to run performantly on CPUs

Examples include:
● Feature detection (SIFT, etc)
● Histogram of Gradients
● Edge detection
● Kernel methods
Object Detection
Neural Network based approaches:

https://www.deeplearningbook.org/contents/intro.html
● Much easier to configure for a
given object class
● Robust and precise
● Much higher-spec hardware
requirements - only enabled due to
advance of GPUs
Object Detection - CNNs
Convolution Networks/Layers:

https://www.deeplearningbook.org/contents/convnets.html
● Related to traditional CV kernel
methods
● Great for processing data with grid
like topology (time series, images)
● Can be thought of as a smoothing
step
Object Detection - CNNs
CNNs for object detection:
● You Only Look Once (YOLO)
● Recursive CNN (R-CNN)
● Single Shot MultiBox Detector (SSD)
● RetinaNet

https://openaccess.thecvf.com/content_cvpr_2016/papers/Redmon_You_Only_Look_CVPR_2016_paper.pdf
Object Detection - YOLO

https://openaccess.thecvf.com/content_cvpr_2016/papers/Redmon_You_Only_Look_CVPR_2016_paper.pdf
Object Tracking
Traditional CV approaches:
● Background subtraction
● Optical Flow
● Kalman Filters
● SORT

Neural Network based approaches:

● ROLO
● SiamMask
● TrackR-CNN
System Overview
● Hybrid ML/CV tracking system for balls and players in
professional sports environment, using multiple views on
the same scene
● Main objectives:
○ Low cost
○ Low complexity
○ Low latency
○ Scalable
Triangulation
Triangulation

High quality camera sensors with Simple RANSAC inspired

embedded Jetson TX2 SoCs triangulation algorithm that
minimises back projection error.
Camera Space Player Tracking
● Attempt to emulate human vision
○ Use basic computer vision approaches to detect and
track movement in the video - these processes can
happen on the CPU and at high frame rate
○ Use detection network (YOLO) to refine candidate
blobs - analogue for attention, lower frame rate, but
higher precision
○ Use Kalman Filter to estimate blob state over time
and track
Camera Space Player Tracking
● Parallel streams of the video source at the appropriate
frame size and color space for each sub-process:

movement
queue
detection (CPU)
candidate blobs

leaky YOLO blob refinement Track builder

tee (CUDA cores) (Kalman Filter)
queue

refined blobs

webrtc & HLS

streams
Getting Multi-view Sports Footage in 2021

https://github.com/BazkieBumpercar/GameplayFootball

Thanks to Rob Kendrick (rjek) for helping patch this game to support fixed cameras and repeatable game state.
Player Tracking

movement
queue
detection (CPU)
candidate blobs

leaky YOLO blob refinement track builder

tee (CUDA cores) (Kalman Filter)
queue

refined blobs

webrtc & HLS

streams
Candidate Blob Generation

movement
queue
detection (CPU)
candidate blobs

leaky YOLO blob refinement track builder

tee (CUDA cores) (Kalman Filter)
queue

refined blobs

webrtc & HLS

streams
Candidate Blob Generation
Blob Refinement

movement
queue
detection (CPU)
candidate blobs

leaky YOLO blob refinement track builder

tee (CUDA cores) (Kalman Filter)
queue

refined blobs

webrtc & HLS

streams
Blob Refinement
Blob Refinement
Track Generation

movement
queue
detection (CPU)
candidate blobs

leaky YOLO blob refinement track builder

tee (CUDA cores) (Kalman Filter)
queue

refined blobs

webrtc & HLS

streams
Track Generation
Future Improvements
● Make track builder less dependent on feedback from
triangulation system
● Add support other high quality detection NN models
beyond YOLO
● Add facility to auto-optimise Kalman Filter settings based
on track quality heuristic
● Infrastructure to retrain model using arbitrary team shirts
- to aid with disambiguation when split out from
triangulation system
Questions

If you need help - please get in touch

Aiden Jeffrey - Codethink

aiden.jeffrey@codethink.co.uk