JCTVC-A114

Description of video coding technology proposal by NTT, NTT DOCOMO, Orange Labs, Panasonic and Technicolor
15-23 April 2010 1st JCT-VC meeting

Panasonic ideas for life

Outline

Proposal overview Coding tools Encoder control Performance & Complexity Conclusion

April 16, 2010

Basic design of CDCM

CDCM Collaborative Design Coding Model

Block-based predictive transform codec

New codec design, not an incremental extension of MPEG-4 AVC/H.264

Main properties

Using simple and fundamental coding tools in the decoder Optimized encoding algorithms Implements multi-thread decoding capabilities

April 16, 2010

Performance

Significant objective improvement / anchors - BD-rate gains:

JCfP version Anchor Alpha - CS1 Beta - CS2 Gamma CS2 BD gain Y 31.6% 30.4% 47.4% BD gain U 29.2% 10.6% 34.1% BD gain V 30.0% 10.9% 35.1% Anchor Alpha -CS1 Beta - CS2 Gamma CS2 Revised version BD gain Y 31.7% 30.6% 47.6% BD gain U 34.5% 18.3% 39.2% BD gain V 35.6% 18.1% 40.0%

Performs equally well in all sequence classes, bitrates and constraint sets Enhanced chroma reproduction

Significant visual improvement compared to Alpha and Beta anchors

For all bitrates in both constraint sets Sharper pictures, less blocking artifacts, better chroma representation

April 16, 2010

Content of the proposal

Stable and robust encoder and decoder reference software

Proprietary, C++ based

Runs on multiple platforms, multiple compilers Developed and tested by many companies Systematic integration steps: valgrind, streams crosscheck, perf. verifying Single and Multi-thread decoding capabilities

Complete documentation

Decoder specification Reference Encoder Model User manual Doxygen support

April 16, 2010

Outline

Proposal overview Coding tools Encoder control Performance & Complexity Conclusion

April 16, 2010

CDCM Encoder and decoder block diagrams

expanding T Q IQ Transform & Quantization IT Entropy coding Entropy Coding

Entropy Decoding Entropy decoding Transform & Quantization

IQ

IT

Intra/Inter Internal precision

Intra Prediction

Intra Prediction collapsing

Intra/Inter

Intra Prediction Internal precision

Intra Prediction

collapsing

ME

MC / IC

Denoising/ Deblocking

MC / IC Motion & Intensity compensation

Reference Memory

Denoising/ Deblocking Loop Filter

Motion & Intensity compensation

Reference Memory

Loop Filter

Encoder
7 April 16, 2010

Decoder

Basic picture structure

Basic structure: Superblocks Basic Block-unit:

Inter: 8x8 Luma + 4x4 Chroma Intra: 16x16 blocks & Transform can be enabled
superblock superblock superblock superblock

superblock

superblock

superblock

superblock

superblock

superblock

superblock

superblock

Intra16x16 disabled
8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block

8x8 block 8x8 Y 4x4 U 4x4 V

Intra16x16 enabled
8x8 block 8x8 block 8x8 block 8x8 block 16x16 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 8x8 block 16x16 block

N=4

N=2

16x16 block
8x8 block 8x8 block

16x16 Y

8x8 U

8x8 V

16x16 block

8x8 block 8x8 block

8x8 block 8x8 block

16x16 block

M=8

M=4

April 16, 2010

Detailed list of tools

Motion Representation & Intensity Compensation
Frame types MV block size MV resolution Motion sharing Intensity Compensation MV Competition in P-pict I-Frame, P-Frame (1 ref list), B-Frame (2 ref lists) 8x8 for luma (4x4 for chroma) -up to 1/8th pel in luma (1/16th pel in chroma) -selected at frame level motion block frontier can be displaced by 2 or 4 pixels (right or down) scale and offset signaled at the block level predictor index signaling predictors set switchable at frame level

Luma/Chroma interpolation for Motion Compensation

Luma 1/4 pel Luma 1/8 pel Chroma -2 modes: 1. fixed AVC/H.264 interpolation 2. SAIF (Wiener) -selected at frame level bilinear interpolation from 1/4th pel samples bilinear interpolation from full-pel samples, similar to MPEG-4 AVC/H.264 chroma interpolation (adaptation for 1/16 pel)

Differences with MPEG-4 AVC/H.264 are in red font

April 16, 2010

Detailed list of tools

Intra Prediction
Intra pred block size (SIP type) Prediction modes Edge-Based Prediction Mode TM Averaging (TMA) -Luma: 16x16 (I-picture only), 8x8, 4x4, 2x8 and 8x2 -16x16 block: each 8x8 block within can be split in 8x8, 4x4, 2x8 or 8x2 -Chroma: 8x8 and 4x4 implicitly inferred from luma SIP type - Consistent MPEG-4 AVC/H.264 like modes for intra4x4, intra8x8 - Generalisation MPEG-4 AVC/H.264 directional modes for intra16x16 DC mode can be replaced by a new directional mode, depending on the presence or not of an edge in neighboring reconstructed blocks 2 TMA modes for 16x16, 8x8, 4x4 - 1TMA mode for luma 2x8, 8x2

Low pass filter

-Luma: Same as MPEG-4 AVC/H.264 for intra8x8 and intra16x16 -Chroma: adaptive filtering implicitly inferred for chroma8x8

Differences with MPEG-4 AVC/H.264 are in red font

10

April 16, 2010

Detailed list of tools

Transform & Quantization
Intra block transform size (inferred from SIP_type) Inter block transform size (inferred from pred mode) Adaptive transform selection (DCT-like & KLT) Quantization -Luma: can use 16x16 (intra blks), 8x8, 4x4, 2x8, 8x2 -Chroma: Intra chroma blocks can use 4x4 or 8x8 transforms -Luma: can use 8x8 and 4x4 transforms -Chroma: use 4x4 transform -For luma, adaptive choice between 2 transforms for each block size -Signaling: For 16x16 Transf, flag explicitly signaled, Otherwise flag embedded in coeffs -qp resolution finer than that of the MPEG-4 AVC / H.264 -quantization matrices support

Entropy coding/decoding
Zero tree coding -In place of Signif. map coding of MPEG-4 AVC/H.264, consistent for Motion and Texture -recursive zero tree for 16x16 transform

CABAC

Similar to MPEG-4 AVC/H.264

In-loop filtering
Non-linear Denoising (NDF) Frame Adaptive Denoising Deblocking Filter
-regularization based on a sparsity constraint on the signal -uses a thresholding-based approach in an over-complete transform domain frame-based Wiener filter using residual, prediction and reconstructed signals -Similar to MPEG-4 AVC/H.264 -possible to place deblocking filter before/after NDF

Differences with AVC/H.264 are in red

11 April 16, 2010

Outline

Proposal overview Coding tools Encoder control Performance & Complexity Conclusion

12

April 16, 2010

Encoder control

Multi-pass encoding

I, P or B frame type

SAIF on/off

1/4th or 1/8th motion precision

To reduce complexity, combinations are constrained depending on frame type/level

3-steps motion estimation

1.Initial ME: search performed in a quadtree structure, from 8x8 to up to 128x128 2.Trellis ME: joint optimization of rows and columns of MVs 3.Region ME: region growing process To reduce complexity, some steps skipped depending on motion precision and interpolation filter type

Intra-picture encoding

SIP type selection using a Quad-tree like selection

2 more SIP types for luma (2x8 and 8x2) For SIP_16x16, 11 prediction modes instead of 4 in MPEG-4 AVC/H.264

Prediction modes tested

For other SIP types, up to 2 TMA modes in addition to the 9 in MPEG-4 AVC/H.264
1 additional mode on top of DC mode: Edge-Based Prediction Mode

13

April 16, 2010

Outline

Proposal overview Coding tools Encoder control Performance & Complexity Conclusion

14

April 16, 2010

Performance & Complexity

Objective performance

A revised version has been produced with bug fixing & improvts

Improvements: KLT function, Simplif

setting, I16x16 support in P/B frames

Bug fixes: 5 encoder related, 1 decoder related (minor impact)

JCfP version Revised version BD gain V Anchor BD gain Y BD gain U BD gain V

Anchor

BD gain Y

BD gain U

Alpha - CS1
Beta - CS2

31.6%
30.4%

29.2%
10.6%

30.0%
10.9%

Alpha -CS1
Beta - CS2

31.7%
30.6%

34.5%
18.3%

35.6%
18.1%

Gamma CS2

47.4%

34.1%

35.1%

Gamma CS2

47.6%

39.2%

40.0%

Comments

Performs equally well in all sequence classes, bitrates and constraint sets Enhanced chroma reproduction

15

April 16, 2010

Performance & Complexity analysis

Encoding time analysis

Tests conditions

Performed with a cluster using 64-bit Linux, with 64 cores

2 types of machines:

4 machines: 2 Quad-core Xeon E5450 @ 3.00GHz, 32GB RAM 4 machines: 2 Quad-core Xeon E5540 @ 2.53GHz, 32GB RAM

Class A CS1 (sec/fr) CS2 (sec/fr) 857

Class B 403 379

Class C 75 71

Class D 21 19

Class E 145

Decoding time analysis

Tests conditions: 1 Quad-core Xeon E5420 @ 2.5GHz

1 Quad-core Xeon E5420 @ 2.5GHz, 4GB Memory, Windows XP - 32bit Time ratio CDCM / JM

CS1 / alpha CS2 / beta CS2 / gamma

11.1 14.7 15.8

16

April 16, 2010

Performance & Complexity analysis

Expected encoder /decoder memory usage

Memory usage were measured using linux top command

Encoder Memory Usage
3000 350.0 300.0 ClassB 2000 1500 ClassE 1000 ClassC 500 0 0 500 1000 1500 2000 2500 3000 3500 4000 4500 luma samples per frame (x1000) ClassD ClassA (chroma MC on the fly used)

Decoder Memory Usage

Memory usage (MB)

Memory usage (MB)

2500

ClassA 250.0 200.0 150.0 100.0 50.0 0.0 0 ClassC ClassD 500 1000 1500 2000 2500 3000 3500 4000 4500 ClassE ClassB

luma samples per frame (x1000)

17

April 16, 2010

Outline

Proposal overview Coding tools Encoder control Performance & Complexity Conclusion

18

April 16, 2010

Conclusion

CDCM codec is proposed in response to the Joint CfP on Video Compression Technology High performance

Superior subjective quality (much better than JM, better than KTA) Excellent objective quality (more than 30% compared to the JM anchors) Better objective and subjective chroma performance observed

Similar quality improvements proving that it is robust, adaptable and not content-specialized

for CS1, CS2 across all resolutions and sequence types

The basic design of the CDCM codec leaves lots of potential for

other decoder tools to be added during the standardization phase further encoder optimization after the standardization phase

19

April 16, 2010

Conclusion

Attention on parallelizability

illustrated by the multi-threaded decoding capabilities of the software

Well documented proposal

decoder specification, user manual, reference encoder,

Doxygen support of the source code

Stable and robust encoder and decoder reference softwares

tested by many companies, with many compilers, using many platforms

CDCM is a good candidate for the JCTVC Test Model

20

April 16, 2010

Annex1: Improvements

21

April 16, 2010

Annex1: Improvements
Improvements after JCfP bitstreams submission

Improvement of KLT basis function

less blocky 7% average BD-rate improvements for chroma components without any significant change for luma components improvement of chroma

Simplification of Lambdas setting

Support of intra16x16 in P/B frames

22

April 16, 2010

Annex1: Bug fixes

Bug fixes

Encoder only (not signaled in bitstream)

interpolation of (7/8th, 7/8th) position of 1/8th pel MC corrected creation of candidate MVs in motion estimation process corrected availability of 8x2 mode for intra prediction corrected sip_type cost now considered when choosing between 4x4, 8x2, 2x8 and 8x8 modes for intra coding bit-cost computation of block coding mode corrected

Decoder related

A bug that causes an overflow after inverse quantization is corrected

Possible impact on decoded sequence

23

April 16, 2010

Annex2: detailed objective performance analysis

24

April 16, 2010

Objective performance

Annex2: detailed objective performance analysis

25

BD-Rate Gains % (+ve means proposal is better)

S0 2P S0

50 45 40 35 30 25 20 15 10 5 0

eo ple On S tre et on o ne m c 3Ki ar kS ce ac tu s riv e 5C llD af fi S0

BD-Rate for CS1 configuration relative to Alpha Anchor

April 16, 2010

1Tr S0 4P S0 as ke tB a Q Te r 7B sk S0 6B S0 S0 8Ba

Average: Y 31.6% - U 29.2% - V 30.0%

Y BD-Rate U BD-Rate

BD-Rate for CS1 configuration relative to Alpha Anchor

Test sequences
AV ER AG BE AV ER AG CE AV ER AG DE AV ER AG Al l-A E VE RA GE A-

V BD-Rate

ra ce et ba llD rill S0 9BQ S1 M 0all Pa rty S1 Sc 1en Ra e S1 ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Sq Bl ow ua in re gB S1 ub 5ble Ra s ce Ho rs es

Annex2: detailed objective performance analysis

Objective performance

BD-Rate for CS1 configuration relative to Alpha Anchor New results

Average: Y 31.7% (31.6%) - U 34.5% (29.2%) - V 35.6% (30.0%)

Revised BD-Rate for CS1 configuration relative to Alpha Anchor
Y BD-Rate
BD-Rate Gains % (+ve means proposal is better)

U BD-Rate

V BD-Rate

60.00 50.00 40.00 30.00 20.00 10.00 0.00

on o ar kS ce ne S0 S0 56Ca Ba ctu sk et s Ba llD S0 riv 7e BQ S0 Te 8Ba rra ce sk et ba llD rill S0 9BQ S1 M 0all Pa rty S1 Sc 1en Ra e S1 ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Sq Bl ow ua in re gB S1 ub 5ble Ra s ce Ho rs es 4P AV ER AG BE AV ER AG CE AV ER AG DE AV ER AG Al l-A E VE RA GE c ra f fi tre et m

On S

1T

S0

ple

3Ki

eo

S0

S0

2P

S0

Test sequences

26

April 16, 2010

A-

Objective performance

Annex2: detailed objective performance analysis

27

BD-Rate Gains % (+ve means proposal is better)

S0 3K

-50 10 20 30 40 50 60 0
S0 4P im ar

-40

-30

-20

-10

BD-Rate for CS2 configuration relative to Beta Anchor

April 16, 2010

Y 30.4% - U 10.6% - V 10.9%

Y BD-Rate U BD-Rate

BD-Rate for CS2 configuration relative to Beta Anchor

Test sequences
BAV ER AG CE AV ER AG DE AV ER AG EAV E ER AG Al l-A E VE RA GE

V BD-Rate

on o kS ce S0 ne S0 56Ca Ba ctu sk et s Ba llD S0 riv 7BQ e S0 Te 8Ba rra sk ce et ba llD rill S0 9BQ S1 0M Pa all rty S1 Sc 1en Ra S1 e ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Bl Sq ow ua in re gB S1 ub 5ble Ra s ce Ho rs es S1 6Vi dy o1 S1 7Vi dy o3 S1 8Vi dy o4

Annex2: detailed objective performance analysis

Objective performance

BD-Rate for CS2 configuration relative to Beta Anchor New results

Y 30.6% (30.4%) - U 18.3% (10.6%) - V 18.1% (10.9%)

Revised BD-Rate for CS2 configuration relative to Beta Anchor
Y BD-Rate 60.00
BD-Rate Gains % (+ve means proposal is better)

U BD-Rate

V BD-Rate

50.00 40.00 30.00 20.00 10.00 0.00

im on o ar kS ce S0 ne S0 56Ca Ba ctu sk et s Ba llD S0 riv 7e BQ S0 Te 8Ba rra sk ce et ba llD rill S0 9BQ S1 0M Pa all rty S1 Sc 1en Ra S1 e ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Sq Bl ow ua in re gB S1 ub 5ble Ra s ce Ho rs es S1 6Vi dy o1 S1 7Vi dy o3 S1 8Vi dy o4 4P

-10.00
S0 3K

-20.00 -30.00 -40.00

S0

Test sequences

28

April 16, 2010

BAV ER AG CE AV ER AG DAV E ER AG EAV E ER AG Al l-A E VE RA GE

Objective performance

Annex2: detailed objective performance analysis

29

BD-Rate Gains % (+ve means proposal is better)

10

20

30

40

50

60

70

BD-Rate for CS2 configuration relative to Gamma Anchor

April 16, 2010

Y 47.4% - U 34.1% - V 35.1%

Y BD-Rate U BD-Rate

BD-Rate for CS2 configuration relative to Gamma Anchor

Test sequences
BAV ER AG CE AV ER AG DE AV ER AG EAV E ER AG Al l-A E VE RA GE

V BD-Rate

S0 3Ki S0 m on 4Pa o rk Sc en S0 S0 e 56Ca Ba ctu sk et s Ba llD S0 riv 7BQ e S0 Te 8Ba rra sk ce et ba llD rill S0 9BQ S1 0M Pa all rty S1 Sc 1en Ra S1 e ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Sq Bl ow ua in re gB S1 ub 5ble Ra s ce Ho rs es S1 6Vi dy o1 S1 7Vi dy o3 S1 8Vi dy o4

Annex2: detailed objective performance analysis

Objective performance

BD-Rate for CS2 configuration relative to Gamma Anchor New results

Y 47.6% (47.4%) - U 39.2% (34.1%) - V 40.0% (35.1%)

Revised BD-Rate for CS2 configuration relative to Gamma Anchor
Y BD-Rate
BD-Rate Gains % (+ve means proposal is better)

U BD-Rate

V BD-Rate

70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00

on o kS ce S0 ne S0 56Ca Ba ctu sk et s Ba llD S0 riv 7BQ e S0 Te 8Ba rra sk ce et ba llD rill S0 9BQ S1 0M Pa all rty S1 Sc 1en Ra S1 e ce 2Ho Ba rs sk es et ba llP S1 as 3S1 s BQ 4Sq Bl ow ua in re gB S1 ub 5ble Ra s ce Ho rs es S1 6Vi dy o1 S1 7Vi dy o3 S1 8Vi dy o4 4Pa r AV ER AG CE AV ER AG DE AV ER AG EAV E ER AG Al l-A E VE RA GE im S0 3K

S0

Test sequences

30

April 16, 2010

B-