Recently on Image Compression ...

MI
A
1 2
Recently on Image Compression ... 3 4
5 6
7 8
What did we learn in the last learning unit? 9 10
For infinite size alphabets, we have several tools at our disposal. 11 12


13 14
Golomb coding can adapt to geometric distributions. 15 16


Fibonacci coding is static, but robust under errors. 17 18



19 20
Questions: 21 22
23 24
For finite alphabets, Huffman Coding is still our best option. 25 26


Can we do better than that? 27 28



29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Recently on Image Compression ... MI

A
1 2
This Learning Unit
3 4
Image Compression
5 6
7 8
Part I: Lossless Part II: Lossy 9 10
11 12
Entropy Coding Transform Coding
Lossless Codecs
Other Approaches 13 14
PNG
Information Theory JPEG
Fractal Compression 15 16
Huffman Coding JPEG2000
gif
HEVC intra
Neural Networks 17 18
Integer Coding
JBIG
Arithmetic Coding
19 20
JBIGLS
RLE, BWT, MTF, bzip 21 22
Inpainting-based Basics
Compression 23 24
quantisation
Dictionary Coding PDE-based Inpainting
error measures
25 26
LZ-Family
Prediction Data Selection 27 28
Tonal Optimisation
Deflate
linear prediction 29 30
Patch-based Inpainting
PPM
Inpainting-based Codecs
Teaser: Video Coding
31 32
PAQ
PDE-based video coding 33 34
MPEG-family 35 36
37 38
39 40
We introduce arithmetic coding as an alternative to Huffman.


41 42
43 44
 Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44

Motivation MI
A
1 2
Motivation 3 4
5 6
Huffman coding is optimal, but it requires at least one bit for each symbol.


7 8
Extended Huffman coding resolves that, but creates overhead. 9 10


11 12
One would like to use a “fractional” amount of bits for each symbol.


13 14
Arithmetic coding 15 16


17 18
• does not have this restriction.
19 20
• can not be described by an encoding scheme 21 22
23 24
• replaces the complete source word by a single code word
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Pure Zeroth-Order Arithmetic Coding MI
A
1 2
Pure Zeroth-Order Arithmetic Coding 3 4
5 6
We start with Pure Zeroth-Order Arithmetic Coding:


7 8
• “Pure”: Practical problems are ignored for the moment. 9 10
11 12
• “Zeroth-Order”: Symbols are independent and identically distributed.
13 14
This algorithm is not used in practise, but shows the main ideas very well. 15 16


17 18
Note that it is not necessary to order the symbols by their probability.


19 20
All examples for Zeroth-Order Coding use the following symbols and probabilities: 21 22


23 24
25 26
27 28
i 1 2 3 4 29 30
31 32
si A D B C
33 34
pi 0.4 0.3 0.2 0.1 35 36
Occurring symbols and their probabilities. 37 38
39 40
41 42
43 44

Arithmetic Coding: Idea MI

A
1 2
Arithmetic Coding: Idea
3 4
The basic idea behind arithmetic coding is simple: 5 6


7 8
• Assign each source word si1 · · · siN to an interval I(i1, . . . , iN ). 9 10
• Store each source word si1 · · · siN as a number r ∈ I(i1, . . . , iN ). 11 12
13 14
15 16
17 18
19 20
21 22
23 24
25 26
27 28
Open questions: 29 30
• Which intervals should be chosen? 31 32
33 34
• Which number r should be stored? 35 36
37 38
39 40
41 42
43 44
 Arithmetic Coding: Idea MI
A
1 2
Illustration of Arithmetic Coding
3 4
1.0 0.70 0.520 0.5080 5 6
0.9 C 0.67 C 0.508 C 0.5056 C 7 8
9 10
B B B B
0.7 0.61 0.484 0.5008 11 12
13 14
D D D D 15 16
17 18
0.4 0.52 0.448 0.4936
19 20
21 22
A A A A
23 24
0.0 0.40 0.400 0.4840 25 26
27 28
Graphical illustration of encoding the word “DABD” with arithmetic coding: any 29 30
number in the interval [0.4936, 0.5008) can be used as code word. 31 32
33 34
35 36
37 38
39 40
41 42
43 44

Pure Zeroth-Order Arithmetic Coding MI

A
1 2
Algorithm to Find the Interval
3 4
1. Start with [α, β) = [0, 1). 5 6
7 8
2. Define the length l as l = β − α. 9 10
3. Subdivide [α, β) into (half-open) intervals with lengths proportional to p1, . . . , pm. 11 12
13 14
4. The j-th interval is thus given by 15 16
      17 18
X X 19 20
α +  pi l, α +  pi  l  21 22
i<j i≤j
23 24
25 26
5. Set [α, β) to the interval corresponding to the next symbol. 27 28
29 30
6. If there are symbols left, continue with step (2).
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Pure Zeroth-Order Arithmetic Coding MI
A
1 2
Illustration of the Subdivision Process
3 4
5 6
next symbol α l
7 8
0 1 9 10
D 0 + 1 · 0.4 = 0.4 1 · 0.3 = 0.3 11 12
13 14
A 0.4 + 0.3 · 0.0 = 0.4 0.3 · 0.4 = 0.12 15 16
B 0.4 + 0.12 · 0.7 = 0.484 0.12 · 0.2 = 0.024 17 18
19 20
D 0.484 + 0.024 · 0.4 = 0.4936 0.024 · 0.3 = 0.0072 21 22
23 24
Illustration of finding the interval used to represent the word “DABD”, using a table.
The resulting interval is: [0.4936, 0.4936 + 0.0072) = [0.4936, 0.5008). 25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Pure Zeroth-Order Arithmetic Coding MI

A
1 2
How to Choose r?
3 4
As stated before, a number r ∈ I(i1, . . . , iN ) is stored instead of the interval. 5 6


7 8
Theoretically, any number r ∈ I(i1, . . . , iN ) can be used to encode the word 9 10


s i1 · · · s iN . 11 12
Since r is stored in binary representation, this representation must be finite, 13 14


though: 15 16
a 17 18
∃a ∈ Z, b ∈ N : r = b
2 19 20
Such numbers are called dyadic fractions or dyadic rationals. 21 22
b bits are necessary to store the number r. Thus, the smaller b, the better. 23 24


25 26
The smaller the intervals, the more digits are usually necessary to store r. Thus,


27 28
the intervals should be as large as possible. 29 30
31 32
33 34
Conclusion: Use the dyadic fraction with smallest denominator. 35 36
37 38
39 40
41 42
43 44
 Pure Zeroth-Order Arithmetic Coding MI
A
1 2
Finding the Dyadic Fraction with Least Denominator
3 4
How to find the dyadic fraction with least denominator in the interval [α, β)? 5 6


7 8
Method I 9 10
11 12
1. Find the smallest integer t with 13 14
15 16
1
17 18
≤l ⇒ 2−t ≤ l < 2−t+1 19 20
2t
21 22
2. Find an integer y satisfying: 23 24
y 25 26
α≤ <β
2t 27 28
Remark: This inequality can have one or two solutions. If there are two solutions,
29 30
you must take the even one.
31 32
y
3. Reduce the fraction r = 2t
33 34
35 36
(The binary representation of y will be stored in pure arithmetic coding)
37 38
39 40
41 42
43 44

Pure Zeroth-Order Arithmetic Coding MI

A
1 2
Example
3 4
In our example from slide 6, we had 5 6


7 8
9 10
α = 0.4936, β = 0.5008, l = 0.0072
11 12
13 14
Computing t yields: 15 16


  17 18
1  
t = log2 = log2 138.8 = log2 256 = 8 19 20
l 21 22
23 24
Next, we solve the inequalities for y ∈ Z 25 26


27 28
y 29 30
α≤ <β
2t 31 32
y
⇔ 0.4936 ≤ < 0.5008 33 34
256
35 36
⇔ 126.3616 ≤ y < 128.2048 37 38
⇔ y ∈ {127, 128} 39 40
41 42
43 44
 Pure Zeroth-Order Arithmetic Coding MI
A
1 2
Example (continued)
3 4
5 6
Since there are two solutions, we chose the even one:


7 8
9 10
y = 128
11 12
13 14
Next, we reduce the fraction: 15 16


17 18
y 128 1
r= = = = 0.12 19 20
2 t 256 2 21 22
23 24
Thus, the word “DABD” is encoded as 1.


25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Pure Zeroth-Order Arithmetic Coding MI

A
1 2
Finding the Dyadic Fraction with Least Denominator
3 4
Method I is not optimal to implement. 5 6


7 8
Remedy: 9 10
11 12
Method II 13 14
15 16
Compute the binary expansions of α and β until they differ: 17 18


19 20
α =(0.a1 · · · at−10at+1 · · · )2 21 22
β =(0.a1 · · · at−11bt+1 · · · )2 23 24
25 26
27 28
The dyadic fraction with least denominator is then given by
29 30
31 32
r = 0.a1 · · · at−11
33 34
35 36
Careful: There are two exceptions (see next slide). 37 38


39 40
41 42
43 44
 Pure Zeroth-Order Arithmetic Coding MI
A
1 2
Exception I: If α = (0.a1 · · · at−1)2, then r = α (since this is shorter than


3 4
(0.a1 · · · at−11)2).
5 6
Exception II: If α > (0.a1 · · · at−1)2 and β = (0.a1 · · · at−11)2, then 7 8


(0.a1 · · · at−11)2 = β 6∈ [α, β). 9 10

In this case, α has the form: 11 12
13 14
α = (0.a1 · · · at−101q 0 · · · )2 15 16
17 18
where q ∈ N0. 19 20
21 22
• If α = (0.a1 · · · at−101q )2, set r = α. 23 24
• If α 6= (0.a1 · · · at−101q )2, set r = (0.a1 · · · at−101q 1). 25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Pure Zeroth-Order Arithmetic Coding MI

A
1 2
Example
3 4
In our example from slide 6, we had 5 6


7 8
9 10
α = 0.4936, β = 0.5008, l = 0.0072
11 12
13 14
Since 15 16


17 18
α =0.01 . . . 19 20
β =0.10 . . . , β 6= 0.12 21 22
23 24
25 26
27 28
it follows that r = 0.12.
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44

Problems and Improvements MI

A
1 2
Problems and Improvements 3 4
5 6
Rescaling 7 8
9 10
Problem: The whole word must be encoded before the first bits of the code


11 12
word are known.
13 14
Solution: Shift out known bits and rescale: 15 16


17 18
If α < β ≤ 12 , then r < 1
= 0.12.


2 19 20
• Thus, the code starts with 0. 21 22
23 24
• We shift out this 0, resulting in the interval [2α, 2β).
25 26
1 1
If ≤ α < β, then r ≥ = 0.12. 27 28


2 2

• Thus, the code starts with 1. 29 30

31 32
• We shift out this 1, resulting in the interval [2α − 1, 2β − 1). 33 34
The bits shifted out are the first bits of the code word, or code stream. 35 36


37 38
In both cases, l is doubled.


39 40
Furthermore, this helps to prevent the interval [α, β) from getting very small. 41 42


43 44
 Problems and Improvements MI
A
1 2
Example
3 4
next symbol α l new bits 5 6
or operation 7 8
0 1 9 10
B 0 + 0.7 · 1 = 0.7 1 · 0.2 = 0.2 11 12
x → 2x − 1 2 · 0.7 − 1 = 0.4 2 · 0.2 = 0.4 1 13 14
B 0.4 + 0.4 · 0.7 = 0.68 0.4 · 0.2 = 0.08 15 16
x → 2x − 1 2 · 0.68 − 1 = 0.36 2 · 0.08 = 0.16 1 17 18
C 0.36 + 0.9 · 0.16 = 0.504 0.16 · 0.1 = 0.016 19 20
21 22
x → 2x − 1 2 · 0.504 − 1 = 0.008 2 · 0.016 = 0.032 1
23 24
x → 2x 2 · 0.008 = 0.016 2 · 0.032 = 0.064 0
25 26
x → 2x 2 · 0.016 = 0.032 2 · 0.064 = 0.128 0
27 28
x → 2x 2 · 0.032 = 0.064 2 · 0.128 = 0.256 0
29 30
x → 2x 2 · 0.064 = 0.128 2 · 0.256 = 0.512 0 31 32
A 0.128 + 0 · 0.512 = 0.128 0.512 · 0.4 = 0.2048 33 34
x → 2x 2 · 0.128 = 0.256 2 · 0.2048 = 0.4096 0 35 36
1 37 38
Encoding of the word “BBCA” with rescaling. Since ∈ [0.256, 0.256 + 0.4096), the
2 39 40
encoded word is 111000001.
41 42
43 44

Problems and Improvements MI

A
1 2
Underflow Expansion
3 4
Question: Can we do even better than rescaling? 5 6


7 8
Answer: Yes, in special cases we can “look two bits into the future” and shift 9 10


out the second bit with an underflow expansion: 11 12

Requirement: 1
≤α< 1
<β≤ 3 13 14


4 2 4
15 16
Then, either 17 18


    19 20
1 1 21 22
r0 ∈ α, ⇒ r0 = (0.01 . . .)2 since α ≥
2 4 23 24
25 26
or    
0 1 0 3 27 28
r ∈ ,β ⇒ r = (0.10 . . .)2 since β ≤ 29 30
2 4
31 32
Thus, the first two bits always differ, i.e. if we know one, we know the other.


33 34
35 36
37 38
39 40
41 42
43 44
 Problems and Improvements MI
A
1 2
Underflow Expansion
3 4
Thus, we can remove the second bit from α and β with the transformation: 5 6


7 8
1 9 10
x → 2x − 11 12
2
13 14
• If x = (0.01x3x4 . . .), then 2x − 12 = (0.0x3x4 . . .) 15 16
17 18
• If x = (0.10x3x4 . . .), then 2x − 21 = (0.1x3x4 . . .) 19 20
This transformation doubles the length of the interval [α, β). 21 22


23 24
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Problems and Improvements MI

A
1 2
Underflow Expansion Rules
3 4
1. Initialise the number k of underflow expansions done to 0. 5 6
7 8
2. When the current interval [α, β) satisfies 14 ≤ α < 21 < β ≤ 43 , remove the second 9 10
bit, resulting in the interval   11 12
1 1 13 14
2α − , 2β −
2 2 15 16
with length 2l, and add one to k. 17 18
3. Whenever the rescaling transformation x → 2x is done, add 01k to the code 19 20
stream (instead of just 0) and reset k to zero. 21 22
23 24
4. Whenever the rescaling transformation x → 2x − 1 is done, add 10k to the code 25 26
stream (instead of just 1) and reset k to zero. 27 28
5. If the underflow counter is not zero when the final interval is found, it must still be 29 30
used when finding the number in the interval. 31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Problems and Improvements MI
A
1 2
Example
3 4
next symbol α l new bits k 5 6
or operation 7 8
0 1 0 9 10
D 0.4 0.3 0 11 12
x → 2x − 12 0.3 0.6 1 13 14
A 0.3 0.24 1 15 16
x → 2x − 12 0.1 0.48 2 17 18
D 0.292 0.144 2 19 20
21 22
x → 2x 0.584 0.288 011 0
23 24
x → 2x − 1 0.168 0.576 1 0
25 26
B 0.5712 0.1152 0
27 28
x → 2x − 1 0.1424 0.2304 1 0
29 30
x → 2x 0.2848 0.4608 0 0 31 32
Encoding of the word “DADB” with rescaling and underflow expansion. Since 33 34
1 35 36
2 ∈ [0.2848, 0.2848 + 0.4608), the encoded word is 0111101.
37 38
39 40
41 42
43 44

Performance MI
A
1 2
Theorem
3 4
5 6
The average length of code words generated with arithmetic coding for source words 7 8
of length N is not greater than N H(S) + 1. 9 10
11 12
This means that H(S) + N1 bits are necessary on average to encode one symbol when 13 14
encoding a word of length N .
15 16
(No proof will be given, here) 17 18
19 20
21 22
23 24
25 26
27 28
29 30
31 32
33 34
35 36
Conclusion: Pure arithmetic coding is optimal among lossless zeroth-order entropy 37 38
coding methods. 39 40
41 42
43 44
 Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44

Decoding MI
A
1 2
Pure Zeroth-Order Decoding 3 4
5 6
To decode r, find the index i1 such that r ∈ I(i1), followed by i2 such that


7 8
r ∈ I(i1, i2), followed by i3 such that . . ..
9 10
11 12
Assumption: The decoder knows the length N of the original source word.
13 14
Details 15 16
17 18
Assume I(i1, . . . , ik ) := [α, α + l) has already been found. 19 20


21 22
Then, ik+1 with r ∈ I(i1, . . . , ik+1) satisfies


23 24
    25 26
X X 27 28
α+ pi  l ≤ r <α+ pi l
29 30
i<ik+1 i≤ik+1
31 32
X r−α X
⇔ pi ≤ < pi 33 34
i<ik+1
l 35 36
i≤ik+1
37 38
39 40
41 42
43 44
 Decoding MI
A
1 2
Example
3 4
r−α 5 6
α l interval decoded symbol
l 7 8
0 1 0.5 0.5 ∈ [0.4, 0.7) D 9 10
0.4 0.3 0.3 0.3 ∈ [0.0, 0.4) A 11 12
(0 + 1 · 0.4) (1 · 0.3) ( 0.5−0.4 13 14
0.3 )
15 16
0.4 0.12 0.83 0.83 ∈ [0.7, 0.9) B
17 18
(0.4 + 0.3 · 0.0) (0.3 · 0.4) ( 0.5−0.4
0.12 )
19 20
0.484 0.024 0.6 0.6 ∈ [0.4, 0.7) D 21 22
(0.4 + 0.12 · 0.7) (0.12 · 0.2) ( 0.5−0.484
0.024 ) 23 24
25 26
Decoding the word 1 with N = 4 yields “DABD”. (Compare slides 4 and 19). 27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Decoding MI
A
1 2
Rescaling and Underflow Expansion While Decoding
3 4
The rescaling and underflow operations introduced for encoding are also possible 5 6


while decoding. 7 8
9 10
The prerequisites stay the same. 11 12


Then, the operations x → 2x, x → 2x − 1, and x → 2x − 12 are applied to α and 13 14



r, and l is always doubled. 15 16

17 18
There is no need to count the underflow expansions.


19 20
21 22
23 24
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Decoding MI
A
1 2
Example
3 4
r−α 5 6
α l r l symbol or operation
7 8
449 449
0 1 512 512 ≈ 0.88 B 9 10
449
0.7 0.2 512 – x → 2x − 1 11 12
193
193 −0.4 13 14
0.4 0.4 256
256
0.4 ≈ 0.88 B
15 16
193
0.68 0.08 256 – x → 2x − 1 17 18
65
0.36 0.16 128 – x → 2x − 12 19 20
33
33 −0.22 21 22
0.22 0.32 64
64
0.32 ≈ 0.92 C
33
23 24
0.508 0.032 64 – x → 2x − 1 25 26
1
0.016 0.064 32 – x → 2x 27 28
0.032 0.128 1
– x → 2x 29 30
16
0.064 0.256 1
– x → 2x 31 32
8
1 −0.128 33 34
1
0.128 0.512 4
4
0.512 ≈ 0.24 A 35 36
37 38
Decoding the code word 111000001 with N = 4 yields “BBCA”. (Compare slide 15). 39 40
41 42
43 44

Advantages and Drawbacks MI

A
1 2
Summary: Pros and Cons of Pure Arithmetic Coding 3 4
5 6
Advantages 7 8
9 10
Pure arithmetic coding can get extremely close to the theoretical optimum. 11 12


13 14
Drawbacks 15 16
17 18
Pure arithmetic coding requires exact computations. 19 20


21 22
Run-time and memory requirements rise drastically with the source word length.


23 24
There are many patents which cover arithmetic coding (mostly IBM). 25 26


27 28
Those patents have (presumably) all expired now.


29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Integer Arithmetic Coding 3 4
5 6
Variants with bounded integers can resolve first two drawbacks.


7 8
Here, we introduce a method by Witten, Neal, and Cleary from 1987, 9 10


abbreviated as WNC-algorithm. 11 12
13 14
Idea: Replace the interval [0, 1) with some finite set {0, . . . , M − 1} and


15 16
perform only integer operations.
17 18
This way, pure arithmetic coding is approximated. 19 20


21 22
Remark: Note that the WNC-algorithm includes EOF as an additional symbol.


23 24
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Intuition Behind the Choice of M
3 4
For the sake of simplicity, M is chosen to be a multiple of 4. 5 6


7 8
The larger M . . . 9 10


• the more accurate the approximation will be, i.e. the better the compression 11 12
ratio. 13 14
15 16
• the slower and more memory consuming the algorithm will be. 17 18
For A, B ∈ N, A < B, we will write sets of the form 19 20


21 22
{A, A + 1, . . . , B − 2, B − 1} = [A, B) ∩ N 23 24
25 26
27 28
as [A, B) to simplify the notation.
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Encoding Algorithm
3 4
1. Start with the interval [L, H) = [0, M ), and the underflow count k = 0. 5 6
7 8
M M 3M
2. (Underflow expansion) While 4 ≤L< 2 <H≤ 4 , increase k by one and 9 10
replace the interval by 11 12
  13 14
M M 15 16
[L, H) → 2L − , 2H −
2 2 17 18
19 20
M 21 22
3. (Rescaling) If H ≤ 2 replace the interval using
23 24
[L, H) → [2L, 2H), 25 26
27 28
add 01k to the code stream (i.e. a zero, followed by k ones), set k to zero, and 29 30
continue with step 2. 31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
M
1 2
4. (Rescaling) If L ≥ 2 replace the interval using 3 4
5 6
[L, H) → [2L − M, 2H − M ), 7 8
9 10
add 10k to the code stream, set k to zero, and continue with step 2. 11 12
5. If all symbols (including EOF) are encoded: 13 14
15 16
M
(a) If L < 4, add 01k+1 to the code stream. 17 18
(b) Otherwise (since H > 3M
add 10k+1 to the code stream. 19 20
4 ),
21 22
Encoding is finished. 23 24
6. Encode the next symbol si via 25 26
27 28
   
 i−1   i  29 30
X  X 
[L, H) → L +  pj (H − L) , L +  pj (H − L) 31 32
j=1 j=1 33 34
35 36
Continue with step 2. 37 38
39 40
Remark: Underflow expansions and rescaling are mandatory in this algorithm! 41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Example
3 4
Symbol L H new bits k 5 6
or operation 7 8
0 16 0 9 10
A 0 9 0 11 12
(0 + b0.6 · (16 − 0)c)
13 14
B 5 8 0
(0 + b0.6 · (9 − 0)c) (0 + b0.9 · (9 − 0)c) 15 16
x → 2x 10 16 0 0 17 18
x → 2x − 16 4 16 1 0 19 20
A 4 11 0 21 22
(4 + b0.0 · (16 − 4)c) (4 + b0.6 · (16 − 4)c)
23 24
x → 2x − 8 0 14 1
25 26
EOF 12 14 1
(0 + b0.9 · (14 − 0)c) (0 + b1 · (14 − 0)c) 27 28
x → 2x − 16 8 12 10 0 29 30
x → 2x − 16 0 8 1 0 31 32
x → 2x 0 16 0 0 33 34
35 36
Encoding the word “ABA” with the WNC algorithm (M = 16, pA = 0.6, pB = 0.3,
37 38
pEOF = 0.1). The final code word is 01101001, since L = 0 < M
4 = 4. 39 40
41 42
43 44
 Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Decoding 3 4
5 6
Decoding is similar to encoding, but there are some important differences: 7 8
9 10
Start with the first N = dlog2 M e bits of the code word r. We denote these bits 11 12


by v. Remove v from r. 13 14
When rescaling or applying underflow expansion, remove the first bit b from r 15 16


and update v: v → v + b. 17 18
19 20
Whenever a symbol is to be decoded, there is no general formula how to find the


21 22
correct interval:
j k 23 24
r−α v−L 25 26
• Emulating l from the pure decoding by H−L does not work in general.
27 28
• We deal with this problem in the last part of the learning unit (slide 40). 29 30
In the following example, we compute all intervals for all symbol to be decoded. 31 32


33 34
35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Example
3 4
v L H [LA, HA) [LB , HB ) [LEOF, HEOF) symbol 5 6
or operation 7 8
(0110)2 = 6 0 16 [0,9) [9,14) [14,16) A 9 10
(0110)2 = 6 0 9 [0,5) [5,8) [8,9) B 11 12
(0110)2 = 6 5 8 x → 2x 13 14
(1101)2 =13 10 16 x → 2x − 16 15 16
(1010)2 =10 4 16 [4,11) [11,14) [14,16) A 17 18
(1010)2 =10 4 11 x → 2x − 8 19 20
21 22
(1100)2 =12 0 14 [0,8) [8,12) [12,14) EOF
23 24
Decoding the code word “01101001” with M = 16, pA = 0.6, pB = 0.3, pEOF = 0.1. 25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Outline MI
A
1 2
Learning Unit 03: 3 4
Arithmetic Coding 5 6
7 8
9 10
11 12
13 14
15 16
Contents 17 18
1. Pure Zeroth-Order Arithmetic Encoding 19 20
21 22
2. Pure Zeroth-Order: Problems and Improvements 23 24
3. Pure Zeroth-Order Decoding 25 26
27 28
4. Integer Arithmetic Encoding 29 30
5. Integer Arithmetic Decoding 31 32
33 34
6. Integer Arithmetic Coding: Practical Issues 35 36
37 38
39 40
c 2023 Christian Schmaltz, Pascal Peter 41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Possible Problems and Solutions 3 4
5 6
Problem: An empty interval might occur.


7 8
Solution: This can be prevented by choosing M appropriately: 9 10


11 12
Where do we modify intervals? 13 14
15 16
Initial interval [0, M ) is non-empty for M ≥ 1. 17 18


19 20
Underflow expansion:


21 22
• For [L, H) → [2L − M M
2 , 2H − 2 ) 23 24
25 26
• Not an issue if L 6= H, independently of choice of M .
27 28
Rescaling: 29 30


• For [L, H) → [2L, 2H) or [L, H) → [2L − M, 2H − M ) 31 32

33 34
• Not an issue if L 6= H, independently of choice of M . 35 36
37 38
39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Where do we modify intervals?
3 4
Encoding the next symbol: 5 6


7 8
     9 10
Xi−1  Xi 
    11 12
[L, H) → L +  pj (H − L) , L +  pj (H − L)
j=1 j=1 13 14
15 16
17 18
Could be problematic due to floor function:


19 20
    21 22
Xi−1  X i 
    23 24
 pj (H − L) =  pj (H − L)
25 26
j=1 j=1
27 28
29 30
No problem for: pi(H − L) ≥ 1


31 32
We have pi ≥ pmin = minj pj , but what can we say about H − L? 33 34


M M
35 36
• after rescaling: L < 2 and H > 2 37 38
• after underflow expansion: L < M
or H > 34 M . 39 40
4
41 42
43 44
 Integer Arithmetic Coding MI
A
M M
1 2
Case 1: L < 4 ,H > 2:


3 4
5 6
M M 1 7 8
H −L> − = M
2 4 4 9 10
11 12
M
Case 2: L < 2 ,H > 34 M : 13 14


15 16
3 M 1 17 18
H −L> M − = M
4 2 4 19 20
21 22
Thus, we always have 23 24


M
pi(H − L) > pmin 25 26
4 27 28
And with 29 30


M 4
pmin >1 ⇔ M> 31 32
4 p min
33 34
we obtain a lower bound for M that guarantees no empty intervals. 35 36
37 38
39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
A slightly sharper lower bound for M can be shown.


3 4
5 6
Theorem
7 8
For a source alphabet S = {s1, . . . , sm} whose symbols appear with probabilities 9 10
p1, . . . pm, no empty interval appears in the WNC algorithm if 11 12
13 14
• M is divisible by 4
15 16
17 18
and
19 20
• 21 22
4
M≥ − 8, where pmin = min pi 23 24
pmin i
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Possible Problems and Solutions
3 4
Problem: It can happen that a new bit is necessary due to a rescaling or 5 6


underflow expansion even if the complete code word has already been used. 7 8
9 10
Solution: Any bit can be used in this situation. 11 12


• This is an effect of the buffering performed by the algorithm: It adds a bit 13 14

to the v-buffer for every rescaling/underflow, no matter if this will really be 15 16
needed for decoding the next symbol or not. 17 18
19 20
• The EoF symbol is already determined by the bits that were written in the 21 22
last encoding step. Additional bits do not affect the result. 23 24
• see Witten, Neal, and Cleary (1987) 25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Possible Problems and Solutions
3 4
5 6
Problem: During decoding, how do we know when we should stop reading new bits? 7 8
9 10
The EOF symbol tells us when the code word is done.


11 12
However, we might need more bits than have actually been written. 13 14


15 16
This is fine if the file ends early, but not if more data follows!


17 18
19 20
Solutions:
21 22
Write more bits during encoding or add arbitrary bits if end of file was reached.
23 24
How to find out how many bits we should write in addition? 25 26


Consider how decoding would work, count excess number of bits needed. 27 28


29 30
cost additional bits, but no EOF symbol required if source word length known 31 32


33 34
35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Approximating Probabilities
3 4
In practice, the probabilities pi are estimated, i.e. they have the form 5 6


7 8
ci 9 10
pi = ci ∈ N0,
C 11 12
13 14
where m 15 16
X
cj = C and 17 18
j=1 19 20
Pi−1 21 22
Similar to the definition of Pi := j=1 pj in Shannon’s method, we can define


23 24
Ci as 25 26
i
X
Ci = cj 27 28
j=1 29 30
with C0 = 0. 31 32
Ci
33 34
Obviously, Cm = C, and = Pi+1.


C 35 36
37 38
39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Finding the Interval in Decoding
3 4
In the situation described on the previous slide, it follows that (compare slide 28) 5 6


7 8
jP k jP k 9 10
i−1 i
L+ j=1 pj (H − L) ≤ v < L+ j=1 pj (H − L) 11 12
jC k j k 13 14
i−1 Ci
⇒ C (H − L) ≤ v−L < C (H − L) 15 16
j k
⇒
Ci−1 Ci 17 18
C (H − L) < v−L+1 ≤ C (H − L)
19 20
⇒ Ci−1(H − L) < (v − L + 1)C ≤ Ci(H − L) 21 22
23 24
⇒ Ci−1(H − L) ≤ (v − L + 1)C − 1 < Ci(H − L)
25 26
j k
⇒ Ci−1 ≤ w := (v−L+1)C−1 < Ci
27 28
H−L
29 30
31 32
Conclusion: w can be used to find the interval containing the next symbol to be 33 34
decoded. 35 36
37 38
39 40
41 42
43 44
 Integer Arithmetic Coding MI
A
1 2
Example
3 4
j k 5 6
(v−L+1)C−1
v L H w= H−L symbol 7 8
or operation 9 10
j k  69 
(6−0+1)10−1 11 12
(0110)2 = 6 0 16 16−0 = 16 = 4 ∈ [0, 6) A
j k 13 14
 69 
(0110)2 = 6 0 9 (6−0+1)10−1
= = 7 ∈ [6, 9) B 15 16
9−0 9
17 18
(0110)2 = 6 5 8 x → 2x 19 20
21 22
(1101)2 =13 10 16 x → 2x − 16 23 24
j k  69 
(10−4+1)10−1 25 26
(1010)2 =10 4 16 16−4 = 12 = 5 ∈ [0, 6) A
27 28
(1010)2 =10 4 11 x → 2x − 8 29 30
j k  129  31 32
(12−0+1)10−1
(1100)2 =12 0 14 14−0 = 14 = 9 ∈ [9, 10) EOF 33 34
35 36
Decoding the code word “01101001” with M = 16, C1 = 6, C2 = 9, C3 = C = 10 37 38
(compare slide 30). 39 40
41 42
43 44

Integer Arithmetic Coding MI

A
1 2
Final Remarks
3 4
In practice M is usually a power of 2, i.e. M = 2a for some a ∈ N. 5 6


⇒ a bits are sufficient to store each intermediate number. 7 8

9 10
Arithmetic coding is significantly slower than Huffman coding. 11 12


There are approaches to reduce the number of multiplications in arithmetic 13 14



coding. 15 16
17 18
19 20
21 22
23 24
25 26
27 28
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44
 Summary MI
A
1 2
Summary 3 4
5 6
Arithmetic coding can outperform any encoding scheme by:


7 8
• Encoding function maps the source word directly to a binary code. 9 10
11 12
• This is achieved by subdividing intervals proportionally to probabilities.
13 14
• Represent interval by dyadic fraction with shortest binary representation. 15 16
17 18
Rescaling and underflow expansions keep the interval size large.


19 20
Integer arithmetic coding approximates pure zeroth order coding. 21 22


23 24
25 26
Outlook 27 28
29 30
Can we reduce overhead generated by the methods introduced so far?


31 32
What should we do if we do not know probabilities a priori? 33 34


35 36
37 38
39 40
41 42
43 44

References MI
A
1 2
References
3 4
C. E. Shannon and W. Weaver. The mathematical theory of communication. 5 6


University of Illinois Press, Urbana, 1998. 7 8

(Detailed introduction to arithmetic coding.) 9 10
11 12
K. Sayood. Introduction to Data Compression. Morgan Kaufmann, 2006. 13 14


(Another detailed introduction to arithmetic coding.) 15 16

I. H. Witten, R. M. Neal, and J. G. Cleary. Arithmetic coding for data 17 18


compression. Communications of the ACM, 30(6):520–540, June 1987. 19 20

(Paper introducing the WNC-Algorithm) 21 22
23 24
T. C. Bell, J. G. Cleary, I. H. Witten. Text compression. Advanced Reference


25 26
Series. Prentice Hall, Englewood Cliffs, 1990.
27 28
(Yet another detailed introduction to arithmetic coding.)
29 30
31 32
33 34
35 36
37 38
39 40
41 42
43 44