Defocus Techniques for

Camera Dynamic Range

Expansion
Matthew Trentacoste, Cheryl Lau, Mushfiqur Rouf,
Rafal Mantiuk, Wolfgang Heidrich

University of British Columbia

Defocus DR expansion
• Sensors limited in dynamic range
Can be expanded, but tradeoffs exist

• Evaluate the opposite, reduce the dynamic

range of the scene incident on the sensor
by optical blurring, restore in software

1/9 1/9 1/9 5/9 5/9 5/9

5 1/9 1/9 1/9

= 5/9 5/9 5/9

1/9 1/9 1/9 5/9 5/9 5/9

Approach

• Use 2 techniques to aid:

coded aperture + deconvolution

• Aperture filter to improve deconvolution quality

PSF preserves more information
[Rashkar 2006][Levin 2007][Veeraraghavan 2007]

• Deconvolution to restore original image

Recent advances using natural image statistics
[Bando 2007][Levin 2007]
Physical setup

• Rays from scene pass through aperture

plane and focused onto sensor

• Cone of rays from out-of-focus points

intersects sensor, forming the shape of the
aperture

• Aonto
pattern in the aperture plane is projected
the sensor for out-of-focus points
Coded Aperture
• Originally from x-ray astronomy
[Fenimore 1978][Gottesman 1989]

• Structured arrays + decoding algorithm with

resolution of pinhole, but better SNR

• Employed in visible light photography

[Rashkar 2006][Levin 2007][Veeraraghavan 2007]

• Improve frequency properties of filter

Aperture filters
• What makes a good filter?
• Frequency response
• Position and spacing of zero frequencies
• Diffraction / transmission
Deconvolution
• Restore image distorted by PSF
[Wiener 1964][Richardson 1972][Lucy 1974]

f = f0 ⊗ k + η

• Ill-posed, infinite solutions

• No exact solution due to noise
• Division in FFT, issues with small
values in OTF of filter
Deconvolution
• Current state-of-the-art methods rely on natural
image statistics

• Real-world images share several properties:

Heavy-tail distribution of gradients

• Prior term in deconvolution algorithms

[Bando 2007][Levin 2007]

• Favors interpretations of the image with all the

gradient intensity at a few pixels
Evaluation
• Goal : determine whether any combo of filter /
deconvolution yields meaningful reduction in DR
with acceptable final image quality

• Measure DR reduction both in terms of image

local contrast and filter

• Measure image quality as difference between

deconvolved and original images
Source material
Atrium Morning Atrium Night

Figure 3.3: Sample images used in evaluation.

Radius Atrium Morning Atrium Night

min max reduction min max reduction

Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65

Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.

Atrium Morning Atrium Night

Source material
Atrium Morning Atrium Night

Figure 3.3: Sample images used in evaluation.

Radius Atrium Morning Atrium Night

min max reduction min max reduction

Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65

2.45 EV
Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.

Atrium Morning Atrium Night

Source material
Atrium Morning Atrium Night

Figure 3.3: Sample images used in evaluation.

Radius Atrium Morning Atrium Night

min max reduction min max reduction

Original 0.00 11.0 0.00 12.0
1 0.00 10.8 0.200 0.452 12.0 0.452
2 0.00 10.6 0.424 0.622 12.0 0.622
3 0.00 10.3 0.716 1.163 11.8 1.34
4 0.02 10.0 1.00 1.436 11.4 1.99
5 0.08 9.94 1.14 1.589 11.4 2.23
6 0.15 9.92 1.24 1.731 11.2 2.51
8 0.31 9.83 1.48 1.890 10.8 3.13
9 0.40 9.79 1.61 1.950 10.5 3.41
11 0.66 9.71 1.94 2.08 10.3 3.74
13 0.86 9.67 2.19 2.18 10.1 4.13
16 1.04 9.59 2.45 2.26 9.61 4.65

2.45 EV 4.56 EV
Figure 3.4: Amount of reduction in dynamic range as a function of radius of a standard aperture (disk)
filter in pixels. All units are in terms of powers of two, referred to as exposure value (EV) stops.

Atrium Morning Atrium Night

Tests
• Filters evaluated: • Deconvolution evaluated:
• Normal aperture • Wiener filtering
• Gaussian • Richardson-Lucy
• Veeraraghavan • Bando
• Levin • Levin
• Zhou
Evaluation (cont)

• Success criteria:
• Reduction of at least 2 stops
to justify the computational cost of deconv

• Quality of at least PSNR 35

Images

Weiner Richardson-Lucy Bando Levin

filter=Zhou, noise = 0, radius = 1

Images

Weiner Richardson-Lucy Bando Levin

filter=Zhou, noise = 0, radius = 5

Images

Weiner Richardson-Lucy Bando Levin

filter=Zhou, noise = 0, radius = 16

 Deconv: no noise
orning deconvolution
Atrium Morning deconvolution
Atrium Morning deconvolution
Atrium Night deconvolution
60 60 60
Weiner
Weiner Richardson−Lucy Weiner Weiner
Richardson−Lucy
55 55
Richardson−Lucy
Bando
Levin
Richardson−Lucy
Bando
Levin

50
55 Bando 50 Bando
Levin Levin
45 45

50
40 40
PSNR (dB)

PSNR (dB)
PSNR

PSNR
35 35

45
30 30

25 25
40
20 20
PSNR (dB)

15 15
35
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)

30 DR reduction DR reduction

25
 Aperture: no noise
Morning aperture filter
Atrium Morning aperture filter
Atrium Morning aperture filter
Atrium Night aperture filter
60 60 60
Standard Aperture
Standard Aperture
Gaussian
Standard Aperture
Standard Aperture
Gaussian
55
Gaussian Veeraraghavan
Zhou
55
Gaussian
Veeraraghavan
Zhou
55 Veeraraghavan
Levin Veeraraghavan
Levin
50 50
Zhou Zhou
45 Levin 45 Levin
50
40 40
PSNR (dB)

PSNR (dB)
PSNR

PSNR
35 35
45
30 30

25 25
40
PSNR (dB)

20 20

15 35 15

10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30DR reduction DR reduction

25

20
 Deconv: noise
orning deconvolution
Atrium Morning deconvolution
Atrium Morning deconvolution
Atrium Night deconvolution
60 60 60
Weiner
Weiner Richardson−Lucy Weiner Weiner
Richardson−Lucy
55 55
Richardson−Lucy
Bando
Levin
Richardson−Lucy
Bando
Levin

50
55 Bando 50 Bando
Levin Levin
45 45

50
40 40
PSNR (dB)

PSNR (dB)
PSNR

PSNR
35 35

45
30 30

25 25
40
20 20
PSNR (dB)

15 15
35
10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)

30 DR reduction DR reduction

25
 Aperture: noise
Morning aperture filter
Atrium Morning aperture filter
Atrium Morning aperture filter
Atrium Night aperture filter
60 60 60
Standard Aperture
Standard Aperture
Gaussian
Standard Aperture
Standard Aperture
Gaussian
55
Gaussian Veeraraghavan
Zhou
55
Gaussian
Veeraraghavan
Zhou
55 Veeraraghavan
Levin Veeraraghavan
Levin
50 50
Zhou Zhou
45 Levin 45 Levin
50
40 40
PSNR (dB)

PSNR (dB)
PSNR

PSNR
35 35
45
30 30

25 25
40
PSNR (dB)

20 20

15 35 15

10 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Dynamic range reduction (EV stops) Dynamic range reduction (EV stops)
30DR reduction DR reduction

25

20
Conclusions

• Levin deconv the best, obtaining results with

coded filters at very low noise levels

• No combination of filter and deconvolution

consistently produced acceptable results

• Efficiency of the approach is scene dependent

Most efficient for small, isolated bright regions