Image Processing,

Retrieval, and Analysis (I)

Prof. Christian Bauckhage
Outline
Lecture 02

Image Acquisition / Digital Photography

Representations of Digital Images (I)

Summary

Exercises
Image Acquisition / Digital Photography

general principle
I visualize, record, and store light
Image Acquisition / Digital Photography

general principle
I visualize, record, and store light

light
I electromagnetic radiation where energy propagates
in form of electromagnetic waves
Image Acquisition / Digital Photography

general principle
I visualize, record, and store light

light
I electromagnetic radiation where energy propagates
in form of electromagnetic waves

I a particle or quantum phenomenon where photons

carry the electromagnetic force
Image Acquisition / Digital Photography

light
I for the speed of light, we have

c = νλ

where ν is frequency and λ is wavelength

Image Acquisition / Digital Photography

light
I for the speed of light, we have

c = νλ

where ν is frequency and λ is wavelength

I for the energy of light, we have

c
E = hν = h
λ
where h = 6.6260689633 × 10−34 Js is Plankck’s constant
Image Acquisition / Digital Photography

light
I for the speed of light, we have

c = νλ

where ν is frequency and λ is wavelength

I for the energy of light, we have

c
E = hν = h
λ
where h = 6.6260689633 × 10−34 Js is Plankck’s constant

⇒ high frequent light carries more energy

Image Acquisition / Digital Photography

electromagnetic spectrum

1m 1mm 1µm 1nm

100 10−1 10−2 10−3 10−4 10−5 10−6 10−7 10−8 10−9
wavelength [m]

108 109 1010 1011 1012 1013 1014 1015 1016 1017 1018
frequency [Hz]

VHF micro waves infrared ultraviolet X−rays

name
visible spectrum
Image Acquisition / Digital Photography

visualizing light: 3 major sources

I reflected light
photographic images, scanned images, . . .
I absorbed light
X-ray images, . . .
I emitted light
infrared images, . . .
Image Acquisition / Digital Photography

visualizing light: active imaging

I time-of-flight cameras record echos of
emitted infrared light (similar to Radar)

tof camera

consumer product depth image

Image Acquisition / Digital Photography

visualizing light: hyperspectral imaging

I record light from across bands of the em-spectrum

hyperscpectral image spectral responses

Image Acquisition / Digital Photography

recording light
I digital (color-)photography relies on CCD cameras
Image Acquisition / Digital Photography

recording light
I digital (color-)photography relies on CCD cameras

I CCD = charged-coupled device

= semiconductor device consisting of
thousands of photosensitive elements
Image Acquisition / Digital Photography

recording light
I digital (color-)photography relies on CCD cameras

I CCD = charged-coupled device

= semiconductor device consisting of
thousands of photosensitive elements

I photo sensors of consumer market cameras contain

at least 2592 × 1944 elements (≈ 3M-pixels)
Image Acquisition / Digital Photography

note:

I in general, CCD elements are not perfectly square

CCD chip schematic CCD matrix

Image Acquisition / Digital Photography

intensity images
I rays of light incident on the camera lens hit the
CCD matrix
I light intensity at each element of the matrix is
recorded as a number
Image Acquisition / Digital Photography

color images (I)

I split light into basic colors (typically red, green, blue)
I use a trichroic prism assembly and record each basic
color on a separate CCD array; this is expensive

trichroic prism three separate CCD arrays

Image Acquisition / Digital Photography

color images (II)

I superimpose a mosaic of color filters (a Bayer filter)
over a single CCD array; this is (much!) cheaper

Bayer filter single array

Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
I an electric field gathers electrons into packets
Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
I an electric field gathers electrons into packets
I packets are read and quantized to gray scale values
Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
I an electric field gathers electrons into packets
I packets are read and quantized to gray scale values
I depending on camera quality, a single CCD element
stores between 100,000 and 350,000 electrons before
saturating
Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
I an electric field gathers electrons into packets
I packets are read and quantized to gray scale values
I depending on camera quality, a single CCD element
stores between 100,000 and 350,000 electrons before
saturating
I representing 100,000 or more gray values would require
a storage capacity of at least 17 bit per CCD element;
Image Acquisition / Digital Photography

recording technology in detail

I photons incident on a CCD element generate a charge;
the more light there is, the more electrons are freed
I an electric field gathers electrons into packets
I packets are read and quantized to gray scale values
I depending on camera quality, a single CCD element
stores between 100,000 and 350,000 electrons before
saturating
I representing 100,000 or more gray values would require
a storage capacity of at least 17 bit per CCD element;
this is
a) too memory intensive
b) unnecessary because of the physiology of the human
visual system (details later!)
Image Acquisition / Digital Photography

intensity quantization
I intensity resolution is reduced using a certain mapping
(details later!)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5
Image Acquisition / Digital Photography

intensity quantization
I intensity resolution is reduced using a certain mapping
(details later!)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5

I mapping can be stored as a lookup table, i.e. the camera

does not have to do the math every time a picture is taken
Image Acquisition / Digital Photography

intensity quantization
I intensity resolution is reduced using a certain mapping
(details later!)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5

I mapping can be stored as a lookup table, i.e. the camera

does not have to do the math every time a picture is taken
I typically, images are stored using 256 = 28 intensity levels
per color channel
Image Acquisition / Digital Photography

intensity quantization
I intensity resolution is reduced using a certain mapping
(details later!)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5

I mapping can be stored as a lookup table, i.e. the camera

does not have to do the math every time a picture is taken
I typically, images are stored using 256 = 28 intensity levels
per color channel
I this requires 8 bit (= 1 byte) per CCD element (high quality
cameras or scanners store up to 14 bit per color channel)
Image Acquisition / Digital Photography

problems of CCD technology

I geometric distortions
I due to rectangular CCD elements; may
necessitate computational correction
Image Acquisition / Digital Photography

problems of CCD technology

I geometric distortions
I due to rectangular CCD elements; may
necessitate computational correction
I blooming
I overexposure may free too many
electrons which may overflow to
neighboring elements thus causing
highlight effects on the picture
Image Acquisition / Digital Photography

problems of CCD technology

I geometric distortions
I due to rectangular CCD elements; may
necessitate computational correction
I blooming
I overexposure may free too many
electrons which may overflow to
neighboring elements thus causing
highlight effects on the picture
I dark noise
I CCD sensors are sensitive to parts
of the invisible em-spectrum (UV or
IR radiation) which may cause
“inexplicable” phantoms or apparitions
Image Acquisition / Digital Photography

storing light
I numerous different file formats
I RAW
I JPG
I GIF
I PNG
I TIFF
I ...
I technical details maybe later
some fun facts

year of lecture 2003 2009 2012

typical resolution of a 3 MPixels 12 MPixels 17 MPixels

consumer digital camera

cell phones with cameras hardly available commodity devices standard technology

100 Euro buy 100 GB 1000 GB 2000 GB

flickr.com non-existent 3.000.000.000 pictures 6.000.000.000 pictures

as of November 2008 as of August 2011

facebook.com non-existent 10.000.000.000 pictures by far the largest image

as of November 2008 repository on the planet
Representations of Digital Images (I)

general implications of digital recording technology

I domain and range of digital images are discrete
⇔ digital images are digitized and quantized

I the smallest, basic elements of a digital image

are called pixels (picture elements)

I a digital image consists of a matrix of pixels

Representations of Digital Images (I)

pixels and pixel arrays

I in memory, digital images are stored as (2D) pixel arrays
which are also called pixmaps

i pij

I color images are stored in 3 to 4 pixmaps, one per color

channel and a possible α-layer
Representations of Digital Images (I)

pixels and pixel arrays

I the resolution of a digital image is characterized through
the width and the height of the pixmap, typically in a ratio
of 4:3, for instance: 1024 × 768
Representations of Digital Images (I)

pixels and pixel arrays

I the resolution of a digital image is characterized through
the width and the height of the pixmap, typically in a ratio
of 4:3, for instance: 1024 × 768

I given width, height, and color depth of an image, its

memory requirement can be readily calculated
Representations of Digital Images (I)

pixels and pixel arrays

I the resolution of a digital image is characterized through
the width and the height of the pixmap, typically in a ratio
of 4:3, for instance: 1024 × 768

I given width, height, and color depth of an image, its

memory requirement can be readily calculated

I stored in an uncompressed image format, a 1024 × 768

color image with 256 intensity levels per color channel
has a size of

1024 × 768 × 3 byte = 2, 414, 592 byte = 2.3 Mbyte

Representations of Digital Images (I)

pixels and pixel arrays

I the resolution of a digital image is characterized through
the width and the height of the pixmap, typically in a ratio
of 4:3, for instance: 1024 × 768

I given width, height, and color depth of an image, its

memory requirement can be readily calculated

I stored in an uncompressed image format, a 1024 × 768

color image with 256 intensity levels per color channel
has a size of

1024 × 768 × 3 byte = 2, 414, 592 byte = 2.3 Mbyte

Representations of Digital Images (I)

image formats
I data formats for storing digital images on Internet
servers, private hard drives, DVDs, . . .

I great variety (JPG, GIF, PNG, TGA, TIFF, PPM, . . . )

I pixmap (a.k.a. bitmap or raster graphics) oriented

storage of image data

I not to be confused with vector graphics formats

(SVG, FIG, . . . )
Representations of Digital Images (I)

image formats
I data formats for compressed storage of images have
their own scientific discipline (details maybe later!)
Representations of Digital Images (I)

image formats
I data formats for compressed storage of images have
their own scientific discipline (details maybe later!)

I at this point, we may note:

I compression reduces memory requirements
I but annihilates information
I differences are hardly visible to the eye . . .
but they are there
Representations of Digital Images (I)

example (I): PPM format

I PPM = portable pixmap
P6
# CREATOR: XV Version 3.10a Rev: 12/29/94 (PNG patch 1.2)
74 82
255
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff fe fe fe fe fe fe ff ff ff fe fe fe fe fe fc
ff ff fd c7 c6 c4 19 15 ...

I 1st line: magic number

I P1, P2, P3: ASCII format
I P4, P5, P6: binary format

I then: comments
I then: image width and image height
I then: maximum color value (same for all color channels)
I finally: image data (hexadecimal values) in (r, g, b) triplets
Representations of Digital Images (I)

example (II)
I enlarged portrait of a Gallic warrior
Representations of Digital Images (I)

example (II)
I PPM hex dump
00000000 50 36 0a 23 20 43 52 45 41 54 4f 52 3a 20 58 56 |P6.# CREATOR: XV|
00000010 20 56 65 72 73 69 6f 6e 20 33 2e 31 30 61 20 20 | Version 3.10a |

...

000000e0 50 4e 47 20 70 61 74 63 68 20 31 2e 32 29 20 20 |PNG patch 1.2) |

000000f0 51 75 61 6c 69 74 79 20 3d 20 31 30 30 2c 20 53 |Quality = 100, S|
00000100 6d 6f 6f 74 68 69 6e 67 20 3d 20 30 0a 37 34 20 |moothing = 0.74 |
00000110 38 32 0a 32 35 35 0a ff ff ff ff ff ff ff ff ff |82.255..........|
00000120 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000130 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000140 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000150 ff ff ff ff ff ff ff ff ff fe fe fe fe fe fe ff |................|
00000160 ff ff fe fe fe fe fe fc ff ff fd c7 c6 c4 19 15 |................|
00000170 12 17 10 0a 2d 1c 12 b1 97 80 ca ae 96 1c 07 00 |....-...........|
00000180 7e 67 55 fc d8 b8 eb c8 aa 39 28 18 08 00 00 07 |˜gU......9(.....|
00000190 00 00 04 00 00 06 00 00 05 00 00 03 00 00 04 03 |................|
000001a0 01 01 00 00 04 02 03 03 01 02 0f 06 00 b7 a1 8a |................|
000001b0 de c8 b1 1f 12 09 0b 06 02 8f 8b 88 ed ed eb e0 |................|
000001c0 e4 e7 d7 dc e0 e0 e4 e7 dd e1 e4 de e3 e7 dd e2 |................|
000001d0 e5 e6 e7 eb e5 e6 e8 3f 40 42 1d 1d 1d e7 e7 e7 |.......?@B......|
000001e0 fe fd fb ff ff fd c9 c9 c7 af ad ae 3a 38 39 c9 |............:89.|
000001f0 c8 c6 ff ff fd ff ff ff ff ff ff ff ff ff ff ff |................|
00000200 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
Representations of Digital Images (I)

example (II)
I JPG hex dump
00000000 ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 |......JFIF......|
00000010 00 01 00 00 ff fe 01 23 43 52 45 41 54 4f 52 3a |.......#CREATOR:|

...

000000e0 0a 0a 43 52 45 41 54 4f 52 3a 20 58 56 20 56 65 |..CREATOR: XV Ve|

000000f0 72 73 69 6f 6e 20 33 2e 31 30 61 20 20 52 65 76 |rsion 3.10a Rev|
00000100 3a 20 31 32 2f 32 39 2f 39 34 20 28 50 4e 47 20 |: 12/29/94 (PNG |
00000110 70 61 74 63 68 20 31 2e 32 29 20 20 51 75 61 6c |patch 1.2) Qual|
00000120 69 74 79 20 3d 20 31 30 30 2c 20 53 6d 6f 6f 74 |ity = 100, Smoot|
00000130 68 69 6e 67 20 3d 20 30 0a ff db 00 43 00 01 01 |hing = 0....C...|
00000140 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
00000150 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
00000160 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
00000170 01 01 01 01 01 01 01 01 01 01 01 01 01 01 ff db |................|
00000180 00 43 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |.C..............|
00000190 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
000001a0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
000001b0 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................|
000001c0 01 01 01 ff c0 00 11 08 00 52 00 4a 03 01 22 00 |.........R.J..".|
000001d0 02 11 01 03 11 01 ff c4 00 1f 00 00 01 05 01 01 |................|
000001e0 01 01 01 01 00 00 00 00 00 00 00 00 01 02 03 04 |................|
000001f0 05 06 07 08 09 0a 0b ff c4 00 b5 10 00 02 01 03 |................|
00000200 03 02 04 03 05 05 04 04 00 00 01 7d 01 02 03 00 |...........}....|
Representations of Digital Images (I)

example (II)
I result of ls -l
-rw-r--r-- 1 cbauckha users 10212 Apr 7 23:30 asterix-head.jpg
-rw-r--r-- 1 cbauckha users 18483 Apr 7 23:30 asterix-head.ppm
Summary

we now know about

I imaging methods

I CCD technology

I image formats (sort of)

Exercises

1. describe in your own words how a CCD camera works

2. what is blooming?

3. what is dark noise?