Sensor Fingerprinting: Challenges and Trends

Walter J. Scheirer
wjs3@vast.uccs.edu VAST Lab University of Colorado at Colorado Springs
IEEE-WVU, Anchorage - 2008

What is Sensor Fingerprinting?

Identifying the data acquisition device that generated a given image
Device Class Identification Specific Device Identification

IEEE-WVU, Anchorage - 2008

Why would we want to do this?

Digital Image Forensics
Over 90% of our of computer forensics cases involve child pornography - unnamed detective from the
Pennsylvania State Police
QuickTime an d a decompressor are need ed to see this p icture .

The Burden of digital photography

Is an image real, or has it been rendered by a computer? Is an image an original, or has it been altered by digital image processing?

? ? ? ?

?
? ?

IEEE-WVU, Anchorage - 2008

Ashcroft v. Free Speech Coalition

Supreme Court Ruling:
Struck down two provisions of the Child Pornography Prevention Act of 1996 Synthetic images depicting what would be considered illegal in a genuine image are legal, and protected by the first amendment

How does a forensic investigator determine what is real, and what is synthetic?
*http://www.law.cornell.edu/supct/html/00-795.ZO.html
IEEE-WVU, Anchorage - 2008

Digital Image Forensics

A three stage process:
1. Image Source Identification (Sensor Fingerprinting) 2. Discrimination of Synthetic Images from Real Images 3. Image Forgery Detection

IEEE-WVU, Anchorage - 2008

Digital Image Forensics

If an image is indeed real
What can we determine about the nature of the source? Can we pinpoint the exact model of the device? Can we prove beyond a reasonable doubt that a certain device produced the image? Evidence!
IEEE-WVU, Anchorage - 2008

QuickTime and a decompressor are need ed to see this picture.

Vision of the Unseen

QuickTime and a decompressor are need ed to see this picture.

Sensor fingerprinting relies on underlying characteristics of sensors and image processing techniques used by digital cameras
Artifacts, distortions, statistical properties

These characteristics are nearly always imperceptible to the human eye

IEEE-WVU, Anchorage - 2008

Image Acquisition Pipeline

Lens

Filter(s)

Color Filter Array

Sensor

Camera Processing

IEEE-WVU, Anchorage - 2008

Sensors
CCD & CMOS Tremendous variation between individual sensors Defects caused by manufacturing or use

IEEE-WVU, Anchorage - 2008

Color Filter Arrays

RGB: Bayer Pattern

CMYK

IEEE-WVU, Anchorage - 2008 10

Sensor Fingerprinting: Source Model Identification

Digital Cameras
Lens Size of sensor Choice of CFA Demosaicing algorithm Color processing algorithm

Many manufacturers use the same components

IEEE-WVU, Anchorage - 2008

11

Image Features
Revisiting some of the techniques of steganalysis: Kharrazi et al. 2004*
Defines a set of 34 features inspired by universal steganlaysis techniques Color features, wavelet coefficient statistics, image quality metrics
*M. Kharrazi, H. T. Sencar, and N. Memon, Blind Source Camera Identication, Proc. of IEEE ICIP (2004)

IEEE-WVU, Anchorage - 2008 12

Image Color Features

Average pixel value
Gray world assumption: average values in RGB channels of an image should average to gray

RGB pairs correlation

Variance in correlation of RG, RB, and GB pairs across sensor manufacturers

Neighbor distribution center of mass

Calculate the number of pixel neighbors for each pixel value Distribution gives an indication of the sensitivity of the camera to different intensity levels

IEEE-WVU, Anchorage - 2008 13

Image Color Features

RGB Pairs Energy Ratio
|G|2 E1 = |B|2
|G|2 E2 = |R|2 |B|2 E1 = |R|2

Wavelet Domain Statistics

Decompose each color band using separable quadric mirror filters* Calculate the mean for each sub-band
*H. Farid and S. Lyu, Detecting hidden messages using higher-order statistics and support vector machines, 5th International Workshop on Information Hiding. 2002

IEEE-WVU, Anchorage - 2008 14

Image Quality Metrics

Pixel difference based measures Mean square error, mean absolute error, modified infinity norm Correlation based measures Normalized cross correlation, Czekanowski correlation Spectral distance based measure Spectral phase and magnitude errors *I. Avcibas, N. Memon, and B. Sankur, Steganalysis using image quality metrics.
IEEE transactions on Image Processing, January 2003.

IEEE-WVU, Anchorage - 2008 15

Classification
Construct feature vector out of the calculated features for a given image Build training sets for each class of camera Use machine learning for classification of images with unknown sources
Camera 1

Camera 2

SVM Hyperplane
IEEE-WVU, Anchorage - 2008 16

Results of Kharrazi et al.

Standard two class SVM
Nikon 99.88 2.4 Sony 0.12 97.6 Nikon 96.08 9.25 Sony 3.91 90.74

Nikon Sony

Nikon Sony

Serious Quantization effects observed, driving up classification performance

Images recompressed to a JPEG quality level of 75

IEEE-WVU, Anchorage - 2008 17

Results of Kharrazi et al.

Multi-class SVM
Nikon Nikon Sony Canon (S110) 89.67 3.56 7.85 Sony 0.22 95.24 0.6 Canon (S110) 4.77 0.31 78.71 Canon (S100) 1.64 0.34 4.78 Canon (S200) 3.7 0.53 8.04

Canon (S100)
Canon (S200)

3.14
5.96

0.32
2.27

3.57
7.88

92.84
0.23

0.11
83.63

* See also M.J. Tsai and G.H. Wu, Using Image Features to Identify Camera Sources, Proc. of IEEE ICASSP (2006).
IEEE-WVU, Anchorage - 2008 18

CFA and Demosaicing Artifacts

Choice of CFA

Bayer Pattern (RGB)

CMYK

Demosaicing

IEEE-WVU, Anchorage - 2008 19

Image Features from CFA - Cell Phone Cameras

eliktutan et al. 2005* Motivation: proprietary interpolation algorithms leave correlations across adjacent bit planes of the images Binary Similarity Measures (also used in steganalysis) Define a stencil function:

n c (k,b) =

1 if xc = 0 xn = 0 2 if xc = 0 xn = 1 3 if xc = 1 xn = 0

4 if xc = 1 xn = 1

*O. Celiktutan, I. Avcibas, B. Sankur and N. Memon, Source Cell-Phone Identication, Proc. of ADCOM (2005).
IEEE-WVU, Anchorage - 2008 20

Image Features - Cell Phone Cameras

Sum n over its four neighbors, and c (k,b) over all MxN pixels Normalize the agreement scores:
p = (k,b) / k
b

(k,b)
k

Define binary Kullback Leibler distance (feature 1): 4 p7

m1 = n=1

p log
7 n

n 8 n

IEEE-WVU, Anchorage - 2008 21

Image Features - Cell Phone Cameras

Define neighborhood weighting mask (feature 2):

S=

x2
i=0
i

1
i

128 64

256 32

8 16

Score Function

m2 =

|
n=0

511

Weighting Pattern of the Neighbors

7 n

8 n|

Absolute Histogram Difference

IEEE-WVU, Anchorage - 2008 22

Image Features - Cell Phone Cameras

Czenakowski distance - feature 3 Classification
KNN and multi-class SVM
Scatter plot of three features for three different cameras from eliktutan et al. 2005

IEEE-WVU, Anchorage - 2008 23

Results of eliktutan et al. 2005

KNN classification
Overall performance = 98.7% Sony K700 Sony K700 Motorolla V3 Nokia 6230 100 0 0 Motorolla V3 0 100 4 Nokia 6230 0 0 96 Sony K700 Motorolla V3 Nokia 7270 Overall performance = 81.3% Sony K750 71 1 18 Motorolla V3 6 97 6 Nokia 7270 23 2 76

IEEE-WVU, Anchorage - 2008 24

Results of eliktutan et al. 2005

S1 S2 4 63 3 0 6 1 1 16 M1 0 1 60 22 2 2 7 2 M2 0 0 11 67 3 6 10 0 N1 0 4 3 4 57 3 1 7 N2 0 0 1 9 3 68 18 0 N3 0 0 3 5 6 22 62 0 N4 3 32 3 0 7 0 1 36 L1 0 3 5 5 18 0 1 12

Multi-class SVM classification Overall accuracy: 62.3% Random guessing: 11.1%

S1 S2 M1 M2 N1 N2 N3 N4

92 5 0 0 0 0 0 3

L1

21

18

56

IEEE-WVU, Anchorage - 2008 25

The Expectation/Maximization Algorithm

Popescu 2005* Motivating assumption: rows and columns of interpolated images are likely to be correlated with their neighbors Two steps:
Expectation: the probability of each sample belonging to each model is estimated Maximization: the specific form of the correlations between samples is estimated Both steps are iterated till convergence
*A. Popescu, Statistical Tools for Digital Image Forensics, Ph.D. Dissertation, Department of Computer Science, Darthmouth College (2005).
IEEE-WVU, Anchorage - 2008 26

The Expectation/Maximization Algorithm

Assume that each sample belongs to one of two models
M1, if a sample is linearly correlated with its neighbors
f(x,y) = u,v=-N u,v
N

f(x+u, y+v) + n(x,y)

M2, if a sample is not correlated with its neighbors

IEEE-WVU, Anchorage - 2008 27

The Expectation/Maximization Algorithm

Expectation Step
The probability of each sample belonging to M1 is estimated using Bayes rule:

Pr{f(x,y) M1 | f(x,y)} =
Pr{ f(x,y) | f(x,y) M1} Pr{f(x,y) M1}

2 i=1

Pr{ f(x,y) | f(x,y) Mi} Pr{ f(x,y) Mi}

IEEE-WVU, Anchorage - 2008 28

The Expectation/Maximization Algorithm

The probability of observing a sample knowing it was generated by a model M1:

Pr{f(x,y) | f(x,y) M1} = 1

exp

[ (
1 - 22

f ( x,y) -

u,v=-N

u,v

f ( x + u, y + v )

)]

IEEE-WVU, Anchorage - 2008 29

The Expectation/Maximization Algorithm

The Maximization step

Generate a new estimate of using weighted least squares:
E() =

w(x,y) (f(x,y) -
x,y u,v=-N

u,vf(x+u,

y+v)

Execute both steps until a stable is achieved. Final result maximizes the likelihood of observed samples
IEEE-WVU, Anchorage - 2008 30

Demosaicing Algorithms*
Image p F(p)

bilinear

bi-cubic

QuickTime and a decompressor are neede d to see this picture.

smooth line
*Popescu 2005

IEEE-WVU, Anchorage - 2008 31

Demosaicing Algorithms*
Image Median 3x3 p
QuickTime and a decompressor are neede d to see this picture.

F(p)

Median 5x5 gradiant

*Popescu 2005

QuickTime and a decompressor are neede d to see this picture.

IEEE-WVU, Anchorage - 2008 32

Demoasicing Algorithms*
Image adaptive color plane p F(p)

Variable number of gradiants

No CFA interpolation
*Popescu 2005

QuickTime an d a decompressor are need ed to see this p icture .

IEEE-WVU, Anchorage - 2008 33

The Expectation/Maximization Algorithm

Results of Popescu 2005 Average accuracy over all pairs of algorithms tested was 97% Minimum testing accuracy was 87% (3x3 median filter vs. variable number of gradients)

Estimated interpolation coef. from 100 images CFA for 8 different algorithms projected on a 2D space

IEEE-WVU, Anchorage - 2008 34

EM Algorithm for Camera Detection

Bayram et al. 2005*
Applies EM algorithm to identify 3 different cameras, classification via SVM
A c t u a l

predicted
nikon nikon sony 95.71 17.14 sony 4.29 82.86 nikon sony nikon 91.43 5.71 sony 8.57 94.29 nikon sony nikon 94.64 3.57 sony 5.36 96.43

3x3 Interpolation kernel

4x4 Interpolation kernel

5x5 Interpolation kernel

*S. Bayram, H. Sencar and N. Memon, Source Camera Identification Based on CFA interpolation, Proc. of the IEEE ICIP (2005)

IEEE-WVU, Anchorage - 2008 35

EM Algorithm for Camera Detection

Bayram et al. 2005
Multiclass SVM
predicted

Nikon
A c t u a l

Sony 10.71 75 10.71

Canon 3.57 14.28 89.28

Nikon Sony Canon

85.71 10.71 0

5x5 interpolation kernel

IEEE-WVU, Anchorage - 2008 36

Enhancements to the EM Approach

Bayram et al. 2006* Better detection of interpolation artifacts in smooth images
Low-order interpolation introduces periodicity in the variance of the second derivative of an interpolated signal
QuickTime and a decompressor are need ed to see this picture.

QuickTime and a decompressor are neede d to se e this picture.

QuickTime a nd a decompressor are need ed to see this picture.

QuickTime and a decompressor are neede d to se e this picture.

*S. Bayram, H.T. Sencar and N. Memon, Improvements on Source Camera -Model Identiciation Based on CFA Interpolation, Proc. of WG 11.9 Int. Conf. on Digital Forensics (2006). IEEE-WVU, Anchorage - 2008 37

Enhancements to the EM Approach

Results of Bayram et al. 2006
predicted A c t u a l

Nikon Nikon Sony Canon 85.71 10.71 0

Sony 10.71 75 10.71

Canon 3.57 14.28 89.28 Nikon Sony

Nikon 76.78 12.5

Sony 8.92 76.78 10.71

Canon 14.28 10.71 69.64

Canon 19.64

5x5 interpolation kernel

Periodicity in the second order derivative

Nikon

Sony

Canon

Nikon
Sony

94.78
2.08

1.50
95.28

3.72
2.64

Canon

2.26

97.74

Combined set of features

IEEE-WVU, Anchorage - 2008 38

Enhancements to the EM Approach

Long et al. 2006
Use modeling error, instead of interpolation filter coefficients

Swaminathan et al. 2006

Assumes a CFA pattern, discriminating between the interpolated and un-interpolated pixel locations and values Estimate interpolation filter coefficients corresponding to the pattern Compute error between an image of newly interpolated interpolated pixels, and the actual image

IEEE-WVU, Anchorage - 2008 39

Lens Distortions

Radial distortion

Rectified Image

Compensation for radial distortion induces unique artifacts in the images Choi et al. introduces a second order radial symmetric distortion model Model parameters are used as classification features Accuracy ~91%
IEEE-WVU, Anchorage - 2008 40

Sensor Fingerprinting: Individual Source Identification

Need more detail beyond what weve looked at so far with Source Model Identification
Hardware and component imperfections, defects, and faults Effects of manufacturing process, environment, operating conditions Aberrations produced by a lens, noisy sensor, dust on the lens
Artifacts may be temporal!

IEEE-WVU, Anchorage - 2008 41

Imaging Sensor Imperfections

Early work: Kurosawa et al. 1999
Detect fixed pattern noise caused by dark current in digital video cameras Dark current: the rate that electrons accumulate in each pixel due to thermal action
Accumulation of dark current on CCD (dark frame brightened 37 times for viewing)
Image credit: http://www.diaginc.com/t echforum/imagecorrecti ons.shtml

*K. Kurosawa, K. Kuroki and N. Saitoh, CCD Fingerprint Method, Proc. of the IEEE ICIP (1999)

IEEE-WVU, Anchorage - 2008 42

Imaging Sensor Imperfections

Kurosawa et al. 1999
DCR-VX1000

QuickTime and a decompressor are neede d to see this picture.

QuickTime an d a decompressor are need ed to see this p icture .

QuickTime an d a decompressor are need ed to see this p icture .

QuickTime and a decompressor are needed to se e this picture.

80642

30821

72567

49967

Serial Numbers

IEEE-WVU, Anchorage - 2008 43

Imaging Sensor Imperfections

Geradts et al. 2001*
Detect hot pixels, cold/dead pixels, pixel traps, and cluster defects
Same Camera Model, Averaged Blank Images

QuickTime and a decompressor are need ed to see this picture.

Camera A
*Z.J. Geradts, J. Bijhold, M. Kieft, K. Kurusawa, K. Kuroki, and N Saitoh, Methods for Identification of Images Acquired with Digital Cameras, Proc. of SPIE, vol. 4232 (2001)

Camera B

IEEE-WVU, Anchorage - 2008 44

Imaging Sensor Imperfections

Effects of temperature on pixel response
QuickTime an d a decompressor are need ed to see this picture .

0 C 20 C Pixel defects in a real image

40 C

QuickTime and a decompressor are need ed to see this picture.

IEEE-WVU, Anchorage - 2008 45

Imaging Sensor Imperfections

No quantitative analysis of previous two methods Lukas et al. 2006*: Formal quantification and analysis of sensor noise for identification
Pattern noise

Fixed pattern noise

Photo-response nonuniformity noise

Pixel non-uniformity

Low-frequency defects

*J. Lukas, J. Fridrich and M. Goljan, Digital Camera Identification From Sensor Pattern Noise, IEEE Trans. On Inf. Forensics and Security, vol. 1, no. 2, pp. 205-214.

IEEE-WVU, Anchorage - 2008 46

Pixel Non-Uniformity Noise

The signal r exhibits properties of a white noise signal with an attenuated high frequency band
Attenuation is likely due to the low-pass character of the CFA interpolation algorithm
QuickTime and a decompressor are neede d to se e this picture.

PNU noise cannot be found in saturated pixels (255)

Magnitude of Fourier transform of one row in an image obtained as average over 118 images of a flat seen

IEEE-WVU, Anchorage - 2008 47

PNU Camera Identification Algorithm

Establish a camera reference pattern Pc, which is an approximation to the PNU noise An approximation of PNU noise P(k):
(P1 + P2 + + PN) / N

Optimization: suppress scene content by applying denoising filter F, and averaging noise residuals n(k):
n(k) = P(k) - F(P(k))

IEEE-WVU, Anchorage - 2008 48

PNU Camera Identification Algorithm

Calculate the correlation C between the noise residual n = p - F(p) and the camera reference pattern PC:
C(p) = (n - n) (PC - PC)
||n - n|| ||PC - PC||

IEEE-WVU, Anchorage - 2008 49

PNU Identification: Results of Lukas et al. 2006

QuickTime and a decompressor are neede d to se e this picture.

QuickTime and a decompressor are neede d to see this picture.

Distribution of the correlation of the reference pattern from Canon S40 with noise residual from 300 Olympus C765 images

Distribution of the correlation of the reference pattern from Nikon D100 with noise residual from 8x300 images from all other cameras

IEEE-WVU, Anchorage - 2008 50

PNU Identification: Results of Lukas et al. 2006

t Nikon C765-1 C765-2 G2 S40 Sigma Kodak C3030 A10 0.0449 0.0170 0.0080 0.0297 0.0322 0.0063 0.0997 0.0209 0.0166 FRR 4.68x10-3 3.79x10-4 5.75x10-11 2.31x10-4 1.42x10-4 2.73x10-4 1.14x10-11 1.87x10-3 7.59x10-5

Decision threshold t and FRR for FAR = 10-3

IEEE-WVU, Anchorage - 2008 51

PNU Identification: Results of Lukas et al. 2006

QuickTime an d a decompressor are need ed to see this p icture .

QuickTime an d a decompressor are need ed to see this picture.

Nikon D100 with noise residual from approximately 6x300 JPEG compressed images with quality factor 90 from 6 other cameras

Correlation of noise residuals from 84 Cannon G2 1600x1200 JPEG images with 6 reference patterns

IEEE-WVU, Anchorage - 2008 52

Application of PNU Camera Identification

Sutcu et al. 2007*
Fuse pattern noise properties with demosaicing characteristics
yes Pattern noise match Classifier based on interpolation coefficients no yes Decision: image was taken by this camera

no Decision: image was not taken by this camera

*Y. Sutcu, S. Bayram, H.T. Sencar and N. Memon, Improvements on Sensor Noise Based Camera Identification, Proc. of IEEE ICME (2007).

IEEE-WVU, Anchorage - 2008 53

Application of PNU Camera Identification

QuickTime an d a decompressor are need ed to see this p icture .
QuickTime and a decompressor are neede d to see this picture.

QuickTime and a decompressor are neede d to se e this picture.

IEEE-WVU, Anchorage - 2008 54