CHAPTER - I

INTRODUCTION TO PHOTO SKETCHING

1.1 Why the face?

The true identity of an individual is invaluable and extremely private information. While the
average person has no concerns with their identity being known, some individuals would
prefer to keep such knowledge secret despite the negative connotations attached to it.
Typically, the motivation for an individual to hide his/her identity is to evade detection by
law enforcement agencies for some type of criminal activity. Ongoing research in biometric
recognition has offered a crucial method to help identify who a person truly is.
Solving unconstrained face recognition often requires a considerable amount of research in
face modelling, feature extraction and matching. The past two decades have witnessed major
landmarks in face recognition algorithms. Turk and Pentland’s holistic Eigenface matching
algorithm, for example, served as the basis for modern face recognition engines. Since the
introduction of the Eigenface algorithm almost two decades back, face recognition accuracy
has increased exponentially, to the point where the face recognition rates under controlled
imaging conditions such as ambient lighting, frontal pose, neutral expression and uniformity
in background, are comparable to fingerprint and iris matching rates. Unfortunately, real
world face recognition scenarios seldom satisfy such controlled conditions. This has
prompted researchers to turn the focus of their research in face recognition to more difficult
problems such as varying degrees of illumination, non-frontal pose and occlusion.
1.2 Sketches and Images
A new and very recent problem of interest in face recognition has emerged that deals with
matching the sketched image of an individual to his/her photograph that may be stored in
government records. The consequence of being able to develop a robust algorithm for the
purpose of matching sketches to photos would be of great value to law enforcement agencies.
When a witness sees a crime being committed, in many instances, the verbal description of
the suspect elicited by the witness is used by a police artist to create a sketch. Many criminals
have been apprehended when they are identified based on such sketches. State of the art face
recognition techniques are not able to successfully match a sketch to the photographs which
are stored in law enforcement databases. The sketch to photo matching capability of an
algorithm implies a camera is not always mandatory to capture the face biometrics.
1.3 Modality
1.3.1. Intra-modality approach
Previous approaches to face sketch-photo recognition have mostly been concentrated on
reducing the modality gap by finding a transformation from sketch to photo aka pseudo-photo
generation or from photo to sketch aka pseudo-sketch generation and subsequently matching
with a photo or sketch respectively (intra-modality approach). Such an ideal transformation
from sketch to photo or otherwise may not exist in the first place. This is because the photo
and sketch come from two independent sources, i.e. a camera and an artist. Thus, it is not
feasible to enforce an assumption of the existence of a perfect transformation from one to
another. Although, an approximate transformation may be learnt from training a set of
sketch-photo pairs via deep learning techniques, but it is likely to fall short to the intrinsic
drawing style of a particular sketch artist and may not hold good for sketches drawn by other
artists, who again, might have their own signature strokes.
Thus, researchers are constantly trying to bridge the gap between the photo and sketch via
either the inter modality approach or the intra modality approach. The intra-modality
approach, as discussed above, aims to bring both sketch and image in the same modality by
generating a pseudo sketch.
1.3.2. Inter-modality approach
Whereas, the inter modality approach uses modality-invariant features to represent the images
and perform the similarity measure based on this representation.
For the inter modality approach, apart from having features that are independent of modality
to represent the image, the sketch and photo quality must also be taken into account because
they may affect the retrieval rate. To shine some more light, a sketch is drawn with no
consideration of lighting conditions (i.e., no illumination, just a greyscale version of an
image) but it may suffer from slight to moderate shape exaggeration (especially for forensic
sketches, because eyewitnesses may have a different perception from the artist). While for
photos, there is no chance of occurrence of shape exaggeration, but there is potential of them
being exposed to lighting variations. Disregarding these imperfections will obviously
sacrifice performance. Also, keeping in mind that sketches are drawn with no regard to the
lighting conditions, matching the features from such representations is inaccurate. If mugshot
photos are free from illumination variance, a better retrieval rate is expected because the
extracted features are devoid of any illumination effects.
1.4 Organisation of Report
In the coming chapters, we will see how important databases and their huge size are to any
study, how to overcome challenges faced while trying to obtain data, various deep learning
techniques, how GANs work to enhance performance, types of GANs, why we used the GAN
we used, how to train a model and subsequently, test it, and many other topics of interest
regarding this project.