International Journal of Advanced Science and Technology

Vol. 29, No. 7, (2020), pp. 5135-5143

Hybrid Tone Mapping with Structural and Edge-preserving Priors

D.Venkat Reddy1, V. V. Satyanarayana Tallapragada2*, K. Raghu3, M. Venkat Naresh4
1
Professor, Department of ECE, Mahatma Gandhi Institute of Technology, Hyderabad.
dasari_reddy@yahoo.com
2*
Associate Professor, Department of ECE, Sree Vidyanikethan Engineering College, Tirupati.
satya.tvv@gmail.com
3
Assistant Professor, Department of ECE, Mahatma Gandhi Institute of Technology,
Hyderabad. raghukasula@mgit.ac.in
4
Assistant Professor, Department of ECE, Sree Vidyanikethan Engineering College, Tirupati.
nareshmvenkat@gmail.com

Abstract
High dynamic range images precisely define a real-world scene in a raw file format. The content of this
raw file may be modified to fit a different display unit. The process of converting the raw file into a standard
dynamic range image is of two kinds. The first is global mapping, where the mapping function is global,
and the same formula applies to all the pixels of the image. The second is a local mapping where different
mapping functions or employing a criterion on different parts of the image. The visual quality of local
mapping functions is better than that of global mapping. Still, there exist various halo effects in global
mapping. In this paper, these effects are minimized using a decomposition-based global mapping. First,
decompose the image into base and detail layers. 1 -norm optimization is used on the base layer while 0
-norm on the detail layer. An edge preserve filter and structural prior are imposed on the detail layer and
base layer respectively. This kind of hybrid use of edge-preserving and structural priors remove the halo
effects while preserving useful edges. The use of 1  0 hybrid decomposition and corresponding
optimization resulted in a high-quality tone mapped image. Simulation results prove that the proposed
scheme produces a Tone mapped image index over 0.8 for most of the test images, which is better than the
existing schemes.
Keywords: Tone mapping, base layer, detail layer, retinex decomposition, TMQI,

I. Introduction

A real-world scene contains a wide range of intensity levels. These are somehow quantized to be within the
range of the capacity of storage mechanism. But, many capturing devices came with the flexibility of
capturing and storing the real-world image with very few approximations. These images include a dynamic
range which quite different from the regular images. As there is a need for a much higher dynamic range to
capture and represent a real-world image, these images have higher dynamic ranges and correspondingly
referred to as High Dynamic Range (HDR) images. The HDR images have flexibility of adapting to
different low dynamic range (LDR) or standard dynamic range (SDR) images on demand.

The visual information in a real-world scene is coded in a raw file. This file contains broader visual
information of the scene area without considering storage and display considerations. Under the
consideration of a specific display unit, an image may be extracted from the raw file that can be displayed
by that display unit. This process of converting the raw file termed as HDR image data into LDR or SDR

ISSN: 2005-4238 IJAST 5135

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

is called tone mapping. The aim of any tone mapping operator (TMO) is to generate LDR image from the
HDR image without losing the crucial visual information of the real scene.

TMOs can be divided into two categories; global and local TMOs. The former considers a single relative
function for the whole image. Gamma correction and Reinhard TMOs are some of the well-known global
TMOs [1]. These TMOs, when mapping the intensity illuminance of the whole image using a single curve,
produce unwanted effects at many places of the image. The mapping if localized on circumstances of local
region of the image, these kinds of effects may be avoided. This is done in local TMOs. The mapping curves
are localized and adapted to each local region.

In case of a global tone mapping operator, the relation between a pixel in raw data and corresponding pixel
in the tone mapped image is same for any pixel. Most of the early TMOs are of this kind. These operators
consider the overall structural appearance of the image. The subjective perception of the natural scene is
preserved in these schemes. Global tone mapping algorithms are simple to design and faster in execution.
But, when the scene includes broad information where there is a need to include multiple backgrounds or
multiple objects and a wide range of exposures. In these situations, if the same relation or function maps
the pixels of raw file to mapped image, much of the visual information is lost.

The local tone mapping operator uses a distinct relation or function to map pixels of the raw file in different
parts of the image. The part may be a group of pixels or an object, or even an individual pixel as well.
Defining the area for which different functions of mapping is a difficult task in global TMOs. After defining
the area, a suitable function needs to be drawn to implement an adaptive TMO with respect to the location
of the raw image file. When the relation is too different for an adjacent location, the global operators may
result in artifacts called halo effect or ringing artifacts.

The local methods are implemented as layer decomposition schemes. Many 2-level encoding of HDR
images are proposed in the literature. In the first layer, only crucial visual information is included by
clipping the dynamic range. The second layer contains the remaining information, which may be obtained
as the difference between the original and the image of the first layer. The benefit of this method is that
there exists backward compatibility to existing formats. The first layer is named as the base layer, and the
residual layer is named as the detail layer. The calculation of the base layer is crucial for the successful
implementation of TMO. The first layer is generally projected using an edge preservative filter. The local
TMOs differ mainly in the design of the filter.

In the beginning age of the TMOs, filters designed based on kernels are used. Erik Reinhard, Micheal S.,
Peter S., and James F. used a filter with a Gaussian density function response [2]. These methods are tailored
to avoid the halo effects that generally arise in early schemes. But the problem with these filters is that they
boost even small details resulting in an over-enhanced image. The multiscale features of wavelet transform
are exploited for efficient tone-mapping. Retinex-based filter for tone mapping [3] is regarded as one of the
early works on TMOs using Retinex concepts. Zeev. F., Raanan. F., Dani L, and Richard S. presented a
weighted least square filter [4]. The filter is capable of effectively preserving the edges and good smoothing
features.
Most of the existing layer decomposition works enforce edge-preserving and constrained smoothing feature
on the base layer but not on the detail layer.

ISSN: 2005-4238 IJAST 5136

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

II. Retinex-based Decomposition

The basic retinex model of an image is:

S  R.L (1)
where S is the original image, R is reflectance, and L is illumination. In the retinex model, an image is
treated as combination of R and L. In the process of Single-scale retinex, which uses only single surrounding
information, halo effects occur at the boundary of objects and edges. These halo effects degrade the quality
of the image as the gradient associated with these artifacts is high. Hence, a multi-scale retinex scheme is
proposed to overcome this problem. In this scheme, multiple single-scale retinex is employed. Simultaneous
reflection and illumination estimation and weighted variation models are retinex-based schemes where the
illumination is predicted and utilized for image enhancement.

The retinex theory [1] is used as a fundamental theory in many core image processing schemes. The problem
of intrinsic images is initially stated by Barrow and Tenenbaum [5], to evaluate the features like depth,
surface orientation, occlusion, optical flow, and motion boundaries. Land and McCann [5] introduced the
retinex theory with an assumption of the Mondrian world. A painting of Dutch painter Mondrian is shown
in Fig. 1, along with the reflectance computation [5]. The intuition behind the theory of Retinex is stated
as the reflectance ratio of two patches chosen from Mondrian images may be found by ensuing a route
between the patches.

Fig. 1 Mondrian painting and the reflectance computation by Land and McCann [5]

Now, multiply the measured ratios of intensity along the path. The intensity ratios at the boundary of patches
resemble the change of reflectance. But the ratio within a patch approaches unity. Thresholding may be
proposed on these ratios to overturn the changes in intensity, which is smooth. These mechanisms are later
formalized [6] and extended to color images. The reflectance of each pixel in the channel I C is calculated

ISSN: 2005-4238 IJAST 5137

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

by building a family xk k 1  H W of J paths. The reflectance

Kj
RC  xn  of a pixel xn in the channel
C is calculated as follows.
K
1 J j
RC  xn      IC  xk   IC  xk 1 
J j 1 k 1
(2)

where
| x | if | x | 
 ( x)   (3)
0 else
xk k 1 so that the above equation can calculate the
Kj
The problem of this method is to choose the paths
reflectance accurately.

III. 1  0 Layer Decomposition Method [8]

The layer decomposition framework is devised by imposing an edge-preservation prior to the first
layer and structural prior to the second layer. Let the original image be denoted by S and the first layer by
B. Then, S-B becomes the detail layer. The optimization is given by

min N  



 S p  B p   1   i Bp  2  F  i  S p  B p  
2
(4)
B p 1 
 i x , y i x , y 

Here, the pixel is indexed by p, and total number of pixels is N. The optimization model contains
three terms. The first term in the expression  S p  Bp  enforces the difference between the base layer
2

original image to be significantly less. The second term  i Bp is a 1 gradient sparsity term that includes

the partial derivative. The third term expresses the detail layer structural property. It is expressed as a 0
gradient sparsity stretch. The term includes a representing function F(x) that is unity for x≠0 and zero for
x=0. The mixed-use of these regularization terms is crucial for an accurate representation of pixel
information of the raw file.

The above objective function is non-convex, as the third term contains 0 the regularization component. An
irregular direction scheme of multipliers is presented to resolve this problem. The objective function is
rewritten below with matrix terms.

min 1
s  b 2  1 b 1  21 F   s  b  
2
(5)
b 2
Here s and b belong to N
which are concatenated versions of S and B.  denotes a combination of
multiple gradient operations given as

  x , y   2 NN
(6)

ISSN: 2005-4238 IJAST 5138

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

The F function performs an elementwise non-zero indication. The resulting Lagrangian function is given
below.
1
L(b, c1 , c2 , y1 , y2 )  s  b 2  1 c1
2
1
2
 21T F  c2    c1  b  y1


  c2    s  b   y2

(7)



2
 c  b
1
2
2
 c2    s  b  2
2

where yi is the Lagranian dual variable. The quadratic programming problem results in splitting c1k , c2k , y1k
and y2k can be solved using FFT effectively.

When this 1  0 decomposition is applied to the radiance map, a piecewise smooth base and constant
detail layers can be obtained. When this single scale scheme is applied, tone mapping is completed. But, to
improve the quality of the tone mapped image, the scheme needs to be applied to the base layer repeatedly.
This actually results in a multiscale decomposition that improves the quality of tone mapping.

IV. Results and Discussions

Tone mapping is one of the image processing operations that has a different perspective with
respect to evaluation. Operations like denoising, deblurring, and compression have a reliable reference. But
the tone mapping does not have any reference image. At the same time, subjective measurement is also a
difficult task because the condition of the real scene is not known to the observer of the tone mapped image.
Tone Mapped image Quality Index (TMQI) is an objective quality assessment that parameter balances the
importance of two existing objective measures viz., Structural fidelity (S) and Naturalness (N)
[7][8][9][10].
TMQI  aS   (1  a) N  (11)
Here the parameter ‘a’ regulates the importance of structural fidelity and naturalness whose sensitivity is
given by α and β. Fig. 2 shows the corresponding standard dynamic range images along with the objective
measures.

ISSN: 2005-4238 IJAST 5139

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

S = 0.0138, N = 0.43276 and Q = 0.5421 S = 0.0125, N = 0.26813 and Q = 0.5268

S = 0.0132, N = 0.69556 and Q = 0.5351 S = 0.0122, N = 0.48825 and Q = 0.5225

S = 0.0537, N = 0.35045 and Q = 0.8209 S = 0.0138, N = 0.43276 and Q = 0.7313

Fig. 2 Simulation Results using the proposed Tone Mapping

Fig. 2 shows the tone mapped images along with structural fidelity, naturalness, and TMQI parameters.
Except for the third image, the remaining contain large content in the image. Images with large content
need to be structurally well-defined to retain much of its visual content. The structural prior imposed on the
base layer will serve this purpose. The structural prior enforces the interrelation among the structural groups
of the image to be intact. The spatial structure of the raw image file will be correspondingly mapped to
another space with a similar relation for a similar structure of the raw file. Hence, defining a similar structure
is crucial for this mechanism. The process of defining a similar structure is taken care of by the structural
prior. The simulation results of a few more images with large content are shown in Fig. 3.

S = 0.0517, N = 0.50878 and Q = 0.8113 S = 0.0560, N = 0.92354 and Q = 0.8314

ISSN: 2005-4238 IJAST 5140

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

S = 0.0138, N = 0.15316 and Q = 0.5425 S = 0.0561, N = 0.65279 and Q = 0.8316

Fig. 3 Tone Mapped Images using the proposed Tone Mapping

The use of structural and edge-preserving priors is very much useful in retaining useful visual
information which avoids the halo effects by bridging the gap between sharp regions using smooth patches.
This shown in Fig. 4 where results from other state-of-the-art schemes is also presented. Table 1 shows the
TMQI values of state-of-the-art schemes. The average TMQI values of large number of images are shown
in Table 1. The TMQI of proposed scheme produced 3.3% better results than the existing schemes.

Radiance Map EPDM [4]

ISSN: 2005-4238 IJAST 5141

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

GOLW [1] RBA [3]

LEMD [12] GDIR [13]

VACP [14] Proposed

Fig. 4. Performance Analysis

Table 1. Performance Analysis

Method TMQI
EPDM [4] 0.7803
GOLW [11] 0.7865
RBA [3] 0.7941
LEMD [12] 0.7986
GDIR [13] 0.7981
VACP [14] 0.8011
Proposed 0.8194

ISSN: 2005-4238 IJAST 5142

Copyright ⓒ 2020 SERSC
 International Journal of Advanced Science and Technology
Vol. 29, No. 7, (2020), pp. 5135-5143

V. Conclusions
In this paper, a layer decomposition-based tone mapping operator is conceived that preserves much of
the visual quality coded in the raw image file. The effects like over-enhancement and aura effects of global
tone mapping operators are dealt. Image is decomposed into base and detail layers, and a structural and
edge-preserve prior are operated on the base layer and detail layer respectively. Reduction of dynamic range
is has taken place in the base layer, while spatial structure boosting was done in detail layer. The multi-
scale tone mapping enforces both the priors to deal with the common artifacts generally present in global
tone mapping functions, and hence the visual quality is highly improved. The superiority of the proposed
mapping operator is proved with the simulation results.

References

[1] E. Reinhard and K. Devlin, “Dynamic range reduction inspired by photoreceptor physiology,”
IEEE Trans. on Visualization and Computer Graphics, vol. 11, no.1, pp. 13–24, 2005.
[2] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, “Photographic tone reproduction for digital
images,” ACM Trans. Graph., 21(3):267–276, July 2002.
[3] L. Meylan and S. Susstrunk, “High dynamic range image rendering with a retinex-based adaptive
filter,” IEEE Trans. on Image Proc., 2006, vol. 15, no. 9, pp. 2820–2830.
[4] Z. Farbman et al., “Edgepreserving decompositions for multi-scale tone and detail manipulation,”
ACM Trans. Graph., 2008, vol. 27, no. 3, pp. 1–67.
[5] https://davidstutz.de/retinex-theory-and-algorithm/ Last accessed 27-04-2020.
[6] B. K. P. Horn, “Determining lightness from an image,” Computer Graphics and Image Processing,
3(4):277-299, December 1974.
[7] Hojatollah Yeganeh, Zhou Wang, “Objective Quality Assessment of Tone-Mapped Images,” IEEE
Trans. on Image Processing, vol. 22, No. 2, February 2013.
[8] Zhetong Liang, Jun Xu, David Zhang, Zisheng Cao, Lei Zhang, “A Hybrid l1-l0 Layer
Decomposition Model for Tone Mapping,” CVPR 2018.
[9] Dr. G.A.E. Satish Kumar, Jaya Krishna Sunkara, “Multiresolution SVD based Image Fusion”,
IOSR Journal of VLSI and Signal Processing, vol. 7, Issue 1, ver. 1, pp. 20-27, Jan-Feb 2017. DOI:
10.9790/4200-0701012027.
[10] Jaya Krishna Sunkara, M Santhosh, C Suneetha, V. V. Satyanarayana Tallapragada, “Vector
Quantization – A Comprehensive Study”, International Journal of Engineering Science Invention,
7(5), pp. 27-38, 2018.
[11] Q. Shan, J. Jia, and Brown, “Globally optimized linear windowed tone mapping,” IEEE
Transactions on Visualization and Computer Graphics, 16(4), pp. 663–675, 2010.
[12] B. Gu, W. Li, M. Zhu, and M. Wang, “Local edge-preserving multiscale decomposition for high
dynamic range image tone mapping,” IEEE Transactions on Image Processing, 22(1), pp. 70–79,
2013.
[13] T. Shibata, M. Tanaka, and M. Okutomi, “Gradient-domain image reconstruction framework with
intensity-range and base-structure constraints,” CVPR, pp. 2745–2753, 2016.
[14] S. Ferradans et al., “An analysis of visual adaptation and contrast perception for tone mapping,”
IEEE Trans. on Pat. Ana. and Machine Intelligence, 33(10), pp. 2002–2012, 2011.

ISSN: 2005-4238 IJAST 5143

Copyright ⓒ 2020 SERSC