See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261635313

Implementation of an Image Signal Processor for Reconfigurable Processors

Conference Paper  in  Digest of Technical Papers - IEEE International Conference on Consumer Electronics · January 2014
DOI: 10.1109/ICCE.2014.6775944

CITATIONS READS
3 674

4 authors:

Seong-Won Lee Seung-Hyun Choi

Kwangwoon University Kwangwoon University
80 PUBLICATIONS   501 CITATIONS    7 PUBLICATIONS   4 CITATIONS   

SEE PROFILE SEE PROFILE

Junguk Cho Yong Min Tai

Samsung Samsung
43 PUBLICATIONS   834 CITATIONS    9 PUBLICATIONS   18 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Research on Multimedia SoC and Platform for Next Generation TV View project

Research on Energy Efficient Smart IoT Devices View project

All content following this page was uploaded by Junguk Cho on 26 September 2018.

The user has requested enhancement of the downloaded file.

 2014 IEEE International Conference on Consumer Electronics (ICCE)
Implementation of an Image Signal Processor for
Reconfigurable Processors
Seung-Hyun Choi1, Junguk Cho2, Yong-Min Tai2 and Seong-Won Lee1, Member, IEEE
1
Kwangwoon University, Seoul, Korea,
2
Samsung Advanced Institute of Technology, Gyeonggi-do, Korea
swlee@kw.ac.kr
Abstract—An image signal processor for a CMOS image
sensor consists of many complicated functions. In this paper, the
full chain of camera ISP functions for smart devices is presented.
The every function in the chain is fully converted to fixed point
arithmetic and no special function is used for easy porting to
reconfigurable processors. To verify the performance of the
proposed ISP chain, series of test are performed, and all
measured values satisfied the quality requirements.
I. INTRODUCTION
Recently, the technology progress of smart devices is
surprisingly fast. Among many performance improvements
and additional special purpose functions, one of the most
commercially successful functions is the digital camera.
Nowadays almost every smart device has the digital camera
feature. Since the environment where the camera with the
smart devices is used is sometimes not quite suitable to get
good quality pictures. Therefore, Bayer pattern image taken
from the CMOS image sensor used in the digital camera
should be converted to the RGB image using many complex
enhancement functions in the ISP (Image Signal Processor).
II. ALGORITHM
In this paper, the full chain of camera ISP functions for
smart devices is presented. The every function of the chain is
fully converted to fixed point arithmetic and no special
function is used for easy porting to reconfigurable processors.
Fig.1 is flow chart of ISP full chain for vector processors.
As in Fig. 1, after performing white balance in the bayer
pattern, Modified AHD(Adaptive Homogeneity-directed
Demosaicing), which is based on AHD[1], is used as
Fig.1 ISP full chain
This work was partly supported by Samsung Advanced Institute of
Technology and the MSIP(Ministry of Science, ICT & Future Planning) ,
Korea, under the ITRC (Information Technology Research Center) support
program supervised by the NIPA (National IT Industry Promotion
Agency)(NIPA-2013-H0301-13-1011), and by the Technology Innovation
Program(10039145, Research of Advanced Power Management technology
for energy-saving smart products) funded by the Ministry of Knowledge
Economy(MKE, Korea).
978-1-4799-1291-9/14/$31.00 ©2014 IEEE 141
demosaicing algorithm. The AHD consist of three steps;
directed interpolation, homogeneity-directed map creation,
iteration. The directed interpolation is made of fixed point
arithmetic with 2bit extension. The coefficient of
homogeneity-directed map creation is converted to fixed
point. Also, CIELab that is used as the color space in the
original homogeneity-directed map is changed to YCbCr color
space. The iteration function is removed for reducing
operational loads. After demosaicing, the color correction
block finds color features and repairs color artifacts.
Linear stretch method is used in the auto contrast. The
linear scale factors in the auto contrast function are calculated
in YPbPr color space. A color control function is also used in
YPbPr color space to control color saturation and color offset.
Gamma Correction with a reduced LUT(LookUp Table) is
used and the gamma values between LUT entries are
calculated by piecewise linear interpolation method. [3]
Modified BF(Bilateral Filter) is used as a noise reduction
algorithm and based on BF[2]. The original BF has two
Gaussian filters. One is for distance weight between pixel
locations. The other is for difference weight between pixel
intensities. In order to simplify the two Gaussian filters, the
Gaussian functions are replaced by fixed point binary
threshold functions. The threshold values are determined by
pre-calculating the Gaussian filter coefficient for pixel
locations and pixel intensities.
For detail enhancement DoG(Difference of Gaussian) based
LTI/CTI[4] is used. The Gaussian mask sizes in the LTI/CTI
are 3x3 and 5x5, which are with pre-calculated coefficients.
TABLE I
CMOS IMAGE SENSOR SPEICIFICATION
spec
Active array size 2592 x 1944
Output formats 10-bit RGB raw
Lens size 1/4 inch
Input clock frequency 6~27 MHz
Max S/N ratio 36dB
 TABLE II
IMAGE QUALITY TEST REQUIREMENTS AND EXPERIMENT RESULTS
Metric MTF30 Over Edge Texture Exposure Gamma SNR
sharpening Roughness Acuity Error

Requirements (20lux) 0.33 ~ 0.8 ” 20% • 0.65 -1.2 ~ 0.5 0.4 ~ 0.9 • 30 dB
Requirements (200lux) 0.35 ~ 0.7 ” 20% ” 0.15 pixel • 0.65 -0.5 ~ 0.5 0.4 ~ 0.7 • 30 dB
Results (20lux) 0.545 0% 0.753 -0.81 0.4 39.23dB
Results (200lux) 0.737 0% 0.062 pixel 0.758 -0.49 0.56 47.03dB
Metric DR Light Falloff Color Uniformity SMIA Delta C00
2900K 3500K 5500K
Requirements (20lux) • 33 dB Mean ” 15 Mean ” 10
Max ” 20 Max ” 15
Requirements (200lux) • 33 dB 70% ” RI ” Max ෙ C ” 10 |SMIA| < Mean ” 10 Mean ” 10
130% 6% Max ” 15 Max ” 15
Results (20lux) 44dB Mean ” 8.1 Mean ” 5.6
Max ” 13.6 Max ” 12.6
Results (200lux) 44dB 72.43% 2.23 -2.55 Mean ” 4.8 Mean ” 4.7
Max ” 14.0 Max ” 14.9

III. EXPERIMENTS TABLE III

OVERALL PERFORMANCE AFTER CGA ACCELERATION
Total cycle 41,080,015 VLIW only cycle 13,865
Software 758 CGRA mapped 41,065,392
pipelinable cycle cycle
Total instructions 323,372,150 Valid instructions 236,116,718
Route 76,990,227 Nullified 10,265,005
instructions instructions
Load instructions 30,796,752 Store instructions 25,664,073
Fig. 3. Color acuity test pattern Spill instructions 995 Load/store spills 262/262
To verify the performance of the proposed ISP chain, series IPC(Effective 7.62(5.75) IPC: VLIW/CGA 0.93/7.62(5.75)
IPC) (Effective)
of test are performed to check if the quality of result images

IV. FUTURE WORKS

Fig. 2. Image resolution test pattern
can pass a commercially arranged test. The experiments are
conducted using a CMOS image sensor whose specification is
shown in table I. There are several test patterns to measure the
image quality properly and Fig. 2 and Fig. 3 are some of the
test patterns. Fig. 2 is image quality resolution test pattern. It
is used to calculate MTF30, over sharpening, etc. Fig. 3 is
color acuity test pattern that is used to evaluate color
performance such as Delta C00.
As shown in Table II, all measured values meet the
requirements of the test. Since whole proposed ISP chain is
designed only with fixed point addition and multiplication, the
proposed ISP chain can be easily ported into any
reconfigurable processor.
Table III is CGA acceleration result for the auto contrast
function. The results show that the algorithm can be easily
converted into the CGA acceleration mode.
Currently the proposed ISP chain is being implemented on a
specific reconfigurable processor that has 8way x 16bit core to
support VLIW and/or CGA(Coarse-Grained Array) units to
verify real-time full HD (1920x1080x30) support with
commercial quality.
REFERENCE
[1] K. Hirakawa and T. W. Parks, “Adaptive homogeneity-directed
demosaicing algorithm,” IEEE Trans. Image Process., vol. 14, no. 3,
pp.360–369, Mar. 2005
[2] C. Tomasi and R. Manduchi, “Bilateral Filtering for Gray and Color
Images”, Proc. 1998 IEEE Int'l Conf. Computer Vision, pp. 839-846,
Jan. 1998
[3] D. Lee, R. Cheung, and J. Villasenor, “A Flexible Architecture for
Precise Gamma Correction,” IEEE Trans. Very Large Scale Integration
(VLSI) Systems, vol. 15, no. 4, pp. 474-478, Apr. 2007.
[4] Yangang Wang, Xi Chen, Hua Han and Silong Peng, ಯ Video
Luminance Transient Improvement Using Difference of Gaussian”, In
Proc. of 15th Asia-Pacific Conference on Communications (APCC
2009), pp. 249 – 253, Oct. 2009.

View publication stats

142