ISSCC2020-05 - Visuals Imagers and ToF Sensors

ISSCC 2020
SESSION 5
Imagers and ToF Sensors
A 240×192-Pixel 10fps 70klux 225m-Range
Automotive LiDAR SoC
Using a 40ch 0.0036mm2 Voltage/Time
Dual-Data-Converter-Based AFE
Satoshi Kondo1, Hiroshi Kubota2, Hisaaki Katagiri2, Yutaka Ota2, Masatoshi Hirono3,
Tuan Thanh Ta1, Hidenori Okuni1, Shinichi Ohtsuka2, Yoshinari Ojima2, Tomohiko Sugimoto2,
Hirotomo Ishii2, Kentaro Yoshioka1, Katsuyuki Kimura2, Akihide Sai1, Nobu Matsumoto1
Toshiba1, Kawasaki, Japan

Toshiba Electronic Devices & Storage2, Kawasaki, Japan
Toshiba3, Yokohama, Japan
© 2020 IEEE 5.1: A 240×192Pixel 10fps 70klux 225m-Range Automotive LiDAR SoC
International Solid-State Circuits Conference Using a 40ch 0.0036mm2 Voltage/Time Dual-Data-Converter-Based AFE 1 of 49
Outline
• Motivation and Target

• LiDAR Fundamentals and Challenges
• Dual-Data Converter(DDC)
• DDC-based AFE Implementation
• Measurement Results
• Conclusion
Motivation
www.tesla.com waymo.com getcuise.com
Tesla / ModelS Waymo / Pacifica General Motors / Cruise
General Benchmark of Depth Sensors

mm-wave
Ultrasonic Camera LiDAR
Radar
Distance    
Image quality    
Environment
robustness    
Automotive LiDAR Requirements
• Highway driving:
– Long range (200m) vehicle detection
Braking distance@120km/h >150m
200m
• Urban area driving:
– High image quality for early and certain pedestrian detection
!! Pedestrian A Pedestrian B
Car C Car B Car A
→Our previous work[ISSCC’18] succeeds 200m long range DM

© 2020 IEEE
International Solid-State Circuits Conference
5.1: A 240×192Pixel 10fps 70klux 225m-Range Automotive LiDAR SoC
Using a 40ch 0.0036mm2 Voltage/Time Dual-Data-Converter-Based AFE DM: distance measurement 4 of 49
Importance of Hi-Resolution LiDAR
Child
Debris
20cm×20cm@100m
LiDAR ⇒0.1deg×0.1deg
resolution is required.
Our Goal: Realization of higher-pixel-resolution LiDAR system

Outline

• Conclusion
Distance Measurement using Optical ToF
• LiDAR detects the distance to object by
calculating ToF
LiDAR System Laser
emitted
Laser
ToF
PD SoC
Light
[Niclass, ISSCC’13] Reflected
ToF
Laser pulse
time Distance to object
ToF Light speed x ToF
PD output =
2
time
D-ToF and I-ToF Overview
D-ToF I-ToF
Modulated ToF
Laser Pulse
ToF
Laser Pulse Reflected
time time
ToF Integrated B
A ToF =
SiPM Output B A
PD Output
time time
 SiPM PDE/gain is high  High resolution

 Low resolution  PD PDE/gain is low
International Solid-State Circuits Conference Using a 40ch 0.0036mm2 Voltage/Time Dual-Data-Converter-Based AFE PDE:Photon Detection Efficiency 8 of 49
Approach for Hi-Res. D-ToF LiDAR
D-ToF LiDAR System
Polygon
Laser
mirror
SiPM SoC
SiPM channels
e.g. 20pix. are acquired in one laser shot
The more pixels at once,

the higher the resolution
Bottleneck: AFE Area
Short-Range(SR) AFE
Time
SR-SiPM
High DR
Data
TIA CFD TDC
Vth
SR SiPM Short-Range(LR) AFE

LR SiPM Long-Range(SR) AFE Long-Range(LR) AFE
SR SiPM Short-Range(LR) AFE Voltage
High PDE
LR-SiPM
LR SiPM Long-Range(SR) AFE Data
TIA ADC
LR SiPM Long-Range(SR) AFE
→Low-Resolution
[Yoshioka, ISSCC’18] A pair of SiPM/AFE is required
Short-Range(SR) AFE
Time
SR-SiPM
High DR
Data
TIA CFD TDC
Vth

High PDE
LR-SiPM
TIA ADC
→Low-Resolution Can be merged by using

[Yoshioka, ISSCC’18]
high PDE/ short dead-time SiPM
Short-Range(SR) AFE
Time
SR-SiPM
High DR
Data
TIA CFD TDC
Vth

High PDE
LR-SiPM
TIA ADC
→Low-Resolution
[Yoshioka, ISSCC’18] Not Scalable
Solution: LR/SR-Unified AFE
SR SiPM Short-Range(LR) AFE SiPM LR/SR-unified AFE

LR SiPM Long-Range(SR) AFE SiPM LR/SR-unified AFE
→Low-Resolution
[Yoshioka, ISSCC’18] →2x Hi-Resolution!!
Solution: LR/SR-Unified AFE
SR
AFE
LR/SR-
SR SiPM
unified
Short-Range(LR) AFE SiPM LR/SR-unified AFE
LRLR SiPM Long-Range(SR) AFE SiPM LR/SR-unified AFE
AFE
LR SiPM
SR SiPM
How?
Long-Range(SR) AFE
Short-Range(LR) AFE
SiPM
SiPM
LR/SR-unified AFE
LR/SR-unified AFE
→Low-Resolution
[Yoshioka, ISSCC’18] →2x Hi-Resolution!!
Outline

• Conclusion
Concept of Dual-Data Conversion
• Simultaneous conversions(Voltage: ADC/

Time: TDC) in a single circuit are realized by
differentiating/ integrating VCO pulse counts
TIA VCO
Diff. Voltage Data
counter
SiPM
Integ. Time Data
Concept of ADC Operation
• Digitize input voltage swing from VCO freq. changes
Detect VCO freq. by differentiating VCO pulses
Laser
Concept of TDC Operation
• ToF is detected from VCO phase information
obtained by integrating VCO pulse counts.
TIA VCO
Diff. Voltage Data
counter
SiPM
Integ. Time Data

Trigger
TIA Output Vth
VCO Integ.
Pulse
How To Realize DDC?
There are TWO issues to realize DDC

TDC Issue: Freq. Modulation and Drifts
• VCO is modulated by input signal for ADC operation
TIA VCO
Diff. Voltage Data
counter
SiPM
Integ. Time Data

Laser Trigger
TIA Output Vth
VCO Integ.
Frequency modulation/drifts
Pulse
VCO frequency modulation/drifts causes serious DM error
ADC Issue: VCO Non-Ideality
• VCO frequency versus input voltage is highly non-
linear.
TIA VCO
Ch-1 Voltage
SiPM
fVCO Diff.
Data
TIA VCO
Ch-2 Voltage
SiPM
Diff.
Data
・・・
VCO
Vin TIA
Ch-40 Voltage
SiPM
Diff.
Data
Non-linearity Mismatch
VCO non-ideality degrades ADC SNDR and image quality

Outline

• Conclusion
DDC Implementation Issues and Solutions
TDC Issue :VCO Freq. Modulation and Drifts

→Subranging
ADC Issue :VCO Non-Ideality
→Multiphase-PDM-based feedback
TDC Issue: Freq. Modulation and Drifts
• VCO is modulated by input signal for ADC
operation
TIA VCO
Diff. Voltage Data
counter
SiPM
Integ. Time Data
Laser Trigger
Vth
VCO Frequency modulation, drifts

Pulse
Solution: Subranging
CLK
Coarse ToF
TIA VCO counter Integ.
SiPM
counter Integ. Time Data

Fine ToF
Subranging: Coarse ToF
CLK
Coarse ToF
SiPM

Fine ToF
Laser
Trigger
VCO Vth
Pulse Coarse ToF: 4
VCO freq. ignored
CLK
Calculate coarse ToF by integrating 400MHz clock
© 2020 IEEE
Subranging: Fine ToF
CLK
Coarse ToF
SiPM

Fine ToF
Laser
Trigger
VCO Vth
Pulse Coarse ToF: 4 15

Voltage Data
9 Fine ToF: 9/15=0.6
CLK
Calculate Fine ToF by integrating VCO pulse count between
© 2020 IEEE 400MHz CLK and Trigger
Using a 40ch 0.0036mm2 Voltage/Time Dual-Data-Converter-Based AFE 27 of 49
Subranging: Fine ToF
CLK
Coarse ToF
SiPM

Fine ToF
Laser
Trigger
VCO Vth
Pulse Coarse ToF: 4 15

Voltage Data
9 Fine ToF: 9/15=0.6
CLK
© 2020 IEEE
Eliminates DM error degradation by modulation/drifts
DDC Implementation Issues and Solutions

→
Conventional Technique
LUT-based DAC-based
VCO LUT
Sampler
Test
1-z-1
Voltage Voltage
signal Data Vin OPA
Data
CLK Calibrated.
Cal. must be conducted to ch.-by-ch.
1-z -1
LUT
Ch-1 DAC
Data
Ch-2
Test 1-z-1 LUT
signal
Data DAC mismatch
Ch-40
1-z-1 LUT
Data
 Cal.-free or simple cal.
 Small area  Bulky, performance degrades
 Long cal. time for 40ch by DAC mismatch
Conventional Technique
LUT-based DAC-based
VCO LUT
Sampler
Test
1-z-1
Voltage Voltage
signal Data Vin OPA
Data
CLK Calibrated.
Cal. must be conducted to ch.-by-ch.

Not suitable for Hi-Res.
1-z -1
LUT
Ch-1 DAC
LiDAR;
Data
Test
signal
Requires novel solutionDAC mismatch
1-z-1 LUT
Ch-2
Data
Ch-40
1-z-1 LUT
Data
 Cal.-free or simple cal.
 Small area  Bulky, performance degrades
 Long cal. time for 40ch by DAC mismatch
Our Approach: PDM-based Feedback
• Pulse density of 1-b cell is modulated by VCO freq.

PDM feedback can reduce Kvco non-linearity by
using mostly-digital circuits only.
VCO VCO Frequency
Voltage
Logic
Vin
Data
PDM Output
Fixed
Pulse-Density Modulator(PDM)
Our Approach: PDM-based Feedback
• Pulse density of 1-b cell is modulated by VCO freq.

PDM feedback can reduce Kvco non-linearity by
using mostly-digital circuits only.
FFT of PDM Output
VCO
Voltage
Logic
Vin
Data Large spur!!
Fixed
Pulse-Density Modulator(PDM) fsignal fVCO

Proposed Multiphase-PDM
• Utilize 8-phases from 4-divided VCO differential output
Singlephase(SP)-PDM Multiphase(MP)-PDM
x8 IPDM
IPDM
VCO_P
fVCO Fixed
÷4
Fixed
VCO_N
Pulser Pulser
IP1
fVCO ITIA
IP1
IP2
IPDM Sum
…
IP8
IPDM
• Utilize 8-phases from 4-divided VCO differential output
SP-PDM MP-PDM
x8 IPDM
IPDM
VCO_P
fVCO Fixed
÷4
Fixed
VCO_N
Pulser Pulser
IP1
fVCO 1x ITIA
4x
IP1
IPDM IP2 1/8x Sum
…
IP8
I
• Switching spur magnitude is reduced by 1/8x PDM
© 2020 IEEE
• Settling time can be relaxed by 4x
• VCO filtering function further suppresses aliasing noise
caused by spur VCO filtering (freq.-to-phase)
Voltage
Logic
Vin
Data
MP-PDM
FFT of PDM Outputs After VCO filtering
freq.- High-freq. spur

to-phase
SP-PDM 1/8x switching spur effectively suppressed
MP-PDM
fsignal fVCO 2fVCO

fsignal fVCO 2fVCO
2x switching freq.
Efficacy of MP-PDM: Linearity and Area
fs=400MS/s VCO Logic
fin=9.8MHz
BW=100MHz
SFDR: 50um
De-cap
MP-PDM
49.8dB
SNDR:
39.3dB 60um
1
Area [mm2]
0.1
0.01 [3]
ISSCC '97-'19
SNDR>35dB
BW=50-400MHz
0.001
0.0001 0.001 0.01 0.1 1
Power [W]
Breaks the tradeoff between area and performance
Efficacy of MP-PDM: SNDR Variation（40ch）
42
Average=39.3dB
1σ=0.36dB
SNDR [dB]
40
38
Small variation across all ch.
36
0 10 20 30 40
Channel #
ADC SNDR variations are minimized without
© 2020 IEEE
any bulky circuits and calibrations
DDC Imp. Issues and Solution: Summary

→Subranging
Outline

• Conclusion
DDC-based SoC for Hi-Res LiDAR
• 28nm CMOS
• 40ch AFE
• 2ch for test
Measurement Setup
70klux sunlight
Effect of 2x Higher Pixel Resolution
70klux sunlight
10% reflection@75m-away
7.5cm
100
90%
Success rate
success
50
[% ]
240x96-pix 240x192-pix
0
50 40 30 20 10 0
Target Height [cm]
Hi-Res. LiDAR detects 2x smaller object

Distance Measurement Results
70klux sunlight
100 1
Success rate [% ]
Standard deviation [m]

90% success 0.8
90
80 100%@225m 0.6
70 0.4
60 0.2
50 0
0 50 100 150 200 250 300 0 50 100 150 200 250 300
Distance [m] Distance [m]
• 225m Long Range with <0.25%-precision.

*Success
© 2020 IEEE
determined
International Solid-State as DM within <±1% error
Circuits Conference 2
Using a 40ch 0.0036mm Voltage/Time Dual-Data-Converter-Based AFE 44 of 49
Comparison between SR and LR
70klux sunlight
160
Standard deviation
140 LR DM, 60% reflect.
120 SR DM, 60% reflect.
100 LR DM, 10% reflect.
SR DM, 10% reflect.
80
[mm]
60
40
20
0
0 5 10 15 20 25
Distance [m]
• Higher precision by SR-DM in <15m.

*Success
© 2020 IEEE
determined
International Solid-State as DM within <±1% error
Circuits Conference 2
Using a 40ch 0.0036mm Voltage/Time Dual-Data-Converter-Based AFE 45 of 49
SR/LR Composite Point Cloud
Daytime 192-Vertical-Pix Image From a 10-FPS Movie
Train Traffic sign

Traffic sign
Pedestrian
Truck
LR DM
SR DM Pedestrian
Signal Intensity Image
Benchmarks
[3] [2]
This work ISSCC'18 JSSC'14
Technology 28nm 28nm 180nm HV
AFE Channels 40 20 16
Pixel-Resolution 240x192 240x96 202x96
Normalized
Pixel-Resolution*
2x 1x 0.9x
Laser Wavelength [nm] 905 905 870
Laser power [mW] 50 50 21
FPS 10 10 10
Distance range [m] 225 200 100

1σ error @max distance 0.25% 0.125% 0.14%
AFE Power Consumption [mW/ch] 10 39 N.A.
AFE area(1ch) [mm2] 0.027 0.134** 0.144**

SoC Area [mm2] 12.5 12.5 31.4
Conclusion
• World’s-first DDC, consolidating the functions of ADC and TDC

into a single circuitry, reduces AFE area 5x smaller.
– Subranging to desensitize VCO frequency modulation and drifts.
– MP-PDM to enhance area efficiency while maintaining ADC performance.
• LiDAR system using a 40-channel DDC-based SoC achieved 2x

higher pixel-resolution and 225m DM.
Demo Session
• High-Res. LiDAR demo!
– 5-7 PM tonight @ Golden Gate Hall
A 1200×900 6μm 450fps Geiger-Mode
Vertical Avalanche Photodiodes CMOS Image Sensor
for a 250m Time-of-Flight Ranging System
Using Direct-Indirect-Mixed Frame Synthesis
with Configurable-Depth-Resolution Down to 10cm
Toru Okino, Shota Yamada, Yusuke Sakata, Shigetaka Kasuga, Masato Takemoto,
Yugo Nose, Hiroshi Koshida, Masaki Tamaru, Yuki Sugiura, Shigeru Saito, Shinzo Koyama,
Mitsuyoshi Mori, Yutaka Hirose, Masayuki Sawada, Akihiro Odagawa, Tsuyoshi Tanaka
Panasonic Corporation, Japan
© 2020 IEEE 5.2: A 1200×900 6μm 450fps Geiger-Mode Vertical Avalanche Photodiodes CMOS Image Sensor for a 250m Time-of-Flight Ranging System Using
International Solid-State Circuits Conference Direct-Indirect-Mixed Frame Synthesis with Configurable-Depth-Resolution Down to 10cm 1 of 25
Outline
1. Motivation
2. Approach
- Phase difference operation into Geiger-mode operation
3. Architecture
- System diagram / Core circuits
- Charge Packet Spatula (CPS) and Photon-Counts-Equalizer (PCE)
4. Realization of finer depth resolution
5. Summary
Motivation and Issue
◆ Need to realize finer and configurable resolution.
Normal image Low depth resolution SRS image
Motivation and Issue
◆ Need to realize finer and configurable resolution.
Normal image High depth resolution SRS image
Principle of Direct-ToF Sub-Range-Synthesis (SRS)
◆ Depth resolution is limited by the pulse width.
∆t c . ∆t
∆z ≥
Light Source 2 ∆z ≥ 1.5m
c = 3.0×108m/s
∆t = 10ns
ToF Sensor
∆z
SR1 SR2 SR3 SR4 SR5
FAR
NEAR
Depth Map Sub-range (SR) images
Phase differentiation operation
Sub-range(SR) SR1 SR2 SR1 SR2

d ∆d
Object
d d
Reflected pulse
t t
Exposure
t t
S1 S1 S2
∆d =d x {(S1 – S2)/(S1 + S2)} x 1/2

Approach: Phase differentiation in photon gating
Sub-range(SR) SR1 SR2 SR1 SR2

d ∆d
Object
d d
Gated pulse
.. t .. t
. .
t' Photon accumulation t'
Photon Count
t t
S1 S1 S2
∆d =d x {(S1 – S2)/(S1 + S2)} x 1/2

Block diagram of APD-ToF system
Pixel Circuits Peripheral FPGA
Circuits
Photon-Counts-
VAPD Charge In-pixel Equalizer (PCE)
 Fast C-Quenching Packet Charge  Auto-zeroing
ADC
 No-after-pulse Spatula Accumulator LVDS  FPN Remove
 High Resolution (1M) (CPS) (ICA)  Linearizer
 DR-Extension
Depth Mapping Engine

Sub-Range
Global Clock Sub-Range Coding
Synthesizer
Generator (SRC)
(SRS)
Photon Imaging Engine

Photon Count
Photon Pulse Frame- Coding
Imager
Generator (FRC)
(PCI)
Circuit diagram of VAPD-CIS
Pixel Circuits Peripheral
Circuits
VAPD Charge In-pixel
 Fast C-Quenching Packet Charge
ADC
 No-after-pulse Spatula Accumulator
 High Resolution (1M) (CPS) (ICA)
PIXEL COLAMP CDS Single Slope-ADC

RSD
CMPINIT
RST VCDS
TRN COMPARATOR
SF ADINIT
FD COUNTER
AMP
CNT Q0~Q6
SEL BUF
VAPD
ICA
VSUB RAMP
Input
・Non-linearity
・Avalanche Noise
・Vt variation
Pixel operation of photon counting
◆ Non-linearity due to charge sharing operation.
CNT (V)
RSD RST CNT
RST
TRN SF
FD
SEL
VAPD
|VICA| (V)
CNT
ICA
VSUB
FD ICA Time
CNT (V)
RSD RST CNT
RST
TRN SF
FD
SEL
VAPD
|VICA| (V)
CNT
ICA
VSUB V1 V1
FD ICA Time
CNT (V)
RSD RST CNT
RST
TRN SF
FD
SEL
VAPD
|VICA| (V)
CNT
ICA
VSUB
FD ICA Time
CNT (V)
RSD RST CNT
RST
TRN SF
FD
SEL
VAPD
|VICA| (V)
CNT
ICA
VSUB V2 V2
FD ICA Time
CNT (V)
RSD RST CNT
RST
TRN SF
FD
SEL
VAPD
|VICA| (V)
CNT
ICA
VSUB V3 V3
FD ICA Time
Charge packet spatula (CPS)
Operation Results
(Before and after CPS)
1.2
FD ICA FD ICA
RST CNT RST CNT 1 Before:σ~120mV
QAN* “half- 0.8
Output (V)
Q∆Vt ON”
0.6
Qsig QAN 0.4

After:σ'~12mV
AN*: Avalanche Noise 0.2
Qsig+QAN+Q∆Vt in FD Spatula Operation 0
0 2 4 6 8 10 12 14 16
Counts
Effect of Vt variance
◆Vt variance of RST: Output voltage vs. counts of each pixel.
～
～
Photon counts equalizer (PCE)
◆ Fixed pattern noise due to Vt variation is removed.
～
～
～
～
0 0
Chip photograph of VAPD CMOS image sensor
HSR
SS-ADC
HSR
COLAMP+CDS
COLAMP+CDS
Vertical
Vertical
circuits
circuits
Pixel Area
circuits
Vertical
Pixel Area
1200 x400
900 Process: 65nm CMOS
x 400
Chip size: 11.3 mm x 12.0 mm
COLAMP+CDS
SS-ADC
COLAMP+CDS
HSR HSR
Indirect-ToF operation and verification
5.6m 6.6m
10cm 10cm
#10
#7 #9
#4 #5#6 #8
#3
#1 #2
Measured distance (cm)
660
640
620
600 Std.Dev.:<7.4cm
Error:<1.4cm
580
560
#1 #2 #3 #4 #5 #6 #7 #8 #9 #10
Position
Geiger-mode Direct-ToF-Indirect-ToF mixed mode vs.
standard Geiger-mode Direct-ToF mode (long range)
Geiger-mode Direct-ToF image Geiger-mode Direct-ToF-Indirect-ToF Mixed image
250m 250m
200m 200m
150m 150m
100m 100m
50m 50m
30m 30m
Geiger-mode Direct-ToF-Indirect-ToF mixed mode vs.
standard Geiger-mode Direct-ToF mode (near range)
Real scene image
Geiger-mode Direct-ToF image Geiger-mode Direct-ToF-Indirect-ToF Mixed image

58.5m 58.5m
51.0m 51.0m
43.5m 43.5m
36.0m 36.0m
28.5m 28.5m
21.0m 21.0m
13.5m 13.5m
12.0m 12.0m
10.5m 10.5m
9.0m 9.0m
7.5m 7.5m
6.0m 6.0m
4.5m 4.5m
3.0m 3.0m
1.5m 1.5m
0.0m 0.0m
Chip specification and comparison table
This Work [1] [2] [3] [4] [5]
CMOS Technology 65nm 65nm 110nm 45nm/65nm 150nm 110nm
Pixel Size 6μm 7μm 32μm 19.88μm 60μm 11.2μm
2D Resolution 1200x900 640x480 64x64 256x256 64x64 688x384
Standard-
Pixel Type VAPD (=SPAD) APD SPAD SPAD VAPD (=SPAD)
PD
APD Mode Geiger N.A. Linear Geiger Geiger Geiger
TOF Type Direct + Indirect Indirect Indirect Direct Direct Direct
Direct: Photon Counting 4-tap phase APD-Gain Time-to-Digital- Time-to-Digital- Photon
Ranging Scheme
Indirect: Phase Difference sampling Modulation Converter Converter Counting
Maximum Range 250m 4m 20m 430m 6000m 220m
Depth Resolution@
1.5m 2.5cm 56cm 80cm 35m 1.5m
Max. Range
Depth Resolution@
10cm 0.2cm 1.7cm 15cm 50cm 1.5m
Min. Range
2D Frame Rate 450fps 60fps 96fps N.A. 7.16fps 60fps
Summary
1. A D-ToF/I-ToF mixed Geiger-mode vertical avalanche photodiode (VAPD)
CMOS imager is demonstrated.
2. # of pixels: 1M Pixels
Speed: 30fps/FR (3D), 450fps/SR (2D)
3. Depth resolution is configurable.

D-ToF ∆Z：1.5m～7.5m（Long range～250m）
I-ToF ∆Z：1cm～10cm（Short range～20m）
4. Key enablers are (i) Charge packet spatula (CPS), (ii) Photon-Counts-
Equalizer (PCE), and (iii) Phase detection operation.
Introduction of our demo session
We will present a VAPD-ToF system in the DS1, 5:00-7:00 PM.
 Real-time D-ToF-I-ToF mixed imaging by developed VAPD-ToF camera.
 A video display of Introduction to the concept of VAPD-ToF system.
 A demonstration video of ranging various scenes.
Real Time Concept- Various-

VAPD-ToF Movie Scenes
VAPD-ToF camera
Thanks for your attention.
An Up-to-1400nm 500MHz Demodulated Time-of-
Flight Image Sensor on a Ge-on-Si Platform
C.-L. Chen*, S.-W. Chu*, B.-J. Chen*, Y.-F. Lyu*, K.-C. Hsu*, C.-F. Liang*, S.-S. Su,
M.-J. Yang, C.-Y. Chen, S.-L. Cheng, H.-D. Liu, C.-T. Lin, K. P. Petrov, H.-W. Chen,
K.-C. Chu, P.-C. Wu, P.-T. Huang, N. Na, S.-L. Chen
Artilux, Hsinchu, Taiwan
*Equally-Credited Authors (ECAs)
© 2020 IEEE 5.3: An Up-to-1400nm 500MHz Demodulated Time-of-Flight Image Sensor on a Ge-on-Si Platform
International Solid-State Circuits Conference 1 of 17
Outline
• Introduction to iToF image sensor

• Ge-on-Si platform
• System architecture
• Circuit details
• Measurement results
• Summary and Conclusion
Introduction
• Indirect Time-of-Flight image sensor is gaining interest in several applications
• Issues: Outdoor SNR is degraded => laser power increased => eye safety
3D Facial Recognition
Autonomous Vehicle
Introduction
• Eye-safety and Wavelength
We would like to apply higher laser

power for better outdoor SNR but this
results in eye safety issue
Long wavelengths enable :
• Ambient light and Wavelength
(i) better eye-safety
(ii) avoiding sunlight interference.
How to implement a long-wavelength

iToF image sensor? => Ge-on-Si platform
Ge-on-Si platform
• Why Ge?
• Ge offers better quantum efficiency and higher demodulation frequency
• The absorption depth is very thin => easier to control the flow of photo charges
• The use of Ge make it possible to integrate this platform on 12-inch wafers
TOP
(Ge-on-Si)
BOT
(Si)
System architecture
• The first prototype with HQVGA resolution (Sensor IC level)
System architecture
• System blocks of the overall 3D-sensing platform (Module/SDK Level)
3D SENSING ENGINE (HW)
Proposed Sensor Reflected light

SENSING
ARRAYS
OPTICS
CONTROL IC
Emitted light
LASERS
3D MODEL RECONSTRUCTION
ALGORITHM (SW)
Circuit details
• Differential 4Tx2 pixel architecture
TOP
• Adjustable integration capacitors
• CLKP/CLKN are routed to the TOP

wafer outside the pixel region for BOT
possible pixel shrink in the future
• Differential and 4-quad demodulation
Circuit details
• Column demodulation CLK driver using tapered pseudo differential inverters
• To alleviate the switching transient, large on-chip decoupling caps are used
• Driver LDOs are further decomposed into 4 groups for better start-up control
BOT
TOP
BOT
500MHz, SS125
Circuit details
• SAR ADC with interleaved frontend
• 0.5bit more ENOB can be achieved
• Programmable delay for optimized
accuracy
• Max 1.4x improvement in accuracy
• Note without the high-bandwidth iToF
pixels on the Ge-on-Si platform, the
improvement will be trivial
Die Photo
• 240x180 iToF pixel array

• 940nm ulens
• Fill factor > 95%
• Backside illumination
• 250mW from 3.3V/1.8V supply
Measurement Result
• Pixel wafer level testing results => multiple splits
• QE approaching 50% at 1550nm for the latest condition
• Considering the reflectivity of different material, wavelength < 1400nm is used for sensor module
1400nm
QE at 1550 nm 40-50%
Dry
wet
37 dies with at least 3 repeats

measured for each wafer
Measurement Result
• 3D point cloud image at 940nm
• Halogen lamps to mimic 100klux daylight
• 10klux halogen lamps = 100klux daylight (940nm)
Measurement Setup
Measurement Result
• Demonstration of 500MHz demodulation capability
• Unambiguous range of iToF system = C/2/Fdemod and it’s 30cm for Fdemod = 500MHz
Same Depth Color

OBJ2
30cm
OBJ1 OBJ1
OBJ2
Measurement Result
• In the digest, we showed 3D point cloud images at 1310nm with 940nm camera lens
• Customized 1310nm camera lens is built recently, the image quality is greatly improved.
Comparison Table
ISSCC’18 [1] VLSI’17[2] VLSI’19[3] This Work
Pixel type Si Si Si Ge-on-Si
Pixel pitch (um) 3.5 10 7 10
Wavelength (nm)
860 850 940 850-1400
940nm 1310nm
Responsivity (A/W) 0.305 0.343 0.227
> 0.49* > 0.54*
Resolution 1024x1024 320x240 640x480 240x180
Modulation Frequency 10-320MHz < 100MHz 10-150MHz 10-500MHz
0.7-0.9 @ 500MHz
Demod. Contrast 0.87@200MHz 0.85@100MHz 0.86@100MHz
depends on setting
Indoor Outdoor
Std. @ 1m (%) < 0.2 0.6 0.5 940nm 940nm
< 0.5 < 0.55
*: Post microlens, microlens efficiency optimized at 940nm
Summary and Conclusion
• A wideband iToF image sensor is proposed on the Ge-on-Si

platform
• Thanks to the use of Ge, high SNR and eye-safe operation
are achieved simultaneously
• Please come to our demo session tonight (5-7pm)
Thank You!!
A Dynamic Pseudo 4-Tap CMOS Time-of-Flight
Image Sensor with Motion Artifact Suppression and
Background Light Cancelling Over 120klux
D. Kim1, S. Lee2, D. Park2, C. Piao1, J. Park1, Y. Ahn1, K. Cho1,

J. Shin3, S. M. Song3, S.-J. Kim2, J.-H. Chun1, J. Choi1
1Sungkyunkwan University, Suwon, Korea

2Ulsan National Institute of Science and Technology, Ulsan, Korea
3Zeean, Hanam, Korea
© 2020 IEEE 5.4: A Dynamic Pseudo 4-Tap CMOS Time-of-Flight Image Sensor
International Solid-State Circuits Conference with Motion Artifact Suppression and Background Light Cancelling Over 120klux 1 of 69
Outline
• Motivation & Background of Research

• Proposed Indirect Time-of-Flight (iToF) CMOS Image Sensor
1) Pixel Array with Trident PPD Structure
2) Dynamic Pseudo 4-tap Scheme
3) ∆Σ Background Light Cancelling Scheme
• Experimental Results
• Summary
Motivation
Moving Object Background Light

(BGL)
Motion Artifact Depth Error

Conventional CMOS iToF sensor has two major problems
Motivation & Basic operation

(BGL)
Background Overview : Basic Operation of IToF
Depth (D)
Sensor
Light Intensity
Emitted Received
Controller
Q0 Qπ/2 Qπ Q3π/2 Q0
t
Emitter
According to the distance

Depth (D)
Sensor
Light Intensity
Emitted Received
Controller
t
Emitter
To calculate the depth,

acquire 4 phase signals
2-tap pixel
PC#1 PC#2
*PC : Pixel circuit unit
Frame 1 Q0 & Qπ (2 Phase)

Frame 2 Qπ/2 & Q3π/2 (2 Phase)
We need 2 frame signals,

to acquire 4 phase signals
Background Overview : Moving Object
2-tap pixel
PC#1 PC#2

PC#1,2
Moving object
is captured
Frame 1
(Q0−Qπ)
2-tap pixel
PC#1 PC#2

PC#1,2 PC#1,2
Positions of the fan
are different
Frame 1 Frame 2
(Q0−Qπ) (Qπ/2 −Q3π/2)
2-tap pixel
PC#1 PC#2

PC#1,2 PC#1,2 Error
Far
Close
Frame 1 Frame 2 Depth
Calculated
(Q0−Qπ) (Qπ/2 −Q3π/2) image
© 2020 IEEE
5.4: A Dynamic Pseudo 4-Tap CMOS Time-of-Flight Image Sensor
with Motion Artifact Suppression and Background Light Cancelling Over 120klux
by 2 frames 10 of 69
2-tap pixel 4-tap pixel
PC#1 PC#2
PC#1 PC#2
8 tr 16 tr
PC#3 PC#4
*PC : Pixel circuit unit Low fill factor

PC#1,2 PC#1,2 Error Low SNR PC#1,2 PC#2,3
Frame 1 Frame 2 Depth Depth Frame 1 Frame 1

(Q0−Qπ) (Qπ/2 −Q3π/2) image image (Q0−Qπ) (Q0−Qπ)
2-tap pixel 4-tap pixel
PC#1 PC#2
PC#1 PC#2
8 tr 16 tr
PC#3 PC#4
*PC : Pixel circuit unit Low fill factor

PC#1,2 PC#1,2 Error
Solution Low SNR PC#1,2 PC#2,3
Pseudo
4-tap
scheme
Frame 1 Frame 2 Depth Depth Frame 1 Frame 1
(Q0−Qπ) (Qπ/2 −Q3π/2) image image (Q0−Qπ) (Q0−Qπ)
Motivation

(BGL)
Background Overview : BGL
Depth (D)
Light Intensity
Sensor Emitted Received
Controller t
Integration time : T ms
Emitter
FD1
FD2
Qπ/2 Q3π/2
Background Overview : BGL
Depth (D) Emitted Received
Light Intensity
Sensor
QBGL
Controller t
Integration time : T ms
Emitter
Qπ/2 Q3π/2
FD1
FD2
Saturation !!
Background Overview : BGL Cancelling
T ms
T/2 ms
Light Intensity (Decreasing)
Q0 Qπ/2 Qπ Q3π/2 Q0 Qπ/2 Qπ Q3π/2 Q0
t
Integration time : T ms Integration time : T/2 ms
Qπ/2 Q3π/2
FD1
FD2
FD1
FD2
Qπ/2 Q3π/2
Saturation !! Avoid saturation, signal degradation!!

T/2 ms = Sub-int #1 Emitted Received
Light Intensity
QBGL
Qπ/2 Q3π/2 ∆Q
FD1 FD2 Analog Q0 Qπ/2 Qπ Q3π/2 Q0
Memory QBGL
Accumulate
(AM) t
To maintain a signal charge,

We have to use ∆Σ operation
T/2 ms = Sub-int #1 Emitted Received
Light Intensity
QBGL
Qπ/2 Q3π/2 ∆Q
FD1 FD2 Analog Q0 Qπ/2 Qπ Q3π/2 Q0
Memory QBGL
Accumulate
(AM) t
T/2 ms = Sub-int #2
Total integration time is T
∆Q Qπ/2 Q3π/2 2∆Q maintain a signal charge
Sub-int #1 AM
T/N ms = Sub-int #1
1 time + − = ∆Q
Qπ/2 Q3π/2
AM
To cancel the
strong sun light
T/N ms = Sub-int #N
(N-1) N
N times ·∆Q without
+
Qπ/2 − Q3π/2 = ·∆Q
QBLG
Σ ∆ AM
Solution → ∆Σ background light cancelling

Outline

• Summary
Proposed iToF CMOS Image Sensor
Even/Odd frame
320 x 240 Over-pixel Analog mem. ISP Main Features
Alternate PHase Driver (APHD)
0&π
π/2&3π/2 VGA upscaler
1. Pixel Array with
Potential (V)
TX2
0&π
π/2&3π/2
Odd
Trident PPD Structure
TX1
Pseudo
...
FD2
0&π Even 4-tap FD1

(P4T) 320 x 240
π/2&3π/2
BSI Field- P4T depth 2. Dynamic
0&π accelerated interpolator Pseudo 4-tap Scheme
π/2&3π/2 Pixels
10-80 MHz Σ
3. ∆Σ BGL Cancelling
∆
PLL
TS1 TS2 TS3 TS4 Denoising (BGLC) Scheme
∆-Σ Background Light Suppression
Bias/DAC ∆Q0, ∆Qπ/2
10b single-slope ADC
Concept of High-Speed Demodulation
Pinned photodiode Storage 1
Potential (V) (PPD) Storage 2
Conventional
Flat E-potential
 Slow transfer
Storage 1
PPD
Storage 2
Potential (V)
Sloped E-potential Proposed

 Fast transfer
Apply
Potential gradient
Conventional PPD
FD1
TX1
X1 X2
TX2
n-
FD2
X1-X2 Distance
Potential (V)
Flat
Pinning potential
Wide
Use 3 solutions to accelerate
© 2020 IEEE
charge transfer
with Motion Artifact Suppression and Background Light Cancelling Over 120klux 23 of 69
Solution : 1. PPD Size Optimization
FD1
TX1
X1 X2
Shrink TX2
n-
Dimension
FD2
X1-X2 Distance
Potential (V)
Flat
Shrink
Shrink the electron moving path
Solution : 2. Adding Secondary n Layer
FD1
TX1
The doping
X1 X2
concentration
n TX2
2ndn > n- n-
FD2
X1-X2 Distance
Potential (V)
Flat By adding 2ndn layer
Potential Potential gradient
gradient around TX
Solution : 3. PPD Shaping
FD1
TX1
Implant
X1 X2
Trident-shaped n-
n TX2
n-
FD2
X1-X2 Distance
Potential (V)
Potential
The e-potential in all
gradient regions has a slope !!
Solution : 3. PPD Shaping
X1-X2 Distance
Potential (V)
YA1 YB1
FD1
XA XB
TX1
XA
X1 X2
Potential (V)
XB n TX2
n-
FD2 YA1-YA2 Distance
YA2 YB2 YB1-YB2 Distance
Trident shaping generates three high potential wells

Trident shape PPD : TCAD Simulation Results
X-Axis (μm)
0.0 1.0 2.0 3.0 4.0 Y1-Y2 Distance (μm)
Y1
0.5 1.0 1.5 2.0 2.5
TX1 = 0.0
Potential (V)
‘OFF’
3.0
Y-Axis (μm)
0.1
0.2
TX2 =
1.0
‘ON’ 0.3
Electron Trajectory
Y2
We have three high-potential wells in the direction of Y
Trident shape PPD : TCAD Simulation Results
X-Axis (μm) X1-X2 Distance (μm)

0.0 1.0 2.0 3.0 4.0 0.5 1.0 1.5 2.0 2.5
TX1 =
Potential (V)
‘OFF’ 0.4
3.0
Y-Axis (μm)
X1 X2 0.8
TX2 =
1.0
‘ON’ 1.2 Shaping Adding n

Electron Trajectory Sizing
All regions of PPD have E-potential gradient in the direction X

Pixel with Trident PPD Structure
R TX1 TX2 R
TX1 TX2
S S
8 μm
TX1 TX2
Column 1
Column 2
TX1 TX2
8 μm
Charge summing based on 4-shared pixel structure

For high responsivity
Dynamic Pseudo 4-tap Scheme
Even/Odd frame
320 x 240 Over-pixel Analog mem. ISP
0&π 2. Dynamic
π/2&3π/2 VGA upscaler Pseudo 4-tap Scheme
Potential (V)
TX2
0&π
π/2&3π/2
Odd TX1
Pseudo
...
FD2
0&π Even 4-tap FD1

(P4T) 320 x 240
π/2&3π/2
BSI Field- P4T depth Off-chip processor
0&π accelerated interpolator
π/2&3π/2 Pixels Hybrid depth imaging
10-80 MHz Σ Depth Depth

∆
(P4T) (2T)
PLL
TS1 TS2 TS3 TS4 Denoising
∆-Σ Background Light Suppression Hybrid
Bias/DAC ∆Q0, ∆Qπ/2 depth
Conventional 2-tap Operation
Frame 1 Frame 2
TX LD TX
LD Pixel Pixel Depth (2T)
Driver Driver
Even
∆Q0 ∆Qπ/2 ∆Q0 ∆Qπ/2

[0] [0] [0] [0]
0 ∆Q0 π/2 ∆Qπ/2 ∆Q0 ∆Qπ/2

Odd
π [1] 3π/2 [1] [1] [1]

2 phases 2 phases
Even
∆Q0 ∆Qπ/2 ∆Q0 ∆Qπ/2

[2] [2] [2] [2]
Depth is calculated by 2 frames

© 2020 IEEE
This is the reason of motion artifact!!
Pseudo 4-tap Operation
Frame 1
LD TX Divide all of phases into

Pixel Depth (P4T)
Driver
even and odd rows
0
Even
∆Q0 ∆Q0
π [0] [0]
? Even
π/2 ∆Qπ/2 ∆Qπ/2

Odd
3π/2 [1] [1] ? Odd

Even
0 ∆Q0 ∆Q0
π [2] [2] ? Even
We can acquire ∆Q0 & ∆Qπ/2 in 1 frame

Pseudo 4-tap Operation
Frame 1
LD TX
Pixel Depth (P4T)
Driver
0
Even
∆Q0 ∆Q0 ∆Qπ/2

π [0] [0] Intp.
π/2 ∆Qπ/2 ∆Qπ/2 ∆Q0 The vacant data

Odd
3π/2 [1] [1] Intp. interpolated
∆Q0 ∆Qπ/2
Even
0 ∆Q0
π [2] [2] Intp.
Depth calculated by 1 frame

© 2020 IEEE
Interpolation method has a low depth accuracy.
Alternate Phased Operation
Frame 1 Frame 2
TX LD TX
LD Pixel Pixel Depth (2T)
Driver Driver
0
Even
∆Q0 π/2 ∆Qπ/2 ∆Q0 ∆Qπ/2

π [0] 3π/2 [0] [0] [0]
π/2 ∆Qπ/2 0 ∆Q0 ∆Q0 ∆Qπ/2

Odd
3π/2 [1] π [1] [1] [1]

Even
0 ∆Q0 π/2 ∆Qπ/2 ∆Q0 ∆Qπ/2

π [2] 3π/2 [2] [2] [2]
Depth is calculated by 2 frames

Alternate Phased Operation
Frame 1 Frame 2
Pixel Depth (P4T) Pixel Depth (P4T) Depth (2T)

Even
∆Q0 ∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2

[0] [0] Intp. [0] [0] Intp. [0] [0]
∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Q0 ∆Qπ/2 ∆Q0 ∆Qπ/2

Odd
[1] [1] Intp. [1] [1] Intp. [1] [1]
∆Q0 ∆Qπ/2
Even
∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2

[2] [2] Intp. [2] [2] Intp. [2] [2]
No motion artifact Select depth High accuracy

We can calculate two depth (P4T & 2T) at the same time.
Hybrid Depth Imaging
Frame 1 Frame 2
Hybrid Imaging
Depth (P4T) Depth (P4T) Depth (2T)
Motion
Even
∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2 detection Motion

[0] Intp. [0] Intp. [0] [0]
detection
∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2 ∆Q0 ∆Qπ/2
Odd
[1] Intp. [1] Intp. [1] [1]

Even
∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2

[2] Intp. [2] Intp. [2] [2]
Close Far
To select the depth, we need a motion detection

(Comparing two depth images from frame 1 & 2)
Frame 1 Frame 2
Hybrid Imaging
Depth (P4T) Depth (P4T) Depth (2T)
Even
∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2

[0] Intp. [0] Intp. [0] [0]
∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2 ∆Q0 ∆Qπ/2

Odd
[1] Intp. [1] Intp. [1] [1]

Even
∆Q0 ∆Qπ/2 ∆Qπ/2 ∆Q0 ∆Q0 ∆Qπ/2

[2] Intp. [2] Intp. [2] [2]
Close Far
Motion No motion
P4T – To reduce motion artifact
Select the depth
2T – To enhance depth accuracy
Interpolation in Pseudo 4-tap
Alternate
Pixel ∆Q0 ∆Qπ/2
Phase Driver
∆Q0 ∆Q0
[0] [0] ?
∆Qπ/2 ∆Qπ/2
LD
[1] ? [1]
0 ∆Q0 ∆Q0
π [2] [2]
?
π/2 ∆Qπ/2 ∆Qπ/2
[3]
? [3]
3π/2
∆Q0 ∆Q0
[4] [4]
?
Pseudo 4-tap depth has low accuracy by interpolation

Linear Interpolation
Alternate
Pixel Real Raw Interpolated
Phase Driver
∆Q0
[0] ∆Qπ/2 ∆Qπ/2 ∆Qπ/2
[1] [1] [1]
LD ∆Qπ/2
∆Qπ/2
[1]
[2]
? ∆Qπ/2
0 ∆Q0 Intp.
π [2]
π/2 ∆Qπ/2 ∆Qπ/2 ∆Qπ/2 ∆Qπ/2

[3] [3] [3]
3π/2 [3]
∆Q0 Depth
[4]
Problems of edge degradation

Pseudo 4-tap interpolation
Alternate
Pixel ∆Q0 ∆Qπ/2
Phase Driver
∆Q0 ∆Q0
[0] [0] ?
∆Qπ/2 ∆Qπ/2
LD
[1] ? [1]
0 ∆Q0 ∆Q0
π [2] [2]
?
π/2 ∆Qπ/2 ∆Qπ/2
[3]
? [3]
3π/2
∆Q0 ∆Q0
[4] [4]
?
Find edge weighted coefficient a and b

Alternate Raw ∆Qπ/2
Pixel
Phase Driver
∆Q0 ∆Q0
[0]
[0]
∆Q0 ∆Qπ/2
LD ∆Qπ/2 [1] [1]
[1] We
∆Q0 Edge??
0 ∆Q0 [2]
cannot ?
π [2] find
π/2 ∆Qπ/2 Real
[3] Edge ∆Q0 ∆Qπ/2
3π/2 [3] [3]
∆Q0
[4] ∆Q0
[4]
We cannot detect an edge

Alternate ∆Q0 ∆Qπ/2
Pixel
Phase Driver
∆Q0 ∆Q0
[0]
[0]
LD ∆Qπ/2
[1]
= No edge ∆Qπ/2
[1]
∆Q0 Edge!!
0 ∆Q0 [2] ? Edge!!
π [2]
∆Qπ/2 Real Occurred
π/2
3π/2 [3] Edge ∆Qπ/2 @Bottom
[3]
∆Q0
[4] ∆Q0
[4] Depth
Now, we can detect an edge
Alternate ∆Q0 ∆Qπ/2
Pixel
Phase Driver Weight
∆Q0 ∆Q0
[0] of edge
[0]
− |∆Q0| + ∆Qπ/2
LD ∆Qπ/2 [0]-[2]
[1] W[1] [1]
∆Q0
0 ∆Q0 [2]
+
π [2] |∆Qπ/2|
−
[1]-[3]
π/2 ∆Qπ/2 − W[2]
[3]
|∆Q0| + ∆Qπ/2
3π/2 [2]-[4]
[3]
∆Q0 Find
[4] ∆Q0
edge [4] Depth
Step 1. Formulate weight of edge by adding two differences
Weight of edge ∆Qπ/2
∆Q0
[0]
|∆Q0| + ∆Qπ/2
High
− ∆Qπ/2
[0]-[2] [1]
W[1] [1]
∆Q0
[2]
+ Low ∆Qπ/2
|∆Qπ/2| Intp.
[1]-[3]
− W[2]
|∆Q0| + ∆Qπ/2
[2]-[4] ∆Qπ/2
[3] [3]
∆Q0 Edge preserving
[4] Depth
Step 2. Calculate edge coefficient &
∆Σ BGL Cancelling Scheme
Even/Odd frame
320 x 240 Over-pixel Analog mem. ISP
0&π
π/2&3π/2 VGA upscaler
3. ∆Σ BGL Cancelling
(BGLC) Scheme
Potential (V)
TX2
0&π
π/2&3π/2
Odd TX1
Pseudo
...
FD2
0&π Even 4-tap FD1
π/2&3π/2 (P4T) 320 x 240

BSI Field- P4T depth
0&π accelerated interpolator
π/2&3π/2 Pixels
10-80 MHz Σ
∆
PLL
TS1 TS2 TS3 TS4 Denoising
∆-Σ Background Light Suppression
Bias/DAC ∆Q0, ∆Qπ/2
∆Σ BGLC : Proposed Circuits & Timing
*Row [0] (TSUB1) *Row [0] (TSUB2)
R TX1 TX2 R
TX1
TX2
SM CM SM
VDD R
FD1 FD2
RINT
S2 RINT
S1
CIN
S1 S2
VREF SM
V∆Σ
1. Double sampling (DS) circuit ∆ Operation is done @ Row [255]

*For a simple illustration,
2. Analog memory (AM) Σ 1 Row operation is shown
∆Σ BGLC (@ TSUB1): 1. Pixel Reset
Row [0] (TSUB1) Row [0] (TSUB2)
R TX1 TX2 R
TX1
VTH1 VTH2
TX2
SM CM SM
R
FD1 FD2
RINT
S2 RINT
S1
CIN
S1 S2
VREF SM
V∆Σ
VTH1 VTH2
∆Σ BGLC (@ TSUB1): 2. Exposure
R TX1 TX2 R Emitter
TX1 0
V0
Vπ TX2 π
SM CM SM
R
FD1 FD2
RINT
S2 RINT
S1
QBGL CIN
S1 S2
VREF SM
V∆Σ
TX1 is synchronized with the emitter

∆Σ BGLC (@ TSUB1): 3. Double Sampling #1
R TX1 TX2 R
TX1
VFD1 TX2
Charge SM CM SM R
FD1 FD2 Flow
RINT
S2 RINT
S1
CIN
VFD1 S1 S2
VREF SM
V∆Σ
*Assume VREF=0 = VREF = 0
AM (CM)
Reset
Analog memory (AM) reset
R TX1 TX2 R
Charge Flow TX1
VFD2 TX2
SM CM SM R
FD1 FD2
RINT
S2 RINT
S1
CIN
VFD2 S2
S1
VREF SM
V∆Σ
*Assume CIN=CM = VFD1 - VFD2
AM (CM)
QBGL
QBGL is cancelled Q0 Qπ ∆Q0
∆QTH (Fixed pattern noise) is stored QTH1 QTH2 ∆QTH
∆Σ BGLC (@ TSUB2): 1. Pixel Reset
R TX1 TX2 R
TX1
VTH1 VTH2 TX2
SM CM SM
R
FD1 FD2
RINT
S2 RINT
S1
CIN
S1 S2
VREF SM
V∆Σ
VTH1 VTH2
∆Σ BGLC (@ TSUB2): 2. Exposure
R TX1 TX2 R Emitter
TX1 0 π
V0 TX2 π 0
Vπ SM CM SM
R
FD1 FD2
RINT
S2 RINT
S1
CIN
S1 S2
VREF SM
V∆Σ
TX1 and TX2 are interchanged

(TX2 is synchronized with Emitter)
R TX1 TX2 R
TX1
VFD2 TX2
Charge SM CM SM
R
FD1 FD2 Flow
RINT
S2 RINT
S1
CIN
VFD2 S2
S1
VREF SM
V∆Σ
*Assume CIN=CM = VFD1 - VFD2
AM (CM)
QBGL
S1 and S2 are interchanged Q0 Qπ ∆Q0
AM conserve V∆Σ of TSUB1 for Σ QTH1 QTH2 ∆QTH
R TX1 TX2 R
Charge Flow TX1
TX2
VFD1 SM CM SM R
FD1 FD2
RINT
S2 S1
CIN
VFD1 S1 S2
VREF SM
V∆Σ
*Assume CIN=CM = 2·∆V0+(∆VTH-∆VTH)
TSUB1 TSUB2 AM (CM)
QBGL
Cancelled QBGL & ∆QTH Q0 Qπ Q0 Qπ 2·
without reset sampling QTH1
∆ QTH2 Σ QTH2 ∆ QTH1 ∆Q0
Over Pixel Analog Memory
Pixel Unit
Pixel Unit
For ∆Σ operation,
R R CM(AM)
TX1 TX2
each pixel requires
Analog
mem.
: 278 fF
one analog memory.
S S SM CM SM
The chip area S2 RINT

CIN
increased S1
VREF V∆Σ
Over pixel analog memory by using BSI process

Outline

• Summary
Chip Photograph
Process 90 nm BSI CMOS
336×256
4.8 mm Pixel Array
(640×480 w/ ISP)
PLL
Alternate Phase
Alternate Phase
Pixel Pitch 8 μm
Pixel Array
Driver
Driver
336Hx256V 2-Tap Pixel : 3.3 V
3.4 mm
ISP
A-Phase Driver : 2.3 V
Supply
Δ-Σ BGLC circuits
Analog : 1.8 V
10b SS ADC / Readout
Digital : 1.2 V
Core Size 3.4 × 4.8 mm2
Measured Depth
Over the range from 0.75 to 4 m
4.0
4 FMOD 30 MHz
Ideal
Measured
Ideal
3.5
3.5 Measured Object White target
3.0 Int Time 20 ms
Measured (m)
Measured (m)
2.5
2.5 Optical
248 μW/cm2 @ 1m
Power
2.0
2
1.5
1.5
1.0
1 Non linearity: <2.2 %
0.5
0.5
0.5
0.5 1.0
1 1.5
1.5 2.0
2 2.5
2.5 3.0
3 3.5
3.5 4.0
4
Distance (m)
Distance (m)
Depth Noise
2.5
1
FMOD 30 MHz
Maximum: 2.1 cm @ 4m
Measured
2.0
0.8 (0.54 %) Object White target
Depth noise (cm)
Depth noise (%)
Int Time 20 ms
1.5
0.6
Optical
248 μW/cm2 @ 1m
Power
1.0
0.4
0.5
0.2
0.0
0
0.5
0.5 1.0
1 1.5
1.5 2.0
2 2.5
2.5 3.0
3 3.5
3.5 4.0
4
Distance (m)
Distance (m)
Raw ∆Q0 (Intp.) ∆Qπ/2 (Intp.) Depth (P4T)
Frame
1 Alternate phases
0, π/2
Frame
1 Alternate phases
0, π/2
Frame
2 Alternate phases
π/2, 0
Depth (2T)
Frame
1, 2
Close Far Motion artifact

© 2020 IEEE
Frame
1 Alternate phases
0, π/2
Frame
2 Alternate phases
π/2, 0
Depth (2T) Depth
No motion
Frame (2T) Motion
1, 2 (P4T) Frame
Difference
Hybrid image (2T + P4T)

© 2020 IEEE
Close Far 5.4: A Dynamic Pseudo 4-Tap CMOS Time-of-Flight Image Sensor
BGL Cancellation & Interpolation Results
BGL : 120 klx BGL : 120 klx
w/o BGLC
w/ BGLC
Corrupted by BGL Error was removed
2-tap Pseudo 4-tap
Edge
Error
No intp. Linear intp. P4T intp. Up-scaled
The edge was not degraded!!
Motion Artifact & BGL Cancellation Video
Synced Time Depth
RGB
(640 × 480)
Distance :
4.5 m
BGL : 120 klx

30 FPS
(Sun + Halogen + LED)
Motion artifact suppression & Background light canceling
Comparison
This Cho Bamji Kato Keel
Work JSSC’14 ISSCC’18 JSSC’18 VLSI’19
320×240
Pixel Array 84×64 1024×1024 320×240 640×480
(640×480 w/ ISP)
Pixel Architecture Pseudo 4-tap 2-tap 2-tap 2-tap 4-tap
Process 90nm BSI 110nm FSI 65nm BSI 90nm BSI 65nm BSI
Pixel Pitch 8μm 23.6μm 3.5μm 10μm 7μm
Fill Factor 43% 24% ~100% (w/ μLens) >80% (w/ μLens) -
Frame Rate 10 to 60fps 10 to 120fps 30fps - 60fps
Modulation Frequency 10 to 80MHz 12.5MHz 10 to 200MHz 50, 100MHz 10 to 150MHz
Demodulation Contrast 65% @80MHz 50% @12.5MHz 87% @200MHz 85% @100MHz 86% @100MHz
ADC Resolution 10bits 10bits 9 or 10bits - 10 or 12bits
Depth Noise <0.54% <4.5% <0.2% 0.59%@ 1m <0.62%
Depth Range 0.75 to 4m 0.75 to 4m 0.4 to 4.2m - 0.4 to 4m
On-Chip BGL Suppression >120klx 100klx 25klx - -
Motion Artifact Suppression Yes No No No Yes
Outline

• Summary
Summary
1. Trident PPD structure : Demodulation contrast of 65% at 80MHz.
2. Dynamic pseudo 4-tap scheme : To suppress a motion artifact with

a high frame rate.
3. ∆Σ BGLC scheme : To cancel a background light error over 120 klx
Demonstration Session 1
Today 5:00-7:00 PM @Golden Gate Hall
Thank you for your attention
A 2.1e- Temporal Noise and -105dB Parasitic Light
Sensitivity Backside-Illuminated 2.3μm-Pixel
Voltage-Domain Global Shutter CMOS Image
Sensor Using High-Capacity DRAM Capacitor
Technology
Jae-kyu Lee, Seung Sik Kim, In-Gyu Baek, Heesung Shim, Taehoon Kim, Taehyoung
Kim, Jungchan Kyoung, Dongmo Im, Jinyong Choi, KeunYeong Cho, Daehoon Kim,
Haemin Lim, Min-Woong Seo, JuYoung Kim, Doowon Kwon, Jiyoun Song, Jiyoon Kim,
Minho Jang, Joosung Moon, HyunChul Kim, Chong Kwang Chang, JinGyun Kim,
Kyoungmin Koh, HanJin Lim, JungChak Ahn, Hyeongsun Hong, Kyupil Lee, Ho-Kyu Kang
Samsung Electronics Co., Hwaseong, Korea
© 2020 IEEE 5.5: A 2.1e- Temporal Noise and -105dB Parasitic Light Sensitivity Backside-Illuminated 2.3μm-Pixel Voltage-Domain Global Shutter MOS Image
International Solid-State Circuits Conference Sensor Using High-Capacity DRAM Capacitor Technology 1 of 23
Outline
1.Motivation
 Application & Technology Comparison
 Issues on Conventional Global Shutter
2.Key Technologies
 Photo Diode, Transistors, and Capacitor
 Optical Properties
 Pixel Architecture & Readout Path
3.Results
4.Summary
Global Shutter Application
■ Accurate information: Acquiring Images without motion artifacts
Factory Automation Jello effect in video sensor
Global Shutter Rolling Shutter
Source: Basler
Mobile application Automotive Drone Camera

Face Identification Driver Monitoring System
Obstacle Avoidance
https://zionmarketresearch.wordpress.com/2017/05/22/global-factory-automation-and-machine-vision-market/
https://www.cpomagazine.com/data-privacy/facial-recognition-technologies-time-to-face-the-music/
https://www.st.com/content/st_com/en/about/media-center/press-item.html/p4100.html
Rolling Shutter Rolling Shutter VS Global Shutter
Line numberCapture Read
start Reset Sequence
Line number
Time Axis
Frame Time
Integration
Capture start
Line number Capture end Read Time
Global Shutter
Line number
Sample & Hold

Global Reset
Time Axis
Integration Time
Readout time
Global Shutter Pixel Size Trend
■ Pixel Shrink competition started
ISSCC 2012
7 Masaki Sakakibara[1]
6
IEDM 2018
Pixel Pitch [um] 5 Y. Kumagai [2]
This Work
4
3
2
1 IEDM 2019
Geunsook Park [3]
0
2012 2014 2016 2018 2020
출시 년도
[1] Masaki Sakakibara1, et al., “An 83dB-Dynamic-Range Single-Exposure Global-Shutter CMOS Image Sensor with In-Pixel Dual Storage,” ISSCC 2012
[2] Y. Kumagai, “Back-illuminated 2.74um-Pixel-Pitch Global Shutter CMOS Image Sensor with Charge-Domain Memory Achieving 10k e- Saturation Signal,” IEDM18
[3] Geunsook Park, et al., “A 2.2µm stacked back side illuminated voltage domain global shutter CMOS image sensor,” IEDM 2019
Charge Domain VS Voltage Domain
Charge Domain Voltage Domain
- Simple structure & low noise - High shutter efficiency, PD Fill Factor ~80%
- Small PD Fill Factor ~30% - Random noise degradation due to KTC
- PLS degradation - Small Capacitance Capacitor
ML ML
Photo
Photo
Diode
Diode
Memory Memory
Photo Switch Tr. Switch Tr.
MIM Cap
Diode
Voltage Domain
RG
Too many Transistors in a limited area.
SF1 S1 S2
TG
SF2
 Large Size Pixel
Bias C1 C2
SEL
RG
1) Small Photo Diode
SF1
 Low FWC
TG
RG SF2
2) Small Capacitance
S1
SF1 CAL Bias C1
SEL
 Low PLS(Parasitic Light Sensitivity)
SF3
TG
SF2 S2
C2  High Random Noise
SAM SEL
Bias
C1
C2
SEL
(kT/C noise dominant)
Global Shutter with DRAM Cap. Technology
⑤ BST(Back Scattering Tech.) ① VTG & Buried PD
 940nm Sensitivity↑ ML  FWC↑
Photo
Top view
Diode
M1
③ DRAM cap ~0.72pF
M2 (two high capacity capacitors)
DRAM  PLS↑
Cap.  Random Noise ↓
② Many Transistors M3
(10T) M4
M5
④ Noise Reduction
① Photo Diode Fill Factor
■ VTG & Buried Photo Diode  Large PD fill factor( ~90%)
FD Transfer gate Off
(photon integration)
PD
VTG(Vertical
Transfer Gate) e-
e-
Transfer gate On
Full (Sig e-  FD transfer)
Buried Isolation
DTI(deep
Photo Diode Trench Isolation)
FD
② Pixel Transistors
■ More than 10 transistors are placed in a 2.3um pixel pitch
Planar Transfer Gate Vertical Transfer Gate
Buried Photo Diode

Surface Photo Diode
e-
e- STI
FD N+ N+ FD N+ N+ N+ N+
TG gate TG gate gate
③ DRAM Capacitor Technology
Inner Metal Electrode
Dielectric material
Capacitance density
450
400
Capacitance (fF/um2)
350 Outer Metal Electrode

300
250
200 DTI tSi ~8um
~ 70X higher
150
100 VTG
50
0
MIS Trench MIM Planar MIM Concave Capacitor
or MIM Trench
Capacitor Type
DTI: Deep Trench Isolation
VTG: Vertical Transfer Gate
④ Parasitic Light Sensitivity & Noise
120 16
Parasitic Light Parasitic Light Sensitivity
14
100
Random Noise [e-]

12
PLS(S.E)[dB]
This work
80
10
60 8
6
40
4
Parasitic Light 20
2
0
Random Noise 0
0 0.2 0.4 0.6 0.8 1 1.2
~1/360 Cap. [pF]
3,600,000e-
10,000e-
⑤ IR Quantum efficiency
BST(Backside Scattering Technology) for Near IR QE improvement
100
BST Optical Power density
90
Quantum Efficiency, QE(%)
w/o BST w/ BST

80
70
60
50
Increase
40
Optical path
30
20  QE ↑
10
0
400 500 600 700 800 900 1000
Wavelength (nm)
Readout Architecture
Top wafer (CIS Technology)
Row Driver
Image Output
Signal I/F
Processor
TSV
ADC
Comparator
Counter
TSV: Through Silicon Via Bottom wafer (Logic Technology)
Pixel Operation: Initialize
RG
TG
CAL
SAM
SEL
Bias
integration reset settling time

start @ X node
1. Charge Integration starts by toggling RG and TG. (consider DCG is on)

2. During the charge integration time, RG & CAL toggle to initialize FD, X and Y.
Pixel Operation: Global dump
Global Dump
RG
TG
CAL
SAM
SEL
Bias
reset settling time signal settling time

@ X node @ X node
3. After reset settling time, TG toggle to transfer signal charge from PD to FD.
4. SF1 transfer voltage level to X via SAM & change voltage level of Y by capacitive coupling.
(2~4 operations are called global dump.)
Pixel Operation: Rolling Readout
Global Dump Rolling Readout
RG
TG
CAL
SAM
SEL
Bias
SIG ADC RST ADC

reset settling time signal settling time
@ X node @ X node
5. After global dump, rolling readout (row-by-row) starts.
6. Each pixel signal stored at Y is first digitized by column-parallel ADC.
7. Reset level of same pixel is digitized sequentially after CAL toggle.
(to remove FPN by SF2 Vth variation and readout circuitry)
Pixel Operation: Rolling Mode
Global shutter operation Rolling shutter operation
RG RG
DCG DCG
CAL CAL
TG TG
SF1 C2 SF1 C2
SF2 SF2
SAM SAM
Bias C1 C1
SEL Bias
SEL
SEL2 SEL2
Analog circuit is designed to support both global and rolling readout operation.
Rolling shutter operation is possible by making normal 4T through SEL2 bypass Tr.
Pixel Operation: Black Sun Prevention Scheme
Global Dump
RG
TG
CAL
…
SAM
SEL
Bias black sun occurrence timing

SEL2
SEL2
Vclamp
Black sun prevention is also possible with the SEL2 transistor

by clamping SF1 output during black sun occurrence timing.
Micrograph & Technology
Parameter Value
Top Chip
Supply Voltage 2.85[V]/1.05[V]
Pixel Size 2.3umX2.3um, Stacked BSI
Resolution 1280X800
Readout rate 8.3[msec]

6,000e-@HCG
Saturation Signal
12,000e-@LCG
Bottom Chip
Sensitivity 18[ke-/lx.s]
Parasitic Light Sensitivity <-115[dB] (-95dB@IR940nm)
QE@940nm IR 42%
Readout Noise 2.1e[e-rms], (Analog gain +18dB)
Comparison of Sensor Performance
This 2015 2016 2017 2018 2019 2019
Work IISW[1] VLSI[2] JSSC[3] ISSCC[4] ISSCC[5] IEDM[6]
6.9um 2.2um
2.3um 4.5um 3.75um 5.86um 2.7um
Pixel Size Stacked Stacked BSI
Stacked BSI FSI BSI FSI Stacked BSI
BSI (With Cu-to-Cu)
Frame Rate 120fps 80fps 50fps 480fps 660fps 120fps 120fps

Shutter Efficiency -115dB@green <-100dB
-74dB -69.5dB -99.6dB -75dB -86dB
(PLS) -95dB@940nm @940nm
12ke-@LCG 12ke-@LCG
FWC 10ke- 8.1ke- 30.5ke- 16.6ke- 10ke-
6ke-@HCG 3.5ke-@HCG
RN 2.1e- 12e- 8.5e- 4.6e- 5.15e- 3.5e- 3.1e-
IRQE@940nm 42% - - - - - 38%
Voltage Voltage Voltage Charge Charge Voltage Voltage
GS Type
Domain Domain Domain Domain Domain Domain Domain
[1] T. Geurts, et al., “A 25 Mpixel, 80fps, CMOS Imager with an In-Pixel-CDS Global Shutter Pixel,” Int. Image Sensor Workshop, pp. 7.02, June 2015.
[2] L. Stark, et al., “Back-Illuminated Voltage-Domain Global Shutter CMOS Image Sensor with 3.75μm Pixels and Dual In-Pixel Storage Nodes,” IEEE Symp. VLSI Tech., pp. 1-2, June 2016.
[3] Y. Oike, et al., “8.3 M-Pixel 480-fps Global-Shutter CMOS Image Sensor with Gain-Adaptive Column ADCs and Chip-on-Chip Stacked Integration,” IEEE JSSC, vol. 52, no. 4, pp. 985 993, Apr. 2017.
[4] M. Sakakibara, et al., “A Back-Illuminated Global-Shutter CMOS Image Sensor with Pixel-Parallel 14b Subthreshold ADC,” ISSCC Dig. Tech. Papers, pp. 80-81, Feb. 2018.
[5] Chen Xu1, et al., “A Stacked Global-Shutter CMOS Imager with SC-Type Hybrid-GS Pixel and Self-Knee Point Calibration Single-Frame HDR and On-Chip Binarization Algorithm for Smart Vision Applications,”, ISSCC Dig. Tech. Paper, pp 94-96, 2019.
[6] Geunsook Park, et al., “A 2.2μm stacked back side illuminated voltage domain global shutter CMOS image sensor”, IEDM pp 378, 16.4.1, 2019
Sample Movie
Rolling Shutter Global Shutter
Condition: Again X16, 200us, F1.8
Summary
 A 2.3um pixel pitch global shutter is implemented

with a DRAM capacitor technology
 Random noise 2.1e- and shutter efficiency <-100dB are

achieved by high-capacity DRAM Capacitor
 IR QE over 40% is achieved by BST and DTI processes
A 1/2.65in 44Mpixel CMOS Image Sensor
with 0.7μm Pixels Fabricated in Advanced
Full-Depth Deep-Trench Isolation Technology
HyunChul Kim, Jongeun Park, Insung Joe, Doowon Kwon, Joo Hyoung Kim,
Dongsuk Cho, Taehun Lee, Changkyu Lee, Haeyong Park, Soojin Hong,
Chongkwang Chang, Jingyun Kim, Hanjin Lim, Youngsun Oh, Yitae Kim, Seungjoo Nah,
Sangill Jung, Jaekyu Lee, JungChak Ahn, Hyeongsun Hong, Kyupil Lee, Ho-Kyu Kang
Samsung Electronics, Hwaseong, Korea
© 2020 IEEE 5.6 : A 1/2.65in 44Mpixel CMOS Image Sensor with 0.7μm Pixels Fabricated in Advanced Full-Depth Deep-Trench Isolation Technology
Outline
□ Pixel scaling down
 Pixel shrink trend
 Challenges for pixel shrinkage – FWC, Crosstalk, Sensitivity
□ Advance of key technology at 0.7um-pixels
 New process integration technology
 Improvement in pixel characteristics
 Advance in optical performance
 Sensor performance and others
□ Summary
Outline
□ Summary
Pixel shrink trend
 Mobile market has been pursuing smaller pixels for higher resolution images.
# of pixels (Mp)
# of pixels (Mp)
Pixel size (µm)
Year Pixel size (µm)

Pixel isolation technology
 Small pixel has been developing by full-depth trench isolation technology.
IEDM2012 ISSCC2018 IEDM2019 ① Micro Lens
① ② Color Filter
③ ②
③ Low Refractive
Sensitivity very good

X-talk very good
X-talk good
Index Grid
⑤ ④ Deeper Photo
④ Diode
⑤ Deep Trench
⑥ Isolation
⑦ ⑥ Vertical Gate
⑧
(Not shown here)
⑦ Metal Contact
1.12um
⑧ Metallization
1.0 / 0.9um 0.8 / 0.7um
2013 2015 / 2017 2018 / 2019
Challenges for pixel scaling * FWC : Full Well Capacity
 Main challenges were FWC*, crosstalk and sensitivity for pixel scaling.
※ Forecasting by Si Volume ※ Forecasting by Si Area
Sensitivity (e-/lux·sec)
Full Well Capacity (e-)
7,500
1.00
5,900
0.75
4,900
3,800 0.60
0.46
1.0 0.9 0.8 0.7 1.0 0.9 0.8 0.7

Pixel size (µm) Pixel size (µm)
Outline
□ Summary
Silicon area and volume * CD : Critical Dimension
 Reducing ① DTI CD* and increasing ② thickness were to increase the
silicon area and volume.

※ =

STI
80 120
V
T
110
Polysilicon gap-fill
G
70 69
100
aspect ratio
aspect ratio
DTI etching
Photodiode
60
② 80 75
DTI
50 45
60 56
① 40
35
38 50
BARL 30 40
LRI
Color Filter
1.0 0.9 0.8 0.7 1.0 0.9 0.8 0.7
Micro Lens
Pixel size (µm) Pixel size (µm)
IEEE 2020 © 5.6 : A 1/2.65in 44Mpixel CMOS Image Sensor with 0.7μm Pixels Fabricated in Advanced Full-Depth Deep-Trench Isolation Technology
International Solid-State Circuits Conference of 23 8
HART dry etching
 Faster Bosch processing was introduced for HART dry etching.
- Bosch process was composed of bottom etching, polymer deposition
and bottom polymer clear.
* HART : High Aspect Ratio Trench
※ F. Roozeboom et al. ECS J. Solid State Sci. Technol. 2015;4:N5067-N5076
Silicon leaning due to cleaning
 Surface energy was decreased through advanced cleaning technology.
Pattern stress Surface tension

Pattern stress (a.u.)
Surface tension (a.u.)

△66% ▽81%
0.8µm 0.7µm DIW* Hot IPA**

*DIW : DeIonized Water
**IPA : IsoPropyl Alcohol
Advanced poly-silicon gap-fill
 High aspect ratio poly-silicon gap-fill was completed by DED (Deposition
– Etching – Deposition) process.
DTI etching & Side-wall oxide DE poly-silicon Final poly-silicon & Etch-back
Process flow
Laterally extended photodiode
 Effective silicon volume was maintained by smaller DTI CD and lower
boron doping concentration.
1.0
0.77
e-e- e-e-
e- e-e- e- e-e-
e- e-
Smaller DTI CD
(▽20%)
& lower B-doping
(▽57%)
0.8µm 0.7µm 0.8µm 0.7µm
Laterally extended photodiode
 Same level of full well capacity was achieved successfully as the
previous generation’s one.
Depletion Junction
Region 0.8µm Line 0.7µm

6,000 6,000
DTI CD
DTI CD
N-type region
(0.382㎛) (0.395㎛)
P-type (0.08㎛)
Region
(0.142 Effective
㎛) photodiode area (0.455
㎛)
(0.422㎛)
0.8µm 0.7µm
Dark current induced by RC delay
 Dark current due to RC delay was increasing in smaller pixel size and
higher pixel density.
Applied voltage
Cox @ chip edge
p+ region
(network resistance)
Poly-
SiOx (Photo
APS array model
Resistivity (Ω cm)
silicon Rox
diode)
RDTI
p
Center n
RDTI, tot Cox, tot
Concentration N (cm-3)
※ Sorab K. Ghandhi (1994), VLSI Fabrication Principles
: Silicon and Gallium Arsenide, New York, John Wiley, p7
Dark current induced by RC delay
 RC delay was on the order of nano-seconds rather than a few seconds
using boron-doped polysilicon whose resistivity is lowered to ~10-6Ω·m.
1st frame Stable state
Stream off/on toggle
Dark current (a.u.)
RC delay
Time (sec) No RC delay
2x2 pixel-binning image sensor
 Pixel-binning was a process that combines data from four pixels into one.
This technique enabled big-pixel-like performance including high sensitivity.
36.0 Binning (2x2)

2x2 binning mode
Y-SNR [dB] @ 20 lux

Bayer
34.0
32.0
Binning mode
30.0
28.0
26.0
1.4 1.12 1.0 0.9 0.8 0.7
Pixel size (µm)
※ Dilemma: Market demands more pixel at limited form factor
① Need smaller pixels, ② Sensitivity decreases as pixel shrinks, ③ Poor low light image quality
Quantum efficiency
 Focused beam size of micro lens did not scale during pixel shrank.
* RI : Refractive Index
Quantum efficiency @540nm

Beam spot / pixel
130%
size ratio
102%
73%
1.4 1.0 0.7

Pixel size (µm)
Pixel size (µm)
※ Metal grid has been used for color filter isolation material.
Optical performance improvement
 Low RI grid CF* isolation & larger PD** volume maximized quantum efficiency.
* CF : Color Filter
** PD : Photodiode Tungsten grid Low RI grid
Tungsten grid Low RI grid

Narrower
DTI 1.4
1.2 Low RI grid
TEM @low RI grid 1
Q.E. (a.u.)
Deeper PD
0.8
0.6
0.4 Tungsten grid
0.2
0
400 450 500 550 600 650 700
Wavelength (nm)
Sensor performances * ISSCC2018
** IDEM2019
0.9µm 0.8µm 0.7µm

Characteristics Unit
Tungsten grid Low RI grid
Full-well capacity e- 6,000* 6,000** 6,000
YSNR (@20lux) dB 33.4 32.7 31.7
Temporal noise e- 1.4* 1.2** 1.4
RTS (≥30LSB) ppm 1* 1* 3
White spot (Tj 60 ℃) ppm 90* 10** 10
Dark current (Tj 60 ℃) e-/s 2.0* 1.5** 1.3
Chip micrograph * TSV : Through Silicon Via
 2-stacked, top and bottom layers were connected by wafer bonding & TSV*.
Top layer Bottom layer

Top
layer Pixel
array
Bottom
CIS logic
layer
Sample images
0.8um 32Mpixels 0.7um 44Mpixels
F/# 1.7, exposure time 1/103sec, F/# 2.0, exposure time 1/62sec,
illumination 1,000lux, gain 1x, frame 30fps illumination 1,000lux, gain 1x, frame 30fps
Outline
□ Pixel sacling down
□ Summary
Summary
□ Key technologies at 0.7um-pixels
- High aspect ratio DTI etch technology
- Highly boron-doped polysilicon gap-fill technology
- Low refractive index grid technology
□ 44Mpixel high-resolution CMOS image sensor

- Full well capacity 6,000e- (equivalent to 0.8um-pixels)
- Dark current 1.3e-/s (better than 0.8um-pixels)
A 132dB Single-Exposure-
Dynamic-Range CMOS Image Sensor
with High Temperature Tolerance
Y. Sakano,1 T. Toyoshima,1 R. Nakamura,1 T. Asatsuma,1 Y. Hattori,1 T. Yamanaka,2
R. Yoshikawa,2 N. Kawazu,1 T. Matsuura,1T. Iinuma,1 T. Toya,1 T. Watanabe,1
A. Suzuki,1 Y. Motohashi,1 J. Azami,1 Y. Tateshita,1 T. Haruta, 1 F. Brady 3
1Sony Semiconductor Solutions Corp.

2Sony Semiconductor Manufacturing Corp.
3Sony Electronics Inc.
© 2020 IEEE
International Solid-State Circuits Conference 5.7: A 132dB Single-Exposure-Dynamic-Range CMOS Image Sensor with High Temperature Tolerance 1 of 36
Outline
• Motivation, Current issues
• Concept
• Key technologies
1. Dual gain read-out
2. Sub-pixel read-out
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Outline
• Concept
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Motivation
✔ Needs for real-time sensing increases
👉 The performance of DR is becoming 100000 lux

a key parameter for CISs DR
>120 dB
0.1 lux
© 2020 IEEE
HDR Technology
✔ Common CIS has a DR of at most 80dB
👉 HDR technology is required to achieve a DR of > 120dB
~80 [dB]
 Time division w/ sensitivity ratio
SNR [dB]
 Non-linear response
 Space division w/ sensitivity ratio
Luminance [cd/m2]
© 2020 IEEE
Time division w/ sensitivity ratio
✔ Composing time division images
with sensitivity ratio cause
flickering and motion artifacts
👉 Single-exposure DR is critical
for real time sensing
Flickering
DR [dB]
SNR [dB]
Luminance [cd/m2] Motion artifact

© 2020 IEEE
Non-linear response
log Ids
dVgs
Human S=
d(logIds )
eyeball
Vgs
Vth
✔ The human eye has a log ✔ Leveraging subthreshold region

response and a DR of >120 dB of transistor is well-known
👉 Considering a log response 👉 Poor low-light performance due
to extend the DR makes sense to no concept of accumulation
© 2020 IEEE
Space division w/ sensitivity ratio
Output [e-]
Retina
DR Expansion
Cone cell Luminance [cd/m2]

Rod cell C of small PD ~ C of large PD
✔ It makes sense to implement multiple PDs with different

sensitivities, such as the retina of the human eye
👉 The correlation between sensitivity and capacity is important
© 2020 IEEE
Output [e-] In-pixel capacitor
Output [e-]
DR Expansion DR Expansion
Luminance [cd/m2] Luminance [cd/m2]
✔ To flip the correlation between sensitivity and capacity is important

👉 In-pixel capacitor is useful to expand DR effectively
© 2020 IEEE
Current issues
SNR [dB] SNR [dB]
DR [dB] Composition DR [dB]

SNR
Luminance [cd/m2] Luminance [cd/m2]
✔ Total DR and composition SNR are in a trade-off relationship

👉 it is important to combine PDs or in-pixel capacities properly
while maintaining composition SNR high
© 2020 IEEE
Outline
• Concept
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Block diagram
Pixel Chip
2897 x 1977 pixels
・・・・・・・・・・・・
VDD FCG TGS
・・・・・・
・・・
・・・
FD3
・・・
RST
・・・・・・・・・・・・・・・
Pixel Chip FD2
SP2
Logic Chip FDG FC
Logic Chip Load MOS Tr. TGL
VDD FCVDD
DAC
Column ADCs AMP
Row Decoder
Row Decoder
Row Driver
FD1
Row Driver
Supply
Voltage
Image Signal SEL SP1
Processor
3.3V
1.8V MIPI I/F CPU PLL
1.1V Pixel size : 3μm x 3 μm
DPHY I2C Input Clock
© 2020 IEEE
Block diagram
Pixel Chip
2897 x 1977 pixels
・・・・・・・・・・・・
・・・・・・
・・・
・・・
・・・
・・・・・・・・・・・・・・・
COUNTER
Pixel Chip AZP0 AZP0
AZP1
AZP1
Logic Chip
Logic Chip Load MOS Tr. DAC DAC0 PIX0 PIX
DAC DAC1
Column ADCs PIX1
Row Decoder
Row Decoder
Row Driver
Row Driver
Supply
Image Signal
Voltage
Processor
3.3V
1.8V MIPI I/F CPU PLL
1.1V
DPHY I2C Input Clock
© 2020 IEEE
Timing diagram
Shutter
Exposure
Read-out
✔ Four images are read out
by one exposure
✔ One image is composited
by selecting the signal with
highest SNR for each pixel
© 2020 IEEE
Summary of the Four Images
Image Condition Calculation
SP1H Low Light CDS of S1-R1 (high CG)
SP1L Medium Light CDS of S2-R2 (low CG)
SP2H High Light CDS of S3-R3 (SP2 PD only)
SP2L Max Light Double Sample of S4-R4 (SP2 PD + FD3)
✔ With one exposure, four images are captured from two

photodiodes using 8 captures (four reset, four image).
✔ SP2L is captured with double sampling, but reset noise is

acceptable due to the very high light use condition.
© 2020 IEEE
Outline
• Concept
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Outline
• Concept
1. Dual gain read-out  lower light
3. Pixel structure
• Results
• Summary
© 2020 IEEE
1: Dual gain read-out (1/4)
SEL
RST
✔ Luminance where FDG
SP1H signal is adopted TGL
FCG
TGS
FCVDD
R2 R1 S1 S2
B A-B X-section
VDD FCG TGS
(E) SP1L Reset Level (R2) (G) SP1H Reset Level (R1) (H) SP1H Transfer
FD3 TGL FDG RST TGL FDG RST TGL FDG RST
RST
FD2
SP2 SP1 SP1 SP1
FDG FC FD1 FD2 VDD FD1 FD2 VDD FD1 FD2 VDD
TGL
VDD FCVDD (I) SP1H Signal Level (S1) (J) SP1L Transfer (K) SP1L Signal Level (S2)
AMP TGL FDG RST TGL FDG RST TGL FDG RST
FD1
SEL SP1
SP1 SP1 SP1
FD1 FD2 VDD FD1 FD2 VDD FD1 FD2 VDD
© 2020 IEEE
A 5.7: A 132dB Single-Exposure-Dynamic-Range CMOS Image Sensor with High Temperature Tolerance 18 of 36
SEL
RST
SP1L signal is adopted TGL
FCG
TGS
FCVDD
R2 R1 S1 S2
B A-B X-section
VDD FCG TGS
(E) SP1L Reset Level (R2) (G) SP1H Reset Level (R1) (H) SP1H Transfer
FD3 TGL FDG RST TGL FDG RST TGL FDG RST
RST
FD2
SP2 SP1 SP1 SP1
FDG FC FD1 FD2 VDD FD1 FD2 VDD FD1 FD2 VDD
TGL
VDD FCVDD (I) SP1H Signal Level (S1) (J) SP1L Transfer (K) SP1L Signal Level (S2)
AMP TGL FDG RST TGL FDG RST TGL FDG RST
FD1
SEL SP1
SP1 SP1 SP1
© 2020 IEEE
A 5.7: A 132dB Single-Exposure-Dynamic-Range CMOS Image Sensor with High Temperature Tolerance 19 of 36
✔ The comparators are characterized
COUNTER
AZP0 AZP0
by having 2-inputs system
AZP1
AZP1
for auto-zero function DAC DAC0 PIX0 PIX
DAC1 PIX1
(D) Auto-zero
COUNTER
AZP0 AZP0
AZP1
AZP1
DAC DAC0 PIX0 PIX
DAC1 PIX1
(E) / (K) ADC

© 2020 IEEE
✔ SP1H is read-out by performing
COUNTER
AZP0 AZP0
auto-zero function by using
AZP1
AZP1
another input of comparators DAC DAC0 PIX0 PIX
DAC1 PIX1
(F) Auto-zero
COUNTER
AZP0 AZP0
AZP1
AZP1
DAC DAC0 PIX0 PIX
DAC1 PIX1
(G) / (I) ADC

© 2020 IEEE
Outline
• Concept
2. Sub-pixel read-out  higher light
3. Pixel structure
• Results
• Summary
© 2020 IEEE
2: Sub-pixel read-out (1/2)
SEL
RST
SP2H signal is adopted TGL
FCG
TGS
FCVDD
R3 S3 S4 R4
C-BX-section
C-B X-section
B
VDD FCG TGS (L) FD1 & FD2 Reset (N) SP2H Reset Level (R3) (O) SP2H Transfer
TGS FCG RST TGS FCG RST TGS FCG RST
FD3
RST
SP2 SP2 SP2
FD2
SP2 FD3 FD2 VDD FD3 FD2 VDD FD3 FD2 VDD
FDG FC C
TGL (P/R) SP2H/L Signal Level (S) FD3 & FC Reset (T) SP2L Reset Level (R4)
VDD FCVDD (S3/4) TGS FCG RST TGS FCG RST TGS FCG RST
AMP
FD1
SP1 SP2 SP2 SP2
SEL
© 2020 IEEE
2: Sub-pixel read-out (2/2)
SEL
RST
SP2L signal is adopted TGL
FCG
TGS
FCVDD
R3 S3 S4 R4
C-B X-section
B (L) FD1 & FD2 Reset (O) SP2H Transfer
VDD FCG TGS (N) SP2H Reset Level (R3)
TGS FCG RST TGS FCG RST TGS FCG RST
FD3
RST
SP2 SP2 SP2
FD2
SP2 FD3 FD2 VDD FD3 FD2 VDD FD3 FD2 VDD
FDG FC C
TGL (P/R) SP2H/L Signal Level (S) FD3 & FC Reset (T) SP2L Reset Level (R4)
VDD FCVDD (S3/4) TGS FCG RST TGS FCG RST TGS FCG RST
AMP
FD1
SP1 SP2 SP2 SP2
SEL
© 2020 IEEE
Outline
• Concept
3. Pixel structure  higher Qsat
• Results
• Summary
© 2020 IEEE
3: Pixel structure
SP1 SP2 SP1 SP2
Reference This Work

✔ The area of SP1 and FC is almost doubled
© 2020 IEEE
Outline
• Concept
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Implementation
10.775mm
Item Data
7.103mm
Process Pixel : FE90nm / BE65nm | 1P4Cu Pixels
Logic : 40nm | 1P6Cu1AL
Pixel Chip
Supply Voltage 3.3 [V] / 1.8 [V] / 1.1 [V]
Pixel Pitch 3.0 [μm]
Pixel Array 2897 (H) x 1977 (V) Column ADCs
Die Size 10.775 [mm] (H) x 7.103 [mm] (V)
Max. Frame Rate 40 [fps] @ 10bit
30 [fps] @ 12bit Logic Chip
© 2020 IEEE
Performance
Item Data
FWC (SP1, SP2+FC) 12800 [e-] , 165800 [e-] @ Tj=85℃
Sensitivity (SP1) 38000 [e-/lux・s]
Sensitivity Ratio (SP1/SP2) 14.5
Conversion Gain (SP1H, SP1L, SP2H/L) 197 [μV/e-] , 54 [μV/e-] , 6.7 [μV/e-]
Random Noise 0.6 [e-rms] @ Tj=85℃
Single-Exposure-Dynamic-Range 132dB @ Tj=85℃
Min. Composition SNR 27dB @ Tj=85℃, 25dB @ Tj=100℃
FWC of SP2+FC · Sensitivity ratio of SP1/SP2

Single−Exposure−Dynamic−Range = 20·log ( )
Random noise
© 2020 IEEE
Noise factor of SP2H & SP2L
✔ The noise of SP2H is relatively lower than that of SP2L

✔ FPN dominates the noise of SP2L at high temperature
© 2020 IEEE
Signal-to-noise ratio curve (Tj=100 )
SNR [dB]
SNR kept > 25 dB at

mode to mode transition
Luminance [a.u.]
✔ Min. composition SNR of above 25dB is maintained at Tj=100℃

© 2020 IEEE
Composite photo-response
Output [e-]
Luminance [a.u.]
✔ Total photo-response characteristics have sufficient linearity
© 2020 IEEE
Composite image comparison
Room Temperature Tj=100℃
This Work
Reference
© 2020 IEEE
Performance comparison
Paper This Work IISW2019 [2] IISW2019 [4] IEDM2018 [5]
Pixel Pitch 3.0 [μm] 3.0 [μm] 2.8 [μm] 3.0 [μm]
HDR Technology sensitivity in-pixel capacitor sensitivity ratio sensitivity ratio
ratio & in-pixel & in-pixel
capacitor capacitor
Random Noise 0.6 [e-rms] 2.7 [e-rms] †† 0.83 [e-rms] 0.68 [e-rms]
@ Tj=85℃ @ RT @ RT
Single-Exposure 132 [dB] 96 [dB] 120 [dB] 121 [dB]
Dynamic-Range @ Tj=85℃ @ RT †† @ Tj=60℃
Min. 27 [dB] 32 [dB] @ 25℃, 23 [dB] > 20 [dB]
Composition @ Tj=85℃ 31 [dB] @ 60℃, @ RT †† @ Tj=60℃
SNR 25 [dB] 29 [dB] @ 80℃,
@ Tj=100℃ 25 [dB] @ 100℃
†† Calculated Values based on disclosed information
© 2020 IEEE
Outline
• Concept
3. Pixel structure
• Results
• Summary
© 2020 IEEE
Summary
A 132dB Single-Exposure-
Dynamic-Range CMOS Image Sensor
with High Temperature Tolerance
✔ Single-Exposure Dynamic-Range of 132 [dB] @ Tj=85

✔ Min. Composition SNR of 25 [dB] @ Tj=100
✔ Robust Vs Flicker and Motion Artifacts
Key Technologies 2. Sub-pixel read-out
3. Pixel structure
© 2020 IEEE
A 0.50e-rms Noise 1.45μm-Pitch CMOS Image Sensor
with Reference-Shared In-Pixel Differential Amplifier
at 8.3Mpixel 35fps
Mamoru Sato1, Yuhi Yorikado1, Yusuke Matsumura1, Hideki Naganuma1,

Eriko Kato1, Takuya Toyofuku1, Akihiko Kato1, Yusuke Oike1
Presenter: Hongbo Zhu2
1Sony Semiconductor Solutions Corp.

2Sony Electronics Inc.
© 2020 IEEE 5.8: A 0.50e-rms Noise 1.45μm-Pitch CMOS Image Sensor with Reference-Shared In-Pixel Differential Amplifier at 8.3Mpixel 35fps
Outline
• Motivation
• Key technology
• Chip characteristics
• Conclusions
Motivation
An in-pixel differential amplifier is a promising technology
if its noise could be improved
HCG : Small FD cap. HCG : In-pixel amp. HCG : In-pixel
LCG : +Extra FD cap. LCG : Source follower differential amp.
LCG : Source follower
Switchable CFD_extra
Differential pair
conv. gain (CG)
technology
CFD_small
Noise Very good Very good Good Very good

Resolution Poor Good Good
PVT robustness Good Poor Good
Previous architecture
No extra transistors are needed in pixel to configure multiple CGs
Source follower (SF) In-pixel differential amplifier (DA)
J.Choi, ISSCC2012
Differential pair
Issue : noise of DA
Key technology : Reference-shared in-pixel differential amplifier (RSDA)
Issue : noise of DA
Random noise of DA increases due to an additional noise source
Column Column Column

k-2 k-1 k
Vout
VSL_REF VSL
σRef σRead
Reference COM Readout

pixel pixel
Random noiseDA≂ σRef2 + σRead2

Key technology : RSDA
Reference-shared in-pixel differential amplifier (RSDA)
reduces noise from the reference side to a negligible level
Column Column Column
Shared k-2 k-1 k
VSL_REF Vout
VSL_REF VSL
σRef σRead
Shared
COM
Reference COM Readout
pixel pixel
Random noiseRSDA≂ σRef2 N:Number of shared reference pixels

+ σRead2
N N = 3840
Reset operation
A negative feedback reset guarantees VSL voltage swing
by setting reset level to an optimal bias VRST
Shared VSL_REF
SELn-1
SELn
VRST RSTn-1
RSTn
VSL
TRGn-1
TRGn
RSTn-1
SELn-1
RSTn
SELn
VRST
TRGn-1 TRGn FDn-1
FDn-1 FDn
FDn
VSL Swing
Reference pixel Readout pixel
Shared COM
Reset level Signal level
Implementation
We can configure two selective CGs; RSDA and SF, with switches
in column circuit.
Column:k-2
Column:k-1
Column:k+1
Column:k+2
Differential pair
Column:k
Bottom part
Top part
Row decoders
Control logic
Row drivers
Pixel array Pixel array

Current sources
Shared VDD VRST
COM
Cu-Cu
Current sources Shared
Connection
VSL_REF
DAC Column ADCs
I/F Control logic
Switchable conversion gains
The sensor can switch CGs between 75μV/e- and 560μV/e-
12
Noise squared [mV2]
10
8
6
RSDA 560 μV/e-
4
2 SF 75 μV/e-
0
0 5 10 15 20
Signal [mV]
Random noise and frame rate
RSDA improves random noise by 30% compared to DA
1.40 70
Random noise [e-rms]
Max. frame rate [fps]

60
1.20 60
49 49
1.00 50
36
0.80 35 40
-30%
0.60 30
1.14e-rms
0.86e-rms
0.90e-rms
0.63e-rms
0.50e-rms
0.40 20
0.20 10
0.00 0
SF SF DA RSDA RSDA
CMS(M=2) CMS(M=2)
12-bit ADC 10-bit ADC
Random noise and frame rate
RSDA achieves 0.50e-rms random noise at 8.3Mpixel 35fps
by combining a correlated-multiple-sampling technique
1.40 70
Max. frame rate [fps]

60
1.20 60
49 49
1.00 50
36
0.80 35 40
0.60 30
1.14e-rms
0.86e-rms
0.90e-rms
0.63e-rms
0.50e-rms
0.40 20
0.20 10
0.00 0
SF SF DA RSDA RSDA
CMS(M=2) CMS(M=2)
12-bit ADC 10-bit ADC
Chip characteristics
Pixel size 1.45 μm (H) x 1.45 μm (V)
Num. of pixels 3840 (H) x 2160 (V)
Full well capacity 5800 e- at 60 C
9777 e- / lux-sec
Sensitivity
(Green pixel, 3200K light source with IR cut filter)
Operation mode SF RSDA w/ CMS (M = 2)
ADC resolution 12-bit 10-bit
Conversion gain 75 μV/e- 560 μV/e-
Random noise 1.14 e-rms @peak 0.50 e-rms @peak
Voltage swing at VSL 435 mV 200 mV
PRNU 0.86 % 2.5 %
Max. frame rate 60 fps 35 fps
Power consumption 450 mW 550 mW
Chip micrographs
- Back-illuminated stacked 1/2.8-inch CIS
- Supply voltage : 2.9V / 1.8V / 1.1V
7.97mm
I/F
DAC
Row decoders
Control logic
5.99mm
Pixel array Column ADCs
/drivers
Test circuits
Current sources
Top process : 90nm 1P4M Bottom process : 55nm 1P7M

Captured images
Sensor images taken under both dark and bright conditions
0.5lux, RSDA mode w/ CMS 1600lux, SF mode
F = 2.8, Exposure time = 1/30sec, γ = 2.0
Performance comparison
IEDM2018 VLSI2015 ISSCC2017 VLSI2015
Paper This Work
[6] [7] [2] [1]
Pixel pitch [μm] 1.45 1.5 1.1 11.2 5.5
H pixels 3840 720 360
8M 8M
V pixels 2160 700 1680
FWC [e-] 5800 13000 N/A 4100 76000
FWC [e-/μm2] 2759 5778 N/A 33 2512
Frame rate [fps] 35 30 7.2 32 14.5
Pixel rate [Mpixel/s] 290 240 5.8 16 8.8
Conv. gain [μV/e-] 560 200 110 172 232
0.50 0.80 0.66 0.44 0.46
@Peak @Peak @Peak @Peak
FOM [e-rms/MHz] 6.6 12 † 42 † 20 19
Pixel rate = Total number of pixels x Frame rate † V pixels 2160 is assumed
FOM = Random noise / (V pixels x Frame rate)
Conclusions
- A 1.45μm pixel, back-illuminated stacked 1/2.8-inch CIS
achieves readout noise of 0.50e-rms at 8.3Mpixel 35fps
and FOM of 6.6[e-rms/MHz]
- Reference-shared in-pixel differential amplifier (RSDA)

Improves readout noise by 30% compared to DA
- The sensor can switch CGs between 75μV/e- and 560μV/e-

without any additional transistors in the pixel
A 0.8V Multimode Vision Sensor for
Motion and Saliency Detection with
Ping-Pong PWM Pixel
Tzu-Hsiang Hsu*, Yen-Kai Chen*, Jun-Shen Wu, Wen-Chien Ting,
Cheng-Te Wang, Chen-Fu Yeh, Syuan-Hao Sie, Yi-Ren Chen,
Ren-Shuo Liu, Chung-Chuan Lo, Kea-Tiong Tang, Meng-Fan Chang,
and Chih-Cheng Hsieh
*Equally-Credited Authors (ECAs)
National Tsing Hua University, Hsinchu, Taiwan
© 2020 IEEE 5.9: A 0.8V Multimode Vision Sensor for Motion and Saliency Detection with Ping-Pong PWM Pixel
Outline
• Motivation
• System Architecture
• Key Technology
- In-pixel Ping-pong Frame Difference Circuit
- Multi-mode Readout Operation
• Measurement
• Conclusion
Motivation
• Energy-efficient image sensor for machine vision application
- Motion detection
High Speed Temporal Difference Imagers
- Saliency detection
Pedestrian Detection Collision Avoidance

Concept of Multimode Vision Sensor
• Real-time frame difference operation w/o sacrificing frame rate
• Supporting multi-mode operation
- Image Capture (IC) 8-bit mode
- Frame Difference (FD) 8-bit mode
- Frame Difference (FD) 2-bit mode with ON/OFF event
- Saliency Detection (SD) mode of sub-block event summation
IC 8-bit mode FD 8-bit mode FD 2-bit mode SD mode

Temporal Difference Pixel Overview
• Event-based pixel • Frame-based pixel
B. Son, ISSCC 2017 / C. Li, VLSI 2019 T. Ohmaru, ISSCC 2015
 Continuous pixel contrast monitoring  Inherent integration time mitigate pixel

w/o signal integration gain stage
 Extra gain stage in pixel frontend  Limited event rate and redundant output
 Event decision comparator  Event decision comparator
System Architecture
Ping-Pong *PWM Pixel

1. PWM Conversion
+
2. Event Decision
Saliency Detection
Event Summation
Single-Slope ADC
© 2020 IEEE 5.9: A 0.8V Multimode Vision Sensor for Motion and Saliency Detection with Ping-Pong PWM Pixel *PWM: pulse-width-modulation
Pixel Prototype
• Inverter-based comparator with auto-zero reset
• Global shutter control RST_PD and TX
• Series capacitor CM for signal storage and coupling
“Storage”
“Coupling”
α=CM/(CM+CPAR) “Differencing”
Pixel Prototype: Issue 1
• Issue 1: Frame difference error from conversion gain mismatch
∆VF1 = IPD1·∆T/(CPD+CM)
∆VG = α(∆VF1 - ∆VF2)
∆VF2 = IPD2·∆T/(CPD+CM||CPAR)
Pixel Prototype: Issue 2
• Issue 2: Frame rate degradation by 0.5×
Circuit Implementation & Operation
Key Technology 1 :
In-pixel Ping-pong Frame Difference Operation
Solution: Ping-Pong Pixel Structure
• EVEN/ODD readout path in each pixel operate in storage, and
differencing phase separately
Ping-Pong Pixel Operation
• ODD: differencing - VGO floating ∆VF1 (CPD + CMO||CPAR,O+CME)-1
• EVEN: storage - VGE fixed
ODD: differencing
EVEN: storage ODD readout

• ODD: storage - VGO fixed ∆VF1 (CPD + CMO||CPAR,O+CME)-1
• EVEN: differencing - VGE floating ∆VF2 (CPD + CME||CPAR,E+CMO)-1
ODD: storage
EVEN: differencing ODD readout EVEN readout

improved by
• ∆VF1 (CPD + CM)-1 ping-pong operation ∆VF1 (CPD + CMO||CPAR,O+CME)-1
• ∆VF2 (CPD + CM||CPAR)-1 ∆VF2 (CPD + CME||CPAR,E+CMO)-1
ODD: storage
EVEN: differencing ODD readout EVEN readout

• Symmetric layout: same conversion gain ∆VF1 (CPD + CMO||CPAR,O+CME)-1
• EVEN/ODD: avoid frame rate degradation ∆VF2 (CPD + CME||CPAR,E+CMO)-1
ODD readout EVEN readout ODD readout

Circuit Implementation & Operation
Key Technology 2 :
Multi-mode Readout Operation
Frame-Difference Mode (FD)
• Series capacitor stores FD signal on VG
• Compared with ramps down threshold voltage VTH_RAMP
• PWFD direct quantized by counter into 8-bit result
Frame Difference 2-bit Mode
• ON/OFF event determined by constant threshold VTH_H and VTH_L
- ON: ∆VF2 > ∆VF1 “00”
- OFF: ∆VF2 < ∆VF1 “11”
- NO: ∆VF2 ≈ ∆VF1 “01”
Saliency-Detection Mode (SD)
• 8 rows selected simultaneously for sub-block event summation
• Mimic current sources controlled by ON/OFF event level
• Current integration and quantization
8 rows
activated
Chip Micrograph & Image Mode
• TSMC 0.18μm standard Global Shutter Rolling Shutter
• Chip size: 2.0mm × 2.1mm
• Array resolution: 64 × 64
• Pixel pitch: 15μm
Distortion
Frame Difference Mode
FD Mode FD Mode
IC Mode 2-bit 8-bit
Lost Information
“01” “00” “11” “01”
Recovered Information
Saliency-Detection Mode
FD Mode: ON/OFF Event
Outer Circle:
Moving Object
Counterclockwise
SD Mode: Sub-block “OFF” Event

Rotating
SD Mode: Max. Sub-block of “OFF” Event

Center Circle:
Static Object
Chip Performance
Conclusion
• Challenge
- Low-power and high-speed frame difference sensor
- Frame rate degradation by 0.5×
- Frame difference error from conversion gain mismatch
• Achieved Result
- 0.8V multi-mode vision sensor
- Ping-pong PWM pixel circuit
- High speed motion detection @ 510fps
- Sub-block saliency detection @ 890fps
Thank you!
QA
A 1280 x 720 Back-Illuminated Stacked Temporal
Contrast Event-based Vision Sensor with 4.86μm
Pixels, 1.066GEPS Readout, Programmable Event
Rate Controller and Compressive Data
Formatting Pipeline
Thomas Finateu1, Atsumi Niwa2, Daniel Matolin1, Koya Tsuchimoto2, Andrea
Mascheroni1, Etienne Reynaud1, Pooria Mostafalu3, Frederick Brady3, Ludovic
Chotard1, Florian LeGoff1, Hirotsugu Takahashi2, Hayato Wakabayashi2, Yusuke Oike2,
Christoph Posch1
Prophesee, Paris, France

1
2 Sony Semiconductor Solutions Corporation, Atsugi, Japan
3 Sony Electronics Inc., Rochester, NY
© 2020 IEEE 5.10: A 1280x720 Back-Illuminated Stacked Temporal Contrast Event-based Vision Sensor with 4.86μm Pixels, 1.066GEPS Readout, Programmable
International Solid-State Circuits Conference Event Rate Controller and Compressive Data Formatting Pipeline 1 of 41
Outline
• Event Sensors - Motivation and Introduction

• Pixel Architecture and Readout
• Chip Architecture
• Results
• Applications examples
• Conclusion
Outline

• Results
• Application examples
• Conclusion
Motivation
How to achieve efficient Machine Vision?

 Where need for speed meets uncontrolled lighting conditions
 automotive, drones & UAVs, aerospace, …
 Where high-speed motion needs real-time operation and closed-loop control
 industrial, robots, drones & UAVs, …
 Where power budget, compute capabilities and bandwidth are limited
 mobile, sensor networks, IoT, always-on, surveillance, edge devices…
Conventional Image Sensors
What’s the problem with video and frames?
 Motion acquisition based on static images
 All pixels are sampling at a given and fixed rate
 But scene dynamics vary
 Any chosen frame rate is wrong
https://commons.wikimedia.org/wiki/File:The_Horse_in_Motion_high_res.jpg
“Sallie Gardner”, 1878
Over-sampling:
 Redundant useless data
 Known from previous acquisition
 Need to acquire, transmit, store,
process,…
Under-sampling:
 Motion blur
https://commons.wikimedia.org/wiki/File:Baseball_pitching_motion_2004.jpg
 Large displacement
Temporal Contrast Event Sensing
 Individual pixels autonomously respond to illuminance changes
 Sample asynchronously on amplitude:
Level-crossing sampling or Asynchronous Delta Modulation (ADM)
 Programmable contrast threshold – threshold crossing = "event"
 Change polarity captured and recorded
Event Sensors Summary
 Efficient acquisition of visual information
 inherent data compression
 focus on relevant dynamic data
 High-dynamic range due to autonomous pixel
 High-speed continuous-time motion capture
 fast pixel reaction times
 high resolution timestamping
 Time is another dimension of information

for vision analysis tasks and
efficient machine-learning approaches
Outline

• Results
• Conclusion
Pixel Architecture
I
t
Iph
PD
 Partially pinned photodiode
Pixel Architecture
I V
t t
Iph log I/V
Vlog
PD

 Subthreshold MOS based logarithmic photocurrent-to-voltage conversion
Pixel Architecture
I V V
t t t
Iph log I/V ADM
t1 t2 t3 t4
Vlog VΔM
PD
C
 ADM / level-crossing sampler
Pixel Architecture
VCON
I V V t
t1 t2
t t t CON
Iph log I/V ADM
t1 t2 t3 t4
COFF
Vlog VΔM
VCOFF
PD t
t3 t4
CTRLADM
 Voltage comparators (for both polarities)
Pixel Architecture
VCON
I V V t
t1 t2 Interface
t t t CON
& State ackY
Iph log I/V ADM
t1 t2 t3 t4
COFF
Vlog VΔM Logic
VCOFF reqY
(ISL)
PD t
t3 t4
ONev OFFev
CTRLADM
 Partially pinned photodiode reqX ON reqX OFF
t1 t2 t3 t4 t
 Voltage comparators (for both polarities)
 Logic with ADM control and interface to the read-out periphery
Stacked Pixel Design
Previous generation: This design:
180nm FSI CIS 90nm BI CIS
15µm pitch on 40nm CMOS
25% fill factor 4.86µm pitch
>77% fill factor
log I/V
Top chip
VDD VDD
Vbias
 Stacked fabrication process Vlog
 Pixel-level Cu-Cu connection Cu-Cu connection
 PD + NMOS on top CIS

PD
 All other pixel circuitry (~50T) on bottom CMOS
Pixel Array Readout Architecture
 Async-to-sync interface supervised by state
machine inside the digital core
 Time-out column ensures correct timing to latch

event data by the digital core
 Precision timestamps attached to the events
 Pipeline row operation

Next row selected while processing current row
In-Pixel Readout Interface
 Input latches to reduce power in reqY
case of slow comparator
switching and to prevent ringing reqX ON
CON reqY ON
S Q
E
R
reqX OFF
COFF
S Q
reqY OFF
E
R
ackY
CTRLADM
 Input latches to reduce power in reqY
switching and prevent ringing reqX ON
CON reqY ON
 Gated latches (K) avoid late S Q
request generation and event E

R
loss
reqX OFF
COFF
S Q
reqY OFF
E
R
ackY
CTRLADM
 Input latches to reduce power in
reqY
Readout req/ack handshake

switching and prevent ringing
reqX ON
CON
 Gated latches (K) avoids late S Q
reqY ON
request generation and event E

R
Pixel reset
loss reqX OFF
COFF
 Only pixels with events stored in S Q
reqY OFF
E
the input latches, reset R
themselves (CTRLADM) ackY
Thereby removing the need for CTRLADM
column-wise acknowledge
signals
Outline

• Results
• Conclusion
Chip Top Block Diagram
 On chip power management
 higher level of integration
 reduced bill of materials
 Bias generator
 for pixel biasing
 Analog compensation over PVT
 Configurable ROI
 any crop or sub-sampling configuration
 LUT-based address filter

 to remove selected events, e.g., from
defective pixels
Event Signal Pipeline (ESP)
Digital core
reqX ON<1279:0>
Event Rate Event Data Event Output

reqX OFF<1279:0> Digital Readout
Controller Formatter Interface 16-bit CMOS IF
addrY<9:0>
& time stamping (ERC) (EDF) (EOI)
Req
Ack Event Signal Processing Interface
Pixel and Vector Readout
The digital core readout:
 Samples event row data
 Timestamps at 1-µs resolution
Px Px Px Px Px Px Px Px Px Px Px Px Px Px Px Px
1280 pixels 0 1 2 29 30 31 32 33 34 61 62 63 1216 1247 1248 1279
40 vectors Vector addr 0 Vector addr 1 Vector addr 38 Vector addr 39
 1280 pixels  40 vectors of 32 pixels

 Each pixel or vector requires the following information
 Timestamp | Y address | Polarity | X address
Event Data Formatter
Assuming a full row of same polarity = 1280 events
And CMOS I/F running at 100 MHz
1280 evts Encoded on 32 bits 1280 evts x 2 (16 bits) x 10 ns
Pixel
40960 bits read out in 25.6 us  32 bits/evt 50 MEPS
Pixel
40 evt
Encoded on 64 bits 40 vectors x 4 (16 bits) x 10 ns
vectors
Vector
Pixel
40 evt
Encoded on 64 bits 40 vectors x 4 (16 bits) x 10 ns
vectors
Vector
40 evt
Encoded on 48 bits 40 vectors x 3 (16 bits) x 10 ns Lossless compressed format
vectors
Vector+ Removes redundant time stamps
1920 bits read out in 1.2 us  1.5 bits/evt 1066 MEPS and Y addresses
Event Rate Controller (ERC)
Event Interest Event
Delay Horizontal Vertical Temporal
Readout Counter Grid Counter Output
FIFO drop rate drop rate drop rate
Input Drop rate Output
Event
Rate
Controller
 ERC caps output event rate to target event rate
Target
 Target rates from 5 kEPS to 1066 MEPS
Event
rate
 Various drop strategies
 Random temporal
 Spatial: horizontal / vertical
 ROI based: 40-by-23 32×32 pixel zones
For each zone, 6-bit weighting on drop rate
Outline

• Results
• Conclusion
Background Activity (Noise)
Peak background rate
Stable Light level
time
Start ON events Stop
recording OFF events recording
 False-positive contrast detection

events (BackGround rate) in Hz
(events/pixel/s) vs illuminance
 Recorded between 1mlux and
100klux at focal plane.
 Peak is 8.3Hz at 0.6lux.
Contrast Sensitivity (S-Curve)
Light High-light (HL)
level Low-light (LL)
time
Contrast calculation

Linear  ·
Log  ln ·
 Contrast step response probability S-

curves at different light levels:
 Nominal contrast threshold (NCT) is
contrast at p=0.5 response probability NCT
at 10lux : 15.7%.
Contrast Detection Probability (CDP)
Light High-light (HL)
level Low-light (LL)
time
 Contrast detection probability (CDP)

curve for contrast c: values of S-
curves at e.g. c = 40% (~ 2.5x NCT).
 Low-Light Cut-Off, High-Light Cut-Off
where CDP curve drops below p=0.5
at low and high light respectively
 LLCO: 40mlx
 HLCO: not seen up to 100klx LLCO HLCO ?
Output Readout Interface
ERC disabled
Theoretical peak event rate:
1280 events / 1.2 us = 1066MEPS
ERC
Event Rate Controller (ERC)
enabled target  368 MEPS
Peak Internal Event Rate
Peak event rate = 2918 MEPS
Internal = out of the
pixel array into the
event pipeline
 3 cycles to process a row of 40 vectors

 Theoretical internal max event rate:
32 events / 10 ns x [ 40 / (40+3) ]
= 2976 MEPS
Power Consumption Vs Event Rate
73 mW
32 mW 300 MEPS
100 kEPS
Outline

• Results
• Conclusion
Outline

• Results
• Conclusion
Chip Specifications and Comparison
This work Samsung Celepixel Inivation
(ISSCC 2017) (ISCAS 2017) (IISW 2019)
Technology
Pixel circuitry array
90nm BI CIS + 90nm CIS BSI 180nm CIS 65nm CMOS
1280 x 720 40nm CMOS
Pixel array
1280 x 720 Supply voltages 2.5, 1.1 2.8, 1.2 3.3 1.2
Resolution 1280 x 720 640 x 480 768 x 640 132 x 104
Pixel size (μm2) 4.86 x 4.86 9x9 18 x 18 10 x 10
Fill Factor >77% 20% 9% 20%
bias generation
+ power management Power 100kEPS 32 27 - 0.25
(mW) 300MEPS 73 50 - 4.9
digital core
Power/Pixel (nW) 35 88 - 18
analog

1280 x 720
Energy/event (pJ) 137 77 - 26
Max event rate (MEPS) 1066 300 200 180
digital core + memory Contrast Sensitivity NCT 11% (15.7%) 9% (19%) - -
Dynamic Range >124dB >80dB (90dB) 120dB -
Summary
 HD 1280x720 1/2" event-based sensor
 Back-Illuminated CIS with partially
1280 x 720pinned photodiode
 Stacked on logic CMOS with pixel-level Cu-Cu connection
 4.86μm pixel pitch
 On-chip power management
 Row-level time-stamping at 1-μs resolution
 High-throughput high-temporal precision readout
(close to 3 GEPS out of the array, 1.066 GEPS out of the chip)
 Lossless effective bit per event compression
 Programmable event rate controller / limiter
Acknowledgments
Prophesee, Paris, France

Naoyuki Hanajima, Tahar Hitana, Adrien Albouy, Moataz Kadry,
Guillaume Schon, Patrice Perrin, Vitor Schwambach, James Wightman,
Raphael Vansebrouck, Stephane Valente, Xavier Lagorce, Sylvain
Brohan, Gerd van den Branden, Jean-Luc Jaffard
Sony Semiconductor Manufacturing Corporation

for chip fabrication

ISSCC2020-05 - Visuals Imagers and ToF Sensors

Uploaded by

Copyright:

Available Formats

You might also like

ISSCC2020-05 - Visuals Imagers and ToF Sensors

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ISSCC2020-05 - Visuals Imagers and ToF Sensors

Uploaded by

Copyright:

Available Formats

ISSCC 2020

Toshiba1, Kawasaki, Japan

• Motivation and Target

Tesla / ModelS Waymo / Pacifica General Motors / Cruise

General Benchmark of Depth Sensors

→Our previous work[ISSCC’18] succeeds 200m long range DM

Our Goal: Realization of higher-pixel-resolution LiDAR system

• Motivation and Target

 SiPM PDE/gain is high  High resolution

e.g. 20pix. are acquired in one laser shot

The more pixels at once,

SR SiPM Short-Range(LR) AFE

SR SiPM Short-Range(LR) AFE

→Low-Resolution Can be merged by using

SR SiPM Short-Range(LR) AFE

SR SiPM Short-Range(LR) AFE SiPM LR/SR-unified AFE

• Motivation and Target

• Simultaneous conversions(Voltage: ADC/

Integ. Time Data

Integ. Time Data

There are TWO issues to realize DDC

Integ. Time Data

VCO non-ideality degrades ADC SNDR and image quality

• Motivation and Target

TDC Issue :VCO Freq. Modulation and Drifts

Integ. Time Data

VCO Frequency modulation, drifts

counter Integ. Time Data

counter Integ. Time Data

Pulse Coarse ToF: 4

VCO freq. ignored

counter Integ. Time Data

Pulse Coarse ToF: 4 15

9 Fine ToF: 9/15=0.6

counter Integ. Time Data

Pulse Coarse ToF: 4 15

9 Fine ToF: 9/15=0.6

TDC Issue :VCO Freq. Modulation and Drifts

Cal. must be conducted to ch.-by-ch.

Cal. must be conducted to ch.-by-ch.

• Pulse density of 1-b cell is modulated by VCO freq.

• Pulse density of 1-b cell is modulated by VCO freq.

Pulse-Density Modulator(PDM) fsignal fVCO

freq.- High-freq. spur

fsignal fVCO 2fVCO

TDC Issue :VCO Freq. Modulation and Drifts

• Motivation and Target

Hi-Res. LiDAR detects 2x smaller object

Standard deviation [m]

• 225m Long Range with <0.25%-precision.

5.1: A 240×192Pixel 10fps 70klux 225m-Range Automotive LiDAR SoC

• Higher precision by SR-DM in <15m.

5.1: A 240×192Pixel 10fps 70klux 225m-Range Automotive LiDAR SoC

Train Traffic sign

Signal Intensity Image

Distance range [m] 225 200 100

AFE area(1ch) [mm2] 0.027 0.134** 0.144**

• World’s-first DDC, consolidating the functions of ADC and TDC

AFE area(1ch) [mm2] 0.027 0.134 0.144