Embedded Linux Conference Europe, October 2020

From the Camera

Sensor to the User
the Journey of a Video Frame

Maxime Chevallier
maxime.chevallier@bootlin.com

© Copyright 2004-2020, Bootlin.

Creative Commons BY-SA 3.0 license.
embedded Linux and kernel engineering
Corrections, suggestions, contributions and translations are welcome!

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 1/1
 Maxime Chevallier

I Linux kernel engineer at Bootlin.

I Linux kernel and driver development, system integration, boot
time optimization, consulting. . .
I Embedded Linux, Linux driver development, Yocto Project &
OpenEmbedded and Buildroot training, with materials freely
available under a Creative Commons license.
I https://bootlin.com
I Contributions:
I Worked on network (MAC, PHY, switch) engines.
I Contributed to the Marvell EBU SoCs upstream support.
I Worked on Rockchip’s Camera interface and Techwell’s
TW9900 decoder.

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 2/1
 Preamble - goals

I Discover the hardware components and protocols involved in

video cameras
I Understand how all these components chain together and how
to configure them
I See various real-life hardware designs and use-cases

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 3/1
 Embedded Linux Conference Europe, October 2020

Video Acquisition
Hardware
Maxime Chevallier
maxime.chevallier@bootlin.com

© Copyright 2004-2020, Bootlin.

Creative Commons BY-SA 3.0 license.
embedded Linux and kernel engineering
Corrections, suggestions, contributions and translations are welcome!

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 4/1
 Necessary components

Optical elements Sensor Signal transcoding SoC

I Optical to Electrical conversion : Lenses and Sensors

I Signal transmission : Digital and Analog protocols
I Signal handling : Controllers and Signal Transcoders
I Image transformation : ISPs, Image Encoders/Decoders

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 5/1
 Image capture

Image acquisition setup

LED Flash

Sensor

Lens and Voice-Coil

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 6/1
 Lens

I Lens :
I Controls the focus of the incoming light
rays
I Can be adjusted for manual or
auto-focus
I Voice Coil Actuator :
I Lens position is adjusted by a Voice Coil
actuator
I A wire coil is attached to the Lens
I The Lens assembly sits in a static
magnetic field
I The coil is drived by a DAC providing
adjustable current
I Mostly controlled through I2 C
I MEDIA_ENT_F_LENS

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 7/1
 Flash

I High-power LED driver

I Sometimes include a privacy indicator
I Controlled through I2 C
I MEDIA_ENT_F_FLASH

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 8/1
 Sensor

Sensor array Amplifier ADC ISP FIFO Data output

Control plane

I Converts an optical signal to an analog electrical signal

I Uses CCD or CMOS technologies
I Include internal ADCs and amplifiers
I Sometimes include an Image Signal Processor
I Data plane uses a dedicated protocol
I Control plane mostly uses I2 C

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 9/1
 Digital Sensors

I Sensors acquire 3 color components, arranged in a grid

I These colors are sometimes sent raw : RGGB, RGBG, etc.
I Conversion from raw sampling to pixel value : debayering
I Also called demosaicing, can be done in-situ
I On basic sensors, this need to be done in the Host
I Possibly costly operation, depending on the algorithm
- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 10/1
 Digital transmission protocols : Raw Parallel

I Simple approach : Transmit data and

sync signals pixel clk
I Consists of the following lines : HSYNC
I Parallel data lines : 8 to 12 bits

Sensor
VSYNC
I Pixel Clock : Ticks every pixel sent
I HSYNC : Toggles on line end data
I VSYNC : Toggles on frame end lines

I Often needs post-processing, such as

demosaicing

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 11/1
 Digital transmission protocols : CCP2

Compact Camera Port 2

I Serialized interface
I Sync signals embedded in data clk+ / clk-

Sensor
I Uses differential pairs to transmit data
d+ / d-
and clocks
I PHY Layer is based on subLVDS
I 4 pins needed : 2 for clk and 2 for data
I Up to 650 mbps

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 12/1
 Transmission protocols : MIPI CSI

Camera Serial Interface

I Standard from the MIPI Alliance

I Multiple layers defined :
1. PHY : Physical transmission
2. Lane Management : Lane distribution and merging
3. Low Level Protocol : Checksuming, Error Correction
4. Application : Pixel to Byte conversion
I CSI-2 is the most used
I CSI-3 : Bi-directional protocol

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 13/1
 MIPI D-PHY Layer

I Serialized interface
I Sync signals embedded in data
clk+ / clk-
I Uses differential pairs to transmit data

Sensor
and clocks
dp1 / dn1
I Minimum 4 pins : 2 for clk and 2 per lane dp2 / dn2
... ...
I Up to 4 lanes
I Up to 6 Gbps

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 14/1
 MIPI C-PHY Layer

I Serialized interface
I Sync signals and clock embedded in data
I 3-levels signals : High, Med and Low A/B/C

Sensor
I 3 lines per lane
I 16 bits transmitted over 7 symbols (in
... ...
quinary)
I Up to 3 lanes
I Up to 41 Gbps

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 15/1
 Analog transmission protocols

NTSC

SECAM

PAL or PAL/SECAM

no info

I Video Broadcasting protocols

I Designed for transmission over an analog media
I Decomposes video into components : Y, Cr, Cb
I Y : Luminance ( Black and White image )
I U, V : Chrominance ( Color information )
I PAL : Europe
I NTSC : USA, Japan
I SECAM : France, Eastern Europe, Russia

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 16/1
 Interlacing

I Increase the perceived framerate with the

same bandwidth
I Transmit the odd lines, then the even lines
I Each part is called a field
I Requires a compatible display
I Else the video need to be deinterlaced
I Requires some signal processing to get Interlacing artifacts
correct results

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 17/1
 Analog to Digital Transcoders

analog video
External signal
Decoder SoC
video source

I Converts an analog video signal into a digital signal

I Can support multiple input standards
I Can embbed a small ISP for simple cropping/scaling
I Converts into a digital video standard :
I BT.656 for PAL and NTSC standards
I BT.1120 for higher resolution standards
I Parallel or Serial interfaces supported by BT.* standards

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 18/1
 Host Interface

SoC

Camera Interface

PHY RAM

ISP Scaler ...

Image processing

I Hardware blocks located in the SoC

I Has lots of different features depending on the Hardware :
I PHY layer support
I Image processing (simple or advanced), including :
I Scaling, Cropping
I Deinterlacing
I Pixel format conversion
I Stores the frame into buffers using DMA
I The V4L2 framework and the media controller API
supports these blocks

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 19/1
 Image processing

I Cropping : Remove areas from the image

I Easy to perform
I Scaling : Resize the image
I Can require complex algorithms
I Deinterlacing : Recompose an interlaced stream
I Joining fields is easy
I Removing artifacts can be complex
I Pixel format conversion
I Debayering : Convert raw sensor data to a usable image format
I Colorspace conversion : RGB to/from YUV
I 3A Processing
I Auto exposure : Adjust the brightness, control the sensor gain
I Auto focus : Adjust the focal point, control the lens position
I Auto white balance : Correct the colors
See the talk on ISP drivers by Helen Koike earlier today !

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 20/1
 Example : Nokia N900

I Smartphone with full Linux support

I Based on TI OMAP3 SoC
I Has a Flash LED driver
I Voice Coil controlled through a DAC
Nokia N900
I CSI Sensor controlled through I2C
flash/strobe Flash LED Driver
I DTS found in i2c
ADP1653

CPU Lens Voice Coil Driver

arch/arm/boot/dts/omap3-n900.dts OMAP3430
i2c
AD5820

Sensor
CSI et8ek8

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 21/1
 Example : Custom Product

PAL Video source Rockchip PX30

BT.656 //
PAL/NTSC Decoder
or VIP
tw9900

NTSC Video source

i2c
analog IN

I Based on Rockchip PX30 SoC

I Remote video source through an analog signal : PAL or NTSC
I Decoded through a tw9900, which auto-detects the standard
I BT.656 Parallel bus to transmit the digital signal
I Uses the VIP Camera Interface (Upstreaming in progress)
I Fields are simply joint, not a full de-interlacing

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 22/1
 Linux Support

I All the above components and more are supported by Linux

I Wide variety of topologies, challenging to have a full-featured
infrastructure
I V4L2 Gives the infrastructure to support Video Cameras
I V4L2 Also deals with buffer management and interaction with
userspace
I The Media Controller API allows configuring each block
through subdevs
I Complex devices can be handled in userspace with libcamera
I Welcoming community :)

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 23/1
 Conclusion

I The number of technologies involved can be overwhelming

I Old analog terminologies and technologies still apply today
I However, Linux support is pretty good
I V4L2 and the media controller API allow complex
use-cases

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 24/1
 Thank you!
Questions? Comments?
Maxime Chevallier — maxime.chevallier@bootlin.com

Slides under CC-BY-SA 3.0

https://bootlin.com/pub/conferences/2019/elce/chevallier-network-
classification-offload/

- Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 25/1