Video Processing on JLR NGI System

David Naegle
Graphics Hardware Architect
Automotive Systems Division

Intel Confidential
  INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY
INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL
ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS
INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR
OTHER INTELLECTUAL PROPERTY RIGHT.
 A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU
PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES,
SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES
AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY
WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR
WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
 Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any
features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or
incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
 The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications.
Current characterized errata are available on request.
 Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
 Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go
to: http://www.intel.com/design/literature.htm.
 Any software source code reprinted in this document is furnished for informational purposes only and may only be used or copied and no license, express or implied, by
estoppel or otherwise, to any of the reprinted source code is granted by this document.
 This document contains information on products in the design phase of development.
 Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor
families. Go to: http://www.intel.com/products/processor_number/.
 Code Names are only for use by Intel to identify products, platforms, programs, services, etc. (“products”) in development by Intel that have not been made
commercially available to the public, i.e., announced, launched or shipped. They are never to be used as “commercial” names for products. Also, they are not intended to
function as trademarks.
 Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
 *Other names and brands may be claimed as the property of others.
 Copyright © 2013, Intel Corporation. All rights reserved.

Intel Confidential
 Agenda
 ISP Overview
 ISP Requested Formats
 KP Video-In SW Architecture
 How to develop device drivers for V4L2
 GEM Buffer sharing between EMGD and ISP
 Early Video issues
 Resolution Scaling
 Instrument Cluster on 2nd HDMI Output
 Audio/Video Synchronization
 Wrap-up

Intel Confidential
 Image Signal Processor (ISP)
(MIPI-CSI Channel Processor)

Intel Confidential
 Why use the ISP ?
• Example: Human eye is better at
judging what is “white” under
different light, but Digitial cameras
have difficulty
• Use DSP to produce a more visually
pleasing image (Auto White Balance)
• Offload the GPU/CPU
• Optimized processor (VLIW)

Good in theory, but for KP 2.0, we just want to capture

video images. Our driver is not mature enough for this
kind of video enhancement on the fly.

Intel Confidential
 ISP blocks – simplified, emphasis on I/O

MIPI-CSI Receiver ISP

DDR

Intel Confidential
 Buffered Sensor mode – Frame Capture
MIPI-CSI Receiver ISP

DDR

Intel Confidential
 Buffered Sensor mode – Frame Acquire
MIPI-CSI Receiver ISP

DDR

Intel Confidential
 ExtMem mode – Triple stream capture
MIPI-CSI Receiver ISP

DDR

Intel Confidential
 ExtMem mode – Triple stream capture + Acquire
MIPI-CSI Receiver ISP

DDR

Intel Confidential
 ExtMem mode – Data path
MIPI-CSI Receiver ISP

DDR

Intel Confidential
 ExtMem mode – Data path width
MIPI-CSI Receiver pixels ISP

32-bit raw MIPI (packets)

DDR
256-bit raw MIPI (words)
shared bandwidth

Intel Confidential
 Data path: Capture Parallel, Acquire Sequential
MIPI-CSI Receiver C B A ISP
A
B
C sequential

parallel A B C
A DDR
B
C
parallel

Intel Confidential
 Input System - Configuration
MIPI-CSI Receiver ISP

Receiver (static)

Routing (static)

DDR
Input buffer partitioning

Intel Confidential
 Requested Formats – JLR Camera
RGB888 960x540 at RGB888 Intel confirmed that the ISP/CSS FW works
Camera MIPI2(1lane) 960x540@30fps, without LineStart/LineEnd tokens in the JLR requirement
30fps, progressive
progressive MIPI-CSI stream.
RGB888 Interim, to support
Basic functionality on RGB888 800x480 at
MIPI2(1lane) 800x480@60fps, Bosch validation testing
all ports. 60fps, progressive
progressive
YUV4:2:2 This format exactly matches the format Interim, to support
Basic functionality on YUV4:2:2 800x480 at
MIPI2(1lane) 800x480@30fps, which is working on Alpine Valley with the Bosch validation testing
all ports. 30fps, progressive progressive Annville camera module.
RGB888 Interim, to support
Video-in via GMSL RGB888 800x480 at
MIPI2(1lane) 800x480@30fps, Allows Bosch to drive MIPI2(1lane) with Bosch validation testing
interface to camera port 30fps, progressive progressive GMSL to camera port.
RGB888
Surround Camera,
RGB888 1280x800 MIPI2(1lane) 1280x800@30fps, to be confirmed: 30fps is all that will fit in JLR requirement
Supserset Chipset progressive 1 MIPI-CSI data lane at this resolution.

Intel Confidential
 Overview of V4L2 ISP driver and V4L2 sub devices

V4l2 sub devices

Intel Confidential
 JLR NGI Component Diagram

Intel Confidential
 Generic File operations required by
V4L2 Video Capture Drivers
 Open
 Release
 MMAP
 IOCTL
 Poll

Intel Confidential
 V4L2 Media Entity

CE Device

CE Device

Camera

Media Entity is the enhanced feature of the V4L2 Sub Device Framework. Each
V4L2 Sub device can be represented as an Entity.
• An entity is a basic media hardware building block.
• A pad is a connection endpoint through which an entity can interact with other
entities.
• A link is a point-to-point oriented connection between two pads, either on the
same entity or on different entities.

Intel Confidential
 ISP-Gfx DMA Buffer Sharing

Intel Confidential
 ExtMem mode – Data path width
MIPI-CSI Receiver pixels ISP

32-bit raw MIPI (packets)

DDR
256-bit raw MIPI (words)
shared bandwidth

Intel Confidential
 Buffered Sensor mode – Acquire Frame
MIPI-CSI Receiver ISP

To GEM Buffer

DDR

Intel Confidential
 Buffered Sensor mode – Write pixels to GEM buffer
MIPI-CSI Receiver ISP

To GEM Buffer

DDR

Intel Confidential
 JLR NGI Component Diagram

Intel Confidential
 JLR NGI End-to-End Camera Data flow

Intel Confidential
 JLR NGI Component Diagram

Intel Confidential
 Kendrick Peak Fast
Boot Sequence

Intel Confidential
 Wayland Compositor Architecture for JLR NGI

Intel Confidential
 Wayland Compositor Architecture for JLR NGI (early Camera)

Intel Confidential
 Instrument Cluster on
Second HDMI Output

Intel Confidential
 Resolution Scaling

 Can be done either in ISP or GPU

– Slightly off-loads GPU if done in ISP
– Less flexible to maintain if done in ISP (?)
– Probably only makes sense to do in ISP if NO other processing needed in Camera
app
 GPU uses OpenGL-ES Textures
 ISP uses Vector Processor
 Memory BW is approximately the same
 If Camera App must display in two different views (e.g., for Center Stack and
Instrument Cluster), it’s easiest to do scaling in GPU.

Intel Confidential
 A/V Sync: How to do it?

HDMI or MHL with A/V phoenix I2S audio

Valleyview
From UIP MIPI-CSI x2

Intel Confidential
 Audio & Video paths

Audio Subsystem

Video
I2S Audio

Intel Confidential
 Opens

 DMA Buffer Sharing

– Not part of Beta release; schedule is WIP
 Concurrent Streams
– Not part of Beta release; schedule is WIP
 V4L2-compatible Drivers for ADI bridge parts
– Ideally, STC develops these
 Audio/Video Sync for CE devices
– Audio Subsystem contains mechanism for up to 100msec audio delay
– Match to latency of CE Media app (equivalent of Camera App) video delay
– If delay through CE Media app is variable, it gets harder to do…
Intel Confidential
Intel Confidential