Mouser Electronics eBook

FPGA Design
eBook
By Adam Taylor

Liu zishan / shutterstock.com

 Mouser Electronics eBook

Contents
• Introduction 3
• What is a Field Programmable Gate Array (FPGA)? 3
• Benefits of Programmable Logic (PL) 3
• FPGA Building Blocks 3
• FPGA Design 4
• Device families 5
• High end solutions 6
• Toolchains 6
• Vivado Design Suite 6
• Application Software Creation 8
• Acceleration 8
• AI and Beyond 9
• Embedded Processing 9
• Embedded Processors 10
• Soft-Core Processors 11
• Big - Little Approach 13
• Communicating Internally and Externally 13
• Internal Data Movement 14
• Programmable Logic Applications 15
• Test Equipment 15
• Automotive and ADAS 15
• Cloud Computing 17
• Industrial 17
• Deep Dive Application - Creating Embedded Vision Systems 18
• Elements of a PL Image Processing System 19
• Software Defined Image Processing 19
• Conclusion 20
• About the author 21

2
 Mouser Electronics eBook
What is an FPGA?
Introduction
Since their introduction in the
mid-1980s, Field Programmable
Gate Array devices have moved
from providing the developer
with the ability to integrate several
discrete logic functions, e.g. glue
logic, to becoming truly the heart
of the system.
Modern FPGA devices and tools
are unrecognizable to those frst
introduced; they contain not only
traditional programmable logic
resources but also high-performance
embedded processors, dedicated
interfaces and memory controllers,
and IO structures capable of
providing multi-gigabit data rates.
This eBook explores modern FPGAs,
how they are programmed and
developed, and examines various
applications, including a deep-dive
into image processing.
Input E
Input A
A
Input B
B
Input C
C
Input D
D
LUT
Figure 1 – Simple LUT Structure
For example, vision and signal processing
and RADAR.
The world we inhabit is analogue. However,
Traditionally programmable logic devices
digital processing enables us to experience
have been available in two device classes:
and interact with the world in new ways: from
Complex Programmable Logic Devices
satellite navigation, to autonomous vehicles,
(CPLD) or Field Programmable Gate Arrays
augmented reality, and the smartphones we
(FPGA). CPLDs offer a simple device
carry in everyday life.
structure of registers and logic functions
Being able to process this information in using a sea-of-gate approach.
real and near real-time requires significant
FPGAs offer a more complex structure
processing capability and, of course, this
than CPLDs and often include dedicated
processing capability has benefited from
hardware elements such as block memory,
Moore’s law. Design engineers also have
digital signal processing, clock management,
several processing technologies from
gigabit serial transceivers, and IO blocks.
which to choose when selecting the most
appropriate for the application at hand. These
processing technology choices range from
traditional processors to Graphics Processing
FPGA Building Blocks
Units (GPU) and Programmable Logic (PL). The basic building blocks of FPGAs are
the lookup table (LUT), registers, and the
Of all processing technologies, programmable
flexible IO cell structures. LUTs enable the
logic is probably the least well known and
implementation of logic equations while
often considered one of the more challenging
registers provide the storage element
to use when implementing solutions.
necessary to implement sequential logic
designs. LUTs and registers are combined
Benefits of PL to provide what is often called a logic slice,
a simple example of which is shown in
Programmable logic enables users to
Figure 1. In modern devices, these
implement truly parallel implementations
slices contain many options which allow
of their algorithms and applications. This
implementation of combinatorial or sequential
parallel implementation enables a more
logic circuits including local distributed
deterministic and responsive solution.
memory and the ability to use the LUT as a
As such, they are used where real-time
shift register depending upon configuration.
processing and responses are required.
Flip Flop Select
OP Select
D Q
3
 Mouser Electronics eBook

Switch
Matrix

CLB CLB

Switch Switch Switch

Matrix Matrix Matrix

CLB CLB

Switch
Matrix

Figure 2 – Configurable routing blocks and interconnection using switch matrixes.

Within the FPGA device, it is common
to group together two slices to form a
Configurable Logic Block (CLB). These
CLBs are interconnected to implement the
necessary functionality using routing and
switching matrixes as shown in Figure 2.
to model the parallel architecture of the
FPGA architecture. It is also increasingly
common to develop FPGA IP blocks using
high level synthesis (HLS) using languages
such as C, C++ or OpenCL, while these
languages do not support parallelism
compiler directives can be used by the
Processing System (PS) delays, and even
serial de-serializer structures. This means
FPGAs offer any-to-any interfacing and are
able to interface with any standard, bespoke
or legacy interface. This flexibility also frees
up the system designers from becoming pin
bound when using Applications Specific

FPGA Design engineer to indicate parallel structures.

Using a higher level language provides the
Standard Products (ASSP).

engineer with a much faster development Generation of a programmable logic

FPGAs are normally designed using a
and verification cycle. design solution therefore requires the
hardware description language (HDL)
following stages:
with the two most common being Verilog
The IO structures of the FPGA devices enable
and VHDL. These languages describe the
direct interfacing with a range of IO standards.
design at a much lower level than traditional • Synthesis – Translates the
This ranges from single-ended standards
software languages by describing the HDL design into a series of
such as LVCMOS to differential standards
register level transfer of the design (e.g. logic equations which are then
like LVDS and TMDS. This interfacing does
implementing state machines, counters etc.). mapped onto the resources
not stop there however. Modern IO structures
Both VHDL and Verilog inherently support available in the target FPGA.
are able to provide on-chip termination, fine
the concept of concurrency which is needed

4
 Mouser Electronics eBook
• Place – The logic resources
determined by the synthesis tool
are placed at available locations
within the target device.
• Routing – The placed logic resources
in the design are interconnected
using routing and switch matrixes
to implement the final application.
• Bit File – The generation of the final
programming file for the target FPGA.
Simulation is used to ensure that
the implemented design functions in
accordance with the design requirements.
Engineers create test benches which
stimulate the RTL modules inputs and
monitors the resulting outputs from the
Register-Transfer Level (RTL) module (no s).
The behavior of the modules can be verified
by viewing the simulation waveform as shown
in Figure 3 or alternatively writing a more
complex test bench which can check and
verify the outputs.
Figure 3 – RTL Simulation Output
While FPGAs offer significant performance
and interfacing benefits, development of
FPGA-based solutions could be considered
more complex than traditional software
development. However, modern design tools,
especially high-level synthesis coupled with
the availability of a range of freely available
IP together with the capabilities of modern
devices, means this is not the case.
Device families
If you are unfamiliar with the history of
FPGAs, they were invented by Ross Freeman
and Bernard Vonderschmitt in 1985 with
the release of the XC2064. This first FPGA
has 64 configurable logic blocks. Today’s
modern Xilinx devices offer the user 8,938,
000 system logic cells, 3840 DSP elements,
and 76 Mb of Block RAM (BRAM) and 90 Mb
of UltraRAM. This is quite a capability step
up from the original offering.
5
Of course, the device presented above is
currently the largest Xilinx FPGA which
would be overkill for many applications.
To help guide engineers in selecting a
suitable FPGA for their application, Xilinx
offers a range of FPGA and System on Chip
devices capable of supporting a wide array
of solutions across several different families.
The cost-optimized portfolio developed
around the 28 nm node provides three
different device families, each optimized
for different user needs.
• Spartan-7 FPGAs – The Spartan-7
family is the successor to the
extremely popular Spartan-6 range
of devices and offers developers
with increased performance
and lower power over the older
technology 45 nm node. The
Spartan-7 devices are I/O optimized,
offering the highest pin count within
 Mouser Electronics eBook

the cost-optimized portfolio.
• Artix-7 FPGAs – A new family
for the Xilinx 7 series which are
transceiver optimized offering
6.6 Gbps transceivers.
• Zynq-7000 SoCs – A revolutionary
family when first debuted, Zynq-
7000 SoCs introduced a new class
of devices which combine hard core
Arm Cortex-A9 processors with FPGA
fabric. This enables a new class of
device which can provide integrated
system solutions, along with the
associated benefits of integration
including reduce power consumption,
a smaller overall solution, and
significantly reduced EMI.
Devices within this portfolio can support
a range of applications from sensor fusion,
to precision control, image processing, and
cloud computing.
High end solutions
For ultra-high performance and more
specialized applications, Xilinx provides
the Kintex and Virtex families across three
technology nodes at 28nm, 20nm and
16nm. This progression of devices provides
significant increases in performance
and capability with the UltraScale and
UltraScale+ family of devices.
Kintex devices offer increasing performance,
logic resources, and transceivers across the
three technology nodes. From 65,500 logic
cells in the Kintex devices to 1,143,00 in
Kintex UltraScale+ devices, they offer both
GTH and GTY transceivers which support
data rates at up to 16.3 Gbps and 32.75
Gbps respectively.
The highest-performance FPGAs are within
the Virtex family. These devices provide
not only logic resources of up to 8,938,000
system logic cells and transceivers capable
of operating at 58 Gbps, but also support
for high bandwidth memory (HBM). These
devices provide between 4GB and 16GB
of on-chip DRAM, with up to 460 Gbps
bandwidth or approximately 20 times more
than provided by a DDR4 DIMM. Virtex
HBM devices are used in applications to
accelerate network and storage applications.
Toolchains
All devices from the smallest Spartan-7
to the largest Virtex UltraScale+ are
supported by Xilinx development tools.
These development tools cover every
aspect of the design life cycle from RTL
capture, to simulation, and software
development for use with processor cores.
• Vivado Design Suite – Vivado enables
the capture of the design, RTL
simulation along with implementation
process of synthesis, place and
route and bit file generation.
• Vivado HLS – High-level
synthesis which enables IP
development using C or C++.
• Vitis Unified Software Platform –
Vitis enables software development
for embedded processors and also
enables acceleration using OpenCL.
• PetaLinux Tools – Petalinux is
an embedded Linux solution
for embedded processors.
This technology stack enables us to
develop solutions for both traditional
FPGA and heterogeneous system on chips
which combine programmable logic with
high-performance Arm processor cores.
Interestingly, as we will see, this technology
stack enables implementation using
traditional hardware design language (HDL)
capture and a higher-level system optimizing
compiler approach depending upon the
users’ desired entry point.
The inter-relationship between the design
tools can be seen below in Figure 4. Each
element of the technology stack provides a
specific capability.
Vivado Design Suite
The lowest level of the technology stack
is Vivado. Vivado enables us to capture
designs using VHDL or Verilog as well as
synthesise the HDL design to the target
device before placing and routing and
generating the programming file.
At the heart of Vivado is the IP integrator
which enables designers to capture designs
quickly and easily using IP provided by Xilinx,
third parties or custom developed. This IP
can be defined using HDL or alternatively, a
higher-level approach can be used with Vitis
HLS which enables the development of IP
blocks using C and C++.
While implementation is Vivado’s focus,
Vivado offers a complete development
ecosystem and provides several different
capabilities which aid the overall
programmable logic development.
One of the key features of any design
is being able to guarantee functional
performance of the HDL prior to
implementation. To verify the HDL
functionality, Vivado includes an HDL
simulator which enables the developer to
stimulate the HDL. Depending upon the
stage of implementation, the test bench can
be applied against the RTL, synthesized
netlist, or the implemented netlist with
associated timing information.
It is through Vivado that we can also
debug designs on the hardware thanks to
its support for integrated logic analysers
(ILA), Virtual IO (VIO), and JTAG to AXI IP.
This allows the designer to instrument the
programmable logic design and observe
behaviour at run time in the actual system.
Both Vivado’s implementation and simulation
capabilities are used by higher-level tools in
the development stack as we will see.

6
 Mouser Electronics eBook

AI / ML Solution
Implementation

Application
Prototyping and Rapid
Development

Embedded and
Accelerated Software
Development

Design Capture and

System Definition
and Implementation

Figure 4 – Xilinx Technology Stack

7
 Mouser Electronics eBook
Application Software
Creation
If we are working with a heterogeneous SoC
or FPGA which contains a soft processor
core such as a MicroBlaze, we will need to
work with higher levels of the stack to create
the operating systems’ application software.
The relationship between Vivado, PetaLinux
and Vitis can be seen below in Figure 5.
Both hard and softcore processor
implementations require the generation
of operating systems and development of
application software. This software can be
developed to run on a BareMetal, real time
operating system (FreeRTOS) or embedded
Linux solution.
This application software can be developed
and debugged using Xilinx’s Vitis unified
software platform. Vitis provides two
development flows embedded and
accelerated. With the embedded flow
developers can create software solutions
for any supported embedded processor,
while the accelerated flow enables users
of Xilinx SoC and Acceleration cards to
accelerate functionality from the processor
system to the programmable logic.
When used in the embedded flow, Vitis
developers can import Vivado designs to
create embedded solutions. To aid developers
in the debugging of the embedded systems,
Vitis includes a complete debugger which
is capable of supporting multi-core
operation. As the software application
XSA
Vivado
XSA
PetaLinux
Figure 5 - relationship between Vivado, PetaLinux and Vitis
will include operations with elements of
the programmable logic design, it is also
possible to cross probe and breakpoint with
ILAs inside the programmable logic to get
a greater understanding of the hardware /
software interaction.
If we desire an embedded Linux solution,
developers utilize the PetaLinux build tool.
PetaLinux is not a Linux distribution but a
build system. As such, PetaLinux allows
developers to configure an embedded
Linux system which includes the drivers
and device tree entries to work with the
programmable logic design.
Once a PetaLinux operating system is
available for a Xilinx SoC, the developer can
deploy the PYNQ framework which enables
Python and Jupiter Notebooks to be used
with programmable logic. PYNQ is perfect
for rapid prototyping because the Python
environment contains many drivers to work
with IP in the programmable logic, thereby
reducing the software development required.
To access higher levels of the technology
stack such as the Vitis acceleration
capabilities and Vitis AI, a hardcore
processor and embedded Linux solution or
an Alveo acceleration card is required.
Acceleration
Vitis also provides an entry point for
solutions which use both the processing
system and the programmable logic thanks
to its ability to accelerate applications into
the programmable logic using OpenCL.
Vitis Platform Vitis Application
XSA
Embedded or
Image.ub
Accelerated
SYSROOT
Applications
Boot Files
QEMU
8
To make use of this acceleration capability,
an acceleration platform must be available
to Vitis. Acceleration platforms can be
created in Vivado for custom hardware and,
alternatively, they are available for download
for many development boards.
Several system resources such as such as
AXI interfaces, processor interrupts, and
clocks are available to the Vitis compiler
through the acceleration platform. The
accelerated elements of the design are
connected to these resources to enable
access from the processing system.
To perform this acceleration, Vitis used
the OpenCL framework. OpenCL is an
open industry standard developed by The
Khronos Group to enable parallel computing
of heterogeneous systems. The OpenCL
model consists of a host, typically an x86
system, and an OpenCL device often called
a kernel. While the host manages the overall
application execution, the OpenCL provides
the parallel processing and application
acceleration. This parallel processing can
be provided by an FPGA, DSP, GPU or CPU.
Using OpenCL enables the source code
that is running on the OpenCL device to be
retargeted without any source code changes.
To support this model, OpenCL provides
OpenCL APIs which can be compiled into
the host application using standard C
compilers such as GCC or G++. When it
comes to the OpenCL device, these are
developed using the OpenCL C language.
OpenCL C language is derived from ISO
C99 but it is not C because the standard
libraries are difficult to support across all the
potential OpenCL devices.
This means that each OpenCL device has a
specific OpenCL compiler provided by the
vendor. In the case of Xilinx devices, this is
provided by Vitis.
In the Xilinx ecosystem, these heterogeneous
systems comprise of the following
architectures:
• Arm processing cores and the
programmable logic are connected
using AXI interconnects
• X86 processor and Alveo acceleration
card connected over PCIe
 Mouser Electronics eBook
Both architectures enable the processing
element to act as the host while the
programmable logic, either on the same
silicon or in a sperate device, is the
OpenCL device.
Implementation of the programmable logic
design will call Vivado in the background but
of course this can take time to implement.
Vitis also offers different build types for use
in developing and verifying the application
before generating the final bit-stream. These
are a software emulation which enable
correction of typos and basic errors while
hardware emulation leverages QEMU and
co-simulation for the programmable logic
design. The hardware emulation enables us
to optimize performance, interfacing, and
resources as shown in Figure 6 above.
To support application development, Vitis
also provides several common libraries which
can be accelerated into the programmable
logic and several domain specific libraries
for implementation of specific applications.
Included in the common libraries are maths,
linear algebra, and DSP libraries, while the
domain-specific libraries include computer
vision, database, quantitative finance, and
security. These libraries enable developers to
leverage programmable logic performance
without needing to start from scratch writing
commonly used and freely available SW
functions to be accelerated.
Source Code
Optimize Performances Software Emulation
Optimize Interfacing
Optimize Resources
Hardware Emulation
Create Boot Files
Figure 6 - Vitis Emulation Flow
AI and Beyond
Building on top of the Vitis acceleration
capabilities is Vitis AI. Using Vitis, the
developer can instantiate a deep learning
processor unit in the programmable logic.
This instantiation then provides significant
acceleration for the machine learning
inference applications.
Of course, instantiating the DPU within the
programmable logic provides significant
benefits to the developer. However, Vitis
AI goes one step further and enables
developers to work with common machine
learning and artificial intelligence
frameworks such as Caffe, TensorFlow
and PyTorch.
Rather than being a single tool, Vitis AI is a
collection of tools to compile, quantize, and
optimize their machine learning applications
from a floating-point implementation to
a fixed-point representation suitable for
implementation within the DPU.
To aid the development and optimization of
the DPU implementation, Vitis AI also provides
a profiler to enable system-level optimization.
Of course, if you wish to start from scratch
there are also pre-optimized models
available in the Xilinx Model Zoo.
Syntax Errors
Verify Functional
Performance
9
There are also other commercial and
open source software tools which can be
used as part of the FPGA development
flow. These tools range from synthesis to
simulation and an increasingly large number
of verification tools that support both
simulation and formal verification.
Embedded Processing
Programmable logic in FPGAs is ideal for
implementing parallel processing structures
for functions like finite impulse response
filters, image processing pipelines, and
motor control algorithms. However, there
are times when serial processing is required.
Implementing communication protocols,
graphical user interfaces or control,
configuration, and status reporting of IP
blocks are good examples of this. Of course,
serial processing is also critical when we
wish to work with higher-level open source
frameworks and languages like TensorFlow,
OpenCV and Python.
Luckily in the programmable logic world,
we have several options which can be used
to implement embedded processors with
programmable logic devices. At the highest
level, these can be classified into two
distinct groups:
• Heterogeneous System-on-
Chip: These devices combine
programmable logic with a processing
system. In these heterogeneous
system-on-chip solutions, the
processing solution is hard in
the silicon of the device and as
such, offers great performance
although the processing solution
then provides only limited
flexibility for configuration.
• Soft-Core Embedded Processors:
Soft-core processors are implemented
using the programmable logic
resources (e.g. the flip-flops (FF),
look-up tables and BRAMs). As
these soft-core processors are
implemented using the logic
resources, they are more configurable
but typically offer lower performance.
 Mouser Electronics eBook

Figure 7 - Zynq MPSoC Processing System Interfacing with PL Image Processing Chain

Both heterogeneous SoC and soft-core
embedded solutions have a range of use
cases across several exciting applications
as we will see. It is not unusual to implement
additional soft-core processors in the
programmable logic of heterogeneous SoCs
to create a Big-Little enabling off-loading of
time for critical tasks.
Embedded Processors
In the Xilinx suite of devices, embedded
processors are provided in the Zynq-7000
SoC and Zynq MPSoC product lines. These
devices offer true heterogeneous processing
systems on the same silicon. Architecturally
in these devices, the processor system
boots first like a traditional processor and
then configures the programmable logic.
The Zynq-7000 SoC was the first introduction
and offers dual or single-core 32-bit Arm
Cortex-A9 processors combined with
programmable logic. As would be expected,
the processing system provides peripherals
used for both volatile and non-volatile
memory along with several interfacing
peripherals such as ethernet, UART and CAN.
To support high-performance applications,
each Cortex-A9 core also includes a
floating-point unit and a NEON engine.
The NEON engine allows processing of
large data sets in parallel using a single
instruction against multiple data (SIMD).
This is especially useful for applications
like image and audio processing, where
algorithms require data sets to be processed
using simple instructions (e.g. multiply and
add) multiple times with little control code.
In applications such as this, leveraging the
SIMD unit can result in a significant increase
in performance.
Data transfer between the processing system
and the programmable logic is implemented
using several Advanced eXtensible Interfaces
(AXI). Using this interface, both the processor
system or the programmable logic can be
the initiator of the transaction. This allows
transfer of data easily to and from the
processor system DDR memory.
This combination of processing system
and programmable logic makes the Zynq-
7000 series excellent for implementing
applications which require both serial and
parallel processing (for example image
processing, robotics, and augmented
reality). To ease connectivity and leverage
the large support of frameworks and
applications, embedded Linux solutions can
be deployed on the processing system, while
the programmable logic accelerates key
elements or algorithms. This combination
of PS and PL provides for a more responsive
and deterministic solution. In the table
below, you will find a simple demonstration
implementing AES encryption.
Operating System Linux
Processor System Clocks 36662
PS Clocks with Programmable 15644
Logic
Reduction in Processing Time 54.8%
With the evolution of the Zynq-7000 SoC
into the next-generation Zynq MPSoC,
a significant step change in processing
capabilities was introduced along with
the latest logic fabric. For the first time,
heterogeneous processors were introduced
within the processing system, enabling the
developer to address several challenges
within the same device.
The processing system within the
Zynq MPSoC contains the following
processor cores:
• Application Processing Unit –
Consists of quad or dual 64-bit
Arm Cortex-A53 processors
• Real Time Processing Unit –
Dual lockstep 32-bit Arm
Cortex-R5 processors
• Platform Management Unit
– Silicon implementation of
a Triple Modular Redundant
32-bit MicroBlaze processor
• Graphics Processor Unit –
Arm Mali-400 MP GPU
In addition to the four processing groups
which can be programmed by the developer,
the MPSoC processing system also contains
a configuration security processor to
implement safety and security processing
and security event responses.
This diverse range of processing solutions
enables the creation of single-chip solutions
for many applications (e.g. automotive)
where both high-level algorithms and user

10
 Mouser Electronics eBook
interfaces can be implemented using the
APU and GPU while real-time control and
interfacing with the vehicle controls can
be via the Block RAM (BRAM) which is
designed and certified for ISO26262
or IEC6508 applications.
Communication between the PS and PL
again uses AXI interfaces however in place
of 32-bit interfaces, 128-bit interfaces
are provided significantly increasing the
throughput between PS and PL. This
high-bandwidth capability enables the
implementation of high-performance vision-
based machine learning applications as
often deployed in automotive applications
or other edge-based solutions. As such, the
embedded processing solution offered by
the Zynq-7000 SoC and Zynq MPSoC class
of devices provides the highest performance
processor systems.
An example image processing application
which transfers image data between the
processor system and the programable logic
to implement the desired algorithm can be
seen in Figure 7.
Soft-Core Processors
The options for soft-core processors in
the Xilinx ecosystem are unlimited. Any
processor which is described in RTL can be
implemented using the FF, LUTs, and RAMs
of Xilinx FPGAs.
However, these are the most deployed soft-
core processors within Xilinx FPGAs:
• MicroBlaze – a 32-bit processor
which can be used in a range of
configurations from controller
to full MMU support capable
of running embedded Linux.
• Arm Cortex-M1 – a 32-bit FPGA
implementation of the popular
Cortex-M0 which offers a small logic
footprint and great code density
thanks to the Thumb Instruction set.
• Arm Cortex-M3 – a 32-bit
implementation of the Cortex-M3
processors which offer full blown
MMU and OS support along with
good code density thanks to
Thumb/Thumb2 instruction set
support. This processor is popular
in Internet of Things applications.
• RISC-V – an open source 32/64/128-
bit instruction set. RISC-V compliant
implementations are available from
several IP vendors for implementation
in Xilinx FPGAs. Of course, RISC-V
is highly customizable and like
MicroBlaze, can also run embedded
Linux and other operating systems.
While the processing performance of a
soft-core processor is below that of a hard
silicon instantiation, the configurability and
flexibility of the solution is higher which
allows for a much more customized solution
to be implemented. This also allows for the
processor to be portable across several
devices or even vendors depending upon
the processor selection.
Like the hard silicon processors, again
AXI is a popular interface for peripheral
connection. However, in the soft-core
processor world, this peripheral also
includes DDR memory interfaces, UARTs,
and other common processor interfaces
such as I2C and SPI. An example of this can
be seen in Figure 8 which uses a MicroBlaze
processor to configure and control a high-
speed image processing pipeline.
So how do we as engineers decide between
the implementation of a hard or soft-core
processor? Of course, performance is one
of the main factors, but others include
application, flexibility, security, resource
availability, portability, and licensing. Of
course, each application will have different
weightings on these factors as engineers
determine what is best for their solution.
To compare the processor capabilities, we
need to be able to compare processing power.
We achieve this using a benchmark called
Dhrystone MIPS or Millions of Instructions
Per Second. By comparing the potential
hard and soft cores in the table below, it is
apparent the embedded processors win due
to their higher clock speeds.
11
Processor DMIPS/ Comment
MHz
Cortex-A53 2.3 Quad or dual
processors
Cortex-A9 2.3 Dual or single
Cortex-R5 1.67 Dual or lockstep
MicroBlaze 1.04 –
1.31
Cortex-M1 0.8
Cortex-M3 1.25
RISC-V 1.7 Depends on
implementation
The application is equally as important. If the
processor core is required to only configure
IP within the processing system or implement
serial communication protocols, then a
softcore-based processor is often a better
selection. However, for high performance
algorithms which need significant processing
capabilities, hard-core processors definitely
offer a performance advantage.
Security can also be a major consideration
especially for edge applications as hard
processing solutions like the Zynq MPSoC
provide inbuilt security solutions e.g. the
Configuration Security Unit, Secure Boot
and Arm Trust Zone. When using softcore
processors, the security protections also
needs to be included in the programmable
logic.
One major difference between hard- and
soft-core processors is in the configuration,
in a hard-core system the processor is the
master and boot first and configures the
programmable logic as desired. This enables
several different power saving modes to
be implemented by the processor e.g.
powering down a processor core, powering
off peripherals or even powering down the
entire programmable logic.
When a soft-core processor is used the
FPGA must first be configured to instantiate
the soft-core processor, only then can the
processor begin operation. This limits the
ability of the soft-core processor to be able
to implement power down / power saving
schemes, although there is still much which
can be achieved by lower clock frequencies.
 Mouser Electronics eBook

Figure 8 - MicroBlaze Image Processing Application

Figure 9 – Big-Little Approach with the Zynq MPSoC and Arm Corex-M3

12
 Mouser Electronics eBook
Big - Little Approach
However, increasingly you will often see
applications which use both hard processors
and soft processors in the same solution as
demonstrated in Figure 9. Such a Big-Little
approach enables the high-performance ap-
plication processor to focus on the high-level
application while real-time applications such
as sensor interfacing and motor control are
off loaded to the soft-core processor in the
programmable logic. This approach provide
a more responsive solution than just using
an application processor.
Correctly architecting the Big-Little
interfacing also enables the main application
to be updated as required without the
requirement to change the code executing
within the little processor. This also means
sensor changes and updates can be
addressed simply by updating the code
running with the soft-core processor.
Communicating
Internally and Externally
One of the genuinely great things
about FPGAs, outside of their truly
parallel nature and the ability to have
heterogeneous systems, is their ‘any-to-
any’ interfacing capability.
What this means is, with the right PHY,
programmable logic can provide interfacing
to many different industry standards, legacy
interfaces, and bespoke interfaces. And
Figure 10 – Camera Link reception and alignment using ISERDES
thanks to the flexibility of the different IO
cells provided in Xilinx programmable logic,
an external PHY is often not needed.
However, as programmable logic designers,
it’s not just the external interfacing
we need to consider. We also need to
understand the efficient methods of moving
data around within the programmable logic
once it is received from the IO cells.
Let us start our journey of connectivity
by examining the different IO structures
that exist for the Xilinx UltraScale and
UltraScale+ families. They include the Kintex
and Virtex UltraScale families as well as the
Kintex, Virtex and Zynq UltraScale+ families.
The UltraScale and UltraScale+ devices
offer three different IO classes, excluding
gigabit transceivers.
• High Performance (HP) – optimised
for high-speed interfaces such
as memory and chip-to-chip.
The maximum operating voltage
for a supported IO standard is 1v8.
• High Range (HR) – supports
interfaces that operate at up to 3v3.
• High Density (HD) – supporting
interfaces that operate at up to 3v3
at data rates of up to 250 Mbps.
UltraScale devices provide a mix of HP and
HR IO banks, while the UltraScale+ devices
provide a mix of HP and HD IO banks.
13
Each of these IP classes can support many
different IO standards: from simple single-
ended CMOS signals to LVDS and HSTL.
At the system level, this has a significant
impact on integration. The wide range
of support for a variety of IO standards
reduces the need for external PHYs, thereby
reducing component count, crucial PCB
area, and power.
However, it is not just the obvious benefits
that programmable logic IO provides. The
IO structures have many features that ease
the implementation at the system level
beyond board area and power. The flexible
IO structures enable the drive strength and
slew rate to be controlled, allowing the
signal integrity and EMI emissions of signals
on the board to be carefully crafted for
optimal performance. They also support on-
chip termination schemes implemented by a
Digitally Controlled Impedance (DCI).
The IO cells support alignment between
high-speed signal traces, offering precision
timing adjustments using the IDELAY and
ODELAY output, while the ISERDES and
OSERDES structures support conversion
between serial and parallel data - especially
useful for aligning data patterns on high-
speed interfaces as shown in Figure 10.
While the HR, HP, and HD IO classes provide
significant interfacing capabilities and
enable significant data movement on and off
chip, the highest interfacing capacities are
provided by the gigabit transceivers.
 Mouser Electronics eBook
These transceivers enable the ultra-fast programmable logic and processing systems,
transfer of data allowing the programmable if desired. Within the Xilinx environment, the
logic to work with some of the fastest serial primary protocol used for data movement
interface standards such as PCIe, SATA, is the Advanced eXtensible Interface (AXI):
100G ethernet, SDI, JESD204A/B, USB 3.0 a subset of the Arm AMBA bus, developed
and DisplayPort. specifically to support implementation in
programmable logic.
Xilinx identifies transceivers as GTx, where x
indicates the specific standards. UltraScale To provide scalability for different use
and UltraScale+ devices provide for data
cases, AXI itself offers three different
rates between 6 Gbps (GTR) and 58 Gbps
interfacing standards.
(GTM). The specific mix of GTx depends
upon the family. Across the UltraScale and
UltraScale+ families of devices, this means • AXI Full / Memory Map - A higher-
we get an excellent range of ultrafast IO and, performance memory-mapped
consequently, a significant peak bandwidth. interface that supports independent
read and write channels. Both
Scale GTx Gbps channels enable bursts to optimize
throughput. In programmable logic
Virtex UltraScale+ GTY/ 32.75/58.0
designs, AXI Full is often used to
GTM
implement direct memory transfers
Kintex UltraScale+ GTH/ 16.3/32.75 between the programable logic and an
GTY external DDR memory, for example.
Zynq UltraScale+ GTR/ 6.0/16.3/ • AXI Lite – A stripped down version
GTH/ 32.75 of AXI Full to provide a memory-
GTY mapped interface which can
be used for configuration and
Virtex UltraScale GTH/ 16.3/30.5
control of IP blocks. AXI Lite does
GTY
not support burst accesses.
Kintex UltraScale GTH 16.3
• AXI Stream - A unidirectional stream
of data from a producer to a consumer.
This stream is point-to-point and
Of course, being able to interface with such
contains no addressing information.
data volumes provided by the HD, HP, HR,
and GTx interfacing capabilities means we
Both the AXI Full and AXI Lite consist
need to be just as efficient if not more within
of independent read and write channels.
the device.
Of course, the complexity of the channel
varies between the two flavours. The read
Internal Data Movement channels consist of two sub-channels: the
read address and control sub-channel and
A key internal feature of the programmable
the read data and response sub-channel.
logic device is the ability to move data:
The write channel consists of three sub-
between IP blocks, even between
Figure 11 - AXI Connections in Zynq UltraScale+
14
channels: the write address, write data,
and write response sub-channels.
AXI Stream interfaces are commonly used to
transfer information from a single producer
to a consumer, typically between IP blocks
as part of a processing chain. Example
processing chains could be image processing
or signal processing, where the signal is
received and processed by each IP block
before being passed on to another.
When AXI interfaces are implemented
in programmable logic, we can benefit
from wide data bus widths to increase the
bandwidth. Data bus widths can vary between
32 to 256 Bits when assuming a clock of
400MHz, giving data rates between 12.8
Gbps and 100 Gbps. This arrangement
of AXI Full, AXI Lite, and AXI Stream is
shown in Figure 11.
AXI interfaces can be locked down using Arm
TrustZone software to support design security
when working with Zynq MPSoC UltraScale+
devices. This capability is increasingly
important both in the cloud and at the
edge. Allowing these orthogonal software
worlds prevents lower security, higher-risk
applications from being able
to access registers and peripherals defined
as secure.
When we are working with heterogeneous
SoCs like the Zynq MPSoC UltraScale+
device, system-level cache coherency
becomes increasingly important. AXI also
provides the additional sideband signals to
provide IO cache coherence and complete
cache coherence with the ACE, ACP and
HP(C) ports available on the Zynq MPSoC
processing system.
 Mouser Electronics eBook

As AXI is implemented within programable

logic, AXI is a point-to-point protocol
between a producer (master) and a consumer
(slave). To enable a single producer to be
able to connect with several consumers,
the developer can make use of a smart
interconnect which enables multiple
consumer (slave) connections to a single
producer (master). Useful when using a
single AXI Lite interface to configure several
IP blocks, for example. Smart interconnects
are also capable of providing clock domain
crossing if required.

The use of standard interfaces based on AXI

enables a much larger ecosystem of IP cores
to be created by both Xilinx and third-party
partners. Standard interfaces also help when
developing custom IP using either RTL or HLS.
Figure 12 – TBS1052C in action debugging an embedded system

Programmable Logic
Test Equipment as part of the STTE enables the

Applications programmable logic element to receive

With its inherent flexibility, it stands to
reason that programmable logic is often
deployed across a range of applications,
offering a more deterministic, high
performance and power-efficient solution
than traditional microprocessor and Digital
Signal Processor applications.
As you might expect, these applications
span a wide range of industries and use
cases. In this article, we are going to examine
four different and diverse use case which
feature programmable logic at the core.
stimulus test patterns and report captured
Developing complex embedded system
data patterns for additional analysis using
solutions can be very difficult, and testing
Gigabit Ethernet links or USB3.
and verification of those systems can be
even more complex. This often requires the
Of course, programmable logic is not
creation of specialized test equipment that
limited to deployment in STTE. Traditional
is able to provide real-time stimulus, data
FPGA and heterogeneous SoCs are
capture, analysis, and storage. For this
increasingly found within common lab
reason, the high performance provided by
equipment such as oscilloscopes, logic
programmable logic is often used to enable
analysers, and signals generators. One such
Special-to-Type Test Equipment (STTE) to
example of this is the Tektronix TBS1052C
stimulate and capture output data. Such
shown in figure one. This uses a single-core
an approach enables real-time samples
Zynq-7000 SoC to implement signal capture
to be generated and captured for most
and provides interfacing with 1GSa/S ADCs
applications. Using a heterogeneous SoC

SAE Name Examples Vehicle Monitoring Fall Back Vehicle

Level Control Control Capability
0 No Automation N/A Human Driver Human Driver Human Driver N/A
1 Driver Assistance Adaptive Cruise Control/Lane Human Driver Human Driver Human Driver Some Driving
Keeping and Parking Assist and Vehicle Modes
2 Partial Automation Traffc Jam Assist Vehicle Human Driver Human Driver Some Driving
Modes
3 Conditional Automation Full Stop and Go Highway Driving Vehicle Vehicle Human Driver Some Driving
Self Parking Modes
4 High Automation Automated Driving Vehicle Vehicle Vehicle Some Driving
Modes
5 Full Automation Driverless Vehicle Operation Vehicle Vehicle Vehicle All Driving Modes

Table 1. SAE Levels for assisted and automated vehicles

15
 Mouser Electronics eBook
along with advanced triggering capabilities
on the digitized data. The ARM processor
core contained within the SoC runs the
operating system and scope application and
provides the USB interfacing capabilities.
Such an approach enables a more tightly
integrated solution.
Automotive and ADAS
In line with the society of Automotive
Engineer’s five-level capability matrix, the
automotive world is on a mission to increase
the level of assisted and automated driving
capabilities in vehicles.
In order to safely interact with the external
world, automotive solutions must use
several diverse sensor modalities and
communications systems, which include
the following:
• Vision Systems – Including Infrared
• 4D RADAR
• LIDAR
• Accelerometers
• Global Positioning Systems
• Vehicle-to-Vehicle and
Infrastructure Communications
These sensors and systems generate
significant data volumes which need to be
aggregated and processed before decisions
on vehicle actions can be implemented,
presenting several challenges to the system
designer. With such a diverse range of
sensors comes a diverse range of sensor
interfaces, ranging from high performance
multi-gigabit serial links (e.g. MIPI) to lower-
speed interfaces such as SPI and I2C as used
by lower-speed sensors. Traditional system-
on-chip solutions provide the user with a
limited number of fixed-function interfaces
of varying types. Using a programmable
logic device enables the system designer
to leverage the flexible IO structure to
implement the specific number and type
of interfaces required, freeing previous
IO limitations.
The parallel structure of programmable
logic enables the implementation of parallel
image/signal processing chains. This parallel
implementation of algorithms provides a
more responsive solution, which is required
to implement safe interaction with other
vehicles and the environment.
Cloud Computing
One of the hottest topics in the
programmable logic world is data centre
acceleration. Deploying programmable
logic in data centres combines large FPGAs
with x86 processors connected using PCIe.
Such an approach enables the x86 software
application to offload highly parallel functions
to the FPGA, accelerating the performance of
the system. To be able to deploy accelerators
with the Cloud or on premises, Xilinx offers
a range of accelerator cards called Alveo.
These cards are design to interface over PCIe
and are programmed using the Vitis unified
software platform and OpenCL. This enables
the developer to use C/C++ and OpenCL to
accelerate algorithms. Typical datacentre
applications include quantitative finance,
database and data analytics, machine
learning, and network acceleration.
Industrial
Programmable logic plays a large part
in Industry 4.0 where one of the main
challenges is to implement a converged
network. Converged networks merge the
Information Technology (IT) and Operation
Technology (OT) networks. Traditionally
the IT network is where the Enterprise
Resource Planning (ERP) is located, while
the OP network contains the sensors and
drives used to manufacture the product.
Connections between the two bring about
challenges using gateways and bridges and
protocol convertors, which can limit the
scalability of the OT network.
As such, the IT and OT networks have
different requirements. The IT network
needs to be able to access multiple systems
and databases etc., while the OT network
needs to be real time and deterministic to be
able to control its sensors and drives.
One of the increasingly popular solutions is
to implement a solution using Time-Sensitive
Networking (TSN). The most popular TSN
standard is Ethernet which is defined
16
by several IEEE 802.1 standards. Time
awareness across the network is implemented
in TSN by allocating scheduled traffic in time-
defined slots, while also supporting cyclic
data transmission and providing pre-emption
for higher priority packets.
Correctly implementing TSN requires a
solution which can provide a low latency and
deterministic response at TSN end points
and switches. This is where Xilinx Zynq SoC
and Zynq MPSoC devices come into play
because they enable the implementation of
TSN ports with programmable logic.
We can implement TSN ports in the Xilinx
ecosystem using their TSN Ethernet
Endpoint MAC LogiCORE IP within a Zynq-
7000 SoC or Zynq UltraScale+ MPSoC.
Each device utilises both the Processing
System (PS) and the Programmable Logic
(PL). The LogiCORE IP consists of FPGA
logic for MAC, TSN Bridge, and TSN
Endpoint, along with software components
for network synchronization, initialization,
and interfacing with network configuration
controllers for Stream Reservation as
defined in P802.1Qcc. The software is
designed to run on PetaLinux and will be
published as Yocto patches.
The logic IP core provides deterministic
behavior in the PL for synchronization (IEEE
802.1AS), scheduled traffic (IEEE 802.1Qbv),
and seamless redundancy (P802.1CB) while
helping to offload the processing unit. It
is also possible to implement an optional
integrated time-aware L2 switch to enable
either chain or tree topology.
Once implemented, the TSN can be
combined with custom applications like motor
control or sensor interfacing within the PL to
be able to act under the control of the TSN.
Deep Dive Application
- Creating Embedded
Vision Systems
There is something special about seeing an
image that you have created on a display.
That display could be demonstrating a
simple, transparent image showcasing an
 Mouser Electronics eBook
image sensor’s capability. Alternatively, it
could be implementing an advanced image
processing solution that identifies and
classifies objects or tracks movement in the
image. Of course, with the correct sensor
selection, we can even extend the range of
vision beyond the visible range of the EM
spectrum into the infrared or X-ray elements
of EM spectrum.
Implementing image processing algorithms is
computationally intensive, especially as image
resolutions increase beyond HD and move
to 4K. A color HD image of 1920 pixels by
1080 lines using a 30-bit pixel must be able to
process 3.73 Gbps to achieve a 60 frames per
second. Moving to 4K resolution, which has
3840 pixels and 2160 lines with a 30-bit pixel
and 60 frames per second, requires quite an
increase of 14.92 Gbps. Each stage of the
image processing algorithm must, therefore,
be able to support this data rate to achieve
the desired frame rates, even when doing
complex calculations on each pixel.
The truly parallel nature of programmable
logic provides an ideal technology to
implement image processing pipelines. The
parallel nature frees the developer from the
sequential software world where each stage
of the image processing algorithm must be
implemented in sequence. In programmable
logic, the algorithm’s elements run in
parallel, enabling an increased throughput
and a more deterministic performance. This
can be critical for many image processing
applications that use embedded vision to
interact safely with the environment. ADAS
or vision-guided robotics are two good
examples.
Figure 13 – Example Image Capture Pipeline
Developers can get the best of both worlds
by using a heterogeneous SoC such as the
Zynq-7000 SoC or Zynq UltraScale+ MPSoC.
These devices combine programmable logic
with high-performance Arm processors.
This provides significant flexibility because
the image processing algorithms can be
implemented within the programmable logic.
While the processing system can provide the
image processing algorithm configuration to
allow easy adaption to new image sensors
or requirements, it can also implement high-
level algorithms that take the output from
the image processing system.
Elements of a PL Image
Processing System
Implementing an image processing system
in programmable logic is not as daunting
as it first may seem. The image processing
pipeline can be broken down into three
distinct elements: image capture, algorithm,
and output pipeline.
Image Capture – The image capture
pipeline connects interfaces directly with
the image sensor or camera. As such, the
image capture interfaces externally to
the programmable logic using interfaces
such as HDMI, SD/HD/UHD-SDI, MIPI
or Parallel/LVDS. Thanks to the flexible
nature of programmable logic IOs, most
standards can be implemented using
the IO structures without the need for an
external PHY. To help capture the image,
Xilinx provides a range of IP cores in
the Vivado IP library that will enable the
image to be captured and made ready for
17
further processing. Once the image has
been captured, further processing might
be required to obtain a useable image
for the image processing algorithm. This
additional image processing may require
color filtering (debayer) to convert raw
pixel values to RGB pixels. The image
capture phase may also include gamma
correction, noise filtering, and color space
conversion. In adaptive systems, the input
video timing and resolution will be detected
to enable the image processing system to
automatically configure itself for the video
format received. An example image capture
pipeline can be seen in Figure 13 below
which includes a MIPI CSI-2, demosaic
(debayer), and frameBuffer to write to PS
DDR memory.
Algorithm – This is the actual implementation
of the image processing algorithm. In many
cases it will consist of several stages of
image processing algorithms, each one
connected to the next stage using an AXI
Stream. These IP blocks may be provided by
the Xilinx Vivado IP library which includes
IP blocks that can scale images up or and
layer video layers on top of each other as
demonstrated in Figure 14.
Alternatively, they can be implemented using
a hardware description language or high-
level synthesis which enables higher-level
languages such as C/C++ to implement
image processing algorithms. Using a
higher-level language enables developers
to leverage the vision domain Xilinx Vitis
accelerated libraries. These libraries
provide several advanced image processing
functions like filters, bounding boxes, bad
pixel correction, warp transformation, and
 Mouser Electronics eBook
Figure 14 – Image Processing Video Mixer mixing live video with a Head Up Display.
stereo block matching. If the image needs
to be made available to the processor
system for higher-level algorithms for
example, Video Direct Memory can be used
to transfer the video stream to the PS DDR.
This transfer can also operate to transfer
data from the processor system to the
programmable logic. Such PS-PL transfers
can be used to provide an overlay on the
image, presenting information on the display
if required.
Output Pipeline – Once the image has
completed the algorithm pipeline, the
processed image needs to be out to the
appropriate display. This could be MIPI-DSI,
HDMI, SD/HD/UHD-SDI or traditional parallel
video with pixel clock, V Sync and H Sync. In
Figure 15 – Example Output Pipeline
the programmable logic, the developer needs
to convert the processed image which is in
an AXI Stream format into the correct output
format. Along with converting the AXI Stream
into the appropriate format, the video must
also be re-timed for output. Just like with the
image capture and algorithm pipeline, the
Vivado IP library contains the necessary IP
cores to generate the output video in the
correct format. Figure 15 below shows a
typical output pipeline with a frame read from
the PS DDR passing data to the AXI Stream
to Parallel Video output, operating under the
control of video timing controller.
It is possible to move the image into the
processing system DDR memory during
the algorithmic processing. This allows the
18
processing system to perform high-level
algorithms on the processed image contents.
The software can further process the image
and output it back into the image processing
stream if so desired.
Software Defined Image
Processing
There are several approaches which can
be undertaken when it comes to working
with the image in the processing system.
Regardless of the approach taken, the
image processing implemented within the
programmable logic is highly configurable
by the developed application software.
 Mouser Electronics eBook
Bare Metal – Bare metal developments
are often used as an initial stage in the
development of the image processing
system. They enable the developers to
demonstrate the design quickly and easily in
the programmable logic and the image sensor
can be correctly configured. This allows for
the creation of a transparent path which
displays the captured image on the selected
display. The bare metal application does not
include the complexity of an embedded Linux
stack. As such, it is very useful for debugging
and commissioning the design using the
Internal Logic Analysers (ILA) and memory
views as well as the debugging capabilities to
inspect register contents.
PYNQ - The open source PYNQ framework
enables developers to leverage the
power of Python to control IP within the
programmable logic thanks to several
PYNQ APIs, drivers and libraries. Thanks
to these PYNQ provisions, developers
can focus on the algorithm development
because the PYNQ frameworks includes
drivers for most AXI connected IP blocks
on the programmable logic. PYNQ runs
on a Ubuntu-like distribution and enables
developers to start focusing on the image
processing algorithms using OpenCV
and other popular Python frameworks.
Using PYNQ enables us to focus on the
algorithm development using the real-
world sensors, which includes being able
to see the limitations of the sensor under
different conditions and the impacts on
the implemented algorithm. Once we know
what the algorithm is, we can implement
the functionality in the programmable logic
using Xilinx IP, HLS or the Vitis accelerated
library function.
PetaLinux – An embedded Linux solution
may be considered if higher-level algorithms
or communication is needed. In this instance,
PetaLinux can be used in conjunction with
the Video4Linux (V4L) and GStreamer
packages to create higher-level image
processing algorithms. This framework may
also be used if USB3 cameras are used that
are connected to the processing system.
Using PetaLinux enables the developer to
also leverage the Vitis AI flow and the Xilinx
Deep Learning Processor Unit (DPU) which
aids the acceleration of the machine learning
inference into the programmable logic using
the DPU and supporting frameworks such as
TensorFlow, Caffe and PyTorch.
Conclusion
In this eBook we have introduced the basics
of what an FPGA is, the tools we use to
work with AMD FPGAs along with looking in
depth structures, elements and features of
a modern FPGA. Along with demonstrating
several different applications which can
benefit from FPGA technology.
If you want to know more about FPGAs,
a range of resources are available at
https://resources.mouser.com/programmable-logic.
19
Adam Taylor is a world-recognised expert
in design and development of embedded
systems and FPGAs for several end
applications. Throughout his career,
Adam has used FPGAs to implement
a wide variety of solutions from RADAR
to safety critical control systems (SIL4)
and satellite systems. He also had
interesting stops in image processing
and cryptography along the way.
Adam is Chartered Engineer, Senior
Member of the IEEE, Fellow of the Institute
of Engineering and Technology, Arm
Innovator, Edge Impulse Ambassador.
He is also the owner of the engineering
and consultancy company Adiuvo
Engineering and Training which develops
embedded solutions for high reliability,
mission critical and space applications.
Current projects include ESA Plato,
Lunar Gateway, Generic Space Imager,
UKSA TreeView and several other clients
across the world.
FPGAs are Adam‘s first love, he is the
author of numerous articles and papers
on electronic design and FPGA design
including over 440 blogs and 30 million
plus views on how to use the Zynq and
Zynq MPSoC for Xilinx.